C01 | Evaluating Mixed Reality Experiences

Prof. Harald Reiterer, University of Konstanz
Email | Website

Harald Reiterer

Prof. Marc Ernst, Ulm University
Email | Website

Marc Ernst

Dr. Johannes Zagermann, University of Konstanz – Email | Website

Sebastian Hubenschmid, University of Konstanz – Email | Website

Jonathan Wieland, University of Konstanz – Email | Website

The main goal of this project is to understand and measure interaction in visual computing systems. While the first funding period (2015-2019) focussed on evaluation of 2D environments (e.g., cross-device interaction), we extended our scope for the current second funding period (2019-2023) to also include evaluation in 3D environments (e.g., mixed reality or hybrid user interfaces). We place particular emphasis on establishing holistic evaluation standards for user studies in the lab and in the wild by 1) combining quantitative and qualitative methods, 2) including collaborative interaction scenarios, and 3) embedding interaction in immersive environments.

Fig. 1: Mixed-immersion evaluation framework ReLive.

Research Questions

How can we find good trade-offs between quantitative and qualitative measurements?

To what extend can lab-based metrics be applied to in the wild experiments?

What kind of metrics and tasks are more suitable for domain-experts than for novices?

What kind of metrics and tasks can help us to describe collaborative activities (e.g., coupling styles)?

How can we measure awareness about group partner’s activities across space and time?

How can immersive environments bridge the gaps between co-located and remote collaboration?

Publications

  1. S. Hubenschmid, D. I. Fink, J. Zagermann, J. Wieland, H. Reiterer, and T. Feuchtner, “Colibri: A Toolkit for Rapid Prototyping of Networking Across Realities,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 2023, pp. 9–13. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10322249
  2. J. Wieland, “Designing and Evaluating Interactions for Handheld AR,” in Companion Proceedings of the 2023 Conference on Interactive Surfaces and Spaces, in Companion Proceedings of the 2023 Conference on Interactive Surfaces and Spaces. New York, NY, USA: Association for Computing Machinery, 2023, pp. 100–103. doi: 10.1145/3626485.3626555.
  3. T. Kosch, J. Karolus, J. Zagermann, H. Reiterer, A. Schmidt, and P. W. Woźniak, “A Survey on Measuring Cognitive Workload in Human-Computer Interaction,” ACM Comput. Surv., Jan. 2023, doi: 10.1145/3582272.
  4. S. Hubenschmid, J. Zagermann, D. Leicht, H. Reiterer, and T. Feuchtner, “ARound the Smartphone: Investigating the Effects of Virtually-Extended Display Size on Spatial Memory,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). New York, NY, USA: ACM, 2023. [Online]. Available: https://kops.uni-konstanz.de/server/api/core/bitstreams/6eecac2f-666f-4399-bec3-d8e607331164/content
  5. A. Zaky, J. Zagermann, H. Reiterer, and T. Feuchtner, “Opportunities and Challenges of Hybrid User Interfaces for Optimization of Mixed Reality Interfaces,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 2023, pp. 215–219. [Online]. Available: https://ieeexplore.ieee.org/document/10322176
  6. J. Zagermann, S. Hubenschmid, D. I. Fink, J. Wieland, H. Reiterer, and T. Feuchtner, “Challenges and Opportunities for Collaborative Immersive Analytics with Hybrid User Interfaces,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). Los Alamitos, CA, USA: IEEE Computer Society, Oct. 2023, pp. 191–195. doi: 10.1109/ISMAR-Adjunct60411.2023.00044.
  7. A. V. Reinschluessel and J. Zagermann, “Exploring Hybrid User Interfaces for Surgery Planning,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 2023, pp. 208–210. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10322244
  8. M. Jenadeleh, J. Zagermann, H. Reiterer, U.-D. Reips, R. Hamzaoui, and D. Saupe, “Relaxed forced choice improves performance of visual quality assessment methods,” in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX), in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX). 2023, pp. 37–42. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10178467
  9. D. I. Fink, J. Zagermann, H. Reiterer, and H.-C. Jetter, “Re-Locations: Augmenting Personal and Shared Workspaces to Support Remote Collaboration in Incongruent Spaces,” Proc. ACM Hum.-Comput. Interact., vol. 6, Nov. 2022, doi: 10.1145/3567709.
  10. P. Fleck, A. Sousa Calepso, S. Hubenschmid, M. Sedlmair, and D. Schmalstieg, “RagRug: A Toolkit for Situated Analytics,” IEEE Transactions on Visualization and Computer Graphics, 2022, [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/35254986/
  11. S. Hubenschmid et al., “ReLive: Bridging In-Situ and Ex-Situ Visual Analytics for Analyzing Mixed Reality User Studies,” in CHI Conference on Human Factors in Computing Systems (CHI ’22), in CHI Conference on Human Factors in Computing Systems (CHI ’22). New York, NY: ACM, 2022, pp. 1–20. doi: 10.1145/3491102.3517550.
  12. J. Zagermann et al., “Complementary Interfaces for Visual Computing,” it - Information Technology, vol. 64, pp. 145–154, 2022, doi: 10.1515/itit-2022-0031.
  13. F. Chiossi et al., “Adapting visualizations and interfaces to the user,” it - Information Technology, vol. 64, pp. 133–143, 2022, doi: 10.1515/itit-2022-0035.
  14. K. Vock, S. Hubenschmid, J. Zagermann, S. Butscher, and H. Reiterer, “IDIAR : Augmented Reality Dashboards to Supervise Mobile Intervention Studies,” in Mensch und Computer 2021 (MuC ’21), in Mensch und Computer 2021 (MuC ’21). New York, NY: ACM, 2021. [Online]. Available: http://nbn-resolving.de/urn:nbn:de:bsz:352-2-22ydtfzvxx3l1
  15. S. Hubenschmid, J. Zagermann, D. I. Fink, J. Wieland, T. Feuchtner, and H. Reiterer, “Towards Asynchronous Hybrid User Interfaces for Cross-Reality Interaction,” in ISS’21 Workshop Proceedings: “Transitional Interfaces in Mixed and Cross-Reality: A new frontier?,” H.-C. Jetter, J.-H. Schröder, J. Gugenheimer, M. Billinghurst, C. Anthes, M. Khamis, and T. Feuchtner, Eds., in ISS’21 Workshop Proceedings: “Transitional Interfaces in Mixed and Cross-Reality: A new frontier?” 2021. [Online]. Available: https://kops.uni-konstanz.de/bitstream/handle/123456789/55453/Hubenschmid_2-84mm0sggczq02.pdf?sequence=1&isAllowed=y
  16. J. Wieland, J. Zagermann, J. Müller, and H. Reiterer, “Separation, Composition, or Hybrid? : Comparing Collaborative 3D Object Manipulation Techniques for Handheld Augmented Reality,” in 2021 IEEE International Symposium on Mixed and Augmented Reality, in 2021 IEEE International Symposium on Mixed and Augmented Reality. Piscataway, NJ: IEEE, 2021, pp. 403–412. [Online]. Available: http://nbn-resolving.de/urn:nbn:de:bsz:352-2-ahkg9sntr33e8
  17. S. Hubenschmid, J. Zagermann, S. Butscher, and H. Reiterer, “STREAM: Exploring the Combination of Spatially-Aware Tablets with Augmented Reality Head-Mounted Displays for Immersive Analytics,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. , New York, NY, USA: Association for Computing Machinery, 2021. doi: 10.1145/3411764.3445298.
  18. J. Zagermann, U. Pfeil, P. von Bauer, D. I. Fink, and H. Reiterer, “‘It’s in my other hand!’: Studying the Interplay of Interaction Techniques and Multi-Tablet Activities,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, in Proceedings of the CHI Conference on Human Factors in Computing Systems. 2020, pp. 413:1-413:13. doi: 10.1145/3313831.3376540.
  19. F. Bishop, J. Zagermann, U. Pfeil, G. Sanderson, H. Reiterer, and U. Hinrichs, “Construct-A-Vis: Exploring the Free-Form Visualization Processes of Children,” IEEE Transactions on Visualization and Computer Graphics, vol. 26, no. 1, Art. no. 1, 2020, [Online]. Available: https://ieeexplore.ieee.org/document/8807271
  20. M. Borowski, J. Zagermann, C. N. Klokmose, H. Reiterer, and R. Rädle, “Exploring the Benefits and Barriers of Using Computational Notebooks for Collaborative Programming Assignments,” in Proceedings of the ACM Technical Symposium on Computer Science Education (SIGCSE), in Proceedings of the ACM Technical Symposium on Computer Science Education (SIGCSE). 2020, pp. 468–474. doi: 10.1145/3328778.3366887.
  21. J. Müller, J. Zagermann, J. Wieland, U. Pfeil, and H. Reiterer, “A Qualitative Comparison Between Augmented and Virtual Reality Collaboration with Handheld Devices,” in Mensch und Computer 2019 – Tagungsband (MuC), F. Alt, A. Bulling, and T. Döring, Eds., in Mensch und Computer 2019 – Tagungsband (MuC). GI, ACM, 2019, pp. 399–410. doi: 10.1145/3340764.3340773.
  22. M. Blumenschein et al., “SMARTexplore: Simplifying High-Dimensional Data Analysis through a Table-Based Visual Analytics Approach,” in Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST), R. Chang, H. Qu, and T. Schreck, Eds., in Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, 2018, pp. 36–47. [Online]. Available: https://ieeexplore.ieee.org/document/8802486
  23. S. Hubenschmid, J. Zagermann, S. Butscher, and H. Reiterer, “Employing Tangible Visualisations in Augmented Reality with Mobile Devices,” in Proceedings of the Working Conference on Advanced Visual Interfaces (AVI), in Proceedings of the Working Conference on Advanced Visual Interfaces (AVI). 2018, pp. 1–4. [Online]. Available: http://nbn-resolving.de/urn:nbn:de:bsz:352-2-1iooenfo4fofm8
  24. J. Zagermann, U. Pfeil, and H. Reiterer, “Studying Eye Movements as a Basis for Measuring Cognitive Load,” Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA), p. LBW095:1-LBW095:6, 2018, doi: 10.1145/3170427.3188628.
  25. L. L. Chuang and U. Pfeil, “Transparency and Openness Promotion Guidelines for HCI,” in Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA), R. L. Mandryk, M. Hancock, M. Perry, and A. L. Cox, Eds., in Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA). ACM, 2018, p. SIG04:1-SIG04:4. doi: 10.1145/3170427.3185377.
  26. D. Jäckle, F. Stoffel, S. Mittelstädt, D. A. Keim, and H. Reiterer, “Interpretation of Dimensionally-Reduced Crime Data: A Study with Untrained Domain Experts,” in Proceedings of the Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP), in Proceedings of the Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP), vol. 3. 2017, pp. 164–175. [Online]. Available: https://bib.dbvis.de/publications/details/697
  27. J. Zagermann, U. Pfeil, D. I. Fink, P. von Bauer, and H. Reiterer, “Memory in Motion: The Influence of Gesture- and Touch-based Input Modalities on Spatial Memory,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, G. Mark, S. R. Fussell, C. Lampe, m. c. schraefel, J. P. Hourcade, C. Appert, and D. Wigdor, Eds., in Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, 2017, pp. 1899–1910. doi: 10.1145/3025453.3026001.
  28. J. Zagermann, U. Pfeil, C. Acevedo, and H. Reiterer, “Studying the Benefits and Challenges of Spatial Distribution and Physical Affordances in a Multi-device Workspace,” in Proceedings of the International Conference on Mobile and Ubiquitous Multimedia (MUM), in Proceedings of the International Conference on Mobile and Ubiquitous Multimedia (MUM). 2017, pp. 249–259. doi: 10.1145/3152832.3152855.
  29. J. Zagermann, U. Pfeil, and H. Reiterer, “Measuring Cognitive Load using Eye Tracking Technology in Visual Computing,” in Proceedings of the Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization (BELIV), M. Sedlmair, P. Isenberg, T. Isenberg, N. Mahyar, and H. Lam, Eds., in Proceedings of the Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization (BELIV). ACM, 2016, pp. 78–85. doi: 10.1145/2993901.2993908.
  30. J. Müller, R. Rädle, and H. Reiterer, “Virtual Objects as Spatial Cues in Collaborative Mixed Reality Environments: How They Shape Communication Behavior and User Task Load,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, J. Kaye, A. Druin, C. Lampe, D. Morris, and J. P. Hourcade, Eds., in Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, 2016, pp. 1245–1249. doi: 10.1145/2858036.2858043.
  31. J. Zagermann, U. Pfeil, R. Rädle, H.-C. Jetter, C. N. Klokmose, and H. Reiterer, “When Tablets meet Tabletops: The Effect of Tabletop Size on Around-the-Table Collaboration with Personal Tablets,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, J. Kaye, A. Druin, C. Lampe, D. Morris, and J. P. Hourcade, Eds., in Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, 2016, pp. 5470–5481. doi: 10.1145/2858036.2858224.
  32. S. Butscher and H. Reiterer, “Applying Guidelines for the Design of Distortions on Focus+Context Interfaces,” in Proceedings of the Working Conference on Advanced Visual Interfaces (AVI), P. Buono, R. Lanzilotti, M. Matera, and M. F. Costabile, Eds., in Proceedings of the Working Conference on Advanced Visual Interfaces (AVI). ACM, 2016, pp. 244–247. doi: 10.1145/2909132.2909284.
  33. L. Lischke et al., “Using Space: Effect of Display Size on Users’ Search Performance,” in Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA), B. Begole, J. Kim, K. Inkpen, and W. Woo, Eds., in Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA). ACM, 2015, pp. 1845–1850. doi: 10.1145/2702613.2732845.

Videos (external links)

Project Group A

Models and Measures

A01 | Uncertainty Quantification and Analysis
A03 | Quantification of Visual Explainability
A07 | Visual Attention Modeling
A08 | Learning and Explaining Dimensionality Reduction
A09 | Scalability and Complexity in Immersive Analytics

 

Completed

A02 | Quantifying Visual Computing Systems
A04 | Quantitative Models for Visual Abstraction
A05 | Visual Quality Assessment
A06 | Computational Photography

 

Project Group B

Adaptive Algorithms

B01 | Adaptive Self-Consistent Visualization
B04 | Motion Estimation
B06 | Adaptive Algorithms for Graph Views and View Transitions

 

Completed

B02 | Adaptive Network Visualization
B05 | Large Scale Variational 3D Reconstruction
B07 | Computational Uncertainty Quantification

 

Project Group C

Interaction

C01 | Evaluating Mixed Reality Experiences
C05 | Human-Machine Interaction with Adaptive Multisensory Systems
C06 | User-Adaptive Mixed Reality
C07 | Optimization for Dynamic XR User Interfaces

 

Completed

C02 | Physiologically Based Interaction
C03 | Immersive Virtual Environments
C04 | Mobile Visualization and Interaction
Quantifying interactions with adaptable multisensory systems

 

Project Group D

Applications

D02 | Visual Analytics in Linguistics
D04 | Immersive Analytics for the Life Sciences
D05 | Visual Computing for ERP-based Brain Activity

 

Completed

D01 | Modeling of 3D Virtual Cities
D03 | Exploration and Analysis of Provenance Data

 

Central Services

INF | Collaboration Infrastructure
Ö | Public Relations

FOR SCIENTISTS

Projects
People
Publications
Graduate School
Equal Opportunity

FOR PUPILS

BOGY
Events for the Public

PRESS AND MEDIA

Press releases
Images

SOCIAL MEDIA

© SFB-TRR 161 | Quantitative Methods for Visual Computing | 2019.