C01 | Evaluating Mixed Reality Experiences
The main goal of this project is to understand and measure interaction in visual computing systems. While the first funding period (2015-2019) focussed on evaluation of 2D environments (e.g., cross-device interaction), we extended our scope for the current second funding period (2019-2023) to also include evaluation in 3D environments (e.g., mixed reality or hybrid user interfaces). We place particular emphasis on establishing holistic evaluation standards for user studies in the lab and in the wild by 1) combining quantitative and qualitative methods, 2) including collaborative interaction scenarios, and 3) embedding interaction in immersive environments.
Fig. 1: Mixed-immersion evaluation framework ReLive.
Research Questions
How can we find good trade-offs between quantitative and qualitative measurements?
To what extend can lab-based metrics be applied to in the wild experiments?
What kind of metrics and tasks are more suitable for domain-experts than for novices?
What kind of metrics and tasks can help us to describe collaborative activities (e.g., coupling styles)?
How can we measure awareness about group partner’s activities across space and time?
How can immersive environments bridge the gaps between co-located and remote collaboration?
Publications
- L. Joos et al., “Evaluating Node Selection Techniques for Network Visualizations in Virtual Reality,” in ACM Symposium on Spatial User Interaction, New York, NY, USA: ACM, 2024, pp. 1–11. doi: 10.1145/3677386.3682102.
- A. Zaky, J. Zagermann, H. Reiterer, and T. Feuchtner, “Opportunities and Challenges of Hybrid User Interfaces for Optimization of Mixed Reality Interfaces,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2023, pp. 215–219. [Online]. Available: https://ieeexplore.ieee.org/document/10322176
- J. Wieland, “Designing and Evaluating Interactions for Handheld AR,” in Companion Proceedings of the 2023 Conference on Interactive Surfaces and Spaces, in ISS Companion ’23. New York, NY, USA: Association for Computing Machinery, 2023, pp. 100–103. doi: 10.1145/3626485.3626555.
- S. Hubenschmid, J. Zagermann, D. Leicht, H. Reiterer, and T. Feuchtner, “ARound the Smartphone: Investigating the Effects of Virtually-Extended Display Size on Spatial Memory,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), New York, NY, USA: ACM, 2023. [Online]. Available: https://kops.uni-konstanz.de/server/api/core/bitstreams/6eecac2f-666f-4399-bec3-d8e607331164/content
- T. Kosch, J. Karolus, J. Zagermann, H. Reiterer, A. Schmidt, and P. W. Woźniak, “A Survey on Measuring Cognitive Workload in Human-Computer Interaction,” ACM Comput. Surv., Jan. 2023, doi: 10.1145/3582272.
- S. Hubenschmid, D. I. Fink, J. Zagermann, J. Wieland, H. Reiterer, and T. Feuchtner, “Colibri: A Toolkit for Rapid Prototyping of Networking Across Realities,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2023, pp. 9–13. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10322249
- A. V. Reinschluessel and J. Zagermann, “Exploring Hybrid User Interfaces for Surgery Planning,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2023, pp. 208–210. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10322244
- M. Jenadeleh, J. Zagermann, H. Reiterer, U.-D. Reips, R. Hamzaoui, and D. Saupe, “Relaxed forced choice improves performance of visual quality assessment methods,” in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX), 2023, pp. 37–42. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10178467
- J. Zagermann, S. Hubenschmid, D. I. Fink, J. Wieland, H. Reiterer, and T. Feuchtner, “Challenges and Opportunities for Collaborative Immersive Analytics with Hybrid User Interfaces,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Los Alamitos, CA, USA: IEEE Computer Society, Oct. 2023, pp. 191–195. doi: 10.1109/ISMAR-Adjunct60411.2023.00044.
- D. I. Fink, J. Zagermann, H. Reiterer, and H.-C. Jetter, “Re-Locations: Augmenting Personal and Shared Workspaces to Support Remote Collaboration in Incongruent Spaces,” Proc. ACM Hum.-Comput. Interact., vol. 6, Nov. 2022, doi: 10.1145/3567709.
- J. Zagermann et al., “Complementary Interfaces for Visual Computing,” it - Information Technology, vol. 64, pp. 145–154, 2022, doi: 10.1515/itit-2022-0031.
- S. Hubenschmid et al., “ReLive: Bridging In-Situ and Ex-Situ Visual Analytics for Analyzing Mixed Reality User Studies,” in CHI Conference on Human Factors in Computing Systems (CHI ’22), New York, NY: ACM, 2022, pp. 1–20. doi: 10.1145/3491102.3517550.
- P. Fleck, A. Sousa Calepso, S. Hubenschmid, M. Sedlmair, and D. Schmalstieg, “RagRug: A Toolkit for Situated Analytics,” IEEE Transactions on Visualization and Computer Graphics, 2022, [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/35254986/
- F. Chiossi et al., “Adapting visualizations and interfaces to the user,” it - Information Technology, vol. 64, pp. 133–143, 2022, doi: 10.1515/itit-2022-0035.
- J. Wieland, J. Zagermann, J. Müller, and H. Reiterer, “Separation, Composition, or Hybrid? : Comparing Collaborative 3D Object Manipulation Techniques for Handheld Augmented Reality,” in 2021 IEEE International Symposium on Mixed and Augmented Reality, Piscataway, NJ: IEEE, 2021, pp. 403–412. [Online]. Available: http://nbn-resolving.de/urn:nbn:de:bsz:352-2-ahkg9sntr33e8
- S. Hubenschmid, J. Zagermann, S. Butscher, and H. Reiterer, “STREAM: Exploring the Combination of Spatially-Aware Tablets with Augmented Reality Head-Mounted Displays for Immersive Analytics,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, New York, NY, USA: Association for Computing Machinery, 2021. doi: 10.1145/3411764.3445298.
- S. Hubenschmid, J. Zagermann, D. I. Fink, J. Wieland, T. Feuchtner, and H. Reiterer, “Towards Asynchronous Hybrid User Interfaces for Cross-Reality Interaction,” in ISS′21 Workshop Proceedings: “Transitional Interfaces in Mixed and Cross-Reality: A new frontier?”, H.-C. Jetter, J.-H. Schröder, J. Gugenheimer, M. Billinghurst, C. Anthes, M. Khamis, and T. Feuchtner, Eds., 2021. [Online]. Available: https://kops.uni-konstanz.de/bitstream/handle/123456789/55453/Hubenschmid_2-84mm0sggczq02.pdf?sequence=1&isAllowed=y
- K. Vock, S. Hubenschmid, J. Zagermann, S. Butscher, and H. Reiterer, “IDIAR : Augmented Reality Dashboards to Supervise Mobile Intervention Studies,” in Mensch und Computer 2021 (MuC ’21), New York, NY: ACM, 2021. [Online]. Available: http://nbn-resolving.de/urn:nbn:de:bsz:352-2-22ydtfzvxx3l1
- J. Zagermann, U. Pfeil, P. von Bauer, D. I. Fink, and H. Reiterer, ““It’s in my other hand!”: Studying the Interplay of Interaction Techniques and Multi-Tablet Activities,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, 2020, pp. 413:1–413:13. doi: 10.1145/3313831.3376540.
- F. Bishop, J. Zagermann, U. Pfeil, G. Sanderson, H. Reiterer, and U. Hinrichs, “Construct-A-Vis: Exploring the Free-Form Visualization Processes of Children,” IEEE Transactions on Visualization and Computer Graphics, vol. 26, Art. no. 1, 2020, [Online]. Available: https://ieeexplore.ieee.org/document/8807271
- M. Borowski, J. Zagermann, C. N. Klokmose, H. Reiterer, and R. Rädle, “Exploring the Benefits and Barriers of Using Computational Notebooks for Collaborative Programming Assignments,” in Proceedings of the ACM Technical Symposium on Computer Science Education (SIGCSE), 2020, pp. 468–474. doi: 10.1145/3328778.3366887.
- J. Müller, J. Zagermann, J. Wieland, U. Pfeil, and H. Reiterer, “A Qualitative Comparison Between Augmented and Virtual Reality Collaboration with Handheld Devices,” in Mensch und Computer 2019 – Tagungsband (MuC), F. Alt, A. Bulling, and T. Döring, Eds., GI, ACM, 2019, pp. 399–410. doi: 10.1145/3340764.3340773.
- S. Hubenschmid, J. Zagermann, S. Butscher, and H. Reiterer, “Employing Tangible Visualisations in Augmented Reality with Mobile Devices,” in Proceedings of the Working Conference on Advanced Visual Interfaces (AVI), 2018, pp. 1–4. [Online]. Available: http://nbn-resolving.de/urn:nbn:de:bsz:352-2-1iooenfo4fofm8
- M. Blumenschein et al., “SMARTexplore: Simplifying High-Dimensional Data Analysis through a Table-Based Visual Analytics Approach,” in Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST), R. Chang, H. Qu, and T. Schreck, Eds., IEEE, 2018, pp. 36–47. [Online]. Available: https://ieeexplore.ieee.org/document/8802486
- L. L. Chuang and U. Pfeil, “Transparency and Openness Promotion Guidelines for HCI,” in Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA), R. L. Mandryk, M. Hancock, M. Perry, and A. L. Cox, Eds., ACM, 2018, pp. SIG04:1–SIG04:4. doi: 10.1145/3170427.3185377.
- J. Zagermann, U. Pfeil, and H. Reiterer, “Studying Eye Movements as a Basis for Measuring Cognitive Load,” Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA), pp. LBW095:1–LBW095:6, 2018, doi: 10.1145/3170427.3188628.
- J. Zagermann, U. Pfeil, C. Acevedo, and H. Reiterer, “Studying the Benefits and Challenges of Spatial Distribution and Physical Affordances in a Multi-device Workspace,” in Proceedings of the International Conference on Mobile and Ubiquitous Multimedia (MUM), 2017, pp. 249–259. doi: 10.1145/3152832.3152855.
- J. Zagermann, U. Pfeil, D. I. Fink, P. von Bauer, and H. Reiterer, “Memory in Motion: The Influence of Gesture- and Touch-based Input Modalities on Spatial Memory,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, G. Mark, S. R. Fussell, C. Lampe, m. c. schraefel, J. P. Hourcade, C. Appert, and D. Wigdor, Eds., ACM, 2017, pp. 1899–1910. doi: 10.1145/3025453.3026001.
- D. Jäckle, F. Stoffel, S. Mittelstädt, D. A. Keim, and H. Reiterer, “Interpretation of Dimensionally-Reduced Crime Data: A Study with Untrained Domain Experts,” in Proceedings of the Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP), 2017, pp. 164–175. [Online]. Available: https://bib.dbvis.de/publications/details/697
- J. Müller, R. Rädle, and H. Reiterer, “Virtual Objects as Spatial Cues in Collaborative Mixed Reality Environments: How They Shape Communication Behavior and User Task Load,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, J. Kaye, A. Druin, C. Lampe, D. Morris, and J. P. Hourcade, Eds., ACM, 2016, pp. 1245–1249. doi: 10.1145/2858036.2858043.
- S. Butscher and H. Reiterer, “Applying Guidelines for the Design of Distortions on Focus+Context Interfaces,” in Proceedings of the Working Conference on Advanced Visual Interfaces (AVI), P. Buono, R. Lanzilotti, M. Matera, and M. F. Costabile, Eds., ACM, 2016, pp. 244–247. doi: 10.1145/2909132.2909284.
- J. Zagermann, U. Pfeil, R. Rädle, H.-C. Jetter, C. N. Klokmose, and H. Reiterer, “When Tablets meet Tabletops: The Effect of Tabletop Size on Around-the-Table Collaboration with Personal Tablets,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, J. Kaye, A. Druin, C. Lampe, D. Morris, and J. P. Hourcade, Eds., ACM, 2016, pp. 5470–5481. doi: 10.1145/2858036.2858224.
- J. Zagermann, U. Pfeil, and H. Reiterer, “Measuring Cognitive Load using Eye Tracking Technology in Visual Computing,” in Proceedings of the Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization (BELIV), M. Sedlmair, P. Isenberg, T. Isenberg, N. Mahyar, and H. Lam, Eds., ACM, 2016, pp. 78–85. doi: 10.1145/2993901.2993908.
- L. Lischke et al., “Using Space: Effect of Display Size on Users’ Search Performance,” in Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA), B. Begole, J. Kim, K. Inkpen, and W. Woo, Eds., ACM, 2015, pp. 1845–1850. doi: 10.1145/2702613.2732845.
Videos (external links)
- Relive: https://www.youtube.com/watch?v=BaNZ02QkZ_k
- Stream: https://www.youtube.com/watch?v=BtwAbytRrJQ
- Separation, Composition, or Hybrid: https://www.youtube.com/watch?v=6PxxDncTBsY
Project Group A
Models and Measures
Completed
Project Group B
Adaptive Algorithms
Completed
Project Group C
Interaction
Completed
Project Group D
Applications
Completed
FOR SCIENTISTS
Projects
People
Publications
Graduate School
Equal Opportunity
FOR PUPILS
PRESS AND MEDIA
SOCIAL MEDIA
© SFB-TRR 161 | Quantitative Methods for Visual Computing | 2019.