C01 | Evaluating Mixed Reality Experiences

Prof. Harald Reiterer, University of Konstanz
Email | Website

Harald Reiterer

Prof. Marc Ernst, Ulm University
Email | Website

Marc Ernst

Project C01

Dr. Johannes Zagermann, University of Konstanz – Email | Website

Sebastian Hubenschmid, University of Konstanz – Email | Website

Jonathan Wieland, University of Konstanz – Email | Website

The main goal of this project is to understand and measure interaction in visual computing systems. While the first funding period (2015-2019) focussed on evaluation of 2D environments (e.g., cross-device interaction), we extended our scope for the current second funding period (2019-2023) to also include evaluation in 3D environments (e.g., mixed reality or hybrid user interfaces). We place particular emphasis on establishing holistic evaluation standards for user studies in the lab and in the wild by 1) combining quantitative and qualitative methods, 2) including collaborative interaction scenarios, and 3) embedding interaction in immersive environments.

Fig. 1: Mixed-immersion evaluation framework ReLive.

Research Questions

How can we find good trade-offs between quantitative and qualitative measurements?

To what extend can lab-based metrics be applied to in the wild experiments?

What kind of metrics and tasks are more suitable for domain-experts than for novices?

What kind of metrics and tasks can help us to describe collaborative activities (e.g., coupling styles)?

How can we measure awareness about group partner’s activities across space and time?

How can immersive environments bridge the gaps between co-located and remote collaboration?

Publications

  1. S. Hubenschmid, D. I. Fink, J. Zagermann, J. Wieland, H. Reiterer, and T. Feuchtner, “Colibri: A Toolkit for Rapid Prototyping of Networking Across Realities,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 2023, pp. 9–13. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10322249
  2. J. Zagermann, S. Hubenschmid, D. I. Fink, J. Wieland, H. Reiterer, and T. Feuchtner, “Challenges and Opportunities for Collaborative Immersive Analytics with Hybrid User Interfaces,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). Los Alamitos, CA, USA: IEEE Computer Society, Oct. 2023, pp. 191–195. doi: 10.1109/ISMAR-Adjunct60411.2023.00044.
  3. J. Wieland, “Designing and Evaluating Interactions for Handheld AR,” in Companion Proceedings of the 2023 Conference on Interactive Surfaces and Spaces, in Companion Proceedings of the 2023 Conference on Interactive Surfaces and Spaces. New York, NY, USA: Association for Computing Machinery, 2023, pp. 100–103. doi: 10.1145/3626485.3626555.
  4. A. V. Reinschluessel and J. Zagermann, “Exploring Hybrid User Interfaces for Surgery Planning,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 2023, pp. 208–210. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10322244
  5. A. Zaky, J. Zagermann, H. Reiterer, and T. Feuchtner, “Opportunities and Challenges of Hybrid User Interfaces for Optimization of Mixed Reality Interfaces,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 2023, pp. 215–219. [Online]. Available: https://ieeexplore.ieee.org/document/10322176
  6. S. Hubenschmid, J. Zagermann, D. Leicht, H. Reiterer, and T. Feuchtner, “ARound the Smartphone: Investigating the Effects of Virtually-Extended Display Size on Spatial Memory,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). New York, NY, USA: ACM, 2023. [Online]. Available: https://kops.uni-konstanz.de/server/api/core/bitstreams/6eecac2f-666f-4399-bec3-d8e607331164/content
  7. M. Jenadeleh, J. Zagermann, H. Reiterer, U.-D. Reips, R. Hamzaoui, and D. Saupe, “Relaxed forced choice improves performance of visual quality assessment methods,” in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX), in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX). 2023, pp. 37–42. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10178467
  8. T. Kosch, J. Karolus, J. Zagermann, H. Reiterer, A. Schmidt, and P. W. Woźniak, “A Survey on Measuring Cognitive Workload in Human-Computer Interaction,” ACM Comput. Surv., Jan. 2023, doi: 10.1145/3582272.
  9. D. I. Fink, J. Zagermann, H. Reiterer, and H.-C. Jetter, “Re-Locations: Augmenting Personal and Shared Workspaces to Support Remote Collaboration in Incongruent Spaces,” Proc. ACM Hum.-Comput. Interact., vol. 6, Nov. 2022, doi: 10.1145/3567709.
  10. P. Fleck, A. Sousa Calepso, S. Hubenschmid, M. Sedlmair, and D. Schmalstieg, “RagRug: A Toolkit for Situated Analytics,” IEEE Transactions on Visualization and Computer Graphics, 2022, [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/35254986/
  11. S. Hubenschmid et al., “ReLive: Bridging In-Situ and Ex-Situ Visual Analytics for Analyzing Mixed Reality User Studies,” in CHI Conference on Human Factors in Computing Systems (CHI ’22), in CHI Conference on Human Factors in Computing Systems (CHI ’22). New York, NY: ACM, 2022, pp. 1–20. doi: 10.1145/3491102.3517550.
  12. F. Chiossi et al., “Adapting visualizations and interfaces to the user,” it - Information Technology, vol. 64, pp. 133–143, 2022, doi: 10.1515/itit-2022-0035.
  13. J. Zagermann et al., “Complementary Interfaces for Visual Computing,” it - Information Technology, vol. 64, pp. 145–154, 2022, doi: 10.1515/itit-2022-0031.
  14. K. Vock, S. Hubenschmid, J. Zagermann, S. Butscher, and H. Reiterer, “IDIAR : Augmented Reality Dashboards to Supervise Mobile Intervention Studies,” in Mensch und Computer 2021 (MuC ’21), in Mensch und Computer 2021 (MuC ’21). New York, NY: ACM, 2021. [Online]. Available: http://nbn-resolving.de/urn:nbn:de:bsz:352-2-22ydtfzvxx3l1
  15. J. Wieland, J. Zagermann, J. Müller, and H. Reiterer, “Separation, Composition, or Hybrid? : Comparing Collaborative 3D Object Manipulation Techniques for Handheld Augmented Reality,” in 2021 IEEE International Symposium on Mixed and Augmented Reality, in 2021 IEEE International Symposium on Mixed and Augmented Reality. Piscataway, NJ: IEEE, 2021, pp. 403–412. [Online]. Available: http://nbn-resolving.de/urn:nbn:de:bsz:352-2-ahkg9sntr33e8
  16. S. Hubenschmid, J. Zagermann, D. I. Fink, J. Wieland, T. Feuchtner, and H. Reiterer, “Towards Asynchronous Hybrid User Interfaces for Cross-Reality Interaction,” in ISS’21 Workshop Proceedings: “Transitional Interfaces in Mixed and Cross-Reality: A new frontier?,” H.-C. Jetter, J.-H. Schröder, J. Gugenheimer, M. Billinghurst, C. Anthes, M. Khamis, and T. Feuchtner, Eds., in ISS’21 Workshop Proceedings: “Transitional Interfaces in Mixed and Cross-Reality: A new frontier?” 2021. [Online]. Available: https://kops.uni-konstanz.de/bitstream/handle/123456789/55453/Hubenschmid_2-84mm0sggczq02.pdf?sequence=1&isAllowed=y
  17. S. Hubenschmid, J. Zagermann, S. Butscher, and H. Reiterer, “STREAM: Exploring the Combination of Spatially-Aware Tablets with Augmented Reality Head-Mounted Displays for Immersive Analytics,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. , New York, NY, USA: Association for Computing Machinery, 2021. doi: 10.1145/3411764.3445298.
  18. M. Borowski, J. Zagermann, C. N. Klokmose, H. Reiterer, and R. Rädle, “Exploring the Benefits and Barriers of Using Computational Notebooks for Collaborative Programming Assignments,” in Proceedings of the ACM Technical Symposium on Computer Science Education (SIGCSE), in Proceedings of the ACM Technical Symposium on Computer Science Education (SIGCSE). 2020, pp. 468–474. doi: 10.1145/3328778.3366887.
  19. F. Bishop, J. Zagermann, U. Pfeil, G. Sanderson, H. Reiterer, and U. Hinrichs, “Construct-A-Vis: Exploring the Free-Form Visualization Processes of Children,” IEEE Transactions on Visualization and Computer Graphics, vol. 26, no. 1, Art. no. 1, 2020, [Online]. Available: https://ieeexplore.ieee.org/document/8807271
  20. J. Zagermann, U. Pfeil, P. von Bauer, D. I. Fink, and H. Reiterer, “‘It’s in my other hand!’: Studying the Interplay of Interaction Techniques and Multi-Tablet Activities,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, in Proceedings of the CHI Conference on Human Factors in Computing Systems. 2020, pp. 413:1-413:13. doi: 10.1145/3313831.3376540.
  21. J. Müller, J. Zagermann, J. Wieland, U. Pfeil, and H. Reiterer, “A Qualitative Comparison Between Augmented and Virtual Reality Collaboration with Handheld Devices,” in Mensch und Computer 2019 – Tagungsband (MuC), F. Alt, A. Bulling, and T. Döring, Eds., in Mensch und Computer 2019 – Tagungsband (MuC). GI, ACM, 2019, pp. 399–410. doi: 10.1145/3340764.3340773.
  22. S. Hubenschmid, J. Zagermann, S. Butscher, and H. Reiterer, “Employing Tangible Visualisations in Augmented Reality with Mobile Devices,” in Proceedings of the Working Conference on Advanced Visual Interfaces (AVI), in Proceedings of the Working Conference on Advanced Visual Interfaces (AVI). 2018, pp. 1–4. [Online]. Available: http://nbn-resolving.de/urn:nbn:de:bsz:352-2-1iooenfo4fofm8
  23. J. Zagermann, U. Pfeil, and H. Reiterer, “Studying Eye Movements as a Basis for Measuring Cognitive Load,” Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA), p. LBW095:1-LBW095:6, 2018, doi: 10.1145/3170427.3188628.
  24. M. Blumenschein et al., “SMARTexplore: Simplifying High-Dimensional Data Analysis through a Table-Based Visual Analytics Approach,” in Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST), R. Chang, H. Qu, and T. Schreck, Eds., in Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, 2018, pp. 36–47. [Online]. Available: https://ieeexplore.ieee.org/document/8802486
  25. L. L. Chuang and U. Pfeil, “Transparency and Openness Promotion Guidelines for HCI,” in Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA), R. L. Mandryk, M. Hancock, M. Perry, and A. L. Cox, Eds., in Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA). ACM, 2018, p. SIG04:1-SIG04:4. doi: 10.1145/3170427.3185377.
  26. D. Jäckle, F. Stoffel, S. Mittelstädt, D. A. Keim, and H. Reiterer, “Interpretation of Dimensionally-Reduced Crime Data: A Study with Untrained Domain Experts,” in Proceedings of the Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP), in Proceedings of the Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP), vol. 3. 2017, pp. 164–175. [Online]. Available: https://bib.dbvis.de/publications/details/697
  27. J. Zagermann, U. Pfeil, D. I. Fink, P. von Bauer, and H. Reiterer, “Memory in Motion: The Influence of Gesture- and Touch-based Input Modalities on Spatial Memory,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, G. Mark, S. R. Fussell, C. Lampe, m. c. schraefel, J. P. Hourcade, C. Appert, and D. Wigdor, Eds., in Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, 2017, pp. 1899–1910. doi: 10.1145/3025453.3026001.
  28. J. Zagermann, U. Pfeil, C. Acevedo, and H. Reiterer, “Studying the Benefits and Challenges of Spatial Distribution and Physical Affordances in a Multi-device Workspace,” in Proceedings of the International Conference on Mobile and Ubiquitous Multimedia (MUM), in Proceedings of the International Conference on Mobile and Ubiquitous Multimedia (MUM). 2017, pp. 249–259. doi: 10.1145/3152832.3152855.
  29. S. Butscher and H. Reiterer, “Applying Guidelines for the Design of Distortions on Focus+Context Interfaces,” in Proceedings of the Working Conference on Advanced Visual Interfaces (AVI), P. Buono, R. Lanzilotti, M. Matera, and M. F. Costabile, Eds., in Proceedings of the Working Conference on Advanced Visual Interfaces (AVI). ACM, 2016, pp. 244–247. doi: 10.1145/2909132.2909284.
  30. J. Müller, R. Rädle, and H. Reiterer, “Virtual Objects as Spatial Cues in Collaborative Mixed Reality Environments: How They Shape Communication Behavior and User Task Load,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, J. Kaye, A. Druin, C. Lampe, D. Morris, and J. P. Hourcade, Eds., in Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, 2016, pp. 1245–1249. doi: 10.1145/2858036.2858043.
  31. J. Zagermann, U. Pfeil, and H. Reiterer, “Measuring Cognitive Load using Eye Tracking Technology in Visual Computing,” in Proceedings of the Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization (BELIV), M. Sedlmair, P. Isenberg, T. Isenberg, N. Mahyar, and H. Lam, Eds., in Proceedings of the Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization (BELIV). ACM, 2016, pp. 78–85. doi: 10.1145/2993901.2993908.
  32. J. Zagermann, U. Pfeil, R. Rädle, H.-C. Jetter, C. N. Klokmose, and H. Reiterer, “When Tablets meet Tabletops: The Effect of Tabletop Size on Around-the-Table Collaboration with Personal Tablets,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, J. Kaye, A. Druin, C. Lampe, D. Morris, and J. P. Hourcade, Eds., in Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, 2016, pp. 5470–5481. doi: 10.1145/2858036.2858224.
  33. L. Lischke et al., “Using Space: Effect of Display Size on Users’ Search Performance,” in Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA), B. Begole, J. Kim, K. Inkpen, and W. Woo, Eds., in Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA). ACM, 2015, pp. 1845–1850. doi: 10.1145/2702613.2732845.

Project Group A

Models and Measures

 

Completed

 

Project Group B

Adaptive Algorithms

 

Completed

 

Project Group C

Interaction

 

Completed

 

Project Group D

Applications

 

Completed