Project C07

C07 | Optimization for Dynamic Mixed Reality User Interfaces

Jun.- Prof. Tiare Feuchtner, University of Konstanz
Email | Website

Jun.- Prof. Sven Mayer, LMU Munich
Email | Website

Abdelrahman Zaky, University of Konstanz – Email | Website

Dr. Anke Reinschlüssel, University of Konstanz – Email | Website

About this project
Results

In this project we aim to dynamically adapt the user interface during interaction with Cross-Reality (XR) applications through head-mounted displays (HMDs), to improve usability and ensure the users safety and comfort.

XR refers to any systems that immerse the user in an interactive virtual or virtually augmented environment, such as Virtual Reality (VR) or Augmented Reality (AR). The virtual content is often placed in the space around the user’s body and can be directly manipulated with hands or controllers. This shift of virtual content from the 2D space of the computer screen to the 3D environment brings along a myriad of new challenges in user interface (UI) and interaction design, due to the complex and dynamic context of interaction. Contexts to consider include the user, the physical environment, and the activity. For example, an AR application presenting a virtual button in mid-air in front of the user may occlude their view of their conversation partner, lead them to knock over the glass of water on the table when trying to reach it, and cause muscle fatigue if interacted with continuously.

We aim to address these challenges by optimizing 3D UIs with regards to the placement of information and interactive elements, as well as the employed interaction techniques and feedback, depending on the context of interaction.

Research Questions

What UI characteristics can and should be adapted and how (e.g., placement, appearance, interaction technique, user representation, immersion, physicality)?

What are meaningful characteristics of the interaction context and how can they be quantified to facilitate real-time computation of optimization functions?

How can we handle conflicting optimization objectives and make their resolution apparent to the designer or user?

How much customizability does the user need during interaction, ranging from pre-defined modes to a fully adaptive system that learns over time?

How can we adapt the UI to effectively support collaboration and social interaction between multiple users?

What are meaningful and measurable success criteria for validating optimization results in user studies?

Fig. 1: Common mid-air interaction with head-mounted Augmented Reality systems

Fig. 2: Illustration of ergonomic cost in the user’s interaction space

Publications

A. V. Reinschluessel and J. Zagermann, “Exploring Hybrid User Interfaces for Surgery Planning,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 2023, pp. 208–210. doi: 10.1109/ISMAR-Adjunct60411.2023.00048.
Abstract
Hybrid user interfaces are a great opportunity to combine complementary interfaces to make use of the best interface for specific steps in a workflow. This position paper outlines one diverse application field: surgery planning. Planning a surgery is a complex task as the surgical team has to get an overview and understanding of a patient’s medical history and the internal anatomical structures of the organ or region of interest. In this position paper, we outline how different hardware (e.g., mixed reality head-worn devices and physical objects) and interaction concepts (e.g., gesture-based interaction or keyboard and mouse) can create an optimal workflow for surgery planning.
BibTeX
@inproceedings{10322244, abstract = {Hybrid user interfaces are a great opportunity to combine complementary interfaces to make use of the best interface for specific steps in a workflow. This position paper outlines one diverse application field: surgery planning. Planning a surgery is a complex task as the surgical team has to get an overview and understanding of a patient’s medical history and the internal anatomical structures of the organ or region of interest. In this position paper, we outline how different hardware (e.g., mixed reality head-worn devices and physical objects) and interaction concepts (e.g., gesture-based interaction or keyboard and mouse) can create an optimal workflow for surgery planning.}, author = {Reinschluessel, Anke V. and Zagermann, Johannes}, booktitle = {2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)}, doi = {10.1109/ISMAR-Adjunct60411.2023.00048}, pages = {208-210}, title = {Exploring Hybrid User Interfaces for Surgery Planning}, url = {https://ieeexplore.ieee.org/abstract/document/10322244}, year = 2023 }
Link
https://ieeexplore.ieee.org/abstract/document/10322244
J. Zagermann, S. Hubenschmid, D. Fink, J. Wieland, H. Reiterer, and T. Feuchtner, “Challenges and Opportunities for Collaborative Immersive Analytics with Hybrid User Interfaces,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). Los Alamitos, CA, USA: IEEE Computer Society, Oct. 2023, pp. 191–195. doi: 10.1109/ISMAR-Adjunct60411.2023.00044.
Abstract
Over the past years, we have seen an increase in the number of user studies involving mixed reality interfaces. As these environments usually exceed standardized user study settings that only measure time and error, we developed, designed, and evaluated a mixed-immersion evaluation framework called ReLive. Its combination of in-situ and ex-situ analysis approaches allows for the holistic and malleable analysis and exploration of mixed reality user study data of an individual analyst in a step-by-step approach that we previously described as an asynchronous hybrid user interface. Yet, collaboration was coined as a key aspect for visual and immersive analytics – potentially allowing multiple analysts to synchronously explore mixed reality user study data from different but complementary angles of evaluation using hybrid user interfaces. This leads to a variety of fundamental challenges and opportunities for research and design of hybrid user interfaces regarding e.g., allocation of tasks, the interplay between views, user representations, and collaborative coupling that are outlined in this position paper.
BibTeX
@inproceedings{10322296, abstract = {Over the past years, we have seen an increase in the number of user studies involving mixed reality interfaces. As these environments usually exceed standardized user study settings that only measure time and error, we developed, designed, and evaluated a mixed-immersion evaluation framework called ReLive. Its combination of in-situ and ex-situ analysis approaches allows for the holistic and malleable analysis and exploration of mixed reality user study data of an individual analyst in a step-by-step approach that we previously described as an asynchronous hybrid user interface. Yet, collaboration was coined as a key aspect for visual and immersive analytics – potentially allowing multiple analysts to synchronously explore mixed reality user study data from different but complementary angles of evaluation using hybrid user interfaces. This leads to a variety of fundamental challenges and opportunities for research and design of hybrid user interfaces regarding e.g., allocation of tasks, the interplay between views, user representations, and collaborative coupling that are outlined in this position paper.}, address = {Los Alamitos, CA, USA}, author = {Zagermann, J. and Hubenschmid, S. and Fink, D. and Wieland, J. and Reiterer, H. and Feuchtner, T.}, booktitle = {2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)}, doi = {10.1109/ISMAR-Adjunct60411.2023.00044}, month = {10}, pages = {191-195}, publisher = {IEEE Computer Society}, title = {Challenges and Opportunities for Collaborative Immersive Analytics with Hybrid User Interfaces}, url = {https://doi.ieeecomputersociety.org/10.1109/ISMAR-Adjunct60411.2023.00044}, year = 2023 }
Link
https://doi.ieeecomputersociety.org/10.1109/ISMAR-Adjunct60411.2023.00044
S. Hubenschmid, J. Zagermann, D. Leicht, H. Reiterer, and T. Feuchtner, “ARound the Smartphone: Investigating the Effects of Virtually-Extended Display Size on Spatial Memory,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). New York, NY, USA: ACM, 2023. doi: https://doi.org/10.1145/3544548.3581438.
Abstract
Smartphones conveniently place large information spaces in the palms of our hands. While research has shown that larger screens positively affect spatial memory, workload, and user experience, smartphones remain fairly compact for the sake of device ergonomics and portability. Thus, we investigate the use of hybrid user interfaces to virtually increase the available display size by complementing the smartphone with an augmented reality head-worn display. We thereby combine the benefits of familiar touch interaction with the near-infinite visual display space afforded by augmented reality. To better understand the potential of virtually-extended displays and the possible issues of splitting the user’s visual attention between two screens (real and virtual), we conducted a within-subjects experiment with 24 participants completing navigation tasks using different virtually-augmented display sizes. Our findings reveal that a desktop monitor size represents a “sweet spot” for extending smartphones with augmented reality, informing the design of hybrid user interfaces.
BibTeX
@inproceedings{Hubenschmid2023ARound, abstract = {Smartphones conveniently place large information spaces in the palms of our hands. While research has shown that larger screens positively affect spatial memory, workload, and user experience, smartphones remain fairly compact for the sake of device ergonomics and portability. Thus, we investigate the use of hybrid user interfaces to virtually increase the available display size by complementing the smartphone with an augmented reality head-worn display. We thereby combine the benefits of familiar touch interaction with the near-infinite visual display space afforded by augmented reality. To better understand the potential of virtually-extended displays and the possible issues of splitting the user’s visual attention between two screens (real and virtual), we conducted a within-subjects experiment with 24 participants completing navigation tasks using different virtually-augmented display sizes. Our findings reveal that a desktop monitor size represents a “sweet spot” for extending smartphones with augmented reality, informing the design of hybrid user interfaces.}, address = {New York, NY, USA}, author = {Hubenschmid, Sebastian and Zagermann, Johannes and Leicht, Daniel and Reiterer, Harald and Feuchtner, Tiare}, booktitle = {Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23)}, doi = {https://doi.org/10.1145/3544548.3581438}, publisher = {ACM}, title = {ARound the Smartphone: Investigating the Effects of Virtually-Extended Display Size on Spatial Memory}, url = {https://kops.uni-konstanz.de/server/api/core/bitstreams/6eecac2f-666f-4399-bec3-d8e607331164/content}, video = {https://youtu.be/p6cHwLxHWJg}, year = 2023 }
Link
https://kops.uni-konstanz.de/server/api/core/bitstreams/6eecac2f-666f-4399-bec3-d8e607331164/content
A. Zaky, J. Zagermann, H. Reiterer, and T. Feuchtner, “Opportunities and Challenges of Hybrid User Interfaces for Optimization of Mixed Reality Interfaces,” in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), in 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 2023, pp. 215–219. doi: 10.1109/ISMAR-Adjunct60411.2023.00050.
Abstract
Current research highlights the importance of adaptive mixed reality interfaces, as increased adoption leads to increasingly diverse, complex and unconstrained interaction scenarios. An interesting approach for adaptation, is the optimization of interface layout and behaviour. We thereby consider three distinct types of context to which the interface adapts: the user, the activity, and the environment. The latter of these includes a myriad of interactive devices surrounding the user, the capabilities of which we propose to take advantage of by integrating them in a hybrid user interface. Hybrid user interfaces offer many opportunities to address distinct usability issues, such as visibility, reachability, and ergonomics. However, considering additional interactive devices for optimizing mixed reality interfaces introduces a number of additional challenges, such as detecting available and suitable devices and modeling the respective interaction costs. Moreover, using different devices potentially introduces a switching cost e.g., in terms of cognitive load and time. In this paper, we aim to discuss different opportunities and challenges of using hybrid user interfaces for the optimization of mixed reality interfaces and thereby highlight directions for future work.
BibTeX
@inproceedings{10322176, abstract = {Current research highlights the importance of adaptive mixed reality interfaces, as increased adoption leads to increasingly diverse, complex and unconstrained interaction scenarios. An interesting approach for adaptation, is the optimization of interface layout and behaviour. We thereby consider three distinct types of context to which the interface adapts: the user, the activity, and the environment. The latter of these includes a myriad of interactive devices surrounding the user, the capabilities of which we propose to take advantage of by integrating them in a hybrid user interface. Hybrid user interfaces offer many opportunities to address distinct usability issues, such as visibility, reachability, and ergonomics. However, considering additional interactive devices for optimizing mixed reality interfaces introduces a number of additional challenges, such as detecting available and suitable devices and modeling the respective interaction costs. Moreover, using different devices potentially introduces a switching cost e.g., in terms of cognitive load and time. In this paper, we aim to discuss different opportunities and challenges of using hybrid user interfaces for the optimization of mixed reality interfaces and thereby highlight directions for future work.}, author = {Zaky, Abdelrahman and Zagermann, Johannes and Reiterer, Harald and Feuchtner, Tiare}, booktitle = {2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)}, doi = {10.1109/ISMAR-Adjunct60411.2023.00050}, pages = {215-219}, title = {Opportunities and Challenges of Hybrid User Interfaces for Optimization of Mixed Reality Interfaces}, url = {https://ieeexplore.ieee.org/document/10322176}, year = 2023 }
Link
https://ieeexplore.ieee.org/document/10322176
F. Chiossi et al., “Adapting visualizations and interfaces to the user,” it - Information Technology, vol. 64, no. 4–5, Art. no. 4–5, 2022, doi: 10.1515/itit-2022-0035.
Abstract
Adaptive visualization and interfaces pervade our everyday tasks to improve interaction from the point of view of user performance and experience. This approach allows using several user inputs, whether physiological, behavioral, qualitative, or multimodal combinations, to enhance the interaction. Due to the multitude of approaches, we outline the current research trends of inputs used to adapt visualizations and user interfaces. Moreover, we discuss methodological approaches used in mixed reality, physiological computing, visual analytics, and proficiency-aware systems. With this work, we provide an overview of the current research in adaptive systems.
BibTeX
@article{ChiossiZagermannKarolusRodriguesBalestrucciWeiskopfEhingerFeuchtnerReitererChuangErnstBullingMayerSchmidt+2022, abstract = {Adaptive visualization and interfaces pervade our everyday tasks to improve interaction from the point of view of user performance and experience. This approach allows using several user inputs, whether physiological, behavioral, qualitative, or multimodal combinations, to enhance the interaction. Due to the multitude of approaches, we outline the current research trends of inputs used to adapt visualizations and user interfaces. Moreover, we discuss methodological approaches used in mixed reality, physiological computing, visual analytics, and proficiency-aware systems. With this work, we provide an overview of the current research in adaptive systems.}, author = {Chiossi, Francesco and Zagermann, Johannes and Karolus, Jakob and Rodrigues, Nils and Balestrucci, Priscilla and Weiskopf, Daniel and Ehinger, Benedikt and Feuchtner, Tiare and Reiterer, Harald and Chuang, Lewis L. and Ernst, Marc and Bulling, Andreas and Mayer, Sven and Schmidt, Albrecht}, doi = {10.1515/itit-2022-0035}, journal = {it - Information Technology}, number = {4-5}, pages = {133–143}, title = {Adapting visualizations and interfaces to the user}, url = {https://www.degruyter.com/document/doi/10.1515/itit-2022-0035}, volume = 64, year = 2022 }
Link
https://www.degruyter.com/document/doi/10.1515/itit-2022-0035
J. Zagermann et al., “Complementary Interfaces for Visual Computing,” it - Information Technology, vol. 64, no. 4–5, Art. no. 4–5, 2022, doi: doi:10.1515/itit-2022-0031.
Abstract
With increasing complexity in visual computing tasks, a single device may not be sufficient to adequately support the user’s workflow. Here, we can employ multi-device ecologies such as cross-device interaction, where a workflow can be split across multiple devices, each dedicated to a specific role. But what makes these multi-device ecologies compelling? Based on insights from our research, each device or interface component must contribute a complementary characteristic to increase the quality of interaction and further support users in their current activity. We establish the term complementary interfaces for such meaningful combinations of devices and modalities and provide an initial set of challenges. In addition, we demonstrate the value of complementarity with examples from within our own research.
BibTeX
@article{ZagermannHubenschmidBalestrucciFeuchtnerMayerErnstSchmidtReiterer+2022+145+154, abstract = {With increasing complexity in visual computing tasks, a single device may not be sufficient to adequately support the user’s workflow. Here, we can employ multi-device ecologies such as cross-device interaction, where a workflow can be split across multiple devices, each dedicated to a specific role. But what makes these multi-device ecologies compelling? Based on insights from our research, each device or interface component must contribute a complementary characteristic to increase the quality of interaction and further support users in their current activity. We establish the term complementary interfaces for such meaningful combinations of devices and modalities and provide an initial set of challenges. In addition, we demonstrate the value of complementarity with examples from within our own research.}, author = {Zagermann, Johannes and Hubenschmid, Sebastian and Balestrucci, Priscilla and Feuchtner, Tiare and Mayer, Sven and Ernst, Marc O. and Schmidt, Albrecht and Reiterer, Harald}, doi = {doi:10.1515/itit-2022-0031}, journal = {it - Information Technology}, lastchecked = {2022-09-20}, number = {4-5}, pages = {145--154}, title = {Complementary Interfaces for Visual Computing}, url = {https://doi.org/10.1515/itit-2022-0031}, volume = 64, year = 2022 }
Link
https://doi.org/10.1515/itit-2022-0031