C05 | Human-Machine Interaction with Adaptive Multisensory Systems

Prof. Dr. Marc Ernst, Ulm University
Email | Website

Daniel Weiskopf

Prof. Dr. Albrecht Schmidt, LMU Munich
Email | Website

New image

Priscilla Balestrucci, Ulm University – Email | Website

User-adaptive systems are a recent trend in technological development. Designed to learn the characteristics of the user interacting with them, they change their own features to provide users with a targeted, personalized experience.

This project investigates human adaptive behavior in mutual-learning situations. A better understanding of adaptive human-machine interactions, and of human sensorimotor learning processes in particular, will provide guidelines, evaluation criteria, and recommendations that will be beneficial for all projects within SFB/Transregio 161 that focus on the design of user-adaptive systems and algorithms.

To achieve this goal, we will carry out behavioral experiments using human participants and base our empirical choices on the framework of optimal decision theory as derived from the Bayesian approach. This approach can be used as a tool to construct ideal observer models against which human performance can be compared.

Previous studies have shown, that adaptive features can create side effects, which might be undesirable. In applications requiring the users to improve their skills, for example, these side effects could impede the human learn process (Fig.1).

Research Questions

What are the determinants of mutual adaptation between an adaptable user and a useradaptive system?

How does mutual adaptation change based on the sensory modalities involved?

Can mutual adaptation enhance immersion in an interactive virtual environment?

Do the interaction capabilities and experiences learnt in an adaptive system generalize to real-life, non-adaptive scenarios?

Fig.1: Median of errors. In the non-adaptive condition (left panel) the error-based algorithm is not implemented, therefore the pointing error is the same as the displayed feedback error. In the adaptive condition (right panel) the system acquires the subject´s pointing error (dashed line) and partially corrects for it in the displayed feedback error (solid line). The pointing error interestingly increases in the adaptive condition over time.

Publications

  1. T. Machulla, L. Chuang, F. Kiss, M. O. Ernst, and A. Schmidt, “Sensory Amplification Through Crossmodal Stimulation,” in Proceedings of the CHI Workshop on Amplification and Augmentation of Human Perception, 2017.
  2. T. Waltemate et al., “The Impact of Latency on Perceptual Judgments and Motor Performance in Closed-loop Interaction in Virtual Reality,” in Proceedings of the ACM Conference on Virtual Reality Software and Technology (VRST), 2016, pp. 27–35.
  3. C. V. Parise, K. Knorre, and M. O. Ernst, “Natural Auditory Scene Statistics Shapes Human Spatial Hearing,” in Proceedings of the National Academy of Sciences, 2014, vol. 111, no. 16, pp. 6104--6108.
  4. M. Rohde, L. C. J. van Dam, and M. O. Ernst, “Predictability is Necessary for Closed-loop Visual Feedback Delay Adaptation,” Journal of Vision, vol. 14, no. 3, pp. 4–4, 2014.
  5. L. C. J. van Dam, D. J. Hawellek, and M. O. Ernst, “Switching Between Visuomotor Mappings: Learning Absolute Mappings or Relative Shifts,” Journal of Vision, vol. 13, no. 2, p. 26, 2013.
  6. E. Steinbach et al., “Haptic Communications,” in Proceedings of the IEEE, 2012, vol. 100, no. 4, pp. 937–956.
  7. M. O. Ernst, “Decisions Made Better,” Science, vol. 329, no. 5995, pp. 1022–1023, 2010.
  8. J. Burge, M. O. Ernst, and M. S. Banks, “The Statistical Determinants of Adaptation Rate in Human Reaching,” Journal of Vision, vol. 8, no. 4, pp. 20:1-20:19, 2008.
  9. M. O. Ernst, “Learning to Integrate Arbitrary Signals from Vision and Touch,” Journal of Vision, vol. 7, no. 5, p. 7, 2007.
  10. M. O. Ernst and M. S. Banks, “Humans Integrate Visual and Haptic Information in a Statistically Optimal Fashion,” Nature, vol. 415, no. 6870, pp. 429–433, 2002.