Project C02

C02 | Physiologically Based Interaction and Adaptive Visualization

Prof. Albrecht Schmidt, LMU
Email | Website

Prof. Harald Reiterer, University of Konstanz
Email | Website

[completed]

Romina Poguntke, University of Stuttgart

About this project
Results

In this project, we research new methods and techniques for cognition-aware visualizations. The basic idea is that a cognition-aware adaptive visualization will observe the physiological response of a person while interacting with a system und use this implicit input. Electrical signals measured on the body (e.g. EEG, EMG, ECG, galvanic skin response), changes of physiological parameters (e.g. body temperature, respiration rate, and pulse) as well as the users’ gaze behavior are used to estimate cognitive load and understanding. Through experimental research, concepts and models for adaptive and dynamic visualizations will be created. Frameworks and tools will be realized and empirically validated.

Research Questions

How can we optimize interactive visualization to create the best possible user experience and to maximize task performance?

To address this, the following questions need to be addressed:

What are relevant quantitative parameters for the user experience in visualization tasks?

How can we estimate these parameters based on physiological measurements?

What forms of adaptation support influencing the experienced cognitive load?

How can these mechanisms be designed and implemented to change and adapt interactive visualizations?

Fig. 1: Possible input modalities for physiological data: EEG, Eye-Tracking, EMG, Thermal Imaging. (left to right, top to bottom).

Fig. 2: Abstract representation of a cognition-aware adaptive visualization.

Publications

T. Kosch, M. Funk, A. Schmidt, and L. L. Chuang, “Identifying Cognitive Assistance with Mobile Electroencephalography: A Case Study with In-Situ Projections for Manual Assembly.,” Proceedings of the ACM on Human-Computer Interaction (ACMHCI), vol. 2, pp. 11:1–11:20, 2018, doi: 10.1145/3229093.
- BibTeX
- Link
BibTeX
@article{journals/pacmhci/KoschFSC18, affiliation = {Chuang, Lewis L., Technische Universität Chemnitz}, author = {Kosch, Thomas and Funk, Markus and Schmidt, Albrecht and Chuang, Lewis L.}, doi = {10.1145/3229093}, journal = {Proceedings of the ACM on Human-Computer Interaction (ACMHCI)}, orcid-numbers = {Chuang, Lewis L./0000-0002-1975-5716}, pages = {11:1-11:20}, title = {Identifying Cognitive Assistance with Mobile Electroencephalography: A Case Study with In-Situ Projections for Manual Assembly.}, url = {https://doi.org/10.1145/3229093}, volume = 2, year = 2018 }
Link
https://doi.org/10.1145/3229093
J. Karolus, H. Schuff, T. Kosch, P. W. Woźniak, and A. Schmidt, “EMGuitar: Assisting Guitar Playing with Electromyography,” in Proceedings of the Designing Interactive Systems Conference (DIS), I. Koskinen, Y.-K. Lim, T. C. Pargman, K. K. N. Chow, and W. Odom, Eds., ACM, 2018, pp. 651–655. doi: 10.1145/3196709.3196803.
- BibTeX
- Link
BibTeX
@inproceedings{conf/ACMdis/KarolusSKW018, author = {Karolus, Jakob and Schuff, Hendrik and Kosch, Thomas and Woźniak, Paweł W. and Schmidt, Albrecht}, booktitle = {Proceedings of the Designing Interactive Systems Conference (DIS)}, doi = {10.1145/3196709.3196803}, editor = {Koskinen, Ilpo and Lim, Youn-Kyung and Pargman, Teresa Cerratto and Chow, Kenny K. N. and Odom, William}, pages = {651-655}, publisher = {ACM}, title = {EMGuitar: Assisting Guitar Playing with Electromyography}, url = {https://doi.org/10.1145/3196709.3196803}, year = 2018 }
Link
https://doi.org/10.1145/3196709.3196803
T. Dingler, R. Rzayev, A. S. Shirazi, and N. Henze, “Designing Consistent Gestures Across Device Types: Eliciting RSVP Controls for Phone, Watch, and Glasses,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, R. L. Mandryk, M. Hancock, M. Perry, and A. L. Cox, Eds., ACM, 2018, pp. 419:1–419:12. doi: 10.1145/3173574.3173993.
- BibTeX
- Link
BibTeX
@inproceedings{conf/chi/DinglerRSH18, affiliation = {Dingler, Tilman, Institut für Visualisierung und Interaktive Systeme. Henze, Niels, Institut für Visualisierung und Interaktive Systeme}, author = {Dingler, Tilman and Rzayev, Rufat and Shirazi, Alireza Sahami and Henze, Niels}, booktitle = {Proceedings of the CHI Conference on Human Factors in Computing Systems}, doi = {10.1145/3173574.3173993}, editor = {Mandryk, Regan L. and Hancock, Mark and Perry, Mark and Cox, Anna L.}, pages = {419:1-419:12}, publisher = {ACM}, title = {Designing Consistent Gestures Across Device Types: Eliciting RSVP Controls for Phone, Watch, and Glasses}, url = {https://doi.org/10.1145/3173574.3173993}, year = 2018 }
Link
https://doi.org/10.1145/3173574.3173993
J. Karolus, P. W. Woźniak, L. L. Chuang, and A. Schmidt, “Robust Gaze Features for Enabling Language Proficiency Awareness,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, G. Mark, S. R. Fussell, C. Lampe, m. c. schraefel, J. P. Hourcade, C. Appert, and D. Wigdor, Eds., ACM, 2017, pp. 2998–3010. doi: 10.1145/3025453.3025601.
- BibTeX
- Link
BibTeX
@inproceedings{conf/chi/KarolusWCS17, affiliation = {Chuang, Lewis L., Technische Universität Chemnitz}, author = {Karolus, Jakob and Woźniak, Paweł W. and Chuang, Lewis L. and Schmidt, Albrecht}, booktitle = {Proceedings of the CHI Conference on Human Factors in Computing Systems}, doi = {10.1145/3025453.3025601}, editor = {Mark, Gloria and Fussell, Susan R. and Lampe, Cliff and schraefel, m. c. and Hourcade, Juan Pablo and Appert, Caroline and Wigdor, Daniel}, orcid-numbers = {Chuang, Lewis L./0000-0002-1975-5716}, pages = {2998-3010}, publisher = {ACM}, title = {Robust Gaze Features for Enabling Language Proficiency Awareness}, url = {https://doi.org/10.1145/3025453.3025601}, year = 2017 }
Link
https://doi.org/10.1145/3025453.3025601
J. Karolus, P. W. Woźniak, and L. L. Chuang, “Towards Using Gaze Properties to Detect Language Proficiency,” in Proceedings of the 9th Nordic Conference on Human-Computer Interaction (NordiCHI), New York, NY, USA: ACM, 2016, pp. 118:1–118:6. doi: 10.1145/2971485.2996753.
- BibTeX
- Link
BibTeX
@inproceedings{noauthororeditor2016towards, address = {New York, NY, USA}, affiliation = {Chuang, Lewis L., Technische Universität Chemnitz}, author = {Karolus, Jakob and Woźniak, Paweł W. and Chuang, Lewis L.}, booktitle = {Proceedings of the 9th Nordic Conference on Human-Computer Interaction (NordiCHI)}, doi = {10.1145/2971485.2996753}, orcid-numbers = {Chuang, Lewis L./0000-0002-1975-5716}, pages = {118:1-118:6}, publisher = {ACM}, title = {Towards Using Gaze Properties to Detect Language Proficiency}, url = {https://doi.org/10.1145/2971485.2996753}, year = 2016 }
Link
https://doi.org/10.1145/2971485.2996753
N. Flad, J. C. Ditz, A. Schmidt, H. H. Bülthoff, and L. L. Chuang, “Data-Driven Approaches to Unrestricted Gaze-Tracking Benefit from Saccade Filtering,” in Proceedings of the Second Workshop on Eye Tracking and Visualization (ETVIS), M. Burch, L. L. Chuang, and A. T. Duchowski, Eds., IEEE, 2016, pp. 1–5. [Online]. Available: https://ieeexplore.ieee.org/document/7851156
- BibTeX
- Link
BibTeX
@inproceedings{conf/etvis/FladDSBC16, affiliation = {Chuang, Lewis L., Technische Universität Chemnitz}, author = {Flad, Nina and Ditz, Jonas C. and Schmidt, Albrecht and Bülthoff, Heinrich H. and Chuang, Lewis L.}, booktitle = {Proceedings of the Second Workshop on Eye Tracking and Visualization (ETVIS)}, editor = {Burch, Michael and Chuang, Lewis L. and Duchowski, Andrew T.}, orcid-numbers = {Chuang, Lewis L./0000-0002-1975-5716}, pages = {1-5}, publisher = {IEEE}, title = {Data-Driven Approaches to Unrestricted Gaze-Tracking Benefit from Saccade Filtering}, url = {https://ieeexplore.ieee.org/document/7851156}, year = 2016 }
Link
https://ieeexplore.ieee.org/document/7851156
L. Lischke, S. Mayer, K. Wolf, N. Henze, H. Reiterer, and A. Schmidt, “Screen arrangements and interaction areas for large display work places,” in PerDis ’16 Proceedings of the 5th ACM International Symposium on Pervasive Displays, ACM, Ed., ACM, 2016, pp. 228–234. doi: 10.1145/2914920.2915027.
- BibTeX
- Link
BibTeX
@inproceedings{Lischke2016perdis, affiliation = {Henze, Niels, Institut für Visualisierung und Interaktive Systeme. Reiterer, Harald, Universtität Konstanz}, author = {Lischke, Lars and Mayer, Sven and Wolf, Katrin and Henze, Niels and Reiterer, Harald and Schmidt, Albrecht}, booktitle = {PerDis '16 Proceedings of the 5th ACM International Symposium on Pervasive Displays}, doi = {10.1145/2914920.2915027}, editor = {ACM}, orcid-numbers = {Reiterer, Harald/0000-0001-8528-8928}, pages = {228-234}, privnote = {uid:2275}, publisher = {ACM}, title = {Screen arrangements and interaction areas for large display work places}, url = {http://dx.doi.org/10.1145/2914920.2915027}, volume = 5, year = 2016 }
Link
http://dx.doi.org/10.1145/2914920.2915027
D. Weiskopf, M. Burch, L. L. Chuang, B. Fischer, and A. Schmidt, Eye Tracking and Visualization: Foundations, Techniques, and Applications. Berlin, Heidelberg: Springer, 2016. [Online]. Available: https://www.springer.com/de/book/9783319470238
- BibTeX
- Link
BibTeX
@book{weiskopf2016tracking, address = {Berlin, Heidelberg}, affiliation = {Weiskopf, Daniel, Visualisierungsinstitut der Universität Stuttgart. Burch, Michael, Visualisierungsinstitut der Universität Stuttgart. Chuang, Lewis L., Technische Universität Chemnitz}, author = {Weiskopf, Daniel and Burch, Michael and Chuang, Lewis L. and Fischer, B. and Schmidt, Albrecht}, orcid-numbers = {Weiskopf, Daniel/0000-0003-1174-1026, Chuang, Lewis L./0000-0002-1975-5716}, publisher = {Springer}, title = {Eye Tracking and Visualization: Foundations, Techniques, and Applications}, url = {https://www.springer.com/de/book/9783319470238}, year = 2016 }
Link
https://www.springer.com/de/book/9783319470238
L. Lischke, V. Schwind, K. Friedrich, A. Schmidt, and N. Henze, “MAGIC-Pointing on Large High-Resolution Displays,” in Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA), ACM, Ed., ACM, 2016, pp. 1706–1712. doi: 10.1145/2851581.2892479.
- BibTeX
- Link
BibTeX
@inproceedings{680a9de709b945b197a921ef21689f06, affiliation = {Henze, Niels, Institut für Visualisierung und Interaktive Systeme}, author = {Lischke, Lars and Schwind, Valentin and Friedrich, Kai and Schmidt, Albrecht and Henze, Niels}, booktitle = {Proceedings of the CHI Conference on Human Factors in Computing Systems-Extended Abstracts (CHI-EA)}, doi = {10.1145/2851581.2892479}, editor = {ACM}, note = {Rezensiertes Poster}, pages = {1706-1712}, privnote = {uid:2180}, publisher = {ACM}, title = {MAGIC-Pointing on Large High-Resolution Displays}, url = {http://dx.doi.org/10.1145/2851581.2892479}, year = 2016 }
Link
http://dx.doi.org/10.1145/2851581.2892479
L. Lischke, J. Grüninger, K. Klouche, A. Schmidt, P. Slusallek, and G. Jacucci, “Interaction Techniques for Wall-Sized Screens,” Proceedings of the International Conference on Interactive Tabletops & Surfaces (ITS), pp. 501–504, 2015, doi: 10.1145/2817721.2835071.
- BibTeX
- Link
BibTeX
@article{d05155d0112446029651f5222cf5502b, author = {Lischke, Lars and Grüninger, Jürgen and Klouche, Khalil and Schmidt, Albrecht and Slusallek, Philipp and Jacucci, Giulio}, doi = {10.1145/2817721.2835071}, editor = {ACM}, journal = {Proceedings of the International Conference on Interactive Tabletops & Surfaces (ITS)}, pages = {501-504}, privnote = {uid:2156}, title = {Interaction Techniques for Wall-Sized Screens}, url = {http://dx.doi.org/10.1145/2817721.2835071}, year = 2015 }
Link
http://dx.doi.org/10.1145/2817721.2835071
L. Lischke, P. Knierim, and H. Klinke, “Mid-Air Gestures for Window Management on Large Displays,” in Mensch und Computer 2015 – Tagungsband (MuC), D. G. Oldenbourg, Ed., De Gruyter, 2015, pp. 439–442. doi: 10.1515/9783110443929-072.
- BibTeX
- Link
BibTeX
@inproceedings{6d9dca05de77410a9b11f62fd1bed9f4, author = {Lischke, Lars and Knierim, Pascal and Klinke, Hermann}, booktitle = {Mensch und Computer 2015 – Tagungsband (MuC)}, doi = {10.1515/9783110443929-072}, editor = {Oldenbourg, De Gruyter}, note = {Rezensiertes Poster}, pages = {439-442}, privnote = {uid:2158}, publisher = {De Gruyter}, title = {Mid-Air Gestures for Window Management on Large Displays}, url = {http://dx.doi.org/10.1515/9783110443929-072}, year = 2015 }
Link
http://dx.doi.org/10.1515/9783110443929-072