CHI 2018 - Special Interest Group

About          TOP guidelines          Agenda          Resources          Organizers

Transparency and Openness Promotion Guidelines in HCI

Special Interest Group at CHI 2018, Montreal, Canada

This special interest group addresses the status quo of HCI research with regards to research practices of transparence and openness. Specifically, it discusses whether current practices are in line with the standards applied to other fields (e.g., psychology, economics, medicine). It seeks to identify current practices that are more progressive and worth communicating to other disciplines, while evaluating whether practices in other disciplines are likely to apply to HCI research constructively. Potential outcomes include: (1) a review of current HCI research policies, (2) a report on recommended practices, and (3) a replication project of key findings in HCI research.

About the Special Interest Group

In 2015, a landmark paper was published that summarized the efforts of 270 researchers in replicating 100 studies, published across 3 highly-esteemed psychology journals [1]. It found that approximately a third of previously reported effects replicated in subsequent studies. Thus, it concluded that the critical features of scientific enterprise, namely transparency, openness, and reproducibility, were significantly undermined in actual research practice. This has given rise to the formalization and advocacy of “Transparency and Openness Promotion” (TOP) guidelines that strive to advance research [2]. Eight standards and three levels of increasing stringency comprise TOP guidelines, with the primary objective of incentivising transparency in research practices. To date, more than 5000 journals and institutions have signed up to review their practices in accordance to TOP guidelines; the Association of Computing Machinery and, notably, its Special Interest Group for Computer-Human Interaction (SIGCHI) are non-signatories.

The objective of the SIG is to arrive at a communal understanding of whether the TOP guidelines and recommended practices of the Open Science Initiative apply to strengthen current HCI research practices.

An outcome of this SIG will be a publication that relates the viable application of TOP guidelines in the HCI research community. This will be of interest to COS as well as CHI. Furthermore, we believe that the HCI community will generate innovative Open Science practices that could be recommended to COS in turn. Next, this SIG could give rise to a new research area, namely the development of better human-computer interfaces for facilitating the adoption of Open Science practices by scientists. Finally, attendees could be interested in implementing a large-scale replication project similar to [1]. Such efforts are currently underway in Cancer Biology and Social Science.

 

TOP Guidelines

The Center of Open Science advocates eight standards that supports the translation of Open Science values (i.e., transparency, openness, and reproducibility) into concrete and tangible actions concerning  [1]:

  1. Citation standards
  2. Data transparency
  3. Analytic methods (code) transparency
  4. Research materials transparency
  5. Design and analysis transparency
  6. Preregistration of studies
  7. Preregistration of analysis plans
  8. Replication

 

Agenda

Welcome and Introduction

Invited talk by COS representative

Debate on whether HCI needs TOP guidelines

Breakout discussion on TOP guidelines

Present decisions on current HCI standards with regards to TOP guidelines and recommend next steps

Summary statement

 

Resources

[1] Brian A Nosek, George Alter, George C Banks, D Borsboom, Sara D Bowman, Steven J Breckler, Stuart Buck, Christopher D Chambers, G Chin, G Christensen, and others. (2015). Promoting an open research culture. Science, 348(6242), 1422-1425.

[2] Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

 

Organizers

Lewis Chuang is an Ambassador of the Center for Open Science. He is interested in the role of human attention during human-machine interactions, in particular closed-loop systems. He is a project leader at the Max Planck Institute for Biological Cybernetics and a principal investigator at the transregional research center for "Quantitative Methods for Visual Computing". He studied psychology at the Universities of York (BSc) and Manchester (MPhil) and received his PhD in neuroscience at the University of Tübingen in 2011.

 

Ulrike Pfeil is a post doctoral researcher at the HCI Group, University of Konstanz. She focuses on research methods in human-computer interaction with a focus on technology supported knowledge work. Within the transregional research center for "Quantitative Methods for Visual Computing", she focuses on the application of quantitative and qualitative measures to develop and validate HCI theories and models. She studied Information Design (BA) at the Stuttgart Media University and Human-Centred Systems (MSc) at City University, London. She received her PhD in HCI at City University, London in 2011.