Project overview

Gaming often drives the early adoption of Natural User Interfaces (NUIs), and serves as a testing ground for their development. Technologies created for gaming, such as the Microsoft Xbox Kinect and the Tobii Eye Tracker 4C, enable new types of playful and immersive experiences — and often go on to impact wider society. However, while there is substantial research into the user experience of NUIs in work-like computer tasks, very little research has examined NUIs in the context of games and immersive environments. We believe that studying NUI in the game context is relevant to the use of these technologies, because the game experience can be quite different when immersed or embodied.

Research under this project generates knowledge about designing with NUIs in social games, through case studies that interrogate social play in games. The research agenda is multi-modal, and aims to contribute a rich understanding regarding the relationship between game embodiment, immersion and interface design.

In addition to academic articles, we communicate results in game industry and non-specialist publications, host talks at game conventions, demonstrate and exemplify our findings through videos, and develop game prototypes that demonstrate the foundations of our design guidelines.

Social voice

Voice is widely used in social and competitive gameplay as a communication modality between players, where it has been shown to vastly improve player experience. However, as an input device for play, voice has a mixed status. It is often marginal and maligned due to the limitations of the technology, but sometimes fêted as a socially engaging way to play. In this project, we seek to understand the parameters of designing for voice interaction with game-like characters, through a study of existing practices and through the evaluation of for-purpose experimental games.

This project is led by Fraser Allison.

Social gaze

EyePlay research seeks to understand how gaze detection technologies can provide new and novel player and spectator experiences. This project examines the effects of gaze and how it can create social experiences in multiplayer games, through the exploration of its detection and display.

This project is led by Joshua Newn.

Social tabletop

Tabletop gaming is a highly social situation due to the configuration of players and the social interactions afforded by tabletop game pieces. This project examines non-digital tabletop games and their digitisations to identify opportunities for NUI technologies to afford social play.

This project is led by Melissa Rogerson.

Social Minecraft

The sandbox-world builder game Minecraft is exceedingly popular, near ubiquitous in children between 8–12 and was recently purchased for $2billion dollars by Microsoft. This popularity, and its emphasis on creative world building, social interactions and cooperation have seen it adopted in schools and released as a standalone Minecraft for Education project. This project examines the social play of Minecraft and how it can support opportunities for social learning.

This project is led by PhD student Jane Mavoa, under the supervision of Martin Gibbs, Marcus Carter and Bjorn Nansen.

Funding

These projects were funded by the University of Melbourne’s Microsoft Research Centre for Social Natural User Interfaces (SocialNUI).

Project team

• 
  Martin Gibbs, Professor
• 
  Eduardo Velloso, Senior Lecturer
• 
  Frank Vetere, Professor
• 
  Wally Smith, Associate Professor
• 
  Fraser Allison, Research Fellow
• 
  Joshua Newn, Research Fellow
• 
  Melissa Rogerson, Lecturer
• 
  Jane Mavoa, PhD Candidate
• 
  Marcus Carter, Senior Lecturer in Digital Cultures, University of Sydney

Contact details

• 
  Martin Gibbs

  Email: martin.gibbs@unimelb.edu.au

Publications

Allison, F., Carter, M., Gibbs, M. & Smith, W. (2018) Design Patterns for Voice Interaction in Games. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY 2018), 28–31 October, Melbourne, Australia.

Rogerson, M., Gibbs, M., Carter, M. & Allison, F. (2018) Understanding Player Elimination in Boardgames as a form of Permadeath. In Proceedings of the 2018 Digital Games Research Association Conference (DiGRA 2018), 25–28 July, Turin, Italy.

Newn, J. (2018) Enabling Intent Recognition Through Gaze Awareness in User Interfaces. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems – Doctoral Colloquium, May 2018, Montreal, Canada.

Newn, J., Allison, F., Velloso, E. & Vetere, F. (2018) Looks Can Be Deceiving: Using Gaze Visualisation to Predict and Mislead Opponents in Strategic Gameplay. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, May 2018, Montreal, Canada.

Rogerson, M., Gibbs, M. & Smith, W. (2018) Collaborating to Compete: the mutuality of cooperation and competition in boardgame play. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, May 2018, Montreal, Canada.

Miller, T., Singh, R., Newn, J., Sonenberg, L., Velloso, E. & Vetere, F. (2018) Combining Planning with Gaze for Online Human Intention Recognition. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2018), 10–15 July, Stockholm, Sweden.

Mavoa, J., Carter, M. & Gibbs, M. (2017) Children, digital games and Minecraft: a survey of young children’s gaming habits. New Media and Society DOI: 10.1177/1461444817745320

Allison, F., Carter, M. & Gibbs, M. (2017) Word Play: A History of Voice Interaction in Digital Games. Games and Culture DOI: 10.1177/1555412017746305

Mavoa, J., Carter, M. & Gibbs, M. (2017) Beyond Addiction: Parent perceptions of Minecraft play. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play, 15nd 18 October 2017, Amsterdam, The Netherlands. DOI: 10.1145/3116595.3116638

Newn, J., Velloso, E., Allison, F., Abdelrahman, Y. & Vetere, F. (2017) Evaluating Real-Time Gaze Representations to Infer Intentions in Competitive Turn-Based Strategy Games. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play, 15–18 October 2017, Amsterdam, The Netherlands. DOI: 10.1145/3116595.3116624

Rogerson, M., Gibbs, M. & Smith, W. (2017) What can we learn from eye tracking boardgame play? In Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in Play, 15–18 October 2017, Amsterdam, The Netherlands. DOI: 10.1145/3130859.3131314

Rogerson, M., Gibbs, M. & Smith, W. (2017) Exploring the digital hinterland: Internet practices surrounding the pursuit of “offline” hobbies. Association of Internet Researchers Conference (AoIR 2017), 18–21 October 2017, Tart, Estonia.

Rogerson, M., Gibbs, M. & Smith, W. (2017) Using gaze data in understanding tabletop boardgame play.  Games Research Methods Symposium, 10–11 July, Sydney, Australia.

Velloso, E. & Gellersen, H. (2017) Gaze Interaction Design: Eye Movement as a Design Material. Designing Interactive Systems Conference (DIS 2017), June 2017, Edinburgh, Scotland.

Mavoa, J., Gibbs, M. & Carter, M. (2017) Constructing the young child media user in Australia: a discourse analysis of Facebook comments. Journal of Children and Media, 11(3): 330-346. [DOI:10.1080/17482798.2017.1308400]

Carter, M. & Allison, F. (in press) Guilt in DayZ. In Transgressions in Games and Play, Eds. Jørgensen, K. & Karlsen, F., MIT Press.

Mavoa, J. (2017) The games children play: an analysis by title, gender and age. Extended Abstract presented at Digital Games Research Association Conference (DiGRA 2017), Melbourne, Australia.

Allison, F., Luger, E. & Hofmann, K. (2017) Player expectations of a learning AI companion in Minecraft. Extended Abstract presented at Digital Games Research Association Conference (DiGRA 2017), Melbourne, Australia.

Rogerson, R. & Cocks, J. (2017) F**king with psychology: psychological tests as boardgame design inspiration. Extended Abstract presented at Digital Games Research Association Conference (DiGRA 2017), Melbourne, Australia.

Rogerson, R., Gibbs, M. & Smith, W. (2017) Me and My Shelfie: hobbyists’ domestication of boardgames. Extended Abstract presented at Digital Games Research Association Conference (DiGRA 2017), Melbourne, Australia.

Carter, M. & Allison, F. (2017) Fear, Loss and Meaningful Play: Permadeath in DayZ. Journal of Gaming and Virtual Worlds 9(2):143–158

Velloso, E., Carter, M., Newn, J., Esteves, A., Clarke, C. & Gellersen, H. (2017) Motion Correlation: Selecting Objects by Matching their Movement ACM Transactions on Computer-Human Interaction (ToCHI) 24(3):article22. [DOI]

Allison, F., Luger, E. & Hofmann, K. (2017) Spontaneous Interactions with a Virtually Embodied Intelligent Assistant in Minecraft. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, Denver, USA. [DOI]

Allison, F. (2016) Are You Out of Your Mind? Focalization in Digital Games. Transactions of the Digital Games Research Association 2(3):31–60.

Carter, M., Fordyce, R., Gibbs, M., Robertson, N., & Witkowski, E., (2016) eSports Spectatorship in Australia. In Proceedings of the 2016 DiGRA Australia Conference, Melbourne [PDF]

Velloso, E. (2016) Game Controller Research from the Lab into the Wild: The Case of Eye Tracking. In Proceedings of the 2016 DiGRA Australia Conference, Melbourne [PDF]

Velloso, E. & Carter, M. (2016) The Emergence of EyePlay: A Survey of Eye Interaction in Games. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (CHI Play 2016). Austin, TX, USA [PDF]

Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Exploring the Effects of Gaze Awareness on Multiplayer Gameplay. In Proceedings of the 2016 Annual Symposium Extended Abstracts on Computer-Human Interaction in Play (CHI PLAY 2016). Austin, TX, USA [DOI] [PDF]

Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Dynamically Exposing Gaze to Foster Playful Experiences in Multiplayer Gameplay. In CHI PLAY 2016 Workshop on Designing for Emotional Complexity in Games: The Interplay of Positive and Negative Affect. Austin, TX, USA [PDF]

Newn, J., Velloso, E., Carter, M. & Vetere, F. (2016). Multimodal Segmentation of a Large Interactive Tabletop: Extending Interaction on Horizontal Surfaces with Gaze. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (ISS 2016). Niagara Falls, ON, Canada [PDF]

Allison, F., Carter, M., & Gibbs, M. (2016). A History of Voice Interaction in Games. In Proceedings of the First Joint International Conference of DiGRA and FDG [PDF]

Rogerson, M. J., Gibbs, M., & Smith, W. (2016) "I Love All the Bits": The Materiality of Boardgames. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016), New York: ACM Press, pp. 3956–3969 [DOI]

Carter, M., Newn, J., Velloso, E., & Vetere, F. (2015) Remote Gaze and Gesture Tracking on the Microsoft Kinect: Investigating the Role of Feedback. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction (OzCHI 2015) New York: ACM Press, pp. 167–176 [DOI]

Allison, F., Carter, M., & Gibbs, M. (2015). Good Frustrations: The Paradoxical Pleasure of Fearing Death in DayZ. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction (OzCHI 2015) New York: ACM Press, pp. 119–123. [DOI]

Carter, M., Allison, F., Downs, J., & Gibbs, M. (2015). Player Identity Dissonance and Voice Interaction in Games. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (CHI Play 2015) New York: ACM Press, pp. 265–269 [DOI]

Rogerson, M. J., Gibbs, M., & Smith, W. (2015) Digitising Boardgames: Issues and Tensions, In Proceedings of the 2015 DiGRA International Conference. Lüneburg, Germany [PDF]

Carter, M., Nansen B., Gibbs, M. (2014) Screen Ecologies, Multi-Gaming and Designing for Different Registers of Engagement. In Proceedings of CHI Play, ACM Press, pp. 37–46. [PDF, DOI]

Media