Augmented learning environment for physiotherapy education

Project overview

Physiotherapy is currently taught via multiple modalities, such as lectures, tutorials, demonstrations and hands-on training in practical classes. However, physiotherapy students often struggle to translate anatomical knowledge from textbooks to dynamic understanding of the mechanics of body movements in real life patients. Moreover, one of the critical skills in physiotherapy, and other clinical domains, is clinical reasoning. Students must learn to negotiate the complexity of multiple factors, starting with the mastery of strong domain knowledge, such as human anatomy, to forming hypotheses through patient inquiry and testing the hypotheses through examination. Clinical reasoning is particularly difficult to teach because it requires students to critically review clinical decisions which are often ephemeral and tacit. The best practice for learning such skills typically involves a class of students observing an expert therapist conducting a consultation with a real patient (or a surrogate patient). However, such settings do not make the therapist’s clinical decision making explicitly available to students. One of the major difficulties for students in learning clinical reasoning is the ability to identify medical cues to develop diagnostic hypotheses.

Augmented Studio setup in the classroom.

This project aims to enhance student learning in physiotherapy via an augmented learning environment titled the ‘Augmented Studio’ developed using projectors and Microsoft Kinect sensors to enable projections of an anatomy model onto a moving body for practical physiotherapy classes. Augmented Studio enables augmentation through projection mapping to turn a human body into a display surface, showing anatomical information such as muscles and skeleton in real time. The moving body of a volunteer student becomes a live canvas for the teacher to illustrate clinical reasoning. Using a tablet interface with pen input, the instructor can draw annotations directly on the virtual body, which are rendered in real time on the patient’s body.

Currently the instructor uses still photos, figures, and diagrams which do not capture the real time and complex dynamics of human movements. With the Augmented Studio, the researchers explore the capability of creating virtual annotations, to identify certain anatomical structures or to illustrate the joints or muscle connections to teach students. The annotations enhance teacher and student communication with the purpose of transferring the real life knowledge and experience of the teacher to a class of students.

Teacher and student interact with annotation on the volunteer

Augmented reality based learning and teaching presents an innovative pedagogical practice delivering benefits to both the staff and the students. The student learning experience is enhanced in multiple ways with the usage of the augmented reality learning environment. Augmented kinaesthetic information and instructor annotations support interactive observation for better clinical understanding of human anatomy and musculoskeletal structures. The ability to track a full body movement in 3 dimensions and examine the changes throughout the range of movement will enhance appreciation by students of the dynamic change in the anatomical configuration of the body.

The project aims to deliver positive learning outcomes for the students by enabling experiential learning and increased connection with the learning materials, lessons and the instructor, catering for both tactile and visual learning, and encouraging cognitive and clinical reasoning.

Collaboration

This project is a collaboration between the Microsoft Research Centre for Social Natural User Interfaces (SocialNUI) in the School of Computing and Information Systems and the Department of Physiotherapy at the University of Melbourne.

The project is supported by an ARC Discovery Project (DP130102142) and a University of Melbourne Learning and Teaching Initiative (LTI) Grant 2016.

Project team

Hasan Shahid Ferdous, Research Fellow, Microsoft Research Centre for SocialNUI, The University of Melbourne
Thuong Hoang, Lecturer, School of Information Technology, Deakin University
Frank Vetere, Professor, The University of Melbourne
Zaher Joukhadar, Lead Software Engineer, Microsoft Research Centre for SocialNUI, The University of Melbourne
Martin Reinoso, PhD Candidate, Microsoft Research Centre for SocialNUI, The University of Melbourne
Louisa Remedios, Associate Professor, Department of Physiotherapy, The University of Melbourne
David Kelly, Lecturer, Department of Physiotherapy, The University of Melbourne
Egemen Tanin, Associate Professor, School of Computing and Information Systems, The University of Melbourne

Contact details

Hasan Shahid Ferdous

Publications

Hoang, T., Ferdous, H., Vetere, F. & Reinoso, M. (2018) Body as a Canvas: An exploration on the role of the body as display of digital information. In Proceedings of the 2018 Designing Interactive Systems (DIS) Conference 2018, pp. 253-263. DOI: 10.1145/3196709.3196724

Kelly, D, Hoang, T, Joukhadar, Z, Reinoso, M, Clements, T & Vetere, F (2017) Usage of an augmented reality system to facilitate dynamic anatomical understanding for physiotherapy students. Australian Physiotherapy Association Conference (APA 2017), Sydney, Australia.

Hoang, T, Reinoso, M, Joukhadar, Z, Vetere, F & Kelly, D (2017) Augmented Studio: Projection Mapping on Moving Body for Physiotherapy Education. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, USA. [DOI]

Hoang, T, Reinoso, M, Vetere, F, & Tanin, E (2016) Onebody: Remote Posture Guidance System using First Person View in Virtual Environment. In Proceedings of Nordic Conference on Human-Computer Interaction, 2016, Gothenburg, Sweden

Exhibitions

Inside Out, an installation which was part of Science Gallery Melbourne’s BLOOD exhibition, explores projection mapping as a novel interaction technique among visitors in a public space. It is an interactive installation that uses the Augmented Studio platform to project an anatomical model of the cardiovascular system onto the visitor’s body in the installation space. Using a wearable Microsoft band, it tracks the visitor’s pulse rate and projects the beating heart and blood flowing through veins & arteries in real-time.

Inside Out was on display from 14–18 August 2017, at the University of Melbourne. Approximately 600 visitors interacted with the installation during this exhibition.