Augmented learning environment for physiotherapy education

Project overview

Physiotherapy  is currently taught via multiple modalities, such as lectures, tutorials,  demonstrations and hands-on training in practical classes. However,  physiotherapy students often struggle to translate anatomical knowledge from  textbooks to dynamic understanding of the mechanics of body movements in real  life patients. Moreover, one of the critical skills in physiotherapy, and other  clinical domains, is clinical reasoning. Students must learn to negotiate the  complexity of multiple factors, starting with the mastery of strong domain knowledge, such as human anatomy, to forming hypotheses through patient inquiry  and testing the hypotheses through examination. Clinical reasoning is particularly  difficult to teach because it requires students to critically review clinical  decisions which are often ephemeral and tacit. The best practice for learning  such skills typically involves a class of students observing an expert  therapist conducting a consultation with a real patient (or a surrogate  patient). However, such settings do not make the therapist’s clinical decision making explicitly available to students. One of the major difficulties for  students in learning clinical reasoning is the ability to identify medical cues  to develop diagnostic hypotheses.

Augmented  Studio setup in the classroom Augmented  Studio setup in the classroom.

This project aims to enhance student learning in physiotherapy via an augmented  learning environment titled the ‘Augmented Studio’ developed using projectors  and Microsoft Kinect sensors to enable projections of an anatomy model onto a  moving body for practical physiotherapy classes. Augmented Studio enables  augmentation through projection mapping to turn a human body into a display  surface, showing anatomical information such as muscles and skeleton in real  time. The moving body of a volunteer student becomes a live canvas for the teacher to illustrate clinical reasoning. Using a tablet interface with pen  input, the instructor can draw annotations directly on the virtual body, which  are rendered in real time on the patient’s body.

Currently  the instructor uses still photos, figures, and diagrams which do not capture  the real time and complex dynamics of human movements. With the Augmented  Studio, the researchers explore the capability of creating virtual annotations,  to identify certain anatomical structures or to illustrate the joints or muscle  connections to teach students. The annotations enhance teacher and student communication with the purpose of transferring the real life knowledge and  experience of the teacher to a class of students.

Teacher and student interact with annotation on the volunteer Teacher and student interact with annotation on the volunteer

Augmented  reality based learning and teaching presents an innovative pedagogical practice  delivering benefits to both the staff and the students. The student learning  experience is enhanced in multiple ways with the usage of the augmented reality  learning environment. Augmented kinaesthetic information and instructor annotations support interactive observation for better clinical understanding  of human anatomy and musculoskeletal structures. The ability to track a full  body movement in 3 dimensions and examine the changes throughout the range of  movement will enhance appreciation by students of the dynamic change in the anatomical  configuration of the body.

The  project aims to deliver positive learning outcomes for the students by enabling  experiential learning and increased connection with the learning materials,  lessons and the instructor, catering for both tactile and visual learning, and  encouraging cognitive and clinical reasoning.


This  project is a collaboration between the Microsoft Research Centre for Social  Natural User Interfaces (SocialNUI) in the School of Computing and Information  Systems and the Department of Physiotherapy at the University of Melbourne.

The project is supported by an ARC Discovery  Project (DP130102142) and a University of Melbourne Learning and  Teaching Initiative (LTI) Grant 2016.

Project team

  • Hasan Shahid Ferdous, Research Fellow, Microsoft Research Centre for SocialNUI, The University of Melbourne
  • Thuong Hoang
    Thuong Hoang, Lecturer, School of Information Technology, Deakin University
  • Frank Vetere
    Frank Vetere, Professor, The University of Melbourne
  • Zaher Joukhadar, Lead Software Engineer, Microsoft Research Centre for SocialNUI, The University of Melbourne
  • Martin Reinoso, PhD Candidate, Microsoft Research Centre for SocialNUI, The University of Melbourne
  • Louisa Remedios, Associate Professor, Department of Physiotherapy, The University of Melbourne
  • David Kelly, Lecturer, Department of Physiotherapy, The University of Melbourne
  • Egemen Tanin, Associate Professor, School of Computing and Information Systems, The University of Melbourne

Contact details


Hoang, T., Ferdous, H., Vetere, F. & Reinoso, M. (2018) Body as a Canvas: An exploration on the role of the body as display of digital information. In Proceedings of the 2018 Designing Interactive Systems (DIS) Conference 2018, pp. 253-263. DOI: 10.1145/3196709.3196724

Kelly, D, Hoang, T, Joukhadar,  Z, Reinoso, M, Clements, T & Vetere, F (2017) Usage of an augmented  reality system to facilitate dynamic anatomical understanding for physiotherapy  students. Australian Physiotherapy  Association Conference (APA 2017), Sydney, Australia.

Hoang, T, Reinoso, M, Joukhadar,  Z, Vetere, F & Kelly, D (2017) Augmented  Studio: Projection Mapping on Moving Body for Physiotherapy Education. In Proceedings of the 2017 CHI Conference on  Human Factors in Computing Systems, Denver, USA.  [DOI]

Hoang, T, Reinoso, M, Vetere, F, & Tanin, E (2016) Onebody: Remote Posture Guidance System using First Person View in Virtual Environment. In Proceedings of Nordic Conference on Human-Computer Interaction, 2016, Gothenburg, Sweden


Inside Out, an installation which was part of Science Gallery Melbourne’s BLOOD exhibition, explores projection mapping as a novel interaction technique among visitors in a public space. It is an interactive installation that uses the Augmented Studio platform to project an anatomical model of the cardiovascular system onto the visitor’s body in the installation space. Using a wearable Microsoft band, it tracks the visitor’s pulse rate and projects the beating heart and blood flowing through veins & arteries in real-time.

Inside Out was on display from 14–18 August 2017, at the University of Melbourne. Approximately 600 visitors interacted with the installation during this exhibition.

Physio Ed