SMART THINKER
A real-life tragedy prompted this specialist in virtual reality to develop an emotion-sensitive space where professionals can ‘meet with’ those who need their help.
The death of a respected psychiatrist motivated Dr Sungchul Jung to improve the safety of psychiatry appointments.
The death of a respected psychiatrist motivated postdoctoral fellow Dr Sungchul Jung to improve the safety of psychiatry appointments by bringing them into the virtual world.
In late 2018, suicideprevention psychiatrist Lim Se-won accepted a patient’s request for a consultation at Kangbuk Samsung Hospital in Seoul, despite the patient having not made an appointment. It was a fateful decision – the patient brought a knife to the consultation and fatally stabbed his doctor. The tragedy sent shockwaves through the South Korean medical community and prompted calls for measures to guarantee the safety of medical staff at hospitals.
Jung was in South Korea at the time, visiting his homeland on a break from his research at the University of Canterbury’s Human Interface Technology Laboratory (HIT Lab NZ). Jung’s work in the field of mixed reality (AR/VR) and humancomputer interaction (HCI) sees him explore the possibilities of socialisation within the digital domain.
Prior to hearing of the incident at Kangbuk Samsung Hospital, Jung had been aware of research into VR therapy and had considered its potential. But the death of Lim made him realise how little research exists into how VR could be used to better protect therapists. If Lim had been able to assess the patient remotely and detect his troubled mental state from afar using ‘telepresence’ technology, the tragedy might have been avoided.
Telepresence in virtual reality sees people wearing VR headsets meet in a virtual space where their digital avatars can interact. The latest VR technology includes headsets and handheld controllers that use cameras and sensors to track head and hand movements, and Jung is exploring the potential for this technology to be used to determine a person’s emotional state. He is working on a Science for Technological Innovation (SfTI) two-year seed project in which he will develop an immersive experience for a service provider to safely carry out ‘sensitive communications’ – such as between a psychiatrist and a person suffering a mental disorder, or a lawyer and a prisoner – in virtual space as if meeting in real world, real time.
In addition to creating an effective immersive telepresence system, Jung will develop an emotion detector and analyser. In the case of doctor-patient interactions, the service provider will be able to see the emotiondetection data from the headsetwearing patient through a monitor. The software would deliver mental-status updates to the doctor and track the patient’s status, informing the doctor’s treatment programme.
Previous research has examined what verbal patterns, eye movements, body language and gestures reveal about a person’s mental state. Meanwhile, physiological signals such as those detected by an EEG, which records brain activity, or an ECG, which records heart activity, can also be indicators for human emotion. Jung’s project will see him develop a platform to gather such gesture and biological data, and present them in a format that can be used by doctors or therapists in their treatment of patients. The project will also explore ways of improving the accuracy of facial-expression recognition.
Jung says there are a number of challenges to overcome before this technology can be put to practical use. “One of the technical questions is, how can we implement the immersive telepresence system in real time with very low latency between data transmitting from doctor to patient?” says Jung. “Then, does the immersive telepresence provide a feeling of trust to the patient as if they are meeting a doctor in the real world?”
The third, most challenging hurdle is being able to accurately classify human emotions based on cues such as facial expressions and physiological signals. For that reason, while virtual therapy has the potential to have many applications, Jung doesn’t foresee it completely replacing in-person consultations. “It’s very hard to convey subtle emotional exchanges with each other, and to share situational changes like temperature, noise or lighting in VR,” he says. “Those things can all be strongly related to the conversation, and thus missing those components can impact on the quality of the therapy.”
Jung hopes VR can be utilised to improve and save lives, but he doesn’t want to see it replace human social activities. He does believe the technology he is developing could be applied to industries looking to improve the quality of online meetings, especially in this era of remote working and learning. He’s also motivated to improve VR technology to help people connect with distant family and friends. “I believe the most exciting part of VR is its unique experience in the created or replicated environment with other users,” says Jung.
“Using VR technology, people have more opportunities for education, training or knowledge, and a better way to meet up with loved ones regardless of the physical distance.”
“IT’S VERY HARD TO CONVEY SUBTLE EMOTIONAL CHANGES.”