Multimodal Attention Network for Trauma Activity Recognition from Spoken Language and Environmental Sound.

Trauma activity recognition aims to detect, recognize, and predict the activities (or tasks) during a trauma resuscitation. Previous work has mainly focused on using various sensor data including image, RFID, and vital signals to generate the trauma event log. However, spoken language and environmental sound, which contain rich communication and contextual information necessary for trauma team cooperation, are still largely ignored. In this paper, we propose a multimodal attention network (MAN) that uses both verbal transcripts and environmental audio stream as input; the model extracts textual and acoustic features using a multi-level multi-head attention module, and forms a final shared representation for trauma activity classification. We evaluated the proposed architecture on 75 actual trauma resuscitation cases collected from a hospital. We achieved 72.4% accuracy with 0.705 F1 score, demonstrating that our proposed architecture is useful and efficient. These results also show that using spoken language and environmental audio indeed helps identify hard-to-recognize activities, compared to previous approaches. We also provide a detailed analysis of the performance and generalization of the proposed multimodal attention network.
AuthorsYue Gu, Ruiyu Zhang, Xinwei Zhao, Shuhong Chen, Jalal Abdulbaqi, Ivan Marsic, Megan Cheng, Randall S Burd
JournalIEEE International Conference on Healthcare Informatics. IEEE International Conference on Healthcare Informatics (IEEE Int Conf Healthc Inform) Vol. 2019 (Jun 2019) ISSN: 2575-2626 [Print] United States
PMID32201857 (Publication Type: Journal Article)

Join CureHunter, for free Research Interface BASIC access!

Take advantage of free CureHunter research engine access to explore the best drug and treatment options for any disease. Find out why thousands of doctors, pharma researchers and patient activists around the world use CureHunter every day.
Realize the full power of the drug-disease research graph!

Choose Username:
Verify Password:
Enter Code Shown: