Intelligent Glasses Sign Language Interaction System Based on Embedded Multimodal Perception

DOI: https://doi.org/10.62517/jbdc.202501419

Author(s)

Zhiyuan Li*, Xiangxuan Ji, Mengmei Wang, Yi Huang, Liyuan Liu, Xiuling Li, Ziru Guo

Affiliation(s)

School of Artificial Intelligence and Software, Ke Wen College of Jiangsu Normal University, Xuzhou, Jiangsu, China *Corresponding Author

Abstract

Addressing the information interaction barriers faced by the Deaf and Hard of Hearing (DHH) community, who encounter difficulties in converting sign language to text, and the Blind and Visually Impaired (BLV) community, who lack sufficient environmental perception, this paper proposes an intelligent glasses sign language interaction system based on embedded multimodal perception. This system integrates Leap Motion SDK, lightweight spatiotemporal graph convolutional networks, computer vision, and intelligent audio processing technologies. It optimizes algorithms to simplify the user experience, thereby building a communication bridge between the DHH community and hearing individuals. Simultaneously, it enhances the convenience of life for the BLV community, ultimately creating a fully accessible and inclusive environment.

Keywords

Barrier-Free Interaction; Multimodal Fusion; Sign Language Recognition; Tactile Feedback; Edge Intelligence

References

[1]Liang Jun. Research on Human Action Recognition Based on Spatio-Temporal Graph Convolution. Central South University for Nationalities, 2024 [2]Wang Qiang, Wang Shuai, Hu Minghuan. Exploration of Dynamic Gesture Recognition Based on Improved YOLOv7. Heilongjiang Science, 2025, 16(04): 94-96+100 [3]Liu Xingzheng. Research on Gesture Recognition Algorithm Based on Improved YOLOv7. Anhui University of Science and Technology, 2023 [4]Zhang Yanhui. Research on the Principle of Stereo Binocular Vision and Its Application in Gesture Recognition. Beijing University of Chemical Technology, 2016 [5]Tu Chong, Jin Liying, Wang Zhongren, et al. Overview of Speech Recognition Technology and Its Applications. Digital Technology and Application, 2025, 43(09): 179-181 [6]Wang Jingyao, Fan Fei, Liu Haoyu, et al. Deaf and Dumb Sign Language Recognition Based on Machine Vision - Voice Interaction System. Internet of Things Technology, 2021, 11(12): 3-5 [7]Zhao Zhiqi. Research on Visual SLAM Algorithms Based on Deep Learning. University of Electronic Science and Technology of China, 2025 [8]Fan Yuhan, Li Qianqian, Meng Xue, et al. Design and Implementation of a Gesture Recognition System Based on Binocular Vision Principle. Intelligent Internet of Things Technology, 2024, 56(06): 81-84 [9]Zhou Kang, Fan Guangyu, Rao Lei, et al. VSLAM Algorithm Based on Image Enhancement and Feature Matching. Journal of Beijing University of Posts and Telecommunications, 2025, 48(03): 53-59 [10]Wu Chundi. Research on Dynamic Sign Language Recognition Algorithm Based on Deep Learning. Shenyang University of Technology, 2024