Audio-to-Sign and Sign-to-Audio: A Real-Time Bidirectional Communication System for the Deaf and Mute Community

Abstract

Over 430 million individuals worldwide live with disabling hearing loss, yet the majority of assistive communication tools address only one direction of translation—converting either speech to text or gestures to text, but never both within a single, affordable platform. This paper presents a real-time, bidirectional communication system that bridges the gap between the deaf/mute and hearing populations. The proposed Python-based application integrates two parallel translation pipelines: an Audio-to-Sign module that converts spoken language into pre-recorded Indian Sign Language (ISL) animations via automatic speech recognition, and a Sign-to-Audio module that maps hand gestures—detected through HSV skin segmentation and Hu-moment feature extraction—into synthesized speech. A Flask web interface unifies both modes under a single user experience. Experimental evaluation with 15 participants across five phrase categories demonstrates 91.2% online speech recognition accuracy, 84.2% gesture recognition accuracy under optimal lighting, and an end-to-end latency of 1.25 seconds on mid-range hardware—within the conversational threshold. User acceptance testing yielded an average satisfaction score of 4.3 out of 5. The system operates entirely on standard webcams and microphones, requires no specialized hardware, and supports an offline fallback mode that preserves user privacy. This work represents a practical, open-source step toward equitable assistive communication technology.