A framework for continuous indian sign language recognition using computer vision

Abstract

Sign language is a non-vocal, visually oriented natural language used by the hearing newlineimpaired and the hard-for-hearing part of society. It combines multiple modalities newlinelike hand movements, facial expressions and body poses. Static gestures involve basic finger movements such as numbers and alphabets, dynamic signs include words, and a sign sentence consists of grammatically connected and meaningful dynamic words. Sign Language Translation (SLT) models have been an actively evolving research topic under computer vision. One of the most challenging aspects in earlier iterations of SLTs was accurately capturing the intricate and constantly changing hand movements and facial expressions characteristic of sign language. newlineHowever, the advent of deep learning models has facilitated significant advancements in the field, particularly in the realm of continuous sign language translation. newlineThe research endeavours to develop a lightweight deep-learning framework newlinespecifically tailored for the translation of Indian Sign Language (ISL) into text and newlineaudio. The proposed framework introduces two collaborative deep-learning components that extract and classify features synergistically. The ISL video sequence serves as the input, which undergoes feature extraction utilizing the Inception V3 architecture, enabling the extraction of features from each frame. Classification models tend to be bulky and intricate, consuming substantial memory space and requiring extended training periods. This challenge has been addressed by introducing a lightweight LSTM model, which effectively utilizes the feature map generated by the Inception model for accurate classification. It is important to note that each sign possesses unique characteristics yet exhibits similar feature maps. The performance of the framework is assessed based on the speed and accuracy achieved in converting the input video into text and audio formats.