An Optimized Deep Learning Framework for Continuous Sign Language Recognition

Abstract

Sign language is a form of movement language that conveys semantic information through hand and arm motions, facial expressions, and head/body postures, serving as a crucial communication medium for the deaf community. Researchers are motivated by the desire to integrate deaf newlineindividuals into mainstream society, leading to a growing interest in automatic sign language recognition systems. This recognition involves interpreting static or dynamic signing within the one-arm distance 3D space around the upper body of the signer.In this work, a comprehensive literature review is conducted within the domains of visionbased Continuous Sign Language Recognition (CSLR) and Sign Language Translation (SLT). The deep Learning (DL) strategy is adopted by all the recent works. Any DL-based CSLR framework has three main modules - feature extraction, sequence learning and alignment learning. Feature extraction is usually done by a CNN. Most of the works have used LSTMs for sequence learning. Notably, it has been observed that the latest Transformer model and its variants are under-explored for these tasks. Furthermore, there is a gap in the literature concerning the investigation of position encoding schemes specific to the Transformer architecture, which is particularly valuable as the architecture lacks inherent sequential information. Therefore, an extensive literature study is conducted on Transformers, their variants, and the available position encoding schemes.This research began with the exploration of new positioning schemes for the Transformer model within the context of CSLR and SLT. Consequently, a novel positioning scheme was introduced, utilizing Gated Recurrent Unit (GRU) as the relative position encoder, and the multi-head attention (MHA) mechanism was modified to integrate relative position embeddings. The resulting Transformer, incorporating both positioning schemes, is referred to as GRU-RST. Furthermore, it was demonstrated that relative positioning outperformed absolute position encoding for Transformer ...