Static and Dynamic Hand Gesture Recognition for Indian Sign Language

Abstract

newline Sign Language Recognition is one of the most recent and challenging applications in the Human Centered Computing. This thesis presents vision based system that can recognize static and dynamic gestures of Indian Sign Language (ISL) and translate them into text and voice. To recognize static gestures, proposed system is first trained using various features like Local Binary Patterns, Histogram of Oriented Gradients and Speeded Up Robust Features. We trained four different classifiers using these features, including the Support Vector Machine (SVM), Artificial Neural Network, K-Nearest Neighbours, and Linear Discriminant Analysis. An RGB camera and a Microsoft kinect sensor have been used as input devices. The proposed system provided accuracy of 99.49% and has performed well even with images with complicated backgrounds. We also introduced a hybrid feature vector by combining Fourier Descriptors, Hu Moments and Zernike Moments. SVM classifier is trained using feature vectors provided invariant with respect to transformation with accuracy of 95.79%. We have also developed an efficient and accurate system using Convolutional Neural Network (CNN) having recognition rate of 99.44%. SVM classifier trained using deep features extracted from our proposed CNN network provided accuracy of 99.75%. In addition, using our ISL dataset, two popular image classification networks, GoogLeNet and VGG16, were retrained and evaluated. The accuracy of GoogLeNet and VGG16 was 99.54% and 99.59%, respectively. Dynamic gesture recognition is challenging task because of the dynamic behavior of gestures. We have also proposed a unified architecture by combining CNN and Long Short-Term Memory (LSTM) network to accurately recognize ISL dynamic hand gestures. newline