Object Tracking and Gesture Recognition for Augmented Reality

Abstract

newlineAugmented Reality (AR) is relatively new domain that deals with rendering real objects in virtual scenes. This research explores one of important application of AR in dance choreography. In dance choreography especially for Indian classical dances like Bharatanatyam there are very few teachers outside India and for Indian diaspora in various countries, it becomes difficult to learn their cultural dance perfectly. With lack of trained professionals, learning becomes challenging and Augmented Reality (AR) based learning is a solution. Learning through AR helps the students with external amplification, internal rewards, challenge, and increased self-confidence. Recognizing Bharatanatyam gestures accurately is one of important challenge in designing augmented reality systems for Indian classical dance. This research address this problem and proposes different deep learning based solutions to recognize the Bharatanatyam gestures i.e., mudras. newlineFrame processing technique has been proposed with active time identification to optimize the latency of CNN in the light of action mapping in Augmented Reality. The technique is evaluated by applying it to the Bharatnatyam mudras, classifying and comparing it with the obtained results with that of the traditional convolution neural network. newlineThe deep learning strategy combining scale invariant feature transform features with convolutional neural network to address the challenge in accurate recognition of mudras is being proposed. The gestures are segmented using active net based segmentation model to reduce the influence of background in gesture recognition. Convolutional neural network with structural conflict minimization kernel is used to classify the gestures. The proposed solution is able to provide an average accuracy of 95% in classification of mudras and it has lower false positives of 2%. newlineFor accurate classification of mudras, the deep learning multi scale feature guided aggregated signature is proposed. The deep learning multi scale features are extracted fr