← All projects
2022Machine Learning

Sign Language Recognition Module

A landmark-based temporal classifier for a limited sign vocabulary, with tools for recording data, training and live feedback.

The module combines MediaPipe landmark extraction with a PyTorch bidirectional LSTM. It supports dataset creation, sequence visualisation and real-time classification for sign-language practice applications.

  • Machine Learning
  • Computer Vision
  • Accessibility Technology

Research and implementation objective

The objective was to build a recognition component for sign-language practice on the augmented mirror. The system works with recorded movement sequences and a defined vocabulary of signs.

Its scope is classification within that vocabulary. It does not perform unrestricted sign-language translation.

Project videosign-language-recognition-module__video_01

Data and application pipeline

MediaPipe extracts body, face and hand landmarks from the camera stream. The coordinates are recorded as temporal sequences and formatted for model training. Shift and scale augmentation expand the recorded examples.

The Python pipeline includes data collection, preprocessing, augmentation, model training or weight loading, and live testing. Visualisation tools show recorded movements and the landmarks detected during practice.

In the tutorial workflow, a user views a demonstrated sign, reproduces it in front of the camera and receives recognition feedback before moving to the next example. The planned JavaScript display integrates this process with the mirror environment.

Image placeholdersign-language-recognition-module__image_01

Model and reported results

The classifier combines a bidirectional LSTM with linear layers, dropout, batch normalisation and ReLU activation. It uses temporal landmark sequences to represent movement over time.

The documented experiment used 1,600 sequences covering 16 signs. The project reported 87% accuracy on its test set and 96% on its training set.

These figures describe the project’s dataset and split. They do not establish performance on unseen signing communities, unrestricted vocabulary or continuous sign-language communication. Broader generalisation would require a separately defined evaluation.

Image placeholdersign-language-recognition-module__image_02