← All projects
2023Augmented Reality

Sign Language Video Game in Augmented Reality

An augmented-mirror visual novel that uses recognised signs as input for dialogue and narrative choices.

This prototype integrates a temporal gesture classifier, animated avatars and a browser-based narrative engine. It explores how short sign-language practice activities can be embedded in a game interaction.

  • Augmented Reality
  • Game Development
  • Sign Language

Design question

The project explores how a narrative game can provide a context for practising a small sign vocabulary. Its design uses characters, demonstrations and immediate recognition feedback to structure repeated interaction.

The intended outcome was an engaging practice interface. The implementation alone does not establish an improvement in learning or retention.

Project videosign-language-video-game-ar__video_01

Interaction scenario

The application is a short visual novel displayed on the augmented mirror. Users perform signs to respond to characters, interact with objects and select narrative branches.

Avatar and tutorial content

Character animation builds on the interactive-avatar project. MediaPipe landmarks and the Kalidokit mapping pipeline were used to record poses and signs for character sequences and tutorials.

The avatar component was also made available as a separate application within the mirror environment.

Recognition component

The game uses a PyTorch classifier based on a bidirectional LSTM with linear layers, dropout, batch normalisation and ReLU activation. Its dataset tools support recording, sequence visualisation and training.

The associated recognition project reported 87% test accuracy and 96% training accuracy on 1,600 sequences covering 16 signs. Those are classifier results; they are not measures of learning effectiveness or complete-game usability.

Connecting recognition to gameplay

Recognition outputs are transmitted through GOSAI’s Redis database to the narrative engine. When the probability assigned to a sign exceeds a threshold, the engine validates the corresponding action and advances the interaction.

Narrative engine integration

The implementation adapts P5VN to the p5.js-based GOSAI environment. The engine was substantially reworked to operate asynchronously, load and play video animations, and accept sign-driven menu selections.

This connected the computer-vision pipeline with the dialogue state and application display.

Contribution and evaluation scope

The contribution is an integrated prototype connecting gesture recognition, avatar demonstrations and narrative progression. Camera-based input allows the interaction to operate without instrumented gloves.

Technical recognition performance and pedagogical value remain different questions. Assessing the latter would require a study of what users learn, retain and transfer beyond the game’s trained vocabulary and scenarios.