Sign Language Video Game in Augmented Reality
An augmented-mirror visual novel that uses recognised signs as input for dialogue and narrative choices.
Project
This prototype integrates a temporal gesture classifier, animated avatars and a browser-based narrative engine. It explores how short sign-language practice activities can be embedded in a game interaction.
Design question
The project explores how a narrative game can provide a context for practising a small sign vocabulary. Its design uses characters, demonstrations and immediate recognition feedback to structure repeated interaction.
The intended outcome was an engaging practice interface. The implementation alone does not establish an improvement in learning or retention.
sign-language-video-game-ar__video_01Interaction scenario
The application is a short visual novel displayed on the augmented mirror. Users perform signs to respond to characters, interact with objects and select narrative branches.
Avatar and tutorial content
Character animation builds on the interactive-avatar project. MediaPipe landmarks and the Kalidokit mapping pipeline were used to record poses and signs for character sequences and tutorials.
The avatar component was also made available as a separate application within the mirror environment.
Recognition component
The game uses a PyTorch classifier based on a bidirectional LSTM with linear layers, dropout, batch normalisation and ReLU activation. Its dataset tools support recording, sequence visualisation and training.
The associated recognition project reported 87% test accuracy and 96% training accuracy on 1,600 sequences covering 16 signs. Those are classifier results; they are not measures of learning effectiveness or complete-game usability.
Connecting recognition to gameplay
Recognition outputs are transmitted through GOSAI’s Redis database to the narrative engine. When the probability assigned to a sign exceeds a threshold, the engine validates the corresponding action and advances the interaction.
Narrative engine integration
The implementation adapts P5VN to the p5.js-based GOSAI environment. The engine was substantially reworked to operate asynchronously, load and play video animations, and accept sign-driven menu selections.
This connected the computer-vision pipeline with the dialogue state and application display.
Contribution and evaluation scope
The contribution is an integrated prototype connecting gesture recognition, avatar demonstrations and narrative progression. Camera-based input allows the interaction to operate without instrumented gloves.
Technical recognition performance and pedagogical value remain different questions. Assessing the latter would require a study of what users learn, retain and transfer beyond the game’s trained vocabulary and scenarios.