Augmented Mirror
A modular augmented mirror combining body tracking, depth sensing and browser-based interactive applications.
Project
Second Self investigates how digital feedback can be aligned with a person’s reflection. The project combines a computer-vision pipeline with a modular application environment for movement-based interaction, music practice and sign-language learning.
View code ↗Research context
An augmented mirror can place visual instructions alongside a person’s own reflection. This project investigated the technical foundations of that interaction: tracking a person, aligning graphics with the reflected body and making those capabilities available to different applications.
The platform supported prototypes for movement exercises, sign-language practice and musical training. These applications provided concrete use cases for testing the interface and its underlying architecture.
augmented-mirror__video_01Tracking and spatial alignment
A pose-estimation model identifies landmarks on the body, face and hands. An Intel RealSense D435i supplies depth information so that image coordinates can be related to the person’s position in physical space.
The pipeline transforms these coordinates to align digital content with the reflection. Computer-vision processing runs in a Docker container, while a separate server forwards the resulting coordinates to a browser-based display. This separation supports deployment and application development independently of the tracking process.
Sign-language practice module
One application presents a demonstration of a sign and asks the user to reproduce it. Live landmark visualisation provides feedback on the movement detected by the camera; the tutorial advances after the action has been recognised.
The associated recognition pipeline uses shift and scale augmentation to expand its recorded training examples. Recognition and training are described in more detail on the Sign Language Recognition Module project page.
Interaction prototypes
Three minigames exercised the tracking pipeline. Poke It asks users to reach on-screen targets within a time limit. Bounce maps hand movement to interaction with a simulated ball. Dance compares the user’s movement with a displayed sequence and advances the demonstration as movements are matched.
These prototypes explored how tracking could drive different forms of feedback. They served as interaction demonstrations rather than a controlled evaluation of training effectiveness.
augmented-mirror__image_01augmented-mirror__image_02augmented-mirror__image_03Contribution and scope
The main outcome was a shared technical basis for augmented-mirror applications: pose tracking, spatial alignment, communication between processes and browser rendering. The work also informed the avatar, sign-language and music-practice projects presented on this site.
The prototype demonstrates the feasibility of these interactions. Application-specific questions, including learning outcomes and sustained use, require evaluation beyond the platform demonstrations.
Selected references
Blum, Kleeberger, Bichlmeier and Navab. “mirracle: An augmented reality magic mirror system for anatomy education.” IEEE Virtual Reality Workshops, 2012. DOI: 10.1109/VR.2012.6180909.
Vera, Gimeno, Coma and Fernández. “Augmented Mirror: Interactive Augmented Reality System Based on Kinect.” INTERACT, 2011.
Anderson, Grossman, Matejka and Fitzmaurice. “YouMove: enhancing movement training with an augmented reality mirror.” UIST, 2013.
Implementation references include the MediaPipe Holistic framework and Intel RealSense depth sensing. These provided the tracking and depth information used by the prototype.