Dynamic Training in AR for Musical Practice
Augmented-mirror prototypes for musical practice, exploring gesture control, pitch estimation and visual feedback.
Project
This work investigates how an augmented mirror can support pitch awareness and musical practice. An initial gesture-controlled theremin prototype exposed latency constraints and led to a microphone-based training interface.
Research objective
Musical practice involves connecting an intended sound with a physical action and the resulting pitch. This project explored visual feedback for that process, with applications to singing and theremin practice.
The augmented mirror provided a surface for displaying guidance while users remained standing and could see their own movement. The design question was how to make pitch and performance feedback legible during practice.
Platform and interaction constraints
The application was developed as a module for the augmented-mirror platform. Its initial interaction used camera-based hand tracking, allowing users to control sound without holding a physical controller.
Gesture-controlled theremin
The first prototype mapped the height of the left hand to gain and the horizontal position of the right hand to pitch. Optional visual bars indicated the available frequency range and volume limits.
Pose estimates positioned the user’s hands relative to the interface. This produced a playable demonstration of a virtual theremin on the mirror.
dynamic-training-ar-musical-practice__image_01Visual feedback
A marker followed each hand to show its estimated position in the pitch and gain controls. The sound was produced by a synthesiser integrated with the mirror’s application environment.
dynamic-training-ar-musical-practice__image_02Latency and design revision
The pose-estimation pipeline introduced a perceptible delay between physical movement and the visual and auditory response. This made rhythmic playing and precise pitch selection difficult in the prototype.
This observation motivated a change in interaction design: the revised application measured sound produced by the user, reducing its dependence on immediate gesture-to-sound mapping.
Microphone-based pitch training
The revised module estimates the pitch of a sung or played note and represents it on a horizontal frequency display. Users can enable note guides, select exercises and follow target notes moving through the interface.
During an exercise, colour changes indicate agreement between the estimated pitch and the target. A score summarises the exercise at its end. These features provide feedback during practice; their effect on learning is a separate evaluation question.
This version retained the mirror as a practice surface while shifting the primary input from hand position to the audio signal.
Software architecture
The application runs in GOSAI with a JavaScript and p5.js display. A microphone driver captures audio and a frequency-estimation module extracts pitch, gain and related parameters.
The interface separates audio analysis from rendering so that the display can update from published features rather than processing the microphone stream directly.
dynamic-training-ar-musical-practice__image_03Data flow and application controls
Audio features are published through Redis. A Python processing component retrieves them and forwards updates to the browser through Socket.IO. The display uses those updates to drive exercises and feedback.
The project also introduced submenu controls for enabling display options within GOSAI applications. The work established a functional practice interface and identified latency, pitch estimation and calibration as relevant constraints for subsequent development.