Interactive Avatar with Pose Estimation
Real-time browser-based avatar animation from facial, body and hand landmarks estimated with MediaPipe.
Project
ESLA is a JavaScript prototype for driving a VRM avatar from a webcam. The work combines MediaPipe tracking, Kalidokit animation parameters and Three.js rendering, with integration into the augmented-mirror environment.
Project objective
This project implements a webcam-driven avatar that runs in a web browser. It provides a reusable basis for interactive applications that need a visual representation of body movement, facial expression and hand articulation.
The accompanying tutorial documented how camera input could be transformed into avatar animation using existing tracking and rendering components.
interactive-avatar-pose-estimation__image_01Tracking approach
Camera-based landmark estimation supplies an alternative input to pre-authored animation. The project uses MediaPipe to extract movement information from the face, body and hands and update a character during interaction.
The work focuses on integrating these estimates with a browser renderer. It does not introduce a new pose-estimation model.
interactive-avatar-pose-estimation__image_02Avatar representation
The application maps the camera user’s movement onto a humanoid character. This allows head movement, expression and selected body gestures to be displayed through an avatar rather than a conventional camera image.
The resulting module was developed for reuse in the augmented-reality communication project and in demonstrations and tutorials for sign-language applications.
Animation pipeline
The implementation connects three stages: landmark detection, conversion of landmarks into animation parameters, and rendering of the character. Kalidokit provides the mapping to rotations and expression parameters for a VRM model.
This approach supports hand and finger articulation as well as facial and body movement. It relies on the estimates available from the webcam view rather than dedicated motion-capture hardware.
Browser and mirror integration
The application is implemented in JavaScript. Three.js renders the character in a browser canvas, while MediaPipe and Kalidokit supply tracking and animation inputs. The module was integrated with GOSAI for use on the augmented mirror.
The contribution is a working integration and a documented implementation workflow. The fidelity of the avatar depends on the tracking pipeline, camera view and mapping to the selected character; the project does not establish equivalence to specialist motion capture.