← All projects
2022Motion Tracking

Interactive Avatar with Pose Estimation

Real-time browser-based avatar animation from facial, body and hand landmarks estimated with MediaPipe.

ESLA is a JavaScript prototype for driving a VRM avatar from a webcam. The work combines MediaPipe tracking, Kalidokit animation parameters and Three.js rendering, with integration into the augmented-mirror environment.

  • Motion Tracking
  • Computer Vision
  • Animation

Project objective

This project implements a webcam-driven avatar that runs in a web browser. It provides a reusable basis for interactive applications that need a visual representation of body movement, facial expression and hand articulation.

The accompanying tutorial documented how camera input could be transformed into avatar animation using existing tracking and rendering components.

Image placeholderinteractive-avatar-pose-estimation__image_01

Tracking approach

Camera-based landmark estimation supplies an alternative input to pre-authored animation. The project uses MediaPipe to extract movement information from the face, body and hands and update a character during interaction.

The work focuses on integrating these estimates with a browser renderer. It does not introduce a new pose-estimation model.

Image placeholderinteractive-avatar-pose-estimation__image_02

Avatar representation

The application maps the camera user’s movement onto a humanoid character. This allows head movement, expression and selected body gestures to be displayed through an avatar rather than a conventional camera image.

The resulting module was developed for reuse in the augmented-reality communication project and in demonstrations and tutorials for sign-language applications.

Animation pipeline

The implementation connects three stages: landmark detection, conversion of landmarks into animation parameters, and rendering of the character. Kalidokit provides the mapping to rotations and expression parameters for a VRM model.

This approach supports hand and finger articulation as well as facial and body movement. It relies on the estimates available from the webcam view rather than dedicated motion-capture hardware.

Browser and mirror integration

The application is implemented in JavaScript. Three.js renders the character in a browser canvas, while MediaPipe and Kalidokit supply tracking and animation inputs. The module was integrated with GOSAI for use on the augmented mirror.

The contribution is a working integration and a documented implementation workflow. The fidelity of the avatar depends on the tracking pipeline, camera view and mapping to the selected character; the project does not establish equivalence to specialist motion capture.