VTM-1.5.1

Turn one anime picture into a live VTuber. A keypoint-driven DiT draws your character in whatever pose and expression your face gives it, frame by frame.

These are the weights behind VTM Spark, the free Windows app that tracks you with a webcam or iPhone and sends the character to OBS, Discord or Zoom as a webcam.

Status: beta · Platform: Windows + NVIDIA CUDA

One still in	Animated by this model

_{Every frame on the right was drawn by VTM-1.5.1.pt from the one picture on the left, recorded straight from the VTM Spark desk with Max FPS at 100 (no cap in practice) and the Ultra decoder on an RTX 5060 Ti: about 60 frames a second shown. The head motion is a recorded iPhone (iFacialMocap) session replayed through the app instead of a live face.}

How it works

Webcam or iPhone, then Track Lab, then 37 keypoints, then the VTM-1.5.1 DiT (fed with your character picture), then the SD VAE, then the VTM Spark camera

Use it

The easy way is VTM Spark. It downloads everything in this repo for you:

Download VTM Spark from vtmstudio.dev/download.
Double-click install.bat.
Double-click run.exe, add your character picture, start tracking, and pick the VTM Spark camera in OBS / Discord / Zoom.

You'll need Windows 10/11 and an NVIDIA RTX 30, 40 or 50 series GPU.

Your character picture

The model expects a still laid out like the one above:

square, 768 × 768, solid green #00FF00 background
chest-up, centred, facing the camera, chin level
both eyes open, small closed-mouth smile
no hands, props, text or extra people

VTM Spark ships this blueprint and a ready-to-paste image-AI prompt in character-blueprint/.

What's in this repo

File	Size	What it is
`VTM-1.5.1.pt`	360 MB	The generator: keypoint-conditioned DiT (this model)
`decoder/vtm-fast-decoder.pt`	4 MB	Ultra fast decoder: turns the DiT's latents into the 768 × 768 frame for live streaming
`trackers/iris_pose.pt`	6 MB	Iris / pupil tracker on the character still (YOLO-pose fine-tune)
`trackers/dwpose_v2.pt`	23 MB	Upper-body keypoints on the still (YOLO-pose fine-tune)
`trackers/animeseg_hair3.pt`	432 MB	Hair-part segmentation (Mask2Former fine-tune), so hair follows the head
`trackers/pose_landmarker_lite.task`	6 MB	MediaPipe pose landmarker for body tracking
`openseeface/*`	21 MB	OpenSeeFace webcam face tracking models
`media/*`		Demo images for this card

VTM Spark places them under models/dit/, models/decoder/, models/trackers/ and vendor/tools/openseeface/models/. On first setup it also fetches a few models straight from their publishers (SD VAE, a tiny VAE and two anime-face detectors), about 1.24 GB in all.

Model

VTM-1.5.1.pt

Architecture: DiT, 30.3M parameters (hidden 320, depth 10, 5 heads, patch 4 on SD-VAE latents), with RoPE on keypoints, QK-norm, SwiGLU and RMSNorm
Output: 768 × 768 through the SD VAE, or through the Ultra fast decoder while streaming
Pose input: 37 keypoints covering the face outline, brows, eyes, irises, nose, mouth and upper body
Identity input: one reference image of the character, as image tokens plus face tokens
Sampling: rectified flow, distilled to run in 1 step from a 30-step teacher (VTM-1.5.1-SFT, a fine-tune of the 1.5.1 base). Guidance is baked into the weights, so run it at 1 step, pose CFG 1.0, identity CFG 1.0 (VTM Spark's defaults)
Speed: built for live use in VTM Spark. On an RTX 5060 Ti the model plus Ultra decoder draws about 117 frames a second on the GPU; the desk shows about 60–65, limited by the app's CPU-side work. Frame rate depends on the GPU and the app's Batch, Inbetweens and Max FPS settings.

The previous VTM-1.5.1.pt (distilled from a 10-step teacher, 1 or 2 steps, CFG not baked in) is still in this repo's commit history.

Ultra fast decoder

decoder/vtm-fast-decoder.pt is a small PixelShuffle decoder distilled from the Hybrid TinyVAE. It decodes the same SD latents in one pass (a whole live frame takes about 11 ms instead of 17 on an RTX 5060 Ti) and is what VTM Spark's "Ultra fast" stream mode runs. Without it the app falls back to the TinyVAE.

Data

Training used a small private character set:

~1,500 characters
~16–30 images per character

Generation quality in this release is limited mainly by model capacity (DiT-30M) and dataset scale.

Download

hf download sinBoo1/VTM-Spark VTM-1.5.1.pt --local-dir ./VTM-Spark

from huggingface_hub import hf_hub_download

ckpt = hf_hub_download(
    repo_id="sinBoo1/VTM-Spark",
    filename="VTM-1.5.1.pt",
)

Everything at once: hf download sinBoo1/VTM-Spark --local-dir ./VTM-Spark

The runtime (live camera → keypoints → this model → virtual camera) is VTM Spark.

Limitations

Beta: soft detail, identity drift and pose errors happen
Small data and DiT-30M capacity limit image quality
Windows + NVIDIA CUDA only (no AMD, macOS, Linux or CPU)
Framing: torso-up only (roughly head to mid-torso). Legs and most of the waist are not supported
Hands: not supported
Character types not supported: realistic humans; non-humanoid / furries
Accessories: glasses and hats generally work; most other accessories are not supported

Intended use

Live VTubing and research on pose → image pipelines (live drive, pose retargeting). Not a finished production renderer.

Licences

No single licence covers every file here, so this repo is tagged license: other. Apache License 2.0 covers our training work: VTM-1.5.1.pt and our tracker fine-tunes. It does not re-license anyone else's weights, and some files here start from third-party weights with their own terms:

File	Licence	Commercial use
`VTM-1.5.1.pt`	Apache-2.0 (ours)	Yes
`decoder/vtm-fast-decoder.pt`	Our training is Apache-2.0, but it is distilled from and warm-started on `cqyan/hybrid-sd-tinyvae`, whose weight licence is not declared	Open question until that publisher states a licence
`trackers/animeseg_hair3.pt`	Our fine-tune of Mask2Former ADE20k weights, which Meta licenses CC BY-NC 4.0	No
`trackers/iris_pose.pt`, `trackers/dwpose_v2.pt`	Our fine-tunes of Ultralytics YOLO-pose pretrained weights, which Ultralytics licenses AGPL-3.0	Under AGPL terms
`trackers/pose_landmarker_lite.task`	Apache-2.0 (Google / MediaPipe)	Yes
`openseeface/*`	BSD 2-Clause (emilianavt/OpenSeeFace)	Yes

Full inventory and sources: THIRD_PARTY_NOTICES.md in the VTM Spark repo.

Downloads last month: -; Downloads are not tracked for this model. How to track