MotionShop: Zero-Shot Motion Transfer in Video Diffusion Models with Mixture of Score Guidance

Hidir Yesiltepe, Tuna Han Salih Meral, Connor Dunlop, Pinar Yanardag
Virginia Tech
Paper Supplementary arXiv Code Data (Coming soon)

Abstract

In this work, we propose the first motion transfer approach in diffusion transformer through Mixture of Score Guidance (MSG), a theoretically-grounded framework for motion transfer in diffusion models. Our key theoretical contribution lies in reformulating conditional score to decompose motion score and content score in diffusion models. By formulating motion transfer as a mixture of potential energies, MSG naturally preserves scene composition and enables creative scene transformations while maintaining the integrity of transferred motion patterns. This novel sampling operates directly on pre-trained video diffusion models without additional training or fine-tuning. Through extensive experiments, MSG demonstrates successful handling of diverse scenarios including single object, multiple objects, and cross-object motion transfer as well as complex camera motion transfer. Additionally, we introduce MotionBench, the first motion transfer dataset consisting of 200 source videos and 1000 transferred sequences, covering single/multi-object transfers, and complex camera motions.

Method Overview

Framework

Our framework consists of three key stages: (1) Reference Motion Extraction captures motion patterns from early-timestep conditional scores through gradient fields ∇z log p(z|y). (2) Motion Transfer with MSG combines content and motion scores through our novel Mixture of Score Guidance formulation, enabling precise control over motion transfer while preserving scene coherence. (3) MSG Path Redirection employs implicit attention-guided dynamics to ensure stable motion transfer by navigating the diffusion process through modified Langevin dynamics. This zero-shot approach operates directly on pre-trained models without additional training, successfully handling diverse scenarios from single object transformations to complex camera trajectories.

Single Object Motion Transfer
Input
Original Video
Motion Transfer 1
“A wooden cart pulled by tiny, winged creatures”
Motion Transfer 2
“A crystal carriage drawn by an ethereal horse in a fog”
Input
Original Video
Motion Transfer 1
"A medieval knight on horseback"
Motion Transfer 2
“A man riding a jetski”
Input
Original Video
Motion Transfer 1
“A pirate captain striding across the deck of a ship”
Motion Transfer 2
“A camel is crossing a road”
Multiple Object Motion Transfer
Input
Original Video
Motion Transfer 1
"Three juvenile unicorns sharing their first sip"
Motion Transfer 2
"Three young phoenixes at their morning ritual"
Input
Original Video
Motion Transfer 1
“A majestic eagle swooping to snatch a shiny gold coin”
Motion Transfer 2
“A medieval knight catches the magical artifact”
Input
Original Video
Motion Transfer 1
“A pair of miniature medieval knights”
Motion Transfer 2
"A duo of adorable robots”
Camera Motion Transfer Results
Camera Trajectory
Camera trajectory visualization
Pan-left with zoom-in trajectory
Input
Original camera movement sequence
Motion Transfer
"A steampunk clockwork butterfly"
Camera Trajectory
Camera trajectory visualization
Circular motion trajectory
Input
Original camera movement sequence
Motion Transfer
"A medieval knight in gleaming armor"
Camera Trajectory
Camera trajectory visualization
Zoom-out trajectory
Input
Original camera movement sequence
Motion Transfer
"A coffee cup on a café table in a miniature cityscape"
Qualitative Comparison
Input
Original cats sequence
Motion Inversion
“A pair of miniature medieval knights”
Space-Time Diffusion
“A pair of miniature medieval knights”
Video Motion Customization
“A pair of miniature medieval knights”
Motion Director
“A pair of miniature medieval knights”
MotionShop (Ours)
“A pair of miniature medieval knights”

BibTeX

@misc{yesiltepe2024motionshop,
    title={MotionShop: Zero-Shot Motion Transfer in Video Diffusion Models with Mixture of Score Guidance},
    author={Hidir Yesiltepe and Tuna Han Salih Meral and Connor Dunlop and Pinar Yanardag},
    year={2024},
    eprint={2412.05355},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}