Generating high-quality, temporally consistent videos using AI has long faced a major roadblock – annoying flickering artifacts that change from frame to frame. But researchers may have finally solved this challenge. In a new paper, scientists present a technique called TokenFlow that enforces consistency across video frames by propagating key edited features throughout the clip.
View more examples on the official website: https://diffusion-tokenflow.github.io/
The Problem:
- Existing AI video models produce flickering between frames
- Each frame is edited separately, leading to inconsistencies
- Makes motions unnatural and backgrounds unrealistic
The TokenFlow Solution:
- Enforces consistency by propagating key edited features
- Samples and edits only a few keyframes
- Uses original video’s features to spread edits smoothly
- Avoids flickering from frame-by-frame manipulation
How It Works:
- Sample and edit keyframes
- Establish correspondences between original video features
- Propagate edited features to all frames based on correspondences
Results:
- Edits like face swaps and cartoon inserts
- No flickering artifacts
- Natural motion and coherent backgrounds
Comparison to Other Methods:
| Method | Flickering | Quality | Edit Control |
|---|---|---|---|
| Per-frame editing | High | Low | High |
| Self-attention | Moderate | Moderate | Moderate |
| TokenFlow | None | High | High |
When AI creates or edits a video, it actually works by processing the video frame-by-frame. So if you want to edit a video to have a different face or object, the AI edits each frame separately to change that face or add that object.
The problem is that small differences creep in between frames when they are edited individually. So the face or object may flicker or shift slightly from one frame to the next, creating an annoying disjointed effect.
Solving the Problem of Flickering in AI Generated Videos
TokenFlow solves this issue by taking a different approach. Instead of editing all frames, it picks out just a few keyframes spread throughout the video to edit extensively.
The key insight is that natural videos have a lot of redundancy – frames tend to have very similar content. TokenFlow uses this by finding connections between the features of the original frames before editing.
It then uses those connections to propagate or copy the edits made to the keyframes to the rest of the frames. So if the keyframes have a new face swapped in cleanly, that edit gets applied to the other frames automatically in a smooth way.
This avoids the flicker effect and maintains natural motion and continuity. So rather than tediously editing each frame, TokenFlow can edit a few frames and fill in the rest based on the video’s inherent similarities.
The end result is the ability to make dramatic edits to video with AI without artifacts or glitches. It helps bring AI-edited video closer to Hollywood-level quality for things like special effects and CGI.
The breakthrough provides a missing piece to unleash the full potential of AI for high-quality video generation and editing. No longer constrained by flickering artifacts, algorithms can focus on creating any scene imaginable. TokenFlow brings us one step closer to easily producing Hollywood-level CGI at home.