Lumiere: Generating the Whole Motion at Once

Question

Lumiere challenges a common generation strategy: create temporally distant keyframes, then fill the gaps with temporal super-resolution. Its Space-Time U-Net instead generates the clip's full temporal duration at once.

The issue is deeper than smooth adjacent frames. A clip can have individually plausible keyframes and smooth interpolation while its subject, space, or action intent drifts. Lumiere treats a short clip as a whole rather than a sequence of images to be filled in.

Technical Reading

Space-Time U-Net models space and time together. It considers visual and temporal structure during generation instead of choosing a few visual points and placing motion between them.

A shot contains action preparation, execution, completion, and a camera's chosen viewing path. Modeling the whole duration gives a model a better opportunity to sustain one continuous intent than simple interpolation between images A and B.

Capability Boundary

Lumiere primarily addresses an individual clip. Better consistency within that clip does not ensure that the next shot will preserve the same character, space, or action consequence.

Its focus is integrity within a shot, rather than continuity across a sequence.

Lab Judgment

After reading Lumiere, I separate video stability into two layers:

local smoothness: no jitter, flicker, or abrupt changes between adjacent frames
global shot consistency: subject, space, action, and style remain coherent across the full shot

Many models reach only the first layer. A shot that can be edited needs the second. Prompts should describe a complete shot, including its ending:

start: the character stands in the doorway, right hand on the doorframe
motion: the camera slowly pushes in as the character looks back to the left
end: light behind the door is revealed; the character remains frame right

The ending state is a key test of whether the model understands time.

Sources

• Lumiere