However you don’t need any picture—you need the picture you specified, sometimes with a textual content immediate. And so the diffusion mannequin is paired with a second mannequin—resembling a big language mannequin (LLM) skilled to match pictures with textual content descriptions—that guides every step of the cleanup course of, pushing the diffusion mannequin towards pictures that the big language mannequin considers a very good match to the immediate.
An apart: This LLM isn’t pulling the hyperlinks between textual content and pictures out of skinny air. Most text-to-image and text-to-video fashions at this time are skilled on giant knowledge units that comprise billions of pairings of textual content and pictures or textual content and video scraped from the web (a observe many creators are very sad about). Which means what you get from such fashions is a distillation of the world because it’s represented on-line, distorted by prejudice (and pornography).
It is best to think about diffusion fashions working with pictures. However the method can be utilized with many varieties of knowledge, together with audio and video. To generate film clips, a diffusion mannequin should clear up sequences of pictures—the consecutive frames of a video—as an alternative of only one picture.
What’s a latent diffusion mannequin?
All this takes an enormous quantity of compute (learn: vitality). That’s why most diffusion fashions used for video technology use a way referred to as latent diffusion. As an alternative of processing uncooked knowledge—the tens of millions of pixels in every video body—the mannequin works in what’s generally known as a latent area, during which the video frames (and textual content immediate) are compressed right into a mathematical code that captures simply the important options of the info and throws out the remainder.
An analogous factor occurs everytime you stream a video over the web: A video is distributed from a server to your display screen in a compressed format to make it get to you quicker, and when it arrives, your laptop or TV will convert it again right into a watchable video.
