Salesforce AI analysis crew current FOFPred, a language pushed future optical move prediction framework that connects giant imaginative and prescient language fashions with diffusion transformers for dense movement forecasting in management and video technology settings. FOFPred takes a number of photos and a pure language instruction corresponding to ‘transferring the bottle from proper to left’ and predicts 4 future optical move frames that describe how each pixel is anticipated to maneuver over time.
Future optical move as a movement illustration
Optical move is the obvious per pixel displacement between two frames. FOFPred focuses on future optical move, which suggests predicting dense displacement fields for future frames given solely present observations and textual content, with out entry to future photos at inference.
Future optical move is a compact movement solely illustration. It removes static look and retains solely pixel degree movement, so it’s effectively suited as an intermediate state for robotic management insurance policies and as a conditioning sign for video diffusion fashions. In comparison with predicting future RGB frames, it reduces the complexity of the output distribution and avoids modeling textures and excessive frequency particulars that aren’t required for movement planning.
To plug into current latent diffusion infrastructure, the analysis crew encode optical move as RGB photos. They map move magnitude and route from polar kind into HSV channels, then convert to RGB. The scaling of every channel is tuned in order that consecutive move frames are visually clean and resemble animated graphics. An ordinary Flux.1 variational autoencoder then encodes and decodes these move photos.
Unified VLM Diffusion spine
FOFPred makes use of a unified structure that mixes a frozen imaginative and prescient language mannequin, a frozen VAE and a trainable diffusion transformer. The pipeline is:
- Qwen2.5-VL is used because the imaginative and prescient language encoder to collectively encode the caption and visible inputs.
- Flux.1 VAE encodes the enter photos and the coaching optical move targets into latent tensors.
- An OmniGen model diffusion transformer, DiT, takes projected visible and textual options as conditional inputs and generates latent future move sequences.
Solely the DiT and small MLP projectors are skilled. The Qwen2.5-VL and Flux.1 weights keep frozen, which lets the mannequin reuse picture modifying pretraining and multimodal reasoning skill from prior work. Temporal modeling is added by extending the RoPE positional encoding and a focus blocks from two dimensional spatial positions to full spatio-temporal positions throughout enter and output body sequences. This provides full spatio-temporal consideration with out including additional parameters, so the DiT can reuse OmniGen picture pretraining straight.
Coaching on noisy net movies with relative optical move
The core mannequin is skilled on net scale human exercise movies with paired captions. The analysis crew makes use of the One thing One thing V2 dataset and the EgoDex selfish manipulation dataset to acquire round 500,000 video caption pairs.
Coaching makes use of an finish to finish move matching goal in latent area. Future optical move sequences are first computed offline, then encoded by the VAE and used as targets in a move matching diffusion loss for the DiT. Throughout coaching the strategy additionally applies classifier free steerage on each textual content and visible situations and masks some frames and viewpoints to enhance robustness.
A important contribution is the relative optical move calculation used to construct clear coaching targets from noisy selfish movies. For every body pair the strategy:
- Computes dense optical move with an off the shelf estimator.
- Estimates digicam movement by way of homography utilizing deep options.
- Makes use of projective geometry to subtract digicam movement and procure object centric relative move vectors.
- Filters body pairs by choosing these the place the highest ok % move magnitudes exceed a threshold, which focuses coaching on segments with significant movement.
These steps are run offline at decrease decision for effectivity, then recomputed at unique decision for the ultimate targets. The ablation research exhibits that static body targets or uncooked move with out digicam movement removing hurt downstream efficiency, whereas disentangled relative move targets give the very best outcomes.
Language pushed robotic manipulation
The primary downstream use case is robotic management. FOFPred is finetuned on robotic video caption information to foretell future optical move from each fastened and wrist mounted cameras. On high of FOFPred, the analysis crew connect a diffusion coverage community that takes predicted move, textual content and robotic state, and outputs steady actions. This setup follows prior diffusion coverage work however makes use of future optical move as a substitute of predicted RGB frames because the core illustration.
On the CALVIN ABCD benchmark, which evaluates lengthy horizon zero shot chains of 5 language specified manipulation duties, FOFPred reaches a mean chain size of 4.48. VPP reaches 4.33 and DreamVLA reaches 4.44 beneath the identical protocol. FOFPred additionally attains a Process 5 success fee of 78.7 %, which is the very best amongst reported strategies. In a low information setting with 10 % of CALVIN demonstrations, FOFPred nonetheless reaches 3.43 common size, increased than the three.25 of VPP.
On RoboTwin 2.0, a twin arm manipulation benchmark with 5 duties that require each arms, FOFPred attains a mean success fee of 68.6 %. The VPP baseline reaches 61.8 % beneath similar coaching settings. FOFPred improves success on each activity within the subset.
Movement conscious textual content to video technology
The second downstream activity is movement management in textual content to video technology. The analysis crew construct a two stage pipeline by connecting FOFPred with the Go together with the Circulation video diffusion mannequin. FOFPred takes an preliminary body and a language description of movement, predicts a sequence of future move frames, and interpolates them right into a dense movement discipline. Go together with the Circulation then makes use of this movement discipline and the preliminary body to synthesize the ultimate video, implementing the described movement sample.
On the movement heavy One thing One thing V2 benchmark, the FOFPred together with Go together with the Circulation pipeline improves over the CogVideoX baseline beneath similar situations. The tactic reaches SSIM 68.4, PSNR 22.26, LPIPS 28.5, FVD 75.39, KVD 11.38, and movement constancy 0.662, that are constantly higher than CogVideoX. Importantly, FOFPred solely makes use of language and a single body at inference, whereas a number of controllable video baselines require hand or object masks or trajectories as additional inputs.
Key Take aways
- FOFPred reframes movement prediction as language pushed future optical move, predicting 4 dense optical move frames from a number of present photos and a textual content instruction, which supplies a compact movement solely illustration for downstream duties.
- The mannequin makes use of a unified VLM Diffusion spine, with Qwen2.5-VL as a frozen imaginative and prescient language encoder, Flux.1-VAE as a frozen latent encoder for photos and move, and an OmniGen model DiT as the one skilled part with spatio temporal RoPE primarily based consideration.
- Coaching depends on giant scale net and selfish video from One thing One thing-V2 and EgoDex, and builds relative optical move targets by estimating ego-motion by way of homography, subtracting digicam move and filtering for top movement segments, which considerably improves downstream efficiency.
- In robotic manipulation, FOFPred acts as a movement spine for a diffusion coverage head and achieves state-of-the-art or higher outcomes on CALVIN ABCD and RoboTwin 2.0, together with 4.48 common activity chain size on CALVIN and 68.6 % common success on RoboTwin, outperforming VPP and DreamVLA variants.
- For textual content to video technology, connecting FOFPred to Go together with the Circulation yields higher SSv2 metrics than CogVideoX, with increased SSIM and PSNR, decrease FVD and KVD, and improved movement constancy, whereas requiring solely language and a single body at inference, making FOFPred a reusable movement controller for each robotics and video synthesis pipelines.
Take a look at the Paper, Mannequin and Repo. Additionally, be happy to observe us on Twitter and don’t overlook to hitch our 100k+ ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you possibly can be part of us on telegram as effectively.
Michal Sutter is a knowledge science skilled with a Grasp of Science in Knowledge Science from the College of Padova. With a stable basis in statistical evaluation, machine studying, and information engineering, Michal excels at remodeling complicated datasets into actionable insights.
