Z-sight joins robot action generation with latent world modeling
Maven Robotics and USC researchers introduced Z-sight, a robot foundation model that couples action generation to future visual tokens predicted in its VLM's native feature space.
Researchers affiliated with Maven Robotics and the University of Southern California introduced Z-sight, a robot foundation model that predicts future visual tokens in its vision-language model’s native feature space as it generates actions.
In the version-one preprint, the authors report an 81.9% average success rate on the SIMPLE humanoid benchmark and 73.3% across six physical Unitree G1 tasks. The findings come from the authors’ own evaluations and have not been independently replicated or peer reviewed in the sources examined.
Z-sight adds explicit foresight without reconstructing future video pixels or using a separate frozen encoder as its prediction target. Its 3.18-billion-parameter mixture-of-transformers combines a 2.22-billion-parameter Qwen3.5-2B vision-language backbone with separate 480-million-parameter foresight and action experts. The foresight expert predicts the visual tokens that the model’s evolving vision encoder would produce for a future observation. The action expert can attend to that prediction as it forms.
At inference, the two experts start with Gaussian noise, then update the future tokens and a 30-step action chunk together over 10 shared Euler steps. The model does not decode the tokens back into pixels. For training, an exponential-moving-average copy of the vision encoder supplies stop-gradient future targets, while the future-prediction loss is blocked from updating the vision-language backbone. The researchers say this setup keeps perception and prediction aligned while limiting representation collapse and the loss of pretrained knowledge.
The authors describe a three-stage training recipe. Pre-training uses 1,977 hours of egocentric human video. Mid-training uses about 2,750 hours of cross-embodiment robot data with human-video replay, bringing the pre- and mid-training source pool to 4,731 hours. Post-training then uses demonstrations from the target robot.
On SIMPLE, the reported 81.9% result spans 12 teleoperated tasks at three difficulty levels, putting it 9.6 percentage points above GR00T N1.7’s 72.3%. The paper says each method was evaluated in three passes, with 10 episodes for every task and difficulty level in each pass.
In the physical G1 evaluation, Z-sight completed 44 of 60 trials, or 73.3%, compared with 53.3% for Fast-WAM. Each method received 10 trials on each of six tasks from matched starting configurations. The paper does not report confidence intervals for those real-world rates.
The researchers also measured a median 109.5 milliseconds to generate a 30-step action chunk on a single RTX 4090, excluding image capture, network transport and low-level control. GR00T N1.7 was faster in that test at 60.0 milliseconds; three other systems in the comparison were slower. The paper also identifies a planning limitation: Z-sight predicts the future from observations alone rather than conditioning foresight on candidate actions, so it cannot yet compare possible action outcomes at test time.
More news

Close&Safe method limits robot detours to obstructed trajectory segments

TacInno trains robot policies on tactile changes vision cannot explain

A2-VLA tests unfamiliar tools before steering robot policy
