RoboDream: Introduction to Data Synthesis of Composite World Model

This introduction corresponds to the paper RoboDream: Compositional World Models for Scalable Robot Data Synthesis, submitted to arXiv on 2026-06-01. The authors come from the USC Physical Superintelligence Lab and Toyota Research Institute. It is suitable to be placed in the 17-具身世界模型 section of this repository as a leading guide on world models after WoG and WALL-WM.

If arranged according to the current directory structure, it is not recommended to reorder the basic chapters such as 1、扩散数理基础, 2、VAE, and 3、DDPM. RoboDream is more suitable as the Article 6 introduction in the world model前沿 methods: LeWM discusses the fundamental issues of latent world models, RISE discusses using composite world models for policy self-improvement, RAW-Dream discusses enhancing VLA within the world model, WoG discusses condition-space world modeling, WALL-WM discusses event-level world action models, and RoboDream focuses on synthesizing robot teaching data using a composable video world model.

After completing this section, everyone needs to focus on one sentence:

RoboDream is not a neural simulator that can perform real-time step(action) -> next_state like MuJoCo / Isaac, but rather a world model that synthesizes data using robot-only trajectories as a framework, along with scene prior and object prior to re-render teaching videos.

This sentence is very important. It explains why RoboDream is worth watching, and also prevents people from being misled by the term “world model”.

1. Paper, project page, and code status

Currently, this does not fit as a reproduction tutorial that explains how to reproduce the paper results from scratch. A more reasonable approach is to first understand its architecture diagram, data flow, experimental conclusions, and boundary conditions thoroughly; after the official code is available, then reproduce it using the official training/generation/policy pipeline.

2. What problem is it actually solving

Figure 1: RoboDream overview animation. Its goal is not to generate robot videos out of thin air, but to re-draw old trajectories onto new objects, new scenes, and new perspectives, given a predetermined robot motion skeleton.

Source: RoboDream official GitHub repository .

Imitation learning for robots lacks data, and this is a well-known issue. A task requires dozens to hundreds of real remote manipulation examples, which is already time-consuming; if you also need to change objects, backgrounds, camera perspectives, or desktop layouts, the cost will explode rapidly. Using large-scale public robot data like DROID is tempting, but the raw DROID data often differs significantly from your target experimental setup:

RoboDream’s problem awareness is: The action trajectories in old data may still be valuable, but what is truly difficult to reuse is the visual domain. Therefore, it is not simply about image enhancement, nor does it generate entirely new actions. Instead, it focuses on a more specific task:

text已有或新采集的机器人轨迹
        ↓
在 Isaac Lab 里渲染成 robot-only motion video
        ↓
给目标场景背景 scene prior
        ↓
给目标物体外观 object prior
        ↓
视频世界模型合成目标域示教视频
        ↓
沿用原轨迹动作标签训练策略

This is what Compositional World Models means in the paper title: Robot movements, object appearances, scene backgrounds, and perspectives are not fixed to a single original demonstration, but can be broken down and recombined.

3. Is it a “manipulable world model”?

This issue needs to be explained in two layers.

In a weak sense, RoboDream is controllable. You can control:

• robot-only trajectory: How the robot body moves and how the gripper opens and closes;
• scene prior: What the background and desktop environment look like;
• object prior: What the target object looks like;
• camera/view: The same trajectory can render robot-only video from a new camera position;
• task instruction: Adding semantic constraints to the generation process using language.

In a strict sense, it is not a manipulable simulator. It does not provide:

text当前状态 s_t + 动作 a_t -> 下一状态 s_{t+1}

It cannot function like Isaac / MuJoCo / Genesis, where a policy or human input determines each step, and then the next frame, collision results, reward, and completion status are returned in real time. Instead, it acts more like a offline teaching video synthesizer: first provide a valid trajectory, and then have it “draw” the target object and scene around that trajectory.

A truly powerful world model with meaningful manipulation should possess at least these capabilities:

So a more accurate term is: controllable video world model for robot data synthesis, rather than learned physics simulator.

4. Overall Architecture Diagram: Three Types of Visual Priors + Two Types of Semantic/Motion Conditions

Figure 2: RoboDream model architecture. When reading this figure, it is recommended to understand it as “under what conditions the channel is processed, and under what conditions attention is applied”: robot-only video and scene prior are concatenated in the channel dimension; object prior becomes tokens and then enters self-attention; task language and global trajectory enter cross-attention.

Source: RoboDream arXiv HTML.

This figure is the most important figure in the entire paper. It answers a core question: If we want the video model to generate robot teachings reliably, what information should we explicitly provide?

The conditional input of RoboDream can be divided into five categories:

There are three key designs in the architecture.

First, robot-only video serves as a pixel-level motion anchor. Many people assume that providing the joint angles or pose of the robot is sufficient, but the paper does not only present low-dimensional states. Instead, it renders the trajectory as video in Isaac Lab first. In this way, the model can directly see where the robotic arm is at each frame, how it moves, and when the gripper opens and closes in pixel space. This design is similar to video-based ControlNet: instead of relying on prompts to let the model infer the robot’s shape, the motion structure of the robot is clearly shown to the model.

Second, the scene prior enters the model through channel extension. The background is usually static in a short video, so the paper expands a single scene prior into a static video along the time dimension, and then concatenates it with the latents of noisy video and robot-only video at the channel dimension. In this way, the same environmental layout can be seen at each time step, reducing background drift.

Third, the object prior is injected via self-attention. The objects are not directly placed at fixed coordinates, but are encoded as latent object tokens, after which the video tokens access these object tokens through self-attention. This detail is crucial: the model learns “what the target object looks like,” rather than “the target object must be at a certain position in the prior map.”

In simplified terms, it is:

mermaidflowchart LR
    A["Noisy video latent"] --> F["Video Diffusion Transformer"]
    B["Robot-only video<br/>kinematic anchor"] --> C["VAE encode"]
    D["Scene prior<br/>static background"] --> E["VAE encode + temporal expand"]
    C --> G["Channel concat"]
    E --> G
    A --> G
    G --> F
    H["Object prior<br/>cropped objects"] --> I["VAE encode as object tokens"]
    I --> J["Extended self-attention"]
    J --> F
    K["Task instruction<br/>T5 encoder"] --> L["Cross-attention"]
    M["Global trajectory<br/>MLP"] --> L
    L --> F
    F --> N["Generated robot demonstration video"]

The value of this design lies not in having a “larger model,” but in breaking down the generation task into several manageable tasks: the movement of the robot is managed by robot-only video, the background is managed by scene prior, the appearance of objects is managed by object prior, and the task semantics are managed by instruction.

5. Why robot-only video is so crucial

If a text-based generation robot is used to produce manipulation videos directly, the model must address four challenges simultaneously:

1. The robotic arm cannot deform;
2. The movement of the robotic arm must comply with joint constraints;
3. The gripper and object contact must be appropriate;
4. The background and target object should resemble the target scene.

RoboDream does not solve these problems from scratch with video models, but instead locks in the robot movement. Its robot-only videos have three functions.

First, it ensures embodiment consistency. Video generation models often bend the robot arm, miss the gripper, and misrepresent the joint shapes. Robot-only videos explicitly provide the shape and position of the robot in each frame, and the model is mainly responsible for adding the objects and scene elements.

Second, it implicitly contains camera information. The paper emphasizes that robotic-only videos include camera information in the pixel domain. If the same trajectory is rendered from a new perspective, the projection of the robotic arm in the image will change; combined with the corresponding perspective scene prior, the model can generate a novel view demonstration.

Third, it retains the availability of action labels. Many text-to-video robot data methods require generating videos first, and then using inverse dynamics to reverse-engineer movements, which easily amplifies errors. RoboDream already has a trajectory for actions, and the video merely repaints the target domain from the observation side. It avoids the problem of “generating videos first and then guessing the actions.”

However, this advantage also has its limitations: If the input trajectory is not suitable for the target object or its layout, RoboDream will not automatically refine it into a well-planned one. It can redraw a grasping trajectory to resemble the target domain, but it cannot guarantee that this trajectory is truly suitable for a new object with completely different size, mass, and position.

6. Prior extraction: How to automatically extract objects and scenes

Figure 3 prior extraction process. RoboDream uses VLM to identify task-related objects from the instruction and first frame, divides the objects using Grounded-SAM, and removes the original objects with OmniPaint to obtain a clean scene prior.

Source: RoboDream official project page .

Training RoboDream requires dividing the existing robot data into three parts: robot motion, target objects, and background scenes. The paper does not require manual frame-by-frame annotation, but uses an automatic pipeline to generate training pairs.

The first step is object identification. Given the first frame and task instructions, use VLM to identify objects that are truly relevant to the task. For example, if the task is “put marker into bowl,” the relevant objects should be the marker and the bowl, rather than the table, wall, or robotic arm base.

Step 2 is the object prior construction. Use Grounded-SAM to segment these task objects, then randomly rotate and scale the cropped objects and place them on a clean canvas. This step is easily overlooked, but it is important: random placement forces the model to learn the appearance of the objects, rather than memorizing the coordinates of the objects in the original image.

The third step is scene prior construction. Remove the task objects from the first frame, then use OmniPaint for inpainting to create a clean background. In this way, scene prior only represents the background and desktop environment, while object prior only represents the task objects. After separating them, the model can truly learn “changing objects” and “changing scenes”.

If this step is not taken, the model will encounter a contradiction: there is already a cup in the scene prior, and the object prior provides another cup. Which one is the target to be manipulated? RoboDream resolves this ambiguity by first removing the objects from the background.

7. Two deployment modes: retrieval and rebirth, and prop-free teleoperation

RoboDream’s approach does not involve a single model structure; instead, it proposes two practical data production models.

7.1 Retrieval and Rebirth: Old Trajectory “Reborn” in New Scenarios

Figure 4 zero-shot demonstration rebirth. The right side shows the original trajectory retrieved from DROID, while the middle part displays the target domain teaching video recreated by RoboDream based on the target object prior and scene prior.

Source: RoboDream arXiv HTML.

Retrieval and Rebirth can be understood as “finding similar action skeletons, and then replacing them with the target domain appearance”. The process is:

1. Enter a new task instruction;
2. Encode the task instruction and each trajectory instruction in the dataset using the T5 encoder;
3. Retrieve DROID trajectories with similar semantics based on cosine similarity;
4. Replay the trajectories in Isaac Lab to render robot-only videos;
5. Prepare the scene prior for the target scene and the object prior for the target object;
6. RoboDream generates a demonstration video in the target domain;
7. Train the downstream policy using the generated video and original action labels.

This pattern is most suitable for scenarios where the public data already contains similar actions, such as approaching, picking up, lifting, placing, and wiping. Your target environment and objects may differ, but the action framework can be reused.

The scenario where it is not suitable is: new tasks require completely different dynamics or contact strategies. For example, when transitioning from picking up blocks to turning bottle caps, zipping up zippers, or plugging in USB devices, the action structures of the old trajectory may not be compatible at all. RoboDream can change its skin, but it cannot create a correct contact strategy out of thin air.

7.2 Prop-free teleoperation: Manipulation through air, then placing objects inside

Prop-free teleoperation is a very insightful concept in the paper. It borrows the “prop-free” idea from film and television: actors perform on air, and props are added later. In robots, this means that the operator does not need to position real objects each time; they can perform robot movements as if they were “pretended operations” on a blank surface or in a simulator, while only recording valid trajectories. Then, RoboDream combines the target object with the scene.

There are two advantages to this:

• No need to repeatedly reset the object position, making remote manipulation collection faster;
• A common pick-and-place trajectory can be transformed into data for multiple tasks using different object priors.

But it also has a clear risk: the subtle adjustments in real grasping, collision feedback, and object sliding do not exist in prop-free trajectories. RoboDream needs to “fill in” these visual phenomena. If done incorrectly, the generated video may look similar, but the action labels and visual contacts could be misaligned.

Therefore, the appropriate positioning of Prop-free is: a near-term solution to reduce data collection costs, not a complete replacement for real teleoperation.

8. Compositional generation: The truly exquisite part of this

Figure 5: RoboDream’s composable generation capability. The same base trajectory can generate novel instances, novel scenes, novel tasks, and novel viewpoints by changing the object prior, scene prior, task context, and camera view.

Source: RoboDream arXiv HTML.

This figure illustrates the core “combinability” of the paper.

Note the boundaries of “zero-shot” here. RoboDream’s zero-shot is mainly visual combination zero-shot, not action planning zero-shot. It does not generate new actions on its own after encountering a completely new task; instead, based on existing or newly collected compatible trajectories, it replaces the visual context with unseen objects, scenes, and perspectives.

This is also where it surpasses ordinary image enhancement. Ordinary enhancement usually only changes colors, textures, and background disturbances, while the underlying interaction layout remains unchanged. RoboDream can regenerate a motion trajectory into a new target domain, making the old trajectory visually closer to the target environment.

9. How to read the experimental results

Figure 6: Real robot evaluation tasks. The paper uses the Franka Panda platform in DROID style to evaluate on four real manipulation tasks: Put Marker into Bowl, Remove Marker from Bowl, Put Cube into Cup, and Wipe Table with Towel.

Source: RoboDream arXiv HTML.

The focus of the paper evaluation is not whether “the video looks similar,” but whether “the generated data can assist downstream policies.” It trains the policy using a Diffusion Policy and evaluates it on real robots.

9.1 Results of Retrieval and Rebirth

This table should not be interpreted only based on the highest Gen-Mix. A more accurate reading is:

First, Orig-100 is 0%. Trying to train the target domain policy using the retrieved DROID raw data fails completely, indicating a severe domain shift. The old data has action value, but the visual domain does not match.

Second, the Gen-100 average is only 15.0%. The pure generated data is not strong enough, clearly lower than the 36.3% of Real-50. This indicates that RoboDream cannot replace real data.

Thirdly, the Gen-Mix has an average of 62.5%, which is higher than Real-50 and Orig-Mix. The most credible conclusion is: A small amount of real data as a foundation, combined with the target domain data generated by RoboDream, can significantly enhance policy training.

Therefore, this paper truly proves that “generated data is useful as real data augmentation”, not that “synthetic data alone can solve robot learning”.

9.2 Prop-free Results

Prop-free has an average of 32.5%, slightly lower than Real-50’s 36.3%. However, collecting 50 real teleoperation trajectories takes about 2 hours, while collecting 50 prop-free trajectories takes about 55 minutes—about 2.2 times faster.

The significance of this result should also be clearly explained: Prop-free does not achieve a better performance than real data, but offers a lower-cost approach that yields results close to real data. It is very useful in scenarios where quick testing tasks and rapid expansion of action skeletons are needed. However, if the task is extremely sensitive to physical contact, real data remains indispensable.

9.3 scaling with generated data

This table indicates that generating data along the trajectory does have a scaling benefit, but the benefit begins to saturate around Mix-200. Possible reasons include: limited diversity in retrieval trajectories, limited generation quality, and a remaining gap between generated data and real deployment.

10. Differences from ordinary video generation, traditional simulation, and other world models

From this table, it can be seen that RoboDream’s role is clear: it is neither a VLA base nor a complete interactive world simulator, but a trajectory-conditioned visual rebirth engine tailored for data bottlenecks in imitation learning.

11. Details Worth Learning from This Article

First, the trajectory dimensionality reduction is impressive. It does not invent robot movements from scratch for the video model, but instead fixes the motion trajectory, allowing the model to learn the visual context. This is much more stable than a text-to-video robot demo.

Second, it avoids inverse dynamics. The action labels come from the original trajectory, and the generation model is mainly responsible for the observation side. This approach is more suitable for policy learning than generating videos first and then applying reverse actions.

Third, the splitting of object prior and scene prior is very detailed. Object prior avoids position leakage through random placement, while scene prior avoids conflicts with object prior by removing objects. All these measures are designed to enable the model to learn true combinations.

Fourth, multi-perspective processing is not a simple puzzle. The paper treats each view as a separate video entry, tokenizes them separately, and stacks the tokens together, thereby reducing spatial ambiguity caused by simple horizontal concatenation.

Fifth, it explains the value of external big data such as DROID in a more practical way. Using public data directly would result in domain shift, but the action skeletons from previous trajectories may still be useful. RoboDream translates these action skeletons into the target visual domain.

12. Major Limitations and Risks

RoboDream is most likely misread as “generating videos can already replace physical simulation.” This is not accurate. It has at least five boundaries.

First, the photorealism of the generated video does not equate to action consistency. A video that looks real does not mean that the gripper, objects, and action labels are aligned at the centimeter and frame levels. In robotic tasks, a deviation of 2 centimeters in an object can cause the policy to learn incorrect associations.

Second, the robot-only trajectory must be reasonable by nature. The paper constraints also emphasize that it assumes that the rendered robot-only video accurately describes the trajectory that the robots should follow in the target scene. RoboDream is not responsible for planning better actions.

Third, physical contact remains a weakness. The contact forces, friction, sliding, and failure modes in grasping, releasing, wiping, and pushing can only be approximately learned from data by the video diffusion model, rather than being explicitly solved like in a physics engine.

Fourth, pure generative data is not sufficient. The average for Gen-100 is 15.0%, which is significantly lower than that of Real-50. The most reliable approach currently is mixed training, rather than fully synthetic training.

Fifth, the code has not been fully released yet. The official repository has exposed the README, license, and assets, but the training, generation, and policy-learning pipelines are still marked as coming soon. Reproduction experiments will require the completion of subsequent code, weights, configurations, and data processing scripts.

13. What should be done if reproduction is needed later

Currently, it is not recommended to immediately engage in computational reproduction experiments. However, you can follow the following checklist to monitor the subsequent open-source status.

A more realistic short-term learning path is:

1. First, read the project page and arXiv to understand Fig. 2, Fig. 3, Fig. 5, and Fig. 6 clearly;
2. Check whether Jay-Ye/RoboDream releases the training/generation scripts;
3. If the official release comes first with inference, you can use a small amount of DROID or self-prepared trajectories to run the generation demo;
4. If the official release is the policy-learning pipeline, then reproduce the core comparison set of Real-50 / Gen-Mix;
5. When reproducing, prioritize verifying action-consistency rather than just checking if the video looks good.

14. Reading order of other world model articles in this tutorial

It is recommended that you read in the following order:

If we had to summarize the location of RoboDream in just one sentence:

It is not about placing robots in an interactive neural universe, but rather transforming existing robot trajectories into data assets that can be reused across objects, scenarios, and perspectives.

15. Reference Links

• RoboDream project page: https://junjieye.com/RoboDream/
• RoboDream paper:arXiv:2606.02577
• RoboDream official repository:Jay-Ye/RoboDream
• DROID dataset:DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
• Cosmos-Predict2：NVIDIA Cosmos Predict2