04 SmolVLA Migration and Sampling Weighting on ROCm

This task focuses on SmolVLA. Compared to ACT, SmolVLA relies more on linguistic conditions and visual features. Therefore, it is particularly suitable for using red cup and blue cup tasks to verify whether the model truly understands the instructions, rather than just memorizing the data distribution.

Supporting practical notebook: 04_smolvla_weighted_sampling.ipynb.

Fixed Red and Blue Cup Evaluation

Don’t just randomly select tasks. It is recommended to force two commands with the same seed:

textPlace the red mug on the plate.
Place the blue mug on the plate.

Then compare:

If the red cup is good but the blue cup is poor, it indicates that the model is not completely unable to capture, but there are biases in task conditions or color distribution.

Figure 1: Comparison of physical_success on fixed commands for Red Cup and Blue Cup with SmolVLA. The key is not to pursue the highest success rate for a single color, but to see if “improving the blue cup results in sacrificing the red cup”.

Why not copy the data directly

A common intuitive approach is to copy the Blue Cup episode multiple times. However, copying the data will alter the statistics of the dataset and the distribution of episodes, potentially causing the model to overfit to the Blue Cup while harming the Red Cup.

A more gentle approach is to apply weighted sampling by frame or episode. For example, use WeightedRandomSampler so that the blue frame is sampled with a higher probability, but the original parquet file remains unchanged.

It is recommended to compare the two methods:

What Did SmolVLA Change This Time?

The original issue with SmolVLA is not that it cannot grasping cups at all, but rather it clearly favors red cups. After fixing the same set of seeds and forcing red and blue cup commands respectively, the baseline results are red cups 8/10 and blue cups 0/10. This phenomenon indicates that the model has the ability to grasp cups, but the language conditions and color distribution are not evenly learned.

We tried the most intuitive way to copy data: copying the Blue Cup episode to 1.5x, 2x, and 3x. As a result, the Blue Cup did appear, but the Red Cup dropped significantly:

So the main line is changed to WeightedRandomSampler. It does not copy the parquet episode, does not alter the original LeRobot dataset, and does not change the norm stats. It only increases the sampling probability of the blue cup frame during training sampling. The two key environment variables used in practice are:

bashexport LEROBOT_TASK_WEIGHT_SUBSTRING=blue
export LEROBOT_TASK_WEIGHT=2.0

The advantage of this approach is simple rollback: the dataset entity remains as demo_data_language, and by simply removing the sampler weights, training returns to normal sampling. At the same time, it does not alter the dataset statistics in a rigid way like copying an episode.

The best checkpoint of this round is not the final step of the same run, but the intermediate 000500:

textckpt/smolvla_weighted_blue2_from5000_1000_20260629_143123/checkpoints/000500/pretrained_model

The loss of 001000 continues to decrease, but the balance between red and blue cups is not as good as that of 000500. Therefore, in this topic, 000500 is set as a protected checkpoint. Any new SmolVLA scheme later on must exceed it under the same fixed seeds and physical_success parameters to be considered worthy of replacement.

The final results are reported in two layers:

The core experience of this approach is straightforward: first break down the tasks according to the instructions for evaluation, and then handle the data distribution bias. Do not focus solely on the overall success rate, nor copy the data immediately just because the blue cup score is poor; copying an episode often damages another task. Sampling weighting is more suitable as a top priority.

ROCm Training Records

When SmolVLA is trained on ROCm, the following is recorded:

• Initial checkpoint;
• Data root directory;
• Training steps;
• Batch size;
• Task weight substring;
• Task weight;
• GPU utilization;
• Temperature range;
• VRAM usage;
• Step to save the checkpoint.

Don’t just look at the final checkpoint. The intermediate checkpoint may be better. For example, 500 steps may be more balanced than 1000 steps.

Video and failed seed reruns

If some seeds fail in the batch eval, run them three times to check if the failure is stable:

The occasional failure indicates that the policy is approaching its boundary. The next step should be to teach the boundary state or perform a small amount of targeted DAgger, rather than blindly performing long training.

Example of the results of this iteration of replication

In this tutorial example, the SmolVLA checkpoint that performs most stably is the end-to-end Notebook reconstruction weighted500. In small-scale comparisons, it achieves red cups 8/10 and blue cups 10/10; after the formal protection evaluation is extended to 30 seeds, red cups 27/30 and blue cups 30/30, totaling 57/60. This indicates that the SmolVLA main line has achieved a relatively stable fixed instruction capture capability on ROCm devices.

Here, it is referred to as the “protection baseline”, which means all subsequent experiments should be compared against it. A new checkpoint is only worth replacing if it exceeds this baseline under the same set of seeds and physical_success parameters. This ensures that we avoid assuming the model is truly better just because a certain training loss is lower.

Figure 2: The failure sequence of the blue cup in baseline. This failure is not due to the environment being unable to capture the cup, but because the model fails to execute the blue cup instructions consistently.

Figure 3: The successful sequence of Blue Cup after weighted sampling. It is suitable for inclusion in tutorials to explain why it is necessary to evaluate by splitting the instruction colors.

Checkpoint

After completing this task, keep these results:

• Success rate of fixed instructions for red and blue cups;
• Controls for at least two sampling policies;
• One protection baseline checkpoint;
• One successful video and one failed video;
• Conclusion on whether the failed seed is stable after re-running.