06 ROCm Debugging Review and Troubleshooting Cases

This chapter compiles the key issues encountered when replicating ACT, SmolVLA, and pi_0 on AMD ROCm devices into learning cases. Instead of listing commands in chronological order, it breaks down each issue into a troubleshooting chain: first observe the phenomenon, then collect evidence, then identify the root cause, and finally verify whether the fix is effective using the same evaluation criteria.

Supporting Practical Notebook: 06_rocm_debug_playbook.ipynb.

This chapter accomplishes several things:

• Determine whether “log success” and “physical success” are consistent;
• Know when to return to data collection and playback auditing;
• Understand why ACT uses a closed loop DAgger instead of only focusing on training loss;
• Understand why the color task bias of SmolVLA requires forced red/blue instructions for evaluation;
• Understand why the pi_0 raw policy and the script terminator diagnosis should be reported separately;
• Identify common process, memory, path, and dependency issues during ROCm remote training;
• Safely handle issues with Hugging Face gated models, tokens, and caches.

From Phenomena to Conclusions

When recreating, do not just save a long log segment. More valuable is to organize each failure into a reproducible “small case”: how the problem was discovered, what evidence exists, which directions were excluded, and finally how the fix was verified to be effective.

It is recommended to organize each question in the following format:

Commands and paths should be written in a portable format, such as $PROJECT_ROOT, $DATA_ROOT, and $OUTPUT_ROOT. When transferred to your own workstation, cloud host, or classroom server, they can also be reproduced according to the same logic.

Case 1: Old success does not equal true grasping success

** Phenomenon**: One assessment of ACT shows success, but in the video, the cup is not lifted, rather it is pushed near the plate by its end, and the cup also tips over.

Evidence: The old env.check_success() is closer to the final state geometry judgment. As long as the cup reaches a near-final state, the gripper opens, and the end height meets the criteria, success is likely to be achieved. Video review reveals that this success does not necessarily involve the complete process of “grabbing, moving, releasing, and placing upright.”

Fix: Introduce physical_success uniformly in the tutorial. At least check:

1. The old geometry success condition is true;
2. The target cup is lifted to a sufficient height, for example, lift is greater than 0.03 m;
3. The lifting state persists for several control ticks;
4. In the final state, the cup is basically upright, for example, upright cosine is greater than the threshold.

Tutorial experience: For all model comparisons, it is necessary to prioritize reporting physical_success. The old success status can serve as an auxiliary indicator, but it cannot be used alone to prove the replication success.

Case 2: Audit the teaching data itself

** Phenomenon**: In early ACT training, it is easy to become “stuck” or experience unstable terminal release. When only focusing on the playback geometry success, the data seems fine, but many closed loop rollouts fail.

Root cause: In the early collection script, there was an issue with how the record of the gripper action at the end was written. It recorded the actual gripper state returned by the environment as the action, resulting in the trailing tag appearing as “closed” rather than “release executed”. This error directly corrupts the policy.

Fix:

• Fix the grasping script to preserve the gripper dimensions of the original control actions;
• Perform post-processing repairs on existing data;
• Before training, conduct an open-loop physical playback audit for each episode to ensure that the teaching data correspond to actual grasping and placement.

Tutorial experience: If the policy always fails at the final stage, don’t suspect that the model is too weak first. First, check whether the data action truly represents the release action.

Case 3: The decrease in ACT’s loss does not indicate closed loop stability

** Phenomenon**: The ACT can complete training, and certain learning effects can be observed through open-loop or teacher-forced diagnostics. However, in closed-loop deployment, the robot deviates from the teaching state, resulting in issues such as not touching the cup, spilling the cup after lifting, and misplacement when placing it.

Troubleshooting Path:

Figure 1: ACT debugging cannot be solved by simply taking more steps. Instead, it requires a step-by-step approach to identify issues based on data distribution, time alignment, and closed loop status correction.

Tutorial Experience: The key for ACT is not just training the checkpoint, but proving that it can continue to resume tasks from its own state distribution.

Case 4: Not all DAgger data will improve the results

** Phenomenon**: After adding more DAgger or failure-bucket data, some branches actually degrade. When full-reset failure-bucket data is mixed directly into the main training, the closed loop success rate decreases significantly.

Root cause: Failed state data and the reset-start main distribution may not be compatible. Direct high-weight mixing might cause the model to forget its initial behavior. Some correction suffixes also have issues where the timestamp/phase does not match the reset-start episode.

Fixing Policy:

• Preserve the checkpoint that performs best currently, so subsequent experiments cannot overwrite it arbitrarily;
• New data are only compared under the same seed group and physical_success format;
• Use timestamp offset for the correction episode;
• Down-sample the correction data, for example using sample weight 0.25;
• Keep the conclusions for the degraded branch, but do not use them as the main line.

Tutorial Insight: The goal of Dagger is to restore the state after a closed loop deviates, rather than discarding all failed trajectories back into the training set. When adding new data, it is necessary to first determine “which state distribution it restores”.

Case 5: SmolVLA does not fail to capture, but there is task bias.

** Phenomenon**: The SmolVLA baseline appears to have some success rate in random evaluations, but after enforcing red and blue cup commands, the difference is very obvious: red cups can succeed, while blue cups almost fail.

Evidence: The same set of seeds is fixed for two commands respectively:

textPlace the red mug on the plate.
Place the blue mug on the plate.

The strict physical results of baseline are the red cup 8/10 and the blue cup 0/10. The number of red/blue episodes in the dataset is not imbalanced, but the blue cup task is longer and more challenging, resulting in instability of the model before grasping the blue cup.

Figure 2: Enforcing red/blue command evaluation can distinguish between “the model does not capture” and “the model has bias toward certain tasks”.

Tutorial experience: The language policy must be evaluated by breaking it down into instructions. It is easy to overlook the bias in color, distance, and task difficulty if only focusing on the overall success rate.

Case 6: Copying data is not necessarily better than weighted sampling

** Phenomenon**: To fix the Blue Cup failure, the intuitive approach is to copy the Blue Cup episode. Copying 1.5x, 2x, and 3x does improve the Blue Cup performance, but the success rate of the Red Cup drops significantly.

Result Comparison:

Root cause: Copying the episode will change the distribution and statistics of the dataset episode, potentially pushing the model towards a specific color task. The Weighted sampler does not alter the original data file; it only adjusts the sampling probability, making it easier to roll back and compare results.

Tutorial experience: When balancing tasks for small datasets, prioritize smoothing and weighting the sampler/loss layer, and then consider duplicating the data. Evaluate the intermediate checkpoint as well; the final step may not be the best one.

Case 7: Videos are evidence, but the success rate is based on batch JSONL

** Phenomenon**: Some seeds fail in batch evaluation but succeed when recorded individually; some seeds succeed in the batch but fail when recorded separately.

Reason: There may be boundary nondeterminism in MuJoCo rollout, rendering, control timing, and policy sampling. A single video is ideal for teaching demonstrations, but it cannot replace batch JSONL statistics.

Recommended reporting format:

• Success rate is evaluated using a fixed seed batch;
• Success/failure videos are used to explain the behavior;
• For boundary failures, run the seed multiple times to determine whether they are fixed or occasional failures;
• The tutorial captions should clearly state that “the video represents the behavior, not the success rate statistics themselves”.

Tutorial experience: Do not replace a statistical table with the most visually appealing video, and do not dismiss an entire checkpoint due to one failed video. Both should be considered together.

Case 8: ROCm remote training should be scripted, not solely done in Notebook

** Phenomenon**: Notebook is suitable for explanation, but long training, batch evaluation, and video recording are more likely to encounter interruptions, paths, processes, and resource issues.

Several typical issues were encountered during this debugging round:

Batch tasks are suitable for being turned into scripts, for example:

bashcd "$PROJECT_ROOT"
export PYTHONPATH="$PWD:${PYTHONPATH:-}"
export TOKENIZERS_PARALLELISM=false

./.venv/bin/python eval_policy_success.py \
  --policy act \
  --act-policy-path "$MODEL_ROOT/act_best/step_5000" \
  --physical-success \
  --seed-start 1000 \
  --episodes 10 \
  --output-jsonl "$OUTPUT_ROOT/act_seen_1000_1009.jsonl"

Tutorial experience: Notebook is used for explanation and step-by-step observation, while scripts are used to generate comparable results. Before reporting the success rate, it is necessary to ensure that logs, JSONL, videos, and GPU status can be interpreted mutually.

Case 8.1: The fact that ROCm can access GPUs does not mean that PyTorch can train stably

现象：rocm-smi can see the device, torch.cuda.is_available() also returns True, but once x.to("cuda") is executed or a larger pi_0 weight is loaded, the Python process crashes directly. Such issues are easily misdiagnosed as model code, VRAM, or temperature problems.

Evidence: The minimum tensor smoke can be reproduced without starting training:

pythonimport torch

print(torch.__version__)
print(torch.cuda.is_available())
x = torch.randn(1024, 1024)
y = x.to("cuda")
print(y.mean())

If this code breaks completely, the subsequent ACT, SmolVLA, and pi_0 training have no diagnostic value. We will discuss the model later after debugging the runtime.

Root cause: Ryzen AI MAX+ 395 corresponds to gfx1151. The AMD ROCm 7.13 documentation has included Ryzen AI Max+ 395 / gfx1151 in the supported list. The ROCm 7.13 release notes also mention the addition of support for the Ryzen AI Max 300 series. Additionally, the AMD RDNA3.5 system optimization documentation indicates that the Ryzen AI Max series requires specific Linux kernel / KFD fixes; otherwise, the compute workload may fail to initialize or behave unpredictably.

So this is not a “inability to use the ROCm ecosystem”, but rather “the APU / RDNA3.5 / gfx1151 generation must strictly align the kernel, drivers, ROCm, and PyTorch wheel”. If a seemingly updated nightly wheel that does not correctly cover gfx1151 is installed, cuda.is_available=True may occur, but the GPU on the tensor will fail.

Fix: First switch to the ROCm / PyTorch wheel combination that explicitly supports gfx1151, and then perform the minimum smoke test. The combination that is available in the example experiment is:

texttorch 2.11.0+rocm7.13.0a20260425
torchvision 0.26.0+rocm7.13.0a20260425
rocm[libraries] 7.13.0a20260425
triton 3.6.0+rocm7.13.0a20260425
HIP 7.13.26162

The version number here is not required to be copied verbatim by everyone, but it indicates the troubleshooting principle: first align the hardware targets of PyTorch and ROCm gfx1151, then proceed with training. After installation, at least four checks should be run:

pythonimport torch

for dtype in [torch.float32, torch.float16, torch.bfloat16]:
    x = torch.randn(1024, 1024, dtype=dtype, device="cuda")
    print(dtype, x.mean().item())

big = torch.empty((512 * 1024 * 1024 // 4,), dtype=torch.float32, device="cuda")
print("512 MiB tensor ok", big.numel())

Verification: After the basic tensor, float32/float16/bfloat16, and 512 MiB tensor pass, then run the steps=0 initialization for pi_0, 1-step expert-only backward, and small checkpoint saving. Only when all these pass can we say that the runtime link is restored.

Tutorial experience: ROCm can now support APUs like Ryzen AI Max. However, “official support” does not mean that any pip wheel will work. The minimum tensor smoke is the first hurdle; if this hurdle is overcome, long training logs are of no value.

References:

• ROCm 7.13 compatibility matrix
• ROCm 7.13 release notes
• AMD RDNA3.5 system optimization

Case 8.2: Loading large model weights may also be a ROCm crash point

现象：Tensor smoke has been resolved, but crashes still occur when loading large model checkpoints like pi_0, especially on the path where safetensors directly loads weights onto the GPU.

Evidence: lerobot/pi0 or model.safetensors from the local pi_0 checkpoint has a large size, causing rapid memory usage during loading. If the crash occurs before training starts, rather than during the forward/backward phase, then the weight loading path should be suspected first.

Fix: Load the safetensors weights onto the CPU first, and then call policy.to("cuda") to move the entire model to the GPU. This patch does not change the model structure or training data; it only changes the loading order:

textsafetensors -> CPU state_dict -> policy.load_state_dict(...) -> policy.to("cuda")

This step resolved the crash issue caused by the large weight of pi_0 directly loading into the GPU in the example experiment. After fixing it, the local 20-step checkpoint can be loaded into the GPU normally, with a memory usage of approximately 7.5 GB.

Verification: Don’t just check if the import succeeded. At least one attempt is required:

1. Load the policy from the local checkpoint;
2. policy.to("cuda");
3. Print torch.cuda.memory_allocated();
4. Construct a minimum batch or run a 1-step smoke.

Tutorial experience: Adapting to ROCm does not always involve “changing the training code”. Often, it involves breaking down the order of data, weights, and tensor movement, so that errors occur at the most easily diagnosable steps.

Case 8.3: Do not overwrite the checkpoint path back to the base model during resume training

现象：When continuing training from the pi_0 500-step checkpoint, the command appears to be resume=true, but inside the training script, cfg.policy.pretrained_path is forcibly changed back to lerobot/pi0. As a result, it may happen that the model weights are reloaded from the base policy, while the optimizer state is restored from the local checkpoint, which semantically does not mean “continuing training from step 500”.

Root cause: For the convenience of new training, the tutorial code includes default logic before creating a policy:

pythonif cfg.policy.type == "pi0":
    cfg.policy.pretrained_path = "lerobot/pi0"

This is convenient for the first training, but it is dangerous for the resume. cfg.validate() has found the pretrained_model directory containing the local checkpoint from train_config.json, and it should not be overwritten later.

Fix: Set the default base model only when not in a resume mode:

pythonif cfg.policy.type == "pi0" and not cfg.resume:
    cfg.policy.pretrained_path = "lerobot/pi0"
elif cfg.policy.type == "smolvla" and not cfg.resume:
    cfg.policy.pretrained_path = "lerobot/smolvla_base"

Additionally, there is a detail in the parser of this old LeRobot version: config_path must be used in an equal sign format, and validate() can read back the path from the original CLI parameters:

bashpython train_model.py \
  --config_path="$MODEL_ROOT/checkpoints/000500/pretrained_model/train_config.json" \
  --resume=true \
  --output_dir="$MODEL_ROOT/pi0_resume_1500" \
  --steps=1500

If written as --config_path path/to/train_config.json, the main parser may be able to read it, but when looking at the parameters in validate(), it cannot be read, resulting in the error “A config_path is expected when resuming a run”.

Verification: The following should appear in the training log:

textresume: True
Creating policy
Loading weights from local directory
Start offline training on a fixed dataset
step:550 ...

If we continue from the 500-step checkpoint, the next training log should start near step:550, not step:50. This detail is very important as it confirms that the training state, scheduler, and optimizer are all properly connected.

Tutorial experience: Resume isn’t just about “starting up and it’s successful”. You need to check the step, learning rate, checkpoint source, and output directory in the logs. Especially for large model policies, incorrect resumes will waste a lot of time and distort the subsequent success rate analysis.

Case 8.4: Tutorials should include steps for accidental deletion and recovery, but private paths should not be exposed

现象：A remote data merge command used double-quoted SSH strings. The remote variables were expanded in the local shell in advance, resulting in an empty output path for fast_merge --overwrite, and a batch of untracked large directories within the project were cleared. The source code can be restored from Git, but untracked assets such as checkpoint, notebook_runs, and temporary data root will not return automatically.

Root cause: In remote multi-line commands, variables like $OUT_ROOT must be expanded on the remote side. If the entire command is placed within local double quotes, the variable may be expanded locally first, resulting in an empty string when received on the remote side.

Fix:

1. Use heredoc for all complex remote commands:

   bashssh user@host 'bash -s' <<'REMOTE'
   set -euo pipefail
   OUT_ROOT="/path/to/output"
   echo "OUT_ROOT=$OUT_ROOT"
   # 后续命令
   REMOTE
   

2. All scripts containing --overwrite, rm -rf, and shutil.rmtree should first apply path protection, rejecting empty paths, ., project root directories, and parent directories.

3. Large checkpoints, datasets, and evaluation results should not exist solely in untracked directories. There must be at least one reconstruction description: which HF repo the data comes from, which script was used for collection, and which config the checkpoint was trained with.

Verification: After fixing, the erroneous command should fail directly instead of continuing to execute:

textrefuse unsafe output_root: .
refuse empty output_root

Then, the teaching baseline is restored from the public dataset: demo_data_language is re-downloaded with 20 episodes / 2621 frames, observation.state is 6D, and action is 7D; followed by running steps=0, 1-step backward, 20-step checkpoint, 500-step checkpoint, and evaluating with fixed red/blue small panels.

Tutorial insights: For public tutorials, do not write “accident details” as private logs, but should retain learnable engineering lessons: remote variable expansion, protective overlaid scripts, reconstructable public data, and checkpoints that do not go into the repository, while training commands and results can be reviewed.

Case 9: pi_0 goes through permissions and smoke first, then discusses training

** Phenomenon**: pi_0 cannot be run successfully with just simple training. It relies on gated model permissions, Hugging Face tokens, weight downloading, caching, and large model initialization. Some failures occur before training begins.

Troubleshooting Order:

1. Ensure the token is valid, but do not print the token in the command line or logs;
2. Verify that the account has accepted the terms of gated models such as PaliGemma;
3. Check whether lerobot/pi0 and related models can be accessed;
4. Confirm that the remote network or proxy can download the weights smoothly;
5. Run a 1-step smoke test first;
6. After the smoke test passes, start the formal training and forced red/blue evaluation.

Stage Status: In this topic example, pi_0 has been trained with pre-gating and entered the post-training diagnosis stage. Permissions, downloading, policy construction, and training are functioning, indicating that the links are available; whether the replication is successful depends on the physical_success with a fixed seed.

Tutorial experience: For large model policies, permissions, downloading, and constructing policies are themselves a layer of experimental gatekeeping. If 1-step smoke is not achieved, do not initiate long-term training.

Case 10: The failure of pi_0 raw occurs mainly at the end, but it does not mean no knowledge is learned for the entire trajectory

** Phenomenon**: After pi_0 is fine-tuned on a small dataset, the raw policy remains 0/4 final_physical_success in the reference open-loop replay of a 2 blue and 2 red small dataset. However, batch rerun with the same settings may result in 1/4 on boundary samples. Therefore, this small dataset is more suitable for identifying failure stages, not for using as a standalone generalization success rate. When expanded to full 20 open-loop replays, the current raw policy becomes 1/20 final_physical_success and 3/20 physical_success_ever. After further 20 seed closed-loop strict evaluations, the raw policy is 0/20 physical_success, while the old geometric口径 only yields 2/20.

Evidence: By examining the final states of each trajectory in more detail, it can be seen that the raw policy often brings the cup close to the plate, but still lacks the final release, elevation, and stable placement. For example, in episode 2, the final state xy distance for the raw policy is only 0.0087 m, with the cup remaining upright, while TCP z is only 0.8953 m, failing to exceed the strict success threshold. In full20 open-loop, there were 3 instances where the cup briefly reached physical_success, but only 1 instance achieved a successful final state; in closed-loop strict, none maintained strict physical success, with the failures mainly due to “the posture is not upright after lifting” and “insufficient elevation”. This clearly shows that “close” and “stably completed” are not the same thing.

Figure 3: The pi_0 raw policy is close to the target, but the final stages of release / raise / stabilize do not stabilize across physical success conditions. The green bar represents the same diagnostic case after the access script terminator.

Technical Background: This type of failure is not due to “pi_0 being inherently capable of learning intermediate actions”; rather, it is more like an amplification effect at the end of a sequence involving small data volumes, long trajectories, and action chunking.

• pi_0 paper places large-scale, multi-task, cross-robot data and post-training recipes in a very core position. It particularly emphasizes the value of corrections and recovery behaviors within diverse data. Here, we only fine-tune with small-scale cup-grasping data, and the coverage and recovery actions are far from being on the same scale, so it is impossible to expect the model to automatically generate unseen tail recovery strategies.
• The Real-Time Action Chunking explanation in Physical Intelligence describes models like pi_0 as predicting one action segment at a time, rather than predicting a single action per time step; in the example, 50 actions correspond to approximately 1 second of control. Action chunking makes actions more coherent, but transitions between chunks, inference delays, and pauses that occur during training will introduce additional issues.
• Real-Time Execution of Action Chunking Flow Policies clarifies the problem further: action chunking improves temporal consistency, but chunk boundaries and delays still affect real-world execution. After small-data fine-tuning, the early phase—“approaching the cup, moving near the plate”—can be averaged out, while the last few centimeters of release, lifting, and holding are easily compromised by cumulative errors.
• The Human-in-the-Loop data collection documentation for LeRobot specifically discusses the issue of compounding errors and state coverage beyond successful teaching: if only successful trajectories are considered, once the policy deviates, it will enter states not seen in the training data; by recording the correction and recovery processes into episodes, the model has a chance to learn to recover from deviations.

Put these back into our gripper task, and the phenomenon can be explained: the raw policy doesn’t completely remain still, but pushes the cup toward or near the plate. However, release / raise / stabilize requires very small end误差, correct gripper timing, and sufficient final state retention. The training loss continues to decrease, but it may not automatically fix this issue, as what is missing are tail-end state coverage, phase boundaries, and physical success goals, not just more steps.

Fix verification: To confirm whether the issue was concentrated at the end, we did not directly announce the success of the raw policy. Instead, we added a diagnostic scripted finisher: a average release / raise / stabilize action was generated from the actions at the end of the dataset, and it stopped immediately after reaching physical_success. This hybrid approach achieved 4/4 final_physical_success in a 2 Blue 2 Red small set.

Then, the same approach was extended to all 20 episodes of demo_data_language: the teacher-forced open-loop result for pi_0 step1500 + all-episode template tail40 + blend20 + stop-on-physical-success is 4/20 final_physical_success, and the successful episodes are 7, 9, 13, 14. The red cup is 2/10, and the blue cup is 2/10. This result strengthens the conclusion: fixing the end section of the script can indeed rescue some trajectories that are close to success, but it does not provide a universal solution. Most of the failed samples still have xy—being remote and not upright, with a few even failing to stabilize at all; the contact, grip, handling, and release at the beginning are coupled, and cannot be solved by averaging the last 40 frames alone.

Next, we conducted another set of more thorough causal analysis: instead of using the pi_0 prefix, we first let the scripted oracle reach different takeover points, and then transfer control to the 10eps suffix-only finisher. The prefix 120/180/240/300 is all 0/2 physical_success on seeds 1000/1001. In some rollouts, the cup is lifted and kept upright, but ultimately xy remains at 0.22-0.60 m, and the cup is moved away from the plate. This result indicates that failures are not just due to “too early or too late takeover”, but also because the target and stage conditions of the current finisher are not properly established.

The dense action-target trace observed later also supports this judgment: the seed 1000 follows the oracle prefix 180, then comes the finisher, and the phase span of the first 80 finisher control ticks is move_pregrasp 28步 -> move_grasp 35步 -> close_gripper 17步. In other words, after the cup is raised, the finisher returns to a stage close to and re-grasping. The corresponding TCP target to the disk center is average 0.481 m, while to the cup it is average 0.147 m; the relative deviation from the disk center y ranges from +0.295 m to +0.554 m. This number is more convincing than just looking at the video, as it directly indicates that the subsequent policy did not turn “disk center pose” into a stable control target.

Just manually fast-forwarding the phase isn’t enough. We need to use the finisher. move_pregrasp/move_grasp/lift dwell pressed to step 1, prefix 180、seeds 1000/1001 Still 0/2 physical_success, terminal state xy About 0.534/0.494 m. Therefore, the subsequent step is not simply adjusting the dwell value, but rather incorporating the phase offset / phase head and plate-relative state into the next policy version.

When analyzing further, the schedule-shift diagnosis shows that the same finisher’s dataset_schedule starts playback from the move_preplace or lower_to_plate start point, without changing the model weights. Starting playback at move_preplace remains at 0/2, but the final state xy for seed 1000 is changed to 0.140 m, indicating that the phase start indeed helps. Seed 1001 is still at 0.410 m, and the gripper opens too early. Starting playback at lower_to_plate results in both seeds being at 0.44-0.45 m, suggesting that jumping directly to the lowering/release stage is too late. Adding phase-scripted gripper still results in 0/2, but seed 1001 changes from 0.410 m to 0.171 m. This step clarifies the problem boundaries: phase and gripper timing are necessary conditions, but not sufficient; the plate-relative transport target remains unstable in the later part of Pi0.

So the next step is not to blindly train the same 19D finisher, but to explicitly put the plate-relative target into the state. The new 22D state is written as:

textjoint6 + timestamp + phase_index_norm + phase_onehot11 + tcp_to_plate3

tcp_to_plate is the current relative vector from TCP to the center of the plate. In the training data, an approximation is made using prefix_end_debug.tcp_pos during collection and eef_abs from the previous frame. During closed loop evaluation, the true tcp_link - body_obj_plate_11 from MuJoCo is directly used. This design corresponds to the root cause in the previous trace: the target of the later-stage policy still remains at the side of the cup. Therefore, “where is the plate relative to me” is used as a learnable input, instead of continuing to let the model implicitly guess from small data images.

This round also encountered a typical migration issue: the normalize_inputs.buffer_observation_state.mean/std of the old checkpoint is 19-dimensional, while the new state is 22-dimensional. Direct loading would result in a shape mismatch. The fix does not involve retraining everything; instead, a copy of the old checkpoint is created, and only the two state normalizer buffers are removed, allowing the 22D dataset statistics to be reinitialized. The main weights of Pi0 continue to use the old finisher. After 250 training steps, the loss is near 0.012-0.019, and the ROCm training is stable, with no OOM or kernel crashes.

Keep the diagnostic boundaries intact during verification: oracle prefix180 runs first with the prefix, and then the 22D finisher takes over from move_preplace schedule. The results of 10 seeds have improved from the old 0/2 counterexample to strict 5/10 and legacy 6/10. Successful seeds are 1000, 1001, 1004, 1005, 1006; among failed seeds, 1002/1007/1008/1009 has been lifted but failed to stabilize at the center of the disk, while 1003 is close to the center but the cup is upside down. These results take the conclusion one step forward: the plate-relative state is a valid clue, but the gripper timing, stability after release, and returning from the scaffold to the raw policy have not been resolved yet.

Another narrow causal comparison: EEF/arm still uses Pi0, but the gripper is changed to phase-scripted open/close. As a result, strict moves from 5/10 to 7/10, and successful seeds become 1000, 1002, 1003, 1004, 1005, 1006, 1008. This result is more reliable than the assumption that there is a problem with the gripper: it indicates that the gripper timing can indeed fix some cases of cup tipping/loosement failures, but it is not a sufficient condition. The remaining failures 1001, 1007, 1009 all involve red cups, and the final state xy still remains in 0.14-0.24 m, which suggests that more stage data or target representations are needed for the handling point and release height of red cups.

Correct the seed口径 first

After completing this debugging cycle, we checked the environment randomization code again and found that y_env2.py had written any seed as np.random.seed(seed=0). Therefore, the historical results in 1010-1039 only changed the policy sampling randomness of Pi0, while the positions of cups and plates in the environment remained unchanged. These results are retained because they still clearly demonstrate how the schedule, gripper head, transition head, and reset protocols affect the stability of fixed scenarios; however, they are no longer referred to as unseen positions or spatial generalization in this section.

After being fixed to np.random.seed(seed=seed), the strict-input visual/history head has only 1/4 on the actual random environment seeds 41-44. This indicates that the old scaffold debugging solved the problem of action timing and state semantics in fixed scenarios, but multi-position data coverage remains another independent hurdle.

Then, the sample size is expanded to the policy sampling seeds (the environment is actually set to seed 0) 1010-1039. If the 10 schedule templates are still used in a loop, this improved scaffold is strict 21/30, legacy 23/30, red cup 12/18, and blue cup 9/12. This result indicates that 22D tcp_to_plate and scripted grippers are not just coincidentally effective, but failures are also very concentrated: all 9 strict failures correspond to schedule episodes 1, 7, 9.

By counting the phase length for each template, the root cause becomes clear. For episodes 1, 7, 9, move_preplace consists of only 72-75 frames; for longer templates, episodes 0, 2, 3, 4, 5, 6, 8 have 120-123 frames. In other words, the failure is not due to “the seed being inherently difficult,” but because the transfer phase is too short—the cup has not even reached above the center of the plate, and the schedule has already entered lower / release. This conclusion is consistent with video behavior: the failed samples are often already upright, but the final state y remains in front of the plate, as if there is still a short distance to transfer before it is released.

Just changing the schedule template can verify this judgment. Replace all the 9 failed seed values with the long template episode 0, resulting in 9/9 strict; then force all 30 policy sampling seeds (the environment is actually set to seed 0) to use the long move_preplace template of episode 0, yielding strict 30/30, legacy 30/30, red cup 18/18, blue cup 12/12, average final state xy of approximately 0.0281 m, and maximum xy of approximately 0.0739 m. This step does not involve changing models or retraining, indicating that the short move_preplace schedule is the biggest fixable bottleneck in this round.

Then start dismantling this scaffold. First, remove the two dependencies separately: no scripted gripper is needed, and while keeping the oracle prefix and long schedule, the policy sampling seeds (the environment is actually fixed to seed 0) 1010-1019 are only strict 5/10; replace the oracle prefix with the current policy prefix. Even if the scripted gripper and long schedule are retained, it is only strict 3/10. These two numbers indicate that gripper timing can be prioritized as a local bottleneck, and the prefix policy cannot immediately replace the oracle prefix.

Thus, a small logistic gripper head was trained to only predict the opening and closing of the gripper, while the EEF/arm still comes from the 22D finisher. In the first version, a full 22D state was used, and the accuracy on the training set was 100%. However, it failed when deployed with seed 1010; it learned not only the phase window but also the correlation between joints/TCP in the training set. Once the closed loop deviated, it became unstable. In the second version, only the phase input was used, focusing solely on timestamp + phase_index_norm + phase_onehot11. The accuracy on the training set remained 100%, but the closed loop review passed: with seed 1010, smoke was 1/1, the policy sampled seeds (the environment was actually set to seed 0) 1010-1019 as strict 10/10, full 1010-1039 as strict 30/30, legacy 30/30, red cup 18/18, blue cup 12/12, and the average final state xy was approximately 0.0272 m, while the maximum value xy was approximately 0.0735 m.

This step is highly educational: the size of the learning head doesn’t necessarily need to be larger. For short events like gripper opening and closing, phase-only input is more stable than full-state input, as it clearly indicates “which gripper window should be in use at this moment,” while avoiding the inclusion of mechanical arm details that could cause the closed loop to drift. Now we can say that the hand-written gripper rules have been replaced by a trainable, saveable small head; however, this is still not the same as raw pi_0 success, because the oracle prefix, manual move_preplace starting point, explicit tcp_to_plate, and the forced long schedule remain.

Break down the second layer: enforce a long schedule for episode0. The new adaptive move_preplace gate still uses schedule episodes 0..9, but no longer releases immediately when the short template reaches lower_to_plate. If live tcp_to_plate_xy is too large, hold the last frame move_preplace; wait until TCP approaches the center or reaches the safe limit, then proceed to lower / release. This change does not alter the checkpoint, EEF/arm output, or the gripper head; only the “fixed-length template” scaffold is replaced.

The most stable policy in this round is xy=0.05m,min_steps=20,max_steps=180: the policy sampling seeds (the environment is actually fixed at seed 0) 1010-1019 is strict 10/10, the complete 1010-1039 is strict 30/30, legacy 30/30, red cup 18/18, blue cup 12/12, and the average final state xy is approximately 0.0281 m. The maximum value xy is approximately 0.0617 m. This indicates that it is possible to complete the short schedule’s transfer phase according to the live progress without forcing all samples to go through episode0.

But it is not yet a complete learned transition head. The trace of seed 1011 shows that 0.05m enters lower_to_plate only as a max-step fallback to step 180. During the switch, live tcp_to_plate_xy is approximately 0.0574 m. When the threshold is relaxed to 0.09m, seed 1011 can switch actively at step 163, which resembles a progress gate more. However, once the number of seeds reaches 30, seed 1034 fails, and the final state goes from xy to 3.16 m. 0.08m is also not a steady state: the probe for seed 1011/1034 succeeds, but among the first 9 entries before full-run, seed 1012 has already failed strictly. This result indicates that, behind the transition head, both xy, z, lift/upright/contact, and phase elapsed must be considered simultaneously; one overly wide xy threshold is not sufficient.

Then replace the adaptive gate with a logistic transition head. In the first version, the head used tcp_to_plate_x/y/z, tcp_to_plate_xy, abs_z, and the number of steps within the phase. The accuracy on the training set was approximately 99.27%, but the seed 1010 failed to go live. The trace shows that the switch occurs at the finisher step 59-60. At that time, the live tcp_to_plate_xy was still around 0.35 m, and the cup had not reached above the center of the plate. Such failures are quite common: a high accuracy on the training set does not mean a reliable closed loop; the full-state head treated direction and height as the release condition.

In the second version of the head, only tcp_to_plate_xy + local_step_norm is retained. The accuracy of the training set drops to approximately 95.01%, but the closed loop becomes more stable: the seed 1010 changes from failure to strict success, with the trigger point at the finisher step 136, not max_steps as a fallback. Subsequently, the policy samples seeds (the environment is actually set to seed 0) 1010-1019, which is strict 10/10; the complete 1010-1039 is strict 30/30, legacy 30/30, red cup 18/18, blue cup 12/12, and the average final state xy is approximately 0.0279 m, while the maximum xy is approximately 0.0741 m. Out of 30 cases, 29/30 is triggered by the transition head, and only seed 1021 reaches max_steps=180 safely as a fallback, yet it still succeeds.

This step is crucial: the second layer of scaffolding has progressed from a “forced long template” to “learnable transition head + safety upper limit”. It is not yet raw pi_0 success, nor is it a completely unstructured dynamic phase policy; the oracle prefix, explicit tcp_to_plate, schedule tail, and max_steps safety net remain. But now we can say more precisely: the gripper timing of the Pi0 posterior finisher and the move_preplace -> lower_to_plate transition have both been equipped with a trainable small head version, and they have passed strict physical evaluations on 30 policy sampling seeds in a fixed environment.

The third layer continues to analyze the contact primitive. The initial all-head counterexample is very valuable: when pregrasp_to_descend / descend_to_close / close_to_lift / lift_to_hold is trained according to phase tail, the accuracy on the training set is decent. However, for smoke seeds 1010/1011, 0/2 occurs. In the trace, descend_to_close triggers when it is about 0.08 m above the grasping floor for TCP, and subsequently close_to_lift occurs earlier, causing the hand to lift before the cup even enters the gripper. This failure indicates that phase tail is not a contact condition; in particular, the phrase “the phase is almost over” cannot be used as a substitute for “it’s safe to close the gripper now”.

The fixed version only replaces pregrasp_to_descend with a geometric label, and adds features such as tcp_to_grasp_xy, abs_tcp_to_pregrasp_z, tcp_to_floor_z, and abs_tcp_to_floor_z; descend_to_close still follows the phase-tail timing, but descend_floor_guard is added during deployment, and close is prohibited until TCP approaches the floor. In this way, smoke 1010/1011 becomes 2/2, and the complete fixed environment policy sampling 1010-1039 is strict 30/30 and legacy 30/30. Among the 30 items, pregrasp->descend, descend->close, and close->lift are all 30/30 triggered by the head, and the floor guard blocks 342 high-altitude closes. This version lacks pregrasp/descend handwritten geometric transitions compared to the previous version, but it still retains target-relative contact scaffold, floor safety guard, and the later finisher.

The fourth layer continues to disassemble the tail portion of dataset_schedule. The new dynamic_timed finisher starts from move_preplace, generating timestamps and phase one-hot using an online phase tracker, without reading the dataset schedule anymore. The first complete policy sampling seeds (the environment is actually fixed to seed 0) 1010-1039 are strict 27/30, and all failed seeds 1021/1031/1036 are not no-contact: 1021 stops outside the disk after being lifted, 1031 is thrown away, and 1036 the blue cup stays in front of the disk. This phenomenon resembles an inappropriate fixed dwell, but code analysis reveals a more specific cause: --tcpplate-prefix-target-state is intended for the tcp_to_target state prepared by the prefix policy, yet it also applies in the dynamic finisher; the finisher model uses tcp_to_plate during training, but receives tcp_to_target when online.

When fixing the issue, change the state builder to be stage-aware: use tcp_link - target_mug for the prefix stage, and tcp_link - plate for the finisher stage. Also limit the contact primitive to only the prefix, so that the dynamic backend is not taken over by the contact scaffold due to the absence of scheduled_states. After the fix, run the failed seeds 1021/1031/1036 first; 0/3 is replaced by 3/3. In the same environment, run all the policy sampling seeds continuously (the environment is actually fixed to seed 0) 1010-1039, including strict/physical 30/30, legacy 30/30, red cup 18/18, blue cup 12/12, mean xy=0.0244 m, and max xy=0.0993 m. The largest xy and the lowest upright cos value both come from seed 1036, which appears to be a boundary success sample.

When continuing the trace, a truly interesting bug emerged: during continuous evaluation in the same environment, the initial position of the last seed sometimes did not fall within the sampling range of that seed. For seed 1035, the initial position was approximately [0.2679, 0.1625, 0.8441], but the normal position after a separate reset should be about [0.2944, 0.1912, 0.8447]. This indicates that the problem is not with the seed itself, but rather the residual contamination from the qvel / ctrl / free-joint dynamics of the previous episode. SimpleEnv2.reset(seed) will reset the object’s qpos, but it does not clear the underlying MuJoCo mjData completely first.

Two additional evaluation protocol switches have been added here: --fresh-env-per-episode to create a new environment for each seed, suitable for review in the small panel; --hard-reset-sim-data to call the underlying parser reset first when reusing the same environment to clear residual dynamics, suitable for complete batch evaluations. The fresh-env small panel 1036/1035/1020/1029 is equivalent to 4/4, but frequent new environment creation occasionally causes errors in asset providers in the large panel, so the hard-reset protocol is ultimately recommended. Using --hard-reset-sim-data, the full policy sampling seeds (the environment is actually set to seed 0) 1010-1039 are run again. The results are still strict/physical 30/30, legacy 30/30, red cups 18/18, blue cups 12/12; mean xy=0.0219 m, max xy=0.0450 m, minimum lift 0.1093 m, minimum upright cos 0.9504. This time, seed 1036 does not touch the edges, while the previous 0.0993 m is more similar to the evaluation artifact caused by residual reset across episodes.

This case is very suitable as a troubleshooting template: when the same wrapper supports both prefix and finisher, the global switch can easily go beyond its limits. tcp_to_target is a good feature for grasping at the front end, but becomes a problem in the back end; tcp_to_plate is a good feature for handling in the back end, but is not sufficient at the front end. Such bugs may not cause errors or be exposed during 2 seed smoke tests, but manifest as “occasional drift, misalignment, and handling issues” in the full seed panel. Similarly, an unclean reset of the evaluation environment may also mask “policy boundary failure” as “model capability issues”. Therefore, during debugging, it is necessary to record the stage, state semantics, reset protocol, and success metrics, rather than just focusing on the final number shown by physical_success.

This is not a successful raw pi_0. A more accurate description is: the later-stage finisher of Pi0 achieved repeatable success for the first time after adding the center relative pose; after further adjusting the gripper window and handling duration, the engineering scaffold could complete the task reliably across 30 policy sampling seeds in a fixed environment; by replacing the gripper rules with phase-only learned head, changing forced long episode0 to learned transition head, and finally substituting dataset_schedule at the end section with stage-aware dynamic_timed finisher, it remained stable under the hard-reset clean protocol. In the next round, in addition to continuing to dismantle the scaffold, priority is given to re-collect data from multiple positions using the corrected environment seed.

Transitioning from scaffold to strict-input visual/history head

Continuing to add training steps to the raw Pi0 main model did not result in successful fresh closed-loop performance. Therefore, the experiment was changed to two layers: retaining the Pi0 visual language backbone and the original policy output, and training a visual/history head that only uses information available during deployment. The new complete task recovery data is divided into two groups, each with 10 episodes, totaling 20 samples; the training seeds are 21-32, and the validation seeds are 33-36. The number of filtered frames for both groups is 3586 and 3164 respectively, including both near and contact scenarios, as well as handling, release, and final stabilization, rather than just training a short segment of the contact suffix.

The input for visual keyframe head is the visual token, language embedding, current 6D robot proprio, and 7D EEF/gripper commands executed by the previous policy, obtained from the agent/wrist image through the Pi0/SigLIP backbone. There are no target/plate coordinates, GT phase, oracle action, or future labels here. The gripper GRU removes the last dimension of the direct head’s previous-gripper feature, and only uses the previously executed EEF action [:6] additionally, to avoid directly copying the gripper state from the previous frame.

The overall validation accuracy of the first version of the linear gripper head appears to be 99.66%, but the held-out release transition is 0/4. The reason is straightforward: a trajectory consists of hundreds of frames, and there are only a few frames where the actual close/release switch occurs. The overall accuracy is almost entirely contributed by maintaining the current state. After changing to a GRU sequence head, a higher loss weight is applied to the transition frame, and the close/release event is used as a separate model selection metric. The best epoch is 20; the validation set has 2164 sequence samples, 5 close events, and 5 release events. Under the selected threshold 0.825, close/release values are 5/5, and the balanced accuracy is approximately 96.8%. During deployment, a more conservative 0.5 threshold is used for the closed loop gate.

The judgment for closed loop success is also tightened. In addition to the old geometric success criteria, it is required that the actual lift be at least 0.03 m, and maintain upright cos by 3 consecutive steps with a value of at least 0.7. The height difference between the cup and plate should not exceed 0.08 m, and the XY displacement of the plate should not exceed 0.05 m. Additionally, the placement conditions must be met continuously for 5 steps. This change will reject two types of old false positives: when the cup hangs above the plate, and when the gripper pushes the plate away together. With a fixed environment seed of 0 and 12 policy sampling seeds in the same batch, the raw protected Pi0 is 0/12, and the visual/history GRU head is finally strictly 6/12; the successful policy seeds are 3, 8, 18, 33, 34, 40. The old relaxed terminal state口径 showed 8/12, where seed36 and seed39 were excluded from the final口径 due to pushing the plate by approximately 12.27 cm and 5.59 cm, respectively.

This set of enhancements and the recompletion of complete task data are directly related, but it cannot be attributed solely to “an increase in data volume”. What truly works is the simultaneous occurrence of four factors: complete task phase coverage of data, observable temporal input from the visual/history head, separate weighting for gripper transition, and the execution end converting gripper probability into clear open/close commands. It also does not solve all problems: after fixing the repair environment seed, the truly random environment 41-44 has only 1/4, indicating that the current 20 recovery data still mainly cover fixed scenarios.

There is another example of a language protocol failure. Using the full instruction Pick up the blue mug and place it on the plate. during training, the head of seed3 activates for 269 steps, and the final strict judgment is reached at step 270. When replaced with a shorter equivalent instruction Place the blue mug on the plate., the prototype distance gate refuses to take over throughout the process, resulting in failure. The current small-data head uses valid language input, but still lacks language rewriting generalization. In the next round, 30–50 multi-position trajectories should be relected after correcting the seed, along with prompt paraphrase enhancement, and then three gates for fixed scene, random position, and language paraphrase should be applied respectively.

Figure 4: Raw-vs-hybrid videos of the same episode. The raw pi_0 on the left did not succeed through strict physics, while it succeeded after the scripted finisher was applied on the right. This video is used to identify the bottleneck at the end, not as evidence of the success rate of raw pi_0.

Next steps: Do not continue blindly increasing the training steps. First, break down the trajectory into several stages: near proximity, grasping contact, handling, release, raising the arm, and final stabilization, and then validate each stage separately. For this time, pi_0, a more reasonable approach is to progress in four parallel directions:

1. Align the episode boundaries with the actual task completion point. If the model is expected to learn “open gripper, lift hand, stabilize wait, return to safe posture,” it must be recorded in an episode; if it is merely a reset after evaluation, do not integrate it into the success criteria of the task.
2. Supplement with phase-specific data. Specifically collect or generate release/raise/stabilize segments that include corrective actions after slight deviations, rather than just extracting a short segment from a perfect trajectory.
3. Use 22D tcp_to_plate finisher as the new post-phase diagnostic baseline. Strict 5/10 for non-scripted gripper, strict 7/10 for phase-scripted gripper, a larger fixed-environment policy sampling 30 seed short schedule template as 21/30, and after forcing the long move_preplace template, use 30/30; all these numbers must be labeled with scaffold conditions and cannot be written as raw success.
4. Keep the phase-only learned gripper head as the first-level scaffold removal result. The full-state head, although trained on 100%, fails at closed-loop seed 1010; the phase-only head is strict 30/30 on fixed-environment policy sampling 1010-1039, indicating that inputs for short-event heads need to be controlled.
5. Keep adaptive move_preplace gate as the second-level scaffold removal engineering version. 0.05m + max_steps=180 is strict 30/30 on fixed-environment policy sampling 1010-1039, and no longer forces long episode0; however, the counterexample of 0.08/0.09m shows that fixed thresholds cannot be written as a fully learned transition head.
6. Keep xy_step learned transition head as the learnable version for the second-level scaffold removal. The full-state transition head training set is more accurate, but seed 1010 closed-loop failure occurs early; after using only tcp_to_plate_xy + local_step_norm, fixed-environment policy sampling 1010-1039 is strict 30/30, and 29/30 involves the head actively triggering, not relying on max-step as a fallback.
7. Keep pregrasp-geometry/contact transition head as the third-level scaffold removal result. Naive all-head will close at high altitude; with geometric labels and descend_floor_guard, fixed-environment policy sampling 1010-1039 is strict 30/30, but it remains a contact scaffold, not a raw policy.
8. Keep stage-aware dynamic finisher as the fourth-level scaffold removal result. Before fixing, dynamic_timed was 27/30; after fixing the target/plate state boundaries of prefix/finisher, it becomes 30/30; after using --hard-reset-sim-data to eliminate cross-episode dynamics, it is still 30/30. This shows that interface semantic boundaries, reset protocols, and model capabilities are equally important.
9. Continue removing the scaffold: The next step is not to use another more complex global switch, but to make the exit conditions of the dynamic phase, release/stabilize conditions, into learnable heads or cleaner data, and gradually release the policy prefix from the contact scaffold.
10. Keep scripted finisher as diagnostic and engineering fallback. It can prove that the tail segment is one of the bottlenecks, but with all 20 items, only 4/20 exists, and the old oracle-prefix handoff scan has only 0/2. Therefore, pi_0 + finisher cannot be written as raw pi_0 success, and fixed tail segments cannot be used as a generalization solution.

The more complete experimental route is described in the section “How to Improve the pi_0 Success Rate” of pi_0 permission smoke and training gate control . This section combines the current baseline, excluded routes, phased policies, EEF/TCP actions, on-policy correction data, and new evaluation thresholds. Subsequent experiments will be carried out according to this roadmap.

Case 11: The frame-by-frame oracle tags may not form an action chunk

** Phenomenon**: In order to simulate a deviation state for Pi0.5, the first version of the collector executes the policy for 40 steps first, and calculates an oracle correction action for each policy state. After adding 5 data points to training, the gripper’s offline error decreases, but both new seeds and old collected seeds remain 0/4. This is not simply due to insufficient Dagger data, but rather because the temporal semantics of the data are invalid.

Pi0.5 is trained with one supervision for the next 50 actions. The first version of the data collection logic is equivalent to:

textobs_0 --student_action_0--> obs_1 --student_action_1--> obs_2 ...
  |                          |
  +-- oracle_label_0         +-- oracle_label_1

oracle_label_0 merely indicates “how to correct the issue in the first step if obs_0 takes over”, but it is not executed. obs_1 is obtained after executing student_action_0, not after executing oracle_label_0. Similarly, oracle_label_1 is just another counterfactual single-step label. Treating [oracle_label_0, oracle_label_1, ...] as a future action chunk is equivalent to telling the model to execute a trajectory that has never occurred in the environment.

This issue may not immediately reveal itself in behavior cloning at single step, as each (observation, one_step_label) is reasonable on its own; however, it is a structural error for the action-chunk policy. In the chunk, actions 2 to 50 are defaulted to be established on the state where the previous tag has been executed, while the observation state here is driven by the student action, resulting in a discrepancy between the two state chains.

Fix: Changed to continuous expert takeover. Pi0.5 executes 10 steps first, and the expert takes real control of the environment from the current deviation state. The collector only saves the continuous suffix after takeover:

textstudent prefix --> takeover_state
takeover_state --oracle_0--> state_1 --oracle_1--> state_2 ...
                    |                    |
                  save                 save

In this way, each future action is based on the state where the previous oracle action has been executed. Only when there are 50-step chunks does it have real dynamics. There are 20 candidate seeds 2100-2119, and 11 of them are saved after successful strict physics execution, totaling 4504 frames; failed trajectories do not enter the training set. During the long-term fresh-env collection, another Too many open files occurred, but eventually it was changed to using 5 candidates per batch and adding them with --resume. After gathering all data, the episode index, frame count, and manifest were checked, and no manual metadata modification was required to fix issues.

Even after correcting the data, it is necessary to honestly evaluate the closed loop. After aggregation training with clean53 and coherent recovery11, the gripper’s MAE decreased from 0.25659 to 0.08261, indicating that continuous recovery data indeed taught some part of the opening-closing timing; however, the xyz MAE did not improve, and the new random seeds 2120-2123 remain strict 0/4. Therefore, there are two independent conclusions from this example: the first version of DAgger chunk is ineffective in terms of time, and it must be corrected; after improving continuity, spatial proximity direction remains another bottleneck, and we cannot claim the model is successful just because data engineering issues have been resolved.

In future audits of action-chunk DAgger, you can check each item individually:

Table 6: The key for Dagger is not having an “oracle tag”, but rather the tag, observations, and environment state transitions must belong to the same executable trajectory.

Case 12: index exists entirely, but the action chunk may still cause frame corruption

** Phenomenon**: After combining 64 EEF-delta data and 27 coherent recovery values, the metadata shows 91 episodes and 35284 frames; all index entries are also arranged exactly as 0..35283. All checks passed, but during training sampling, some positions’ action_chunk[0] values do not match the action in the current parquet row. Continuing training will cause misalignment between observations and future actions, and the loss may still decrease, but the closed loop cannot learn the order of the stable phase.

The chunk query for LeRobot can be simplified as follows:

textcurrent_observation = physical_rows[idx]
future_actions = physical_rows[clamp(idx + delta, episode_start, episode_end - 1)]

Here, idx is the physical row position of the Hugging Face Dataset. It does not involve reading the value of index in the current row and then checking again. If the parquet file is loaded by file name, and the stored index 21510 is located at the physical position 21105, even if the index set remains complete, idx + delta will point to an incorrect frame. There are a total of 4909 misaligned physical positions in the original aggregated data, and the first misalignment is 21105 -> 21510.

This issue will also contaminate the sampler. In the old training summary, the gripper transition center jumps from 21028 to 25707/25937, and then back to 21195; the transition center should increase monotonically with the episode order, and this abnormal sequence is evidence of the disorder in the physical lines.

Root cause: The aggregation tool verified that “the global index is sequential after sorting” and “all episodes exist”, but it did not verify that “according to the actual loading order of parquet, the stored index at position i is i”. After the missing episode is added to restore the order, the file number, episode number, and global index are no longer consistent; the old per-episode action stats still remain as eef_abs, further increasing the risk of misjudgment.

Fix: Keep the source data read-only, and create an independent canonical copy:

1. Find the source parquet for each trajectory using the actual episode_index;
2. Sort each episode according to frame_index;
3. Write each episode into a separate file file-{episode_index}.parquet;
4. Rebuild the global index and dataset_from/to_index in episode order;
5. Fix the data/meta file mapping and recalculate per-episode/global stats from actual values;
6. Write back and compare all non-index Arrow columns, image bytes, status, actions, and task information must remain unchanged.

The supporting scripts are [pi05_canonicalize_lerobot_v3.py](…/…/…/16-Special Team Learning/04-AMD-ROCm Policy Replication Topic/code/pi0/pi05_canonicalize_lerobot_v3.py) and [pi05_validate_chunk_alignment.py](…/…/…/16-Special Team Learning/04-AMD-ROCm Policy Replication Topic/code/pi0/pi05_validate_chunk_alignment.py). The structural misalignment in canonical91 this round has decreased from [4909] to [0]. Subsequently, through the real [delta_timestamps] interface of LeRobot, 1318 representative positions were checked for horizon 10/50; all 553560 action scalars and 28240 episode end padding are aligned, with the maximum absolute error being [0].

Training handling: Checkpoints that have been trained on disordered data cannot be used as a clean base just because the loss looks good. The supervision of observation-action chunks for some frames is misaligned. The safest approach is to retrain from the official Pi0.5 base. pi05_rocm_finetune.py also includes startup access control: if there is any inconsistency in physical location, global index, episode boundaries, frame_index, or boundary chunk[0], an error will be reported and the program will exit before loading the 7GB model.

In the future, when aggregating action-chunk data, check at least the following four levels simultaneously:

Table 7: For the action-chunk policy, only the fourth layer is closest to the actual data read during training. Ignoring the first layer will result in the most dangerous silent errors.

Case 13: Offline error decreases, but both the training internal seed and the new seed are 0/4

** Phenomenon**: After passing through the physical row order and the action-chunk access, canonical91 was retrained for 800 steps using the official Pi0.5 base. The flow loss decreased from 1.86613 to 0.72765, the action MAE decreased from 0.07255 to 0.04588, and the gripper MAE also decreased from 0.49882 to 0.31419. Just looking at these numbers, it is easy to directly write the next step as “expanding unseen evaluation”.

The strict closed-loop approach yields different conclusions: the location covered by the training data 1000-1003 is 0/4, and the new location 2180-2183 is also 0/4. Neither group achieved a continuous 3-step effective uplift of at least 3 cm. Since no locations similar to those in the training data passed, the current issue cannot be mainly attributed to spatial generalization.

This evaluation also encountered a false fault at the tool level. The new launcher does not inherit desktop variables when started via SSH. pyautogui reports KeyError: 'DISPLAY' during the import phase, and the rollout has not begun at all. There is actually a :0 graphical session on the remote side, and the old ACT/SmolVLA/Pi0 scripts also use DISPLAY=:0 and MUJOCO_GL=glfw. After adding these two environment variables, the model’s strict load, MuJoCo initialization, and the four rollouts completed normally. The first failure should be attributed to an evaluation environment issue and cannot be counted in the model success rate denominator.

The progressive trajectory provides more information than the final 0/4. Among the four new seeds, TCP is closest to the plane distance when it touches the cup, which is approximately 5.4-6.7 cm, but it is still about 10.4-13.8 cm above the center of the cup, and the gripper is nearly closed. Then the robotic arm crosses over the cup, and it descends below the cup height only at a plane distance of approximately 24.6-34.3 cm. The failure sequence can be summarized as:

text识别并靠近蓝杯 -> 高处提前闭合 -> 继续平移越过目标 -> 杯子外下降

Root cause judgment: 800 steps and batch size 4 sampled only 3200 current observations, which is approximately 9.1% of 35284 frames. The weighted sampler also repeats the episode start, recovery end, and gripper transition windows. Although each sample monitors the next 10 actions, these 10 actions share the same current visual state, and the coverage rate cannot be calculated based on 10 independent observations. Therefore, this is more consistent with “overall underfitting + unstable learning during phase switching”, rather than a separate unseen generalization failure.

Subsequent gating: Keep the 800-step checkpoint as a clean baseline, and continue training in segments with a new sampler seed and a lower learning rate. After each segment, run two sets of panels consistently:

Only when the main problem is successfully resolved at positions similar to those in the training data and fails at other positions, the generalization and data coverage mode is switched. If neither group succeeds, continue to check training coverage, stage labels, action representations, and on-policy correction. Do not make progress by relaxing the success criterion.

Case 14: GT action replay failed; it may not be due to an error in the action value

现象：canonical91 has been aligned with the parquet row order and chunk, and theoretically sends each frame of GT EEF-delta/gripper action back to the environment. It should be close to 91/91. However, the first batch playback only yields 6/10; even after changing the reset method, it drops to 2/10. If this number is interpreted as “poor expert trajectory quality,” it will result in the deletion of many valid recoveries.

There are three types of states that can easily get mixed up:

1. obj_init records logical spawn information, which is not necessarily equal to the final free-joint state after reset and gravitational/contact settling;
2. seed only corresponds to the random position during collection when used together with the original position_profile;
3. The recovery data does not start from reset. The first frame of the expert suffix is based on the takeover state of the Pi0 executed prefix.

After aligning each item, a useful diagnostic ladder is formed:

Root cause: GT action is the control variable defined relative to the current simulation state. If any of environment reset, object settle, position random profile, or takeover prefix is inconsistent, even if the actions are identical bit by bit, they no longer correspond to the same dynamic trajectory.

Fix: batch_replay_pi0_dataset_predictions.py added and standardized these protocols:

• Read the original environment seed from episode metadata;
• Explicitly use the original position_profile during reset;
• Reuse the pose of objects that have been naturally settled after reset;
• Hard reset all MuJoCo dynamic states before each episode;
• Recovery is achieved by replaying the prefix saved during collection via --prefix-replay-map-json;
• Use the same environment for large panels to avoid exhausting file handles/asset providers due to repeated environment creation.

Finally, all 9 strict failures came from base episodes 1,13,17,20,22,25,37,45,47; 38 recovery attempts were successful under the correct prefix. The new replayable82 only filters out these 9 base episodes, keeping all recoveries, then renumbers them, recalculates stats, and validates the horizon of 10/50.

The general conclusion of this case is: GT replay is not as simple as “reading the action and then calling env.step”. For delta actions, contact tasks, and suffix DAgger, the state chain when the action was defined must be restored. When GT replay fails, first audit reset/profile/settle/prefix; do not blame the model or delete data first.

Case 15: Nomally unfreezing the visual, but not actually updating the visual

** Phenomenon**: After action-expert-only was transferred from s800 to s2400, the gripper MAE decreased from 0.31419 to 0.08073, but during training on s800/s2400 and at the new position 10-seed panel, all values remained 0/10. The fixed 4 mm actuator bound is still 0/10. Continuing to update only the action expert may result in a more accurate fit of average actions under existing visual representations, without fixing random position positioning.

The first version of the visual parameter probe is placed on BF16 post-layernorm. A update of the 1e-6 magnitude may still appear as 0 after being quantized by BF16, which can lead to the misjudgment of “insufficient probe accuracy” as “no gradient in vision”. After the fix, the check is changed to FP32 vision patch embedding, while keeping the probes for the action head and language layer:

textexpert_vision 通过条件：
action head delta > 0
vision FP32 patch embedding delta > 0
language layer delta == 0

The maximum change in action/vision with 1-step smoke is approximately 1.0e-6, and the language is 0. This indicates that the current mode indeed updates the action expert, vision tower, and multimodal projector, while freezing the language model.

The second issue is the uniform determination of memory capacity. A single-step smoke with batch 16 and off gradient checkpointing can run, but the PyTorch peak value has reached 52.80 GiB. This does not include desktop, ROCm runtime, system services, and caches, so it cannot be used as a long training configuration. With batch 32 and checkpointing enabled, the 10-step continuous smoke peak value is 24.89 GiB, with visual updates ongoing but language still at 0. Therefore, the latter option is chosen.

The third issue is the recovery sampler. The first 44 episodes of replayable82 are complete reset-to-success episodes, and the next 38 episodes are takeover suffixes. If the first 60 frames of all episodes are multiplied by the start weight, the start of the suffix will be mistaken for the task start point. After adding --episode-start-episode-count 44, the start window is changed from 4920 frames to 2640 frames; --tail-episode-count 38 is responsible for the recovery weighting separately. The sampling information is not used as model input, so there is no leakage of target/phase/oracle.

This gate control breaks down the problem into four verifiable issues: whether the data can be strictly restored, whether the action chunks are aligned, whether the visual parameters are actually updated, and whether there is sufficient capacity in the 64 GiB unified memory configuration. Only when all four pass must it take hours to train; after completion, the result should still be judged using raw strict closed-loop, and success rate cannot be replaced by parameter delta or offline loss.

Case 16: After correcting the prefix DAgger, raw strict is still 0/10

The previous case has fixed two easily overlooked data issues: the oracle tag of the first version of prefix DAgger did not actually execute, and it was later changed to continuous expert takeover; the aggregated LeRobot data has also been re-canonicalized according to physical parquet row order to avoid index appearing continuous while being actual action chunk frames. Next, we need to verify whether, after the data semantics are corrected, Pi0.5 truly learns the closed loop approach starting from visual input.

In this experiment, three branches were created first, all evaluated using raw strict: only the policy image, language, robot proprio, and historical execution actions are provided during deployment, without reading the target/plate coordinates, GT phase, oracle action, or scaffold status.

What is most worth preserving here is not a specific loss value, but the failure elimination order: action-only can keep the offline error very low, but it fails to improve the closed loop success rate at visible positions; after解冻 vision, the FP32 vision patch embedding has indeed been updated, while the language layer remains frozen, indicating that “no training of vision” is not the cause; changing the execution chunk from 10 to 5 still fails, suggesting that the problem isn’t just due to the long action chunk; increasing the global z-direction loss weight also fails without strict gate.

The step-by-step trajectory provides a more specific behavior pattern. For some seeds, the initial actions toward the cup move closer; the closest distance to the plane can reach 4–15 cm, but the predicted range in the z direction is significantly smaller than that of the prefix label: in the early 20 recording steps of the early rollout, the z delta mean is approximately -0.0002 m, while the corresponding label is about -0.004 m. Subsequently, the model often closes the gripper prematurely and continues to move along the plane direction across the cup, finally descending outside the cup; some seeds can approach the cup, but no continuous effective lifting occurs. This is a case of “seeing the target but failing to stabilize the sequence of approaching—contact—closing,” not simply a failure in spatial generalization.

This round also verified a data volume boundary. The stable lift suffix data on 15 samples can achieve 15/15’s stable-lift, but the actual physical success remains 0/15, as these samples only begin after contact, and cannot teach the model how to rely on vision to reach the contact state after reset. After the effective correction data for prefix-only increases to canonical92, both the physical sequence, episode boundaries, and chunk start points are passed, but “correct data structure” does not equal “a closed loop that covers enough cases”.

Therefore, the current conclusion is: Pi0.5 has not achieved successful raw end-to-end performance yet. Continuing to blindly increase the number of ordinary training steps, repeat the suffix, scan chunks, or adjust only one global z weight lacks new falsifiable hypotheses. The next experiment should collect a longer continuous on-policy prefix correction, such as 80–120 execution steps, allowing the oracle to continuously take over from the actually deviated states generated by the policy and store each true subsequent state, followed by a small-step comparison starting from the clean S400 baseline. The new data must record whether the prefix action was actually executed, whether the prefix state was written, the source of the oracle phase, the episode seed/profile, and the physical success audit; the evaluation still uses only raw strict metrics, without treating the oracle labels during training as deployment inputs.

Acceptance threshold for this round of experiments

1. First, check the data: physical row order, episode boundaries, chunk[0], and the state and action dimensions all pass.
2. Then conduct an offline inspection: report the xyz, gripper, and total action MAE separately to avoid the fact that the gripper lowering hides no improvement in spatial actions.
3. After each small training segment, run at least one set of fixed training positions and one set of new positions; during training, when 0 occurs, do not label it as a generalization failure.
4. Only when raw strict shows success first during training, continue to expand the unseen panel; if it remains 0, return to the z/ stage for covering and on-policy data semantics, without using a more relaxed success criterion instead.

Case 17: Modifying the HF cache variable before training may trigger a crash of libamdhip64 in ROCm

These issues are often mistaken for “too large model” or “insufficient memory”. Before the Pi0.5 training phase, the physical row order, episode boundaries, and action chunk start points of the dataset were processed through the gate, but the process terminated when the model was moved to the GPU. The kernel log showed libamdhip64.so.7 SIGSEGV; there were no OOM records.

During troubleshooting, verify the cache variables separately instead of moving all temporary directories into tmpfs at once:

text纯 HIP tensor smoke                         通过
CPU 加载模型                                通过
默认环境加载模型到 GPU                     通过
只设置 HF_DATASETS_CACHE                   通过
同时把 HF_HOME/TMPDIR 指向 tmpfs            SIGSEGV

On this ROCm runtime, a safe approach is to place the dataset Arrow/cache in a dedicated cache directory with a clear capacity; HF_HOME and TMPDIR can remain at their default values, or be placed in a stable disk directory. Do not arbitrarily point to /dev/shm. After fixing the issue, run a 1-step smoke test first, then start the long training; the fact that the 1-step passes indicates only that the runtime is stable, not that the model has converged.

The 1-step smoke pass is achieved after this round of fixes, with a peak value of approximately 14.27 GiB; the 400-step training is also completed smoothly, with a peak value of approximately 15.68 GiB, and there are no new kernel errors. However, the final offline MAE remains almost unchanged, and the raw strict value is still legacy 1/10、physical 0/10. Therefore, the “runtime fix” and “policy capability fix” should be considered as two separate conclusions.

Case 18: Longer prefix correction and 400 steps of training cannot replace closed loop contact learning

To verify the hypothesis that “the prefix is too short and the number of training steps is insufficient,” continue with 80 consecutive prefix corrections using the clean S400 expert-vision base: the merged data totals 102 episodes / 32792 frames, and the physical row order mismatch of EEF-delta action-chunk is 0. The training uses expert_vision, batch8, chunk10, and 400 steps; deployment evaluation does not use oracle prefix, target/plate coordinates, GT phase, scripted gripper, or finisher.

Offline results are as follows:

On 10 train-seeds, the old geometric aperture is only 1/10, and the strict physical success is 0/10. The only seed with successful old geometry achieved a lift of 0.1286 m, but the cup eventually tips over, resulting in an upright cosine of only 0.207; most other seeds did not reach 3 cm continuous lifting. This result indicates that the format of longer prefix data and the chunk semantics may be correct, and the visual parameters are indeed updated, but when coverage is insufficient or the action target is not suitable for contact with the closed loop, continuing to add steps will not automatically yield raw VLA success.

The troubleshooting conclusion should be written in three layers:

1. Data layer: Canonical merging and chunk alignment are completed;
2. Execution layer: ROCm cache configuration is fixed, and 400 steps are completed stably;
3. Capability layer: The timing of visual approach, descent contact, closure, and lift is still not stable.

Next, modify the falsifiable modeling assumptions. For example, rely only on images, language, and the phase/progress/contact head of robot proprio, learning-based EEF residual, or temporal aggregation. First, verify using a fixed train-seed gate; do not continue to mask raw failures with a broader success criterion.

Case 19: 800 steps training is completed, but raw Pi0.5 is still 0/10

Problem Background

Case 18 has shown that there is no issue with aligning the physical row order of data in canonical102 with the action chunk, and the 400-step expert-vision continuation can also be completed smoothly. However, raw strict still fails physically. Therefore, in this round, we only verify a narrower hypothesis: whether the number of training steps is insufficient. We continue to train with clean S400 for 800 steps to see if we can learn the contact phase.

The input and evaluation protocol remain unchanged: dual-camera images, language, robot proprio; EEF-delta 7D action; chunk10; fresh environment and hard reset. The prefix correction in the training labels is used solely for supervision; at deployment, target, plate coordinates, oracle prefix, GT phase, scripted gripper, or external finisher/contact head are not read.

First, handle the checkpoint truncation, rather than misjudging the model failure.

The optimizer update for the first 800-step task has been completed, but during the saving phase, it was found that the /dev/shm space is insufficient. The model file header indicates a need of 7473096344 bytes, while the actual output size is only 6865059840 bytes. When checked with safe_open, it shows “file not fully covered”, resulting in a strict loading failure.

This phenomenon should be recorded separately from training failures: the logs do not contain OOM, NaN, SIGSEGV, or kernel errors. The cause is that old checkpoints, model caches, and temporary files fill up the tmpfs, causing the save and write operations to be truncated. Delete the failed output of this round and expired caches, then run again to avoid using half-finished files for further evaluation.

Re-run Results

The size of the checkpoint after rerunning is consistent with the expected file header, and safe_open successfully read 812 tensors. The peak unified memory is approximately 33.56 GiB. There were no new ROCm crashes during the training process. The offline metrics before and after training are as follows:

The decrease in flow loss does not equal a successful closed loop. The action MAE and gripper MAE remain unchanged, and there is only a slight change in xyz, indicating that this further training did not produce sufficiently strong new behaviors.

Strict raw evaluation

On the 10 fresh seeds (1000, 1009, 1018, 1023, 1037, 1044, 1055, 1060, 1068, 1078), the legacy success for old geometric aperture is 0/10, and the strict physical success is also 0/10. During evaluation, the number of visual contact head active steps is 0, so this is not due to the head or finisher not being triggered, but rather because the pure raw Pi0.5 behavior failed to complete the task.

The failure type provides more information than a single success rate: most trajectories fail to form an effective lift; 1009 and 1068 close high or tilt at a higher position shortly thereafter; 1078’s maximum lift is approximately 0.0278 m, nearing the 0.03 m threshold but ultimately inverts. In other words, the model occasionally enters a “climbing” local state, but fails to stabilize the entire closed loop of contact, closing claws, lifting, and maintaining upright position.

Three-level Conclusion

1. The running layer has passed: the official Pi0.5 weight can be strictly loaded on ROCm, 800 steps can be updated smoothly, and the checkpoint truncation has been identified and fixed.
2. The data layer has passed: the physical row order of canonical102, episode boundaries, and the gate at the start of chunks all have zero mismatch.
3. The capability layer still has not passed: the direction from vision to the first EEF-delta, contact timing, gripper events, and lifting retention do not form a generalizable closed loop.

Therefore, the next step should not continue to repeat the same data and action representations in the steps/lr/chunk grid. A more informative experiment is to have the current Pi0.5 run into failure states such as no-lift, high-close, and tilt-release, and then let experts take over these real states continuously to form on-policy correction. At the same time, only the learned phase/progress/contact or EEF residual of image, language, and robot proprio can be read, and temporal aggregation can be used as a candidate for stabilizing action blocks. All candidates must be independently evaluated on raw strict and independent seeds, and no oracle or manual finisher should be introduced again.

Minimum Troubleshooting Record Template

If any new issues arise later, you can directly copy this template to write the experiment record:

markdown### 问题名

- 现象：
- 复现命令：
- 关键证据：
- 排除项：
- 根因：
- 修复：
- 修复后验证：
- 学习结论：

The “learning conclusion” should be phrased as a statement that can guide the next experiment. For example, “Old success cannot replace physical verification of success,” or “Copying an episode will improve the Blue Cup but may harm the Red Cup.”

Checkpoint

After completing this chapter, you should be able to answer:

• Why this topic does not directly use the old success report results;
• Why ACT should use DAgger, but still cannot blindly mix in all failed data;
• Why SmolVLA should be split into fixed instructions for red and blue cups evaluation;
• Why the weighted sampler is more suitable than simply copying episodes for this blue cup fix;
• Why pi_0 raw policy cannot be considered a successful imitation, while pi_0 + scripted finisher can only serve as a final diagnostic;
• Why the index set of LeRobot data still lacks continuity, and further verification of parquet physical row order and real action chunks is needed;
• How to organize one’s debugging records into a reviewable experimental case, rather than simply copying the original log.