Problem List and Solutions

The following “solutions” are divided into three categories:

• Verified fixes: Clear re-test results or official acknowledgments;
• Engineering mitigations: Reduced risks, but no evidence of an improvement in online scores;
• Unsolved hypotheses: Observed correlations, but insufficient experiments to establish causality.

1. The training data files are complete, but the data semantics still need to be verified.

Phenomenon

550 processed files are correctly counted, and the model can be trained. However, issues such as action/qpos time offset, three-camera sequence error, expert profile mixing, or inconsistent action patterns in rescue trajectory may still exist.

Verified Fix

Establish strict audit: Check the file collection, 16-D, left and right arm metadata, finite values, three camera lengths, and frame-by-frame next-state alignment. canonical processed550 reaches next_state_max_error=0, and the manifest SHA is also fixed.

Still Needs Improvement

Solidify the collection configuration, object model, pose, yaw, expert profile, and stock/retry/rescue sources for each layout into a separate manifest; do not use a single global episode number to cover up the gaps in joint coverage.

2. High proportion of rescue data may cause distribution bias

Phenomenon

In exact550, rescue=257 accounts for a high proportion. The rescue trajectory resolves the data overwrite and disk write issues, but it may have different takeover stages, speeds, end poses, and action sequences.

Engineering Processing

Keep canonical550 unchanged and do not modify the main line. Avoid changing data, gripper weights, and sampling simultaneously near the deadline. Create a new clean50 for diagnosis to see if the cleanest subset can learn quickly.

Conclusion Boundary

The performance difference between clean50 and exact550 supports the idea that “data distribution warrants further study”, but it is not sufficient to prove that rescue is the only reason for the low scores in online tests. Future studies should involve stock/rescue ablation with the same architecture, seed, and testing topology, and verification using a new seed panel.

3. Insufficient training steps; low loss cannot cover up the issue.

Phenomenon

The official ACT goal is 6000 epochs, but the main branch has finally reached e1250. Subsequently, attempts were made in the directions of e1500/e1750/e2000. Short-term training near the deadline cannot replace full training from scratch, nor can it ensure the optimizer state, data order, and normalization statistics are equivalent.

Verified Facts

The e1250 section completed the 1000/1000 phase normally, not OOM. The reason for stopping is the manual stage limit and hardware/deadline constraints.

Correct Approach

Lock the 6000 epoch computing budget before the start; save an immutable checkpoint every 250 epochs; perform fixed fresh3 only on idle, clean, single-process evaluators for each complete checkpoint; select candidates based on success with a fixed seed, rather than just looking at train/val losses.

4. The scores of a regular runner and a trace runner are different

Phenomenon

The normal fresh3 for clean50 e6500 is 0/3, 1/3, and the trace diagnostic version is 1/3, 5/9. There are significant differences in the same weight.

Root Cause Judgment

The trace version has performed a monkey-patch on run_act_eval.py to record action boundaries and final states. The running topology, reset order, initialization, and RNG consumption cannot be considered the same as those of the official runner.

Solution

The results of a regular runner are used as official access controls; the trace is only used to identify failed stages. Both are saved side by side without elevation, averaging, or submitting high scores of the trace to the online system.

Things to do in the future

The final gate uses an organizer-matched persistent process. The public seed order is fixed, and the model is loaded only once. The path is reset per episode, which is consistent with the official method. To isolate hang cases, a separate process can be used for filtering, but the filtering results cannot be directly used as the final score.

5. The gap between the local public seed and the online private seed is significant

Phenomenon

The historical control candidate local fresh3 is 2/3, and the online T4 has only 25.3/100. The success rate of 3 local public entries does not represent the average depth of 100 private entries.

Solution

Establish hierarchical evaluation: fresh3 only performs interface/obvious bad model filtering, while fresh10/20 evaluate candidate stability. Subsequently, the full fixed public panel is used; candidates must also pass the preflight checks for code packages, checkpoints, stats, and the official environment.

Not to do

Do not treat a successful clean50 trace, training set replay, expert trajectory video, or “no errors” as an online success. Do not add failed seeds to training after observing them, and then report generalization improvement using the original seed.

6. Confirmation of correct account and submission queue

Phenomenon

Once an account that was only unlocked to a lower track was misused to submit T4, the server returned 403. Although the second version started uploading before the deadline, the official receipt arrived after the deadline.

Verified Fix

Before submission, only the token suffix is kept for verification. Use the unlocked T4 account to resubmit, and obtain clear #719 T4 and submit_exit=0. After receiving 提交已截止 in the second version, stop re-submission.

Things to do in the future

The submission script should first check the account unlock status. After the upload is completed, the official response, queue id, server SHA, and timestamp must be saved. If there is no receipt before the deadline, it should not be assumed that the item has been queued based on “the process is still running”.

7. Submission package boundaries and checkpoint naming

Phenomenon

T2/T3/T4 online evaluations only support official ACT dependencies and the minimum code provided by contestants. Including the entire project root directory, soft links, wandb, raw data, or additional dependencies in the package will result in unpacking failures, import errors, or configuration/weight mismatches.

Solution

Before submission, limit the code root to a minimal immutable snapshot of external/robotwin_local; set the checkpoint package to policy_last.ckpt + dataset_stats.pkl; reject symbolic links, hard links, parent directory traversal, and additional packages; perform ACT import smoke after local simulation packaging.

Things to do in the future

All submitted candidates must include: code tree SHA, deploy config SHA, weight SHA, stats SHA, list of packaged files, and server response. Ensure that policy_best.ckpt, policy_last.ckpt, or the semantics of config are consistent across training, evaluation, and upload.

8. Official action alignment and local modifications

Phenomenon

The ACT loader uses next-state semantics in the sim data path. If action[max(0, start_ts - 1):] is changed to action[start_ts:] locally, it seems like one frame is missing, but actually it changes the target of each training sample.

Solution

The main line fixed action shift=0, and an action[t] == qpos[t+1] audit is performed on the raw data. All alignment experiments should use independent run names, independent stats, and independent checkpoints. They cannot be changed simultaneously with the model size, gripper loss, or data sampling.

9. 4090 Overdue Shutdown and PRO 6000 Migration

Phenomenon

The remote 4090 shut down due to insufficient balance, disrupting the training and evaluation timeline; temporary applications/recharge, restarting, and environment synchronization consumed the end-of-season window. The cloud-based PRO 6000 also has constraints regarding other user tasks, disk space, and GPU ownership.

Engineering Mitigation

• During operation, resume transfer using breakpoints, and after migration, check the number of files, size, SHA, and import access;
• Compatible with Python 3.10 / PyTorch / SAPIEN / MPlib / cuRobo environments. Prioritize creating verifiable images instead of temporary reconstruction before the deadline;
• Store big data, checkpoints, and logs separately to avoid filling up the root drive with environment resources;
• Do not occupy tasks of other users, and do not start multiple SAPIEN evaluators concurrently on the same GPU.

Conclusion

This is an engineering usability issue, not evidence that the model quality is “determined” by a specific GPU. Subsequently, a persistent disk, balance alerts, auto-start recovery upon boot, and daily recoverable checkpoints should be set up.

10. True Repair Priority After Low Scores

If a rematch is to be held, the priority should be:

1. First, lock in the official environment and persistent evaluator;
2. Re-audit the coverage of layout/object/pose/yaw joints;
3. Fully train on clean official code up to the desired horizon;
4. Perform single-variable paired ablation on rescue / retry / stock;
5. Conduct failure clustering on gripper timing, lift/transport/release stages;
6. Verify the improvement using an independent fresh seed panel;
7. Finally, package and submit the results.

Unrecommended approaches include simply adding the epoch from 1250 to 2000 mechanically, focusing only on loss, selecting a trace with a high score once, or modifying five data and model variables simultaneously.