logo Mr.Robota 具身智能学习手册
/
中文
Chapter 15: ATEC2026 Lightweight Code

Workspace Memory

ATEC2026 L0 Task E

  • Scope:

    • Current priority is L0 Task E tabletop manipulation in $PROJECT_ROOT; do not switch to L1 unless the user asks.
    • 2026-06-07 Official website file upload submission records: XSA-ACT uploaded and online score achieved 6.00, build number BUILD2001385_2038204482147323904_20260607142442_bae481, task “desk organization / Piper”, taking about 00:11:22.62 time. The get_action_spec not found warning is not the reason for deduction; it has been eliminated with get_action_spec(self): return None in demo/solution_act.py and submissions/task_e_xsa_act_20260607_upload/solution.py. The updated solution.py SHA256 is a08790e7512c914e5b78132303e1c6098c8aba3b01a373933e6ddc617fa3a485.
    • 2026-06-07 PCA/GraspGen-style submit branch re-test: ATEC_PCA_OBJECTS=3,2,1, seed11, demo/solution_pca.py completed 5200 steps and achieved 0.00, log logs/pca_submit_eval/pca_submit_seed11_20260607.log, video logs/videos/task_e_pca_submit_eval/pca_submit_seed11_20260607.mp4. Failure phenomenon: RGB-D/PCA center estimation can be generated, but the actual position of submit-side IK/finger never catches up with the desired finger center, and object3/object2/object1 are not scored. Currently, solution_pca.py should not be considered for the ranking submission.
    • 2026-06-07/08 Two ACT seed2 checkpoint packages were prepared for online trial ranking; both have been slimmed down to inference checkpoints (about 64MB) and initialized via AlgSolution(): submissions/task_e_act_seed2_45000_20260607_upload, zip SHA256 4b5d1178db3060883db4400f24eaa04b11b259a5179a220a7a622d7347ea2895, policy_act.pt SHA256 d9badbf17b0f29000c4fb21c3a2a89f66f7450e3ec1cb7362e733f4a4a7a0cb2. Historical local seed11 achieved 18, seed12 achieved 0, online user results are 12.00 submissions/task_e_act_seed2_50000_20260607_upload, zip SHA256 1d777cfa95ee40d92afafd70f5e3cbfd3ee6477f0b036ed9d459c94237858118, policy_act.pt SHA256 8ac243e8427c54704a0445fa8f0321450618fa6c2d2d7296db027c9c5fe1e2c9. Historical local seed11 achieved 15, seed12 achieved 6, online user results are 6.00. Both packages use the updated solution.py SHA a08790e... and IPython’s act/detr/transformer.py SHA fa290d.... Conclusion: seed2 checkpoint has high variance and cannot be considered a stable 18-point solution.
    • 2026-06-08 Preparation of the old ACT seed1 best “drawing candidate” submission package: submissions/task_e_act_seed1_best_20260608_upload, zip submissions/task_e_act_seed1_best_20260608_upload.zip, zip SHA256 e265ebe77b989a968cf4607ea70d4f3adc2322902538828b37fe84877f5b1391, policy_act.pt SHA256 282614de9673dc01e229557448e380b6a02c196b5c3953cec77b2aebd2305a8e, solution.py SHA256 a08790e7512c914e5b78132303e1c6098c8aba3b01a373933e6ddc617fa3a485. The source is the old seed1 best (original backup SHA 76bbef2c81c85fc6cc60448476ff09075ec8bc97217c2d2fb70a3253fd3e307d), which has been slimmed down to only norm_stats, model_args, ema_agent and initialized via AlgSolution(). Historical local independent run1 achieved 18, run2/run3 achieved 6, not yet submitted online; if users can try the ranking multiple times, this is the most valuable package among the currently packaged candidates for the next submission.

  • 2026-07-19 Post-match open source archive: The model repository is https://huggingface.co/Datawhale/atec2026-task-e-act-seed1-best, and policy_act.pt along with the complete submission zip have been uploaded; the data/logging repository is https://huggingface.co/datasets/Datawhale/atec2026-task-e-reproducibility. The training data for reproduction uses filtered HDF5: datasets/atec_task_e_obj321_servo_100demos/trajectory_filtered.hdf5, approximately 33GiB in size, with SHA256 value 4b85980e72b8e76261ad037bfba97b4795210e3f77cdb5b71479846b9230ac02; on HF, it is split into data/final_100demos_filtered_split_100m/trajectory_filtered.hdf5.part-0000..0336 with a size of 100MiB, along with RESTORE_FILTERED_DATASET.sh, SHA256_FILTERED_ORIGINAL.txt, and SHA256SUMS_FILTERED_SPLIT.txt. Uploading is performed using a temporary storage directory $DATA_ROOT/ATEC2026/hf_task_e_filtered_100m and the user service atec-task-e-hf-filtered-100m-upload.service, with an entry point $DATA_ROOT/ATEC2026/upload_task_e_filtered_100m_large_folder.py, and settings HF_HUB_DISABLE_XET=1 and upload_large_folder(num_workers=8) for breakpoint resume. On 2026-07-19, remote verification was completed: 337/337 fragments are complete, README.md, RESTORE_FILTERED_DATASET.sh, two SHA256 lists, and logs/task_e_logs_20260520_20260609.tar.zst exist; the temporary storage directory can be cleared, but the original trajectory_filtered.hdf5 remains intact unless confirmed by the user.

  • 2026-06-08 The user submitted task_e_act_seed1_best_20260608_upload/policy_act.pt to the official website, resulting in an online submission 15.00; due to limited opportunities for submissions/submissions on that day, the user stated that there is only one more submission opportunity. This result is better than XSA online 6, seed2_45000 online 12, and seed2_50000 online 6, and currently represents the best online outcome. The last submission must have clear evidence of local performance exceeding 15; otherwise, do not use it.

  • Risk assessment for the final attempt, 2026-06-08: Historical logs show that XSA final and seed2_50000 usually result in object_1 failing while object_2/object_3 are successful; seed2_45000 locally with seed11 results in all object_1/2/3 being successful but seed12 failing completely. Online 12 indicates that the hidden evaluation still loses at least one object. For the PCA/GraspGen-style submit branch, current seed11 is 0, and a complete GraspNet/AnyGrasp does not meet the short-term single-file stable submission criteria. Do not use the PCA/GraspNet approach with failed local submissions for the last attempt.

  • Last attempt hybrid test, 2026-06-08: A solution submissions/task_e_act_seed1_hybrid_obj1_pca_20260608_upload was developed. The idea is to first perform ACT seed1/XSA on the first two objects, then apply the object_1-only geometry remedy from solution_pca.py. Initialization fixes include copying solution_pca.py and Piper URDF third_party/Agilex-College/piper/handpose_det/models/modified_piper_without_camera.urdf into the submission directory, and changing the solution_pca.py URDF search to prioritize the current submission directory. The result was failed: seed1-best hybrid seed11 score=6.00, with only object_3 inside, as shown in video logs/videos/task_e_hybrid_eval/hybrid_obj1_pca_seed11_20260608.mp4; for XSA final hybrid, fixed ACT step 1320 followed by PCA resulted in seed11 score=3.00, with object_1/2/3 not inside, as shown in video logs/videos/task_e_hybrid_eval/hybrid_xsa_fixed1320_obj1_pca_seed11_20260608.mp4.DO_NOT_SUBMIT.md has been added to the directory; do not use this hybrid package for the last online submission.

  • On 2026-06-08 afternoon, continued efforts to achieve a negative score: new experimental files demo/solution_hybrid_obj1_pca.py (only cut object_1 PCA after current_score>=15) and demo/solution_dual_act.py (two sets of ACT synchronized observations, cut secondary policy after score threshold) have not yet been submitted as a proposal. logs/eval_task_e_hybrid_xsa_score15_obj1_pca_seed11_20260608.log gets 0.00; logs/eval_task_e_hybrid_seed1best_score15_obj1_pca_seed12_20260608.log gets 0.00; logs/eval_task_e_hybrid_seed1_95000_score15_obj1_pca_seed11_20260608.log gets 0.00, with video logs/videos/task_e_hybrid_eval/hybrid_seed1_95000_score15_obj1_pca_seed11_20260608.mp4. For dual ACT: logs/eval_task_e_dualact_seed1best_seed2_45000_switch12_seed11_20260608.log gets 0.00; logs/eval_task_e_dualact_seed1_95000_seed2_45000_switch12_seed11_20260608.log cuts the secondary policy in score=15 with primary_steps=900, but ultimately remains 15.00, outside object_1 and inside object_2/3. Conclusion: The PCA/double ACT sequence after score=15 failed to include object_1 in the basket, so the last submission opportunity cannot be used.

  • On 2026-06-08, continued efforts toward reaching 18 with hybrid policies: runs/.../seed1/checkpoints/95000.pt trace seed11 shows step 300 score=6, step 600 score=12. Object_2/object_3 have been scored, while object_1 remains on the desktop; log logs/trace_task_e_seed1_95000_seed11_20260608.log. Based on this, tested score=12 + min_act_policy_steps=600 cutting object_1 PCA: logs/eval_task_e_hybrid_seed1_95000_score12_step600_obj1_pca_seed11_20260608.log still remains 15.00, with object_1 finally outside (0.988,0.336,0.872), inside object_2/3; the old object_1 rescue logs/eval_task_e_hybrid_seed1_95000_score12_step600_obj1_pca_rescue_seed11_20260608.log is returned to 12.00 and no hybrid switch occurs. Explicit ATEC_HYBRID_FORCE_ENV_STEPS counting and reasons for switching have been added to demo/solution_hybrid_obj1_pca.py to avoid relying solely on ACT internal _ts/score callbacks; an experiment entry(ATEC_PCA_OBJ1_DIRECT_RESCUE=1) has been added to demo/solution_pca.py, and the obvious bug in _build_object1_drag_rescue() has been fixed: the old version start_y=TABLE_CENTER_Y+0.085 missed from the front of the square when object_1 is at y≈0.26..0.34, now it dynamically generates lanes from the center y-side of the current square, with lane x also centered around the current detection center. Due to heavy machine GPU/Isaac concurrency from 16:15–16:45 on 2026-06-08, multiple force-env eval processes fail to progress to valid steps after Isaac starts, so they have been manually stopped; these new direct-rescue patches have not yet produced >15 local evidence and cannot be submitted.

  • On 2026-06-08, seed1 checkpoint sweep: rough screening of seed11 for runs/act-task-e-rgb-obj321-servo-100demos-seed1/checkpoints. Results: 35000=0, 45000=12, 50000=0, 55000=6, 65000=0, 70000=0, 75000=6, 80000=6, 90000=9, 95000=15, best_loss=0, corresponding logs: logs/eval_task_e_seed1_*_20260608_sweep*.log. Extended testing for 95000.pt: seed12 6, seed13 6, thus 95000 is not a more stable alternative to the online seed1-best 15. The current best remains the online task_e_act_seed1_best_20260608_upload with 15.00, and neither 95000 nor dual/hybrid should be submitted.

  • On 2026-06-04, Chinese handover documentation has been organized: docs/ATEC2026_TaskE_规则物理配置与提交说明_20260604.md; official README snapshots have been downloaded to docs/sources/ATEC2026_Simulation_Challenge_official_readme_20260604.md. Documentation conclusion: Currently, the official ATEC-TaskE-Piper / Isaac Lab v2.3.2 environment is used, with Piper solver, object mass, and friction/contact parameters sourced from local official source code; the banana sliding issue resembles problems with the two-finger gripper contact and submit-side primitive, and cannot be submitted by modifying the judge physics; physical enhancements are only suitable for diagnosis.

  • 2026-06-04 fresh clone official repository pushed to /tmp/ATEC2026_Simulation_Challenge_official_check, HEAD is dbe7c251f680b02f357a6db67430b18d3ba45ea1. Verification results: piper.py, object.py, Task E terrain.py, rewards.py, and terminations.py match the local SHA exactly; task_e/env_cfg.py only differs in the action group field name, while the logic for objects/cages/cameras/Y-band/physics material remains consistent. The latest official version adds demo/server.py’s /get_action_spec and custom action examples, which are optional customizations to the action interface but do not allow modification of the judge physics.

  • 2026-06-04 again pushed the official repository to git fetch origin, while origin/main remains at dbe7c25 with no new commits. Checking GitHub issues/PRs: no public issue mentions Task E/Piper/grasp/physics/friction/banana falling; Piper targets PR #2 eye-in-hand camera for B2Piper/B2WPiper/G1, not Task E Piper; PR #5 is a participant action spec, and PR #8 is Tron2A joint configuration. A unique low-risk compatibility fix has been synchronized: adding the official /get_action_spec endpoint to the local demo/server.py; if the solution lacks this method, {} is returned, maintaining the default action config.

  • The official Task E robot is ATEC-TaskE-Piper; the newer Tron2AWheel / Tron2ALegged models are not relevant for this tabletop task.

  • Debugging review summary, 2026-05-30:

    • Current conclusion: the deployable baseline is still XSA-ACT’s demo/policy_act.pt, SHA256 is c3eacae3f1b0ec8ccde9fdc05d680fff119cc2ca70e1f79e4ddd5610c4bbfa93. It is not a perfect stable solution, but local evidence remains stronger than seed2 ACT, GraspNet/SAM3 direct control, and pi0.5/OpenPI branch. Do not use the current checkpoint of pi0.5 to override demo/policy_act.pt.
    • Robot/task identity: Task E uses ATEC-TaskE-Piper, local env registration is in source/atec_rl_lab/atec_rl_lab/tasks/task_e/__init__.py, Piper configuration comes from TaskEEnvPiperCfg / PIPER_CFG. PiPER is part of the AgileX/Songling robotic arm series; this task is a desktop-level fixed Piper, not a L1 Tron/Unitree wheeled/foot robot.
    • Data generation mainline: the official/initial scripted oracle is relatively available for object_3, but object_1/object_2 fail. The root cause is not the camera, but rather mismatches in USD/root geometry center, Piper gripper TCP, actual contact geometry of the gripper, and transport/placement primitives. Finally, three object 3 2 1 success data were generated using per-object calibrated primitives and closed loop servos. The core data directory is datasets/atec_task_e_obj321_servo_100demos, and after filtering, HDF5 is datasets/atec_task_e_obj321_servo_100demos/trajectory_filtered.hdf5.
    • ACT seed1 mainline: after training ordinary ACT with the above 100 demos, the first evaluation failure was demo/solution_act.py due to misunderstanding of the home action: Task E action is (joint_target - default_joint_pos) / 0.5, not speed/PD delta. After fixing the home action, ordinary ACT can achieve partial success, but best_loss.pt/final.pt are unstable, and training loss has little relation to task success rate. The old seed1 best has been backed up as demo/policy_act.seed1_best_backup_20260522_012719.pt.
    • XSA-ACT mainline: the user-provided xsa_update_20260520/lerobot_policy_act_xsa.zip is the LeRobot act_xsa plugin, and the original example cannot be directly run with ATEC 8D Piper. source/atec_rl_lab/atec_rl_lab/train/act/act/detr/transformer.py and demo/act/detr/transformer.py have added --use_xsa adaptation; scripts/act/train_task_e.py saves model_args, and demo/solution_act.py can automatically load use_xsa based on checkpoint. XSA final directly independent evaluation of seeds 11/12/13 once achieved 15/15/15, with logs logs/eval_task_e_xsa_final_seed11_recheck.log, logs/eval_task_e_xsa_final_seed12_recheck.log, and logs/eval_task_e_xsa_final_seed13_recheck.log; however, post-deployment review revealed contact/non-deterministic fluctuations in seed11 9 and seed12 0. It remains the strongest deployable baseline currently.

  • ACT gripper/resume main line: seed2 is trained from scratch with resume support. Related changes are in scripts/act/train_task_e.py and scripts/act/run_task_e_pipeline.sh. It supports --resume_checkpoint / --resume_iter, RESUME_CHECKPOINT / RESUME_ITER, and preserves optimizer/scheduler/EMA. During seed2 training, there are local snapshots that appear to yield high scores, such as 45000.pt with seed11 18, but 0 with seed12; ultimately final.pt with seed11 0, and best_loss.pt with seed11 9. Conclusion: seed2 should not be deployed, and high scores from a single seed should not be regarded as reliable evidence.

  • GraspNet/TunTunClaw main line: The OpenClaw/Datawhale TunTunClaw reference provided by users has been received in third_party/tuntunclaw/. The GraspNet checkpoint is third_party/tuntunclaw/temp/logs/log_rs/checkpoint-rs.tar; the bridging scripts are scripts/graspnet_task_e/tuntun_adapter.py and scripts/graspnet_task_e/run_graspnet_pick.py. Dependencies are connected under $HOME/envs/genmanip-isaac5-py311/bin/python, including open3d==0.19.0, local graspnetAPI, transforms3d==0.4.2, and the pointnet2 CUDA extension for Blackwell TORCH_CUDA_ARCH_LIST=12.0. GraspNet can provide candidates, but raw candidates cannot directly solve real Isaac interactions; object_3 can succeed through SAM3/GraspNet+ real finger/staged transport, while object_1/object_2 still slip/overstep during transportation and cannot serve as the final controller.

  • SAM3 main line: The SAM3 checkpoint is located in $HOME/Documents/01Proj/sam3d_gs/submodule/Prompt-Inpaint/checkpoints/sam3.pt, and independent Python is in $HOME/Documents/01Proj/sam3d_gs/.venv/bin/python. Do not install it into the Isaac environment. New scripts/graspnet_task_e/sam3_segment_image.py is added for RGB prompt segmentation; run_graspnet_pick.py supports --mask_provider sam3, --sam3_prompt, and --basket_center_release. Important conclusion: SAM3 segmentation is not the main bottleneck; masks for object_1/2/3 can all be generated correctly; early failures were due to incorrect selection of multiple candidates’ masks, which was later fixed by world-band/RGB-D point filtering. The real bottleneck is Piper gripper slipping during contact and transportation.

  • GraspNet/SAM3 placement rules: The user’s suggestion of “above the target center point, stabilize before releasing the gripper” is correct. Parameters such as transport-end logging, short stabilization hold, --release_z, --basket_xy_tol, --basket_hold_steps, --basket_stable_steps, --basket_recovery_steps, --staged_transport, and --transport_servo_fraction have been added. Experience: Do not hold for a long time at high altitude, as the object will slip/bounce; short stabilization hold is better for object_3. Object_1/object_2 are still unstable, and simply adjusting the gripper release timing is insufficient; the grasp/contact primitive needs to be redesigned.

  • pi0.5/OpenPI main line: The OpenPI repo is $HOME/src/openpi-ebench-clean, the environment is $HOME/envs/openpi-pi05, and the general checkpoint is $HOME/projects/robotics_shared/checkpoints/openpi/pi05-ebench-generalist/pi05_generalist/params. HF_LEROBOT_HOME=$HOME/projects/robotics_shared/datasets/lerobot, HF_HOME=$HOME/projects/robotics_shared/hf_cache, and PYTHONPATH=$HOME/src/openpi-ebench-clean/src:$HOME/src/openpi-ebench-clean/packages/openpi-client/src must be set. Do not use uv run in the OpenPI repo, as it will create a huge .venv and cause redundant downloads.

  • pi0.5 data conversion: scripts/pi05/convert_task_e_hdf5_to_lerobot.py maps the 8D qpos/action from ATEC HDF5 to the OpenPI fixed-base 16D padded format: state/action.joints[:6] corresponds to Piper’s 6 arm joints, state/action.gripper[:2] corresponds to two gripper joints, and zeros are filled for the rest; the three-channel camera overlook/left/right is temporarily written as a single 224x224 RGB image. The official dataset is $HOME/projects/robotics_shared/datasets/lerobot/atec/task_e_obj321_servo_100demos_s5_224, approximately 4.7G in size, with stride=5 and fps=10.

  • pi0.5 key action constraints: The actions in ATEC HDF5 is already the environment action (joint_target - default_joint_pos) / 0.5, not the absolute qpos. Therefore, all ATEC OpenPI configurations must be extra_delta_transform=False. If the OpenPI default delta transform is used, the state will be reduced again, causing misalignment in training objectives.

    • pi0.5 training results: 20-demo fast branch pi05_atec_task_e_20demos_s5_224 2k step in seeds 11/12/13 Full 0, the checkpoint has been deleted, only the content is retained logs/pi05_eval_20260522074137 And the video logs/videos/task_e_pi05_eval/pi05_pi05_atec_task_e_20demos_s5_224_seed11_20260522074137.mp4. 100-demo official branch pi05_atec_task_e_s5_224 Train 29999 steps, checkpoint .../pi05_atec_task_e_s5_224/atec_task_e_pi05_s5_224_30k_20260522065716/29999 Approximately 42G, evaluation logs/pi05_eval_20260522112701：seed11 0，seed12 0，seed13 6，mean 2.0, video logs/videos/task_e_pi05_eval/pi05_pi05_atec_task_e_s5_224_seed11_20260522112701.mp4. Conclusion: The pi0.5 current branch is not deployed.
    • Analysis of pi0.5 failure reasons: This is not direct evidence that the “pi0.5 model is weak”, but rather that the adaptation layer is likely incorrect. The current bridge forces the ATEC Piper 8D action into OpenPI EBench fixed-base 16D, uses the same image for all three cameras, employs coarse-grained prompts, the data follows a scripted oracle style, and the evaluation parameters max_steps=1500/action_repeat=5 are not fully equivalent to ACT 1800-step trajectories. Another strong suspect is the OpenPI loader log discrete_state_input=True, and the public experience of Piper pi0.5 posttrain is mentioned discrete_state_input=False Avoids state token saturation under z-score norm. If pi0.5 is continued, prioritize using native Piper/OpenPI config, check q01/q99/norm stats, and try discrete_state_input=False, and consider trying pi0 instead of pi0.5 first.

  • Addition of verification for the old branch pi0.5, 2026-06-08: Already reviewed scripts/pi05/convert_task_e_hdf5_to_lerobot.py、demo/solution_pi05.py、OpenPI config.py And logs/pi05_train_task_e_s5_224_20260522065716.log. The old converter converts the 8D Piper pad into EBench fixed-base 12 joints + 4 gripper, where the real arm is in 0:6, real gripper in 12:14The remaining 8 valid EBench dimensions are all 0; then PadStatesAndActions(model_action_dim=32) Pad to 32D again. Confirm the training log. TokenizePrompt(... discrete_state_input=True), and three-channel images head/hand_left/hand_right It is all the same 224 RGB image. The HDF5 raw data has no NaN/Inf values, and the action is indeed the environment action, not absolute qpos;extra_delta_transform=False Correct. Evaluation logs/pi05_eval_20260522112701 The seed11/12 three objects basically remain on the original y-axis surface of the table. For seed13, only object_1 is scored into the basket. The main issues appear to be in the embodiment/action-time bridge and input distribution problems, rather than simply “grasping and sliding”. Recommended restart: keep pi0.5 generalist. action_dim=32 To be compatible with pre-trained weights, but adding ATEC-native transform. The real 8D state/action is placed in the first 8 dimensions and only the first 8 dimensions are output; set Pi0Config(pi05=True, action_horizon=50, discrete_state_input=False); Use a single camera real mask or multiple real cameras, do not use fake three paths; first set stride=1 or action_repeat=1/2, save 1k/2k/5k/10k/15k multiple checkpoints for fixed-seed eval, then decide whether to perform long training.

  • pi0.5 native8 restart branch, 2026-06-08: new additions scripts/pi05/convert_task_e_hdf5_to_lerobot_native8.py、demo/solution_pi05_native8.py、OpenPI transform $HOME/src/openpi-ebench-clean/src/openpi/policies/atec_piper_policy.py, as well as configs pi05_atec_task_e_native8_20demos_s1_224 / pi05_atec_task_e_native8_100demos_s1_224. Design intention: Input real 8D qpos, output real 8D env action. The model still pads to 32D internally to be compatible with pi0.5 weights; only use video.base_camera_view Real images throughout; zero images are used for left/right wrists, and image_mask=False；discrete_state_input=False;stride=1/action_repeat=1, to avoid damaging the grasping timing caused by the old stride=5. Script path:scripts/pi05/run_pi05_native8_20demos_smoke.sh、scripts/pi05/run_pi05_native8_100demos_prepare.sh、scripts/pi05/run_pi05_native8_eval_checkpoint.sh。Due to the root disk / Only about 68G remains. The LeRobot data and OpenPI assets/checkpoints for native8 must be written down. $DATA4_ROOT/01Proj/13atec/...; Do not write the new 42G checkpoint again $HOME/projects/...。

  • pi0.5 native8 is currently running, 2026-06-08 22:30 CST: The old atec-pi05-native8-20demo-smoke-20260608222155.service was actively stopped due to slow image synchronization; atec-pi05-native8-20demo-smoke-20260608222519.service was also stopped because the stride=1 conversion takes approximately 1.5–2 hours, making it unsuitable for the adapter inspection. The converter has been defaulted to image_writer_processes=5, image_writer_threads=10. Currently, the fast gating user unit atec-pi05-native8-20demo-s2-smoke-20260608223030.service is being started, executing 20-demo native8 stride=2 smoke (conversion -> norm stats -> 3k training -> automatic evaluation of seeds 11/12/13), with log prefixes logs/pi05_native8_convert_20demos_s2_224_*, logs/pi05_native8_norm_20demos_s2_224_*, logs/pi05_native8_train_20demos_s2_224_*, logs/pi05_native8_eval_20demos_s2_224_*, and data directory $DATA4_ROOT/01Proj/13atec/robotics_shared/datasets/lerobot/atec/task_e_obj321_servo_20demos_native8_s2_224. This branch is for adapter inspection only and is not a candidate for submission; evaluation uses CONFIG=pi05_atec_task_e_native8_20demos_s2_224 ACTION_REPEAT=2 scripts/pi05/run_pi05_native8_eval_checkpoint.sh <ckpt>. Only when seeds 11/12/13 are evaluated locally and show significantly better results than those obtained online via 15.00, will the last official submission opportunity be used.

  • pi0.5 native8 quick sanity check, 2026-06-08 22:39 CST: The 20-demo s2 conversion still takes about 50 minutes, so atec-pi05-native8-20demo-s2-smoke-20260608223030.service was actively stopped first. pi05_atec_task_e_native8_4demos_s2_224, scripts/pi05/run_pi05_native8_4demos_s2_sanity.sh were added, and atec-pi05-native8-4demo-s2-sanity-20260608223911.service was started. This branch uses only 4 demos, stride=2, 1k steps; after training, automatic ACTION_REPEAT=2 evaluation of seeds 11/12/13 is performed. The goal is simply to quickly confirm whether the native8 action bridge / OpenPI websocket / Isaac rollout can move the robotic arm toward the target, not to determine the final score. If the sanity video shows no movement or random motion, fix the bridge/norm first; if the motion direction is correct, then resume 20-demo/100-demo.

  • pi0.5 native8 sanity results, 2026-06-08 23:15 CST: 4-demo s2 training is completed, and the loss has dropped from 0.5766 to approximately 0.0405. However, all seeds 11/12/13 have scores of 0.00, with video logs/videos/task_e_pi05_native8_eval/pi05_native8_pi05_atec_task_e_native8_4demos_s2_224_seed11_20260608231046.mp4 and montage logs/videos/task_e_pi05_native8_eval/pi05_native8_4demo_s2_seed11_montage_20260608231046.jpg. The video shows the robotic arm basically stationary near the home position; an offline action probe uses checkpoint 999 to推理 training frames, and the output is not entirely zero (the multi-dimensional motion amplitude is approximately 1–3), indicating that websocket/unnormalize works well, but the 4 data points are too few and the trajectory has not been generalized, so it cannot serve as a submission basis. This 4-demo checkpoint accounts for 83G and is worthless, thus it has been deleted; logs, data, and video are retained.

  • pi0.5 native8 current gating operation, 2026-06-08 23:19 CST: The save_interval/keep_period in native8 configs has been changed to only save the final version, to avoid the 40GB intermediate checkpoint slowing down training. The 20-demo s2 user unit atec-pi05-native8-20demo-s2-smoke-20260608231948.service was restarted, and script scripts/pi05/run_pi05_native8_20demos_s2_smoke.sh will convert 20 demos stride=2 -> norm stats -> 3k train -> automatically perform ACTION_REPEAT=2 evaluation of seeds 11/12/13. If the score remains 0/0/0 or the video still stops near home, do not add more data at this time; first fix action timing/home gate/bridge. If there is clear movement and non-zero scores, then run 100-demo native8.

  • pi0.5 native8 20-demo s2 results, 2026-06-09 00:42 CST:pi05_atec_task_e_native8_20demos_s2_224 Training completed with 3k samples, loss minimized at approximately 0.0188, checkpoint $DATA4_ROOT/01Proj/13atec/openpi_atec_runs/checkpoints/pi05_atec_task_e_native8_20demos_s2_224/atec_task_e_pi05_native8_20demos_s2_224_3k_20260608231948/2999. Evaluation logs logs/pi05_native8_eval_20260609003633 / summary logs/pi05_native8_eval_20demos_s2_224_20260608231948.log, seeds 11/12/13 scores 6/0/0, mean 2.00; for seed11, only object_3 was scored, video logs/videos/task_e_pi05_native8_eval/pi05_native8_pi05_atec_task_e_native8_20demos_s2_224_seed11_20260609003633.mp4. Conclusion: native8 bridge can produce some effective policies, but 20 data points are insufficient to exceed online performance 15.00, thus not to be submitted.

  • pi0.5 native8 100-demo s2 gating, 2026-06-09 00:46 CST: New OpenPI config added pi05_atec_task_e_native8_100demos_s2_224 (100 demos, stride=2/fps=25, discrete_state_input=False, 10k step, final-only checkpoint, assets/checkpoints in $DATA4_ROOT/01Proj/13atec/openpi_atec_runs) and script scripts/pi05/run_pi05_native8_100demos_s2_gate.sh. User unit atec-pi05-native8-100demo-s2-gate-20260609004546.service has been started, workflow: conversion trajectory_filtered.hdf5 -> norm stats -> 10k train -> ACTION_REPEAT=2 PORT=8023, evaluating seeds 11/12/13. Log prefixes: logs/pi05_native8_convert_100demos_s2_224_20260609004546.log, logs/pi05_native8_norm_100demos_s2_224_20260609004546.log, logs/pi05_native8_train_100demos_s2_224_20260609004546.log, logs/pi05_native8_eval_100demos_s2_224_20260609004546.log. Only when local multi-seeds significantly exceed online performance 15.00 is the last official submission considered.

  • pi0.5 native8 100-demo s2 results, 2026-06-09 05:49 CST: Conversion completed 100 episodes / 84877 frames, data directory $DATA4_ROOT/01Proj/13atec/robotics_shared/datasets/lerobot/atec/task_e_obj321_servo_100demos_native8_s2_224 approximately 3.9G; norm stats written in $DATA4_ROOT/01Proj/13atec/openpi_atec_runs/assets/pi05_atec_task_e_native8_100demos_s2_224/atec/task_e_obj321_servo_100demos_native8_s2_224. 10k training completed, checkpoint $DATA4_ROOT/01Proj/13atec/openpi_atec_runs/checkpoints/pi05_atec_task_e_native8_100demos_s2_224/atec_task_e_pi05_native8_100demos_s2_224_10k_20260609004546/9999 approximately 42G; loss decreased from 0.4456 to approximately 0.009-0.013 at the end, but task evaluation is poor. Evaluation logs logs/pi05_native8_eval_20260609054339 / summary logs/pi05_native8_eval_100demos_s2_224_20260609004546.log, seeds 11/12/13 scores 3/0/0, mean 1.00, none of the three objects scored; video for seed11 logs/videos/task_e_pi05_native8_eval/pi05_native8_pi05_atec_task_e_native8_100demos_s2_224_seed11_20260609054339.mp4. Conclusion: this full 100-demo native8-s2 branch should not be submitted, nor should it use the last official submission opportunity; online 15.00 ACT seed1 best remains the protected best.

  • pi0.5 native8 LoRA entry/startup, 2026-06-09 09:51 CST: Based on the experience of the third-party Pantheon Piper pi0.5 LoRA (119 Piper single-arm episodes, discrete_state_input=False, 7500 steps, peak LR 5e-5, LoRA preserving prior knowledge), the OpenPI config pi05_atec_task_e_native8_100demos_s2_224_lora has been modified to be a true LoRA gatekeeping mechanism: paligemma_variant="gemma_2b_lora", action_expert_variant="gemma_300m_lora", explicit freeze_filter=Pi0Config(...).get_freeze_filter(), num_train_steps=7500, peak_lr=5e-5, ema_decay=None, discrete_state_input=False. Currently, the local $HOME/projects/robotics_shared/checkpoints/openpi/pi05-ebench-generalist/pi05_generalist/params is still used as the weight loader, as no local pi05_base directory was found; this is not exactly the same as Pantheon’s “pi05_base no warm-start”. If this branch fails, the next step is to download/verify the gs://openpi-assets/checkpoints/pi05_base/params branch. A new script scripts/pi05/run_pi05_native8_100demos_s2_lora_gate.sh has been added, reusing the converted data $DATA4_ROOT/01Proj/13atec/robotics_shared/datasets/lerobot/atec/task_e_obj321_servo_100demos_native8_s2_224 and existing norm stats without redundant conversion. The user unit atec-pi05-native8-100demo-s2-lora-gate-20260609095114.service has been started, training logs logs/pi05_native8_lora_train_100demos_s2_224_20260609095114.log, service outer logs logs/atec-pi05-native8-100demo-s2-lora-gate-20260609095114.service.log, checkpoint root directory $DATA4_ROOT/01Proj/13atec/openpi_atec_runs/checkpoints/pi05_atec_task_e_native8_100demos_s2_224_lora, and automatic ACTION_REPEAT=2 PORT=8024 evaluation of seeds after training 11/12/13 have been carried out. The threshold remains unchanged: only when the local multi-seed significantly exceeds the existing online 15.00, the last official submission opportunity will be considered; otherwise, the ACT seed1 best is preserved.

  • pi0.5 native8 LoRA results, 2026-06-09 10:16 CST: atec-pi05-native8-100demo-s2-lora-gate-20260609095114.service has been completed. The training ended normally after 7500 steps, with the loss decreasing from 0.4452 to approximately 0.013-0.019 at the end, and the final checkpoint $DATA4_ROOT/01Proj/13atec/openpi_atec_runs/checkpoints/pi05_atec_task_e_native8_100demos_s2_224_lora/atec_task_e_pi05_native8_100demos_s2_224_lora_7500_20260609095114/7499 is around 8.6G. Automatic evaluation ACTION_REPEAT=2, logs logs/pi05_native8_lora_eval_100demos_s2_224_20260609095114.log / logs/pi05_native8_eval_20260609101354, all seeds 11/12/13 are 0.00, and none of the three objects were scored; video for seed11 is logs/videos/task_e_pi05_native8_eval/pi05_native8_pi05_atec_task_e_native8_100demos_s2_224_lora_seed11_20260609101354.mp4. Conclusion: Although referring to Pantheon’s LR/steps/discrete_state_input=False/LoRA freeze still results in failure, it indicates that the current OpenPI native8 bridge/data distribution/action timing issues are not resolved by LoRA. This LoRA should not be submitted, and the last official submission opportunity should not be used. The temporary heartbeat atec-pi0-5-native8-lora-gate-monitor has been removed.

    • Pi0.5 LoRA troubleshooting, 2026-06-09 10:30 CST: After checking the web page and OpenPI issue #672 and Pantheon cards, the current LoRA configuration is… freeze_filter、ema_decay=None、discrete_state_input=False Correct;num_steps=10/25/50 Offline sampling does not significantly improve the training frame error, indicating that it is not simply due to too few ODE sampling steps. Added demo/solution_pi05_native8.py Environment variables ATEC_PI05_CHUNK_EXEC_STEPS(Default is still 50, which does not affect the old behavior), test ATEC_PI05_CHUNK_EXEC_STEPS=5 seed11 still 0.00, video logs/videos/task_e_pi05_native8_eval/pi05_native8_lora_chunk5_seed11.mp4The explanation is also not simple; a 50-step open loop is too long. A stronger cause: the current training data for pi0.5 comes from trajectory_filtered.hdf5, this file is composed of scripts/act/filter_demos.py Delete the static/hold step, 100 demos from 749560 Frame pressed to 169700 Frame, approximately removed 77%; Unfiltered data qpos diff p50 1.5e-5, after filtering, qpos diff p50 0.00936、max 0.419This will disrupt the real-time semantics of the OpenPI/pi0.5 50-step action chunk. Another issue: the state is absolutely qpos, but the action is an ATEC environment action. (target-default)/0.5, which is inconsistent with the common “joint-position action” semantics seen in publicly available Piper LeRobot data; the subsequent pi0.5 should be preferentially changed to unfiltered data + absolute joint target action, and then transformed back into env action via the output transform.

  • pi0.5 rawabs/action-expert investigation, 2026-06-09 12:30 CST: The raw unfiltered + absolute target action has been implemented according to the previous suggestion. Key change:scripts/pi05/convert_task_e_hdf5_to_lerobot_native8.py Supported --action_mode absolute_target And --max_frames_per_episode；$HOME/src/openpi-ebench-clean/src/openpi/policies/atec_piper_policy.py Supported AtecPiperOutputs(action_mode="absolute_target")During reasoning, convert the model’s absolute joint target back to ATEC env action;demo/solution_pi05_native8.py Default ATEC_PI05_CHUNK_EXEC_STEPS=10, and add optional options ATEC_PI05_ZERO_NOISE=1/ATEC_PI05_ACTION_HORIZON。1-demo dataset is $DATA4_ROOT/01Proj/13atec/robotics_shared/datasets/lerobot/atec/task_e_obj321_servo_rawabs_1demo_native8_s10_224（raw trajectory.hdf5, 1 episode, stride=10, 250 frames, absolute target action).

  • pi0.5 rawabs results, 2026-06-09: new configs added pi05_atec_task_e_native8_rawabs_1demo_s10_224_lora_base_h10_2k、..._aefull_h10_2k、..._aefull_h10_10k_lr5 And script scripts/pi05/run_pi05_native8_rawabs_1demo_s10_aefull_h10_10k_lr5.sh。aefull_h10_10k_lr5 From local pi05_base/params Start, only train action expert/action heads/time MLP.discrete_state_input=False、horizon=10、LR 5e-5、EMA off；checkpoint $DATA4_ROOT/01Proj/13atec/openpi_atec_runs/checkpoints/pi05_atec_task_e_native8_rawabs_1demo_s10_224_aefull_h10_10k_lr5/atec_task_e_pi05_native8_rawabs_1demo_s10_224_aefull_h10_10k_lr5_202606091220_aefull_h10_10k_lr5/9999 About 8.8G。Offline probe from 2k mean first/chunk L2 1.51/1.65 Improved to 10k 0.28/1.10; Fix the zero noise approximately 0.22/0.87, but the chunk is still unstable. Tested with Isaac Seed11:CHUNK_EXEC_STEPS=1 Get 0.00(Video) logs/videos/task_e_pi05_native8_eval/pi05_native8_pi05_atec_task_e_native8_rawabs_1demo_s10_224_aefull_h10_10k_lr5_seed11_20260609122037.mp4），CHUNK_EXEC_STEPS=10 Get 0.00(Video) ..._20260609122341.mp4，montage ..._20260609122341.montage.jpg），CHUNK_EXEC_STEPS=10 + ATEC_PI05_ZERO_NOISE=1 Still 0.00(Log) logs/pi05_native8_eval_20260609122748). Conclusion: The OpenPI server, unnormalize, and env action scale have been verified to execute, but the policy lacks a continuous phase near the home/high-level observation area. The action exhibits jitter between the home and mid-trajectory, preventing effective pick/place operations; this pi0.5 rawabs branch should still not be submitted as the final candidate on the official website.

  • pi0.5 data policy, 2026-06-08/09: Do not add GraspGen/PCA data for now. Reason: The main issue with the old pi0.5 is the embodiment/action-time bridge. If the native8 videos remain unchanged, adding data will only exacerbate errors. The 20-demo native8 s2 already provides seed11. 6, so currently we run the 100-demo stride=2 gateing, which is faster than stride=1 and preserves the grasping sequence. Only when 100-demo native8 s2 is still below 15 will data be added: LoRA/small adapter target 50-150 High-quality real observation-action demo; full/continued fine-tuning targets 300-500 Lines, covering object_1/2/3, with different initial positions and grasping orders. GrabGen/PCA can only serve as a data generator for oracle/candidate generators; the final training data must be competition observations and valid 8D env actions, and cannot rely on oracle object pose for submission reasoning.

  • pi0.5 evaluation tool:demo/solution_pi05.py It is a websocket bridge;scripts/pi05/eval_task_e_pi05.py It is the Isaac evaluation entry point;scripts/pi05/run_pi05_eval_checkpoint.sh <checkpoint> Will start scripts/serve_policy.py, run seeds 11/12/13Record a video with seed11;scripts/pi05/summarize_pi05_eval.py Summary score;scripts/pi05/compare_pi05_vs_act_baseline.py Compare with XSA baseline and print HOLD_ACT_XSA Or REVIEW_FOR_DEPLOYThe 100-demo branch has been automatically output. HOLD_ACT_XSA。

    • Next steps for AnyGrasp: AnyGrasp/AnyGrasp SDK is worth trying as a short-term engineering solution, but it is not a “100% one-click solution” that directly replaces the entire policy. The recommended approach is: use SAM3/task prior mask to define the target point cloud region, generate 6D grasp pose and collision-free candidates with AnyGrasp, then integrate IK/trajectory planning from ATEC Piper, real finger center contact, and short-term stabilization basket-center release. Prioritize verifying object_3 first, followed by object_1 and object_2; strictly avoid using the oracle object pose/mask for the final submission. Additional checks are required for AnyGrasp license, source of model weights, and competition reproduction review materials.
    • AnyGrasp configuration attempts, 2026-05-30: The official SDK has been cloned. third_party/anygrasp_sdk; Python 3.11 binary has been placed as third_party/anygrasp_sdk/grasp_detection/gsnet.so And third_party/anygrasp_sdk/grasp_detection/lib_cxx.so. Isaac/Training Python is $HOME/envs/genmanip-isaac5-py311/bin/python, environmental observation is in Python 3.11.15、torch 2.7.0+cu128、CUDA runtime 12.8、GPU NVIDIA RTX PRO 6000 Blackwell Workstation，open3d==0.19.0、graspnetAPI==1.2.11 Available,pointnet2._ext Must be done first import torch Import again later. SDK dependencies libcrypto.so.1.1Currently, it is solved temporarily using the OpenSSL 1.1 library built into the Isaac ROS2 bridge:$HOME/projects/manipdojo2026/micromamba/envs/genmanip-sim/lib/python3.10/site-packages/isaacsim/exts/omni.isaac.ros2_bridge/humble/lib, it must be placed inside during execution LD_LIBRARY_PATH。
    • AnyGrasp license/weight history blockage, on 2026-05-30, removed on 2026-06-03: The official SDK is an authorized binary.license_checker -f Need ifconfig; This machine has no system. ifconfig, with compatible packaging added tools/anygrasp/ifconfig. The current machine feature id is 9024425463046318996. Not received yet on 2026-05-30 licenseCfg.json And checkpoint_detection.tarAt that time, it could not be claimed that AnyGrasp was able to run ATEC grasping; on 2026-06-03, the license/weight was received and installed, as shown in the below smoke record.
    • AnyGrasp MinkowskiEngine is installed, 2026-05-30: Official modified MinkowskiEngine Cloned to third_party/anygrasp_sdk/dependencies/MinkowskiEngine, branch cuda-12-1. The system only has GCC/G++. 13.3.0, will trigger CUDA 12.8 std::__to_address Ambiguity; new isolation compiler prefix created $HOME/envs/anygrasp-gcc11, containing conda-forge GCC/G++ 11.4.0, and only patch the private header files for this prefix $HOME/envs/anygrasp-gcc11/lib/gcc/x86_64-conda-linux-gnu/11.4.0/include/c++/bits/shared_ptr_base.h, keep the two places auto __raw = __to_address(__r.get()); Change to auto __raw = std::__to_address(__r.get());, no system modification /usr/include. The system lacks a BLAS devel symlink; the link in the created project must be retained third_party/anygrasp_sdk/dependencies/blas_lib/libblas.so -> /usr/lib/x86_64-linux-gnu/blas/libblas.so.3, and use the local version third_party/anygrasp_sdk/dependencies/blas_include/cblas.h Header file. Key points for the successful compilation command:CUDA_HOME=/usr/local/cuda-12.8、TORCH_CUDA_ARCH_LIST=12.0、MAX_JOBS=4、CC/CXX/CUDAHOSTCXX Point to $HOME/envs/anygrasp-gcc11/bin/x86_64-conda-linux-gnu-gcc/g++，--blas_include_dirs=third_party/anygrasp_sdk/dependencies/blas_include，--blas_library_dirs=third_party/anygrasp_sdk/dependencies/blas_lib. After installation $HOME/envs/genmanip-isaac5-py311/bin/python Importable MinkowskiEngine 0.5.4。
    • AnyGrasp local inspection entry:tools/anygrasp/check_anygrasp_env.sh It will be set up PATH/LD_LIBRARY_PATH, check SDK files, torch/CUDA, Open3D, GraspNetAPI, pointnet2, MinkowskiEngine, feature id, and checkpoint gaps. Review results on 2026-05-30:gsnet.so、lib_cxx.so、open3d 0.19.0、graspnetAPI 1.2.11、pointnet2._ext、MinkowskiEngine 0.5.4 All OK; still missing official version licenseCfg.json and checkpoint_detection.tar/AnyGrasp Weight.
    • AnyGrasp license/weight installation script, 2026-05-30: Obtain the official license zip and checkpoint_detection.tar After that, run tools/anygrasp/install_license_and_checkpoint.sh <license_zip_or_dir> <checkpoint_detection.tar>; it will install the license third_party/anygrasp_sdk/grasp_detection/license And third_party/anygrasp_sdk/grasp_tracking/license, place the weights in third_party/anygrasp_sdk/grasp_detection/log/checkpoint_detection.tar, and call it license_checker -c Verification. After installation, use tools/anygrasp/run_detection_demo.sh Run the official example data detection demo; this script has been verified to report an error quickly when the license is missing, and will not skip half of the process by mistake.
    • The AnyGrasp license/windowing factor is in place and the script runs successfully. On June 3, 2026: The user received an email from Chenxi Wang with the AnyGrasp SDK license, and the attachment is located at $HOME/Downloads/license_KeweiChen.zip; detection weight is downloaded from the official Google Drive $HOME/Downloads/anygrasp_weights/checkpoint_detection.tar, approximately in size 283M. Already executed tools/anygrasp/install_license_and_checkpoint.sh $HOME/Downloads/license_KeweiChen.zip $HOME/Downloads/anygrasp_weights/checkpoint_detection.tar, install to third_party/anygrasp_sdk/grasp_detection/license、third_party/anygrasp_sdk/grasp_tracking/license and third_party/anygrasp_sdk/grasp_detection/log/checkpoint_detection.tar。license_checker Print KeweiChen.lic check passedAlthough the check command returns a non-0 code occasionally, the SDK is actually loaded when needed. license passed: True, state: FvrLicenseState.PASSED。

  • AnyGrasp detection smoke results, 2026-06-03: Fixed a issue where after installing pointnet2, $HOME/envs/genmanip-isaac5-py311/bin/python can import pointnet2._ext, pointnet2.pointnet2_utils, and pointnet2.pointnet2_modules. Running tools/anygrasp/run_detection_demo.sh successfully completed the official grasp_detection/example_data inference, producing 20 grasp scores with a maximum value of approximately 0.47641015. This proves that AnyGrasp SDK detection is available on this machine; next, an ATEC bridge can be written to extract the target region from Isaac RGB-D/point cloud, call AnyGrasp to generate grasp pose, and then perform Piper IK/execution primitives.

  • AnyGrasp ATEC bridge has been integrated, 2026-06-03: Added scripts/graspnet_task_e/anygrasp_adapter.py; scripts/graspnet_task_e/run_graspnet_pick.py supports --grasp_provider anygrasp, and the encapsulation entry is scripts/graspnet_task_e/run_anygrasp_pick.sh <object> <seed>. The adapter pre-loads ROS2 bridge’s libcrypto.so.1.1 / libssl.so.1.1 within the Isaac Python process, and injects tools/anygrasp/ifconfig into PATH. Otherwise, the SDK will encounter OpenSSL or machine-id/license verification issues. The detector is loaded solely with smoke: $HOME/envs/genmanip-isaac5-py311/bin/python -c 'from scripts.graspnet_task_e.anygrasp_adapter import _load_anygrasp_detector; _load_anygrasp_detector()', and license passed: True should be observed.

  • AnyGrasp Task-E smoke results, 2026-06-03: object_3 succeeded under --use_task_quat --tcp_z_offset 0.040 --close_z_offset -0.020, inside=True, and z_gain_lift=0.137, with video logs/videos/task_e_anygrasp/anygrasp_obj3_seed11_taskquat_lowclose.mp4; object_2 also succeeded with the same parameters, inside=True, with video logs/videos/task_e_anygrasp/anygrasp_obj2_seed11_taskquat_lowclose.mp4. object_1 currently failed in two rounds: anygrasp_obj1_seed11_taskquat_lowclose_retry.mp4 and anygrasp_obj1_seed11_defaultquat_nooffset_lowclose.mp4. The common issue is that the SDK only provides low-score/clearly narrow width candidates (approximately 0.006-0.008m), and the gripper closes but the object does not rise with lift. Do not mistake the current AnyGrasp bridge for a three-object full score policy; it has passed integration verification, but object_1 still needs to continue tuning along the existing Task-E calibration template.

  • Recommendations for using AnyGrasp, 2026-06-03: The most stable approach currently is to use AnyGrasp as a RGB-D grasp candidate/center/yaw prior, then force the use of Task-E verified execution parameters (--use_task_quat, low close z, finger-center/staged transport, basket stable release). Do not directly use AnyGrasp raw 6D wrist pose; object_3’s raw pose failed before switching to Task-E quaternion. For object_1, prioritize reusing the control template from the previous GraspNet success evidence: center at the upper-surface median, add OBJ_GRASP_CENTER_OFFSETS[1], --close_z_offset -0.020, and keep the grasp quat through transport/place/open, along with long transport and real finger-center servo.

  • Review of third-party routes for GraspGen/AgileX, 2026-06-03: Online research revealed that AgileX community/tutorials recommend PiPER’s GraspGen route. AgilexRobotics/GraspGen is currently inaccessible, but the accessible mirror/author repository vanstrong12138/GraspGen has a ros2_jazzy_version branch; it has been shallow-cloned into third_party/GraspGen. AgileX’s official teaching repository agilexrobotics/Agilex-College has also been shallow-cloned into third_party/Agilex-College, and its piper/GraspGen README describes the PiPER + YOLOv8-Pose/TensorRT + PCA/AABB + IK/gripper control workflow. The core code is located at third_party/GraspGen/sam3/realsense-sam.py::compute_grasp_from_pca_aabb(): 3D PCA -> principal axis AABB -> shortest edge as the gripping direction.

  • GraspGen-style PCA/AABB ATEC integration, 2026-06-03: Added scripts/graspnet_task_e/pca_aabb_adapter.py, run_graspnet_pick.py supports --grasp_provider pca. The object_1 smoke command uses --grasp_provider pca --object 1 --seed 11 --mask_provider oracle --use_task_quat --tcp_z_offset 0.040 --close_z_offset -0.020 --close_steps 160 --move_steps 200 --transport_steps 1400 --place_steps 260, and the output is [PCA_AABB] points=5231 extents=(0.140,0.122,0.056) axis=2 width=0.056. This indicates that the gripper width calculated using the geometric method is reasonable, distinctly different from the abnormally narrow width of 0.006-0.008m in AnyGrasp. However, this attempt still failed: pick=(0.950,0.261), z_gain_lift=-0.024, inside=False, with a video showing logs/videos/task_e_pca/pca_obj1_seed11_taskquat_lowclose.mp4. Cause analysis: The single-view RGB-D/mask only shows the visible side of object_1, and the execution center of PCA/upper-surface median tends to be on the left side. It is necessary to combine known object dimensions, multiple views, learned completion, or the successful center from old GraspNet to fill in the true gripping center.

  • Fix for “poking the side of the block without it entering the gripper” for object_1, 2026-06-03: Based on the video, when the user indicated the block was on the left side, the Piper gripper only pushed/penetrated the block but did not allow it to enter between two fingers before closing. Re-testing confirmed: PCA/AABB with --preclose_insert_dx 0.040 --close_z_offset -0.020 still failed, z_gain_lift=-0.025. In the trace, during the close phase, min_finger_dist≈0.058m, indicating that the fingertip center is too far from the target. A failed video is logs/videos/task_e_pca/pca_obj1_seed11_insert_dx004.mp4. The actual effective fix is not to manually lower the close height, but to align the parameters of object_1 with the Task-E collector: OBJ_GRASP_CENTER_OFFSETS[1]=(0.025,0,0), OBJ_GRASP_Z_OFFSETS[1]=0.072, OBJ_CLOSE_Z_OFFSETS[1]=0.020, and use the actual link7/link8 finger-center servo. Successful command: OMNI_KIT_ACCEPT_EULA=YES PYTHONUNBUFFERED=1 $HOME/envs/genmanip-isaac5-py311/bin/python scripts/graspnet_task_e/run_graspnet_pick.py --grasp_provider pca --object 1 --seed 11 --mask_provider oracle --use_task_quat --tcp_z_offset 0.072 --close_steps 180 --move_steps 200 --transport_steps 1400 --place_steps 260 --video_path logs/videos/task_e_pca/pca_obj1_seed11_cfgclose.mp4 --save_debug_npz logs/pca_task_e/obj1_seed11_cfgclose_debug.npz --headless; results are inside=True, z_gain_lift=0.144, with trace reach min_finger_dist≈0.006m and close level ≈0.024m. Conclusion: The failure of object_1 is not due to “insufficient gripper force” or “need to switch to AnyGrasp”, but rather a pre-closing collision caused by errors in TCP/finger-center/close height calibration. run_graspnet_pick.py has incorporated preclose_insert stats into [TRACE]. If parameter sweeps continue, it is necessary to examine the finger-center errors for reach/insert/close/lift before making a judgment.

  • demo/solution_pca.py object_1 submit-style repair progress, 2026-06-04: Do not overwrite the demo/solution.py/ACT baseline; solution_pca.py remains an experimental branch. The latest single-object seeds 11/12/13 have all successfully entered the basket: seed11 produced score=6.00, initial (1.0945,0.2979,0.8833), and final (1.0582,-0.3256,0.8820); under the latest tight parameters, seed12 achieved score=6.00 and final (1.0338,-0.2790,0.8584), while seed13 achieved score=6.00 and final (1.0257,-0.2567,0.8871). Verification entry: ATEC_PCA_OBJECTS=1 OMNI_KIT_ACCEPT_EULA=YES PYTHONUNBUFFERED=1 $HOME/envs/genmanip-isaac5-py311/bin/python scripts/graspnet_task_e/debug_solution_pca_execution.py --seed <11|12|13> --object 1 --max_steps 4800 --headless. Key parameters: object_1 disables ROI fill-hole point cloud center; visible face center completion is performed via OBJ_CENTER_COMPLETION_OFFSETS[1]=(0.020,0.0); OBJ_GRASP_CENTER_OFFSETS[1]=(0,0); during the reach phase, do not use low-finger-z servo; object_1 release/open height is calibrated using runner TABLE_TOP_Z+0.32/+0.24; OBJ1_HOLD_GAP=0.0415, OBJ_LIFT_STEPS[1]=300, close/lift/mid are recorded after reaching the gap threshold for the first time _obj1_hold_grip, and subsequent lift/mid/release/settle maintain this gripper opening to prevent the object from being squeezed out when closing to 0. Error experience: OBJ_PLACE_XY_OFFSETS[1]=(0,-0.250) causes the release target to become y=-0.55, resulting in the object falling into the basket before completion; the wide ROI/white grid in object_1 rescue is prone to false detection and should be disabled by default, unless reliable foreground segmentation is recreated.

  • demo/solution_pca.py object_2/3 status, 2026-06-04: object_2/3 still lacks submitable stability; do not claim that the PCA solution has achieved full score. In the runner comparison, object_3 seed11 succeeded once, but retesting similar runner parameters on 2026-06-04 failed, indicating that the previous success was not stable. The obj321_servo collector log on May 20 proved that all three objects can achieve full score under official state_machine/real object root; object_3 successfully traced common z_gain≈0.42, gap_close≈0.063-0.069, finger_close≈0.016-0.030, finger_vec_close.x≈-0.01~-0.03, and basket_inside=True. Key new progress for submit-style object_3: both low and high defaults are unstable; the issue progressed from “no contact at all” to “strong lift possible but sliding during transport”; current experimental default values are adjusted to OBJ_CLOSE_Z_OFFSETS[3]=0.030, OBJ3_HOLD_GAP=0.055, OBJ_FINGER_XY_OFFSETS[3].x=-0.030, and OBJ_LIFT_STEPS[3]=40, which can produce strong lifts in some seed11 placements, such as logs/pca_solution_obj3_fxneg003_full_seed11_20260604.log producing score=3.00 and max_z_gain=0.195, with short lift logs logs/pca_solution_obj3_shortlift_seed11_20260604.log outputting max_z_gain=0.236; however, mid/release phases still slide back onto the table and do not enter the basket. Failure boundaries: hold_gap=0/0.045/0.060/0.065, close_z=-0.020/0/0.020/0.050/0.070/0.090, PCA yaw, ATEC_PCA_USE_FULL_IK=1, and ATEC_PCA_OBJ3_OBJECT_SERVO=1 remain unsolved. Next, stop questioning whether “no object is caught at all”; focus on maintaining object_3 transport/low-position transfer: immediately after grasping, perform lateral movement, reduce pure vertical lift, or use closed-drag/under-scoop to enter the basket. object_2 still needs individual realignment; do not apply compensation similar to that used for object_1/object_3.

  • demo/solution_pca.py additional fix for object_3 sliding, 2026-06-04 18:20 CST: The “short lift + dynamic visual object-servo + low-position fallback” approach has been implemented in the experimental branch according to the user’s scheme, but seed11 still fails; do not deploy it. Current file changes include: OBJ3_HOLD_GAP=0.066, OBJ3_HOLD_GRIP_DEFAULT=(0.032,-0.034), OBJ3_CLOSE_MIN_STEPS=160, OBJ3_LOW_HOLD_STEPS=140, OBJ3_APPROACH_FINGER_Z=TABLE_TOP_Z+0.090, OBJ3_CLOSE_FINGER_Z=TABLE_TOP_Z+0.010, OBJ3_LIFT_FINGER_Z=TABLE_TOP_Z+0.080, OBJ3_OBJECT_SERVO_GAIN=0.12, OBJ3_OBJECT_SERVO_MAX_XY=0.040, and OBJ3_FINGER_SERVO_MAX_XY=0.080, and changing object_3 fallback to trigger earlier than obj3_mid stagnation, OBJ3_FALLBACK_DRAG_Z=TABLE_TOP_Z+0.035, and fallback steps 60/260/260/100. Note: do not freeze the actual qpos as object_3 hold grip, which previously locked the asymmetric gripper opening like (0.035,-0.026); instead, use the calibrated default hold grip. Do not rely solely on gripper_base z; object_3 reach/close/lift requires explicit finger-center z.

  • object_3 latest trace boundary, 2026-06-04: logs/pca_solution_fix/obj3_seed11_runner_vec_match_debug_20260604_181342.log demonstrates that the old default OBJ_FINGER_Y_OFFSET=+0.003 causes the close finger-center to fall on the y-side of the banana center by approximately +0.024m, pushing the banana sideways apart; the default y offset has been changed to -0.020. logs/pca_solution_fix/obj3_seed11_yoffset_neg020_debug_20260604_181533.log shows that y returns to normal, but x may be too negative; temporarily covering ATEC_PCA_OBJ3_FINGER_X_OFFSET=-0.004 ATEC_PCA_OBJ3_FINGER_Y_OFFSET=-0.020 with logs/pca_solution_fix/obj3_seed11_xyoffset_xneg004_yneg020_debug_20260604_181706.log yields a close vector closer to the runner: the actual finger is approximately (x=+0.002,y=-0.015,z=-0.014) relative to the current visual center, but still score=0, indicating that the issue has progressed from “finger not in place” to “no viable contact after closing, and the visual center may be obscured by the gripper”. Early trigger of fallback logs/pca_solution_fix/obj3_seed11_early_low_fallback_20260604_181908.log only pushes object_3 to (1.083,0.055,0.841), with x close to the basket but y still far from the basket center +0.355m; the current closed-drag height/contact direction still fails to move the banana.

  • Next steps for object_3, 2026-06-04: Continue focusing on submit-style grasping, avoid retreating to ACT/training, and do not change the official physics. Prioritize implementing and verifying true table-level under-scoop/sweep: entering from the +y side of the current visual center or outside the arc, with finger z lower than TABLE_TOP_Z+0.035 in small steps toward BASKET_CENTER_Y, while maintaining the current center re-estimation for object_3; fallback should trigger earlier at obj3_mid and complete within 1500 steps. If continuing to adjust clamping, the target trace still aims for success with runner logs/graspnet_task_e_obj3_sam3_closed_drag.log: inside=True, z_gain_lift=0.207, reach/close/lift min_finger_vec≈(-0.007~-0.010, -0.002~+0.002, -0.014~-0.011), gap≈0.066-0.068. Any new parameters must use ATEC_PCA_DEBUG_TARGET=1 or debug_solution_pca_execution.py to refer to finger-center/object-center values, and do not rely on video analysis alone.

  • User-specified record for object_3: “short lift + low cradle/servo + stable basket release”, 2026-06-04 20:35 CST: Implemented in the demo/solution_pca.py experimental branch, but still not deployable. Key changes: OBJ3_HOLD_GAP=0.0675, OBJ3_CLOSE_STEPS=90, OBJ3_LIFT_STEPS=35, OBJ3_LOW_HOLD_STEPS=0, OBJ3_LIFT_FINGER_Z=TABLE_TOP_Z+0.045, OBJ3_FINGER_X_OFFSET=-0.010; hold grip changed to latch only at obj3_lift and when the gap reaches a threshold, avoiding premature locking during the close phase; obj3_basket_hold fixes a clear bug: do not set dynamic_finger_xy to the current banana center c_now, but keep object_target_xy + carry_offset, otherwise the hand will be pulled back to the old position. Validation logs: logs/pca_solution_fix/obj3_seed11_eval_close90_shortlift_fasttransport_20260604_201931.log shows successful latch but error in basket_hold desired, and ultimately object_3 remains at y≈0.068; after fixing, logs/pca_solution_fix/obj3_seed11_eval_fixservo_transport440_20260604_202219.log and ...fingerik_20260604_202355.log still get stuck at finger y≈-0.09; the low-position closed-drag version logs/pca_solution_fix/obj3_seed11_eval_low_closed_drag_20260604_202611.log also only reaches finger y≈-0.09, with the object not entering the basket; position-only DLS logs/pca_solution_fix/obj3_seed11_eval_lowdrag_positiononly_*.log allows the gripper_base to get closer to the basket, but finger z rises too high to ≈1.0m, resulting in loss of contact. Conclusion: The current object_3 grasping pose/short lift can be achieved, and the main blocker is the submit-side Pinocchio DLS/traffic control, which does not reproduce the real PhysX Jacobian behavior of runner/collector Isaac CartesianController, especially the y motion of the basket and the retention of finger z; do not attribute the primary issue to GraspNet/AnyGrasp candidate models anymore.

    • object_3 subsequent shortest path, 2026-06-04 20:35 CST: prioritize replicating IsaacLab CartesianController Details of action generation, rather than continuing to switch to the grasp model. The official version has been confirmed get_action_spec() Only the action group can be modified. mode/scale/clipIt cannot directly submit the Cartesian action. If used, only the arm velocity/scale diagnosis can be attempted, and pose control cannot be completely resolved. Next steps: 1) Use scripts/graspnet_task_e/debug_solution_pca_execution.py Add the “stuck phase” q6/joint limit printing, and confirm whether it is a joint limit or DLS/rotation error; 2) Change demo/solution_pca.py _rot_error/DLS Damping/Clamp alignment per step source/atec_rl_lab/atec_rl_lab/utils/cartesian_controller.py, prioritize using position+current wrist quat instead of forcing topdown; 3) If still not working, create a small joint-space transport library: The runner generates relative joint waypoints from the initial positions of several object_3 y-band, and submits them through visual center selection/interpolation, rather than calculating Cartesian IK at each step. Keep it as is. demo/solution.py / demo/policy_act.pt The ACT/XSA baseline remains unchanged.solution_pca.py Used only as an experimental branch.
    • PCA/AABB + RGB-D band mask for three-object single smoke, 2026-06-03: The goal is to avoid the non-distribution restrictions of the AnyGrasp SDK, thereby establishing a reproducible alternative approach to geometric grasping.scripts/act/task_e/config.py OBJ_SPAWN_Y_BANDS Official alignment achieved source/atec_rl_lab/atec_rl_lab/tasks/task_e/env_cfg.py：object_1 [+0.25,+0.29]、object_2 [+0.14,+0.20]、object_3 [+0.03,+0.09]；scripts/graspnet_task_e/tuntun_adapter.py rgbd_band_object_mask() Added object-specific z gate to prevent mixing with table/arm/ basket points. Successful command for object_1 band:... run_graspnet_pick.py --grasp_provider pca --object 1 --seed 11 --mask_provider band --use_task_quat --tcp_z_offset 0.072 --close_steps 180 --move_steps 200 --transport_steps 1400 --place_steps 260 --video_path logs/videos/task_e_pca/pca_obj1_seed11_band_cfgclose_yband_zfilter.mp4 ..., result inside=True、z_gain_lift=0.144. object_2 passed the re-test after the official y-band narrowing:... --object 2 --seed 11 --mask_provider band --use_task_quat --tcp_z_offset 0.040 --close_z_offset -0.020 --transport_steps 1400 --place_steps 260 --video_path logs/videos/task_e_pca/pca_obj2_seed11_band_yalign_taskquat_lowclose.mp4 ..., result inside=True、z_gain_lift=0.119. object_3 initially has the ability to grab with band/PCA, but will slide out during high-altitude release; the fix is in scripts/graspnet_task_e/pca_aabb_adapter.py In particular object_index==3 Use the oriented AABB center instead of the upper-surface median, and run_graspnet_pick.py Add --open_release_z Supports “high-position transportation and low-position claw opening”. object_3 seed11 AABB-center High-position release can be successful once:logs/videos/task_e_pca/pca_obj3_seed11_band_aabbcenter_taskquat_lowclose.mp4, result inside=True、z_gain_lift=0.161; the high-level release of seed12 will slip out, so switch to it --release_z 0.32 --open_release_z 0.24 --place_steps 220 Successed later, video logs/videos/task_e_pca/pca_obj3_seed12_band_aabbcenter_highcarry_lowopen.mp4, result inside=True, the final in-the-court delta is approximately (-0.031,-0.049). Note: These are single-object runner tests. They still need to be combined into a three-object continuous episode and migrated to demo/solution.py the observation-only control logic; current run_graspnet_pick.py The env object state will still be read for verification/dynamic servo, and it cannot be directly used as the final submission.

  • Debugging of the submit-style PCA controller, 2026-06-04: demo/solution_pca.py is a experimental branch and not deployed; demo/solution.py remains the ACT/XSA baseline. scripts/act/eval_task_e_act.py now supports --solution_module solution_pca, and scripts/graspnet_task_e/debug_solution_pca_execution.py can quantify finger_dist/max_z_gain/pin_mean_err using Isaac internals. It has been confirmed that Pinocchio FK aligns with Isaac link7/link8/gripper_base, pin_mean_err finger=0.0000 gb=0.0000, so the main reason for “not grasping or failing to grasp” is not a FK coordinate error, but the submit-style IK/controller fails to reproduce the low-level contact ability of run_graspnet_pick.py’s Isaac CartesianController.

  • Current status of submit-style object_1, 2026-06-04: Attempts have been made to migrate the successful template of object_1 from run_graspnet_pick.py to demo/solution_pca.py, including RGB-D world AABB center completion, OBJ_CENTER_COMPLETION_OFFSETS[1]=(0,+0.018), object_1 x execution offset received from +0.025 as +0.010, a low close plane TABLE_TOP_Z+0.030, and direct finger-center IK. Conclusion: the perception center can achieve centimeter-level accuracy, but current Pinocchio IK often traps link7/link8 finger-center above the object during the close phase by approximately +0.05m and max_z_gain≈0, preventing object_1 from being lifted. Direct finger-center IK and the forced object_1 task quaternion (0,0.707,0.707,0) are even worse, and we have returned to the gripper_base compensation approach; do not mistake this for a failure in Grasp/PCA/AnyGrasp itself. The next step should be to replicate the differential IK mathematics/parameters of Isaac CartesianController, or implement a more reliable DLS resolved-rate joint update within the submit policy, rather than continuing to switch grasp models.

  • Update on preclose insert for submit-style object_1, 2026-06-04: demo/solution_pca.py added _PiperIK.step_dls(), runner-style finger-center XY correction, object_1 task quaternion [0,0.709,0.705,0], object_1 relative-z correction, preclose insert, and object_1 release overshoot. Video logs/videos/task_e_pca_solution/pca_solution_obj1_seed11_best_trace_20260604_114715.mp4 clearly shows that without insert, the gripper mouth stays on the side of the cube, and the cube remains on the table when lifted. After adding insert, OBJ_PRECLOSE_INSERT_OFFSETS[1]=(0,-0.040) achieved score=3, max_z_gain=0.137m, but slipped onto y≈-0.047 during transportation; (0,+0.025) achieved max_z_gain=0.193m, and ultimately y≈-0.083, yet still failed to enter the basket, and another attempt failed to grasp at a random position. Conclusion: the direction/amplitude of insert is key to effective grasping, but object_1 remains unstable; current demo/solution_pca.py is still only for debugging, and do not override demo/solution.py or demo/policy_act.pt.

  • Comparison with GraspNet object_1, 2026-06-03: Attempted to replicate old known successful commands. Video target logs/videos/task_e_graspnet/graspnet_pick_obj1_seed11_recheck_20260603.mp4, and the process took over 3 minutes in the GraspNet inference/candidate generation phase without any output from [GRASP], consuming approximately 6.6GB of GPU resources and high CPU load, so it was manually killed. This is not evidence of successful or failed grasping; if we want to continue comparing with GraspNet, first investigate why the current recheck inference phase is slow/blocked.

  • Reference conclusion from AnyGrasp third-party, 2026-05-30: No official AnyGrasp for Piper project exists for Songling/AgileX; AgileX mainly provides PiPER Python/ROS/ROS2/URDF/simulation control stacks. Third-party/paper approaches support “using AnyGrasp as a pre-contact/contact-pose prior, then applying robot IK/trajectory planning”: DKT’s work on transparent objects mentions RealSense D435 + PiPER Arm + AnyGrasp + CuRobo grasp stack; BayesVLA splits the policy into visual action prior and language likelihood, and assigns the pre-contact phase to frozen contact-pose models like AnyGrasp. Insights for this project: AnyGrasp should first generate candidate grasping poses and perform ranking, while ATEC Piper still requires separate handling of TCP calibration, IK/trajectory, gripper closing/anti-slip during transportation, and high-stability release at the basket center.

  • Current work recommendations: 1) Keep demo/policy_act.pt’s XSA-ACT baseline unchanged; 2) If aiming for short-term improvement, prioritize using AnyGrasp/SAM3 as a grasp candidate generator to integrate into the existing calibrated Piper primitive, rather than continuing blind training with pi0.5; 3) If continuing VLA, redo the OpenPI/Piper embodiment adapter (native Piper dims/cameras/norm/discrete_state_input), and do not use fixed-base 16D padding as the final conclusion; 4) Before each deployment, must independently evaluate seeds 11/12/13, and keep at least one video and [BASKET] basket entry logs.

  • Latest status, 2026-05-20 20:55 CST:

    • Supersedes the older “object 1/2 unresolved” notes below; keep them only as historical debugging context.
    • L0 Task E three-object scripted data generation now works well enough to collect demos. Verified all-object order is 3 2 1 with --only_success --trace --abort_failed_lift.
    • Verified object-1-only success: datasets/atec_task_e_probe_obj1_graspoffset_servo/trajectory.hdf5, 3/3 successful demos; trace z-gain about 0.387-0.417, basket_inside=True.
    • Verified object-2-only success: datasets/atec_task_e_probe_obj2_original_rootservo/trajectory.hdf5, 3/3 successful demos; trace z-gain about 0.302-0.304, basket_inside=True.
    • Verified three-object success: datasets/atec_task_e_probe_obj321_final_servo/trajectory.hdf5, 3 successful demos in 4 attempts; success traces have object_1/2/3 basket_inside=True.
    • Review video: logs/videos/task_e_obj321_final_servo/demo_0000.mp4; keyframe summary: logs/videos/task_e_obj321_final_servo/summary_5s.jpg.
    • Root cause/fix: data generation was wrong, not ACT training. Object 1 must keep the finger-centre servo target on obj_pos + OBJ_GRASP_CENTER_OFFSETS[1]; object 2 must track the USD/root object position, not the offset center. Both need closed-loop finger-centre servo during REACH/CLOSE/LIFT/TRANSPORT/PLACE.
    • Stable constants in scripts/act/task_e/config.py: object 1 offset (0.025,0,0), grasp z 0.072, close z 0.020, transport/place steps 2200/300; object 2 offset (0.060,0,0), grasp z 0.135, transport/place steps 2200/300; default PICK_OBJECTS="3 2 1".
    • scripts/act/task_e/collector.py has the important per-object servo distinction: object 1 servo/trace target uses obj_pos + OBJ_GRASP_CENTER_OFFSETS; object 2/3 use obj_pos. Do not “simplify” this back to one formula.
    • scripts/act/train_task_e.py uses lr_drop = max(int(2/3 * total_iters), 1) so tiny smoke tests no longer fail with StepLR(step_size=0).
    • CAP-X / RoboTwin lesson: use contact/TCP calibrated grasp candidates and closed-loop place logic first. GraspNet/ContactGraspNet can be added later as a candidate generator, but it is not required to fix this current oracle data bug.
    • Real pipeline command: DATA_DIR="datasets/atec_task_e_obj321_servo_100demos" PICK_OBJECTS="3 2 1" NUM_DEMOS=100 MAX_ATTEMPTS=260 TOTAL_ITERS=100000 BATCH_SIZE=512 SEED=1 LOG_FREQ=100 SAVE_FREQ=5000 RUN_NAME="act-task-e-rgb-obj321-servo-100demos-seed1" bash scripts/act/run_task_e_pipeline.sh.
    • Active long run started via systemd user unit because plain background/nohup could be killed with the Codex command: unit atec-task-e-obj321-servo-20260520211650.service, log logs/task_e_obj321_servo_pipeline_20260520_211650.log, latest symlink logs/task_e_obj321_servo_pipeline.latest.
    • As of 2026-05-20 21:22 CST the long run is active, Demo 1/100 succeeded and wrote datasets/atec_task_e_obj321_servo_100demos/trajectory.hdf5 (~1.3 GB), then proceeded to Demo 2/100.

  • Latest train/eval status, 2026-05-21 14:55 CST:

    • atec-task-e-obj321-servo-20260520211650.service completed normally; raw dataset is datasets/atec_task_e_obj321_servo_100demos/trajectory.hdf5 (~130 GB), filtered dataset is trajectory_filtered.hdf5 (~35 GB), run is runs/act-task-e-rgb-obj321-servo-100demos-seed1.
    • Checkpoints include best_loss.pt and final.pt; training reached 100000 iters, final logged loss around 0.0100.
    • First ACT eval of best_loss.pt scored 0/3 episodes; video logs/videos/task_e_act_eval/obj321_best_loss_100demos_20260521_145104.mp4 shows the arm staying near the basket/home pose instead of starting the object grasp.
    • Root cause found in demo/solution_act.py: the home stage treated env action as PD/velocity, but Task E action is (joint_target - default_joint_pos) / 0.5. Fix is to command the static teleop-home target action until stable, then switch to ACT. Re-run eval after this fix before judging model quality.
    • After the home fix, best_loss.pt eval improved: 1-episode score 9.00 with video logs/videos/task_e_act_eval/obj321_best_loss_homefix_1ep_20260521_145529.mp4; same-process 3-episode eval had valid first episode score 18.00 and video logs/videos/task_e_act_eval/obj321_best_loss_homefix_3ep_20260521_145652.mp4, but subsequent episodes reset to done=True at step 1, so do not treat that mean score as independent.
    • final.pt single-episode eval scored 6.00, so deploy best_loss.pt.
    • Deployment: previous demo/policy_act.pt backed up as demo/policy_act.prev_20260521_145951.pt; current demo/policy_act.pt is copied from runs/act-task-e-rgb-obj321-servo-100demos-seed1/checkpoints/best_loss.pt and sha256 matches 76bbef2c81c85fc6cc60448476ff09075ec8bc97217c2d2fb70a3253fd3e307d.

  • XSA ACT experiment, 2026-05-21:

    • Local XSA package source: $HOME/Documents/01Proj/13atec/xsa_update_20260520/lerobot_policy_act_xsa.zip; it is a LeRobot act_xsa plugin with 12-dim example config and is not directly runnable for ATEC Task E’s 8-dim Piper setup.
    • Minimal XSA adaptation added behind --use_xsa: source/atec_rl_lab/atec_rl_lab/train/act/act/detr/transformer.py and demo/act/detr/transformer.py support XSATransformerEncoderLayer / XSATransformerDecoderLayer; default ACT remains unchanged.
    • scripts/act/train_task_e.py now saves model_args in checkpoints; demo/solution_act.py reads model_args including use_xsa, so ordinary ACT and XSA checkpoints can both load.
    • Smoke test passed: runs/act-task-e-rgb-xsa-smoke-10demos-seed7/checkpoints/final.pt loads as XSATransformerEncoderLayer.
    • Active XSA comparison run: systemd user unit atec-task-e-xsa-30k-20260521165009.service, log logs/task_e_xsa_30k_20260521_165009.log, run name act-task-e-rgb-xsa-100demos-30k-seed11, command uses --use_xsa --total_iters 30000 --batch_size 256 --num_demos 100.
    • Mid-training evals are noisy but not yet encouraging: 19k-ish snapshot scored 9/18 with video logs/videos/task_e_act_eval/xsa_mid_best_1ep_20260521_174124.mp4; later current-best snapshot at 20260521_175038 scored 0/18 with video logs/videos/task_e_act_eval/xsa_current_best_1ep_20260521_175038.mp4. Do not infer ranking from training loss alone; run independent single-episode processes because same-process multi-episode reset can be invalid after success.

  • GraspNet / TunTunClaw integration, 2026-05-21:

    • User reference cloned into third_party/tuntunclaw/ from Datawhale/OpenClaw home assistant material. Official GraspNet checkpoint is third_party/tuntunclaw/temp/logs/log_rs/checkpoint-rs.tar.
    • Local bridge files: scripts/graspnet_task_e/tuntun_adapter.py and smoke runner scripts/graspnet_task_e/run_graspnet_pick.py.
    • Dependencies were made to work in $HOME/envs/genmanip-isaac5-py311/bin/python: open3d==0.19.0, local graspnetAPI, transforms3d==0.4.2, and the GraspNet pointnet2 CUDA extension built for Blackwell with TORCH_CUDA_ARCH_LIST=12.0.
    • Important lesson: raw GraspNet point/orientation alone did not lift object 1. The first successful object-1 smoke used GraspNet/TunTunClaw for candidate generation, but executed with Task-E calibrated controls: robust mask upper-surface median as execution center, object-1 OBJ_GRASP_CENTER_OFFSETS added, --close_z_offset -0.020, finger-center servo, keep grasp quaternion through transport/place/open, and long transport.
    • Known-good object-1 command: python scripts/graspnet_task_e/run_graspnet_pick.py --object 1 --seed 11 --tcp_z_offset 0.040 --close_z_offset -0.020 --use_task_quat --close_steps 160 --move_steps 200 --transport_steps 1400 --place_steps 260 --headless --device cuda:0 --video_path logs/videos/task_e_graspnet/graspnet_pick_obj1_seed11_keepquat_slow.mp4 --save_debug_npz logs/graspnet_task_e/obj1_seed11_keepquat_slow_debug.npz.
    • Verified result for that command: inside=True, z_gain_lift=0.164, final object_1 position about (1.043,-0.247,0.934), basket delta (-0.037,+0.053). Video: logs/videos/task_e_graspnet/graspnet_pick_obj1_seed11_keepquat_slow.mp4.
    • Object 3 GraspNet smoke also succeeded with refined AABB oracle mask and lower close height: command uses --object 3 --tcp_z_offset 0.055 --close_z_offset 0.015 --use_task_quat --place_x_offset 0.04 --place_y_offset 0.05; result inside=True, z_gain_lift=0.188, video logs/videos/task_e_graspnet/graspnet_pick_obj3_seed11_lowclose.mp4.
    • Object 2 GraspNet smoke is not solved yet. Best attempts can move/lift slightly (z_gain_lift≈0.07-0.10) but final y remains outside basket; simple --post_push moved it only to about y=-0.056. Keep using the existing Task-E calibrated state-machine/ACT path for object 2 until a better GraspNet contact primitive is found.
    • Added legal-ish RGB-D y-band mask provider rgbd_band_object_mask() and runner flag --mask_provider band; it avoids sim object state by using camera intrinsics, depth, table height, and official spawn y-bands. Current band-mask object-3 test did not lift yet, so the GraspNet success videos above still rely on oracle_object_mask()/AABB debug masks. Do not use oracle masks in final submission without replacement.
    • Three-object stable oracle/state-machine verification after GraspNet changes: scripts/act/collect_demos_task_e.py --pick_objects 3 2 1 --num_demos 1 --max_attempts 3 --only_success --trace --abort_failed_lift --save_video ... succeeded on attempt 2, with all object_1/2/3 basket_inside=True. Video: logs/videos/task_e_obj321_current_verify/demo_0000.mp4.

  • SAM3 segmentation / basket-center release, 2026-05-21:

    • Local SAM3 checkpoint exists at $HOME/Documents/01Proj/sam3d_gs/submodule/Prompt-Inpaint/checkpoints/sam3.pt; SAM3 is installed in $HOME/Documents/01Proj/sam3d_gs/.venv/bin/python, not in the Isaac env. Keep SAM3 isolated as a subprocess rather than installing it into $HOME/envs/genmanip-isaac5-py311.
    • Added scripts/graspnet_task_e/sam3_segment_image.py: standalone RGB image -> SAM3 binary mask. It uses local Prompt-Inpaint SAM3Predictor and writes .npy mask plus optional JSON metadata.
    • scripts/graspnet_task_e/run_graspnet_pick.py now supports --mask_provider sam3, --sam3_prompt, --sam3_threshold, and --sam3_python. It also defaults to --basket_center_release: during transport/place/open it servo-corrects the EE target by current object center error so the object center is driven to (BASKET_CENTER_X, BASKET_CENTER_Y) before opening.
    • SAM3 smoke on saved frames passed for all three prompts: object 1 best prompt box score 0.863, mask 31206 px; object 2 best prompt yellow bottle score 0.961, mask 2509 px; object 3 best prompt banana score 0.953, mask 1311 px. Overlays are logs/graspnet_task_e/sam3_smoke_obj{1,2,3}_overlay.png.
    • First live object-3 SAM3+GraspNet smoke was blocked before environment creation by ModuleNotFoundError: scripts.act; fixed by explicitly inserting repo root into sys.path at the top of run_graspnet_pick.py. Re-run live smoke after this fix.
    • Live object-3 SAM3+GraspNet with default high close failed (z_gain_lift=0.000). Lowering close offset fixed the contact: command used --mask_provider sam3 --object 3 --seed 11 --tcp_z_offset 0.055 --close_z_offset -0.005 --use_task_quat --basket_center_release; result inside=True, z_gain_lift=0.211, final (0.995,-0.254,0.852), basket delta (-0.085,+0.046). Video: logs/videos/task_e_graspnet/graspnet_pick_obj3_sam3_basket_center_lowclose.mp4.
    • run_graspnet_pick.py now has a local GraspNet-only default close offset for object 3 (GRASPNET_CLOSE_Z_DEFAULTS[3] = -0.005), without changing global Task-E scripted collector constants.
    • Debug finding after user insisted the SAM-center-to-basket rule is valid: object-1 initial SAM3 failure was not a SAM capability issue. sam3_segment_image.py returned multiple masks and run_graspnet_pick.py initially picked the highest-confidence distractor near the basket/table (y≈-0.316) instead of the sugar box (y≈+0.261). Fix: sam3_segment_image.py now can write all candidates, and run_graspnet_pick.py selects/refines the candidate with the most RGB-D points inside the target object’s official spawn x/y band and table-height range.
    • After world-band candidate selection, object-1 SAM3 perception is correct (selected_candidate=1 prompt=box, mask 3327 px, GraspNet/pointcloud target about (1.020,0.261,0.917)). It partially lifts (z_gain_lift=0.120) but misses the basket at final y -0.051; stronger basket servo made grasping worse, so current blocker is grasp/contact stability plus transport slip, not segmentation.
    • Object-2 SAM3 perception is also correct (band_ratio≈0.98-0.99, mask about 2500-2700 px). With default offset it partially lifts (z_gain_lift≈0.098-0.112) but stops around y -0.08..-0.10; no-offset is worse and lower close height still misses. Current blocker is bottle grasp/contact stability and transport slip, not SAM3/SAM2.
    • Do not switch to SAM2 as the primary fix unless SAM3 latency becomes unacceptable. The failure evidence points to candidate filtering (fixed) and execution/contact tuning (still pending), while SAM3 masks are good once selected by world-band constraints.
    • User correctly identified early release: added explicit transport-end logging, short stable-hold gating, and “do not open unless stable” logic in run_graspnet_pick.py. New args include --release_z, --basket_xy_tol, --basket_hold_steps, --basket_stable_steps, --basket_recovery_steps, --auto_table_push_on_slip, and --drag_recovery_steps.
    • Important nuance: long high hold can cause slip/bounce. Object-3 high-hold test reached min_xy_err≈0.055 but failed because it waited too long and then dropped/bounced. Use a short stable hold (basket_stable_steps about 6-12, basket_xy_tol about 0.09) rather than hundreds of hold steps.
    • Verified object-3 SAM3 short-stable release success after adding TRANSPORT_END: command used --object 3 --mask_provider sam3 --close_z_offset -0.005 --release_z 0.18 --basket_xy_tol 0.09 --basket_hold_steps 40 --basket_stable_steps 6 --transport_steps 220; result inside=True, TRANSPORT_END xy_err=0.006, final (1.077,-0.313,0.852). Video: logs/videos/task_e_graspnet/graspnet_pick_obj3_sam3_closed_drag.mp4.
    • Object-1 and object-2 still fail under SAM3+GraspNet despite correct masks. At TRANSPORT_END, both have slipped/dropped before release: object-1 example final transport (0.885,-0.056,0.872), xy_err=0.313; object-2 example (0.978,0.034,0.882), xy_err=0.349. Closed-drag/table-push recovery as currently implemented does not move them, likely because Piper gripper_base low push pose is not actually contacting the object side. Next step should switch recovery to the known Task-E calibrated state-machine transport/contact primitive or redesign pusher using actual finger/link contact geometry, not just gripper_base target z.
    • Follow-up on 2026-05-21 night: user explicitly required the method to be competition-real, so scripts/graspnet_task_e/run_graspnet_pick.py was changed to use actual link7/link8 finger centers for transport/recovery, not fake gripper_base contact assumptions. Added staged transport (--staged_transport, --transport_servo_fraction) so the arm first carries smoothly and only applies basket-center object servo in the final segment.
    • Verified SAM3+GraspNet object-3 with real finger/staged transport: command used --object 3 --mask_provider sam3 --close_z_offset -0.005 --transport_steps 700 --transport_servo_fraction 0.20 --basket_servo_gain 0.75 --basket_servo_max_xy 0.18; result inside=True, final about (0.995,-0.220,0.853), video logs/videos/task_e_graspnet/graspnet_pick_obj3_sam3_staged_transport.mp4.
    • SAM3+GraspNet object-1/2 are still not robust enough for submission. Object-1 no-offset test can partially lift (z_gain_lift≈0.130) but slips near y≈-0.055; object-2 offset test can partially lift (z_gain_lift≈0.120-0.137) but slips around y≈-0.10. GraspNet native yaw/quat made object-2 worse. The blocker is real contact stability during transport, not SAM3 segmentation.

  • ACT obj321 servo training/eval, 2026-05-21:

    • Dataset exists: datasets/atec_task_e_obj321_servo_100demos/trajectory.hdf5 (~122G) and trajectory_filtered.hdf5 (~33G).
    • Training completed: run runs/act-task-e-rgb-obj321-servo-100demos-seed1, checkpoints through final.pt; training log symlink logs/task_e_obj321_servo_pipeline.latest.
    • Default submission checkpoint is already installed: demo/policy_act.pt SHA256 matches runs/act-task-e-rgb-obj321-servo-100demos-seed1/checkpoints/best_loss.pt (76bbef2c81c85fc6cc60448476ff09075ec8bc97217c2d2fb70a3253fd3e307d).
    • Evaluation with real observations: final.pt single episode scored 6.00, video logs/videos/task_e_act_eval/obj321_servo_final_1ep_20260521_realobs.mp4. best_loss.pt produced one successful 18.00 episode with video logs/videos/task_e_act_eval/obj321_servo_bestloss_3ep_20260521_realobs.mp4, but independent single-process repeats scored 6/12/12; treat current expected score as roughly 10-12 with occasional 18, not yet robust.
    • Known eval gotcha: same-process multi-episode reset can produce bogus later episodes with steps=1; use separate Python processes for trustworthy episode estimates.
    • 2026-05-21 night inference tuning: demo/solution_act.py now supports env vars ATEC_ACT_TEMPORAL_AGG=0, ATEC_ACT_PREFER_NEW=1, and ATEC_ACT_TEMPORAL_K=<float> without changing default behavior. scripts/act/eval_task_e_act.py now supports --seed and prints [BASKET] object status at the end.
    • Fixed-seed quick checks with best_loss.pt: prefer-new (ATEC_ACT_PREFER_NEW=1 ATEC_ACT_TEMPORAL_K=0.05) scored 6/15/0 on seeds 11/12/13; no-agg (ATEC_ACT_TEMPORAL_AGG=0) initially scored 6/9/15 without basket debug, but seed handling/debug rerun of seed13 scored 0 and showed all objects still outside basket. More controlled seed sweeps are needed before changing submission inference defaults.
    • Started ACT seed2 training from the existing filtered dataset, no recollection: systemd user unit atec-task-e-act-seed2-20260521225511.service, log logs/task_e_act_seed2_20260521225511.log, run name act-task-e-rgb-obj321-servo-100demos-seed2.
    • Seed2 mid-training evals: 5000.pt on fixed seed 11 scored 0.00 (all objects outside); then-current best_loss.pt scored 6.00 with only object_3 inside. 10000.pt on fixed seed 11 scored 0.00. 15000.pt on fixed seed 11 also scored 0.00; video logs/videos/task_e_act_eval/seed2_15000_seed11.mp4. 20000.pt on fixed seed 11 scored 6.00 with only object_3 inside; video logs/videos/task_e_act_eval/seed2_20000_seed11.mp4. Do not replace demo/policy_act.pt with seed2 before later checkpoints prove better. Training continued past Iter 20000, latest logged loss around 0.020.
    • 2026-05-22 01:12 CST seed2 reached 30000.pt; loss about 0.0158, and best_loss.pt was updated. Attempted fixed-seed eval logs/eval_task_e_seed2_30000_seed11.log, but it is invalid: GPU was already at ~91/98 GB because unrelated Ray/FSDP processes used ~76 GB, Isaac/Warp hit CUDA OOM and fell back to software physics. The eval process was killed before completion; do not treat this as model success or failure. Next trustworthy seed2 eval should wait for enough free CUDA memory, or after the unrelated Ray job finishes; avoid stopping seed2 because scripts/act/train_task_e.py has no resume argument.
    • 2026-05-22 01:19 CST valid CUDA eval of seed2 30000.pt on fixed seed 11 scored 0.00; all objects outside. Video: logs/videos/task_e_act_eval/seed2_30000_seed11_valid.mp4. The original seed2 service then failed with status 143 at iter ~30400, leaving 30000.pt as the last full checkpoint.
    • Added resume support to scripts/act/train_task_e.py and scripts/act/run_task_e_pipeline.sh: --resume_checkpoint, --resume_iter, and env vars RESUME_CHECKPOINT / RESUME_ITER. New checkpoints save train_iter, optimizer, scheduler, and EMA state; legacy checkpoints can resume weights with a fresh optimizer. Smoke test passed: logs/train_task_e_resume_smoke_seed2_30000.log, run runs/act-task-e-resume-smoke-seed2-30000.
    • Started resumed seed2 continuation: systemd unit atec-task-e-act-seed2-resume-20260522013429.service, log logs/task_e_act_seed2_resume_20260522013429.log, command resumes from runs/act-task-e-rgb-obj321-servo-100demos-seed2/checkpoints/30000.pt with RESUME_ITER=30000 and TOTAL_ITERS=100000.
    • Resume status 2026-05-22 01:52 CST: service successfully loaded 100 trajectories (169700 timesteps), resumed at iter 30000, and reached about Iter 31000; GPU utilization was 100%, process 722380 used about 13 GB VRAM. New 30000.pt / best_loss.pt are about 333 MB because they now include optimizer/scheduler/EMA state.
    • Seed2 resumed eval: 35000.pt exists and valid seed-11 eval scored 0.00 (logs/eval_task_e_seed2_35000_seed11_valid.log); all object_1/2/3 were outside. Do not deploy seed2 at 35k. Continue training toward 50k/100k and only evaluate later checkpoints or best_loss when GPU is not crowded.
    • Seed2 best_loss snapshot at 2026-05-22 02:22 (runs/act-task-e-rgb-obj321-servo-100demos-seed2/checkpoints/best_loss_snapshot_20260522_022223.pt, sha256 5e52307e84684a98429273102e136f0398e458ab1bd02a0dc6efc025493d4f02) scored 12.00 on seed 11 (logs/eval_task_e_seed2_bestloss_snapshot_20260522_022223_seed11.log): object_2 and object_3 inside, object_1 outside. This is better than 35000.pt but still not enough to replace XSA; evaluate later best_loss/50k across multiple seeds before deployment.
    • Resume status 2026-05-22 02:32 CST: training reached about Iter 41200; 40000.pt exists (runs/act-task-e-rgb-obj321-servo-100demos-seed2/checkpoints/40000.pt). No 40k eval yet because GPU was crowded by another Ray/FSDP job (~70 GB) plus seed2 training; wait for 50k or a clear GPU window.
    • Seed2 best_loss snapshot at 2026-05-22 02:35 (runs/act-task-e-rgb-obj321-servo-100demos-seed2/checkpoints/best_loss_snapshot_20260522_023530.pt, sha256 4048a27189b6539e9c5a699ae39eb4e2ea9496e197649d5fa5434141b5495ad7) scored only 6.00 on seed 11 (logs/eval_task_e_seed2_bestloss_snapshot_20260522_023530_seed11.log): only object_3 inside. Lower training loss did not correlate with better task score; do not deploy this snapshot.
    • Seed2 numbered checkpoint evals after resume: 45000.pt scored seed11 18.00 but seed12 0.00; 50000.pt scored seed11 15.00 and seed12 6.00. This is still too unstable to deploy over current XSA candidate. Continue training and evaluate later checkpoints; object_1 remains the main missing piece.
    • Resume status 2026-05-22 03:24 CST: 55000.pt exists and best_loss.pt updated at the same time, but no 55k eval was run because 45k/50k already proved instability. Next planned evaluation gate is 75k or final/100k unless training fails.
    • Resume status 2026-05-22 04:00 CST: training reached about Iter 64300; 60000.pt exists and best_loss.pt updated at about 03:58:59. No 60k eval; continue to 75k gate.
    • Seed2 75000.pt eval on seed 11 scored only 6.00 (logs/eval_task_e_seed2_75000_seed11.log): only object_3 inside. Do not spend more seed12/13 eval on 75k; wait for final/100k or a specific promising snapshot.
    • Seed2 mid/late checks on 2026-05-22: 85000.pt seed11 scored 0.00 (logs/eval_task_e_seed2_85000_seed11.log), while the then-current best_loss.pt around iter 87300 scored 15.00 (logs/eval_task_e_seed2_bestloss_87300_seed11.log) with object_2 and object_3 inside and object_1 outside. This reinforces that training loss and task score are weakly correlated; object_1 remains the main blocker and ACT alone is not yet stable full-score.
    • Seed2 finished 100k on 2026-05-22: final.pt SHA256 36493deefaf05694351db776435c2d53d9fe4a274c4f95febfe0fe8b36fea652 scored 0.00 on seed11 (logs/eval_task_e_seed2_final_seed11.log); latest best_loss.pt SHA256 f14c9d7c536c1d4156a8603064ec65cc593d8659147da175212fd8d7cbad2950 scored 9.00 on seed11 (logs/eval_task_e_seed2_bestloss_96900_seed11.log, only object_3 inside). Do not deploy seed2 over current XSA candidate.
    • Strategy judgment as of 2026-05-22: keep ACT/XSA as the best deployable baseline and finish seed2 to 100k for evidence, but do not assume more ACT iterations will reach stable 18. Prepare a pi0.5/OpenPI fine-tune branch using the existing HDF5 demos converted to LeRobot format; use GraspNet/SAM3 mainly as a grasp/data-generation aid unless object_1/2 contact stability is proven in ATEC Isaac with real observations/actions.
    • XSA final checkpoint had the best current evidence but is not full-score stable. Direct eval of runs/act-task-e-rgb-xsa-100demos-30k-seed11/checkpoints/final.pt scored 15/15/15 on seeds 11/12/13 (object_2 and object_3 inside, object_1 outside), with seed-11 video logs/videos/task_e_act_eval/xsa_final_seed11_recheck.mp4. The same checkpoint deployed as demo/policy_act.pt later scored 9 on seed 11 and 0 on seed 12, so there is residual nondeterminism/contact instability. Current demo/policy_act.pt SHA256 is XSA final c3eacae3f1b0ec8ccde9fdc05d680fff119cc2ca70e1f79e4ddd5610c4bbfa93; previous seed1 best was backed up as demo/policy_act.seed1_best_backup_20260522_012719.pt.

  • pi0.5 / OpenPI Task E branch, 2026-05-22:

    • OpenPI repo/env: $HOME/src/openpi-ebench-clean, Python env $HOME/envs/openpi-pi05; run with PYTHONPATH=$HOME/src/openpi-ebench-clean/src:$HOME/src/openpi-ebench-clean/packages/openpi-client/src, HF_LEROBOT_HOME=$HOME/projects/robotics_shared/datasets/lerobot, HF_HOME=$HOME/projects/robotics_shared/hf_cache.
    • Avoid: do not use uv run in the OpenPI repo; it tries to create a large local .venv and download duplicated packages.
    • Converter: scripts/pi05/convert_task_e_hdf5_to_lerobot.py maps ATEC 8D Piper qpos/action to OpenPI fixed-base 16D padded state/action.{joints,gripper} and writes 3 identical 224x224 camera streams. 224 smoke dataset atec/task_e_obj321_servo_smoke2_stride10_224 wrote 343 frames in about 70s and occupies 48M.
    • OpenPI configs added in $HOME/src/openpi-ebench-clean/src/openpi/training/config.py: pi05_atec_task_e_smoke_224 and pi05_atec_task_e_s5_224. The formal s5_224 config uses repo atec/task_e_obj321_servo_100demos_s5_224, batch size 1, 30k steps, save interval 30k to avoid 42G checkpoint spam.
    • Important fix: ATEC HDF5 actions are env actions (joint_target-default)/0.5, not absolute joint targets, so all ATEC OpenPI configs must keep extra_delta_transform=False. Using the OpenPI delta transform here would subtract state again and train on the wrong target.
    • Smoke verification passed: pi05_atec_task_e_smoke_224 norm stats wrote under $HOME/projects/robotics_shared/checkpoints/openpi/atec_runs/assets/pi05_atec_task_e_smoke_224/...; 10-step train saved successfully once, but the temporary 42G checkpoint was deleted to preserve disk space. Log: logs/pi05_atec_task_e_smoke_224_10step_20260522.log.
    • Inference bridge added: demo/solution_pi05.py talks to an OpenPI websocket server and maps ATEC observation/action to/from the 16D fixed-base format. Eval entry point: scripts/pi05/eval_task_e_pi05.py. Default ATEC_PI05_ACTION_REPEAT=5 matches the current stride-5/fps-10 training dataset. After a checkpoint exists, use scripts/pi05/run_pi05_eval_checkpoint.sh <checkpoint_step_dir> to start scripts/serve_policy.py policy:checkpoint ..., run seeds 11/12/13, and record a seed-11 video.
    • Eval summary helper: scripts/pi05/summarize_pi05_eval.py <logs/pi05_eval_*> parses seed logs for score/object basket status and is called automatically at the end of run_pi05_eval_checkpoint.sh.
    • Formal s5 conversion service started: unit from logs/pi05_convert_task_e_s5_224.latest_unit, log from logs/pi05_convert_task_e_s5_224.latest; auto continuation service logs/pi05_s5_train_after_convert.latest_unit waits for conversion, then computes norm stats and trains pi05_atec_task_e_s5_224. Eval watcher logs/pi05_100demos_eval_after_train.latest_unit waits for training, then calls CONFIG=pi05_atec_task_e_s5_224 PORT=8017 scripts/pi05/run_pi05_eval_checkpoint.sh <latest_ckpt>.
    • Early-signal branch: pi05_atec_task_e_20demos_s5_224 config uses repo atec/task_e_obj321_servo_20demos_s5_224, 2k steps, batch size 1, no delta transform. Conversion service/log are in logs/pi05_convert_task_e_s5_224_20demos.latest_unit and .latest; auto continuation service logs/pi05_20demos_train_after_convert.latest_unit waits, computes norm stats, then trains. Eval watcher logs/pi05_20demos_eval_after_train.latest_unit waits for training, then calls CONFIG=pi05_atec_task_e_20demos_s5_224 PORT=8016 scripts/pi05/run_pi05_eval_checkpoint.sh <latest_ckpt>. Use this for quick pi0.5 viability checks before the 100-demo run finishes.
    • Note: the first 20-demo training run started from the old script name atec_task_e_pi05_20demos_s5_224_10k_20260522070624, but the loaded config was already patched to num_train_steps=2000; the name is cosmetic. The script was patched afterward to use _2k_ in future run names.
    • 20-demo result: checkpoint .../pi05_atec_task_e_20demos_s5_224/atec_task_e_pi05_20demos_s5_224_10k_20260522070624/1999 scored 0/0/0 on seeds 11/12/13. Logs: logs/pi05_eval_20260522074137; video: logs/videos/task_e_pi05_eval/pi05_pi05_atec_task_e_20demos_s5_224_seed11_20260522074137.mp4. The 42G 20-demo checkpoint directory was deleted after evaluation to preserve disk; dataset and logs were kept.
    • 100-demo formal run status, 2026-05-22 08:56 CST: conversion and norm stats are complete; training unit atec-pi05-s5-train-after-convert-20260522065716.service is actively running scripts/train.py pi05_atec_task_e_s5_224 --exp-name atec_task_e_pi05_s5_224_30k_20260522065716 --overwrite. Latest observed progress was about 731/30000 steps at roughly 3.3-3.6 it/s, GPU util 100%, ~80 GB VRAM. Log: logs/pi05_train_task_e_s5_224_20260522065716.log.
    • 100-demo checkpoint/eval gate: config pi05_atec_task_e_s5_224 saves only at 30000 steps (save_interval=30000), expected final dir .../checkpoints/pi05_atec_task_e_s5_224/atec_task_e_pi05_s5_224_30k_20260522065716/30000. Eval watcher unit atec-pi05-100demos-eval-after-train-20260522070900.service waits for the training unit, then runs CONFIG=pi05_atec_task_e_s5_224 PORT=8017 scripts/pi05/run_pi05_eval_checkpoint.sh <checkpoint> for seeds 11/12/13 and records a seed-11 video.
    • Deployment decision gate: current deployed ACT/XSA baseline is still demo/policy_act.pt SHA256 c3eacae3f1b0ec8ccde9fdc05d680fff119cc2ca70e1f79e4ddd5610c4bbfa93. Do not replace it with pi0.5 unless the 100-demo pi0.5 checkpoint has real ATEC Isaac eval evidence on seeds 11/12/13 and is clearly better. Required completion artifacts for this branch are: final checkpoint dir, logs/pi05_eval_* with seed11/12/13 [RESULT] and [BASKET], and logs/videos/task_e_pi05_eval/pi05_pi05_atec_task_e_s5_224_seed11_*.mp4.
    • Post-eval comparison helper: scripts/pi05/compare_pi05_vs_act_baseline.py <logs/pi05_eval_*> --repo <repo> compares pi0.5 seed logs with current ACT/XSA evidence (eval_task_e_xsa_final_seed11/12/13_recheck.log plus deployed seed11 recheck) and prints HOLD_ACT_XSA unless pi0.5 is clearly better. scripts/pi05/run_pi05_eval_checkpoint.sh calls this automatically after summarize_pi05_eval.py. It was sanity-checked on the 20-demo failed eval and correctly reported HOLD_ACT_XSA.
    • 100-demo result, 2026-05-22 11:27 CST: checkpoint .../pi05_atec_task_e_s5_224/atec_task_e_pi05_s5_224_30k_20260522065716/29999 (about 42G) evaluated at logs/pi05_eval_20260522112701 and scored seed11 0.00, seed12 0.00, seed13 6.00 (mean 2.00). Seed11 video: logs/videos/task_e_pi05_eval/pi05_pi05_atec_task_e_s5_224_seed11_20260522112701.mp4. The comparison helper printed HOLD_ACT_XSA; do not deploy pi0.5 over current XSA.
    • Monitoring: the old heartbeat automation atec-task-e-seed2-resume-monitor was deleted after the 100-demo pi0.5 train/eval completed, because the monitored branch is done. Recreate a new monitor only if starting another long run.

  • Known-good environment:

    • Python: $HOME/envs/genmanip-isaac5-py311/bin/python.
    • Required env prefix: OMNI_KIT_ACCEPT_EULA=YES PYTHONUNBUFFERED=1.
    • Required PYTHONPATH: IsaacLab source dirs under $HOME/envs/genmanip-isaac5-py311/lib/python3.11/site-packages/isaaclab/source/*, plus source/atec_rl_lab and scripts/act from this repo.

  • Current ACT baseline:

    • Object 3 only works with dataset datasets/atec_task_e, run runs/act-task-e-rgb-100demos-seed1, observed eval score about 6.
    • Baseline eval video: logs/videos/task_e_act_eval/current_single_object_6pt.mp4.

  • Scripted oracle status:

    • Official/local Task E collector comment said object 3 works and object 1/2 need distance modifications.
    • Piper gripper joint limits observed on 2026-05-20: joint7 is [0.0, 0.035], joint8 is [-0.035, 0.0]. Use legal close target [0.0, 0.0], not [-0.015, 0.015].
    • Object 2 had a previously observed basket candidate: offset (dx=0.006, dy=0.046), z=0.045, carry_z=TABLE_TOP_Z+0.075, place_z=TABLE_TOP_Z+0.075, target_y=-0.24, yaw=-pi/2, keep gripper closed in transport. Treat this as not yet robust and likely push/slide rather than true lift: on 2026-05-20 a trace rerun showed the earlier search used candidate-index-dependent RNG, so success could be placement-specific.
    • On 2026-05-20 13:47, object 2 candidate search first 20 all failed under fixed trial seed. Best was cand 011 (dx=0.030, dy=0.026, z=0.045, carry_z=TABLE_TOP_Z+0.075, target_y=-0.24, yaw=+pi/2), final (1.224,-0.009,1.213), reward_lifted=True, outside by dy=+0.291. Object 2 is not ready for demo collection/training.
    • Object 1 is unresolved as of 2026-05-20. Prior search candidates mostly pushed/slid the object; final basket failure alone was too weak to prove a real grasp. Add/keep per-stage diagnostics before trusting any object 1 candidate.

  • Important local changes:

    • scripts/act/eval_task_e_act.py evaluates ACT and can record video.
    • demo/solution_act.py supports ATEC_ACT_POLICY_PATH and reset_episode().
    • scripts/act/cli_args.py adds --max_attempts; scripts/act/collect_demos_task_e.py uses it and prints object basket status.
    • scripts/act/task_e/collector.py uses _BASKET_MAX_Z = TABLE_TOP_Z + 0.15, has basket_status_lines, and can return numeric trace diagnostics when trace=True including object z-gain, gripper_base distance, and link7/link8 finger-body center/gap metrics.
    • scripts/act/search_task_e_grasps.py is the per-object candidate bank/search entry point.
    • scripts/act/search_task_e_grasps.py should keep trial seeds independent of candidate index so candidates are compared on the same object placements.
    • scripts/act/search_task_e_grasps.py prints [BEST] even when no candidate succeeds, using basket distance plus z-gain as a debugging score.
    • scripts/act/search_task_e_grasps.py supports --start_candidate to resume long searches without re-testing earlier candidates.
    • scripts/act/search_task_e_grasps.py now supports per-candidate push_x/push_y/push_z plus close/lift/transport/place/open step overrides. Use this to test trajectory timing without editing global config for each trial.
    • scripts/act/task_e/state_machine.py now has object-servo transport/place/open support for object 1/2: OBJ_SERVO_TO_BASKET_STATES, OBJ_SERVO_XY_GAINS, and OBJ_SERVO_MAX_XY in scripts/act/task_e/config.py. The TRANSPORT correction is ramped by state progress to avoid yanking objects out of the gripper.
    • scripts/act/collect_demos_task_e.py now disables default cameras and cfg.observations.image when need_camera=False; this fixes headless collection without --enable_cameras.

  • Latest object 1 trace:

    • On 2026-05-20 13:19, foreground search candidates 1-12 all failed: lifted=False, reward_lifted=False, z_gain about 0.000-0.003m; best candidate was 004 but still final y=0.210, far outside basket. Next search should resume at candidate 13 to test push/sweep fallback candidates.
    • On 2026-05-20 13:24, foreground search candidates 13-24 tested push/sweep fallback. No basket success yet, but contact is real: candidate 019 (dx=0.040, dy=0.140, z=-0.020, carry_z=TABLE_TOP_Z+0.035, target_y=-0.240, yaw=0, push_y=-0.18) reached lifted=True, reward_lifted=True, z_gain=0.388, final pos about (1.276, -0.072, 1.260). It overshoots upward/outside basket, so next search should refine around push_y -0.14..-0.18, lower carry/place behavior, and possibly static place release.
    • On 2026-05-20 13:36, focused candidates 25-34 around candidate 019 all failed. Best was cand 034: dx=0.040, dy=0.140, z=-0.020, carry_z=TABLE_TOP_Z+0.035, target_x=-0.17, target_y=0.0, push_y=-0.18, final (1.280,-0.064,1.257), reward_lifted=True, still outside by dy=+0.236.
    • On 2026-05-20 13:42, timing candidates with longer/lower TRANSPORT and PLACE/OPEN overrides all failed. Best was cand 002 with transport_steps=900, final (1.280,-0.042,1.257), outside by dy=+0.258. This proves simply extending the push/sweep duration does not solve object 1; the contact geometry slips/throws the box before basket placement.
    • On 2026-05-20 14:15, object-servo ramp still failed object 1 first 8 candidates. Best was cand 003 (push_y=-0.14, transport_steps=1200), final (1.294,-0.020,1.270), reward_lifted=True, outside by dx=+0.214, dy=+0.280. Object 1 still needs a different contact/grasp primitive, not more global servo.

  • Latest object 2 trace:

    • On 2026-05-20 14:14, object-servo ramp made fixed-seed search candidate 001 succeed for object 2: final (1.050,-0.272,0.865), inside=True, z_gain=0.102, candidate {dx=0.006, dy=0.046, z=0.045, carry_z=TABLE_TOP_Z+0.075, target_y=-0.24, yaw=-pi/2}.
    • On 2026-05-20 14:20, random only_success collection for object 2 still failed 0/4 attempts: final y errors remained about +0.206..+0.310, one early termination. So object 2 is promising but not robust enough for dataset/training yet.
    • On 2026-05-20 14:26, adding PLACE hold-until-near-basket still failed random object 2 collection 0/4: final y errors stayed about +0.206..+0.334.
    • On 2026-05-20 14:28, stronger object 2 servo (gain=1.15, max_xy=0.36, TRANSPORT=480) also failed 0/4 and worsened x drift; reverted object 2 to the softer fixed-seed-success servo values (gain=0.85, max_xy=0.24, no transport override). Next object 2 work should focus on grasp/contact robustness before transport, not stronger basket servo.
    • On 2026-05-20 14:41, object 2 multi-seed candidate validation (--trials 3 --max_candidates 12) found 0/12 robust candidates. The previous fixed-seed-success candidate failed immediately under this stricter check (result=0/3, final y error about +0.262). Best was yaw 0.0 with z_gain=0.355 but final (1.197,-0.001,1.277), outside by dy=+0.299. Treat object 2 as contact/grasp unstable, not transport-servo-limited.

  • Current debugging judgment:

    • Do not integrate GraspNet first. For this small three-object simulated task, a reproducible scripted candidate bank with explicit lift/grasp diagnostics is faster and easier to pass code-review reproduction.
    • If object 1 remains blocked, collect/train object 2+3 first for a practical score bump, then return to object 1. Next object 1 work should change the motion primitive/contact mode, not merely sweep more static offsets or longer transport times.

  • Useful verification commands:

    • Single object 1 debug video: python scripts/act/collect_demos_task_e.py --pick_objects 1 --num_demos 1 --max_attempts 3 --headless --enable_cameras --only_success --save_video --video_dir logs/videos/task_e_oracle_obj1_fixed --output_dir logs/debug_task_e_obj1_fixed.
    • Object 2+3 sanity check: python scripts/act/collect_demos_task_e.py --pick_objects 2 3 --num_demos 1 --max_attempts 3 --headless --enable_cameras --only_success --save_video --video_dir logs/videos/task_e_oracle_obj23_fixed --output_dir logs/debug_task_e_obj23_fixed.

  • External reference checked:

    • https://github.com/fiveages-sim/lerobot_ros2 is a public ROS2/LeRobot integration repo. It exposes generic ROS2 robot/camera wrappers and examples with single_arm.pick/place YAML skills, but it does not directly provide an ATEC Piper-compatible grasp planner. Its robot_action_composer submodule is private/inaccessible by HTTPS clone in this environment.