Supersedes the older “object 1/2 unresolved” notes below; keep them only as historical debugging context.
L0 Task E three-object scripted data generation now works well enough to collect demos. Verified all-object order is 3 2 1 with --only_success --trace --abort_failed_lift.
Root cause/fix: data generation was wrong, not ACT training. Object 1 must keep the finger-centre servo target on obj_pos + OBJ_GRASP_CENTER_OFFSETS[1]; object 2 must track the USD/root object position, not the offset center. Both need closed-loop finger-centre servo during REACH/CLOSE/LIFT/TRANSPORT/PLACE.
Stable constants in scripts/act/task_e/config.py: object 1 offset (0.025,0,0), grasp z 0.072, close z 0.020, transport/place steps 2200/300; object 2 offset (0.060,0,0), grasp z 0.135, transport/place steps 2200/300; default PICK_OBJECTS="3 2 1".
scripts/act/task_e/collector.py has the important per-object servo distinction: object 1 servo/trace target uses obj_pos + OBJ_GRASP_CENTER_OFFSETS; object 2/3 use obj_pos. Do not “simplify” this back to one formula.
scripts/act/train_task_e.py uses lr_drop = max(int(2/3 * total_iters), 1) so tiny smoke tests no longer fail with StepLR(step_size=0).
CAP-X / RoboTwin lesson: use contact/TCP calibrated grasp candidates and closed-loop place logic first. GraspNet/ContactGraspNet can be added later as a candidate generator, but it is not required to fix this current oracle data bug.
Active long run started via systemd user unit because plain background/nohup could be killed with the Codex command: unit atec-task-e-obj321-servo-20260520211650.service, log logs/task_e_obj321_servo_pipeline_20260520_211650.log, latest symlink logs/task_e_obj321_servo_pipeline.latest.
As of 2026-05-20 21:22 CST the long run is active, Demo 1/100 succeeded and wrote datasets/atec_task_e_obj321_servo_100demos/trajectory.hdf5 (~1.3 GB), then proceeded to Demo 2/100.
Latest train/eval status, 2026-05-21 14:55 CST:
atec-task-e-obj321-servo-20260520211650.service completed normally; raw dataset is datasets/atec_task_e_obj321_servo_100demos/trajectory.hdf5 (~130 GB), filtered dataset is trajectory_filtered.hdf5 (~35 GB), run is runs/act-task-e-rgb-obj321-servo-100demos-seed1.
Checkpoints include best_loss.pt and final.pt; training reached 100000 iters, final logged loss around 0.0100.
First ACT eval of best_loss.pt scored 0/3 episodes; video logs/videos/task_e_act_eval/obj321_best_loss_100demos_20260521_145104.mp4 shows the arm staying near the basket/home pose instead of starting the object grasp.
Root cause found in demo/solution_act.py: the home stage treated env action as PD/velocity, but Task E action is (joint_target - default_joint_pos) / 0.5. Fix is to command the static teleop-home target action until stable, then switch to ACT. Re-run eval after this fix before judging model quality.
After the home fix, best_loss.pt eval improved: 1-episode score 9.00 with video logs/videos/task_e_act_eval/obj321_best_loss_homefix_1ep_20260521_145529.mp4; same-process 3-episode eval had valid first episode score 18.00 and video logs/videos/task_e_act_eval/obj321_best_loss_homefix_3ep_20260521_145652.mp4, but subsequent episodes reset to done=True at step 1, so do not treat that mean score as independent.
final.pt single-episode eval scored 6.00, so deploy best_loss.pt.
Deployment: previous demo/policy_act.pt backed up as demo/policy_act.prev_20260521_145951.pt; current demo/policy_act.pt is copied from runs/act-task-e-rgb-obj321-servo-100demos-seed1/checkpoints/best_loss.pt and sha256 matches 76bbef2c81c85fc6cc60448476ff09075ec8bc97217c2d2fb70a3253fd3e307d.
XSA ACT experiment, 2026-05-21:
Local XSA package source: $HOME/Documents/01Proj/13atec/xsa_update_20260520/lerobot_policy_act_xsa.zip; it is a LeRobot act_xsa plugin with 12-dim example config and is not directly runnable for ATEC Task E’s 8-dim Piper setup.
Minimal XSA adaptation added behind --use_xsa: source/atec_rl_lab/atec_rl_lab/train/act/act/detr/transformer.py and demo/act/detr/transformer.py support XSATransformerEncoderLayer / XSATransformerDecoderLayer; default ACT remains unchanged.
scripts/act/train_task_e.py now saves model_args in checkpoints; demo/solution_act.py reads model_args including use_xsa, so ordinary ACT and XSA checkpoints can both load.
Smoke test passed: runs/act-task-e-rgb-xsa-smoke-10demos-seed7/checkpoints/final.pt loads as XSATransformerEncoderLayer.
Active XSA comparison run: systemd user unit atec-task-e-xsa-30k-20260521165009.service, log logs/task_e_xsa_30k_20260521_165009.log, run name act-task-e-rgb-xsa-100demos-30k-seed11, command uses --use_xsa --total_iters 30000 --batch_size 256 --num_demos 100.
Mid-training evals are noisy but not yet encouraging: 19k-ish snapshot scored 9/18 with video logs/videos/task_e_act_eval/xsa_mid_best_1ep_20260521_174124.mp4; later current-best snapshot at 20260521_175038 scored 0/18 with video logs/videos/task_e_act_eval/xsa_current_best_1ep_20260521_175038.mp4. Do not infer ranking from training loss alone; run independent single-episode processes because same-process multi-episode reset can be invalid after success.
GraspNet / TunTunClaw integration, 2026-05-21:
User reference cloned into third_party/tuntunclaw/ from Datawhale/OpenClaw home assistant material. Official GraspNet checkpoint is third_party/tuntunclaw/temp/logs/log_rs/checkpoint-rs.tar.
Local bridge files: scripts/graspnet_task_e/tuntun_adapter.py and smoke runner scripts/graspnet_task_e/run_graspnet_pick.py.
Dependencies were made to work in $HOME/envs/genmanip-isaac5-py311/bin/python: open3d==0.19.0, local graspnetAPI, transforms3d==0.4.2, and the GraspNet pointnet2 CUDA extension built for Blackwell with TORCH_CUDA_ARCH_LIST=12.0.
Important lesson: raw GraspNet point/orientation alone did not lift object 1. The first successful object-1 smoke used GraspNet/TunTunClaw for candidate generation, but executed with Task-E calibrated controls: robust mask upper-surface median as execution center, object-1 OBJ_GRASP_CENTER_OFFSETS added, --close_z_offset -0.020, finger-center servo, keep grasp quaternion through transport/place/open, and long transport.
Verified result for that command: inside=True, z_gain_lift=0.164, final object_1 position about (1.043,-0.247,0.934), basket delta (-0.037,+0.053). Video: logs/videos/task_e_graspnet/graspnet_pick_obj1_seed11_keepquat_slow.mp4.
Object 3 GraspNet smoke also succeeded with refined AABB oracle mask and lower close height: command uses --object 3 --tcp_z_offset 0.055 --close_z_offset 0.015 --use_task_quat --place_x_offset 0.04 --place_y_offset 0.05; result inside=True, z_gain_lift=0.188, video logs/videos/task_e_graspnet/graspnet_pick_obj3_seed11_lowclose.mp4.
Object 2 GraspNet smoke is not solved yet. Best attempts can move/lift slightly (z_gain_lift≈0.07-0.10) but final y remains outside basket; simple --post_push moved it only to about y=-0.056. Keep using the existing Task-E calibrated state-machine/ACT path for object 2 until a better GraspNet contact primitive is found.
Added legal-ish RGB-D y-band mask provider rgbd_band_object_mask() and runner flag --mask_provider band; it avoids sim object state by using camera intrinsics, depth, table height, and official spawn y-bands. Current band-mask object-3 test did not lift yet, so the GraspNet success videos above still rely on oracle_object_mask()/AABB debug masks. Do not use oracle masks in final submission without replacement.
Three-object stable oracle/state-machine verification after GraspNet changes: scripts/act/collect_demos_task_e.py --pick_objects 3 2 1 --num_demos 1 --max_attempts 3 --only_success --trace --abort_failed_lift --save_video ... succeeded on attempt 2, with all object_1/2/3 basket_inside=True. Video: logs/videos/task_e_obj321_current_verify/demo_0000.mp4.
Local SAM3 checkpoint exists at $HOME/Documents/01Proj/sam3d_gs/submodule/Prompt-Inpaint/checkpoints/sam3.pt; SAM3 is installed in $HOME/Documents/01Proj/sam3d_gs/.venv/bin/python, not in the Isaac env. Keep SAM3 isolated as a subprocess rather than installing it into $HOME/envs/genmanip-isaac5-py311.
Added scripts/graspnet_task_e/sam3_segment_image.py: standalone RGB image -> SAM3 binary mask. It uses local Prompt-Inpaint SAM3Predictor and writes .npy mask plus optional JSON metadata.
scripts/graspnet_task_e/run_graspnet_pick.py now supports --mask_provider sam3, --sam3_prompt, --sam3_threshold, and --sam3_python. It also defaults to --basket_center_release: during transport/place/open it servo-corrects the EE target by current object center error so the object center is driven to (BASKET_CENTER_X, BASKET_CENTER_Y) before opening.
SAM3 smoke on saved frames passed for all three prompts: object 1 best prompt box score 0.863, mask 31206 px; object 2 best prompt yellow bottle score 0.961, mask 2509 px; object 3 best prompt banana score 0.953, mask 1311 px. Overlays are logs/graspnet_task_e/sam3_smoke_obj{1,2,3}_overlay.png.
First live object-3 SAM3+GraspNet smoke was blocked before environment creation by ModuleNotFoundError: scripts.act; fixed by explicitly inserting repo root into sys.path at the top of run_graspnet_pick.py. Re-run live smoke after this fix.
Live object-3 SAM3+GraspNet with default high close failed (z_gain_lift=0.000). Lowering close offset fixed the contact: command used --mask_provider sam3 --object 3 --seed 11 --tcp_z_offset 0.055 --close_z_offset -0.005 --use_task_quat --basket_center_release; result inside=True, z_gain_lift=0.211, final (0.995,-0.254,0.852), basket delta (-0.085,+0.046). Video: logs/videos/task_e_graspnet/graspnet_pick_obj3_sam3_basket_center_lowclose.mp4.
run_graspnet_pick.py now has a local GraspNet-only default close offset for object 3 (GRASPNET_CLOSE_Z_DEFAULTS[3] = -0.005), without changing global Task-E scripted collector constants.
Debug finding after user insisted the SAM-center-to-basket rule is valid: object-1 initial SAM3 failure was not a SAM capability issue. sam3_segment_image.py returned multiple masks and run_graspnet_pick.py initially picked the highest-confidence distractor near the basket/table (y≈-0.316) instead of the sugar box (y≈+0.261). Fix: sam3_segment_image.py now can write all candidates, and run_graspnet_pick.py selects/refines the candidate with the most RGB-D points inside the target object’s official spawn x/y band and table-height range.
After world-band candidate selection, object-1 SAM3 perception is correct (selected_candidate=1 prompt=box, mask 3327 px, GraspNet/pointcloud target about (1.020,0.261,0.917)). It partially lifts (z_gain_lift=0.120) but misses the basket at final y -0.051; stronger basket servo made grasping worse, so current blocker is grasp/contact stability plus transport slip, not segmentation.
Object-2 SAM3 perception is also correct (band_ratio≈0.98-0.99, mask about 2500-2700 px). With default offset it partially lifts (z_gain_lift≈0.098-0.112) but stops around y -0.08..-0.10; no-offset is worse and lower close height still misses. Current blocker is bottle grasp/contact stability and transport slip, not SAM3/SAM2.
Do not switch to SAM2 as the primary fix unless SAM3 latency becomes unacceptable. The failure evidence points to candidate filtering (fixed) and execution/contact tuning (still pending), while SAM3 masks are good once selected by world-band constraints.
User correctly identified early release: added explicit transport-end logging, short stable-hold gating, and “do not open unless stable” logic in run_graspnet_pick.py. New args include --release_z, --basket_xy_tol, --basket_hold_steps, --basket_stable_steps, --basket_recovery_steps, --auto_table_push_on_slip, and --drag_recovery_steps.
Important nuance: long high hold can cause slip/bounce. Object-3 high-hold test reached min_xy_err≈0.055 but failed because it waited too long and then dropped/bounced. Use a short stable hold (basket_stable_steps about 6-12, basket_xy_tol about 0.09) rather than hundreds of hold steps.
Verified object-3 SAM3 short-stable release success after adding TRANSPORT_END: command used --object 3 --mask_provider sam3 --close_z_offset -0.005 --release_z 0.18 --basket_xy_tol 0.09 --basket_hold_steps 40 --basket_stable_steps 6 --transport_steps 220; result inside=True, TRANSPORT_END xy_err=0.006, final (1.077,-0.313,0.852). Video: logs/videos/task_e_graspnet/graspnet_pick_obj3_sam3_closed_drag.mp4.
Object-1 and object-2 still fail under SAM3+GraspNet despite correct masks. At TRANSPORT_END, both have slipped/dropped before release: object-1 example final transport (0.885,-0.056,0.872), xy_err=0.313; object-2 example (0.978,0.034,0.882), xy_err=0.349. Closed-drag/table-push recovery as currently implemented does not move them, likely because Piper gripper_base low push pose is not actually contacting the object side. Next step should switch recovery to the known Task-E calibrated state-machine transport/contact primitive or redesign pusher using actual finger/link contact geometry, not just gripper_base target z.
Follow-up on 2026-05-21 night: user explicitly required the method to be competition-real, so scripts/graspnet_task_e/run_graspnet_pick.py was changed to use actual link7/link8 finger centers for transport/recovery, not fake gripper_base contact assumptions. Added staged transport (--staged_transport, --transport_servo_fraction) so the arm first carries smoothly and only applies basket-center object servo in the final segment.
Verified SAM3+GraspNet object-3 with real finger/staged transport: command used --object 3 --mask_provider sam3 --close_z_offset -0.005 --transport_steps 700 --transport_servo_fraction 0.20 --basket_servo_gain 0.75 --basket_servo_max_xy 0.18; result inside=True, final about (0.995,-0.220,0.853), video logs/videos/task_e_graspnet/graspnet_pick_obj3_sam3_staged_transport.mp4.
SAM3+GraspNet object-1/2 are still not robust enough for submission. Object-1 no-offset test can partially lift (z_gain_lift≈0.130) but slips near y≈-0.055; object-2 offset test can partially lift (z_gain_lift≈0.120-0.137) but slips around y≈-0.10. GraspNet native yaw/quat made object-2 worse. The blocker is real contact stability during transport, not SAM3 segmentation.
ACT obj321 servo training/eval, 2026-05-21:
Dataset exists: datasets/atec_task_e_obj321_servo_100demos/trajectory.hdf5 (~122G) and trajectory_filtered.hdf5 (~33G).
Training completed: run runs/act-task-e-rgb-obj321-servo-100demos-seed1, checkpoints through final.pt; training log symlink logs/task_e_obj321_servo_pipeline.latest.
Evaluation with real observations: final.pt single episode scored 6.00, video logs/videos/task_e_act_eval/obj321_servo_final_1ep_20260521_realobs.mp4. best_loss.pt produced one successful 18.00 episode with video logs/videos/task_e_act_eval/obj321_servo_bestloss_3ep_20260521_realobs.mp4, but independent single-process repeats scored 6/12/12; treat current expected score as roughly 10-12 with occasional 18, not yet robust.
Known eval gotcha: same-process multi-episode reset can produce bogus later episodes with steps=1; use separate Python processes for trustworthy episode estimates.
2026-05-21 night inference tuning: demo/solution_act.py now supports env vars ATEC_ACT_TEMPORAL_AGG=0, ATEC_ACT_PREFER_NEW=1, and ATEC_ACT_TEMPORAL_K=<float> without changing default behavior. scripts/act/eval_task_e_act.py now supports --seed and prints [BASKET] object status at the end.
Fixed-seed quick checks with best_loss.pt: prefer-new (ATEC_ACT_PREFER_NEW=1 ATEC_ACT_TEMPORAL_K=0.05) scored 6/15/0 on seeds 11/12/13; no-agg (ATEC_ACT_TEMPORAL_AGG=0) initially scored 6/9/15 without basket debug, but seed handling/debug rerun of seed13 scored 0 and showed all objects still outside basket. More controlled seed sweeps are needed before changing submission inference defaults.
Started ACT seed2 training from the existing filtered dataset, no recollection: systemd user unit atec-task-e-act-seed2-20260521225511.service, log logs/task_e_act_seed2_20260521225511.log, run name act-task-e-rgb-obj321-servo-100demos-seed2.
Seed2 mid-training evals: 5000.pt on fixed seed 11 scored 0.00 (all objects outside); then-current best_loss.pt scored 6.00 with only object_3 inside. 10000.pt on fixed seed 11 scored 0.00. 15000.pt on fixed seed 11 also scored 0.00; video logs/videos/task_e_act_eval/seed2_15000_seed11.mp4. 20000.pt on fixed seed 11 scored 6.00 with only object_3 inside; video logs/videos/task_e_act_eval/seed2_20000_seed11.mp4. Do not replace demo/policy_act.pt with seed2 before later checkpoints prove better. Training continued past Iter 20000, latest logged loss around 0.020.
2026-05-22 01:12 CST seed2 reached 30000.pt; loss about 0.0158, and best_loss.pt was updated. Attempted fixed-seed eval logs/eval_task_e_seed2_30000_seed11.log, but it is invalid: GPU was already at ~91/98 GB because unrelated Ray/FSDP processes used ~76 GB, Isaac/Warp hit CUDA OOM and fell back to software physics. The eval process was killed before completion; do not treat this as model success or failure. Next trustworthy seed2 eval should wait for enough free CUDA memory, or after the unrelated Ray job finishes; avoid stopping seed2 because scripts/act/train_task_e.py has no resume argument.
2026-05-22 01:19 CST valid CUDA eval of seed2 30000.pt on fixed seed 11 scored 0.00; all objects outside. Video: logs/videos/task_e_act_eval/seed2_30000_seed11_valid.mp4. The original seed2 service then failed with status 143 at iter ~30400, leaving 30000.pt as the last full checkpoint.
Added resume support to scripts/act/train_task_e.py and scripts/act/run_task_e_pipeline.sh: --resume_checkpoint, --resume_iter, and env vars RESUME_CHECKPOINT / RESUME_ITER. New checkpoints save train_iter, optimizer, scheduler, and EMA state; legacy checkpoints can resume weights with a fresh optimizer. Smoke test passed: logs/train_task_e_resume_smoke_seed2_30000.log, run runs/act-task-e-resume-smoke-seed2-30000.
Started resumed seed2 continuation: systemd unit atec-task-e-act-seed2-resume-20260522013429.service, log logs/task_e_act_seed2_resume_20260522013429.log, command resumes from runs/act-task-e-rgb-obj321-servo-100demos-seed2/checkpoints/30000.pt with RESUME_ITER=30000 and TOTAL_ITERS=100000.
Resume status 2026-05-22 01:52 CST: service successfully loaded 100 trajectories (169700 timesteps), resumed at iter 30000, and reached about Iter 31000; GPU utilization was 100%, process 722380 used about 13 GB VRAM. New 30000.pt / best_loss.pt are about 333 MB because they now include optimizer/scheduler/EMA state.
Seed2 resumed eval: 35000.pt exists and valid seed-11 eval scored 0.00 (logs/eval_task_e_seed2_35000_seed11_valid.log); all object_1/2/3 were outside. Do not deploy seed2 at 35k. Continue training toward 50k/100k and only evaluate later checkpoints or best_loss when GPU is not crowded.
Seed2 best_loss snapshot at 2026-05-22 02:22 (runs/act-task-e-rgb-obj321-servo-100demos-seed2/checkpoints/best_loss_snapshot_20260522_022223.pt, sha256 5e52307e84684a98429273102e136f0398e458ab1bd02a0dc6efc025493d4f02) scored 12.00 on seed 11 (logs/eval_task_e_seed2_bestloss_snapshot_20260522_022223_seed11.log): object_2 and object_3 inside, object_1 outside. This is better than 35000.pt but still not enough to replace XSA; evaluate later best_loss/50k across multiple seeds before deployment.
Resume status 2026-05-22 02:32 CST: training reached about Iter 41200; 40000.pt exists (runs/act-task-e-rgb-obj321-servo-100demos-seed2/checkpoints/40000.pt). No 40k eval yet because GPU was crowded by another Ray/FSDP job (~70 GB) plus seed2 training; wait for 50k or a clear GPU window.
Seed2 best_loss snapshot at 2026-05-22 02:35 (runs/act-task-e-rgb-obj321-servo-100demos-seed2/checkpoints/best_loss_snapshot_20260522_023530.pt, sha256 4048a27189b6539e9c5a699ae39eb4e2ea9496e197649d5fa5434141b5495ad7) scored only 6.00 on seed 11 (logs/eval_task_e_seed2_bestloss_snapshot_20260522_023530_seed11.log): only object_3 inside. Lower training loss did not correlate with better task score; do not deploy this snapshot.
Seed2 numbered checkpoint evals after resume: 45000.pt scored seed11 18.00 but seed12 0.00; 50000.pt scored seed11 15.00 and seed12 6.00. This is still too unstable to deploy over current XSA candidate. Continue training and evaluate later checkpoints; object_1 remains the main missing piece.
Resume status 2026-05-22 03:24 CST: 55000.pt exists and best_loss.pt updated at the same time, but no 55k eval was run because 45k/50k already proved instability. Next planned evaluation gate is 75k or final/100k unless training fails.
Resume status 2026-05-22 04:00 CST: training reached about Iter 64300; 60000.pt exists and best_loss.pt updated at about 03:58:59. No 60k eval; continue to 75k gate.
Seed2 75000.pt eval on seed 11 scored only 6.00 (logs/eval_task_e_seed2_75000_seed11.log): only object_3 inside. Do not spend more seed12/13 eval on 75k; wait for final/100k or a specific promising snapshot.
Seed2 mid/late checks on 2026-05-22: 85000.pt seed11 scored 0.00 (logs/eval_task_e_seed2_85000_seed11.log), while the then-current best_loss.pt around iter 87300 scored 15.00 (logs/eval_task_e_seed2_bestloss_87300_seed11.log) with object_2 and object_3 inside and object_1 outside. This reinforces that training loss and task score are weakly correlated; object_1 remains the main blocker and ACT alone is not yet stable full-score.
Seed2 finished 100k on 2026-05-22: final.pt SHA256 36493deefaf05694351db776435c2d53d9fe4a274c4f95febfe0fe8b36fea652 scored 0.00 on seed11 (logs/eval_task_e_seed2_final_seed11.log); latest best_loss.pt SHA256 f14c9d7c536c1d4156a8603064ec65cc593d8659147da175212fd8d7cbad2950 scored 9.00 on seed11 (logs/eval_task_e_seed2_bestloss_96900_seed11.log, only object_3 inside). Do not deploy seed2 over current XSA candidate.
Strategy judgment as of 2026-05-22: keep ACT/XSA as the best deployable baseline and finish seed2 to 100k for evidence, but do not assume more ACT iterations will reach stable 18. Prepare a pi0.5/OpenPI fine-tune branch using the existing HDF5 demos converted to LeRobot format; use GraspNet/SAM3 mainly as a grasp/data-generation aid unless object_1/2 contact stability is proven in ATEC Isaac with real observations/actions.
XSA final checkpoint had the best current evidence but is not full-score stable. Direct eval of runs/act-task-e-rgb-xsa-100demos-30k-seed11/checkpoints/final.pt scored 15/15/15 on seeds 11/12/13 (object_2 and object_3 inside, object_1 outside), with seed-11 video logs/videos/task_e_act_eval/xsa_final_seed11_recheck.mp4. The same checkpoint deployed as demo/policy_act.pt later scored 9 on seed 11 and 0 on seed 12, so there is residual nondeterminism/contact instability. Current demo/policy_act.pt SHA256 is XSA final c3eacae3f1b0ec8ccde9fdc05d680fff119cc2ca70e1f79e4ddd5610c4bbfa93; previous seed1 best was backed up as demo/policy_act.seed1_best_backup_20260522_012719.pt.
pi0.5 / OpenPI Task E branch, 2026-05-22:
OpenPI repo/env: $HOME/src/openpi-ebench-clean, Python env $HOME/envs/openpi-pi05; run with PYTHONPATH=$HOME/src/openpi-ebench-clean/src:$HOME/src/openpi-ebench-clean/packages/openpi-client/src, HF_LEROBOT_HOME=$HOME/projects/robotics_shared/datasets/lerobot, HF_HOME=$HOME/projects/robotics_shared/hf_cache.
Avoid: do not use uv run in the OpenPI repo; it tries to create a large local .venv and download duplicated packages.
Converter: scripts/pi05/convert_task_e_hdf5_to_lerobot.py maps ATEC 8D Piper qpos/action to OpenPI fixed-base 16D padded state/action.{joints,gripper} and writes 3 identical 224x224 camera streams. 224 smoke dataset atec/task_e_obj321_servo_smoke2_stride10_224 wrote 343 frames in about 70s and occupies 48M.
OpenPI configs added in $HOME/src/openpi-ebench-clean/src/openpi/training/config.py: pi05_atec_task_e_smoke_224 and pi05_atec_task_e_s5_224. The formal s5_224 config uses repo atec/task_e_obj321_servo_100demos_s5_224, batch size 1, 30k steps, save interval 30k to avoid 42G checkpoint spam.
Important fix: ATEC HDF5 actions are env actions (joint_target-default)/0.5, not absolute joint targets, so all ATEC OpenPI configs must keep extra_delta_transform=False. Using the OpenPI delta transform here would subtract state again and train on the wrong target.
Smoke verification passed: pi05_atec_task_e_smoke_224 norm stats wrote under $HOME/projects/robotics_shared/checkpoints/openpi/atec_runs/assets/pi05_atec_task_e_smoke_224/...; 10-step train saved successfully once, but the temporary 42G checkpoint was deleted to preserve disk space. Log: logs/pi05_atec_task_e_smoke_224_10step_20260522.log.
Inference bridge added: demo/solution_pi05.py talks to an OpenPI websocket server and maps ATEC observation/action to/from the 16D fixed-base format. Eval entry point: scripts/pi05/eval_task_e_pi05.py. Default ATEC_PI05_ACTION_REPEAT=5 matches the current stride-5/fps-10 training dataset. After a checkpoint exists, use scripts/pi05/run_pi05_eval_checkpoint.sh <checkpoint_step_dir> to start scripts/serve_policy.py policy:checkpoint ..., run seeds 11/12/13, and record a seed-11 video.
Eval summary helper: scripts/pi05/summarize_pi05_eval.py <logs/pi05_eval_*> parses seed logs for score/object basket status and is called automatically at the end of run_pi05_eval_checkpoint.sh.
Formal s5 conversion service started: unit from logs/pi05_convert_task_e_s5_224.latest_unit, log from logs/pi05_convert_task_e_s5_224.latest; auto continuation service logs/pi05_s5_train_after_convert.latest_unit waits for conversion, then computes norm stats and trains pi05_atec_task_e_s5_224. Eval watcher logs/pi05_100demos_eval_after_train.latest_unit waits for training, then calls CONFIG=pi05_atec_task_e_s5_224 PORT=8017 scripts/pi05/run_pi05_eval_checkpoint.sh <latest_ckpt>.
Early-signal branch: pi05_atec_task_e_20demos_s5_224 config uses repo atec/task_e_obj321_servo_20demos_s5_224, 2k steps, batch size 1, no delta transform. Conversion service/log are in logs/pi05_convert_task_e_s5_224_20demos.latest_unit and .latest; auto continuation service logs/pi05_20demos_train_after_convert.latest_unit waits, computes norm stats, then trains. Eval watcher logs/pi05_20demos_eval_after_train.latest_unit waits for training, then calls CONFIG=pi05_atec_task_e_20demos_s5_224 PORT=8016 scripts/pi05/run_pi05_eval_checkpoint.sh <latest_ckpt>. Use this for quick pi0.5 viability checks before the 100-demo run finishes.
Note: the first 20-demo training run started from the old script name atec_task_e_pi05_20demos_s5_224_10k_20260522070624, but the loaded config was already patched to num_train_steps=2000; the name is cosmetic. The script was patched afterward to use _2k_ in future run names.
20-demo result: checkpoint .../pi05_atec_task_e_20demos_s5_224/atec_task_e_pi05_20demos_s5_224_10k_20260522070624/1999 scored 0/0/0 on seeds 11/12/13. Logs: logs/pi05_eval_20260522074137; video: logs/videos/task_e_pi05_eval/pi05_pi05_atec_task_e_20demos_s5_224_seed11_20260522074137.mp4. The 42G 20-demo checkpoint directory was deleted after evaluation to preserve disk; dataset and logs were kept.
100-demo formal run status, 2026-05-22 08:56 CST: conversion and norm stats are complete; training unit atec-pi05-s5-train-after-convert-20260522065716.service is actively running scripts/train.py pi05_atec_task_e_s5_224 --exp-name atec_task_e_pi05_s5_224_30k_20260522065716 --overwrite. Latest observed progress was about 731/30000 steps at roughly 3.3-3.6 it/s, GPU util 100%, ~80 GB VRAM. Log: logs/pi05_train_task_e_s5_224_20260522065716.log.
100-demo checkpoint/eval gate: config pi05_atec_task_e_s5_224 saves only at 30000 steps (save_interval=30000), expected final dir .../checkpoints/pi05_atec_task_e_s5_224/atec_task_e_pi05_s5_224_30k_20260522065716/30000. Eval watcher unit atec-pi05-100demos-eval-after-train-20260522070900.service waits for the training unit, then runs CONFIG=pi05_atec_task_e_s5_224 PORT=8017 scripts/pi05/run_pi05_eval_checkpoint.sh <checkpoint> for seeds 11/12/13 and records a seed-11 video.
Deployment decision gate: current deployed ACT/XSA baseline is still demo/policy_act.pt SHA256 c3eacae3f1b0ec8ccde9fdc05d680fff119cc2ca70e1f79e4ddd5610c4bbfa93. Do not replace it with pi0.5 unless the 100-demo pi0.5 checkpoint has real ATEC Isaac eval evidence on seeds 11/12/13 and is clearly better. Required completion artifacts for this branch are: final checkpoint dir, logs/pi05_eval_* with seed11/12/13 [RESULT] and [BASKET], and logs/videos/task_e_pi05_eval/pi05_pi05_atec_task_e_s5_224_seed11_*.mp4.
Post-eval comparison helper: scripts/pi05/compare_pi05_vs_act_baseline.py <logs/pi05_eval_*> --repo <repo> compares pi0.5 seed logs with current ACT/XSA evidence (eval_task_e_xsa_final_seed11/12/13_recheck.log plus deployed seed11 recheck) and prints HOLD_ACT_XSA unless pi0.5 is clearly better. scripts/pi05/run_pi05_eval_checkpoint.sh calls this automatically after summarize_pi05_eval.py. It was sanity-checked on the 20-demo failed eval and correctly reported HOLD_ACT_XSA.
100-demo result, 2026-05-22 11:27 CST: checkpoint .../pi05_atec_task_e_s5_224/atec_task_e_pi05_s5_224_30k_20260522065716/29999 (about 42G) evaluated at logs/pi05_eval_20260522112701 and scored seed11 0.00, seed12 0.00, seed13 6.00 (mean 2.00). Seed11 video: logs/videos/task_e_pi05_eval/pi05_pi05_atec_task_e_s5_224_seed11_20260522112701.mp4. The comparison helper printed HOLD_ACT_XSA; do not deploy pi0.5 over current XSA.
Monitoring: the old heartbeat automation atec-task-e-seed2-resume-monitor was deleted after the 100-demo pi0.5 train/eval completed, because the monitored branch is done. Recreate a new monitor only if starting another long run.
Required PYTHONPATH: IsaacLab source dirs under $HOME/envs/genmanip-isaac5-py311/lib/python3.11/site-packages/isaaclab/source/*, plus source/atec_rl_lab and scripts/act from this repo.
Current ACT baseline:
Object 3 only works with dataset datasets/atec_task_e, run runs/act-task-e-rgb-100demos-seed1, observed eval score about 6.
Official/local Task E collector comment said object 3 works and object 1/2 need distance modifications.
Piper gripper joint limits observed on 2026-05-20: joint7 is [0.0, 0.035], joint8 is [-0.035, 0.0]. Use legal close target [0.0, 0.0], not [-0.015, 0.015].
Object 2 had a previously observed basket candidate: offset (dx=0.006, dy=0.046), z=0.045, carry_z=TABLE_TOP_Z+0.075, place_z=TABLE_TOP_Z+0.075, target_y=-0.24, yaw=-pi/2, keep gripper closed in transport. Treat this as not yet robust and likely push/slide rather than true lift: on 2026-05-20 a trace rerun showed the earlier search used candidate-index-dependent RNG, so success could be placement-specific.
On 2026-05-20 13:47, object 2 candidate search first 20 all failed under fixed trial seed. Best was cand 011 (dx=0.030, dy=0.026, z=0.045, carry_z=TABLE_TOP_Z+0.075, target_y=-0.24, yaw=+pi/2), final (1.224,-0.009,1.213), reward_lifted=True, outside by dy=+0.291. Object 2 is not ready for demo collection/training.
Object 1 is unresolved as of 2026-05-20. Prior search candidates mostly pushed/slid the object; final basket failure alone was too weak to prove a real grasp. Add/keep per-stage diagnostics before trusting any object 1 candidate.
Important local changes:
scripts/act/eval_task_e_act.py evaluates ACT and can record video.
demo/solution_act.py supports ATEC_ACT_POLICY_PATH and reset_episode().
scripts/act/cli_args.py adds --max_attempts; scripts/act/collect_demos_task_e.py uses it and prints object basket status.
scripts/act/task_e/collector.py uses _BASKET_MAX_Z = TABLE_TOP_Z + 0.15, has basket_status_lines, and can return numeric trace diagnostics when trace=True including object z-gain, gripper_base distance, and link7/link8 finger-body center/gap metrics.
scripts/act/search_task_e_grasps.py is the per-object candidate bank/search entry point.
scripts/act/search_task_e_grasps.py should keep trial seeds independent of candidate index so candidates are compared on the same object placements.
scripts/act/search_task_e_grasps.py prints [BEST] even when no candidate succeeds, using basket distance plus z-gain as a debugging score.
scripts/act/search_task_e_grasps.py supports --start_candidate to resume long searches without re-testing earlier candidates.
scripts/act/search_task_e_grasps.py now supports per-candidate push_x/push_y/push_z plus close/lift/transport/place/open step overrides. Use this to test trajectory timing without editing global config for each trial.
scripts/act/task_e/state_machine.py now has object-servo transport/place/open support for object 1/2: OBJ_SERVO_TO_BASKET_STATES, OBJ_SERVO_XY_GAINS, and OBJ_SERVO_MAX_XY in scripts/act/task_e/config.py. The TRANSPORT correction is ramped by state progress to avoid yanking objects out of the gripper.
scripts/act/collect_demos_task_e.py now disables default cameras and cfg.observations.image when need_camera=False; this fixes headless collection without --enable_cameras.
Latest object 1 trace:
On 2026-05-20 13:19, foreground search candidates 1-12 all failed: lifted=False, reward_lifted=False, z_gain about 0.000-0.003m; best candidate was 004 but still final y=0.210, far outside basket. Next search should resume at candidate 13 to test push/sweep fallback candidates.
On 2026-05-20 13:24, foreground search candidates 13-24 tested push/sweep fallback. No basket success yet, but contact is real: candidate 019 (dx=0.040, dy=0.140, z=-0.020, carry_z=TABLE_TOP_Z+0.035, target_y=-0.240, yaw=0, push_y=-0.18) reached lifted=True, reward_lifted=True, z_gain=0.388, final pos about (1.276, -0.072, 1.260). It overshoots upward/outside basket, so next search should refine around push_y -0.14..-0.18, lower carry/place behavior, and possibly static place release.
On 2026-05-20 13:36, focused candidates 25-34 around candidate 019 all failed. Best was cand 034: dx=0.040, dy=0.140, z=-0.020, carry_z=TABLE_TOP_Z+0.035, target_x=-0.17, target_y=0.0, push_y=-0.18, final (1.280,-0.064,1.257), reward_lifted=True, still outside by dy=+0.236.
On 2026-05-20 13:42, timing candidates with longer/lower TRANSPORT and PLACE/OPEN overrides all failed. Best was cand 002 with transport_steps=900, final (1.280,-0.042,1.257), outside by dy=+0.258. This proves simply extending the push/sweep duration does not solve object 1; the contact geometry slips/throws the box before basket placement.
On 2026-05-20 14:15, object-servo ramp still failed object 1 first 8 candidates. Best was cand 003 (push_y=-0.14, transport_steps=1200), final (1.294,-0.020,1.270), reward_lifted=True, outside by dx=+0.214, dy=+0.280. Object 1 still needs a different contact/grasp primitive, not more global servo.
Latest object 2 trace:
On 2026-05-20 14:14, object-servo ramp made fixed-seed search candidate 001 succeed for object 2: final (1.050,-0.272,0.865), inside=True, z_gain=0.102, candidate {dx=0.006, dy=0.046, z=0.045, carry_z=TABLE_TOP_Z+0.075, target_y=-0.24, yaw=-pi/2}.
On 2026-05-20 14:20, random only_success collection for object 2 still failed 0/4 attempts: final y errors remained about +0.206..+0.310, one early termination. So object 2 is promising but not robust enough for dataset/training yet.
On 2026-05-20 14:26, adding PLACE hold-until-near-basket still failed random object 2 collection 0/4: final y errors stayed about +0.206..+0.334.
On 2026-05-20 14:28, stronger object 2 servo (gain=1.15, max_xy=0.36, TRANSPORT=480) also failed 0/4 and worsened x drift; reverted object 2 to the softer fixed-seed-success servo values (gain=0.85, max_xy=0.24, no transport override). Next object 2 work should focus on grasp/contact robustness before transport, not stronger basket servo.
On 2026-05-20 14:41, object 2 multi-seed candidate validation (--trials 3 --max_candidates 12) found 0/12 robust candidates. The previous fixed-seed-success candidate failed immediately under this stricter check (result=0/3, final y error about +0.262). Best was yaw 0.0 with z_gain=0.355 but final (1.197,-0.001,1.277), outside by dy=+0.299. Treat object 2 as contact/grasp unstable, not transport-servo-limited.
Current debugging judgment:
Do not integrate GraspNet first. For this small three-object simulated task, a reproducible scripted candidate bank with explicit lift/grasp diagnostics is faster and easier to pass code-review reproduction.
If object 1 remains blocked, collect/train object 2+3 first for a practical score bump, then return to object 1. Next object 1 work should change the motion primitive/contact mode, not merely sweep more static offsets or longer transport times.
https://github.com/fiveages-sim/lerobot_ros2 is a public ROS2/LeRobot integration repo. It exposes generic ROS2 robot/camera wrappers and examples with single_arm.pick/place YAML skills, but it does not directly provide an ATEC Piper-compatible grasp planner. Its robot_action_composer submodule is private/inaccessible by HTTPS clone in this environment.
