Evaluation Protocol and Model Selection
1. Metric Definition
T4 is not a simple binary success classification. For each episode, the final stacked layer count is returned, and the score can only be a discrete layer value: 0 layer, 1 layer, 2 layers, or 3 layers. If the total number of layers for three fresh3 instances is 2, then the mean graded value is 2/9. This does not indicate which specific three instances have each layer count, unless the final states per seed are saved.
Therefore, the report must include at least the following simultaneously:
textseed 列表
每条 episode 的层数
0/1/2/3 层计数
完整三碗 success 数
mean graded
失败阶段（如果有可靠 trace）
2. Access Control Levels
Gate A: Data and Interfaces
- Manifest, raw/processed quantities, and SHA;
- ACT loader, checkpoint, stats, and deploy config can be loaded;
- Three cameras, action/state dimensions, and next-state alignment are completed;
- The code package has no soft links, dangerous paths, or additional dependencies.
Gate B：fresh3
Only answer whether “this candidate is obviously damaged”. Three successes or high mean grades are not proof of online generalization. If fresh3 does not meet the preset threshold, stop upgrading and diagnose first.
Gate C：fresh10 / fresh20
Require a fixed seed, the same runner, and the same process topology. This stage is used for filtering candidates and identifying seed-specific failures, but it is still not equivalent to the final online private 100.
Gate D: Complete public panel
Only the complete and pre-fixed public panel can be used for candidate comparison. The seed file cannot be modified after viewing the results, nor can setup failures be removed.
Gate E: Online Submission
The private seeds and environment of the online service are the final truth values. They must be frozen before submission: checkpoint, stats, code root, deploy config, dependency versions, packaging list, time, and account unlock status.
3. Boundaries of ordinary runners, traces, and videos
| Execution Mode | Purpose | Can be used as online score? |
|---|---|---|
| Official regular runner | Formal local access control | Only serves as local evidence; cannot replace online results |
| trace/monkey-patch runner | Diagnosis during failure phase | Cannot be directly used as a formal score |
| cold single seed process | Isolated suspension, rapid filtering | Cannot be directly compared with persistent scores |
| persistent official topology | Closest to online results | Still requires online verification |
| Expert trajectory replay | Verification environment and success predicate | Not considered a successful policy |
| Online official testing | Final result of private rankings | Is the final true value |
The 1/3, 5/9 in this clean50 trace cannot cover the 0/3, 1/3 of the regular runner, nor can it cover the online T4 25.3/100.
4. Failure stage records
Do not just write success=false. It is recommended to categorize them according to the following stages:
textsetup_demo / reset
approach
grasp
lift
transport
lower
release
stabilize
final stacked-state check
However, it can be categorized only when the individual action traces, status, or reliable videos are preserved. For a regular runner with an aggregate grade, one should not guess the failure stage based on the score alone.
5. Model Selection Rules
- First, protect a code package; the candidate should have complete stats, weights, and evaluation evidence.
- The candidate comparison uses the same official wrapper, seed order, and process topology.
- First, check the complete three bowls of success, then mean graded, followed by the layer distribution and failure stages.
- Loss is only used as a training health signal and does not determine submission independently.
- Clean50 diagnosis only determines the next experiment and does not replace exact550 directly.
- Any changes to action alignment, data sampling, gripper loss, chunk, or model size must be single variable, independent run name, and independent stats.
- Successful replay of training set/expert data is not counted in the generalization panel.
- If the subsequent candidate is lower than the protected candidate, the protected file cannot be overwritten.
6. When there are differences between online and local results
Check in the following order:
- Ensure that the uploaded weights match the local candidate SHA;
- Verify that the stats correspond to the same set as the weights;
- Confirm that the code used in production, deploy config, and local preflight are consistent;
- Make sure there are no additional Python dependencies, soft links, or absolute paths;
- Check whether the public/private seed distributions, task initial state, and failure setup differ;
- Verify the differences in cold/persistent reset, planner warm-up, RNG, and renderer initialization;
- Finally, discuss whether the model lacks sufficient generalization.
This online value of 25.3 is less than 2/3 of the local historical control fresh3, which at least indicates that the public 3-seed results are not stable enough; however, this does not prove that a specific cause has been identified.