{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "# 06 ROCm 调试复盘与排障案例\n\n        这一节把复刻中的折腾整理成排障工作簿。不需要记住所有命令，关键是把一个失败现象拆成证据、根因、修复和验证。\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "from pathlib import Path\nimport json\nimport os\nimport shutil\nimport subprocess\nimport sys\n\n\ndef find_topic_root():\n    cwd = Path.cwd().resolve()\n    for candidate in [cwd, *cwd.parents]:\n        if (candidate / \"assets\" / \"metrics_snapshot.json\").exists():\n            return candidate\n    raise RuntimeError(\"请从 AMD ROCm 专题目录或 notebooks 子目录启动 Jupyter。\")\n\n\nTOPIC_ROOT = find_topic_root()\nASSET_DIR = TOPIC_ROOT / \"assets\"\nNOTEBOOK_DIR = TOPIC_ROOT / \"notebooks\"\nPROJECT_ROOT = Path(os.environ.get(\"PROJECT_ROOT\", \"/path/to/every-embodied/mujoco_pnp\"))\nDATA_ROOT = Path(os.environ.get(\"DATA_ROOT\", \"/path/to/datasets/every_embodied\"))\nOUTPUT_ROOT = Path(os.environ.get(\"OUTPUT_ROOT\", TOPIC_ROOT / \"outputs\"))\nMODEL_ROOT = Path(os.environ.get(\"MODEL_ROOT\", PROJECT_ROOT / \"ckpt\"))\n\nprint(\"TOPIC_ROOT =\", TOPIC_ROOT)\nprint(\"PROJECT_ROOT =\", PROJECT_ROOT)\nprint(\"DATA_ROOT =\", DATA_ROOT)\nprint(\"OUTPUT_ROOT =\", OUTPUT_ROOT)\nprint(\"MODEL_ROOT =\", MODEL_ROOT)\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "try:\n    from IPython.display import Image, Markdown, display\nexcept Exception:\n    class Markdown(str):\n        pass\n\n    def display(obj):\n        print(obj)\n\n    def Image(filename=None, width=None):\n        return f\"[image] {filename}\"\n\n\ndef show_asset(filename, width=960):\n    path = ASSET_DIR / filename\n    if path.exists():\n        display(Image(filename=str(path), width=width))\n    else:\n        print(f\"缺少素材：{path}\")\n\n\ndef md_table(headers, rows):\n    lines = [\"| \" + \" | \".join(headers) + \" |\", \"| \" + \" | \".join([\"---\"] * len(headers)) + \" |\"]\n    for row in rows:\n        lines.append(\"| \" + \" | \".join(str(x) for x in row) + \" |\")\n    display(Markdown(\"\\n\".join(lines)))\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## Checkpoint 1：排障记录模板\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "template = \"\"\"\n### 问题名\n\n- 现象：\n- 复现命令：\n- 关键证据：\n- 排除项：\n- 根因：\n- 修复：\n- 修复后验证：\n- 学习结论：\n\"\"\"\nprint(template)\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## Checkpoint 2：九个典型案例索引\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "rows = [\n    (\"旧 success 误判\", \"必须引入 physical_success\"),\n    (\"示教数据 gripper 标签问题\", \"先审计数据，再怀疑模型\"),\n    (\"ACT loss 下降但闭环失败\", \"open-loop 与 closed-loop 分开看\"),\n    (\"DAgger 数据混入后退化\", \"新增数据要说明补的是哪个状态分布\"),\n    (\"SmolVLA 红蓝杯偏置\", \"按 instruction 拆开评估\"),\n    (\"复制数据伤害另一类任务\", \"优先试 sampler/loss 层加权\"),\n    (\"视频与 batch 统计不一致\", \"视频解释行为，成功率看 batch JSONL\"),\n    (\"ROCm 长训练资源问题\", \"脚本化、分批评估、检查进程和温度\"),\n    (\"pi0 训练前门控\", \"权限和 smoke 通过后再谈长训练\"),\n    (\"pi0 reset 残留\", \"同一环境连续评估前先清掉 MuJoCo 动力学状态\"),\n]\nmd_table([\"案例\", \"学习结论\"], rows)\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## Checkpoint 3：pi0 hard-reset 评估协议案例\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "`dynamic_timed` finisher 的一次排障很典型：stage-aware state 修复后，旧版连续环境评估已经能跑到 strict `30/30`，但 seed `1036` 的终态 `xy` 一度贴近阈值。继续 trace 时发现，后一个 seed 的初始杯子位置偶尔跑出该 seed 的采样范围，说明前一个 episode 的 qvel / ctrl / free-joint 动态状态污染了下一个 episode。\n\n        这里不要急着把它归因成模型泛化问题。更干净的做法是先修评估协议：小面板可以用 `--fresh-env-per-episode` 每个 seed 新建环境；完整批量评估推荐 `--hard-reset-sim-data`，在每次 `env.reset(seed)` 前先清掉底层 MuJoCo sim data。用 hard-reset clean protocol 重跑 策略采样 seeds（环境实际固定为 seed 0） `1010-1039` 后，结果仍是 strict `30/30`、legacy `30/30`，mean `xy=0.0219 m`，max `xy=0.0450 m`，seed `1036` 不再贴边。\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "rows = [\n    (\"现象\", \"旧连续环境评估 30/30，但 seed 1036 一度 max xy=0.0993 m\"),\n    (\"异常证据\", \"seed 1035 初始杯子位置出现采样范围外的值\"),\n    (\"根因\", \"env.reset(seed) 重设物体位置，但没有先清掉底层 MuJoCo 动力学残留\"),\n    (\"修复\", \"新增 --fresh-env-per-episode 和 --hard-reset-sim-data\"),\n    (\"clean 结果\", \"hard-reset 后 1010-1039 为 strict 30/30，mean xy=0.0219 m，max xy=0.0450 m\"),\n]\nmd_table([\"项\", \"记录\"], rows)\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## Checkpoint 4：把指标和视频放在一起看\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "show_asset(\"act_dagger_progress_curve.png\", width=900)\nshow_asset(\"smolvla_red_blue_success.png\", width=1000)\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "图表回答“整体表现如何”，视频回答“行为到底像不像成功”。这两类证据最好同时放进实验报告。\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": "## Checkpoint 5：结果摘要模板\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": "rows = [\n    (\"设备\", \"GPU/APU 型号、ROCm、PyTorch、温度和显存\"),\n    (\"数据\", \"episode 数量、任务类型、是否通过物理回放审计\"),\n    (\"ACT\", \"best checkpoint、physical_success、主要失败类型\"),\n    (\"SmolVLA\", \"红杯/蓝杯成功率、采样策略、保护基线\"),\n    (\"pi0\", \"gated 权限、1-step smoke、正式训练状态\"),\n    (\"视频\", \"一个真实成功、一个典型失败\"),\n    (\"复盘\", \"最关键的一个坑和修复证据\"),\n]\nmd_table([\"项目\", \"应该写什么\"], rows)\n"
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}