HIMLoco Four-Legged Robot Motion Control: From Paper Understanding to Reproduction in Isaac Lab New Stack

This chapter guides you through a practical learning experience with HIMLoco: first, understand what the ICLR 2024 paper Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response aims to solve, then examine the open-source repository to clarify the observations, internal model, PPO training, and Sim2Real design. Finally, migrate the training process that originally relied on Isaac Gym Preview 4 to the new Isaac Sim/Isaac Lab environment, and run the smoke test on Blackwell GPU.

The goal of this chapter is not to train a paper-level final policy, but to help everyone understand the most critical engineering processes clearly:

• Why HIMLoco is not an ordinary obs -> action PPO;
• Where do the historical observations of 45 x 6 = 270 and 238’s privileged critic observation come from;
• How does HIMEstimator learn speed and hidden variables from historical proprioception;
• Why the old Isaac Gym stack fails on Blackwell GPU;
• How to run tasks and HIM algorithms using the new Isaac Lab / PyTorch CUDA 12.8 stack;
• What is the relationship between the official demo video, local rendering video, and complete training results;
• What smoke tests can prove, and what they cannot prove.

Note: The reproduction in this chapter uses the open-source repository OpenRobotLab/HIMLoco, the paper project page is the official HIMLoco homepage, and the paper is arXiv:2312.11460. The README of the official repository indicates that the training code is available, but the deployment guide is not yet available. Therefore, this chapter focuses on the training pipeline, algorithm structure, and migration to the Isaac Lab new stack, without interpreting the smoke test results as the real robot deployment being completed.

Reproduction Roadmap

You can view this chapter as a reinforcement learning experiment class for quadruped robots. We start by analyzing the original paper and repository to understand why the author designed the Hybrid Internal Model. Then, we examine how each tensor flows in the engineering implementation. Finally, we complete the minimum training and validation using a local migration version task.

It is necessary to clarify the significance of the smoke test: it proves that environment creation, observation construction, action execution, reward calculation, rollout storage, backpropagation, and optimizer update all work properly. It does not prove that the policy has converged, nor does it indicate that the policy can be directly applied to a real robot.

1. What problem does HIMLoco solve

In quadruped robot motion control, reinforcement learning policies often face a contradiction: simulation tools possess many information that real robots cannot obtain during deployment, such as accurate base linear velocity, external disturbances, terrain height, contact status, and rigid body mass changes. However, when using a real robot, the policy usually relies only on IMU, joint encoders, previous action, and a small amount of state estimation.

If the actor directly reads these simulation privileges during training, the policy will look good in the simulation. However, when deployed, the information is missing, making Sim2Real vulnerable. If these information are completely omitted, the simulation can be wasted by using the supervision signals it provides for training.

The core idea of HIMLoco is to build a bridge between the two:

• During training, the critic can use privileged observations to help PPO learn a more accurate value function;
• The actor does not directly read privileged observations, but only the historical proprioception that is likely to be obtained in real deployment;
• An additional HIMEstimator is trained to infer the base velocity and latent hidden variables from historical observations;
• Finally, the actor uses 当前观测 + 估计速度 + latent to output 12 joint actions.

In other words, HIMLoco does not allow the policy to “peek at answers,” but instead trains an internal model so that the robot can infer the current dynamics state from its own historical responses.

Figure 1: Overview of the HIMLoco official policy. This figure illustrates the core concept of HIMLoco: the policy does not rely solely on the current frame of observations, but uses historical robot responses to learn internal state representations, which are then integrated into the motion control policy. Focus on two aspects: first, the internal model connects historical responses and current control; second, simulation data during training is used for learning representations, rather than simply serving as inputs required for deployment.

_{Source: HIMLoco official project page, image file `overview.jpeg`.}

Figure 2: Simplified diagram of the HIMLoco official internal model control. This figure emphasizes the concept of internal model control: after the robot performs an action, a physical response is generated, and this historical response is then encoded into the internal states required by the policy. This is crucial for those working on robot control: HIMLoco does not simply add an MLP layer to the network structure, but explicitly teaches the network to learn the dynamic information embedded in the robot’s responses during training.

_{Source: HIMLoco official project page, image file `imc.jpeg`.}

The relationship between official videos and local remakes

The official HIMLoco homepage also provides multiple demo videos. It is recommended that users watch the official videos first while learning, to gain an intuitive understanding of “the behavior effects desired in the paper”; then watch the local smoke-test videos at the end of this chapter to confirm that the Isaac Lab migration process on our machine can render, conduct simulation, output actions, and write checkpoints. The two types of videos have different meanings: the official videos demonstrate the effects of the research project, while the local videos show the minimum viable evidence after migration.

_{Source: YouTube demo video embedded on the HIMLoco official project page.}

II. Code organization in the original repository

The HIMLoco open-source repository is based on modifications of legged_gym and rsl_rl. When reading the code, do not use the original legged_gym or rsl_rl for installation, as the author has added a runner, storage, actor-critic, and PPO specifically for HIM in the repository.

The core directory can be understood as follows:

The training entry itself is very short:

pythonenv, env_cfg = task_registry.make_env(name=args.task, args=args)
ppo_runner, train_cfg = task_registry.make_alg_runner(env=env, name=args.task, args=args)
ppo_runner.learn(num_learning_iterations=train_cfg.runner.max_iterations, init_at_random_ep_len=True)

True logic lies in the environment and runner created by the registry. When reading the code, you can follow the order of “the environment generates tensors first, and the runner consumes them later”, and avoid getting stuck in the PPO formula from the beginning.

III. How the observation tensor flows

The observation design in HIMLoco is the most valuable part to learn in the entire paper and repository. It separates the information visible to the actor from that visible to the critic.

1. one-step observation of actor

Each frame of actor observation in the original repository has 45 dimensions:

Then HIMLoco stacks 6 frames of history:

textactor observation = 45 x 6 = 270

This is similar to the “short-term memory” on a robot. Single-frame proprioception may not be able to accurately determine speed, delay, contact, and external disturbances, but 6 frames of history allow the network to infer these states from the response trends.

2. Critic’s privileged observation

critic uses 238-dimensional privileged observation:

The design here is a typical asymmetric actor-critic approach: the actor simulates deployment constraints, while the critic uses simulation true values to reduce training difficulty.

3. The HIM actor actually consumes 64 dimensions

Many people think that the input of actor MLP is 270 dimensions when they first see the code. However, it isn’t true. HIMEstimator First, encode the 270-dimensional historical observations as:

textestimated base velocity: 3
latent: 16

Then the input for actor MLP is:

text当前 one-step obs 45
+ estimated base velocity 3
+ latent 16
= 64

This is the form in which the “internal model” of HIMLoco appears in the code.

Figure 3: HIMLoco’s tensor flow. This figure is a reproduction version of the data flow organized in this chapter based on the source code. Note that the inputs of actor and critic are different: actor obtains its internal state from historical observations via the estimator, while critic directly reads the privileged observations during training.

IV. Training Objectives of Estimator

HIMEstimator is not just an ordinary encoder. It has two training objectives:

1. Predict the base linear velocity;
2. Learn structured latent through swap loss in prototype/Sinkhorn style.

The key segment in the code is:

pythonvel = next_critic_obs[:, self.num_one_step_obs:self.num_one_step_obs+3].detach()
next_obs = next_critic_obs.detach()[:, 3:self.num_one_step_obs+3]

Here, vel comes from the real base linear velocity in next privileged observation, serving as a supervision signal for the estimator. next_obs is used by the target network to generate a latent target. Subsequently, the code will calculate:

pythonestimation_loss = F.mse_loss(pred_vel, vel)
swap_loss = -0.5 * (q_s * log_p_t + q_t * log_p_s).mean()

For robot learning, this design is very insightful: it doesn’t just let the policy follow the reward end-to-end, but incorporates an interpretable intermediate task within the policy, enabling the network to learn how to estimate dynamic quantities that are difficult to measure directly during deployment.

5. Why the original Isaac Gym stack doesn’t work on new graphics cards

The recommended environment for the official README is:

textUbuntu 20.04
Python 3.7.16
PyTorch 1.10.0+cu113
Isaac Gym Preview 4

This environment is reasonable for old graphics cards, but a CUDA architecture mismatch occurs with the Blackwell GPU. Our tested graphics card is NVIDIA RTX PRO 6000 Blackwell Workstation Edition, and its compute capability is sm_120. PyTorch 1.10.0+cu113 only supports older architectures, which results in errors during training:

textCUDA error: no kernel image is available for execution on the device

We also tried using the newer PyTorch 2.4.1+cu124 under Python 3.8, which is one of the higher Python versions that the Isaac Gym Preview 4 binding can tolerate. However, the torch.cuda.get_arch_list() in this wheel only goes up to sm_90, and it cannot cover sm_120.

Therefore, for the Blackwell route, a more reliable approach is not to continue fixing Isaac Gym Preview 4, but to migrate to:

textIsaac Sim / Isaac Lab
Python 3.11
PyTorch 2.7.0+cu128
rsl-rl-lib 3.0.1

This is also why the new stack of Isaac Lab is used in the latter part of this chapter.

VI. Engineering Structure after Isaac Lab Migration

The goal after migration is not to rewrite all physical details, but to preserve the training semantics of HIMLoco as much as possible:

• Unitree A1, 12-dimensional joint position action;
• action scale 0.25;
• PD stiffness 40.0, damping 1.0;
• dt=0.005, decimation 4, episode length 20s;
• rough terrain curriculum;
• command range x/y = [-1, 1], yaw/heading [-pi, pi];
• policy observation 270;
• critic observation 238;
• HIM runner includes HIMActorCritic, HIMEstimator, HIMPPO.

The local structure of the migrated project is as follows:

textHIMLoco/
  source/himloco_isaaclab_tasks/
    himloco_isaaclab_tasks/
      a1/
        rough_env_cfg.py
        agents/rsl_rl_ppo_cfg.py
      rsl_rl_him.py
  scripts/
    isaaclab_train_himloco.py
    train_himloco_isaaclab.sh
    train_himloco_isaaclab_him.sh

Among them, rough_env_cfg.py is responsible for transforming the A1 rough terrain task in Isaac Lab into a HIMLoco-style observation/reward/domain-randomization configuration; rsl_rl_him.py adapts the HIM-related algorithms from the original repository to the observation group and TensorDict interface of rsl-rl-lib 3.x.

Figure 4: Local reproduction link after Isaac Lab migration. This figure is the engineering reproduction diagram for this chapter. Unlike the official diagram, it focuses on which modules interact during actual operation. The environment generates an observation group, the runner collects rollout data, PPO updates the actor/critic, and the estimator learns the internal model through next critic observations.

VII. Environment Preparation

It is recommended to define several path variables in advance to avoid hardcoding commands on a specific machine:

bashexport HIMLOCO_ROOT=/path/to/HIMLoco
export ISAACLAB_ROOT=/path/to/IsaacLab
export PYTHON_BIN=/path/to/isaaclab-python/bin/python

The key versions of the actual testing environment in this chapter are as follows:

textGPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, sm_120
Torch: 2.7.0+cu128
Isaac Sim: 5.1
rsl-rl-lib: 3.0.1

The migration task package needs to be installed in an editable format:

bashcd "$HIMLOCO_ROOT"

"$PYTHON_BIN" -m pip install -e \
  "$HIMLOCO_ROOT/source/himloco_isaaclab_tasks" \
  --no-deps

Here, --no-deps is used to avoid pip re-parsing the heavy dependencies of Isaac Sim / Isaac Lab. It is best to keep the Isaac environment managed by the existing Isaac Lab environment, and not let a small task package trigger a full reinstall of dependencies.

VIII. Checkpoint 1: Normal PPO baseline runs successfully

After the migration, a normal PPO baseline is retained to confirm that the Isaac Lab task itself is functioning properly:

bashcd "$HIMLOCO_ROOT"
./scripts/train_himloco_isaaclab.sh --num_envs 1 --max_iterations 1

When successful, everyone should see a similar output from the observation manager:

textActive Observation Terms in Group: 'policy' (shape: (270,))
Active Observation Terms in Group: 'critic' (shape: (238,))
Action Terms (shape: 12)

Then the actor/critic structure will be displayed as:

textActor MLP: Linear(in_features=270, ...)
Critic MLP: Linear(in_features=238, ...)

The purpose of this baseline is to verify whether task registration, environment creation, terrain, reward, observation group, and the rsl-rl 3.x standard runner are functioning properly. It is not a reproduction of the original HIM algorithm, as the actor directly receives the 270-dimensional historical observations without using the 64-dimensional actor input generated by the estimator.

9. Checkpoint 2: HIM algorithm smoke test

The closer reproduction entry to the original HIMLoco is:

bashcd "$HIMLOCO_ROOT"
./scripts/train_himloco_isaaclab_him.sh --num_envs 1 --max_iterations 1

When successful, everyone should see:

textHIMActorCritic ...
Actor MLP: Linear(in_features=64, out_features=512, ...)
Critic MLP: Linear(in_features=238, out_features=512, ...)
Estimator: Linear(in_features=270, out_features=128, ...)

This indicates that all three key dimensions are aligned:

The single-environment smoke test can prove that the code flow works, but issues related to batch processing may not be exposed. Therefore, it is recommended to run a small-scale debugging test that is slightly closer to the actual training scenario:

bashcd "$HIMLOCO_ROOT"
./scripts/train_himloco_isaaclab_him.sh --num_envs 64 --max_iterations 2

The key logs measured in this chapter are as follows:

textIteration 0/2:
Computation: 2630 steps/s
Mean value_function loss: 0.3333
Mean surrogate loss: 0.0228
Mean entropy loss: 17.0127
Mean estimation loss: 0.7230
Mean swap loss: 0.1080
Mean reward: -2.80
Total timesteps: 6400

Iteration 1/2:
Computation: 4110 steps/s
Mean value_function loss: 0.1658
Mean surrogate loss: -0.0045
Mean entropy loss: 17.0122
Mean estimation loss: 0.6086
Mean swap loss: 0.1079
Mean reward: -3.24
Total timesteps: 12800

The complete excerpt is saved assets/logs/him_smoke_64env_2iter.txt. These losses are not used to evaluate the policy quality in 2 iterations; they are only used to confirm that PPO loss, estimator loss, and swap loss all participate in the update.

Figure 5: The loss curve of HIM smoke test. This is a local validation plot for 64 environments and 2 iterations. The curve is short and cannot be used to discuss convergence, but it confirms that value loss, surrogate loss, estimation loss, and swap loss are recorded in TensorBoard.

Figure 6: Performance records of HIM smoke test. This figure shows the total_fps and action noise std during local small-scale training. For the smoke test, the performance values serve only as a reference to whether the environment runs properly on the GPU, and do not represent the full training throughput.

Checkpoint 3: Record local rendering video

If you want to confirm that the Isaac Lab rendering chain is also working properly, you can start video recording at the HIM training entry point:

bashcd "$HIMLOCO_ROOT"
./scripts/train_himloco_isaaclab_him.sh \
  --num_envs 4 \
  --max_iterations 2 \
  --video \
  --video_length 120 \
  --video_interval 100000

When recording videos, attention should also be paid to the camera position. The migration script defaults to having the viewport camera follow robot/trunk in the first environment when --video is enabled; otherwise, Isaac Lab may only record a large area of rough terrain and fail to capture the robot dog. video_interval should not be set too small either; if recording triggers every step, Gymnasium RecordVideo will produce many 1-frame clips, which are not suitable for direct use in tutorials.

This video demonstrates that the A1 rough-terrain environment, camera capture, action steps, and training cycles are connected. However, it only has 2 iterations, and the policy has not yet learned to stabilize the gait. Therefore, do not compare it with the final movement results in the official video. The blue and green arrows in the video come from Isaac Lab’s command visualization, used to show the velocity commands and current velocity direction.

Video 5: Local Isaac Lab smoke-test rendering. This is a debugging clip of 4 environments and 2 iterations recorded in this chapter using Blackwell GPU + Isaac Lab new stack. Its value lies in proving that the rendering and training closed loop works properly; since the policy is still in the very early training stage after random initialization, unstable standing or flipping is normal in the video. Achieving the actual paper-level results requires long-term training, appropriate checkpoints, and subsequent play/record processes.

Ten. Normal Training Entry

If the smoke test passes, you can adjust the number of environments and the number of iterations back to the training scale:

bashcd "$HIMLOCO_ROOT"
./scripts/train_himloco_isaaclab_him.sh --num_envs 4096 --max_iterations 200000

Default logging writing:

textlogs/rsl_rl/himloco_a1_rough_him_isaaclab

If it is just continuing the debugging, it is recommended not to run 4096 environments from the start. A more reliable debugging step-by-step is:

bash./scripts/train_himloco_isaaclab_him.sh --num_envs 1 --max_iterations 1
./scripts/train_himloco_isaaclab_him.sh --num_envs 64 --max_iterations 2
./scripts/train_himloco_isaaclab_him.sh --num_envs 512 --max_iterations 10
./scripts/train_himloco_isaaclab_him.sh --num_envs 4096 --max_iterations 200000

At each level, check whether the observation shape is correct, whether there are NaN values in the loss, whether the GPU memory is stable, and whether the checkpoint is saved properly. This is easier to identify issues than simply running the training until completion.

11. Several boundaries in the migration implementation

Migration is not a line-by-line translation. When replicating, you should know which parts are strictly aligned and which parts are approximate engineering solutions.

Parts that must be aligned strictly:

• The policy observation dimension remains 270;
• The critic observation dimension remains 238;
• The actor input remains 64 = 45 + 3 + 16;
• A1 action scale, PD gains, dt, decimation, and episode length are aligned;
• PPO hyperparameters are mapped according to the original repository configuration;
• The velocity loss and swap loss of the estimator are retained.

Parts with approximate engineering:

• The external force event in Isaac Lab does not directly expose the disturbance buffer of the original legged_gym, so the 3 slots for external force in the privileged observation are kept as 0 to maintain a 238-dimensional layout;
• The original repository has special handling for termination privileged observations, and the Isaac Lab migration version uses next critic observation as an approximation;
• The terrain generator, USD assets, and contact names come from Isaac Lab, not the pixel-by-pixel equivalent scenarios of Isaac Gym Preview 4;
• The smoke test only verifies the training process, not the paper metrics or real robot performance.

These boundaries do not affect the core pipeline of learning HIMLoco, but for serious paper reproduction, further long-term training, multiple random seeds, official metric alignment, and real robot deployment verification are required.

XII. Common Errors and Troubleshooting

1. no kernel image is available for execution on the device

This is usually because the PyTorch CUDA wheel does not support the current GPU architecture. For Blackwell GPUs, you need to check:

bash"$PYTHON_BIN" - <<'PY'
import torch
print(torch.__version__, torch.version.cuda)
print(torch.cuda.get_device_name(0))
print(torch.cuda.get_arch_list())
PY

If sm_120 or compute_120 are not present in the list, the old stack is likely to fail to run. It is recommended to migrate to a wheel that supports Blackwell, such as PyTorch 2.7+ CUDA 12.8.

2. Isaac Gym cannot find libpython3.7m.so.1.0

This is a common issue in the old Isaac Gym Preview 4. You need to add the lib from the conda env to LD_LIBRARY_PATH:

bashexport LD_LIBRARY_PATH=/path/to/conda/env/lib:${LD_LIBRARY_PATH:-}

However, if everyone has already moved to the new stack of Isaac Lab, this issue usually is no longer a major problem.

3. Task registration failed

If the report cannot find Isaac-HIMLoco-Rough-Unitree-A1-v0, check first:

bashecho "$HIMLOCO_ROOT"
echo "$ISAACLAB_ROOT"
"$PYTHON_BIN" -m pip show himloco_isaaclab_tasks

Also confirm that the training script has added the following paths to sys.path:

textHIMLoco/source/himloco_isaaclab_tasks
IsaacLab/source/isaaclab
IsaacLab/source/isaaclab_tasks
IsaacLab/source/isaaclab_assets
IsaacLab/source/isaaclab_rl

4. The actor input dimension is not 64

If the actor input printed by the HIM runner is not 64, prioritize checking whether the policy observation remains 45 x 6 = 270. The default order of the observation history in Isaac Lab is different from that of the old repository. In the migration code, the history needs to be processed in order, and then the current frame’s 45-dimensional data is fed into the actor.

5. NaN occurs for ‘loss’

First, reduce the scale to:

bash./scripts/train_himloco_isaaclab_him.sh --num_envs 1 --max_iterations 1

After confirming that the observation shape, reward terms, and estimator loss are all normal, gradually increase num_envs. If NaN occurs in the 4096 environment from the start, the problem may stem from a certain reset, height scan, terrain, or batch dimension, and it is not easy to identify directly.

XIII. What should be mastered after completing this chapter

After completing this chapter, everyone should be able to describe the implementation of HIMLoco as a complete pipeline:

textUnitree A1 / rough terrain
  -> 并行仿真环境
  -> 45 维 one-step observation
  -> 6 帧历史堆叠成 270 维 policy observation
  -> 238 维 privileged critic observation
  -> HIMEstimator 从历史观测估计 velocity 和 latent
  -> Actor 使用 64 维输入输出 12 维动作
  -> Critic 使用 238 维 privileged observation 估计 value
  -> HIMPPO 同时更新 policy、critic 和 estimator
  -> checkpoint / TensorBoard / 后续 play 或部署

More importantly, it should be understood that the engineering core of such quadruped robot reinforcement learning projects is not just “running PPO”. What truly determines the quality of the project is:

• Which quantities in the observations can be obtained during deployment, and which can only be used during training;
• How historical observations compensate for unobservable states;
• How the privileged critic utilizes simulation true values;
• How domain randomization creates Sim2Real robustness;
• What each of smoke test, debug training, and full training verifies.

These ideas are applicable not only to HIMLoco, but also to many leg-and-foot robots, mobile robots, and simulation-to-reality migration projects.

References

• HIMLoco official project page: https://junfeng-long.github.io/HIMLoco/
• HIMLoco paper:Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response
• HIMLoco open-source repository: https://github.com/OpenRobotLab/HIMLoco
• Isaac Lab official repository: https://github.com/isaac-sim/IsaacLab
• Original legged_gym project: https://github.com/leggedrobotics/legged_gym