Habitat-lab Simulation Basics

Habitat-Lab is an algorithm layer based on Habitat-Sim, encapsulating standardized tasks, evaluation metrics, and APIs. This allows researchers to focus on the design, training, and evaluation of embodied AI algorithms without having to deal with the underlying simulation details.

1. Introduction to Habitat-lab Features

1. Predefined embodied tasks

• Navigation types: PointGoalNav (point target navigation), ObjectNav (object target navigation), VLN (visual language navigation).
• Interaction types: Rearrange (object rearrangement), PickPlace (pick and place), OpenDoor (open door).
• Support for custom tasks (such as multi-agent collaboration, long-term planning).

2. Standardized Evaluation System

• Built-in core metrics (success rate SR, path length efficiency SPL, DTW alignment score, etc.) allow direct comparison of algorithm performance.
• Supports alignment with public benchmarks (such as Habitat Challenge), facilitating paper reproduction and result comparison.

3. Modular Agent Architecture

• Decouple “sensor → encoder → policy → controller”, enabling quick replacement of components (e.g., use CNN or Transformer for visual encoding, and RL/IL for decision-making);
• Deeply integrate PyTorch, supporting mainstream algorithms such as reinforcement learning (RL), imitation learning (IL), and end-to-end deep learning.

4. User-friendly API

• Provides a simple Python interface for one-click loading of scenarios, initialization of agents, and execution of simulation loops.

II. Habitat-lab Environment Setup

You can directly use the conda environment created by habitat-sim here.

pythonconda create -n habitat python=3.9 cmake=3.14.0
conda activate habitat

When downloading habitat-lab, make sure it is consistent with the habitat-sim version installed. For example, if the habitat-sim version used in this tutorial is 0.2.5, you need to download the 0.2.5 version of habitat-lab. When reproducing other papers, also ensure that the habitat-sim and habitat-lab versions are the same.

pythongit clone --branch v0.2.5 https://github.com/facebookresearch/habitat-lab.git
cd habitat-lab
pip install -e habitat-lab

Install habitat-baselines simultaneously

pythonpip install -e habitat-baselines

Download 3D scene data and point navigation data

pythonpython -m habitat_sim.utils.datasets_download --uids habitat_test_scenes --data-path data/

pythonpython -m habitat_sim.utils.datasets_download --uids habitat_test_pointnav_dataset --data-path data/

III. Initial Exploration of Habitat-lab and Basic Explanation of yaml Configuration

This section will focus on the interactive navigation code habitatlab_test.py provided in the habitat-lab project, and explain in detail the writing logic and loading process of the yaml configuration files in Habitat-Lab. It provides a basic approach for reading more complex yaml files in the future.

1. Basic and implementation of YAML configuration for habitatlab_test.py

Running process of habitatlab_test.py:

1. Load YAML configuration

2. Initialize the environment

3. Reset the environment to obtain observations

4. Key interaction control agent

5. Terminate the loop based on the maximum step number configured.

It can be seen that the most distinct part from Habitat-sim is the loading and configuration of YAML. After importing the YAML configuration, initialization still uses Habitat-sim. Below, we will provide a detailed explanation of the pointnav_habitat_test.yaml file.

python# @package _global_

defaults:
  - pointnav_base
  - /habitat/dataset/pointnav: habitat_test
  - _self_

habitat:
  environment:
    max_episode_steps: 500
  simulator:
    agents:
      main_agent:
        sim_sensors:
          rgb_sensor:
            width: 256
            height: 256
          depth_sensor:
            width: 256
            height: 256

YAML configuration input:

1. Load pointnav_base.yaml (basic PointNav configuration)
2. Overwrite /habitat/dataset/pointnav to habitat_test
3. Override the basic configuration with habitat.environment.max_episode_steps:500 and simulator.agents.main_agent.sim_sensors from the current file

Beginners who are using habitat-lab for the first time may be confused: Is that all the configuration is done? In fact, it isn’t. In the code, there are many default configurations. Ultimately, these are parsed by habitat-lab and passed to habitat-sim to create the environment and the agent. We will explain this in detail below to clear your doubts.

For specific YAML configuration items, please refer to: https://github.com/facebookresearch/habitat-lab/blob/main/habitat-lab/habitat/config/CONFIG_KEYS.md

The code related to the YAML configuration below refers to the functions in the default_structured_configs.py file in habitat-lab. You can refer to the source file for details.

Detailed Explanation of YAML Configuration:

1) Detailed Explanation of Dataset Configuration in yaml

pythonclass DatasetConfig(HabitatBaseConfig):
    type: str = "PointNav-v1"
    split: str = "train"
    scenes_dir: str = "data/scene_datasets"
    content_scenes: List[str] = field(default_factory=lambda: ["*"])
    data_path: str = (
        "data/datasets/pointnav/"
        "habitat-test-scenes/v1/{split}/{split}.json.gz"
    )
    # TODO: Make this field a structured dataclass.
    metadata: Optional[Any] = None

Explanation of default values for key configuration items:

• habitat.dataset.type: Default value is PointNav-v1, indicating the point navigation task dataset
• habitat.dataset.split: Default value is train, indicating the training set split
• habitat.dataset.scene_dir: Default value is data/scene_datasets, indicating the root directory of the scene dataset
• habitat.dataset.data_path: Default value is data/datasets/pointnav/habitat-test-scenes/v1/{split}/{split}.json.gz, indicating the dataset file path, where {split} will be replaced by the actual split value
• The dataset configuration also supports specifying the list of scenes to load through the content_scenes field. By default, all available scenes will be loaded

2) Detailed Explanation of Task Configuration in YAML

python@dataclass
class TaskConfig(HabitatBaseConfig):
    physics_target_sps: float = 60.0
    reward_measure: Optional[str] = None
    success_measure: Optional[str] = None
    success_reward: float = 2.5
    slack_reward: float = -0.01
    end_on_success: bool = False
    # NAVIGATION task
    type: str = "Nav-v0"
    # Temporary structure for sensors
    lab_sensors: Dict[str, LabSensorConfig] = field(default_factory=dict)
    measurements: Dict[str, MeasurementConfig] = field(default_factory=dict)
    # Measures to only construct in the first environment of the first rank for
    # vectorized environments.
    rank0_env0_measure_names: List[str] = field(
        default_factory=lambda: ["habitat_perf"]
    )
    # Measures to only record in the first rank for vectorized environments.
    rank0_measure_names: List[str] = field(default_factory=list)
    goal_sensor_uuid: str = "pointgoal"
    # REARRANGE task
    count_obj_collisions: bool = True
    settle_steps: int = 5
    constraint_violation_ends_episode: bool = True
    constraint_violation_drops_object: bool = False
    # Forced to regenerate the starts even if they are already cached
    force_regenerate: bool = False
    # Saves the generated starts to a cache if they are not already generated
    should_save_to_cache: bool = False
    object_in_hand_sample_prob: float = 0.167
    min_start_distance: float = 3.0
    gfx_replay_dir = "data/replays"
    render_target: bool = True
    # Spawn parameters
    filter_colliding_states: bool = True
    num_spawn_attempts: int = 200
    spawn_max_dist_to_obj: float = 2.0
    base_angle_noise: float = 0.523599
    spawn_max_dist_to_obj_delta: float = 0.02
    # Factor to shrink the receptacle sampling volume when predicates place
    # objects on top of receptacles.
    recep_place_shrink_factor: float = 0.8
    # EE sample parameters
    ee_sample_factor: float = 0.2
    ee_exclude_region: float = 0.0
    base_noise: float = 0.05
    spawn_region_scale: float = 0.2
    joint_max_impulse: float = -1.0
    desired_resting_position: List[float] = field(
        default_factory=lambda: [0.5, 0.0, 1.0]
    )
    use_marker_t: bool = True
    cache_robot_init: bool = False
    success_state: float = 0.0
    # Measurements for composite tasks.
    should_enforce_target_within_reach: bool = False
    # COMPOSITE task CONFIG
    task_spec_base_path: str = "habitat/task/rearrange/pddl/"
    task_spec: str = ""
    # PDDL domain params
    pddl_domain_def: str = "replica_cad"
    obj_succ_thresh: float = 0.3
    # Disable drop except for when the object is at its goal.
    enable_safe_drop: bool = False
    art_succ_thresh: float = 0.15
    robot_at_thresh: float = 2.0

    # The minimum distance between the agents at start. If < 0
    # there is no minimal distance
    min_distance_start_agents: float = -1.0
    actions: Dict[str, ActionConfig] = MISSING

Explanation of default values for key configuration items:

• habitat.task.type: Default value is Nav-v0, indicating the navigation task​
• habitat.task.physics_target_sps: Default value is 60.0, indicating the physics engine update frequency of 60 times per second​
• habitat.task.reward_measure: Default value is None, and the reward measurement metric must be specified in the specific configuration​
• habitat.task.success_measure: Default value is None, and the success measurement metric must be specified in the specific configuration​
• habitat.task.success_reward: Default value is 2.5, indicating the reward for successfully completing the task​
• habitat.task.slack_reward: Default value is -0.01, indicating the base reward (penalty) per step​
• habitat.task.end_on_success: Default value is False, indicating that the episode does not end automatically after success​
• The task configuration also includes many default values related to rearrangement tasks, such as settle_steps (default 5 steps), constraint_violation_ends_episode (default True), etc.

3) Detailed Explanation of Environment Configuration in YAML

pythonclass EnvironmentConfig(HabitatBaseConfig):
    max_episode_steps: int = 1000
    max_episode_seconds: int = 10000000
    iterator_options: IteratorOptionsConfig = IteratorOptionsConfig()

Explanation of default values for key configuration items:

• habitat.environment.max_episode_steps: The default value is 1000, indicating the maximum number of steps for an episode.​
• habitat.environment.max_episode_seconds: The default value is 10000000, indicating the maximum time limit for an episode.​
• habitat.environment.iterator_options: It contains the default configurations for multiple iterator options, such as the cycle mode being True by default, and the shuffle mode being True by default, etc.

4) Detailed Explanation of Simulator Configuration in YAML

pythonclass SimulatorConfig(HabitatBaseConfig):
    type: str = "Sim-v0"
    forward_step_size: float = 0.25  # in metres
    turn_angle: int = 10  # angle to rotate left or right in degrees
    create_renderer: bool = False
    requires_textures: bool = True
    # Sleep options
    auto_sleep: bool = False
    step_physics: bool = True
    concur_render: bool = False
    # If markers should be updated at every step:
    needs_markers: bool = True
    # If the articulated_agent camera positions should be updated at every step:
    update_articulated_agent: bool = True
    scene: str = "data/scene_datasets/habitat-test-scenes/van-gogh-room.glb"
    # The scene dataset to load in the metadatamediator,
    # should contain simulator.scene:
    scene_dataset: str = "default"
    # A list of directory or config paths to search in addition to the dataset
    # for object configs. should match the generated episodes for the task:
    additional_object_paths: List[str] = field(default_factory=list)
    # Use config.seed (can't reference Config.seed) or define via code
    # otherwise it leads to circular references:
    #
    seed: int = II("habitat.seed")
    default_agent_id: int = 0
    debug_render: bool = False
    debug_render_articulated_agent: bool = False
    kinematic_mode: bool = False
    # If False, will skip setting the semantic IDs of objects in
    # `rearrange_sim.py` (there is overhead to this operation so skip if not
    # using semantic information).
    should_setup_semantic_ids: bool = True
    # If in render mode a visualization of the rearrangement goal position
    # should also be displayed
    debug_render_goal: bool = True
    robot_joint_start_noise: float = 0.0
    # Rearrange agent setup
    ctrl_freq: float = 120.0
    ac_freq_ratio: int = 4
    load_objs: bool = True
    # Rearrange agent grasping
    hold_thresh: float = 0.15
    grasp_impulse: float = 10000.0
    # we assume agent(s) to be set explicitly
    agents: Dict[str, AgentConfig] = MISSING
    # agents_order specifies the order in which the agents
    # are stored on the habitat-sim side.
    # In other words, the order to return the observations and accept
    # the actions when using the environment API.
    # If the number of agents is greater than one,
    # then agents_order has to be set explicitly.
    agents_order: List[str] = MISSING

    # Simulator should use default navmesh settings from agent config
    default_agent_navmesh: bool = True
    # if default navmesh is used, should it include static objects
    navmesh_include_static_objects: bool = False

    habitat_sim_v0: HabitatSimV0Config = HabitatSimV0Config()
    # ep_info is added to the config in some rearrange tasks inside
    # merge_sim_episode_with_object_config
    ep_info: Optional[Any] = None
    # The offset id values for the object
    object_ids_start: int = 100
    # Configuration for rendering
    renderer: RendererConfig = RendererConfig()

Explanation of default values for key configuration items:

• habitat.simulator.type: Default value is Sim-v0​
• habitat.simulator.forward_step_size: Default value is 0.25 meters, controlling the step size of the forward motion​
• habitat.simulator.turn_angle: Default value is 10 degrees, controlling the angle of the rotation motion​
• habitat.simulator.create_renderer: Default value is False​
• habitat.simulator.requires_textures: Default value is True​
• habitat.simulator.lag_observations: Default value is 0​
• habitat.simulator.auto_sleep: Default value is False​
• habitat.simulator.step_physics: Default value is True

5) Detailed Explanation of Agent Configuration in YAML

pythonclass AgentConfig(HabitatBaseConfig):
    height: float = 1.5
    radius: float = 0.1
    max_climb: float = 0.2
    max_slope: float = 45.0
    grasp_managers: int = 1
    sim_sensors: Dict[str, Any] = field(default_factory=dict)
    is_set_start_state: bool = False
    start_position: List[float] = field(default_factory=lambda: [0, 0, 0])
    start_rotation: List[float] = field(default_factory=lambda: [0, 0, 0, 1])
    joint_start_noise: float = 0.1
    joint_that_can_control: Optional[List[int]] = None
    # Hard-code the robot joint start. `joint_start_noise` still applies.
    joint_start_override: Optional[List[float]] = None
    articulated_agent_urdf: Optional[str] = None
    articulated_agent_type: Optional[str] = None
    ik_arm_urdf: Optional[str] = None
    # File to motion data, used to play pre-recorded motions
    motion_data_path: str = ""
    auto_update_sensor_transform: bool = True

• habitat.simulator.agents.main_agent.height: Default value is 1.5 meters, defining the base height of the agent, which directly affects the observation angle of sensors (such as RGB and depth cameras). The default matches the human viewing height.
• habitat.simulator.agents.main_agent.radius: Default value is 0.1 meters, setting the collision radius of the agent for collision detection in the simulator, preventing the agent from penetrating scene objects.
• habitat.simulator.agents.main_agent.auto_update_sensor_transform: Default value is True, automatically updating the pose of sensors relative to the agent per frame; if turned off, manual updates are required, suitable for scenarios with custom sensor layouts.

Combining the above default configurations with pointnav_habitat_test.yaml and pointnav_base.yaml, a sample complete YAML file can be obtained as follows. (The actual code does not generate this YAML file during execution; it is just a sample YAML for reference.)

python# @package _global_
defaults:
  # 基础PointNav配置（包含核心默认逻辑，优先级低于后续配置）
  - pointnav_base
  # 引用habitat_test数据集预定义配置
  - /habitat/dataset/pointnav: habitat_test
  # 加载导航任务所需离散动作（前进、转向、停止等）
  - /habitat/task/actions:
    - move_forward
    - turn_left
    - turn_right
    - stop
    - look_up
    - look_down
  # 加载导航任务核心测量指标（步数、距离、成功率、SPL等）
  - /habitat/task/measurements:
    - num_steps
    - distance_to_goal
    - success
    - spl
    - distance_to_goal_reward
  # 加载代码依赖的传感器（GPS+罗盘+目标点传感器）
  - /habitat/task/lab_sensors:
    - pointgoal_with_gps_compass_sensor
    - compass_sensor
    - gps_sensor
  # 自身配置（优先级最高，覆盖前面的默认值）
  - _self_

# 核心配置主体
habitat:
  # 环境配置（用户定义500步上限，保留并补充默认项）
  environment:
    max_episode_steps: 500
    max_episode_seconds: 10000000  # 默认值，防止超时
    iterator_options:
      cycle: true  # 循环迭代episode
      shuffle: true  # 训练时打乱episode顺序（默认开启）

  # 数据集配置（补充habitat_test数据集默认参数，与预定义配置对齐）
  dataset:
    type: PointNav-v1  # 默认数据集类型，适配PointNav任务
    split: train  # 默认分割集（可在代码中动态修改为val/test）
    scenes_dir: data/scene_datasets  # 默认场景文件目录
    data_path: data/datasets/pointnav/habitat-test-scenes/v1/{split}/{split}.json.gz  # 默认数据路径
    content_scenes: []  # 空表示加载所有场景，可指定具体场景名过滤

  # 任务配置（补充导航任务默认参数，适配奖励和成功判定）
  task:
    type: Nav-v0  # 默认导航任务类型
    physics_target_sps: 60.0  # 物理引擎更新频率（默认60Hz）
    reward_measure: distance_to_goal_reward  # 奖励指标（基于距离变化）
    success_measure: success  # 成功判定指标
    success_distance: 0.2  # 成功阈值（距离目标<0.2米视为成功，与代码判定一致）
    end_on_success: false  # 成功后不自动结束episode（需手动按F触发stop动作）
    slack_reward: -0.01  # 每步基础惩罚（防止无意义徘徊，默认值）

  # 模拟器配置（保留用户传感器分辨率，补充默认物理和代理参数）
  simulator:
    type: Sim-v0  # 默认模拟器类型
    forward_step_size: 0.25  # 前进步长（默认0.25米）
    turn_angle: 10.0  # 转向角度（默认10度/次）
    auto_sleep: false  # 禁用自动休眠（保证实时交互）
    step_physics: true  # 启用物理步进（导航任务必需）
    requires_textures: true  # 启用纹理渲染（保证RGB图正常显示）
    agents:
      main_agent:
        height: 1.5  # 代理高度（默认1.5米，符合人类视角）
        radius: 0.1  # 代理碰撞半径（默认0.1米）
        sim_sensors:
          # RGB传感器（用户定义256x256，补充默认视场角等参数）
          rgb_sensor:
            width: 256
            height: 256
            hfov: 90.0  # 水平视场角（默认90度）
            sensor_type: COLOR
            position: [0.0, 1.5, 0.0]  # 传感器位置（与代理高度一致）
          # 深度传感器（用户定义256x256，补充默认参数）
          depth_sensor:
            width: 256
            height: 256
            hfov: 90.0
            sensor_type: DEPTH
            position: [0.0, 1.5, 0.0]
            min_depth: 0.0  # 最小探测深度
            max_depth: 10.0  # 最大探测深度（默认10米）

  # 补充传感器细化配置（与lab_sensors对应，确保观测正常输出）
  task:
    lab_sensors:
      pointgoal_with_gps_compass_sensor:
        type: PointGoalWithGPSCompassSensor  # 目标点+GPS+罗盘集成传感器
      compass_sensor:
        type: EpisodicCompassSensor  # 罗盘传感器（输出与初始朝向的角度差）
      gps_sensor:
        type: EpisodicGPSSensor  # GPS传感器（输出与初始位置的坐标差）

  # 测量指标细化配置（确保奖励和成功判定逻辑正确）
  task:
    measurements:
      distance_to_goal:
        distance_to: POINT  # 测量到目标点的距离（默认）
      success:
        success_distance: 0.2  # 与任务配置一致，确保判定统一
      spl:
        success_distance: 0.2  # SPL计算依赖的成功阈值

After successfully running habitatlab_test.py and understanding the basic principles of YAML configuration, you can skip the following sections. If you want to know about the execution process of the YAML file configuration import code, refer to the subsequent content to learn the basic approach to importing YAML files.

2. Detailed Explanation of the YAML File Configuration Running Process

1) Obtain the configuration file and merge the basic settings

The construction of the configuration object is completed by the get_config function, which is located in habitat-lab/habitat/config/default.py. The construction process is as follows:

pythondef get_config(
    config_path: str,
    overrides: Optional[List[str]] = None,
    configs_dir: str = _HABITAT_CFG_DIR,
) -> DictConfig:
    r"""Returns habitat config object composed of configs from yaml file (config_path) and overrides.

    :param config_path: path to the yaml config file.
    :param overrides: list of config overrides. For example, :py:`overrides=["habitat.seed=1"]`.
    :param configs_dir: path to the config files root directory (defaults to :ref:`_HABITAT_CFG_DIR`).
    :return: composed config object.
    """
    register_configs()
    config_path = get_full_config_path(config_path, configs_dir)
    # If get_config is called from different threads, Hydra might
    # get initialized twice leading to issues. This lock fixes it.
    with lock, initialize_config_dir(
        version_base=None,
        config_dir=osp.dirname(config_path),
    ):
        cfg = compose(
            config_name=osp.basename(config_path),
            overrides=overrides if overrides is not None else [],
        )

    return patch_config(cfg)

1. Path resolution: First, use the get_full_config_path function to parse the absolute path of the configuration file. This function checks whether the configuration path exists; if not, it searches within the configuration directory.
2. Configuration combination: Use the compose function to combine configurations. The parameters of the compose function include the configuration name (the basic name of the configuration file) and a list of override items.
3. Configuration patching: Before returning the configuration object, call the patch_config method to perform internal patching on the configuration, infer missing keys, and ensure that certain keys exist and are compatible with each other.

2) The config is passed to the Env class for initialization

After the get_config function, instantiation of the simulator, tasks, and sensors is completed in the habitat-lab/habitat/core/env.py file. The habitat.Env class is the core class of the Habitat Lab environment, integrating three main components: dataset, simulator, and tasks.

Detailed steps for the Env class initialization process:

1. Constructor parameter handling:

pythondef __init__(self, config: "DictConfig", dataset: Optional[Dataset[Episode]] = None) -> None:

• config parameter: Environment configuration, which should include the simulator ID and task name. These will be passed to the make_sim and make_task functions.

• dataset parameter: A reference to the dataset containing task instance-level information. Can be None

2. Configure preprocessing:

pythonif "habitat" in config:
    config = config.habitat
self._config = config

If the configuration contains the habitat key, the habitat sub-configuration is extracted as the actual configuration used.

3. Dataset initialization:

pythonself._dataset = dataset​
if self._dataset is None and config.dataset.type:​
    self._dataset = make_dataset(​
        id_dataset=config.dataset.type,​
        config=config.dataset​
    )

If no dataset is provided and the dataset type is specified in the configuration, use make_dataset to create the dataset.

4. Current episode and episode iterator initialization:

pythonself._current_episode = None
self._episode_iterator = None
self._episode_from_iter_on_reset = True
self._episode_force_changed = False

5. Episode iterator settings (if there is a dataset):

pythonif self._dataset:
    assert (
        len(self._dataset.episodes) > 0
    ), "dataset should have non-empty episodes list"
    self._setup_episode_iterator()
    self.current_episode = next(self.episode_iterator)
    with read_write(self._config):
        self._config.simulator.scene_dataset = (
            self.current_episode.scene_dataset_config
        )
        self._config.simulator.scene = self.current_episode.scene_id

    self.number_of_episodes = len(self.episodes)
else:
    self.number_of_episodes = None

• Call the setup_setup_episode_iterator method to set up the episode iterator
• Retrieve the first episode and set it as the current episode
• Update the scene dataset and scene ID in the configuration
• Set the total number of episodes

6. Simulator initialization:

pythonself._sim = make_sim(
    id_sim=self._config.simulator.type,
    config=self._config.simulator
)

Initialize the simulator using the simulator type and configuration in the configuration.

7. Task initialization:

pythonself._task = make_task(
    self._config.task.type,
    config=self._config.task,
    sim=self._sim,
    dataset=self._dataset,
)

Use the task type, task configuration, simulator instance, and dataset initialization tasks specified in the configuration.

8. Observe the space and action space settings:

pythonself.observation_space = spaces.Dict({
    **self._sim.sensor_suite.observation_spaces.spaces,
    **self._task.sensor_suite.observation_spaces.spaces,
})
self.action_space = self._task.action_space

The observation space is formed by combining the observation space of the simulator sensor kit and the task sensor kit, while the action space comes from the task.

9. Maximum steps and time limit settings:

pythonself._max_episode_seconds = self._config.environment.max_episode_seconds
self._max_episode_steps = self._config.environment.max_episode_steps
self._elapsed_steps = 0
self._episode_start_time: Optional[float] = None
self._episode_over = False

Read the maximum episode steps and time limit from the configuration, and initialize the relevant state variables.

In the above process, component instantiation driven by configuration is the core mechanism of Habitat Lab. Different types of environments can be created flexibly through configuration files.

The code in the above example shows that the simulator is instantiated as make_sim, the task is instantiated as make_task, and the sensor is instantiated with the following code and paths:

• Simulator instantiation

(habitat-lab/habitat/sims/registration.py)

pythondef make_sim(id_sim, **kwargs):
    logger.info("initializing sim {}".format(id_sim))
    _sim = registry.get_simulator(id_sim)
    assert _sim is not None, "Could not find simulator with name {}".format(id_sim)
    return _sim(**kwargs)

• Task instantiation

(habitat-lab/habitat/tasks/registration.py)

pythondef make_task(id_task, **kwargs):
    logger.info("Initializing task {}".format(id_task))
    _task = registry.get_task(id_task)
    assert _task is not None, "Could not find task with name {}".format(id_task)
    return _task(**kwargs)

• Sensor instantiation

(habitat-lab/habitat/sims/habitat_simulator/habitat_simulator.py)

The sensor instantiation has no function; it is defined directly in habitat_simulator.py and the habitat-sim environment is set up.

pythonsim_sensors = []
for agent_config in self.habitat_config.agents.values():
    for sensor_cfg in agent_config.sim_sensors.values():
        sensor_type = registry.get_sensor(sensor_cfg.type)
        assert sensor_type is not None, "invalid sensor type {}".format(sensor_cfg.type)
        sim_sensors.append(sensor_type(sensor_cfg))

Read the sensor configurations of each agent from the configuration, then use registry.get_sensor to obtain the sensor class, and finally create a sensor instance using the sensor configuration.

3) Set up the habitat-sim environment according to the configuration settings

After processing the yaml file as described earlier, use the final configuration to set up the habitat-sim environment.

1. Simulator configuration structure

pythonhabitat.simulator
├── type: "Sim-v0"  # 模拟器类型
├── scene_dataset: "data/scene_datasets/habitat-test-scenes"  # 场景数据集路径
├── scene: "van-gogh-room.glb"  # 当前场景
├── agents:  # 智能体配置
│   └── rgbd_agent:
│       ├── height: 1.5
│       ├── radius: 0.1
│       └── sim_sensors:  # 传感器配置
│           ├── rgb_sensor:
│           └── depth_sensor:
├── forward_step_size: 0.25  # 前进步长
└── turn_angle: 10  # 旋转角度

In the create_sim_config method of the HabitatSim class (habitat-lab/habitat/sims/habitat_simulator/habitat_simulator.py), the configuration is converted into the internal configuration format of Habitat-Sim.

pythondef create_sim_config(self, _sensor_suite: SensorSuite) -> habitat_sim.Configuration:
    sim_config = habitat_sim.SimulatorConfiguration()

    # 从Habitat Lab配置复制到Habitat-Sim配置
    overwrite_config(
        config_from=self.habitat_config.habitat_sim_v0,
        config_to=sim_config,
        ignore_keys={"gpu_gpu"},
    )

    sim_config.scene_dataset_config_file = self.habitat_config.scene_dataset
    sim_config.scene_id = self.habitat_config.scene

    # 配置智能体
    agent_config = habitat_sim.AgentConfiguration()
    overwrite_config(
        config_from=lab_agent_config,
        config_to=agent_config,
        # 这些键只在Hab-Lab中使用，忽略
        ignore_keys={
            "is_set_start_state", "sensors", "sim_sensors", "start_position",
            "start_rotation", "articulated_agent_urdf", "articulated_agent_type",
            ...
        },
    )

    # 配置传感器
    sensor_specifications = []
    for sensor in _sensor_suite.sensors.values():
        sim_sensor_cfg = sensor._get_default_spec()
        overwrite_config(
            config_from=sensor.config,
            config_to=sim_sensor_cfg,
            ignore_keys=sensor._config_ignore_keys,
            trans_dict={
                "sensor_model_type": lambda v: getattr(habitat_sim.FisheyeSensorModelType, v),
                "sensor_subtype": lambda v: getattr(habitat_sim.SensorSubType, v),
            },
        )
        sensor_specifications.append(sim_sensor_cfg)

    agent_config.sensor_specifications = sensor_specifications

    return habitat_sim.Configuration(sim_config, [agent_config])

2. Task configuration structure (PointNav task)

pythonhabitat.task
├── type: "Nav-v0"  # 任务类型
├── reward_measure: "distance_to_goal_reward"  # 奖励度量
├── success_measure: "spl"  # 成功度量
├── success_reward: 2.5  # 成功奖励
├── slack_reward: -0.01  # 失败惩罚
├── end_on_success: true  # 是否在成功时结束
└── lab_sensors:  # 实验室传感器
    └── pointgoal_with_gps_compass_sensor:
        ├── type: "PointGoalWithGPSCompassSensor"
        └── goal_format: "POLAR"

During the initialization of the NavigationTask class (habitat-lab/habitat/tasks/nav/nav.py), configurations are used to set various attributes of the task:

python@registry.register_task(name="Nav-v0")
class NavigationTask(EmbodiedTask):
    def __init__(
        self,
        config: "DictConfig",
        sim: Simulator,
        dataset: Optional[Dataset] = None,
    ) -> None:
        super().__init__(config=config, sim=sim, dataset=dataset)

    def overwrite_sim_config(self, config: Any, episode: Episode) -> Any:
        with read_write(config):
            config.simulator.scene = episode.scene_id
            if (
                episode.start_position is not None
                and episode.start_rotation is not None
            ):
                agent_config = get_agent_config(config.simulator)
                agent_config.start_position = episode.start_position
                agent_config.start_rotation = [
                    float(k) for k in episode.start_rotation
                ]
                agent_config.is_set_start_state = True
        return config

    def _check_episode_is_active(self, *args: Any, **kwargs: Any) -> bool:
        return not getattr(self, "is_stop_called", False)

3. Agent configuration structure

pythonhabitat.simulator.agents.rgbd_agent
├── height: 1.5  # 智能体高度
├── radius: 0.1  # 智能体半径
├── sim_sensors:  # 传感器配置
│   ├── rgb_sensor:
│   │   ├── type: "HabitatSimRGBSensor"
│   │   ├── width: 256
│   │   └── height: 256
│   └── depth_sensor:
│       ├── type: "HabitatSimDepthSensor"
│       ├── width: 256
│       └── height: 256
├── start_position: [0.0, 0.0, 0.0]  # 初始位置
└── start_rotation: [0.0, 0.0, 0.0, 1.0]  # 初始旋转（四元数）

In the create_sim_config method of the HabitatSim class (habitat-lab/habitat/sims/habitat_simulator/habitat_simulator.py), the agent configuration is converted into the internal format of Habitat-Sim:

pythondef create_sim_config(self, _sensor_suite: SensorSuite) -> habitat_sim.Configuration:
    # ... 其他配置处理 ...

    # 配置智能体
    agent_config = habitat_sim.AgentConfiguration()

    # 从Habitat Lab配置复制到Habitat-Sim配置
    overwrite_config(
        config_from=lab_agent_config,
        config_to=agent_config,
        ignore_keys={
            "is_set_start_state", "sensors", "sim_sensors", "start_position",
            "start_rotation", "articulated_agent_urdf", "articulated_agent_type",
            ...
        },
    )

    agent_config.sensor_specifications = sensor_specifications

    agent_config.action_space = {
        0: habitat_sim.ActionSpec("stop"),
        1: habitat_sim.ActionSpec(
            "move_forward",
            habitat_sim.ActuationSpec(
                amount=self.habitat_config.forward_step_size
            ),
        ),
        2: habitat_sim.ActionSpec(
            "turn_left",
            habitat_sim.ActuationSpec(
                amount=self.habitat_config.turn_angle
            ),
        ),
        3: habitat_sim.ActionSpec(
            "turn_right",
            habitat_sim.ActuationSpec(
                amount=self.habitat_config.turn_angle
            ),
        ),
    }

    # ... 传感器配置处理 ...

    return habitat_sim.Configuration(sim_config, [agent_config])

4. Sensor configuration structure

pythonhabitat.simulator.agents.rgbd_agent.sim_sensors.rgb_sensor
├── type: "HabitatSimRGBSensor"  # 传感器类型
├── width: 256  # 图像宽度
├── height: 256  # 图像高度
├── position: [0.0, 1.25, 0.0]  # 传感器位置
├── orientation: [0.0, 0.0, 0.0]  # 传感器方向
└── hfov: 90  # 水平视场角

In the implementation of the sensor class (habitat-lab/habitat/sims/habitat_simulator/habitat_simulator.py), configurations are used to set the properties of the sensors:

python@registry.register_sensor
class HabitatSimRGBSensor(RGBSensor, HabitatSimSensor):
    def __init__(self, config: DictConfig) -> None:
        super().__init__(config=config)
        self._width = config.width
        self._height = config.height
        self._hfov = config.hfov
        self._position = config.position
        self._orientation = config.orientation

In the sensor initialization process of the HabitatSim class (habitat-lab/habitat/sims/habitat_simulator/habitat_simulator.py), the sensor configuration is converted into the Habitat-Sim sensor specification:

pythonfor sensor in _sensor_suite.sensors.values():
    sim_sensor_cfg = sensor._get_default_spec()

    # 从Habitat Lab配置复制到Habitat-Sim传感器配置
    overwrite_config(
        config_from=sensor.config,
        config_to=sim_sensor_cfg,
        ignore_keys=sensor._config_ignore_keys,
        trans_dict={
            "sensor_model_type": lambda v: getattr(habitat_sim.FisheyeSensorModelType, v),
            "sensor_subtype": lambda v: getattr(habitat_sim.SensorSubType, v),
        },
    )

    sim_sensor_cfg.uuid = sensor.uuid
    sim_sensor_cfg.resolution = [sensor.observation_space.shape[1], sensor.observation_space.shape[0]]
    sim_sensor_cfg.sensor_type = sensor.sim_sensor_type
    sensor_specifications.append(sim_sensor_cfg)

After introducing the basic configuration of habitat-lab and its operation in code, it can be seen that the habitat-lab configuration system allows developers to flexibly create various complex embodied AI environments, enabling diverse applications ranging from simple navigation to complex manipulation tasks.

References:

https://github.com/facebookresearch/habitat-lab