Basics of Simulation and Robotics

This article introduces some basic concepts of simulation and robotics, helping to understand the technical details of ManiSkill more deeply. For example, it covers what a pose is, how quaternions are used, and what an Actor and Articulation are, etc.

Common Terms / Conventions

• Pose: The combination of the position and orientation of an object in three-dimensional space. The pose in ManiSkill/SAPIEN consists of a three-dimensional position and a four-dimensional quaternion.
• Quaternion : A mathematical tool commonly used to represent rotation/orientation, consisting of four values. ManiSkill/SAPIEN uses quaternions in the wxyz format. For more information on rotation representation in simulation, refer to the article and .
• Z axis is “up”: In ManiSkill/SAPIEN, the Z axis is considered the standard “upward” direction. Therefore, objects such as upright wine glasses typically have their long axis along the Z axis.

Simulation Objects

For rigid body simulation, which simulates objects that are physically difficult to deform (such as wooden blocks, computers, walls, etc.), ManiSkill/SAPIEN provides two main object types: Actor and Articulation.

During the simulation process, it usually starts with a reconfiguration step, where all objects are loaded into the simulation environment. After loading, we set the pose of all objects and initialize them.

Actor

An Actor is usually a “single” object. When subjected to external forces, the entire object moves together without deformation. For example, a baseball bat, a glass, or a wall can be considered Actors. An Actor has the following properties:

• Pose: The position and orientation of an object in three-dimensional space. The position is measured in meters.
• Linear velocity: The translation speed of an object along the x, y, and z axes, measured in meters per second.
• Angular velocity: The angular speed of an object along the x, y, and z axes, measured in radians per second.

In simulation, an Actor consists of two main elements: the collision shape and the visual shape.

Collision shape:

The collision shape defines the physical interaction of an object in simulation. An Actor can consist of multiple convex collision shapes.

It should be noted that Actors do not necessarily need to have a collision shape; they can be “ghost” objects used solely for visual indication, without participating in physical interactions.

Visual Shape:

Visual shape defines the appearance of an object in simulation for rendering and display, but it does not affect physical simulation.

Actor type: dynamic, kinematic, static

There are three types of Actor:

• Dynamic: These Actors are fully involved in the physical simulation. When external forces are applied, they respond according to physical laws.
• Kinematic: These Actors are partially involved in the physical simulation. When external forces are applied, they do not deform or move. However, dynamic objects interacting with these Actors are affected by reaction forces. Compared to Dynamic Actors, Kinematic Actors consume less CPU/GPU resources and run faster in simulation.
• Static: These Actors are similar to Kinematic Actors, but their pose cannot be changed after being loaded into the simulation environment. They consume less CPU/GPU resources and run faster in simulation. Walls, floors, cabinets, etc. are usually modeled as Kinematic or Static Actors because they typically do not move or break in reality.

Depending on the task to be simulated, some objects may need to be set as dynamic. For example, when simulating the task of moving a cup from the table to a shelf, the cup should be set as a dynamic Actor, while the shelf can be set as a kinematic or static Actor.

Articulation (Joint Mechanism)

Articulation consists of links and joints. In ManiSkill/SAPIEN, a joint connects every two links. Articulation is usually defined using XML or a tree structure to represent more complex joint mechanisms. For example, cabinets, refrigerators, and cars can all be considered as Articulations.

Links

The link is similar to an Actor, with the same properties, and can undergo physical simulation and manipulation. The difference is that the link must be connected to another link through a specific joint.

Joints (Joints)

There are mainly three types of joints: fixed joint, rotating joint, and sliding joint.

• Fixed Joint: Connects two links, keeping their relative position fixed. This is useful when defining Articulation, as the links connected by a fixed joint can be considered as a single unit.
• Revolute Joint: Similar to a hinge, where the connected links can rotate around the joint axis.
• Prismatic Joint: The connected links can slide along the joint axis.

Example

Consider an example of a cabinet. The cabinet has a base link, a top drawer link, and a bottom door link.

• The bottom drawer is connected to the base via a sliding joint, allowing the drawer to slide in a specific direction.
• The door panel is connected to the base via a rotating joint, enabling the door to rotate around the axis.

By understanding these basic concepts, you can more deeply participate in the development and application of ManiSkill, and design more complex and realistic simulation scenarios.

GPU Simulation

ManiSkill uses PhysX for physical simulation on the GPU. This approach differs from traditional simulation on the CPU, and the details are as follows. It is recommended to read this section to understand the basics of GPU simulation and the design principles of ManiSkill, which will help in writing more efficient code and building optimized GPU parallel tasks.

Scenarios and Sub-scenarios

Under the framework of GPU parallelization, thousands of tasks can be simulated simultaneously on the GPU. In ManiSkill/SAPIEN, this is achieved by placing all Actors and Articulation into a single PhysX scene, and creating a small working space called sub-scene for each task within that scene. 

The design of sub-scenes ensures that when reading data such as Actor pose, it is automatically preprocessed as data relative to the center of the sub-scene, rather than the entire PhysX scene. The following figure shows how 64 sub-scenes are organized. Note that the distance between each sub-scene is defined by the sim_config.spacing parameter in the simulation configuration, which can be set during task construction.

SAPIEN allows sub-scenes to be placed anywhere, but ManiSkill typically chooses a square layout with fixed spacing parameters. It should be noted that if an object in a sub-scene exceeds its working space, it may affect other sub-scenes. This is a common issue users encounter when simulating large scenes (such as houses or outdoor environments). For example, when the spacing parameter is set too low, objects in Sub-Scene 0 may interact with objects in Sub-Scene 1.

GPU Simulation Lifecycle

In ManiSkill, the Gym API is used to create, reset, and advance the environment. The env.reset process includes a one-time reconfiguration, followed by initialization steps: 

1. Reconfiguration: Load the objects (including Actors, Articulations, and light sources) into the scene, i.e., generate them in their initial pose. 
2. Call physx_system.gpu_init() to initialize all GPU memory buffers and set up the rendering groups required for parallel rendering. 
3. Initialize all Actors and Articulations (set pose, qpos values, etc.). 
4. Run physx_system.gpu_apply_* to save the data initialized in step 3 into the GPU buffer, preparing for simulation. 
5. Run physx_system.gpu_update_articulation_kinematics() to update the Articulation data (e.g., link poses) for acquisition. 
6. Run physx_system.gpu_fetch_* to update relevant GPU buffers and generate observation data. 

In the code, we save the physx_system variable as env.scene.px. 

env.step process includes repeatedly performing the following steps to handle actions and generate output: 

1. Obtain the user’s actions (and trim them as needed). 
2. Process the actions and convert them into control signals (such as joint positions/velocities) to control the Agent. 
3. Run physx_system.gpu_apply_articulation_target_position and physx_system.gpu_apply_articulation_target_velocity, and apply the objectives from step 2. 
4. Run physx_system.step() to advance the simulation. 
5. Run physx_system.gpu_fetch_* to update relevant GPU buffers and generate observation data. 
6. Return step data: observations, rewards, termination flag, truncation flag, and other information. 

Data Organization on GPU

In each sub-scene, the rigid body data of each rigid body Actor and Articulation link (including pose (7D), linear velocity (3D), and angular velocity (3D)) is tightly packaged in physx_system.cuda_rigid_body_data, forming a large matrix. The organization is as follows: 

plaintext[Actor 1 Data] [Actor 2 Data] ... [Articulation 1 Link 1 Data] [Articulation 1 Link 2 Data] ... [Articulation N Link M Data]



For users who plan to directly manipulate GPU buffers, understanding this organization might be helpful. Otherwise, if using the API provided by ManiSkill, these details will be handled automatically.

It is worth noting that this organization of the GPU buffer may not follow a intuitive structure (for example, each k rows represent data for a sub-scene), which is a trade-off made for better performance. For example, the following example shows the data organization when a PhysX scene contains 3 rigid body Actors (represented by red) and 3 Articulations with different number of links/degrees of freedom (DOF) (represented by green). SAPIEN will assign the number of rows for each Articulation to the highest degree of freedom in the entire PhysX scene.

ManiSkill Design Principles

Process everything in batches

ManiSkill is designed to support parallel simulation schemes for CPU and GPU. The reason is that for certain tasks, even in non-industrial settings, using GPU simulation may not be faster than using more CPU. Therefore, almost all the code in ManiSkill exposes data in batch form to users (batch dimension = number of parallel environments), and CPU simulations with a batch size of 1 are treated as a special case.

Management Objects and Views

ManiSkill can be regarded as the Python interface of SAPIEN.

Controller / Action Space

The controller is the interface between the user/policy and the robot. Whenever a step is taken in the environment and an action is provided, this action is sent to the selected controller, which converts the action into control signals for the robot. At the lowest level, all robots in simulation are controlled through joint position or joint velocity control, effectively specifying the position or speed that each joint should reach.

For example, the arm_pd_ee_delta_pose controller takes the relative motion of the end effector as input, and uses inverse kinematics to convert the input motion into the target positions of the robot joints. The robot uses PD controller to drive the motors to achieve the desired joint positions.

In ManiSkill, there are several key points to note regarding the controller:

• The controller defines the action space of the task.
• The robot can have independent controllers for different joint groups. The action space is a series of action spaces from all controllers.
• A single robot may have multiple sets of available controllers.

The following section will provide a detailed introduction to each pre-built controller and its functions.

Passive

pythonfrom mani_skill.agents.controllers import PassiveControllerConfig

This controller allows the forcibly specified joints to be uncontrolled by motion. For example, in the CartPole environment , the hinge joints of the CartPole robot are set to passive control (i.e., only sliding boxes are allowed to be controlled).

PD Joint Position (PD Joint Position)

pythonfrom mani_skill.agents.controllers import PDJointPosControllerConfig

Use the PD controller to control the position of joints via action control.

PD EE (End effector) Pose

pythonfrom mani_skill.agents.controllers import PDEEPoseControllerConfig

This controller has two variants: posture and position, allowing for more intuitive control of the robot’s end effector (or any link). The default settings of the controller are optimized for simplicity, but various options are also available.

To understand how it works, you first need to know about the following three related transformation coordinate systems:

1. World coordinate system
2. Root link coordinate system
3. End effector/link coordinate system

In the following figure, these coordinate systems are represented via the RGB axis, where red = X axis, green = Y axis, and blue = Z axis. The target link to be controlled is a virtual link representing the tool control point (TCP), which is a simple offset of the end effector link (hence its origin lies between the two grippers). Note that in ManiSkill/SAPIEN, the Z axis is typically considered the natural “up-down direction,” which differs from some other simulators.

In this controller, decoupled control for the translation and rotation of the end effector is implemented. This means that the motion used for translation will not affect the motion of the rotating end effector. This provides 6-dimensional control: 3 for 3D translation and 3 for rotation, as follows.

The controller provides two modes: incremental control and absolute control. In each environmental time step, given an action, ManiSkill calculates the new target pose of the end effector, and uses inverse kinematics to determine the joint movements that can best achieve this target pose. The configuration of the controller effectively changes the way the new target pose is calculated.

Incremental Control

This mode is enabled by default and configured via the use_delta property. It allows users to submit actions that define incremental end effector poses to move toward the target. There are 4 combinations of control frameworks, derived from two options: translation and rotation. The framework is defined by the frame property, and the naming rules are as follows:

python# 命名规则是 <frame>_translation:<frame>_rotation
# 以下是 4 种可能的框架组合
"body_translation:root_aligned_body_rotation",
"root_translation:root_aligned_body_rotation",
"body_translation:body_aligned_body_rotation",
"root_translation:body_aligned_body_rotation",
# 默认的框架组合
"root_translation:root_aligned_body_rotation"

Translation

In this controller, the user specifies the increments on the X, Y, and Z axes (in meters if not normalized), representing the distance moved in all these dimensions. Then, inverse kinematics is used to determine the joint movements required for the desired translation.

In ManiSkill, two position translation frameworks are defined: the root coordinate system and the solid coordinate system. By setting the corresponding dimensions to be greater than 0 and all other dimensions to 0 in the action, the following two types of translation can be achieved.

Rotation

For rotation, the user specifies the incremental rotations on the X, Y, and Z axes (in radians if not normalized), representing the angle of rotation in all these dimensions. They are processed as XYZ Euler angles and internally converted to quaternions. Then, inverse kinematics is used to determine the joint movements required to achieve the desired rotation.

ManiSkill implements two types of rotation control, which are generally considered the most intuitive and commonly used methods in real-world robots—that is, aligning/positioning the rotation of another coordinate system within a single coordinate system. Specifically, it supports two rotation frameworks: root-align body and body-align body. Root-align A means rotating within a framework that has the same direction as coordinate system A but the same position as coordinate system B. These two rotation methods are demonstrated by setting the corresponding dimensions in the setup actions to greater than 0, while the other dimensions are set to 0.

Observation Mode

Observation Space

Observation mode defines the observation space. All ManiSkill tasks will pass the observation mode (obs_mode) as an input parameter to gym.make(env_id, obs_mode=...). Typically, observations are organized into a dictionary (with an observation space of gym.spaces.Dict).

There are three original observation modes: state_dict (privileged state), sensor_data (original sensor data, such as unprocessed visual data), and state+sensor_data (both). state is a flattened version of state_dict. rgb+depth, rgb+depth+segmentation (or any combination of rgb, depth, and segmentation) and pointcloud post-process sensor_data to provide a convenient visual data representation. state_dict+rgb will return the unflattened privileged state and visual data, which can be mixed and matched with different modalities as needed.

The unprocessed shapes are described in detail below. Typically, the returned data always has batching dimensions, unless a CPU simulation is used and it is returned as a torch tensor. Additionally, we annotate the data types of certain values.

state_dict

Observation is a state dictionary. It usually contains privileged information, such as the object’s pose. The soft task does not support this mode.

• agent: Robot body perception (return value of the _get_obs_agent function of the task)
  • qpos: Current joint position, shaped as [nq]; nq is the number of degrees of freedom.
  • qvel: Current joint velocity, also shaped as [nq].
  • controller: Depends on the controller state of the used controller; usually an empty dictionary.
• extra: Dictionary of task-specific information, such as target position and end effector position. This is the return value of the _get_obs_extra function of the task.

state

state is a flattened version of state_dict. The observation space is gym.spaces.Box.

sensor_data

In addition to agent and extra, sensor_data and sensor_param have also been introduced. Currently, only camera-type sensors are available. The camera behavior may vary when using different shaders. The default shader is called minimal, which is the fastest and most memory-efficient option. The selected shader determines the data stored in this observation mode. The original data format under the minimal shader is described below. For detailed information on how to customize the sensor/camera, please refer to the sensors section.

• sensor_data: Data captured by sensors configured in the environment

  • {sensor_uid}：

    If the data comes from a camera sensor:

    • Color: Shape is [H, W, 4], and data type is torch.uint8 or np.uint8. RGB+Alpha values.
    • PositionSegmentation: Shape is [H, W, 4], and data type is torch.int16 or np.int16. The first three dimensions represent (x, y, z) coordinates in millimeters. The last dimension represents the segmentation ID. For more details, see Segmentation data section.

• sensor_param: Parameters for each sensor, depending on the sensor type

  • {sensor_uid}：

    If sensor_uid corresponds to the camera:

    • cam2world_gl: Shape is [4, 4], conversion from camera coordinate system to world coordinate system (OpenGL/Blender specification).
    • extrinsic_cv: Shape is [4, 4], camera external parameters (OpenCV specification).
    • intrinsic_cv: Shape is [3, 3], camera internal parameters (OpenCV specification).

rgb+depth+segmentation

By simply connecting the required image textures (rgb, depth, segmentation, albedo, normal, etc.) with +, various combinations can be requested in the observation mode. For example, the data format of rgb+depth+segmentation is the same as that of 's sensor_data mode , but all sensor data from the camera is replaced by the following structure:

• sensor_data：

  • {sensor_uid}：

    If the data comes from a camera sensor:

    • rgb: Shape is [H, W, 3], and data type is torch.uint8 or np.uint8. RGB image.
    • depth: Shape is [H, W, 1], and data type is torch.int16 or np.uint16. Unit is in millimeters. A value of 0 indicates invalid pixels (outside the camera’s maximum range of view).
    • segmentation: Shape is [H, W, 1], and data type is torch.int16 or np.uint16. For more details, please refer to Segmentation data section.

Note that these data are not scaled/normalized to [0, 1] or [-1, 1] to save memory. Therefore, if you plan to train on RGB, depth, or segmentation data, make sure to scale the data before training.

ManiSkill defaults to flexible support for various combinations of RGB, depth, and segmentation data, namely rgb, depth, segmentation, rgb+depth, rgb+depth+segmentation, rgb+segmentation, and depth+segmentation. Unselected image modalities will not be included in the observation, thus saving memory and GPU bandwidth.

Here are examples of RGBD images from two cameras:

---
alt: 来自两个相机的 RGBD 图像，显示了在 ReplicaCAD 数据集场景中 Fetch 机器人的视图。
---

In ManiSkill, the pointcloud observation mode provides a point cloud representation that integrates data from multiple cameras, making it suitable for tasks that require handling spatial geometric information.

• pointcloud:
  • xyzw: Shape is [N, 4]; data type is torch.float32 or np.float32. It integrates point cloud data from all cameras and is represented in homogeneous coordinates. w=0 represents points at infinity (beyond the camera’s maximum range), while w=1 represents other points. 
  • rgb: Shape is [N, 3]; data type is torch.uint8 or np.uint8. It contains color information of the corresponding point cloud. 
  • segmentation: Shape is [N, 1]; data type is torch.int16 or np.int16. It is the segmentation ID of each point, indicating the object or region to which the point belongs. For more details, refer to the Segmentation data section. 

Note that the point cloud data itself does not contain additional information, unless otherwise specified. 

Here is an example of point cloud data obtained using two cameras in the ReplicaCAD scene: 

---
alt: 来自两个相机的点云数据，显示了在 ReplicaCAD 数据集场景中 Fetch 机器人的视图。
---



To quickly demonstrate the visualization of point clouds, run the following command: 

bashpython -m mani_skill.examples.demo_vis_pcd -e "PushCube-v1"



This will display the point cloud data in the task environment “PushCube-v1”. 

Split Data

In ManiSkill, objects automatically receive a segmentation ID when loaded (i.e., the sapien.Entity object’s per_scene_id attribute). To obtain the Actor or Link information corresponding to each ID, run the following code: 

pythonimport gymnasium as gym
import mani_skill.envs
from mani_skill.utils.structs import Actor, Link

env = gym.make("PushCube-v1", obs_mode="rgbd")
for obj_id, obj in sorted(env.unwrapped.segmentation_id_map.items()):
    if isinstance(obj, Actor):
        print(f"{obj_id}: Actor, name - {obj.name}")
    elif isinstance(obj, Link):
        print(f"{obj_id}: Link, name - {obj.name}")



Note that ID 0 usually indicates a distant background. 

To quickly demonstrate the visualization of segmented data, run the following command: 

bashpython -m mani_skill.examples.demo_vis_segmentation -e "PushCube-v1"  # 绘制所有分割
python -m mani_skill.examples.demo_vis_segmentation -e "PushCube-v1" --id cube  # 仅显示名称为 "cube" 的对象



This will help understand and analyze the segmentation information of various objects in the environment. 

Sensors / Cameras

This page provides detailed instructions on how to use/customize the sensors and cameras of ManiSkill during runtime and task/environment definitions. In ManiSkill, a sensor is a “device” that can capture certain data modalities. Currently, only the camera sensor type is available.

Camera

The camera in ManiSkill can capture data of various different modalities/textures. By default, ManiSkill limits these to rgb, depth, position (used for deriving depth) and segmentation. The internal ManiSkill uses SAPIEN as a highly optimized rendering system, which renders different data modalities through shaders. The full configuration of the camera can be found in {py:class}mani_skill.sensors.camera.CameraConfig.

Each shader has a preset configuration that generates data for image styles. This data is often in a difficult-to-use format due to high optimization. ManiSkill uses Python’s shader configuration system to convert these different shaders into more user-friendly texture formats, such as rgb and depth. For more detailed information on which textures can be generated by which shaders, please refer to the shader and texture section.

Each ManiSkill environment has 3 types of cameras (although some categories may be empty): sensors for agent/policy observation, human_render_cameras for high-quality video capture of humans, and a single observation camera used by a GUI application to render the environment.

When creating an environment at runtime, you can pass runtime overrides for these cameras. For example, the following code changes the human rendering camera to use a ray-tracing shader for photorealistic rendering, modifies the width and height of the sensor camera to 320 and 240, and changes the field of view (FOV) value of the observation camera.

pythongym.make("PickCube-v1",
  sensor_configs=dict(width=320, height=240),
  human_render_camera_configs=dict(shader_pack="rt"),
  viewer_camera_configs=dict(fov=1),
)

These overlays will affect the category of each camera in the environment. Therefore, sensor_configs=dict(width=320, height=240) will change the width and height of each sensor camera, but it will not affect human rendering cameras or observation cameras.

To cover a specific camera, you can do it by the camera name. For example, if you want to cover the sensor camera named camera_0 with different width and height, you can do it like this:

pythongym.make("PickCube-v1",
  sensor_configs=dict(camera_0=dict(width=320, height=240)),
)

Now, all other sensor cameras will maintain the default width and height, while camera_0 will have the specified width and height.

These specific custom options are very useful for users who want to customize the rendering method or generate policy observations to meet their needs.

Shaders and Textures

Here are the shaders available in ManiSkill:

The following are the textures available in ManiSkill. Note that, unless specified otherwise, all data has not been scaled/normalized.

Reproducibility and Random Number Generation

Like many simulators, ManiSkill uses a Random Number Generator (RNG) to randomize everything in the environment, including object geometry, poses, and textures. These randomizations typically occur when the environment is reset.

However, randomization poses challenges for reproducibility. Especially when generating demonstrations on one computer and hoping to replay them on another machine, particularly in a GPU simulation/parallelization environment, usually a seed determines the randomization in each environment.

There are two ways to ensure the reproduction of the trajectory/demo: batch seed RNG and environment state. Typically, it is sufficient to ensure the reproduction of random numbers when the environment is reconfigured/loaded (determining which objects/textures to include in the environment) and during round initialization (determining the initial positions, speeds, etc. of the objects).

Ensuring Reproduction via RNG

To solve this problem, ManiSkill recommends using the _batched_episode_rng object in the environment, which is an instance of [BatchedRNG]. This batch RNG object is the same as [np.random.RandomState, but now all outputs have an additional batch dimension equal to the number of parallel environments. Additionally, the batch RNG object uses a seed list to initialize the seeds for each parallel environment. Now, regardless of the number of parallel environments, RNG generation is deterministic. If needed, _batched_episode_rng[i] will return the RNG object for the i-th environment so that sampling can be performed only for that environment.

When the user calls env.reset(seed=x), if x is an integer, we will use the seed x to initialize the first parallel environment. Other environments will generate random seeds based on x. For complete reproduction, the user can call env.reset(seed=[x1, x2, ..., xn]), and the seed list will determine the RNG for each parallel environment. If no seed is provided and env.reset() is called, a random seed will be generated for each parallel environment.

One disadvantage of this method is that RNG generation is slower than using the default torch/numpy RNG function to generate random numbers in batches (at present). Considering this, we recommend using batch rounds of RNG at least during environment reconfiguration (in ML workflows, environment reconfiguration does not occur frequently during training). Round initialization (occurring only once per round) can be handled using the environment state with the next method.

Ensuring Reproduction through Environmental Status

Environmental status includes the joint angles, joint velocities, poses, and speeds of all objects. It does not include details such as object textures, fixed camera poses, and robot controller stiffness.

Details such as the geometric shape/texture of objects are deterministic when randomized using the _batched_episode_rng object discussed above, given the same seed.

For other types of states included in the environment status, to ensure reproduction, these environment statuses can be saved and then set after creating the environment. Assuming the same seed is used to ensure all objects are identical, setting the environment status will ensure that the objects are in the correct positions.