Embodied AI Reasoning and Deployment Framework: From APXInf to Jetson-PI, Embodied.cpp

What problem does this chapter solve

The VLA/WAM model training is completed, but it does not mean the robot can operate stably. Real deployment requires handling model forward processing, flow/diffusion sampling, camera and robot state synchronization, action chunk scheduling, communication, IK, control cycle, data recording, and multi-robot resource allocation.

This chapter compares the items that are currently easy to mix together in one diagram:

• Edge inference Runtime: APXInf, Embodied.cpp, vla.cpp, Jetson-PI-Edge;
• Asynchronous control and delay hiding: Jetson-PI, OpenPI-RTC, EVA-Client;
• Real robot deployment and closed loop: EVA-Client;
• Multiple robots/cloud Serving: ROSA;
• Training, rollout, and evaluation infrastructure: RLinf;
• Model reference implementation and policy service: OpenPI;
• General NVIDIA deployment foundation: TensorRT, TensorRT-LLM, Isaac ROS.

They are not the same software, and no ranking based on the same metrics should be used. This chapter uses APXInf as a reproduction example, while the other projects are used to establish the selection criteria and the framework for reading papers.

1. First, stratify the “inference framework”

Robot systems usually follow this chain:

text训练 / 后训练
  RLinf、LeRobot、OpenPI 等
          ↓ checkpoint
模型与策略服务
  OpenPI、GR00T、StarVLA 或自定义 policy server
          ↓ observation / action chunk
推理与执行层
  APXInf、Embodied.cpp、vla.cpp、TensorRT
  Jetson-PI、OpenPI-RTC、EVA-Client
          ↓ action
机器人中间件与控制器
  ROS1/ROS2、ZeroMQ、IK、驱动、硬件

There are two concepts that are often confused:

1. Inference Runtime determines how the model forward process is executed, such as operators, quantization, memory, graph capture, and the backend.
2. Inference policy/execution client decides when to request the model, how to combine action chunks, how to compensate for latency, and finally how to send actions to the robot.

APXInf mainly belongs to Category 1; Jetson-PI and EVA-Client mainly address Category 2; RLinf is located at the upper level of training and evaluation.

2. Panoramic Comparison

Table 1 Positioning of Embodied AI Reasoning and Deployment Projects

How to Read the Paper Status

This table distinguishes between “formal publication” and “public preprint”. The official repository for Jetson-PI clearly states the acceptance information for CoRL 2026, so it is labeled as “official repository with accepted submission”; the links to Embodied.cpp, vla.cpp, EVA-Client, and ROSA are currently arXiv entries, and the tutorials do not indicate that they have been formally published in a conference or journal. APXInf is currently discussed as an open-source project and official technical introduction, without fabricating the source of the papers.

3. APXInf: Specialized for Optimizing NVIDIA Edge VLA

APXInf addresses the issue of “how existing models can run fast and stable on robot bodies.” The official open-source implementation is RLinf/APXinf-robo, where the ApxInf engine is introduced via Git submodule.

Its typical link is:

textPython API / OpenPI-compatible server
                 ↓
          Rust 系统运行时
                 ↓
  融合算子、CUTLASS/cuBLAS、CUDA Graph、量化
                 ↓
      Jetson Thor / Orin / RTX 4090

Figure 1: Official layering diagram of APXInf. The Python interface maintains the usability of model loading, preprocessing, and inference calls; the Rust Runtime is responsible for scheduling and resource lifecycle management; the underlying operator library is optimized for Jetson/RTX and embodied models.

Source: Unquestionable Core Dome APXInf Official Introduction .

The official launch focuses on π0.5, and supports BF16, FP8, and INT8. The delays provided by the official documentation for π0.5, batch=1, two-stream view, 10 flow steps, and action horizon=10 are as follows:

Figure 2: The optimized path provided by the official source. The reduction in latency comes from the combination of fusion operators, static graphs, CUDA Graph, quantization, Kernel optimization, and action generation pruning. It cannot be simply attributed to Rust or a specific CUDA Kernel.

Source: No Wenxinqiong APXInf Official Introduction .

APXInf Minimum Verification

The following process only verifies the installation and inference process of APXInf. There is no need to submit model weights to the tutorial repository. The command is executed in the root directory of the APXinf-robo repository or in a Linux NVIDIA environment:

bashgit clone --recursive https://github.com/RLinf/APXinf-robo.git
cd APXinf-robo

python3 -m venv .venv
source .venv/bin/activate
pip install maturin

CARGO_TARGET_DIR=target/wheel maturin build --release --features cuda \
  --auditwheel skip -m apxinf/crates/apxinf-py/Cargo.toml
pip install --force-reinstall target/wheel/wheels/apxinf_py-*.whl
pip install -e "./apxinf/python/apxinf[serving]" --config-settings editable_mode=strict
pip install -e ".[serve]"

First, perform the import check:

bashpython -c 'import apxinf_py; print(apxinf_py.__version__)'
python -c 'import apxinf_robo; print(apxinf_robo.__version__)'

Run a smoke test with random weights during execution:

bashpython scripts/bench_pi05.py \
  --random-weights \
  --precision bf16 \
  --layer l1 \
  --views 2 \
  --token-count 10 \
  --action-horizon 10 \
  --num-flow-steps 10 \
  --warmup 10 \
  --samples 30 \
  --autotune

Random weight only proves the availability of input shape, CUDA Kernel, graph capture, and timing links, but does not prove the model’s success rate. Real testing also requires compatible π0.5 checkpoint, norm_stats.json, LIBERO, and MuJoCo. The official documentation recommends running a single task and round first, before conducting the full LIBERO-10 test. APXInf LIBERO documentation

4. Embodied.cpp: A more “general Runtime” for embodied models

Embodied.cpp What the code attempts to solve is the fragmentation of the model and robot deployment stack: different VLA/WAMs use different Python code, backend, and robot glue code, resulting in the need to re-wire when switching models, devices, or ontologies.

The paper breaks down Runtime into five layers:

textInput adapters
      ↓
Sequence builders
      ↓
Backbone execution
      ↓
Head plugins
      ↓
Deployment adapters

Its research focus is on multi-rate execution, batch-1 latency-first inference, fusion execution, and heterogeneous device deployment. Compared to APXInf, its goal is more general: instead of merely optimizing π0.5 on certain NVIDIA chips, it aims to develop a portable C++ runtime that can support VLA/WAM.

The paper on Embodied.cpp is currently a arXiv technical report 2607.02501. This chapter does not consider it as a formally published conference paper. Its engineering potential and the status of the paper publication are two different things.

5. vla.cpp: Turning VLA into a lightweight engine in GGUF/ggml style

vla.cpp follows a similar approach to llama.cpp: converting the model into a self-contained GGUF file and executing it in C++/ggml Runtime, so that it does not rely on a complete Python/PyTorch environment during inference.

Its public repository covers various policies and backends, including π0/π0.5, SmolVLA, BitVLA, Evo-1, OpenVLA-OFT, GR00T, etc., and provides paths such as CUDA, CPU, Apple Silicon, SYCL/OpenVINO. It is suitable for answering the following questions:

Can a lightweight, portable, and packable Runtime be used to run multiple VLAs uniformly?

6. Jetson-PI: Not just making reasoning faster, but reducing robot waiting times

Jetson-PI’s core is not simply replacing PyTorch, but parallelizing action execution with the next inference. It introduces Foresight-Aligned Asynchronous Correction: predicting future environmental representations based on submitted actions, and generating actions at future moments to mitigate the observation-execution mismatch caused by asynchronous inference.

text当前 action chunk 执行 ───────────────→ 机器人控制器
          │
          └── 后台推理 → 未来校正 → 下一段 action chunk

It also incorporates confidence scheduling, CUDA Graph reuse, GPU-resident buffer, and flow unrolling. The official paper reports that on Jetson Orin, the control frequency is increased by 8.66× and 5.41× compared to plain PyTorch and vla.cpp. This result encompasses both algorithms, scheduling, and system optimizations, and cannot be directly used as a raw model speed for a specific runtime. Jetson-PI paper

The official repository for Jetson-PI is marked as Accepted at CoRL 2026. Here, the “acceptance information from the official repository” is written separately from the arXiv page to facilitate subsequent paper status updates.

7. EVA-Client: Real robot execution layer after model inference

EVA-Client should not be classified as a CUDA inference engine. It resides between the policy service and the robotic agent and is responsible for:

• ROS1, ROS2, ZeroMQ, and offline data playback;
• OpenPI, OpenPI-RTC, StarVLA, GR00T, and other policy backends;
• sync, async, naive, ACT ensemble, and RTC;
• action buffer, delay compensation, EEF/Joint action conversion, and IK;
• Debug, Collect, Eval, Replay;
• Record both the original policy output, the smoothed action, and the actual executed action.

This type of framework addresses the issue of “the model can output actions, but why does the real robot still pause, shake, misalign, or fail to review”. It is complementary to APXInf: APXInf can serve as the model inference backend, while EVA-Client is responsible for connecting the output to the robot and forming a data closed loop. EVA-Client Technical Report

8. ROSA: Moving from single-machine inference to robot cluster Serving

ROSA: A Robotics Foundation Model Serving System for Robot Factories This discusses another deployment model: there are many robots in the factory, but not all of them are equipped with a high-performance GPU.

text机器人集群
   ↓ 网络请求
共享 GPU Pool / Ray Serve
   ↓
多模型推理与机器人感知调度

It focuses on shared GPU pools, multi-model pipelines, task-level performance requirements, and fault handling. The goal is to optimize the productivity of the entire factory, rather than just improving the latency of a single robot. Its relationship with APXInf is a complement between edge and cloud: APXInf can be used at the robot level, while ROSA-like Serving systems can be used at the server side.

The public paper entry for ROSA is arXiv:2607.01088. This page is not currently equivalent to a formal conference or journal publication proof.

9. Where are RLinf, OpenPI, and TensorRT placed

RLinf: Training and rollout infrastructure

RLinf supports embodied reinforcement learning, simulation, rollout, distributed execution, and various model backends. It can call APXInf via APXinf-robo for π0.5 evaluation, but this does not mean that RLinf itself is a on-device inference engine.

A clear combination is:

textRLinf：训练 / 强化学习 / rollout / 评测
  ↓ checkpoint
APXInf 或 vla.cpp：模型端侧推理
  ↓ action chunk
EVA-Client 或自研客户端：异步执行、IK、记录和真机评测

OpenPI: Policy Reference Implementations

OpenPI provides policy services for models, data transformation, training, and implementation references of π0/π0.5. APXInf, vla.cpp, Jetson-PI, and EVA-Client can all interface with OpenPI at different levels. While OpenPI addresses “how models are defined and called,” it does not solve all the extreme deployment issues related to hardware.

TensorRT / Isaac ROS: Industrial Deployment Foundation

TensorRT, TensorRT-LLM, and Isaac ROS are not single runtimes specifically designed for VLA, but they are important in the NVIDIA robotics ecosystem. When using fixed models, fixed hardware, and a clear input shape, the TensorRT/NVFP4/FP8 approach may achieve very low latency; Isaac ROS provides hardware acceleration components and system integration on the ROS side.

This route and APXInf should not be simply chosen between the two: APXInf can integrate VLA sampling, action horizon, policy interface, and robot deployment constraints, while TensorRT is responsible for general graph optimization and hardware backend. Ultimately, testing should be conducted under the same model, input, flow steps, temperature, and statistical metrics.

10. Select a solution according to requirements

Table 2 Selection Suggestions

11. How should these systems be compared?

Don’t just compare a single “inference speed” number. At least, record them separately:

1. Runtime latency: The execution time of the model under fixed input and fixed sampling steps;
2. Policy latency: Including resize, tokenization, normalization, sampling, and unnormalization;
3. Transport latency: Inter-process communication and network round-trip time;
4. Control frequency: The actual frequency of actions being executed, as well as the effective frequency under asynchronous policies;
5. Task success: The success rate on the same checkpoint, seed, and task set;
6. Long-term stability: VRAM, memory, queue, threads, and exception recovery;
7. Migration cost: Whether rewriting the Runtime is required when switching models, hardware, or ontologies.

A 26 ms bare model result is not necessarily more suitable for real robots than a 40 ms system that can execute asynchronously continuously, supports failure recovery, and is traceable. Conversely, a deployed client cannot be called the underlying inference engine just because of its high control frequency.

12. Summary of this Chapter

Embodied AI reasoning infrastructure is emerging in several complementary directions:

text模型和训练：OpenPI、RLinf
        ↓
端侧 Runtime：APXInf、Embodied.cpp、vla.cpp、TensorRT
        ↓
异步控制：Jetson-PI、OpenPI-RTC
        ↓
真机闭环：EVA-Client、ROS/机器人控制器
        ↓
集群 Serving：ROSA

APXInf’s value lies in deepening the inference of embodied AI models on NVIDIA side; Embodied.cpp and vla.cpp focus on the general runtime; Jetson-PI addresses the time mismatch between inference and control; EVA-Client solves the issue of real robot execution and closed loop; ROSA extends the problem to robot clusters. Together, they illustrate that the “inference” of embodied AI is not just a Transformer forward, but a system issue composed of models, scheduling, hardware, and robot control.

References

1. APXinf-robo Official Repository
2. RLinf Official Repository
3. OpenPI Official Repository
4. Embodied.cpp Official Repository / arXiv:2607.02501
5. vla.cpp Official Repository / arXiv:2606.08094
6. Jetson-PI Official Repository / arXiv:2607.12659
7. EVA-Client Official Repository / arXiv:2607.02646
8. ROSA Official Project Page / arXiv:2607.01088
9. NVIDIA Isaac ROS