1. Why choose this method? Have you tried other Baselines? What was the comparison of results?

Example Answers and Analysis (using the SAC algorithm as an example):

In my robotic arm grasping project, I finally chose the Soft Actor-Critic (SAC) algorithm.

• Tried Baseline:

  1. DDPG (Deep Deterministic Policy Gradient): As a deterministic policy, its exploration ability is very poor in embodied continuous control, and it easily gets trapped in local optima.
  2. PPO (Proximal Policy Optimization): Although stable, as an on-policy algorithm, its sample efficiency is too low, making data collection costly in real robotic arm or high-fidelity physics engines (such as Isaac Gym).

• Reason for selecting SAC (combined with formula):

SAC introduces a Maximum Entropy reinforcement learning framework, where the objective function is optimized to maximize both the cumulative reward and the entropy of the policy:

• $s_t, a_t$: The state and actions at the moment $t$.

• $r(s_t, a_t)$: Reward function.

• $\alpha$: Temperature parameter, which determines the importance of the entropy term relative to the reward.

• $\mathcal{H}(\pi(\cdot | s_t))$: The entropy of the policy in a given state (i.e., the randomness of the action distribution).

• Effect Comparison: SAC (Off-policy) has nearly one order of magnitude higher sample efficiency than PPO. Additionally, due to the presence of entropy regularization, the robotic arm exhibits stronger robustness when facing external disturbances.

2. How was your reward function designed? Which versions were iterated?

Example Answers and Analysis:

My reward function has gone through three iterations, from sparse rewards to dense rewards that include a potential function.

• V1 version: Sparse Reward

Reward is only given when the task succeeds.

• $p_{\text{gripper}}$: Three-dimensional coordinates of the robotic arm end effector.

• $p_{\text{obj}}$: Three-dimensional coordinates of the target object.

• $d(\cdot, \cdot)$: Euclidean distance.

• $\epsilon$: Distance threshold for successful judgment.

• $\mathbb{I}(\cdot)$: Indicator function, 1 when the condition is satisfied, 0 otherwise.

• Problem: In complex high-dimensional action spaces, robotic arms can hardly obtain positive feedback through random exploration, leading to gradient vanishing.

• V2 version: Distance-based Dense Reward

  $$r_t = -w_1 \cdot ||p_{\text{gripper}} - p_{\text{obj}}||_2$$
  • $w_1$: Weight of the distance penalty.
  • Issue: The robotic arm approaches objects quickly, but it exhibits “Jittering” behavior near the objects, and cannot learn the lifting motion after grasping.

• V3 version: Reward Shaping (the final version with introduced potential and action penalties)

I introduced a potential-based reward shaping policy to ensure the invariance of the optimal policy, and added action penalties.

• $d_t$: The distance from the end of the motion to the object at $t$.
• $(d_{t-1} - d_t)$: Distance difference, encouraging the robotic arm to get closer to the target with each step than the previous one.
• $||a_t||_2^2$: L2 norm penalty for motion amplitude, used to reduce the energy consumption of the robotic arm and minimize rough, violent movements.

Python# V3 奖励函数伪代码示例
def compute_reward(state, action, next_state, success_threshold=0.05):
    # 解析状态变量
    gripper_pos = state['gripper_pos']
    obj_pos = state['obj_pos']
    next_gripper_pos = next_state['gripper_pos']
    next_obj_pos = next_state['obj_pos']

    # 1. 距离惩罚
    dist_current = np.linalg.norm(gripper_pos - obj_pos)
    dist_next = np.linalg.norm(next_gripper_pos - next_obj_pos)
    r_dist = -1.0 * dist_next

    # 2. 势能函数 (向目标靠近给予正反馈)
    r_approach = 5.0 * (dist_current - dist_next)

    # 3. 成功奖励
    r_success = 100.0 if dist_next < success_threshold else 0.0

    # 4. 动作惩罚 (平滑控制)
    r_action_penalty = -0.1 * np.linalg.norm(action)**2

    total_reward = r_dist + r_approach + r_success + r_action_penalty
    return total_reward

3. What do you think are the advantages and disadvantages of imitation learning (IL) and reinforcement learning (RL) in embodied tasks?

Example Answers and Analysis:

In embodied AI, IL and RL are two sides of the same coin.

Imitation Learning (e.g., Behavior Cloning)

• Core logic: Transform the control problem into a supervised learning problem.

  $$\arg\min_{\theta} \mathbb{E}_{(s, a) \sim \mathcal{D}_{\text{expert}}} \left[ -\log \pi_\theta(a|s) \right]$$
  • $\mathcal{D}_{\text{expert}}$: Dataset for expert human telemanipulation.
  • $\pi_\theta(a|s)$: Policy for model parameterization.

• Advantages: High sample efficiency. By directly learning human intuition, the model can quickly master complex semantic tasks (such as putting an apple in a drawer).

• Disadvantages: Covariate Shift. Once the $s'$ state during testing deviates from the expert distribution (e.g., the robotic arm is hit), the error will accumulate exponentially with time step $T$, leading to task failure. Additionally, the upper limit of IL performance is locked at the expert level.

Reinforcement Learning

• Core logic: Maximize long-term rewards through trial and error interactions with the environment.
• Advantages: It can detect “Emergent Behaviors” that surpass those of human experts, and is highly robust in underlying locomotion (e.g., quadruped robot gait).
• Disadvantages: The reward function design is difficult (Reward exploitation), and exploring from scratch (Tabula Rasa) in a physical world is extremely costly and may damage robot hardware.

Summary: The future trend is IL + RL (such as RLHF in embodied settings, or offline RL). First, use IL to pre-train a policy with common sense based on large-scale human data, and then use RL to fine-tune and align it in simulation or reality.

4. What is your opinion on the VLA (Vision-Language-Action) model? Which application scenarios do you think it is most suitable for?

Example Answers and Analysis:

I hold a highly optimistic view of the VLA model (such as Google’s RT-2 and RT-X series). It is the essential path for embodied AI to achieve General Purpose Robotics.

• The nature of VLA: It maps the robot’s actions $a$ into a Token space similar to that of language $l$ and vision $v$, essentially being a multimodal conditional generation model.

  $$a_t = \text{Decoder}(h_t) \quad \text{where} \quad h_t = \text{Transformer}(v_{\le t}, l, a_{<t})$$

• Most suitable scenarios:

  1. Open-vocabulary Manipulation: By leveraging the extensive pre-trained knowledge of the LLM, VLA can understand instructions it has never seen before (such as “give me that toy that looks like Thanos”), which is impossible for traditional RL.
  2. Long-horizon Task Planning: Combined with Chain-of-Thought (CoT), VLA models can directly integrate high-level logical reasoning and low-level action outputs in the end-to-end output.

• Current limitations: The inference delay is too high (usually greater than 100ms), making it unsuitable for dexterous hand manipulation that requires very high control frequencies (such as 500Hz) or high-dynamic tasks (such as table tennis).

5. The attention mechanism of Transformer, the principle of RoPE positional encoding, and the underlying logic of LoRA fine-tuning method.

Example Answers and Analysis:

(1) Attention Mechanism (Scaled Dot-Product Attention)

The underlying logic is to calculate the correlation weights between different Tokens in the input sequence, thereby aggregating information.

• $Q$ (Query): The vector of other features that need to be focused on currently.
• $K$ (Key): The vector in which other features are matched.
• $V$ (Value): The actual content vector of the feature.
• $\sqrt{d_k}$: Scaling factor ($d_k$ is the dimension of the Key), to prevent the dot product result from being too large and causing the Softmax gradient to vanish.

(2) RoPE (Rotary Position Embedding) Rotational Position Embedding

The underlying logic involves manipulating the rotation matrix at absolute positions, thereby cleverly implementing relative position encoding in the dot product attention.

For the feature vector $x = [x_1, x_2]^T$ at position $m$ (taking two dimensions as an example):

• $m$: Absolute position index of the token.

• $\theta$: The preset rotation base angle.

When the vectors $q_m$ and $k_n$ compute their inner product, the result is only related to their relative position $(m-n)$, which greatly enhances the model’s generalization ability to the length of extrapolation sequences.

(3) The underlying logic of LoRA (Low-Rank Adaptation)

The underlying logic is as follows: during fine-tuning of a pre-trained large model, the update amount of its parameter matrix $\Delta W$ has a very low “intrinsic rank”. Therefore, there is no need to fine-tune all the parameters; only the product of two low-rank matrices needs to be trained.

• $x \in \mathbb{R}^{d}$: Input vector.
• $W_0 \in \mathbb{R}^{d \times k}$: Frozen pre-trained weight matrix.
• $B \in \mathbb{R}^{d \times r}, A \in \mathbb{R}^{r \times k}$: Trainable low-rank matrix.
• $r$: Rank size, and $r \ll \min(d, k)$.

Python# LoRA 底层逻辑伪代码 (PyTorch 风格)
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, in_features, out_features, r=8, alpha=16):
        super().__init__()
        # 冻结的原始权重 (不参与梯度更新)
        self.pretrained_weight = nn.Linear(in_features, out_features, bias=False)
        self.pretrained_weight.weight.requires_grad = False

        # LoRA 的低秩矩阵 A 和 B
        self.lora_A = nn.Linear(in_features, r, bias=False)
        self.lora_B = nn.Linear(r, out_features, bias=False)

        # 缩放因子
        self.scaling = alpha / r

        # 初始化: A 为高斯分布，B 必须为 0 矩阵，保证初始状态下 Delta W = 0
        nn.init.normal_(self.lora_A.weight)
        nn.init.zeros_(self.lora_B.weight)

    def forward(self, x):
        # 原始输出 + 低秩通路输出 * 缩放因子
        return self.pretrained_weight(x) + self.lora_B(self.lora_A(x)) * self.scaling

6. The essential differences between VLA and World Model; the differences in the application of autoregressive (AR) and diffusion models (DM) in embodied control.

Example Answers and Analysis:

(1) The essential difference between VLA and World Model

• VLA is “Policy Model”: It addresses the issue of “what I see, what I should do”. The input is observation (visual/language), and the output is an action.

  $$\pi_{\theta}: (o_{\le t}, l) \rightarrow a_t$$

• World Model (world model) is “Dynamics Model”: It addresses the question of “what will the world be like if I do this”. It learns the state transition probabilities of the environment.

  $$\mathcal{M}_{\phi}: (s_t, a_t) \rightarrow (s_{t+1}, r_t)$$

Summary of Differences: VLA is used to directly control the robot; World Model resembles the “physical intuition” in the human brain, enabling “imagining and reasoning” within the brain. It is usually used in conjunction with planning algorithms (such as MPC) or as a simulator for RL.

(2) Differences between autoregressive (AR) and diffusion models (DM) in embodied control

• Autoregressive Model:

  Usually, it is necessary to discretize the continuous action space (Tokenization). It breaks down multi-dimensional actions into individual units and generates them sequentially.

  $$P(A|o) = \prod_{i=1}^{N} P(a^i | a^{<i}, o)$$

• $a^i$: The $i$th dimension of the action (such as the angle token of a joint in a robotic arm).

• Features: Good at introducing multi-modal semantics like language, but has a slow inference speed, and is prone to cumulative errors between dimensions during continuous control.

• Diffusion Model (Diffusion Model, such as Diffusion Policy):

Modeling is carried out directly in the continuous action space, and the action sequence is generated through a reverse diffusion process to remove noise step by step.

• $a_t^k$: The action after the $k$th step diffusion with noise.
• $k$: Number of diffusion steps.
• Features: Perfectly compatible with high-dimensional continuous control! It can accurately fit the multimodal distribution in human manipulation data (for example: avoiding obstacles can be done from the left or the right. Traditional MSE loss takes an average, leading to collisions with trees, while DM can precisely output either the left or the right direction).

7. The principles, advantages and disadvantages of mainstream algorithms such as PPO and SAC, as well as the improvements for specific tasks.

Answer Examples and Analysis:

(1) PPO (Proximal Policy Optimization)

• Principle: Based on the Actor-Critic architecture, a clipping mechanism is used to limit the magnitude of policy updates (Trust Region), ensuring monotonic increase and stability during training.

  $$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \left[ \min(r_t(\theta)\hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t) \right]$$
  • $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$: Probability ratio of old and new policies.
  • $\hat{A}_t$: Advantage function, which evaluates how much an action is better than the average.
  • $\epsilon$: Cutoff range (usually set to 0.2).

• Advantages and Disadvantages: The hyperparameters are highly robust and the convergence is stable; however, since it is an on-policy method, only data collected by the current policy can be used, resulting in very low sample efficiency.

• Improvement: In visual embodied tasks, Asymmetric Actor-Critic (non-symmetric architecture) can be used. The Critic input can be the privileged state in the simulator (such as precise object coordinates and mass), while the Actor input is only the camera image. This not only ensures feasibility for real-world deployment, but also greatly accelerates the convergence of the value network.

(2) SAC (Soft Actor-Critic)

• Principle: As described in Question 1, it is an off-policy algorithm based on maximum entropy. It not only rewards high performance but also ensures sufficient randomness in actions. Two Q networks (Double Q-learning) are used to take the minimum value to mitigate the over-estimation problem of Q values.
• Advantages and Disadvantages: It has extremely high sample efficiency, strong exploration ability, and good anti-interference capability; however, the hyperparameter $\alpha$ (temperature coefficient) is very difficult to tune, and it is sensitive to the state space dimension and Reward scale.
• Improvements: In robotic arm long-range tasks, the native SAC algorithm has difficulty in convergence. The improvement involves introducing Automatic Entropy Adjustment, which automatically updates $\alpha$ through gradient descent based on heuristic objectives, or combining Hindsight Experience Replay (HER) to address the buffer construction problem under sparse rewards.