Test Specification

This page defines the offline and rollout tests used before trusting a learned AR4 manipulation policy on hardware. The tests differ in one respect: what observation the policy receives after it predicts an action. That choice determines whether the test measures local imitation accuracy, short-horizon plan quality, or closed-loop task completion.

Teacher forcing and open-loop execution are diagnostic tests. Closed-loop rollout in simulation or on hardware is the deployment-relevant test. A policy can pass the first two and still fail the third.

Observation and Action

At control step t, the AR4 SmolVLA observation is

where:

• I_wrist and I_scene are RGB images from the wrist and fixed scene cameras.
• q_t is observation.state: six joint angles plus normalized gripper opening.
• ell is the natural-language instruction, held constant within the episode.

SmolVLA predicts an action chunk:

For the AR4 checkpoints discussed here, each action is an absolute target:

[joint_1, joint_2, joint_3, joint_4, joint_5, joint_6, gripper]

The default chunk horizon is H = 50, about 3.3 seconds at 15 Hz before inference latency.

Teacher Forcing

Teacher forcing evaluates the learned one-step mapping on recorded expert states. For a recorded episode, at each frame t:

1. Read the dataset observation o*_t.
2. Run the policy and take the first action from the predicted chunk: a_hat_t = pi_theta(o*_t)[0].
3. Compare a_hat_t with the recorded expert action a*_t.
4. Discard a_hat_t.
5. Advance to the next recorded observation o*_{t+1}.

The policy never sees the state that its own action would have created. Each prediction is made from a known-good dataset state. We log per-joint error, typically

Teacher forcing is useful for checking the model, data schema, normalization, image keys, and action layout. If teacher-forced error is large, deployment failure is not surprising. If it is small, deployment can still fail because the policy has not been tested on its own state distribution.

Open Loop

Open-loop execution tests the predicted plan without policy feedback. The procedure is:

1. Reset the arm and scene.
2. Capture one initial observation o_0.
3. Predict an action chunk A_0 = pi_theta(o_0).
4. Execute the chunk without giving the policy another observation.
5. Compare the planned and realized joint trajectories, and score the task outcome if relevant.

In notation:

with no new call to pi_theta for t > 0.

Open loop isolates short-horizon plan quality. If the trajectory diverges within one chunk, the predicted actions are already poor. If open loop is accurate but closed loop fails, the failure is not the first chunk alone; it is likely chunk-to-chunk drift, compounding, missing recovery behavior, or mismatch between the policy's visited states and the training distribution.

Closed Loop

Closed-loop evaluation executes policy actions and feeds the resulting live observation back into the policy:

This is the only mode that directly measures task completion under feedback. It also exposes the failure modes that teacher forcing hides: drift, missed grasps, delayed recovery, chunk-boundary discontinuities, stale observations, and termination problems.

The current AR4 SmolVLA inference node consumes SmolVLA's action chunks as a queue of absolute joint targets and publishes them to the joint trajectory controller. It does not route actions through MoveIt Servo in the current direct-inference path.

Plan, Realized Motion, and Feedback

The predicted chunk is a plan, not a test mode by itself. The mode is defined by how that plan is used.

In an open-loop comparison, the relevant curves are usually the policy plan and the realized joint trajectory. Their difference reflects controller tracking and dynamics. It is not feedback to the policy, because the policy is not called again.

Action Chunking and Replanning Cadence

SmolVLA predicts chunks of 50 actions. If all 50 actions are executed before the next policy call, the arm runs for about 3.3 seconds without visual feedback, plus inference latency. In practice, the measured cadence was about 3.9 seconds. The demonstrated pick-and-place sequence was 302 control steps, or seven chunks at n_action_steps = 50. A rollout can continue beyond that because the policy has no learned termination action.

The execution horizon can be shorter than the prediction horizon:

Reducing n_action_steps turns inference into receding-horizon control: predict 50 actions, execute the first k, observe again, then predict a new chunk. This is an inference-time change; the model still emits the same 50-action chunk. Smaller k reduces drift and chunk-boundary jumps, but it also increases the number of forward passes. With synchronous inference, too small a value can stall the controller. Async inference or real-time chunking is the cleaner solution.

Temporal ensembling is another smoothing option: average overlapping predictions from recent chunks rather than switching abruptly at each boundary.

AURA-739 Cadence Sweep

The AURA-739 sweep evaluated n_action_steps values of 5, 10, 25, and 50 for 300 control steps on the same checkpoint and cube. The main measurement was the discontinuity at chunk boundaries.

Executing the whole chunk (n_action_steps = 50) produced roughly five to six times the boundary jump of the shorter horizons. In this sweep, n_action_steps = 25 was the only setting that lifted the cube. The lower effective rate at n_action_steps = 5 shows the latency cost of replanning too often without an asynchronous execution path.

Run: aura-739-naction-steps-sweep.

On the strength of this result the inference backend now defaults n_action_steps to 25 rather than the smolvla_base value of 50, so the deployed replan cadence matches what the sweep found. The value is overridable per run through the --n-action-steps flag or the SMOLVLA_N_ACTION_STEPS environment variable.

Interpreting the Modes

The modes answer different questions and should not be substituted for each other.

For the AURA-739 checkpoint, teacher-forced and one-chunk open-loop behavior looked good, while closed-loop rollout diverged and missed the grasp. The failure was therefore not a broken one-step mapping. It was longer-horizon drift and grasp precision under the policy's own visited states.

Training Loss and Task Success

Training loss is not used as a capability claim. The mechanisms and corpus evidence are treated in Impact of Training Data. This specification uses loss only as a training diagnostic; rollout success is the evaluation target.

Reference Runs

Worked examples from AURA-739 use SmolVLA on AR4 with smolvla_ar4_moveit_single_v3. The W&B runs and Rerun recordings are: