Impact of Training Data

Abstract

Vision-language-action policies are often discussed as if model architecture is the dominant determinant of robotic competence. In this study, we hold the model, optimizer, and training budget fixed and vary only the demonstration corpus used to fine-tune a compact VLA policy for a 6-DOF AR4 manipulator. Across closely related language-conditioned pick-and-place corpora, closed-loop atomic placement success ranges from 0% to 95%, despite similar final training loss. The results isolate three corpus effects with large consequences for closed-loop performance: deterministic trajectories induce a proprioceptive replay shortcut; thinly represented object-destination cells can degrade cells that are otherwise well covered; and beyond a modest per-cell floor, additional clean demonstrations saturate unless they add missing behaviors. A fourth set of experiments shows that corrective demonstrations can be harmful when their seed states are biased, even when the correction labels are correct. Balancing correction seeds removes the harm but does not, by itself, improve over the clean-demonstration baseline. Finally, held-out probes show that the recovered in-distribution success rate does not imply visual generalization: success collapses when the cube is displaced off the training grid or a distractor is introduced. These results argue for reporting corpus structure directly: task-cell coverage, trajectory ambiguity, correction-state coverage, and job-level evaluation should be treated as primary experimental variables rather than hidden details behind a total episode count.

1. Introduction

Generalist robot policies promise to reuse broad visual, linguistic, and motor priors across tasks and embodiments. In practice, however, a deployed policy is only as useful as its closed-loop behavior in the target environment. This creates a tension for small fine-tuning campaigns. The pretrained model may provide useful perception and language priors, but the final policy is still shaped by a limited, local set of demonstrations. If those demonstrations contain shortcut structure, missing behaviors, or uneven task coverage, the learned policy can fail even when training loss appears healthy.

We examine this issue in a controlled AR4 manipulation setting. A fine-tuned SmolVLA policy receives wrist and scene images, proprioception, and a natural-language instruction, then emits absolute joint targets for the arm and gripper. The task family is language-conditioned pick-and-place onto a shelf with multiple destinations. Demonstrations are generated by a scripted MoveIt expert in Gazebo and recorded in LeRobot format using realized joint configurations as action labels.

We ask how much closed-loop performance can move when the architecture is fixed and only the training corpus changes. In these experiments, the movement is large enough to determine whether the learned policy grounds its action in the scene and instruction or follows a memorized joint trajectory. The claim is therefore not broad robotic generalization, but a controlled account of why a local fine-tuning run succeeds or fails.

2. Experimental Setting

All experiments fine-tune smolvla_base, a 450M-parameter VLA with a frozen vision encoder, for 50k-100k training steps using LeRobot 0.5.2. The robot is a 6-DOF AR4 arm in Gazebo. Each atomic rollout spawns one cube, issues one language instruction, and scores whether the object ends at the specified shelf or cubby. Unless otherwise stated, evaluation uses 40 closed-loop rollouts, ten per destination, on a fixed grid.

The action representation is important. Each action is a realized 7-D configuration:

[joint_1, joint_2, joint_3, joint_4, joint_5, joint_6, gripper]

This convention makes human teleoperation and scripted MoveIt demonstrations schema-compatible. It also makes shortcut learning plausible. If the current joint state almost determines the future joint path, a behavior-cloned policy can reduce loss without grounding the action in the image or the instruction.

We distinguish three levels of task:

The reported success rates are atomic placement rates. They should not be read as job completion rates. If atomic success is p, a k-object job scales roughly as p^k in the absence of recovery. Thus 78% atomic success corresponds to approximately 22% success for a six-object job under an independence approximation. This compounding motivates job-level evaluation, but the experiments in this paper remain at the atomic level.

3. Results

3.1 Training loss is not a capability measure

Three checkpoints with similar final losses, approximately 0.009-0.017, produced 0%, 95%, and 45% closed-loop atomic success. In another run, success decreased from 38% at 40k steps to 28% at 100k steps while optimization continued. These results are consistent with a standard exposure-bias argument: behavior cloning optimizes one-step action reconstruction on expert states, whereas deployment evaluates sequential closed-loop behavior on states induced by the policy.

We therefore use training loss only as a coarse diagnostic. Model selection is based on fixed closed-loop rollouts with per-instruction or per-cell scoring.

3.2 Deterministic trajectories induce proprioceptive replay

The first corpus contained 63 near-identical demonstrations. The resulting policy reproduced expert actions under teacher forcing but achieved 0% closed-loop success. A diagnostic probe held the arm at a fixed mid-descent state while moving only the cube. The predicted grasp point moved by about 3 mm while the cube moved by 40-50 mm, implying a visual tracking gain near 0.05. Adding cube coordinates to the instruction did not change the prediction. The policy had learned to replay a joint trajectory.

A regenerated corpus changed the trajectory distribution without changing the model. It added uniform spatial coverage, randomized approach heights, recorded via-waypoints, and substantial pacing variation. A nearest-neighbor future-divergence measure rose from 2.0x to 6.0x at three seconds, meaning that similar proprioceptive states no longer implied the same future action. The same architecture trained on this corpus reached 95% success on the earlier tabletop setup.

The relevant change was not only corpus size. It was the conditional structure of the data: after regeneration, proprioception alone no longer predicted the demonstrated future, so low loss required information from pixels and language.

3.3 Thin task cells can create negative transfer

The next experiments use a Furinno shelf twin with four destinations, including tight cubbies. A single-object, four-destination corpus scored 45%. Merging it with a multi-object corpus produced 468 total episodes across 24 object-destination cells, but many cells had only 4-16 episodes. This merge reduced success to 38% at 40k steps and 28% at 100k steps, with one instruction failing completely.

The failure was not explained by obvious dataset corruption. An alignment audit found no task-episode mismatches, and a language probe still produced accurate grasp plans for all four instructions. The control experiment restricted the merged corpus to its 275 single-object episodes, removing the thin multi-object fragments while preserving the rest of the setup. Success recovered to 75%.

This is a small-scale instance of mixture interference. Underrepresented task fragments were not ignored by the optimizer; they changed the learned behavior on better-covered fragments. The relevant budgeting unit is therefore the task cell rather than the aggregate corpus.

3.4 Balanced volume saturates before missing behaviors are learned

After removing the thin fragments, the balanced single-object corpus was expanded from 275 to 481 episodes, or roughly 69 to 120 episodes per destination. Success changed from 75% to 78%, within the noise expected from 40 rollouts. The added demonstrations were clean successes, and they did not contain the behaviors that dominated the residual failures: recovery from missed grasps, mid-carry drops, and object nudges.

This is saturation of a particular data distribution, not evidence that the task is saturated. Additional demonstrations should help when they cover new states or behaviors; additional clean successes from the same distribution did not teach recovery from states outside that distribution.

3.5 Corrective demonstrations depend on seed-state coverage

Corrective data was introduced to address the residual failures. Two initial strategies were tested. The first scripted a deliberate miss followed by a recovery, which placed the bad action itself into the labels. The second followed a DAgger-style reset: seed the scene from harvested policy-failure states and record only the expert completion.

Both reduced performance. The clean 481-episode baseline scored 78%. The scripted-miss corpus scored 62%, with the nudge-and-drop failure class increasing. The DAgger-style corpus scored 45%. It avoided the miss-imitation failure but introduced a spatial grasp bias: success was 12/16 when the cube started at x >= 0.41 and 6/24 when x <= 0.39.

The source of the bias was the seed distribution. The harvested failures used for correction were concentrated in far and elevated workspace regions. Adding them shifted the policy's grasp distribution toward those regions and degraded near-table grasping. The labels were correct; the state coverage was not.

When the same failure seeds were bucketed by shelf, near/far grasp region, and failure type, and then sampled uniformly, the damage disappeared. The uniformly seeded DAgger corpus scored 78%, matching the no-correction baseline, and near-table grasping recovered from 6/24 to 19/24. The result does not establish a gain from corrective data over the baseline. It establishes a precondition for testing such a gain: the correction seeds must cover the relevant failure-state distribution rather than reproduce the geometry of the first harvested failures.

3.6 In-distribution success did not imply visual generalization

The recovered 78% policy was tested on held-out scene changes. A same-session control on the trained purple-cube grid scored 12/20. Moving the cube off the trained grid dropped success to 3/20. Adding a second object while leaving the purple target at a trained pose also scored 3/20. Unseen color and shape conditions scored 3/20 and 1/20, respectively, although those two conditions were confounded by a parked purple object in view.

The two clean conditions qualify the in-distribution result. The policy was competent on the trained manifold but fragile to object displacement and distractors. Its dominant failure was not wrong placement but failure to acquire the object, consistent with a policy that still partly aims at remembered coordinates rather than tracking the seen object.

4. Discussion

For small-robot VLA fine-tuning, the dataset is better described by its structured coverage than by its total number of episodes. Four properties mattered in these experiments:

1. Trajectory ambiguity. Demonstrations must prevent the current joint state from predicting the future action by itself.
2. Cell coverage. Each object-destination condition needs enough support, or the mixture can degrade.
3. Behavior coverage. Clean demonstrations do not teach recovery from states they never visit.
4. Seed-state coverage. Corrective data inherits the geometry of the failures from which it is sampled.

These properties also change how multi-object shelving should be measured. Equal episodes per (object, shelf) pair are not enough. Two cells with the same count can differ in source-zone coverage, grasp pose families, approach paths, shelf-entry geometries, and failure modes. A useful sampler should balance a finer factorization:

object x shelf x source-zone x grasp-pose-family x approach-family x shelf-entry-family

DAgger data needs the same treatment. Harvested failures should be bucketed and sampled across relevant failure factors rather than replayed in proportion to their observed frequency; otherwise the correction set can reinforce the policy's existing failure geometry.

Finally, the action representation may limit transfer. This system fine-tunes a pretrained SmolVLA checkpoint whose prior is heavily shaped by SO-100/SO-101-style arms, while inference on AR4 emits absolute joint targets. Absolute joint space is not embodiment-neutral. A future ablation should compare the current fine-tune to a model with the action head reinitialized and to a model that predicts end-effector deltas through a robot-specific MoveIt or IK adapter.

5. Limitations

The study is intentionally narrow. It evaluates atomic single-object placement, not full multi-object job completion. It uses simulation, scripted demonstrations, one robot, and a small number of corpora. Several comparisons are controlled only within the Furinno shelf world; the earlier 0%-to-95% result used a different tabletop shelf setup and should not be treated as part of a single monotonic scaling curve.

The correction-data experiment also has a narrow interpretation. Balanced DAgger recovered baseline performance but did not exceed it. The supported claim is that biased correction seeds can make corrective data harmful; whether balanced corrections improve the baseline remains open.

The held-out generalization probe is diagnostic rather than comprehensive. It shows fragility to displacement and distractors, but the color and shape conditions need to be rerun without a parked target-colored object in view before object-appearance generalization can be isolated.

6. Conclusion

For this AR4 SmolVLA fine-tuning campaign, demonstration composition dominated closed-loop atomic success. Deterministic demonstrations produced a policy that replayed joints. Trajectory diversity forced visual and language grounding and recovered high success in the simpler world. In the harder Furinno setting, thin task cells caused mixture interference, balanced clean volume saturated, and corrective demonstrations were safe only when seeded from a balanced state distribution. The recovered policy remained narrow, failing under modest scene changes.

VLA fine-tuning reports should therefore include more than model size, training loss, and episode count. Cell coverage, trajectory ambiguity, correction-state sampling, held-out scene perturbations, and job-level completion are part of the experimental result. Without those measurements, atomic benchmark success can mask failure on the task the robot is meant to perform.

Evidence and Artifacts

Dataset artifacts with full provenance and ground-truth sidecars: v5, v6, v7, and v4. Evaluation modes and protocol definitions are in Test Specification.