SAIL: Test-Time Scaling through Iterative Refinement
for VLMs as Robot Trajectory Generators
Robot control with foundation models
What does a robot need before it can tackle a new task? A common starting point is to collect demonstrations and train a policy on the resulting trajectories. Foundation models, however, have already learned from images, video, code, language and robotics-related data. The question, then, is how much of that knowledge we can draw out for robot control without updating the model's weights.
In fact, GPT-6 Astra, in addition to its general capabilities in language and vision, has demonstrated the ability to perform multiple manipulation tasks on a physical robot by specifying motion targets from camera images and robot state [Robocurve, 2026]. Earlier work uses tokenized visual observations and a few demonstrations to generate trajectories through in-context learning with a language model [Di Palo and Johns, 2024]. The figure below illustrates direct trajectory generation with a VLM or LLM.
Context
VLM/LLM
Generates a trajectory
Generated trajectory
Position and orientation
Gripper state
Limits of trajectory generation with foundation models and our approach (SAIL)
However, a foundation model does not necessarily produce a reliable robot trajectory in a single generation. Predictions depend on the context provided, such as the demonstrations and their relevance to the current scene, and can also vary with sampling. Even when the intended action is appropriate, small errors in the motion targets generated by the model can cause failure. Executing a single prediction without evaluation leaves no opportunity to correct such errors before the robot moves.
Language models face a similar limitation, and training is not the only way to address it. Test-time scaling can improve accuracy beyond a single prediction by generating, evaluating and revising candidates at inference time without updating the weights. We ask whether the same trade between computation and accuracy extends to physical action.
We propose SAIL (Scaling In-Context Imitation Learning), a method for more reliable VLM-based robot trajectory generation through test-time scaling. SAIL uses a policy VLM to generate candidate trajectories from a few successful demonstrations provided in context. SAIL executes the trajectories in simulation, evaluates task progress with an evaluation VLM and returns step-level feedback to guide refinement through Monte Carlo Tree Search (MCTS). Candidate testing takes place in simulation, and only the selected trajectory is sent to the physical robot.
We then test how far this framework can improve success rates by increasing computation at inference time. Across six manipulation tasks in simulation, increasing the search budget from one node to 45 nodes raised the average rate of finding a successful trajectory from 25% to 73%. We also evaluated SAIL on a physical robot. In all experiments, we used Gemini Robotics-ER 1.5 as both the policy and evaluation VLM without updating its weights.
A single generated trajectory can fail
Test-time scaling through refinement in simulation
How SAIL refines trajectories
SAIL uses MCTS to combine refinement of promising trajectories with exploration of alternatives. Three components support the search. (1) Generation from similar demonstrations. Each node represents a complete trajectory generated by a VLM through in-context learning with retrieved demonstrations. (2) Evaluation of task completion. The trajectory is executed in simulation, and an evaluation VLM estimates task completion from the resulting video. (3) Refinement with step-level feedback. The parent trajectory, annotated with progress scores at selected waypoints, is passed to the generating VLM, which produces a revised trajectory as a child node. Each edge represents this refinement.
1. Generation from similar demonstrations
An archive stores successful trajectories. The system retrieves demonstrations from scenes that look similar to the current scene. The generating VLM receives these demonstrations together with the current object configuration and robot state, then generates a complete trajectory. Successful trajectories found during search are added to the archive for later use.
2. Evaluation of task completion
The trajectory generated in step 1 is executed in simulation, producing a video of its execution. An evaluation VLM estimates how much of the task has been completed by assessing progress through predefined subtasks in this video. Completion estimates across the execution are combined into a trajectory score, allowing comparison even when neither candidate finishes the task. This evaluation builds on ROVER's approach to scoring task progress over time [Schroeder et al., 2025].
3. Refinement with step-level feedback
Completion scores from sampled video frames are aligned with selected trajectory waypoints and returned to the generating VLM. The feedback prompts it to preserve successful segments and revise those where progress stalls. The revised trajectory forms a child node, where it is evaluated again in simulation to guide further search.
Success rates increase with the search budget
We evaluated six manipulation tasks in the ALOHA simulator, with 20 initial configurations per task. Increasing the search budget raised the average success rate from 25% with a single generated trajectory to 73% with a budget of 45 candidate trajectories (up to 45 search nodes).
Here, the success rate measures how often the search finds a trajectory that passes the simulator's ground truth success check within the search budget. This binary check is separate from the VLM estimates of task completion used to guide search.
| Method | Nodes | Banana | Pen | Bowl | Drawer | Laptop | Marker | Average | ||
|---|---|---|---|---|---|---|---|---|---|---|
| Baselines | ||||||||||
| Single generation | 1 | 40 | 40 | 40 | 10 | 15 | 5 | 25 | ||
| Breadth first search | 15 | 85 | 50 | 60 | 35 | 60 | 15 | 51 | ||
| Depth first search | 15 | 45 | 50 | 80 | 5 | 30 | 10 | 37 | ||
| Our method (SAIL) | ||||||||||
| Single generation (repeated) | More nodes | 1 | 40 | 40 | 40 | 10 | 15 | 5 | 25 | Higher success |
| SAIL | 6 | 80 | 55 | 100 | 20 | 50 | 25 | 55 | ||
| SAIL | 15 | 90 | 70 | 100 | 40 | 50 | 40 | 65 | ||
| SAIL | 30 | 95 | 80 | 100 | 50 | 55 | 45 | 71 | ||
| SAIL | 45 | 95 | 80 | 100 | 50 | 70 | 45 | 73 | ||
Average success rates increased across the tested budgets, reaching 55%, 65%, 71% and 73% with 6, 15, 30 and 45 nodes, respectively. These results demonstrate test-time scaling through search and refinement, improving success rates with additional inference computation without updating the VLM's weights.
From simulation to physical execution
We evaluated physical execution on a block placement task using a LeRobot SO-101 arm. We reconstruct the scene from color and depth observations, search for a successful trajectory in simulation, and execute it on the robot. With randomized object positions and a budget of 15 candidate trajectories, the method succeeded in five of six trials. We attribute the remaining failure to pose estimation errors and differences in contact dynamics between simulation and the physical environment.
Furthermore, successful search trajectories can serve as training data for an imitation policy. This approach also succeeded in five of six trials and reduced the reported execution time compared with MCTS search.
Limitations and future work
SAIL has three main limitations. First, trajectories are executed open-loop, without visual feedback during motion. Second, searching and refining trajectories in simulation requires additional computation. Third, real-world evaluation is limited to one task and six trials per method, so broader validation is needed.
SAIL shows that allocating inference computation to simulation feedback and iterative refinement can improve robot trajectories generated by VLMs. See the paper for the method and experimental details.