SAIL: Test-Time Scaling through Iterative Refinement
for VLMs as Robot Trajectory Generators

* This work was conducted during Makoto Sato's internship at Sakana AI.

Physical execution after trajectory refinement in simulation. SAIL evaluates and refines candidates in a reconstructed scene, then executes the selected trajectory on the robot.

Robot control with foundation models

What does a robot need before it can tackle a new task? A common starting point is to collect demonstrations and train a policy on the resulting trajectories. Foundation models, however, have already learned from images, video, code, language and robotics-related data. The question, then, is how much of that knowledge we can draw out for robot control without updating the model's weights.

In fact, GPT-6 Astra, in addition to its general capabilities in language and vision, has demonstrated the ability to perform multiple manipulation tasks on a physical robot by specifying motion targets from camera images and robot state [Robocurve, 2026]. Earlier work uses tokenized visual observations and a few demonstrations to generate trajectories through in-context learning with a language model [Di Palo and Johns, 2024]. The figure below illustrates direct trajectory generation with a VLM or LLM.

Context

Robot policy

VLM/LLM

Generates a trajectory

Generated trajectory

e1→e2→…→eT

Position and orientation
Gripper state

Direct trajectory generation with a VLM or LLM. The model uses context to generate a sequence of end-effector poses and gripper states.

Limits of trajectory generation with foundation models and our approach (SAIL)

However, a foundation model does not necessarily produce a reliable robot trajectory in a single generation. Predictions depend on the context provided, such as the demonstrations and their relevance to the current scene, and can also vary with sampling. Even when the intended action is appropriate, small errors in the motion targets generated by the model can cause failure. Executing a single prediction without evaluation leaves no opportunity to correct such errors before the robot moves.

Language models face a similar limitation, and training is not the only way to address it. Test-time scaling can improve accuracy beyond a single prediction by generating, evaluating and revising candidates at inference time without updating the weights. We ask whether the same trade between computation and accuracy extends to physical action.

We propose SAIL (Scaling In-Context Imitation Learning), a method for more reliable VLM-based robot trajectory generation through test-time scaling. SAIL uses a policy VLM to generate candidate trajectories from a few successful demonstrations provided in context. SAIL executes the trajectories in simulation, evaluates task progress with an evaluation VLM and returns step-level feedback to guide refinement through Monte Carlo Tree Search (MCTS). Candidate testing takes place in simulation, and only the selected trajectory is sent to the physical robot.

We then test how far this framework can improve success rates by increasing computation at inference time. Across six manipulation tasks in simulation, increasing the search budget from one node to 45 nodes raised the average rate of finding a successful trajectory from 25% to 73%. We also evaluated SAIL on a physical robot. In all experiments, we used Gemini Robotics-ER 1.5 as both the policy and evaluation VLM without updating its weights.

Limitation

A single generated trajectory can fail

Conceptual illustration of a missed grasp. A robot follows a red trajectory and closes its gripper beside a blue block. A red bowl is nearby.
Our solution (SAIL)

Test-time scaling through refinement in simulation

Conceptual illustration of trajectory refinement. Candidates are evaluated in simulation, and a green trajectory from the blue block to the red bowl is selected.
Trajectory errors and refinement in simulation. The left panel shows a missed grasp as one example of a trajectory error. The right panel illustrates evaluating and refining candidates in simulation. These are conceptual illustrations.

How SAIL refines trajectories

SAIL stores successful trajectories in a shared archive. A VLM estimates subtask completion, and scores are aligned with trajectory steps and returned to the generating model for refinement.
Trajectory refinement in SAIL. (1) The system retrieves demonstrations and generates a trajectory. (2) It estimates task completion from its simulated execution. (3) Completion scores aligned with selected trajectory waypoints guide the next refinement.

1. Generation from similar demonstrations

An archive stores successful trajectories. The system retrieves demonstrations from scenes that look similar to the current scene. The generating VLM receives these demonstrations together with the current object configuration and robot state, then generates a complete trajectory. Successful trajectories found during search are added to the archive for later use.

2. Evaluation of task completion

The trajectory generated in step 1 is executed in simulation, producing a video of its execution. An evaluation VLM estimates how much of the task has been completed by assessing progress through predefined subtasks in this video. Completion estimates across the execution are combined into a trajectory score, allowing comparison even when neither candidate finishes the task. This evaluation builds on ROVER's approach to scoring task progress over time [Schroeder et al., 2025].

3. Refinement with step-level feedback

Completion scores from sampled video frames are aligned with selected trajectory waypoints and returned to the generating VLM. The feedback prompts it to preserve successful segments and revise those where progress stalls. The revised trajectory forms a child node, where it is evaluated again in simulation to guide further search.

Success rates increase with the search budget

We evaluated six manipulation tasks in the ALOHA simulator, with 20 initial configurations per task. Increasing the search budget raised the average success rate from 25% with a single generated trajectory to 73% with a budget of 45 candidate trajectories (up to 45 search nodes).

Simulation examples from the six evaluation tasks.

Here, the success rate measures how often the search finds a trajectory that passes the simulator's ground truth success check within the search budget. This binary check is separate from the VLM estimates of task completion used to guide search.

Table 1. Rates of finding a successful trajectory (%) over 20 initial configurations per task
Success rates for Banana, Pen, Bowl, Drawer, Laptop, and Marker. Baseline averages: single generation 25%, breadth first search 51%, depth first search 37%. SAIL averages increase from 55% at 6 nodes to 65% at 15 nodes, 71% at 30 nodes, and 73% at 45 nodes.
Table 1. Rates of finding a successful trajectory (%) over 20 initial configurations per task
MethodNodesBananaPenBowlDrawerLaptopMarkerAverage
Baselines
Single generation14040401015525
Breadth first search1585506035601551
Depth first search154550805301037
Our method (SAIL)
Single generation (repeated)More
nodes
14040401015525Higher
success
SAIL6805510020502555
SAIL15907010040504065
SAIL30958010050554571
SAIL45958010050704573

Average success rates increased across the tested budgets, reaching 55%, 65%, 71% and 73% with 6, 15, 30 and 45 nodes, respectively. These results demonstrate test-time scaling through search and refinement, improving success rates with additional inference computation without updating the VLM's weights.

From simulation to physical execution

We evaluated physical execution on a block placement task using a LeRobot SO-101 arm. We reconstruct the scene from color and depth observations, search for a successful trajectory in simulation, and execute it on the robot. With randomized object positions and a budget of 15 candidate trajectories, the method succeeded in five of six trials. We attribute the remaining failure to pose estimation errors and differences in contact dynamics between simulation and the physical environment.

A simulated SO-101 robot and its physical counterpart grasp a blue block, move it over a red bowl and release it.
Simulation and physical environments for placing a block in a bowl.

Furthermore, successful search trajectories can serve as training data for an imitation policy. This approach also succeeded in five of six trials and reduced the reported execution time compared with MCTS search.

Limitations and future work

SAIL has three main limitations. First, trajectories are executed open-loop, without visual feedback during motion. Second, searching and refining trajectories in simulation requires additional computation. Third, real-world evaluation is limited to one task and six trials per method, so broader validation is needed.

SAIL shows that allocating inference computation to simulation feedback and iterative refinement can improve robot trajectories generated by VLMs. See the paper for the method and experimental details.