Learning to Anticipate Physical Failures via
Agentic Adversarial Harness

Setting

A coding agent receives a task e with a language goal and a scene. It writes a pipeline p, a single script or a set of skills driven by a state machine. s(p, e) is the success rate of p on e over 10 seeds. A variation v changes one aspect of the scene; the designed task e ⊕ v keeps the goal and the success function of e. All training happens in simulation, where the agent can break its programs safely.

Shared robot API

The agent never reads simulator state. Every backend, simulated or real, exposes the same Python API: perception (detect, get_obstacles, track, camera_frame), motion (move_to with obstacles to avoid, grasp with a grip strength, place, open_gripper, close_gripper, wait_until) and state checks (tcp_pose, gripper_holding, in_region). Simulated detections carry 1 cm and 3° of noise, so a program cannot rely on exact simulator state. Because the vocabulary is shared, a rule learned in one simulator names the same calls on the real robot.

Constraints that may or may not appear

Each accepted variation is one more constraint on the task. After k rounds the agent holds C_k = {v₁, …, v_k}, and the task it must solve is e ⊕ {v_j ∈ C_k : b_j = 1} with b_j ~ Bernoulli(π_j), where π_j is the unknown chance that constraint v_j is present. A good pipeline succeeds whether or not each constraint appears. In training we check the base task and every constraint on its own: s(p, e) ≥ τ and s(p, e ⊕ v_j) ≥ τ for every v_j.

Training loop (one run per task)

1. Solve. Write p and evaluate it on 10 seeds until s(p, e) ≥ τ, with τ = 0.8. Record s₀ = s(p, e).
2. Question and design. List the assumptions the pipeline makes about the scene, pick the one a real robot is most likely to violate and that has not been attacked yet, and write a variation that violates it. Variations are scene edits from a fixed vocabulary: object pose, orientation and size, mass and friction, distractors, obstacles, object motion during execution, lighting, and camera pose. The scene must stay physically solvable.
3. Gates. A gate tool, not the agent, decides. It runs a frozen copy of the pipeline from before the round.
   • Break gate: the frozen pipeline must lose at least half its performance, s(p, e ⊕ v) ≤ s₀/2.
   • Variance gate: pure parameter tuning (for example, moving the same object further in the same direction) may break the pipeline at most two times in a row; the next variation must change a different aspect. The agent labels each variation and the tool checks the label against the previous one.
   • Validity and integrity: scenes that are invalid at reset (objects falling or interpenetrating) are rejected, and a hash of the success function and the tools stops the agent from passing a gate by editing the evaluator.
   A rejected variation sends the agent back to step 2.
4. Recover. Repair the pipeline, at most 6 iterations, until it reaches τ on the new variation without dropping below τ on the base task or any earlier constraint. Otherwise the variation is marked unsolved, the pipeline is reverted, and the variation does not join C; it still produces a reflection.
5. Reflect. One entry per round: the weakness, how it was found, why it was missed, the fix, and a one-sentence general rule that names no task or object, marked novel or not with respect to earlier entries.
6. Converge. Return to step 2 with the repaired pipeline. Stop after five accepted variations or two consecutive accepted rounds whose rules are not novel.

p ← A(e);  s0 ← s(p, e);  C ← ∅;  R ← ∅;  n_tune ← 0
while |C| < 5 and the last two accepted rules are not both non-novel:
    v ← A.DesignTask(p, e)                       # question the pipeline
    if v is parameter tuning and n_tune ≥ 2: reject v      # variance gate
    if s(p, e ⊕ v) > s0 / 2: reject v                      # break gate, frozen p
    n_tune ← n_tune + 1 if v is parameter tuning else 0
    p' ← p
    repeat p' ← A.Repair(p', e ⊕ v)              # at most 6 times
    until s(p', e) ≥ τ and s(p', e ⊕ v') ≥ τ for all v' in C ∪ {v}
    if recovered: p ← p';  C ← C ∪ {v}           # new constraint
    R ← R ∪ A.Reflect(p, v)                      # also for unsolved v
return R

Summarizing the harness

After all runs, a fresh agent reads every reflection and writes the harness H, a Markdown file of at most 800 words. It keeps only lessons supported by at least one reflection, names no training task or object, merges duplicates and orders rules by how many failures they would have prevented. Each rule carries one line on the failure it prevents. H has four sections: questions to ask before writing code, constraints every program must satisfy, self-checks to run without a simulator, and a workflow. A hidden source map links every rule to its reflections. For a real task that also appears in training (hanging mug, plug charger), the harness is built leave-one-out from all other reflections. The harness used on the robot will be released upon acceptance.

Deployment protocol

A fresh agent gets an unseen real task with no simulator, no access to training tasks and no human supervisor. Before writing, it may inspect the scene with read-only perception calls, which never move the robot and do not count as attempts. The robot moves only when the operator runs a submitted program through a runner that enforces velocity, acceleration and workspace limits; each run is one attempt. After a failure the agent sees camera frames and the execution log, revises, and tries again, for at most four attempts, following the single- and multi-turn protocols of CaP-X with the number of turns capped. Every attempt records all camera videos, the robot state, every API call, tokens, time, failure type and safety events.