Discovering better agent designs
Our starting point is Polaris, our library for composing agentic systems. We want to make its harnesses the subject of automated experiments: identify a recurring failure, propose a change to the agent's design, implement it and measure what happens.
There is a concrete precedent in Automated Design of Agentic Systems, where a meta agent writes new agent designs in code and draws on an archive of previous experiments. The Darwin Gödel Machine extends the idea through an evolving archive of coding agents, allowing the search to branch rather than keeping only a single incumbent.
In Radiant, the Meta Agent would use execution traces to formulate the next experiment and hand a bounded implementation task to a coding agent. A change might replace a memory strategy, alter tool selection or introduce a verification step. The result should tell us which behaviour changed, on which tasks, and at what cost.
Harness² explores a further step: improving the harness that performs the improvement itself. Our initial experiments keep the Meta Agent fixed, so we can attribute changes in performance to the research agent it is modifying.
Learning from execution
Harness search leaves the underlying model unchanged. We’re also interested in what agents can learn through ongoing post-training on their production experience: successful demonstrations, corrections, alternative attempts and outcomes accumulated across real tasks.
The useful unit of data depends on the learning method. A successful trajectory may provide a demonstration for supervised fine-tuning. Two attempts at the same task, with a reliable preference between them, may support preference optimisation. Tasks with attributable rewards open a route to reinforcement learning.
Agent Lightning is particularly relevant here: it separates execution from training and introduces credit assignment over agent trajectories. This suggests a role for observability beyond debugging—a record of experience that can be selected, labelled and used to train the next model.
Our research questions concern that transition. Which traces are worth learning from? How should credit be assigned across tool calls and multiple agents? How do we account for data collected by older policies, and improve on new tasks without losing earlier capabilities?
Making improvement measurable
Every candidate has a harness revision and a model checkpoint. We want to study changes to each independently before testing how they interact. That requires an experimental record linking the proposed change to its training data, execution conditions, traces and evaluation results.
Search uses development tasks; independent validation tests whether an improvement carries beyond them. Across successive updates, we’re interested in transfer, retention and the cost of solving a task, as well as success rate. Evaluation protocols can evolve as production exposes new problems, while remaining fixed within each controlled comparison.
Building environments from real work
A production trace records one path through a task. To experiment with a different action, we need somewhere to execute it. We’re exploring how to reconstruct the relevant state, tools and success conditions of production workflows as repeatable environments.
Consider an agent that fails halfway through a workflow. Can we restore the state just before the failure, try a different tool or reasoning strategy, and evaluate the resulting outcome? Making that possible would connect failure investigation to both harness experiments and reinforcement learning.
The challenge is preserving the parts of the environment that determine success: tool behaviour, dependencies, state transitions and feedback. These are research problems we want to work on with teams deploying agents in complex settings.