An AI scientist you can audit
We rebuilt Sakana's Agentic Scientist as a Polaris graph. Same results, safer by design, and every step traceable.
Sakana AI's Agentic Scientist, the third version of their AI Scientist, takes a research task and returns a finished paper. It comes up with ideas, writes and scores programs, runs ablations, analyses the results, and writes the paper with its own evidence checks. Each stage is handed to a general coding agent with a full shell.
We rebuilt it with Polaris, our open-source framework for building agents as composable graphs. In the graph, every model call is its own node. The rules around it (what it can read, what it can run, how its output is checked) are nodes and edges written in code, not instructions in a prompt. Each model call runs as its own session of Polaris Code, our coding agent built on Polaris, which matched Claude Code on SWE-bench Verified with about a sixth of the tokens.
It works. On the same task, with the same model, the graph reaches the same score and writes a paper of the same length with the same overall review score, and higher marks for clarity, soundness and presentation. We were not trying to beat Sakana's result, and we did not. What changes is how much of the run you can see and control.
| Measure | Original | Polaris graph |
|---|---|---|
| Best score, re-scored by the task’s own evaluator | 2.636 | 2.630 |
| Paper | 13 pages, compiles | 13 pages, compiles |
| Automated reviewer, overall score | 2 (reject) | 2 (reject) |
| Paper’s own evidence audit | failed after 11 rounds | passed in round 4 |
| Printed numbers backed by a saved result | 62 of 143 | 116 of 116 |
| Model calls with every message and tool call saved | 33 of 56 | 33 of 33 |
| Roles given an unrestricted shell | all | none |
| Wall-clock time, one worker | 75 min | 57 min |
| Token cost, from the recorded usage | $9.09 | $9.01 |
The original and our graph version, one run each. The task asks for a program that packs circles into a square so that their radii add up to as much as possible.
TLDR
- Same task, same model: the graph scored 2.630 to the original's 2.636. Both papers got the same overall review score, and ours scored higher on clarity, soundness and presentation.
- No role in the graph has an unrestricted shell, and the score comes from code the model cannot reach.
- Every model call keeps its full conversation: 33 of 33, against 33 of 56 in the original.
- Every number printed in our paper traces back to a saved result: 116 of 116, against 62 of 143.
- The graph ran in 57 minutes to the original's 75, at about the same token cost. One run each does not prove a speed gain.
Four ways to wire an AI scientist
Automated research agents have taken a few shapes. The first AI Scientist was a fixed pipeline. The second added a search tree over experiments. The Agentic Scientist puts a coding agent in charge of each stage. Ours keeps the same stages and builds them as one graph.
The first three are Sakana's designs, as described in their papers. The fourth is ours.
The difference between the last two is where decisions are made. In the Agentic Scientist, the controller starts a stage, and the coding agent decides which commands to run, in what order, and when it is done. In the graph, the order of steps is fixed by the graph. The model decides only what goes into each step: an idea, a program, a paragraph.
The whole run as one graph
Because the graph is the program, the run's log is also a picture of what happened. The figure below is our full run, drawn from its event log. Hover over a step to see which role ran it, for how long, and in how many turns.
Show all 33 model calls
| # | Phase | Step | Role | Time | Turns | Note |
|---|---|---|---|---|---|---|
| 1 | Investigation | idea 1 | ideation | 0:36 | 7 | |
| 2 | Investigation | idea 2 | ideation | 0:35 | 6 | |
| 3 | Investigation | program author | program author | 2:05 | 8 | |
| 4 | Investigation | idea | ideation | 0:56 | 8 | cycle 2 |
| 5 | Investigation | program author | program author | 2:06 | 9 | cycle 2 |
| 6 | Ablations | retrofit | variant retrofit | 2:44 | 11 | adds switches so parts of the best program can be turned off |
| 7 | Ablations | propose | variant proposer | 0:30 | 4 | |
| 8 | Ablations | select | variant selector | 0:17 | 3 | |
| 9 | Ablations | edit | variant author | 2:08 | 8 | no hole reinsertion moves |
| 10 | Ablations | verify | variant verifier | 0:29 | 9 | no hole reinsertion moves |
| 11 | Ablations | edit | variant author | 2:02 | 7 | no basin hopping phase |
| 12 | Ablations | verify | variant verifier | 0:27 | 8 | no basin hopping phase |
| 13 | Ablations | edit | variant author | 2:06 | 7 | no pair pruning |
| 14 | Ablations | verify | variant verifier | 0:27 | 6 | no pair pruning |
| 15 | Ablations | summarise | variant summariser | 0:10 | 4 | |
| 16 | Post-hoc | propose | post-hoc proposer | 1:07 | 7 | |
| 17 | Post-hoc | select | post-hoc selector | 0:15 | 4 | |
| 18 | Post-hoc | analysis | post-hoc analyst | 1:58 | 15 | contact-graph rigidity |
| 19 | Paper | refs | reference planner | 0:27 | 4 | |
| 20 | Paper | plan | paper strategist | 1:11 | 6 | rejected by its check: asked for a figure type that does not exist |
| 21 | Paper | plan | paper strategist | 1:42 | 7 | second try, with the reason |
| 22 | Paper | rank | post-hoc ranker | 0:09 | 3 | |
| 23 | Paper | draft | paper writer | 11:42 | 69 | |
| 24 | Paper | gaps | citation gap analyst | 0:29 | 5 | |
| 25 | Paper | refs | reference planner, after the draft | 0:21 | 4 | |
| 26 | Paper | enrich | paper enricher | 1:52 | 20 | |
| 27 | Paper | audit 1 | evidence auditor (read-only) | 0:55 | 12 | found problems |
| 28 | Paper | repair 1 | paper repairer | 1:42 | 22 | |
| 29 | Paper | audit 2 | evidence auditor (read-only) | 0:57 | 9 | found problems |
| 30 | Paper | repair 2 | paper repairer | 0:17 | 7 | |
| 31 | Paper | audit 3 | evidence auditor (read-only) | 1:07 | 11 | found problems |
| 32 | Paper | repair 3 | paper repairer | 0:44 | 15 | |
| 33 | Paper | audit 4 | evidence auditor (read-only) | 1:10 | 14 | passed |
Every model call in the run, in order. Task setup and scoring are code; they use no model.
Two moments stand out. In the paper phase, the first plan asked for a figure type that does not exist. Its check rejected the plan and sent the reason back, and the second try passed. Later, the evidence audit found claims that the results did not support. The graph sent the paper to a repairer and audited it again, and it passed on the fourth audit. Both loops are edges in the graph with a fixed limit (at most 10 repair rounds), so a run cannot loop forever, and every round leaves a record.
Safer: the rules are part of the graph
An agent that is judged on a number has a reason to change how the number is computed. Sakana's paper reports its agents doing this: extending their own time limits, and changing how results were evaluated, for example taking a majority vote over several samples where the task asked for a single attempt.
The original is not careless about this. Its controller checks the files each stage leaves behind, and many of our checks follow its rules. The difference is when the rules apply. In the original, an agent with a full shell does its work, and the controller checks the result afterwards. In the graph, the rules decide what each step can touch before it starts.
- a read-only copy of its inputs, and an empty outputs folder
- a list of 12 named programs it may run, not a general shell
- a test runner with no network that can write only to a scratch folder
- its own limits on turns, dollars and time
- runs the task’s own evaluator, unchanged
- on a frozen copy of the program
- in an empty environment and its own process group
- with a fixed 300-second time limit
- records fingerprints of the program and the evaluator
Each time the model writes a program, it can test it once in a sandbox. Only the score step, which the model cannot reach, decides the number the run is judged on.
The same idea applies to every role. A role that only needs to think, such as choosing between ideas, gets no shell at all. A role that needs to run things gets a short list of named programs. The evidence auditor can read the paper and the results but cannot change either.
Shell access for the 20 roles that ran. In the original, each role is a coding-agent session with the same full shell.
Checks in code also keep the paper honest about its own numbers. The graph's last step looks up every number printed in the paper in the result files the run saved, and the paper does not pass until each one is found.
Checked with our own script, which looks for each printed number in the result files a run saved. Some of the original’s unmatched numbers may come from results it computed but did not save, which is itself the problem: a reader cannot check them.
Show as a table
| Run | Numbers printed | Found in a saved result | Not found |
|---|---|---|---|
| Original | 143 | 62 | 81 |
| Polaris graph | 116 | 116 | 0 |
A number that cannot be traced to a saved result cannot be checked by a reader, even if it is correct.
The rules have a cost. Of the 244 shell commands our roles tried to run, 29 were refused, mostly
harmless ones such as echo, sed or a shell loop. The paper writer had 10
of its 73 refused. Each refusal costs the model a turn. A better list would refuse less; we have not
tuned it yet.
This run does not test whether the rules stop an agent that tries to cheat. We test that in a new post, coming soon, on ImpossibleBench: public coding tasks whose tests were changed so that the only way to pass them is to cheat. There we compare the same rules written in a prompt and enforced by the graph, and check that the agent still solves normal tasks.
Transparent: every call is on record
When an automated research run goes wrong, the first question is what the agent actually did. Answering it needs the full conversation of each model call, with every message and tool call, not only the files it left behind.
A full conversation is every message and tool call inside one model call, kept after the run ends. The original kept no full log for its 17 ablation sessions. In our graph, saving the conversation is a step of the graph itself, so it happens for every call.
Show as a table
| Phase | Original, saved | Original, calls | Ours, saved | Ours, calls |
|---|---|---|---|---|
| Investigation | 4 | 4 | 5 | 5 |
| Ablations | 0 | 17 | 10 | 10 |
| Post-hoc | 1 | 3 | 3 | 3 |
| Paper | 28 | 32 | 15 | 15 |
In the original, 23 of 56 sessions have no usable full conversation. 22 were never saved, including all 17 ablation sessions, and one was overwritten by a later session.
The records also showed us our own mistakes. Two examples:
- In an early run, two of our three ablation variants were unchanged copies of the program they were meant to differ from, so the ablation measured nothing. We found it in the records and added a check that rejects a variant whose code matches the program it is compared against. The run reported here passed that check.
- Later, we found a way to run a command that was not on a role's list, by passing it through a heredoc. We have a fix, but it was not yet in the version used for the run reported here.
A graph does not make the rules right. It makes them visible, testable and fixable.
Faster and cheaper: promising, not proven
Our run took 57 minutes against the original's 75. The token cost was about the same: $9.01 against $9.09.
These two runs differ in ways that favour ours. Our best program searched for 110 seconds each time it was scored, the original’s for 200. Our paper passed its audit in round 4; the original’s audit loop ran 11 rounds. A fair speed test needs equal search time and several runs per side.
Show as a table
| Phase | Original, min | Polaris graph, min |
|---|---|---|
| Investigation | 10.4 | 8.8 |
| Ablations | 24.5 | 18.3 |
| Post-hoc | 1.5 | 3.3 |
| Paper | 38.8 | 26.6 |
Wall-clock minutes per phase. Task setup and final evaluation are left out; neither uses the model.
We do not count this as a speed result. One run per side cannot separate the design from luck, and both differences in the note above favour us.
Cost is a similar story. Every request in the original carries about 15,400 tokens of fixed text (tool descriptions, a system prompt and reminders). Ours carries about 2,500. That saving did not reach the total. Our paper writer sent 67 requests against the original's 28, each 1.6 times larger on average. Ours came out slightly lower mostly because it needed fewer audit and repair rounds. These figures come from the recorded token usage.
What the graph does give us is a clear place to look. Each node reports its own tokens, so we know which role to fix next.
Why this matters for safe AI research
We are a research lab focused on safe agentic AI, and we think it starts with being able to see what an agent did. Much of our work so far looks at how agents behave under pressure: how quickly they give way over a long conversation, what that looks like inside the model, and how hard it is to steer a model away from giving in.
Automated research raises the same problem at a larger scale. An AI scientist is an agent judged on a number, with the means to change how that number is computed, and its results are meant to be trusted by people who did not watch it run. Sakana's own report of agents extending their time limits shows this already happens. As these systems start doing research on AI itself, being able to check what they did becomes a safety requirement.
That is what this rebuild is for. Before a step runs, anyone can see what it is allowed to read and run. After the run, anyone can read the full conversation of every call and trace every number in the paper to a saved result. It is the same idea behind Radiant Platform, which records each run so that people, and our Meta Agent, can find out why it went the way it did.
What this does and does not show
- It shows that the Agentic Scientist can be built as a graph and reach the same result on this task with the same model.
- It shows that a graph can enforce rules the original leaves to the agent: what each step can read, what it can run, how the score is computed, and what gets recorded.
- It does not show that the graph is faster or cheaper. That needs equal settings and several runs per side.
- It does not test whether the graph stops an agent that tries to cheat. A new post on that is coming soon: coding agents built as graphs on ImpossibleBench, where the only way to pass is to cheat.
- It covers one task, one model, and one run per side.
What comes next
Next, we want to run both systems with equal search time and several runs each. We also want a version of this task where the agent can edit its own time limit and evaluation, to see whether the graph's rules hold up in research as well as in coding tasks.
The code is open source at auditable-scientist. The full records of both runs, with the scripts that recompute the numbers in this post, are attached to its first release.
Thanks to Sakana AI for publishing the Agentic Scientist and its code under the Apache-2.0 licence. This work builds directly on theirs.
If you're working on automated research or agent safety, or want to build your own agents as graphs with Polaris, we'd love to hear from you at [email protected].