An AI scientist you can audit

We rebuilt Sakana's Agentic Scientist as a Polaris graph. Same results, safer by design, and every step traceable.

Radiant Labs

A small research graph of model calls and checks over a blurred colour field

Sakana AI's Agentic Scientist, the third version of their AI Scientist, takes a research task and returns a finished paper. It comes up with ideas, writes and scores programs, runs ablations, analyses the results, and writes the paper with its own evidence checks. Each stage is handed to a general coding agent with a full shell.

We rebuilt it with Polaris, our open-source framework for building agents as composable graphs. In the graph, every model call is its own node. The rules around it (what it can read, what it can run, how its output is checked) are nodes and edges written in code, not instructions in a prompt. Each model call runs as its own session of Polaris Code, our coding agent built on Polaris, which matched Claude Code on SWE-bench Verified with about a sixth of the tokens.

It works. On the same task, with the same model, the graph reaches the same score and writes a paper of the same length with the same overall review score, and higher marks for clarity, soundness and presentation. We were not trying to beat Sakana's result, and we did not. What changes is how much of the run you can see and control.

MeasureOriginalPolaris graph
Best score, re-scored by the task’s own evaluator2.6362.630
Paper13 pages, compiles13 pages, compiles
Automated reviewer, overall score2 (reject)2 (reject)
Paper’s own evidence auditfailed after 11 roundspassed in round 4
Printed numbers backed by a saved result62 of 143116 of 116
Model calls with every message and tool call saved33 of 5633 of 33
Roles given an unrestricted shellallnone
Wall-clock time, one worker75 min57 min
Token cost, from the recorded usage$9.09$9.01

The original and our graph version, one run each. The task asks for a program that packs circles into a square so that their radii add up to as much as possible.

TLDR

  • Same task, same model: the graph scored 2.630 to the original's 2.636. Both papers got the same overall review score, and ours scored higher on clarity, soundness and presentation.
  • No role in the graph has an unrestricted shell, and the score comes from code the model cannot reach.
  • Every model call keeps its full conversation: 33 of 33, against 33 of 56 in the original.
  • Every number printed in our paper traces back to a saved result: 116 of 116, against 62 of 143.
  • The graph ran in 57 minutes to the original's 75, at about the same token cost. One run each does not prove a speed gain.

Circle-packing task · Claude Sonnet 5.5 on both sides · one run each · Sakana AI's Agentic Scientist vs our rebuild as a Polaris graph

Four ways to wire an AI scientist

Automated research agents have taken a few shapes. The first AI Scientist was a fixed pipeline. The second added a search tree over experiments. The Agentic Scientist puts a coding agent in charge of each stage. Ours keeps the same stages and builds them as one graph.

From a fixed pipeline to a graph
Three designs from Sakana AI, and ours.
model call (ours) check or code step coding-agent stage (original)
The AI Scientist2024
IdeaExperimentsedit, run, repeatWrite-upReview
A fixed pipeline. One code assistant edits a template experiment.
The AI Scientist v22025
Ideasearch tree over experimentsWrite-upReview
A search tree over experiments, run by an experiment manager, then the write-up.
The Agentic Scientist (v3)2026
controller script: starts each stage, then checks the files it leaves behindInvestigation4 sessionscoding agentshell: any commandAblations17 sessionscoding agentshell: any commandPost-hoc3 sessionscoding agentshell: any commandPaper32 sessionscoding agentshell: any command
A short controller runs the four stages that use the model. Each stage hands the work to a general coding agent with a full shell. Inside a stage, the agent decides what to run.
The Agentic Scientist as a Polaris graphours
Investigation5 callsAblations10 callsPost-hoc3 callsPaper15 calls×2×4
The same stages as one graph. Each model call is its own node, followed by a check written in code. Scoring, retries and loops are part of the graph, not part of a prompt. Simplified here; the full run is drawn further down.

The first three are Sakana's designs, as described in their papers. The fourth is ours.

The difference between the last two is where decisions are made. In the Agentic Scientist, the controller starts a stage, and the coding agent decides which commands to run, in what order, and when it is done. In the graph, the order of steps is fixed by the graph. The model decides only what goes into each step: an idea, a program, a paragraph.

The whole run as one graph

Because the graph is the program, the run's log is also a picture of what happened. The figure below is our full run, drawn from its event log. Hover over a step to see which role ran it, for how long, and in how many turns.

Our full run, drawn from its own log
33 model calls in 57 minutes. Hover over any step for its role, time and turns.
model call, with its check attached code step: check, score or fetch branch ×a check rejected the output
Task setup13 sno model callsInvestigation8.8 min5 calls, 2 cyclesAblations18.3 min10 callsPost-hoc3.3 min3 callsPaper26.6 min15 callsvariant: no hole reinsertion movesvariant: no basin hopping phasevariant: no pair pruningretrypassTask setup: reads the task. Code, no model.inspectTask setup: checks the task’s runtime. Code, no model.runtimeTask setup: records the starting point. Code, no model.baselineCall 1 of 33 · ideation · 0:36 · 7 turnsidea 1Call 2 of 33 · ideation · 0:35 · 6 turnsidea 2Call 3 of 33 · program author · 2:05 · 8 turnsprogram authorScores the program with the task’s own evaluator, unchanged, on a frozen copy. Code, no model.scoreChecks the attempt against a fixed rubric. Code, no model.rubricKeeps whichever program has the best score so far. Code, no model.keep bestRecords this attempt for the next cycle. Code, no model.notesCall 4 of 33 · ideation · 0:56 · 8 turns · cycle 2ideaCall 5 of 33 · program author · 2:06 · 9 turns · cycle 2program authorScores the program with the task’s own evaluator, unchanged, on a frozen copy. Code, no model.scoreChecks the attempt against a fixed rubric. Code, no model.rubricKeeps whichever program has the best score so far. Code, no model.keep bestRecords this attempt for the next cycle. Code, no model.notesCall 6 of 33 · variant retrofit · 2:44 · 11 turns · adds switches so parts of the best program can be turned offretrofitCall 7 of 33 · variant proposer · 0:30 · 4 turnsproposeCall 8 of 33 · variant selector · 0:17 · 3 turnsselectCall 9 of 33 · variant author · 2:08 · 8 turns · no hole reinsertion moves · check: the code must differ from the controleditScores the program with the task’s own evaluator, unchanged, on a frozen copy. Code, no model.Call 10 of 33 · variant verifier · 0:29 · 9 turns · no hole reinsertion movesverifyCall 11 of 33 · variant author · 2:02 · 7 turns · no basin hopping phase · check: the code must differ from the controleditScores the program with the task’s own evaluator, unchanged, on a frozen copy. Code, no model.Call 12 of 33 · variant verifier · 0:27 · 8 turns · no basin hopping phaseverifyCall 13 of 33 · variant author · 2:06 · 7 turns · no pair pruning · check: the code must differ from the controleditScores the program with the task’s own evaluator, unchanged, on a frozen copy. Code, no model.scoreCall 14 of 33 · variant verifier · 0:27 · 6 turns · no pair pruningverifyCall 15 of 33 · variant summariser · 0:10 · 4 turnssummariseCall 16 of 33 · post-hoc proposer · 1:07 · 7 turnsproposeCall 17 of 33 · post-hoc selector · 0:15 · 4 turnsselectCall 18 of 33 · post-hoc analyst · 1:58 · 15 turns · contact-graph rigidityanalysisCall 19 of 33 · reference planner · 0:27 · 4 turnsrefsFetches the planned references. Code, no model.fetchCall 20 of 33 · paper strategist · 1:11 · 6 turns · rejected by its check: asked for a figure type that does not existplanCall 21 of 33 · paper strategist · 1:42 · 7 turns · second try, with the reasonplanCall 22 of 33 · post-hoc ranker · 0:09 · 3 turnsrankCall 23 of 33 · paper writer · 11:42 · 69 turnsdraftCall 24 of 33 · citation gap analyst · 0:29 · 5 turnsgapsCall 25 of 33 · reference planner, after the draft · 0:21 · 4 turnsrefsFetches the new references. Code, no model.fetchCall 26 of 33 · paper enricher · 1:52 · 20 turnsenrichCall 27 of 33 · evidence auditor (read-only) · 0:55 · 12 turns · found problemsauditCall 29 of 33 · evidence auditor (read-only) · 0:57 · 9 turns · found problemsauditCall 31 of 33 · evidence auditor (read-only) · 1:07 · 11 turns · found problemsauditCall 33 of 33 · evidence auditor (read-only) · 1:10 · 14 turns · passedauditAudit found problems: the paper goes to the repairer.Audit found problems: the paper goes to the repairer.Audit found problems: the paper goes to the repairer.Audit passed.Call 28 of 33 · paper repairer · 1:42 · 22 turnsrepairCall 30 of 33 · paper repairer · 0:17 · 7 turnsrepairCall 32 of 33 · paper repairer · 0:44 · 15 turnsrepairChecks every claim and printed number against saved results. Code, no model.claimsCompiles the paper. Code, no model.PDF
Show all 33 model calls
#PhaseStepRoleTimeTurnsNote
1Investigationidea 1ideation0:367
2Investigationidea 2ideation0:356
3Investigationprogram authorprogram author2:058
4Investigationideaideation0:568cycle 2
5Investigationprogram authorprogram author2:069cycle 2
6Ablationsretrofitvariant retrofit2:4411adds switches so parts of the best program can be turned off
7Ablationsproposevariant proposer0:304
8Ablationsselectvariant selector0:173
9Ablationseditvariant author2:088no hole reinsertion moves
10Ablationsverifyvariant verifier0:299no hole reinsertion moves
11Ablationseditvariant author2:027no basin hopping phase
12Ablationsverifyvariant verifier0:278no basin hopping phase
13Ablationseditvariant author2:067no pair pruning
14Ablationsverifyvariant verifier0:276no pair pruning
15Ablationssummarisevariant summariser0:104
16Post-hocproposepost-hoc proposer1:077
17Post-hocselectpost-hoc selector0:154
18Post-hocanalysispost-hoc analyst1:5815contact-graph rigidity
19Paperrefsreference planner0:274
20Paperplanpaper strategist1:116rejected by its check: asked for a figure type that does not exist
21Paperplanpaper strategist1:427second try, with the reason
22Paperrankpost-hoc ranker0:093
23Paperdraftpaper writer11:4269
24Papergapscitation gap analyst0:295
25Paperrefsreference planner, after the draft0:214
26Paperenrichpaper enricher1:5220
27Paperaudit 1evidence auditor (read-only)0:5512found problems
28Paperrepair 1paper repairer1:4222
29Paperaudit 2evidence auditor (read-only)0:579found problems
30Paperrepair 2paper repairer0:177
31Paperaudit 3evidence auditor (read-only)1:0711found problems
32Paperrepair 3paper repairer0:4415
33Paperaudit 4evidence auditor (read-only)1:1014passed

Every model call in the run, in order. Task setup and scoring are code; they use no model.

Two moments stand out. In the paper phase, the first plan asked for a figure type that does not exist. Its check rejected the plan and sent the reason back, and the second try passed. Later, the evidence audit found claims that the results did not support. The graph sent the paper to a repairer and audited it again, and it passed on the fourth audit. Both loops are edges in the graph with a fixed limit (at most 10 repair rounds), so a run cannot loop forever, and every round leaves a record.

Safer: the rules are part of the graph

An agent that is judged on a number has a reason to change how the number is computed. Sakana's paper reports its agents doing this: extending their own time limits, and changing how results were evaluated, for example taking a majority vote over several samples where the task asked for a single attempt.

The original is not careless about this. Its controller checks the files each stage leaves behind, and many of our checks follow its rules. The difference is when the rules apply. In the original, an agent with a full shell does its work, and the controller checks the result afterwards. In the graph, the rules decide what each step can touch before it starts.

One step of the investigation, up close
The model writes the program. Code it cannot reach decides the score.
the model’s step, in its own foldercode the model cannot changeThe files this step needs, copied in read-only.inputs/read-only copyOne model call with its own limits on turns, dollars and time.program authormodel callThe only place the step can leave results. Checked in code before anything moves on.outputs/empty at startRuns the task’s own evaluator on a frozen copy of the program.scorecode, no modelKeeps the program only if its score beats the best so far.keep bestcode, no model
What the model gets
  • a read-only copy of its inputs, and an empty outputs folder
  • a list of 12 named programs it may run, not a general shell
  • a test runner with no network that can write only to a scratch folder
  • its own limits on turns, dollars and time
What the score step does
  • runs the task’s own evaluator, unchanged
  • on a frozen copy of the program
  • in an empty environment and its own process group
  • with a fixed 300-second time limit
  • records fingerprints of the program and the evaluator

Each time the model writes a program, it can test it once in a sandbox. Only the score step, which the model cannot reach, decides the number the run is judged on.

The same idea applies to every role. A role that only needs to think, such as choosing between ideas, gets no shell at all. A role that needs to run things gets a short list of named programs. The evidence auditor can read the paper and the results but cannot change either.

What each role is allowed to run
20 roles. Original: every role has a shell that accepts any command. Ours: 12 have no shell, and 8 may run only a named list of programs.
any commandoriginal no shellours 12 programsours, named list only
Role Original Polaris graph
Investigation
ideation any command no shell
program author any command 12 programs
Ablations
variant retrofit any command 15 programs
variant proposer any command no shell
variant selector any command no shell
variant author any command 12 programs
variant verifier any command no shell
variant summariser any command no shell
Role Original Polaris graph
Post-hoc
post-hoc proposer any command no shell
post-hoc selector any command no shell
post-hoc analyst any command 15 programs
Paper
reference planner any command no shell
paper strategist any command no shell
post-hoc ranker any command no shell
paper writer any command 18 programs
citation gap analyst any command no shell
reference planner, after the draft any command no shell
paper enricher any command 18 programs
paper repairer any command 18 programs
evidence auditorread-only any command 13 programs

Shell access for the 20 roles that ran. In the original, each role is a coding-agent session with the same full shell.

Checks in code also keep the paper honest about its own numbers. The graph's last step looks up every number printed in the paper in the result files the run saved, and the paper does not pass until each one is found.

Numbers in the final paper that trace back to a saved result
One square per number printed in the paper.
original, found in a saved result ours, found in a saved result no saved result found
Original: 62 of 143 printed numbers found in a saved resultOriginal62 of 143 backedPolaris graph: 116 of 116 printed numbers found in a saved resultPolaris graph116 of 116 backed

Checked with our own script, which looks for each printed number in the result files a run saved. Some of the original’s unmatched numbers may come from results it computed but did not save, which is itself the problem: a reader cannot check them.

Show as a table
RunNumbers printedFound in a saved resultNot found
Original1436281
Polaris graph1161160

A number that cannot be traced to a saved result cannot be checked by a reader, even if it is correct.

The rules have a cost. Of the 244 shell commands our roles tried to run, 29 were refused, mostly harmless ones such as echo, sed or a shell loop. The paper writer had 10 of its 73 refused. Each refusal costs the model a turn. A better list would refuse less; we have not tuned it yet.

This run does not test whether the rules stop an agent that tries to cheat. We test that in a new post, coming soon, on ImpossibleBench: public coding tasks whose tests were changed so that the only way to pass them is to cheat. There we compare the same rules written in a prompt and enforced by the graph, and check that the agent still solves normal tasks.

Transparent: every call is on record

When an automated research run goes wrong, the first question is what the agent actually did. Answering it needs the full conversation of each model call, with every message and tool call, not only the files it left behind.

Model calls whose full conversation was saved
One dot per model call, by phase.
original, saved ours, saved no full conversation kept
OriginalPolaris graphInvestigationOriginal, Investigation: 4 of 4 calls saved in full4 of 4Polaris graph, Investigation: 5 of 5 calls saved in full5 of 5AblationsOriginal, Ablations: 0 of 17 calls saved in full0 of 17Polaris graph, Ablations: 10 of 10 calls saved in full10 of 10Post-hocOriginal, Post-hoc: 1 of 3 calls saved in full1 of 3Polaris graph, Post-hoc: 3 of 3 calls saved in full3 of 3PaperOriginal, Paper: 28 of 32 calls saved in full28 of 32Polaris graph, Paper: 15 of 15 calls saved in full15 of 15All phases33 of 5633 of 33

A full conversation is every message and tool call inside one model call, kept after the run ends. The original kept no full log for its 17 ablation sessions. In our graph, saving the conversation is a step of the graph itself, so it happens for every call.

Show as a table
PhaseOriginal, savedOriginal, callsOurs, savedOurs, calls
Investigation4455
Ablations0171010
Post-hoc1333
Paper28321515

In the original, 23 of 56 sessions have no usable full conversation. 22 were never saved, including all 17 ablation sessions, and one was overwritten by a later session.

The records also showed us our own mistakes. Two examples:

  • In an early run, two of our three ablation variants were unchanged copies of the program they were meant to differ from, so the ablation measured nothing. We found it in the records and added a check that rejects a variant whose code matches the program it is compared against. The run reported here passed that check.
  • Later, we found a way to run a command that was not on a role's list, by passing it through a heredoc. We have a fix, but it was not yet in the version used for the run reported here.

A graph does not make the rules right. It makes them visible, testable and fixable.

Faster and cheaper: promising, not proven

Our run took 57 minutes against the original's 75. The token cost was about the same: $9.01 against $9.09.

Minutes per phase, one run each
Promising, not proven. See the note under the chart.
Original Polaris graph
010203040 minInvestigationOriginal, Investigation: 10.4 min10.4Polaris graph, Investigation: 8.8 min8.8AblationsOriginal, Ablations: 24.5 min24.5Polaris graph, Ablations: 18.3 min18.3Post-hocOriginal, Post-hoc: 1.5 min1.5Polaris graph, Post-hoc: 3.3 min3.3PaperOriginal, Paper: 38.8 min38.8Polaris graph, Paper: 26.6 min26.6
Wall-clock time, whole run
75 min 57 min
Token cost
$9.09 $9.01
Fixed text sent with every request
15.4k tokens about 2.5k tokens

These two runs differ in ways that favour ours. Our best program searched for 110 seconds each time it was scored, the original’s for 200. Our paper passed its audit in round 4; the original’s audit loop ran 11 rounds. A fair speed test needs equal search time and several runs per side.

Show as a table
PhaseOriginal, minPolaris graph, min
Investigation10.48.8
Ablations24.518.3
Post-hoc1.53.3
Paper38.826.6

Wall-clock minutes per phase. Task setup and final evaluation are left out; neither uses the model.

We do not count this as a speed result. One run per side cannot separate the design from luck, and both differences in the note above favour us.

Cost is a similar story. Every request in the original carries about 15,400 tokens of fixed text (tool descriptions, a system prompt and reminders). Ours carries about 2,500. That saving did not reach the total. Our paper writer sent 67 requests against the original's 28, each 1.6 times larger on average. Ours came out slightly lower mostly because it needed fewer audit and repair rounds. These figures come from the recorded token usage.

What the graph does give us is a clear place to look. Each node reports its own tokens, so we know which role to fix next.

Why this matters for safe AI research

We are a research lab focused on safe agentic AI, and we think it starts with being able to see what an agent did. Much of our work so far looks at how agents behave under pressure: how quickly they give way over a long conversation, what that looks like inside the model, and how hard it is to steer a model away from giving in.

Automated research raises the same problem at a larger scale. An AI scientist is an agent judged on a number, with the means to change how that number is computed, and its results are meant to be trusted by people who did not watch it run. Sakana's own report of agents extending their time limits shows this already happens. As these systems start doing research on AI itself, being able to check what they did becomes a safety requirement.

That is what this rebuild is for. Before a step runs, anyone can see what it is allowed to read and run. After the run, anyone can read the full conversation of every call and trace every number in the paper to a saved result. It is the same idea behind Radiant Platform, which records each run so that people, and our Meta Agent, can find out why it went the way it did.

What this does and does not show

  • It shows that the Agentic Scientist can be built as a graph and reach the same result on this task with the same model.
  • It shows that a graph can enforce rules the original leaves to the agent: what each step can read, what it can run, how the score is computed, and what gets recorded.
  • It does not show that the graph is faster or cheaper. That needs equal settings and several runs per side.
  • It does not test whether the graph stops an agent that tries to cheat. A new post on that is coming soon: coding agents built as graphs on ImpossibleBench, where the only way to pass is to cheat.
  • It covers one task, one model, and one run per side.

What comes next

Next, we want to run both systems with equal search time and several runs each. We also want a version of this task where the agent can edit its own time limit and evaluation, to see whether the graph's rules hold up in research as well as in coding tasks.

The code is open source at auditable-scientist. The full records of both runs, with the scripts that recompute the numbers in this post, are attached to its first release.

Thanks to Sakana AI for publishing the Agentic Scientist and its code under the Apache-2.0 licence. This work builds directly on theirs.

If you're working on automated research or agent safety, or want to build your own agents as graphs with Polaris, we'd love to hear from you at [email protected].