I'm an undergrad in RL. I built agent-mcts, which treats each Claude Code run as a node in a search tree. Expansion starts a new agent session, either a fresh approach from the root or a revision of a parent that sees its test output. The value is the visible test pass ratio, optionally averaged with an LLM judge score. Selection is UCT, and the top candidates get a final head-to-head comparison.
To measure it I used tasks with hidden test suites the agent never sees, and compared the chosen answer with the first attempt of the same run, which is the no-search baseline. Two tasks, haiku, 5-node budget, 10 runs per setting.
- Tests-only value: 0 better / 9 same / 1 worse. The first attempt already passed the visible tests in 9 of 10 runs, so the reward had nothing left to say.
- Tests + judge: 4 better / 4 same / 2 worse, mean change -0.1 pt.
- For pairs of attempts that both pass the visible tests but differ on the hidden ones, the judge ordered them correctly 38 times out of 62 (4 ties).
- Revisions under the judge were mostly neutral: 7 better / 12 same / 1 worse.
So the search does what the reward tells it to; the reward just isn't informative once the visible tests saturate. Small n, so take it as one data point.
I'd be interested in ideas for a better value estimate here: pairwise comparisons instead of absolute scores, generated tests, ensembles of judges, a learned verifier?
Code and benchmark: https://github.com/natsu0529/agent-mcts
Update (v0.4): I changed the value function and it now beats a single run on my benchmark.
Before the search starts, an agent writes extra pytest files from the task description only (it never sees an attempt). Each attempt is scored half on the project's own tests and half on these. The generated tests also go into the revision prompt, so a failing test tells the next attempt what to fix.
Same two tasks, haiku, 5-node budget, 20 runs: the final answer beat the first attempt 16 times, tied 2, lost 2. Mean hidden score went from 88% to 93%. Most of the gain comes from revisions (32 of 40 raised the hidden score), not from picking among first attempts.
One way it fails: on a vague bug report, the test writer made the same mistake as the bug, so the unfixed code scored highest and got returned every time. v0.4.1 only returns the unchanged code if every attempt does worse on the project's own tests.
Still small (three tasks, one model). Numbers are in bench/README.md.