Methodology
Methodology
How to read the results, in short. The rules are in protocol v1.
What we test
Small front-end builds you can see and click: an effect, a page, an animation. We write our own tasks, so every input and every result can be shown.
What we compare
An agent is a harness, such as Claude Code or Codex CLI, with a model and an effort level. Every result names all three.
Works
A build works when every blocker check passes. One failed blocker means it does not. A check with no result yet keeps it pending, never a pass or a fail.
Checks passed
Across an agent’s attempts we show the median and the range, so you can see how much the attempts varied.
Run order
Runs go one at a time, in a shuffled order saved before the first one.
How it looks
People review the captures blind, without knowing which agent made them. The review never changes works.
Time, tokens and cost
Shown as each harness reports them. A value it does not report reads “not reported”, never zero. They describe the run, not its quality.
Leaderboard
One table per episode, and no overall rank.
What we don't do
No points score, no hidden adjustments, no mixing of versions, no silent retries, and no agent code running on this site.
Limits
A few runs per agent show what happened, not a settled rate. One task is one task: it does not predict how an agent does on your code. Checks can be wrong; fixes go to corrections.
Versions
Published versions never change. A fix is a new version at a new address.