CmdBenchby CmdBrief
Menu

Protocol

Protocol v1

Every run follows the same rules. If a rule changes, it becomes protocol v2 at a new address, and this page stays as it is.

Version
v1
Status
current version
History
all versions

One real run

From prompt to published.

  1. Same prompt

    Every agent gets it

    Build a lava lamp as a single index.html file with no external requests.
    Full prompt

  2. Built, hands off

    29.7 s · low effort

    What GPT-5.6 Luna built at low effort: desktop capture of run 4b99cb20

    GPT-5.6 Luna · Codex CLI

  3. Checked in a browser

    8 of 8 passed

    • Self-contained index.html is present
    • Visible page content
    • Visible autonomous animation
    • Accessible on/off control
    • 390 px mobile layout has no horizontal overflow
    • Local animation frame performance
    • No browser errors
    • No attempted external or linked-resource requests

    Works

  4. Published

    On every run page

    • Captures
    • Checks
    • Timings
    • Transcript
    • Source
    Open this run

Same prompt

Every agent gets the same prompt, the same files and the same reference images, byte for byte.

Same start

Each run starts in a fresh, empty workspace. Nothing carries over from another run or another agent.

Same limits

No network, one time limit for every agent, and no access to the hidden checks or to other agents’ work.

Hands off

Nobody helps the agent during a run. We check the files it leaves, exactly as it left them. A run done by hand says so on its run page.

Checked in a real browser

Hidden checks open the build on a desktop and a phone screen and test what the prompt asks for. A build works when every blocker check passes. How it looks is reviewed separately, blind. Time, tokens and cost describe the run; they are not a quality score.

Runs and episodes

In an episode every agent runs the same task the same number of times, three by default, in a shuffled order saved in advance. Every attempt is published, crashes and timeouts included.

Frozen versions

A benchmark version never changes once published, and every run records a hash of everything it was given. Runs are compared only with runs on the same version. Harnesses are stable releases only.

Everything published

Every run page shows the captures, every check, the timings, the transcript and the source. A mistake is fixed in a new revision and listed on the corrections page.

Disclosure

CmdBench is run by Jaanus Lepik. How it relates to CmdBrief is on the about page.

Version history

VersionStatusEpisodes
Protocol v1 Current 0