Passing ≠ correct
Every submission passes the scored fixtures, but a finite suite doesn't prove complete correctness. A review found two grader bugs, which have been fixed and rescored. No pass/fail result changed.
LANGUAGE TOKENS BENCHMARK
Token Golf
The same program, written in five languages by the same AI agent.
Workflow engine
median of 6 runs
Expression evaluator
1 run
JSON Patch
1 run
Output tokens per session, fewest first. Hover or tap a language to find it in every column.
Rust uses the most tokens on the workflow engine and the fewest on the expression evaluator. Python wins JSON Patch and comes last on expressions. No language holds its place across all three tasks.
01 Same prompt, same model, very different runs
The workflow engine was run six times per language. Python's cheapest run used 78.4k output tokens and its most expensive used 154.2k. The gap between runs of the same language is bigger than the gap between the top three languages.
95.0k median · 2.3× spread
96.1k median · 2.0× spread
102.1k median · 2.4× spread
120.0k median · 1.4× spread
128.9k median · 1.3× spread
60k90k120k150k180k
3.76M median · 7.2× spread
4.63M median · 3.7× spread
3.80M median · 3.4× spread
6.17M median · 3.1× spread
6.78M median · 2.1× spread
03M6M9M12M
On total tokens, JavaScript's most expensive run used 7.2× as many as its cheapest.
Each dot is one run. The amber tick is the median.
Six runs per language aren't enough to say which language is cheapest.
02 What the token count includes
97% of all tokens processed were cached context: the agent re-reading conversation it had already seen. Per session it ranged from 92% to 98%. The longer a session runs, the more it re-reads, so total tokens mostly track how long the agent kept going. That's why the leaderboard counts output tokens.
- Cached context re-read97.0%
- Output (code, tool calls, reasoning)2.2%
- Fresh input0.7%
- About 65% of output tokens were reasoning. The rest is tool calls, tests, rewrites and the final source.
- Cached, fresh and output tokens are priced differently, so ranking by cost can give a different order. Costs are what was paid at the time.
- Everything here comes from one model. Another model could produce a different order.
03 The course
- One spec. A fully specified CLI program: JSON in, JSON out, deterministic, with a closed set of error codes.
- Five players. Python, JavaScript, Go, Rust and C. Each gets an independent agent session with tools, a shell and room to self-test.
- One referee. A language-agnostic conformance suite that only talks to the binary through its CLI. Fail it and your score doesn't count.
- The score. Session output tokens: everything the agent wrote, including reasoning, tool calls and rewrites.
Workflow engineDAG execution, $ref, conditions, retries, 40+ ops
Expression evaluatortokenizer, precedence parser, i64 math, exact error positions
JSON PatchRFC 6901 pointers, six ops, atomicity, error taxonomy
Model: DeepSeek-V4.1-Flash-1017, thinking=high, one autonomous session per run. Every run counted here passes its scored suite.
04 The fine print
Not a frozen eval
Some specs, fixtures and oracles were corrected after submissions were examined. The Git history records every change, but this wasn't a pre-registered evaluation.
Trust, partially verified
The raw session logs aren't published, so usage totals and compliance with the no-peeking rules can't be audited from the repo. The launcher isn't an OS sandbox.
05 Contribute
Every submission and the full test suite are in the repo, so you can check these numbers without running a model.
Rescore the results
Rebuild every committed submission and rerun the test suite. No API key needed.
git clone https://github.com/GodGMN/language-tokens-benchmark
python3 benchmark/rescore.py Add a test case
One JSON file per behaviour. It has to follow from the spec and pass against the reference implementation.
benchmark/holdout/<category>/your_case.json
python3 benchmark/runner.py --language oracle Add a model or a language
A new round for another model, or an adapter for Kotlin, Swift, Zig and so on. Use a new round id and commit failed runs too.
benchmark/rounds/<round>/round.json
benchmark/adapters/<lang>.json 06 Every sample
All 45 sessions, nothing filtered. Download dataset.csv or dataset.json.
Show all 45 rows
| Round | Task | Language | Tests | Output | Total | Turns | Cost |
|---|---|---|---|---|---|---|---|
| round-8-expr | expr v1 | C | 56/56 | 71,378 | 1,887,134 | 29 | $0.103 |
| round-8-expr | expr v1 | Go | 56/56 | 60,130 | 1,408,634 | 30 | $0.086 |
| round-8-expr | expr v1 | JavaScript | 56/56 | 60,022 | 1,241,698 | 22 | $0.084 |
| round-8-expr | expr v1 | Python | 56/56 | 73,983 | 2,524,500 | 38 | $0.112 |
| round-8-expr | expr v1 | Rust | 56/56 | 59,463 | 1,067,224 | 23 | $0.080 |
| round-9-patch | patch v1 | C | 38/38 | 77,186 | 2,355,515 | 41 | $0.112 |
| round-9-patch | patch v1 | Go | 38/38 | 42,905 | 709,941 | 21 | $0.059 |
| round-9-patch | patch v1 | JavaScript | 38/38 | 66,718 | 1,562,150 | 29 | $0.095 |
| round-9-patch | patch v1 | Python | 38/38 | 37,514 | 700,543 | 20 | $0.054 |
| round-9-patch | patch v1 | Rust | 38/38 | 44,765 | 925,327 | 28 | $0.065 |
| round-1 | workflow v1 | C | 106/106 | 121,998 | 5,224,057 | 55 | $0.191 |
| round-1 | workflow v1 | Go | 106/106 | 113,306 | 3,365,415 | 40 | $0.162 |
| round-1 | workflow v1 | JavaScript | 106/106 | 90,736 | 2,959,548 | 40 | $0.134 |
| round-1 | workflow v1 | Python | 106/106 | 125,031 | 6,222,745 | 65 | $0.200 |
| round-1 | workflow v1 | Rust | 106/106 | 156,063 | 9,405,161 | 85 | $0.255 |
| round-2 | workflow v2 | C | 112/112 | 118,885 | 6,107,267 | 60 | $0.192 |
| round-3 | workflow v2 | C | 112/112 | 142,225 | 6,068,702 | 57 | $0.215 |
| round-4 | workflow v2 | C | 112/112 | 117,365 | 6,633,406 | 66 | $0.198 |
| round-5 | workflow v2 | C | 112/112 | 121,115 | 6,223,051 | 57 | $0.205 |
| round-6 | workflow v2 | C | 112/112 | 108,581 | 3,997,438 | 45 | $0.163 |
| round-7 | workflow v2 | C | 112/112 | 154,778 | 12,365,110 | 94 | $0.287 |
| round-2 | workflow v2 | Go | 112/112 | 110,406 | 3,455,929 | 37 | $0.163 |
| round-3 | workflow v2 | Go | 112/112 | 69,835 | 1,891,864 | 34 | $0.101 |
| round-4 | workflow v2 | Go | 112/112 | 170,119 | 6,404,923 | 58 | $0.251 |
| round-5 | workflow v2 | Go | 112/112 | 101,824 | 2,538,895 | 32 | $0.143 |
| round-6 | workflow v2 | Go | 112/112 | 102,388 | 4,517,854 | 60 | $0.159 |
| round-7 | workflow v2 | Go | 112/112 | 78,537 | 4,149,141 | 61 | $0.128 |
| round-2 | workflow v2 | JavaScript | 112/112 | 119,392 | 4,913,756 | 53 | $0.180 |
| round-3 | workflow v2 | JavaScript | 112/112 | 67,714 | 1,351,236 | 23 | $0.095 |
| round-4 | workflow v2 | JavaScript | 112/112 | 157,602 | 9,721,332 | 80 | $0.260 |
| round-5 | workflow v2 | JavaScript | 112/112 | 93,405 | 2,785,938 | 39 | $0.137 |
| round-6 | workflow v2 | JavaScript | 112/112 | 96,639 | 4,513,010 | 55 | $0.153 |
| round-7 | workflow v2 | JavaScript | 112/112 | 87,820 | 3,007,457 | 41 | $0.132 |
| round-2 | workflow v2 | Python | 112/112 | 82,123 | 3,323,523 | 49 | $0.128 |
| round-3 | workflow v2 | Python | 112/112 | 106,168 | 5,585,811 | 64 | $0.171 |
| round-4 | workflow v2 | Python | 112/112 | 78,385 | 3,121,625 | 45 | $0.121 |
| round-5 | workflow v2 | Python | 112/112 | 154,250 | 11,614,289 | 103 | $0.266 |
| round-6 | workflow v2 | Python | 112/112 | 86,096 | 3,667,690 | 46 | $0.135 |
| round-7 | workflow v2 | Python | 112/112 | 123,874 | 6,737,225 | 69 | $0.200 |
| round-2 | workflow v2 | Rust | 112/112 | 104,408 | 4,194,694 | 54 | $0.158 |
| round-3 | workflow v2 | Rust | 112/112 | 112,031 | 4,635,085 | 58 | $0.172 |
| round-4 | workflow v2 | Rust | 112/112 | 125,886 | 8,193,632 | 85 | $0.211 |
| round-5 | workflow v2 | Rust | 112/112 | 140,846 | 6,020,679 | 56 | $0.218 |
| round-6 | workflow v2 | Rust | 112/112 | 135,279 | 8,987,971 | 85 | $0.230 |
| round-7 | workflow v2 | Rust | 112/112 | 131,824 | 7,532,337 | 87 | $0.215 |