How the arena works

01

The round protocol

A round is one minute long and runs continuously — about a thousand of them a day. Every round follows exactly the same clock, and every model in it works to the same deadline.

Segments are drawn to scale — the bar is one round, one minute, plus the gap before the next. Reveal is the exception: it takes an instant, and is drawn wide enough to press.

t + 0:00 The round opens

WHAT THE ARENA DOES

Publishes the round: asset, entry price, the 180 one-minute candles every model will see, and the exact expiry second.
Sends one identical call to all fifteen models through the same code path — same temperature, same token cap, same timeout.
Records each answer the moment its model replies. Every answer in the round is judged on the same expiry, whenever it arrived.

WHAT YOU SEE

The chart, the countdown, and how many models have already committed.
Your own call, if you want one — it is logged next to theirs and scored the same way.
Never which way any model went. That field does not exist in the payload yet.

IF SOMETHING FAILS A model that has not replied by the response window is out of this round only. The other fourteen carry on, and the round still settles.

One deadline for everyone Models answer at different speeds — some in two seconds, some in thirty. Every answer in a round is judged on the same open price and the same close price whenever it arrived, so a slow vendor costs its model latency and nothing else. Nobody gets a shorter horizon for being slow, and nobody waits for the slowest model.

02

Identical inputs

A benchmark where the flagship gets a richer prompt than the budget model measures the prompt, not the model. So there is exactly one prompt, published here in full — the same text the API sends, character for character.

System prompt · every model, every round

You are a participant in a public forecasting benchmark.
You will be shown recent one-minute candles for a single asset and asked
which way the price will move over a fixed, short horizon.

Answer with a single JSON object and nothing else:
{"direction": "up" | "down", "rationale": "<one short sentence>"}

Rules:
- "direction" must be exactly "up" or "down". No neutral option, no abstention.
- "rationale" must be at most 280 characters, in English, and must state
  your actual reason.
- No markdown, code fences, disclaimers, or any text outside the JSON object.
- A short-horizon call like this is close to a coin flip. Commit to one side
  anyway; hedged answers are discarded.

User message · the same market data for all fifteen

Asset: BTC/USD
Current price: 118412.50
Horizon: 1 minute from now.

Last 180 one-minute candles (UTC, oldest first):
14:32  O 118380.0  H 118421.5  L 118376.0  C 118402.0
14:33  O 118402.0  H 118433.0  L 118398.5  C 118427.5
…
14:51  O 118401.0  H 118419.0  L 118395.5  C 118412.5

Will the price at the end of the horizon be above or below 118412.50?
Reply with the JSON object only.

PARAMETER	VALUE	WHY
Provider	OpenRouter	One key, one request format, one bill — and one identical code path for all fifteen models.
Temperature	0.3	Low but not zero. Identical for every model; nobody gets tuned.
Reasoning	disabled	The task is a direction plus one sentence. Extended thinking would change cost and latency per vendor, not accuracy on a coin-flip horizon.
Max output tokens	320	Enough for the JSON object and a 280-character rationale.
Response window	45 s	Hard abort. Missing it costs that model the round, and nobody else.
Calls per round	1 per model	The answer is cached and served to every visitor — no re-rolls, no best-of-N.
Prompt version	2026-07-16.1	Pinned. Any edit to the text above bumps it, because rounds are only comparable within one version.

Read strictly, extracted leniently Models wrap JSON in prose or code fences at different rates, so the parser digs the object out of whatever came back — punishing formatting habits would measure formatting, not forecasting. But the direction itself must literally be "up" or "down". Anything else is a skipped round, never a guess made on the model's behalf.

03

How a round is decided

One rule, stated in advance, over a price series anyone can pull: a round went up if the price at expiry is above the price at open.

CONDITION	VALUE	NOTE
Series	BTC/USD spot	The same series the chart draws and the same candles the models are prompted with. Trades around the clock, so there is no “market closed” gap in the record.
Horizon	1 minute	From the second the round opens to the second it expires. Both timestamps are published with the round.
Open price	read at the open	The last traded price when the round is created. It is what the models see as “current price”.
Close price	read at expiry	Both ends come from one source. A move built from two sources is a move the market never had.
Verdict	close > open → up	Strictly greater. An exact tie is a draw and scores nobody.
Stake	none	Nothing is staked and no order is placed. A call is right or wrong; it is not worth money.

We decide the outcome, and that is why you can check it The arena used to defer to a broker's settlement, on the reasoning that a third party cannot be accused of marking its own homework. It settled an instrument published nowhere, and it took a database query to find that its verdicts and the chart on this site agreed on barely half of rounds — for a whole season, unnoticed. A referee nobody can check is not neutrality; it is an unaudited dependency. A rule stated in advance over a public series is worse for us and better for you: every round publishes both prices and both timestamps, and the exchange will serve you the same candles.

No money is involved at any point. The arena holds no account, places no order and takes no position. What you are watching is fifteen language models calling a coin-flip-hard question every minute, on the record, with the answers sealed until the clock runs out.

04

How the score is computed

Every figure on the leaderboard comes from these four lines. There is no composite index and no weighting anyone could tune.

HIT — the ranking metric hit = called it ÷ (called it + missed) Draws and skipped rounds are excluded from both sides. A round a model sat out cannot help or hurt its hit rate — which is exactly why answer rate is published next to it.

±95% — the interval ± 1.96 × √(hit × (1 − hit) ÷ calls) Published beside every rate, because the gaps this board is read for are usually smaller than it. Over 500 calls it is about ±4.4 points: two rows a point apart are the same result.

Rank sort by hit ▸ then calls ▸ then name Calls break the tie because two models on the same rate are not the same evidence. Ties resolve downward through the list, so the same data always produces the same board.

Streak correct → +1 · missed → −1 · draw → unchanged A round that closed exactly where it opened was nobody's call to get right, so it neither extends nor breaks a run.

Provider and tier rows

The Providers and Tiers views are not averages of the model rates. A group's hit rate is pooled from its members' calls — every call counted once, wherever it came from — so three models that answered 500, 500 and 5 times do not each get a third of the say. Pooling also triples the evidence behind a row, which is why a provider's interval narrows enough to separate providers long before any model's is narrow enough to separate models.

VIEW	ROWS	AGGREGATION
Models	15	The model's own calls.
Providers	5	Σ correct ÷ Σ decided across that provider's three models.
Tiers	3	The same arithmetic, grouped TOP / MID / ECO across all five providers.

Seasons

A season runs 30 days, then every model starts again at nought of nought. It ends on that date and on nothing else — a season used to also end early when an account neared the point where it could no longer cover its stake, and a hit rate cannot run out. Past seasons keep their records; the curve on the board is the season's rate as it stood at each of the last 40 settled rounds, which is why it tightens as the season runs rather than jumping about.

05

50% is the bar, and 53% may not clear it

A two-sided call over one minute is close to a coin flip, so the bar is 50%. The harder question is how many rounds it takes before a number above 50% means anything.

Calls scored 500

505,000

INTERVAL AT 50% ±4.4pp 1.96 × √(0.25 ÷ calls)

SMALLEST REAL EDGE 54.4% Below this, chance explains it

THAT IS 11 hours At ~1,080 rounds a day

WHERE THE LINE SITS

50 calls5,000 calls

How wide the interval is as calls accumulate. At 500 calls it is ±4.4pp — the gold line. It narrows with the square root, so ten times the evidence buys about three times the precision.

Read every hit rate against its own interval At 500 calls the interval is ±4.4 points, so anything under 54.4% is a rate chance alone would produce often enough not to be news. It is why the board sets a rate in ink only where its interval clears 50% outright, and why the first weeks of a season separate almost nobody. It is also why the board can honestly show one model above another and mean nothing by it: rank is an ordering, not a claim that the order is real.

06

Skipped rounds

A round can end without a call for a model. That is recorded as a skip — not a loss, not a win, and never a direction invented on the model's behalf.

No answer in timetimeout

The response window closed with nothing back. The round continues for the other fourteen.

Unreadable answerinvalid

No JSON object, no direction field, or a value that is not “up” or “down” — including a refusal to answer.

Answer arrived too latelate

Fewer than thirty seconds were left before expiry. A call that short is answering a different question, so it is not scored.

Answer after the lockafter_lock

The answer arrived after the round locked. What it said is stored with the round; it is not scored.

No closing priceunsettled

No closing price arrived within 45 seconds of expiry, so the round is voided rather than guessed at.

Whole round voidvoid

No model completed a round — a market-data or provider outage. The round scores nobody and is marked void.

Skipping is not free Because skips are excluded from hit rate, a model that quietly sat out its hardest rounds would look better than one that always commits. So the arena publishes answer rate beside every hit rate, counts skips in the outcome split on each model page, and states the number of settled calls on the board. A 55% hit rate over 300 calls and a 55% hit rate over 700 are not the same claim.

07

How much of this is luck

Predicting a one-minute price move is as close to a coin flip as this gets, and we expect the models to be close to coin flips. The interesting question is not who is on top — it is whether the gap between top and bottom is bigger than chance would produce anyway. Usually it is not.

Is this hit rate distinguishable from chance? 95% confidence interval on a binomial proportion.

Observed hit rate 53.1%

Settled calls 750

COIN FLIP A COIN

45%50%55%60%

95% INTERVAL 49.5 – 56.7% ± 3.6 points

CALLS TO PROVE IT 2,040 ≈ 1.9 days of rounds

LUCKIEST OF 15 COIN FLIPS 53.2% Expected top score with no skill at all

Indistinguishable from chance At 750 calls, a 53.1% hit rate has a confidence interval that still contains 50%. It is consistent with skill and equally consistent with a lucky coin.

What that means for the board

The top row is the maximum of fifteen noisy numbers. Even if every model were a coin flip, one of them would finish first — and its hit rate would look convincingly above 50%. The calculator above shows what that leader's score would be at any sample size.
Rank order changes far more than skill does. Over a one-minute horizon, a day of rounds moves the board a lot and tells you less than it looks. A day is about a thousand rounds now rather than 270, and a thousand still leaves every rate ±3 points wide — enough for the order to rearrange itself overnight without a single model having changed. The season is the shortest window worth reading, and even that is thin.
“More expensive ≠ more accurate” is a hypothesis being tested in public, not a result we are claiming. Tiers are grouped precisely so the comparison can be made honestly, including when it shows no difference at all.
Nothing here forecasts the next round. A model's record is a record. The arena publishes it because it is checkable, not because it predicts anything.

Known limits of this benchmark One instrument (BTC/USD) and one horizon (one minute) — results say nothing about other markets or timeframes. One prompt: a different prompt could rank the models differently. Vendors update models behind a fixed API id, so a model's own behaviour can change mid-season. And the roster is fifteen competitors on one provider, which is a sample of the field, not the field.

08

Why calls stay hidden until expiry

Model calls become visible at the moment the round expires — never before. That is a decision about what this product is, and it is enforced in the data, not in the interface.

Before the reveal the fields do not exist. Direction and rationale are omitted from the public payload, not blanked out. There is nothing to read in a network tab, no hidden element to inspect, and no ordering to infer.
Nor is the split. While a round is open you can see how many models have committed, never how many chose up. A live tally of fifteen opinions is a signal even when no individual call is visible.
By the time you can see a call, it has already settled. Copying it is not “risky” — for that round it is arithmetically impossible.

This is a spectator benchmark The arena publishes what fifteen models did and what the market did to them. It does not recommend trades, does not sell predictions, and has no way to place one for you.

09

Check it yourself

A methodology page is a claim. These are the four ways to test it against what the arena actually publishes.

The exact model behind each name is public Every model page prints its provider id — anthropic/claude-opus-4.8, deepseek/deepseek-v4-pro — and the same list is served from the config endpoint. No “an advanced AI model” placeholders.

Every call is listed, not just the good ones Each model page carries its full log: round, timestamp, direction, the round’s open and close price, and the result. Misses and skipped rounds in the same table as the hits.

The rationale is the one given at the time Each row expands to the sentence the model wrote before the outcome was known. It is stored when the answer arrives and never edited afterwards — including the confident ones that turned out wrong.

The prices can be checked against the exchange Every round publishes its open price, its close price and both timestamps. Pull the same minute from Binance’s public API and you have everything you need to re-derive the verdict — and to catch us if it disagrees.

Something here that does not match what you see on the site is a bug worth reporting. Corrections are logged below rather than quietly applied.

10

Changes to the method

Anything that affects comparability — the prompt, the roster, the asset, the round length — is recorded here with the date it took effect. Rounds are only comparable within one prompt version.

16 Jul 2026
Season S1 opened All fifteen models start again at nought of nought. Minute-long rounds, BTC/USD.
16 Jul 2026
Prompt version 2026-07-16.1 First published version of the prompt in §2. Earlier internal test rounds are not part of any published record.
16 Jul 2026
Roster: 15 models, 5 providers, 3 tiers Anthropic, OpenAI, Kimi, Qwen and DeepSeek, three tiers each. A model added mid-season starts from the current season's baseline and is marked as such on the board.
27 Jul 2026
Answer rate published beside hit rate Skips were already excluded from hit rate but not visible on the board. The Models view now carries answer rate in the same row.