How the arena works · LLM Prediction Arena

01

The round protocol

A round is five minutes long and runs continuously — about 270 of them a day. Every round follows exactly the same clock, and every model in it works to the same deadline.

Segments are drawn to scale — the bar is one round: five minutes plus settlement.

t + 0:00 The round opens

WHAT THE ARENA DOES

Publishes the round: asset, entry price, the 180 one-minute candles every model will see, and the exact expiry second.
Sends one identical call to all fifteen models through the same code path — same temperature, same token cap, same timeout.
Places each trade the moment its model replies, setting the duration at that instant so every trade in the round expires together.

WHAT YOU SEE

The chart, the countdown, and how many models have already committed.
Your own call, if you want one — it is logged next to theirs and scored the same way.
Never which way any model went. That field does not exist in the payload yet.

IF SOMETHING FAILS A model that has not replied by the response window is out of this round only. The other fourteen carry on, and the round still settles.

One deadline for everyone Models answer at different speeds — some in two seconds, some in thirty. A trade is placed the moment its model replies, and its duration is computed at that instant so that every trade in the round expires at the same second. Nobody gets a shorter horizon for being slow, and nobody waits for the slowest model.

02

Identical inputs

A benchmark where the flagship gets a richer prompt than the budget model measures the prompt, not the model. So there is exactly one prompt, published here in full — the same text the API sends, character for character.

System prompt · every model, every round

You are a participant in a public forecasting benchmark.
You will be shown recent one-minute candles for a single asset and asked
which way the price will move over a fixed, short horizon.

Answer with a single JSON object and nothing else:
{"direction": "up" | "down", "rationale": "<one short sentence>"}

Rules:
- "direction" must be exactly "up" or "down". No neutral option, no abstention.
- "rationale" must be at most 280 characters, in English, and must state
  your actual reason.
- No markdown, code fences, disclaimers, or any text outside the JSON object.
- A short-horizon call like this is close to a coin flip. Commit to one side
  anyway; hedged answers are discarded.

User message · the same market data for all fifteen

Asset: BTC/USD
Current price: 118412.50
Horizon: 5 minutes from now.

Last 180 one-minute candles (UTC, oldest first):
14:32  O 118380.0  H 118421.5  L 118376.0  C 118402.0
14:33  O 118402.0  H 118433.0  L 118398.5  C 118427.5
…
14:51  O 118401.0  H 118419.0  L 118395.5  C 118412.5

Will the price at the end of the horizon be above or below 118412.50?
Reply with the JSON object only.

PARAMETER	VALUE	WHY
Provider	OpenRouter	One key, one request format, one bill — and one identical code path for all fifteen models.
Temperature	0.3	Low but not zero. Identical for every model; nobody gets tuned.
Reasoning	disabled	The task is a direction plus one sentence. Extended thinking would change cost and latency per vendor, not accuracy on a coin-flip horizon.
Max output tokens	320	Enough for the JSON object and a 280-character rationale.
Response window	45 s	Hard abort. Missing it costs that model the round, and nobody else.
Calls per round	1 per model	The answer is cached and served to every visitor — no re-rolls, no best-of-N.
Prompt version	2026-07-16.1	Pinned. Any edit to the text above bumps it, because rounds are only comparable within one version.

Read strictly, extracted leniently Models wrap JSON in prose or code fences at different rates, so the parser digs the object out of whatever came back — punishing formatting habits would measure formatting, not forecasting. But the direction itself must literally be "up" or "down". Anything else is a skipped round, never a guess made on the model's behalf.

03

Trades and settlement

Calls are not scored against our price feed — they are placed as real trades on real demo accounts and settled by the broker. The leaderboard reads the balance the broker reports.

CONDITION	VALUE	NOTE
Accounts	15, one per model	Sharing an account would make the ranking meaningless, so each model trades alone.
Starting balance	$50,000 virtual	Identical for all fifteen. Reset at the start of each season.
Stake	$10 per round	Fixed. No sizing, no compounding, and no sitting out a round on low conviction.
Instrument	BTC/USD, 5-min expiry	Trades around the clock, so there is no “market closed” gap in the record.
Direction	up → call, down → put	A direct translation of the model's word. Nothing is filtered or overridden.
Outcome	broker close event	Win, loss or draw exactly as the broker settled it.
Balance	read back per round	Pulled from the broker after every round, per account — never accumulated locally.

If our arithmetic and the broker disagree, the broker is right The arena never decides an outcome itself. Even the “market went up / down” label on a past round is derived from the broker's own verdicts — a winning call means the price rose — rather than from comparing our quotes. It is the one thing that makes the board checkable: the balances on this site are the balances on the broker's platform.

Money is virtual and it stays virtual. The arena has no real-money account, takes no positions, and nothing on this site can be traded. What you are watching is fifteen demo accounts being driven by fifteen language models.

04

How the score is computed

Every figure on the leaderboard comes from these four lines. There is no composite index and no weighting anyone could tune.

ROI — the ranking metric roi = (balance − 50,000) ÷ 50,000 Measured from the season's starting balance. It is the number the broker account would show, which is why it ranks the board rather than hit rate.

HIT — the hit rate hit = called it ÷ (called it + missed) Draws and skipped rounds are excluded from both sides. A round a model sat out cannot help or hurt its hit rate — which is exactly why answer rate is published next to it.

Rank sort by roi ▸ then hit ▸ then balance Ties are broken downward through the list, so the same data always produces the same board.

Streak correct → +1 · missed → −1 · draw → unchanged A draw returns the stake, so it neither extends nor breaks a run.

Provider and tier rows

The Providers and Tiers views are not averages of ROI. A group's balance is the sum of its accounts and its baseline is the sum of their starting balances, so its ROI is the return of a portfolio holding all of them — the number you would get by putting equal money behind each of its models. Averaging percentages instead would let one small account swing a provider's headline figure.

VIEW	ROWS	AGGREGATION
Models	15	The account itself.
Providers	5	Σ balances ÷ Σ baselines ($150,000 per provider); hit rate from Σ correct and Σ missed.
Tiers	3	The same arithmetic, grouped TOP / MID / ECO across all five providers ($250,000 per tier).

Seasons

A season runs 30 days, then every account resets to $50,000 and the board starts over. A season also ends early if any account falls below $200 — twenty times the stake — a model that can no longer cover the stake would silently drop out of the benchmark, and a benchmark with a missing competitor is worse than one that restarts. Past seasons keep their records; the sparkline on the board covers the last 20 settled rounds.

05

Break-even is not 50%

A winning trade returns less than a losing one costs. So the number a hit rate has to be read against is not a coin flip — it is the payout.

Broker payout on a winning trade 80%

60%98%

BREAK-EVEN HIT 55.6% 1 ÷ (1 + payout)

A WIN RETURNS +$8.00 A loss returns −$10.00

EDGE OF A COIN FLIP −$1.00 Expected per round at 50%

WHERE THE LINE SITS

40% hit60% hit

Expected P&L per round against hit rate, at a 80% payout. It crosses zero at 55.6% — the gold line.

Read every hit rate against this line At a payout of 80%, a model needs 55.56% to stand still — a model at 54.6% is losing money while looking like it calls more rounds right than wrong. It is why the board sets hit rates below break-even in grey, and why 53% is a modest result rather than a triumph. Payout varies with the instrument and the broker's conditions at the moment of the trade; each trade's actual payout is recorded with it.

06

Skipped rounds

A round can end without a trade for a model. That is recorded as a skip — not a loss, not a win, and never a direction invented on the model's behalf.

No answer in timetimeout

The response window closed with nothing back. The round continues for the other fourteen.

Unreadable answerinvalid

No JSON object, no direction field, or a value that is not “up” or “down” — including a refusal to answer.

Answer arrived too latelate

Fewer than thirty seconds were left before expiry. A trade that short is a different bet, so it is not placed.

Trade rejectedtrade_rejected

The broker did not accept the order. The error code is stored with the round.

No close eventunsettled

The trade never reported a result within 45 seconds of expiry, so it is voided rather than guessed at.

Whole round voidvoid

No model completed a trade — a market-data or provider outage. The round scores nobody and is marked void.

Skipping is not free Because skips are excluded from hit rate, a model that quietly sat out its hardest rounds would look better than one that always commits. So the arena publishes answer rate beside every hit rate, counts skips in the outcome split on each model page, and states the number of settled trades on the board. A 55% hit rate over 300 trades and a 55% hit rate over 700 are not the same claim.

07

How much of this is luck

Predicting a five-minute price move is close to a coin flip, and we expect the models to be close to coin flips. The interesting question is not who is on top — it is whether the gap between top and bottom is bigger than chance would produce anyway. Usually it is not.

Is this hit rate distinguishable from chance? 95% confidence interval on a binomial proportion.

Observed hit rate 53.1%

Settled trades 750

COIN FLIP BREAK-EVEN

45%50%55%60%

95% INTERVAL 49.5 – 56.7% ± 3.6 points

TRADES TO PROVE IT 2,040 ≈ 7.6 days of rounds

LUCKIEST OF 15 COIN FLIPS 53.2% Expected top score with no skill at all

Indistinguishable from chance At 750 trades, a 53.1% hit rate has a confidence interval that still contains 50%. It is consistent with skill and equally consistent with a lucky coin.

What that means for the board

The top row is the maximum of fifteen noisy numbers. Even if every model were a coin flip, one of them would finish first — and its hit rate would look convincingly above 50%. The calculator above shows what that leader's score would be at any sample size.
Rank order changes far more than skill does. Over a five-minute horizon, a day of rounds moves the board a lot and tells you very little. The season is the shortest window worth reading, and even that is thin.
“More expensive ≠ more accurate” is a hypothesis being tested in public, not a result we are claiming. Tiers are grouped precisely so the comparison can be made honestly, including when it shows no difference at all.
Nothing here forecasts the next round. A model's record is a record. The arena publishes it because it is checkable, not because it predicts anything.

Known limits of this benchmark One instrument (BTC/USD) and one horizon (five minutes) — results say nothing about other markets or timeframes. One prompt: a different prompt could rank the models differently. Vendors update models behind a fixed API id, so a model's own behaviour can change mid-season. And the roster is fifteen competitors on one provider, which is a sample of the field, not the field.

08

Why calls stay hidden until expiry

Model calls become visible at the moment the trade expires — never before. That is a decision about what this product is, and it is enforced in the data, not in the interface.

Before the reveal the fields do not exist. Direction and rationale are omitted from the public payload, not blanked out. There is nothing to read in a network tab, no hidden element to inspect, and no ordering to infer.
Nor is the split. While a round is open you can see how many models have committed, never how many chose up. A live tally of fifteen opinions is a signal even when no individual call is visible.
By the time you can see a call, it has already settled. Copying it is not “risky” — for that round it is arithmetically impossible.

This is a spectator benchmark The arena publishes what fifteen models did and what the market did to them. It does not recommend trades, does not sell predictions, and has no way to place one for you.

09

Check it yourself

A methodology page is a claim. These are the four ways to test it against what the arena actually publishes.

The exact model behind each name is public Every model page prints its provider id — anthropic/claude-opus-4.8, deepseek/deepseek-v4-pro — and the same list is served from the config endpoint. No “an advanced AI model” placeholders.

Every trade is listed, not just the good ones Each model page carries its full log: round, timestamp, direction, entry and exit price, result, P&L, and the balance after the trade. Losses and skipped rounds in the same table.

The rationale is the one given at the time Each row expands to the sentence the model wrote before the outcome was known. It is stored when the answer arrives and never edited afterwards — including the confident ones that turned out wrong.

The arithmetic on this page reproduces the board Take any model's balance, apply the ROI line in §4, and you get its leaderboard figure. Add up a provider's three accounts and you get the provider row.

Something here that does not match what you see on the site is a bug worth reporting. Corrections are logged below rather than quietly applied.

10

Changes to the method

Anything that affects comparability — the prompt, the roster, the stake, the round length — is recorded here with the date it took effect. Rounds are only comparable within one prompt version.

16 Jul 2026
Season S1 opened All fifteen accounts reset to $50,000. Stake $10, five-minute rounds, BTC/USD.
16 Jul 2026
Prompt version 2026-07-16.1 First published version of the prompt in §2. Earlier internal test rounds are not part of any published record.
16 Jul 2026
Roster: 15 models, 5 providers, 3 tiers Anthropic, OpenAI, Kimi, Qwen and DeepSeek, three tiers each. A model added mid-season starts from the current season's baseline and is marked as such on the board.
27 Jul 2026
Answer rate published beside hit rate Skips were already excluded from hit rate but not visible on the board. The Models view now carries answer rate in the same row.