Reasoning effort experiment
TL;DR: Per-token price is not per-task cost: in this experiment, across 7 models and 22 short-answer tasks, hidden reasoning changed the observed bill by up to 31.51× versus a model’s baseline effort.
Experiment run 2026-10-22 · 22 short-answer tasks · one pass per cell · benchmark list-price snapshot, not a current rate quote. Results JSON · Runner/suite · Estimate with your own token counts
Methodology and limitations
Single pass per task and effort; small suite (22 tasks). Reasoning-token counts vary run to run and by task type, so treat per-task costs as indicative ranges, not fixed prices. Truncated answers (finish_reason=length) count as incorrect.
These tasks are short-answer: absolute reasoning-token counts are far below long or agentic workloads, where multipliers can be larger and must be measured separately. This is a single pass per cell, not a repeated-trial estimate or a prediction for your workload.
45 failed calls: 44 in unavailable-effort cells and 1 other failure(s); 0 budget skips. Failed calls do not enter successful-call accuracy or cost averages; unavailable is not zero-cost success.
Runner settings, grading and effort mapping
- question
- Per-token list prices do not change with reasoning effort; the number of hidden reasoning tokens does, and they are billed as output tokens. For each model, what is the list-price cost per task at each effort level, and how does accuracy change?
- inference Route
- OpenRouter chat completions with the unified reasoning.effort parameter (reasoning text excluded from the response; still billed)
- cost Basis
- AI Pricing Guru direct-provider list prices from public/pricing.json: prompt tokens x inputPerM + (reasoning + visible output tokens) x outputPerM. No cache discounts. OpenRouter billed spend is reported separately.
- reasoning Token Source
- usage.completion_tokens_details.reasoning_tokens; cells where a provider does not report it are flagged reasoningReported=false and their cost still uses total completion tokens.
- effort Mapping
- OpenAI/xAI receive effort natively. For Anthropic, OpenRouter converts effort into a thinking budget proportional to max_tokens, so Anthropic results are only comparable at the same max_tokens. Gemini 3 maps effort to thinkingLevel. 'none' disables reasoning where the route allows it.
- max Output Tokens
- 16000
- temperature
- provider default (not sent)
- grading
- deterministic exact/number/regex/contains/json_fields graders; no LLM judge. number/exact graders ignore markdown emphasis, code ticks, a leading $ and thousands separators; a worked explanation instead of the bare answer still fails.
- monthly Cap Usd
- 100
- run Budget Usd
- 15
- concurrency
- 4
Cost versus hard-task accuracy
Per-model effort ladders
Costs scale the observed task mix, not a volume quote; accuracy uses successful calls only. Baseline is the first available effort. Tables scroll horizontally.
| Effort | Cost / 1,000 tasks | × baseline | Accuracy | Hard accuracy | Avg reasoning tokens |
|---|---|---|---|---|---|
| low | $0.5089 | 1.00× | 95.5% | 92.9% | 22.0 |
| medium | $0.5598 | 1.10× | 100% | 100% | 26.7 |
| high | $0.7789 | 1.53× | 100% | 100% | 48.5 |
| none | Not available — reasoning cannot be disabled on this endpoint. | ||||
Experiment run 2026-10-22 · 22 short-answer tasks · one pass per cell. Not a general model-cost estimate or current quote. Experiment methodology · Pricing methodology.
| Effort | Cost / 1,000 tasks | × baseline | Accuracy | Hard accuracy | Avg reasoning tokens |
|---|---|---|---|---|---|
| none | $0.0142 | 1.00× | 59.1% | 42.9% | 0.0 |
| low | $0.0529 | 3.71× | 95.5% | 92.9% | 76.3 |
| medium | $0.0553 | 3.88× | 95.5% | 92.9% | 81.1 |
| high | $0.0583 | 4.10× | 95.5% | 92.9% | 87.2 |
Experiment run 2026-10-22 · 22 short-answer tasks · one pass per cell. Not a general model-cost estimate or current quote. Experiment methodology · Pricing methodology.
| Effort | Cost / 1,000 tasks | × baseline | Accuracy | Hard accuracy | Avg reasoning tokens |
|---|---|---|---|---|---|
| low | $2.0176 | 1.00× | 95.5% | 92.9% | 62.5 |
| medium | $2.1204 | 1.05× | 100% | 100% | 69.0 |
| high | $2.3004 | 1.14× | 100% | 100% | 78.0 |
| none | Not available — reasoning cannot be disabled on this endpoint. | ||||
Experiment run 2026-10-22 · 22 short-answer tasks · one pass per cell. Not a general model-cost estimate or current quote. Experiment methodology · Pricing methodology.
| Effort | Cost / 1,000 tasks | × baseline | Accuracy | Hard accuracy | Avg reasoning tokens |
|---|---|---|---|---|---|
| none | $1.4575 | 1.00× | 54.5% | 35.7% | 0.0 |
| low | $1.0457 | 0.72× | 95.5% | 92.9% | 67.8 |
| medium | $1.183 | 0.81× | 100% | 100% | 80.8 |
| high | $1.7162 | 1.18× | 100% | 100% | 133.9 |
Experiment run 2026-10-22 · 22 short-answer tasks · one pass per cell. Not a general model-cost estimate or current quote. Experiment methodology · Pricing methodology.
| Effort | Cost / 1,000 tasks | × baseline | Accuracy | Hard accuracy | Avg reasoning tokens |
|---|---|---|---|---|---|
| minimal | $0.0954 | 1.00× | 76.2% | 64.3% | 0.0 · 1 failed call(s) |
| low | $1.0773 | 11.30× | 100% | 100% | 261.9 |
| medium | $2.0796 | 21.81× | 100% | 100% | 529.7 |
| high | $3.0044 | 31.51× | 100% | 100% | 776.0 |
Experiment run 2026-10-22 · 22 short-answer tasks · one pass per cell. Not a general model-cost estimate or current quote. Experiment methodology · Pricing methodology.
| Effort | Cost / 1,000 tasks | × baseline | Accuracy | Hard accuracy | Avg reasoning tokens |
|---|---|---|---|---|---|
| none | $0.072 | 1.00× | 45.5% | 28.6% | 0.0 |
| high | $0.7095 | 9.86× | 100% | 100% | 264.8 |
Experiment run 2026-10-22 · 22 short-answer tasks · one pass per cell. Not a general model-cost estimate or current quote. Experiment methodology · Pricing methodology.
| Effort | Cost / 1,000 tasks | × baseline | Accuracy | Hard accuracy | Avg reasoning tokens |
|---|---|---|---|---|---|
| low | $2.3889 | 1.00× | 95.5% | 100% | 296.3 |
| high | $4.1595 | 1.74× | 100% | 100% | 591.4 |
Experiment run 2026-10-22 · 22 short-answer tasks · one pass per cell. Not a general model-cost estimate or current quote. Experiment methodology · Pricing methodology.
What this experiment shows
- In this test, the largest cost multiplier: Gemini 3.6 Flash at high, 31.51× its baseline.
- Claude Sonnet 5: turning reasoning off costs more per task than low effort in this run.
- In this test, reasoning could not be disabled: GPT-6.1 Sol, Claude Opus 5.5; the endpoint rejects the none setting.
- In this test, GPT-6.1 Sol: medium was the cheapest measured effort reaching its best overall accuracy, 100%, at $0.5598 per 1,000 tasks.
- In this test, GPT-6 Luna: low was the cheapest measured effort reaching its best overall accuracy, 95.5%, at $0.0529 per 1,000 tasks.
- In this test, Claude Opus 5.5: medium was the cheapest measured effort reaching its best overall accuracy, 100%, at $2.1204 per 1,000 tasks.
- In this test, Claude Sonnet 5: medium was the cheapest measured effort reaching its best overall accuracy, 100%, at $1.183 per 1,000 tasks.
- In this test, Gemini 3.6 Flash: low was the cheapest measured effort reaching its best overall accuracy, 100%, at $1.0773 per 1,000 tasks.
- In this test, DeepSeek V4 Pro: high was the cheapest measured effort reaching its best overall accuracy, 100%, at $0.7095 per 1,000 tasks.
- In this test, Grok 4.6: high was the cheapest measured effort reaching its best overall accuracy, 100%, at $4.1595 per 1,000 tasks.