Frontier Model Tokenizers: Real Pricing Impact
PlayCode measured the same code across frontier model tokenizers. Here is why Claude, GPT, Gemini, and Grok prices are not comparable by $/M tokens alone.
By AI Pricing Guru Editorial Team
AI Pricing Guru articles are maintained by the editorial workflow behind the site: daily pricing snapshots, provider source checks, and review passes for model launches, subscription limits, and billing changes.
A new PlayCode analysis is spreading because it attacks the most comfortable shortcut in AI pricing: tokens multiplied by price.
That shortcut only works if a token means the same thing across providers. It does not. The same file can become a different number of billable tokens depending on the model tokenizer, and PlayCode’s measured example is sharp: the same 2,888-character TypeScript file counted as 681 tokens on GPT-5.x’s o200k tokenizer and 1,178 tokens on Claude’s newest tokenizer. That is 1.73x as many billable tokens before comparing the published per-token price.
This is not a public rate-card change from Anthropic, OpenAI, Google, or xAI. The pricing impact is about effective cost. If two frontier models both advertise “$5 per million input tokens,” but one tokenizer turns your codebase into 30%, 50%, or 73% more tokens, the invoice is not equal.
For live sticker prices, compare Anthropic pricing, OpenAI pricing, Google AI pricing, xAI pricing, and the full AI API pricing table. Then model your workload in the AI token calculator. For a related hidden-cost angle, read our breakdown of Claude Code token overhead.
What changed
The source is PlayCode’s July 13 post, “The Same TypeScript Costs 73% More on Claude Than on GPT”, surfaced by today’s HN Model Launch alert.
PlayCode measured real fixtures through provider tokenizers: English prose, HTML, JavaScript, Python, TypeScript, Rust, JSON tool schemas, Chinese text, and an agent system prompt. The important part is that the analysis separates two different effects:
| Effect | Comparison | Reported result | Pricing read |
|---|---|---|---|
| Same-vendor tokenizer change | Claude old tokenizer vs Claude new tokenizer | +31% on TypeScript, +29% on Rust, about +32% on a blended agent turn | Same sticker price can produce more billed tokens |
| Cross-vendor tokenizer gap | Claude new tokenizer vs GPT o200k | 1.73x on TypeScript, 1.58x on Rust, 1.52x on JavaScript | $/M token comparisons understate code workload cost |
| English prose gap | Claude new tokenizer vs GPT o200k | About 1.40x | Still material, but less extreme than code |
| Chinese prose | Claude old vs new tokenizer | Roughly unchanged | Tokenizer changes are workload-specific |
PlayCode says Anthropic counts were taken from Anthropic’s official count_tokens endpoint, OpenAI counts were checked against o200k and live usage deltas, and Google/xAI were counted through provider token-count endpoints. The article also says it excluded DeepSeek and GLM from the main tables because it did not have real tokenizer counts for them.
That last point matters. The lesson is not “use one universal characters-per-token estimate.” The lesson is the opposite: use each provider’s counter on the exact content you send.
Current pricing comparison
Here are the live sticker rates in our tracker for the models most relevant to the PlayCode comparison. These are still necessary, but not sufficient, for budget math.
| Model | Input | Cached input | Output | Status |
|---|---|---|---|---|
| Claude Sonnet 5 | $2.00/M | $0.20/M | $10.00/M | Intro price through Aug. 31, 2026 |
| Claude Opus 4.8 | $5.00/M | $0.50/M | $25.00/M | Active premium Claude |
| Claude Fable 5 | $10.00/M | $1.00/M | $50.00/M | Preview |
| GPT-5.6 Sol | $5.00/M | $0.50/M | $30.00/M | Active OpenAI flagship (limited preview) |
| GPT-5.6 Sol | $5.00/M | $0.50/M | $30.00/M | Limited preview |
| Gemini 3 Flash | $0.50/M | $0.05/M | $3.00/M | Preview |
| Grok 4.5 | $2.00/M | $0.50/M | $6.00/M | Active |
Now adjust the input side by PlayCode’s TypeScript tokenizer ratios, using GPT’s o200k as the 1.00x reference.
| Route | Sticker input price | TypeScript token multiplier vs GPT | Effective input cost for same TS bytes |
|---|---|---|---|
| GPT-5.x/o200k | $5.00/M on GPT-5.6 Sol | 1.00x | $5.00 per GPT-equivalent 1M tokens |
| Grok 4.5 | $2.00/M | 1.05x | About $2.10 per GPT-equivalent 1M tokens |
| Gemini 3 Flash | $0.50/M | 1.16x | About $0.58 per GPT-equivalent 1M tokens |
| Claude Opus 4.8 | $5.00/M | 1.73x | About $8.65 per GPT-equivalent 1M tokens |
| Claude Sonnet 5 launch | $2.00/M | 1.73x | About $3.46 per GPT-equivalent 1M tokens |
| Claude Sonnet 5 standard | $3.00/M | 1.73x | About $5.19 per GPT-equivalent 1M tokens |
This table is intentionally narrow. It only adjusts input tokens for a TypeScript-heavy workload using PlayCode’s reported ratios. It does not say which model is best, and it does not adjust output tokenization or quality. But it shows the budget trap: the sticker input price is not the same thing as the cost to process a file.
The Claude Sonnet 5 timing issue
The most important near-term detail is Claude Sonnet 5’s launch pricing.
In our live data, Sonnet 5 is $2 input / $0.20 cached input / $10 output per million tokens through August 31, 2026. The pricing note says standard pricing starts September 1 at $3 input / $0.30 cached input / $15 output.
PlayCode’s claim is that the new Claude tokenizer makes the same blended agent request about 32% larger than the previous Claude tokenizer. During launch pricing, the lower sticker price mostly cushions that. After the launch window, the cushion disappears while the tokenizer behavior remains.
That is the practical warning for teams signing annual budgets in July: if your evals use Sonnet 5 during the intro window, model both the sticker-price change and the token-count change. Otherwise a workload that looks attractive in August can become materially more expensive in September.
What this means for coding-agent buyers
Coding agents are the worst place to ignore tokenizer differences because their inputs are not plain chat messages.
A typical coding-agent turn can include:
- a long system prompt
- repository instructions
- tool schemas
- JSON tool results
- TypeScript, JavaScript, Python, Rust, or HTML files
- previous patches and test output
- summaries from prior turns
That content mix is exactly where PlayCode reports the largest gaps: code, JSON, and long agent prompts. The same repository context can look compact under one tokenizer and expensive under another.
This does not mean Claude is a bad buy for coding. Many teams pay for Claude because it produces better code, handles long context well, or reduces failed attempts. A model that costs more per byte can still be cheaper per merged change if it solves the task faster.
But it does mean “$2/M input” or “$5/M input” is not enough for procurement. The right metric is cost per accepted task:
| Metric | Why it matters |
|---|---|
| Tokens per file under each provider | Converts sticker price into comparable input cost |
| Cache hit rate | Large repeated prompts can be much cheaper if stable |
| Request count per task | A cheap tokenizer can lose if the model needs more turns |
| Output length | Output tokens often dominate coding-agent bills |
| Pass rate | Failed edits, retries, and human review are part of cost |
Practical advice
First, build a tokenizer audit set. Include the content your app really sends: TypeScript files, JSON schemas, markdown docs, tool results, system prompts, logs, multilingual snippets, and long retrieved context. Do not rely on a generic “four characters per token” estimate.
Second, run each provider’s official token counter before a migration. Use Anthropic’s token-count endpoint for Claude, OpenAI’s documented tokenizer and live usage checks for GPT, and provider counters for Gemini and Grok when available. Keep the fixtures versioned so pricing changes are comparable over time.
Third, separate sticker price from effective price. For each model, calculate:
effective input cost = provider token count / reference token count * input price
Then repeat the same exercise for cached input, output-heavy prompts, and batch jobs.
Fourth, re-run the audit when a model version changes. PlayCode’s most actionable same-vendor finding is that a newer tokenizer can raise billed tokens even if the public $/M token rate does not move.
Finally, do not optimize only for tokenizer compactness. Add quality, latency, retry rate, and review time. The cheapest tokenizer is not automatically the cheapest product decision.
Who benefits
OpenAI benefits from having a stable, compact o200k reference for many code-heavy workloads. Its sticker price may not always be the lowest, but the token count can make the real input cost more competitive than a raw rate card suggests.
Google and xAI benefit when their lower or mid-tier prices combine with token counts close to GPT’s on code. In PlayCode’s TypeScript row, Grok 4.5 was only 1.05x GPT’s token count and Gemini 3 Flash was 1.16x, before considering their lower sticker prices in our tracker.
Anthropic still benefits where Claude’s coding quality, long-context behavior, and prompt caching beat the alternatives. The risk is budget surprise: if buyers compare only $/M tokens, Claude workloads can look cleaner in the spreadsheet than they do on the invoice.
Bottom line
PlayCode’s post is a useful reminder that frontier-model pricing has three layers:
| Layer | Question |
|---|---|
| Sticker price | What does the provider charge per million tokens? |
| Tokenizer price | How many tokens does my actual content become? |
| Outcome price | How many attempts, outputs, reviews, and retries does the task need? |
The rate card shows only the first layer. For code-heavy teams, the second layer can move costs by double-digit percentages before quality enters the picture.
If you are evaluating Claude Sonnet 5, Claude Opus 4.8, GPT-5.6, Gemini 3, or Grok 4.5 for a coding agent this week, run the same repository fixtures through every tokenizer and price the resulting counts. Tokens times price is still the formula, but only after you measure the tokens for your workload.
Sources: PlayCode’s frontier tokenizer pricing analysis, AI Pricing Guru’s live pricing dataset, and our Claude Code token overhead pricing analysis.
FAQ
Why can the same file cost more on one model?
Each provider tokenizer splits text differently. If one tokenizer turns the same file into more billable tokens, the effective input cost rises even when the published $/M token price is identical.
Is Claude always more expensive for coding?
No. PlayCode’s tokenizer data says some code-heavy inputs become more tokens under Claude’s newest tokenizer than under GPT’s o200k tokenizer. But total cost still depends on model quality, number of turns, cache behavior, output length, and whether the task succeeds.
What should teams measure before switching models?
Measure provider token counts on real prompts, code files, tool schemas, and tool results. Then combine those counts with input price, cached-input price, output price, pass rate, and retry count.