Castform Beats GPT-5.6 Sol: Cost Impact (August 2026)

Castform and Neon published a retrieval case study on August 5 showing a reinforcement-learning post-trained Qwen3.5-4B model earning a higher mean evaluation reward than GPT-5.6 Sol at a fraction of the measured inference cost.

The headline is directionally supported by the source chart, but it needs two qualifiers. The exact ratio from the displayed rollout costs is 94.9x, not 100x, and the result covers one private retrieval setup rather than general intelligence. The live chart and calculator above use Castform’s official hosted Qwen3.5-4B rate, but that public base-model endpoint is not proof that callers receive the private post-trained checkpoint used in the benchmark.

For public rates, compare Castform pricing, OpenAI pricing, and the AI token calculator.

What happened

Castform built an agentic-search workflow over documents stored in Neon Postgres. Neon Lakebase Search supplied hybrid full-text and vector retrieval, while Castform generated question-answer tasks and used reinforcement learning to teach a small open model when and how to search.

The reward function evaluated retrieval, citation, and answer correctness. Instead of calling a frontier model for every search step, the trained specialist could repeatedly query the same search tool and produce a grounded answer.

This is specialization, not a new foundation-model launch. The untrained 4B model performed poorly; the gain appeared after training it against the task and environment.

What the benchmark shows

Source-reported model	Cost per rollout	Mean evaluation reward
Qwen3.5-4B, Castform fine-tune	$0.000920	1.447
Qwen3.5-4B base	$0.000929	0.382
GPT-5.4	$0.028158	1.377
GPT-5.6 Sol	$0.087338	1.369

The tuned 4B model’s reward was about 5.7% higher than Sol’s while its measured rollout cost was about 98.9% lower. It also narrowly exceeded GPT-5.4 in the source chart.

Those are Castform and Neon’s measurements, not independently reproduced AI Pricing Guru results. The article does not publish the complete evaluation set, sample count, reward weights, variance, confidence intervals, training bill, hardware configuration, or a reproducible script for the plotted comparison. Castform’s public Neon RAG example defaults to a small generated dataset and explicitly says it is not production-ready; it should not be treated as proof of the private chart’s exact methodology.

Pricing impact: what “100x cheaper” means

The claim is cost per benchmark rollout, not a new list price or a guaranteed invoice discount. Neon says a typical multi-turn Sol search can exceed ten seconds and cost roughly three cents end to end, while the plotted evaluation reports a different Sol rollout cost. That difference alone shows why buyers must price their own traces.

Total production cost includes post-training, synthetic-data generation, embeddings, database compute, inference hosting, evaluation, monitoring, retraining, and frontier fallbacks. A small specialist wins only when those fixed costs are spread across enough repeated work.

Castform’s public pricing API lists hosted Qwen3.5-4B at $0.03 input and $0.15 output per 1M tokens. It separately lists an introductory training rate of $0.10 per 1M training tokens. The hosted inference row is now in our canonical pricing dataset and is maintained by the daily pipeline; the training rate remains a separate billing unit on the Castform pricing page.

Neither number recreates the benchmark’s all-in economics. The source does not disclose total training tokens, rollouts, database usage, embeddings, or engineering cost, and it does not publish the tuned derivative as a stable public model ID. Teams that want another hosted open-model route can compare open-model endpoints on Novita, but those endpoints do not reproduce Castform’s private fine-tune.

Affiliate disclosure: we may earn a commission from the sponsored link above. It does not affect the analysis.

Labs availability blocker

The benchmark fine-tune is not in the Cost-per-Task Labs leaderboard. Castform publishes a hosted base-model rate and a public training example, but not a stable callable ID for the tuned checkpoint, the complete evaluation set, reward weights, or enough usage records to reproduce the plotted cost.

Testing the hosted Qwen3.5-4B base endpoint would answer a different question—the source chart already shows the base model at 0.382 reward versus 1.447 after training. Labs will add the specialist only when the exact artifact or endpoint, fixed retrieval corpus, grader, stopping rules, and measured usage are available.

Who benefits—and who loses

High-volume teams with stable knowledge bases, repeated search patterns, and machine-checkable answers benefit most. Support retrieval, internal policy lookup, and product-documentation agents are plausible candidates because citations and answer correctness can be scored.

Low-volume teams may lose once training and operations are included. Workflows with changing policies, rare edge cases, or subjective answers are also weaker candidates. Frontier models lose routine retrieval loops only after a specialist clears a blind holdout; they remain useful for generating training data, handling exceptions, and serving as fallbacks.

What developers should do now

1. Build a held-out set from real failed and successful searches; never score only on synthetic training questions.
2. Measure retrieval, citation, final-answer accuracy, latency, and total cost per accepted answer.
3. Compare the base model, tuned specialist, and frontier baseline under identical tools and stopping rules.
4. Include training, Neon or other database compute, embeddings, idle inference capacity, and fallback calls in the break-even model.
5. Route gradually and preserve a frontier fallback for low-confidence or out-of-distribution queries.

Our self-hosted AI break-even guide covers the fixed-cost calculation. The earlier Qwen3.5 specialist case study shows the same economic pattern on a separate catalog-review task.

Bottom line

Castform’s result is a strong argument for training small models to use one retrieval environment well. The displayed numbers support a roughly 95x rollout-cost advantage and a higher composite reward than GPT-5.6 Sol on this evaluation.

They do not prove that Qwen3.5-4B is generally better than Sol or that every RAG system will save 100x. Treat the benchmark as a pilot target: reproduce it on private traces, calculate full-system cost, and switch traffic only if the specialist holds up on a blind test.

Sources: Neon and Castform’s case study, Castform’s public Neon RAG example, the Hacker News discussion, and AI Pricing Guru’s live pricing dataset.