What did Memdoor report?
On October 31, the Memdoor team-solve workflow report described a reproduction of the GVS5H multi-agent method using Qwen3.8-27B. Its 50 scored runs passed all hidden tests 49 times.
The headline needs a narrow reading: this is ten distinct programming problems, each attempted five times. It is not a new Qwen checkpoint, an official leaderboard announcement or proof of superior coding ability across workloads.
| System on this ten-problem subset | Solved per pass, 1–5 | Mean per ten |
|---|---|---|
| Memdoor Qwen3.8-27B workflow | 10, 10, 10, 10, 9 | 9.8 |
| Original authors’ Qwen3.8-27B harness | 9, 10, 10, 9, 10 | 9.6 |
| Original authors’ Fable 5 single-call baseline | 7, 8, 10, 8, 7 | 8.0 |
We recalculated Memdoor’s success counts and costs from its published runs.tsv, and checked the comparator totals in authors.tsv. We did not independently rerun the models or grader.
How does the agent team work?
The method separates a planner, two ideators, a solver and a manager, with a repair worker when needed. Fresh contexts communicate through files containing the plan, proposals, notes and current program. “Five models” describes orchestration roles using one model family, not five newly trained models.
The scored reproduction used the authors’ prompts, a 128K reply cap and up to ten rounds. Sample tests guide the loop; hidden tests determine the reported score. The ordinary example workflow and this parity benchmark configuration should not be assumed identical.
The original GVS5H project reports results across 100 hard problems. Memdoor’s reproduction covers only the first ten. Its hosted inference also differs from the original authors’ local FP8 vLLM setup.
What did the experiment cost?
These are historical experiment charges, not today’s API rate card or a quoted price for reproducing the result.
| Recorded Memdoor expense | Reported amount | Scope |
|---|---|---|
| Scored runs | $35.79 | 50 runs; about 15.3 million output tokens |
| Including interrupted attempts | $38.77 | Adds 16 attempts stopped, paused or cut by a clock |
| Median scored run | $0.39 | Not the mean; excludes interrupted attempts |
| Matched Fable 5 expense | Not established here | No same-harness Fable cost experiment |
| Pricing data: How we verify → | ||
We independently summed the scored-run charges. The larger interrupted-attempt total is the maintainer’s reported figure, not a total we reconstructed from that TSV.
Memdoor reports a median of 40 minutes of model time, with the longest run taking four hours. A low token bill does not imply interactive latency, low hosting overhead or an acceptable developer waiting time.
Inference ran through OpenRouter across Modal, CoreWeave and Wafer. These charges must not be substituted with another host’s Qwen rate. For separate maintained provider references, see Qwen pricing and Anthropic pricing. Neither page establishes the cost of this specific experiment or a subscription entitlement.
Where does the comparison break down?
Fable 5 was not run in Memdoor’s harness. Its numbers come from the original authors’ single-call results. Comparing iterative agents with a single call measures systems with different inference budgets, not an equal-budget model contest.
The problems date from March 29 to April 6, 2025. The maintainer explicitly says training contamination cannot be ruled out. Five repetitions on ten familiar problems do not supply 50 independent tests of generalization.
The logs also expose regression risk. In the sole failed scored run, the manager reportedly invented a counterexample with an incorrect expected answer, causing a correct program to be rewritten incorrectly. In two other runs, correct programs were overwritten and later recovered at additional expense.
Passing public samples is therefore not a sufficient stopping rule. The most useful result may be the failure trace, not the headline score.
What should coding-agent builders do now?
Teams building asynchronous coding workflows gain a concrete, inspectable design to test. Buyers choosing a model or subscription still lack a matched task-cost comparison; this report identifies no general cost winner or loser.
If you already use Qwen, compare orchestration against a single-call baseline on your own held-out tasks. Fix the total token budget, wall-clock limit and grading rules. Count failed calls, retries and interrupted work, not just successful completions.
Validate a manager’s counterexamples with executable checks, preserve earlier candidate solutions and track cost per accepted solution alongside latency. Use our token calculator for hypothetical API usage and API cost guide for the accounting framework; neither reproduces this benchmark bill.
Bottom line: Memdoor supplies a promising small-subset reproduction with unusually useful logs. It does not establish a universal Fable 5 replacement, a new rate card or a price cut.