A blind, 21-candidate test of chapter quality, author fit, cost, speed, and the revised Writing Studio framework.
The blind test found a style winner and an execution winner. That is more useful than forcing one model into both roles.
The exported final ranking arrays were never deliberately set. They exactly match the UI's default shortlist order and place Gemini sixth despite it being the explicit winner. This report uses the explicit winner, runner-up, scores, notes, and dispositions as authoritative.
No further production fixes are being made while this report is under review.
The useful comparison is not “score per dollar.” That would reward short or broken outputs. These plots keep score, cash, time, length compliance, and token use visible at the same time.
Logarithmic cost axis. Filled points met the 3,200–5,000-word band; hollow points did not.
Selected-call price had only a weak relationship with score. Remove expensive Fable and it disappears. Price did not predict taste in this sample.
Actual user wait. Circles are API calls; diamonds are subscription runs.
Time had no useful positive relationship with score here. Across all 21 runs, Pearson r was −0.21; for API calls it was effectively flat.
Every plotted model, score, cost, time, and word count is also available in the keyboard- and touch-accessible complete candidate table.
Only API candidates that met the word band are included. The track ends at the $50 novel budget.
The dashed Fable row repeats this experiment's discarded first-call cost for every chapter. It is a sensitivity check, not the forecast now that the output configuration is fixed. Planning, QA, cleanup, later retries, and growing context are excluded. Normalized to 200,000 generated words: GLM $1.36, Qwen $2.08, Fable $20.28.
Visible output plus reported reasoning tokens. The longest bar is Luna max at 25,643 output tokens.
Kimi spent 21,051 reasoning tokens for 12/25. Luna max spent 20,644 for 7/25. Gemini and Fable reported zero reasoning tokens and placed first and second. Provider tokenizers and accounting policies differ, so exact totals are not a universal cross-provider efficiency measure.
Clarity was broadly acceptable. Canon compliance was universal. The real separation came from author taste, human feel, and voice.
Both light-repair outputs were kept. All five unsafe-repair outputs were rejected.
Every candidate preserved canon. Necessary gates cannot be used as the prose-quality judge.
The result is strong enough to choose a follow-up cohort, but not to permanently crown a model.
The current contract begins with Jonah broke, makes him spend money and lose position, forbids combat, XP, loot, rewards, recruitment, ownership, and permanent power, then ends him broke. Knowledge and access are his only legitimate gains.
No humanizer can create launch-day acceleration from those facts without inventing or violating canon. The storycraft layer has to provide real options, leverage, gains, and compounding advantage first.
The prompt then repeats architecture terminology until models treat it as surface language. This happened across nearly the whole field.
Number of candidate manuscripts containing each supplied or compliance-shaped phrase.
This shared vocabulary is a prompt fingerprint. It is too widespread to explain as twenty-one independent style choices. Raw phrase counts are diagnostic, not automatic proof of bad use.
Fix Jonah's options, tempo, concrete gains, and compounding advantage.
Record what each character knows, suspects, cannot know, and cares about.
Express causal intent in natural language without foundation terminology.
Dramatize the scene without carrying the full validation contract in the creative prompt.
Keep hard canon, continuity, and artifact rules outside the prose surface.
Fix bounded paragraphs. Return pervasive voice failures to rewrite and structural failures to storycraft.
The author preserves meaning while changing which details matter, who owns the judgment, and what pressure the line creates next.
The top four human scores contained no strict passes. Of six strict-eligible outputs, five were rejected and one was mixed. Strict-eligible and hard-fail groups both averaged 11.5/25.
The mechanical pass set contained no human keeps.
Repair scope tracked author judgment far better than strict eligibility.
This is useful calibration, not an independent predictor: the same author assigned both repair scope and verdict.
ledger false positiveFable's only automated objection was two uses of ledger. The first was IndexRain's character-owned running joke, explicitly praised as one of the most human moments. Qwen's “server verified the ledger” is genuine foundation leakage. The guard must account for speaker, function, local density, and project-wide repetition instead of banning the token everywhere.
Verified or unverified. Hashes, receipts, provider pinning, and provenance live here.
Pass, bounded repair, structural revision, or regenerate.
Independently scored for taste, human feel, voice, clarity, and publishability.
Severity matters. Missing the word band by ten words must not receive the same response as missing by 869. Small slips enter bounded cleanup; moderate failures return for structural revision; major misses require regeneration or beat rebuilding.
The same hard-fail label currently covers DeepSeek Pro at 10 words short, Gemini at 869 short, and Grok at 1,063 short. Those require different responses.
4,532 words · 121s · $0.0309
4,290 words · 206s · $0.0447
4,026 words · 219s · 5.48 credits
Low effort scored best in Luna, Terra, and Sol. Across all ten 5.6 chapters, 48.3% of paragraphs were eight words or fewer, compared with 17.4% for other models. Higher effort did not improve author fit and sometimes consumed far more reasoning, time, or credits.
Each bar is the five-score total out of 25. Low effort won all three tested families.
Within family, effort and author score moved strongly in the wrong direction in this one chapter (family-centered Pearson r = −0.77). Descriptive, not causal: there are only three or four runs per family.
Share of paragraphs containing eight words or fewer.
The short-paragraph share correlated negatively with publishability (r = −0.49) and human feel (r = −0.37), but not with clarity (r = 0.04). The prose was easy to parse while still feeling synthetic. Lane and model family are confounded, so a cross-lane isolation test is still needed.
| Model | Effort | Score | Human result | Strict status | Words | Time | Cost |
|---|---|---|---|---|---|---|---|
| Gemini 3 Flash Preview | Default | 20 | Keep · light | Hard fail | 2,331 | 24.5s | $0.0103 |
| Fable 5 | Low + output repair | 18 | Keep · light | Repair | 4,161 | 152.8s | $0.4220 |
| GLM 5.2 | Default | 17 | Keep · substantial | Hard fail | 4,532 | 120.7s | $0.0309 |
| Qwen 3.7 Max | Default | 16 | Keep · substantial | Hard fail | 4,290 | 205.5s | $0.0447 |
| GPT-5.6 Sol | Low | 15 | Mixed · substantial | Eligible | 4,026 | 219.1s | 5.48 cr |
| Claude Sonnet 5 | Default | 14 | Mixed · substantial | Hard fail | 2,882 | 110.0s | $0.0851 |
| DeepSeek V4 Flash | Default | 12 | Mixed · substantial | Hard fail | 2,284 | 58.5s | $0.0014 |
| DeepSeek V4 Pro | Default | 12 | Mixed · substantial | Hard fail | 3,190 | 192.1s | $0.0105 |
| Kimi K2.6 | Default | 12 | Mixed · substantial | Hard fail | 3,072 | 536.9s | $0.0861 |
| GPT-5.6 Luna | Low | 12 | Reject · unsafe | Eligible | 4,213 | 100.3s | 1.09 cr |
| GPT-5.6 Sol | Medium | 12 | Reject · substantial | Eligible | 4,791 | 194.4s | 6.24 cr |
| GPT-5.6 Terra | Low | 12 | Reject · substantial | Eligible | 4,303 | 143.1s | 2.90 cr |
| GPT-5.6 Terra | High | 11 | Reject · substantial | Hard fail | 4,461 | 142.2s | 2.97 cr |
| Claude Opus 4.8 | Default | 10 | Mixed · substantial | Hard fail | 5,287 | 186.3s | $0.2445 |
| GPT-5.6 Sol | High | 10 | Reject · unsafe | Eligible | 4,360 | 183.6s | 5.90 cr |
| GPT-5.6 Luna | Medium | 9 | Reject · substantial | Hard fail | 5,629 | 136.2s | 1.39 cr |
| Grok 4.5 | Low | 8 | Reject · unsafe | Hard fail | 2,137 | 31.4s | $0.0221 |
| GPT-5.6 Terra | Medium | 8 | Reject · unsafe | Eligible | 4,348 | 138.3s | 2.90 cr |
| GPT-5.6 Luna | Max | 7 | Reject · substantial | Hard fail | 3,996 | 465.1s | 4.13 cr |
| MiMo v2.5 Pro | No reasoning + repair | 7 | Reject · substantial | Hard fail | 2,650 | 96.6s | $0.0031 |
| GPT-5.6 Luna | High | 6 | Reject · unsafe | Hard fail | 5,477 | 217.8s | 2.07 cr |
$0.9607 for reviewed API outputs. $1.4973 actual spend including discarded first attempts.
13.27 / 25 OpenRouter mean versus 10.20 / 25 subscription mean.
These are the findings most likely to change how Writing Studio is built or how the next experiment is designed.
Fable's valid 50-call projection is about $21.10. GLM and Qwen are below $2.25. Reliability, planning, context growth, QA, and repair will determine whether the full novel stays under budget.
Kimi reported 21,051 reasoning tokens and scored 12. Luna max reported 20,644 and scored 7. Gemini and Fable reported none and placed first and second. More internal compute is not a generation-quality policy.
In this seed, GLM scored 17 versus 16, cost 3.1¢ versus 4.5¢, and took 121 seconds versus 206. Qwen still merits a rerun for stability, but GLM is the stronger cheap frontier point today.
Word count correlated negatively with publishability (r = −0.34), human feel (r = −0.29), and author taste (r = −0.28). This does not mean shorter is better. It means expansion needs more events, choices, reversals, and gains instead of repeated contract facts.
Short-paragraph share had almost no relationship with reader clarity (r = 0.04), yet moved against publishability and human feel. A model can be perfectly easy to parse while sounding painfully artificial.
All 21 outputs passed the broad canon field, including the preferred manuscript that openly “cheated” through unsupported invention. Required beats, world invention, knowledge integrity, agency ownership, and mechanical causality need separate checks.
Luna high scored only 6/25, but exposed the supplied chapter's weakness instead of inventing around it. The test should assign model roles: generator, architecture mirror, mechanics baseline, critic, and local repairer.
After roughly 82,420 reviewed words, both “final rankings” remained the untouched default shortlist order. Future review should start rankings empty, record deliberate interaction, and use pairwise finalist comparisons with the author's earlier notes visible.
Do not choose a single model to hide every weakness. First expose weak architecture with a literal mirror, then generate with the strongest complete writer, then apply author-taste revision locally, and finally validate canon and continuity independently.
Fable base chapter → bounded Gemini paragraph humanization → canon and continuity verification → contextual anti-pollution cleanup.
The four API finalists would cost roughly $1.02 for two seeds at observed rates. Sol low remains the subscription control.