Inference Layer Cost: Routing by Cost Alone Isn't Enough

Is the most expensive model worth it? I was watching the token counts at the end of Claude Code runs and had no idea what those numbers actually meant. Was this expensive? Did it matter?

That question has been running underneath everything since. Three articles and four experiments later, I have more data than I started with. What I didn't expect is that every time more data was collected, it would make the question harder to answer, not easier.

The routing argument isn't new. It just hasn't been framed in economic terms for most practitioners. Chris Hutchins put it plainly on a recent podcast:¹ "If I needed someone to wash my car, I'm not going to call my neurosurgeon to come in and wash my car." Right model for the right task. The question is what it costs when you get that wrong.

Also on that podcast, Anish Acharya observed: "On a per-token basis, consumers are going to need to get more efficient at some point." The subsidies that make this feel optional today won't last. When pricing normalises, the teams that have been defaulting to the most expensive model for every task will feel it.

This article is about what happens when you measure cost and quality together.

One of the experiments earlier in this series tested four models on three DevOps incident types: a PostgreSQL storage exhaustion, a Kubernetes node eviction, and a fictionalised S3-style outage. Two judge models scored each response independently.

Three incidents. Directional, not definitive. S = Sonnet judge, G = GPT-5.5 judge.

The cost spread is real. So is what it obscures. GPT-5.4-mini at $0.001 matched Sonnet at $0.016 on quality scores across most scenarios, but the responses were structurally different. Haiku produced tight, actionable responses with prioritised steps and owner assignments. GPT-5.5 produced comprehensive responses with architectural depth, edge cases, and specific operational warnings. GPT-5.5's more comprehensive responses came with a tradeoff: harder to act on quickly, but more likely to catch something specific. On the PostgreSQL incident, that tradeoff mattered.

The VACUUM FULL finding from the PostgreSQL incident is the clearest example. Haiku scored 8/10. GPT-5.5 scored 10/10. The two-point gap contained a specific warning: do not run VACUUM FULL without careful planning, as it requires an exclusive lock on the table and additional disk space. On a storage-constrained system that warning could prevent someone from making the situation significantly worse.

That finding cuts against a bias most of us carry. Chris put it plainly: "I never want to use Haiku. So I'm like, oh god, that's the dumb model."¹ But the data doesn't hold that consistently in either direction. On the most complex incident, Haiku scored 10/10. GPT-5.5 failed silently on the first run: the token limit that worked for every other model wasn't enough for it. It produced no output and billed for it. The expensive model must be better is a feeling, not a finding. Neither model won every scenario.

The incident triage experiment answered one question: does routing to a cheaper model cost you quality? The answer was mostly no, with the caveat that what you lose isn't always visible in the score. That raised a harder question: what happens when the task isn't just operational, but compliance-sensitive? Does cost optimisation create liability when the stakes are an audit finding rather than a missed operational warning?

The SOC2 compliance experiment tested the same four models on three readiness scenarios:

• CC6: Access Control: An IAM role that had not been reviewed in 18 months, with unrestricted permissions on a data pipeline processing customer transaction data
• CC8: Change Management: A CTO bypassing branch protection 14 times in a quarter with no documented emergency change procedure
• CC7: System Operations: A monitoring setup with dashboards but no alert policies. The last incident was discovered by manual testing, not automated signal

The question was direct: does routing to the cheapest model create compliance risk?

Three scenarios. No critical findings were missed by any model. S = Sonnet judge, G = GPT-5.5 judge.

Every model caught every audit-critical gap. The cheapest model identified the correct SOC2 controls, correctly classified the gaps, and accurately assessed the compliance consequence. GPT-5.5 at 42x the cost caught nothing additional.

But there was one finding that didn't show up in the scores.

Sonnet gave the only unambiguous Adverse classification across both the IAM role and change management scenarios. Haiku flagged Adverse in its response headers but hedged in the body text, leaving the overall classification ambiguous. GPT-5.4-mini and GPT-5.5 were consistently Qualified.

That distinction matters. Qualified Opinion Risk means an auditor would flag this as a finding, but you can still achieve SOC2 certification if you remediate. Adverse Opinion Risk means this is significant enough that it could prevent certification or cause customers to question your security posture.

Sonnet's reasoning was specific. For the IAM role, multiple compounding Design Gaps (unrestricted S3, RDS, and IAM enumeration permissions on a data pipeline processing customer transaction data, with no access review in 18 months) reach Adverse rather than Qualified. For the change management scenario: 14 bypass events in one quarter is not an isolated deviation. It's a systemic pattern that an auditor would treat as the control not operating at all. The full responses from all four models are in the results dropdown at the end of this article.

A team routed to any model other than Sonnet gets Qualified Opinion Risk and prepares accordingly. A team routed to Sonnet gets Adverse Opinion Risk and may treat it as a certification blocker. The routing decision changed the urgency of the response, not the accuracy of the diagnosis.

Both experiments pointed in the same direction: cost optimisation doesn't cost you the diagnosis. The cheap model caught the same gaps as the expensive one. What the data also showed, and what the scores didn't capture, is that models make different assumptions about what you need from a response.

Haiku gives you something you can act on immediately. GPT-5.5 gives you something you can learn from. Sonnet gives you a more conservative risk assessment that may or may not reflect how an auditor would actually rule. None of those is wrong. They're just different, in ways that matter depending on the moment you're in.

The VACUUM FULL warning appeared on the simplest incident. The audit opinion risk divergence appeared on scenarios every model scored identically. Neither showed up in the scores. A routing layer optimising on quality scores alone has no signal for either.

That's where this series runs out of road. Not because the question is unanswerable. Because answering it properly requires a different kind of experiment.

¹ Chris Hutchins and Anish Acharya, All the Hacks podcast, May 13 2026. Cited twice in this article. youtube.com/watch?v=nBPpwz4H85Y

This is the third and final piece in the Infinite Drive Field Notes series on LLM inference layer architecture. The work was run using the Anthropic and OpenAI APIs in May 2026.

You are a senior security and compliance engineer conducting
a SOC2 Type II readiness assessment. When given a description
of an infrastructure configuration, deployment process, or
operational setup, respond with a structured assessment covering:
1. The SOC2 Common Criteria control at risk (e.g. CC6.1, CC8.1)
   with a brief explanation of why it applies.
2. The specific gap: classify as Control Deficiency (control exists but does not operate effectively), Deviation (process exists but is not being followed), or Design Gap (no control exists for this criteria at all).
3. Audit opinion risk: Unqualified (no risk), Qualified Opinion
   Risk (auditor would flag this; you can still pass), or
   Adverse Opinion Risk (could prevent certification or cause
   customers to question your security posture).
4. Immediate remediation steps required before an audit.