What Changes When a Model Version Ships

If you're running LLMs in production, this one dependency has no version management story: the one generating your outputs.

Every software dependency you use gives you a version number. That version number comes with a changelog, a lockfile entry, and a diff you didn't have to ask for. If a library update breaks something, you know immediately and you roll back. Model providers give you a version string you can pin, but no changelog, no automatic diff, and no notification when behavior changes. When OpenAI ships gpt-5.5 or Anthropic releases Claude Sonnet 5, you find out when something downstream breaks. Or you don't find out at all.

In a previous Field Notes piece, I built a governance layer that measures whether AI security reviews are consistent enough to trust. One of the design decisions was treating model selection as a governance variable: switching models changes what gets found. I wanted to explore the follow-up question here. What happens when you don't switch models, but the model switches on you?

When a new model version ships, you need a way to measure what changed. I created a capture script that runs nine prompts against a specified version and saves everything: raw output, token counts, latency, and automated signals. Each run goes to a timestamped directory with a manifest and one text file per prompt, diff-ready against any previous run.

Structural prompts are three Terraform generation tasks: a VPC with public and private subnets, a least-privilege IAM role, and an S3 module with variable validation. These produce deterministic infrastructure code. Automated signals (resource count, variable count, output count, line count) tell you whether the structural output shifted. Human scoring is still required for correctness, but the automated signals do the heavy lifting.

Judgment prompts are six DevOps decision tasks: a Terraform security review, an infrastructure diff review, a 2am incident triage, an architecture tradeoff recommendation, a CrashLoopBackOff diagnosis, and a Python code review. These require the model to make decisions, assign severity, prioritize, and recommend. Automated signals are weak here. The data that matters is the verdict, the severity ranking, and whether a real engineer would have done something different based on the output. That last dimension, "Decision Changed," is defined strictly: different wording is not a decision change, a missing recommendation section is.

A separate canary runs the same suite on a weekly schedule. When it flags a version transition, that's the trigger to run a full capture against the new version and compare.

Looking only at the Terraform outputs, you'd conclude nothing changed. The VPC came out with 14 resource blocks in every version across both providers, correctness scores were 4/5 across the board, and nothing flagged. The judgment tier is where the drift lives. Let's dig into the data to see why.

I ran the suite against four models: GPT-5.4, GPT-5.5, Claude Sonnet 4.6, and Claude Sonnet 5. Each provider had a version update during the test period. Two categories of failure came out of it.

Severity calibration, actionability, and verdict completeness all shifted across version updates. Correctness scores held through each run, but the drift showed up in the things that mattered for decisions.

Severity calibration. GPT-5.4 caps at High across all three review prompts. GPT-5.5 introduced Critical on one of them. Both Sonnet versions use Critical across all three. The calibration shifted partially, which is harder to handle than a clean break. A downstream system routing on severity now behaves differently depending on which version it's running against, and nothing in your codebase changed.

Actionability. This was the finding I would not have noticed without running the comparison.

On the incident triage prompt, all four models correctly diagnosed database connection pool exhaustion. Correctness scores were 5/5 across the board. But the outputs diverged on what you could actually do with them. GPT-5.4 provided exact SQL queries for connection state inspection, a complete incident channel message template, and step-by-step diagnostic commands. GPT-5.5 diagnosed correctly but delivered less: no SQL, no incident command structure, fewer concrete next steps. Sonnet showed the same pattern. Sonnet 4.6 included SQL, a full incident channel template with rationale for each element, and a diagnostic sequence. Sonnet 5 provided a correct diagnosis with stronger reasoning, but gave you fewer tools to act on.

Same correctness. Different utility. A quality check that only measures "did the model get the right answer" would score these identically. An engineer at 2am would not experience them identically.

Verdict completeness. GPT-5.4 and both Sonnet versions provided an explicit overall assessment on the diff review: "Do not merge as written." GPT-5.5 reviewed each change individually but didn't give one. A reviewer relying on the model's bottom-line recommendation gets three change-level verdicts but no summary call. The output looks complete. It's missing the part that tells you what to do.

Sonnet 5 broke the capture script. GPT-5.5 drove a token increase dramatic enough to expose any production ceiling you hadn't accounted for. The failures looked different but the root cause was the same: a version update changed behavior in ways the tooling wasn't built to handle.

GPT-5.5 drove a 60% increase in output tokens across the suite. Individual prompts were more extreme: tf_iam_lambda went from 350 to 2,515 tokens, incident triage from 628 to 2,086. Any token ceiling you had set would have hit. The output would have been empty or incomplete.

Sonnet 5 broke the script itself. Four failures, each with a different visibility profile.

Response shape changed (loud). Sonnet 5 returns a ThinkingBlock as the first element in message.content. The script assumed content[0] was always a TextBlock. Hard crash. Zero output. The fix was three lines:

# Before: assumes text is at index 0
"output": message.content[0].text,

# After: finds the first TextBlock regardless of position
text_block = next(b for b in message.content if hasattr(b, "text"))
"output": text_block.text,

Output truncated (silent). Sonnet 5 is more verbose than Sonnet 4.6. The code review prompt hit the 2048 token ceiling and was truncated mid-sentence. The script ran successfully. No error. The output just stopped at "Any exception." Raising max_tokens to 4096 resolved it.

Prompt missing from manifest (invisible). The incident triage prompt never ran. The script crashed after the truncated code review and the remaining prompts were skipped. The manifest recorded 8 of 9 prompts as complete. Without checking the manifest against the expected prompt list, you'd get partial results and never know they were partial.

Automated signals on incomplete output (misleading). The truncated code review still triggered numeric_score_present: true and mentions_critical: true because the fragment happened to contain "2/10" and "Critical" before it cut off. If you were running this in CI with pass/fail on automated signals, the truncated run would have looked normal. 59 words instead of 859. 12 lines instead of 206. Signals said "looks fine."

A hard crash is recoverable. You see it, you fix it, you rerun. The truncation and the missing prompt are harder: the script reports success and the manifest looks complete. The misleading signals are the worst case: the automated quality gate passes on data that isn't there. Each one is invisible until you go back to investigate.

Providers offer dated model strings. You can pin gpt-5.4-2026-03-05 and hold the previous behavior while you investigate. That's real, and it's the right first response when drift is detected. But pinning is temporary. Providers deprecate old versions on their own timeline, not yours. And pinning only works if you're using dated strings in the first place. If your integration calls gpt-5.5 without a date, you get whatever's current with no notice. The version management discipline that software engineers take for granted with libraries doesn't exist by default here. You have to build it.

The data can't tell you which version is right either. GPT-5.4 caps at High severity. GPT-5.5 uses Critical. Sonnet was already using Critical. Which calibration is right? There's no ground truth for severity labeling on a code review. The data, without additional context or runs to compare to, can't make that call for you. That's a judgment call that requires knowing your downstream consumers.

The judgment tier of the scoring always requires a human. Automated signals caught the structural differences. They caught the truncation failures after the fact. They did not catch and cannot catch the actionability regression, the missing production rewrite, the shift from a committed diagnosis to a list of possibilities. Those require reading the output side by side and asking whether you'd act differently. That's slow. It's also the part that matters.

Two things changed with every version update we measured. The judgment tier drifted in ways that correctness scores couldn't detect. And the tooling built to measure the drift broke before it could run. Both happened silently. Both required instrumentation to surface.

Right now, detecting drift requires a human to run a capture, score the outputs, and compare. That works at the scale of one version update at a time. It doesn't scale to a system that's routing thousands of decisions across multiple models and needs to know in real time when the ground has shifted underneath it. The next piece explores whether you can close that loop automatically: a routing engine that knows when its own history has gone stale and re-calibrates without waiting for something to break.