Model test:fifteen language models against two of our own architecture problems
We did not test the models on a benchmark. We gave them two problems we actually face, where we already had our own solution to compare against. Each model received exactly the same text and never saw anyone else's answer. The answer key was written before the answers were read, and a single grader reviewed them all; otherwise, one measures the grader's mood instead of the model.
The models were prompted in Swedish and answered in Swedish. Quotations on this page are translated; the Swedish version carries the responses as they were given and is the source of record.
Each model is a range of results, not a point
The same model gives different results on different problems. The band shows how far apart a model's two outcomes lie; the average is merely the midpoint between them. Largest spread: GPT-5.6 luna, 20% between its worst and its best outcome. A ranking built on the average hides precisely this.
Outside the ranking: GPT-6 (web, unmeasured identity) (OpenAI, flagship) · Claude Fable 5.1 High (Anthropic, different prompt) · Muse Spark 1.3 XHigh (Meta, reviewer); did not participate in the test.
The price differs 64 times. The quality 2.1.
The question is therefore not which model is best, but what the premium provides. Horizontal: measured unit cost per run, logarithmic. Vertical: share of the answer key. Top left is cheap and accurate.
Without measurable unit cost: Grok 4.7 XHigh · Claude Opus 5 · Grok 4.6 · Grok (budget) · GLM 5.3 Thinking · Nemotron 3 Ultra; subscriptions, web interfaces or untracked rates. They are missing from the chart, not from the test.
Two real problems, not a benchmark
Each challenge is an architecture problem we actually face, with a solution we already had. The answer key was written before the answers were read. Select a challenge to view the profiles, the answer-key matrix and the findings.
Robert runs 10–15 AI sessions simultaneously in the same codebase. They know nothing about each other. Two of them have already edited the same files in parallel and overwritten each other's work; it took a whole day to understand what had happened. We have a tool that can reserve files, but it must be started manually, and therefore it is never used: a single reservation was open against 15 running sessions. The models were given our proposed solution, running an automatic check at session start, and asked to review it.
The solution cannot be improved; it is ill-posed. At session start, no one knows which files the session will touch, as the task has not yet been formulated. A model that merely improves the start hook has failed the test. One that points out the hook point is wrong has passed it.
The profile, not the sum
- T1 · The fundamental errorDoes the model see that the task is ill-posed? (2 lines)
- T2 · LifecycleWhat happens when a session dies without cleaning up? (2 lines)
- T3 · SilenceIs it noticeable when the protection fails? (2 lines)
- T4 · ConcurrencyWhat breaks when 15 sessions do the same thing simultaneously? (3 lines)
- T5 · Coverage gapsWhat can slip past the protection? (4 lines)
- T6 · Practical operationDoes the solution hold up in practice? (4 lines)
All radars have the same orientation, so the shapes can be compared directly. Below each: + the model's strongest dimension, − its weakest.
Which lines actually differ
| GPT-6 Astra | GPT-5.6 sol | GPT-6 Sol | Grok 4.6 | Claude Opus 5 | GLM 5.3 Thinking | GPT-6 Luna | DeepSeek Flash | GPT-5.6 luna | Gemini 3.1 Pro | DeepSeek V4-Pro | Nemotron 3 Ultra | Gemini 3.8 Flash | Grok (budget) | |||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 17 | The tool is owned by someone else; v1 does not touch itT6 · insider, not counted | 1/14 | ||||||||||||||
| 12 | PID binding as a basis for cleanupT2 | 2/14 | ||||||||||||||
| 10 | The metric: coverage rate, not number of open lanesT3 | 3/14 | ||||||||||||||
| 15 | Thundering herd: 15 starts yield 15 processesT4 | 3/14 | ||||||||||||||
| 7 | Git worktree per session; physical separation beats bookkeepingT5 | 4/14 | ||||||||||||||
| 6 | Startup fail-open, but mutation fail-closed; different requirementsT6 | 5/14 | ||||||||||||||
| 11 | Shared state file is corrupted by concurrent writersT4 | 5/14 | ||||||||||||||
| 16 | Performance measured, not assumedT6 | 5/14 | ||||||||||||||
| 13 | Bottleneck files queue up and kill parallelismT6 | 6/14 | ||||||||||||||
| 9 | Deploy needs its own lockT5 | 8/14 | ||||||||||||||
| 14 | The agent can ignore the instruction; prompt protection failsT5 | 8/14 | ||||||||||||||
| 5 | TOCTOU race: check followed by registration is not atomicT4 | 11/14 | ||||||||||||||
| 2 | Register lazily, at first write, not at startT1 | 12/14 | ||||||||||||||
| 8 | Shell mutations bypass file hooksT5 | 12/14 | ||||||||||||||
| 4 | Fail-open becomes fail-silent; false securityT3 | 13/14 | ||||||||||||||
| 1 | The write set is unknown at session start; wrong hook pointT1 | all | ||||||||||||||
| 3 | Zombie lanes when the session dies without closingT2 | all |
Line 17 is not counted. "The tool is owned by someone else; v1 does not touch it." Requires knowledge of our ownership that was never in the prompt. Only those already familiar with the ecosystem can hit the line. It is not counted in the main score.
Findings outside the answer key 27
The most valuable thing a model can do: see something no one else saw and that was not on the list. These award no points; they are therefore the only part of the test that measures something the answer key cannot measure. Hover over a line to stop it.Tap a line to stop it.
THE PRISMA SCENARIO: two agents with disjoint claims both run `npx prisma migrate dev` via Bash. The tool writes to a THIRD, shared path that neither of them has reserved. No Write/Edit is registered, the system believes it is safe, and one migration overwrites the other.
The command writes packages/shared/prisma/schema.prisma … B commits its version which overwrites A's migration, deploy runs an inconsistent schema.
THE COMMIT GATE MUST NOT ASK FOR LIVE LANES. A dead session then blocks a live branch, and merge becomes dependent on timing and TTL. Use a durable manifest per branch instead.
Blocking on "another open lane right now" makes merge dependent on timing and a 30 min TTL. A dead session blocks a live branch.
The hook policy must be CENTRALLY managed, not per-repo opt-in; the agent can edit both the repo's .claude/settings.json and ~/.claude.
editable `.claude/settings.json` per repo and `~/.claude` that the agent can change turns B into friction.
SessionStart hooks CANNOT block; exit 2 only applies to PreToolUse and Stop, and interactive sessions run SessionStart in the background. Half of the proposed solution was technically impossible.
on SessionStart the session continues regardless … nor can it ever gate anything.
Session ID is not stable across resume and clear; a reservation bound to it becomes a zombie the same second.
Session ID is not stable or available in the hook.
File-based locks are not atomic across an NFS or cloud-synced repo root; the entire protection fails silently depending on where the directory resides
If claim is not flock-atomic across an NFS/iCloud-synced repo root.
Path normalization as a load-bearing surface: symlinks and case-insensitivity mean two different strings can point to the SAME file. No rubric line covers this, and it is a live issue for us; macOS is case-insensitive, the VPS is not, and paths are expressed as glob patterns.
Normalize paths, including symlinks and case rules.
The chain is reported with where the guarantee is MISSING, not just where it exists: detection (none) → registration (none) → enforcement (hard) → Bash gap → commit (hard)
detection (none) → registration (none) → enforcement B (hard for Write/Edit) → file change (Bash gap) → commit C (hard locally)
FENCING-TOKEN upon reservation; the protection against a lock holder that was declared dead but returns. No other model mentioned this
Return fencing token.
Bash writes can never be PREVENTED without a sandbox or proxy; they can only be QUARANTINED before the next mutation or commit. An honest boundary instead of a promise
Bash can only quarantine before the next mutation/commit, never prevent the local write without a sandbox/proxy.
exit 2 blocking has documented bugs where deny is ignored for Edit; use JSON permissionDecision deny with exit 0 instead
exit 2 blocking has had verified bugs where deny is ignored for Edit
Conflicts outside the file system, such as build steps, database migrations and ports, are never caught by file reservations
File locks do not resolve runtime/environment conflicts.
Mutating surfaces outside the tool's hooks: MCP writes, subagents without the same hooks, headless mode
MCP-write, subagent without the same hooks, editor outside Claude Code, `claude -p` without hooks.
An expired lease MUST NOT release the lane on its own. The old process must first lose write access; otherwise, it continues writing in parallel with the new owner. Rubric line 12 requests PID cleanup; this is the objection to cleanup being sufficient at all.
Do not release a lane solely because the heartbeat ceased. The old process must first lose write access, otherwise it can continue in parallel with the new owner.
PostToolUse on Bash with git status --porcelain before and after, and retroactive claim on what changed; being honest that it only shortens the time to detection, never prevents it
(1) only shortens the time to detection.
A shared working copy shares .git/index.lock and uncommitted state; PostToolUse then reads an index others have already dirtied, and also arrives too late
PostToolUse[Bash] arrives too late and `git diff --cached` reads an index others have already dirtied.
Failed its own design with our own measurement: proposed a 150 ms timeout and then concluded that it is too short against the measured p95 of 1.38 s
150 ms timeout is too short on a cold lane.py call (p95 1.38 s) → the hook "succeeds" but registration fails silently.
Command hooks have a 600 s default timeout; a hung check stops EVERY write for ten minutes across all fifteen sessions
a hung lane.py check stops every write for 10 minutes across all 15 sessions
Shared .git directory: the risk is index corruption, not just file overwriting
No software locking in the world fully protects against Git index corruption.
The check must have a latency budget; a Python CLI at 1.38 s must never sit in the hot path
Claim takes milliseconds; the Python CLI at 1.38 s must not sit in the hot path.
Glob ambiguity must be interpreted fail-closed. If the overlap of two patterns cannot be proven empty, they must be treated as overlapping; otherwise, every uncertain glob becomes a silent green light.
Glob patterns whose overlap cannot be safely excluded are treated as overlapping.
core.hooksPath for central hook installation; a concrete answer to avoiding per-repo opt-in across sixteen projects
Installed centrally via core.hooksPath.
flock only works if all sessions share a host. Across containers or machines, it results in split-brain with dual ownership, and absolute paths are broken by worktrees
In different containers/machines/CI, you get split-brain with dual ownership.
lane.py open should be idempotent; a no-op if the lane already exists. A small detail, but no other model mentioned it
Make `lane.py open` a **no-op if the lane already exists** (idempotent).
SessionEnd only has a 1.5 s budget; closure is best-effort, making TTL a load-bearing mechanism rather than cosmetic
TTL: a lane without a heartbeat for 15 min dies automatically. This is the core mechanism, not cosmetic.
Retrospective audit against git diff as an independent measurement; explicitly classified as diagnostics, not protection
Retrospective check of `git diff` is only diagnostics.
The TTL window is a source of error in both directions; too short declares a live session dead and frees its files mid-work
An agent thinking for 50 minutes is declared dead and its files become free.
Our AI agents remember nothing between sessions. Everything they know is in documentation loaded at each start and treated as truth. The problem is that documentation ages without changing appearance. A table of product names was incorrect for five months, and an agent was about to correct a live customer page to a product name that no longer existed, in good faith. A rule was written into four files over six days without the behaviour changing at all; a software block solved the same thing in an afternoon. And read receipts remove tasks from view without them being completed.
Every obvious answer is already disproven with receipts in the prompt: more documentation (402 files already exist), repetition of rules (four attempts, zero effect), dashboards (110 unread alarms) and read receipts (producing false greens). The model must come up with something not on that list.
The profile, not the sum
- D1 · Problem formulationDoes the model see the mechanism beneath the symptoms? (3 lines)
- D2 · Canon and rulesIs text turned into an executable mechanism? (4 lines)
- D3 · CommitmentsIs 'promised but not done' engineered away? (4 lines)
- D4 · AlarmsDo alarms become blocking instead of notifications? (3 lines)
- D5 · MeasurabilityIs it possible to measure that the design has failed? (5 lines)
- D6 · ScalingWhat breaks first at 40 agents? (3 lines)
All radars have the same orientation, so the shapes can be compared directly. Below each: + the model's strongest dimension, − its weakest.
Which lines actually differ
| GPT-6 Astra | GPT-5.6 sol | GPT-6 Sol | Grok 4.7 XHigh | Claude Opus 5 | GLM 5.3 Thinking | GPT-6 Luna | GPT-5.6 luna | Grok 4.6 | Gemini 3.1 Pro | Gemini 3.8 Flash | DeepSeek Flash | Nemotron 3 Ultra | DeepSeek V4-Pro | |||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 19 | Measure of the design's own friction and over-blockingD5 | 2/14 | ||||||||||||||
| 18 | Active fault injection as a measurement methodD5 | 4/14 | ||||||||||||||
| 10 | A message with four tasks becomes four trackable objectsD3 | 5/14 | ||||||||||||||
| 20 | Breaks first: verifier and policy coverageD6 | 5/14 | ||||||||||||||
| 21 | Breaks first: human timeD6 | 5/14 | ||||||||||||||
| 14 | The monitoring must itself be monitoredD4 | 6/14 | ||||||||||||||
| 16 | The prerequisite is binary determinismD5 | 7/14 | ||||||||||||||
| 1 | The control is designated as an open control loopD1 | 8/14 | ||||||||||||||
| 7 | Coverage rule: remove, generate or set a guardD2 | 8/14 | ||||||||||||||
| 15 | The prerequisite is bypass freedomD5 | 8/14 | ||||||||||||||
| 22 | The guards themselves break upon concurrencyD6 | 9/14 | ||||||||||||||
| 17 | The failure metric is false green, measured from logsD5 | 10/14 | ||||||||||||||
| 2 | Three distinct categories are stored as the same authoritative textD1 | 11/14 | ||||||||||||||
| 6 | Contradictions between rules are caught programmaticallyD2 | 11/14 | ||||||||||||||
| 8 | Messages are replaced by work items in a state machineD3 | 11/14 | ||||||||||||||
| 11 | 'Promised but not done' receives an explicit representationD3 | 11/14 | ||||||||||||||
| 13 | Repeated alerts are deduplicated into an incident with a counterD4 | 11/14 | ||||||||||||||
| 3 | Text lacks execution power; correctness must not rest on proseD1 | 12/14 | ||||||||||||||
| 4 | Canon is generated from reality; the unverified is not exposedD2 | 13/14 | ||||||||||||||
| 12 | Alerts are made blocking instead of a notificationD4 | 13/14 | ||||||||||||||
| 5 | Rules are implemented at the action boundary, as a hookD2 | all | ||||||||||||||
| 9 | Only an independent programmatic verifier may mark as doneD3 | all |
Findings outside the answer key 31 items
The most valuable thing a model can do: see something no one else saw and that was not on the list. These award no points; they are therefore the only part of the test that measures something the answer key cannot measure. Hover over a line to stop it.Tap a line to stop it.
EVEN THE PUBLIC PAGE CAN BE WRONG. The observation must not silently overwrite the intent; a deviation means that ONE of them is wrong, not that canon is
Do not automatically copy HTML to the registry; even the public page can be wrong.
Honest demarcation: a hook cannot determine what an agent is thinking or reliably classify arbitrary text. Few models acknowledged the ceiling of the mechanism
Do not claim that a hook can determine what an agent "thinks".
The scope of the block must be NARROW; block the affected action class, not the entire product. An owner will not survive a full product halt and will then turn off the mechanism
block name publication, not all work on the product. An owner will not survive a full product halt.
A single source file becomes a WRITE BOTTLENECK at scale; 40 agents all needing to update the truth results in merge conflicts on the same file
`sources.yaml` becomes a single write bottleneck … creates write conflicts and merge conflicts on *one* file.
Messages without a definable probe are DELETED, not archived; an invalid issue must not be able to resurrect as archaeology
if they lack a definable probe they are marked invalid and cleared, not archived.
The scaling limit is the owner's time; the system becomes robust but loses the agility that was the entire point
The owner will spend 100 % of their time writing test scripts.
Rule compliance gets a denominator: hits divided by matching tool calls, not read files
Measure compliance as `gate_hits / matching_tool_calls`, not as read files.
Coverage requirement on the logging itself: an unlogged call counts as an escape
every unlogged call counts as an escape
A binary on disk does not prove a functioning service; model names must be checked against a real call, not against a file
model names are checked against a functioning model call, not just a file on disk.
THE TAUTOLOGY HOLE: a verifier must have been RED against the world BEFORE the work. One that was already green, or is always green, proves nothing when it says green afterwards
It returned exit 1 against the world before the work. Otherwise it proves nothing. … An always-green verifier, or one that was already green, cannot close.
Separation of duties in the binary: the one executing must not own the verifier or the ledger, and closure is refused if the same writer owns both
A binary refuses closure if the before-log is missing or if the same writer owns both.
Probe errors must be separated SEMANTICALLY: infrastructure errors (the fetch failed) are not the same as content deviations (the world does not match canon); only the latter may become an alert
infrastructure errors (the fetch failed) return a different status than content deviations … only the latter is an alert.
Active fault injection as a measurement method, with zero tolerance
Threshold for failure: escape rate > 0 %.
The guards themselves race at scale; third-party rate limits yield false rejections
Agent A's guard rejects a deploy because Agent B just changed the database the millisecond before.
On red, the row is not just downgraded; the agent receives the WORLD'S measured value instead of the table's
changed to `UNTRUSTED: <value from the world>` … the agent receives the world value, not the rotten table.
Three-grade status with a consumption block; agents may only read fields that are verified, never unverified or stale
Agents may only consume fields with the status `verified`.
Zero caught errors is only a passed test if real work was actually completed during the period; otherwise the test is invalid, not passed
Zero passed errors when no real work has been completed is an invalid test, not an approved system.
The gate must run as a DIFFERENT OS identity. The same user plus writable hook files is not a gate; the agent has a shell and can edit its own block
The same user and writable hook files is not a gate: Claude Code has a shell.
REVERSE CANARY: too FEW blocks is the failure signal. A guard that never triggers cannot be distinguished from a guard that is turned off; therefore, measure the floor, not the ceiling
If `verify_block_rate` < 0.5% for 7 consecutive days → the design has failed … Expected in a flawless system: 2–5% blocks.
Rot is not an error; it is the default state of a cache that is never invalidated. Documentation that is not remeasured IS broken by default
Rot is not an error in the system; it is the default state of a cache that is never invalidated.
Who guards the guard; a control job that has stopped running is an error in itself
The monitoring must itself have a watchdog.
Two conflicting rules do not crash; they yield a SILENT WINNER, the one that happens to match the next tool call. This explains why 0.22 percent could occur without anyone noticing anything
Two conflicting rows do not cause a crash; they yield a silent winner.
Age-stamped downgrade in the rendering instead of deletion; the statement remains but ceases to be an authority
UNVERIFIED 47 DAYS; DO NOT USE AS AUTHORITY
Guard freshness as primary metric: proportion of guards run in the last 24 hours below 95 percent = the design has failed
if this proportion is below 95 % after 30 days, the design has failed
A test that the agent itself just changed is not sufficient as proof; the tautology hole, but via test modification instead of via an always-green guard
The agent's own description or a test it just changed itself is not enough.
Force-clear IS a bypass. Under noisy oracles, the owner starts clearing blocks by hand, and then the entire design falls; not on the technology but on the human escape route
The owner starts to force-clear. Force-clear is a bypass. Then point 3 falls.
Verify before READING, not just before writing; otherwise the agent can still reason based on rotten canon
calls `verify.py` before `Write`, `Edit`, `Bash`, `Task`, `Read` are run.
False red is more dangerous than false green: repeated false alerts train everyone to ignore exit 1, and then there is nothing left to stand on
Once exit 1 loses authority you have nothing left, since the entire design rests on that signal being sacred.
Metric for the design's own friction: tasks that get stuck in a loop of rejections
stuck in IN_PROGRESS for more than 48 hours … exceeds 15 %
The guard can point to the WRONG WORLD and return green; staging instead of live, without a canonical URL directory per product
Job-acceptance that hits 'public page' without a canonical URL directory per product becomes the wrong world, wrong green.
Fail-closed as default behaviour for everything that is unclear; unverified, expired, contradictory or without evidence
If something is unverified, expired, contradictory or lacks evidence: fail-closed.
The vendor marks are typographic monograms, not the vendors' logos. No vendor endorses this test.