Llm on David Parry

🖥️ You Don't Need a Frontier Model. You Need a Spec.

Fri, 14 Aug 2026 15:00:00 -0500

You do not need a frontier model to build professional software. You need a spec, a test bar, and a workflow that will not let anyone — human or model — skip either. Frontier models are what you reach for when you want day-zero results: something that compiles this afternoon and looks finished until you have to live with it. A responsible solution is slower in the screenshot and faster over the life of the system. The surprise of the last year, for me, is that this path is now cheap enough to run on a laptop.

I already wrote that spec-first was always right, and that agents finally made it fast . That post was the argument. This one is what I learned once I stopped treating the hosted model as the product and started treating a local model as one tool inside a spec-driven loop. The working proof is the bdd CLI that grew out of that workshop — one native binary, Ollama by default, no cloud calls, no token meter.

The CLI is the discipline, compiled

The workshop needed an MCP server so an agent could not wander. The CLI is that same loop as a program you run yourself. The command manual is the full surface; the idea is one sentence:

The requirements spec is the source of truth, and the discipline is enforced by tooling, not by convention.

Everything else follows from that. The spec is machine-validated (bdd spec validate) and its wording is quality-gated (bdd spec refine) before any scenario or line of production code exists. A valid-but-vague requirement is caught and reworded first. Behavior flows downhill from the approved spec: Gherkin scenarios tagged with requirement ids, step definitions, unit tests, and only then production code. The Red/Green/Refactor cycle is a state machine, not a suggestion — start_refactor is refused on a red bar, and a requirement is only marked implemented behind a green one.

Agents get no escape hatches. No arbitrary file writes, no shell, no “just install this for me.” Every mutation goes through a typed, validated tool, lands in staging (.bdd-staged/), and waits for a human to review it. The same tools serve a person at a prompt and an agent over MCP. The LLM is local, discovered rather than assumed, and generation falls back to deterministic templates when Ollama is down or empty. Nothing is ever installed for you.

Run bare bdd and you get the loop, the version, and whatever local model is already on the machine:

$ bdd

  ╭──────────────────────────────────╮
  │                                  ▼
  │    > bdd  v0.2.4                 │
  │    spec → RED → GREEN → REFACTOR │
  ▲                                  │
  ╰──────────────────────────────────╯

Model set for this session: qwen3-coder-next:q4_K_M (not saved - keep it with: bdd model use qwen3-coder-next:q4_K_M).
bdd>

From an empty directory, bdd greenfield runs the whole creation order with exactly two human gates: the wording of the driving spec, and the review of generated tests before they are committed. From an existing project, you drive the same phases yourself — spec draft, spec validate, spec refine, test, implement, refactor — and bdd status names the one next step that actually moves the loop forward.

The model is not sitting above this process. It is boxed inside a few commands: drafting a requirement from plain words, polishing step definitions and unit tests, attempting an implementation against a recorded RED bar. test, state, and refactor never call it. That is not a prompt instruction. It is the architecture.

What I actually learned about local models

The lesson is not “an 8B model beats Opus.” The lesson is: once you move the deterministic work out of the model, a local model becomes enough.

I measured a version of that claim in Skills vs MCP . For one workload, putting the business rules in code instead of in the model’s head cut hosted cost by 32% on Claude Opus 4.8 and 76% on GPT-5. The same implementation on qwen3:30b, served locally through Ollama on an Apple M4 Max, cost nothing and dropped mean latency from 43.4 seconds (instruction-only Skill) and 54.4 seconds (prompt only) to 10.8 seconds with the tool. There is no per-token invoice to blame on that last comparison. Generating fewer tokens still consumed less wall-clock time on my hardware.

The CLI takes that architecture and applies it to the development loop itself. Spec validation is code. Wording critique is code. The TDD state machine is code. Gherkin parsing is code. Test execution is Maven, cucumber-js, dotnet test, or cargo test — the project’s own runner, not a model’s impression of a runner. The model is asked to propose, not to be the system of record. When it proposes a step definition, the CLI prefers that output only if it validates; otherwise the deterministic template ships. That is why qwen3-coder-next:q4_K_M is a reasonable default on this tool, and why qwen3:30b is a luxury rather than a requirement.

Open-weight coding models have also closed enough of the raw-capability gap that this is no longer a thought experiment. Alibaba’s Qwen3-Coder reports state-of-the-art results among open models on SWE-Bench Verified without test-time scaling, and describes the 480B variant as comparable to Claude Sonnet 4 on agentic coding. You can argue with any one leaderboard. You cannot argue with the direction: the model you can run next to the repo is no longer a toy, and the remaining gap is exactly where unconstrained agent loops still fall down — long-horizon, multi-file, “figure out what I meant” work. Spec-driven development is how you stop asking the model to do that work.

METR’s early-2025 randomized trial is the other data point I keep coming back to. Experienced open-source developers working on their own mature repositories, using frontier tools of the day (primarily Cursor with Claude), took 19% longer when AI was allowed. They expected a 24% speedup. After living through the slowdown, they still believed they had been sped up by 20%. Perception and the clock disagreed. METR was careful about what that does not prove. They said, explicitly, that they do not provide evidence “there are not ways of using existing AI systems more effectively” — scaffolding, prompting, repository-specific context. Their later 2026 follow-up ran into a different problem: too many developers now refuse to work without AI, so the experiment can no longer see the tasks where people expect the biggest lift. I read both results the same way. Unconstrained generation on a real codebase is not the same activity as a gated spec-to-green loop. The first can feel fast and still be slow. The second is slower to start and cheaper to finish, and it does not require the most expensive model on the market.

Day-zero is a product. Professional software is a process.

I will say this as cleanly as I can. Frontier models are extraordinary at day-zero. You describe a thing, files appear, a demo boots, a screenshot looks like a product. That is a real capability, and it is the capability the labs demo, because it is the capability that converts. If what you wanted was a prototype before lunch, pay for the tokens. I do that too.

Professional software is a different shape. Someone has to write down what “done” means in a form that can be checked. Tests have to fail for the right reason before they pass for the right reason. The code has to be the simplest thing that makes that true, and then it has to be reviewed. Edge cases have to be found on purpose, not stumbled into in production. That work has been preached at us for as long as I have been doing this — test-first, behavior-first, requirements-first. Cucumber and Gherkin were the first tools that let the requirement itself become executable, and I never went back. What changed is not the advice. What changed is that an agent will now execute the advice if you put it in a tool instead of a slide.

GitClear’s 2025 look at 211 million changed lines — Google, Microsoft, Meta, and enterprise C-corps, 2020 through 2024 — is what day-zero looks like when it becomes the default. Lines associated with refactoring fell from 25% of changed code in 2021 to less than 10% in 2024. Copy/pasted lines rose from 8.3% to 12.3% in the same window, and 2024 was the first year in their dataset where copy/paste exceeded moved (refactored) code. Assistants do beget more lines. Senior developers, asked what would unlock their team, do not answer “more lines.” They answer reuse, tests, and the courage to change old code. Those are the habits a day-zero loop does not practice, because they do not show up in the demo.

GitHub putting a name and a toolkit on this — Spec Kit , “define what to build before building it” — is the industry catching up to a sentence some of us have been repeating for twenty years. I am glad it exists. Specs that live as markdown for an agent to interpret are still a step up from a vibe. They are not the same as a spec that is structurally validated, wording-gated, and tied to a real Cucumber suite and a TDD state machine that will refuse to refactor on red. That difference is why bdd exists. The closest tools I know of each do part of this. None of them combine a machine-validated spec, a wording gate, real Cucumber across Java, JavaScript/TypeScript, .NET, and Rust, an enforced test-state machine, typed mutations with no shell escape hatch, an embedded MCP server, and a local-only LLM, in one binary.

It does not behoove them to teach you this

I want to be as careful here as I was in the token-bill post. This is an incentive, not a conspiracy. Model providers have good reasons to make models more capable, and customers are free to buy that capability. Still, the bill is not subtle. On the two hosted models I priced, output tokens cost five to eight times input tokens. When a model does more work, customers usually buy more inference.

It does not behoove a frontier lab to teach you a workflow in which an 8B local model drafts a requirement, a validator rejects the sloppy wording, a test runner produces a red bar, and the model is only then allowed to attempt the smallest patch that turns it green. That workflow spends its calories in code you already own. It does not spend them on a metered API. The labs need the habit: open the chat, describe a feeling, accept a tree of files, come back when it breaks. Addiction is an ugly word for a pricing model, but the loop is the same shape. You stay because the first screenshot was free in time and expensive in everything that came after.

I do not think the models are the enemy. I think the missing guidance is the enemy, and the missing guidance is the same guidance we have had since before any of this: write the spec, write the test, write the code last, keep all three in sync. Frontier models do not ship with that. A CLI can.

The rehire articles are a symptom. They are not the point.

I have seen the articles, and the videos, about companies hiring people back after betting that AI would replace them. Some of that reporting is looser than it sounds. The number that is actually sourced is not “developers,” and I am not going to pretend it is.

Gartner’s February 2026 forecast is about customer service. By 2027, they say, 50% of companies that attributed headcount reduction to AI will rehire staff to perform similar functions, under different job titles. Their own October 2025 survey of 321 customer service and support leaders found that only 20% had actually reduced agent staffing because of AI. Kathy Ross, a Gartner analyst on that practice, said most recent workforce reductions were influenced by broader economic conditions rather than automation alone. Emily Potosky, in the same release: “AI simply isn’t mature enough to fully replace the expertise, empathy, and judgment that human agents provide.”

Klarna is the case everyone uses, and it is also customer service. In May 2025, CEO Sebastian Siemiatkowski told Bloomberg that cost had been too predominant a factor, and that what you end up with is lower quality. He started recruiting so customers could always speak to a person. That is not “AI failed, delete the chatbot.” It is “we over-indexed on the demo, and the demo was not the job.”

IBM is moving the other direction on purpose. CHRO Nickle LaMoreaux said the company plans to triple US entry-level hiring in 2026 , across software and the rest of the business, because “if we don’t continue to invest in entry-level hires, what happens in 3–5 years? There’s no pipeline; the well simply dries up.” That is not a rehire of the old org chart. It is an admission that if you delete the bottom of the profession, you do not get a more efficient profession. You get a cliff.

The engineering version of this story is not a Gartner headcount number I can cite. It is the GitClear chart: more lines, less reuse, more churn, code that is written to be shipped today and expensive to touch tomorrow. Companies that staffed for day-zero generation and starved the people who can tell a spec from a vibe will hire some of those people back. Of course they will.

If anyone who knows me is watching that wave, I hope what comes back is only the 30%. I hope the 70% are now out of the equation. I am going to leave the 70–20–9–1 split to another post. Trust me, it will offend a lot of people. It is not meant to. It is about priorities, and about other aspects of life, and about whether filling a seat was ever the right use of someone’s years. If this era does one decent thing, it will be to stop asking the 70% to occupy a spot they were never going to love, and to let them go explore the rest of a life. That is only possible if the 20% take the AI and use it for good — not to manufacture day-zero code, and not to skip the techniques that have been taught for as long as I have been doing this. The 20% have to keep the spec, the tests, and the code in sync. Nobody else is going to do that for them. The frontier model will not. It is not in its interest.

Try it on the machine you already have

Clone the repo , or install bdd from the latest release . Install Ollama if you do not have it, pull something small, and point the CLI at it:

ollama pull qwen3-coder-next:q4_K_M
bdd model use qwen3-coder-next:q4_K_M
mkdir calculator && cd calculator
bdd greenfield

Describe a calculator in a sentence. Watch a local model split that sentence into a requirement. Watch validate_spec and refine_requirement refuse the sloppy wording. Approve the spec when it is actually the behavior you want. Watch the bar go RED, then GREEN. You will not have called a frontier API. You will have practiced the same discipline I have been arguing for for twenty years, at the speed an agent can finally sustain.

To learn more about Spec-Driven Development — the workshop, the CLI, the talk, and how the loop actually runs — start at davidparry.github.io/tdd-bdd-agentic .

That is the whole claim. Frontier models are optional. The spec is not. Day-zero is a screenshot. Software is what is still true after the screenshot.

💰 Skills vs MCP: What the Token Bill Actually Measures

Mon, 20 Jul 2026 09:00:00 -0500

Move deterministic business rules out of the model and into code. For the one workload I benchmarked, that single decision cut hosted-model cost by 32% on Claude Opus 4.8 and 76% on GPT-5, ran two to five times faster, and eliminated a silent age-calculation error because an authoritative clock replaced the model’s guess. That is the conclusion. Everything below is how I measured it, why the popular “Skills versus MCP” framing is the wrong dividing line, and where the result stops being defensible.

The question that sent me down this path came after a conference talk I gave about building MCP servers. Another engineer asked me something blunt: Why are you still talking about MCP when Skills can do all of this?

My first reaction was equally blunt. If I worked for a model provider, I might prefer the design that keeps more work inside the model. I work for the organization paying the bill, so I want deterministic business rules to run in ordinary code whenever that is practical.

That is my opinion, not evidence of a vendor conspiracy. Providers have good reasons to make models more capable, and customers are free to choose the architecture. Still, the incentive is worth noticing: when a model does more work, customers usually buy more inference. The two hosted models in this benchmark also price output tokens at five to eight times their input-token rate.

So I built a small benchmark to see how much that choice mattered. The result supported my architectural instinct, but it also exposed a problem with the original framing of this article. “Skills versus MCP” is not the real dividing line. The real one is model-executed rules versus code-executed rules.

What I actually compared

A Skill is a package, not an execution environment. The open Agent Skills specification allows a Skill to contain instructions, reference material, assets, and executable scripts. OpenAI’s description of Skills likewise says a Skill can include code.

MCP is a protocol through which a model-facing client can discover and call tools. An MCP server can contain business logic, but it can just as easily wrap a database, an API, a clock, or a bad nondeterministic service. The protocol itself does not make the result correct.

The benchmark therefore compared these specific implementations:

Instruction-only Skill: the Skill contains the rules and catalog in Markdown. The model performs the calculations.
MCP-backed tool: a Rust server performs the calculations and returns structured data. The model calls it and formats the response.
Prompt-only baseline: the same rules are placed directly in the system prompt and the model performs the calculations.

That is a narrower and more useful comparison: model-executed rules versus code-executed rules.

An executable Skill would make a legitimate fourth configuration, but “the Skill contains code” is not enough information. There are two very different ways an agent can use that code. It can run the script in a real software runtime, or it can read the source and attempt to follow the logic itself. The second option is still model-executed business logic. In fact, it may be worse than giving the model concise rules: the source consumes more context, while the model remains free to miss a branch, mishandle a boundary, or improvise around the implementation.

Running the script is different. Once ordinary software receives the same validated inputs, it can produce the same authoritative output as the Rust server. The remaining uncertainty sits in the orchestration around it. Will the model load the Skill, notice the script, invoke it instead of doing the calculation itself, construct the command and arguments correctly, and present the returned value without “correcting” it? If script execution is merely suggested in prose, the architecture still depends on the model choosing the deterministic path on every request.

If the agent runtime exposes the bundled script as a required, typed capability and reliably routes the request through it, I would expect the token and latency profile to resemble the MCP-backed path. At that point, however, the important win comes from executable code being treated as authoritative, not from the Skill label. If the model is expected to read and mentally execute the bundled source, I would expect more tokens without gaining the guarantee that made the code worth writing.

I have not tested either version of that fourth configuration yet. A useful follow-up would measure them separately: Skill plus enforced script execution and Skill plus source code for the model to interpret. Combining those into one “executable Skill” result would hide the architectural difference, but both versions still hit the core economic question: how much material must the model read, reason about, and generate before the code runs? Loading Skill instructions or source, deciding to invoke a script, constructing the command and arguments, and then consuming the result all use tokens. Executable code makes the calculation authoritative; it does not make the orchestration free. My expectation is that asking the model to interpret source will be the most expensive version, while enforced script execution should be closer to MCP. Whether it is cheaper or more expensive than a compact, typed MCP tool call is something the fourth benchmark must measure rather than assume.

The workload

The customer record contains a birth date, ZIP code, ordered interests, and a maximum budget. The system must calculate age, classify the customer, filter a small activity catalog, apply discounts, rank eligible activities, and return the best matches.

The harness ran all three configurations 15 times on each of three models:

Claude Opus 4.8
GPT-5
qwen3:30b, served locally through Ollama

That produced 135 responses. The harness recorded provider-reported input, output, and total tokens; model and tool calls; end-to-end latency; estimated API cost; and the final text. The Rust server, Skill, prompts, harness, tests, and raw JSON are in the public benchmark repository .

There is one important wrinkle. The customer request did not include the current date. The MCP server read the host’s clock, while the instruction-only configurations had no equivalent clock tool. An earlier draft of this article said the date was part of the scenario. It was not.

That means the cost and latency comparison uses the same user request, but the later age comparison is not a controlled test of code versus model arithmetic. It also tests access to current state. I discuss those results separately rather than pretending the difference does not exist.

Results

Every number below is the mean of 15 successful runs. Prices are standard, non-cached API prices at the time of the test. GPT-5 was $1.25 per million input tokens and $10 per million output tokens . Claude Opus 4.8 was $5 and $25 , respectively.

The “text variation” column is the mean pairwise character-sequence difference among the 15 final answers, calculated with Python’s SequenceMatcher. It measures how different the rendered responses were. It does not measure semantic correctness, business-rule determinism, or auditability.

Claude Opus 4.8

Config	Input tokens	Output tokens	Total tokens	Model calls	Input $	Output $	Total $	Latency (s)	Text variation
Instruction-only Skill	1,210	1,193	2,403	1	$0.00605	$0.02983	$0.03588	12.5	48.5%
MCP-backed tool	2,071	561	2,632	2	$0.01036	$0.01403	$0.02438	9.0	38.8%
Prompt only	1,131	1,231	2,362	1	$0.00566	$0.03078	$0.03643	12.9	61.9%

GPT-5

Config	Input tokens	Output tokens	Total tokens	Model calls	Input $	Output $	Total $	Latency (s)	Text variation
Instruction-only Skill	802	2,089	2,891	1	$0.00100	$0.02089	$0.02189	16.1	76.8%
MCP-backed tool	963	408	1,371	2	$0.00120	$0.00408	$0.00529	4.8	38.7%
Prompt only	783	2,320	3,103	1	$0.00098	$0.02320	$0.02418	20.8	67.8%

qwen3:30b (local via Ollama on an Apple M4 Max with 128 GB RAM)

Config	Input tokens	Output tokens	Total tokens	Model calls	Input $	Output $	Total $	Latency (s)	Text variation
Instruction-only Skill	898	4,192	5,090	1	$0.00	$0.00	$0.00	43.4	76.7%
MCP-backed tool	1,480	1,086	2,565	2	$0.00	$0.00	$0.00	10.8	58.9%
Prompt only	882	5,112	5,994	1	$0.00	$0.00	$0.00	54.4	73.2%

Cost per 1,000 requests

Model	MCP-backed tool	Instruction-only Skill	Prompt only	Tool vs Skill	Tool vs Prompt
Claude Opus 4.8	$24.38	$35.88	$36.43	32% less	33% less
GPT-5	$5.29	$21.89	$24.18	76% less	78% less

What those numbers support

The tool-backed path sent more input because the request included a tool schema and required a second model call. It still cost less because it used far fewer completion tokens. On these models, for this task, the price difference between input and output was large enough to overwhelm the extra round trip.

The same implementation was also faster in this sweep. It reduced mean latency from 12.5 to 9.0 seconds against the instruction-only Skill on Opus, from 16.1 to 4.8 seconds on GPT-5, and from 43.4 to 10.8 seconds on the local model. The local comparison matters because there is no per-token invoice to blame. Generating fewer tokens still consumed less wall-clock time on my hardware.

The provider APIs reported completion-token usage; those counts are what I priced. Depending on the provider and model, that usage can include billed reasoning tokens that never appear in the visible answer. It would be inaccurate to say every calculation was literally printed to the user.

The text-variation result is interesting but weaker. Tool-backed answers were more alike in all three sets, which makes sense because the model received the same structured result each time. A character-level similarity score is sensitive to headings, wording, and answer length, though. I would not use it as evidence that the business logic is deterministic. For that, I would test the function outputs directly.

Most important, this is one synthetic workload with one customer and one catalog. It shows that moving this set of rules into code reduced tokens, latency, and hosted-model cost. It does not prove that an MCP call is always cheaper. A large tool schema, a chatty tool response, network latency, retries, or a tiny calculation could reverse the result.

The clock result: useful, but not a fair arithmetic contest

All configurations returned the same three activities in the same order:

Guided Nature Walk — $20
Mountain Hiking Tour — $45
Jazz Club Evening — $60

The reported age differed:

Config	Claude Opus 4.8	GPT-5	qwen3:30b
MCP-backed tool	36 (15/15)	36 (15/15)	36 (15/15)
Instruction-only Skill	35 (14/15)	36 (15/15)	33 (12/15)
Prompt only	35 (15/15)	36 (15/15)	33 (14/15)

The benchmark ran on July 19, 2026, and the customer was born on May 15, 1990, so 36 was correct. The Rust service read that date from the system clock. Opus often behaved as if the year were 2025, and qwen often behaved as if it were 2023. GPT-5 returned the exact benchmark date even though the user request and harness system prompt did not supply it. The data does not tell me whether that date came from model behavior or provider-side context, so calling it a “guess” would go beyond the evidence.

This result demonstrates a real production lesson: if an answer depends on current state, give the system an authoritative source for that state. It does not demonstrate that MCP is uniquely able to provide one. A bundled Skill script, a local command, a conventional API, or an MCP server could all read a clock.

It also explains why the bad ages did not change the recommendations. Ages 33, 35, and 36 fall into the same age band, generation, and discount tier in this catalog. Near a boundary, the error could matter. A 65-year-old calculated as 62 would miss the senior discount; an 18-year-old calculated as 17 could cross discount and eligibility rules. Those are examples of what the defect could cause, not outcomes observed in this run.

For repeatable testing, the server already has a FixedClock implementation. The benchmark should use it, or pass the same explicit date to all three configurations. I plan to add that controlled case before making broader correctness claims.

Where I draw the boundary

This experiment reminded me of reading a SQL execution plan. Returning the right rows is necessary, but it is not the end of the engineering work. At scale, I also care about the cost of getting those rows, the behavior under failure, and whether I can test the logic without asking a probabilistic model to repeat it.

My rule of thumb is now:

Use Skill instructions for judgment, workflow, and reusable operating procedures.
Use executable code for calculations, eligibility, prices, policy rules, and access to authoritative state.
Use MCP when that executable capability should be a shared, discoverable tool with a stable interface. A bundled Skill script or ordinary service may be simpler when it should not.

That is also what I look for when interviewing engineers who use AI. This is a personal hiring preference, not a universal standard. I care less about whether a candidate can make a model produce working code and more about whether they can explain which parts belong in model reasoning, which parts need a deterministic boundary, and what the choice costs at production volume.

The benchmark changed my wording, not my conclusion. “MCP beats Skills” is too broad. The claim I can defend is this: for this workload, code-executed business rules beat model-executed rules on cost and latency, and an authoritative clock prevented a silent data error.

That is enough to influence an architecture. It is not enough to declare a universal winner.

Reproduce the experiment or challenge it in the skill-vs-mcp repository . The raw 15-run result set used here is included.