Agentic on David Parry

🖥️ You Don't Need a Frontier Model. You Need a Spec.

Fri, 14 Aug 2026 15:00:00 -0500

You do not need a frontier model to build professional software. You need a spec, a test bar, and a workflow that will not let anyone — human or model — skip either. Frontier models are what you reach for when you want day-zero results: something that compiles this afternoon and looks finished until you have to live with it. A responsible solution is slower in the screenshot and faster over the life of the system. The surprise of the last year, for me, is that this path is now cheap enough to run on a laptop.

I already wrote that spec-first was always right, and that agents finally made it fast . That post was the argument. This one is what I learned once I stopped treating the hosted model as the product and started treating a local model as one tool inside a spec-driven loop. The working proof is the bdd CLI that grew out of that workshop — one native binary, Ollama by default, no cloud calls, no token meter.

The CLI is the discipline, compiled

The workshop needed an MCP server so an agent could not wander. The CLI is that same loop as a program you run yourself. The command manual is the full surface; the idea is one sentence:

The requirements spec is the source of truth, and the discipline is enforced by tooling, not by convention.

Everything else follows from that. The spec is machine-validated (bdd spec validate) and its wording is quality-gated (bdd spec refine) before any scenario or line of production code exists. A valid-but-vague requirement is caught and reworded first. Behavior flows downhill from the approved spec: Gherkin scenarios tagged with requirement ids, step definitions, unit tests, and only then production code. The Red/Green/Refactor cycle is a state machine, not a suggestion — start_refactor is refused on a red bar, and a requirement is only marked implemented behind a green one.

Agents get no escape hatches. No arbitrary file writes, no shell, no “just install this for me.” Every mutation goes through a typed, validated tool, lands in staging (.bdd-staged/), and waits for a human to review it. The same tools serve a person at a prompt and an agent over MCP. The LLM is local, discovered rather than assumed, and generation falls back to deterministic templates when Ollama is down or empty. Nothing is ever installed for you.

Run bare bdd and you get the loop, the version, and whatever local model is already on the machine:

$ bdd

  ╭──────────────────────────────────╮
  │                                  ▼
  │    > bdd  v0.2.4                 │
  │    spec → RED → GREEN → REFACTOR │
  ▲                                  │
  ╰──────────────────────────────────╯

Model set for this session: qwen3-coder-next:q4_K_M (not saved - keep it with: bdd model use qwen3-coder-next:q4_K_M).
bdd>

From an empty directory, bdd greenfield runs the whole creation order with exactly two human gates: the wording of the driving spec, and the review of generated tests before they are committed. From an existing project, you drive the same phases yourself — spec draft, spec validate, spec refine, test, implement, refactor — and bdd status names the one next step that actually moves the loop forward.

The model is not sitting above this process. It is boxed inside a few commands: drafting a requirement from plain words, polishing step definitions and unit tests, attempting an implementation against a recorded RED bar. test, state, and refactor never call it. That is not a prompt instruction. It is the architecture.

What I actually learned about local models

The lesson is not “an 8B model beats Opus.” The lesson is: once you move the deterministic work out of the model, a local model becomes enough.

I measured a version of that claim in Skills vs MCP . For one workload, putting the business rules in code instead of in the model’s head cut hosted cost by 32% on Claude Opus 4.8 and 76% on GPT-5. The same implementation on qwen3:30b, served locally through Ollama on an Apple M4 Max, cost nothing and dropped mean latency from 43.4 seconds (instruction-only Skill) and 54.4 seconds (prompt only) to 10.8 seconds with the tool. There is no per-token invoice to blame on that last comparison. Generating fewer tokens still consumed less wall-clock time on my hardware.

The CLI takes that architecture and applies it to the development loop itself. Spec validation is code. Wording critique is code. The TDD state machine is code. Gherkin parsing is code. Test execution is Maven, cucumber-js, dotnet test, or cargo test — the project’s own runner, not a model’s impression of a runner. The model is asked to propose, not to be the system of record. When it proposes a step definition, the CLI prefers that output only if it validates; otherwise the deterministic template ships. That is why qwen3-coder-next:q4_K_M is a reasonable default on this tool, and why qwen3:30b is a luxury rather than a requirement.

Open-weight coding models have also closed enough of the raw-capability gap that this is no longer a thought experiment. Alibaba’s Qwen3-Coder reports state-of-the-art results among open models on SWE-Bench Verified without test-time scaling, and describes the 480B variant as comparable to Claude Sonnet 4 on agentic coding. You can argue with any one leaderboard. You cannot argue with the direction: the model you can run next to the repo is no longer a toy, and the remaining gap is exactly where unconstrained agent loops still fall down — long-horizon, multi-file, “figure out what I meant” work. Spec-driven development is how you stop asking the model to do that work.

METR’s early-2025 randomized trial is the other data point I keep coming back to. Experienced open-source developers working on their own mature repositories, using frontier tools of the day (primarily Cursor with Claude), took 19% longer when AI was allowed. They expected a 24% speedup. After living through the slowdown, they still believed they had been sped up by 20%. Perception and the clock disagreed. METR was careful about what that does not prove. They said, explicitly, that they do not provide evidence “there are not ways of using existing AI systems more effectively” — scaffolding, prompting, repository-specific context. Their later 2026 follow-up ran into a different problem: too many developers now refuse to work without AI, so the experiment can no longer see the tasks where people expect the biggest lift. I read both results the same way. Unconstrained generation on a real codebase is not the same activity as a gated spec-to-green loop. The first can feel fast and still be slow. The second is slower to start and cheaper to finish, and it does not require the most expensive model on the market.

Day-zero is a product. Professional software is a process.

I will say this as cleanly as I can. Frontier models are extraordinary at day-zero. You describe a thing, files appear, a demo boots, a screenshot looks like a product. That is a real capability, and it is the capability the labs demo, because it is the capability that converts. If what you wanted was a prototype before lunch, pay for the tokens. I do that too.

Professional software is a different shape. Someone has to write down what “done” means in a form that can be checked. Tests have to fail for the right reason before they pass for the right reason. The code has to be the simplest thing that makes that true, and then it has to be reviewed. Edge cases have to be found on purpose, not stumbled into in production. That work has been preached at us for as long as I have been doing this — test-first, behavior-first, requirements-first. Cucumber and Gherkin were the first tools that let the requirement itself become executable, and I never went back. What changed is not the advice. What changed is that an agent will now execute the advice if you put it in a tool instead of a slide.

GitClear’s 2025 look at 211 million changed lines — Google, Microsoft, Meta, and enterprise C-corps, 2020 through 2024 — is what day-zero looks like when it becomes the default. Lines associated with refactoring fell from 25% of changed code in 2021 to less than 10% in 2024. Copy/pasted lines rose from 8.3% to 12.3% in the same window, and 2024 was the first year in their dataset where copy/paste exceeded moved (refactored) code. Assistants do beget more lines. Senior developers, asked what would unlock their team, do not answer “more lines.” They answer reuse, tests, and the courage to change old code. Those are the habits a day-zero loop does not practice, because they do not show up in the demo.

GitHub putting a name and a toolkit on this — Spec Kit , “define what to build before building it” — is the industry catching up to a sentence some of us have been repeating for twenty years. I am glad it exists. Specs that live as markdown for an agent to interpret are still a step up from a vibe. They are not the same as a spec that is structurally validated, wording-gated, and tied to a real Cucumber suite and a TDD state machine that will refuse to refactor on red. That difference is why bdd exists. The closest tools I know of each do part of this. None of them combine a machine-validated spec, a wording gate, real Cucumber across Java, JavaScript/TypeScript, .NET, and Rust, an enforced test-state machine, typed mutations with no shell escape hatch, an embedded MCP server, and a local-only LLM, in one binary.

It does not behoove them to teach you this

I want to be as careful here as I was in the token-bill post. This is an incentive, not a conspiracy. Model providers have good reasons to make models more capable, and customers are free to buy that capability. Still, the bill is not subtle. On the two hosted models I priced, output tokens cost five to eight times input tokens. When a model does more work, customers usually buy more inference.

It does not behoove a frontier lab to teach you a workflow in which an 8B local model drafts a requirement, a validator rejects the sloppy wording, a test runner produces a red bar, and the model is only then allowed to attempt the smallest patch that turns it green. That workflow spends its calories in code you already own. It does not spend them on a metered API. The labs need the habit: open the chat, describe a feeling, accept a tree of files, come back when it breaks. Addiction is an ugly word for a pricing model, but the loop is the same shape. You stay because the first screenshot was free in time and expensive in everything that came after.

I do not think the models are the enemy. I think the missing guidance is the enemy, and the missing guidance is the same guidance we have had since before any of this: write the spec, write the test, write the code last, keep all three in sync. Frontier models do not ship with that. A CLI can.

The rehire articles are a symptom. They are not the point.

I have seen the articles, and the videos, about companies hiring people back after betting that AI would replace them. Some of that reporting is looser than it sounds. The number that is actually sourced is not “developers,” and I am not going to pretend it is.

Gartner’s February 2026 forecast is about customer service. By 2027, they say, 50% of companies that attributed headcount reduction to AI will rehire staff to perform similar functions, under different job titles. Their own October 2025 survey of 321 customer service and support leaders found that only 20% had actually reduced agent staffing because of AI. Kathy Ross, a Gartner analyst on that practice, said most recent workforce reductions were influenced by broader economic conditions rather than automation alone. Emily Potosky, in the same release: “AI simply isn’t mature enough to fully replace the expertise, empathy, and judgment that human agents provide.”

Klarna is the case everyone uses, and it is also customer service. In May 2025, CEO Sebastian Siemiatkowski told Bloomberg that cost had been too predominant a factor, and that what you end up with is lower quality. He started recruiting so customers could always speak to a person. That is not “AI failed, delete the chatbot.” It is “we over-indexed on the demo, and the demo was not the job.”

IBM is moving the other direction on purpose. CHRO Nickle LaMoreaux said the company plans to triple US entry-level hiring in 2026 , across software and the rest of the business, because “if we don’t continue to invest in entry-level hires, what happens in 3–5 years? There’s no pipeline; the well simply dries up.” That is not a rehire of the old org chart. It is an admission that if you delete the bottom of the profession, you do not get a more efficient profession. You get a cliff.

The engineering version of this story is not a Gartner headcount number I can cite. It is the GitClear chart: more lines, less reuse, more churn, code that is written to be shipped today and expensive to touch tomorrow. Companies that staffed for day-zero generation and starved the people who can tell a spec from a vibe will hire some of those people back. Of course they will.

If anyone who knows me is watching that wave, I hope what comes back is only the 30%. I hope the 70% are now out of the equation. I am going to leave the 70–20–9–1 split to another post. Trust me, it will offend a lot of people. It is not meant to. It is about priorities, and about other aspects of life, and about whether filling a seat was ever the right use of someone’s years. If this era does one decent thing, it will be to stop asking the 70% to occupy a spot they were never going to love, and to let them go explore the rest of a life. That is only possible if the 20% take the AI and use it for good — not to manufacture day-zero code, and not to skip the techniques that have been taught for as long as I have been doing this. The 20% have to keep the spec, the tests, and the code in sync. Nobody else is going to do that for them. The frontier model will not. It is not in its interest.

Try it on the machine you already have

Clone the repo , or install bdd from the latest release . Install Ollama if you do not have it, pull something small, and point the CLI at it:

ollama pull qwen3-coder-next:q4_K_M
bdd model use qwen3-coder-next:q4_K_M
mkdir calculator && cd calculator
bdd greenfield

Describe a calculator in a sentence. Watch a local model split that sentence into a requirement. Watch validate_spec and refine_requirement refuse the sloppy wording. Approve the spec when it is actually the behavior you want. Watch the bar go RED, then GREEN. You will not have called a frontier API. You will have practiced the same discipline I have been arguing for for twenty years, at the speed an agent can finally sustain.

To learn more about Spec-Driven Development — the workshop, the CLI, the talk, and how the loop actually runs — start at davidparry.github.io/tdd-bdd-agentic .

That is the whole claim. Frontier models are optional. The spec is not. Day-zero is a screenshot. Software is what is still true after the screenshot.

🔺 Spec-First Was Always Right — Agents Just Made It Fast

Fri, 07 Aug 2026 16:00:00 -0500

For twenty years I have argued that requirements, tests, and code are the same information at three altitudes, and that the job is to keep them in sync. The argument has not changed — the economics finally have. The correct way was always to nail the spec first, but that was the slower way, and when a human sits down at a keyboard there has always been an unspoken rule that whatever they type had better be code. Spec-writing looked like stalling. So teams typed the code, and the spec — if it ever existed — drifted into fiction. That trade-off is now dead. With an agent doing the transformation from requirement to test to implementation, the spec-first path is no longer the slow path. Getting it right is now faster than winging it, because a well-captured requirement is the thing an agent can actually execute against, and a vague one is the thing you pay for in review, rework, and slop.

I recently decided to polish my TDD talk, TDD in the Agentic Era, with exactly this in mind. The talk always rested on having the spec; the revision makes the spec mandatory. The workflow now begins at the requirement and refuses to move without one — the agent cannot write a test or a line of implementation until it has pulled the spec and turned its acceptance criteria into an executable scenario. In the 60-minute hands-on version, every artifact is real, runnable code — an MCP server, a client agent, a requirements backlog, Gherkin scenarios, and a Red/Green/Refactor state machine that refuses to let anyone, human or AI, refactor on a red bar. It is public in the tdd-bdd-agentic repository , and I use it through this post as the working proof of the argument. But the workshop is the demonstration, not the point. The point is that my old conviction is holding true, and the time to do this — and get it right — is now: why this style finally pays for itself, and why this exact idea, spec-driven development as a triangle, is the reason I joined CodiumAI. I chose them for Itamar’s vision — a vision that was not completely deliverable at the time, given where the models were, but was spot on about where this was all going. That company is the one you now know as Qodo.

What we accomplish in the hour

Everyone has watched an agent demo. Almost nobody has driven the workflow underneath one. That is the gap the workshop closes — and deliberately not by building plumbing. The MCP server and client come completed in the repo (Java, stdio transport, tested to 100% coverage); MCP gets about seven minutes as the standard way to feed your tools into whatever agent you use — Cursor, Claude Desktop, or the bundled CLI client, which narrates the handshake, discovery, and invocation that every IDE does under the hood. The rest of the hour, with 35 of the 60 minutes at your keyboard instead of looking at mine, every attendee:

Drafts and refines a requirement with an agent. You describe the intent in a sentence; the agent writes the requirement into the backlog; and then the spec iterates through two server feedback loops. First structure: validate_spec reports every issue — a criterion missing its Then, a duplicate id, broken JSON — and the agent fixes and re-validates until the spec is valid. Then wording: refine_requirement critiques the draft — “‘quickly’ is ambiguous,” “the story is missing its why,” “only happy paths, add an edge case” — and the LLM rewords against that deterministic feedback until the critique comes back clean. The agent drafts, the server arbitrates, the human approves the final wording.
Drives the full spec-to-green loop. A real LLM agent validates the spec, picks up a pending requirement through the seven workflow tools (list_requirements, get_requirement, validate_spec, refine_requirement, run_tests, get_tdd_state, start_refactor), writes the Gherkin scenario for its acceptance criteria, runs the tests to show RED, implements the simplest code to reach GREEN, and then — only then — refactors.

The kata under all of this is deliberately boring: the String Calculator. The point was never the calculator. The point is the workflow wrapped around it.

Three altitudes, one discipline

The center of the talk is a slide I call the three altitudes. It is not “just TDD.” It composes the three spec-first methodologies, each pinning the system at a different height, and the agent works across all of them:

Methodology	Pins	Canonical artifact
SDD (spec-driven)	the feature	a versioned spec with acceptance criteria — in the repo, `requirements/requirements.json`
BDD (behavior-driven)	one behavior	a Gherkin scenario, executed by Cucumber
TDD (test-driven)	one unit	a failing JUnit test

The flow is spec-down. The agent reads a requirement (SDD), turns its acceptance criteria into a tagged Gherkin scenario (BDD), adds unit tests where useful (TDD), and the run_tests tool runs Cucumber and JUnit together — one bar, one color. Tests are generated from the spec, not reverse-engineered from the code afterward. That direction is the entire spec-driven claim.

Here is what the agent actually produces during the spec-to-green exercise, for a requirement whose acceptance criteria were already phrased Given/When/Then in the spec:

  @REQ-003
  Scenario: Two numbers separated by a comma are summed
    Given a string calculator
    When I add "1,2"
    Then the result is 3

And here is the prompt the audience pastes into their agent — notice how much of it is workflow, not code:

Using the tdd-workflow tools: validate the spec first, then find the next pending requirement, add a Gherkin scenario for its acceptance criteria to the feature file (tag it with the requirement id), reuse or add step definitions, run the tests to show RED, then implement the simplest code to reach GREEN, then refactor. Ask me before each phase change.

There are two hard human checkpoints in the loop. The first is a spec review: when the agent writes the scenario, the room stops and reads it aloud — is this the behavior we want? The second is diff approval after GREEN. The agent turns the crank; the human steers. And the discipline is not a suggestion living in a prompt — it lives in the tool. Ask the server to start_refactor while the bar is red and it refuses. Guardrails belong in code, not in vibes.

Twenty years a convert

I have been a convert to this style for twenty years. Test-first, behavior-first, and above all requirements-first — the conviction that you cannot build the right thing until someone has written down, in a form that can be checked, what the right thing is. I lived through the era when the spec was a Word document that was obsolete before the first sprint ended. Cucumber and Gherkin were the first tools that let the requirement itself become executable, and I never went back.

But I should be honest: being a convert and getting to practice it were two different things. The conviction was mostly private, because in most places I worked the requirement either did not exist at all or arrived from someone outside engineering — a product owner, a stakeholder, a business analyst — who was never equipped to turn what they wanted into a technical requirement or a PRD an engineer could build against. What came across was rarely close. That was not their failing. They knew the business cold; they simply were not trained to interrogate a request the way it needs to be interrogated — to dig for the edge cases and the unstated assumptions, to ask the questions that actually need asking, and to pin the answers down in a form that can be checked. So I spent twenty years believing in requirements-first while working in organizations that rarely produced a requirement worth the name.

Which is the whole point, because it tells you where the scarce skill always lived. It was never typing the code. It was extracting what is actually needed — from the ticket, from the stakeholder, from the silence between what they said and what they meant — and pinning it down in a testable form. That is why the agentic era feels like vindication rather than disruption to me. An agent is a phenomenal crank-turner, but it can only turn the crank on requirements someone captured well — and if that skill was scarce when a human wrote the code, it is scarcer and more valuable now that an agent will faithfully execute whatever you hand it, good spec or bad. Which leads me to believe the developer’s real emerging role is requirements gatherer for the agent — but that is another post, and I intend to write it.

The triangle that made me join

In July 2024 I was deciding whether to join a startup called CodiumAI. Plenty of companies were pitching AI code generation; what convinced me was that the CEO, Itamar Friedman , was saying the same thing I had spent twenty years believing — and he had drawn it as a triangle: spec, tests, code, kept in sync. He lays it out most explicitly in his Gen 3.5 framing , where agents collaborate around three core artifacts, Specs → Code → Tests, improving the spec, transforming it into code, generating and executing tests, and keeping all three tightly aligned as each evolves. I was not joining a company that bolted testing onto generation as an afterthought; I was joining one whose founding thesis was that the three corners are the same information at different altitudes and the job of AI is to keep them aligned. When I joined, that vision was ahead of what the models could deliver. Reading it now, with the workshop in this post running exactly that loop on my laptop, we are close — if not already there.

The workshop is the triangle, running

Look back at the workshop with the triangle in mind and the mapping is one-to-one:

Spec corner: requirements.json, versioned, with acceptance criteria — drafted with the agent, held structurally valid by validate_spec and iterated to clean wording by refine_requirement, and the source of truth the agent implements from.
Tests corner: the Gherkin feature file and the JUnit tests, generated from the spec, executed together as one bar.
Code corner: StringCalculator.java, written last, as the simplest thing that makes the bar green.

The MCP server is the connective tissue — the standard way for an agent to discover your workflow tools and respect your discipline. The human sits at the two checkpoints where judgment lives: is this the right spec? and is this the right code? Everything between those checkpoints is crank-turning, and the crank no longer needs to be turned by hand.

Twenty years ago, keeping the three corners in sync was a manual discipline that most teams abandoned under deadline pressure. The spec drifted, the tests decayed, and the code became the only truth — unreadable, unverifiable truth. What the agentic era changes is the cost of the discipline. The alignment work that teams always skipped is exactly the work agents are good at.

The talk ends with homework: on trunk — the exact starting point the class clones — requirements REQ-004 through REQ-006 are still pending, and the requirement the room drafted in Exercise 1 is waiting to be taken to green. Clone the repo , point your agent at the tools, and finish the kata on the plane home. Watch the bar go RED, then GREEN. And if you want the answer key, the complete branch is the kata fully driven through the loop — every requirement implemented, every scenario tagged and green — with CI telling the story on both branches: trunk deliberately fails a class-completeness check because the work is still ahead of you; complete passes everything. Then ask yourself who really wrote the requirement — because that person, not the agent, decided what got built. That’s the next post.

💰 Skills vs MCP: What the Token Bill Actually Measures

Mon, 20 Jul 2026 09:00:00 -0500

Move deterministic business rules out of the model and into code. For the one workload I benchmarked, that single decision cut hosted-model cost by 32% on Claude Opus 4.8 and 76% on GPT-5, ran two to five times faster, and eliminated a silent age-calculation error because an authoritative clock replaced the model’s guess. That is the conclusion. Everything below is how I measured it, why the popular “Skills versus MCP” framing is the wrong dividing line, and where the result stops being defensible.

The question that sent me down this path came after a conference talk I gave about building MCP servers. Another engineer asked me something blunt: Why are you still talking about MCP when Skills can do all of this?

My first reaction was equally blunt. If I worked for a model provider, I might prefer the design that keeps more work inside the model. I work for the organization paying the bill, so I want deterministic business rules to run in ordinary code whenever that is practical.

That is my opinion, not evidence of a vendor conspiracy. Providers have good reasons to make models more capable, and customers are free to choose the architecture. Still, the incentive is worth noticing: when a model does more work, customers usually buy more inference. The two hosted models in this benchmark also price output tokens at five to eight times their input-token rate.

So I built a small benchmark to see how much that choice mattered. The result supported my architectural instinct, but it also exposed a problem with the original framing of this article. “Skills versus MCP” is not the real dividing line. The real one is model-executed rules versus code-executed rules.

What I actually compared

A Skill is a package, not an execution environment. The open Agent Skills specification allows a Skill to contain instructions, reference material, assets, and executable scripts. OpenAI’s description of Skills likewise says a Skill can include code.

MCP is a protocol through which a model-facing client can discover and call tools. An MCP server can contain business logic, but it can just as easily wrap a database, an API, a clock, or a bad nondeterministic service. The protocol itself does not make the result correct.

The benchmark therefore compared these specific implementations:

Instruction-only Skill: the Skill contains the rules and catalog in Markdown. The model performs the calculations.
MCP-backed tool: a Rust server performs the calculations and returns structured data. The model calls it and formats the response.
Prompt-only baseline: the same rules are placed directly in the system prompt and the model performs the calculations.

That is a narrower and more useful comparison: model-executed rules versus code-executed rules.

An executable Skill would make a legitimate fourth configuration, but “the Skill contains code” is not enough information. There are two very different ways an agent can use that code. It can run the script in a real software runtime, or it can read the source and attempt to follow the logic itself. The second option is still model-executed business logic. In fact, it may be worse than giving the model concise rules: the source consumes more context, while the model remains free to miss a branch, mishandle a boundary, or improvise around the implementation.

Running the script is different. Once ordinary software receives the same validated inputs, it can produce the same authoritative output as the Rust server. The remaining uncertainty sits in the orchestration around it. Will the model load the Skill, notice the script, invoke it instead of doing the calculation itself, construct the command and arguments correctly, and present the returned value without “correcting” it? If script execution is merely suggested in prose, the architecture still depends on the model choosing the deterministic path on every request.

If the agent runtime exposes the bundled script as a required, typed capability and reliably routes the request through it, I would expect the token and latency profile to resemble the MCP-backed path. At that point, however, the important win comes from executable code being treated as authoritative, not from the Skill label. If the model is expected to read and mentally execute the bundled source, I would expect more tokens without gaining the guarantee that made the code worth writing.

I have not tested either version of that fourth configuration yet. A useful follow-up would measure them separately: Skill plus enforced script execution and Skill plus source code for the model to interpret. Combining those into one “executable Skill” result would hide the architectural difference, but both versions still hit the core economic question: how much material must the model read, reason about, and generate before the code runs? Loading Skill instructions or source, deciding to invoke a script, constructing the command and arguments, and then consuming the result all use tokens. Executable code makes the calculation authoritative; it does not make the orchestration free. My expectation is that asking the model to interpret source will be the most expensive version, while enforced script execution should be closer to MCP. Whether it is cheaper or more expensive than a compact, typed MCP tool call is something the fourth benchmark must measure rather than assume.

The workload

The customer record contains a birth date, ZIP code, ordered interests, and a maximum budget. The system must calculate age, classify the customer, filter a small activity catalog, apply discounts, rank eligible activities, and return the best matches.

The harness ran all three configurations 15 times on each of three models:

Claude Opus 4.8
GPT-5
qwen3:30b, served locally through Ollama

That produced 135 responses. The harness recorded provider-reported input, output, and total tokens; model and tool calls; end-to-end latency; estimated API cost; and the final text. The Rust server, Skill, prompts, harness, tests, and raw JSON are in the public benchmark repository .

There is one important wrinkle. The customer request did not include the current date. The MCP server read the host’s clock, while the instruction-only configurations had no equivalent clock tool. An earlier draft of this article said the date was part of the scenario. It was not.

That means the cost and latency comparison uses the same user request, but the later age comparison is not a controlled test of code versus model arithmetic. It also tests access to current state. I discuss those results separately rather than pretending the difference does not exist.

Results

Every number below is the mean of 15 successful runs. Prices are standard, non-cached API prices at the time of the test. GPT-5 was $1.25 per million input tokens and $10 per million output tokens . Claude Opus 4.8 was $5 and $25 , respectively.

The “text variation” column is the mean pairwise character-sequence difference among the 15 final answers, calculated with Python’s SequenceMatcher. It measures how different the rendered responses were. It does not measure semantic correctness, business-rule determinism, or auditability.

Claude Opus 4.8

Config	Input tokens	Output tokens	Total tokens	Model calls	Input $	Output $	Total $	Latency (s)	Text variation
Instruction-only Skill	1,210	1,193	2,403	1	$0.00605	$0.02983	$0.03588	12.5	48.5%
MCP-backed tool	2,071	561	2,632	2	$0.01036	$0.01403	$0.02438	9.0	38.8%
Prompt only	1,131	1,231	2,362	1	$0.00566	$0.03078	$0.03643	12.9	61.9%

GPT-5

Config	Input tokens	Output tokens	Total tokens	Model calls	Input $	Output $	Total $	Latency (s)	Text variation
Instruction-only Skill	802	2,089	2,891	1	$0.00100	$0.02089	$0.02189	16.1	76.8%
MCP-backed tool	963	408	1,371	2	$0.00120	$0.00408	$0.00529	4.8	38.7%
Prompt only	783	2,320	3,103	1	$0.00098	$0.02320	$0.02418	20.8	67.8%

qwen3:30b (local via Ollama on an Apple M4 Max with 128 GB RAM)

Config	Input tokens	Output tokens	Total tokens	Model calls	Input $	Output $	Total $	Latency (s)	Text variation
Instruction-only Skill	898	4,192	5,090	1	$0.00	$0.00	$0.00	43.4	76.7%
MCP-backed tool	1,480	1,086	2,565	2	$0.00	$0.00	$0.00	10.8	58.9%
Prompt only	882	5,112	5,994	1	$0.00	$0.00	$0.00	54.4	73.2%

Cost per 1,000 requests

Model	MCP-backed tool	Instruction-only Skill	Prompt only	Tool vs Skill	Tool vs Prompt
Claude Opus 4.8	$24.38	$35.88	$36.43	32% less	33% less
GPT-5	$5.29	$21.89	$24.18	76% less	78% less

What those numbers support

The tool-backed path sent more input because the request included a tool schema and required a second model call. It still cost less because it used far fewer completion tokens. On these models, for this task, the price difference between input and output was large enough to overwhelm the extra round trip.

The same implementation was also faster in this sweep. It reduced mean latency from 12.5 to 9.0 seconds against the instruction-only Skill on Opus, from 16.1 to 4.8 seconds on GPT-5, and from 43.4 to 10.8 seconds on the local model. The local comparison matters because there is no per-token invoice to blame. Generating fewer tokens still consumed less wall-clock time on my hardware.

The provider APIs reported completion-token usage; those counts are what I priced. Depending on the provider and model, that usage can include billed reasoning tokens that never appear in the visible answer. It would be inaccurate to say every calculation was literally printed to the user.

The text-variation result is interesting but weaker. Tool-backed answers were more alike in all three sets, which makes sense because the model received the same structured result each time. A character-level similarity score is sensitive to headings, wording, and answer length, though. I would not use it as evidence that the business logic is deterministic. For that, I would test the function outputs directly.

Most important, this is one synthetic workload with one customer and one catalog. It shows that moving this set of rules into code reduced tokens, latency, and hosted-model cost. It does not prove that an MCP call is always cheaper. A large tool schema, a chatty tool response, network latency, retries, or a tiny calculation could reverse the result.

The clock result: useful, but not a fair arithmetic contest

All configurations returned the same three activities in the same order:

Guided Nature Walk — $20
Mountain Hiking Tour — $45
Jazz Club Evening — $60

The reported age differed:

Config	Claude Opus 4.8	GPT-5	qwen3:30b
MCP-backed tool	36 (15/15)	36 (15/15)	36 (15/15)
Instruction-only Skill	35 (14/15)	36 (15/15)	33 (12/15)
Prompt only	35 (15/15)	36 (15/15)	33 (14/15)

The benchmark ran on July 19, 2026, and the customer was born on May 15, 1990, so 36 was correct. The Rust service read that date from the system clock. Opus often behaved as if the year were 2025, and qwen often behaved as if it were 2023. GPT-5 returned the exact benchmark date even though the user request and harness system prompt did not supply it. The data does not tell me whether that date came from model behavior or provider-side context, so calling it a “guess” would go beyond the evidence.

This result demonstrates a real production lesson: if an answer depends on current state, give the system an authoritative source for that state. It does not demonstrate that MCP is uniquely able to provide one. A bundled Skill script, a local command, a conventional API, or an MCP server could all read a clock.

It also explains why the bad ages did not change the recommendations. Ages 33, 35, and 36 fall into the same age band, generation, and discount tier in this catalog. Near a boundary, the error could matter. A 65-year-old calculated as 62 would miss the senior discount; an 18-year-old calculated as 17 could cross discount and eligibility rules. Those are examples of what the defect could cause, not outcomes observed in this run.

For repeatable testing, the server already has a FixedClock implementation. The benchmark should use it, or pass the same explicit date to all three configurations. I plan to add that controlled case before making broader correctness claims.

Where I draw the boundary

This experiment reminded me of reading a SQL execution plan. Returning the right rows is necessary, but it is not the end of the engineering work. At scale, I also care about the cost of getting those rows, the behavior under failure, and whether I can test the logic without asking a probabilistic model to repeat it.

My rule of thumb is now:

Use Skill instructions for judgment, workflow, and reusable operating procedures.
Use executable code for calculations, eligibility, prices, policy rules, and access to authoritative state.
Use MCP when that executable capability should be a shared, discoverable tool with a stable interface. A bundled Skill script or ordinary service may be simpler when it should not.

That is also what I look for when interviewing engineers who use AI. This is a personal hiring preference, not a universal standard. I care less about whether a candidate can make a model produce working code and more about whether they can explain which parts belong in model reasoning, which parts need a deterministic boundary, and what the choice costs at production volume.

The benchmark changed my wording, not my conclusion. “MCP beats Skills” is too broad. The claim I can defend is this: for this workload, code-executed business rules beat model-executed rules on cost and latency, and an authoritative clock prevented a silent data error.

That is enough to influence an architecture. It is not enough to declare a universal winner.

Reproduce the experiment or challenge it in the skill-vs-mcp repository . The raw 15-run result set used here is included.