Why “facts vs. specs” is the wrong fight — and how to wire OpenSpec to fast-check, proptest, and Hypothesis so your living spec actually compiles to something falsifiable.


There’s a Medium piece making the rounds — Jarosław Wasowski’s “Stop Writing Specs. Start Writing Facts. The Entire SDD Movement Is Already Obsolete.” The hook is good, and I think it’s worth taking seriously precisely because Wasowski isn’t an outsider throwing rocks. He has spent the last half year writing pro-SDD essays. This is a practitioner recanting, which is the most honest kind of critique.

His evidence is a single anecdote that lands hard: one executable test, written in June 2025, passed CI unmodified through Sonnet 3.5, 3.7, 4, and Opus 4.5+. The 1,500-word prose specification describing the same endpoint needed four reinterpretations over the same window. One artifact was stable across four model generations. The other drifted every time the model underneath it changed its mind about what the words meant.

He reaches for Popper to explain it — “a theory that explains everything explains nothing” — and concludes that the whole spec-driven edifice (Spec Kit, Kiro, OpenSpec, BMAD) is built on sand.

I’ve been living inside this exact problem. I run OpenSpec against Claude Code, I write spec-driven changes for a Rust-and-TypeScript stack, and I’ve watched prose requirements get re-interpreted out from under me. So I want to agree with him. But I don’t, not quite. He’s diagnosed the disease correctly and prescribed an amputation. The actual fix is smaller and stranger: a spec and a property test are the same object seen from two sides. The prose is the name. The property is the shadow it casts onto the machine. You don’t choose between them — you bind them together, and you make the binding executable.

Let me show you what I mean, with real tools, real code that runs, and a concrete path to wire it into OpenSpec.

The disagreement, stated precisely

Wasowski is right about one thing and wrong about its consequence.

He’s right that prose specs are non-deterministic under regeneration. This isn’t a quality problem you can prompt your way out of. A regenerated implementation is a fresh sample from the model’s distribution. Feed the same 1,500 words to two model versions — or the same model twice — and you draw two different points. The spec doesn’t constrain the output so much as bias it. That’s the whole game, and it’s why his test outlived his spec: a test is a fixed oracle, evaluated identically forever, while prose is re-interpreted on every generation.

But here’s where the amputation goes wrong. Tests have a failure mode that is exactly as fatal, and Dijkstra named it in 1969 at the NATO conference: testing shows the presence of bugs, not their absence. A test is an existential claim — “there exists at least one input where the code does the right thing.” It says nothing about the inputs you didn’t enumerate. Worse, in the LLM era there’s a second-order trap that the academic literature has started calling the cycle of self-deception: when you ask a model to write both the implementation and its tests, the tests inherit the same blind spots as the code. Green CI, shipped bug. The fox guarding the henhouse, and the fox wrote the henhouse’s threat model.

So we have two artifacts, each load-bearing, each broken in a complementary way:

  • Prose specs carry intent and universally-quantified meaning (”the total is never negative”), but they’re re-interpreted on every regeneration and a human has to read them to know if the code complied.

  • Example-based tests are stable and falsifiable, but they only cover the handful of inputs you thought of, and an LLM writing them tends to test the bugs it already shipped.

The synthesis is not to pick one. It’s to find the artifact that has the universality of a spec and the falsifiability of a test. That artifact already exists. It’s a property.

What a property actually is — and why it survives model upgrades

A property is a universally-quantified, executable, falsifiable claim about all inputs in a domain. “For every non-negative subtotal and every percentage, the resulting total is between zero and the subtotal, inclusive.” That sentence is simultaneously a spec requirement and a test you can run ten thousand times against randomly generated inputs.

This is not new. Koen Claessen and John Hughes shipped QuickCheck for Haskell in 2000, and the idea has propagated into every serious ecosystem: Hypothesis for Python, fast-check for TypeScript, proptest and quickcheck for Rust, jqwik for Java, ScalaCheck. The machinery has three parts worth naming because they map directly onto what we want from a spec:

  1. 1.

    Properties — the invariant, the universally-quantified claim. This is your spec requirement, expressed so a machine can check it.

  2. 2.

    Generators — strategies that produce random inputs across the domain. This is the part prose specs can’t do: actively probing the input space instead of describing it.

  3. 3.

    Shrinking — when a property fails, the framework automatically reduces the failing input to a minimal counterexample. This is the part that turns “it broke” into a fact you can act on.

The industrial track record is real, not academic decoration. Quviq’s QuickCheck team translated over 3,000 pages of AUTOSAR textual specification into executable models and surfaced more than 200 issues against shipping automotive software — at an estimated fraction of the effort of manual testing. Three thousand pages of prose spec, made falsifiable.

And here’s the thing that directly answers Wasowski’s anecdote. Why did his test survive four model generations while his spec drifted? Because a property is the fixed point of the system. The implementation is the variable — it gets resampled on every regeneration. The property holds, or fails, over the entire input domain regardless of which implementation the model happened to draw this week. When Opus 4.5 rewrites the endpoint, the round-trip property decode(encode(x)) == x doesn’t need reinterpretation. It re-runs. If the new implementation is wrong, the property fails and hands you a shrunk counterexample. If it’s right, green. The model changed; the fact didn’t.

There’s recent, specific research backing this for LLM-generated code. He et al.’s “Use Property-Based Testing to Bridge LLM Code Generation and Validation” (arXiv 2506.18315, June 2025) builds a “Property-Generated Solver” and reports relative pass@1 gains of 23–37% over standard TDD methods. Their key move is the one that breaks the cycle of self-deception: generate the properties first, independently, before generating the code. A property like “the output list is a permutation of the input” is independent of whatever sorting bug the model is about to write. The invariant doesn’t know the implementation’s mistakes, so it can’t share them.

What OpenSpec actually gives you — and the gap at its center

Now let’s get concrete about OpenSpec, because this is where the theory meets a tool I actually use.

OpenSpec (github.com/Fission-AI/OpenSpec) is the lightweight, brownfield-first member of the SDD family. Its core idea is a clean separation I genuinely like: openspec/specs/ holds the living source of truth — what the system does today — and openspec/changes/ holds proposed modifications as self-contained folders. Each change carries a proposal.md (why), a design.md (how), a tasks.md (the checklist), and delta specs that mark requirements as ## ADDED, ## MODIFIED, or ## REMOVED. You drive it with slash commands — /opsx:propose, /opsx:apply, /opsx:verify, /opsx:archive — and archiving folds the deltas into the living spec. The delta model is the brownfield differentiator: you’re not regenerating the world, you’re proposing a diff against a spec that already exists.

The spec format is already property-shaped, which is the whole reason this essay is possible. Requirements use RFC 2119 keywords and scenarios use Given/When/Then:

## ADDED Requirements

### Requirement: Discount Application
The system SHALL apply percentage discounts to an order subtotal
without ever producing a negative total.

#### Scenario: Discount never exceeds subtotal
- **GIVEN** an order with a non-negative subtotal
- **WHEN** any percentage discount is applied
- **THEN** the resulting total is between zero and the subtotal, inclusive

Read that THEN clause again. “The resulting total is between zero and the subtotal, inclusive.” That is a property. It is universally quantified (”any percentage discount”), it is falsifiable, and it is one short translation away from an executable check. The OpenSpec docs even say scenarios should be “testable — you could write an automated test for them.”

Here’s the gap, and it’s the entire reason Wasowski’s critique has teeth: OpenSpec never writes that test, and never runs it. /opsx:verify is not a test runner. It’s an LLM agent that reads your code and your spec and reasons, in prose, about whether they match. It will helpfully emit something like ⚠ Scenario 'discount never exceeds subtotal' has no test coverage — and then do nothing about it. It’s optional. It does not block /opsx:archive. So the living spec, the supposed source of truth, gets updated with no executable gate confirming the code ever satisfied it. The scenario was “testable in principle” and untested in fact. The prose said SHALL; nothing checked.

This isn’t me inventing a strawman. It’s the subject of live issues in the repo: #900 (a proposed test-generation step bridging SDD and TDD, labeled future-roadmap), #880 (a /opsx:validate command to check code against the living specs, also future-roadmap), and #433 (translate scenarios to Jest/Playwright and gate CI on them). The maintainers know. The future-roadmap label means “strategically important, not yet scoped.” This essay is my attempt to scope it.

The bridge, made of code that runs

Let me take that exact discount scenario and compile it down the rungs of falsifiability. I’ll use TypeScript with fast-check because that’s where most of you live, then the Rust proptest equivalent because that’s where I live.

First, the implementation — the kind of thing Claude Code generates from the spec:

function clampPercent(p) {
  if (Number.isNaN(p)) return 0;
  return Math.max(0, Math.min(100, p));
}

function applyDiscount(subtotalCents, percent) {
  const p = clampPercent(percent);
  const discount = Math.round(subtotalCents * (p / 100));
  return Math.max(0, subtotalCents - discount);
}

Now the scenario’s THEN clause, as a fast-check property. Note the generator deliberately includes out-of-range percentages (−50 to 200), because the spec says any percentage, and “any” is where the bugs live:

const fc = require('fast-check');
const { applyDiscount, clampPercent } = require('./pricing');

const cents   = fc.integer({ min: 0, max: 100_000_00 }); // up to $100k
const percent = fc.integer({ min: -50, max: 200 });        // includes out-of-range

// Scenario: "the total is between zero and the subtotal, inclusive"
fc.assert(fc.property(cents, percent, (sub, p) => {
  const t = applyDiscount(sub, p);
  return t >= 0 && t <= sub;
}));

That passes — I ran it. So does its sibling, a property that isn’t even in the prose but is clearly implied: a bigger discount must never yield a bigger total. This is the under-specification gap that PBT closes — the generator forces you to state invariants the prose author forgot:

// Monotonicity: a larger discount never produces a larger total
fc.assert(fc.property(cents, percent, percent, (sub, a, b) => {
  const lo = Math.min(clampPercent(a), clampPercent(b));
  const hi = Math.max(clampPercent(a), clampPercent(b));
  return applyDiscount(sub, hi) <= applyDiscount(sub, lo);
}));

Now the payoff. Suppose the model had generated a plausible-but-wrong implementation — one that forgot to clamp the percentage and forgot the max(0, …) floor. A handful of example-based tests would sail right past it. The property does not:

function applyDiscountBuggy(subtotalCents, percent) {
  const discount = Math.round(subtotalCents * (percent / 100)); // no clamp
  return subtotalCents - discount;                              // no floor
}

Running the bounds property against this produces, after shrinking:

Property failed after 3 tests
Counterexample: [17, 103]
Shrunk 17 time(s)

A subtotal of 17 cents and a 103% “discount” yields a total of −1 cent. That is the fact Wasowski is talking about. Not a prose warning that someone has to read and interpret. A minimal, reproducible counterexample — [17, 103] — that fails identically under Sonnet 3.5 and Opus 4.5, because the property doesn’t care which model wrote the code. fast-check even persists the seed so the failure is replayable in CI forever. The shrinking is the difference between “something’s wrong somewhere” and “here is the smallest input that breaks your SHALL.”

The Rust proptest version expresses the same scenario, and here the type system discharges part of the spec for free — u64 can’t be negative, so the lower bound is enforced by the compiler, which is the cheapest possible “fact”:

use proptest::prelude::*;

fn apply_discount(subtotal_cents: u64, percent: i64) -> u64 {
    let p = percent.clamp(0, 100) as u64;
    let discount = (subtotal_cents * p + 50) / 100; // round half up
    subtotal_cents.saturating_sub(discount)
}

proptest! {
    // THEN the resulting total is between zero and the subtotal, inclusive.
    // (>= 0 is guaranteed by the u64 type — a fact discharged at compile time.)
    #[test]
    fn total_within_bounds(sub in 0u64..=10_000_000, p in -50i64..=200) {
        prop_assert!(apply_discount(sub, p) <= sub);
    }

    // A larger discount never produces a larger total.
    #[test]
    fn monotonic_in_discount(
        sub in 0u64..=10_000_000,
        a in -50i64..=200,
        b in -50i64..=200,
    ) {
        let lo = a.clamp(0, 100).min(b.clamp(0, 100));
        let hi = a.clamp(0, 100).max(b.clamp(0, 100));
        prop_assert!(apply_discount(sub, hi) <= apply_discount(sub, lo));
    }
}

When proptest fails, it writes the minimal counterexample to a proptest-regressions/ file and commits it as a permanent, replayable seed. Your repository accumulates a ledger of every falsified SHALL it has ever seen. That ledger is exactly the stable, model-independent artifact Wasowski found by accident — except now it’s generated from the spec, on purpose.

What I’d actually build into OpenSpec

So here is the concrete proposal, in OpenSpec’s own vocabulary. None of this throws away prose. It makes prose cash out.

Make the property a first-class scenario element. Extend the scenario grammar so a THEN clause can carry an explicit, machine-parseable property block beside its prose. Something like:

#### Scenario: Discount never exceeds subtotal
- **GIVEN** an order with a non-negative subtotal
- **WHEN** any percentage discount is applied
- **THEN** the resulting total is between zero and the subtotal, inclusive

  #### Property: total_within_bounds
  - **forall** subtotal in 0..=10_000_000, percent in -50..=200
  - **invariant** 0 <= total && total <= subtotal

The prose stays for the human reviewer. The property block is what compiles. For the trigger and precondition clauses, lean on EARS-style constrained syntax (the structured requirements grammar Kiro already borrows from Rolls-Royce) so the GIVEN/WHEN parses deterministically into generators.

Ship a spec → property test compilation step, and generate the properties before the code. This is issue #900, done right. Add a tests/ artifact to the change folder. When you /opsx:apply, an agent reads the delta spec and emits property-test stubs in the project’s framework — fast-check, proptest, or Hypothesis — each tagged with its scenario ID for traceability. Critically, following He et al.: the agent generates the properties first, you review those, and only then does it generate the implementation against them. The property is the contract; the code is what gets sampled against it. This is the single move that breaks the self-deception cycle, because the invariant is authored before — and independently of — the code it judges.

Make /opsx:verify run the suite instead of reasoning about it. Today verify is an LLM prose comparison. Make it a hybrid: run the compiled property suite, capture pass/fail and any shrunk counterexamples, and feed the counterexamples back to the implementing agent for regeneration rather than handing a human a ⚠. The minimal counterexample [17, 103] is a far better prompt than “the implementation may not satisfy this scenario.” You’re closing the loop with a fact.

Gate /opsx:archive on green CI. This is the keystone. Right now a change folds into the living spec with no executable confirmation, which is precisely how a “source of truth” rots into fiction — half the archived requirements never verified against running code. Make the default policy: a change cannot be archived unless its scenario-linked property tests pass in CI. Now the living spec is a continuously-verified artifact. And — this is the part that should make Wasowski reconsider the amputation — regeneration under a new model becomes safe, because the property suite re-validates the new implementation automatically. Opus 5 rewrites your endpoint? The properties re-run. Green means the new sample still satisfies every SHALL. The spec didn’t need reinterpreting; it needed a CI gate.

For the hard cases, reach for stateful and metamorphic properties. Brownfield systems are full of sequential logic where a single invariant won’t do. Compile state-machine scenarios into model-based tests (proptest‘s state-machine support, fast-check‘s commands model). And for endpoints with no clean oracle — the ones where you can’t state the exact right answer without re-implementing the function — support metamorphic relations as a scenario type: idempotence, round-trip, paraphrase-invariance. The metamorphic-testing literature reports catching ~75% of erroneous GPT-4-generated programs this way, on functions where a conventional assert-equals oracle is impossible to write. This is where OpenSpec could genuinely leapfrog Spec Kit and Kiro, neither of which has an executable-invariant story at all.

The part where I disagree with the headline

Facts and specs are not enemies. The opposition is an artifact of looking at only two rungs of a ladder that has more rungs than that.

At the top is prose: intent, the thing a human reads to understand why. At the bottom is the type system: facts so cheap the compiler discharges them before you’ve finished typing (a u64 that can never be negative). In between sits the property — the rung that has the meaning of the prose and the falsifiability of the test. Wasowski found that a test outlived a spec and concluded the spec was the problem. I think he found that the property was doing the spec’s job better than the prose was, and that the prose had no way to become a property. That’s not a reason to delete the prose. It’s a reason to give it a compile target.

Popper, whom he quotes against specs, actually argues for the synthesis. Falsifiability isn’t a property of tests as opposed to specs — it’s a property of any well-formed claim. The fix for a non-falsifiable prose requirement is not to throw the requirement away. It’s to attach a falsifiable check to it. A spec that says SHALL and is never checked explains everything and therefore nothing. A spec that says SHALL and compiles to a property that fails loudly on [17, 103] is a theory that can be proven wrong — which, per Popper, is the only kind worth having.

So: keep writing specs. Just stop letting them stay prose. A spec is the name we give to a property. Make OpenSpec write the property down, generate it before the code, run it in verify, and gate archive on it staying green. Do that, and the next model upgrade stops being a CI emergency and starts being a non-event — because the facts your spec compiled to don’t change their minds when the model changes its.


Tooling referenced, all real: OpenSpec (github.com/Fission-AI/OpenSpec), fast-check, proptest, Hypothesis, QuickCheck. The fast-check examples above were run as written; the proptest example is written against proptest’s API but worth a cargo test before you trust it. Research threads worth pulling: He et al., “Use Property-Based Testing to Bridge LLM Code Generation and Validation” (arXiv 2506.18315); OpenSpec issues #900, #880, #433; and, of course, Wasowski’s piece that started the argument.

https://leanpub.com/clarityforge - practical guide to openspec

Originally published on Sovereign Agentic AI on Substack.