The Journey of Building PatLang, Continued: Teaching It to Write Itself

A direct continuation of The Journey of Building PatLang (Acts I-VI: the one-day MVP sprint through the self-hosting fixpoint through four "PatLang first" features) — split into its own page because this arc, on its own, ended up as long as everything before it combined. Acts VII through IX cover a genuinely different kind of work: not extending the language, but teaching it to write PatLang code from examples, then testing that new capability against a real, external, pre-existing project instead of another purpose-built demo. Same warts-included approach, same commit-history sourcing, same house style as the main page — see there for the full framing and for what these lessons draw on before this page picks up.

Act VII: teaching the language to write itself from examples

The most recent stretch layered something genuinely new on top of everything before it: an inductive logic programming (ILP) system, built entirely in PatLang, that reads BDD-style Given/When/Then scenarios and induces real, compilable PatLang rule clauses from them — see the full write-up for worked examples. What makes this act worth its own entry isn't the ILP theory (Metagol-style metarule search, bottom-up witness search, anti-unification by longest-common-prefix — all standard technique) but the shape of the five-milestone build that got there, and one late course-correction that's its own lesson.

Each milestone was scoped as its own working, tested increment — the same staging discipline from Act I, applied to a much more abstract problem: flat classification (group examples by output label), then a fixed recursive metarule (reproducing a hand-written dependency-graph rule from example queries), then arbitrary conjunctions with distractor rejection, then multi-hop relational chains (the classic ILP "grandparent" benchmark), then finally bottom-up evidence-driven induction that can diagnose why it failed, not just that it failed. Every milestone was proven against a real demo already in the codebase, not just synthetic data — reproducing the router DSL demo's routing table caught a real bug the toy data never could:

Found a real gotcha in the native A1 resolver: any argument string matching
^[A-Z] is treated as a Prolog logic VARIABLE, not a ground constant.
Request strings like "GET /users" start with an uppercase letter, so they
silently unified with EVERYTHING -- the compiled router always returned
the first category, no matter the input. The toy digit corpus ("1", "5",
"9") never hit this because digits never match ^[A-Z].

Lesson: a toy corpus and a real corpus test different things. The digit classifier proved the mechanism worked; only pointing the same mechanism at real data (HTTP verbs, which are conventionally uppercase) surfaced a genuine bug in a resolution engine that had already shipped and passed its own tests. If a system is meant to generalize, testing it on data drawn from an actual domain — not just data invented to be easy — belongs in the suite from early on, not as an afterthought.

One test design mistake is worth naming honestly, because it's a useful example of a test author's intuition being wrong in an instructive way. A "conflict" scenario meant to prove the engine correctly rejects an over-general rule used a negative example one relation-hop shorter than the positive examples, expecting the shorter chain to wrongly pass. It didn't — the anti-unification step correctly generalized to the shorter, genuinely shared chain length instead, which was the right answer, not a bug. The test had to be redesigned around a negative example with the exact same witnessed structure as the positives (standing in for a real-world distinction the given facts simply can't express) to produce a genuine conflict. Lesson: when a test's failure mode surprises you, check whether the system found a better answer than the one you assumed was correct before assuming the system is wrong.

The act closes with a small but deliberate constraint, applied after the fact: the whole eleven-suite selftest corpus had been wired into cargo test via the same cucumber harness Act VI's browser-layer bugs were caught by — appropriate there, since those bugs lived at a real Rust/WASM/DOM boundary, but this system has no such boundary; it's PatLang calling PatLang throughout, with the base interpreter as its only real dependency. The suite was pulled back out of cucumber and given a small PatLang-native runner instead (self_hosting/tools/run_synthesis_selftests.patlang) that reproduces the exact same check — process succeeded, stdout says "ALL TESTS PASSED" — using nothing but exec_capture and the base pat runtime. Lesson: a test harness should depend on exactly the boundary it's actually testing across, not the toolchain that happened to be convenient to wire up first. Act VI's four features genuinely needed a Rust/JS/WASM harness because their real bugs lived at that boundary; a system that never leaves PatLang doesn't, and carrying the heavier dependency anyway is a cost with no corresponding benefit.

The write-up's own honest "current limits" section became the next milestone's spec: the bottom-up witness search took the first matching background fact at every relation hop, so a fact graph with genuine branching — one parent with two children, say — only ever explored whichever child's fact happened to be declared first. The fix (explore every witnessed branch, anti-unify across each example's full set of witnessed chains rather than a single greedily-chosen one) came with its own new failure mode worth naming as a diagnosis in its own right, not lumped into an existing one: no_common_structure, for when every example has real evidence but that evidence never agrees on a shape — distinct from a missing fact (no_witness) or a hypothesis that mismatches one example (conflict). The regression test built to prove the fix does something is itself worth noting: a family tree where the dead-end branch is declared before the real one, specifically so a first-match-only search would have committed to the wrong branch and failed the test — proof the fix isn't just plausible-looking code, but code that changes the answer on a case constructed to require it. Lesson: a published list of honest limitations isn't just a disclaimer — treated as a live backlog, it's a ready-made source of well-scoped next milestones, each with its own natural regression test already implied by the limitation's own description.

The next item on that same limitations list turned out to be the deepest change in the whole arc: replacing a fixed library of separately hard-coded clause-shape strategies — plain conjunctions, relation chains, each blind to the other's existence — with one genuinely general representation both could go through. The tell that the two were really separate all along was a rule neither could express alone: "a grandparent whose grandchild is young" needs a relation chain and a condition on where it ends, and milestone 3 (conjunctions only) and milestones 4-6 (chains only) each covered exactly one half. The fix generalized the clause body into a real literal list and replaced "keep the common prefix of a predicate-name sequence" with actual anti-unification: at every node visited along a witnessed chain, not just its endpoint, record every fact true there, then intersect that evidence across examples the same way. A predicate survives into the induced rule only if it held at the same position for every example merged — proven not just at the chain's endpoint but at an intermediate node too, where a distractor fact on one example's midpoint got correctly dropped for not appearing at the other's. Both earlier, narrower cases still worked identically once run through the general engine — nothing was lost collapsing two special cases into one general one, which is usually the actual test of whether a generalization was a genuine simplification or just a rewrite. Lesson: when a system has grown two or more separately-implemented special cases that could never combine, look for the rule that neither one can express alone — it's usually the sharpest test of whether they were ever truly separate problems, or one problem that hadn't been named yet.

The third item on that limitations list was the one case where the honest answer wasn't "generalize the representation further" — it was "some concepts genuinely shouldn't be generalized toward each other at all." A predicate true for two structurally unrelated reasons (eligible as a veteran, or as an enrolled student, sharing no evidence whatsoever) has no single anti-unified structure to converge on, and forcing one anyway would silently invent a shared condition that doesn't exist. The fix didn't touch the anti-unification engine at all — it wrapped it with a clustering step: group examples whose evidence does share structure, open a new group only when one can't join an existing one, then emit one clause per group. The insight that made this cheap rather than a second resolution engine: multiple clauses for the same predicate head are already ordinary logical disjunction in the underlying resolver — the entire feature is "emit N clauses instead of always exactly 1," nothing more exotic. The regression test worth naming here inverted the usual shape: rather than checking the new capability fires, it checked that examples which do share real structure still collapse into one clause, proving the fix only splits when a split is actually needed rather than fragmenting every induction into unnecessary pieces. Lesson: when a generalization mechanism keeps failing on a specific category of case, check whether the fix is "generalize harder" or "stop trying to generalize these together at all" — treating every failure as evidence the representation needs to grow can miss the simpler fix of admitting some things are genuinely disjoint.

The fourth and final item asked to be tackled specifically alongside a second, separate one — the user's own hunch, stated as a question rather than an instruction: "search-cost scaling and clustering are related, can we aim for a better clustering at the same time?" The literal scaling problem was real: every candidate hypothesis needed its own subprocess, because the resolver has no way to un-register a rule once tried. But the fix wasn't to give it one — it was noticing that several different hypotheses could safely share one process anyway, as long as each used its own uniquely-named head predicate, since resolution only ever follows rules for the predicate actually asked about. That single trick — batch several candidates into one script, one subprocess spawn instead of many — is what turned the previous item's greedy, order-dependent clustering into something worth improving at all: a second, order-insensitive clustering strategy was cheap to add only because comparing it against the original no longer cost a second process spawn. Proving the comparison actually did something required resisting the temptation to hand-pick a favorable example — an actual probing run against real code turned up a domain where the new order-insensitive strategy did worse (four clauses against the original's two), which became the regression test: not "does the new strategy win," but "does the system correctly keep the better of the two regardless of which one that turns out to be." Lesson: when a user connects two problems that look separate, take the connection seriously before assuming it's coincidental — here, the thing that made the second problem worth solving well was solving the first problem first. And when testing a "pick the better of two" mechanism, don't test it on a case you already know favors your new option — go find, empirically, a case where the new option loses, and confirm the mechanism still makes the right call.

Act VIII: throwing it at a real project instead of another toy

Every milestone in Act VII, including the four "current limits" that got closed one by one, was proven against either a purpose-built toy corpus or a real demo that already lived inside PatLang's own codebase — a fair test of the mechanism, but not a fair test of what happens when the input is genuinely someone else's mess. The next step was explicit about that gap: "I suspect we need to throw it at some much bigger problems, and will find some bugs crawl out of the woodwork as we do so." The chosen target was an entirely separate, pre-existing project — a Ruby fantasy-population simulator with no meaningful test suite of its own, so its source code was the only honest record of what it actually did, not what it was supposed to do.

The first pass wasn't code at all — it was reverse-engineering. A research pass over the Ruby surfaced real, concrete bugs: two different relatedness calculations in the same codebase that disagree with each other (one correctly special-cases siblings, the other doesn't and reports them as "0th cousin"); a self-as-sibling bug traced to a hash being subtracted from an array of integer IDs, a type mismatch that silently does nothing; a marriage routine that randomly samples one candidate but proposes to a different one. None of this was guessed at — it came from reading the actual code, the same discipline the "toy vs real corpus" lesson from earlier in this history already argued for, just applied to someone else's project this time instead of PatLang's own router demo.

Then came the part that mattered most: writing BDD requirements for the *corrected*, intended behavior and handing them to the induction engine — and finding, before a single line of new engine code was written, that the engine structurally could not express two of the three things the domain actually needed. Not "hasn't been tried yet" — genuinely couldn't, by construction. A family-tree sibling relationship needs two people to each independently trace a path back to a shared ancestor; every chain-search function built across four milestones only ever walked outward from one starting point. Digging one level further revealed an even more basic gap underneath that one: no chain-based relation exposed its endpoint as a real second argument at all — every induced chain could only prove "X reaches *something*," never bind what that something actually was. The "easy" case and the "hard" case turned out to share the same root cause, which only became visible by trying to build the hard case and watching the easy case fail too.

Rather than force a bad induction attempt at a shape the engine couldn't hold, the honest move was to scope down to what genuinely was inducible (an existential "has *some* grandchild" predicate, a standalone eligibility check with no cross-person comparison), ship that, and write the blocked capabilities up as named, numbered gaps — the same "limitations list as a live backlog" habit from earlier in Act VII, just applied one level deeper: two new milestones (binding a chain's endpoint as a real second argument; searching outward from two points at once to find where they meet) closed two of the three gaps on the first real attempt at each, reusing the machinery each earlier fix had already built rather than starting over. Lesson: when a domain feels obviously too hard for a system to express, don't assume the system just needs to try harder — see whether the domain is quietly asking for a structural capability that was never actually built, and whether the "hard" version of that capability and an "easy" version you thought already worked share the same missing piece.

One gap was deliberately left open, and demonstrated rather than hidden: the newly-induced sibling relation has no way to say "these must be two different people," because the engine still has no equality or inequality comparison between two entities at all. The regression test doesn't pretend this away — it explicitly queries the induced rule with the same individual for both arguments and asserts that it wrongly succeeds, a passing test whose entire purpose is to prove a known limitation is real and unaddressed, not to prove the code works. That's a different use of a test than every other one in this history: not "does this do what it should," but "does this honestly fail exactly where we said it would."

A correction arrived mid-session, from the user, not from the code, and reshaped the very domain that had just been modeled: marriage across social classes and between people of the same sex were treated as hard exclusions in the first pass, and both are actually meant to be *possible, just less common* — a probability, not a rule. That distinction matters structurally: a single example can confirm or refute a boolean rule, but it says nothing about a probability. The fix was a small, separate piece of testing infrastructure — run an implementation many times, and check that the observed success rate falls inside a statistically-sized tolerance band around the declared probability, rather than asserting one outcome. Kept deliberately apart from the induction engine itself: this doesn't derive a probability from data, it verifies that a human-declared probability and an implementation's actual behavior agree.

The chapter closed with a small but telling fix, prompted by simply trying to use the finished engine the way an outside project actually would: every hypothesis-testing function called its own interpreter via a path relative to the current working directory, so the engine only worked correctly by accident of which directory a script happened to be run from. It had never been run from anywhere else before, so the bug had never had a chance to matter. Lesson: a capability that has only ever been exercised from inside its own project hasn't really been tested as a capability *other* projects can use — the moment something is meant to be a reusable engine rather than a self-contained demo, actually calling it from somewhere else, even once, is worth doing before assuming it's ready.

Act IX: fantpop-patlang becomes a real target, not just a stress test

The fantpop-patlang stress test earned its keep so thoroughly it stopped being a stress test and became a real, ongoing application in its own right — the user's actual goal all along, it turned out: simulate a population across potentially thousands of generations, track full family trees and life-event history, and eventually let a caller query the living population for a character to play, complete with real family history. Development continued the same way the engine itself was built: small, single BDD-scoped milestones, checked in on before moving to the next one.

The first milestone — a per-individual life-event ledger, recording birth/death/marriage/migration — was deliberately hand-authored rather than induced, and said so plainly in its own BDD file: "at most one birth event" is a counting constraint, "chronological order" is a numeric-comparison constraint, and the induction engine's equality-only unification can express neither. Not every capability needs to come from the new machinery; naming which ones don't, and why, is as much a part of the discipline as building the machinery itself.

A small suggestion arrived mid-session that turned into the most productive detour of the whole arc: swap the event ledger's storage from an in-memory list to the real SQL engine built in an earlier session, since the actual target ("hundreds or thousands of generations") needs data that survives past one process. The immediate payoff was one already-established habit paying for itself again: the existing BDD acceptance suite, unchanged, became the regression test for the storage swap — and it failed on the first run. The cause: this SQL engine's tables are backed by a persistent virtual filesystem, not scoped to the handle a caller passes around, so "start fresh" doesn't actually clear anything unless you know to ask it to. A one-line fix, found only because a real test was pointed at real (if still small) behavior instead of being trusted to just work.

Testing "at scale" for the first time — something the user flagged as genuinely untested territory before it was tried — surfaced something much bigger: a database whose INSERT silently rewrote its ENTIRE table on every single row added. Invisible at demo scale (a handful of rows), catastrophic at real scale (a planned few-thousand-row test simply never finished). The underlying cause traced all the way down to the virtual filesystem itself, which had a way to replace a file's whole contents but no way to add to it — so a proper fix meant adding a genuine new primitive to the runtime, not just patching the SQL layer on top of what it already had, then routing the common case through it. Confirmed by direct before-and-after timing rather than assumed: a batch of inserts that had taken seconds before completed in a fraction of that afterward, with the growth curve visibly flattening from a curve to a line. A second, structurally identical bug turned up one layer up, in the newly-SQL-backed application code itself — a duplicate-check query run before every insert, silently re-introducing the same quadratic cost the database-level fix had just removed, fixed with a lightweight index kept outside the database entirely rather than by trying to add real indexing to the database in the same pass. Lesson: a performance bug that's invisible at demo scale and catastrophic at real scale is not rare — it's the default outcome of never having tested at real scale, and the fix for one instance of "no indexing" doesn't mean every instance of the same root cause has been found; the same shape of bug reappeared one layer up in code written specifically to fix the first one.

Not yet started, named openly rather than left implicit: whether the next milestone (a genuinely threaded, multi-generation simulation loop) can lean on the induction engine at all is an open question, not a settled one — the user pushed back directly on an assumption that it couldn't, pointing out that BDD requirements about thread safety and result collation might make it inducible after all. Worth taking seriously when that milestone actually starts, rather than resolving by assumption in either direction ahead of time.

Lessons from this arc, the short version

Test a real corpus, not just a convenient one. Toy data proves a mechanism works in principle; only real-shaped data (with real-shaped quirks, like uppercase HTTP verbs) reliably surfaces the bugs that matter.
A test harness should depend on exactly the boundary it's actually testing across. A heavier toolchain wired up because it was convenient for an earlier, different feature is a cost worth removing once you notice it's no longer buying you anything for the feature at hand.
A published list of honest limitations is a backlog, not just a disclaimer. Each one usually implies its own regression test already — build the test around the specific case that would have failed under the old behaviour, so the fix is provably a fix, not just plausible-looking code.
Two special cases that can never combine are a sign a more general rule hasn't been found yet. Look for the case neither one can express alone — collapsing them into one system that still handles both original cases identically is the real test of whether the generalization was genuine.
Not every case that resists generalization needs a more powerful generalizer. Sometimes the honest fix is admitting two things are genuinely unrelated and handling them as separate cases (or, here, separate clauses) rather than stretching one mechanism to cover both.
Take a user's hunch that two problems are related seriously before assuming it's coincidental. Solving the cheaper one first sometimes turns out to be exactly what makes the harder one affordable to attempt at all.
When testing "pick the better of two options," don't test on a case that already favors the new one. Go find, empirically, a case where the new option actually loses, and confirm the mechanism still chooses correctly.
When a domain feels too hard for a system to express, look for a missing structural capability before assuming the system just needs to try harder. An "easy" case that quietly fails alongside a "hard" one is often a sign both share the same actual gap.
Some tests should prove a limitation is real, not that the code works. A test that deliberately demonstrates a known gap, rather than hiding it, is more honest than silently working around it.
A capability only ever exercised from inside its own project hasn't really been tested as reusable. Try calling it from somewhere else, even once, before assuming it's ready for that.
Not every needed capability has to come from your newest, fanciest machinery. Naming which parts of a problem a given tool genuinely can't handle, plainly, is as much a discipline as building the tool.
A performance bug invisible at demo scale and catastrophic at real scale is the default outcome of never testing at real scale, not a rare surprise. Test the size you actually need, not the size that's convenient to type out.
Fixing one instance of a root cause doesn't mean every instance is fixed. The same shape of bug can reappear one layer up, in code written specifically to work around the first one.