What Building a Code-Synthesis Engine Taught About Any Software Project

A companion to Project Navigation & Triangulation, The Report as the Front-End, and What Four Real Features Taught About Navigating a Build, grounded in Acts VII-IX of the PatLang journey: an eleven-milestone inductive-synthesis system, each milestone scoped, built, and tested as its own increment before the next began — the last several found not by planning ahead, but by pointing the finished system at a real, external, pre-existing project and watching where it broke. None of what follows depends on the subject matter being a programming-language feature — the same shape of lesson applies to any project built as a sequence of genuinely separable steps.

Abstract, unfamiliar work stages the same way concrete, familiar work does

Inductive logic programming has no obvious "arithmetic calculator" starting point the way a new language does — there's no small, self-evidently-correct first slice to build outward from. The actual first milestone (group examples by output label into one branch per real category) wasn't the easiest technically interesting idea available; it was the smallest thing that could be built, tested, and proven correct in isolation before anything harder was attempted. Each of the six milestones after it added exactly one new capability to a working system, never two at once, and each was proven against a real target before the next began. The lesson generalizes past any one domain: when a problem feels too abstract or unfamiliar to know where to start staging it, the answer isn't to design the whole thing up front — it's to find the smallest genuinely separable capability, however unglamorous, and build only that first.

Toy data proves the mechanism works; only real data proves the system does

A synthetic test corpus (single digits, deliberately easy) passed cleanly through the first milestone's mechanism and genuinely proved the underlying idea was sound. It took pointing the exact same code at real, already-existing data — reproducing an actual demo's routing table — to surface a real bug: values starting with an uppercase letter were silently misinterpreted by the underlying engine as something other than what they were, because uppercase HTTP verbs are common and single digits never are. The toy corpus wasn't wrong to build first; it was the wrong place to stop. Any system meant to generalize needs both a synthetic case that proves the mechanism in isolation and a real-shaped case with the domain's actual quirks, and the two catch genuinely different classes of bug.

When a test's failure surprises you, check whether the system found a better answer before assuming it's wrong

A test built to prove the system correctly rejects an over-general rule used a negative example that was simply shorter than the positive examples, expecting the shorter case to wrongly pass. It didn't — the system generalized to the shorter, genuinely shared structure instead, which was the actually-correct answer, not a bug. The test had to be redesigned around a case that shared the exact same structure as the positives but was still meant to be excluded, which is a different and harder thing to construct. The instinct to treat a test's failure as proof the code is wrong is usually right, but not always — sometimes the test's own assumption about what "correct" looks like was the thing that needed correcting.

A published list of honest limitations is a backlog, not a disclaimer

A written, public account of what a system doesn't yet do turned directly into the next several milestones' specifications, each addressed by name, in priority order chosen by simply asking which one mattered most. Each limitation's own description usually implies its own proof-of-fix test almost for free — "the search only explores the first matching option" directly implies "build a case where the first option is wrong and the second is right, and confirm the fix picks the second." Treating an honestly-written limitations section as a live backlog, rather than a permanent disclaimer, turns documentation debt directly into a scoped work plan.

A test harness should depend on exactly the boundary it's actually testing across

An earlier, unrelated part of the same project genuinely needed a heavyweight test harness, because its real bugs lived at a genuine boundary between two different execution environments. When a later, self-contained system was wired into that same harness by default — because it was already there and convenient — the extra dependency bought nothing: the new system never crossed the boundary the harness existed to test. Removing it was a small, deliberate correction once noticed, not a redesign. The general form: inheriting a project's existing tooling by default is often right, but it's worth periodically asking whether a given piece of infrastructure is testing something the code in front of you actually does, or just something convenient that happened to be lying around.

Not every case that resists generalization needs a more powerful generalizer

A mechanism built to find shared structure across examples kept failing on a specific category of case: examples that were both individually valid but had genuinely nothing in common. The instinct each earlier fix had reinforced was "the representation needs to grow to cover this" — and for several rounds, that instinct was right. Here it wasn't: the honest answer was that these particular cases shouldn't be forced toward each other at all, and the fix was a lightweight grouping step in front of the existing mechanism, not a more powerful version of the mechanism itself. Worth noting how cheap the actual fix turned out to be once correctly diagnosed — the underlying system already had a native way to express "true for either of these reasons," so the whole feature reduced to routing examples into the right groups rather than building new logic to combine them. The general trap: when a tool keeps failing on one category of input, the default assumption is often "make the tool smarter," when the actual answer is sometimes "stop asking the tool to unify things that were never meant to unify."

Asked whether an expensive-search problem and a not-quite-optimal-clustering problem were secretly related, the honest answer turned out to be yes, but not in the obvious direction: the search-cost fix (batch several candidate hypotheses into one process instead of spawning one per candidate) wasn't valuable mainly for its own sake — it was what made a second improvement affordable at all. Comparing two clustering strategies and keeping the better one only became cheap once comparing them stopped costing an extra process spawn. Solving the cheaper-sounding problem first is sometimes the actual precondition for the harder one being worth attempting.

The regression test built to prove the "keep the better of two options" mechanism worked is worth naming specifically: rather than construct a case designed to make the new strategy look good, an actual probing run against the real code was used to go looking for a case where it didn't — and found one, honestly, where the new strategy did worse than the original. That became the test: not "does the new option ever win," but "does the system correctly recognize when it loses and keep the original instead." A comparison mechanism tested only on cases that favor the new option hasn't been tested at all.

A real, untested external project finds bugs a self-written toy corpus never will

Every capability so far had been proven against either a purpose-built toy example or a real demo that already lived inside the same codebase — a fair test of whether a mechanism worked, but not a fair test of what happens when the input is someone else's genuine mess. Pointing the finished system at an entirely separate, pre-existing, essentially untested project (a Ruby simulator whose only real specification was its own source code) surfaced real bugs in that project within the first research pass — and, more importantly, surfaced real, previously-invisible gaps in the tool doing the reverse-engineering. Reading someone else's actual, un-idealized code is a fundamentally different exercise than exercising a system against inputs designed to demonstrate it working.

A domain that seems too hard may be revealing that an "easy" case was never actually solved either

The first attempt to express a real domain's requirement (two people sharing a common ancestor) failed outright — the system had no way to represent it at all, not a wrong answer, a structurally absent capability. Investigating further, to scope a smaller, supposedly-easier version of the same idea (one person tracing a path to a specific other person, not two people meeting in the middle), turned up an even more basic gap underneath: that "easy" case had never actually been solved either, just never been tried in a way that would have exposed it. The apparently-hard problem and the apparently-easy problem shared one missing piece, not two separate ones. Worth remembering: a request that seems to need a big new capability sometimes reveals that a smaller capability everyone assumed already existed, in fact, doesn't.

Some tests should prove a limitation is real, not that the code works

Closing two of three newly-found gaps left one deliberately open — the new capability had no way to express "these two things must be different," because nothing in the system could compare two entities against each other at all. Rather than silently avoid exercising that weak spot, the regression test does the opposite: it deliberately queries the new capability in exactly the way that would expose the gap, and asserts that it fails in the expected way. A test whose entire purpose is "prove this doesn't work yet, on purpose" is an unusual thing to write, but it's more honest than a test suite that simply never goes near what it can't yet do — the next person (or the next session) reading the code sees the boundary stated as a fact, not left to be rediscovered the hard way.

A capability that has only ever been called from inside its own project hasn't really been tested as reusable

A piece of infrastructure had quietly always shelled out to its own interpreter using a path relative to whatever directory happened to be current — invisible for as long as it was only ever run from inside its own project, where that directory was always correct by habit. The first time an entirely separate project tried to call it from its own directory, it broke immediately. Nothing about the code had changed; what changed was that the capability was finally exercised the way it was actually meant to be used. If something is meant to be reusable by other projects rather than a self-contained demo, calling it from somewhere else — even once, even before anyone asks — is a cheap way to find this class of bug before a real user does.

Lessons, generalized

When a problem is too abstract to know where to start staging it, find the smallest genuinely separable capability and build only that first. The easiest-to-build slice and the most-interesting slice are rarely the same one — build the easiest.
Toy test data proves a mechanism works in principle; only real-shaped data reliably proves the system works in practice. Budget for both, and don't treat the toy case passing as the finish line.
A surprising test failure sometimes means the test's assumption was wrong, not the code. Before rewriting the code to match the test, check whether the system actually found a better answer.
An honestly-published list of limitations is a ready-made backlog. Each limitation usually implies its own regression test almost for free — build the case that would have failed under the old behaviour.
Match your test harness to the boundary you're actually testing. Inherited tooling that was right for an earlier feature isn't automatically right for the next one — periodically ask whether it's still buying you anything.
A tool that keeps failing on one category of case doesn't always need to get smarter. Sometimes the fix is recognizing those cases were never meant to be handled by the same mechanism, and routing them separately instead.
Take a suggestion that two problems are related seriously before assuming coincidence. Solving the cheaper one first can be exactly what makes the harder one worth attempting.
Test a "pick the better option" mechanism on a case where the new option actually loses. A comparison only tested on cases favoring the new choice hasn't really been tested.
Test a system against a real, external, untested project, not just inputs designed to show it working. Reading someone else's un-idealized code surfaces both their bugs and your own tool's blind spots at once.
A domain that seems too hard for a system may reveal that an "easy" adjacent case was never actually solved either. Look for the single missing piece both the hard and easy versions of a problem might share.
Some tests should deliberately prove a limitation is real, not that the code works. Stating a boundary as a passing, explicit test is more honest than a suite that quietly avoids it.
A capability only ever called from inside its own project hasn't really been tested as reusable. Call it from somewhere else, even once, before assuming it's ready for that.