Risk Management: Lessons from Testing Real Systems

The Risk Management & Mitigation page lays out the standard framework — taxonomy, likelihood/impact scoring, mitigation strategies. This page is a companion, not a replacement: it walks through what that framework actually looks like applied to real work, grounded in four systems built and tested in a single stretch of development (a durable message queue, a live REPL, a click-to-play simulation, and a transactional database — all documented in The Journey of Building PatLang, Act VI). Every example below is a real bug that was actually found, not a hypothetical.

The core idea this page walks through

A passing test suite tells you the system behaves correctly in every state your tests actually put it into — not in every state it can actually be in. Risk management, applied to testing, is mostly the discipline of asking: what states haven't I actually visited yet, and which of them matter? That question turns out to have a small number of recurring, learnable answers, which is what the rest of this page walks through: what to test (layers, each catching a different class of risk), when to test (timing changes what a test can find), how to test (technique), and how to think about threat surfaces (which failures you can afford to let happen, and which you actively have to fence off).

Vocabulary: error, fault, and failure aren't the same thing

Standard software engineering usage distinguishes three separate things that everyday speech collapses into one word — "bug":

Error (or mistake): the human act that goes wrong — a wrong assumption, a misread signature, editing the function that looked right instead of the one that was right.
Fault (or defect, or flaw): the incorrect thing the error leaves behind in the artifact itself — the wrong line of code, the wrong design decision, sitting there whether or not anything has triggered it yet.
Failure: the observable, incorrect behaviour that happens when a fault is actually triggered at runtime — the thing a test (or a user) actually sees go wrong.

The gap between fault and failure is exactly where a lot of risk lives. A fault can sit in a codebase for a long time without producing a single failure, simply because nothing has yet exercised the specific condition that triggers it — which is the whole reason "no failures reported yet" is such a weak signal of "no faults present." The SQL database's rollback problem is a clean example of all three, kept separate: the error was writing a serializer that treated a table's currently-staged content as interchangeable with its committed content; the fault was the one line of code that read the wrong data into the wire format under an ambiguous marker; the failure — the actual wrong behaviour anyone could observe — was ROLLBACK not undoing an insert, and it only appeared once something exercised the specific condition the fault depended on (serializing mid-transaction), not the moment the fault was written.

What to test: layers, each catching a different risk

Five layers of verification were used across this work, and each one caught faults the others structurally could not — including one layer that catches errors before they ever become a fault at all:

Layer	What it actually proves	What it cannot see
Static review — reading the real source before writing code that depends on it	An assumption is wrong before it ever becomes a fault (see the dedicated section below) — this is the one layer that can prevent a fault from being written at all, rather than detect a failure after the fact	Its own coverage. Nobody tracks what was and wasn't actually read carefully, so a gap in review is invisible until something downstream fails
Unit / engine tests — calling internal functions directly	The core logic is correct in isolation (e.g. a database engine's transaction staging, tested by calling `rdb_begin`/`rdb_commit`/`rdb_rollback` directly, no parser involved)	Whether the real interface on top of that logic (a parser, a protocol, a UI) actually calls it correctly
Integration tests through the real interface — issuing real input, not calling functions directly	The interface and the engine agree (a SQL test suite that types `"INSERT INTO people VALUES (1, 'alice')"` as text, not `rdb_save(...)` as a function call, catches parser/executor mismatches the engine tests never could)	Whether the system behaves the same way across every environment it will actually run in
Cross-environment parity — the same input, run through every real execution path	The system doesn't quietly diverge between an interpreter, a compiled binary, and a self-hosted/alternate compiler — a divergence class that is invisible if you only ever test one path	Whether the artifact behaves correctly when driven the way its real, deployed caller actually drives it
Functional testing against the real, deployed artifact — driving the actual compiled/shipped output, with a harness that simulates real input	The boundary between two real systems (a browser's DOM and a compiled WebAssembly module, in this case) behaves correctly under something close to real use	Whether a human can actually understand and use what's correct underneath — see the last layer below
Real human use — someone actually using the finished thing	Usability, legibility, "does this feel right" — an entirely different risk dimension from correctness	—

That first row deserves its own treatment, not just a table cell — it's the layer most often left out of a testing discussion entirely, which is itself a risk.

Static review is testing too, and it's the one nobody logs

A meaningful amount of what actually prevented faults in this work wasn't a test at all — it was reading real source code before writing anything that depended on assumptions about it. Before calling a disk-flush function, its actual Rust implementation was read line by line, which is how a wrong assumption about its argument count was caught and corrected before a single line of code that called it wrongly was ever compiled — no test ever needed to fail, because the error never became a fault in the first place. The same pattern repeated: reading a parser's actual error-node handling before designing a console's error-recovery logic around it; reading an existing string-rendering function's already-implemented error case before deciding it was safe to reuse rather than duplicate. Static review — reading, not running — is a genuine testing activity (traditionally called static testing, distinct from dynamic testing, which requires execution), and in every one of these cases it was cheaper than the dynamic test that would otherwise have had to discover the same problem at runtime.

But it has a real weakness, and it's a process risk, not a technical one: a dynamic test suite leaves a permanent, growing, countable record behind it — "29 checks, up from 26" is a fact anyone can see in a commit history forever. A code-reading session that catches a wrong assumption leaves nothing, unless someone deliberately writes down that it happened and what it found. Across this work, that review activity was real and substantial, but it shows up (if at all) as a sentence buried in a commit message or a code comment, never as a number anyone can point to the way a test count can be pointed to. That asymmetry is itself worth naming as a risk: work that doesn't produce a visible, growing artifact tends to be under-valued and under-repeated relative to work that does, even when it's doing an equal or greater share of actually preventing faults — and once nobody remembers what was checked and why it mattered, the exact same wrong assumption can get made again later by someone (or something) with no record to consult. Logging static review findings as deliberately as dynamic test results — a commit message that says "confirmed via reading the actual implementation, not assumed" rather than just "fixed", a code comment explaining why a design choice is safe rather than just what it does — is a cheap, durable mitigation for a risk that's otherwise invisible until it repeats itself.

The sharpest example of the third and fourth rows is the SQL database's transaction bug. Twenty-six integration checks, issuing real SQL text end to end, passed cleanly on the first run — every transaction test began BEGIN, ran some statements, and ended COMMIT or ROLLBACK, all inside one continuous, uninterrupted program call. That's how a CLI test naturally gets written. But the browser makes a genuinely fresh call for every single statement a user types — so the database's whole state has to round-trip through a serialized text blob between every keystroke. The serializer had conflated "this table's committed content" with "this table's currently-staged, mid-transaction content" under the same marker; invisible to every integration test, because none of them ever serialized and deserialized the database while a transaction was still open. A functional-testing harness that simulated the browser's actual per-statement call pattern — not a human, but a script driving the real compiled artifact the same way a human eventually would — found it in minutes. The fix was closed for good by writing a new integration test that deliberately does the thing no earlier test had done: serialize, then deserialize, mid-transaction.

The fifth row — real human use — caught something no amount of correctness testing could have: a hex-grid gather-and-build simulation, proven correct by both an engine test suite and a click-simulation harness driving the real compiled game, was reported back as "I'm not sure the trainer ever really got built and I can't tell if I still had ore in stock." Nothing was incorrect. The system was completely correct and completely illegible, because no test — however thorough — had ever asked whether a person could tell what was happening. The fix wasn't a bug fix; it was adding a feedback channel (an event log, a prominent resource counter) that no assertion had any reason to demand, because "can a human follow this" isn't a property any of the earlier layers were built to check.

When to test: timing changes what a test can find

Test the core before the layers on top of it. The database's storage/transaction engine got its own dedicated test suite, calling engine functions directly, before the SQL parser was even written. When something breaks later, you already know the engine works — the search space for the bug is smaller by construction.
Test the specific new boundary a new capability introduces, not just its happy path. Adding transactions to the database didn't just need "does BEGIN/COMMIT/ROLLBACK work" — it needed a test of the exact new state that transactions specifically introduce: what happens if the system is serialized and reconstructed while one is open. That's not a generic test; it's the one test that only this new feature makes meaningful.
Get real usage as its own, later, distinct testing phase. Correctness testing and usability testing answer different questions and neither substitutes for the other. Budget for both, and expect the second one to happen after the first one has already passed, not instead of it.

How to test: technique

Reproduce minimally before fixing. A failure caused by reassigning an immutable variable was isolated to a three-line repro before anyone tried to explain it — the smaller the reproduction, the less there is to misdiagnose, and the less likely you are to "fix" a fault that wasn't actually the one causing the failure in front of you.
Trace state step by step when the failure is "eventually wrong," not "immediately wrong." A game unit that should loop between a resource tile and its base forever instead went idle after one trip — the failure only appeared several steps after the triggering condition. A tick-by-tick print of its own state (position, order, cargo) across thirty ticks showed exactly the tick where behaviour diverged from expectation, which traced back to the actual fault: a design error where one field was being used to mean two different things, invisible from reading the code casually.
Build disposable scratch test harnesses freely, and delete them. A Node.js script that fakes just enough of a browser (a canvas, a DOM element, an event-handler registry) to drive real compiled WebAssembly is not production code and doesn't need to be kept — it needs to exist for the ten minutes it takes to answer one question, then go away.
Verify the fix, not the disappearance of the symptom. The first attempt to fix the REPL's crash-persistence failure was itself a fresh error: it edited a plausible-looking function — a near-duplicate of the real one, sharing most of its code, sitting in the same file — leaving the actual fault completely untouched. It compiled. It looked right. It was the wrong function. The mistake was only caught by re-running the exact original failing scenario against the actual rebuilt artifact and seeing the failure was still there — not by re-reading the diff and being satisfied it looked correct. A "fix" is only verified once the specific failure that motivated it is confirmed gone, not once the code that produced it looks different.

Threat surfaces: deciding what's allowed to fail, and how far

A threat surface isn't only about malicious actors — it's the full set of ways a system's state can go somewhere you didn't intend, whether the cause is an attacker, a typo, or an honest bug. Every system built this session made an explicit decision about which failures were allowed to happen, and how far their consequences were allowed to spread:

Decision	Mitigation strategy (see the framework page)	Concrete mechanism
Which host-level capabilities a program is even allowed to call	Avoid — the capability simply isn't reachable	An explicit allowlist the compiler checks before it will generate a call to any given host function at all; a new capability that isn't on the list can't be lowered into a program, full stop
Where a sandboxed virtual filesystem is allowed to write real files on disk	Reduce (impact) — the failure mode (writing somewhere unintended) is fenced to a narrow, explicit boundary	A native-only "flush to disk" operation that refuses to write outside an explicitly configured root directory, checked by resolving the real, canonical path and rejecting anything outside it — not by trusting the caller's own path string
Which errors are allowed to be survived versus which stay fatal	A genuine Accept/Reduce split, made deliberately rather than by default	A user's typo in an interactive console is caught and reported as ordinary data, not a crash; a genuine internal contract violation is left fatal, on the reasoning that silently continuing past a broken invariant is worse than stopping
Whether concurrent writers are supported at all	Accept — named and documented, not hidden	A database engine built with exactly one writer at a time, explicitly because no real locking primitive existed to build genuine concurrency control on top of — an honest limitation stated in the code and the documentation, not a gap discovered later by a user

The common thread: none of these were "we didn't think about it." Each was a specific, named choice about which part of the system's behaviour was allowed to be permissive and which part had to be a hard boundary — exactly the Avoid / Reduce / Transfer / Accept vocabulary from the risk framework page, just applied while the code was actually being written rather than retrofitted afterward in a risk register.

A worked example: risk-scoring one system's real gaps

Applying the framework page's own likelihood × impact scoring to the transactional database's actual, stated limitations (not hypothetical ones):

Risk	Likelihood	Impact	Score	Actual decision
CSV values containing a comma silently corrupt a row	Possible (3)	Moderate (3)	9 — High	Accept, documented. Real CSV quoting is a distinct, sizeable piece of work; stating the limitation plainly in the code and choosing not to build it was a conscious v1 scope decision, not an oversight.
Two concurrent writers corrupt shared state	Unlikely in this deployment shape (2)	Major (4)	8 — High	Accept, by design. Single-writer transactions were chosen specifically because the language has no real locking primitive to build genuine concurrency control on — pretending otherwise would have been worse than naming the constraint.
A crashed mid-transaction state gets misread as committed	The failure was actually observed (5)	Major (4)	20 — Critical	Reduced. This is the wire-format fault described above — found, fixed, and given a standing regression test so the same fault class can't silently reappear.
A malformed SQL statement crashes the whole session	Likely, in an interactive console (4)	Moderate (3)	12 — High	Reduced. Parse and execution errors are returned as ordinary recoverable values, never allowed to abort the process.

Notice the shape: the highest-scoring row is the one that was actually found and fixed, not the one that sounds scariest in the abstract. That's usually how it goes — a live bug earns its score by having actually happened, where a theoretical one has to be argued for.

A short checklist

Have you tested the core logic in isolation, separately from whatever real interface sits in front of it?
Have you tested the interface with real input, not just by calling the functions underneath it directly?
If your system runs in more than one environment, have you run the identical input through all of them?
Have you tested the artifact the way its real caller will actually drive it — one call per user action, not one long continuous run?
Has a real person actually used the finished thing, separately from it passing every automated check?
For every new capability you've added: what NEW state does it introduce that nothing tested before it could have reached? Have you written a test for exactly that state?
For every failure mode you know about: is it Avoided, Reduced, Transferred, or Accepted — and did you choose that on purpose?
When you fix a bug, have you re-run the original failing case against the real rebuilt artifact — not just read the diff and trusted it?
Have you actually read the real implementation of anything you're about to depend on, rather than assumed its behaviour — and if that reading changed something, did you write down what it found somewhere durable, not just in the resulting code?