The Swarm Compiler: Compilation as a Graph of Owned Artefacts

TheThe idea behind the design is that a compiler should organise its own work the way PatLang asks programs to organise theirs. A well-structured PatLang program exposes its state through parameters, closures and narrow objects that carry specific capabilities, with no ambient mutable globals; the Block Ownership Model makes that structure visible to the compiler. If the program's dependencies are explicit, the compiler can take its schedule from them, treating a build as a distributed fixed-point computation over owned artefacts and dropping the separate scheduling scheme it would otherwise impose.the same ownership rules guide the compiler

Two kinds of ownership, kept apart FoundationalKnowledge that endures for decades — core principles

TheThe design uses ownership vocabulary twice, on purpose, and insists the two are never mixed:

A worker may own the job of compiling a function outright while that function's body handles a value that is shared at run time. The codebase had already been bitten by confusing the two: counters kept in PatLang's old shared global table raced between parallel_map workers and produced silent miscompiles, which is compiler-work state accidentally sharing identity with something that looked like program state. The rule is to keep the two distinct in names, types and diagnostics everywhere.

Jobs and the three barriers FoundationalKnowledge that endures for decades — core principles

TheThe pipeline becomes a set of job kinds. Most run per unit: a file, a declaration, a function. Three run once per program, and those three are the only places where the whole program has to be seen at once.

Each barrier exists for a stated reason. B0 is there because PatLang lets a program define its own syntax, and a syntax definition in one file changes how another file parses. B1 is there because whether f(x) is a call to a declared function or to a built-in can only be decided by knowing every declaration. B2 holds the handful of facts that are program-wide by design, such as which functions are reached only by name through apply or parallel_map. Each barrier is a cheap scan or fold over small summaries; the expensive work sits between them and runs per unit.cf. pipeline: stages with barriers

Three rules make the jobs independent of the order they run in:

• Names come from declarations. A synthesised name (a closure's, a loop block's) is the declaring function's identity plus a counter local to it, never a shared counter, so any order of work produces the same names.
• Registrations are recorded where they are declared. when handlers, logic rules and functions reached by name become explicit entries in the program facts. The design lists inferring dependencies by searching strings after lowering as an anti-pattern, and the project had a direct scar from it: a linear membership scan that dominated an earlier compile-time regression.
• Output order is source order. Chunks, symbols and diagnostics are ordered by file, then declaration, then synthesised block, and never by which worker finished first.

The ledger: content-addressed, complete or absent FoundationalKnowledge that endures for decades — core principles

EveryEvery artefact is stored under an identity computed from what it depends on: the job kind, a fingerprint of the compiler source that implements that stage, the artefact's own payload, and the identities of its inputs. Worker identity, timestamps and completion order never enter it. Two builds that need the same artefact compute the same key and can share the stored result.like git hashes, keys are reproducible

The encoding is a compact binary form of nested PatLang values. A first prototype used JSON, and measurement showed JSON round-trips accounted for the entire 25–38% overhead the prototype added; the existing per-function code cache had already moved off JSON for the same reason. Every validator that checks an artefact also gets a negative test proving it can reject a bad one, because the prototype's validator had never been shown to reject anything.

This is a build system as much as a compiler, and the build-systems literature has names for its parts. Mokhov, Mitchell and Peyton Jones classify build systems by how they schedule tasks and how they decide a task is up to date, and describe constructive traces, stored results keyed by a task's inputs, as the mechanism that lets cloud build systems share outputs between machines¹. The ledger is a constructive-trace store at the granularity of single functions.

Measure first: two quadratic costs and an 8.5× speed-up Applied / MethodologicalKnowledge with a 5–10 year half-life — stable practice

TheThe plan's first phase changed nothing in the compiler. It timed every stage of a full self-compile of the old compiler, about 2.4 MB of PatLang source, by polling the compiler's stage markers from an outside script once a second. The whole build took 8,970 seconds. Five functions took 75% of it.

Those five, regex_match_node and its neighbours, each lowered to around 6 MB of assembly, about 69,600 instructions apiece. Two causes, both quadratic, were behind it:

• Splitting assembly text into lines was O(n²). split_lines read the text one character at a time with char_code, and each char_code call checked whether the whole string was ASCII. Assembly time per function grew with roughly the square of its size: 3.8× more text cost 13× more time. The fix cuts the text into pieces of at most 1,024 characters by repeated halving and scans only those: 80,000 lines went from 12,597 ms to 373 ms.
• Every closure call inlined a dispatch over the whole program. Each dynamic call site expanded into a chain of string comparisons with one branch for every callable in the program taking that many arguments, around 6,300 instructions per site in the full compiler, and the same expanded code was copied into five separate roots. Each site now makes one call to a shared dispatcher per arity. A function with two closure calls went from 3,028 instructions to 66.

Together the two fixes took the full self-compile from 8,970 seconds to 1,050. Neither fix involved any parallelism. The measurements also bounded what parallelism could ever have achieved: per-function parallel compilation cannot make a build faster than its largest single function, which had been about 1,600 seconds, longer than the whole build after the fixes. The plan names, among its anti-patterns, "claiming distributed-compilation success when only a thread pool was added around otherwise-monolithic lowering"; this phase is the reason that line is there.the longest task sets the floor

The measurements turned up one more thing: the same input, producing byte-identical output, ran 30% slower on a second run. Any speed-up claim since has had to rest on repeated runs.

Building it in the swarm's shape from the start Applied / MethodologicalKnowledge with a 5–10 year half-life — stable practice

TheThe first plan wrapped the existing compiler in a new driver and kept its output byte-identical, using the old compiler as the oracle. Its hardest phase was untangling program-wide state that the old lowering kept in a shared global table. Once the memory design review decided to rewrite the compiler and runtime anyway, the swarm structure moved into the new version's foundations:

• Each stage is a pure job: explicit input artefacts, one output artefact, no shared mutable state.
• Per-function ownership summaries are artefacts and contracts. A caller is compiled against its callee's summary (its arity, which parameters it borrows, whether its result is owned), never against its body.
• The runtime is compiled by the same pipeline as one more program, cached per function.
• Byte parity with the old compiler gave way to behavioural parity, since two-word values make the new output different by design. Byte parity still holds inside the new version, between a sequential and a parallel build of the same input.

The summary is what turns the ledger into an incremental compiler. A job's key hashes the function's IR, the specific program facts its code generation reads, the summary of every function it calls, and the compiler stages' source. Change the body of g without changing its summary, and its caller f keeps its key and is reused; only g is rebuilt. Our own reading of this, in the terms of the build-systems literature cited above, is that it applies early cutoff at the level of a function's contract: a rebuild stops propagating as soon as a callee's interface comes out unchanged, however much its body moved.early cutoff: stop propagating rebuilds

What it does now Ephemeral / ToolingKnowledge that evolves in months to a year — check for updates

MeasuredMeasured on the v2 compiler in November 2026:

• A second program built after the first reuses all 238 of the runtime's function jobs and builds only its own 5.
• Run on eight threads in a shuffled order (--workers 8 --shuffle 7), a build's jobs took 46 seconds against 206 on one thread, and the two executables were byte-identical.
• Two cold builds of the same program run at once, sharing one artefact store, at 100, 200, 400 and 800 new functions: both succeed and both executables are byte-identical.
• Each job record states the length of every artefact it wrote and ends with a marker line, so an artefact cut short by an interrupted build, or by another build writing it at that moment, is rebuilt. Because a job's artefacts are the same bytes whoever writes them, no locking or rename step is needed.
• The store is trimmed explicitly, least recently used first, by --evict KEEP_MB.
• v2 compiles itself to a fixed point in about 7.8 minutes cold on eight workers; generations two and three are byte-identical.

Threads became usable because the stages write no shared state, which the first plan could not assume and why it chose separate worker processes. Making threads safe still found bugs: every thread had written system-call byte counts to one shared word, so the compiler run with eight workers failed at random on 4 of 21 compiles, until each thread's context got its own word and buffer.

A deterministic IR as the bootstrap Applied / MethodologicalKnowledge with a 5–10 year half-life — stable practice

DeterministicDeterministic output gives one more option. v2 can write a program's complete IR, every reachable function including the runtime's, as text. A small reader written in Python with only its standard library builds a Windows executable from that IR, and it builds the compiler from the compiler's own IR. That compiler builds the compiler, the result builds it again, and the last two are byte-identical. The same IR also produces a WebAssembly module for the browser. Bootstrapping PatLang therefore needs no saved seed binary and no Rust, only the IR and a reader short enough to read. Packages, resolution and trust takes up what that means for trusting the toolchain.

See also

Blocks,Blocks, ownership and memory covers the runtime design this compiler implements. The compiler pipeline describes the single-pass front end the earlier compilers used. Journey Chapter 13 records an earlier optimisation measured, found slower, and reverted, the same discipline as this page's measure-first phase.