The PatLang Compiler Pipeline: One Front End, Four Final Stages

PatLang has always had more than one way to run a program — the tree-walking interpreter (--ir-run), the rustc-backed native compiler (--patc), and (in progress, currently on hold) a from-scratch x64/NASM backend. What changed recently is that this stopped being three separate, loosely-related tools and became one genuinely shared pipeline: every path lexes, parses, and lowers a program through the exact same code, and only the final stage — what to actually DO with the resulting IR — differs per target. This page documents that shape directly, including the newest addition: a fourth final stage, a meta-circular interpreter written in PatLang itself.

The shared pipeline

The lexer/parser/lowerer were already shared between the rustc and x64 backends before this addition — the IR they produce is a plain, list-shaped value (["ProgramIR", entry_name, funcs, events], each function a ["FuncIR", name, params, instrs] entry, instructions drawn from a small fixed set: Const/Load/Store/Bin/Un/Jump/JumpIfFalse/Return/CallHost/Call/BuildList/MakeClosure/CallValue). Nothing about that IR is specific to any one backend; it's just data. The interpreter slots into that same shape as a third possible consumer of it, needing no codegen step at all — just a function that walks the IR directly and computes.

Explicit pipeline-stage commands

This modularity used to be an internal implementation detail, only reachable through patc1's single combined <input> <output> invocation (still fully supported, unchanged, for every existing caller). It's now also directly available as separate commands, so the same intermediate representation can be saved, inspected, and fed into any final stage without recompiling from source each time:

patc1 lower       mysource.patlang mysource.ir     # tokenize + parse + lower, save IR
patc1 emit_rust   mysource.ir      mysource.rs      # rustc-backend text, from a saved IR
patc1 emit_x64    mysource.ir      mysource.asm     # NASM-backend text, from a saved IR
patc1 emit_wasm   mysource.ir      mysource.wasm    # reserved, not yet implemented
patc1 interpret   mysource.ir                       # meta-circular interpreter, prints the result

Candidates in the same shape, not yet built: an optimise command (an IR-to-IR pass), a cfg command (dump a control-flow graph for a function), and a debug command (an interactive, inspectable step-through of interpret's own execution). If any of those eventually need name/type information beyond what the plain IR carries, a real symbol table would slot in as another value threaded alongside the IR — a small addition to this same shape, not a redesign of it.

The meta-circular interpreter

self_hosting/lib/interp.patlang's interpret_ir(ir) is an interpreter FOR PatLang, written IN PatLang — distinct from both the Rust-native tree-walker (rust-runtime/src/ir/interpreter.rs, what --ir-run actually runs) and from run_ir (a host function that just delegates to that same Rust interpreter, useful for the live playground but not self-hosted at all). Runtime values are plain PatLang numbers/strings/lists/booleans — no separate tagged value type is needed, since the interpreter runs AS a PatLang program and can already compute with PatLang's own native values and operators directly. A closure is represented as a small internal tagged list (["__closure__", func_name, captured_values]), the same convention IR/AST nodes themselves already use throughout this codebase.

Built and verified in four slices, each tested against a real program before moving to the next:

1. Arithmetic, locals, control flow — Const/Load/Store/Bin/Un/Jump/JumpIfFalse/Return. Verified: 3 + 4 * 2 correctly returns 11 (operator precedence intact), and an if/else correctly takes the branch a comparison selects.
2. Host calls and lists — CallHost, plus BuildList for list literals.
3. Recursive user-defined function calls — Call. The interpreter threads the whole program's function list through itself so a call can look up and recurse into any other function by name, reusing PatLang's own call stack for the recursion rather than maintaining a separate one. Verified with a recursive factorial: fact(5) correctly returns 120.
4. Closures — MakeClosure/CallValue. Verified with a closure capturing an enclosing function's parameter: make_adder(3) returns a closure that, called with 7, correctly returns 10.

All four slices passed on their first real test except the very first (a small, unrelated bug: a call to a Rust-internal formatting function that was never actually registered as a callable host function, caught and fixed immediately). All of this has since been wired into the automated regression suite (a dedicated Cucumber feature, one scenario per slice, run through the real patc1 lower/interpret CLI path) rather than left as one-off manual smoke tests.

Host-function coverage started narrow (a curated dozen: print, list_*, a few string/number primitives) and was later asked to become complete: "we should get the whole of the language supported in the interpreter." The first attempt at scoping this had assumed oo/logic/contracts/networking couldn't be supported because their state (object tables, facts, rules, actions) lives in Rust-side thread-local registries the interpreter has no PatLang mirror of — that assumption turned out to be simply wrong, caught by checking rather than by being told: interp.patlang is itself an ordinary PatLang program (interpreted or compiled into patc1.exe either way), and those registries live in the SAME native process regardless of which PatLang code is running. Delegating new/fact/send/tcp_listen/etc. to the exact same host functions any other program already calls turned out to be exactly as easy as delegating sqrt. interp_call_host now covers essentially the entire real host surface — core, strings, collections, files, I/O, math, objects, logic/facts, contracts, networking — verified directly: an object's field set via send and read back via get returns the right value; a fact recorded via fact and queried via query reports the right count. Only the compiler's own bootstrap-meta tools (parse_tiny_source, rustc_build, run_ir, and similar) remain out of scope — those are about compiling or interpreting OTHER programs, not this program's own behavior, a deliberate line rather than an oversight.

Error handling was tightened to match: every function that can fail now returns ["Ok", value] or ["Err", message] instead of printing a message and silently returning 0, propagating cleanly up through recursive Call/CallValue chains — calling an unsupported host function now reports a clear interpret: error: host function 'X' not supported... rather than a puzzling wrong answer.

Finally, two purpose-built primitives — compile_patlang(source, out_path) and interpret_patlang(source) — let a running INTERPRETED program compile or interpret brand-new PatLang source text at runtime, reusing the exact same tokenize/parse/lower/emit_program_rs_chunked/rustc_build_chunked_texts pipeline the CLI's own lower/emit_rust/interpret subcommands use. Verified end to end: a program calling compile_patlang("return 6 * 7", ...) produces a real, independently-runnable executable that, run directly, prints 42 — the foundation for an eventual interactive REPL/friendly-cli session able to compile, not just interpret, whatever a user types.

Where this sits among the dev tools

Both the native pat binary and the self-hosted patc1.exe can drive the chunk-precompile-and-link mechanism (each PRELUDE_* runtime chunk compiled once into a cached library, keyed by a hash of its own fully-assembled source, instead of every compile re-optimizing the whole shared prelude text from scratch) — a standalone compiled program has no direct access to the native process's internal chunk-text lookup, so it carries its own self-contained, std-library-only copy of the same caching algorithm, fed chunk text from its own compiled-in mirror functions instead. Both paths were verified to produce a working, correctly-behaving binary at every generation: the initial patc1.exe build, patc1.exe compiling its own combined source into a second-generation patc2.exe, and that second-generation binary correctly compiling a third, ordinary program on its own.

See also

Act XXI tells the fuller story of the chunk-precompile mechanism, including the real bugs found along the way. The standard library reference lists the underlying host functions (rustc_build_chunked/rustc_build_chunked_texts). Capabilities & Honest Limitations for what interpret_ir and emit_wasm genuinely don't support yet, and the x64 backend's current on-hold status.