Last updated: 2026-09-16

Testing Non-Deterministic Systems: Randomness, Timing, and LLMs

Most testing advice quietly assumes a program is a pure function of its input: same input, same output, every time. Two whole categories of system break that assumption on purpose. One consults a source of randomness — games, simulations, sampling, cryptographic nonces, machine learning. The other has a completion time that isn't under the program's control at all — disk and network I/O, another process, a human, or a language model whose generation time and even its own output content both vary. Testing either kind by "run it and see if it passes" either passes by luck or fails by bad luck, and neither result tells you anything you can act on. This page covers both, and a third case that combines them: testing calls to a large language model.

Randomness: make it deterministic on purpose

Testing Fundamentals already lists randomness alongside the filesystem, the network, and the clock as a thing worth mocking. The underlying principle: a test suite should never depend on an unseeded source of randomness. A test that calls the platform's default random number generator without controlling its seed isn't testing your logic. It's testing whether today happened to be a lucky day.

Inject the seed; don't reach for a global RNG. Pass the random number generator, or at minimum its seed, in as an explicit parameter or dependency, the same way a well-tested function takes the clock or the filesystem as a parameter rather than calling a hidden global. A function that reaches into ambient global state for randomness is no more testable than one that reaches into ambient global state for anything else.
Log the seed the moment a test fails. A property-based framework that finds a failing case (QuickCheck, Hypothesis, PropCheck, fast-check, and their equivalents) should print the exact seed and the exact generated input that broke the test, so the failure is reproducible on demand rather than "it failed once, twenty minutes ago, and I can't get it to happen again."
Prefer property-based testing over one fixed example wherever the input space is random. Property-based testing generates many seeded inputs automatically and shrinks a failure down to the smallest input that still reproduces it — the natural fit for this problem, and a deeper treatment than a single hard-coded example test can offer.
Know that "reproducible" and "representative" are different properties. A fixed seed makes a test deterministic, but if that one seed never happens to generate the edge case that actually breaks the code, the test gives false confidence forever. A fixed-seed regression test and a property-based test that varies the seed are solving two different problems, and most systems that use randomness need both.
A real example with no way to cheat: PatLang has no random-number host function at all. The CGA interval classifier demo needs pseudo-randomness to place its dataset, and since the language provides no ambient rand() to reach for, it hand-rolls a small seeded linear congruential generator instead — a Park–Miller/MINSTD-style generator (s2 = (s * 48271) % 2147483647), seeded explicitly from a value passed in through the command line. The result is the discipline argued for above, but enforced by the language rather than remembered by the programmer: the same seed always reproduces the same dataset, and only a genuinely new seed produces a new one.

Timing: I/O, other processes, and the clock

Timing non-determinism is a different shape of the same underlying problem. The value a call returns might be perfectly deterministic; when it arrives, or whether it arrives in any bounded time at all, isn't.

Don't assert on wall-clock duration unless duration is the thing under test. "This must complete in under 100ms" fails constantly on a shared or loaded CI runner for reasons that have nothing to do with correctness. Where possible, assert on order and causality instead of duration — "the response was received strictly after the request completed" rather than "completed within 100ms."
Control the clock explicitly rather than sleeping and hoping. A test that calls a real sleep() to "wait long enough" is both slow and still occasionally wrong. Injecting a fake, advanceable clock lets a test move simulated time forward instantly and deterministically instead of gambling on a real delay being long enough.
Test the retry and timeout logic itself, not just the code it protects. If production code retries a flaky call three times with backoff, a test should verify that policy actually fires — by making a fake dependency fail exactly the right number of times before succeeding, for instance. A retry policy is part of the program's specification, and an untested one is the kind of code most likely to hide an off-by-one: retrying twice instead of three times, or never giving up at all.
Distinguish "eventually happens, in unknown but bounded time" from "may never happen." The first calls for polling or an awaited callback with a generous timeout, not a guessed sleep duration. The second is a genuine hang, and a test needs its own timeout, separate from whatever timeout the code under test has — otherwise a hang in the code under test becomes a hang in the entire test run, with no diagnostic left behind.
Mocking removes non-determinism, but it also removes the chance of catching a real integration bug. That's a deliberate trade-off between test-pyramid layers, not a reason to avoid live dependencies entirely: a suite that only ever talks to mocks can pass while the real integration is quietly broken.

LLMs: a harder kind of non-determinism, made of the first two

A call to a large language model combines both problems above and adds a third. Generation time is variable and network-bound, like any other remote call. And even at temperature 0 — which makes token selection greedy, always the single highest-probability token — a hosted API's response to an identical prompt is still not guaranteed to be bit-for-bit reproducible across separate calls. The usual explanation blames floating-point non-associativity in GPU matrix operations, but a more precise account traces most of the effect to something narrower: individual GPU kernels are actually deterministic for a fixed batch size, but their numerical output changes with the batch size itself, and a hosted server's batch size varies with how many other requests happen to be running at that exact moment — something with nothing to do with your prompt at all¹. On top of that, the output itself lives in a third space neither plain randomness nor plain timing testing has to deal with: unbounded natural language, not a small fixed set of return values, where "assert equals" is rarely even the right shape of check to write.

Test the property, not the string. Don't assert that a model's response equals fixed text. Assert a property of it instead — it contains a required field, it parses as valid JSON against a schema, it avoids a banned pattern, it falls within an expected length or structure. This is the same shift property-based testing makes for random input, applied to unpredictable output instead.
Keep "the call returned well-formed output" separate from "the output was good," and test them on different schedules. The first is a fast, close-to-deterministic contract test (shape and schema validation) that belongs in every commit's test run. The second is closer to an evaluation than a unit test — often run less often, against a curated set of scenarios, sometimes scored by a second model or a person, and tracked for drift over time rather than asserted pass or fail on every run.
Pin what can be pinned, and budget for what can't. Fix the model version, the prompt, and the sampling parameters (temperature, top-p, and the seed if the provider exposes one) in a test, since an unannounced model update can silently change behaviour underneath a passing suite. Even fully pinned, accept that a hosted model's output may still drift slightly, and write assertions loose enough to survive that drift without becoming so loose they would also pass on nonsense.
Record real responses once, and replay them for the tests that don't need a live call every run. A cassette or fixture captured from one real call, then replayed, covers most of the suite; reserve genuinely live calls for a smaller, slower set of tests that exist specifically to catch drift or regressions in the real service. Live calls also cost money and add real latency to every run, which is its own reason to keep them a deliberate minority.
Treat "the model will keep behaving the way it did when this prompt was written" as an assumption to test, not one to hold on faith. Trustworthy Software's verifiability pillar makes the general version of this point: an assumption nobody has actively tried to break isn't the same claim as an assumption that holds.

Non-determinism is not a reason to skip testing. It's a reason to test differently: control what can be controlled (seeds, clocks, pinned model versions), assert on properties and invariants instead of exact values where control isn't possible, and make failures reproducible — log the seed, log the actual timing, log the actual model response — so a rare failure is something to diagnose rather than something to shrug off.

The single worst habit common to all three is re-running a failing test until it passes, or quarantining a flaky test and never returning to it. A flaky test is not noise. It is a signal about a real non-determinism the code doesn't yet handle correctly, and it deserves the same standing as a test that fails for a known, documented, currently-accepted reason: kept in the suite, labelled honestly, and not narrowed or deleted just to make the run go green.

Where this connects

Testing Fundamentals — the mocking and property-based testing foundations this page builds on directly.
Types of Testing — the test-pyramid trade-off between fast, mocked layers and slower, real-dependency layers referenced in the timing section above.
Trustworthy Software — the verifiability pillar this page's LLM section borrows directly for treating model behaviour as an assumption to test.
PatLang CGA Interval Classifier — the live demo behind this page's seeded-LCG example.

References

He, H., in collaboration with Thinking Machines Lab (2025). Defeating Nondeterminism in LLM Inference. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ ↩