Testing probabilistic software

Section 01

Testing systems with variable output

Traditional software testing assumes deterministic behavior: given input X, the system produces output Y. LLM systems break this assumption. The same input can produce different valid outputs. Some variation is expected; some indicates a bug. How do you test a system where the correct answer is not a single value?

Section 02

Deterministic shell, probabilistic core

An LLM application has two layers: a deterministic shell (input validation, routing, tool execution, output checking, state management) and a probabilistic core (the model's generation). The deterministic shell can and should be tested traditionally — exact assertions, schema validation, permission checks. The probabilistic core requires different strategies.

Where the analogy breaks: in traditional software, the boundary between tested and untested code is usually clear. In LLM systems, the model's output flows directly into the application shell (as function arguments, as routing decisions, as user-facing text). The boundary is porous — model output becomes application state.

Section 03

Testing strategies for probabilistic output

Invariants: properties that must always hold, regardless of the specific output. 'The response must be valid JSON.' 'The response must not contain PII.' 'The tool call must include a valid API name.' These can be tested with exact assertions on every output.

Repeated trials: run the same input N times and measure the distribution of outputs. Check that the variance is within acceptable bounds. Flag outputs that appear in less than X% of trials as potential instabilities.

Calibrated semantic checks: use a model grader to evaluate outputs on a rubric. Track the distribution of scores over time. A drop in median score indicates a regression.

Confidence intervals: when measuring pass rates, report the confidence interval. A 92% pass rate on 50 samples has a wide interval. Do not treat point estimates as truth.

Tolerant snapshots: instead of asserting exact output, assert structural properties (contains these sections, references these sources, is under this length). Snapshot tests that are too brittle will fail on acceptable variation.

Section 04

What to test deterministically

These surfaces should be tested with exact assertions:

• JSON schema validation (does the model's output parse? do all required fields exist?)
• Permission checks (does the system enforce access control on retrieved documents?)
• State machine transitions (does the agent move through valid states only?)
• Tool policy enforcement (are disallowed tools blocked?)
• Token budget enforcement (does the system reject over-limit requests?)
• Error handling (does the system degrade gracefully when the API is unavailable?)

Section 05

Running evaluations as tests

Connect your evaluation suite (Lesson 09) to your CI pipeline. Every model change, prompt change, or retrieval change runs the full evaluation set. The results are compared against the baseline. A regression on any slice blocks the release.

This is different from traditional unit tests: you are not checking for exact matches. You are checking that the distribution of outputs has not degraded beyond an acceptable threshold. False positives (flagging acceptable variation as a regression) are costly but manageable. False negatives (shipping a real regression) are worse.

# Run evaluation suite
results = run_evals(model_version, eval_dataset)

# Check for regressions by slice
for slice in results.by_slice:
    if slice.pass_rate < baseline[slice.name].pass_rate - threshold:
        fail(f'Regression on {slice.name}: '
              f'{slice.pass_rate:.1%} vs baseline '
              f'{baseline[slice.name].pass_rate:.1%}')

# Check for new failure patterns
new_failures = diff_failures(results, baseline)
if new_failures:
    warn(f'{len(new_failures)} new failure cases')

Section 06

Common testing pitfalls

Testing with the same seed always: if your tests use a fixed seed and temperature 0, they pass deterministically but do not reflect production variance. Run tests across multiple seeds and temperatures.

Asserting exact text: LLM output varies. Asserting exact text creates brittle tests that fail on acceptable variation. Assert properties instead.

Ignoring slice performance: aggregate pass rates hide slice regressions. Always break down by slice.

No universal flakiness threshold: there is no standard 'acceptable flakiness rate.' It depends on the task consequence. Document your threshold and justify it.

Section 07

Unscored lab: design a test strategy

Design a test strategy for a tool-calling agent that reads from a database and sends emails.

1. Identify the deterministic surfaces: what can be tested with exact assertions?
2. Identify the probabilistic surfaces: what requires invariants, repeated trials, or semantic checks?
3. Write 5 invariant tests (properties that must always hold).
4. Design a repeated-trial test: run the same input 20 times. What do you measure?
5. Define the CI integration: what blocks a release? What is a warning?