xtc/docs/PERFORMANCE.md

# Performance and realtime qualification

The original Phase 9 `test-performance` ran a hardware-independent 48 kHz, 128-frame stereo
workload: 64 graph nodes, 4096 MIDI notes (32 simultaneous), 16 tracks and gain
automation. Preparation and block timing use CLOCK_MONOTONIC outside the engine;
callback code itself does not clock or log. The test checks finite output and
reports mean, p99, maximum and CPU-time/audio-duration ratio. A run is a workload
measurement, not a universal realtime threshold. Run more blocks with:

```sh
./build/test_performance 10000
```

Initial Phase 9 measurement on an AMD Ryzen Threadripper 1920X, GCC 16.2.1, -O2,
2000 blocks: preparation 3.045 ms, mean 308.301 us, p99 422.153 us, maximum
665.096 us, versus a 2666.667 us block deadline; CPU/audio ratio 0.1156. These are
one host/run's observed values and depend on scheduling/load/compiler. The test
excludes WAV disk preparation, external/native plugin complexity, GUI redraw,
server scheduling and actual hardware latency. Native callbacks must provide
their own bounded realtime implementation.

Existing software JACK loopback, stereo MIDI/save/reopen, record/reopen,
restart/close and concurrent graph handoff tests remain available. Runtime
fault injection covers loss/rate/xrun/null-buffer and lifecycle failures without
altering the shared server. ASan/UBSan validate memory/error paths; TSan checks
recorder/handoff races, not timing. Sanitized runs are not latency qualification.

Current worst-case storage includes up to 256 MiB preloaded mono WAV samples,
paired graph buffers (128 MiB at 256 nodes/65536 frames; GUI limit 32 MiB), bounded
MIDI/mix metadata and up to 33 metadata/graph history snapshots. Pending/retired
plans can retain additional graph memory. Native preparation can allocate more;
the API cannot enforce an extension's memory budget. There is no CPU overload
scheduler, disk streaming, latency compensation or callback allocator audit of
arbitrary third-party code. High-density interactive projects, physical devices,
long-duration stereo/recording and live server termination remain unqualified.

## Phase 19 matrix, capacity and contention checks

`make benchmark` records 24 warmed graph cases: 2/16/64/256 nodes, mono/stereo,
44.1k/31, 48k/128 and 96k/1024. Each reports control-thread compile time separately
from 300 process calls, mean/p99/max, block deadline, mean load, and paired graph
buffer bytes. Timing, sorting, allocation and logging happen outside processing.
Metered gain chains test graph scaling; they do not represent expensive plugins.
The checked-in `performance-baseline.json` records CPU/compiler/flags/kernel and
all measurements. Copy a new run only after reviewing changes and conditions.

```sh
make benchmark test-performance test-scale test-stress
make benchmark-compare BENCHMARK_MAX_REGRESSION=1.5
# Select a baseline from this same host/compiler/flags:
make benchmark-compare BENCHMARK_BASELINE=/path/to/performance.json
```

Comparison rejects mismatched CPU/architecture/compiler/flags or missing cases.
It fails if mean time exceeds both the ratio budget and a 1 microsecond absolute
noise allowance. p99/max are reported for investigation, not universally gated.
Repeat surprising results under comparable load; do not silently replace the
baseline to make a regression disappear. Scheduling, clocks, thermal state and
shared-machine contention still affect results. No real-time scheduling/pinning
or external profiler is required. CPU-pressure runs are correctness checks, not
comparable timing baselines. Tests/benchmark outputs live below BUILD_DIR.

Initial matrix (Threadripper 1920X, GCC 16.2.1, -O2, 300 blocks): 256-node stereo
48k/128 mean 353.214 us, p99 396.495 us, load .132; 96k/1024 mean 2478.520 us,
p99 3092.434 us, load .232. The full JSON is the evidence. These met the local
investigation target of mean load <.5 for ordinary graph chains; this is a
host-specific development target, not a supported maximum CPU or latency promise.

The mixed 64-node/4096-note workload measured 340–357 us before removing unused
live-controller processing and 313 us after (mean over 2000 blocks); p99 varied
with system load. The measured hot-path change skips live voices/controller ramps
when neither is active and avoids finished-ramp calls. Exact CC/odd-block tests
continue to pass. No other graph optimization was justified by this measurement.

`test-scale` constructs all 64 tracks, 1024 clips, 1024 automation points and
4096 notes, compiles a standalone 256-node metered graph, renders stereo odd-sized
blocks and rejects capacity excesses. Project/GUI graphs remain capped at 64
nodes. `test-stress` runs publication, allocation/retirement, capacity and optional
GUI/timer/redraw tests while two ordinary CPU workers compete; subprocesses have
timeouts and workers are always terminated. Existing callback guards verify no
host allocation/free/loading/unloading. This does not qualify arbitrary plugins.

Callback work is bounded by documented node/frame/clip/note/event/point capacities.
Exceeding model/plan/mailbox capacities returns CAPACITY transactionally; invalid
or overlarge processing buffers are silenced and reported, JACK xruns are counted.
There is no CPU scheduler or preemption inside a DSP callback: an expensive trusted
plugin can miss a deadline or hang. Automated LADSPA lanes currently use run(1)
and bounded per-sample lane searches and may be costly; they are excluded from
the gain-chain baseline. Large WAV memory limits remain those in AUDIO_FILES.md.
Physical-device endurance, CPU-overload sound quality and a full GUI latency
benchmark remain unqualified.

## Phase 29 qualification workloads

The current `test-performance` emits one JSON record (the old prose output is
retired). Default: 2,000 stereo blocks at 48 kHz/128, 64 graph nodes, 16 tracks,
1,024 stored notes and 256 automation points, at most 32 active voices. This
changes the Phase 9 workload; its historical numbers above are not a baseline
for this revision. `benchmark-project` runs nine cases: 1/16/64 tracks,
64/1,024/4,096 stored notes, 16/256/1,024 points and 31/128/1,024-frame buffers.
Two long notes per first 16 tracks remain audible. Other notes are scheduled
later to remain within the 32-voice plan limit. The position wraps every 96,000
samples to keep long runs active. Future notes exercise storage/preparation and
bounded scans rather than additional simultaneous synthesis. Each track has
16 gain points; graph processing uses the same 64-node chain in every case.
The graph matrix separately scales nodes and chain connections from 2 to 256.
These synthetic cases do not qualify every possible dense graph or plugin chain.

```sh
make benchmark benchmark-project
make benchmark-compare
make benchmark-project-compare
# Baselines must match the current CPU/compiler/flags and workload.
make benchmark-project-compare PROJECT_BENCHMARK_BASELINE=/path/to/report.json
./build/test_performance 10000 64 128
make test-benchmark-tools
```

Both matrices report wall-clock mean/p99/max processing duration and the count
of measured calls exceeding frames/sample-rate. These are offline deadline
exceedances, not actual JACK xruns: there is no server scheduling in the timing
loop. Graph compilation or project construction/preparation is timed separately.
`process_cpu_ms` measures CLOCK_PROCESS_CPUTIME_ID over the complete measured
loop, including timing and finite-output checks; it excludes preparation and
sorting. Project `cpu_audio_ratio` divides that CPU time by simulated audio time.
`peak_rss_kib` is Linux getrusage process high-water RSS, including model, plans,
measurement storage and executable/runtime overhead; it is not current heap size
or a leak test. Graph buffer bytes remain a separate exact buffer-size estimate.
Timing probes, memory queries, sorting and output occur outside engine calls.

Checked-in `project-performance-baseline.json` provides initial project cases;
the original graph baseline is retained. `graph-performance-phase29.json` adds
a current graph report with CPU/RSS/deadline fields. Gates use the existing
1.5x mean-time budget and 1 us noise allowance, with p99/max/misses and memory
reported for investigation. There is no universal hardware-independent deadline
or RSS pass threshold. `test-benchmark-tools` verifies same-context success,
regression rejection, missing cases and host/workload mismatch without relying
on machine timing. Do not replace a failing baseline without investigating.

## Endurance, faults and callback audit

```sh
make test-soak test-stress test-stress-plugins
make test-stress STRESS_ROUNDS=10
./build/test_soak 10000000
make check-development
```

`test-soak` renders exactly 1,000,000 alternating 31/128/1,024-frame stereo
blocks: 2.282 hours of virtual 48 kHz audio. It checks every output sample,
processed block counts and final publication acknowledgement, with 9,999
quiescent position/rebuild cycles. It runs faster than wall time and models
position wrap/seek and restart publication; it is not a real device start/stop
or hardware endurance test. The project benchmark separately wraps active MIDI.

`test-stress` defaults to three rounds under two ordinary CPU workers. Each round
runs concurrent publication, allocation/lifetime guards, maximum model capacity,
project relocation/reload and missing-media retention, save-fault diagnostics,
and the soak test. With JACK headers present it also repeats the deterministic
backend fake's start/stop/seek, rate/buffer/server-loss and failure/retry cases.
Optional GUI tests run after those rounds under the same CPU load.
`test-stress-plugins` adds the original LADSPA fixture's identity/layout/range,
instantiation-error and invalid-metadata cases each round; it needs LADSPA headers.
No command changes a physical device, kills the shared server or loads unknown
plugins. Subprocess deadlines bound each test and workers are cleaned up on exit.

The reliability executable now interposes fopen/fread/fwrite/read/write,
pthread_mutex_lock and pthread_cond_wait in addition to malloc/calloc/realloc/free.
Guarded host graph/realtime render calls fail immediately on these operations.
This complements the existing native lifetime and plugin callback guards and
control/callback ownership review in RELIABILITY.md. It does not intercept every
libc entry point, raw syscalls, hidden dynamic-module calls, or GUI code by magic;
third-party DSP remains trusted. No known host callback safety violation was found.

`check-development` combines the ordinary build/tests, capacity/soak checks,
benchmark gate tests, static analysis and ASan/UBSan. The sanitizer target now
includes maximum capacity and virtual endurance. These are opt-in developer
checks requiring installed Clang/Python; no runtime dependency was added.
Sanitized/contended/profiled runs must not become timing baselines.

## Profiling evidence and remaining costs

Before considering DSP changes, Phase 29 sampled a 10,000-block, 64-track/128-frame
project with GCC `-O2 -g -pg` and GNU gprof on the documented baseline host.
Flat samples attributed about 30% to mixer automation `value`, 29% to MIDI
`segment`, and 11% to finite-value handling. Instrumentation and inlining affect
attribution; this identifies investigation candidates, not exact production
cost or evidence that checks should be removed. MIDI scans bounded stored-note
arrays and evaluates per-voice, per-sample gain/pan lanes. No DSP optimization
was introduced without a separate correctness/performance justification.

Reproduce in an isolated build (GNU gprof must already be installed):

```sh
make BUILD_DIR=build/profile CFLAGS='-O2 -g -pg -std=c99 -Iinclude' LDFLAGS=-pg build/profile/test_performance
GMON_OUT_PREFIX="$PWD/build/profile/gmon" ./build/profile/test_performance 10000 64 128
gprof build/profile/test_performance build/profile/gmon.* > build/profile/profile.txt
```

Remaining limits include per-sample LADSPA automation run(1), stored-note/clip
scans, preloaded WAV memory and immutable-plan/history overlap. This phase does
not qualify physical-device changes, arbitrary plugin hangs/crashes, multi-hour
wall-clock sessions or GUI redraw latency. Virtual endurance and deterministic
faults provide repeatable software evidence, not a hard realtime guarantee.

Phase 29 observed results (same Threadripper/GCC host, 300 blocks per case):
64-track/128-frame project mean 380.417 us, p99 720.271 us, prepare 11.841 ms,
CPU/audio .142 and peak process RSS 19,296 KiB. The 256-node stereo/128 graph
mean was 286.398 us, p99 484.712 us, compile .300 ms. These case runs observed
zero deadline exceedances. See JSON for all cases and host metadata.

The project comparison passed. Repeated graph comparisons flagged variable
small cases despite unchanged graph processing code; a 3,000-block run against
the retained Phase 19 baseline still flagged 16-node mono/31-frame mean 5.089 us
above its 1.5x budget. The new graph report is evidence, not a replacement chosen
to make that comparison pass. This shared-host timing alert remains an
investigation item; no CPU/latency guarantee is inferred from the correctness
suite. Longer comparisons can use `BENCHMARK_BLOCKS=3000`.