xtc/docs/PERFORMANCE.md

# Performance and Realtime Qualification

The original `test-performance` ran a hardware-independent 48 kHz, 128-frame
stereo workload: 64 graph nodes, 4096 MIDI notes (32 simultaneous), 16 tracks
and gain automation. Preparation and block timing use CLOCK_MONOTONIC outside
the engine; callback code itself does not clock or log. The test checks finite
output and reports mean, p99, maximum and CPU-time/audio-duration ratio. A run
is a workload measurement, not a universal realtime threshold. Run more blocks
with:

```sh
./build/test_performance 10000
```

Initial measurement on an AMD Ryzen Threadripper 1920X, GCC 16.2.1, -O2, 2000
blocks: preparation 3.045 ms, mean 308.301 us, p99 422.153 us, maximum 665.096
us, versus a 2666.667 us block deadline; CPU/audio ratio 0.1156. These are one
host/run's observed values and depend on scheduling/load/compiler. The test
excludes WAV disk preparation, external/native plugin complexity, GUI redraw,
server scheduling and actual hardware latency. Native callbacks must provide
their own bounded realtime implementation.

Existing software JACK loopback, stereo MIDI/save/reopen, record/reopen,
restart/close and concurrent graph handoff tests remain available. Runtime fault
injection covers loss/rate/xrun/null-buffer and lifecycle failures without
altering the shared server. ASan/UBSan validate memory/error paths; TSan checks
recorder/handoff races, not timing. Sanitized runs are not latency
qualification.

Current worst-case storage includes up to 256 MiB preloaded mono WAV samples,
paired graph buffers (128 MiB at 256 nodes/65536 frames; GUI limit 32 MiB),
bounded MIDI/mix metadata and up to 33 metadata/graph history snapshots.
Pending/retired plans can retain additional graph memory. Native preparation can
allocate more; the API cannot enforce an extension's memory budget. There is no
CPU overload scheduler, disk streaming, latency compensation or callback
allocator audit of arbitrary third-party code. High-density interactive
projects, physical devices, long-duration stereo/recording and live server
termination remain unqualified.

---

## Matrix, Capacity and Contention Checks

`make benchmark` records 24 warmed graph cases: 2/16/64/256 nodes, mono/stereo,
44.1k/31, 48k/128 and 96k/1024. Each reports control-thread compile time
separately from 300 process calls, mean/p99/max, block deadline, mean load, and
paired graph buffer bytes. Timing, sorting, allocation and logging happen
outside processing. Metered gain chains test graph scaling; they do not
represent expensive plugins. The checked-in `performance-baseline.json` records
CPU/compiler/flags/kernel and all measurements. Copy a new run only after
reviewing changes and conditions.

```sh
make benchmark test-performance test-scale test-stress
make benchmark-compare BENCHMARK_MAX_REGRESSION=1.5
# Select a baseline from this same host/compiler/flags:
make benchmark-compare BENCHMARK_BASELINE=/path/to/performance.json
```

Comparison rejects mismatched CPU/architecture/compiler/flags or missing cases.
It fails if mean time exceeds both the ratio budget and a 1 microsecond absolute
noise allowance. p99/max are reported for investigation, not universally gated.
Repeat surprising results under comparable load; do not silently replace the
baseline to make a regression disappear. Scheduling, clocks, thermal state and
shared-machine contention still affect results. No real-time scheduling/pinning
or external profiler is required. CPU-pressure runs are correctness checks, not
comparable timing baselines. Tests/benchmark outputs live below BUILD_DIR.

Initial matrix (Threadripper 1920X, GCC 16.2.1, -O2, 300 blocks): 256-node
stereo 48k/128 mean 353.214 us, p99 396.495 us, load .132; 96k/1024 mean
2478.520 us, p99 3092.434 us, load .232. The full JSON is the evidence. These
met the local investigation target of mean load <.5 for ordinary graph chains;
this is a host-specific development target, not a supported maximum CPU or
latency promise.

The mixed 64-node/4096-note workload measured 340–357 us before removing unused
live-controller processing and 313 us after (mean over 2000 blocks); p99 varied
with system load. The measured hot-path change skips live voices/controller
ramps when neither is active and avoids finished-ramp calls. Exact CC/odd-block
tests continue to pass. No other graph optimization was justified by this
measurement.

`test-scale` constructs all 64 tracks, 1024 clips, 1024 automation points and
4096 notes, compiles a standalone 256-node metered graph, renders stereo
odd-sized blocks and rejects capacity excesses. Project/GUI graphs remain capped
at 64 nodes. `test-stress` runs publication, allocation/retirement, capacity and
optional GUI/timer/redraw tests while two ordinary CPU workers compete;
subprocesses have timeouts and workers are always terminated. Existing callback
guards verify no host allocation/free/loading/unloading. This does not qualify
arbitrary plugins.

Callback work is bounded by documented node/frame/clip/note/event/point
capacities. Exceeding model/plan/mailbox capacities returns CAPACITY
transactionally; invalid or overlarge processing buffers are silenced and
reported, JACK xruns are counted. There is no CPU scheduler or preemption inside
a DSP callback: an expensive trusted plugin can miss a deadline or hang.
Automated LADSPA lanes currently use run(1) and bounded per-sample lane searches
and may be costly; they are excluded from the gain-chain baseline. Large WAV
memory limits remain those in AUDIO_FILES.md. Physical-device endurance,
CPU-overload sound quality and a full GUI latency benchmark remain unqualified.

---

## Qualification Workloads

The current `test-performance` emits one JSON record (the old prose output is
retired). Default: 2,000 stereo blocks at 48 kHz/128, 64 graph nodes, 16 tracks,
1,024 stored notes and 256 automation points, at most 32 active voices. This
changes the original workload; its historical numbers above are not a baseline
for this revision. `benchmark-project` runs nine cases: 1/16/64 tracks,
64/1,024/4,096 stored notes, 16/256/1,024 points and 31/128/1,024-frame buffers.
Two long notes per first 16 tracks remain audible. Other notes are scheduled
later to remain within the 32-voice plan limit. The position wraps every 96,000
samples to keep long runs active. Future notes exercise storage/preparation and
bounded scans rather than additional simultaneous synthesis. Each track has 16
gain points; graph processing uses the same 64-node chain in every case. The
graph matrix separately scales nodes and chain connections from 2 to 256. These
synthetic cases do not qualify every possible dense graph or plugin chain.

```sh
make benchmark benchmark-project
make benchmark-compare
make benchmark-project-compare
# Baselines must match the current CPU/compiler/flags and workload.
make benchmark-project-compare PROJECT_BENCHMARK_BASELINE=/path/to/report.json
./build/test_performance 10000 64 128
make test-benchmark-tools
```

Both matrices report wall-clock mean/p99/max processing duration and the count
of measured calls exceeding frames/sample-rate. These are offline deadline
exceedances, not actual JACK xruns: there is no server scheduling in the timing
loop. Graph compilation or project construction/preparation is timed separately.
`process_cpu_ms` measures CLOCK_PROCESS_CPUTIME_ID over the complete measured
loop, including timing and finite-output checks; it excludes preparation and
sorting. Project `cpu_audio_ratio` divides that CPU time by simulated audio
time. `peak_rss_kib` is Linux getrusage process high-water RSS, including model,
plans, measurement storage and executable/runtime overhead; it is not current
heap size or a leak test. Graph buffer bytes remain a separate exact buffer-size
estimate. Timing probes, memory queries, sorting and output occur outside engine
calls.

Checked-in `project-performance-baseline.json` provides initial project cases;
the original graph baseline is retained. `graph-performance-baseline.json` adds
a current graph report with CPU/RSS/deadline fields. Gates use the existing 1.5x
mean-time budget and 1 us noise allowance, with p99/max/misses and memory
reported for investigation. There is no universal hardware-independent deadline
or RSS pass threshold. `test-benchmark-tools` verifies same-context success,
regression rejection, missing cases and host/workload mismatch without relying
on machine timing. Do not replace a failing baseline without investigating.

---

## Endurance, Faults and Callback Audit

```sh
make test-soak test-stress test-stress-plugins
make test-stress STRESS_ROUNDS=10
./build/test_soak 10000000
make check-development
```

`test-soak` renders exactly 1,000,000 alternating 31/128/1,024-frame stereo
blocks: 2.282 hours of virtual 48 kHz audio. It checks every output sample,
processed block counts and final publication acknowledgement, with 9,999
quiescent position/rebuild cycles. It runs faster than wall time and models
position wrap/seek and restart publication; it is not a real device start/stop
or hardware endurance test. The project benchmark separately wraps active MIDI.

`test-stress` defaults to three rounds under two ordinary CPU workers. Each
round runs concurrent publication, allocation/lifetime guards, maximum model
capacity, project relocation/reload and missing-media retention, save-fault
diagnostics, and the soak test. With JACK headers present it also repeats the
deterministic backend fake's start/stop/seek, rate/buffer/server-loss and
failure/retry cases. Optional GUI tests run after those rounds under the same
CPU load. `test-stress-plugins` adds the original LADSPA fixture's
identity/layout/range, instantiation-error and invalid-metadata cases each
round; it needs LADSPA headers. No command changes a physical device, kills the
shared server or loads unknown plugins. Subprocess deadlines bound each test and
workers are cleaned up on exit.

The reliability executable now interposes fopen/fread/fwrite/read/write,
pthread_mutex_lock and pthread_cond_wait in addition to
malloc/calloc/realloc/free. Guarded host graph/realtime render calls fail
immediately on these operations. This complements the existing native lifetime
and plugin callback guards and control/callback ownership review in
RELIABILITY.md. It does not intercept every libc entry point, raw syscalls,
hidden dynamic-module calls, or GUI code by magic; third-party DSP remains
trusted. No known host callback safety violation was found.

`check-development` combines the ordinary build/tests, capacity/soak checks,
benchmark gate tests, static analysis and ASan/UBSan. The sanitizer target now
includes maximum capacity and virtual endurance. These are opt-in developer
checks requiring installed Clang/Python; no runtime dependency was added.
Sanitized/contended/profiled runs must not become timing baselines.

---

## Profiling Evidence and Remaining Costs

Before considering DSP changes, a 10,000-block, 64-track/128-frame project was
sampled with GCC `-O2 -g -pg` and GNU gprof on the documented baseline host.
Flat samples attributed about 30% to mixer automation `value`, 29% to MIDI
`segment`, and 11% to finite-value handling. Instrumentation and inlining affect
attribution; this identifies investigation candidates, not exact production cost
or evidence that checks should be removed. MIDI scans bounded stored-note arrays
and evaluates per-voice, per-sample gain/pan lanes. No DSP optimization was
introduced without a separate correctness/performance justification.

Reproduce in an isolated build (GNU gprof must already be installed):

```sh
make BUILD_DIR=build/profile CFLAGS='-O2 -g -pg -std=c99 -Iinclude' LDFLAGS=-pg build/profile/test_performance
GMON_OUT_PREFIX="$PWD/build/profile/gmon" ./build/profile/test_performance 10000 64 128
gprof build/profile/test_performance build/profile/gmon.* > build/profile/profile.txt
```

Remaining limits include per-sample LADSPA automation run(1), stored-note/clip
scans, preloaded WAV memory and immutable-plan/history overlap. This work does
not qualify physical-device changes, arbitrary plugin hangs/crashes, multi-hour
wall-clock sessions or GUI redraw latency. Virtual endurance and deterministic
faults provide repeatable software evidence, not a hard realtime guarantee.

Observed results (same Threadripper/GCC host, 300 blocks per case):
64-track/128-frame project mean 380.417 us, p99 720.271 us, prepare 11.841 ms,
CPU/audio .142 and peak process RSS 19,296 KiB. The 256-node stereo/128 graph
mean was 286.398 us, p99 484.712 us, compile .300 ms. These case runs observed
zero deadline exceedances. See JSON for all cases and host metadata.

The project comparison passed. Repeated graph comparisons flagged variable small
cases despite unchanged graph processing code; a 3,000-block run against the
retained baseline still flagged 16-node mono/31-frame mean 5.089 us above its
1.5x budget. The new graph report is evidence, not a replacement chosen to make
that comparison pass. This shared-host timing alert remains an investigation
item; no CPU/latency guarantee is inferred from the correctness suite. Longer
comparisons can use `BENCHMARK_BLOCKS=3000`.