xtc/docs/PERFORMANCE.md

Performance and realtime qualification

The original Phase 9 test-performance ran a hardware-independent 48 kHz, 128-frame stereo workload: 64 graph nodes, 4096 MIDI notes (32 simultaneous), 16 tracks and gain automation. Preparation and block timing use CLOCK_MONOTONIC outside the engine; callback code itself does not clock or log. The test checks finite output and reports mean, p99, maximum and CPU-time/audio-duration ratio. A run is a workload measurement, not a universal realtime threshold. Run more blocks with:

./build/test_performance 10000

Initial Phase 9 measurement on an AMD Ryzen Threadripper 1920X, GCC 16.2.1, -O2, 2000 blocks: preparation 3.045 ms, mean 308.301 us, p99 422.153 us, maximum 665.096 us, versus a 2666.667 us block deadline; CPU/audio ratio 0.1156. These are one host/run's observed values and depend on scheduling/load/compiler. The test excludes WAV disk preparation, external/native plugin complexity, GUI redraw, server scheduling and actual hardware latency. Native callbacks must provide their own bounded realtime implementation.

Existing software JACK loopback, stereo MIDI/save/reopen, record/reopen, restart/close and concurrent graph handoff tests remain available. Runtime fault injection covers loss/rate/xrun/null-buffer and lifecycle failures without altering the shared server. ASan/UBSan validate memory/error paths; TSan checks recorder/handoff races, not timing. Sanitized runs are not latency qualification.

Current worst-case storage includes up to 256 MiB preloaded mono WAV samples, paired graph buffers (128 MiB at 256 nodes/65536 frames; GUI limit 32 MiB), bounded MIDI/mix metadata and up to 33 metadata/graph history snapshots. Pending/retired plans can retain additional graph memory. Native preparation can allocate more; the API cannot enforce an extension's memory budget. There is no CPU overload scheduler, disk streaming, latency compensation or callback allocator audit of arbitrary third-party code. High-density interactive projects, physical devices, long-duration stereo/recording and live server termination remain unqualified.

Phase 19 matrix, capacity and contention checks

make benchmark records 24 warmed graph cases: 2/16/64/256 nodes, mono/stereo, 44.1k/31, 48k/128 and 96k/1024. Each reports control-thread compile time separately from 300 process calls, mean/p99/max, block deadline, mean load, and paired graph buffer bytes. Timing, sorting, allocation and logging happen outside processing. Metered gain chains test graph scaling; they do not represent expensive plugins. The checked-in performance-baseline.json records CPU/compiler/flags/kernel and all measurements. Copy a new run only after reviewing changes and conditions.

make benchmark test-performance test-scale test-stress
make benchmark-compare BENCHMARK_MAX_REGRESSION=1.5
# Select a baseline from this same host/compiler/flags:
make benchmark-compare BENCHMARK_BASELINE=/path/to/performance.json

Comparison rejects mismatched CPU/architecture/compiler/flags or missing cases. It fails if mean time exceeds both the ratio budget and a 1 microsecond absolute noise allowance. p99/max are reported for investigation, not universally gated. Repeat surprising results under comparable load; do not silently replace the baseline to make a regression disappear. Scheduling, clocks, thermal state and shared-machine contention still affect results. No real-time scheduling/pinning or external profiler is required. CPU-pressure runs are correctness checks, not comparable timing baselines. Tests/benchmark outputs live below BUILD_DIR.

Initial matrix (Threadripper 1920X, GCC 16.2.1, -O2, 300 blocks): 256-node stereo 48k/128 mean 353.214 us, p99 396.495 us, load .132; 96k/1024 mean 2478.520 us, p99 3092.434 us, load .232. The full JSON is the evidence. These met the local investigation target of mean load <.5 for ordinary graph chains; this is a host-specific development target, not a supported maximum CPU or latency promise.

The mixed 64-node/4096-note workload measured 340–357 us before removing unused live-controller processing and 313 us after (mean over 2000 blocks); p99 varied with system load. The measured hot-path change skips live voices/controller ramps when neither is active and avoids finished-ramp calls. Exact CC/odd-block tests continue to pass. No other graph optimization was justified by this measurement.

test-scale constructs all 64 tracks, 1024 clips, 1024 automation points and 4096 notes, compiles a standalone 256-node metered graph, renders stereo odd-sized blocks and rejects capacity excesses. Project/GUI graphs remain capped at 64 nodes. test-stress runs publication, allocation/retirement, capacity and optional GUI/timer/redraw tests while two ordinary CPU workers compete; subprocesses have timeouts and workers are always terminated. Existing callback guards verify no host allocation/free/loading/unloading. This does not qualify arbitrary plugins.

Callback work is bounded by documented node/frame/clip/note/event/point capacities. Exceeding model/plan/mailbox capacities returns CAPACITY transactionally; invalid or overlarge processing buffers are silenced and reported, JACK xruns are counted. There is no CPU scheduler or preemption inside a DSP callback: an expensive trusted plugin can miss a deadline or hang. Automated LADSPA lanes currently use run(1) and bounded per-sample lane searches and may be costly; they are excluded from the gain-chain baseline. Large WAV memory limits remain those in AUDIO_FILES.md. Physical-device endurance, CPU-overload sound quality and a full GUI latency benchmark remain unqualified.

Phase 29 qualification workloads

The current test-performance emits one JSON record (the old prose output is retired). Default: 2,000 stereo blocks at 48 kHz/128, 64 graph nodes, 16 tracks, 1,024 stored notes and 256 automation points, at most 32 active voices. This changes the Phase 9 workload; its historical numbers above are not a baseline for this revision. benchmark-project runs nine cases: 1/16/64 tracks, 64/1,024/4,096 stored notes, 16/256/1,024 points and 31/128/1,024-frame buffers. Two long notes per first 16 tracks remain audible. Other notes are scheduled later to remain within the 32-voice plan limit. The position wraps every 96,000 samples to keep long runs active. Future notes exercise storage/preparation and bounded scans rather than additional simultaneous synthesis. Each track has 16 gain points; graph processing uses the same 64-node chain in every case. The graph matrix separately scales nodes and chain connections from 2 to 256. These synthetic cases do not qualify every possible dense graph or plugin chain.

make benchmark benchmark-project
make benchmark-compare
make benchmark-project-compare
# Baselines must match the current CPU/compiler/flags and workload.
make benchmark-project-compare PROJECT_BENCHMARK_BASELINE=/path/to/report.json
./build/test_performance 10000 64 128
make test-benchmark-tools

Both matrices report wall-clock mean/p99/max processing duration and the count of measured calls exceeding frames/sample-rate. These are offline deadline exceedances, not actual JACK xruns: there is no server scheduling in the timing loop. Graph compilation or project construction/preparation is timed separately. process_cpu_ms measures CLOCK_PROCESS_CPUTIME_ID over the complete measured loop, including timing and finite-output checks; it excludes preparation and sorting. Project cpu_audio_ratio divides that CPU time by simulated audio time. peak_rss_kib is Linux getrusage process high-water RSS, including model, plans, measurement storage and executable/runtime overhead; it is not current heap size or a leak test. Graph buffer bytes remain a separate exact buffer-size estimate. Timing probes, memory queries, sorting and output occur outside engine calls.

Checked-in project-performance-baseline.json provides initial project cases; the original graph baseline is retained. graph-performance-phase29.json adds a current graph report with CPU/RSS/deadline fields. Gates use the existing 1.5x mean-time budget and 1 us noise allowance, with p99/max/misses and memory reported for investigation. There is no universal hardware-independent deadline or RSS pass threshold. test-benchmark-tools verifies same-context success, regression rejection, missing cases and host/workload mismatch without relying on machine timing. Do not replace a failing baseline without investigating.

Endurance, faults and callback audit

make test-soak test-stress test-stress-plugins
make test-stress STRESS_ROUNDS=10
./build/test_soak 10000000
make check-development

test-soak renders exactly 1,000,000 alternating 31/128/1,024-frame stereo blocks: 2.282 hours of virtual 48 kHz audio. It checks every output sample, processed block counts and final publication acknowledgement, with 9,999 quiescent position/rebuild cycles. It runs faster than wall time and models position wrap/seek and restart publication; it is not a real device start/stop or hardware endurance test. The project benchmark separately wraps active MIDI.

test-stress defaults to three rounds under two ordinary CPU workers. Each round runs concurrent publication, allocation/lifetime guards, maximum model capacity, project relocation/reload and missing-media retention, save-fault diagnostics, and the soak test. With JACK headers present it also repeats the deterministic backend fake's start/stop/seek, rate/buffer/server-loss and failure/retry cases. Optional GUI tests run after those rounds under the same CPU load. test-stress-plugins adds the original LADSPA fixture's identity/layout/range, instantiation-error and invalid-metadata cases each round; it needs LADSPA headers. No command changes a physical device, kills the shared server or loads unknown plugins. Subprocess deadlines bound each test and workers are cleaned up on exit.

The reliability executable now interposes fopen/fread/fwrite/read/write, pthread_mutex_lock and pthread_cond_wait in addition to malloc/calloc/realloc/free. Guarded host graph/realtime render calls fail immediately on these operations. This complements the existing native lifetime and plugin callback guards and control/callback ownership review in RELIABILITY.md. It does not intercept every libc entry point, raw syscalls, hidden dynamic-module calls, or GUI code by magic; third-party DSP remains trusted. No known host callback safety violation was found.

check-development combines the ordinary build/tests, capacity/soak checks, benchmark gate tests, static analysis and ASan/UBSan. The sanitizer target now includes maximum capacity and virtual endurance. These are opt-in developer checks requiring installed Clang/Python; no runtime dependency was added. Sanitized/contended/profiled runs must not become timing baselines.

Profiling evidence and remaining costs

Before considering DSP changes, Phase 29 sampled a 10,000-block, 64-track/128-frame project with GCC -O2 -g -pg and GNU gprof on the documented baseline host. Flat samples attributed about 30% to mixer automation value, 29% to MIDI segment, and 11% to finite-value handling. Instrumentation and inlining affect attribution; this identifies investigation candidates, not exact production cost or evidence that checks should be removed. MIDI scans bounded stored-note arrays and evaluates per-voice, per-sample gain/pan lanes. No DSP optimization was introduced without a separate correctness/performance justification.

Reproduce in an isolated build (GNU gprof must already be installed):

make BUILD_DIR=build/profile CFLAGS='-O2 -g -pg -std=c99 -Iinclude' LDFLAGS=-pg build/profile/test_performance
GMON_OUT_PREFIX="$PWD/build/profile/gmon" ./build/profile/test_performance 10000 64 128
gprof build/profile/test_performance build/profile/gmon.* > build/profile/profile.txt

Remaining limits include per-sample LADSPA automation run(1), stored-note/clip scans, preloaded WAV memory and immutable-plan/history overlap. This phase does not qualify physical-device changes, arbitrary plugin hangs/crashes, multi-hour wall-clock sessions or GUI redraw latency. Virtual endurance and deterministic faults provide repeatable software evidence, not a hard realtime guarantee.

Phase 29 observed results (same Threadripper/GCC host, 300 blocks per case): 64-track/128-frame project mean 380.417 us, p99 720.271 us, prepare 11.841 ms, CPU/audio .142 and peak process RSS 19,296 KiB. The 256-node stereo/128 graph mean was 286.398 us, p99 484.712 us, compile .300 ms. These case runs observed zero deadline exceedances. See JSON for all cases and host metadata.

The project comparison passed. Repeated graph comparisons flagged variable small cases despite unchanged graph processing code; a 3,000-block run against the retained Phase 19 baseline still flagged 16-node mono/31-frame mean 5.089 us above its 1.5x budget. The new graph report is evidence, not a replacement chosen to make that comparison pass. This shared-host timing alert remains an investigation item; no CPU/latency guarantee is inferred from the correctness suite. Longer comparisons can use BENCHMARK_BLOCKS=3000.