Phase 29: add project benchmarks, endurance stress and callback safety checks
This commit is contained in:
parent
ca6c869b0b
commit
afc049e2c8
14 changed files with 1032 additions and 43 deletions
|
|
@ -1,6 +1,6 @@
|
|||
# Performance and realtime qualification
|
||||
|
||||
`make test-performance` runs a hardware-independent 48 kHz, 128-frame stereo
|
||||
The original Phase 9 `test-performance` ran a hardware-independent 48 kHz, 128-frame stereo
|
||||
workload: 64 graph nodes, 4096 MIDI notes (32 simultaneous), 16 tracks and gain
|
||||
automation. Preparation and block timing use CLOCK_MONOTONIC outside the engine;
|
||||
callback code itself does not clock or log. The test checks finite output and
|
||||
|
|
@ -89,3 +89,130 @@ and bounded per-sample lane searches and may be costly; they are excluded from
|
|||
the gain-chain baseline. Large WAV memory limits remain those in AUDIO_FILES.md.
|
||||
Physical-device endurance, CPU-overload sound quality and a full GUI latency
|
||||
benchmark remain unqualified.
|
||||
|
||||
## Phase 29 qualification workloads
|
||||
|
||||
The current `test-performance` emits one JSON record (the old prose output is
|
||||
retired). Default: 2,000 stereo blocks at 48 kHz/128, 64 graph nodes, 16 tracks,
|
||||
1,024 stored notes and 256 automation points, at most 32 active voices. This
|
||||
changes the Phase 9 workload; its historical numbers above are not a baseline
|
||||
for this revision. `benchmark-project` runs nine cases: 1/16/64 tracks,
|
||||
64/1,024/4,096 stored notes, 16/256/1,024 points and 31/128/1,024-frame buffers.
|
||||
Two long notes per first 16 tracks remain audible. Other notes are scheduled
|
||||
later to remain within the 32-voice plan limit. The position wraps every 96,000
|
||||
samples to keep long runs active. Future notes exercise storage/preparation and
|
||||
bounded scans rather than additional simultaneous synthesis. Each track has
|
||||
16 gain points; graph processing uses the same 64-node chain in every case.
|
||||
The graph matrix separately scales nodes and chain connections from 2 to 256.
|
||||
These synthetic cases do not qualify every possible dense graph or plugin chain.
|
||||
|
||||
```sh
|
||||
make benchmark benchmark-project
|
||||
make benchmark-compare
|
||||
make benchmark-project-compare
|
||||
# Baselines must match the current CPU/compiler/flags and workload.
|
||||
make benchmark-project-compare PROJECT_BENCHMARK_BASELINE=/path/to/report.json
|
||||
./build/test_performance 10000 64 128
|
||||
make test-benchmark-tools
|
||||
```
|
||||
|
||||
Both matrices report wall-clock mean/p99/max processing duration and the count
|
||||
of measured calls exceeding frames/sample-rate. These are offline deadline
|
||||
exceedances, not actual JACK xruns: there is no server scheduling in the timing
|
||||
loop. Graph compilation or project construction/preparation is timed separately.
|
||||
`process_cpu_ms` measures CLOCK_PROCESS_CPUTIME_ID over the complete measured
|
||||
loop, including timing and finite-output checks; it excludes preparation and
|
||||
sorting. Project `cpu_audio_ratio` divides that CPU time by simulated audio time.
|
||||
`peak_rss_kib` is Linux getrusage process high-water RSS, including model, plans,
|
||||
measurement storage and executable/runtime overhead; it is not current heap size
|
||||
or a leak test. Graph buffer bytes remain a separate exact buffer-size estimate.
|
||||
Timing probes, memory queries, sorting and output occur outside engine calls.
|
||||
|
||||
Checked-in `project-performance-baseline.json` provides initial project cases;
|
||||
the original graph baseline is retained. `graph-performance-phase29.json` adds
|
||||
a current graph report with CPU/RSS/deadline fields. Gates use the existing
|
||||
1.5x mean-time budget and 1 us noise allowance, with p99/max/misses and memory
|
||||
reported for investigation. There is no universal hardware-independent deadline
|
||||
or RSS pass threshold. `test-benchmark-tools` verifies same-context success,
|
||||
regression rejection, missing cases and host/workload mismatch without relying
|
||||
on machine timing. Do not replace a failing baseline without investigating.
|
||||
|
||||
## Endurance, faults and callback audit
|
||||
|
||||
```sh
|
||||
make test-soak test-stress test-stress-plugins
|
||||
make test-stress STRESS_ROUNDS=10
|
||||
./build/test_soak 10000000
|
||||
make check-development
|
||||
```
|
||||
|
||||
`test-soak` renders exactly 1,000,000 alternating 31/128/1,024-frame stereo
|
||||
blocks: 2.282 hours of virtual 48 kHz audio. It checks every output sample,
|
||||
processed block counts and final publication acknowledgement, with 9,999
|
||||
quiescent position/rebuild cycles. It runs faster than wall time and models
|
||||
position wrap/seek and restart publication; it is not a real device start/stop
|
||||
or hardware endurance test. The project benchmark separately wraps active MIDI.
|
||||
|
||||
`test-stress` defaults to three rounds under two ordinary CPU workers. Each round
|
||||
runs concurrent publication, allocation/lifetime guards, maximum model capacity,
|
||||
project relocation/reload and missing-media retention, save-fault diagnostics,
|
||||
and the soak test. With JACK headers present it also repeats the deterministic
|
||||
backend fake's start/stop/seek, rate/buffer/server-loss and failure/retry cases.
|
||||
Optional GUI tests run after those rounds under the same CPU load.
|
||||
`test-stress-plugins` adds the original LADSPA fixture's identity/layout/range,
|
||||
instantiation-error and invalid-metadata cases each round; it needs LADSPA headers.
|
||||
No command changes a physical device, kills the shared server or loads unknown
|
||||
plugins. Subprocess deadlines bound each test and workers are cleaned up on exit.
|
||||
|
||||
The reliability executable now interposes fopen/fread/fwrite/read/write,
|
||||
pthread_mutex_lock and pthread_cond_wait in addition to malloc/calloc/realloc/free.
|
||||
Guarded host graph/realtime render calls fail immediately on these operations.
|
||||
This complements the existing native lifetime and plugin callback guards and
|
||||
control/callback ownership review in RELIABILITY.md. It does not intercept every
|
||||
libc entry point, raw syscalls, hidden dynamic-module calls, or GUI code by magic;
|
||||
third-party DSP remains trusted. No known host callback safety violation was found.
|
||||
|
||||
`check-development` combines the ordinary build/tests, capacity/soak checks,
|
||||
benchmark gate tests, static analysis and ASan/UBSan. The sanitizer target now
|
||||
includes maximum capacity and virtual endurance. These are opt-in developer
|
||||
checks requiring installed Clang/Python; no runtime dependency was added.
|
||||
Sanitized/contended/profiled runs must not become timing baselines.
|
||||
|
||||
## Profiling evidence and remaining costs
|
||||
|
||||
Before considering DSP changes, Phase 29 sampled a 10,000-block, 64-track/128-frame
|
||||
project with GCC `-O2 -g -pg` and GNU gprof on the documented baseline host.
|
||||
Flat samples attributed about 30% to mixer automation `value`, 29% to MIDI
|
||||
`segment`, and 11% to finite-value handling. Instrumentation and inlining affect
|
||||
attribution; this identifies investigation candidates, not exact production
|
||||
cost or evidence that checks should be removed. MIDI scans bounded stored-note
|
||||
arrays and evaluates per-voice, per-sample gain/pan lanes. No DSP optimization
|
||||
was introduced without a separate correctness/performance justification.
|
||||
|
||||
Reproduce in an isolated build (GNU gprof must already be installed):
|
||||
|
||||
```sh
|
||||
make BUILD_DIR=build/profile CFLAGS='-O2 -g -pg -std=c99 -Iinclude' LDFLAGS=-pg build/profile/test_performance
|
||||
GMON_OUT_PREFIX="$PWD/build/profile/gmon" ./build/profile/test_performance 10000 64 128
|
||||
gprof build/profile/test_performance build/profile/gmon.* > build/profile/profile.txt
|
||||
```
|
||||
|
||||
Remaining limits include per-sample LADSPA automation run(1), stored-note/clip
|
||||
scans, preloaded WAV memory and immutable-plan/history overlap. This phase does
|
||||
not qualify physical-device changes, arbitrary plugin hangs/crashes, multi-hour
|
||||
wall-clock sessions or GUI redraw latency. Virtual endurance and deterministic
|
||||
faults provide repeatable software evidence, not a hard realtime guarantee.
|
||||
|
||||
Phase 29 observed results (same Threadripper/GCC host, 300 blocks per case):
|
||||
64-track/128-frame project mean 380.417 us, p99 720.271 us, prepare 11.841 ms,
|
||||
CPU/audio .142 and peak process RSS 19,296 KiB. The 256-node stereo/128 graph
|
||||
mean was 286.398 us, p99 484.712 us, compile .300 ms. These case runs observed
|
||||
zero deadline exceedances. See JSON for all cases and host metadata.
|
||||
|
||||
The project comparison passed. Repeated graph comparisons flagged variable
|
||||
small cases despite unchanged graph processing code; a 3,000-block run against
|
||||
the retained Phase 19 baseline still flagged 16-node mono/31-frame mean 5.089 us
|
||||
above its 1.5x budget. The new graph report is evidence, not a replacement chosen
|
||||
to make that comparison pass. This shared-host timing alert remains an
|
||||
investigation item; no CPU/latency guarantee is inferred from the correctness
|
||||
suite. Longer comparisons can use `BENCHMARK_BLOCKS=3000`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue