xtc/docs/RELIABILITY.md

Engine correctness and reliability

Phase 10 audits the existing engine rather than adding musical features. The review covers graph edits/compilation, offline DSP, plan publication, JACK callbacks/lifecycle, audio/MIDI source plans and the recorder handoff.

Ownership and synchronization

Resource	Owner and lifetime
Editable graph, project, history and asset paths	One serialized control thread (the GUI thread in the application). Borrowed model pointers expire on mutation/destruction. Never traverse these objects in the callback.
Compiled graph routing/specifications	Copied during compilation; independent of the editable graph. Immutable after publication. All nodes, including disconnected nodes, are scheduled; only explicit outputs (or compatibility terminals) are mixed.
DSP buffers, native instances, phases, meter peaks and graph-local clock	One processing consumer. Reset and meter reads require that owner or quiescence; these are not atomic GUI snapshots. Native callback code must remain loaded until all instances are destroyed.
Pending graph	Single control producer publishes with release; the consumer acquires it at a block boundary. The revision is read before releasing the pending slot, preventing producer reuse of its metadata. A full pending slot rejects submission.
Active graph	Consumer only until the consumer is stopped/joined. No producer access to mutable DSP state.
Retired graph	Consumer releases only after its final access. Control acquires/exchanges the slot and destroys the plan. If the slot is occupied, adoption waits while processing the current graph continues. The callback never destroys plans.
Realtime status	Individual atomic counters/revisions; a status read is not a transaction spanning every field. Required pointer/integer atomics must be always lock-free.
JACK client, ports and backend control fields	Serialized control thread. JACK owns callback scheduling. Successful deactivate establishes processing quiescence; successful close synchronizes callbacks before freeing context. Failed deactivate/close preserves ownership for retry. Server-loss notification alone is not permission to free context.
Preloaded audio/MIDI plans and source buffers	Backend owns them; prepare/replace while stopped, callback alone renders while running. MIDI voices/panic state follow the same rule; panic requests use atomics.
Recorder	Caller owns it; backend borrows it. Attach/detach only while stopped. Audio is sole ring producer, control is sole drain/file writer. Acquire/release indices protect samples; errors are sticky. Detach/quiesce before finish/destroy.
Widgets and GUI timers	GUI thread only; status polling and retired-plan collection occur there, never through FLTK calls from audio.

The public realtime API is SPSC: multiple producers, simultaneous process calls, collection from arbitrary threads, and destruction during processing are not supported. Atomics do not remove those ownership requirements.

Bounded callback work

Compilation, topological sorting, asset I/O, allocation and native preparation are control-thread operations. Graph processing traverses at most 256 nodes, 65536 frames per accepted block and the precomputed edges (at most 256 squared storage entries). Graph edits validate port direction/type/layout and reject cycles transactionally using Kahn ordering; fixed public mono ports have no implicit event/audio conversions. Stereo uses two mono graph passes, independent native instances and one oscillator phase advance per frame.

Audio regions, MIDI spans, voices, events and automation are bounded by public project/MIDI capacities. Rendering scans bounded region/note arrays; automation uses binary searches in prepared lanes. JACK MIDI accepts at most 256 events per block; overflow requests panic and reports drops. Recorder push performs at most one ring-capacity-sized copy and never drains a file. No callback path waits for a producer, collector, GUI or disk. Realtime pointer handoff has no retry loop.

Oversized JACK blocks are silenced without graph/source processing. Clearing backend-provided output and computing peaks still costs the server's frame count; the server's buffer contract is trusted. Public render APIs likewise require caller-provided valid buffers. Lock-free does not promise a hardware latency bound for atomic instructions, libc memory functions or libm. Native callbacks are trusted: the host cannot prevent their allocation, blocking, crashes or excessive runtime. Maximum supported capacities are safety bounds, not a promise that every workload meets its audio deadline.

Findings and regression coverage

Graph rendering previously mixed the left output before reading right-channel sources. With external input equal to the left output, the right pass could read already processed samples (gain applied twice). Both passes now finish source reads before either output is written. Tests check mono input aliasing and cross-channel source aliasing against exact expected samples. Output buffers must not overlap each other. A redundant assignment in missing-asset preparation was also removed after Clang's dead-store diagnostic; behavior is unchanged.

test_reliability uses GNU-compatible linker allocation interposition in the test executable only. It rejects malloc/calloc/realloc/free during rendering, fails each preparation allocation in turn until compilation succeeds, verifies that failed replacement preserves output and revisions, and counts both stereo native instances through retirement, collection and destruction of a pending plan. It repeats this lifecycle 100 times. A maximum 256-node reverse-insertion chain checks dependency scheduling, long-cycle rejection/rollback and compiled snapshot independence. The pthread handoff test submits 2000 native graphs while processing, checks whole-block output/revision consistency, asserts that prepare/destruction run on the control thread, and balances all native lifetimes after joining the consumer.

Existing backend fault tests cover repeated start/stop, partial registration, callback setup, open/activate/deactivate/close failures and retries, server loss, rate/buffer changes, missing buffers, MIDI disconnect/panic, and recorder detach. Other suites cover deterministic variable-block DSP, finite-output handling, project errors, record-ring saturation and concurrent draining, and crash/write failure preservation. No critical realtime safety defect remains known within these contracts; these tests are evidence for tested paths, not a formal proof.

Reproducing the audit

From the repository root:

make BUILD_DIR=build/audit all gui test-reliability test-recovery
make BUILD_DIR=build/audit test-sanitize
make analyze
make BUILD_DIR=build/audit test-gui test-gui-jack test-jack test-jack-audio test-jack-midi
make BUILD_DIR=build/nojack PKG_CONFIG=false all test-reliability gui

test-sanitize needs installed Clang and runs core, allocation-failure and recovery tests with ASan/UBSan and Linux's default ASan leak checking. analyze uses Clang's path-sensitive analyzer on every core translation unit, including JACK when its development files are available, and fails on diagnostics. Python and Clang are development tools, not new runtime dependencies. The allocation guard requires a GNU-compatible linker. Live tests require an existing JACK-compatible server; GUI tests use installed FLTK and Xvfb. No target installs dependencies or changes server/system configuration.

Phase 10 verification on GCC 16.2.1 / Clang 23.1.3 passed normal and no-JACK builds, all core tests, ASan/UBSan/leak checks, Clang static analysis, GUI/mock and live JACK integrations, plus GCC TSan on the pthread native-publication and recorder concurrency tests. The GUI also passed ASan/UBSan with the existing tests/integration/lsan.supp fontconfig-only suppression; core tests use no leak suppression. The live graph test observed both replacement gains, ten restart cycles and zero xruns during its five-second software run.

Physical audio devices, hard realtime scheduling, multi-hour endurance and arbitrary third-party native callbacks remain unqualified. Sanitizers change timing and should not be used to measure production realtime performance. See PERFORMANCE.md for workload measurements, RECOVERY.md for process-crash limits, and NATIVE_API.md for the trusted extension boundary.

Phase 12 transition ownership

Source fade configuration is written before publication and immutable during rendering. The master ramp and stop-observed flag belong to the audio consumer while active; start initializes them only while quiescent. Control publishes an atomic stop request; audio publishes completion after the last ramp sample. Collection acquires completion and deactivates the backend before resetting voices/flags. A completed callback continues silence/holds position until that deactivation. No callback waits or calls JACK control functions. Immediate stop/close handles faults or missing callbacks. See AUDIO_QUALITY.md.

Phase 29 repeatable development checks

make check-development combines normal tests, capacity/endurance, deterministic benchmark-gate tests, static analysis and ASan/UBSan (including scale and soak). make test-stress repeats project/backend/publication faults under CPU contention; make test-stress-plugins adds the original LADSPA failure fixture. The reliability callback guard now also rejects common host file I/O and blocking mutex/condition calls. It audits directly linked host code, not arbitrary dynamic-plugin internals. See PERFORMANCE.md for exact workloads, metrics, profiles and qualification limits.