xtc/docs/RELIABILITY.md

# Engine correctness and reliability

Phase 10 audits the existing engine rather than adding musical features. The
review covers graph edits/compilation, offline DSP, plan publication, JACK
callbacks/lifecycle, audio/MIDI source plans and the recorder handoff.

## Ownership and synchronization

| Resource | Owner and lifetime |
| --- | --- |
| Editable graph, project, history and asset paths | One serialized control thread (the GUI thread in the application). Borrowed model pointers expire on mutation/destruction. Never traverse these objects in the callback. |
| Compiled graph routing/specifications | Copied during compilation; independent of the editable graph. Immutable after publication. All nodes, including disconnected nodes, are scheduled; only explicit outputs (or compatibility terminals) are mixed. |
| DSP buffers, native instances, phases, meter peaks and graph-local clock | One processing consumer. Reset and meter reads require that owner or quiescence; these are not atomic GUI snapshots. Native callback code must remain loaded until all instances are destroyed. |
| Pending graph | Single control producer publishes with release; the consumer acquires it at a block boundary. The revision is read before releasing the pending slot, preventing producer reuse of its metadata. A full pending slot rejects submission. |
| Active graph | Consumer only until the consumer is stopped/joined. No producer access to mutable DSP state. |
| Retired graph | Consumer releases only after its final access. Control acquires/exchanges the slot and destroys the plan. If the slot is occupied, adoption waits while processing the current graph continues. The callback never destroys plans. |
| Realtime status | Individual atomic counters/revisions; a status read is not a transaction spanning every field. Required pointer/integer atomics must be always lock-free. |
| JACK client, ports and backend control fields | Serialized control thread. JACK owns callback scheduling. Successful deactivate establishes processing quiescence; successful close synchronizes callbacks before freeing context. Failed deactivate/close preserves ownership for retry. Server-loss notification alone is not permission to free context. |
| Preloaded audio/MIDI plans and source buffers | Backend owns them; prepare/replace while stopped, callback alone renders while running. MIDI voices/panic state follow the same rule; panic requests use atomics. |
| Recorder | Caller owns it; backend borrows it. Attach/detach only while stopped. Audio is sole ring producer, control is sole drain/file writer. Acquire/release indices protect samples; errors are sticky. Detach/quiesce before finish/destroy. |
| Widgets and GUI timers | GUI thread only; status polling and retired-plan collection occur there, never through FLTK calls from audio. |

The public realtime API is SPSC: multiple producers, simultaneous process calls,
collection from arbitrary threads, and destruction during processing are not
supported. Atomics do not remove those ownership requirements.

## Bounded callback work

Compilation, topological sorting, asset I/O, allocation and native preparation
are control-thread operations. Graph processing traverses at most 256 nodes,
65536 frames per accepted block and the precomputed edges (at most 256 squared
storage entries). Graph edits validate port direction/type/layout and reject
cycles transactionally using Kahn ordering; fixed public mono ports have no
implicit event/audio conversions. Stereo uses two mono graph passes, independent
native instances and one oscillator phase advance per frame.

Audio regions, MIDI spans, voices, events and automation are bounded by public
project/MIDI capacities. Rendering scans bounded region/note arrays; automation
uses binary searches in prepared lanes. JACK MIDI accepts at most 256 events per
block; overflow requests panic and reports drops. Recorder push performs at most
one ring-capacity-sized copy and never drains a file. No callback path waits for
a producer, collector, GUI or disk. Realtime pointer handoff has no retry loop.

Oversized JACK blocks are silenced without graph/source processing. Clearing
backend-provided output and computing peaks still costs the server's frame
count; the server's buffer contract is trusted. Public render APIs likewise
require caller-provided valid buffers. Lock-free does not promise a hardware
latency bound for atomic instructions, libc memory functions or libm. Native
callbacks are trusted: the host cannot prevent their allocation, blocking,
crashes or excessive runtime. Maximum supported capacities are safety bounds,
not a promise that every workload meets its audio deadline.

## Findings and regression coverage

Graph rendering previously mixed the left output before reading right-channel
sources. With external input equal to the left output, the right pass could
read already processed samples (gain applied twice). Both passes now finish
source reads before either output is written. Tests check mono input aliasing
and cross-channel source aliasing against exact expected samples. Output buffers
must not overlap each other. A redundant assignment in missing-asset preparation
was also removed after Clang's dead-store diagnostic; behavior is unchanged.

`test_reliability` uses GNU-compatible linker allocation interposition in the
test executable only. It rejects malloc/calloc/realloc/free during rendering,
fails each preparation allocation in turn until compilation succeeds, verifies
that failed replacement preserves output and revisions, and counts both stereo
native instances through retirement, collection and destruction of a pending
plan. It repeats this lifecycle 100 times. A maximum 256-node reverse-insertion
chain checks dependency scheduling, long-cycle rejection/rollback and compiled
snapshot independence. The pthread handoff test submits 2000 native graphs
while processing, checks whole-block output/revision consistency, asserts that
prepare/destruction run on the control thread, and balances all native lifetimes
after joining the consumer.

Existing backend fault tests cover repeated start/stop, partial registration,
callback setup, open/activate/deactivate/close failures and retries, server loss,
rate/buffer changes, missing buffers, MIDI disconnect/panic, and recorder detach.
Other suites cover deterministic variable-block DSP, finite-output handling,
project errors, record-ring saturation and concurrent draining, and crash/write
failure preservation. No critical realtime safety defect remains known within
these contracts; these tests are evidence for tested paths, not a formal proof.

## Reproducing the audit

From the repository root:

```sh
make BUILD_DIR=build/audit all gui test-reliability test-recovery
make BUILD_DIR=build/audit test-sanitize
make analyze
make BUILD_DIR=build/audit test-gui test-gui-jack test-jack test-jack-audio test-jack-midi
make BUILD_DIR=build/nojack PKG_CONFIG=false all test-reliability gui
```

`test-sanitize` needs installed Clang and runs core, allocation-failure and recovery
tests with ASan/UBSan and Linux's default ASan leak checking. `analyze` uses
Clang's path-sensitive analyzer on every core translation unit, including JACK
when its development files are available, and fails on diagnostics. Python and
Clang are development tools, not new runtime dependencies. The allocation guard
requires a GNU-compatible linker. Live tests require an existing JACK-compatible
server; GUI tests use installed FLTK and Xvfb. No target installs dependencies or
changes server/system configuration.

Phase 10 verification on GCC 16.2.1 / Clang 23.1.3 passed normal and no-JACK builds,
all core tests, ASan/UBSan/leak checks, Clang static analysis, GUI/mock and live
JACK integrations, plus GCC TSan on the pthread native-publication and recorder
concurrency tests. The GUI also passed ASan/UBSan with the existing
`tests/integration/lsan.supp` fontconfig-only suppression; core tests use no leak
suppression. The live graph test observed both replacement gains, ten restart
cycles and zero xruns during its five-second software run.

Physical audio devices, hard realtime scheduling, multi-hour endurance and
arbitrary third-party native callbacks remain unqualified. Sanitizers change
timing and should not be used to measure production realtime performance. See
PERFORMANCE.md for workload measurements, RECOVERY.md for process-crash limits,
and NATIVE_API.md for the trusted extension boundary.

## Phase 12 transition ownership

Source fade configuration is written before publication and immutable during
rendering. The master ramp and stop-observed flag belong to the audio consumer
while active; start initializes them only while quiescent. Control publishes an
atomic stop request; audio publishes completion after the last ramp sample.
Collection acquires completion and deactivates the backend before resetting
voices/flags. A completed callback continues silence/holds position until that
deactivation. No callback waits or calls JACK control functions. Immediate
stop/close handles faults or missing callbacks. See AUDIO_QUALITY.md.

## Phase 29 repeatable development checks

`make check-development` combines normal tests, capacity/endurance, deterministic
benchmark-gate tests, static analysis and ASan/UBSan (including scale and soak).
`make test-stress` repeats project/backend/publication faults under CPU contention;
`make test-stress-plugins` adds the original LADSPA failure fixture. The reliability
callback guard now also rejects common host file I/O and blocking mutex/condition
calls. It audits directly linked host code, not arbitrary dynamic-plugin internals.
See PERFORMANCE.md for exact workloads, metrics, profiles and qualification limits.