Phase 10: Audit engine reliability and strengthen lifecycle regression tests

docs/AUDIO_ENGINE.md

@@ -273,3 +273,10 @@ processing. Callback errors silence the block and reset the failing processor,
 without rolling back already advanced DSP. Native event instruments can render
 explicitly into graph instrument buffers. See NATIVE_API.md for exact lifecycle
 and serialization limits and PERFORMANCE.md for measured workload and budgets.
+
+## Phase 10 ownership audit
+
+[RELIABILITY.md](RELIABILITY.md) records resource owners, publication and
+reclamation ordering, callback work bounds and repeatable diagnostic tests.
+Graph stereo rendering completes both channels before writing caller outputs;
+either output may equal a source, but the output buffers must not overlap.

docs/BUILD.md

@@ -222,3 +222,13 @@ No-JACK core/GUI/install checks and Clang ASan/UBSan core/GUI checks also passed
 (the GUI retains only its existing fontconfig leak suppression). TSan recorder
 and realtime-handoff checks passed. Two independent optimized source exports
 and their repacked source archives were byte-identical on this toolchain.
+
+## Phase 10 reliability audit
+
+Run `make test-reliability` for the core suites, allocation-failure sweep,
+callback allocation/free guard and native reclamation checks. Run
+`make test-sanitize` for Clang ASan/UBSan/leak checks and `make analyze` for
+Clang static analysis of all core sources. These need installed development
+tools; the allocation guard uses a GNU-compatible linker. See
+[RELIABILITY.md](RELIABILITY.md) for ownership, audit findings,
+full verification commands and remaining limits.

docs/RELIABILITY.md

@@ -0,0 +1,116 @@
+# Engine correctness and reliability
+
+Phase 10 audits the existing engine rather than adding musical features. The
+review covers graph edits/compilation, offline DSP, plan publication, JACK
+callbacks/lifecycle, audio/MIDI source plans and the recorder handoff.
+
+## Ownership and synchronization
+
+| Resource | Owner and lifetime |
+| --- | --- |
+| Editable graph, project, history and asset paths | One serialized control thread (the GUI thread in the application). Borrowed model pointers expire on mutation/destruction. Never traverse these objects in the callback. |
+| Compiled graph routing/specifications | Copied during compilation; independent of the editable graph. Immutable after publication. All nodes, including disconnected nodes, are scheduled; only explicit outputs (or compatibility terminals) are mixed. |
+| DSP buffers, native instances, phases, meter peaks and graph-local clock | One processing consumer. Reset and meter reads require that owner or quiescence; these are not atomic GUI snapshots. Native callback code must remain loaded until all instances are destroyed. |
+| Pending graph | Single control producer publishes with release; the consumer acquires it at a block boundary. The revision is read before releasing the pending slot, preventing producer reuse of its metadata. A full pending slot rejects submission. |
+| Active graph | Consumer only until the consumer is stopped/joined. No producer access to mutable DSP state. |
+| Retired graph | Consumer releases only after its final access. Control acquires/exchanges the slot and destroys the plan. If the slot is occupied, adoption waits while processing the current graph continues. The callback never destroys plans. |
+| Realtime status | Individual atomic counters/revisions; a status read is not a transaction spanning every field. Required pointer/integer atomics must be always lock-free. |
+| JACK client, ports and backend control fields | Serialized control thread. JACK owns callback scheduling. Successful deactivate establishes processing quiescence; successful close synchronizes callbacks before freeing context. Failed deactivate/close preserves ownership for retry. Server-loss notification alone is not permission to free context. |
+| Preloaded audio/MIDI plans and source buffers | Backend owns them; prepare/replace while stopped, callback alone renders while running. MIDI voices/panic state follow the same rule; panic requests use atomics. |
+| Recorder | Caller owns it; backend borrows it. Attach/detach only while stopped. Audio is sole ring producer, control is sole drain/file writer. Acquire/release indices protect samples; errors are sticky. Detach/quiesce before finish/destroy. |
+| Widgets and GUI timers | GUI thread only; status polling and retired-plan collection occur there, never through FLTK calls from audio. |
+
+The public realtime API is SPSC: multiple producers, simultaneous process calls,
+collection from arbitrary threads, and destruction during processing are not
+supported. Atomics do not remove those ownership requirements.
+
+## Bounded callback work
+
+Compilation, topological sorting, asset I/O, allocation and native preparation
+are control-thread operations. Graph processing traverses at most 256 nodes,
+65536 frames per accepted block and the precomputed edges (at most 256 squared
+storage entries). Graph edits validate port direction/type/layout and reject
+cycles transactionally using Kahn ordering; fixed public mono ports have no
+implicit event/audio conversions. Stereo uses two mono graph passes, independent
+native instances and one oscillator phase advance per frame.
+
+Audio regions, MIDI spans, voices, events and automation are bounded by public
+project/MIDI capacities. Rendering scans bounded region/note arrays; automation
+uses binary searches in prepared lanes. JACK MIDI accepts at most 256 events per
+block; overflow requests panic and reports drops. Recorder push performs at most
+one ring-capacity-sized copy and never drains a file. No callback path waits for
+a producer, collector, GUI or disk. Realtime pointer handoff has no retry loop.
+
+Oversized JACK blocks are silenced without graph/source processing. Clearing
+backend-provided output and computing peaks still costs the server's frame
+count; the server's buffer contract is trusted. Public render APIs likewise
+require caller-provided valid buffers. Lock-free does not promise a hardware
+latency bound for atomic instructions, libc memory functions or libm. Native
+callbacks are trusted: the host cannot prevent their allocation, blocking,
+crashes or excessive runtime. Maximum supported capacities are safety bounds,
+not a promise that every workload meets its audio deadline.
+
+## Findings and regression coverage
+
+Graph rendering previously mixed the left output before reading right-channel
+sources. With external input equal to the left output, the right pass could
+read already processed samples (gain applied twice). Both passes now finish
+source reads before either output is written. Tests check mono input aliasing
+and cross-channel source aliasing against exact expected samples. Output buffers
+must not overlap each other. A redundant assignment in missing-asset preparation
+was also removed after Clang's dead-store diagnostic; behavior is unchanged.
+
+`test_reliability` uses GNU-compatible linker allocation interposition in the
+test executable only. It rejects malloc/calloc/realloc/free during rendering,
+fails each preparation allocation in turn until compilation succeeds, verifies
+that failed replacement preserves output and revisions, and counts both stereo
+native instances through retirement, collection and destruction of a pending
+plan. It repeats this lifecycle 100 times. A maximum 256-node reverse-insertion
+chain checks dependency scheduling, long-cycle rejection/rollback and compiled
+snapshot independence. The pthread handoff test submits 2000 native graphs
+while processing, checks whole-block output/revision consistency, asserts that
+prepare/destruction run on the control thread, and balances all native lifetimes
+after joining the consumer.
+
+Existing backend fault tests cover repeated start/stop, partial registration,
+callback setup, open/activate/deactivate/close failures and retries, server loss,
+rate/buffer changes, missing buffers, MIDI disconnect/panic, and recorder detach.
+Other suites cover deterministic variable-block DSP, finite-output handling,
+project errors, record-ring saturation and concurrent draining, and crash/write
+failure preservation. No critical realtime safety defect remains known within
+these contracts; these tests are evidence for tested paths, not a formal proof.
+
+## Reproducing the audit
+
+From the repository root:
+
+```sh
+make BUILD_DIR=build/audit all gui test-reliability test-recovery
+make BUILD_DIR=build/audit test-sanitize
+make analyze
+make BUILD_DIR=build/audit test-gui test-gui-jack test-jack test-jack-audio test-jack-midi
+make BUILD_DIR=build/nojack PKG_CONFIG=false all test-reliability gui
+```
+
+`test-sanitize` needs installed Clang and runs core, allocation-failure and recovery
+tests with ASan/UBSan and Linux's default ASan leak checking. `analyze` uses
+Clang's path-sensitive analyzer on every core translation unit, including JACK
+when its development files are available, and fails on diagnostics. Python and
+Clang are development tools, not new runtime dependencies. The allocation guard
+requires a GNU-compatible linker. Live tests require an existing JACK-compatible
+server; GUI tests use installed FLTK and Xvfb. No target installs dependencies or
+changes server/system configuration.
+
+Phase 10 verification on GCC 16.2.1 / Clang 23.1.3 passed normal and no-JACK builds,
+all core tests, ASan/UBSan/leak checks, Clang static analysis, GUI/mock and live
+JACK integrations, plus GCC TSan on the pthread native-publication and recorder
+concurrency tests. The GUI also passed ASan/UBSan with the existing
+`tests/integration/lsan.supp` fontconfig-only suppression; core tests use no leak
+suppression. The live graph test observed both replacement gains, ten restart
+cycles and zero xruns during its five-second software run.
+
+Physical audio devices, hard realtime scheduling, multi-hour endurance and
+arbitrary third-party native callbacks remain unqualified. Sanitizers change
+timing and should not be used to measure production realtime performance. See
+PERFORMANCE.md for workload measurements, RECOVERY.md for process-crash limits,
+and NATIVE_API.md for the trusted extension boundary.

docs/ROADMAP.md

@@ -136,6 +136,17 @@ physical-device/endurance qualification is claimed.
 - No known critical real-time safety defects remain.
 - Resource ownership and cleanup behavior are documented.
 
+Implemented: graph/DSP and ownership audit; stereo source-alias correction with
+exact-output regression tests; maximum-size reverse-order graph/cycle rollback;
+allocation-failure sweeps and callback allocation/free guards; 2000 concurrent
+native-plan replacements with control-thread destruction and balanced lifetimes;
+repeatable ASan/UBSan/leak and Clang static-analysis targets. Core, no-JACK,
+GUI/mock/live JACK, recovery and TSan publication/recorder tests pass. See
+RELIABILITY.md for ownership, synchronization, bounded-work analysis, commands
+and evidence. Physical-device/endurance validation and trusted native callback
+behavior remain explicit limits; no critical realtime safety defect is known
+within the documented contracts.
+
 ## Phase 11 — Complete Minimal Music-Making Workflow
 
 **Goal:** Ensure a user can create, save, reopen, and play a small musical project without manually editing project files.