Skip to content

[logging] Add default TX capture health summaries for TCI handoffs#4233

Open
jensenpat wants to merge 3 commits into
aethersdr:mainfrom
jensenpat:aether/tx-capture-health-logging
Open

[logging] Add default TX capture health summaries for TCI handoffs#4233
jensenpat wants to merge 3 commits into
aethersdr:mainfrom
jensenpat:aether/tx-capture-health-logging

Conversation

@jensenpat

@jensenpat jensenpat commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds default-on, bounded TX microphone capture health summaries for #4230 without changing audio behavior.

The #4230 field log has now confirmed the failure signature: while TCI audio keeps the 200 ms suppression window fresh, AudioEngine::onTxAudioReady() returns without consuming the local QAudioSource. The Linux/PipeWire capture ring reached 12000/12000 bytes while Qt still reported Active/NoError; the last successful microphone read was 14,173 ms old. This PR's diagnostic now detects that full-while-Active condition directly, while retaining Active-to-Idle with unread bytes as a fallback when capacity is unavailable.

Merge order and follow-up fix

Merge this diagnostic PR before #4251. The Linux-only behavioral fix in #4251 is intentionally stacked on this branch. After this PR lands, #4251 will reduce to its single behavior commit and the default-on diagnostics here will remain available to validate the fix in affected PipeWire systems.

What is recorded

  • Tracks suppressed microphone callbacks and peak unread bytes while TCI audio is fresh.
  • Records saturation when buffered bytes reach the reported source capacity, even if Qt continues reporting the source as Active.
  • Retains the previously suspected Active-to-Idle-with-unread-bytes signature as a fallback when source capacity is unavailable.
  • Counts later local, non-DAX TX starts while the post-TCI source remains saturated.
  • Excludes initial Idle state, TX owned by another client on the same radio, DAX TX, and TX while TCI audio is still fresh.
  • Includes device, Qt state/error, buffer availability/capacity, source lifetime, last successful microphone-read age, and aggregate counters.

AetherSDR's own local TX remains tracked normally when the radio is operating in Multi-Flex mode. Only TX owned by another client is excluded, because that remote transmission does not consume this AetherSDR process's microphone capture stream and must not be counted as a local capture failure.

Anomalies use the existing aether.audio.summary category, whose info summaries are force-enabled for support logs. Logging is bounded to one warning for each anomaly class per source lifecycle, plus one aggregate info summary when a relevant source lifecycle ends. No per-callback logging is added.

The existing get audio automation snapshot exposes the same TX endpoint evidence:

  • buffer_bytes_available
  • buffer_capacity_bytes
  • source_was_active
  • saturation_observed
  • tci_suppressed_callbacks
  • idle_during_tci_transitions
  • full_buffer_during_tci_observations
  • post_tci_local_tx_while_saturated
  • last_mic_read_age_ms

Related PipeWire / Linux audio tracking

These references are the working set of PipeWire/Linux audio failure and lifecycle reports relevant to field correlation. They are not asserted to share one root cause; the diagnostics distinguish local QAudioSource capture saturation from DAX routing, stream ownership, clocking, and device-change failures.

Current reports:

Prior PipeWire/Linux audio lifecycle and quality reports:

Issues that only concern packaging, PipeWire node naming, or channel-pair selection are intentionally not listed as capture-stall candidates.

Validation

  • Configured and built the complete application with Ninja and 8 build jobs.
  • Full CTest suite: 114/114 passed; the quarantined clean-room test was skipped by its existing policy.
  • Hardware-free state-machine test: 18/18 assertions passed, including Active/full-buffer saturation, the Idle fallback, rate limiting, repeated local TX counts, DAX exclusion, and exclusion of TX owned by another client.
  • tools/check_engine_boundary.py --strict: zero blocking findings.
  • git diff --check: clean.
  • Launched the built app with AETHER_AUTOMATION_NO_AUTOCONNECT=1; ping and get audio passed and exposed the corrected fields. No radio connection or TX was performed.

Field-validation result

The first affected PipeWire support log supplied the decisive evidence this PR was designed to collect: the ring was full while the Qt source remained Active. #4251 applies the narrow Linux-only drain/discard fix; these summaries remain enabled to verify that affected systems no longer reach saturation.

Screenshots

Not applicable: this is support-log and diagnostics instrumentation with no visual UI change.

👨🏼‍💻 Generated with OpenAI Codex (GPT-5.6 Sol) and tested by @jensenpat

@jensenpat jensenpat changed the title Add default TX capture health summaries for TCI handoffs logging: Add default TX capture health summaries for TCI handoffs Jul 13, 2026
@jensenpat
jensenpat marked this pull request as ready for review July 13, 2026 15:08
@jensenpat
jensenpat requested review from a team as code owners July 13, 2026 15:08

@aethersdr-agent aethersdr-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely scoped diagnostic PR. The TxCaptureHealthTracker is a clean hardware-free state machine, the conservative saturation signature (Active→Idle + suppressed callbacks + unread bytes + TCI fresh) is well-reasoned, the anomaly rate-limiting is correct, and the state-machine test pins each class (initial-Idle-is-healthy, saturation, DAX exclusion, foreign-owner exclusion, one-log-per-lifecycle). Threading care is evident — recordTxCaptureLocalTxAttempt is marshaled onto the owning thread and the stateChanged lambda guards against a stale source. All files are in scope, no AppSettings/QSettings misuse, no new deps, commit is signed. Default-on + bounded + no per-callback logging matches the stated field-evidence goal.

Non-blocking notes

  • audioEndpointDiagnostics() reads the tracker (and m_micDevice->bytesAvailable()) from the caller's thread while the tracker mutates on the audio thread — inline comment. Consistent with the existing pattern in that function, so not a blocker.

Thanks for the thorough writeup and the field-validation boundary being called out honestly — that's exactly the right framing for a diagnostic-only change.


🤖 aethersdr-agent · cost: $4.9911 · model: claude-opus-4-8

Comment thread src/core/AudioEngine.cpp
Comment on lines +2528 to +2530
const TxCaptureHealthTracker::Snapshot txHealth =
m_txCaptureHealth.snapshot(txCaptureNowMs());
tx["buffer_bytes_available"] = static_cast<double>(txCaptureBufferedBytes());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

audioEndpointDiagnostics() is const and invoked from AutomationServer / DeviceDiagnostics on the main thread, but m_txCaptureHealth and m_micDevice are owned and mutated on the audio thread (onTxAudioReady, the queued stateChanged lambda, recordMicRead). So snapshot() here — and txCaptureBufferedBytes()'s m_micDevice->bytesAvailable() call one line below — race the audio thread's writes on non-atomic members.

This matches the pre-existing pattern in this same function (it already reads m_audioSink->state(), m_micDevice->isOpen(), etc. cross-thread), so it's not a regression and the torn reads are diagnostic-only. Worth flagging only because you deliberately marshaled recordLocalTxAttempt onto the owning thread "so diagnostics cannot race readyRead/stateChanged" — the read side of the same data isn't marshaled. Fine to leave as-is for a support-log snapshot; just noting the asymmetry.

@jensenpat jensenpat changed the title logging: Add default TX capture health summaries for TCI handoffs [logging] Add default TX capture health summaries for TCI handoffs Jul 14, 2026
@NF0T

NF0T commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Reviewed this alongside #4251 (which stacks on top of this one). Two findings that are specifically this PR's own — reproducible with no drain/Linux-specific code involved at all:

1. Cross-thread read of audioEndpointDiagnostics() is a real race, not just a "widened surface." The stacked PR's bot review flagged this as pre-existing and non-blocking, but I traced both call paths: AutomationServer's get audio correctly marshals via Qt::BlockingQueuedConnection onto the audio thread before calling audioEndpointDiagnostics() — but SliceTroubleshootingDialog::refreshSnapshot()DeviceDiagnostics::buildAudioDevicesSnapshot() calls it directly from the GUI thread with no marshaling at all. AudioEngine lives on a dedicated audio thread (moveToThread in MainWindow.cpp), so this is genuine UAF risk on m_audioSource (nulled and deleted on the audio thread with no lock while the GUI thread can be mid-read), and genuine UB on the new TxCaptureHealthTracker::Snapshot this PR adds (9 plain non-atomic fields — torn/inconsistent reads are possible) and on the new m_txCaptureHealthClock (a QElapsedTimer raced between .elapsed() on one thread and .restart() on the other). The m_audioSource UAF pattern pre-dates this PR, but this PR adds three new non-atomic fields and a second racy timer into that exact unsafe path. Worth fixing by marshaling SliceTroubleshootingDialog's path the same way AutomationServer already does, rather than carrying it forward as non-blocking.

2. m_saturationReported latches for the entire source lifecycle. In TxCaptureHealthTracker::recordLocalTxAttempt(), saturated is m_saturationReported || (...), and nothing resets that flag except a full reset(). So once any saturation event fires once, every later local-TX attempt for the rest of the session gets counted as "stalled" regardless of whether the buffer has since recovered — inflating exactly the field evidence this instrumentation exists to collect cleanly. If "sticky forever" is intentional, worth a comment saying so; if not, it probably wants a way to clear once the buffer's healthy again.

Both are independent of #4251's Linux drain — they reproduce in this PR alone. I posted two different, drain-interaction-specific findings on #4251 itself, since those only manifest once the drain commit is layered on top of this one.

@NF0T NF0T self-assigned this Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

help wanted Extra attention needed priority: medium Medium priority

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants