Separating Mic and System Audio in Rust Without Diarization
Notes from building a dual-stream macOS audio capture and transcription pipeline with cpal, a Swift Process Tap helper, rtrb, and a cross-stream echo guard.
When building a local meeting assistant in Rust and Tauri, it is tempting to capture all audio into one stream, run Whisper, and bolt on speaker diarization later to figure out who said what.
In my current side project (audio-capture), I took a simpler and more reliable route: never merge the microphone and system audio streams in the first place.
Why Keep SourceKind From Capture to LLM?
On a call, you already have a clean physical boundary:
SourceKind::Microphoneis you.SourceKind::SystemOutputis everyone else.
If that tag survives every stage (from raw f32 PCM frames, through resampling and local whisper.cpp inference, all the way into the suggestion agent), you get accurate attribution without running a separate diarization model.
Isolating macOS System Audio in a Swift Subprocess
Microphone capture in Rust is straightforward with cpal. System loopback audio on macOS (14.4+) requires Apple's Core Audio Process Tap API, which is sensitive to lifecycle edge cases and OS quirks.
Rather than binding that FFI directly inside the main Tauri process, I isolated it into a standalone swift-helper subprocess that writes length-framed binary PCM to stdout and logs to stderr.
- Real-time discipline: Inside both the
cpalcallback and the Swift Core AudioIOProc, callbacks never block, allocate, or do I/O; they push into a lock-freertrbring buffer drained by an ordinary worker thread. - Fault isolation & supervision: If either source goes silently stale (a real bug we hit where the mic callback kept firing while publishing nothing),
StalenessWatcherdetects the quiet stream and a circuit-brokenRestartPolicyrestarts only that source.
The Hidden Catch: Acoustic Echo Bleed
Separating the streams creates one subtle real-world problem: if the user isn't wearing headphones, the microphone picks up voices coming out of the laptop speakers, meaning the same sentence gets transcribed twice (once tagged SystemOutput and once tagged Microphone).
To prevent duplicate transcript lines from confusing the suggestion agent, cue-agent runs a lightweight cross-stream echo guard over a 10-second window using word-set Jaccard similarity:
/// Word-set Jaccard similarity catches near-duplicate phrases produced by
/// two separate Whisper passes over the same acoustic bleed.
fn is_echo_of(a: &HashSet<String>, b: &HashSet<String>) -> bool {
if a.is_empty() || b.is_empty() {
return false;
}
let intersection = a.intersection(b).count();
let union = a.len() + b.len() - intersection;
intersection as f32 / union as f32 > 0.55
}What's Next
The Rust crates (audio-core, transcribe, and cue-agent) are in solid shape and verified with integration tests and CLI examples. Next up is overhauling the Tauri desktop interface and floating overlay so the UI feels as solid as the capture pipeline underneath.