Dual-Channel Audio Capture in Rust
Local Rust, Tauri, and Swift Core Audio pipeline separating mic and system audio without diarization
A personal project in Rust and Tauri that captures microphone and macOS system audio as two separate streams, runs local Whisper transcription on Metal, and drives an experimental live prompt overlay.
Why I'm Building This
Having built a cloud-based real-time meeting assistant at Uppr, I wanted to explore what a local-first desktop version looks like today using Rust, local GPU speech-to-text, and native macOS audio APIs.
The project is still in progress: the capture, transcription, and cue-agent pipeline works end-to-end via Rust examples and integration tests, while the Tauri frontend and overlay UI are undergoing an overhaul.
Core Design Decision: Keep the Streams Separate
Most meeting tools merge microphone and system audio into a single buffer and rely on post-hoc speaker diarization to guess who spoke. Instead, this pipeline keeps SourceKind::Microphone ("you") and SourceKind::SystemOutput ("everyone else") strictly separated from capture all the way through transcription and LLM context.
Architecture
The workspace splits into four Rust crates plus a standalone macOS Swift sidecar:
crates/audio-core(Capture & Supervision): Defines theAudioSourcetrait (MicrophoneSourceviacpal,SystemOutputSourceviaswift-helper, andFakeAudioSourcefor hardware-free tests). Real-time audio callbacks pushf32PCM into a lock-freertrbring buffer; a non-blockingtokio::sync::broadcastAudioBusfans frames out to subscribers, watched by aStalenessWatcher+RestartPolicycircuit breaker.swift-helper(Isolated System Audio Tap): An independent Swift package using macOS 14.4+'s Core Audio Process Tap API, streaming framed binary PCM overstdoutso any OS audio quirk is isolated to a supervised subprocess.crates/transcribe(Local Whisper): Subscribes toAudioBus, segments speech with an energy-based VAD (chunker.rs), resamples to16 kHzviarubato, and runswhisper-rswith Apple Metal acceleration, preservingSourceKindon everyTranscriptSegment.crates/cue-agent&src-tauri(Suggestion Engine & Overlay): Maintains a rolling 12k-char transcript with cooldown/word-count gating, a Jaccard cross-stream echo guard (dropping speaker bleed picked up by the mic), and a screen-capture-hidden floating macOS overlay (overlay.rs).
/// Non-blocking multi-consumer event bus (`crates/audio-core/src/bus.rs`).
/// Lagging subscribers receive `RecvError::Lagged(n)` and resume without
/// backpressuring the real-time audio producer.
#[derive(Clone)]
pub struct AudioBus {
tx: broadcast::Sender<AudioFrame>,
}
impl AudioBus {
pub fn new(capacity: usize) -> Self {
let (tx, _rx) = broadcast::channel(capacity);
Self { tx }
}
pub fn subscribe(&self) -> broadcast::Receiver<AudioFrame> {
self.tx.subscribe()
}
pub fn publish(&self, frame: AudioFrame) {
let _ = self.tx.send(frame);
}
}Key Trade-offs
| Decision Axis | Chosen Path | Rejected Alternative | Why It Matters |
|---|---|---|---|
| macOS System Audio Capture | Isolated Swift subprocess (Process Tap) over framed stdout | Linking fragile Core Audio tap FFI directly into Tauri | If the macOS audio tap hangs or stalls, the Rust supervisor detects silence and restarts the helper without crashing the app. |
| Speaker Attribution | Independent Microphone + SystemOutput streams + echo guard | Single mixed stream + ML speaker diarization | Guarantees clean 'you vs. them' attribution while filtering speaker-to-mic acoustic bleed via word-set Jaccard similarity. |
| Bus Backpressure Contract | Best-effort broadcast (`tokio::sync::broadcast`) | Bounded MPSC blocking producers | A slow downstream consumer can lag and recover, but can never stall real-time audio capture. |
Current Status & Next Steps
- Working today: Dual capture verified via standalone WAV/transcription examples,
StalenessWatchersilent-failure recovery, local Metal Whisper transcription, and unit/integration tests acrossaudio-core,transcribe, andcue-agent. - In progress: Overhauling the Tauri control window and floating suggestion overlay UI so the desktop experience matches the reliability of the underlying Rust pipeline.