Skip to main content
Joshua Foster
Back to selected work
2026 (WIP)Personal Project · Rust & TauriActive side project

Dual-Channel Audio Capture in Rust

Local Rust, Tauri, and Swift Core Audio pipeline separating mic and system audio without diarization

A personal project in Rust and Tauri that captures microphone and macOS system audio as two separate streams, runs local Whisper transcription on Metal, and drives an experimental live prompt overlay.

RustTauri v2Swift (Process Tap)cpal & rtrbWhisper-rs (Metal)React & TypeScript
Source reference
Dual-Channel Audio Capture in Rust
Interactive Architecture Preview
CRATE BOUNDARY
crates/audio-core
CONTRACT
AudioBus (broadcast)
BUILD STATUS
Active R&D (WIP)
DUAL-CHANNEL PIPELINE STAGES (NO DIARIZATION)
SourceKind::Microphone (cpal)Dedicated OS audio callback pushes raw f32 PCM into a lock-free rtrb ring buffer without blocking or allocating.
Interactive architecture & workflow preview

Why I'm Building This

Having built a cloud-based real-time meeting assistant at Uppr, I wanted to explore what a local-first desktop version looks like today using Rust, local GPU speech-to-text, and native macOS audio APIs.

The project is still in progress: the capture, transcription, and cue-agent pipeline works end-to-end via Rust examples and integration tests, while the Tauri frontend and overlay UI are undergoing an overhaul.

Core Design Decision: Keep the Streams Separate

Most meeting tools merge microphone and system audio into a single buffer and rely on post-hoc speaker diarization to guess who spoke. Instead, this pipeline keeps SourceKind::Microphone ("you") and SourceKind::SystemOutput ("everyone else") strictly separated from capture all the way through transcription and LLM context.

Architecture

The workspace splits into four Rust crates plus a standalone macOS Swift sidecar:

  1. crates/audio-core (Capture & Supervision): Defines the AudioSource trait (MicrophoneSource via cpal, SystemOutputSource via swift-helper, and FakeAudioSource for hardware-free tests). Real-time audio callbacks push f32 PCM into a lock-free rtrb ring buffer; a non-blocking tokio::sync::broadcast AudioBus fans frames out to subscribers, watched by a StalenessWatcher + RestartPolicy circuit breaker.
  2. swift-helper (Isolated System Audio Tap): An independent Swift package using macOS 14.4+'s Core Audio Process Tap API, streaming framed binary PCM over stdout so any OS audio quirk is isolated to a supervised subprocess.
  3. crates/transcribe (Local Whisper): Subscribes to AudioBus, segments speech with an energy-based VAD (chunker.rs), resamples to 16 kHz via rubato, and runs whisper-rs with Apple Metal acceleration, preserving SourceKind on every TranscriptSegment.
  4. crates/cue-agent & src-tauri (Suggestion Engine & Overlay): Maintains a rolling 12k-char transcript with cooldown/word-count gating, a Jaccard cross-stream echo guard (dropping speaker bleed picked up by the mic), and a screen-capture-hidden floating macOS overlay (overlay.rs).
rust
/// Non-blocking multi-consumer event bus (`crates/audio-core/src/bus.rs`).
/// Lagging subscribers receive `RecvError::Lagged(n)` and resume without
/// backpressuring the real-time audio producer.
#[derive(Clone)]
pub struct AudioBus {
    tx: broadcast::Sender<AudioFrame>,
}

impl AudioBus {
    pub fn new(capacity: usize) -> Self {
        let (tx, _rx) = broadcast::channel(capacity);
        Self { tx }
    }

    pub fn subscribe(&self) -> broadcast::Receiver<AudioFrame> {
        self.tx.subscribe()
    }

    pub fn publish(&self, frame: AudioFrame) {
        let _ = self.tx.send(frame);
    }
}

Key Trade-offs

Architectural trade-offs and engineering rationale
Decision AxisChosen PathRejected AlternativeWhy It Matters
macOS System Audio CaptureIsolated Swift subprocess (Process Tap) over framed stdoutLinking fragile Core Audio tap FFI directly into TauriIf the macOS audio tap hangs or stalls, the Rust supervisor detects silence and restarts the helper without crashing the app.
Speaker AttributionIndependent Microphone + SystemOutput streams + echo guardSingle mixed stream + ML speaker diarizationGuarantees clean 'you vs. them' attribution while filtering speaker-to-mic acoustic bleed via word-set Jaccard similarity.
Bus Backpressure ContractBest-effort broadcast (`tokio::sync::broadcast`)Bounded MPSC blocking producersA slow downstream consumer can lag and recover, but can never stall real-time audio capture.

Current Status & Next Steps

  • Working today: Dual capture verified via standalone WAV/transcription examples, StalenessWatcher silent-failure recovery, local Metal Whisper transcription, and unit/integration tests across audio-core, transcribe, and cue-agent.
  • In progress: Overhauling the Tauri control window and floating suggestion overlay UI so the desktop experience matches the reliability of the underlying Rust pipeline.