Real‑time Whisper transcription with Candle.

Provides a real‑time Whisper engine using Hugging Face Candle. Handles model download/cache, mel preprocessing, encoder/decoder inference, and emits Partial/Final events from a worker thread. Uses segment-aware windowing (≤30s per Whisper context) with live preview and final transcription on segment end.

• Candle‑based Whisper (CPU, Metal, CUDA) device selection
• Quantized model support - 3-4x faster inference with GGUF models
• Quality improvements - Per‑unit silence‑drop + overlap dedupe, temperature fallback, quality metrics
• HF Hub auto‑download and local cache
• High‑quality arbitrary-rate resampling via rubato (16k, 32k, 44.1k, 48k, 96k all supported)
• Streaming interface: push audio frames, poll events
• Segment-aware: live preview while speaking; single Final on segment end
• Windowed inference: ≤30s windows for long utterances; automatic concatenation
• Language auto-detection - Automatic language identification when not specified
• Performance optimized - Zero-copy model access, single encoder pass, rubato resampling

✅ Normal models (safetensors):

• openai/whisper-tiny - fastest, 75 MB
• openai/whisper-base - good balance, 142 MB
• openai/whisper-small - higher accuracy, 466 MB
• ...

⚡ Quantized models (GGUF) - 3-4x faster:

• Any model with .gguf files in the directory
• Automatically detected and loaded
• ~1% accuracy loss, significant speed gain
• Ideal for lower-end hardware

keyless-whisper = { path = "../keyless-whisper" }

Build backends:

• macOS: Metal is auto‑enabled via target‑specific dependencies; nothing extra to do.
• Windows/Linux with NVIDIA: enable CUDA explicitly:

cargo run -p keyless --features "keyless-whisper/cuda"

use keyless_whisper::{Whisper, WhisperConfig, RealtimeTranscriber, WhisperLoadPhase, PhaseState};
use std::path::PathBuf;

let cfg = WhisperConfig {
    model_path: PathBuf::from("openai/whisper-base"),  // or whisper-base.en
    language: Some("en".to_string()),
    source_sample_hz: 48_000,
};
// Basic constructor
let mut t = Whisper::new(cfg.clone())?;
// t.push_audio(&frame)?; // push mono f32 frames
// t.end_segment()?; // on PTT release
// if let Some(evt) = t.try_next_event() { /* handle */ }

let mut t = Whisper::new_with_progress(cfg, Some(|phase: WhisperLoadPhase, state: PhaseState| {
    // Map to your UI/logging; called for Begin/End of each phase
    eprintln!("{:?}: {:?}", phase, state);
}))?;

• WhisperConfig model path can be an HF ID (auto‑download) or a local directory with config.json, model.safetensors, tokenizer.json.
• Language: For multilingual models, provide language: Some("en") for English transcription. Auto-detection is slower and less accurate.
• Model selection: Both multilingual and .en models work. .en models are slightly smaller but English-only.

• Unified Preview = Final: Both run the same voiced‑mask pipeline. Previews reuse cached unit texts via a ~128 ms tail hash and only decode the newest tail units; Final runs over the full segment.
• Voiced‑mask units: Build voiced spans, split into ≤10s units with 0.5s overlap, decode, dedupe overlap, stitch.
• Mel cap: Cap mel frames to full Whisper context (2×max_source_positions = 3000 frames) to avoid encoder narrow errors.
• Silence‑drop: Per‑unit guard combines RMS and Whisper’s no_speech_prob to drop only silent units (no global gate).
• Temperature fallback: Keep greedy first; retry with higher temperatures only when metrics indicate low confidence or repetition.

Audio Flow:

1. Audio captured at device sample rate → rubato resampler → 16 kHz mono
2. VAD gates speech; only voice frames accumulate in the utterance buffer
3. While speaking (PTT held): emit Preview by running the voiced‑mask pipeline on the current buffer (≤10s units, 0.5s overlap); reuse cached unit texts via a ~128 ms tail hash; decode only tail units
4. On PTT release: emit single Final by running the same pipeline on the entire utterance; stitch units after overlap dedupe

Decoder:

• Based on Candle WASM Whisper example pattern
• Feed full token sequence every step (not last-token-only)
• Flush KV cache only on first iteration per window
• Use decoder.final_linear() for vocab projection
• Greedy sampling with repetition guards (max steps, identical tokens, tri-grams)

We analyzed the candle-wasm-examples/whisper reference implementation and extracted several critical improvements:

Auto-detects .gguf files and loads appropriate format. No user configuration needed.

What we learned:

• On silent/noise-only segments, Whisper can hallucinate text
• First decoder iteration provides reliable no-speech probability
• Threshold of 0.6 effectively filters silent segments

Implementation:

• Extract no_speech_prob from the first decoder pass for each unit
• Combine with unit RMS; drop the unit only if high no‑speech and very low RMS
• Avoids global gating so non‑silent content is never discarded

What we learned:

• Greedy decoding (temp=0.0) sometimes fails on difficult audio
• Quality metrics indicate when to retry: compression_ratio, avg_logprob
• Temperature schedule [0.0, 0.2, 0.4, 0.6, 0.8, 1.0] from OpenAI reference

Implementation:

• Try greedy first (fastest)
• If quality metrics indicate failure, retry with higher temperature
• Early stop when quality is acceptable (95%+ succeed on first try)

What we learned:

• Whisper can detect language from audio features
• Run one decoder step with just SOT token, examine language token probabilities
• More accurate than specifying wrong language, but slower than correct hint

Implementation:

• Optional: auto-detect when language not specified
• Reuses encoder output (no duplicate encoder pass)
• Logs detected language and confidence

Critical fixes during development:

1. Vocab Projection Must Use decoder.final_linear()

   • Problem: Initially tried manual vocab projection via token embedding
   • Issue: This produced gibberish because we were argmaxing over the wrong value
   • Solution: Use Candle's built-in decoder.final_linear() method which handles the projection correctly

2. Decoder Loop Pattern: Full Sequence Every Step

   • Problem: Attempted to feed only the last token on subsequent steps (trying to optimize KV cache)
   • Issue: Candle's Whisper decoder expects the full token sequence on every forward pass
   • Solution: Feed tokens[..] every step; KV cache is managed internally, only flush on first iteration (i == 0)

3. Repetition Guards Essential

   • Problem: Short/noisy audio segments cause decoder loops (e.g., "And And And...")
   • Solution: Implement guards for identical token repetition (≥8 of last 10) and tri-gram repetition (≥3 occurrences)

Reference Implementation: This implementation follows the pattern from candle-wasm-examples/whisper, which is the official working Candle Whisper example. Note:

• Full token sequence feeding pattern
• Proper use of decoder.final_linear() instead of manual projection
• KV cache semantics (flush once per window, not per step)

Emits tracing logs for device selection, model loading, and inference timing/errors. Libraries do not initialize logging.

• Whisper::new() delegates to new_with_progress(None); no duplication.
• WhisperLoadPhase::as_label() provides user-friendly labels for UI overlays.

cargo test -p keyless-whisper

Unit tests cover preprocessing and token filtering. Full model inference is exercised in integration flows.

• candle-core, candle-nn, candle-transformers for Whisper
• tokenizers for text decoding
• keyless-models for cache path resolution (models are downloaded by the TUI)
• rubato for high-quality resampling
• tracing for logging

• src/lib.rs: Public API re-exports and module declarations
• src/config.rs: Configuration types (WhisperConfig, WhisperLoadPhase, PhaseState)
• src/device.rs: Device selection and caching (Metal > CUDA > CPU)
• src/transcriber.rs: RealtimeTranscriber trait definition
• src/whisper.rs: Main Whisper transcriber implementation
  • whisper/construct.rs: Construction and initialization
  • whisper/inference_thread.rs: Inference thread (runs Whisper model)
  • whisper/worker_thread.rs: Worker thread (resampling, accumulation, partial previews)
  • whisper/trait_impl.rs: Trait implementations (RealtimeTranscriber, Drop)
  • whisper/types.rs: Type definitions (Whisper, WhisperCmd, InferReq)
• src/model.rs: Model loading and management
  • model/loader.rs: Model loading with progress callbacks
  • model/mel_filters.rs: Mel filter bank generation
  • model/files.rs: File detection and location helpers
  • model/types.rs: Model type definitions (Model, WhisperModel, WhisperTokens)
• src/decode.rs: Token generation and text decoding
  • decode/fallback.rs: Temperature fallback decoding
  • decode/language.rs: Language detection from audio features
  • decode/temperature.rs: Single-temperature decoding
  • decode/helpers.rs: Helper functions (token decoding, repetition detection)
  • decode/constants.rs: Decoding constants (thresholds, temperatures)
  • decode/result.rs: Decoding result type with quality metrics
• src/preprocessing.rs: PCM→mel spectrogram conversion (uses pre-generated mel filters)
• src/inference.rs: End‑to‑end inference pipeline

• Device selection at runtime: Metal (macOS) → CUDA (if enabled) → CPU fallback. The CLI logs the chosen backend during startup.
• Offline-first: models are downloaded and cached by the TUI (via keyless-models); this crate loads from the local cache.

Critical Discovery: .en and multilingual models have different token IDs for special tokens:

// From candle-transformers
let sot = tokenizer.token_to_id(m::SOT_TOKEN)?;  // Gets correct ID for any model
let eot = tokenizer.token_to_id(m::EOT_TOKEN)?;

This pattern automatically works for both model types.

MIT