Automatic speech recognition (ASR) has become a core technology in modern voice products. But knowing which model to pick, how to train it, and when to fine-tune it remains unclear for many teams. This guide breaks it all down.
What Is Speech Recognition?
Speech recognition is the technology that converts spoken audio into written text. It powers everything from voice assistants and call center automation to real-time translation platforms like Palabra.
How Does Speech Recognition Work?
When a person speaks, the system captures raw audio, processes it into numeric features, and maps those features to the most probable sequence of words. This process happens in milliseconds and requires tight coordination between acoustic and language components.
Key Components: Acoustic Model, Language Model, Decoder
The acoustic model maps audio features to phonemes. The language model estimates which word sequences are statistically probable. The decoder combines both signals to produce the final transcript. In older systems these were separate modules; modern architectures merge them into a single neural network.
ASR Model Architectures: From Pipelines to End-to-End
ASR has gone through several architectural generations, each with different trade-offs in accuracy, speed, and ease of use.
Traditional Pipeline Models (Kaldi)
Kaldi uses a step-by-step pipeline: feature extraction, acoustic modeling, and language modeling are handled separately. It offers fine-grained control but requires significant engineering effort – bash scripts, manual pre-processing, and deep expertise to configure.
Self-Supervised Models (wav2vec 2.0)
wav2vec 2.0 by Meta learns audio representations from raw unlabeled speech first, then fine-tunes on labeled transcripts. This dramatically reduces the need for annotated data and makes it practical for low-resource languages and specialized domains.
Encoder-Decoder Models (Whisper)
OpenAI’s Whisper was trained on 680,000+ hours of diverse internet audio. Its encoder-decoder architecture jointly models acoustics and language, producing outputs that include punctuation, capitalization, and timestamps out of the box – with no extra post-processing needed.
What Differentiates One ASR Model from Another?
Not all ASR models are equal. Four factors drive most of the performance gap between them.
Model Architecture
Pipeline models treat ASR as a sequence of independent tasks. End-to-end models optimize the entire process jointly, which generally improves accuracy on noisy or conversational audio.
Model Size and Capacity
Larger models capture more complex patterns but require more compute. Whisper medium strikes a practical balance between accuracy and resource usage. Smaller models are faster but struggle with accents, noise, and domain-specific vocabulary.
Training Data
This is the single biggest differentiator. Whisper’s broad, diverse training corpus explains its low WER (5-20%) on real-world audio, while Kaldi trained on narrower datasets regularly exceeds 40-70% WER on conversational speech.
Audio Pre-processing
Different models expect different input formats. Whisper handles raw audio directly. Kaldi requires manually resampled, mono-channel audio at specific sample rates. The less pre-processing required, the faster you can ship.
Key Techniques Used in ASR Training
Modern ASR draws on several machine learning techniques that each solve a different part of the recognition problem.
Hidden Markov Models (HMM)
HMMs model the temporal structure of speech by representing phonemes as sequences of states with transition probabilities. They remain the foundation of hybrid systems and are still relevant in production pipelines that prioritize interpretability.
Neural Networks
Deep neural networks replaced hand-crafted acoustic features across most ASR tasks. Convolutional and recurrent architectures learn audio representations directly from data, capturing patterns that traditional signal processing cannot.
Natural Language Processing (NLP)
NLP layers on top of ASR to improve transcription quality. Techniques like named entity recognition, language modeling, and contextual re-ranking reduce errors on domain-specific terms and ambiguous homophones.
N-gram Language Models
N-gram models assign probabilities to word sequences based on co-occurrence statistics. They are lightweight and fast, making them a common choice for re-ranking ASR outputs or biasing recognition toward specific vocabulary.
CTC and Self-Supervised Learning
Connectionist Temporal Classification (CTC) allows training without explicit frame-level alignment between audio and text. Combined with self-supervised pre-training, it enables wav2vec 2.0 to learn from large amounts of unlabeled audio before seeing any transcripts.
Fine-Tuning ASR Models for Your Domain
A model trained on general internet speech will struggle with medical consultations, legal proceedings, or live multilingual events. Fine-tuning adapts a pre-trained model to your specific domain without training from scratch.
When to Fine-Tune
Fine-tuning makes sense when your audio differs significantly from the training distribution – heavy accents, specialized vocabulary, background noise, or a language variant not covered by the base model. If WER on your data is more than 10-15 percentage points above benchmark results, fine-tuning is likely worth the investment.
How to Fine-Tune: Step-by-Step
1.Collect 10-100+ hours of labeled, domain-specific audio
2.Clean and normalize transcripts to match training conventions
3.Load a pre-trained checkpoint (e.g., Whisper or wav2vec 2.0) via HuggingFace
4.Train with a low learning rate to avoid catastrophic forgetting
5.Evaluate on a held-out test set using WER
6.Apply data augmentation (speed perturbation, noise injection) if data is scarce
Text Normalization and Evaluation Metrics (WER)
Word Error Rate (WER) measures substitutions, insertions, and deletions relative to the reference transcript. Before computing WER, normalize both the hypothesis and reference: lowercase, remove punctuation, expand numbers. Without consistent normalization, WER comparisons across models are meaningless.
Accuracy vs. Speed: Benchmarks That Matter
Choosing a model always involves a trade-off between recognition quality and inference speed.
Accuracy Results by Domain
| Model | Conversational WER | Phone Call WER | Meetings WER |
| Kaldi Gigaspeech XL | 40-70% | 50-75% | 45-70% |
| wav2vec 2.0 large | 15-30% | 20-35% | 18-30% |
| Whisper medium.en | 5-20% | 8-22% | 6-18% |
Whisper leads across all domains. Kaldi lags significantly on unscripted, conversational speech.
Speed Results by Hardware
| Model | Throughput (2080 Ti) | Throughput (A5000) |
| Kaldi Gigaspeech XL | Moderate | Moderate |
| wav2vec 2.0 large | 15-40x real-time | 25-55x real-time |
| Whisper medium.en | 3-8x real-time | 6-14x real-time |
wav2vec 2.0 processes audio significantly faster, but at the cost of accuracy on noisy data. For streaming use cases, this trade-off requires careful consideration.
ASR in Production: Real-World Use Cases
Benchmarks measure clean conditions. Production environments are messier.
Meetings and Calls
Multi-speaker audio, overlapping speech, and variable audio quality all degrade ASR accuracy. Production pipelines need speaker diarization, noise suppression, and language model adaptation to domain vocabulary.
Healthcare
Medical ASR must handle dense clinical terminology, accented speech from practitioners of diverse backgrounds, and strict accuracy requirements. Fine-tuning on clinical corpora and adding medical vocabulary to the language model are standard practice.
Live Events and Real-Time Translation
Live ASR demands low latency above all else. Streaming models process audio in short chunks and output partial transcripts as speech happens. Palabra combines ASR with real-time translation and voice synthesis to deliver sub-second multilingual output for live conferences, calls, and broadcasts.
Build vs. Buy: What Fine-Tuning Really Costs
Open-source models appear free but carry hidden costs. Fine-tuning requires GPU infrastructure, labeled data collection, engineering time, evaluation pipelines, and ongoing maintenance as models age. For teams without dedicated ML resources, the total cost of ownership often exceeds a managed API – especially when you factor in the latency, reliability, and compliance requirements of a production system.
The real question is not which model scores best on a benchmark. It is which solution delivers the accuracy your users need, at the latency your product requires, within the budget your team can sustain.
How Palabra Solves This for You
Palabra controls the full voice AI pipeline – ASR, translation, and TTS – with custom glossaries, accent-aware models, and voice cloning built in. You get domain-accurate speech recognition and real-time multilingual output without managing model training, GPU clusters, or evaluation pipelines yourself.
Whether you are running a global product demo, a live international conference, or a multilingual customer support line – Palabra handles the complexity so your team can focus on building.
