Guide to WhisperX with Python: Speech Recognition, Word-Level Alignment, and Speaker Diarization - ITFROMZERO - Share tobe shared! Transcribe audio with WhisperX and Python: leverage forced alignment for precise word-level timestamps and automated speaker diarization running locally on your GPU.

Table of Contents

Real-World Speech Recognition: When Transcribing Text Simply Isn’t Enough

Your customer support team just handed you a 45-minute call recording from Asterisk or Zoom. The requirement: extract the complete transcript to audit call quality. If you feed that audio into vanilla Whisper, you’ll end up with an unbroken block of thousands of words without any context. Who is the agent? At what exact minute did the customer complain? Where in the conversation was the deal closed?

Whether generating synchronized karaoke-style captions in CapCut or drafting automated meeting minutes with assigned action items, a production STT system must solve three challenges simultaneously: What was said?, At what exact second was it said? (word-level timestamps), and Who said it? (speaker diarization).

Comparing Modern Speech-to-Text Solutions

Here are real-world takeaways after benchmarking various architectures across hundreds of hours of conversational audio:

Vanilla OpenAI Whisper: Impressive accuracy across many languages. However, timestamps are limited to the segment or sentence level. Audio files longer than 30 minutes frequently drift out of sync by several seconds or fall into repetitive hallucination loops. Its biggest limitation: Whisper does not support speaker diarization out of the box.
Cloud Speech APIs (Google Cloud Speech-to-Text, AWS Transcribe): Feature-rich and highly reliable, but cost remains a major barrier. At an average rate of around $0.024/minute ($1.44/hour), a contact center processing 500 hours of recordings monthly will spend over $700. Additionally, sending internal audio containing credit card details or national IDs to third-party cloud services raises compliance and data privacy concerns.
WhisperX: An open-source project combining three tools: the faster-whisper core (using CTranslate2 for optimized inference), forced alignment phoneme models, and the pyannote.audio diarization pipeline. WhisperX eliminates timestamp drift, cleanly separates speakers, and runs entirely offline on self-hosted infrastructure.

Pros and Cons of Each Approach

1. Vanilla Whisper

Pros: Quick installation via a single pip install openai-whisper command. Official library with robust multilingual context handling.
Cons: Heavy VRAM consumption (requires 10–12GB VRAM for the large-v3 model). Slow inference on long audio files. No native diarization and cannot support word-by-word video editing.

2. Commercial Cloud APIs

Pros: Zero hardware infrastructure management. Simply query a REST API and receive structured results.
Cons: Cloud bills scale exponentially with volume. Sensitive corporate data leaves your private network.

3. WhisperX

Pros: 2x to 4x faster than vanilla Whisper thanks to CTranslate2 and Voice Activity Detection (VAD) that cuts out silence. Word-level accuracy powered by forced alignment. Built-in Pyannote 3.1 integration provides exceptional speaker separation accuracy.
Cons: Requires an NVIDIA GPU with at least 6GB–8GB of VRAM (an RTX 3060 12GB or cloud T4/A10 is recommended). Initial setup requires accepting terms and configuring a Hugging Face user access token.

Why WhisperX Is the Best Choice for Self-Hosted Systems

If you are building an automated short-form video captioning tool, a Google Meet meeting summarizer, or a QA call auditing pipeline, WhisperX strikes the perfect balance. Hardware expenses are fixed, data privacy is guaranteed, and the engine outputs granular JSON data that can be piped directly into LLMs for downstream analysis.

Step-by-Step Implementation Guide with Python

Step 1: Set Up the Environment and Dependencies

WhisperX requires Python 3.10 or 3.11 along with ffmpeg to extract and process audio streams.

# Install ffmpeg on Ubuntu/Debian
sudo apt update && sudo apt install -y ffmpeg

# Create a virtual environment
python3 -m venv venv && source venv/bin/activate

# Install PyTorch compatible with CUDA 12.1
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Install WhisperX
pip install git+https://github.com/m-bain/whisperx.git

Step 2: Obtain a Hugging Face Access Token

The speaker diarization pipeline relies on pre-trained weights from pyannote.audio. You need a Hugging Face account to accept the user conditions on the following two model repositories:

pyannote/speaker-diarization-3.1
pyannote/segmentation-3.0

Next, navigate to Settings > Access Tokens on Hugging Face, generate a new token with Read permissions, and copy it to use in your Python code.

Step 3: Build the Complete Audio Processing Script

The script below executes the complete 3-stage pipeline: Speech Recognition → Word-Level Timestamp Alignment → Speaker Diarization.

import whisperx
import gc
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
audio_file = "meeting_demo.wav"
batch_size = 8  # Lower to 4 or 2 if VRAM is under 8GB
compute_type = "float16" if device == "cuda" else "int8"
HF_TOKEN = "hf_your_personal_access_token_here"

# 1. Transcribe audio using faster-whisper
print("--- [1/3] Transcribing audio content ---")
model = whisperx.load_model("large-v3", device, compute_type=compute_type, language="vi")
audio = whisperx.load_audio(audio_file)
result = model.transcribe(audio, batch_size=batch_size)

# Free GPU VRAM before loading the next model
del model
gc.collect()
torch.cuda.empty_cache()

# 2. Word-level timestamp alignment (Forced Alignment)
print("--- [2/3] Aligning word-level timestamps ---")
model_a, metadata = whisperx.load_align_model(language_code=result["language"], device=device)
result = whisperx.align(result["segments"], model_a, metadata, audio, device, return_char_alignments=False)

del model_a
gc.collect()
torch.cuda.empty_cache()

# 3. Speaker diarization
print("--- [3/3] Assigning speaker labels with Pyannote ---")
diarize_model = whisperx.DiarizationPipeline(use_auth_token=HF_TOKEN, device=device)
diarize_segments = diarize_model(audio_file)

# Assign speaker information to word timestamps
final_result = whisperx.assign_word_speakers(diarize_segments, result)

# Display results in the console
for segment in final_result["segments"]:
    speaker = segment.get("speaker", "UNKNOWN")
    start = segment["start"]
    end = segment["end"]
    text = segment["text"].strip()
    print(f"[{start:.2f}s -> {end:.2f}s] {speaker}: {text}")
    
    # Print detailed word timestamps to check alignment
    words_detail = [f"{w['word']} ({w.get('start', 0):.2f}-{w.get('end', 0):.2f})" for w in segment.get("words", [])]
    print("   Word details:", ", ".join(words_detail))

Step 4: Leveraging the Output Data

WhisperX returns a structured JSON object containing millisecond-accurate timestamps and speaker IDs (such as SPEAKER_00, SPEAKER_01). With this output, you can readily build three practical use cases:

Dynamic Subtitles: Export to .srt or .ass to render synchronized karaoke subtitles in Premiere Pro, ensuring frame-perfect alignment.
Audio Search: Enable users to search for keywords and jump straight to the exact second spoken across an archive of thousands of calls.
Meeting Summarization with LLMs: Feed speaker-labeled transcripts into GPT-4o or Claude 3.5 Sonnet to accurately extract individual action items without confusing speakers.