LIVE
Individual Project  ·  AI & Automation

Live Translation
Overlay for Desktop

A real-time audio translation tool that runs on top of any app: TikTok, YouTube, Zoom, WeChat. Built entirely from free, local tools. No subscriptions. No API keys. Zero cost.

$0 to build Python 3.11 OpenAI Whisper PyTorch + CUDA Runs locally All languages

Proof of concept: live translation overlay running in real time

Overview

Built to solve a
real problem.

The idea was simple: be able to watch or listen to content in any language and get a live translation on screen without switching apps, paying for a service, or copying text into a translator.

The app listens to whatever audio is playing on your Windows machine, runs it through a local speech recognition model, translates the output, and displays it as a floating overlay that sits on top of everything else. It works on TikTok livestreams, YouTube videos, Zoom calls, WeChat voice notes, and anything else that produces audio.

Every tool used is free and runs on your own machine. No data leaves your computer. No subscription. No API key. The total cost to build and run it is $0.

By the Numbers
$0
Total cost
~5GB
Disk space needed
1
Build session
10
Bugs fixed
Tech Stack
Python 3.11
Core language. Not 3.14, because PyTorch compatibility required 3.11 specifically.
OpenAI Whisper (medium)
Converts speech to text, runs entirely on your machine. No internet needed.
deep-translator
Free Google Translate wrapper. No API key required.
pyaudiowpatch
Captures Windows system audio output, whatever the computer is playing.
PyTorch + CUDA
GPU acceleration for Whisper. Critical, because CPU-only runs too slowly for real time.
tkinter
Builds the floating overlay window that sits on top of other apps.
scipy
Resamples audio from 44,100 Hz down to 16,000 Hz before sending to Whisper.
silero-vad
Voice activity detection that filters out music and background noise before translation.
Problems Solved
01
Python Version Compatibility
Started with Python 3.14. PyTorch does not support it yet, so GPU acceleration would not install at all. The whole AI stack depends on PyTorch being able to talk to the GPU.
Switched to Python 3.11 for the best AI package compatibility available
02
Audio Sample Rate Mismatch
This was the biggest one. Whisper requires audio at 16,000 Hz. Windows audio devices output at 44,100 Hz. Whisper was receiving the wrong sample rate and generating random fake words instead of transcribing what was actually playing.
Added a scipy resampling step to convert audio before sending it to Whisper
audio_np = scipy.signal.resample(audio_np, int(len(audio_np) * 16000 / sample_rate))
03
Whisper Hallucinations
Whisper kept generating text that was not in the audio. Watermarks like "Subtitles by Solanya", single characters, and long strings of repeated characters showed up constantly.
Added a blacklist, repetition filter, set temperature=0, and raised the silence threshold
04
GPU Not Being Used
PyTorch kept defaulting to the CPU version. Running torch.cuda.is_available() returned False even after installation. The default pip install always pulls the CPU build silently.
Always install PyTorch using the CUDA-specific wheel URL
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
05
silero-vad Broke GPU Support
Installing silero-vad automatically pulled in a CPU version of PyTorch and silently replaced the GPU version. The app went back to running slowly with no warning or error.
Run torch.cuda.is_available() after every new package install to catch breakage early
06
Last Sentence Not Translating
Short audio clips like WeChat voice notes were always missing their final sentence. The app waited for a full audio chunk before processing, but the audio ended before the buffer filled up.
Added a silence detection flush, so after 1 second of silence, whatever is in the buffer gets processed immediately
07
Music Getting Translated
Livestreams with background music produced garbage translations. Whisper picked up whatever was loudest and tried to transcribe the music as speech.
Added silero-vad with a 0.8 threshold, strict enough to reject most music and background noise
What to Do Differently Next Time
1
Start with Python 3.11 from the very beginning. Don't experiment with newer versions on AI projects until the ecosystem catches up.
2
Install the CUDA version of PyTorch before anything else: before Whisper, before VAD, before any other package.
3
Test the audio sample rate early. Most of the debugging time traced back to the 44,100 Hz vs 16,000 Hz mismatch.
4
Add VAD from the start. Trying to handle music and noise after the fact is harder than building it in at the beginning.
5
Run torch.cuda.is_available() after every new package install. Some packages silently overwrite the GPU build with a CPU one.
6
Set temperature=0 and prompt=None in Whisper's decode options from day one. These two settings alone eliminate most hallucination issues.
Final Settings That Worked
Setting Value Reason
WHISPER_MODELmediumBest accuracy for non-English speech
CHUNK_SECONDS5Good balance of speed vs accuracy
fp16FalseMore accurate than True on this setup
temperature0Stops hallucination
SILENCE_LIMIT1 secondCatches last sentence of short clips
VAD threshold0.8Strict enough to filter music
Total Cost
$0
Money spent
Every tool used is free and open source. No subscriptions, no API keys, no hidden costs.
~5GB
Disk space
PyTorch is 2.5GB, Whisper medium is 1.5GB, everything else is under 200MB.
1
Build session
From blank file to working proof of concept in a single session including all debugging.

Want to see
more of my work?

Browse all projects or get in touch directly.

All Projects Get in Touch