Connect the World, Frame by Frame

English｜简体中文｜繁體中文｜日本語｜Español｜Русский｜Français

🌟 Overview (Try VL Now!)

VideoLingo is an all-in-one video translation, localization, and dubbing tool aimed at generating Netflix-quality subtitles. It eliminates stiff machine translations and multi-line subtitles while adding high-quality dubbing, enabling global knowledge sharing across language barriers.

Key features:

• 🎥 YouTube video download via yt-dlp

• 🎙️ Word-level and Low-illusion subtitle recognition with WhisperX

• 📝 NLP and AI-powered subtitle segmentation

• 📚 Custom + AI-generated terminology for coherent translation

• 🔄 3-step Translate-Reflect-Adaptation for cinematic quality

• ✅ Netflix-standard, Single-line subtitles Only

• 🗣️ Local MLX voice-cloning dubbing with nine selectable backends

• 🚀 One-click startup and processing in Streamlit

• 🌍 Multi-language support in Streamlit UI

• 📝 Detailed logging with progress resumption

Difference from similar projects: Single-line subtitles only, superior translation quality, seamless dubbing experience

🎥 Demo

Dual Subtitles trans.mp4

Voice Clone Demo dubbing.mp4

Voice Clone Demo sovits.mp4

Language Support

Input Language Support(more to come):

🇺🇸 English 🤩 | 🇷🇺 Russian 😊 | 🇫🇷 French 🤩 | 🇩🇪 German 🤩 | 🇮🇹 Italian 🤩 | 🇪🇸 Spanish 🤩 | 🇯🇵 Japanese 😐 | 🇨🇳 Chinese* 😊

*Chinese uses a separate punctuation-enhanced whisper model, for now...

Translation supports all languages, while dubbing language depends on the chosen TTS method.

Installation

You don't have to read the whole docs, here is an online AI agent to help you.

Note: For Windows users with NVIDIA GPU, follow these steps before installation:

1. Install CUDA Toolkit 12.6
2. Install CUDNN 9.3.0
3. Add C:\Program Files\NVIDIA\CUDNN\v9.3\bin\12.6 to your system PATH
4. Restart your computer

Note: FFmpeg is required. Please install it via package managers:

• Windows: choco install ffmpeg (via Chocolatey)
• macOS: brew install ffmpeg (via Homebrew)
• Linux: sudo apt install ffmpeg (Debian/Ubuntu)

1. Clone the repository

git clone https://github.com/Huanshere/VideoLingo.git
cd VideoLingo

2. Install dependencies(requires python=3.10)

conda create -n videolingo python=3.10.0 -y
conda activate videolingo
python install.py

3. Start the application

streamlit run st.py

Docker

Alternatively, you can use Docker (requires CUDA 12.4 and NVIDIA Driver version >550), see Docker docs:

docker build -t videolingo .
docker run -d -p 8501:8501 --gpus all videolingo

APIs

VideoLingo supports OpenAI-compatible LLM APIs and local MLX speech backends:

• LLM: claude-3-5-sonnet-20240620, deepseek-chat(v3), gemini-2.0-flash-exp, gpt-4o, ... (sorted by performance)
• WhisperX: Run whisperX locally or use 302.ai API
• TTS: the local MLX router with IndexTTS2, OmniVoice, Qwen3-TTS, VoxCPM2, Higgs Audio, dots.tts, ZONOS2, and MOSS-TTS.

Note: Translation can still use an OpenAI-compatible API; dubbing in this fork is local MLX only.

2026 Local Upgrade Notes

This fork keeps the original workflow but adds provider-based configuration and environment checks:

• Secrets are no longer expected in config.yaml. Set keys through .env / shell variables based on .env.example.
• Run conda run -n videolingo python -m core.doctor before long jobs to check FFmpeg, Python packages, services, secrets, and recoverable step status.
• LLM routing is provider-based. The default is local oMLX with Qwen3.6-35B-A3B-Qwable-Holo3-Qwopus-oQ6-mtp; the original openai_compatible API block and cloud presets remain explicitly selectable.
• Source text and word timing are restricted to WhisperX and stable-ts. MOSS-Transcribe-Diarize runs as a local MLX speaker sidecar: it labels validated native words without replacing their timestamps, and the pipeline fails closed below 98% speaker coverage.
• TTS routes are registry-based. --tts auto / tts_method: mlx_router auto-routes through proven defaults and exposes eight local voice-cloning adapters: IndexTTS2, OmniVoice, Qwen3-TTS, VoxCPM2, Higgs Audio, dots.tts, ZONOS2, and MOSS-TTS. Experimental additions must be explicitly selected.
• High-sync dubbing now uses duration budgets, optional LLM shortening/retry, IndexTTS2 target_duration passthrough, absolute-timeline audio overlay, ASR/leak score fields when available, and output/audio/dubbing_eval.json / .xlsx metrics.
• Shared CLI entrypoint: conda run -n videolingo python -m core.cli doctor, python -m core.cli models list, python -m core.cli run --input <video-or-srt> --source zh --target vi --profile cinematic --tts auto, and python -m core.cli eval dubbing.
• Local video smoke runs can be trimmed directly with python -m core.cli run --input <local-video> --source zh --target en --profile cinematic --llm omlx --tts mlx_indextts2 --smoke-seconds 60 --no-resume; omit --llm to use the same configured oMLX default.
• ASR readback defaults to dubbing_quality.asr_readback_backend: moss-mlx; run python -m core.cli eval dubbing --readback. builtin keeps stable-whisper available, while command accepts {audio}, {language}, and {output_dir} placeholders.
• Dubbing repair loop: python -m core.cli repair dubbing --limit 20 writes output/audio/dubbing_repair_plan.json; add --reasons missing_audio,over_duration to target specific failure classes, --apply to mutate selected rows, --batches 3 to repeat several plan/apply batches, and --full-remap only when chunk timing should be recomputed globally. Applied repairs rebuild output/dub.mp3 and output/AI配音.mp4 by default; use --no-rebuild-output only for intermediate tuning. Batch runs append summaries to output/audio/dubbing_repair_history.jsonl, and over-duration triage writes output/audio/dubbing_over_duration_report.json.
• For English smoke runs, ASR-low rows can switch to qwen3_tts via dubbing_repair.low_content_fallback_backend. Repeated ASR-quality failures become explicit manual_review warnings.
• Repair/rewrite/readback/TTS use the current output/pipeline_state.json target language before config.yaml, so a resumed English smoke is not rewritten or scored as Vietnamese when config defaults drift.
• ASR readback stores a fingerprint of target text, source reference, language, backend/model, and segment audio signatures. Cached scores are reused only when the fingerprint still matches.
• Audio merge fails fast on missing or corrupt segment audio unless silence fallback is explicitly enabled, preventing a broken repair run from producing a silent dubbed video.
• Timeline rescue loop: python -m core.cli rescue timeline writes output/audio/timeline_rescue_report.json / .xlsx; add --write-candidate-tasks to create non-destructive output/audio/tts_tasks_timeline_rescue.xlsx before deciding whether to regenerate all dubbing audio.
• Translation provenance check: python -m core.cli translation status reports whether LLM-generated translation artifacts match the current configured model. Use translation adopt-current --apply to write a manifest for trusted existing artifacts. translation archive is dry-run by default; add --apply to move translation-derived artifacts and GPT logs into output/history/translation_* before a clean retranslation. CLI run / resume blocks stale translation artifacts unless --auto-archive-stale-translation is explicit.
• Repair-time LLM rewrite uses dubbing_repair.llm_timeout_seconds and dubbing_repair.llm_retry_attempts, separate from the longer global translation retry settings.

Common modes:

1. Subtitle-only: download/import video -> ASR or SRT/subtitle extraction -> translation -> SRT/video subtitle output.
2. Subtitle + dubbing: run subtitle pipeline, then TTS task generation -> reference extraction -> TTS -> audio/video merge.
3. Existing SRT / embedded subtitles: skip ASR with Mode 2 or Mode 3, then continue from segmentation/translation.

For quick tuning, enable smoke_test.enabled in config.yaml or the Streamlit sidebar to trim newly downloaded videos to a short sample before running the full dubbing chain.

For detailed installation, API configuration, and batch mode instructions, please refer to the documentation: English | 中文

Current Limitations

1. WhisperX transcription performance may be affected by video background noise, as it uses wav2vac model for alignment. For videos with loud background music, please enable Voice Separation Enhancement. Additionally, subtitles ending with numbers or special characters may be truncated early due to wav2vac's inability to map numeric characters (e.g., "1") to their spoken form ("one").

2. Using weaker models can lead to errors during intermediate processes due to strict JSON format requirements for responses. If this error occurs, please delete the output folder and retry with a different LLM, otherwise repeated execution will read the previous erroneous response causing the same error.

3. The dubbing feature may not be 100% perfect due to differences in speech rates and intonation between languages, as well as the impact of the translation step. However, this project has implemented extensive engineering processing for speech rates to ensure the best possible dubbing results.

4. Multilingual video transcription recognition will only retain the main language. This is because whisperX uses a specialized model for a single language when forcibly aligning word-level subtitles, and will delete unrecognized languages.

5. Speaker identity is diarization-based rather than character-name recognition; MOSS separates voices as S01, S02, etc., but mapping those IDs to named characters still requires user metadata.

📄 License

This project is licensed under the Apache 2.0 License. Special thanks to the following open source projects for their contributions:

whisperX, yt-dlp, json_repair, BELLE

📬 Contact Me

• Submit Issues or Pull Requests on GitHub
• DM me on Twitter: @Huanshere
• Email me at: team@videolingo.io

⭐ Star History

---

If you find VideoLingo helpful, please give me a ⭐️!