Real-time Subtitle Tool for CPU Released

The open-source project hayamimi has been released, generating real-time multilingual subtitles in CPU environments without GPU or cloud API costs. This tool solves the issue of real-time processing latency on CPUs by adopting a method that identifies the language for each utterance and routes it to a model dedicated to that language, instead of using a single large model.

Key Technologies and Performance

All models are INT8 quantized and run in ONNX format, using the sherpa-onnx runtime which does not require PyTorch or a GPU. Benchmark results (based on CPU, not compared against RTX 3090) show an RTF (Real-Time Factor) of 0.07–0.10 on a Ryzen 5 5600X CPU, stably performing real-time processing (below 1.0). Accuracy records WER/CER levels of 2.3%–8.1% by language, with particularly high accuracy in Cantonese compared to existing models.

Key Features and Configuration

• Language-specific dedicated models: Loads optimized models after language detection, such as Korean (Paraformer), English (NVIDIA Parakeet), and Japanese (ReazonSpeech).
• Two-pass processing: Displays finalized subtitles approximately 100ms after the end of an utterance, and re-infers considering the full context when silence is detected to improve accuracy.
• Speaker diarization and translation: Uses DAMO Academy's 3D-Speaker embeddings to label speakers and supports real-time translation via Fugu and M2M-100 models.
• Installation and usage: Installable via pip install hayamimi and runs in environments with FFmpeg and Python 3.10 or higher. The default model download size is approximately 3.1GB.