Real-Time Multilingual Video Subtitle Spotting on Resource-Constrained Devices

For on-device real-time subtitle spotting, a 5-stage pipeline was designed consisting of content classification, subtitle tracking, text line detection, text recognition, and TTS.

Input frames are first filtered to determine whether they are movie/entertainment content with subtitles, reducing false detections and unnecessary computation on non-subtitle screens. A subtitle tracker using two consecutive frames detects the moment a new subtitle appears.

Subtitle location detection is built on an SSD + MobileNetV3 backbone, with quantization-aware training (QAT) applied to reduce rounding error while maintaining a lightweight model. Detected text lines are normalized to a height of 48px before being passed to the recognizer.

Text recognition uses a CRNN-CTC architecture, combined with a probabilistic language model and token-passing algorithm for contextual correction. On the CNN side, the last 5 layers were removed from MobileNetV3-small, and the stride was adjusted to reduce the X-axis sequence length by a factor of 8.

Training data was built from a self-generated synthetic dataset.

• Content classification data included approximately 2 million frames of target content and approximately 600K frames of non-target content.
• The tracker used a binary classification dataset made of consecutive frame pairs.
• Detection data was organized as JSON containing subtitle layout metadata and reference text.
• For the recognition model, 6 synthetic datasets were created per language, with some English data reused for training other languages.

Evaluation was conducted on TV sets using independent test sets per language, each consisting of 64 five-minute videos and 8 subtitle styles. Overall results achieved 97%+ E2E word accuracy across 7 languages, with latency under 200ms.

This technology has already been commercialized in Samsung TV's Audio Subtitles and Live Translate services, and presents potential for expansion to IoT devices and other digital appliances.