Inflearn Develops 'Video Screen Translation' Feature That Translates Text Within Video Frames

Inflearn is developing a 'Video Screen Translation' feature that recognizes text within the screen during video playback and overlays translated text. This feature has been experimentally applied to some lectures, such as DeNA × AI Day 2026, and adopts an overlay method that draws background color boxes and translated text on top of the original video.

Technical Implementation

Unlike existing encoding methods, this approach uses a 3-layer structure that masks the original text area with a background color box and then renders the translated text on top. This method offers advantages such as ease of adding languages, the ability to improve quality simply by modifying translation data, and reduced network transmission burden. The data is stored in JSON format and includes screen configuration retention time (events), text positions (bbox), and color information (texts).

Text Extraction and Preprocessing

The basic pipeline consists of OCR (text recognition) followed by translation and position overlay. To enhance the naturalness of translations, a VLM (Vision Language Model) is used to exclude unnecessary text such as UI buttons or code and group text into paragraphs. To address the issue of existing document layout analysis models being unsuitable for video, the screen is divided into blocks using the XY Cut algorithm from 1984, after which the VLM determines whether to group them into paragraphs.

Frame Processing Optimization

To address the processing burden of approximately 3,600 frames for a 1-hour video and the issue of overlay flickering, a reuse strategy was introduced. Previous results are reused if the pixels are nearly identical to the previous frame or if they meet the criteria for text similarity (≥0.85) and IoU (≥0.5). Additionally, boxes containing only numbers or symbols are excluded without model calls, and the VLM is called only when the screen changes to ensure stability.

Color Extraction and Translation Stage

Background and text colors are estimated by sampling pixels within the box and performing k-means (k=2) clustering in the RGB space. The group with more pixels is classified as the background color, and the group with fewer pixels is classified as the text color. In the translation stage, deduplicated original text is batch-requested to an LLM, and surrounding text such as the video title is included in the prompt to prevent terminology inconsistencies. However, there are limitations in processing text over photos or gradient backgrounds.