VisTW: A Taiwan-Specific VLM Benchmark Released

NTU MiuLab has released the VisTW benchmark to evaluate the real-world reading capabilities of Vision-Language Models (VLMs) in Traditional Chinese environments. It provides data reflecting local culture and context, such as Taiwanese road signs, receipts, and textbook diagrams, which were overlooked by existing benchmarks centered on English and Simplified Chinese.

VisTW Benchmark Composition

VisTW consists of two subsets.

• VisTW-MCQ: Comprises 4,770 multiple-choice questions extracted from actual exam questions across 21 academic disciplines, including various visual data such as charts, circuit diagrams, sheet music, and medical images.
• VisTW-Dialogue: Contains 141 open-ended questions about everyday scenes in Taiwan, graded on a scale of 0 to 10 by an LLM judge to measure understanding of cultural context.

Twinkle Eval Integration and Validation

The open-source evaluation framework Twinkle Eval supports both VisTW subsets starting from v2.10.0. Comparison validation against the official implementation using 253 sample data points showed a very low accuracy difference of 0.45 percentage points (81.01% vs 80.56%), ensuring reliability.

Key Technical Considerations

• Extraction Logic Differences: The official implementation uses a three-stage extraction method that calls an LLM if regex fails, but Twinkle Eval excludes this to prevent confusion between model performance and parser performance, reporting unparsed_rate instead.
• Token Limit Caution: For reasoning VLMs, setting max_tokens to 2048 can result in 19% of responses being truncated, potentially causing a 19 percentage point drop in accuracy; therefore, the template is set to 8192.