[ACL 2024] A New Approach to Chart Understanding and Reasoning - LG AI Research BLOG

Charts are central to data visualization, but it remains a challenging task for AI models to grasp the relationships among complex elements within a chart and answer questions about it. Existing approaches have had limitations in processing visual information and applying it in real-world settings.

To address this, ChartInstruct proposes a general-purpose approach that develops a chart-specific Vision-Language Model(VLM) capable of performing various tasks such as question answering, summarization, and reasoning.

The data construction process is as follows:

• Data collection: Charts of various styles (bar, line, pie charts, etc.) were collected from public datasets such as UniChart and through web crawling
• Information extraction: Gemini Pro Vision was used to automatically extract data tables and titles from chart images
• Question generation: Based on the extracted tables, GPT-3.5/4 was used to generate various types of questions, including summarization, fact-checking, Chain of Thought(CoT) reasoning, and code generation

In the end, a total of 191,000 instruction data points were generated for 70,882 charts. The model architecture uses an end-to-end system combining a Vision Encoder, an Adapter Module, and an LLM to maximize both visual information understanding and language generation capability.