Hugging Face Releases 8B Multimodal Model Idefics2

Hugging Face has released Idefics2, a general-purpose multimodal model capable of processing text and images simultaneously. This model can handle image-based question answering, visual content description, document information extraction, and basic arithmetic operations.

Key Features and Improvements:

• It has 8B parameters and is released under the Apache 2.0 license, allowing free use.
• Compared to the previous model, Idefics1, its OCR (Optical Character Recognition) capability has been significantly enhanced, showing performance competitive with large models such as LLaVA-Next-34B.
• It is immediately integrated into the Transformers library, making it easy to fine-tune for various multimodal applications.

Datasets and Ecosystem:

• It was trained using various public datasets such as Wikipedia, OBELICS, and LAION-COCO, as well as image-code data.
• It also releases 'The Cauldron', a collection of 50 curated datasets for multi-turn conversations, to support the community's multimodal research and development.