LG AI Research: Applying Image Understanding and Generation Technology through the Hwadamsup Project

AI technology is innovating the creative process of art, serving as a companion to artists. The Hwadamsup Project is a successful media art case in which AI collaborated with humans, performing everything from theme setting to recommending visual and auditory effects.

Image Understanding is a technology that uses multimodal models such as VLM and LMM to perform image-based question answering. The existing LLaVA structure passes information to the LLM through a Vision Encoder and Projection Layer, but it has the drawback of relying on GPT-generated data, which can raise licensing issues.

In response, LG AI Research proposed the EXAONE LMM structure. This model has the following characteristics.

Image Generation technology also plays a key role. LG AI Research developed its own image generation model, named EXAONE Atelier, and is using it in the design processes of affiliate companies.

To overcome the limitations of existing models that had not been trained on specific data for the Hwadamsup Project, a Customize Image Generation model was developed. Image-Text Pair data was extracted from Hwadamsup footage using the EXAONE Image Understanding model, and tuning techniques such as LoRA and Dreambooth were applied to complete a customized model embodying the philosophy of Hwadamsup.