[ACL 2022] Retrieval-Based Knowledge Augmentation Methods for Natural Language Understanding - LG AI Research BLOG

General language models rely on their trained corpus, but combining 'knowledge' information such as relationships between specific words or common sense can significantly boost performance. Various retrieval-based studies are being conducted for this purpose.

REALM introduces a Knowledge Retriever into the language model training process, training the retrieval model and encoder simultaneously in an unsupervised manner. It drew attention at the time by achieving higher performance than the T5(11b) model.

DPR (Dense Passage Retrieval) uses two encoders to find the passage most relevant to a question, and is a study that maximizes training efficiency by utilizing In-batch negatives. This demonstrated deep learning-based retrieval performance that surpassed conventional Elasticsearch.

RETRO, instead of increasing parameters, utilizes trillions of tokens, adding knowledge information from long texts into the model through a Cross-attention structure. This proved that even with the same number of parameters, performance can be greatly improved depending on how data is utilized.

WebGPT is a study that fine-tunes GPT-3, inspired by the way humans use web browsers, to fetch up-to-date information through the Bing search engine. Meanwhile, REINA presents a method that leverages the idea of prompt learning, improving performance by finding similar data within the training data and feeding it into the model together.