LG AI Research 221

General language models are trained using basic corpora, but there have been ongoing attempts to internalize Knowledge Information, such as relationships between specific words or common sense, into models.

REALM is a model that introduces a Knowledge Retriever into the language model training process. Unlike previous research, it trains the retriever in an unsupervised learning mode, and by combining external information into the input through a Knowledge-Augmented Encoder, it achieved higher performance than existing models at the time, such as ORQA or T5(11b).

DPR(Dense Passage Retrieval) is a model widely cited in vector search research, using two encoders each specialized for questions and passages to find the most relevant passage. In particular, it improved training efficiency by utilizing In-batch negatives, and demonstrated performance surpassing existing TF-IDF/BM25 methods.

RETRO is an approach that utilizes data containing trillions of tokens instead of increasing parameters. Similar to REALM, it uses the idea of adding relevant knowledge to the input, but it has the characteristic of being able to process longer document chunks.