AI BriefingExtending LLM Context from 8k to 1M Using Qwen-Agent
Key point
It presents a strategy for building an agent that handles 1M context using an 8k context model, and using this to train long-context models.
Details
The recent trend in LLMs is moving toward long-context models that handle millions of tokens. However, preparing sufficient fine-tuning data for this remains one of the technical challenges.
By leveraging Qwen-Agent, it is possible to build a powerful agent capable of handling 1M context even with a weak model that only has 8k context. This agent is used to generate synthetic data, which is then used to fine-tune a final, powerful model that supports 1M context, following a 3-stage strategy.
The agent evolves through three levels:
- Level 1: RAG (Retrieval-Augmented Generation) - A keyword-based approach. It separates information and instructions from the user query, extracts multilingual keywords, and retrieves relevant chunks via the BM25 algorithm.
- Level 2: Chunk-by-Chunk Reading - To overcome the limitations of keyword matching, the model directly evaluates the relevance of every 512-token chunk. It uses a brute-force strategy of extracting relevant sentences and reusing them as search queries.
- Level 3: Step-by-Step Reasoning - To solve complex questions requiring multi-hop reasoning, the agent thinks step by step and finds answers using tool-calling and other methods.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.