[Usage Guide] How to Use kt cloud AI RAG (Retrieval-Augmented Generation) - Boosting Performance Through Context Optimization

The quality of LLM answers is more heavily influenced by what context is fed in than by the prompt. For questions where accountability is on the line—such as the latest documents, terms and conditions, SLAs, or security guides—how you select and refine the supporting documents matters more than polishing the prompt.

The prompt is the instruction, and the context is the evidence. Instructions like "answer based on policy" or "organize into a table" only set the direction; the actual power to produce an answer depends on how clearly accurate evidence—document excerpts, tables, logs, clauses—is provided.

There are four representative patterns where context breaks down.

• When many irrelevant paragraphs are mixed in
• When old and new versions are included together
• When there are many duplicate sentences
• When the key evidence is far from the question

This problem is not well solved by writing a longer prompt. The solution is to put in less but more accurately, which lies in reconstructing the input bundle.

TopK is the value that determines how many candidates to retrieve during the search stage. Blindly increasing TopK can raise the probability that the correct answer is included, but it also increases noise, duplication, and conflicts at the same time. In practice, it's more stable to separate the number of search candidates from the number actually inserted into the context, and use rerank to keep only the top few pieces of evidence.

Even with the same information, reasoning becomes much more stable when it's structured. The recommended input structure is as follows.

• Rules (priority) → question reformulation → evidence (citations) → answer format
• Include metadata such as title, section, date, version, and source
• Limit evidence to 3-5 items, and connect each to which part of the question it supports

There are clear improvement points that show immediate effects in operations.

1. Reduce TopK + strengthen rerank to keep only the few accurate items
2. Remove duplicates to reduce cost, latency, and distortion of importance
3. Filter by date/version to block the mixing of old versions
4. Extract only sentences directly connected to the question to increase evidence density
5. Instead of generic summarization, use query-aware compression to compress only the parts relevant to the question
6. Make evidence citation the default, and if there's no evidence, switch to asking a follow-up question or stating "insufficient evidence"

External public materials point in the same direction. Anthropic disclosed that combining Contextual Retrieval with reranking reduced the Top-20 retrieval failure rate from 5.7% to 1.9%, and OpenAI notes that output tokens directly affect latency and cost. Ultimately, RAG quality is determined far more by input refinement and token management than by the model itself.

Fixing a template like the one below stabilizes quality.

Rules (priority)
1) Latest document > old version
2) No speculation on content not in the evidence
3) Answers must cite evidence; if none, ask a follow-up question

Question reformulation
- Output the user wants: (e.g., summary including conditions/exceptions, table format)

Evidence (3-5 items, including metadata)
[Evidence 1] Title/Section/Date/Version/Source + sentence excerpt directly connected to the question
[Evidence 2] ...
[Evidence 3] ...

Answer format
- Conclusion
- Evidence citation
- Exceptions/cautions
- Follow-up confirmation question (if needed)

Ultimately, there are three keys to stabilizing kt cloud RAG quality: version criteria that prevent old and new versions from mixing, an approach that reduces TopK and precisely selects via rerank, and a response policy that enforces evidence citation as the default. The result is not a more plausible-sounding answer, but a verifiable answer.