We Had an AI Solve Financial Security Quizzes: A 2025 Financial AI Challenge Excellence Award Review

The Kakao Bank LostCow team recorded 1st place in Public, 2nd place in Private at the 2025 Financial AI Challenge, ultimately winning the Excellence Award. The goal was to build an AI model usable in financial security practice that could handle both multiple-choice and short-answer questions well with a single LLM.

The core constraints were clear: performance had to be maximized within a configuration that used 24GB VRAM, a 270-minute inference limit, and no external dependencies, while remaining reproducible. In particular, multiple-choice questions required getting the exact correct number, while short-answer questions depended not only on semantic similarity but also on keyword recall, making the balance between retrieval and generation important.

After reviewing several candidates, the team settled on Qwen3-30B-A3B-Instruct-2507 as the model. The reasons were its strength in Korean and long-text processing, and the fact that its MoE architecture reduced the actual active inference parameters despite being a 30B-class model, making it operable within a 24GB environment.

Training was designed as a 2-stage CPT.

• Stage-1: Law-centered CPT to first inject the standards and precise expressions of financial security
• Stage-2: Guideline-centered CPT to reinforce procedural explanation and case-based narrative ability

To reduce catastrophic forgetting, LoRA was also applied differently at each stage. In Stage-1, the scope was opened broadly to include MLP + MoE Gate + embedding, training about 18% of parameters, and rsLoRA was used at Rank=128. In Stage-2, the scope was narrowed to training only Query/Value attention projection, at about 2%.

Data was collected largely divided into laws and guidelines/manuals, with preprocessing varied according to purpose. For CPT guidelines, the original flow and context were preserved as much as possible, while for RAG data, a 13-step filtering process was applied to remove control characters, CSS/HTML residue, repeated characters, and abnormal whitespace for the sake of BM25 performance. Even for the same source material, RAG-specific cleaning was necessary, and that difference alone resulted in a 4.5%p performance gap.

For retrieval, BM25 was chosen over embedding-based methods. This is because the finance/legal domain places importance on core keywords and precise terminology, and short-answer grading also weighs keyword recall heavily. Chunking used a Sliding Window with 1,000 tokens / overlap 100, and MeCab morphological analysis was used to make the system robust to Korean particle variations.

In Top-k experiments, 10 was optimal. With 5, information was insufficient; from 20 onward, noise increased; and at 40, the average token count exceeded 15,000, even causing OOM. Ultimately, this confirmed that operational stability—being able to run reliably to completion—matters just as much as performance.

Overall, this pipeline was a case that combined domain knowledge internalization (CPT) with evidence-based response (RAG) to raise response quality suited to financial security questions even within limited resources.