Building a Backoffice AI Agent: A RAG+MCP-based Knowledge Search System Tailored for PlaceAI

The PlaceAI Platform (PAIP) MLOps team built a Backoffice-dedicated RAG+MCP semantic search agent, creating a knowledge search environment that enables quick discovery of scattered technical documents and issues. The goal was to reduce document search time during OJT and collaboration, and to ease the knowledge gap between technologies and services within the team.

The data layer was designed with OpenSearch serving as both the source repository and log repository, and Milvus as the vector store. The ingestion targets were GitHub Issues, Confluence Wiki, and official documentation such as Milvus, vLLM, and GoFastMCP. Operations were run through a Data Loader that automated collection-cleaning-chunking-embedding-indexing, and a separate Milvus Loader dedicated to embedding experiments. Crawl4AI was applied for body-focused crawling, and an incremental update API was put in place to maintain freshness.

As of now, 11,173 documents from 5 data sources have been indexed. This structure absorbs both internal collaboration documents and external technical documents together, unifying context that was previously scattered across services into a single search layer.

To overcome the limitations of single-method search, the search layer was built as Dense + Sparse hybrid search. An evaluation framework based on BEIR-SCIFACT and LLM-as-a-Judge was established first, and 20 experiments were run to compare embedding models, indexes, and chunking strategies. In the end, the combination of an HNSW index, 2048-token chunks, RRF (Reciprocal Rank Fusion), and an LLM Reranker was adopted to raise the contextual quality of search results.

Notably, for chunking, TokenTextSplitter proved more stable than Markdown header splitting, and 2048 tokens scored better on long meeting notes and log documents. In the reranking stage, the Medium model Reranker improved Judge scores by an average of +0.7 points, producing a more precise final Context for the LLM to reference.

The interface was unified around MCP (Model Context Protocol). On top of FastMCP, the Search Module and Document Module were separated, providing vector search and original document retrieval as standardized Tools, allowing the same calling method to be used in environments like Cursor or Dify. In addition, mcp_log was accumulated in OpenSearch to track queries, tool calls, response times, and returned documents.

Log analysis revealed that unstructured queries themselves, rather than retriever performance, were the main cause of quality degradation. To address this, Smart Search was introduced to first structure the queries. After performing time normalization, author name variant generation, query rewriting, and document type filtering, Hybrid Search and Rerank were called sequentially, completing the flow from search to response as an Agentic RAG pipeline.

The operational results were clear: response satisfaction doubled (2x), response speed improved 1.4x, token usage decreased by 66%, and tool call count decreased by 49%. By raising both search quality and operational efficiency together, backoffice knowledge search was brought up to a level usable in actual work.