Latent Briefing: Efficient Multi-Agent Memory Sharing via KV Cache Compression

Multi-agent systems excel at collaboration and reasoning, but as intermediate reasoning keeps accumulating between the orchestrator and workers, token efficiency degrades sharply. Existing LLM summarization is slow, RAG is brittle, and passing all context as-is drives up cost and latency.

Latent Briefing, proposed here, uses the model's attention patterns instead of text summarization to keep only the important parts and discard the rest at the KV cache level. In other words, it compresses and passes along only the accumulated memory of the orchestrator that the worker actually needs.

The foundation is Attention Matching(AM)-based KV cache compaction. Originally, AM processed each layer and head independently, requiring 40 layers × 8 KV heads = 320 sequential solves for Qwen3-14B. Each head goes through token selection, NNLS-based β estimation, and ridge regression-based C2 reconstruction—good quality, but taking over 30 seconds, making it unsuitable for real-time agents.

To address this, three things were changed.

• Task-guided query vectors: Instead of extracting queries from the context itself, queries are generated from the current worker call's task prompt, prioritizing preservation of memories important to that specific task.
• Shared token selection: Rather than per-head top-k, a common mask is built from a global score that sums scores across all layers/heads. Since Qwen3-14B has no optimized head budget, uniform weighting was used, but consensus across multiple heads proved sufficiently effective.
• MAD normalization thresholding: Instead of a fixed count, positions exceeding median + threshold · MAD are retained, allowing compaction strength to adapt to document and question difficulty.

Thanks to this shared mask, the 320 solves could be combined into a batched tensor operation, and adaptive batch sizing along with KV prefix caching allowed reuse of 90%+ of prior representations. With in-place softmax, staged sequential execution, CPU offloading, chunked prefills, and automatic batch size halving on OOM, compaction overhead was reduced from 30+ seconds to a median of about 1.7 seconds.

Experiments were conducted on LongBench v2 using a Claude Sonnet 4 orchestrator paired with a Qwen-14B worker. Across a total of 126 questions with documents ranging from 0 to 100k tokens, the baseline was compared against four thresholds: t = -1.0, 0.0, 1.0, 2.0. The results were as follows.

• Accuracy was on par with or better than baseline, improving by up to +3pp across all conditions.
• At the optimal threshold, worker tokens decreased by 42–57%, and total tokens dropped by 21–31%.
• For medium-length (32k–100k) documents, median token savings reached as high as 49%.
• Worker model token consumption decreased by 65%.

The optimal point varied by threshold. Longer documents needed broader coverage, so weaker compaction like t = -1.0 worked better, while for harder questions, the orchestrator's speculative reasoning became noise, so stronger compaction like t = 2.0 fit better. Conversely, for shorter, easier documents, moderate compaction around t = 1.0 was most stable.

The limitations are also clear. The Claude Sonnet 4 orchestrator is non-deterministic, so decomposition can vary even for the same question, and the experiments were limited to LongBench v2 alone. Nonetheless, overall, Latent Briefing addresses multi-agent memory sharing through representation-level compaction rather than text summarization or retrieval, achieving near-real-time token savings while maintaining accuracy.