How to Implement Continuous Trace Intelligence at Scale

The trace data generated during AI agent operations can amount to millions of tokens, and its structure is far too complex to be handled by typical NLP tools or embedding models. Embedding raw traces as-is introduces severe noise, summarizing the entire trace is prohibitively expensive, and sampling risks missing important bugs.

To solve this, Braintrust developed a Topics solution inspired by Anthropic's Clio paper. The core idea is that instead of embedding the entire trace, an LLM is used to summarize the trace into one or two sentences along a specific dimension (facet). By embedding and clustering these generated summaries, large-scale analysis becomes possible without requiring the trace itself to fit into the embedding model's context window.

The full pipeline consists of the following 6 stages:

• Preprocess: Clean and tokenize the raw trace
• Facet: Summarize along specific dimensions (Task, Sentiment, Issues, etc.) using an LLM
• Embed: Vectorize the summarized text with an embedding model
• Cluster: Cluster the embedded data
• Name: Assign names to clusters
• Classify: Classify the results

For cost efficiency, an optimization was applied that batches multiple facets into a single LLM call. This allows analysis across various dimensions while paying the trace token cost only once.