Cohere Framework for Choosing Between Shared and Dedicated Inference for Embedding and Rerank Models

Sizing infrastructure for Embedding and Rerank models cannot rely on request volume alone; the request shape and utilization rate are the primary drivers of cost-efficiency. A new decision framework from Cohere outlines how to choose between shared and dedicated deployment based on these factors rather than simple throughput metrics.

Request Shape and Compute Load

The compute load varies significantly by request type. Batch indexing handles large volumes of tokens per request (e.g., ~20,000 tokens for 100 texts), prioritizing throughput, while search queries involve minimal tokens (~8 tokens) and prioritize latency. For Rerank models, compute load scales linearly with the number of candidate documents. Retrieval depth is a critical variable: reranking 20 candidates requires ~4,400 tokens per search, while 50 candidates require ~11,000 tokens, and 100 candidates require ~22,000 tokens.

Traffic Patterns and Utilization

Sustained traffic favors dedicated inference because infrastructure remains highly utilized, whereas bursty traffic favors consumption-based (shared) inference to avoid paying for idle capacity. The economic crossover depends on the workload's specific request profile and traffic consistency, not just peak volume. For example, a catalog embedding workload with ~20 requests per minute may reach the break-even point for dedicated infrastructure, while query embeddings require tens of thousands of requests per minute to justify dedicated capacity.

Latency and Deployment Strategy

Peak throughput does not equal achievable capacity. Batch indexing can operate at maximum throughput, but interactive workloads like search and reranking require headroom to meet latency targets. Reranking specifically trades off documents processed per second against searches served per second as candidate lists grow. Many applications adopt a hybrid approach, using dedicated capacity for stable catalog indexing and shared inference for spiky end-user query traffic. The decision framework prioritizes understanding the work contained in each request and the consistency of traffic over simple volume metrics.