AI Briefing14x Faster Embeddings: How the ONNX Path Was Rebuilt in Manticore
Key point
Manticore Search has introduced an ONNX Runtime backend, boosting embedding inference speed by an average of 14x compared to before.
Details
Manticore Search has released a new ONNX Runtime backend with Manticore Search 27.1.5. It is the result of engineering work aimed at resolving the CPU resource waste and bottlenecks that occurred in the existing SentenceTransformers/Candle-based path.
The key performance improvement metrics are as follows:
- Average 14x speed improvement: On the same hardware (16 cores/32 threads) and the same model, performance has been dramatically improved from the previous 5~11 docs/sec to a range of 70~230 docs/sec.
- Reduced latency: Single-insert latency for a single client has been dramatically cut to ~14ms, down from the previous 200ms or more.
- Optimized parallel processing: It delivers consistent performance improvements regardless of the number of client threads or batch size, with maximum performance achieved especially when using high batch sizes (32~128).
This update was achieved by disabling intra_op_spinning and changing the way document batches are processed inside workers. Now, any model that supports Hugging Face's .onnx files will use this new high-speed path by default in Manticore.
This summary was generated automatically by AI. Check the original for the author's claims and context. Copyright belongs to the original author.
Our guide explains how the AI works. Report summary errors, attribution issues, or removal requests via Contact.
Related
Embedding Space Conversion Without Paired Data Raises Vector DB Security Threats
Amazon DynamoDB Supports Real-Time Vector Search at Any ScaleStanford and NVIDIA Release 'Contrastive Language Model' CLM with 9x Faster Inference
Meta Introduces Muse Realtime Avatar for Synchronized, Expressive Video Generation