AI Briefing#model-serving
The latest AI and developer news about #model-serving, with the original source and a short summary.
The latest AI and developer news about #model-serving, with the original source and a short summary.
PaddleOCR 3.5 has added Transformers backend support.
Osaurus has been released as an MCP server that lets you use local and cloud AI models together on Mac.
Thinking Machines has released a research preview that handles audio, video, and text in real time.
AWS has organized its infrastructure stack for foundation model training and inference, along with the latest GPU/EFA configurations.
ExLlamaV3 has added Gemma 4 support, DFlash optimization, and bug fixes.
Supports real-time conversation and high-quality TTS simultaneously through OpenAI API compatibility and a process isolation architecture.
Analyzes memory and compute scaling in LLMs along with batching strategies for efficient inference.
Netflix routed model serving traffic by purpose using Switchboard.
Z.AI identified the cause of GLM-5's intermittent output anomalies as an infrastructure issue.
The AI inference market is fragmenting by use case.
vLLM Recipes has been revamped to support hardware-specific serving configs and command generation.
It presents a methodology for optimizing hardware-specific costs and latency for large-scale classification and embedding workloads.
Hugging Face's TGI is introducing a multi-backend architecture that supports various inference engines such as vLLM and TensorRT-LLM.
Hugging Face has launched HUGS, an inference service that makes it easy to deploy open models on your own infrastructure.
HuggingFace kept the base model in memory and dynamically swapped only LoRA adapters, improving inference speed by 300%.
Hugging Face has launched 'Inference for PRO,' offering PRO subscribers a dedicated API for high-performance models with higher usage limits.
Rocket Money successfully deployed a BERT model with more than 4,000 classes into production using the Hugging Face Inference API.
This explains how to efficiently deploy the DeepFloyd IF model, which has a multi-stage architecture, using BentoML.
This guide explains how to deploy the MusicGen model using Hugging Face's Inference Endpoints and a custom handler.
Hugging Face has launched a TGI-based LLM inference container dedicated to Amazon SageMaker.
Hugging Face revealed the performance and cost benefits gained from migrating its internal model serving from AWS ECS to Inference Endpoints.
Kakaopay shared its process of comparing and optimizing frameworks to improve the consistency and performance of model serving.
Hugging Face has launched Enterprise Hub, integrating enterprise-grade security and collaboration tools.
This explains how to deploy Hugging Face's TensorFlow vision models using TF Serving.