AI Briefing#llm-serving
The latest AI and developer news about #llm-serving, with the original source and a short summary.
The latest AI and developer news about #llm-serving, with the original source and a short summary.
SGLang now supports the Transformers backend, enabling high-performance inference for any Transformers-compatible model.
This analyzes the queue-blocking problem caused by long prompts in vLLM's chunked-prefill strategy and presents an optimization approach.
Hugging Face now supports Featherless AI, a serverless inference service, as a new Inference Provider.
This proposes introducing fair scheduling at the API server layer to prevent specific users from monopolizing requests.
Fireworks.ai has been added as a serverless inference provider on the Hugging Face Hub, supporting major LLMs.
One-click model deployment via FriendliAI is now available on the Hugging Face Hub.
Hugging Face's TGI has launched a feature that dynamically serves multiple LoRA adapters from a single base model.
Hugging Face has launched a Messages API compatible with the OpenAI API to support transitions to open LLMs.
Hugging Face's LLM serving solution TGI is now officially supported on AWS Inferentia2 and Amazon SageMaker.
HuggingFace explains the technical process of optimizing the BLOOM model's inference server, improving latency by 5x and throughput by 50x.
Compares the optimal hardware configuration and benchmarks for maximizing inference performance of the 176B-parameter BLOOM model using DeepSpeed and Accelerate.
Hugging Face has unveiled a technical approach that improves Transformer inference speed by 100x through library optimization and hardware-specific compilation.
Separating Encoder, Decoder, and LLM and applying llm-d and USP optimization increased throughput by 2.1x.