AI Briefing#vllm
The latest AI and developer news about #vllm, with the original source and a short summary.
The latest AI and developer news about #vllm, with the original source and a short summary.
Fujitsu has released OneComp, an open-source Python library that improves the quantization accuracy of LLMs.
A 32-bit overflow in vLLM's Mamba CUDA kernel caused logprob mismatches.
Mistral Small 4 has been released, combining reasoning, multimodality, and coding into one model.
Netflix built a flexible LLM Post-Training Framework on top of Ray, PyTorch, and vLLM.
Tuning and autoscaling vLLM for bursty traffic reduced throughput bottlenecks and latency.
The cause of rare gibberish was traced to vLLM's Mamba scheduler and cache.
Mistral AI shared the technical process of tracking down a memory leak that occurred in a distributed serving environment of vLLM.
Mistral has announced the Mistral 3 series, its next-generation models released under the Apache 2.0 license.
Rebellions and SqueezeBits held a hands-on workshop where participants ran vLLM directly on NPUs.
Hugging Face has added Public AI as a new inference provider to support public AI models.
This is an event page reporting on the first vLLM meetup held in Korea.
Rebellions unveiled a high-efficiency LLM serving technology compatible with vLLM by optimizing FlashAttention and other techniques for NPUs.
Hugging Face and AMD collaborated to release custom kernels specifically for MI300X that maximize the performance of Llama 3.1 405B.
This analyzes the queue-blocking problem caused by long prompts in vLLM's chunked-prefill strategy and presents an optimization approach.
TRL has integrated vLLM into the training process, reducing GPU resource waste and improving GRPO training efficiency.
Hugging Face has launched an Inference Endpoints option that boosts Whisper transcription performance by up to 8x using vLLM.
Qwen3 is an open-weight model with a hybrid thinking mode and support for 119 languages.
This proposes introducing fair scheduling at the API server layer to prevent specific users from monopolizing requests.
Qwen has released the Qwen2.5-1M series, open-source models supporting up to 1 million tokens of context length.
Hugging Face's TGI is introducing a multi-backend architecture that supports various inference engines such as vLLM and TensorRT-LLM.