AI Briefing#llm-inference
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The latest AI and developer news about #llm-inference, with the original source and a short summary.
AWS has secured M3 Ultra Mac Studio units, which are not available for general purchase, and is offering them via the cloud.
An update has been released for llama.cpp that optimizes prompt decoding performance when using MTP.
The llama.cpp b9180 release adds MTP support and partial rollback.
MTP (Multi-Token Prediction) support has been officially merged into the llama.cpp project.
This analyzes the components of the GGUF file format—chat templates, special tokens, sampler settings—and the areas that still need improvement.
Orthrus, based on Qwen3, achieved up to 7.8x speedup through lossless parallel generation.
Orthrus claims up to a 7.8x speedup on Qwen3-8B inference while keeping the backbone frozen.
Orthrus, which uses a Diffusion module to boost token generation speed by up to 7.8x while maintaining the accuracy of existing Transformers, has been unveiled.
The llama.cpp b9158 release includes Flash Attention fixes for AMD RDNA3 GPUs.
This explains in detail the structure of the GGUF file format and the role of metadata it includes, such as chat templates and special tokens.
Hugging Face presented up to a 24% speed improvement by asynchronizing continuous batching.
Sipeed unveiled the K3 RISC-V SBC series, touting 60 TOPS and 30B LLM operation.
The **attention drift** phenomenon in speculative decoding drafters was identified and its performance was improved.
Luce DFlash/PFlash boosts LLM inference performance by up to more than 3x on the AMD Strix Halo iGPU.
It's an MLX-based local AI engine that supports inference up to 4.2x faster than Ollama on Apple Silicon.
ExLlamaV3 has added Gemma 4 support, DFlash optimization, and bug fixes.
Taiwan's Skymizer has unveiled the HTX301, a PCIe AI accelerator that uses older-generation chips and memory to run a 700B-scale LLM at low power.
Triggered by GitHub Copilot's billing shift, local LLM hardware is proposed as a solution for AI self-sufficiency.
NVIDIA has unveiled Star Elastic, a technology that can extract models of various sizes from a single checkpoint.
Fixed vLLM V1's logprob, cache, and update discrepancies to align with the V0 trajectory.
A UCSD research team achieved an average 3.13x speedup with DFlash on Google TPU v5p.
Benchmark analysis results on LLM inference performance, power efficiency, and MoE model running performance across Apple Silicon chipsets have been released.
Analysis results of LLM inference performance and power efficiency for Apple M4/M5 and A18 Pro chips have been released.