AI Briefing#llm-inference
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The latest AI and developer news about #llm-inference, with the original source and a short summary.
Ollama has raised $65 million in a Series B round, accelerating its push into the enterprise AI agent market.
AI21 proposes a strategy that optimizes cost and speed for SWE agents by flexibly allocating compute resources according to task difficulty.
vLLM's Transformers modeling backend has been optimized to deliver throughput on par with or faster than existing custom implementations.
The paper proposes MBD-LM, a Multi-Block approach designed to improve the parallel decoding performance of Block Diffusion Language Models (BD-LMs).
An experimental Scatter sampler has been introduced to llama.cpp, which locally smooths the probability distribution to improve generation quality.
ReFreeKV, a threshold-free KV Cache compression technique that automatically adjusts the compression budget based on input data, is proposed.
Proposes a new paradigm that compiles natural language specifications into lightweight neural network artifacts to dramatically lower inference costs.
This work proposes a new policy learning method that maximizes the efficiency of token selection (unmasking) in Diffusion LLMs through reinforcement learning.
DeepSeek has released DSpark, a speculative decoding framework that dramatically improves LLM response speed, under the MIT license.
A Pull Request to support the DeepSeek V4 model has been merged into the llama.cpp project.
CUDA backend synchronization optimizations and asynchronous copy functionality have been added to the ggml library.
A JetSpec research paper has been released, dramatically increasing LLM inference speed through parallel tree drafting technology.
OpenAI partnered with Broadcom to unveil 'Jalapeño,' a proprietary intelligence processor that maximizes LLM inference performance.
An analysis of large-scale data from the r/LocalLLaMA community diagnoses the current state of the local AI ecosystem.
Lemonade v10.8 introduces automatic memory management, cloud offload, and MCP gateway functionality.
DFlash technology, developed jointly by Z Lab and Modal, achieved LLM inference performance far superior to existing MTP methods.
The NVIDIA Blackwell GB300 NVL72 platform recorded overwhelming performance in the AgentPerf benchmark, with power efficiency 20 times higher than Hopper.
Proposes a new infrastructure model for sharing and trading pre-computed KV Cache to reduce redundant computation by AI agents.
MiniMax has announced MSA, a new sparse attention technology that maximizes computational efficiency for large-scale context processing.
EAGLE3, an inference acceleration technology, has been merged into llama.cpp, improving inference efficiency.
InfiniteKV has been released, dramatically extending a model's context length by compressing the KV cache and storing it in RAM or on disk.
Google has unveiled DiffusionGemma, an experimental model that moves away from the sequential generation approach of existing LLMs and generates blocks of text simultaneously.
A lightweight retriever that compresses DeepSeek-V4's CSA KV-cache to maximize GPU memory efficiency has been released.
OSCAR, a technique using spectral covariance-aware rotation to reduce quantization loss in KV Cache, has been released.