AI Briefing#llm-inference
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The release uses an LLM to natively edit weights, outperforming Unsloth and EXL3 in KL divergence and top-1 accuracy at 128K context.
The project replaces all floating-point operations with exact integer arithmetic, ensuring bit-identical logits across different GPUs and CPUs without requiring pinned kernel orders.
The method saves 28 seconds on 120K-token prompts using ridge regression maps, with RULER scores dropping 1.2-1.4 points.
The Apache 2.0 research prototype verifies that a specific model generated a response using zero-knowledge proofs, with a demo running Llama 4 Maverick at 110 prefill tokens/sec.
The second-generation accelerator supports 20B to 100B parameter models at 10-30 tokens per second, with projected 6.7x performance gains via Samsung LPDDR5x-PIM.
Built on llama.cpp, the tool automatically allocates model weights to VRAM, RAM, or flash storage to maximize prefill and decode speeds on heterogeneous hardware.
The method removes inter-round dependencies by drafting successor blocks during verification, achieving higher average speedups than DFlash across seven benchmarks.
The new routers weigh eleven factors per request and handle over 2.5T tokens monthly, with Auto Code matching Claude Opus 5.5 performance at 74% lower cost.
Modular has released a free 57-page handbook covering LLM inference metrics, optimization, and operational costs.
The framework parallelizes draft sampling and supervision across independent branches to improve inference efficiency for large language models.
ChatGPT, Claude, and Gemini now use metered usage limits, but none publishes the specific cost of individual messages or the total allowance size.
An autonomous agentic loop discovers lossless system optimizations that nearly double the speedup of parallel drafters over autoregressive decoding.
The company routed 18 million requests through Vercel AI Gateway in its first 10 days, enabling over 2,000 internal projects and provisioning expert access in seconds.
Baseten unveiled joey-1, a 2B decision model, offering a price of $0.01 per 1M API input tokens.
The method maintains near-full-KV accuracy while supporting 700B scale models at 96-99% of DSA throughput.
The open-source tool unlocks hardware limits on GA102-GA106 chips, boosting BF16 GEMM throughput by up to 81% and LLM prefill speeds by 14-36%.
ISTA DASLab releases GSQ-RCO quantized GGUFs for the 763.2B parameter DeepSeek-V4.1-Flash-1017, achieving 100% benchmark retention at 2.00 bpw on a 12GB GPU.
The method uses pair-norm analysis to allocate precision to salient channel pairs, maintaining accuracy down to 1.5 bits where baselines degrade sharply.
The method outperforms DSpark by improving average throughput by 7.97% and acceptance length by 8.50% across seven benchmarks.
Unclosed connections after user cancellation or retries cause vLLM to generate up to 175 wasted tokens per task, reducing on-time answer rates to 11%.
The update introduces a command to drain engines and pass the GPU to a VM, alongside gateway fallbacks and security fixes.
The tool operates as a llama.cpp fork that modifies models at runtime, requiring no additional VRAM or pre-processed files.
The draft models combine a five-layer DFlash backbone with a learned bigram selector to propose multiple candidate continuations for target-model verification.
The open-source Kubernetes-native stack supports vLLM and SGLang across heterogeneous hardware, claiming up to 3x higher output throughput and 2x faster TTFT.