AI Briefing#inference
The latest AI and developer news about #inference, with the original source and a short summary.
The latest AI and developer news about #inference, with the original source and a short summary.
Qwen3.6 27B and 35B-A3B were compared across multiple engines on 2x 5060 Ti 16GB.
Qwen3.6-27B was run on 2x 3090 GPUs at 118 tokens per second using a combination of vLLM Docker with INT4 and MTP.
The HTX301-based PCIe card supports 700B LLM inference at about 240W.
A single H100 80GB benchmark confirmed the advantage of Gemma 4 E2B and Qwen 3.6 A3B FP8.
A PR supporting Cohere MoE models has been submitted to vLLM.
On 4x Tesla M40 12GB with an fp4 setup, Qwen3.6 27B speed was reported at 26 tok/s PP and 8 tok/s TG.
The top score in the public inference challenge for Tesla T4·Qwen2.5-0.5B is 3,536 tok/s.
vLLM Recipes has been revamped to support hardware-specific serving configs and command generation.
Google unveiled TPU 8t for training and TPU 8i for inference.
Kimi Vendor Verifier validates implementation deviations across inference providers.
A single checkpoint switches among 8 presets, delivering up to 10.7x faster performance.
Google unveiled the training-focused TPU 8t and the inference-focused TPU 8i.
TPU 8i and 8t for agents were unveiled, strengthening inference and training performance respectively.
A Metal kernel implementation combining INT3 model compression with INT2 KV cache has been released.
Even with 32GB VRAM, llama.cpp's `--fit` ran Qwen3.6 Q8 at a 256k context at 57 t/s.
Kimi has open-sourced a tool for verifying inference accuracy across open-source model deployment environments.
METR's time horizon keeps growing, but we examine whether the hourly cost of AI agents is rising along with it.
Unweight reduces LLM weights by 15-22%, freeing up about 3GB of VRAM on H100.
Kimi Linear makes cross-datacenter Prefill/Decode disaggregation practical, lowering costs.
AI inference chip startups raised **$8.3 billion in 2026**, intensifying the race against Nvidia.
Parcae uses a looped recurrent structure to make 770M aim for 1.3B Transformer-level performance.
OpenAI will spend over $20 billion on Cerebras chips over three years and receive an equity stake as well.
Nvidia is solidifying its AI infrastructure dominance through supply chain, financial, and China strategies.