AI Briefing#gpu-inference
The latest AI and developer news about #gpu-inference, with the original source and a short summary.
The latest AI and developer news about #gpu-inference, with the original source and a short summary.
The release expands hardware support to include two-GPU setups and achieves 75.5 tok/s on a single RTX PRO 6000 card.
The system uses a lightweight quantized draft model to predict expert usage, reducing PCIe I/O bottlenecks on commodity hardware.
The open-source Project Maya allows running the 321-billion-parameter GLM-5.3-Flash model on consumer hardware like a single RTX 5090 or four RTX 4090s.
The new Apache 2.0 licensed engine abstracts OpenGL ES, OpenCL, Metal, and WebGPU to enable real-time ML experiences across platforms.
Switching to g7e.2xlarge instances improved generation speed by 3-5x and reduced new user wait times by 57.5%
Instead of using hosted APIs, Netflix built its own LLM serving infrastructure to directly operate the entire stack from model deployment to inference.
MTP speculative decoding has been added to llama.cpp, making Qwen3.6 27B up to 2.44x faster.
On dual RTX 5060 Ti 16GB GPUs, Qwen3.6 27B ran at about 60 tok/s.
Qwen3.6 27B IQ4_XS recorded 22 tok/s on an RTX 5060 Ti 16GB.
Running Qwen3.6-27B on a single RTX 3090 achieved 85 TPS at 125K context.
By sharing Wasm memory with the GPU, inference runs zero-copy on Apple Silicon.
On A100 40GB, MIG significantly lowered protected-tenant p95 compared to shared vLLM.
Using turbo3, achieved about 40 tok/s running Qwen3.6 35B A3B at 260k ctx on a 12GB VRAM 3080.
On an RTX 5080·64GB Windows 11 setup, Llama.cpp was about 2x faster than LM Studio.
Verified Qwen3.5-122B running at 198 tok/s on 2x RTX PRO 6000 Blackwell.
To increase the expressiveness of ad ranking models, the team moved away from the existing Two-Tower architecture and built a new GPU-based serving architecture.