AI Briefing#memory-optimization
The latest AI and developer news about #memory-optimization, with the original source and a short summary.
The latest AI and developer news about #memory-optimization, with the original source and a short summary.
The new ASRN architecture uses learned hash tables to locate and copy prior context occurrences, achieving memory usage linear in sequence length.
Hayder Tirmazi's optimizations for prompt lookup decoding in llama.cpp, combined with a PR by Daniel Lemire, achieve up to 140x faster drafting and up to 2.6x less memory usage.
Cloudflare recovered 100TB of RAM across its global infrastructure by optimizing the consistent hashing structure in Pingora.
Alexia Jolicoeur-Martineau's research team published a paper combining sliding window attention and attention sinks to replace linear attention.
Ollama's KV cache quantization settings are applied globally, which can cause performance degradation for specific models.
Cloudflare optimized its DNS cache storage method, saving 100TB of memory and improving performance.
Sakana AI's DiffusionBlocks technique, presented at ICLR 2026, significantly reduces memory usage during deep learning training.
AirLLM has been updated to enable Kimi K3 (2.8T) to run on 4GB VRAM via sparse MoE model streaming.
Chunked KL loss reduced Knowledge Distillation memory usage from approximately 85GB to 5GB for 32K contexts.
WASTE ran the 2.78T Kimi K3 via disk streaming on a 64GB MacBook Pro.
Analyzes the memory spike problem that occurs during Cross Entropy calculation in LLM training and presents solutions.
ReFreeKV, a threshold-free KV Cache compression technique that automatically adjusts the compression budget based on input data, is proposed.
InfiniteKV has been released, dramatically extending a model's context length by compressing the KV cache and storing it in RAM or on disk.
A 2-bit quantization technique has been released that compresses KV cache memory usage by about 7x while minimizing performance degradation.
CODA, a GPU kernel abstraction that solves the memory bottleneck in Transformer training, has been released
DeepSeek V4's FP16 KV cache usage was recalculated based on a 1M context.
Compressed the KV cache on Qwen 3.6's 1M context, reducing it from 10.7GB to 6.9GB.
It argues that storing only the residual stream, without a KV cache, can produce identical outputs.
LazyMoE aims for ultra-low-memory execution of a 120B LLM using lazy expert loading and TurboQuant.
Liger Kernel has been integrated into TRL, reducing memory usage by 40% during GRPO training.
This presents a method for efficiently reducing the memory usage of Diffusion Transformer models by leveraging the Quanto quantization tool.
KV Cache quantization reduces LLM memory usage and enables longer context generation.