AI Briefing#inference-speed
The latest AI and developer news about #inference-speed, with the original source and a short summary.
The latest AI and developer news about #inference-speed, with the original source and a short summary.
The model runs at 72x real-time on CPU and 1120x on GPU, though it may slur difficult words or long numbers.
A drop-in component for transformers that reduces peak memory by 1.5-3.4x while maintaining accuracy.
The new Jeff-Code agent uses a small decision model to pre-fetch information and manage thinking levels, reducing average task time to 0.68x of the baseline with no significant drop in accuracy.
Although its intelligence index is below average, it offers approximately 7x faster output speed and lower cost compared to models in the same price range.
Applying the model grafting technique, which converts existing models into a causal encoder-decoder architecture, increased Qwen3.5-4B's prompt processing speed by up to 3.7x.
Kuleshov's research team summarized diffusion language model (dLLM) technology that overcomes the speed limits of AR models through parallel generation, along with the current status of commercialization efforts such as Mercury 2.
A researcher released 'Laya', a fully open-source decision model responding to the closed model 'Jev', achieving 6-8x faster speed and hallucination blocking compared to Jev.
IFM has released K2-Horizon-7B, which combines a diffusion adapter with a causal LLM, claiming inference speeds of up to 5200 tps and lossless performance.
Cerebras has started serving the Qwen 3.8 27B model on its public endpoint at a speed of approximately 1,500 tokens per second.
Multiverse Computing has released Quasar 438B, an inference model that achieved the highest score among European models with 43 points on the Artificial Analysis Index.
Discrete diffusion-based open-weight LLM 'DiffusionGemma' demonstrates ultra-fast inference performance.
llama.cpp has improved prompt processing speed by up to 31% through vectorization using F16C intrinsics.
Through CUDA kernel optimization in llama.cpp, the inference speed of the Bonsai model was improved by about 5%.
Hugging Face and Cerebras have introduced ultra-low-latency real-time speech-to-speech technology powered by Gemma 4.
Based on HBM bandwidth and pipeline architecture, this forecasts how model parameter sizes will change through 2031.
Xiaomi has unveiled MiMo V2.5Pro, a model that delivers inference speeds of up to 3,000 tokens per second.
Xiaomi collaborated with TileRT to achieve a generation speed of over 1000 tokens per second on a model with 1 trillion parameters.
I-DLM simultaneously boosts DLM quality and speed through introspective consistency.
It introduces how to apply fp16 and use memory-efficient attention to speed up inference and reduce memory usage for the SDXL model.