AI Briefing#llm-inference
The latest AI and developer news about #llm-inference, with the original source and a short summary.
The latest AI and developer news about #llm-inference, with the original source and a short summary.
Google Cloud C4 VMs based on Intel Xeon 6 improved TCO by 1.7x for GPT OSS inference.
Two RNGD cards served GPT-OSS 120B at 5.8ms TPOT with low power consumption.
Arm KleidiAI is now enabled by default in ExecuTorch 0.7, greatly improving LLM inference performance on mobile and edge devices.
FuriosaAI has raised a $125 million Series C round to scale AI computing.
This explains the mechanisms and key performance metrics of Prefill and Decode, the core stages of LLM inference.
HuggingFace's LLM serving solution TGI now natively supports Intel Gaudi hardware.
FuriosaAI demonstrated high-speed batch inference performance that efficiently processes large-scale LLM requests through its RNGD chip and SDK.
Hugging Face's TGI is introducing a multi-backend architecture that supports various inference engines such as vLLM and TensorRT-LLM.
Introducing Self-Speculative Decoding, a technique that leverages the early layers of the same model to improve text generation speed and memory efficiency.
A technology has been developed that can speed up LLM inference using any small model, even if the tokenizer is different.
AMD's 5th generation EPYC CPU 'Turin' has been unveiled, improving LLM inference performance by up to 2x compared to the previous generation.
Dynamic speculative decoding, developed by Intel Labs and Hugging Face, has been applied to the Transformers library, boosting generation speed by up to 2.7x.
FuriosaAI globally unveiled RNGD, an AI accelerator for LLM inference, at Hot Chips 2024.
FuriosaAI unveiled RNGD, an AI accelerator for high-performance LLM inference, at Hot Chips 2024.
FuriosaAI has unveiled its new product **RNGD**, built for low-power, high-performance AI inference.
Introduces the features and core concepts of ggml, a low-level C/C++ library optimized for Transformer inference.
It introduces a benchmarking tool that analyzes TGI's performance metrics and how to use it to optimize LLM deployment.
KV Cache quantization reduces LLM memory usage and enables longer context generation.
Hugging Face has launched the Optimum-NVIDIA library, which speeds up LLM inference by up to 28x on NVIDIA platforms.
It provides a method for optimizing and deploying Llama 2 models on AWS Inferentia2 using Hugging Face's optimum-neuron.
Hugging Face has released benchmark results for over 60 configurations to optimally deploy Llama 2 on Amazon SageMaker.
Compares the characteristics of the bitsandbytes and auto-gptq quantization methods natively supported by the 🤗 Transformers library.
Hugging Face has launched a TGI-based LLM inference container dedicated to Amazon SageMaker.
Introduces how to efficiently deploy and run inference on the 176B-scale BLOOMZ model using 🤗 Optimum Habana on Habana Gaudi2 accelerators.