AI Briefing#distributed-inference
The latest AI and developer news about #distributed-inference, with the original source and a short summary.
The latest AI and developer news about #distributed-inference, with the original source and a short summary.
A cluster of four M5 Ultra Mac Studios achieves over 36 tokens per second on the 816B-parameter Muse Spark Open model using new RDMA-based distributed inference.
The Petals project uses a BitTorrent-style approach to let consumer GPUs run huge LLMs like Llama 3.1 405B in a distributed manner.
The Shard project dramatically improved distributed inference speed for large-scale models by leveraging speculative decoding and other techniques in a WAN environment.