Microsoft Details Azure Maia 300 Architecture for Cloud-Scale Agentic Inference

Microsoft has detailed the architecture of Azure Maia 300, its third-generation AI accelerator designed specifically for cloud-scale agentic inference. The system is co-designed across software, silicon, rack infrastructure, networking, and datacenters to optimize useful tokens per watt and useful tokens per dollar. This approach addresses the computational demands of agentic workloads, which require sustaining locality, minimizing data movement, and managing fixed power envelopes.

Software-Defined Dataflow and Memory Efficiency

Maia 300 evolves the Software Defined Locally Accessed Dataflow Architecture (SDLA) introduced in Maia 200. Key architectural changes include:

• NUMA Organization: Memory shifts from uniform access (UMA) to a locality-aware non-uniform (NUMA) organization across chiplets, allowing software to place data near the compute that uses it.
• Narrow-Precision Formats: Introduction of enhanced MXFP4 and sub-3-bit storage formats with hardware-accelerated conversion to reduce bytes moved per token.
• Memory Hierarchy: Combines software-managed on-die SRAM with eight HBM4 stacks, providing 288 GB of capacity and 18 TB/s of bandwidth per accelerator. This delivers 2.5 times the memory bandwidth, 33% more capacity, and approximately 2 times the HBM bandwidth per watt compared to Maia 200.
• Multi-Chiplet Design: Compute is placed on TSMC N2P process nodes for performance-per-watt benefits, while memory-controller and I/O functions are on separate dies. This increases compute FLOPs by 36% and scale-up bandwidth by 14%.

Network Scale-Up and Superpod Architecture

The networking layer treats the network as a native extension of the dataflow architecture, using custom AI transport based on Ethernet. A two-tier network builds a superpod of up to 12,288 accelerators.

• In-Rack Topology: Each rack contains 96 accelerators across 24 trays. Direct links connect pairs within trays, while a switched fabric connects all 96 accelerators in the rack (the pod).
• Optical Interconnect: Second-tier switches connect up to 128 racks to form a superpod, using optics for reach.
• Unified Communication: Accelerators can directly initiate communication to any peer in the superpod without host involvement. Bandwidth allocation is dynamic; an accelerator can drive up to 750 GB/s to any peer within the superpod when cross-rack traffic is limited.
• Disaggregated Inference: Prefill and decode workers can be sized and placed independently, with KV state moving directly over the scale-up fabric.

Rack-Scale Deployment and Efficiency

The physical rack design focuses on density and thermal efficiency within a fixed power envelope.

• Power and Density: Integrates 96 accelerators within a 167-kW rack, doubling the accelerator density of Maia 200. Each tray includes four accelerators with host processors, memory, and storage.
• Cooling: Liquid cooling removes more than 80% of the thermal load from accelerators, voltage regulators, and switch ASICs. The system supports direct facility water connection or heat-exchanger units for air-cooled environments.
• Switching: Six high-radix switch ASICs are integrated into three liquid-cooled switch trays to provide rack-local connectivity.

Performance Outcomes

Relative to Maia 200, Maia 300 delivers:

• Up to 7 times greater rack-scale token throughput in a full 96-accelerator configuration.
• More than 2.5 times the performance per watt.
• 2.5 times the memory bandwidth and 33% more capacity per accelerator.

Microsoft states that Maia 300 is designed to support products like Microsoft Copilot, GitHub Copilot, and Microsoft Foundry by maximizing capacity and capability within datacenter constraints.