Extropic Releases Z1T Research Based on Z1+FPGA… Presents Estimated Energy Efficiency for Some Operations

Extropic researchers released the Z1T architecture, which improves inference energy efficiency for Transformer-based models using the Z1 chip. This initial study addresses a decoding configuration that combines Z1's sparse operations with the remaining operations on an FPGA. Using GPUs or other accelerators as auxiliary processors is a subject for future exploration, and the study notes that GPUs are currently more suitable for prefill. Z1T applies Gated Convolutional Attention (GCA) instead of Softmax Attention, DyT instead of RMSNorm, and compiles the MLP's sparse weights into Z1's tanh-linear units. For data processing, it uses the Dy4P representation, which approximates 4-bit precision.

According to the researchers' extrapolation, Z1T requires approximately 10x more FLOPs to reach the same loss as GPT-2, but the low energy cost of Z1 operations may offset this. The presented energy figures are estimates based on Z1's theoretical power consumption and exclude the final dense logit readout and data movement between Z1 and FPGA. Within this scope, the estimate is 294.52 nJ per token, which is approximately 139x better than the H100 assuming 10% MFU (40.9 µJ). Including the cost of performing the final logit readout on the FPGA changes the figure to approximately 136.4 µJ/token. Even in the estimate covering only the included operations, the FPGA accounts for over 95% of the energy.

Throughput is also not a measured result for the entire system. The Z1T estimate assuming a single serial stream is approximately 17,000 tok/s, while the H100's batch-1 torch.compile measurement is approximately 9,764 tok/s. This comparison also excludes the final vocabulary logits operations on both sides. The researchers explain that small models and batch-1 conditions are unfavorable for H100 efficiency, and that efficiency can improve significantly with GPU batch processing.