An RPN Calculator Built with a Transformer

By treating the residual stream like a register and using attention as an information router, a deterministic RPN interpreter is embedded inside a standard Transformer.

• The residual of a standard decoder-only Transformer is interpreted as persistent state, used like a modify operation that reads and writes values at each layer.
• W_Q/W_K/W_V/W_O serve to select and copy registers and then relocate them to different positions in the residual space, with W_O in particular moving head outputs to the target register.
• Register, Head, Layer, and @compute abstractions are set up so that a compiler fills in offsets and matrices based on names. It starts with a one-hot and block-identity approach.
• RPN was chosen because it requires a stack but involves almost no grammar parsing, making it well-suited to explaining interpreter structure. For example, 3 4 + 3 3 + * evaluates to 42.
• Since standard Transformers have no loops, state is carried forward either by securing sufficient depth or by leaving a breadcrumb to pass on to the next execution.

The related code is published at radarsat1/rpn_transformer, and the Transformer is not presented as the fixed operating principle of actual LLMs, but rather as one example of a possible state space.