LLM Turns into Opcodes

The text generation head was removed from a frozen Qwen 1.5B, and instead a 38M parameter cross-attention head was made to emit CHIP-8 opcodes directly.

Instruction tokens from the input are used as the query, and machine state (display, registers, previous opcode) is fed in as keys/values, designed so the model outputs commands directly without generating text.

In the demo, the following was possible.

• "add 7 and 8": performed BCD extraction and digit drawing in 17ms with 16 opcodes
• "draw a star using a subroutine called twice": drew two stars in different positions with 18 opcodes including CALL/RET
• "wait for 10 ticks": created a busy-wait loop with GETDT, SKE, JUMP and executed after the timer
• "count from 1 to 5": built a loop with increment, conditional branch, and backward jump

All opcodes ran on an actual CHIP-8 emulator, with execution speed at around 1~3ms per opcode.

However, "two plus three" failed. The arithmetic program itself was generated, but the operands were wrong, and the author explains the cause through hidden state interpretation. The representations of number words and numeral tokens fell nearly orthogonal to each other in the arithmetic context, and that gap seemed to be bridged only during the text generation process.

The core conclusion is that understanding lives in hidden states, and computation lives in decoding. Remove the decoder, and computational ability disappears along with it. Though this is a PoC for CHIP-8, the claim is that the instruction query + machine state cross-attention pattern itself is extensible to other VMs or software control.