Multi-Stream LLMs: Resolving Language Model Bottlenecks with Parallel Streams of Thoughts, Inputs, and Outputs

Multi-Stream LLMs identifies the single message stream structure of existing chat-type LLMs as a problem.

Current agents process users, systems, their own thoughts (chain-of-thought), and tool outputs all as a single sequential conversation flow. As a result,

• they cannot act while reading,
• it's hard to incorporate new information while writing, and
• even while thinking, reading or acting is limited at the same time.

To solve this, the paper proposes turning instruction-tuning itself into multiple parallel streams. Each role is separated into its own stream, and at every forward pass, multiple input streams are read simultaneously while tokens are generated into multiple output streams. Each stream causally depends on the previous timestep.

The authors argue this design can improve the following:

• Usability: mitigating conflicts between reading, thinking, and output
• Efficiency: improved processing through parallelization
• Security: strengthened boundaries of responsibility through role separation
• Monitorability: improved observability of agent behavior