Missing Executive Control in Transformer Attention

Research has been published showing that the Attention mechanism, the core of Transformer models, lacks higher-order Executive Control ability in the process of selecting and focusing on information.

According to the research, while Transformers excel at learning statistical patterns in input data, they have limited ability to strategically filter and manage information in tasks requiring complex reasoning or logical steps.

Key findings are as follows:

• The Attention mechanism tends to over-focus on certain tokens or disperse attention toward unnecessary information
• Model performance drops sharply in tasks requiring complex cognitive control
• This is a core challenge that must be solved for the model to function as a logical reasoning engine beyond simple pattern matching