Apple Researchers Optimize Transformer Attention: Let Tokens Ignore Their Own Info for Consistent Performance Boost

The Transformer attention mechanism is core to many AI models, and optimizing it has long been a research focus. Recently, Shuangfei Zhai, a machine learning research scientist at Apple, published a paper proposing the 'Exclusive Self-Attention' (XSA) mechanism.

Unlike standard Transformers, XSA forces each token to exclude its own position's contribution when calculating attention, extracting information only from the context—intuitively, tokens already know their own info; the value of attention lies in understanding surrounding content.

Experiments show that XSA consistently outperforms standard self-attention in models up to 2.7 billion parameters, with more obvious advantages for longer sequences. Zhai was also the author of Attention Free Transformer (AFT) before, continuing to explore alternative attention mechanisms.