Academic reviews of the history of the attention mechanism are provided in Niu et al. and Soydaner.
Predecessors
Selective attention in … 展开

Many variants of attention implement soft weights, such as
• fast weight programmers, or fast weight controllers (1992). A "slow" neural network outputs the "fast" weights of another neural network through outer products. The slow network learns … 展开

• Dan Jurafsky and James H. Martin (2022) Speech and Language Processing (3rd ed. draft, January 2022), ch. 10.4 Attention and ch. 9.7 Self-Attention Networks: Transformers
• Alex Graves (4 May 2020), Attention and Memory in Deep Learning (video … 展开

The attention network was designed to identify high correlations patterns amongst words in a given sentence, assuming that it has learned word correlation patterns from the training data. This correlation is captured as neuronal weights learned during training with 展开

Tasks dealing with language can be cast as a problem of translating general sequences, called seq2seq. One way to build such a machine in 2014 is to graft an attention unit to the recurrent Encoder-Decoder (diagram below). With the advent of Transformers in 2017, … 展开

• Transformer (deep learning architecture) § Efficient implementation
• Dynamic neural network 展开