Predecessors
For many years, sequence modelling and generation was done by using plain recurrent neural networks (RNNs). A well-cited early example was the 展开

Methods for stabilizing training
The plain transformer architecture had difficulty converging. In the original paper the authors recommended using learning rate warmup. That is, the learning rate should linearly scale up from 0 to maximal value for the first part of … 展开

Alternative activation functions
The original transformer uses ReLU activation function. Other activation functions were developed. … 展开

The transformer has had great success in natural language processing (NLP). Many large language models such as GPT-2, GPT-3, 展开

All transformers have the same primary components:
• Tokenizers, which convert text into tokens. 展开

Sublayers
Each encoder layer contains 2 sublayers: the self-attention and the feedforward network. Each decoder layer contains 3 sublayers: the causally masked self-attention, the cross-attention, and the feedforward network. 展开

• seq2seq – Family of machine learning approaches
• Perceiver – Variant of Transformer designed for multimodal data
• Vision transformer – Variant of Transformer designed for vision processing 展开