Transformer Networks: The Transformer Machine Learning Model

Lecture 28 min.



Transformer is a deep neural network architecture introduced in 2017 by researchers at Google Brain.

Like recurrent neural networks (RNNs), transformers are designed to process sequences, such as natural-language text, and to solve tasks such as machine translation and automatic summarization. Unlike RNNs, transformers do not require sequences to be processed in order. For example, if the input is text, the transformer does not need to process the end of the text after processing its beginning. Because of this, transformers are easier to parallelize than RNNs and can be trained faster.

Network architecture

The transformer architecture consists of an encoder and a decoder. The encoder receives as input a vectorized sequence with positional information. The decoder receives as input part of this sequence and the output of the encoder. The encoder and decoder consist of layers. The encoder layers pass their result sequentially to the next layer as its input. The decoder layers pass their result sequentially to the next layer, together with the encoder's output, as its input.

Each encoder consists of a self-attention mechanism (input from the previous layer) and a feed-forward neural network (input from the self-attention mechanism). Each decoder consists of a self-attention mechanism (input from the previous layer), an attention mechanism over the encoding results (input from the self-attention mechanism and the encoder) and a feed-forward neural network (input from the attention mechanism).

Transformer Networks: The Transformer Machine Learning Model
Transformer

Transformer Networks: The Transformer Machine Learning Model
Encoder layer

Transformer Networks: The Transformer Machine Learning Model
Decoder layer

Scaled dot-product attention

Each attention mechanism is parameterized by a query weight matrix Transformer Networks: The Transformer Machine Learning Model, a key weight matrix Transformer Networks: The Transformer Machine Learning Model, and a value weight matrix Transformer Networks: The Transformer Machine Learning Model. To compute the attention of an input vector 𝑋 to a vector 𝑌 , the vectors Transformer Networks: The Transformer Machine Learning Model, Transformer Networks: The Transformer Machine Learning Model, Transformer Networks: The Transformer Machine Learning Model are computed. These vectors are used to compute the attention result by the formula:

Transformer Networks: The Transformer Machine Learning Model

Timeline

  • In 1990, the Elman network, using a recurrent neural network, encoded each word in the training set as a vector called a word embedding, and the whole vocabulary as a vector database, which allowed it to perform tasks such as sequence prediction that are beyond a simple multilayer perceptron. A drawback of static embeddings was that they did not distinguish between multiple meanings of words with the same spelling.
  • In 1992, Jürgen Schmidhuber published the "fast weight controller". It learns to answer queries by programming the attention weights of another neural network through outer products of key vectors and value vectors, called FROM and TO. The fast weight controller was later shown to be equivalent to an unnormalized linear transformer. The term "learning internal spotlights of attention" was introduced in 1993.
  • In 1993, IBM alignment models were used for statistical machine translation.
  • In 1997, a precursor of the large language model was proposed, using recurrent neural networks such as long short-term memory.
  • In 2001, a large text corpus of one billion words taken from the Internet, which at the time was called "very very large", was used for word sense disambiguation.
  • In 2012, AlexNet demonstrated the effectiveness of large neural networks for image recognition, encouraging the use of large artificial neural networks in place of older statistical approaches.
  • In 2014, Sutskever et al. proposed a seq2seq model with 380 million parameters for machine translation using two long short-term memory (LSTM) networks. [19] The architecture consists of two parts. The encoder is an LSTM , that takes a sequence of tokens and turns it into a vector. The decoder is another LSTM , that converts the vector into a sequence of tokens.
  • In 2014, gating proved useful in a seq2seq model with 130 million parameters, which used simplified gated recurrent units (GRU). Bahdanau et al. showed that GRUs are neither better nor worse than gated LSTMs.
  • In 2014, Bahdanau et al. improved the previous seq2seq model by using an "additive" attention mechanism between two LSTM networks. However, this was not yet the kind of attention that can be parallelized (the scaled "dot-product"), which was later proposed in the 2017 transformer paper.
  • In 2015, the relative performance of global and local (windowed) attention model architectures was evaluated by Luong et al.: a mixed attention architecture was found to improve on the translations offered by the Bahdanau architecture, while using a local attention architecture reduced translation time.
  • In 2016, Google Translate gradually replaced the old statistical machine translation approach with a new approach based on neural networks, which included a seq2seq model combined with LSTM and an "additive" attention mechanism. They reached a higher level of performance than the statistical approach, which had taken ten years to develop, in only nine months.
  • In 2017, the original (100M-sized) encoder-decoder transformer model with a faster (parallelizable or decomposable) attention mechanism was proposed in the paper "Attention Is All You Need". Because the model had difficulty converging, it was proposed to scale the learning rate linearly from 0 to a maximum value for the first part of training (i.e. 2% of the total number of training steps). The goal of the transformer model is to take the seq2seq model and remove its recurrent neural networks, but keep its additive attention mechanism.
  • In 2018, in the ELMo paper, the entire sentence was processed before an embedding vector was assigned to each word in the sentence. A bidirectional LSTM was used to compute such deeply contextualized embeddings for each word, improving on the line of research built on Bag of Words and word2vec .
  • In 2018, the BERT model (over 1B in size) used an encoder-only transformer, which improved on ELMo.
  • In 2020, the vision transformer and the convolution-augmented transformer for speech processing , outperformed the recurrent neural networks previously used for vision and speech.
  • In 2020, the convergence difficulties of the original transformer were resolved by Xiong et al. by normalizing the layers before (rather than after) multi-head attention. This is called the pre-LN transformer.
  • In 2023, unidirectional ("autoregressive") transformers were used in GPT-3 (over 100B in size) and other OpenAI GPT models.

Predecessors

Before transformers, predecessors of the attention mechanism, such as LSTM and gated recurrent units (GRU), were added to gated recurrent neural networks, which processed datasets sequentially. The dependence on previous token computations prevented them from parallelizing the attention mechanism. In 1992, the fast weight controller was proposed as an alternative to recurrent neural networks that can learn "internal spotlights of attention". In theory, information from one token can propagate arbitrarily far down the sequence, but in practice the vanishing gradient problem leaves the model's state at the end of a long sentence without precise, retrievable information about preceding tokens.

The performance of earlier models was improved by adding an attention mechanism, which allowed the model to access any previous point in the sequence. The attention layer weights all previous states according to a learned measure of relevance, providing relevant information about distant tokens. This proved especially useful in language translation, where distant context can be critical to the meaning of a word in a sentence. The state vector was available only after the last English word had been processed while, for example, translating it from French with an LSTM model. Although in theory such a vector retains information about the whole source sentence, in practice that information is poorly preserved. If an attention mechanism is added, the decoder is given access to the state vectors of every input word, not just the last one, and it can learn attention weights that determine how much attention to pay to each input state vector. Augmenting seq2seq models with an attention mechanism was first implemented in the context of machine translation by Bahdanau, Cho, and Bengio in 2014.

Decomposable attention

In 2016, highly parallelizable decomposable attention was successfully combined with a feedforward network. This indicated that attention mechanisms are powerful in themselves and that sequential recurrent processing of data is not necessary to achieve the quality gains of recurrent neural networks with attention. In 2017, Vaswani et al. also proposed replacing recurrent neural networks with self-attention and began efforts to evaluate this idea. Transformers, using an attention mechanism and processing all tokens simultaneously, computed "soft" weights between them in successive layers. Since the attention mechanism uses information only about other tokens from lower layers, it can be computed for all tokens in parallel, which leads to faster training.

Training

Methods for stabilizing training

The plain transformer architecture had difficulty converging. In the original paper the authors recommended using learning rate warmup. That is, the learning rate should increase linearly from 0 to its maximum value over the first part of training (usually recommended to be 2% of the total number of training steps), and then decrease again.

A 2020 paper found that applying layer normalization before (rather than after) the multi-head attention and feedforward layers stabilizes training, without requiring learning rate warmup.

The GT 3 model combines CWTE, SWTE, and TTE using a self-adaptive gate layer, which enables an efficient and effective fusion of the three types of features for end-to-end text-based stock market prediction.

Pretraining and fine-tuning

Transformers typically undergo self-supervised learning, involving unsupervised pretraining followed by supervised fine-tuning. Pretraining is usually done on a larger dataset than fine-tuning, due to the limited availability of labeled training data. Pretraining and fine-tuning tasks commonly include:

  • language modeling
  • next sentence prediction
  • question answering
  • reading comprehension
  • sentiment analysis
  • paraphrasing

The paper on the T5 transformer documents a large number of pretraining tasks. Some examples are:

  • restoring corrupted text: Thank you <X> me to your party <Y> week.-> <X> for inviting <Y> last <Z>where <Z>means "end of output".
  • translation: translate English to German: That is good.-> Das ist gut..
  • judging the grammatical acceptability of a sentence (CoLA sentenceThe course is jumping well. ): -> not acceptable.

Implementations

The transformer model is implemented in standard deep learning frameworks such as TensorFlow and PyTorch .

Transformers is a library created by Hugging Face , which provides transformer-based architectures and pretrained models.

Architecture

Transformer Networks: The Transformer Machine Learning Model

An illustration of the main components of the transformer model from the original paper, where layer normalization was applied after multi-head attention. A 2020 paper found that placing layer normalization before multi-head attention (rather than after it) improves training stability .

All transformers have the same primary components:

  • Tokenizers, which convert text into tokens.
  • A single embedding layer, which converts tokens and token positions into vector representations.
  • Transformer layers, which carry out repeated transformations on the vector representations, extracting more and more linguistic information. They consist of alternating attention and feedforward layers.
  • (optional) An un-embedding layer, which converts the final vector representations back into a probability distribution over tokens.

Transformer layers can be one of two types: encoder and decoder . The original paper used both, whereas later models included only one of them. BERT is an example of an encoder-only model; GPT models are decoder-only.

Input

The input text is parsed into tokens by a tokenizer, most often a byte-pair encoding tokenizer , and each token is converted into a vector by looking it up in a word embedding table . Then the positional information of the token is added to the word embedding.

Encoder-decoder architecture

Like earlier seq2seq models , the original transformer model used an encoder-decoder architecture . The encoder consists of encoding layers that iteratively process the input tokens one layer after another, while the decoder consists of decoding layers that iteratively process the encoder's output as well as the decoder's output tokens.

The function of each encoder layer is to generate contextualized token representations, where each representation corresponds to a token that "mixes" information from the other input tokens via the self-attention mechanism. Each decoder layer contains two attention sublayers: (1) cross-attention for incorporating the encoder output (contextualized input token representations) and (2) self-attention for "mixing" information among the input tokens to the decoder (i.e. the tokens generated so far during inference)

Both the encoder and decoder layers have a feedforward neural network for additional processing of the outputs, and contain residual connections and layer normalization steps.

Scaled dot-product attention

The building blocks of the transformer are scaled dot-product attention units . For each attention unit, the transformer model learns three weight matrices: the query weights Transformer Networks: The Transformer Machine Learning Model, the key weights Transformer Networks: The Transformer Machine Learning Model, and the value weights Transformer Networks: The Transformer Machine Learning Model. For each token Transformer Networks: The Transformer Machine Learning Model, the input token representationxiTransformer Networks: The Transformer Machine Learning Modelis multiplied by each of the three weight matrices to produce a query vector qTransformer Networks: The Transformer Machine Learning Model, a key vectorki=xiWKTransformer Networks: The Transformer Machine Learning Model, and a value vector vTransformer Networks: The Transformer Machine Learning Model. The attention weights are calculated using the query and key vectors: the attention weight Transformer Networks: The Transformer Machine Learning Modelfrom token Transformer Networks: The Transformer Machine Learning Modelto token Transformer Networks: The Transformer Machine Learning Modelis the dot product betweenqiTransformer Networks: The Transformer Machine Learning ModelandkjTransformer Networks: The Transformer Machine Learning Model. The attention weights are divided by the square root of the dimension of the key vectors,dkTransformer Networks: The Transformer Machine Learning Model, which stabilizes gradients during training, and passed through a softmax , which normalizes the weights. The fact that Transformer Networks: The Transformer Machine Learning Modeland Transformer Networks: The Transformer Machine Learning Modelare different matrices allows attention to be non-symmetric: if tokeniTransformer Networks: The Transformer Machine Learning Modelattends to tokenjTransformer Networks: The Transformer Machine Learning Model(i.e. Transformer Networks: The Transformer Machine Learning Modelis large), this does not necessarily mean that token Transformer Networks: The Transformer Machine Learning Modelwill attend to tokeniTransformer Networks: The Transformer Machine Learning Model(i.e. Transformer Networks: The Transformer Machine Learning Modelmay be small). The output of the attention unit for tokeniTransformer Networks: The Transformer Machine Learning Modelis the weighted sum of the value vectors of all tokens, weighted by Transformer Networks: The Transformer Machine Learning Model, the attention from token Transformer Networks: The Transformer Machine Learning Modelto each token.

The attention calculation for all tokens can be expressed as one large matrix calculation using the softmax function , which is useful for training because of computational matrix operation optimizations that allow matrix operations to be computed quickly. The matrices Transformer Networks: The Transformer Machine Learning Model, Transformer Networks: The Transformer Machine Learning Modeland Transformer Networks: The Transformer Machine Learning Modelare defined as the matrices whoseiTransformer Networks: The Transformer Machine Learning Modelth rows are the vectors Transformer Networks: The Transformer Machine Learning Model, Transformer Networks: The Transformer Machine Learning Model, and Transformer Networks: The Transformer Machine Learning Modelrespectively. Then we can represent the attention as

Attention(Q,K,V)=softmax(QKTdk)V
Transformer Networks: The Transformer Machine Learning Model

where the softmax is taken over the horizontal axis.

Multi-head attention

One set of Transformer Networks: The Transformer Machine Learning Modelmatrices is called an attention head , and each layer in a transformer model has multiple attention heads. While each attention head attends to the tokens that are relevant to each token, multiple attention heads allow the model to do this for different definitions of "relevance". In addition, the field of influence representing relevance can become progressively dilated in successive layers. Many transformer attention heads encode relevance relations that are meaningful to humans. For example, some attention heads may attend mostly to the next word, while others mainly attend from verbs to their direct objects. The computations for each attention head can be performed in parallel , which allows for fast processing. The outputs of the attention layer are concatenated to be passed into the feedforward neural network layers .

Concretely, let the multiple attention heads be indexed byiTransformer Networks: The Transformer Machine Learning Model, then we have

MultiHeadedAttention(Q,K,V)=Concati∈[#heads](Attention(XWiQ,XWiK,XWiV))WO
Transformer Networks: The Transformer Machine Learning Model

where the matrix Transformer Networks: The Transformer Machine Learning Model is the concatenation of word embeddings, and the matrices Transformer Networks: The Transformer Machine Learning Modelare the "projection matrices" belonging to an individual attention head Transformer Networks: The Transformer Machine Learning Model, and Transformer Networks: The Transformer Machine Learning Modelis a final projection matrix belonging to the whole multi-head attention head.

Masked attention

It may be necessary to cut out attention links between some pairs of words. For example, the decoder at token positiontTransformer Networks: The Transformer Machine Learning Modelshould not have access to token position Transformer Networks: The Transformer Machine Learning Model. This can be done before the softmax stage by adding a mask matrixM that is Transformer Networks: The Transformer Machine Learning Modelat entries where the attention link must be disabled, and0 elsewhere:

MaskedAttention(Q,K,V)=softmax(M+QKTdk)V
Transformer Networks: The Transformer Machine Learning Model

For example, autoregressive modeling uses the following mask matrix:

M=[0−∞−∞…−∞00−∞…−∞000…−∞⋮⋮⋮⋱⋮000…0]
Transformer Networks: The Transformer Machine Learning Model

In words, this means that each token can attend to itself and every token before it, but not to any token after it.

Encoder

Each encoder consists of two major components: a self-attention mechanism and a feedforward neural network. The self-attention mechanism accepts input encodings from the previous encoder and weighs their relevance to each other to generate output encodings. The feedforward neural network further processes each output encoding individually. These output encodings are then passed to the next encoder as its input, as well as to the decoders.

The first encoder takes positional information and embeddings of the input sequence as its input, rather than encodings. The positional information is necessary for the transformer to make use of the order of the sequence, because no other part of the transformer makes use of it.

The encoder is bidirectional. Attention can be paid to tokens both before and after the current token. Tokens are used instead of words to account for polysemy .

Transformer Networks: The Transformer Machine Learning Model

Diagram of sinusoidal positional encoding with parameters Transformer Networks: The Transformer Machine Learning Model

Positional encoding

Positional encoding is a fixed-size vector representation that encapsulates the relative positions of tokens within a target sequence: it provides the transformer model with information about where the words are in the input sequence.

Positional encoding is defined as a function of type Transformer Networks: The Transformer Machine Learning Model, where Transformer Networks: The Transformer Machine Learning Modelis a positive even integer . The full positional encoding defined in the original paper is given by the equation:

(f(t)2k,f(t)2k+1)=(sin⁡(θ),cos⁡(θ))∀k∈{0,1,…,d/2−1}
Transformer Networks: The Transformer Machine Learning Model

where Transformer Networks: The Transformer Machine Learning Model.

Here, Transformer Networks: The Transformer Machine Learning Modelis a free parameter that should be significantly larger than the biggest Transformer Networks: The Transformer Machine Learning Modelthat would be input into the positional encoding function. In the original paper the authors chose Transformer Networks: The Transformer Machine Learning Model.

The function has a simpler form when written as a complex function of typef: Transformer Networks: The Transformer Machine Learning Model

f(t)=(eit/rk)k=0,1,…,d2−1
Transformer Networks: The Transformer Machine Learning Model

where Transformer Networks: The Transformer Machine Learning Model.

The main reason the authors chose this as the positional encoding function is that it allows shifts to be performed as linear transformations:

f(t+Δt)=diag(f(Δt))f(t)
Transformer Networks: The Transformer Machine Learning Model

where Transformer Networks: The Transformer Machine Learning Modelis the distance one wishes to shift. This allows the transformer to take any encoded position and find the encoding of the position n steps ahead or n steps behind, by a matrix multiplication.

By taking a linear sum, any convolution can also be implemented as linear transformations:

∑jcjf(t+Δtj)=(∑jcjdiag(f(Δtj)))f(t)
Transformer Networks: The Transformer Machine Learning Model

for any constantscjTransformer Networks: The Transformer Machine Learning Model. This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors. This sum of encoded positions, when fed into the attention mechanism, would create attention weights on its neighbors, much like what happens in a convolutional neural network language model . In the authors' words, "we hypothesized it would allow the model to easily learn to attend by relative position."

In typical implementations, all operations are done over the real numbers, not the complex numbers, but since complex multiplication can be implemented as real 2-by-2 matrix multiplication , this is just a difference in notation.

Decoder

Each decoder consists of three major components: a self-attention mechanism, an attention mechanism over the encodings, and a feedforward neural network. The decoder functions in a similar way to the encoder, but an additional attention mechanism is inserted which instead draws relevant information from the encodings generated by the encoders. This mechanism can also be called encoder-decoder attention .

Like the first encoder, the first decoder takes positional information and embeddings of the output sequence as its input, rather than encodings. The transformer must not use the current or future output to predict an output, so the output sequence must be partially masked to prevent this reverse information flow. This allows for autoregressive text generation. For all attention heads, attention cannot be paid to following tokens. The last decoder is followed by a final linear transformation and softmax layer, to produce the output probabilities over the vocabulary.

All members of the OpenAI GPT series have a decoder-only architecture.

Terminology

In large language models, the terminology is somewhat different from that used in the original Transformer paper:

  • "encoder-only": full encoder, full decoder.
  • "encoder-decoder": full encoder, autoregressive decoder.
  • "decoder-only": autoregressive encoder, autoregressive decoder.

Here "autoregressive" means that a mask is inserted in the attention head to zero out all attention from one token to all tokens following it, as described in the "Masked attention" section.

Generally, transformer-based language models are of two types: causal (or "autoregressive") and masked. The GPT series is causal and decoder-only. BERT is masked and encoder-only. The T5 series is encoder-decoder, with a full encoder and an autoregressive decoder.

Subsequent work

Alternative activation functions

The original transformer uses the ReLU activation function. Other activation functions have been developed, such as SwiGLU

Alternative positional encodings

Transformers may use positional encoding methods other than sinusoidal.

RoPE

RoPE (rotary positional embedding) is best explained by considering a list of 2-dimensional vectors Transformer Networks: The Transformer Machine Learning Model. Now pick an angleθTransformer Networks: The Transformer Machine Learning Model. Then the RoPE encoding is

RoPE(xm(1),xm(2),m)=(cos⁡mθ−sin⁡mθsin⁡mθcos⁡mθ)(xm(1)xm(2))=(xm(1)cos⁡mθ−xm(2)sin⁡mθxm(2)cos⁡mθ+xm(1)sin⁡mθ)
Transformer Networks: The Transformer Machine Learning Model

Equivalently, if we write the 2-dimensional vectors as complex numberszm=xm(1)+ixm(2)Transformer Networks: The Transformer Machine Learning Model, then the RoPE encoding is just multiplication by an angle:

RoPE(zm,m)=eimθzm
Transformer Networks: The Transformer Machine Learning Model

For a list of Transformer Networks: The Transformer Machine Learning Model-dimensional vectors, a RoPE encoder is defined by a sequence of angles Transformer Networks: The Transformer Machine Learning Model. Then the RoPE encoding is applied to each pair of coordinates.

The advantage of RoPE is that the dot product between two vectors depends only on their relative location:

RoPE(x,m)TRoPE(y,n)=RoPE(x,m+k)TRoPE(y,n+k)
Transformer Networks: The Transformer Machine Learning Model

for any integer Transformer Networks: The Transformer Machine Learning Model.

ALiBi

ALiBi (Attention with Linear Biases) is not a replacement for the positional encoder of the original transformer. Instead, it is an additional positional encoder that is directly plugged into the attention mechanism. Specifically, the ALiBi attention mechanism is

Attention(Q,K,V)=softmax(QKTdk+sB)V
Transformer Networks: The Transformer Machine Learning Model

Here,sTransformer Networks: The Transformer Machine Learning Modelis a real number (a "scalar"), and Transformer Networks: The Transformer Machine Learning Modelis the linear bias matrix, defined by

B=(0123⋯−1012⋯−2−101⋯−3−2−10⋯⋮⋮⋮⋮⋱)
Transformer Networks: The Transformer Machine Learning Model

in other words, Transformer Networks: The Transformer Machine Learning Model.

ALiBi allows pretraining on short context windows, then fine-tuning on longer context windows. Since it is directly plugged into the attention mechanism, it can be combined with any positional encoder that is plugged into the "bottom" of the entire network (which is where the sinusoidal encoder on the original transformer, as well as RoPE and many others, are located).

Relative positional encodings

Relative positional encoding is similar to ALiBi, but more generic:

Attention(Q,K,V)=softmax(QKTdk+B)V
Transformer Networks: The Transformer Machine Learning Model

where Transformer Networks: The Transformer Machine Learning Modelis a Toeplitz matrix , that is, Transformer Networks: The Transformer Machine Learning Modelwheneveri−Transformer Networks: The Transformer Machine Learning Model.

Efficient implementation

FlashAttention

FlashAttention is an algorithm that implements the transformer attention mechanism efficiently on a GPU. It performs matrix multiplications in blocks , such that each block fits within the GPU cache, and by careful management of the blocks it minimizes data copying between GPU caches (as data movement is slow).

An improved version, FlashAttention-2, was developed to meet the growing demand for language models capable of processing longer contexts. It offers improvements in work partitioning and parallelism, enabling it to achieve up to 230 TFLOPs/s on A100 GPUs ( FP16 / BF16 ), a 2x speedup over the original FlashAttention.

Key advances in FlashAttention-2 include reducing the number of non-matmul FLOPs, improved parallelism over the sequence length dimension, better work partitioning between GPU warps, and added support for head sizes up to 256 as well as multi-query attention (MQA) and grouped-query attention (GQA).

Benchmarks revealed that FlashAttention-2 is up to 2x faster than FlashAttention and up to 9x faster than a standard attention implementation in PyTorch. Future developments include optimization for new hardware such as H100 GPUs and new data types such as FP8.

Multi-query attention

Multi-query attention changes the multi-head attention mechanism. Whereas normally,

MultiHeadedAttention(Q,K,V)=Concati∈[#heads](Attention(XWiQ,XWiK,XWiV))WO
Transformer Networks: The Transformer Machine Learning Model

with multi-query attention there is just oneWK,WVTransformer Networks: The Transformer Machine Learning Model, thus:

MultiQueryAttention(Q,K,V)=Concati∈[#heads](Attention(XWiQ,XWK,XWV))WO
Transformer Networks: The Transformer Machine Learning Model

This has a neutral effect on model quality and training speed, but increases inference speed.

Speculative decoding

Transformers are used in large language models for autoregressive sequence generation: generating a stream of text, one token at a time. However, in most settings, decoding from language models is memory-bound, meaning that we have spare compute power available. Speculative decoding [ uses this spare compute power by computing several tokens in parallel. Similarly to speculative execution in CPUs, future tokens are computed concurrently, by speculating on the value of previous tokens, and are then discarded if it turns out the speculation was incorrect.

In particular, consider a transformer model like GPT-3, which has a context window size of 512. To generate an entire context window autoregressively with greedy decoding, it would have to be run 512 times, generating one token each time. Transformer Networks: The Transformer Machine Learning Model. However, if we had some educated guess for the values of these tokens, we could verify all of them in parallel, in one run of the model, by checking that each Transformer Networks: The Transformer Machine Learning Modelis indeed the token with the largest log-likelihood in the Transformer Networks: The Transformer Machine Learning Model-th output.

In speculative decoding, a smaller model or some other simple heuristic is used to generate a few speculative tokens that are subsequently verified by the larger model. For example, suppose a small model generated four speculative tokens: Transformer Networks: The Transformer Machine Learning Model. These tokens are run through the larger model, and only Transformer Networks: The Transformer Machine Learning Modeland Transformer Networks: The Transformer Machine Learning Modelare accepted. The same run of the large model has already generated a new token Transformer Networks: The Transformer Machine Learning Modelto replace Transformer Networks: The Transformer Machine Learning Model, andx4~Transformer Networks: The Transformer Machine Learning Modelis completely discarded. The process then repeats (starting from the 4th token) until all tokens are generated.

For non-greedy decoding, similar ideas apply, except that the speculative tokens are accepted or rejected stochastically, in such a way as to guarantee that the final output distribution is the same as if speculative decoding were not used.

Subquadratic transformers

Training transformer-based architectures can be expensive, especially for long inputs. Alternative architectures include the Reformer (which reduces the computational load from Transformer Networks: The Transformer Machine Learning Modelto Transformer Networks: The Transformer Machine Learning Model ) or models like ETC/BigBird (which can reduce it to Transformer Networks: The Transformer Machine Learning Model) where Transformer Networks: The Transformer Machine Learning Modelis the length of the sequence. This is done using locality-sensitive hashing and reversible layers.

Ordinary transformers require a memory size that is quadratic in the size of the context window. Attention-free transformers reduce this dependence to linear while still retaining the advantages of the transformer by linking the key to the value.

Long Range Arena (2020) is a standard benchmark for comparing the behavior of transformer architectures over long inputs.

Random Feature Attention (2021) uses random Fourier features:

𝜑(x)"="1D[cos⁡⟨w1,x⟩,sin⁡⟨w1,x⟩,⋯cos⁡⟨wD,x⟩,sin⁡⟨wD,x⟩]T
Transformer Networks: The Transformer Machine Learning Model

where Transformer Networks: The Transformer Machine Learning Modelare independent samples from the normal distribution Transformer Networks: The Transformer Machine Learning Model. This choice of parameters satisfies Transformer Networks: The Transformer Machine Learning Model, or

e⟨x,y⟩/𝜎2"="E[⟨e‖x‖2/2𝜎2𝜑(x),e‖y‖2/2𝜎2𝜑(y)⟩]≈⟨e‖x‖2/2𝜎2𝜑(x),e‖y‖2/2𝜎2𝜑(y)⟩
Transformer Networks: The Transformer Machine Learning Model

Consequently, the unidirectional attention with a single query can be written as

Attention(q,K,V)"="softmax(qKTdk)V≈𝜑(q)T∑ie‖ki‖2/2𝜎2𝜑(ki)viT𝜑(q)T∑ie‖ki‖2/2𝜎2𝜑(ki)
Transformer Networks: The Transformer Machine Learning Model

where Transformer Networks: The Transformer Machine Learning Model. Similarly for multiple queries and multi-headed attention.

This approximation can be computed in linear time, since we can compute the matrix Transformer Networks: The Transformer Machine Learning Modelfirst, then multiply it by the query. Essentially, we have managed to obtain a more precise version of

Attention(q,K,V)"="softmax(qKTdk)V≈q(KTV/dk)
Transformer Networks: The Transformer Machine Learning Model

Performer (2022) uses the same random feature attention, but Transformer Networks: The Transformer Machine Learning Modelare first independently sampled from the normal distribution Transformer Networks: The Transformer Machine Learning Model, and then they are processed with Gram-Schmidt.

Multimodality

Transformers can also be used or adapted for modalities (input or output) beyond text, usually by finding a way to "tokenize" the modality.

Vision transformers adapt the transformer to computer vision by breaking input images into a series of patches, turning them into vectors, and treating them as tokens in a standard transformer.

Conformer and the later Whisper follow the same pattern for speech recognition , first turning the speech signal into a spectrogram , which is then treated like an image, i.e., broken into a series of patches, turned into vectors, and treated as tokens in a standard transformer.

Perceivers , by Andrew Jaegle et al. (2021) can learn from large amounts of heterogeneous data.

As for image output , Peebles et al. introduced the Diffusion Transformer (DiT), which facilitates the use of the transformer architecture for diffusion-based image generation . In addition, Google released a transformer-centric image generator called "Muse", based on parallel decoding and the masked generative transformer technique. (Transformers played a less central, though still significant, role in earlier image generation technologies. )

Applications

The transformer has had great success in natural language processing (NLP), for example in machine translation and time series prediction tasks. Many large language models , such as GPT-2 , GPT-3 , GPT-4 , Claude , BERT , XLNet, RoBERTa and ChatGPT, demonstrate the ability of transformers to perform a wide variety of such NLP-related tasks, and have the potential to find real-world applications. These may include:

  • machine translation
  • document summarization
  • document generation
  • named entity recognition (NER)
  • biological sequence analysis
  • writing computer code based on requirements expressed in natural language
  • video understanding .

Beyond NLP applications, it has also had success in other areas, such as computer vision or protein folding applications (such as AlphaFold ).

As an illustrative example, consider Ithaca, an encoder-only transformer with three output heads. It takes an ancient Greek inscription as input in the form of a sequence of characters, but with illegible characters replaced by "-". Its three output heads respectively output probability distributions over Greek characters, the location of the inscription, and the date of the inscription.

See also

  • Perceiver - a machine learning algorithm for non-text data.
  • BERT (language model) – a language model developed by Google.
  • GPT – a text generation language model
  • ChatGPT — a chatbot developed by OpenAI.
  • Wu Dao - a Chinese multimodal artificial intelligence program
  • Vision transformer — a machine learning algorithm for image processing
  • BLOOM (language model) – a multilingual open-access language model.
created: 2024-04-25
updated: 2026-09-29
93



Was this answer useful?
Choose a quick rating so we can improve the next answer for you.
How satisfied are you?


Comments

To leave a comment

If you have any suggestion, idea, thanks or comment, feel free to write. We really value feedback and are glad to hear your opinion.
To reply

Lectures and tutorial on "Neural networks"

Terms: Neural networks

Sign in and the lectures you have studied will be marked in this list.

Choose a course