Transformer-based Encoder-Decoder Models

!pip install transformers==4.2.1
!pip install sentencepiece==0.1.95

The transformer-based encoder-decoder model was introduced by Vaswani
et al. within the famous Attention is all you wish
paper and is today the de-facto
standard encoder-decoder architecture in natural language processing
(NLP).

Recently, there was a variety of research on different pre-training
objectives for transformer-based encoder-decoder models, e.g. T5,
Bart, Pegasus, ProphetNet, Marge, etc…, however the model architecture
has stayed largely the identical.

The goal of the blog post is to present an in-detail explanation of
how the transformer-based encoder-decoder architecture models
sequence-to-sequence problems. We’ll give attention to the mathematical model
defined by the architecture and the way the model might be utilized in inference.
Along the way in which, we’ll give some background on sequence-to-sequence
models in NLP and break down the transformer-based encoder-decoder
architecture into its encoder and decoder parts. We offer many
illustrations and establish the link between the idea of
transformer-based encoder-decoder models and their practical usage in
🤗Transformers for inference. Note that this blog post does not explain
how such models might be trained – this can be the subject of a future blog
post.

Transformer-based encoder-decoder models are the results of years of
research on representation learning and model architectures. This
notebook provides a brief summary of the history of neural
encoder-decoder models. For more context, the reader is suggested to read
this awesome blog
post by
Sebastion Ruder. Moreover, a basic understanding of the
self-attention architecture is advisable. The next blog post by
Jay Alammar serves as refresher on the unique Transformer model
here.

On the time of writing this notebook, 🤗Transformers comprises the
encoder-decoder models T5, Bart, MarianMT, and Pegasus, which
are summarized within the docs under model
summaries.

The notebook is split into 4 parts:

• Background – A brief history of neural encoder-decoder models
  is given with a give attention to RNN-based models.
• Encoder-Decoder – The transformer-based encoder-decoder model
  is presented and it’s explained how the model is used for
  inference.
• Encoder – The encoder a part of the model is explained in
  detail.
• Decoder – The decoder a part of the model is explained in
  detail.

Each part builds upon the previous part, but may also be read on its
own.

Background

Tasks in natural language generation (NLG), a subfield of NLP, are best
expressed as sequence-to-sequence problems. Such tasks might be defined as
finding a model that maps a sequence of input words to a sequence of
goal words. Some classic examples are summarization and
translation. In the next, we assume that every word is encoded
right into a vector representation. $nn$ input words can then be represented as
a sequence of $nn$ input vectors:

$${\mathbf{X}}_{1:n}=\{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_n\}\mathrm{.}mathbf\{X\}\_\{1:n\}\; =\; \{mathbf\{x\}\_1,\; ldots,\; mathbf\{x\}\_n\}.$$

Consequently, sequence-to-sequence problems might be solved by finding a
mapping $ff$ from an input sequence of $nn$ vectors ${\mathbf{X}}_{1:n}mathbf\{X\}\_\{1:n\}$

$$f:{\mathbf{X}}_{1:n}\to {\mathbf{Y}}_{1:m}\mathrm{.}f:\; mathbf\{X\}\_\{1:n\}\; to\; mathbf\{Y\}\_\{1:m\}.$$

Sutskever et al. (2014) noted that
deep neural networks (DNN)s, “*despite their flexibility and power can
only define a mapping whose inputs and targets might be sensibly encoded
with vectors of fixed dimensionality.*” ${}^1\{\}^1$

Using a DNN model ${}^2\{\}^2$

In 2014, Cho et al. and
Sutskever et al. proposed to make use of an
encoder-decoder model purely based on recurrent neural networks (RNNs)
for sequence-to-sequence tasks. In contrast to DNNS, RNNs are capable
of modeling a mapping to a variable variety of goal vectors. Let’s
dive a bit deeper into the functioning of RNN-based encoder-decoder
models.

During inference, the encoder RNN encodes an input sequence ${\mathbf{X}}_{1:n}mathbf\{X\}\_\{1:n\}$

$$f_{{\theta }_{enc}}:{\mathbf{X}}_{1:n}\to \mathbf{c}\mathrm{.}f\_\{theta\_\{enc\}\}:\; mathbf\{X\}\_\{1:n\}\; to\; mathbf\{c\}.$$

Then, the decoder’s hidden state is initialized with the input encoding
and through inference, the decoder RNN is used to auto-regressively
generate the goal sequence. Let’s explain.

Mathematically, the decoder defines the probability distribution of a
goal sequence ${\mathbf{Y}}_{1:m}mathbf\{Y\}\_\{1:m\}$

$$p_{{\theta }_{dec}}({\mathbf{Y}}_{1:m}\mathrm{\mid }\mathbf{c})\mathrm{.}p\_\{theta\_\{dec\}\}(mathbf\{Y\}\_\{1:m\}\; |mathbf\{c\}).$$

By Bayes’ rule the distribution might be decomposed into conditional
distributions of single goal vectors as follows:

$$p_{{\theta }_{dec}}({\mathbf{Y}}_{1:m}\mathrm{\mid }\mathbf{c})=\prod _{i=1}^m{p}_{{\theta }_{\text{dec}}}({\mathbf{y}}_i\mathrm{\mid }{\mathbf{Y}}_{0:i-1},\mathbf{c})\mathrm{.}p\_\{theta\_\{dec\}\}(mathbf\{Y\}\_\{1:m\}\; |mathbf\{c\})\; =\; prod\_\{i=1\}^\{m\}\; p\_\{theta\_\{text\{dec\}\}\}(mathbf\{y\}\_i\; |\; mathbf\{Y\}\_\{0:\; i-1\},\; mathbf\{c\}).$$

Thus, if the architecture can model the conditional distribution of the
next goal vector, given all previous goal vectors:

$$p_{{\theta }_{\text{dec}}}({\mathbf{y}}_i\mathrm{\mid }{\mathbf{Y}}_{0:i-1},\mathbf{c}),\mathrm{\forall }i\in \{1,\dots ,m\},p\_\{theta\_\{text\{dec\}\}\}(mathbf\{y\}\_i\; |\; mathbf\{Y\}\_\{0:\; i-1\},\; mathbf\{c\}),\; forall\; i\; in\; \{1,\; ldots,\; m\},$$

then it will probably model the distribution of any goal vector sequence given
the hidden state $\mathbf{c}mathbf\{c\}$ by simply multiplying all conditional
probabilities.

So how does the RNN-based decoder architecture model $p_{{\theta }_{\text{dec}}}({\mathbf{y}}_i\mathrm{\mid }{\mathbf{Y}}_{0:i-1},\mathbf{c})p\_\{theta\_\{text\{dec\}\}\}(mathbf\{y\}\_i\; |\; mathbf\{Y\}\_\{0:\; i-1\},\; mathbf\{c\})$

In computational terms, the model sequentially maps the previous inner
hidden state ${\mathbf{c}}_{i-1}mathbf\{c\}\_\{i-1\}$

$$f_{{\theta }_{\text{dec}}}({\mathbf{y}}_{i-1},{\mathbf{c}}_{i-1})\to {\mathbf{l}}_i,{\mathbf{c}}_i\mathrm{.}f\_\{theta\_\{text\{dec\}\}\}(mathbf\{y\}\_\{i-1\},\; mathbf\{c\}\_\{i-1\})\; to\; mathbf\{l\}\_i,\; mathbf\{c\}\_i.$$

$$p({\mathbf{y}}_i\mathrm{\mid }{\mathbf{l}}_i)=\mathbf{\text{Softmax}}({\mathbf{l}}_i),\text{ with }{\mathbf{l}}_i=f_{{\theta }_{\text{dec}}}({\mathbf{y}}_{i-1},{\mathbf{c}}_{\text{prev}})\mathrm{.}p(mathbf\{y\}\_i\; |\; mathbf\{l\}\_i)\; =\; textbf\{Softmax\}(mathbf\{l\}\_i),\; text\{\; with\; \}\; mathbf\{l\}\_i\; =\; f\_\{theta\_\{text\{dec\}\}\}(mathbf\{y\}\_\{i-1\},\; mathbf\{c\}\_\{text\{prev\}\}).$$

For more detail on the logit vector and the resulting probability
distribution, please see footnote ${}^4\{\}^4$

The space of possible goal vector sequences ${\mathbf{Y}}_{1:m}mathbf\{Y\}\_\{1:m\}$

Given such a decoding method, during inference, the following input vector ${\mathbf{y}}_{i}mathbf\{y\}\_i$

A vital feature of RNN-based encoder-decoder models is the
definition of special vectors, reminiscent of the $\text{EOS}text\{EOS\}$ and $\text{BOS}text\{BOS\}$ vector. The $\text{EOS}text\{EOS\}$ vector often represents the ultimate
input vector ${\mathbf{x}}_{n}mathbf\{x\}\_n$

The unfolded RNN encoder is coloured in green and the unfolded RNN
decoder is coloured in red.

The English sentence “I need to purchase a automobile”, represented by ${\mathbf{x}}_1=\text{I}mathbf\{x\}\_1\; =\; text\{I\}$

To generate the primary goal vector, the decoder is fed the $\text{BOS}text\{BOS\}$
vector, illustrated as ${\mathbf{y}}_{0}mathbf\{y\}\_0$

$$p_{{\theta }_{dec}}(\mathbf{y}\mathrm{\mid }\text{BOS},\mathbf{c})\mathrm{.}p\_\{theta\_\{dec\}\}(mathbf\{y\}\; |\; text\{BOS\},\; mathbf\{c\}).$$

The word $\text{Ich}text\{Ich\}$ is sampled (shown by the grey arrow, connecting ${\mathbf{l}}_{1}mathbf\{l\}\_1$

$$\text{will}\sim p_{{\theta }_{dec}}(\mathbf{y}\mathrm{\mid }\text{BOS},\text{Ich},\mathbf{c})\mathrm{.}text\{will\}\; sim\; p\_\{theta\_\{dec\}\}(mathbf\{y\}\; |\; text\{BOS\},\; text\{Ich\},\; mathbf\{c\}).$$

And so forth until at step $i=6i=6$

To sum it up, an RNN-based encoder-decoder model, represented by $f_{{\theta }_{\text{enc}}}f\_\{theta\_\{text\{enc\}\}\}$

$$p_{{\theta }_{\text{enc}},{\theta }_{\text{dec}}}({\mathbf{Y}}_{1:m}\mathrm{\mid }{\mathbf{X}}_{1:n})=\prod _{i=1}^m{p}_{{\theta }_{\text{enc}},{\theta }_{\text{dec}}}({\mathbf{y}}_i\mathrm{\mid }{\mathbf{Y}}_{0:i-1},{\mathbf{X}}_{1:n})=\prod _{i=1}^m{p}_{{\theta }_{\text{dec}}}({\mathbf{y}}_i\mathrm{\mid }{\mathbf{Y}}_{0:i-1},\mathbf{c}),\text{ with }\mathbf{c}=f_{{\theta }_{enc}}(X)\mathrm{.}p\_\{theta\_\{text\{enc\}\},\; theta\_\{text\{dec\}\}\}(mathbf\{Y\}\_\{1:m\}\; |\; mathbf\{X\}\_\{1:n\})\; =\; prod\_\{i=1\}^\{m\}\; p\_\{theta\_\{text\{enc\}\},\; theta\_\{text\{dec\}\}\}(mathbf\{y\}\_i\; |\; mathbf\{Y\}\_\{0:\; i-1\},\; mathbf\{X\}\_\{1:n\})\; =\; prod\_\{i=1\}^\{m\}\; p\_\{theta\_\{text\{dec\}\}\}(mathbf\{y\}\_i\; |\; mathbf\{Y\}\_\{0:\; i-1\},\; mathbf\{c\}),\; text\{\; with\; \}\; mathbf\{c\}=f\_\{theta\_\{enc\}\}(X).$$

During inference, efficient decoding methods can auto-regressively
generate the goal sequence ${\mathbf{Y}}_{1:m}mathbf\{Y\}\_\{1:m\}$

The RNN-based encoder-decoder model took the NLG community by storm. In
2016, Google announced to completely replace its heavily feature engineered
translation service by a single RNN-based encoder-decoder model (see
here).

Nevertheless, RNN-based encoder-decoder models have two pitfalls. First,
RNNs suffer from the vanishing gradient problem, making it very
difficult to capture long-range dependencies, cf. Hochreiter et al.
(2001). Second,
the inherent recurrent architecture of RNNs prevents efficient
parallelization when encoding, cf. Vaswani et al.
(2017).

---

${}^1\{\}^1$

${}^2\{\}^2$

${}^3\{\}^3$

${}^4\{\}^4$

${}^5\{\}^5$

${}^6\{\}^6$

Encoder-Decoder

In 2017, Vaswani et al. introduced the Transformer and thereby gave
birth to transformer-based encoder-decoder models.

Analogous to RNN-based encoder-decoder models, transformer-based
encoder-decoder models consist of an encoder and a decoder that are
each stacks of residual attention blocks. The important thing innovation of
transformer-based encoder-decoder models is that such residual attention
blocks can process an input sequence ${\mathbf{X}}_{1:n}mathbf\{X\}\_\{1:n\}$

As a reminder, to unravel a sequence-to-sequence problem, we want to
discover a mapping of an input sequence ${\mathbf{X}}_{1:n}mathbf\{X\}\_\{1:n\}$

Much like RNN-based encoder-decoder models, the transformer-based
encoder-decoder models define a conditional distribution of goal
vectors ${\mathbf{Y}}_{1:m}mathbf\{Y\}\_\{1:m\}$

$$p_{{\theta }_{\text{enc}},{\theta }_{\text{dec}}}({\mathbf{Y}}_{1:m}\mathrm{\mid }{\mathbf{X}}_{1:n})\mathrm{.}p\_\{theta\_\{text\{enc\}\},\; theta\_\{text\{dec\}\}\}(mathbf\{Y\}\_\{1:m\}\; |\; mathbf\{X\}\_\{1:n\}).$$

The transformer-based encoder part encodes the input sequence ${\mathbf{X}}_{1:n}mathbf\{X\}\_\{1:n\}$

$$f_{{\theta }_{\text{enc}}}:{\mathbf{X}}_{1:n}\to {\bar{\mathbf{X}}}_{1:n}\mathrm{.}f\_\{theta\_\{text\{enc\}\}\}:\; mathbf\{X\}\_\{1:n\}\; to\; mathbf\{overline\{X\}\}\_\{1:n\}.$$

The transformer-based decoder part then models the conditional
probability distribution of the goal vector sequence ${\mathbf{Y}}_{1:n}mathbf\{Y\}\_\{1:n\}$

$$p_{{\theta }_{dec}}({\mathbf{Y}}_{1:n}\mathrm{\mid }{\bar{\mathbf{X}}}_{1:n})\mathrm{.}p\_\{theta\_\{dec\}\}(mathbf\{Y\}\_\{1:n\}\; |\; mathbf\{overline\{X\}\}\_\{1:n\}).$$

By Bayes’ rule, this distribution might be factorized to a product of
conditional probability distribution of the goal vector ${\mathbf{y}}_{i}mathbf\{y\}\_i$

$$p_{{\theta }_{dec}}({\mathbf{Y}}_{1:n}\mathrm{\mid }{\bar{\mathbf{X}}}_{1:n})=\prod _{i=1}^n{p}_{{\theta }_{\text{dec}}}({\mathbf{y}}_i\mathrm{\mid }{\mathbf{Y}}_{0:i-1},{\bar{\mathbf{X}}}_{1:n})\mathrm{.}p\_\{theta\_\{dec\}\}(mathbf\{Y\}\_\{1:n\}\; |\; mathbf\{overline\{X\}\}\_\{1:n\})\; =\; prod\_\{i=1\}^\{n\}\; p\_\{theta\_\{text\{dec\}\}\}(mathbf\{y\}\_i\; |\; mathbf\{Y\}\_\{0:\; i-1\},\; mathbf\{overline\{X\}\}\_\{1:n\}).$$

The transformer-based decoder hereby maps the sequence of encoded hidden
states ${\bar{\mathbf{X}}}_{1:n}mathbf\{overline\{X\}\}\_\{1:n\}$

Having defined the conditional distribution $p_{{\theta }_{\text{dec}}}({\mathbf{y}}_i\mathrm{\mid }{\mathbf{Y}}_{0:i-1},{\bar{\mathbf{X}}}_{1:n})p\_\{theta\_\{text\{dec\}\}\}(mathbf\{y\}\_i\; |\; mathbf\{Y\}\_\{0:\; i-1\},\; mathbf\{overline\{X\}\}\_\{1:n\})$

Let’s visualize the entire technique of auto-regressive generation of
transformer-based encoder-decoder models.

The transformer-based encoder is coloured in green and the
transformer-based decoder is coloured in red. As within the previous section,
we show how the English sentence “I need to purchase a automobile”, represented by ${\mathbf{x}}_1=\text{I}mathbf\{x\}\_1\; =\; text\{I\}$

To start with, the encoder processes the entire input sequence ${\mathbf{X}}_{1:7}mathbf\{X\}\_\{1:7\}$

Next, the input encoding ${\bar{\mathbf{X}}}_{1:7}mathbf\{overline\{X\}\}\_\{1:7\}$

$$p_{{\theta }_{enc,dec}}(\mathbf{y}\mathrm{\mid }{\mathbf{y}}_0,{\mathbf{X}}_{1:7})=p_{{\theta }_{enc,dec}}(\mathbf{y}\mathrm{\mid }\text{BOS},\text{I want to buy a automobile EOS})=p_{{\theta }_{dec}}(\mathbf{y}\mathrm{\mid }\text{BOS},{\bar{\mathbf{X}}}_{1:7})\mathrm{.}p\_\{theta\_\{enc,\; dec\}\}(mathbf\{y\}\; |\; mathbf\{y\}\_0,\; mathbf\{X\}\_\{1:7\})\; =\; p\_\{theta\_\{enc,\; dec\}\}(mathbf\{y\}\; |\; text\{BOS\},\; text\{I\; need\; to\; purchase\; a\; automobile\; EOS\})\; =\; p\_\{theta\_\{dec\}\}(mathbf\{y\}\; |\; text\{BOS\},\; mathbf\{overline\{X\}\}\_\{1:7\}).$$

Next, the primary goal vector ${\mathbf{y}}_{1}mathbf\{y\}\_1$

$$p_{{\theta }_{dec}}(\mathbf{y}\mathrm{\mid }\text{BOS Ich},{\bar{\mathbf{X}}}_{1:7})\mathrm{.}p\_\{theta\_\{dec\}\}(mathbf\{y\}\; |\; text\{BOS\; Ich\},\; mathbf\{overline\{X\}\}\_\{1:7\}).$$

We are able to sample again and produce the goal vector ${\mathbf{y}}_{2}mathbf\{y\}\_2$

$$\text{EOS}\sim p_{{\theta }_{dec}}(\mathbf{y}\mathrm{\mid }\text{BOS Ich will ein Auto kaufen},{\bar{\mathbf{X}}}_{1:7})\mathrm{.}text\{EOS\}\; sim\; p\_\{theta\_\{dec\}\}(mathbf\{y\}\; |\; text\{BOS\; Ich\; will\; ein\; Auto\; kaufen\},\; mathbf\{overline\{X\}\}\_\{1:7\}).$$

And so forth in auto-regressive fashion.

It is crucial to know that the encoder is simply utilized in the primary
forward pass to map ${\mathbf{X}}_{1:n}mathbf\{X\}\_\{1:n\}$

As might be seen, only in step $i=1i=1$

In 🤗Transformers, this auto-regressive generation is finished under-the-hood
when calling the .generate() method. Let’s use considered one of our translation
models to see this in motion.

from transformers import MarianMTModel, MarianTokenizer

tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de")
model = MarianMTModel.from_pretrained("Helsinki-NLP/opus-mt-en-de")


input_ids = tokenizer("I need to purchase a automobile", return_tensors="pt").input_ids


output_ids = model.generate(input_ids)[0]


print(tokenizer.decode(output_ids))

Output:

     Ich will ein Auto kaufen

Calling .generate() does many things under-the-hood. First, it passes
the input_ids to the encoder. Second, it passes a pre-defined token, which is the $\text{<pad>text{<pad>}</pad> symbol within the case of<br> <code>MarianMTModel</code> together with the encoded <code>input\_ids</code> to the decoder.<br> Third, it applies the beam search decoding mechanism to<br> auto-regressively sample the following output word of the <em>last</em> decoder<br> output <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow></mrow><mn>1</mn></msup></mrow>{}^1</math></span></span></pad>}$