Transformer family

Read the Attention is column first, because the other three follow from it.

Bidirectional means every position sees the whole sequence, which is only possible if the training objective hides something. BERT hides it by masking, and the paper is explicit that this is not a stylistic choice: standard language models "can only be trained left-to-right or right-to-left, since bidirectional conditioning would allow each word to indirectly 'see itself'" [1, §3.1]. Remove the mask and the task becomes trivial.

Causal means every position sees only what precedes it, which is exactly what generation requires and exactly what makes a model worse at judging text it already has — the BERT page has the ablation.

Encoder plus decoder keeps both, at the cost of running two stacks.

Everything else in the table is downstream. A bidirectional model is shaped for understanding, a causal one for generation, and the objective is whichever task the attention pattern makes non-trivial.

After BERT, almost nobody added a capability

Look down the Chasing column. BERT bought a new capability; nearly every model under it bought a resource back.

RoBERTa bought accuracy with nothing but a recipe. Its modifications are listed plainly — "(1) training the model longer, with bigger batches, over more data; (2) removing the next sentence prediction objective; (3) training on longer sequences; and (4) dynamically changing the masking pattern" [2, §1] — and the finding is that "BERT was significantly undertrained, and can match or exceed the performance of every model published after it" [2, abstract]. A year of architectural progress turned out to be partly a training-budget artefact, which is the single most useful result in this table.

ALBERT and DistilBERT bought size back, by opposite means. ALBERT applies "two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT", reaching state of the art "while having fewer parameters compared to BERT-large" [3, abstract] — the network is not smaller, its weights are shared. DistilBERT instead trains a genuinely smaller student, and moves the distillation into pretraining: "while most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase", which "reduce[s] the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster" [4, abstract]. One is cheaper to train, the other cheaper to run.

ELECTRA bought compute back by fixing masking's inefficiency. Its objective corrupts the input "by replacing some tokens with plausible alternatives sampled from a small generator network" and then predicts which ones were replaced; the reason it is faster is stated exactly — "the task is defined over all input tokens rather than just the small subset that was masked out" [5, abstract]. That is the arithmetic on the BERT page turned into a design. The payoff is large enough to be worth quoting: a model trained "on one GPU for 4 days" that outperforms GPT trained "using 30x more compute" [5, abstract].

The two that went sideways

T5 attacked the interface rather than the model. Its move is to convert "all text-based language problems into a text-to-text format" [6, abstract], which makes classification, translation and summarisation the same problem with different strings. Its pretraining objective came out of a systematic comparison rather than an argument: span corruption, which corrupts 15% of the sequence in contiguous runs, with an average span length of 3 slightly outperforming independently chosen tokens [6, §3.3.4].

GPT-3 attacked the fine-tuning step itself. The complaint is that pretraining "still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples", where "humans can generally perform a new language task from only a few examples or from simple instructions" [7, abstract]. The answer is scale — "175 billion parameters, 10x more than any previous non-sparse language model" — applied so that "for all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction" [7, abstract]. Fine-tuning stops being a training step and becomes part of the prompt.

And ViT left the field. The claim is negative and that is what makes it interesting: "this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks" [8, abstract]. The condition matters as much as the claim — the result holds "when pre-trained on large amounts of data" [8, abstract], which is the same weak-inductive-bias bargain the transformer page describes, paid in images.

Which to reach for

Classifying, tagging or extracting from text you already have? An encoder. Start from a RoBERTa-family checkpoint rather than the original BERT, for the reason RoBERTa established [2, abstract].

Generating text? A decoder. The causal mask is the requirement, not a limitation to work around.

Turning one text into another — summarising, translating, rewriting? An encoder-decoder, or a decoder with the input in the prompt. T5's argument is that the distinction between these tasks is smaller than it looks [6, abstract].

Constrained on latency or memory? Distillation and parameter sharing are the two established answers, and they trade differently: DistilBERT is smaller at inference [4, abstract], ALBERT is lighter to train [3, abstract].

One caution for the whole table. Every row reports benchmark scores, and every row was pretrained on a corpus of web text whose content the authors did not choose term by term. The numbers are results on specific datasets; they are not statements about language, and the biases in the corpus are inherited by the representation whichever objective was used.