Natural language processing

Intuition

Text is the wrong shape for a model. A model consumes vectors of numbers; a sentence is a variable-length sequence of symbols whose meaning depends on order, context, and a great deal that is never written down. Everything in the field is downstream of that mismatch.

The historical arc is one long retreat from hand-built structure. First linguists wrote the rules; then features were counted and the model learned the weights; then the representation itself was learned; and now the representation is learned from unlabelled text before your task is even known. Each step removed a place where a human had to specify what mattered.

D2L states the scope plainly: "natural language processing studies interactions between computers and humans using natural languages" [1, ch. 15, introduction]. The engineering question underneath is narrower and more useful — what is the vector, and where did it come from?

Formal definition

The field has no single defining equation. What it has is a two-stage architecture that almost every modern system shares, and both stages have precise statements in the literature.

Stage one — pretraining. A model is fitted to unlabelled text by predicting hidden parts of it from the surrounding context. D2L organises its treatment around exactly this, distinguishing context-independent representations, where a word gets one vector regardless of use, from context-sensitive ones, where the vector depends on the surrounding tokens [1, §15.8.1]. The canonical context-sensitive model is BERT, which is "designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers" [2, abstract].

Stage two — fine-tuning. The pretrained model is adapted to a specific task. The economy of this is the whole reason the field reorganised around it: a pretrained BERT "can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications" [2, abstract].

How it works

Tokenization. Text becomes a sequence of units. Words are the obvious choice and a poor one — vocabularies are unbounded and every unseen word becomes a hole. Subword units are the standard answer, and D2L treats them as a topic in their own right [1, §15.6].

Embedding. Each token becomes a vector. The context-independent family — word2vec [1, §15.1] and GloVe [1, §15.5] — assigns one vector per word type, which means bank gets a single representation for the river and the money.

Encoding in context. This is where the architecture matters. The Transformer was introduced as "a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely" [3, abstract].

The mechanism is one formula. In "Scaled Dot-Product Attention", the model computes "the dot products of the query with all keys, divide each by √d_k, and apply a softmax function to obtain the weights on the values" [3, §3.2.1]. Run that several times over in parallel — the paper "found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections" [3, §3.2.2] — and you have multi-head attention.

Removing recurrence is not a detail, and the paper's own justification names the three things it was trading: "the total computational complexity per layer", "the amount of computation that can be parallelized", and "the path length between long-range dependencies in the network" [3, §4]. A recurrent model must process token 5 after token 4, so the path between distant tokens is long and the work is sequential. Attention makes that path a constant.

The cost of throwing away recurrence is that word order goes with it. "Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence" [3, §3.5] — which is what positional encodings are, and why the worked example below is a problem attention does not solve for free.

The task head. A small layer on top, trained on your labelled data. The procedure is as plain as it sounds: "for each task, we simply plug in the task-specific inputs and outputs into BERT and fine-tune all the parameters end-to-end" [2, §3.2].

The results that made this the default are worth quoting rather than asserting. The Transformer reported 28.4 BLEU on WMT 2014 English-to-German, "improving over the existing best results, including ensembles by over 2 BLEU", and 41.8 BLEU on English-to-French "after training for 3.5 days on eight GPUs" [3, abstract]. BERT then reported new state-of-the-art results "on eleven natural language processing tasks", including a GLUE score of 80.5 — a 7.7-point absolute improvement — and SQuAD v1.1 test F1 of 93.2 [2, abstract].

Where it's used

D2L picks two tasks as representative, and the choice is instructive: sentiment analysis and natural language inference, "which analyze single text and relationships of text pairs, respectively" [1, ch. 16, introduction]. Almost every applied task is one of those two shapes.

• Single-text tasks — sentiment, topic labelling, spam detection, intent classification, toxicity filtering [1, §16.1].
• Text-pair tasks — natural language inference, duplicate-question detection, retrieval and reranking [1, §16.4].
• Token-level tasks, which D2L treats alongside the sequence-level ones [1, §16.6] — named-entity recognition, part-of-speech tagging, extractive question answering: a label per token rather than per text.
• Sequence-to-sequence tasks — translation and summarization, the setting the Transformer was introduced on [3, abstract].
• Search and semantic similarity, built directly on embeddings rather than on a classifier.

Why & when

Start from a pretrained model. Not because it is fashionable but because of the arithmetic of both labels and compute: the pretraining used unlabelled text, and adapting it needs "just one additional output layer" [2, abstract]. The second half of that bargain is easy to understate — "compared to pre-training, fine-tuning is relatively inexpensive", with a paper's worth of results reproducible "in at most 1 hour on a single Cloud TPU" [2, §3.2]. A few thousand labelled examples now reach places that used to need hundreds of thousands.

Prefer the simple baseline when the signal is lexical. Some problems really are keyword problems. A linear model over word counts is faster, interpretable, runs anywhere, and is not obviously worse at detecting whether an email mentions an invoice.

Watch the language. The results above are English benchmarks. Coverage, tokenizer quality and pretraining data all thin out sharply outside a handful of languages, and a pipeline that works in English can fail quietly in another.

Budget for inference, not just accuracy. The pretrained model is large. That cost is paid on every request, forever, whereas the accuracy is measured once.

Worked example

The single most important thing a representation can throw away, in nine lines.

import numpy as np

sentences = ["the dog bit the man",
             "the man bit the dog"]

vocab = sorted({w for s in sentences for w in s.split()})
print("vocabulary:", vocab)

def bag_of_words(s):
    counts = np.zeros(len(vocab), dtype=int)
    for w in s.split():
        counts[vocab.index(w)] += 1
    return counts

a, b = (bag_of_words(s) for s in sentences)
print("sentence 1:", a)
print("sentence 2:", b)
print("identical vectors:", np.array_equal(a, b))

vocabulary: ['bit', 'dog', 'man', 'the']
sentence 1: [1 1 1 2]
sentence 2: [1 1 1 2]
identical vectors: True

Two sentences with opposite meanings, one vector. No classifier downstream can recover the difference, because the difference was destroyed before the classifier ran — and the model will report a confident, consistent, wrong answer on one of them without any sign that something is missing.

Everything after bag-of-words is an answer to this. Word order is the missing information, and recurrent networks handled it by reading in sequence.

Attention did not simply inherit that. Self-attention lets every token look at every other one directly, which fixes the path length [3, §4] — but a set of vectors attending to each other is still a set, and permuting it changes nothing. The Transformer therefore has to put the order back by hand: "in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens" [3, §3.5].

With those positional encodings in place, context-sensitive embeddings are the result — the vector for bit differs between the two sentences because its neighbours, and their positions, do [1, §15.8.1].

Pitfalls

Assuming the tokenizer is neutral. It decides what the model can represent. A tokenizer built for English splits other scripts badly, inflating sequence lengths and costs, and it is upstream of everything — which is why subword methods are a subject rather than a setting [1, §15.6].

Reading benchmark numbers as capability. GLUE at 80.5 and SQuAD F1 at 93.2 are results on specific datasets [2, abstract], not statements about language understanding in general. Test on your own data.

One vector per word, for words with several senses. This is the known limit of the context-independent family and the reason the context-sensitive one exists [1, §15.8.1].

Leakage through pretraining data. Your evaluation set may already have been in the pretraining corpus. The train/test discipline that applies to your own data does not extend to a corpus you did not assemble and cannot inspect.

Treating fluency as correctness. A model that produces well-formed text is demonstrating that it models the distribution of text, which is what it was fitted to do. It is not evidence that the content is true.