difficulty 4/5
BERT
A transformer encoder pretrained by hiding 15% of the tokens and predicting them from both sides at once, then fine-tuned on a task with one extra output layer and no architectural surgery.
Intuition
A language model predicts the next word. That objective is free — every text in the world is already labelled with what comes next — which is why it became the way to pretrain on unlabelled data. But it forces a constraint that has nothing to do with language: to predict token i, the model may only look at tokens before it, or the task would be trivial.
For generating text that constraint is exactly right. For understanding it is a self-inflicted wound. Deciding whether "he went to the bank" means a river or a chequebook depends entirely on what comes afterwards, and a left-to-right model is forbidden from looking.
BERT's move is to change the task rather than the architecture. Hide a few tokens at random and predict those [1, §3.1]. Now every prediction may use the whole sentence, because the answer is not sitting in the context waiting to be copied — it has been removed. Bidirectionality stops being cheating and becomes available.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
On the cheat sheet: NLP tasks →
Sources
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019)paperhttps://ar5iv.labs.arxiv.org/html/1810.04805
- Dive into Deep Learningbook · retrieved 2026-09-06https://d2l.ai/