difficulty 4/5
Instruction tuning
Fine-tuning a pretrained model on tasks phrased as instructions, and then on human preferences between its own outputs. What turns a text completer into something that answers the question you asked.
Intuition
A pretrained language model continues documents. Asked a question, the most probable continuation is often another question, because that is what the web does. Nothing in the objective ever asked it to be helpful — and Ouyang et al. put the consequence in their first sentence: "making language models bigger does not inherently make them better at following a user's intent" [2, Abstract].
Prompting is one answer: describe the task in the input. It works, and GPT-3 was built to show how far it goes [3, Abstract]. But it works much better with examples than without — "GPT-3's zero-shot performance is much worse than few-shot performance on tasks such as reading comprehension, question answering, and natural language inference" [1, §1] — and every request pays for the examples again.
Instruction tuning moves that work into the weights. Train the model on many tasks phrased as instructions, and it learns the general skill of doing what an instruction says, including for instructions it has never seen. Then, optionally, train it further on which of its own outputs people actually prefer. The two halves are different techniques with different evidence, and this page keeps them apart.
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Finetuned Language Models Are Zero-Shot Learners (2022)paperhttps://ar5iv.labs.arxiv.org/html/2109.01652
- Training Language Models to Follow Instructions with Human Feedback (2022)paperhttps://ar5iv.labs.arxiv.org/html/2203.02155
- Language Models are Few-Shot Learners (2020)paperhttps://ar5iv.labs.arxiv.org/html/2005.14165