Hallucination

Intuition

A language model is trained to produce probable text. Nothing in that objective mentions truth, and nothing in it mentions the document you handed the model either. When the two come apart, what comes out is fluent, well-formed and unsupported — which Ji et al. describe as generation that is "prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios" [1, Abstract].

The word covers two different failures, and conflating them is the most common mistake made about it. Saying something the source contradicts is one problem. Saying something the source simply does not mention is a different one — and, importantly, is not necessarily false.

It is also not rare or exotic, and the sentence reporting how common it is contains its own punchline: "state-of-the-art abstractive summarization systems, evaluated with metrics such as ROUGE, BLEU, and METEOR, have hallucinated content in 25% of their generated summaries" [1, §4]. Those metrics are the ones such systems are tuned on, and they are doing no work against this failure at all — which the worked example below makes concrete.

Formal definition

How it works

Where it shows up

Why & when

Worked example

Pitfalls