Sentence embeddings

Intuition

BERT will tell you how similar two sentences are, and it is very good at it. The catch is how it does it: both sentences go through the network together, so a similarity score costs one full forward pass per pair. Reimers and Gurevych put the consequence in a sentence: "Finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations (~65 hours) with BERT" [1, Abstract].

The fix is to make the representation the unit of work rather than the comparison. Encode each sentence once, independently, into a vector; compare vectors with cosine similarity. Now a collection of n sentences costs n forward passes, and every pairwise question afterwards is arithmetic. The same paper reports "about 5 seconds with SBERT" for the collection that took 65 hours [1, Abstract].

This is the vector space model with a better vector. Manning et al. set up the frame — documents as vectors, similarity as cosine — and note precisely what that representation throws away: "this representation loses the relative ordering of the terms in each document ... the documents Mary is quicker than John and John is quicker than Mary are identical in such a bag of words representation" [2, §6.3.1]. A sentence embedding is an attempt to keep the geometry and stop losing the order.

Formal definition

How it works

Where it's used

Why & when

Worked example

Pitfalls