Embedding

Computers cannot work with words directly, so AI systems turn them into numbers. An embedding is that numerical form. OpenAI's documentation defines it as "a vector (list) of floating point numbers" and says the distance between two embeddings measures how related they are: small distances suggest high relatedness¹.

The idea of learning a "distributed representation" for each word was set out by Yoshua Bengio and colleagues in 2003. Their model could handle sentences it had never seen, because words with nearby representations were treated as similar². In 2013 Tomas Mikolov and colleagues at Google showed that good word vectors could be learned from a 1.6 billion word data set in less than a day³. Their paper notes that vector("King") minus vector("Man") plus vector("Woman") lands closest to the vector for "Queen"³.

Modern embedding models work on whole sentences and documents, not just single words. OpenAI's text-embedding-3-small produces 1,536 numbers per input by default, and text-embedding-3-large produces 3,072¹. OpenAI lists search, clustering, recommendations, anomaly detection and classification as typical uses¹.

Embeddings also sit inside every language model. The transformer begins by using learned embeddings to convert each input token into a vector⁴.

An embedding is not the text itself, and you cannot read it like a sentence. It only tells a system how close one item is to another.