Vector
In short
A vector is an ordered list of numbers. In AI, almost everything a model handles, from words to images, is turned into vectors so that it can be calculated with and compared, for example by measuring how close two vectors are.
- Also called
- Vectors, Feature vector
- Topic
- How AI models work
- Level
- Beginner
- First described
- Vectors are standard mathematics. In language AI, Mikolov and colleagues' 2013 Google paper on learning word vectors from very large data sets is a landmark use5.
In maths, a vector is an array of numbers arranged in order, and each number can be picked out by its position1. You can think of a vector as a point in space, with each number giving the position along a different axis1. A list of three numbers is a point in 3D; AI systems use the same idea with hundreds or thousands of numbers, called dimensions.
Vectors are how meaning gets into numbers. An embedding is a vector: OpenAI's documentation describes it as "a vector (list) of floating point numbers" and says the distance between two vectors measures how related they are2. OpenAI recommends cosine similarity for comparing them2, which looks at whether two vectors point in the same direction.
Vectors run all the way through a language model. The transformer converts each token into a vector, and the queries, keys and values used by its attention step are all vectors3. In Meta's first Llama models, the 7 billion parameter version held each token as a vector of 4,096 numbers4.
Word vectors made the idea famous. Mikolov and colleagues' 2013 paper notes that taking the vector for "King", subtracting "Man" and adding "Woman" gives a vector closest to "Queen"5.
A vector on its own has no meaning. The numbers only make sense relative to other vectors produced by the same model.
Example
OpenAI's text-embedding-3-small turns a sentence such as "Where is my nearest GP surgery?" into a vector of 1,536 numbers2. "Find a doctor near me" gets a vector pointing in a similar direction, so a search tool can match them.
Why it matters
Nearly every AI feature you use, from search to chatbots, works by turning things into vectors and comparing them. It is the basic currency of modern AI.
Questions people ask
What is the difference between a vector and an embedding?
A vector is any ordered list of numbers1. An embedding is a vector produced by a model so that similar things end up close together2. All embeddings are vectors, but not the other way round.
What does vector dimension mean?
It is how many numbers the vector holds. OpenAI's text-embedding-3-small produces 1,536 by default and text-embedding-3-large 3,0722. More dimensions can capture more detail but take more storage.
What is cosine similarity?
A way to compare two vectors by the angle between them rather than their length. Vectors pointing the same way score close to 1. OpenAI recommends it for comparing its embeddings2.
Related terms
Tools that use it
- PineconeManaged vector database that lets AI apps and agents search company data by meaning
- Gemini Embedding 2Google's first embedding model that maps text, images, video, audio and PDFs into one space
Sources
- Goodfellow, Bengio and Courville (2016), Deep Learning, MIT Press, chapter 2 (Linear Algebra)
- OpenAI API documentation, Vector embeddings guide
- Vaswani et al. (2017), Attention Is All You Need, arXiv:1706.03762
- Touvron et al. (2023), LLaMA: Open and Efficient Foundation Language Models, arXiv:2302.13971
- Mikolov, Chen, Corrado and Dean (2013), Efficient Estimation of Word Representations in Vector Space, arXiv:1301.3781