Latency
In short
Latency is the delay between sending a request to an AI model and getting its answer. It is often measured as time to first token, when the reply starts to appear, and as total time until the reply is finished.
- Also called
- Response time, Time to first token, TTFT
- Topic
- Using AI tools
- Level
- Beginner
Anthropic defines latency as the time a model takes to process a prompt and generate an output, and says it depends on the size of the model, the complexity of the prompt and the infrastructure behind it1.
Two measures matter most. Time to first token (TTFT) is how long you wait before any of the reply appears1. Nvidia notes that it includes queuing, network time and reading the prompt, and grows with longer prompts3. End-to-end latency is the time until the full reply arrives, which is TTFT plus the time spent writing3. Writing is usually the slow part: OpenAI says generating tokens "is almost always the highest latency step", and that cutting output by half may cut latency by about half, while halving the prompt saves only around 1 to 5%2.
The usual fixes are a smaller or faster model, shorter prompts and answers, fewer back-and-forth requests, and running independent steps in parallel12. Streaming does not make the model faster, but it shows text as it is written, so the wait feels shorter1.
Latency is not throughput. Throughput is how many tokens a system produces per second across all users at once. Serving more people at the same time raises total throughput up to a point, but each person's reply gets slower3.
Example
On a voice assistant, time to first token is the awkward pause after you finish speaking and before it starts to answer. A short, fast reply feels natural; a long pause makes people talk over it.
Why it matters
Speed shapes whether an AI tool feels usable. It is also a trade-off: the slower, more thorough models and settings often give better answers but keep you waiting.
Questions people ask
Why are reasoning models slower?
They write extra hidden thinking before the answer, and generating tokens is usually the slowest step2. Anthropic lets developers turn the amount of thinking down with an "effort" setting, and points to its fastest model, Claude Haiku 5.5, for speed-critical apps1.
Does streaming reduce latency?
Not the total time. Streaming sends the reply piece by piece as it is generated, so people can start reading sooner, which Anthropic says improves how responsive an app feels1.