Inference cost

In short

Inference cost is what it costs to run a trained AI model each time someone uses it: the chips, electricity and data centre time needed to produce an answer. For customers it usually shows up as a price per million tokens.

Also called: Cost of inference, Serving cost, Cost per token
Topic: The AI business
Level: Some background helps

Training a model is a large one-off bill. Inference is the running cost that comes after, paid every time the model answers a question. The more people use a product, the bigger this bill gets.

Providers pass it on as token prices, and the range is wide. OpenAI lists GPT-6 Astra at $10 per million input tokens and $50 per million output tokens, against $0.10 and $0.50 for GPT-6 Luna¹. Output costs more than input at both OpenAI and Anthropic¹².

The cost of a given level of ability has fallen fast. Stanford's 2025 AI Index found that the cost of running a system as good as GPT-3.5 dropped more than 280-fold between November 2022 and October 2024³. Epoch AI found prices for a fixed level of performance falling between 9 and 900 times a year, depending on the task, while warning the fastest drops may not last⁴.

Cheaper tokens do not always mean cheaper answers. Reasoning models "think" in hidden tokens that are billed as output tokens⁶, so one answer can use many more tokens than before. Anthropic says its newer tokeniser produces about 30% more tokens for the same text².

For the companies serving AI, inference eats into margins. Microsoft said its cloud gross margin fell to 66% in the year to June 2026, "driven by continued investments in AI infrastructure and growing AI product usage"⁵.

Example

Asking a chatbot to summarise a long PDF costs the provider far more than answering "What is the capital of France?", because the whole document has to be read as input tokens every time.

Why it matters

It sets what AI products cost and whether they make money. It is also why free tiers have limits and why companies route easy questions to cheaper models.

Questions people ask

Why is AI inference so expensive?

Every answer runs a large model on specialised chips in a data centre, using electricity and costly hardware. The bigger the model and the longer the answer, the more it costs. Microsoft, for example, says AI infrastructure and usage pushed its cloud gross margin down to 66% in its 2026 financial year⁵.

Is the cost of AI going down?

Per unit of capability, yes, sharply. Stanford's AI Index measured a more than 280-fold fall in the cost of GPT-3.5-level performance in about two years³. But reasoning models and agents use far more tokens per task⁶, so total spending can still rise.

What is the difference between training cost and inference cost?

Training cost is the one-off expense of building a model from data. Inference cost is the ongoing expense of running that model every time someone uses it. For a popular product, inference costs keep growing with usage long after training ends.

Related terms

Sources

Back to the glossary