Chips, compute and data centres
Inference chip
In short
An inference chip is a processor used to run a trained AI model on new data, so it can produce a prediction, a label or a reply. Training, the step that sets the model's weights, is a different job.
- Also called
- AI inference accelerator, Inference accelerator
- Topic
- Chips, compute and data centres
- Level
- Some background helps
- First described
- Norman P. Jouppi and colleagues, in a 2017 study, describe the Tensor Processing Unit as a custom ASIC deployed in data centres since 2015 to accelerate neural-network inference. The project was to build that inference ASIC, with off-the-shelf GPUs used for training2.
Inference is the production step. The International Energy Agency defines it as deploying a trained model on new or live data to produce predictions, classifications, decisions or responses. Training is the earlier step, in which the model learns from data. The processors used include graphics processors and task-specific accelerators such as tensor processing units and inference-optimised application-specific integrated circuits1.
A clear early case is Google's Tensor Processing Unit. Norman Jouppi and colleagues describe a custom ASIC, in data centres since 2015, built to accelerate neural-network inference. They bought GPUs for training and built the TPU for inference. Its core is a matrix-multiply unit. The models they measured held 5 million to 100 million weights, kept in memory and read while the chip ran. On the TPU those weights are read-only. Speech, vision, translation and search ranking, written in TensorFlow, were compiled onto the chip2.
These inference applications usually put response time ahead of throughput, because they are user-facing, and reading the weights takes time and energy2. The IEA adds that memory access and data movement are a growing share of AI system energy, and that compiling for the hardware, or running the trained model in lower-precision numbers, can cut energy per query. Those accelerators outperform a general-purpose CPU on machine-learning work, and on some workloads they beat a contemporary AI GPU on performance per watt1.
The TPU measurements are from data-centre servers. The IEA separately describes processing on or near the device, which it calls edge computing, for autonomous driving, industrial robots and on-device inference. In each setting the chip uses weights that training has already set12.
Example
Jouppi and colleagues describe the projection that started the TPU project. If people used voice search for three minutes a day, speech-recognition models would have doubled the computation in their data centres. The chip was built to run those trained models, not to retrain them on each query2.
Why it matters
Most AI a person actually touches is inference: a trained model answering a new request. The IEA says the balance of AI electricity use has already shifted from training to that day-to-day inference1.
Questions people ask
Is an inference chip the same as a GPU?
No. A GPU can run inference, and Jouppi and colleagues measured one doing so. The IEA also classes tensor processing units and inference-optimised ASICs as task-specific accelerators for this work. The name describes the job, not one design12.
What is the difference between training and inference chips?
Training is how a model learns from data and its weights get set. Inference deploys those weights on new input. Jouppi and colleagues bought GPUs for training and built a custom ASIC for inference. The same paper also times a GPU on inference, so the split is not absolute2.
Where are inference chips used?
Jouppi and colleagues measured theirs in data centres, on speech, vision, translation and search ranking. The IEA separately describes on-device inference, autonomous driving and industrial robots as uses of computing at the edge, where processing stays near the device12.