Small language model
In short
A small language model (SLM) is an AI text model compact enough to run on an everyday device, such as a phone or laptop, instead of in a data centre. Most have under about 10 billion parameters1.
- Also called
- SLM, Small LLM, On-device language model
- Topic
- How AI models work
- Level
- Beginner
There is no official size cut-off. One working definition, from NVIDIA researchers, is a language model that fits on a common consumer device and can answer one person quickly enough to be practical1. The same paper says that, as of 2025, most models below 10 billion parameters would count1. Parameters are the learned numbers inside a model.
Size drives cost. The NVIDIA paper says serving a 7 billion parameter model is 10 to 30 times cheaper in latency, energy and computing operations than serving a 70 to 175 billion parameter model1.
Small does not have to mean weak. Microsoft's Phi-3-mini has 3.8 billion parameters and was trained on 3.3 trillion tokens of heavily filtered web data and synthetic data. It scored 69% on the MMLU knowledge test, which its authors say rivals GPT-3.5, while being small enough to run on a phone2.
You meet them in phones and apps. Apple's on-device foundation model has about 3 billion parameters and runs on Apple silicon. App developers can call it through Apple's Foundation Models framework, and Apple says that inference is free of cost to them3.
An SLM is not a different kind of AI. It is a language model with fewer parameters, so it usually knows less and handles fewer kinds of task than a large one. That makes it a good fit for narrow, repeated jobs. The NVIDIA researchers argue this is what most AI agent steps are, and that small models are often the better and cheaper choice there1.
Example
A note-taking app on an iPhone that tidies up your shopping list without an internet connection could use Apple's built-in model of about 3 billion parameters, through the Foundation Models framework3.
Why it matters
Small models can run privately on your own device, work offline and cost far less to serve. They are why more AI features now run on phones rather than in the cloud.
Questions people ask
What is the difference between a small and a large language model?
Mostly size and where it runs. One research definition calls a model small if it fits on a consumer device and responds fast enough for one user, and large if it does not. Serving a 7 billion parameter model can be 10 to 30 times cheaper than serving a 70 to 175 billion parameter one1.
Can a small language model run on my phone?
Yes. Microsoft designed Phi-3-mini, with 3.8 billion parameters, to be small enough to deploy on a phone2. Apple runs a model of about 3 billion parameters directly on its devices3.
Are small language models less accurate?
Usually somewhat, all else being equal. In Microsoft's Phi-3 family, the 3.8 billion parameter mini scored 69% on MMLU, while the larger 14 billion parameter medium model scored 78%2. For narrow tasks the gap can matter less.
Related terms
Tools that use it
- Gemma 4Google's open-weight model family in four sizes for reasoning and agents on local hardware
- Qwen3.5 SmallThe small open models in Alibaba's Qwen3.5 series, from 0.8B to 9B parameters, for phones and laptops
- OllamaFree tool for running open AI models on a home computer, with optional paid cloud models
Sources
- Belcak et al. (2025), Small Language Models are the Future of Agentic AI, arXiv:2506.02153
- Abdin et al. (2024), Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone, arXiv:2404.14219
- Apple Machine Learning Research (2025), Updates to Apple's On-Device and Server Foundation Language Models