Local model

Most chatbots send what you type to a data centre. A local model is downloaded once and then runs on your own hardware. LM Studio, for example, says it can work entirely offline once you have the model files, including chatting with models and with documents¹.

Privacy is the main draw. Ollama's documentation says that when you run models locally, the company does not see your prompts or data². Ollama also offers cloud-hosted models, so not everything it runs is local².

You can only run a model locally if its weights, the trained numbers that make up the model, are available to download. Open-weight models qualify; closed models do not. A widely used engine is llama.cpp, whose stated goal is running language models with minimal setup on a wide range of hardware³.

To squeeze a model into a laptop's memory, these tools use quantisation, storing each number with fewer bits. llama.cpp supports from 1.5-bit to 8-bit formats to cut memory use³. The cost is some accuracy: Apple, which compresses its own phone model, says the result "often shows a moderate level of quality loss"⁴.

Speed depends on your computer's chip and memory, so the same model can feel quick on one machine and sluggish on another.