What local AI is
A chat assistant like ChatGPT runs on a model: a very large file of numbers, built by training on huge amounts of text, that predicts a good reply to whatever you type. The online services keep their models on the company’s computers. Local AI means downloading a model and running it on your own computer instead.
That changes three things. What you type never leaves the machine, so you can paste in private letters, medical notes or work documents. It keeps working with the internet switched off. And there’s no plan or message limit: once a model is downloaded, every question is free apart from the electricity.
The catch is size. The models behind the big online services are far larger than anything a home computer can hold, so a local model knows less and makes more mistakes. For summarising, rewording, drafting, explaining and answering questions about your own documents, the better small models are good enough to use every day.
One computer or the whole house: decide this first
- One person, one computer. Install Ollama on a Windows PC or a Mac and chat in its own window. Nothing else is needed, and this is where everyone should start.
- Everyone in the house, from any device. Run Ollama on a computer that stays on, usually a home server with a graphics card, and add Open WebUI, a chat page that opens in any browser on your network, phones included. Each person gets their own login and chat history.
You can start with the first and move to the second later. The models work the same way in both.
How the pieces fit together
There are three parts, and it helps to keep them separate in your head:
| Part | What it is | Examples |
|---|---|---|
| The model | The file that does the thinking, from a few gigabytes to tens of gigabytes | Families such as Llama, Gemma, Qwen, Mistral and gpt-oss |
| The runner | The program that loads the model and does the work | Ollama, LM Studio |
| The chat window | Where you type and read the answers | Ollama’s own window, Open WebUI |
Each model family comes in sizes, counted in billions of parameters (the numbers inside the model) and written like 4b, 12b or 27b. Bigger is cleverer but needs more memory. Most downloads are quantised, meaning compressed so they fit in less memory with only a small loss in quality.
What it needs
Memory decides which models you can run. For replies at reading speed, the whole model has to fit in your graphics card’s own memory, or in the shared memory of an Apple Silicon Mac. If it doesn’t fit, Ollama and LM Studio run the rest on the main processor, and replies slow to a trickle. The hardware for AI page explains what fits where.
- A Windows PC running Windows 10 22H2 or newer. An NVIDIA graphics card helps most, and recent AMD Radeon cards work too.
- A Mac with an Apple M-series chip and macOS 14 Sonoma or newer. Older Intel Macs run models on the processor only, which is slow.
- Disk space. Each model is several gigabytes, and a small collection soon reaches tens of gigabytes.
- No account. Ollama doesn’t need one for models you download.
How Ollama and Open WebUI fit in
Ollama does the heavy lifting:
- A model library. Choose a model by name and it downloads the right file.
- Its own chat window on Windows and Mac. You can drag in PDFs and text files, and pictures for models that can see them.
- Automatic hardware settings. It finds your graphics card and puts as much of the model on it as fits.
- A connection point for other apps. Open WebUI and many coding tools can use the models Ollama has downloaded.
Open WebUI adds what a household needs:
- A chat page that works like the online assistants, in any browser, including on phones.
- Accounts. The first person to sign up becomes the administrator and decides who else can join.
- Your own documents. Upload files and ask questions about them.
- Model downloads from the page, so you rarely need to touch the server itself.
One thing to know: Ollama also offers cloud models, which run on Ollama’s servers rather than on your computer. They only work after you sign in to an Ollama account, and their names include “cloud”. They’re a way to try a big model, but what you type leaves your house, so for anything private use models you’ve downloaded. Ollama also has a local-only mode that switches cloud features off.
The easy way to set it up
Neither route needs the terminal.
On one Windows PC or Mac:
- Download Ollama from its download page. On Windows, run the installer, which doesn’t need administrator rights. On a Mac, open the file and drag Ollama into Applications.
- Open Ollama. Skip signing in if you only want local models.
- Choose a model in the chat window. Start small, with a 4b or 8b model from a well-known family such as Gemma or Qwen, and let it download.
- Type a question. The first reply takes a little longer while the model loads.
If you’d rather browse models in a shop-like window, LM Studio, in the options below, does the whole job as one desktop app.
On a home server running ZimaOS:
- Make sure the server has an NVIDIA graphics card. ZimaOS 1.6.2 and later detect NVIDIA cards and load a driver for them.
- Open the App Store and install Open WebUI. The store’s version has Ollama built in, so this one app is the whole setup.
- Open it from the dashboard and create the first account. That account is the administrator.
- In the admin settings, download a model, then choose it at the top of a new chat.
- Add accounts for the rest of the house, and use Tailscale to reach it from your phone when you’re out.
The store’s Open WebUI is set up for an NVIDIA card. A server without one answers slowly anyway, so the PC or Mac route is the better start.
If you’re comfortable with the terminal, Ollama installs on Linux with one command from its download page, and Open WebUI runs in Docker. See Docker and containers.
Questions people ask
Is it as good as ChatGPT?
No. The online services use much bigger models. A good local model handles drafting, summaries, explanations and questions about your own files well, but it knows less, reasons less carefully and makes things up more often. Many people use both: a local model for anything private and an online one for the rest. See AI tools.
Which model should I start with?
A recent model from a well-known family, in the largest size that fits your memory with some room to spare. Try two or three on the same real task and keep the one whose answers you like.
Does it work without the internet?
Yes, once the model is downloaded. You only need the internet to download models and updates.
Can my family use it from their phones?
Yes, with Open WebUI on a computer that stays on. At home they open its address in the phone’s browser. Away from home, Tailscale connects the phone to your network without opening anything on your router. See remote access.
Will it slow my computer down?
While it’s answering, yes, and the graphics card works hard. Ollama keeps a model in memory for five minutes after you last use it, then frees the memory.
