Why memory matters more than speed
An AI model is a large file that has to sit in memory while it works, and for every word it writes, the computer reads through the whole thing. So the question isn’t only how fast the chip is, but where the model is kept.
A graphics card has its own memory, called VRAM (video memory), wired right next to its processor and much faster than the computer’s ordinary memory. If the whole model fits in VRAM, replies arrive faster than you can read them. If it doesn’t fit, the leftover part runs on the main processor and ordinary memory, and replies slow to a word or two at a time.
An Apple Silicon Mac works differently. It has one pool of memory shared by the processor and the graphics, so most of the Mac’s memory can hold a model. That’s why a Mac with plenty of memory can load models that would need an expensive graphics card on a PC.
Use what you have, or buy: decide this first
Don’t buy anything until you’ve tried a small model on the computer you own. Install Ollama or LM Studio (see Local AI), download a small model and use it on real tasks for a week. One of three things happens:
- It’s good enough. You’re done, and you’ve spent nothing.
- It’s good but slow. The model doesn’t fit in fast memory. More VRAM is the fix, and that’s what this page is about.
- It isn’t clever enough. You need a bigger model, which needs more memory still. Rent a large graphics card for an afternoon, in the options below, to find out whether the bigger model is worth it before you buy.
Where the memory can come from
| Option | Memory for models | The trade-off |
|---|---|---|
| NVIDIA graphics card | The card’s VRAM | Fastest when the model fits, and every tool supports it |
| AMD Radeon card | The card’s VRAM | Good for chat, more work for images |
| Mac with Apple Silicon | Most of the Mac’s memory | Quiet and simple, but memory can’t be added later |
| Shared-memory mini PC | Up to 128 GB | Holds huge models, runs them more slowly |
| Main processor only | The computer’s ordinary memory | Free, but slow beyond small models |
As a rule of thumb, a model at the usual 4-bit compression needs a little over half a gigabyte for every billion parameters (the numbers inside the model), plus room for the conversation. The list further down works this through for common sizes.
What to check before buying a card
- The memory, on the listing itself. Some cards were sold in two memory sizes under the same name. The memory size is the number that matters, so make sure the listing states it.
- Its age. NVIDIA’s 580 drivers are the last to support the GTX 900 and GTX 10 series, so those cards won’t get driver updates after that. For a card you’ll keep, the RTX 20 series or newer is the safer start.
- Power. The card’s maker lists its power draw and a recommended power supply size. Check the wattage on your power supply’s label, and that it has the power cables the card needs.
- Size. Measure the room in your case. Big cards are long and take up two or three slots’ worth of width.
- A slot to put it in. It needs a full-length PCIe x16 slot, the long one nearest the processor. Most mini PCs, laptops and small NAS boxes don’t have one.
- Heat and noise. A busy card is loud. If the server lives in a bedroom or living room, see power, noise and UPS.
How an NVIDIA card fits in
NVIDIA’s advantage is CUDA, its software layer that nearly every AI tool is written for first. In practice:
- Ollama and LM Studio find the card on their own and load as much of the model onto it as fits.
- Image generation in ComfyUI works without extra steps.
- Speech to text with Whisper runs much faster than on the processor.
- Other home server apps can share it. Immich can use it for face and object recognition in your photos, and Jellyfin for converting video while it streams.
- The widest help. Most guides, forum answers and fixes assume an NVIDIA card, which matters when something goes wrong.
The easy way to set it up
None of this needs the terminal on Windows, a Mac or ZimaOS.
- See what you have. On Windows, open Task Manager, choose Performance, then GPU, and read “Dedicated GPU memory”. On a Mac, open the Apple menu and choose About This Mac to see the chip and memory.
- Try a model first. Install Ollama or LM Studio and run something small. LM Studio shows, before you download a model, whether it’s likely to fit in your graphics memory.
- Fit the card. Shut the computer down and unplug it. Put the card in the long slot, screw it to the case, and connect the power cables from the power supply.
- Install the driver. On Windows, download the NVIDIA app from NVIDIA’s website. It installs the driver and keeps it up to date. On ZimaOS 1.6.2 or later, the system detects NVIDIA cards and loads a driver itself. On a Linux desktop such as Ubuntu or Linux Mint, use the driver tool in the system settings.
- Check it’s being used. Load a model in Ollama or LM Studio and watch “Dedicated GPU memory” rise in Task Manager. If replies are slow and the main processor is busy instead, the model didn’t fit. Choose a smaller one.
On a plain Linux server, installing the NVIDIA driver and the extra piece that lets Docker containers use the card takes terminal commands. Giving a card to a virtual machine, called passthrough, is harder still. See server operating systems and virtual machines.
Questions people ask
Is a second-hand card a good idea?
Often, yes, because memory is what you’re paying for and older cards with plenty of it cost much less used. Buy where you can return it, check the memory size and the card’s age against the advice above, and run a model for a while in the first days to make sure it stays stable.
Can I add a graphics card to a mini PC or NAS?
Usually not. Most have no full-size slot, no room and no spare power. If your server is small, a Mac or a shared-memory mini PC is the simpler way to get more memory for models.
Do AMD and Intel cards work?
Recent AMD Radeon cards work with Ollama, LM Studio and llama.cpp, and Ollama’s GPU page lists exactly which ones on Windows and Linux. Intel Arc cards can work through Vulkan, a general graphics standard that LM Studio and Ollama can use, but fewer tools and guides support them.
Is 8 GB enough?
For small models, yes. Models up to about 8 billion parameters fit and are fine for chat, summaries and simple scripts. If you already own an 8 GB card, use it before buying anything bigger.
How much electricity will it use?
A card draws little while idle and a lot while it’s answering. On a server that’s on all day, the idle figure matters most, so look for it in reviews of the card before you buy.
