Open Source Special Topic · Running It on Your Own Machine

How Large a Model Can Your Computer Run

Everything so far has been concepts; this lesson is hands-on. Pick your GPU or your Mac model and read the answer straight off. By the end you will see that the bar for running models locally is lower than most people assume.

Run the Numbers First

Platform

Device

Precision

Every candidate model is Apache 2.0 weights you can actually download. The conversion produces estimates, meant to help you judge feasibility rather than replace real measurement.

The last row, Qwen3.8-Max, stays red no matter what you pick. That is not a bug in the list. It is the open-weight flagship from the third lesson of this chapter: run 2.4 trillion parameters through the formula above and you need 1560 GB, more than even the most expensive machine on the list can offer. It stays in the table because "the weights are open" and "you can run it" are two different things, and seeing that number lands better than reading the words "very large."

Where the Formula Comes From

The result above is not looked up in a table. It is one multiplication:

VRAM needed (GB) ≈ parameter count (B) × precision factor
Example: an 8B model at INT4, 8 × 0.65 ≈ 5.2 GB

The factor is the part that needs explaining. To load the weights alone, INT4 takes 0.5 bytes per parameter, so an 8B model needs only 4 GB. But a running model also needs extra room to hold the intermediate state that piles up during a conversation, the KV Cache covered in Part 2. That overhead is already baked into the factor, so do not multiply in another safety margin on top of it, or you will end up concluding that no machine can run anything.

Four Precision Levels

Quantization means storing each parameter with fewer bits. Fewer bits means a smaller footprint, and the price is lost precision.

FP32

× 4.0

Full precision. Used essentially only during training; nobody runs local inference this way.

FP16

× 2.6

Half precision. The original format models ship in, and the baseline for quality.

INT8

× 1.3

Half the size, with quality loss you usually cannot notice. A safe pick if you have the VRAM.

INT4

× 0.65

A quarter of the original size. On most everyday tasks the loss is noticeable but acceptable.

Most common locally

Switch the precision from FP16 to INT4 in the calculator above and watch which models become runnable. Quantization is the single most effective way to lower the bar for local deployment: an 8 GB card cannot handle an 8B model at FP16, but at INT4 it has room to spare.

Quantization loss is not spread evenly. For everyday questions, summarizing, and rewriting, INT4 is generally good enough, but on tasks that need long reasoning chains or exact calculation the drop is more visible. For demanding work, take a model one size down at INT8 rather than one size up at INT4.

Two Kinds of Hardware, Two Sets of Rules

NVIDIA GPU

VRAM is dedicated; you can use just about all of the rated capacity
High bandwidth and fast generation; at the same model size it feels noticeably smoother
Capacity is a hard ceiling — consumer cards currently top out around 32 GB
The most mature software ecosystem; almost any problem you hit already has a searchable fix

Apple Silicon

CPU and GPU share unified memory, and the system will not let you hand all of it to the GPU
The calculator assumes roughly 75% is assignable, which is a conservative estimate
A big capacity advantage; high-end configurations fit sizes consumer GPUs never reach
Bandwidth usually trails a similarly priced discrete GPU, so large models generate more slowly

Put simply: NVIDIA competes on speed, Apple on capacity. If you want to run models above 30B, a Mac with plenty of memory is often more realistic than a consumer GPU; if you are after response speed, a discrete GPU suits you better.

The share of unified memory available to the GPU can be adjusted on macOS through iogpu.wired_limit_max; 75% is a conservative estimate for the default configuration, not a hard ceiling.

What Makes MoE Models Different

Entries in the results list with an A in the name are MoE models, such as Qwen3-30B-A3B, meaning 30B parameters in total with 3B actually activated each pass. These come with a trap that is easy to fall into:

VRAM scales with total parameters; speed scales with activated parameters. For a 30B MoE model you need enough VRAM to hold 30B, but it runs at close to the speed of a 3B. Big footprint, fast execution.

That makes MoE a good fit when VRAM is plentiful but you want quick responses — a high-memory Mac, for example. The reverse also holds: if VRAM is tight, a smaller dense model gives you more for the same footprint.

A Few Caveats

Every number above is an estimate. Actual usage is also shaped by these factors:

Context length. This is the biggest variable. Open the context from 4K up to 128K and the KV Cache footprint grows several times over, which can break a model that was running fine.
Concurrency. Handling several requests at once means each one needs its own copy of the intermediate state. Usually not a concern for personal use, but you have to redo the math when you build a service.
The specific quantization implementation. Two tools can both call it INT4 and still differ in implementation details, so the sizes come out different.
System overhead. The GPU still has to leave something for the desktop and other programs, so do not plan on using every last byte.

So the calculator gives you a feasibility judgment, not a precise budget. When the verdict is "tight," plan as if it will not run.

Next Step

Now that you know what you can run, the next lesson gets it installed. Two tools — Ollama on the command line and LM Studio with a GUI — and ten minutes to a working setup.