Open Source Special Topic · Running It on Your Own Machine

Getting Started with Ollama and LM Studio

The last lesson worked out what you can run; this one installs it. Two tools, one on the command line and one with a graphical interface, and ten minutes to a working setup the first time.

Pick Your Tool First

Ollama

Command line

One command downloads and runs a model, with no extra steps
Once installed it serves in the background automatically, so programs can call it directly
The interface matches OpenAI's format, so existing API code only needs a new address
No graphical interface — switching models and adjusting parameters both mean typing commands

Right for you if: you want to wire a local model into your own program, or you are at home in a terminal.

LM Studio

Graphical interface

A built-in model browser lets you check size and quantization level before you commit to a download
It tells you whether your machine can handle a model, which helps a lot when you are new
It can start a local server too, also OpenAI-compatible
On Apple silicon it supports the MLX engine, which is faster than the generic format

Right for you if: you want to try a few models and compare them, or you would rather not touch the command line.

No need to agonize over it — install both. The model files they download do not conflict, and plenty of people use LM Studio to pick models and test them, then run Ollama as the always-on service.

Ollama: From Nothing to Running

Once Ollama is installed, one command is all you need. Say the last lesson told you an 8B is within reach:

Download and start chatting right away

ollama run qwen3:8b

That is the whole thing. If it is not on your machine yet it downloads first, then drops you into a chat when it finishes. To download without running, swap run for pull. The other commands you will reach for:

Day-to-day management

# See which models you have downloaded and how much space each one takes
ollama list

# Delete what you no longer use; local models eat a lot of disk
ollama rm qwen3:8b

# See which models are currently holding VRAM
ollama ps

After installation Ollama serves on port 11434 on your machine, with the same interface format as OpenAI. That means the calling code from Part 7 runs as-is once you point base_url here, with the tag name in the model field.

Calling a local model with the standard OpenAI SDK

from openai import OpenAI

# The local server does not verify keys; any non-empty api_key will do
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

resp = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Explain what quantization is in one sentence"}],
)
print(resp.choices[0].message.content)

The 8b in that command is only an example. Which number belongs there depends on your machine. Make a few choices below and you get the exact line to copy.

Pick your setup, hit copy, and this command works the moment Ollama is installed.

Platform

Model

Main use

How to Read a Tag

The string after the colon is not arbitrary. Every segment means something.

qwen3:30b-a3b-q4_K_M

qwen3The model family name.

30b-a3bThe size. An a means MoE; here it is 30B total parameters with 3B activated. As the last lesson explained, plan VRAM for 30B.

q4_K_MThe quantization level. Leave it off and you get the default level, covered in the next section.

Trap one: by default you are already running a quantized version. With no quantization suffix, what Ollama pulls is usually the Q4 level, not the original precision. Plenty of people compare that against an official API, conclude open-source models fall short, and never notice they were not comparing the same thing. Before you compare quality, confirm both sides are running the same level.

Choosing a Quantization Level

Q4_K_M

Pick this by default. The balance point between size and quality, and the default level in almost every tool. Good enough for everyday questions, summarizing, and rewriting.

Q5_K_M

Slightly larger, more consistent quality. If you mainly use it for code or math, this level is the safer bet, since both are fairly sensitive to precision loss.

Q8_0

Close to original quality, and close to double the size. For when VRAM is plentiful and you want the best result available.

Q3 and below

Not recommended unless you genuinely cannot fit anything else. The quality drop starts to show, and you are better off with a model one size smaller at a higher level.

A practical rule: the larger the model, the safer quantization is. Whatever a 32B model gives up going to Q4, it usually still beats an 8B at Q8. So when VRAM is limited, protect the size first and think about the level second.

Available tags change with each release, so before you pull, open that model's tags page at ollama.com/library and confirm which sizes and levels currently exist. Verified 2026-08-07.

LM Studio: Just a Few Clicks

Search the model library

Search a model name and the list shows the size of each quantized version. The interface flags which ones your machine can handle, which is a lot more direct than the command line.

Chat as soon as the download finishes

When loading a model you can adjust context length and GPU allocation. Both directly affect VRAM usage, so get it working on the defaults before you touch them.

Start the local server when a program needs to call it

Launch it from the server panel. This is also an OpenAI-compatible interface, used exactly like the Ollama example above, just on a different port.

Mac users: choose the MLX build

If the model comes in an MLX format, prefer it. That engine is tuned for Apple silicon, and the same model runs noticeably faster than in the generic format.

Three More Common Traps

Opening the context too wide blows out your VRAM. This is the most frequent problem by far. VRAM looks sufficient when the model loads, but pull the context up to tens of thousands of tokens and the growing KV Cache breaks it. What you see is generation freezing halfway through, or speed suddenly dropping to unusable. Get it working at the default context first, then raise it one step at a time when you need long text.
Running out of VRAM does not always produce an error. Some tools quietly push whatever does not fit into system memory and compute it on the CPU. The result runs, but more than ten times slower. If generation speed is absurdly slow, first check whether the model fully fit into VRAM.
Your disk gets eaten. An 8B model at Q4 is three or four GB, so trying a handful of models adds up to tens of GB. Check in with ollama list now and then and delete what you do not use.

Three Routes, and When to Take Each

At this point you have three ways to use a model. They do not replace one another; each has its own place:

Run it locally. When data cannot leave your machine, when you will be calling it heavily over a long period, or when you want to be completely independent of any provider. The cost is that your capability ceiling is set by your hardware.
The official API. When you want the strongest capability and no operations work. Pay as you go, for exactly what you use.
A relay reseller. Convenience and risk come together. The specific trade-offs were covered in "What Are API Relay Resellers"; your data passes through a third party, so it is worth going back to that lesson before you pick one.

That is the end of this chapter. You should now be able to judge for yourself how open a given model is, know where small models come from and what they cost, and get one running on your own machine.