Choose a model (local LLM or cloud provider)¶
Understand ships with several local models and can also use external runtimes (LM Studio, Ollama) or cloud providers. There's no single "best" model — only the best fit for your hardware, your codebase, and any compliance rules you work under.
Open the picker from Data preferences → the ... button next to the AI Provider row.
The picker checks your machine
Understand runs the bundled ullama-recommend when it builds this list, and marks the entry
it recommends for your computer. Models your machine cannot hold are labeled (more memory
than this computer has) — still selectable, in case you know better or are about to upgrade,
but flagged so the choice is deliberate. "Hold" means the model's weights and a working
context together: each model's context memory is costed at its own rate, so a sliding-window
model such as Gemma4-E4B fits on a card that a size-only rule would rule it out of. The
recommendation moves down to a smaller model on a constrained machine; it does not push you to
a larger one just because it fits, because the larger models are several times slower per
overview and overview generation runs in the background by default.
Picking a model also fills in Context Window Size with the context that model can use on this computer — the memory left after its weights, divided by what one token of its context costs. The number stays yours to edit: the field caps at the window the model was trained for, and a value above the estimate is allowed with a warning that it may run slowly or fail. On a machine with a discrete GPU the estimate tracks what the server itself settles on — within a few percent in SciTools' measurements.
Quick recommendation¶
| If you… | Use |
|---|---|
| Want the best overall balance | Granite4.1-3B — the recommended model |
| Want the fastest responses, or have little memory | Qwen3-1.7B |
| Want the most accurate overviews, fast | Qwen3.5-2B |
| Want stronger chat answers and have the hardware | Gemma4-E4B or Qwen3.5-9B |
| Want the best answers a local model can give | Gemma4-12B |
| Have a workstation and generate overviews in bulk | Qwen3.6-35B-A3B |
| Must use US-origin software | Granite4.1-3B (IBM) or any Gemma (Google) |
Bundled local models¶
The model picker offers a range of models, in increasing size. On Windows, the installer downloads the model you pick during installation (Granite by default — Advanced Install chooses another or none). Beyond that, missing models are downloaded on demand (from SciTools' Hugging Face page) the first time you select them.
- Qwen3-1.7B — 1.0 GB. The fastest responses and the smallest footprint. Choose it when memory is tight; the larger models answer better. Alibaba.
- Qwen3.5-2B — 1.2 GB. The most accurate overviews measured, and very fast at generating them in bulk; weaker at chat than the similarly sized models. Alibaba.
- Granite4.1-3B — 2.0 GB. Recommended: the best balance in SciTools' testing — strong chat answers at the lowest latency of the qualifying models, and by far the fastest at generating overviews in bulk. IBM, US-origin, Apache-2.0.
- Gemma4-E2B — 2.9 GB. Accurate overviews on very little graphics memory (only about 1.5 GB of it loads onto the card). It thinks before answering in chat, which makes its chat replies slower. Google, US-origin, Gemma Terms of Use.
- Gemma4-E4B — 4.6 GB. Better chat answers than Granite, but several times slower to respond and much slower at bulk overviews. Google, US-origin, Gemma Terms of Use.
- Qwen3.5-9B — 5.3 GB. Strong chat answers, but slow, and it needs the most graphics memory per token of context of any model here. Alibaba.
- Gemma4-12B — 6.6 GB. The best answers measured, and better than E4B on every axis except memory. Needs a high-end machine: budget roughly 14 GB of VRAM, or an Apple Silicon machine with plenty of unified memory. Google, Gemma Terms of Use.
- Qwen3.6-35B-A3B — 20.8 GB. The most accurate overviews of any model here, and fast for its size because it activates only a fraction of its weights per token; needs a workstation-class GPU or a large Apple Silicon machine. Alibaba.
Answer quality is what separates these models
In SciTools' testing, models differ far more in whether the answer is correct than in whether they can drive Understand's tools — small models investigate the code competently and still get the conclusion wrong. That is why the recommendations weigh answer quality first. Pick the largest model your hardware runs comfortably.
VRAM guidance¶
Larger models need more VRAM. As a rough guide from SciTools' benchmarking, a small model (≈1–2B) fits comfortably in ~4–6 GB, a mid model (≈4B) wants ~8 GB, and an 8B model wants ~11 GB+ on discrete GPUs. Apple unified-memory and integrated GPUs use less but run slower. When in doubt, start small and move up only if you rely heavily on AI explanations.
Size it against your own machine
The bundled ullama-recommend tool reads the machine's memory — and on Windows and Linux the
free memory of a discrete GPU — and reports the largest model size and parameter tier that will
fit, a suggested model, and a recommended total context. Give it a model file and it also reports
the context this machine can run that model with:
ullama-recommend --json
ullama-recommend --model /path/to/model.gguf
Run it from the Understand program directory (see Run the Understand AI server). It accounts for your actual hardware, which a static table cannot. For a sliding-window model the per-token cost it reads from the file is an upper bound, so the context it reports for one is a floor.
The table below shows approximate peak VRAM (unified memory on Apple ARM) that SciTools observed in benchmarking, per model and hardware type. Figures are approximate and hardware-dependent — treat them as a starting point, not a guarantee. Blank cells were not tested on that configuration; not every model was measured on every setup. These runs predate the current model lineup, so the recommended Granite4.1-3B is not listed; its 2.0 GB download puts it in the same class as the other 3–4B models here.
| Model | Nvidia GPU (Windows) | AMD GPU (Windows) | Intel iGPU (Windows) | Nvidia GPU (Linux) | Unified memory (Apple ARM) |
|---|---|---|---|---|---|
| Qwen3-0.6B¹ | ~5 GB | ~5.1 GB | — | — | ~3.7 GB |
| Qwen3-1.7B | ~5.6 GB | ~5.8 GB | ~6 GB | ~5.8 GB | ~3.7 GB |
| Qwen3-4B¹ | — | — | ~8.6 GB | ~8.1 GB | ~4.8 GB |
| Qwen3-8B¹ | — | — | — | ~11.2 GB | ~8.8 GB |
| Granite 4.0-1B¹ | — | — | — | ~4.6 GB | ~2.7 GB |
| Granite 4.0-Micro (3B)¹ | ~6.3 GB | — | ~6.7 GB | ~7.2 GB | ~4.1 GB |
| Llama 3.2-1B¹ | ~3.3 GB | — | ~4.6 GB | — | ~2.2 GB |
| Llama 3.2-3B¹ | — | ~8.1 GB | ~8.7 GB | — | ~5 GB |
| Llama 3.2-8B¹ | — | — | ~11.6 GB | — | ~9.5 GB |
¹ Benchmark-only rows: the in-app model picker offers exactly four models — Qwen3-1.7B, Granite4.1-3B, Gemma4-E4B, and Gemma4-12B. Everything else here was measured for comparison and needs the Custom GGUF route.
Bring your own GGUF¶
In the model picker choose Custom GGUF file and browse to any compatible .gguf model —
useful when your organization has standardized on a specific model. You can also set the context
window size for local models.
The bundled server tracks current llama.cpp, so recent model families load as they are released, and it carries per-model settings for the ones that need them. Families it recognizes today include Qwen3, Qwen3-Coder, Qwen3.5 and Qwen3.6, Gemma 3 and Gemma 4, Granite 4, Phi-4-mini, GPT-OSS, Mistral Small, Devstral, Nemotron, Apriel, and DeepSeek-R1 distills. Anything else falls back to the model's own built-in settings.
Those per-model settings matter
Some models simply do not work without them: Phi-4-mini emits no tool calls under its own chat template, Apriel's template stops the server from starting, and the Qwen3.5 models never stop reasoning on summarization tasks. The server applies the right fix based on the model's filename, so keep the published filename when you copy a GGUF into place — a renamed file falls back to defaults, and the launcher's startup log says when no rule matched.
Use an external runtime or a cloud provider¶
The Provider dropdown offers, besides the bundled local model:
| Provider | Endpoint (default) | API key |
|---|---|---|
| LM Studio | 127.0.0.1:1234 |
not required |
| Ollama | 127.0.0.1:11434 |
not required |
| ChatGPT (OpenAI) | api.openai.com |
required |
| Claude (Anthropic) | api.anthropic.com |
required |
| Gemini (Google) | Google endpoint | required |
| Grok (xAI) | api.x.ai |
required |
| Other | your OpenAI-compatible URL | as configured |
Cloud providers send your prompts off the machine
LM Studio and Ollama still run locally. The cloud providers (ChatGPT, Claude, Gemini, Grok, or a custom endpoint) send prompts — including the code context in them — to that third party. An API key is required, HTTPS is enforced, and you must tick the data-sharing and fee acknowledgements before the setting is accepted. See What can the AI access?, and for a deployment that must guarantee no cloud calls at all, see Local AI with no cloud calls.
For LM Studio and Ollama setup — including serving one model to a whole team from a machine on your network — see Set up a more robust local AI server.