mirror of
https://github.com/ollama/ollama.git
synced 2026-06-03 05:53:55 -04:00
9db4bdbad6
* broad lint fixes to sidestep CI scope glitch * runner: Remove CGO engines, use llama-server exclusively for GGML models Remove the vendored GGML and llama.cpp backend, CGO runner, Go model implementations, and sample. llama-server (built from upstream llama.cpp via FetchContent) is now the sole inference engine for GGUF-based models. (Safetensor based models continue to run on the new MLX engine.) This allows us to more rapidly pick up new capabilities and fixes from llama.cpp as they come out. On windows this now requires recent AMD driver versions to support ROCm v7 as llama.cpp currently does not support building against v6. * llama/compat: load Ollama-format GGUFs in llama-server Squashed from upstream/jmorganca/llama-compat on 2026-04-29. Source tip:0c33775d37. Original source commits: -25223160dllama/compat: add in-memory shim so llama-server can load Ollama-format GGUFs -7449b539allm,server: route Ollama-format gemma3 blobs through llama/compat -436f2e2b1llama/compat: make patch-apply idempotent -8c2c9d4c8llama/compat: extend gemma3 handler to cover 1B and 270M blobs -021389f7bllama/compat: shrink clip.cpp injection from 18 lines to 1 -61b367ec2llama/compat: shrink patch to pure call-site hooks (34 -> 20 lines) -36049361cllama/compat: simplify shim (gemma3-tested) -8fa664865llama/compat: add qwen35moe text handler -db0c74530llama/compat: add qwen35moe vision (clip) support -2a388da77llama/compat: split shared infra into a util TU -9a69a17dcllama/compat: document non-public API dependencies -d0f38a915llama/compat: add gpt-oss and lfm2 handlers -086071822llama/compat: add mistral3 text handler (vision TODO) -63bde9ff7llama/compat: add mistral3 vision (clip) support -3a57b89d5llama/compat: apply LLaMA RoPE permute to mistral3 vision Q/K -99cb87439llama/compat: add qwen35, gemma4, deepseek-ocr handlers -2c7850dballama/compat: add nemotron_h_moe handler (latent FFN + MTP skip) -9e3b54225llama/compat: add llama4 text + clip handlers -034fee349llama/compat: add gemma4 clip handler (gemma4v projector) -9945c5a93server: remove dhiltgen/* compat redirect table -5d4539101llama/compat: rewrite gemma4 tokenizer model to BPE -7e0765327llama/compat: add glm-ocr text handler + text-loader load-op hook -f1bd1a25allama/compat: add glm-ocr clip handler (glm4v projector) -4b5cf3420llama/compat: collapse text-loader hook back to one new patch line -eb4ecf4fcllama/compat: extend gemma4 clip handler to gemma4a (audio) -a23a5e76fllama/compat: fix gemma4a per-block norm tensor mapping -cd2dcaff4llama/compat: add embeddinggemma handler -1ce8a6b26llama/compat: add qwen3-vl + qwen2.5-vl handlers -fd98ffa1ellama/compat: add gemma3n + glm4moelite handlers -cc7bdf0bcllama/compat: handle null buft in maybe_load_tensor -0c33775d3llama/compat: disable mmap when load_op transforms text-side tensors * refine implementation * ci: fix windows MLX build * ci: fix windows llama-server build * ci: fix windows rocm build * ci: windows mlx tuning Shorten long-tail on build, and get OllamaSetup.exe back under 2g limit * ci: fix windows dependencies * win: fix dependency gathering * disable openmp * win: arm64 cross-compile build also DRY out CI steps * scheduler improvements * ci: improvements from #15982 * win: favor ninja for faster developer builds * win: fix build * win: fix arm64 cross-compile * win: avoid spaces in compiler path * misc discovery fixes, and bos handling * lint fixes * win: fix arm cross-compile build/CI bugs * llama.cpp update * win: handle multiple CRT dirs * vulkan: add windows iGPU detection * fix creation bugs for patched models, other refactoring work * tune batch size for better performance * ci and lint fixes * fix repeat_last_n bug * build: revamp build for better developer UX * amd, sampler, qwen3next fixes * version bump * fix mlx build * revamp GPU discovery Scanning the output of llama-server is turning out to be too error prone across llama.cpp updates, so this switches to a thin dynamic library load against the bundled GGML libraries so more details can be gathered from the API. * version bump * missing file * ci: fix cache miss on rocm build * refine vulkan dep handling * fix ps reporting bug on full GPU load * improve cmake wiring for customized local builds * version bump * docker build arg cleanup * improve windows exit error logs * fix community gemma4 support and ci flakes * fix mlx unit test * tighten up ps logic to avoid double counting fit log lines * version bump * fix ps view for full gpu layer offload * add MTP wiring for llama-server and create with GGUFs * pick best template by capabilities * version bump * ci: harden apt repos * remove unused cpu core discovery * adjust batch default logic to reduce OOMs * support larger tool calls * fix audio support, template show * qwen35 mtp patch support * flesh out dtypes * rocm deps * version bump * lint fix * block broken gfx1150 on windows * fix qwen3.5 moe mtp tensors in patch * mmproj oom fallback and vulkan on by default * qwen MTP compat fix * version bump * ci: fix WoA cross-compile * ci: workaround ui tool in cross-compile * version bump * win: enable OpenMP for CPU builds * build: improve developer UX * ci: windows path workaround for CPU build * win: fix WoA dependencies * win: fix large offset reads for mmproj patched loads * version bump * fix vulkan dup detection * add OLLAMA_IGPU_ENABLE and largely disable iGPUs by default * opt-in MTP, win large offset, integraton fixes * fix unit test scheduler interaction hang * fix multi-gpu filtering * version bump * review comments * fix thinking level * fix linux rocm ordering and granite 3.3 template * version bump * ci fix - non-shallow MLX checkout * bypass linux sysfs unit test on windows --------- Co-authored-by: jmorganca <jmorganca@gmail.com>
180 lines
12 KiB
Plaintext
180 lines
12 KiB
Plaintext
---
|
|
title: Hardware support
|
|
---
|
|
|
|
## Nvidia
|
|
Ollama supports Nvidia GPUs with compute capability 5.0+ and driver version 531 and newer.
|
|
Nvidia GPUs with compute capability 5.0 through 6.2 require driver version 570 or newer.
|
|
|
|
Check your compute compatibility to see if your card is supported:
|
|
[https://developer.nvidia.com/cuda-gpus](https://developer.nvidia.com/cuda-gpus)
|
|
|
|
| Compute Capability | Family | Cards |
|
|
| ------------------ | ------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
|
|
| 12.1 | NVIDIA | `GB10 (DGX Spark)` |
|
|
| 12.0 | GeForce RTX 50xx | `RTX 5060` `RTX 5060 Ti` `RTX 5070` `RTX 5070 Ti` `RTX 5080` `RTX 5090` |
|
|
| | NVIDIA Professional | `RTX PRO 4000 Blackwell` `RTX PRO 4500 Blackwell` `RTX PRO 5000 Blackwell` `RTX PRO 6000 Blackwell` |
|
|
| 9.0 | NVIDIA | `H200` `H100` |
|
|
| 8.9 | GeForce RTX 40xx | `RTX 4090` `RTX 4080 SUPER` `RTX 4080` `RTX 4070 Ti SUPER` `RTX 4070 Ti` `RTX 4070 SUPER` `RTX 4070` `RTX 4060 Ti` `RTX 4060` |
|
|
| | NVIDIA Professional | `L4` `L40` `RTX 6000` |
|
|
| 8.6 | GeForce RTX 30xx | `RTX 3090 Ti` `RTX 3090` `RTX 3080 Ti` `RTX 3080` `RTX 3070 Ti` `RTX 3070` `RTX 3060 Ti` `RTX 3060` `RTX 3050 Ti` `RTX 3050` |
|
|
| | NVIDIA Professional | `A40` `RTX A6000` `RTX A5000` `RTX A4000` `RTX A3000` `RTX A2000` `A10` `A16` `A2` |
|
|
| 8.0 | NVIDIA | `A100` `A30` |
|
|
| 7.5 | GeForce GTX/RTX | `GTX 1650 Ti` `TITAN RTX` `RTX 2080 Ti` `RTX 2080` `RTX 2070` `RTX 2060` |
|
|
| | NVIDIA Professional | `T4` `RTX 5000` `RTX 4000` `RTX 3000` `T2000` `T1200` `T1000` `T600` `T500` |
|
|
| | Quadro | `RTX 8000` `RTX 6000` `RTX 5000` `RTX 4000` |
|
|
| 7.0 | NVIDIA | `TITAN V` `V100` `Quadro GV100` |
|
|
| 6.1 | NVIDIA TITAN | `TITAN Xp` `TITAN X` |
|
|
| | GeForce GTX | `GTX 1080 Ti` `GTX 1080` `GTX 1070 Ti` `GTX 1070` `GTX 1060` `GTX 1050 Ti` `GTX 1050` |
|
|
| | Quadro | `P6000` `P5200` `P4200` `P3200` `P5000` `P4000` `P3000` `P2200` `P2000` `P1000` `P620` `P600` `P500` `P520` |
|
|
| | Tesla | `P40` `P4` |
|
|
| 6.0 | NVIDIA | `Tesla P100` `Quadro GP100` |
|
|
| 5.2 | GeForce GTX | `GTX TITAN X` `GTX 980 Ti` `GTX 980` `GTX 970` `GTX 960` `GTX 950` |
|
|
| | Quadro | `M6000 24GB` `M6000` `M5000` `M5500M` `M4000` `M2200` `M2000` `M620` |
|
|
| | Tesla | `M60` `M40` |
|
|
| 5.0 | GeForce GTX | `GTX 750 Ti` `GTX 750` `NVS 810` |
|
|
| | Quadro | `K2200` `K1200` `K620` `M1200` `M520` `M5000M` `M4000M` `M3000M` `M2000M` `M1000M` `K620M` `M600M` `M500M` |
|
|
|
|
For building locally to support older GPUs, see [developer](./development#linux-cuda-nvidia)
|
|
|
|
### GPU Selection
|
|
|
|
If you have multiple NVIDIA GPUs in your system and want to limit Ollama to use
|
|
a subset, you can set `CUDA_VISIBLE_DEVICES` to a comma separated list of GPUs.
|
|
Numeric IDs may be used, however ordering may vary, so UUIDs are more reliable.
|
|
You can discover the UUID of your GPUs by running `nvidia-smi -L` If you want to
|
|
ignore the GPUs and force CPU usage, use an invalid GPU ID (e.g., "-1")
|
|
|
|
### Linux Suspend Resume
|
|
|
|
On linux, after a suspend/resume cycle, sometimes Ollama will fail to discover
|
|
your NVIDIA GPU, and fallback to running on the CPU. You can workaround this
|
|
driver bug by reloading the NVIDIA UVM driver with `sudo rmmod nvidia_uvm &&
|
|
sudo modprobe nvidia_uvm`
|
|
|
|
## AMD Radeon
|
|
|
|
Ollama supports the following AMD GPUs via the ROCm library:
|
|
|
|
> **NOTE:**
|
|
> Additional AMD GPU support is provided by the Vulkan Library - see below.
|
|
|
|
|
|
### Linux Support
|
|
|
|
Ollama requires the AMD ROCm v7 driver on Linux. You can install or upgrade
|
|
using the `amdgpu-install` utility from
|
|
[AMD's ROCm documentation](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/).
|
|
|
|
| Family | Cards and accelerators |
|
|
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
| AMD Radeon RX | `9070 XT` `9070 GRE` `9070` `9060 XT` `9060 XT LP` `9060` `7900 XTX` `7900 XT` `7900 GRE` `7800 XT` `7700 XT` `7700` `7600 XT` `7600` `6950 XT` `6900 XTX` `6900XT` `6800 XT` `6800` `5700 XT` `5700` `5600 XT` `5500 XT` |
|
|
| AMD Radeon AI PRO | `R9700` `R9600D` |
|
|
| AMD Radeon PRO | `W7900` `W7800` `W7700` `W7600` `W7500` `W6900X` `W6800X Duo` `W6800X` `W6800` `V620` |
|
|
| AMD Ryzen AI | `Ryzen AI Max+ 395` `Ryzen AI Max 390` `Ryzen AI Max 385` `Ryzen AI 9 HX 475` `Ryzen AI 9 HX 470` `Ryzen AI 9 465` `Ryzen AI 9 HX 375` `Ryzen AI 9 HX 370` `Ryzen AI 9 365` |
|
|
| AMD Instinct | `MI350X` `MI300X` `MI300A` `MI250X` `MI250` `MI210` `MI100` |
|
|
|
|
### Windows Support
|
|
|
|
Ollama requires an AMD ROCm v7 / HIP7-capable driver stack on Windows.
|
|
|
|
| Family | Cards and accelerators |
|
|
| -------------- | -------------------------------------------------------------------------------------------------------------------- |
|
|
| AMD Radeon RX | `7900 XTX` `7900 XT` `7900 GRE` `7800 XT` `7700 XT` `7600 XT` `7600` `6950 XT` `6900 XTX` `6900XT` `6800 XT` `6800` |
|
|
| AMD Radeon PRO | `W7900` `W7800` `W7700` `W7600` `W7500` `W6900X` `W6800X Duo` `W6800X` `W6800` `V620` |
|
|
|
|
### Overrides on Linux
|
|
|
|
Ollama leverages the AMD ROCm library, which does not support all AMD GPUs. In
|
|
some cases you can force the system to try to use a similar LLVM target that is
|
|
close. For example The Radeon RX 5400 is `gfx1034` (also known as 10.3.4)
|
|
however, ROCm does not currently support this target. The closest support is
|
|
`gfx1030`. You can use the environment variable `HSA_OVERRIDE_GFX_VERSION` with
|
|
`x.y.z` syntax. So for example, to force the system to run on the RX 5400, you
|
|
would set `HSA_OVERRIDE_GFX_VERSION="10.3.0"` as an environment variable for the
|
|
server. If you have an unsupported AMD GPU you can experiment using the list of
|
|
supported types below.
|
|
|
|
If you have multiple GPUs with different GFX versions, append the numeric device
|
|
number to the environment variable to set them individually. For example,
|
|
`HSA_OVERRIDE_GFX_VERSION_0=10.3.0` and `HSA_OVERRIDE_GFX_VERSION_1=11.0.0`
|
|
|
|
At this time, the known supported GPU types on linux are the following LLVM Targets.
|
|
This table shows some example GPUs that map to these LLVM targets:
|
|
| **LLVM Target** | **An Example GPU** |
|
|
|-----------------|---------------------|
|
|
| gfx908 | Radeon Instinct MI100 |
|
|
| gfx90a | Radeon Instinct MI210/MI250 |
|
|
| gfx942 | Radeon Instinct MI300X/MI300A |
|
|
| gfx950 | Radeon Instinct MI350X |
|
|
| gfx1010 | Radeon RX 5700 XT |
|
|
| gfx1012 | Radeon RX 5500 XT |
|
|
| gfx1030 | Radeon PRO V620 |
|
|
| gfx1100 | Radeon PRO W7900 |
|
|
| gfx1101 | Radeon PRO W7700 |
|
|
| gfx1102 | Radeon RX 7600 |
|
|
| gfx1103 | Radeon 780M |
|
|
| gfx1150 | Ryzen AI 9 HX 375 |
|
|
| gfx1151 | Ryzen AI Max+ 395 |
|
|
| gfx1200 | Radeon RX 9070 |
|
|
| gfx1201 | Radeon RX 9070 XT |
|
|
|
|
Reach out on [Discord](https://discord.gg/ollama) or file an
|
|
[issue](https://github.com/ollama/ollama/issues) for additional help.
|
|
|
|
### GPU Selection
|
|
|
|
If you have multiple AMD GPUs in your system and want to limit Ollama to use a
|
|
subset, you can set `ROCR_VISIBLE_DEVICES` to a comma separated list of GPUs.
|
|
You can see the list of devices with `rocminfo`. If you want to ignore the GPUs
|
|
and force CPU usage, use an invalid GPU ID (e.g., "-1"). When available, use the
|
|
`Uuid` to uniquely identify the device instead of numeric value.
|
|
|
|
### Container Permission
|
|
|
|
In some Linux distributions, SELinux can prevent containers from
|
|
accessing the AMD GPU devices. On the host system you can run
|
|
`sudo setsebool container_use_devices=1` to allow containers to use devices.
|
|
|
|
## Metal (Apple GPUs)
|
|
|
|
Ollama supports GPU acceleration on Apple devices via the Metal API.
|
|
|
|
|
|
## Vulkan GPU Support
|
|
|
|
Additional GPU support on Windows and Linux is provided via
|
|
[Vulkan](https://www.vulkan.org/). Vulkan is enabled by default when the
|
|
backend is installed. On Windows most GPU vendors drivers come
|
|
bundled with Vulkan support and require no additional setup steps. Most Linux
|
|
distributions require installing additional components, and you may have
|
|
multiple options for Vulkan drivers between Mesa and GPU Vendor specific packages
|
|
|
|
- Linux Intel GPU Instructions - https://dgpu-docs.intel.com/driver/client/overview.html
|
|
- Linux AMD GPU Instructions - https://amdgpu-install.readthedocs.io/en/latest/install-script.html#specifying-a-vulkan-implementation
|
|
|
|
For AMD GPUs on some Linux distributions, you may need to add the `ollama` user to the `render` group.
|
|
|
|
The Ollama scheduler leverages available VRAM data reported by the GPU libraries to
|
|
make optimal scheduling decisions. Vulkan requires additional capabilities or
|
|
running as root to expose this available VRAM data. If neither root access or this
|
|
capability are granted, Ollama will use approximate sizes of the models
|
|
to make best effort scheduling decisions.
|
|
|
|
```bash
|
|
sudo setcap cap_perfmon+ep /usr/local/bin/ollama
|
|
```
|
|
|
|
### GPU Selection
|
|
|
|
To select specific Vulkan GPU(s), you can set the environment variable
|
|
`GGML_VK_VISIBLE_DEVICES` to one or more numeric IDs on the Ollama server as
|
|
described in the [FAQ](faq#how-do-i-configure-ollama-server). If you
|
|
encounter any problems with Vulkan based GPUs, you can disable all Vulkan GPUs
|
|
by setting `OLLAMA_VULKAN=0` or `GGML_VK_VISIBLE_DEVICES=-1`.
|
|
|
|
On mixed iGPU/dGPU systems where the Vulkan iGPU is unstable, keep Vulkan
|
|
enabled and set `GGML_VK_VISIBLE_DEVICES` to the discrete GPU index. For
|
|
example, use `GGML_VK_VISIBLE_DEVICES=1` when `Vulkan1` is the discrete
|
|
GPU.
|