Compare commits

...
Author SHA1 Message Date
Bruce MacDonald d9ac9664ea no ollama serve command 2026-05-27 14:21:45 -07:00
Bruce MacDonald 3812e8820e docs: omp usage instructions 2026-05-27 14:12:54 -07:00
Patrick Devine f63eea3d27 mlx: fix reported information in ollama show (#16289)
This change updates the show API for MLX models to:
  * display the correct quantization in mixed precision models
  * not display the global_scale scalar value
  * not duplicate the `tools` capability
2026-05-24 14:08:06 -07:00
Anto jones 632ff00798 server: remove duplicate template parsing (#16287) 2026-05-24 13:27:24 -07:00
Jesse Gross 275f122cda mlxrunner: keep gated-delta recurrent state in float32
Split the gated-delta Metal/CUDA kernels' dtype template into separate
input (InT) and state (StT) types so activations can stay in bf16/fp16
while the accumulated delta state stays in float32. Allocate the delta
state and qwen3_5's no-cache zero state in float32 to match.
2026-05-22 09:32:09 -07:00
Jesse Gross 32568531bd create: read draft architecture from its config.json
Previously the draft architecture was hardcoded to
Gemma4AssistantForCausalLM. Read it from the draft model's config so
any draft architecture can be packaged.
2026-05-22 09:32:09 -07:00
Jesse Gross 438fb991e4 mlxrunner: move YaRN RoPE helpers into x/models/nn
Move RopeParameters, BuildYarnRopeFreqs, and ScaleRotaryPart out of
laguna and into x/models/nn so other models can reuse them.
2026-05-22 09:32:09 -07:00
Jesse Gross 358af4af23 Revert "mlxrunner: add DFlash speculative decoding (#16134)"
This reverts commit 98e26b8c37.

The DFlash integration is too invasive to keep at this stage: it
threads DFlash-specific logic through the pipeline, base model
interfaces, and the cache layer. The recurrent cache also now
has qwen3.5 model-specific code. Revert it now and reintroduce
the self-contained, generally-useful pieces (YaRN RoPE DRY-out, draft
architecture autodetection, gated-delta fp32 state) as separate
follow-up commits.
2026-05-22 09:32:09 -07:00
Parth Sareen 91c8e5e1a8 launch: enriched model inventory (#16230) 2026-05-21 11:57:20 -07:00
Daniel Hiltgen 4b2d529966 Reduce startup model hydration (#16215)
* Reduce startup model hydration

Add a lightweight model list cache for tags and launch inventory, while keeping show cache population lazy. This avoids loading every local model at startup on large model stores.

* harden flaky scheduler unit test

* remove extra launch model metadata text

* review comments

* review comments
2026-05-19 15:53:08 -07:00
Bruce MacDonald e6b1d751f2 codex: omit patch tool type (#16231)
Including this value can cause schema compatibility issues. It was removed from codex in a new version.
2026-05-19 13:37:05 -07:00
Eva HandBruce MacDonald 56b319f457 launch: add codex model metadata catalog (#15795)
Co-authored-by: Bruce MacDonald <brucewmacdonald@gmail.com>
2026-05-18 15:26:43 -07:00
Daniel Hiltgen 42e6f56c2a ci: speed up release builds (#15982)
* ci: speed up release builds

This should help speed things up for release.  It also will help
speed up local developer builds a little.

* ci: dedup linux build steps and optimize

* review comments
2026-05-15 14:53:15 -07:00
Daniel Hiltgen da679adcde quiet down kv log spew (#16105) 2026-05-15 13:28:32 -07:00
Parth Sareen b9c0421f03 docs: add codex app (#16163) 2026-05-14 17:53:15 -07:00
Patrick Devine 98e26b8c37 mlxrunner: add DFlash speculative decoding (#16134)
This change adds dflash block diffusion speculative decoding to the MLX runner. Included in this change:

support for qwen3.6 moe/dense speculative decoding
draft model recurrent cache playback
RoPE/YaRN changes (DRY out the laguna/dflash MoE YaRN implementation)
support for greedy sampling / leviathan/chen sampling
2026-05-14 14:02:34 -07:00
Parth Sareen c28ddc0a7b launch: codex app restarts (#16155) 2026-05-14 12:08:13 -07:00
Parth Sareen 3ad2fa3fb5 launch: update codex app UI copy (#16157) 2026-05-14 12:08:08 -07:00
Parth Sareen 6b6f45ef0e docs: hide codex app till launch (#16153) 2026-05-14 10:56:52 -07:00
Patrick Devine 4860130f83 mlx: rework the MLX sampler (#16122)
* mlx: rework the MLX sampler

Replace the MLX sampler transform chain with an explicit distribution pipeline that applies:
  1. penalties
  2. top-k
  3. temperature/softmax
  4. top-p
  5. min-p
  6. normalize
  7. categorical

The common top_k path now keeps sparse [B,K] token ids/probabilities on GPU instead of carrying full-vocab
scores, and sampled MTP reuses those draft/target distributions for acceptance, bonus, and residual sampling.

This change also fixes the seed parameter so that temperature sampling and sampled MTP are reproducible.
2026-05-13 17:18:27 -07:00
Parth Sareen ac7295ccab launch: codex app integration (#16120) 2026-05-13 17:11:52 -07:00
Daniel Hiltgen 6398cd5b78 mlx: add memory trace logging (#16131)
This should help narrow down the root cause of #16030
2026-05-13 13:37:31 -07:00
Eva H 3af1a008e2 launch/opencode: add image modalities for vision models (#15922) 2026-05-12 15:51:46 -04:00
Eva H 6bdb73073b anthropic: Preserve Claude local image-path tool results in renderer-owned prompt formatting (#16047) 2026-05-12 00:02:17 -04:00
Daniel Hiltgen 421faa0263 mlx: fix macOS 26 target leakage in v3 metallib (#16053)
MLX compiles the AIR objects with the requested -mmacosx-version-min, but its final metallib step invokes metal instead of metallib. With the macOS 26 SDK, that can stamp the Metal v3 library with a macOS 26 deployment target.

Relink the generated AIR files with metallib before install until this is fixed upstream.
2026-05-11 16:37:57 -07:00
Daniel Hiltgen 206b049508 mlx: avoid status timeout during inference (#16086)
The MLX runner now routes model work through a locked worker thread. Status also used that worker only to sample memory, so a scheduler health ping could sit behind long prefill or generation until its 10s context expired, causing /v1/status to return 500 and the server to treat the runner as unhealthy.

While Metal doesn't change VRAM reporting, CUDA does. Cache the last memory sample and make status perform only a short best-effort refresh. If the worker is busy, status returns the cached value while a single background refresh continues and updates the cache when the worker becomes available. The in-flight guard and lifecycle context keep this from spawning unbounded refreshes while preserving live VRAM refresh behavior for CUDA.

Fixes #16081
2026-05-11 16:03:38 -07:00
111 changed files with 8237 additions and 1505 deletions

No files matched your search

+156 -53
View File
@@ -141,6 +141,14 @@ jobs:
env:
GOFLAGS: ${{ needs.setup-environment.outputs.GOFLAGS }}
steps:
# Increase pagefile to handle momentary spikes in RAM from NVCC compiles
- if: startsWith(matrix.preset, 'MLX ')
name: Increase pagefile to 200 GB
uses: al-cheb/configure-pagefile-action@v1.5
with:
minimum-size: 16GB
maximum-size: 200GB
disk-root: "D:"
- name: Install system dependencies
run: |
choco install -y --no-progress ccache ninja
@@ -237,8 +245,9 @@ jobs:
Import-Module 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\Tools\Microsoft.VisualStudio.DevShell.dll'
Enter-VsDevShell -VsInstallPath 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise' -SkipAutomaticLocation -DevCmdArguments '-arch=x64 -no_logo'
cmake --preset "${{ matrix.preset }}" ${{ matrix.flags }} --install-prefix "$((pwd).Path)\dist\${{ matrix.os }}-${{ matrix.arch }}"
cmake --build --parallel ([Environment]::ProcessorCount) --preset "${{ matrix.preset }}"
cmake --build --preset "${{ matrix.preset }}" -- -l $([Environment]::ProcessorCount)
cmake --install build --component "${{ startsWith(matrix.preset, 'MLX ') && 'MLX' || startsWith(matrix.preset, 'CUDA ') && 'CUDA' || startsWith(matrix.preset, 'ROCm ') && 'HIP' || startsWith(matrix.preset, 'Vulkan') && 'Vulkan' || 'CPU' }}" --strip
if ('${{ matrix.preset }}'.StartsWith('MLX ')) { cmake --install build --component MLX_VENDOR }
Remove-Item -Path dist\lib\ollama\rocm\rocblas\library\*gfx906* -ErrorAction SilentlyContinue
env:
CMAKE_GENERATOR: Ninja
@@ -380,20 +389,36 @@ jobs:
dist/*.ps1
dist/OllamaSetup.exe
linux-build:
# Pre-build each Dockerfile stage on its own runner in parallel and push the
# resulting layers to a per-stage registry cache. The downstream
# docker-build-push job then assembles cache-hit-only.
linux-depends:
strategy:
matrix:
include:
- os: linux
arch: amd64
target: archive
- os: linux
arch: amd64
target: rocm
- os: linux
arch: arm64
target: archive
runs-on: ${{ matrix.arch == 'arm64' && format('{0}-{1}', matrix.os, matrix.arch) || matrix.os }}
- arch: amd64
target: cpu
- arch: amd64
target: cuda-12
- arch: amd64
target: cuda-13
- arch: amd64
target: mlx
- arch: amd64
target: rocm-7
- arch: amd64
target: vulkan
- arch: arm64
target: cpu
- arch: arm64
target: cuda-12
- arch: arm64
target: cuda-13
- arch: arm64
target: jetpack-5
- arch: arm64
target: jetpack-6
runs-on: ${{ matrix.arch == 'arm64' && 'linux-arm64' || 'linux' }}
environment: release
needs: setup-environment
env:
@@ -401,53 +426,53 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
username: ${{ vars.DOCKER_USER }}
password: ${{ secrets.DOCKER_ACCESS_TOKEN }}
# Increase swap to handle momentary spikes in RAM from NVCC compiles
- if: matrix.target == 'mlx'
name: Increase Linux swap to 200 GB
shell: bash
run: |
set -e
SWAP_PATH=/swapfile-mlx
SWAP_SIZE_GB=200
if [ -f "$SWAP_PATH" ]; then
sudo swapoff "$SWAP_PATH" 2>/dev/null || true
sudo rm -f "$SWAP_PATH"
fi
if ! sudo fallocate -l ${SWAP_SIZE_GB}G "$SWAP_PATH" 2>/dev/null; then
echo "fallocate unsupported, falling back to dd"
sudo dd if=/dev/zero of="$SWAP_PATH" bs=1M count=$((SWAP_SIZE_GB * 1024))
fi
sudo chmod 600 "$SWAP_PATH"
sudo mkswap "$SWAP_PATH"
sudo swapon "$SWAP_PATH"
swapon --show
free -h
- uses: docker/build-push-action@v6
with:
context: .
platforms: ${{ matrix.os }}/${{ matrix.arch }}
platforms: linux/${{ matrix.arch }}
target: ${{ matrix.target }}
provenance: false
sbom: false
build-args: |
GOFLAGS=${{ env.GOFLAGS }}
CGO_CFLAGS=${{ env.CGO_CFLAGS }}
CGO_CXXFLAGS=${{ env.CGO_CXXFLAGS }}
outputs: type=local,dest=dist/${{ matrix.os }}-${{ matrix.arch }}
cache-from: type=registry,ref=${{ vars.DOCKER_REPO }}:latest
cache-to: type=inline
- name: Deduplicate CUDA libraries
run: |
./scripts/deduplicate_cuda_libs.sh dist/${{ matrix.os }}-${{ matrix.arch }}
- run: |
for COMPONENT in bin/* lib/ollama/*; do
case "$COMPONENT" in
bin/ollama*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}.tar.in ;;
lib/ollama/*.so*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}.tar.in ;;
lib/ollama/cuda_v*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}.tar.in ;;
lib/ollama/vulkan*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}.tar.in ;;
lib/ollama/mlx*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}-mlx.tar.in ;;
lib/ollama/include*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}-mlx.tar.in ;;
lib/ollama/cuda_jetpack5) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}-jetpack5.tar.in ;;
lib/ollama/cuda_jetpack6) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}-jetpack6.tar.in ;;
lib/ollama/rocm) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}-rocm.tar.in ;;
esac
done
working-directory: dist/${{ matrix.os }}-${{ matrix.arch }}
- run: |
echo "Manifests"
for ARCHIVE in dist/${{ matrix.os }}-${{ matrix.arch }}/*.tar.in ; do
echo $ARCHIVE
cat $ARCHIVE
done
- run: |
for ARCHIVE in dist/${{ matrix.os }}-${{ matrix.arch }}/*.tar.in; do
tar c -C dist/${{ matrix.os }}-${{ matrix.arch }} -T $ARCHIVE --owner 0 --group 0 | zstd --ultra -22 -T0 >$(basename ${ARCHIVE//.*/}.tar.zst);
done
- uses: actions/upload-artifact@v4
with:
name: bundles-${{ matrix.os }}-${{ matrix.arch }}-${{ matrix.target }}
path: |
*.tar.zst
GOFLAGS=${{ env.GOFLAGS }}
APT_MIRROR=http://azure.archive.ubuntu.com/ubuntu
OLLAMA_MLX_BUILD_JOBS=16
OLLAMA_MLX_NVCC_THREADS=6
cache-from: |
type=registry,ref=ollama/release:cache-${{ matrix.arch }}-${{ matrix.target }}
type=registry,ref=${{ vars.DOCKER_REPO }}:latest
cache-to: type=registry,ref=ollama/release:cache-${{ matrix.arch }}-${{ matrix.target }},mode=max
# Build each Docker variant (OS, arch, and flavor) separately. Using QEMU is unreliable and slower.
# Heavy stages were pre-built by linux-depends; this job is cache-hit-only for those layers
# and just assembles, runs the Go build, and pushes the final image.
docker-build-push:
strategy:
matrix:
@@ -459,6 +484,15 @@ jobs:
CGO_CXXFLAGS
GOFLAGS
APT_MIRROR=http://azure.archive.ubuntu.com/ubuntu
OLLAMA_MLX_BUILD_JOBS=16
OLLAMA_MLX_NVCC_THREADS=6
cache-from: |
type=registry,ref=${{ vars.DOCKER_REPO }}:latest
type=registry,ref=ollama/release:cache-arm64-cpu
type=registry,ref=ollama/release:cache-arm64-cuda-12
type=registry,ref=ollama/release:cache-arm64-cuda-13
type=registry,ref=ollama/release:cache-arm64-jetpack-5
type=registry,ref=ollama/release:cache-arm64-jetpack-6
- os: linux
arch: amd64
build-args: |
@@ -466,6 +500,15 @@ jobs:
CGO_CXXFLAGS
GOFLAGS
APT_MIRROR=http://azure.archive.ubuntu.com/ubuntu
OLLAMA_MLX_BUILD_JOBS=16
OLLAMA_MLX_NVCC_THREADS=6
cache-from: |
type=registry,ref=${{ vars.DOCKER_REPO }}:latest
type=registry,ref=ollama/release:cache-amd64-cpu
type=registry,ref=ollama/release:cache-amd64-cuda-12
type=registry,ref=ollama/release:cache-amd64-cuda-13
type=registry,ref=ollama/release:cache-amd64-mlx
type=registry,ref=ollama/release:cache-amd64-vulkan
- os: linux
arch: amd64
suffix: '-rocm'
@@ -475,9 +518,15 @@ jobs:
GOFLAGS
FLAVOR=rocm
APT_MIRROR=http://azure.archive.ubuntu.com/ubuntu
OLLAMA_MLX_BUILD_JOBS=16
OLLAMA_MLX_NVCC_THREADS=6
cache-from: |
type=registry,ref=${{ vars.DOCKER_REPO }}:latest
type=registry,ref=ollama/release:cache-amd64-cpu
type=registry,ref=ollama/release:cache-amd64-rocm-7
runs-on: ${{ matrix.arch == 'arm64' && format('{0}-{1}', matrix.os, matrix.arch) || matrix.os }}
environment: release
needs: setup-environment
needs: [setup-environment, linux-depends]
env:
GOFLAGS: ${{ needs.setup-environment.outputs.GOFLAGS }}
steps:
@@ -492,9 +541,11 @@ jobs:
with:
context: .
platforms: ${{ matrix.os }}/${{ matrix.arch }}
provenance: false
sbom: false
build-args: ${{ matrix.build-args }}
outputs: type=image,name=${{ vars.DOCKER_REPO }},push-by-digest=true,name-canonical=true,push=true
cache-from: type=registry,ref=${{ vars.DOCKER_REPO }}:latest
cache-from: ${{ matrix.cache-from }}
cache-to: type=inline
- run: |
mkdir -p ${{ matrix.os }}-${{ matrix.arch }}
@@ -505,6 +556,58 @@ jobs:
name: digest-${{ matrix.os }}-${{ matrix.arch }}-${{ matrix.suffix }}
path: |
${{ runner.temp }}/${{ matrix.os }}-${{ matrix.arch }}-${{ matrix.suffix }}.txt
# Re-run buildx with --target archive against buildkit's local cache to
# extract the release directory layout. All upstream stages were just
# built above, so this is a cache-hit-only pass that just writes files.
- uses: docker/build-push-action@v6
with:
context: .
platforms: ${{ matrix.os }}/${{ matrix.arch }}
target: archive
provenance: false
sbom: false
build-args: ${{ matrix.build-args }}
outputs: type=local,dest=dist/${{ matrix.os }}-${{ matrix.arch }}
cache-from: ${{ matrix.cache-from }}
- name: Deduplicate CUDA libraries
run: |
./scripts/deduplicate_cuda_libs.sh dist/${{ matrix.os }}-${{ matrix.arch }}
- run: |
for COMPONENT in bin/* lib/ollama/*; do
case "$COMPONENT" in
bin/ollama*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}.tar.in ;;
lib/ollama/*.so*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}.tar.in ;;
lib/ollama/cuda_v*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}.tar.in ;;
lib/ollama/vulkan*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}.tar.in ;;
lib/ollama/mlx*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}-mlx.tar.in ;;
lib/ollama/include*) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}.tar.in ;;
lib/ollama/cuda_jetpack5) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}-jetpack5.tar.in ;;
lib/ollama/cuda_jetpack6) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}-jetpack6.tar.in ;;
lib/ollama/rocm) echo $COMPONENT >>ollama-${{ matrix.os }}-${{ matrix.arch }}-rocm.tar.in ;;
esac
done
working-directory: dist/${{ matrix.os }}-${{ matrix.arch }}
# rocm builds cpu + rocm libs for the container image, which
# creates a CPU-only amd64 tarball that would collide with the full
# bundle when the release job merges artifacts.
- if: matrix.suffix == '-rocm'
run: rm -f dist/${{ matrix.os }}-${{ matrix.arch }}/ollama-${{ matrix.os }}-${{ matrix.arch }}.tar.in
- run: |
echo "Manifests"
for ARCHIVE in dist/${{ matrix.os }}-${{ matrix.arch }}/*.tar.in ; do
echo $ARCHIVE
cat $ARCHIVE
done
- run: |
for ARCHIVE in dist/${{ matrix.os }}-${{ matrix.arch }}/*.tar.in; do
tar c -C dist/${{ matrix.os }}-${{ matrix.arch }} -T $ARCHIVE --owner 0 --group 0 | zstd -19 -T0 >$(basename ${ARCHIVE//.*/}.tar.zst) &
done
wait
- uses: actions/upload-artifact@v4
with:
name: bundles-${{ matrix.os }}-${{ matrix.arch }}${{ matrix.suffix }}
path: |
*.tar.zst
# Merge Docker images for the same flavor into a single multi-arch manifest
docker-merge-push:
@@ -544,7 +647,7 @@ jobs:
release:
runs-on: ubuntu-latest
environment: release
needs: [darwin-build, windows-app, linux-build]
needs: [darwin-build, windows-app, docker-build-push]
permissions:
contents: write
env:
+4 -4
View File
@@ -65,7 +65,7 @@ jobs:
- preset: 'MLX CUDA 13'
container: nvidia/cuda:13.0.0-devel-ubuntu22.04
extra-packages: libcudnn9-dev-cuda-13 libopenblas-dev liblapack-dev liblapacke-dev git curl
flags: '-DCMAKE_CUDA_ARCHITECTURES=87 -DBLAS_INCLUDE_DIRS=/usr/include/x86_64-linux-gnu -DLAPACK_INCLUDE_DIRS=/usr/include/x86_64-linux-gnu'
flags: '-DCMAKE_CUDA_ARCHITECTURES=87 -DMLX_CUDA_ARCHITECTURES=80-virtual -DBLAS_INCLUDE_DIRS=/usr/include/x86_64-linux-gnu -DLAPACK_INCLUDE_DIRS=/usr/include/x86_64-linux-gnu'
install-go: true
runs-on: linux
container: ${{ matrix.container }}
@@ -105,7 +105,7 @@ jobs:
key: ccache-${{ runner.os }}-${{ runner.arch }}-${{ matrix.preset }}-${{ needs.changes.outputs.vendorsha }}
- run: |
cmake --preset "${{ matrix.preset }}" ${{ matrix.flags }}
cmake --build --preset "${{ matrix.preset }}" --parallel
cmake --build --preset "${{ matrix.preset }}" -- -l $(nproc)
windows:
needs: [changes]
@@ -134,7 +134,7 @@ jobs:
- preset: 'MLX CUDA 13'
install: https://developer.download.nvidia.com/compute/cuda/13.0.0/local_installers/cuda_13.0.0_windows.exe
cudnn-install: https://developer.download.nvidia.com/compute/cudnn/redist/cudnn/windows-x86_64/cudnn-windows-x86_64-9.18.1.3_cuda13-archive.zip
flags: '-DCMAKE_CUDA_ARCHITECTURES=80'
flags: '-DCMAKE_CUDA_ARCHITECTURES=80 -DMLX_CUDA_ARCHITECTURES=80-virtual'
cuda-components:
- '"cudart"'
- '"nvcc"'
@@ -240,7 +240,7 @@ jobs:
Import-Module 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\Tools\Microsoft.VisualStudio.DevShell.dll'
Enter-VsDevShell -VsInstallPath 'C:\Program Files\Microsoft Visual Studio\2022\Enterprise' -SkipAutomaticLocation -DevCmdArguments '-arch=x64 -no_logo'
cmake --preset "${{ matrix.preset }}" ${{ matrix.flags }}
cmake --build --parallel --preset "${{ matrix.preset }}"
cmake --build --preset "${{ matrix.preset }}" -- -l $([Environment]::ProcessorCount)
env:
CMAKE_GENERATOR: Ninja
+10 -5
View File
@@ -228,15 +228,20 @@ if(MLX_ENGINE)
list(APPEND MLX_INCLUDE_REGEXES "^dl\\.dll$")
endif()
# Split mlx/mlxc libraries from runtime deps to avoid stripping deps
install(TARGETS mlx mlxc
RUNTIME_DEPENDENCIES
DIRECTORIES ${MLX_RUNTIME_DIRS}
PRE_INCLUDE_REGEXES ${MLX_INCLUDE_REGEXES}
PRE_EXCLUDE_REGEXES ".*"
RUNTIME_DEPENDENCY_SET mlx_runtime_deps
RUNTIME DESTINATION ${OLLAMA_INSTALL_DIR} COMPONENT MLX
LIBRARY DESTINATION ${OLLAMA_INSTALL_DIR} COMPONENT MLX
FRAMEWORK DESTINATION ${OLLAMA_INSTALL_DIR} COMPONENT MLX
)
install(RUNTIME_DEPENDENCY_SET mlx_runtime_deps
DIRECTORIES ${MLX_RUNTIME_DIRS}
PRE_INCLUDE_REGEXES ${MLX_INCLUDE_REGEXES}
PRE_EXCLUDE_REGEXES ".*"
RUNTIME DESTINATION ${OLLAMA_INSTALL_DIR} COMPONENT MLX_VENDOR
LIBRARY DESTINATION ${OLLAMA_INSTALL_DIR} COMPONENT MLX_VENDOR
)
if(TARGET jaccl)
install(TARGETS jaccl
@@ -366,7 +371,7 @@ if(MLX_ENGINE)
if(MLX_CUDA_LIBS)
install(FILES ${MLX_CUDA_LIBS}
DESTINATION ${OLLAMA_INSTALL_DIR}
COMPONENT MLX)
COMPONENT MLX_VENDOR)
endif()
endif()
endif()
+1 -1
View File
@@ -112,7 +112,7 @@
"name": "MLX CUDA 13",
"inherits": [ "MLX", "CUDA 13" ],
"cacheVariables": {
"MLX_CUDA_ARCHITECTURES": "86;89;90;90a;100;103;75-virtual;80-virtual;110-virtual;120-virtual;121-virtual",
"MLX_CUDA_ARCHITECTURES": "75-virtual;80-virtual;86-virtual;89-virtual;90-virtual;90a-virtual;100-virtual;103-virtual;110-virtual;120-virtual;121-virtual",
"OLLAMA_RUNNER_DIR": "mlx_cuda_v13"
}
}
+7 -3
View File
@@ -144,6 +144,9 @@ RUN --mount=type=cache,target=/root/.ccache \
FROM base AS mlx
ARG CUDA13VERSION=13.0
# OLLAMA_MLX_BUILD_JOBS empty -> ninja gates by load average (-l $(nproc))
ARG OLLAMA_MLX_BUILD_JOBS=
ARG OLLAMA_MLX_NVCC_THREADS=2
RUN dnf install -y cuda-toolkit-${CUDA13VERSION//./-} \
&& dnf install -y openblas-devel lapack-devel \
&& dnf install -y libcudnn9-cuda-13 libcudnn9-devel-cuda-13 \
@@ -170,9 +173,10 @@ RUN --mount=type=cache,target=/root/.ccache \
&& if [ -f /tmp/local-mlx-c/CMakeLists.txt ]; then \
export OLLAMA_MLX_C_SOURCE=/tmp/local-mlx-c; \
fi \
&& cmake --preset 'MLX CUDA 13' -DBLAS_INCLUDE_DIRS=/usr/include/openblas -DLAPACK_INCLUDE_DIRS=/usr/include/openblas \
&& cmake --build --preset 'MLX CUDA 13' -- -l $(nproc) \
&& cmake --install build --component MLX --strip
&& cmake --preset 'MLX CUDA 13' -DBLAS_INCLUDE_DIRS=/usr/include/openblas -DLAPACK_INCLUDE_DIRS=/usr/include/openblas -DCMAKE_CUDA_FLAGS="-t ${OLLAMA_MLX_NVCC_THREADS}" \
&& cmake --build --preset 'MLX CUDA 13' -- -l $(nproc) ${OLLAMA_MLX_BUILD_JOBS:+-j ${OLLAMA_MLX_BUILD_JOBS}} \
&& cmake --install build --component MLX --strip \
&& cmake --install build --component MLX_VENDOR
FROM base AS build
WORKDIR /go/src/github.com/ollama/ollama
+106 -28
View File
@@ -78,6 +78,11 @@ type MessagesRequest struct {
ToolChoice *ToolChoice `json:"tool_choice,omitempty"`
Thinking *ThinkingConfig `json:"thinking,omitempty"`
Metadata *Metadata `json:"metadata,omitempty"`
OutputConfig *OutputConfig `json:"output_config,omitempty"`
}
type OutputConfig struct {
Effort string `json:"effort,omitempty"`
}
// MessageParam represents a message in the request
@@ -161,7 +166,7 @@ type WebSearchToolResultError struct {
// ImageSource represents the source of an image
type ImageSource struct {
Type string `json:"type"` // "base64" or "url"
Type string `json:"type"` // "base64"
MediaType string `json:"media_type,omitempty"`
Data string `json:"data,omitempty"`
URL string `json:"url,omitempty"`
@@ -373,9 +378,26 @@ func FromMessagesRequest(r MessagesRequest) (*api.ChatRequest, error) {
}
var think *api.ThinkValue
normalizedEffort := ""
if r.OutputConfig != nil {
normalizedEffort = strings.ToLower(strings.TrimSpace(r.OutputConfig.Effort))
if normalizedEffort == "xhigh" {
normalizedEffort = "high"
}
}
if r.Thinking != nil && r.Thinking.Type == "enabled" {
think = &api.ThinkValue{Value: true}
}
if r.Thinking != nil && r.Thinking.Type == "disabled" {
think = &api.ThinkValue{Value: false}
}
if think == nil && r.OutputConfig != nil {
switch normalizedEffort {
case "high", "medium", "low", "max":
think = &api.ThinkValue{Value: normalizedEffort}
}
}
stream := r.Stream
convertedRequest := &api.ChatRequest{
@@ -425,17 +447,12 @@ func convertMessage(msg MessageParam) ([]api.Message, error) {
return nil, errors.New("invalid image source")
}
if block.Source.Type == "base64" {
decoded, err := base64.StdEncoding.DecodeString(block.Source.Data)
if err != nil {
logutil.Trace("anthropic: invalid base64 image data", "role", role, "error", err)
return nil, fmt.Errorf("invalid base64 image data: %w", err)
}
images = append(images, decoded)
} else {
logutil.Trace("anthropic: unsupported image source type", "role", role, "source_type", block.Source.Type)
return nil, fmt.Errorf("invalid image source type: %s. Only base64 images are supported.", block.Source.Type)
decoded, err := resolveImageSource(block.Source)
if err != nil {
logutil.Trace("anthropic: unsupported image source", "role", role, "source_type", block.Source.Type, "error", err)
return nil, err
}
images = append(images, decoded)
case "tool_use":
toolUseBlocks++
@@ -457,26 +474,16 @@ func convertMessage(msg MessageParam) ([]api.Message, error) {
case "tool_result":
toolResultBlocks++
var resultContent string
switch c := block.Content.(type) {
case string:
resultContent = c
case []any:
for _, cb := range c {
if cbMap, ok := cb.(map[string]any); ok {
if cbMap["type"] == "text" {
if text, ok := cbMap["text"].(string); ok {
resultContent += text
}
}
}
}
resultContent, resultImages, err := convertToolResultContent(block.Content)
if err != nil {
logutil.Trace("anthropic: invalid tool_result content", "role", role, "error", err)
return nil, err
}
toolResults = append(toolResults, api.Message{
Role: "tool",
Content: resultContent,
Images: resultImages,
ToolCallID: block.ToolUseID,
})
@@ -508,6 +515,10 @@ func convertMessage(msg MessageParam) ([]api.Message, error) {
}
}
if role == "user" && len(toolResults) > 0 {
messages = append(messages, toolResults...)
}
if textContent.Len() > 0 || len(images) > 0 || len(toolCalls) > 0 || thinking != "" {
m := api.Message{
Role: role,
@@ -519,8 +530,10 @@ func convertMessage(msg MessageParam) ([]api.Message, error) {
messages = append(messages, m)
}
// Add tool results as separate messages
messages = append(messages, toolResults...)
// Add tool results as separate messages.
if role != "user" || len(toolResults) == 0 {
messages = append(messages, toolResults...)
}
logutil.Trace("anthropic: converted block message",
"role", role,
"blocks", len(msg.Content),
@@ -969,6 +982,71 @@ func GenerateMessageID() string {
return generateID("msg")
}
func resolveImageSource(source *ImageSource) (api.ImageData, error) {
if source.Type != "base64" {
return nil, fmt.Errorf("invalid image source type: %s. Only base64 images are supported.", source.Type)
}
decoded, err := base64.StdEncoding.DecodeString(source.Data)
if err != nil {
return nil, fmt.Errorf("invalid base64 image data: %w", err)
}
return decoded, nil
}
func convertToolResultContent(content any) (string, []api.ImageData, error) {
switch c := content.(type) {
case nil:
return "", nil, nil
case string:
return c, nil, nil
case []any:
var text strings.Builder
var images []api.ImageData
for _, cb := range c {
cbMap, ok := cb.(map[string]any)
if !ok {
continue
}
switch cbMap["type"] {
case "text":
if t, ok := cbMap["text"].(string); ok {
text.WriteString(t)
}
case "image":
rawSource, ok := cbMap["source"].(map[string]any)
if !ok {
return "", nil, errors.New("invalid tool_result image source")
}
var source ImageSource
if rawType, ok := rawSource["type"].(string); ok {
source.Type = rawType
}
if rawMediaType, ok := rawSource["media_type"].(string); ok {
source.MediaType = rawMediaType
}
if rawData, ok := rawSource["data"].(string); ok {
source.Data = rawData
}
img, err := resolveImageSource(&source)
if err != nil {
return "", nil, err
}
images = append(images, img)
}
}
return text.String(), images, nil
default:
return "", nil, nil
}
}
// ptr returns a pointer to the given string value
func ptr(s string) *string {
return &s
+235
View File
@@ -271,6 +271,241 @@ func TestFromMessagesRequest_WithToolResult(t *testing.T) {
}
}
func TestFromMessagesRequest_WithToolResultImage(t *testing.T) {
imgData, _ := base64.StdEncoding.DecodeString(testImage)
req := MessagesRequest{
Model: "test-model",
MaxTokens: 1024,
Messages: []MessageParam{
{
Role: "user",
Content: []ContentBlock{
{
Type: "tool_result",
ToolUseID: "call_img",
Content: []any{
map[string]any{"type": "text", "text": "Attached image"},
map[string]any{
"type": "image",
"source": map[string]any{
"type": "base64",
"media_type": "image/png",
"data": testImage,
},
},
},
},
},
},
},
}
result, err := FromMessagesRequest(req)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(result.Messages) != 1 {
t.Fatalf("expected 1 message, got %d", len(result.Messages))
}
msg := result.Messages[0]
if msg.Role != "tool" {
t.Errorf("expected role 'tool', got %q", msg.Role)
}
if msg.ToolCallID != "call_img" {
t.Errorf("expected tool_call_id 'call_img', got %q", msg.ToolCallID)
}
if msg.Content != "Attached image" {
t.Errorf("unexpected content: %q", msg.Content)
}
if len(msg.Images) != 1 {
t.Fatalf("expected 1 image, got %d", len(msg.Images))
}
if string(msg.Images[0]) != string(imgData) {
t.Error("image data mismatch")
}
}
func TestFromMessagesRequest_WithToolResultFollowedByUserText(t *testing.T) {
req := MessagesRequest{
Model: "test-model",
MaxTokens: 1024,
Messages: []MessageParam{
{
Role: "assistant",
Content: []ContentBlock{
{
Type: "tool_use",
ID: "call_read",
Name: "Read",
Input: makeArgs("file_path", "/Users/hoyyeva/Desktop/aaa.png"),
},
},
},
{
Role: "user",
Content: []ContentBlock{
{
Type: "tool_result",
ToolUseID: "call_read",
Content: "Read image (311.5KB)",
},
{
Type: "text",
Text: ptr("Please describe it."),
},
},
},
},
}
result, err := FromMessagesRequest(req)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(result.Messages) != 3 {
t.Fatalf("expected 3 messages, got %d", len(result.Messages))
}
if result.Messages[1].Role != "tool" {
t.Fatalf("expected second message to be tool, got %q", result.Messages[1].Role)
}
if result.Messages[1].ToolCallID != "call_read" {
t.Fatalf("expected tool_call_id 'call_read', got %q", result.Messages[1].ToolCallID)
}
if result.Messages[2].Role != "user" {
t.Fatalf("expected third message to be user, got %q", result.Messages[2].Role)
}
if result.Messages[2].Content != "Please describe it." {
t.Fatalf("unexpected user content: %q", result.Messages[2].Content)
}
}
func TestFromMessagesRequest_WithOutputConfigEffort(t *testing.T) {
req := MessagesRequest{
Model: "gemma4",
MaxTokens: 32000,
Messages: []MessageParam{
{
Role: "user",
Content: textContent("Describe the image."),
},
},
OutputConfig: &OutputConfig{
Effort: "high",
},
}
result, err := FromMessagesRequest(req)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if result.Think == nil {
t.Fatal("expected think to be set from output_config.effort")
}
if got := result.Think.String(); got != "high" {
t.Fatalf("expected think level 'high', got %q", got)
}
}
func TestFromMessagesRequest_WithOutputConfigEffortXHighMapsToHigh(t *testing.T) {
req := MessagesRequest{
Model: "gemma4",
MaxTokens: 32000,
Messages: []MessageParam{
{
Role: "user",
Content: textContent("Describe the image."),
},
},
OutputConfig: &OutputConfig{
Effort: "xhigh",
},
}
result, err := FromMessagesRequest(req)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if result.Think == nil {
t.Fatal("expected think to be set from output_config.effort")
}
if got := result.Think.String(); got != "high" {
t.Fatalf("expected think level 'high' for xhigh effort, got %q", got)
}
}
func TestFromMessagesRequest_ThinkingDisabledOverridesOutputConfigEffort(t *testing.T) {
req := MessagesRequest{
Model: "gemma4",
MaxTokens: 32000,
Messages: []MessageParam{
{
Role: "user",
Content: textContent("Describe the image."),
},
},
Thinking: &ThinkingConfig{
Type: "disabled",
},
OutputConfig: &OutputConfig{
Effort: "high",
},
}
result, err := FromMessagesRequest(req)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if result.Think == nil {
t.Fatal("expected think to be set")
}
if got := result.Think.Value; got != false {
t.Fatalf("expected think=false when thinking is disabled, got %v", got)
}
}
func TestFromMessagesRequest_ThinkingAdaptiveUsesOutputConfigEffort(t *testing.T) {
req := MessagesRequest{
Model: "gemma4",
MaxTokens: 32000,
Messages: []MessageParam{
{
Role: "user",
Content: textContent("Describe the image."),
},
},
Thinking: &ThinkingConfig{
Type: "adaptive",
},
OutputConfig: &OutputConfig{
Effort: "high",
},
}
result, err := FromMessagesRequest(req)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if result.Think == nil {
t.Fatal("expected think to be set from output_config.effort")
}
if got := result.Think.String(); got != "high" {
t.Fatalf("expected think level 'high' for adaptive thinking, got %q", got)
}
}
func TestFromMessagesRequest_WithTools(t *testing.T) {
req := MessagesRequest{
Model: "test-model",
+11 -8
View File
@@ -824,14 +824,15 @@ type ProcessResponse struct {
// ListModelResponse is a single model description in [ListResponse].
type ListModelResponse struct {
Name string `json:"name"`
Model string `json:"model"`
RemoteModel string `json:"remote_model,omitempty"`
RemoteHost string `json:"remote_host,omitempty"`
ModifiedAt time.Time `json:"modified_at"`
Size int64 `json:"size"`
Digest string `json:"digest"`
Details ModelDetails `json:"details,omitempty"`
Name string `json:"name"`
Model string `json:"model"`
RemoteModel string `json:"remote_model,omitempty"`
RemoteHost string `json:"remote_host,omitempty"`
ModifiedAt time.Time `json:"modified_at"`
Size int64 `json:"size"`
Digest string `json:"digest"`
Details ModelDetails `json:"details,omitempty"`
Capabilities []model.Capability `json:"capabilities,omitempty"`
}
// ProcessModelResponse is a single model description in [ProcessResponse].
@@ -924,6 +925,8 @@ type ModelDetails struct {
Families []string `json:"families"`
ParameterSize string `json:"parameter_size"`
QuantizationLevel string `json:"quantization_level"`
ContextLength int `json:"context_length,omitempty"`
EmbeddingLength int `json:"embedding_length,omitempty"`
}
// UserResponse provides information about a user.
+10 -9
View File
@@ -1201,15 +1201,16 @@ func (db *database) getSettings() (Settings, error) {
func (db *database) setSettings(s Settings) error {
lastHomeView := strings.ToLower(strings.TrimSpace(s.LastHomeView))
validLaunchView := map[string]struct{}{
"launch": {},
"openclaw": {},
"claude": {},
"hermes": {},
"codex": {},
"copilot": {},
"opencode": {},
"droid": {},
"pi": {},
"launch": {},
"openclaw": {},
"claude": {},
"hermes": {},
"codex": {},
"codex-app": {},
"copilot": {},
"opencode": {},
"droid": {},
"pi": {},
}
if lastHomeView != "chat" {
if _, ok := validLaunchView[lastHomeView]; !ok {
+15
View File
@@ -122,6 +122,21 @@ func TestStore(t *testing.T) {
}
})
t.Run("settings codex app home view is accepted", func(t *testing.T) {
if err := s.SetSettings(Settings{LastHomeView: "codex-app"}); err != nil {
t.Fatal(err)
}
loaded, err := s.Settings()
if err != nil {
t.Fatal(err)
}
if loaded.LastHomeView != "codex-app" {
t.Fatalf("expected codex-app LastHomeView to be preserved, got %q", loaded.LastHomeView)
}
})
t.Run("window size", func(t *testing.T) {
if err := s.SetWindowSize(1024, 768); err != nil {
t.Fatal(err)
Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

+13 -5
View File
@@ -22,11 +22,12 @@ const LAUNCH_COMMANDS: LaunchCommand[] = [
iconClassName: "h-7 w-7",
},
{
id: "openclaw",
name: "OpenClaw",
command: "ollama launch openclaw",
description: "Personal AI with 100+ skills",
icon: "/launch-icons/openclaw.svg",
id: "codex-app",
name: "Codex App",
command: "ollama launch codex-app",
description: "An AI agent you can delegate real work to, by OpenAI",
icon: "/launch-icons/codex-app.png",
iconClassName: "h-full w-full",
},
{
id: "hermes",
@@ -36,6 +37,13 @@ const LAUNCH_COMMANDS: LaunchCommand[] = [
icon: "/launch-icons/hermes-agent.svg",
iconClassName: "h-7 w-7",
},
{
id: "openclaw",
name: "OpenClaw",
command: "ollama launch openclaw",
description: "Personal AI with 100+ skills",
icon: "/launch-icons/openclaw.svg",
},
{
id: "opencode",
name: "OpenCode",
+1 -1
View File
@@ -2231,7 +2231,7 @@ func runLauncherAction(cmd *cobra.Command, action tui.TUIAction, deps launcherDe
func launcherActionExitsLoop(integration string) bool {
switch integration {
case "vscode":
case "codex-app", "vscode":
return true
default:
return false
+1 -1
View File
@@ -249,7 +249,7 @@ func TestRunLauncherAction_GUIAppsExitTUILoop(t *testing.T) {
cmd := &cobra.Command{}
cmd.SetContext(context.Background())
for _, integration := range []string{"vscode"} {
for _, integration := range []string{"codex-app", "vscode"} {
continueLoop, err := runLauncherAction(cmd, tui.TUIAction{Kind: tui.TUIActionLaunchIntegration, Integration: integration}, launcherDeps{
resolveRunModel: unexpectedRunModelResolution(t),
launchIntegration: func(ctx context.Context, req launch.IntegrationLaunchRequest) error {
+1 -1
View File
@@ -44,7 +44,7 @@ func (c *Claude) findPath() (string, error) {
return fallback, nil
}
func (c *Claude) Run(model string, args []string) error {
func (c *Claude) Run(model string, _ []LaunchModel, args []string) error {
claudePath, err := c.findPath()
if err != nil {
return fmt.Errorf("claude is not installed, install from https://code.claude.com/docs/en/quickstart")
+1 -1
View File
@@ -130,7 +130,7 @@ func (c *ClaudeDesktop) SkipModelReadiness() bool {
return true
}
func (c *ClaudeDesktop) Run(_ string, _ []string) error {
func (c *ClaudeDesktop) Run(_ string, _ []LaunchModel, _ []string) error {
return errClaudeDesktopUnsupported()
}
+1 -1
View File
@@ -932,7 +932,7 @@ func TestClaudeDesktopRunReturnsUnsupported(t *testing.T) {
)
for _, args := range [][]string{nil, {"--foo"}} {
err := (&ClaudeDesktop{}).Run("qwen3.5", args)
err := (&ClaudeDesktop{}).Run("qwen3.5", nil, args)
if err == nil {
t.Fatal("expected Run to fail")
}
+4 -4
View File
@@ -16,7 +16,7 @@ type Cline struct{}
func (c *Cline) String() string { return "Cline" }
func (c *Cline) Run(model string, args []string) error {
func (c *Cline) Run(model string, _ []LaunchModel, args []string) error {
if _, err := exec.LookPath("cline"); err != nil {
return fmt.Errorf("cline is not installed, install with: npm install -g cline")
}
@@ -40,7 +40,7 @@ func (c *Cline) Paths() []string {
return nil
}
func (c *Cline) Edit(models []string) error {
func (c *Cline) Edit(models []LaunchModel) error {
if len(models) == 0 {
return nil
}
@@ -66,10 +66,10 @@ func (c *Cline) Edit(models []string) error {
baseURL := envconfig.Host().String()
config["ollamaBaseUrl"] = baseURL
config["actModeApiProvider"] = "ollama"
config["actModeOllamaModelId"] = models[0]
config["actModeOllamaModelId"] = models[0].Name
config["actModeOllamaBaseUrl"] = baseURL
config["planModeApiProvider"] = "ollama"
config["planModeOllamaModelId"] = models[0]
config["planModeOllamaModelId"] = models[0].Name
config["planModeOllamaBaseUrl"] = baseURL
config["welcomeViewCompleted"] = true
+5 -5
View File
@@ -43,7 +43,7 @@ func TestClineEdit(t *testing.T) {
t.Run("creates config from scratch", func(t *testing.T) {
os.RemoveAll(filepath.Join(tmpDir, ".cline"))
if err := c.Edit([]string{"kimi-k2.5:cloud"}); err != nil {
if err := c.Edit(testLaunchModels("kimi-k2.5:cloud")); err != nil {
t.Fatal(err)
}
@@ -77,7 +77,7 @@ func TestClineEdit(t *testing.T) {
data, _ := json.Marshal(existing)
os.WriteFile(configPath, data, 0o644)
if err := c.Edit([]string{"glm-5:cloud"}); err != nil {
if err := c.Edit(testLaunchModels("glm-5:cloud")); err != nil {
t.Fatal(err)
}
@@ -93,10 +93,10 @@ func TestClineEdit(t *testing.T) {
t.Run("updates model on re-edit", func(t *testing.T) {
os.RemoveAll(filepath.Join(tmpDir, ".cline"))
if err := c.Edit([]string{"kimi-k2.5:cloud"}); err != nil {
if err := c.Edit(testLaunchModels("kimi-k2.5:cloud")); err != nil {
t.Fatal(err)
}
if err := c.Edit([]string{"glm-5:cloud"}); err != nil {
if err := c.Edit(testLaunchModels("glm-5:cloud")); err != nil {
t.Fatal(err)
}
@@ -124,7 +124,7 @@ func TestClineEdit(t *testing.T) {
t.Run("uses first model as primary", func(t *testing.T) {
os.RemoveAll(filepath.Join(tmpDir, ".cline"))
if err := c.Edit([]string{"kimi-k2.5:cloud", "glm-5:cloud"}); err != nil {
if err := c.Edit(testLaunchModels("kimi-k2.5:cloud", "glm-5:cloud")); err != nil {
t.Fatal(err)
}
+571 -48
View File
@@ -1,13 +1,17 @@
package launch
import (
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"github.com/ollama/ollama/cmd/internal/fileutil"
"github.com/ollama/ollama/envconfig"
"github.com/ollama/ollama/types/model"
"github.com/pelletier/go-toml/v2"
"golang.org/x/mod/semver"
)
@@ -16,10 +20,22 @@ type Codex struct{}
func (c *Codex) String() string { return "Codex" }
const codexProfileName = "ollama-launch"
const (
codexProfileName = "ollama-launch"
codexProviderName = "Ollama"
codexFallbackContextWindow = 128_000
func (c *Codex) args(model string, extra []string) []string {
codexRootProfileKey = "profile"
codexRootModelKey = "model"
codexRootModelProviderKey = "model_provider"
codexRootModelCatalogJSONKey = "model_catalog_json"
)
func (c *Codex) args(model, modelCatalogPath string, extra []string) []string {
args := []string{"--profile", codexProfileName}
if modelCatalogPath != "" {
args = append(args, "-c", fmt.Sprintf("%s=%q", codexRootModelCatalogJSONKey, modelCatalogPath))
}
if model != "" {
args = append(args, "-m", model)
}
@@ -27,16 +43,21 @@ func (c *Codex) args(model string, extra []string) []string {
return args
}
func (c *Codex) Run(model string, args []string) error {
func (c *Codex) Run(model string, models []LaunchModel, args []string) error {
if err := checkCodexVersion(); err != nil {
return err
}
if err := ensureCodexConfig(); err != nil {
if err := ensureCodexConfig(model, models); err != nil {
return fmt.Errorf("failed to configure codex: %w", err)
}
cmd := exec.Command("codex", c.args(model, args)...)
catalogPath, err := codexModelCatalogPath()
if err != nil {
return fmt.Errorf("failed to configure codex: %w", err)
}
cmd := exec.Command("codex", c.args(model, catalogPath, args)...)
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
@@ -46,79 +67,581 @@ func (c *Codex) Run(model string, args []string) error {
return cmd.Run()
}
// ensureCodexConfig writes a [profiles.ollama-launch] section to ~/.codex/config.toml
// with openai_base_url pointing to the local Ollama server.
func ensureCodexConfig() error {
home, err := os.UserHomeDir()
// ensureCodexConfig writes a Codex profile and model catalog so Codex uses the
// local Ollama server and has model metadata available.
func ensureCodexConfig(modelName string, models []LaunchModel) error {
configPath, err := codexConfigPath()
if err != nil {
return err
}
codexDir := filepath.Join(home, ".codex")
codexDir := filepath.Dir(configPath)
if err := os.MkdirAll(codexDir, 0o755); err != nil {
return err
}
configPath := filepath.Join(codexDir, "config.toml")
return writeCodexProfile(configPath)
catalogPath := codexModelCatalogPathForConfig(configPath)
if err := writeCodexModelCatalog(catalogPath, codexCatalogModel(modelName, models)); err != nil {
return err
}
return writeCodexProfile(configPath, catalogPath)
}
func codexConfigPath() (string, error) {
home, err := os.UserHomeDir()
if err != nil {
return "", err
}
return filepath.Join(home, ".codex", "config.toml"), nil
}
func codexModelCatalogPath() (string, error) {
configPath, err := codexConfigPath()
if err != nil {
return "", err
}
return codexModelCatalogPathForConfig(configPath), nil
}
func codexModelCatalogPathForConfig(configPath string) string {
return filepath.Join(filepath.Dir(configPath), "model.json")
}
// writeCodexProfile ensures ~/.codex/config.toml has the ollama-launch profile
// and model provider sections with the correct base URL.
func writeCodexProfile(configPath string) error {
baseURL := envconfig.Host().String() + "/v1/"
func writeCodexProfile(configPath string, modelCatalogPath ...string) error {
opts := codexLaunchProfileOptions{
forceAPIAuth: true,
}
if len(modelCatalogPath) > 0 {
opts.modelCatalogPath = modelCatalogPath[0]
}
return writeCodexLaunchProfile(configPath, opts)
}
type codexLaunchProfileOptions struct {
activate bool
profileName string
forceAPIAuth bool
setRootModelConfig bool
model string
modelCatalogPath string
backupIntegration string
}
func writeCodexLaunchProfile(configPath string, opts codexLaunchProfileOptions) error {
baseURL := codexBaseURL()
profileName := codexLaunchProfileName(opts)
profileHeader := codexProfileHeaderFor(profileName)
providerHeader := codexProviderHeaderFor(profileName)
content, readErr := os.ReadFile(configPath)
text := ""
if readErr == nil {
text = string(content)
} else if !os.IsNotExist(readErr) {
return readErr
}
parsed, err := codexParseConfig(text)
if err != nil {
return err
}
model := strings.TrimSpace(opts.model)
if model == "" {
model = parsed.ProfileString(profileName, codexRootModelKey)
}
modelCatalogPath := strings.TrimSpace(opts.modelCatalogPath)
if modelCatalogPath == "" {
modelCatalogPath = parsed.ProfileString(profileName, codexRootModelCatalogJSONKey)
}
profileLines := []string{}
if model != "" {
profileLines = append(profileLines, fmt.Sprintf("%s = %q", codexRootModelKey, model))
}
profileLines = append(profileLines,
fmt.Sprintf("openai_base_url = %q", baseURL),
fmt.Sprintf("%s = %q", codexRootModelProviderKey, profileName),
)
if opts.forceAPIAuth {
profileLines = append(profileLines, `forced_login_method = "api"`)
}
if modelCatalogPath != "" {
profileLines = append(profileLines, fmt.Sprintf("%s = %q", codexRootModelCatalogJSONKey, modelCatalogPath))
}
sections := []struct {
header string
lines []string
}{
{
header: fmt.Sprintf("[profiles.%s]", codexProfileName),
lines: []string{
fmt.Sprintf("openai_base_url = %q", baseURL),
`forced_login_method = "api"`,
fmt.Sprintf("model_provider = %q", codexProfileName),
},
header: profileHeader,
lines: profileLines,
},
{
header: fmt.Sprintf("[model_providers.%s]", codexProfileName),
header: providerHeader,
lines: []string{
`name = "Ollama"`,
fmt.Sprintf("name = %q", codexProviderName),
fmt.Sprintf("base_url = %q", baseURL),
`wire_api = "responses"`,
},
},
}
content, readErr := os.ReadFile(configPath)
text := ""
if readErr == nil {
text = string(content)
if opts.activate {
text = codexSetRootStringValue(text, codexRootProfileKey, profileName)
}
for _, s := range sections {
block := strings.Join(append([]string{s.header}, s.lines...), "\n") + "\n"
if idx := strings.Index(text, s.header); idx >= 0 {
// Replace the existing section up to the next section header.
rest := text[idx+len(s.header):]
if endIdx := strings.Index(rest, "\n["); endIdx >= 0 {
text = text[:idx] + block + rest[endIdx+1:]
} else {
text = text[:idx] + block
}
} else {
// Append the section.
if text != "" && !strings.HasSuffix(text, "\n") {
text += "\n"
}
if text != "" {
text += "\n"
}
text += block
if opts.setRootModelConfig {
if model != "" {
text = codexSetRootStringValue(text, codexRootModelKey, model)
}
text = codexSetRootStringValue(text, codexRootModelProviderKey, profileName)
if modelCatalogPath != "" {
text = codexSetRootStringValue(text, codexRootModelCatalogJSONKey, modelCatalogPath)
}
}
return os.WriteFile(configPath, []byte(text), 0o644)
for _, s := range sections {
text = codexUpsertSection(text, s.header, s.lines)
}
parsed, err = codexParseConfig(text)
if err != nil {
return err
}
if err := codexValidateLaunchProfileText(parsed, profileName, opts, model, modelCatalogPath, baseURL); err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(configPath), 0o755); err != nil {
return err
}
return fileutil.WriteWithBackup(configPath, []byte(text), opts.backupIntegration)
}
func codexLaunchProfileName(opts codexLaunchProfileOptions) string {
if name := strings.TrimSpace(opts.profileName); name != "" {
return name
}
return codexProfileName
}
func codexBaseURL() string {
return strings.TrimRight(envconfig.ConnectableHost().String(), "/") + "/v1/"
}
func codexProfileHeader() string {
return codexProfileHeaderFor(codexProfileName)
}
func codexProviderHeader() string {
return codexProviderHeaderFor(codexProfileName)
}
func codexProfileHeaderFor(profileName string) string {
return fmt.Sprintf("[profiles.%s]", profileName)
}
func codexProviderHeaderFor(profileName string) string {
return fmt.Sprintf("[model_providers.%s]", profileName)
}
func codexValidateLaunchProfileText(config codexParsedConfig, profileName string, opts codexLaunchProfileOptions, model, modelCatalogPath, baseURL string) error {
for _, check := range []struct {
path []string
want string
}{
{[]string{"profiles", profileName, "openai_base_url"}, baseURL},
{[]string{"profiles", profileName, codexRootModelProviderKey}, profileName},
{[]string{"model_providers", profileName, "name"}, codexProviderName},
{[]string{"model_providers", profileName, "base_url"}, baseURL},
{[]string{"model_providers", profileName, "wire_api"}, "responses"},
} {
if got, ok := config.String(check.path...); !ok || got != check.want {
return fmt.Errorf("generated Codex config missing %s = %q", strings.Join(check.path, "."), check.want)
}
}
if opts.forceAPIAuth {
if got, ok := config.String("profiles", profileName, "forced_login_method"); !ok || got != "api" {
return fmt.Errorf("generated Codex config missing profiles.%s.forced_login_method = %q", profileName, "api")
}
}
if model != "" {
if got, ok := config.String("profiles", profileName, codexRootModelKey); !ok || got != model {
return fmt.Errorf("generated Codex config missing profiles.%s.model = %q", profileName, model)
}
}
if modelCatalogPath != "" {
if got, ok := config.String("profiles", profileName, codexRootModelCatalogJSONKey); !ok || got != modelCatalogPath {
return fmt.Errorf("generated Codex config missing profiles.%s.model_catalog_json = %q", profileName, modelCatalogPath)
}
}
if opts.activate {
if got := config.RootString(codexRootProfileKey); got != profileName {
return fmt.Errorf("generated Codex config missing profile = %q", profileName)
}
}
if opts.setRootModelConfig {
if model != "" {
if got := config.RootString(codexRootModelKey); got != model {
return fmt.Errorf("generated Codex config missing model = %q", model)
}
}
if got := config.RootString(codexRootModelProviderKey); got != profileName {
return fmt.Errorf("generated Codex config missing model_provider = %q", profileName)
}
if modelCatalogPath != "" {
if got := config.RootString(codexRootModelCatalogJSONKey); got != modelCatalogPath {
return fmt.Errorf("generated Codex config missing model_catalog_json = %q", modelCatalogPath)
}
}
}
return nil
}
func codexUpsertSection(text, header string, lines []string) string {
block := strings.Join(append([]string{header}, lines...), "\n") + "\n"
if targetPath, ok := codexTableHeaderPath(header); ok {
if start, end, found := codexSectionRange(text, targetPath); found {
return text[:start] + block + text[end:]
}
}
if text != "" && !strings.HasSuffix(text, "\n") {
text += "\n"
}
if text != "" {
text += "\n"
}
return text + block
}
func codexRemoveSection(text, header string) string {
targetPath, ok := codexTableHeaderPath(header)
if !ok {
return text
}
start, end, found := codexSectionRange(text, targetPath)
if !found {
return text
}
return text[:start] + text[end:]
}
type codexParsedConfig struct {
values map[string]any
}
func (c codexParsedConfig) String(path ...string) (string, bool) {
if len(path) == 0 {
return "", false
}
var current any = c.values
for _, part := range path {
table, ok := current.(map[string]any)
if !ok {
return "", false
}
current, ok = table[part]
if !ok {
return "", false
}
}
value, ok := current.(string)
if !ok {
return "", false
}
return value, true
}
func (c codexParsedConfig) RootString(key string) string {
value, _ := c.RootStringOK(key)
return value
}
func (c codexParsedConfig) RootStringOK(key string) (string, bool) {
return c.String(key)
}
func (c codexParsedConfig) ProfileString(profileName, key string) string {
value, _ := c.String("profiles", profileName, key)
return value
}
func (c codexParsedConfig) ProviderString(profileName, key string) string {
value, _ := c.String("model_providers", profileName, key)
return value
}
func codexRootStringValue(text, key string) string {
config, err := codexParseConfig(text)
if err != nil {
return ""
}
return config.RootString(key)
}
func codexRootStringValueOK(text, key string) (string, bool) {
config, err := codexParseConfig(text)
if err != nil {
return "", false
}
return config.RootStringOK(key)
}
func codexStringValue(text string, path ...string) (string, bool) {
config, err := codexParseConfig(text)
if err != nil {
return "", false
}
return config.String(path...)
}
func codexSectionStringValue(text, header, key string) string {
path, ok := codexTableHeaderPath(header)
if !ok {
return ""
}
value, _ := codexStringValue(text, append(path, key)...)
return value
}
func codexParseConfig(text string) (codexParsedConfig, error) {
values, err := codexParseConfigText(text)
if err != nil {
return codexParsedConfig{}, err
}
return codexParsedConfig{values: values}, nil
}
func codexParseConfigText(text string) (map[string]any, error) {
cfg := map[string]any{}
if strings.TrimSpace(text) == "" {
return cfg, nil
}
if err := toml.Unmarshal([]byte(text), &cfg); err != nil {
return nil, fmt.Errorf("invalid Codex config TOML: %w", err)
}
return cfg, nil
}
func codexValidateConfigText(text string) error {
_, err := codexParseConfig(text)
return err
}
func codexSectionRange(text string, targetPath []string) (int, int, bool) {
lines := strings.SplitAfter(text, "\n")
offset := 0
start := -1
for _, line := range lines {
trimmed := strings.TrimSpace(line)
if !strings.HasPrefix(trimmed, "[") || strings.HasPrefix(trimmed, "#") {
offset += len(line)
continue
}
if start >= 0 {
return start, offset, true
}
if path, ok := codexTableHeaderPath(trimmed); ok && codexSamePath(path, targetPath) {
start = offset
}
offset += len(line)
}
if start >= 0 {
return start, len(text), true
}
return 0, 0, false
}
func codexTableHeaderPath(header string) ([]string, bool) {
trimmed := strings.TrimSpace(header)
if !strings.HasPrefix(trimmed, "[") || strings.HasPrefix(trimmed, "[[") {
return nil, false
}
const probeKey = "__ollama_launch_probe"
cfg := map[string]any{}
if err := toml.Unmarshal([]byte(trimmed+"\n"+probeKey+" = true\n"), &cfg); err != nil {
return nil, false
}
return codexFindProbePath(cfg, probeKey, nil)
}
func codexFindProbePath(value any, probeKey string, path []string) ([]string, bool) {
table, ok := value.(map[string]any)
if !ok {
return nil, false
}
if probe, ok := table[probeKey].(bool); ok && probe {
return path, true
}
for key, child := range table {
if key == probeKey {
continue
}
if childPath, ok := codexFindProbePath(child, probeKey, append(path, key)); ok {
return childPath, true
}
}
return nil, false
}
func codexSamePath(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
func codexSetRootStringValue(text, key, value string) string {
lines := strings.SplitAfter(text, "\n")
rootEnd := len(lines)
for i, line := range lines {
if strings.HasPrefix(strings.TrimSpace(line), "[") {
rootEnd = i
break
}
}
assignment := fmt.Sprintf("%s = %q", key, value)
for i := range rootEnd {
line := lines[i]
trimmed := strings.TrimSpace(line)
if trimmed == "" || strings.HasPrefix(trimmed, "#") {
continue
}
if codexRootLineHasKey(trimmed, key) {
if strings.HasSuffix(line, "\n") {
lines[i] = assignment + "\n"
} else {
lines[i] = assignment
}
return strings.Join(lines, "")
}
}
insert := assignment + "\n"
root := strings.Join(lines[:rootEnd], "")
rest := strings.Join(lines[rootEnd:], "")
if root != "" && !strings.HasSuffix(root, "\n") {
root += "\n"
}
if rest != "" && !strings.HasSuffix(insert, "\n\n") {
insert += "\n"
}
return root + insert + rest
}
func codexRemoveRootValue(text, key string) string {
lines := strings.SplitAfter(text, "\n")
rootEnd := len(lines)
for i, line := range lines {
if strings.HasPrefix(strings.TrimSpace(line), "[") {
rootEnd = i
break
}
}
out := make([]string, 0, len(lines))
for i, line := range lines {
if i < rootEnd {
trimmed := strings.TrimSpace(line)
if trimmed != "" && !strings.HasPrefix(trimmed, "#") && codexRootLineHasKey(trimmed, key) {
continue
}
}
out = append(out, line)
}
return strings.Join(out, "")
}
func codexRootLineHasKey(line, key string) bool {
cfg := map[string]any{}
if err := toml.Unmarshal([]byte(line+"\n"), &cfg); err != nil {
return false
}
_, ok := cfg[key]
return ok
}
func codexCatalogModel(modelName string, models []LaunchModel) LaunchModel {
if model, ok := findLaunchModel(models, modelName); ok {
return model.WithCloudLimits()
}
return fallbackLaunchModel(modelName)
}
func writeCodexModelCatalog(catalogPath string, model LaunchModel) error {
entry := buildCodexModelEntry(model)
catalog := map[string]any{
"models": []any{entry},
}
data, err := json.MarshalIndent(catalog, "", " ")
if err != nil {
return err
}
return os.WriteFile(catalogPath, data, 0o644)
}
func buildCodexModelEntry(launchModel LaunchModel) map[string]any {
modelName := launchModel.Name
contextWindow := codexFallbackContextWindow
systemPrompt := ""
if launchModel.ContextLength > 0 {
contextWindow = launchModel.ContextLength
} else if launchModel.Details.ContextLength > 0 {
contextWindow = launchModel.Details.ContextLength
}
if l, ok := lookupCloudModelLimit(modelName); ok {
contextWindow = l.Context
}
if !isCloudModelName(modelName) && launchModel.Details.Format != "safetensors" {
if ctxLen := envconfig.ContextLength(); ctxLen > 0 {
contextWindow = int(ctxLen)
}
}
modalities := []string{"text"}
if launchModel.HasCapability(model.CapabilityVision) {
modalities = append(modalities, "image")
}
truncationMode := "bytes"
if isCloudModelName(modelName) {
truncationMode = "tokens"
}
return map[string]any{
"slug": modelName,
"display_name": modelName,
"context_window": contextWindow,
"shell_type": "default",
"visibility": "list",
"supported_in_api": true,
"priority": 0,
"truncation_policy": map[string]any{"mode": truncationMode, "limit": 10000},
"input_modalities": modalities,
"base_instructions": systemPrompt,
"support_verbosity": true,
"default_verbosity": "low",
"supports_parallel_tool_calls": false,
"supports_reasoning_summaries": false,
"supported_reasoning_levels": []any{},
"experimental_supported_tools": []any{},
}
}
func checkCodexVersion() error {
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+380 -14
View File
@@ -1,15 +1,23 @@
package launch
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"github.com/ollama/ollama/api"
"github.com/ollama/ollama/cmd/internal/fileutil"
modelpkg "github.com/ollama/ollama/types/model"
)
func TestCodexArgs(t *testing.T) {
c := &Codex{}
catalogPath := filepath.Join("tmp", "model.json")
catalogArg := fmt.Sprintf("%s=%q", codexRootModelCatalogJSONKey, catalogPath)
tests := []struct {
name string
@@ -17,15 +25,15 @@ func TestCodexArgs(t *testing.T) {
args []string
want []string
}{
{"with model", "llama3.2", nil, []string{"--profile", "ollama-launch", "-m", "llama3.2"}},
{"empty model", "", nil, []string{"--profile", "ollama-launch"}},
{"with model and extra args", "qwen3.5", []string{"-p", "myprofile"}, []string{"--profile", "ollama-launch", "-m", "qwen3.5", "-p", "myprofile"}},
{"with sandbox flag", "llama3.2", []string{"--sandbox", "workspace-write"}, []string{"--profile", "ollama-launch", "-m", "llama3.2", "--sandbox", "workspace-write"}},
{"with model", "llama3.2", nil, []string{"--profile", "ollama-launch", "-c", catalogArg, "-m", "llama3.2"}},
{"empty model", "", nil, []string{"--profile", "ollama-launch", "-c", catalogArg}},
{"with model and extra args", "qwen3.5", []string{"-p", "myprofile"}, []string{"--profile", "ollama-launch", "-c", catalogArg, "-m", "qwen3.5", "-p", "myprofile"}},
{"with sandbox flag", "llama3.2", []string{"--sandbox", "workspace-write"}, []string{"--profile", "ollama-launch", "-c", catalogArg, "-m", "llama3.2", "--sandbox", "workspace-write"}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := c.args(tt.model, tt.args)
got := c.args(tt.model, catalogPath, tt.args)
if !slices.Equal(got, tt.want) {
t.Errorf("args(%q, %v) = %v, want %v", tt.model, tt.args, got, tt.want)
}
@@ -37,8 +45,9 @@ func TestWriteCodexProfile(t *testing.T) {
t.Run("creates new file when none exists", func(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")
catalogPath := filepath.Join(tmpDir, "model.json")
if err := writeCodexProfile(configPath); err != nil {
if err := writeCodexProfile(configPath, catalogPath); err != nil {
t.Fatal(err)
}
@@ -63,21 +72,28 @@ func TestWriteCodexProfile(t *testing.T) {
if !strings.Contains(content, `model_provider = "ollama-launch"`) {
t.Error("missing model_provider key")
}
if !strings.Contains(content, fmt.Sprintf("model_catalog_json = %q", catalogPath)) {
t.Error("missing model_catalog_json key")
}
if !strings.Contains(content, "[model_providers.ollama-launch]") {
t.Error("missing [model_providers.ollama-launch] section")
}
if !strings.Contains(content, `name = "Ollama"`) {
t.Error("missing model provider name")
}
if err := codexValidateConfigText(content); err != nil {
t.Fatalf("generated config should be valid TOML: %v\n%s", err, content)
}
})
t.Run("appends profile to existing file without profile", func(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")
catalogPath := filepath.Join(tmpDir, "model.json")
existing := "[some_other_section]\nkey = \"value\"\n"
os.WriteFile(configPath, []byte(existing), 0o644)
if err := writeCodexProfile(configPath); err != nil {
if err := writeCodexProfile(configPath, catalogPath); err != nil {
t.Fatal(err)
}
@@ -95,10 +111,11 @@ func TestWriteCodexProfile(t *testing.T) {
t.Run("replaces existing profile section", func(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")
catalogPath := filepath.Join(tmpDir, "model.json")
existing := "[profiles.ollama-launch]\nopenai_base_url = \"http://old:1234/v1/\"\n\n[model_providers.ollama-launch]\nname = \"Ollama\"\nbase_url = \"http://old:1234/v1/\"\n"
os.WriteFile(configPath, []byte(existing), 0o644)
if err := writeCodexProfile(configPath); err != nil {
if err := writeCodexProfile(configPath, catalogPath); err != nil {
t.Fatal(err)
}
@@ -114,15 +131,160 @@ func TestWriteCodexProfile(t *testing.T) {
if strings.Count(content, "[model_providers.ollama-launch]") != 1 {
t.Errorf("expected exactly one [model_providers.ollama-launch] section, got %d", strings.Count(content, "[model_providers.ollama-launch]"))
}
if err := codexValidateConfigText(content); err != nil {
t.Fatalf("generated config should be valid TOML: %v\n%s", err, content)
}
})
t.Run("replaces equivalent quoted profile table", func(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")
existing := "" +
`profile = "default"` + "\n\n" +
`[profiles."ollama-launch"]` + "\n" +
`openai_base_url = "http://old:1234/v1/"` + "\n\n" +
`[model_providers."ollama-launch"]` + "\n" +
`name = "Old"` + "\n" +
`base_url = "http://old:1234/v1/"` + "\n\n" +
`[profiles.default]` + "\n" +
`model = "gpt-5.5"` + "\n"
os.WriteFile(configPath, []byte(existing), 0o644)
if err := writeCodexProfile(configPath); err != nil {
t.Fatal(err)
}
data, _ := os.ReadFile(configPath)
content := string(data)
if strings.Contains(content, `profiles."ollama-launch"`) {
t.Fatalf("quoted profile table should be replaced, got:\n%s", content)
}
if strings.Contains(content, "old:1234") {
t.Fatalf("old URL was not replaced, got:\n%s", content)
}
if got := codexSectionStringValue(content, codexProfileHeader(), "model_provider"); got != codexProfileName {
t.Fatalf("profile model_provider = %q, want %q", got, codexProfileName)
}
if got := codexSectionStringValue(content, codexProviderHeader(), "base_url"); !strings.Contains(got, "/v1/") {
t.Fatalf("provider base_url = %q, want /v1/ URL", got)
}
if err := codexValidateConfigText(content); err != nil {
t.Fatalf("generated config should be valid TOML: %v\n%s", err, content)
}
})
t.Run("rejects invalid existing toml without writing", func(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")
existing := "profile = \n"
os.WriteFile(configPath, []byte(existing), 0o644)
err := writeCodexProfile(configPath)
if err == nil || !strings.Contains(err.Error(), "invalid Codex config TOML") {
t.Fatalf("writeCodexProfile error = %v, want invalid TOML", err)
}
data, _ := os.ReadFile(configPath)
if string(data) != existing {
t.Fatalf("invalid config should be left untouched, got:\n%s", data)
}
})
t.Run("rejects malformed existing toml variants without writing", func(t *testing.T) {
tests := map[string]string{
"duplicate root key": "profile = \"default\"\nprofile = \"other\"\n",
"unterminated string": "model = \"gpt-5.5\n",
"bad table": "[profiles.ollama-launch\nmodel = \"llama3.2\"\n",
"duplicate table key": "[profiles.ollama-launch]\nmodel = \"a\"\nmodel = \"b\"\n",
}
for name, existing := range tests {
t.Run(name, func(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")
if err := os.WriteFile(configPath, []byte(existing), 0o644); err != nil {
t.Fatal(err)
}
err := writeCodexProfile(configPath)
if err == nil || !strings.Contains(err.Error(), "invalid Codex config TOML") {
t.Fatalf("writeCodexProfile error = %v, want invalid TOML", err)
}
data, _ := os.ReadFile(configPath)
if string(data) != existing {
t.Fatalf("invalid config should be left untouched, got:\n%s", data)
}
})
}
})
t.Run("backs up previous config before overwrite", func(t *testing.T) {
tmpDir := t.TempDir()
setTestHome(t, tmpDir)
configPath := filepath.Join(tmpDir, ".codex", "config.toml")
if err := os.MkdirAll(filepath.Dir(configPath), 0o755); err != nil {
t.Fatal(err)
}
existing := "# original-codex-backup-marker\n[profiles.default]\nmodel = \"gpt-5.5\"\n"
if err := os.WriteFile(configPath, []byte(existing), 0o644); err != nil {
t.Fatal(err)
}
if err := writeCodexProfile(configPath); err != nil {
t.Fatal(err)
}
assertBackupContains(t, filepath.Join(fileutil.BackupDir(), "config.toml.*"), "original-codex-backup-marker")
})
t.Run("updates equivalent quoted root keys", func(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")
existing := "" +
`"profile" = "default"` + "\n" +
`"model" = "gpt-5.5"` + "\n" +
`"model_provider" = "openai"` + "\n\n" +
`[profiles.default]` + "\n" +
`model = "gpt-5.5"` + "\n"
os.WriteFile(configPath, []byte(existing), 0o644)
err := writeCodexLaunchProfile(configPath, codexLaunchProfileOptions{
activate: true,
setRootModelConfig: true,
model: "llama3.2",
})
if err != nil {
t.Fatal(err)
}
data, _ := os.ReadFile(configPath)
content := string(data)
for key, want := range map[string]string{
"profile": codexProfileName,
"model": "llama3.2",
"model_provider": codexProfileName,
} {
if got := codexRootStringValue(content, key); got != want {
t.Fatalf("root %s = %q, want %q in:\n%s", key, got, want, content)
}
}
if strings.Contains(content, `"profile"`) || strings.Contains(content, `"model_provider"`) {
t.Fatalf("quoted root keys should be rewritten once, got:\n%s", content)
}
if err := codexValidateConfigText(content); err != nil {
t.Fatalf("generated config should be valid TOML: %v\n%s", err, content)
}
})
t.Run("replaces profile while preserving following sections", func(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")
catalogPath := filepath.Join(tmpDir, "model.json")
existing := "[profiles.ollama-launch]\nopenai_base_url = \"http://old:1234/v1/\"\n[another_section]\nfoo = \"bar\"\n"
os.WriteFile(configPath, []byte(existing), 0o644)
if err := writeCodexProfile(configPath); err != nil {
if err := writeCodexProfile(configPath, catalogPath); err != nil {
t.Fatal(err)
}
@@ -143,10 +305,11 @@ func TestWriteCodexProfile(t *testing.T) {
t.Run("appends newline to file not ending with newline", func(t *testing.T) {
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")
catalogPath := filepath.Join(tmpDir, "model.json")
existing := "[other]\nkey = \"val\""
os.WriteFile(configPath, []byte(existing), 0o644)
if err := writeCodexProfile(configPath); err != nil {
if err := writeCodexProfile(configPath, catalogPath); err != nil {
t.Fatal(err)
}
@@ -166,8 +329,9 @@ func TestWriteCodexProfile(t *testing.T) {
t.Setenv("OLLAMA_HOST", "http://myhost:9999")
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")
catalogPath := filepath.Join(tmpDir, "model.json")
if err := writeCodexProfile(configPath); err != nil {
if err := writeCodexProfile(configPath, catalogPath); err != nil {
t.Fatal(err)
}
@@ -178,6 +342,26 @@ func TestWriteCodexProfile(t *testing.T) {
t.Errorf("expected custom host in URL, got:\n%s", content)
}
})
t.Run("uses connectable host for unspecified bind address", func(t *testing.T) {
t.Setenv("OLLAMA_HOST", "http://0.0.0.0:11434")
tmpDir := t.TempDir()
configPath := filepath.Join(tmpDir, "config.toml")
if err := writeCodexProfile(configPath); err != nil {
t.Fatal(err)
}
data, _ := os.ReadFile(configPath)
content := string(data)
if strings.Contains(content, "0.0.0.0") {
t.Fatalf("config should not write bind-only host, got:\n%s", content)
}
if !strings.Contains(content, "127.0.0.1:11434/v1/") {
t.Fatalf("expected connectable loopback URL, got:\n%s", content)
}
})
}
func TestEnsureCodexConfig(t *testing.T) {
@@ -185,7 +369,7 @@ func TestEnsureCodexConfig(t *testing.T) {
tmpDir := t.TempDir()
setTestHome(t, tmpDir)
if err := ensureCodexConfig(); err != nil {
if err := ensureCodexConfig("llama3.2", launchModelsFromNames([]string{"llama3.2"})); err != nil {
t.Fatal(err)
}
@@ -202,16 +386,25 @@ func TestEnsureCodexConfig(t *testing.T) {
if !strings.Contains(content, "openai_base_url") {
t.Error("missing openai_base_url key")
}
catalogPath := filepath.Join(tmpDir, ".codex", "model.json")
data, err = os.ReadFile(catalogPath)
if err != nil {
t.Fatalf("model.json not created: %v", err)
}
if !strings.Contains(string(data), `"slug": "llama3.2"`) {
t.Error("missing model catalog entry for selected model")
}
})
t.Run("is idempotent", func(t *testing.T) {
tmpDir := t.TempDir()
setTestHome(t, tmpDir)
if err := ensureCodexConfig(); err != nil {
if err := ensureCodexConfig("llama3.2", launchModelsFromNames([]string{"llama3.2"})); err != nil {
t.Fatal(err)
}
if err := ensureCodexConfig(); err != nil {
if err := ensureCodexConfig("llama3.2", launchModelsFromNames([]string{"llama3.2"})); err != nil {
t.Fatal(err)
}
@@ -227,3 +420,176 @@ func TestEnsureCodexConfig(t *testing.T) {
}
})
}
func assertBackupContains(t *testing.T, pattern, marker string) {
t.Helper()
backups, err := filepath.Glob(pattern)
if err != nil {
t.Fatal(err)
}
for _, backupPath := range backups {
data, err := os.ReadFile(backupPath)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(data), marker) {
return
}
}
t.Fatalf("backup matching %q with marker %q not found", pattern, marker)
}
func TestModelInfoContextLength(t *testing.T) {
tests := []struct {
name string
modelInfo map[string]any
want int
}{
{"float64 value", map[string]any{"qwen3_5_moe.context_length": float64(262144)}, 262144},
{"int value", map[string]any{"llama.context_length": 131072}, 131072},
{"no context_length key", map[string]any{"llama.embedding_length": float64(4096)}, 0},
{"empty map", map[string]any{}, 0},
{"nil map", nil, 0},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, _ := modelInfoContextLength(tt.modelInfo)
if got != tt.want {
t.Errorf("modelInfoContextLength() = %d, want %d", got, tt.want)
}
})
}
}
func TestBuildCodexModelEntryContextWindow(t *testing.T) {
tests := []struct {
name string
model LaunchModel
envContextLen string
wantContext int
}{
{
name: "inventory context length as fallback",
model: LaunchModel{
Name: "llama3.2",
ContextLength: 131072,
Details: api.ModelDetails{Format: "gguf"},
},
wantContext: 131072,
},
{
name: "details context length is used when model context is empty",
model: LaunchModel{
Name: "llama3.2",
Details: api.ModelDetails{Format: "gguf", ContextLength: 131072},
},
wantContext: 131072,
},
{
name: "OLLAMA_CONTEXT_LENGTH overrides local gguf inventory context",
model: LaunchModel{
Name: "llama3.2",
ContextLength: 131072,
Details: api.ModelDetails{Format: "gguf"},
},
envContextLen: "64000",
wantContext: 64000,
},
{
name: "safetensors uses inventory context only",
model: LaunchModel{
Name: "llama3.2",
ContextLength: 131072,
Details: api.ModelDetails{Format: "safetensors"},
},
envContextLen: "64000",
wantContext: 131072,
},
{
name: "cloud model uses hardcoded limits",
model: LaunchModel{
Name: "qwen3.5:cloud",
ContextLength: 131072,
Details: api.ModelDetails{Format: "gguf"},
},
envContextLen: "64000",
wantContext: 262144,
},
{
name: "unknown cloud model without metadata uses fallback context",
model: LaunchModel{
Name: "deepseek-v4-pro:cloud",
},
envContextLen: "64000",
wantContext: codexFallbackContextWindow,
},
{
name: "vision capability without reasoning advertisement",
model: LaunchModel{
Name: "llama3.2",
ContextLength: 131072,
Details: api.ModelDetails{Format: "gguf"},
Capabilities: []modelpkg.Capability{modelpkg.CapabilityVision, modelpkg.CapabilityThinking},
},
wantContext: 131072,
},
{
name: "missing metadata uses fallback context",
model: LaunchModel{Name: "llama3.2"},
wantContext: codexFallbackContextWindow,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if tt.envContextLen != "" {
t.Setenv("OLLAMA_CONTEXT_LENGTH", tt.envContextLen)
} else {
t.Setenv("OLLAMA_CONTEXT_LENGTH", "")
}
entry := buildCodexModelEntry(tt.model)
gotContext, _ := entry["context_window"].(int)
if gotContext != tt.wantContext {
t.Errorf("context_window = %d, want %d", gotContext, tt.wantContext)
}
if tt.name == "vision capability without reasoning advertisement" {
modalities, _ := entry["input_modalities"].([]string)
if !slices.Contains(modalities, "image") {
t.Error("expected image in input_modalities")
}
levels, _ := entry["supported_reasoning_levels"].([]any)
if len(levels) != 0 {
t.Errorf("supported_reasoning_levels length = %d, want 0", len(levels))
}
if got, _ := entry["supports_reasoning_summaries"].(bool); got {
t.Error("supports_reasoning_summaries = true, want false")
}
}
if tt.name == "cloud model uses hardcoded limits" {
truncationPolicy, _ := entry["truncation_policy"].(map[string]any)
if mode, _ := truncationPolicy["mode"].(string); mode != "tokens" {
t.Errorf("truncation_policy mode = %q, want %q", mode, "tokens")
}
}
requiredKeys := []string{"slug", "display_name", "shell_type"}
for _, key := range requiredKeys {
if _, ok := entry[key]; !ok {
t.Errorf("missing required key %q", key)
}
}
if _, ok := entry["apply_patch_tool_type"]; ok {
t.Error("apply_patch_tool_type should be omitted so Codex CLI defaults can handle schema changes")
}
if _, err := json.Marshal(entry); err != nil {
t.Errorf("entry is not JSON serializable: %v", err)
}
})
}
}
+1 -1
View File
@@ -43,7 +43,7 @@ func (c *Copilot) findPath() (string, error) {
return fallback, nil
}
func (c *Copilot) Run(model string, args []string) error {
func (c *Copilot) Run(model string, _ []LaunchModel, args []string) error {
copilotPath, err := c.findPath()
if err != nil {
return fmt.Errorf("copilot is not installed, install from https://docs.github.com/en/copilot/how-tos/set-up/install-copilot-cli")
+9 -11
View File
@@ -40,7 +40,7 @@ type modelEntry struct {
func (d *Droid) String() string { return "Droid" }
func (d *Droid) Run(model string, args []string) error {
func (d *Droid) Run(model string, _ []LaunchModel, args []string) error {
if _, err := exec.LookPath("droid"); err != nil {
return fmt.Errorf("droid is not installed, install from https://docs.factory.ai/cli/getting-started/quickstart")
}
@@ -64,7 +64,7 @@ func (d *Droid) Paths() []string {
return nil
}
func (d *Droid) Edit(models []string) error {
func (d *Droid) Edit(models []LaunchModel) error {
if len(models) == 0 {
return nil
}
@@ -99,7 +99,7 @@ func (d *Droid) Edit(models []string) error {
return fileutil.WriteWithBackup(settingsPath, data, "droid")
}
func updateDroidSettings(settingsMap map[string]any, settings droidSettings, models []string) map[string]any {
func updateDroidSettings(settingsMap map[string]any, settings droidSettings, models []LaunchModel) map[string]any {
// Keep only non-Ollama models from the raw map (preserves extra fields)
// Rebuild Ollama models
var nonOllamaModels []any
@@ -119,20 +119,18 @@ func updateDroidSettings(settingsMap map[string]any, settings droidSettings, mod
var defaultModelID string
for i, model := range models {
maxOutput := 64000
if isCloudModelName(model) {
if l, ok := lookupCloudModelLimit(model); ok {
maxOutput = l.Output
}
if model.MaxOutputTokens > 0 {
maxOutput = model.MaxOutputTokens
}
modelID := fmt.Sprintf("custom:%s-%d", model, i)
modelID := fmt.Sprintf("custom:%s-%d", model.Name, i)
newModels = append(newModels, modelEntry{
Model: model,
DisplayName: model,
Model: model.Name,
DisplayName: model.Name,
BaseURL: envconfig.Host().String() + "/v1",
APIKey: "ollama",
Provider: "generic-chat-completion-api",
MaxOutputTokens: maxOutput,
SupportsImages: false,
SupportsImages: model.HasCapability("vision"),
ID: modelID,
Index: i,
})
+35 -35
View File
@@ -63,7 +63,7 @@ func TestDroidEdit(t *testing.T) {
t.Run("fresh install creates models with sequential indices", func(t *testing.T) {
cleanup()
if err := d.Edit([]string{"model-a", "model-b"}); err != nil {
if err := d.Edit(testLaunchModels("model-a", "model-b")); err != nil {
t.Fatal(err)
}
@@ -99,7 +99,7 @@ func TestDroidEdit(t *testing.T) {
t.Run("sets sessionDefaultSettings.model to first model ID", func(t *testing.T) {
cleanup()
if err := d.Edit([]string{"model-a", "model-b"}); err != nil {
if err := d.Edit(testLaunchModels("model-a", "model-b")); err != nil {
t.Fatal(err)
}
@@ -116,10 +116,10 @@ func TestDroidEdit(t *testing.T) {
t.Run("re-indexes when models removed", func(t *testing.T) {
cleanup()
// Add three models
d.Edit([]string{"model-a", "model-b", "model-c"})
d.Edit(testLaunchModels("model-a", "model-b", "model-c"))
// Remove middle model
d.Edit([]string{"model-a", "model-c"})
d.Edit(testLaunchModels("model-a", "model-c"))
settings := readSettings()
models := getCustomModels(settings)
@@ -155,7 +155,7 @@ func TestDroidEdit(t *testing.T) {
]
}`), 0o644)
d.Edit([]string{"model-a"})
d.Edit(testLaunchModels("model-a"))
settings := readSettings()
models := getCustomModels(settings)
@@ -184,7 +184,7 @@ func TestDroidEdit(t *testing.T) {
"sessionDefaultSettings": {"autonomyMode": "auto-high"}
}`), 0o644)
d.Edit([]string{"model-a"})
d.Edit(testLaunchModels("model-a"))
settings := readSettings()
@@ -203,7 +203,7 @@ func TestDroidEdit(t *testing.T) {
t.Run("required fields present", func(t *testing.T) {
cleanup()
d.Edit([]string{"test-model"})
d.Edit(testLaunchModels("test-model"))
settings := readSettings()
models := getCustomModels(settings)
@@ -239,7 +239,7 @@ func TestDroidEdit(t *testing.T) {
"sessionDefaultSettings": {"reasoningEffort": "off"}
}`), 0o644)
d.Edit([]string{"model-a"})
d.Edit(testLaunchModels("model-a"))
settings := readSettings()
session := settings["sessionDefaultSettings"].(map[string]any)
@@ -256,7 +256,7 @@ func TestDroidEdit(t *testing.T) {
"sessionDefaultSettings": {"reasoningEffort": "high"}
}`), 0o644)
d.Edit([]string{"model-a"})
d.Edit(testLaunchModels("model-a"))
settings := readSettings()
session := settings["sessionDefaultSettings"].(map[string]any)
@@ -281,7 +281,7 @@ func TestDroidEdit_CorruptedJSON(t *testing.T) {
os.WriteFile(settingsPath, []byte(`{corrupted json content`), 0o644)
// Corrupted JSON should return an error so user knows something is wrong
err := d.Edit([]string{"model-a"})
err := d.Edit(testLaunchModels("model-a"))
if err == nil {
t.Fatal("expected error for corrupted JSON, got nil")
}
@@ -306,7 +306,7 @@ func TestDroidEdit_WrongTypeCustomModels(t *testing.T) {
os.WriteFile(settingsPath, []byte(`{"customModels": "not an array"}`), 0o644)
// Should not panic - wrong type should be handled gracefully
err := d.Edit([]string{"model-a"})
err := d.Edit(testLaunchModels("model-a"))
if err != nil {
t.Fatalf("Edit failed with wrong type customModels: %v", err)
}
@@ -338,7 +338,7 @@ func TestDroidEdit_EmptyModels(t *testing.T) {
os.WriteFile(settingsPath, []byte(originalContent), 0o644)
// Empty models should be no-op
err := d.Edit([]string{})
err := d.Edit(testLaunchModels())
if err != nil {
t.Fatalf("Edit with empty models failed: %v", err)
}
@@ -359,7 +359,7 @@ func TestDroidEdit_DuplicateModels(t *testing.T) {
settingsPath := filepath.Join(settingsDir, "settings.json")
// Add same model twice
err := d.Edit([]string{"model-a", "model-a"})
err := d.Edit(testLaunchModels("model-a", "model-a"))
if err != nil {
t.Fatalf("Edit with duplicates failed: %v", err)
}
@@ -388,7 +388,7 @@ func TestDroidEdit_MalformedModelEntry(t *testing.T) {
// Model entry is a string instead of a map
os.WriteFile(settingsPath, []byte(`{"customModels": ["not a map", 123]}`), 0o644)
err := d.Edit([]string{"model-a"})
err := d.Edit(testLaunchModels("model-a"))
if err != nil {
t.Fatalf("Edit with malformed entries failed: %v", err)
}
@@ -415,7 +415,7 @@ func TestDroidEdit_WrongTypeSessionSettings(t *testing.T) {
// sessionDefaultSettings is a string instead of map
os.WriteFile(settingsPath, []byte(`{"sessionDefaultSettings": "not a map"}`), 0o644)
err := d.Edit([]string{"model-a"})
err := d.Edit(testLaunchModels("model-a"))
if err != nil {
t.Fatalf("Edit with wrong type sessionDefaultSettings failed: %v", err)
}
@@ -490,7 +490,7 @@ func TestDroidEdit_RoundTrip(t *testing.T) {
os.WriteFile(settingsPath, []byte(testDroidSettingsFixture), 0o644)
// Edit with new models
if err := d.Edit([]string{"llama3", "mistral"}); err != nil {
if err := d.Edit(testLaunchModels("llama3", "mistral")); err != nil {
t.Fatal(err)
}
@@ -615,7 +615,7 @@ func TestDroidEdit_PreservesUnknownFields(t *testing.T) {
}`
os.WriteFile(settingsPath, []byte(original), 0o644)
if err := d.Edit([]string{"model-a"}); err != nil {
if err := d.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
@@ -660,7 +660,7 @@ func TestDroidEdit_PreservesUnknownFields(t *testing.T) {
}`
os.WriteFile(settingsPath, []byte(original), 0o644)
if err := d.Edit([]string{"llama3"}); err != nil {
if err := d.Edit(testLaunchModels("llama3")); err != nil {
t.Fatal(err)
}
@@ -715,10 +715,10 @@ func TestDroidEdit_Idempotent(t *testing.T) {
os.WriteFile(settingsPath, []byte(testDroidSettingsFixture), 0o644)
// Edit twice with same models
d.Edit([]string{"llama3", "mistral"})
d.Edit(testLaunchModels("llama3", "mistral"))
firstData, _ := os.ReadFile(settingsPath)
d.Edit([]string{"llama3", "mistral"})
d.Edit(testLaunchModels("llama3", "mistral"))
secondData, _ := os.ReadFile(settingsPath)
// Results should be identical
@@ -744,7 +744,7 @@ func TestDroidEdit_MultipleConsecutiveEdits(t *testing.T) {
if i%2 == 0 {
models = []string{"model-x", "model-y", "model-z"}
}
if err := d.Edit(models); err != nil {
if err := d.Edit(launchModelsFromNames(models)); err != nil {
t.Fatalf("edit %d failed: %v", i, err)
}
}
@@ -803,7 +803,7 @@ func TestDroidEdit_UnicodeAndSpecialCharacters(t *testing.T) {
}`
os.WriteFile(settingsPath, []byte(original), 0o644)
if err := d.Edit([]string{"model-a"}); err != nil {
if err := d.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
@@ -845,7 +845,7 @@ func TestDroidEdit_LargeNumbers(t *testing.T) {
}`
os.WriteFile(settingsPath, []byte(original), 0o644)
if err := d.Edit([]string{"model-a"}); err != nil {
if err := d.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
@@ -889,7 +889,7 @@ func TestDroidEdit_EmptyAndNullValues(t *testing.T) {
}`
os.WriteFile(settingsPath, []byte(original), 0o644)
if err := d.Edit([]string{"model-a"}); err != nil {
if err := d.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
@@ -943,7 +943,7 @@ func TestDroidEdit_DeeplyNestedStructures(t *testing.T) {
}`
os.WriteFile(settingsPath, []byte(original), 0o644)
if err := d.Edit([]string{"model-a"}); err != nil {
if err := d.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
@@ -988,7 +988,7 @@ func TestDroidEdit_ModelNamesWithSpecialCharacters(t *testing.T) {
"model_with_underscores",
}
if err := d.Edit(specialModels); err != nil {
if err := d.Edit(launchModelsFromNames(specialModels)); err != nil {
t.Fatal(err)
}
@@ -1025,7 +1025,7 @@ func TestDroidEdit_MissingCustomModelsKey(t *testing.T) {
t.Fatal(err)
}
settings = updateDroidSettings(settings, settingsStruct, []string{"model-a"})
settings = updateDroidSettings(settings, settingsStruct, testLaunchModels("model-a"))
// Original fields preserved
if settings["diffMode"] != "github" {
@@ -1062,7 +1062,7 @@ func TestDroidEdit_NullCustomModels(t *testing.T) {
}`
os.WriteFile(settingsPath, []byte(original), 0o644)
if err := d.Edit([]string{"model-a"}); err != nil {
if err := d.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
@@ -1090,7 +1090,7 @@ func TestDroidEdit_MinifiedJSON(t *testing.T) {
original := `{"diffMode":"github","enableHooks":true,"hooks":{"imported":["cmd1","cmd2"]},"customModels":[],"sessionDefaultSettings":{}}`
os.WriteFile(settingsPath, []byte(original), 0o644)
if err := d.Edit([]string{"model-a"}); err != nil {
if err := d.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
@@ -1120,7 +1120,7 @@ func TestDroidEdit_CreatesDirectoryIfMissing(t *testing.T) {
t.Fatal("directory should not exist before test")
}
if err := d.Edit([]string{"model-a"}); err != nil {
if err := d.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
@@ -1157,7 +1157,7 @@ func TestDroidEdit_PreservesFileAfterError(t *testing.T) {
os.WriteFile(settingsPath, []byte(original), 0o644)
// Empty models list is a no-op, should not modify file
d.Edit([]string{})
d.Edit(testLaunchModels())
data, _ := os.ReadFile(settingsPath)
if string(data) != original {
@@ -1181,7 +1181,7 @@ func TestDroidEdit_BackupCreated(t *testing.T) {
original := fmt.Sprintf(`{"diffMode": "%s", "customModels": [], "sessionDefaultSettings": {}}`, uniqueMarker)
os.WriteFile(settingsPath, []byte(original), 0o644)
if err := d.Edit([]string{"model-a"}); err != nil {
if err := d.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
@@ -1231,7 +1231,7 @@ func TestDroidEdit_LargeNumberOfModels(t *testing.T) {
models = append(models, fmt.Sprintf("model-%d", i))
}
if err := d.Edit(models); err != nil {
if err := d.Edit(launchModelsFromNames(models)); err != nil {
t.Fatal(err)
}
@@ -1261,7 +1261,7 @@ func TestDroidEdit_LocalModelDefaultMaxOutput(t *testing.T) {
settingsDir := filepath.Join(tmpDir, ".factory")
settingsPath := filepath.Join(settingsDir, "settings.json")
if err := d.Edit([]string{"llama3.2"}); err != nil {
if err := d.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
@@ -1312,7 +1312,7 @@ func TestDroidEdit_ArraysWithMixedTypes(t *testing.T) {
}`
os.WriteFile(settingsPath, []byte(original), 0o644)
if err := d.Edit([]string{"model-a"}); err != nil {
if err := d.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
+1 -1
View File
@@ -65,7 +65,7 @@ type Hermes struct{}
func (h *Hermes) String() string { return "Hermes Agent" }
func (h *Hermes) Run(_ string, args []string) error {
func (h *Hermes) Run(_ string, _ []LaunchModel, args []string) error {
// Hermes reads its primary model from config.yaml. launch configures that
// default model ahead of time so we can keep runtime invocation simple and
// still let Hermes discover additional models later via its own UX.
+7 -7
View File
@@ -552,7 +552,7 @@ func TestHermesRunPassthroughArgs(t *testing.T) {
}
h := &Hermes{}
if err := h.Run("", []string{"--continue"}); err != nil {
if err := h.Run("", nil, []string{"--continue"}); err != nil {
t.Fatalf("Run returned error: %v", err)
}
@@ -603,7 +603,7 @@ fi
}
h := &Hermes{}
if err := h.Run("", nil); err != nil {
if err := h.Run("", nil, nil); err != nil {
t.Fatalf("Run returned error: %v", err)
}
@@ -655,10 +655,10 @@ func TestHermesRun_SetUpLaterRepromptsOnLaterLaunches(t *testing.T) {
}
h := &Hermes{}
if err := h.Run("", nil); err != nil {
if err := h.Run("", nil, nil); err != nil {
t.Fatalf("first Run returned error: %v", err)
}
if err := h.Run("", nil); err != nil {
if err := h.Run("", nil, nil); err != nil {
t.Fatalf("second Run returned error: %v", err)
}
@@ -713,7 +713,7 @@ func TestHermesRun_SkipsMessagingPromptWhenConfigured(t *testing.T) {
}
h := &Hermes{}
if err := h.Run("", nil); err != nil {
if err := h.Run("", nil, nil); err != nil {
t.Fatalf("Run returned error: %v", err)
}
@@ -753,7 +753,7 @@ func TestHermesRun_SkipsMessagingPromptWithYesPolicy(t *testing.T) {
}
h := &Hermes{}
if err := h.Run("", nil); err != nil {
if err := h.Run("", nil, nil); err != nil {
t.Fatalf("Run returned error: %v", err)
}
@@ -798,7 +798,7 @@ fi
}
h := &Hermes{}
err := h.Run("", nil)
err := h.Run("", nil, nil)
if err == nil {
t.Fatal("expected messaging setup failure")
}
+49 -11
View File
@@ -25,7 +25,7 @@ type stubEditorRunner struct {
editErr error
}
func (s *stubEditorRunner) Run(model string, args []string) error {
func (s *stubEditorRunner) Run(model string, _ []LaunchModel, args []string) error {
s.ranModel = model
return nil
}
@@ -34,11 +34,11 @@ func (s *stubEditorRunner) String() string { return "StubEditor" }
func (s *stubEditorRunner) Paths() []string { return nil }
func (s *stubEditorRunner) Edit(models []string) error {
func (s *stubEditorRunner) Edit(models []LaunchModel) error {
if s.editErr != nil {
return s.editErr
}
cloned := append([]string(nil), models...)
cloned := launchModelNames(models)
s.edited = append(s.edited, cloned)
return nil
}
@@ -58,6 +58,9 @@ func TestIntegrationLookup(t *testing.T) {
{"claude desktop", "claude-desktop", true, "Claude Desktop"},
{"claude desktop alias", "claude-app", true, "Claude Desktop"},
{"codex", "codex", true, "Codex"},
{"codex app", "codex-app", true, "Codex App"},
{"codex app desktop alias", "codex-desktop", true, "Codex App"},
{"codex app gui alias", "codex-gui", true, "Codex App"},
{"kimi", "kimi", true, "Kimi Code CLI"},
{"droid", "droid", true, "Droid"},
{"opencode", "opencode", true, "OpenCode"},
@@ -80,7 +83,7 @@ func TestIntegrationLookup(t *testing.T) {
}
func TestIntegrationRegistry(t *testing.T) {
expectedIntegrations := []string{"claude", "claude-desktop", "codex", "kimi", "droid", "opencode", "hermes", "pool"}
expectedIntegrations := []string{"claude", "claude-desktop", "codex", "codex-app", "kimi", "droid", "opencode", "hermes", "pool"}
for _, name := range expectedIntegrations {
t.Run(name, func(t *testing.T) {
r, ok := integrations[name]
@@ -203,7 +206,7 @@ func TestAllIntegrations_HaveRequiredMethods(t *testing.T) {
if displayName == "" {
t.Error("String() should not return empty")
}
var _ func(string, []string) error = r.Run
var _ func(string, []LaunchModel, []string) error = r.Run
})
}
}
@@ -478,11 +481,11 @@ func TestBuildModelList_ExistingRecommendedMarked(t *testing.T) {
func TestBuildModelList_PreservesRecommendationRequiredPlanForExistingCloudModel(t *testing.T) {
recommendations := []ModelItem{
{
Name: "glm-5:cloud",
Description: "Reasoning and code generation",
Recommended: true,
RequiredPlan: "pro",
ContextLength: 202_752,
Name: "glm-5:cloud",
Description: "Reasoning and code generation",
Recommended: true,
RequiredPlan: "pro",
Details: api.ModelDetails{ContextLength: 202_752},
},
}
existing := []modelInfo{{Name: "glm-5:cloud", Remote: true}}
@@ -863,7 +866,7 @@ func TestPrepareEditorIntegration_SavesOnlyAfterSuccessfulEdit(t *testing.T) {
}
editor := &stubEditorRunner{editErr: errors.New("boom")}
err := prepareEditorIntegration("droid", editor, []string{"new-model"})
err := prepareEditorIntegration("droid", editor, testLaunchModels("new-model"))
if err == nil || !strings.Contains(err.Error(), "setup failed") {
t.Fatalf("expected setup failure, got %v", err)
}
@@ -1737,6 +1740,11 @@ func TestIntegration_InstallHint(t *testing.T) {
input: "codex",
wantURL: "https://developers.openai.com/codex/cli/",
},
{
name: "codex app has hint",
input: "codex-app",
wantURL: "https://developers.openai.com/codex/quickstart",
},
{
name: "openclaw has hint",
input: "openclaw",
@@ -1813,11 +1821,38 @@ func TestListIntegrationInfos(t *testing.T) {
}
want = filtered
}
if codexAppSupported() != nil {
filtered := make([]string, 0, len(want))
for _, name := range want {
if name != "codex-app" {
filtered = append(filtered, name)
}
}
want = filtered
}
if diff := compareStrings(got, want); diff != "" {
t.Fatalf("launcher integration order mismatch: %s", diff)
}
})
t.Run("prioritizes primary launcher integrations", func(t *testing.T) {
got := make([]string, 0, len(infos))
for _, info := range infos {
got = append(got, info.Name)
}
wantPrefix := []string{"claude", "codex-app", "hermes", "openclaw"}
if codexAppSupported() != nil {
wantPrefix = []string{"claude", "hermes", "openclaw", "opencode"}
}
if len(got) < len(wantPrefix) {
t.Fatalf("expected at least %d integrations, got %v", len(wantPrefix), got)
}
if diff := compareStrings(got[:len(wantPrefix)], wantPrefix); diff != "" {
t.Fatalf("unexpected primary launcher order: %s", diff)
}
})
t.Run("all fields populated", func(t *testing.T) {
for _, info := range infos {
if info.Name == "" {
@@ -1831,6 +1866,9 @@ func TestListIntegrationInfos(t *testing.T) {
t.Run("includes known integrations", func(t *testing.T) {
known := map[string]bool{"claude": false, "codex": false, "opencode": false}
if codexAppSupported() == nil {
known["codex-app"] = false
}
if poolsideGOOS != "windows" {
known["pool"] = false
}
+1 -1
View File
@@ -36,7 +36,7 @@ func (k *Kimi) args(config string, extra []string) []string {
return args
}
func (k *Kimi) Run(model string, args []string) error {
func (k *Kimi) Run(model string, _ []LaunchModel, args []string) error {
if strings.TrimSpace(model) == "" {
return fmt.Errorf("model is required")
}
+2 -2
View File
@@ -307,7 +307,7 @@ func TestKimiRun_RejectsConflictingArgsBeforeInstall(t *testing.T) {
}
t.Cleanup(func() { DefaultConfirmPrompt = oldConfirm })
err := k.Run("llama3.2", []string{"--model", "other"})
err := k.Run("llama3.2", nil, []string{"--model", "other"})
if err == nil || !strings.Contains(err.Error(), "--model") {
t.Fatalf("expected conflict error mentioning --model, got %v", err)
}
@@ -337,7 +337,7 @@ exit 0
t.Setenv("OLLAMA_HOST", srv.URL)
k := &Kimi{}
if err := k.Run("llama3.2", []string{"--quiet", "--print"}); err != nil {
if err := k.Run("llama3.2", nil, []string{"--quiet", "--print"}); err != nil {
t.Fatalf("Run() error = %v", err)
}
+74 -66
View File
@@ -12,6 +12,7 @@ import (
"github.com/ollama/ollama/api"
"github.com/ollama/ollama/cmd/config"
modelpkg "github.com/ollama/ollama/types/model"
"github.com/spf13/cobra"
"golang.org/x/term"
)
@@ -137,16 +138,17 @@ var isInteractiveSession = func() bool {
return term.IsTerminal(int(os.Stdin.Fd())) && term.IsTerminal(int(os.Stdout.Fd()))
}
// Runner executes a model with an integration.
// Runner executes an integration with the selected model and its resolved
// launch metadata. models is ordered with the primary model first.
type Runner interface {
Run(model string, args []string) error
Run(model string, models []LaunchModel, args []string) error
String() string
}
// Editor can edit config files for integrations that support model configuration.
type Editor interface {
Paths() []string
Edit(models []string) error
Edit(models []LaunchModel) error
Models() []string
}
@@ -165,7 +167,7 @@ type ManagedSingleModel interface {
// ManagedModelListConfigurer lets managed single-model integrations receive
// the launcher's model list while still preserving one primary selected model.
type ManagedModelListConfigurer interface {
ConfigureWithModels(primary string, models []string) error
ConfigureWithModels(primary string, models []LaunchModel) error
}
// ManagedAutodiscoveryIntegration is for managed integrations that do not need
@@ -239,24 +241,18 @@ type SupportedIntegration interface {
Supported() error
}
type modelInfo struct {
Name string
Remote bool
ToolCapable bool
}
// ModelInfo re-exports launcher model inventory details for callers.
type ModelInfo = modelInfo
// ModelItem represents model metadata before selector-only UI state is derived.
type ModelItem struct {
Name string
Description string
Recommended bool
VRAMBytes int64
ContextLength int
MaxOutputTokens int
RequiredPlan string
ToolCapable bool
Capabilities []modelpkg.Capability
Size int64
Details api.ModelDetails
}
// SelectionItem represents a model row after launch has derived selector-only UI state.
@@ -285,22 +281,25 @@ Flags and extra arguments require an integration name.
Supported integrations:
claude Claude Code
cline Cline
codex-app Codex App (aliases: codex-desktop, codex-gui)
hermes Hermes Agent
openclaw OpenClaw (aliases: clawdbot, moltbot)
opencode OpenCode
codex Codex
copilot Copilot CLI (aliases: copilot-cli)
droid Droid
hermes Hermes Agent
kimi Kimi Code CLI
opencode OpenCode
openclaw OpenClaw (aliases: clawdbot, moltbot)
pi Pi
pool Pool
cline Cline
vscode VS Code (aliases: code)
Examples:
ollama launch
ollama launch claude
ollama launch claude --model <model>
ollama launch codex-app
ollama launch codex-app --restore
ollama launch hermes
ollama launch droid --config (does not auto-launch)
ollama launch codex -- -p myprofile (pass extra args to integration)
@@ -406,8 +405,7 @@ func launchCommandIsClaudeDesktop(name string) bool {
type launcherClient struct {
apiClient *api.Client
modelInventory []ModelInfo
inventoryLoaded bool
inventory *modelInventory
recommendationsLoaded bool
recommendationItems []ModelItem
accountState *AccountState
@@ -424,10 +422,18 @@ func newLauncherClient(policy LaunchPolicy) (*launcherClient, error) {
return &launcherClient{
apiClient: apiClient,
inventory: newModelInventory(apiClient),
policy: policy,
}, nil
}
func (c *launcherClient) modelInventory() *modelInventory {
if c.inventory == nil {
c.inventory = newModelInventory(c.apiClient)
}
return c.inventory
}
// BuildLauncherState returns the launch-owned root launcher menu snapshot.
func BuildLauncherState(ctx context.Context) (*LauncherState, error) {
launchClient, err := newLauncherClient(defaultLaunchPolicy(isInteractiveSession(), false))
@@ -549,7 +555,7 @@ func prepareIntegrationLaunch(name string, policy LaunchPolicy) (*launcherClient
}
func (c *launcherClient) buildLauncherState(ctx context.Context) (*LauncherState, error) {
_ = c.loadModelInventoryOnce(ctx)
_, _ = c.modelInventory().Load(ctx)
state := &LauncherState{
LastSelection: config.LastSelection(),
@@ -723,7 +729,7 @@ func (c *launcherClient) launchSingleIntegration(ctx context.Context, name strin
}
}
return launchAfterConfiguration(name, runner, target, req)
return launchAfterConfiguration(name, runner, target, c.resolveRunModels(ctx, []string{target}), req)
}
func (c *launcherClient) launchEditorIntegration(ctx context.Context, name string, runner Runner, editor Editor, saved *config.IntegrationConfig, req IntegrationLaunchRequest) error {
@@ -745,13 +751,17 @@ func (c *launcherClient) launchEditorIntegration(ctx context.Context, name strin
return nil
}
var launchModels []LaunchModel
if (needsConfigure || req.ModelOverride != "") && !savedMatchesModels(saved, models) {
if err := prepareEditorIntegration(name, editor, models); err != nil {
launchModels = c.modelInventory().Resolve(ctx, models)
if err := prepareEditorIntegration(name, editor, launchModels); err != nil {
return err
}
} else {
launchModels = c.resolveRunModels(ctx, models)
}
return launchAfterConfiguration(name, runner, models[0], req)
return launchAfterConfiguration(name, runner, models[0], launchModels, req)
}
func (c *launcherClient) launchManagedSingleIntegration(ctx context.Context, name string, runner Runner, managed ManagedSingleModel, saved *config.IntegrationConfig, req IntegrationLaunchRequest) error {
@@ -769,12 +779,18 @@ func (c *launcherClient) launchManagedSingleIntegration(ctx context.Context, nam
return nil
}
if needsConfigure || req.ModelOverride != "" || (current != "" && target != current) || !savedMatchesModels(saved, []string{target}) {
// current is the live managed app config; target may come from saved launch
// state. Rewrite when the live config is missing or has drifted so the app
// config converges with the model which launch is about to use.
liveConfigMissing := current == ""
liveConfigDrifted := current != "" && target != current
configured := false
if needsConfigure || req.ModelOverride != "" || liveConfigMissing || liveConfigDrifted || !savedMatchesModels(saved, []string{target}) {
configureModels, err := c.managedSingleConfigureModels(ctx, managed, target)
if err != nil {
return err
}
if err := prepareManagedSingleIntegration(name, managed, target, configureModels); err != nil {
if err := prepareManagedSingleIntegration(name, managed, target, c.modelInventory().Resolve(ctx, configureModels)); err != nil {
return err
}
if refresher, ok := managed.(ManagedRuntimeRefresher); ok {
@@ -782,6 +798,7 @@ func (c *launcherClient) launchManagedSingleIntegration(ctx context.Context, nam
return err
}
}
configured = true
}
if !managedIntegrationOnboarded(saved, managed) {
@@ -793,11 +810,17 @@ func (c *launcherClient) launchManagedSingleIntegration(ctx context.Context, nam
}
}
if configured {
if !printConfigurationSuccess(managed) {
printRestoreHint(managed)
}
}
if req.ConfigureOnly {
return nil
}
return runIntegration(runner, target, req.ExtraArgs)
return runIntegration(runner, target, c.resolveRunModels(ctx, []string{target}), req.ExtraArgs)
}
func (c *launcherClient) launchManagedAutodiscoveryIntegration(ctx context.Context, name string, runner Runner, autodiscovery ManagedAutodiscoveryIntegration, saved *config.IntegrationConfig, req IntegrationLaunchRequest) error {
@@ -840,7 +863,7 @@ func (c *launcherClient) launchManagedAutodiscoveryIntegration(ctx context.Conte
return nil
}
return runIntegration(runner, target, req.ExtraArgs)
return runIntegration(runner, target, c.resolveRunModels(ctx, []string{target}), req.ExtraArgs)
}
func (c *launcherClient) managedAutodiscoveryUsable(ctx context.Context, autodiscovery ManagedAutodiscoveryIntegration) bool {
@@ -941,7 +964,7 @@ func (c *launcherClient) resolveSingleIntegrationTarget(ctx context.Context, run
}
}
if needsConfigure {
if needsConfigure && req.ModelOverride == "" {
selected, err := c.selectSingleModelWithSelectorReady(ctx, fmt.Sprintf("Select model for %s:", runner), target, DefaultSingleSelector, !skipReadiness)
if err != nil {
return "", false, err
@@ -1092,13 +1115,14 @@ func runMultiSelector(title string, items []SelectionItem, preChecked []string,
}
func (c *launcherClient) loadSelectableModels(ctx context.Context, preChecked []string, current, emptyMessage string) ([]ModelItem, []string, error) {
if err := c.loadModelInventoryOnce(ctx); err != nil {
inventory, err := c.modelInventory().Load(ctx)
if err != nil {
return nil, nil, err
}
recommendations := c.recommendations(ctx)
cloudDisabled, _ := cloudStatusDisabled(ctx, c.apiClient)
items, orderedChecked, _, _ := buildModelListWithRecommendations(c.modelInventory, recommendations, preChecked, current)
items, orderedChecked, _, _ := buildModelListWithRecommendations(inventory, recommendations, preChecked, current)
if cloudDisabled {
items = filterCloudItems(items)
orderedChecked = c.filterDisabledCloudModels(ctx, orderedChecked)
@@ -1163,9 +1187,11 @@ func (c *launcherClient) requestRecommendations(ctx context.Context) ([]ModelIte
Description: description,
Recommended: true,
VRAMBytes: rec.VRAMBytes,
ContextLength: rec.ContextLength,
MaxOutputTokens: rec.MaxOutputTokens,
RequiredPlan: strings.TrimSpace(rec.RequiredPlan),
Details: api.ModelDetails{
ContextLength: rec.ContextLength,
},
})
}
@@ -1286,10 +1312,11 @@ func (c *launcherClient) filterDisabledCloudModels(ctx context.Context, models [
}
func (c *launcherClient) savedModelUsable(ctx context.Context, name string) (bool, error) {
if err := c.loadModelInventoryOnce(ctx); err != nil {
inventory, err := c.modelInventory().Load(ctx)
if err != nil {
return c.showBasedModelUsable(ctx, name)
}
return c.singleModelUsable(ctx, name), nil
return c.singleModelUsable(ctx, name, inventory), nil
}
func (c *launcherClient) showBasedModelUsable(ctx context.Context, name string) (bool, error) {
@@ -1315,7 +1342,7 @@ func (c *launcherClient) showBasedModelUsable(ctx context.Context, name string)
return true, nil
}
func (c *launcherClient) singleModelUsable(ctx context.Context, name string) bool {
func (c *launcherClient) singleModelUsable(ctx context.Context, name string, inventory []LaunchModel) bool {
if name == "" {
return false
}
@@ -1323,11 +1350,11 @@ func (c *launcherClient) singleModelUsable(ctx context.Context, name string) boo
cloudDisabled, _ := cloudStatusDisabled(ctx, c.apiClient)
return !cloudDisabled
}
return c.hasLocalModel(name)
return hasLocalModel(inventory, name)
}
func (c *launcherClient) hasLocalModel(name string) bool {
for _, model := range c.modelInventory {
func hasLocalModel(inventory []LaunchModel, name string) bool {
for _, model := range inventory {
if model.Remote {
continue
}
@@ -1338,37 +1365,18 @@ func (c *launcherClient) hasLocalModel(name string) bool {
return false
}
func (c *launcherClient) loadModelInventoryOnce(ctx context.Context) error {
if c.inventoryLoaded {
return nil
}
resp, err := c.apiClient.List(ctx)
if err != nil {
return err
}
c.modelInventory = c.modelInventory[:0]
for _, model := range resp.Models {
c.modelInventory = append(c.modelInventory, ModelInfo{
Name: model.Name,
Remote: model.RemoteModel != "",
})
}
cloudDisabled, _ := cloudStatusDisabled(ctx, c.apiClient)
if cloudDisabled {
c.modelInventory = filterCloudModels(c.modelInventory)
}
c.inventoryLoaded = true
return nil
func (c *launcherClient) resolveRunModels(ctx context.Context, models []string) []LaunchModel {
return c.modelInventory().Resolve(ctx, models)
}
func runIntegration(runner Runner, modelName string, args []string) error {
return runner.Run(modelName, args)
func runIntegration(runner Runner, modelName string, models []LaunchModel, args []string) error {
if len(models) == 0 && modelName != "" {
models = launchModelsFromNames([]string{modelName})
}
return runner.Run(modelName, models, args)
}
func launchAfterConfiguration(name string, runner Runner, model string, req IntegrationLaunchRequest) error {
func launchAfterConfiguration(name string, runner Runner, model string, models []LaunchModel, req IntegrationLaunchRequest) error {
if req.ConfigureOnly {
launch, err := ConfirmPrompt(fmt.Sprintf("Launch %s now?", runner))
if err != nil {
@@ -1381,7 +1389,7 @@ func launchAfterConfiguration(name string, runner Runner, model string, req Inte
if err := EnsureIntegrationInstalled(name, runner); err != nil {
return err
}
return runIntegration(runner, model, req.ExtraArgs)
return runIntegration(runner, model, models, req.ExtraArgs)
}
func loadStoredIntegrationConfig(name string) (*config.IntegrationConfig, error) {
+235 -17
View File
@@ -11,6 +11,7 @@ import (
"runtime"
"slices"
"strings"
"sync/atomic"
"testing"
"time"
@@ -24,7 +25,7 @@ type launcherEditorRunner struct {
ranModel string
}
func (r *launcherEditorRunner) Run(model string, args []string) error {
func (r *launcherEditorRunner) Run(model string, _ []LaunchModel, args []string) error {
r.ranModel = model
return nil
}
@@ -33,8 +34,8 @@ func (r *launcherEditorRunner) String() string { return "LauncherEditor" }
func (r *launcherEditorRunner) Paths() []string { return r.paths }
func (r *launcherEditorRunner) Edit(models []string) error {
r.edited = append(r.edited, append([]string(nil), models...))
func (r *launcherEditorRunner) Edit(models []LaunchModel) error {
r.edited = append(r.edited, launchModelNames(models))
return nil
}
@@ -44,7 +45,7 @@ type launcherSingleRunner struct {
ranModel string
}
func (r *launcherSingleRunner) Run(model string, args []string) error {
func (r *launcherSingleRunner) Run(model string, _ []LaunchModel, args []string) error {
r.ranModel = model
return nil
}
@@ -68,18 +69,21 @@ func (r *launcherRestorableRunner) RestoreSuccessMessage() string {
}
type launcherManagedRunner struct {
paths []string
currentModel string
configured []string
ranModel string
onboarded bool
onboardCalls int
onboardingComplete bool
refreshCalls int
refreshErr error
paths []string
currentModel string
configured []string
ranModel string
onboarded bool
onboardCalls int
onboardingComplete bool
refreshCalls int
refreshErr error
restoreHint string
configSuccessMessage string
skipModelReadiness bool
}
func (r *launcherManagedRunner) Run(model string, args []string) error {
func (r *launcherManagedRunner) Run(model string, _ []LaunchModel, args []string) error {
r.ranModel = model
return nil
}
@@ -110,6 +114,14 @@ func (r *launcherManagedRunner) RefreshRuntimeAfterConfigure() error {
return r.refreshErr
}
func (r *launcherManagedRunner) RestoreHint() string { return r.restoreHint }
func (r *launcherManagedRunner) ConfigurationSuccessMessage() string {
return r.configSuccessMessage
}
func (r *launcherManagedRunner) SkipModelReadiness() bool { return r.skipModelReadiness }
type launcherHeadlessManagedRunner struct {
launcherManagedRunner
}
@@ -121,8 +133,8 @@ type launcherManagedListRunner struct {
configuredModelLists [][]string
}
func (r *launcherManagedListRunner) ConfigureWithModels(primary string, models []string) error {
r.configuredModelLists = append(r.configuredModelLists, append([]string(nil), models...))
func (r *launcherManagedListRunner) ConfigureWithModels(primary string, models []LaunchModel) error {
r.configuredModelLists = append(r.configuredModelLists, launchModelNames(models))
return r.Configure(primary)
}
@@ -480,6 +492,116 @@ func TestLaunchIntegration_ManagedSingleIntegrationConfigOnlySkipsFinalRun(t *te
}
}
func TestLaunchIntegration_ManagedSingleIntegrationPrintsConfigurationSuccessAfterConfigure(t *testing.T) {
tmpDir := t.TempDir()
setLaunchTestHome(t, tmpDir)
withInteractiveSession(t, true)
withLauncherHooks(t)
runner := &launcherManagedRunner{
configSuccessMessage: "configured successfully\nrestore via success message",
restoreHint: "run restore command",
skipModelReadiness: true,
}
withIntegrationOverride(t, "stubmanaged", runner)
stderr := captureStderr(t, func() {
if err := LaunchIntegration(context.Background(), IntegrationLaunchRequest{
Name: "stubmanaged",
ModelOverride: "gemma4",
ForceConfigure: true,
ConfigureOnly: true,
}); err != nil {
t.Fatalf("LaunchIntegration returned error: %v", err)
}
})
if diff := compareStrings(runner.configured, []string{"gemma4"}); diff != "" {
t.Fatalf("configured models mismatch: %s", diff)
}
if !strings.Contains(stderr, "configured successfully") {
t.Fatalf("expected configuration success in stderr, got %q", stderr)
}
if !strings.Contains(stderr, "restore via success message") {
t.Fatalf("expected restore guidance in configuration success, got %q", stderr)
}
if strings.Contains(stderr, "run restore command") {
t.Fatalf("restore hint should not print separately after configure, got %q", stderr)
}
}
func TestLaunchIntegration_ManagedSingleIntegrationDoesNotPrintRestoreHintWhenUnchanged(t *testing.T) {
tmpDir := t.TempDir()
setLaunchTestHome(t, tmpDir)
withInteractiveSession(t, true)
withLauncherHooks(t)
runner := &launcherManagedRunner{
currentModel: "gemma4",
onboardingComplete: true,
configSuccessMessage: "configured successfully",
restoreHint: "run restore command",
skipModelReadiness: true,
}
withIntegrationOverride(t, "stubmanaged", runner)
if err := config.SaveIntegration("stubmanaged", []string{"gemma4"}); err != nil {
t.Fatalf("failed to save managed integration config: %v", err)
}
if err := config.MarkIntegrationOnboarded("stubmanaged"); err != nil {
t.Fatalf("failed to mark integration onboarded: %v", err)
}
stderr := captureStderr(t, func() {
if err := LaunchIntegration(context.Background(), IntegrationLaunchRequest{Name: "stubmanaged"}); err != nil {
t.Fatalf("LaunchIntegration returned error: %v", err)
}
})
if len(runner.configured) != 0 {
t.Fatalf("expected Configure to be skipped when saved matches, got %v", runner.configured)
}
if strings.Contains(stderr, "configured successfully") {
t.Fatalf("configuration success should not print when config is unchanged, got %q", stderr)
}
if strings.Contains(stderr, "run restore command") {
t.Fatalf("restore hint should not print when config is unchanged, got %q", stderr)
}
}
func TestLaunchIntegration_ManagedSingleIntegrationForceConfigureUsesModelOverride(t *testing.T) {
tmpDir := t.TempDir()
setLaunchTestHome(t, tmpDir)
withInteractiveSession(t, true)
withLauncherHooks(t)
runner := &launcherManagedRunner{
paths: nil,
skipModelReadiness: true,
}
withIntegrationOverride(t, "stubmanaged", runner)
DefaultSingleSelector = func(title string, items []SelectionItem, current string) (string, error) {
return "", fmt.Errorf("selector should not run with an explicit model override")
}
if err := LaunchIntegration(context.Background(), IntegrationLaunchRequest{
Name: "stubmanaged",
ModelOverride: "gemma4",
ForceConfigure: true,
ConfigureOnly: true,
}); err != nil {
t.Fatalf("LaunchIntegration returned error: %v", err)
}
if diff := compareStrings(runner.configured, []string{"gemma4"}); diff != "" {
t.Fatalf("configured models mismatch: %s", diff)
}
if runner.ranModel != "" {
t.Fatalf("expected configure-only flow to skip final launch, got %q", runner.ranModel)
}
}
func TestLaunchIntegration_ManagedSingleIntegrationSkipsRewriteWhenSavedMatches(t *testing.T) {
tmpDir := t.TempDir()
setLaunchTestHome(t, tmpDir)
@@ -505,7 +627,9 @@ func TestLaunchIntegration_ManagedSingleIntegrationSkipsRewriteWhenSavedMatches(
t.Fatalf("failed to save managed integration config: %v", err)
}
runner := &launcherManagedRunner{}
runner := &launcherManagedRunner{
currentModel: "gemma4",
}
withIntegrationOverride(t, "stubmanaged", runner)
DefaultSingleSelector = func(title string, items []SelectionItem, current string) (string, error) {
@@ -532,6 +656,53 @@ func TestLaunchIntegration_ManagedSingleIntegrationSkipsRewriteWhenSavedMatches(
}
}
func TestLaunchIntegration_ManagedSingleIntegrationRewritesWhenSavedMatchesButLiveConfigMissing(t *testing.T) {
tmpDir := t.TempDir()
setLaunchTestHome(t, tmpDir)
withInteractiveSession(t, true)
withLauncherHooks(t)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/api/show":
fmt.Fprint(w, `{"model_info":{"general.context_length":131072}}`)
default:
http.NotFound(w, r)
}
}))
defer srv.Close()
t.Setenv("OLLAMA_HOST", srv.URL)
if err := config.SaveIntegration("stubmanaged", []string{"gemma4"}); err != nil {
t.Fatalf("failed to save managed integration config: %v", err)
}
runner := &launcherManagedRunner{}
withIntegrationOverride(t, "stubmanaged", runner)
DefaultSingleSelector = func(title string, items []SelectionItem, current string) (string, error) {
t.Fatal("selector should not be called when saved model is usable")
return "", nil
}
DefaultConfirmPrompt = func(prompt string, options ConfirmOptions) (bool, error) {
return true, nil
}
if err := LaunchIntegration(context.Background(), IntegrationLaunchRequest{Name: "stubmanaged"}); err != nil {
t.Fatalf("LaunchIntegration returned error: %v", err)
}
if diff := compareStrings(runner.configured, []string{"gemma4"}); diff != "" {
t.Fatalf("expected Configure to rewrite missing live config: %s", diff)
}
if runner.refreshCalls != 1 {
t.Fatalf("expected runtime refresh once after rewrite, got %d", runner.refreshCalls)
}
if runner.ranModel != "gemma4" {
t.Fatalf("expected launch to run saved model, got %q", runner.ranModel)
}
}
func TestLaunchIntegration_ManagedSingleIntegrationRewritesWhenSavedDiffers(t *testing.T) {
tmpDir := t.TempDir()
setLaunchTestHome(t, tmpDir)
@@ -1292,6 +1463,53 @@ func TestBuildLauncherState_ToleratesInventoryFailure(t *testing.T) {
}
}
func TestBuildLauncherState_UsesTagsInventoryWithoutShow(t *testing.T) {
tmpDir := t.TempDir()
setLaunchTestHome(t, tmpDir)
if err := config.SetLastModel("llama3.2"); err != nil {
t.Fatalf("failed to seed last model: %v", err)
}
if err := config.SaveIntegration("codex", []string{"qwen3:8b"}); err != nil {
t.Fatalf("failed to seed codex config: %v", err)
}
var showCalls atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/api/tags":
fmt.Fprint(w, `{"models":[`+
`{"name":"llama3.2","capabilities":["completion","tools"],"context_length":131072,"size":3200000000},`+
`{"name":"qwen3:8b","capabilities":["completion","tools"],"context_length":65536,"size":4500000000}`+
`]}`)
case "/api/show":
showCalls.Add(1)
fmt.Fprint(w, `{"model_info":{"general.context_length":131072}}`)
default:
http.NotFound(w, r)
}
}))
defer srv.Close()
t.Setenv("OLLAMA_HOST", srv.URL)
state, err := BuildLauncherState(context.Background())
if err != nil {
t.Fatalf("BuildLauncherState returned error: %v", err)
}
if !state.RunModelUsable {
t.Fatal("expected saved run model to be usable from tags inventory")
}
if state.Integrations["codex"].CurrentModel != "qwen3:8b" {
t.Fatalf("expected codex current model from saved config, got %q", state.Integrations["codex"].CurrentModel)
}
if !state.Integrations["codex"].ModelUsable {
t.Fatal("expected saved codex model to be usable from tags inventory")
}
if got := showCalls.Load(); got != 0 {
t.Fatalf("show calls = %d, want 0 for broad launcher state", got)
}
}
func TestResolveRunModel_UsesSavedModelWithoutSelector(t *testing.T) {
tmpDir := t.TempDir()
setLaunchTestHome(t, tmpDir)
+201
View File
@@ -0,0 +1,201 @@
package launch
import (
"context"
"slices"
"strings"
"sync"
"github.com/ollama/ollama/api"
modelpkg "github.com/ollama/ollama/types/model"
)
// LaunchModel is the model metadata Launch passes to integration config
// writers after resolving selected model names through the per-run inventory.
type LaunchModel struct {
Name string
Remote bool
ToolCapable bool
Capabilities []modelpkg.Capability
ContextLength int
MaxOutputTokens int
EmbeddingLength int
Size int64
Details api.ModelDetails
}
type modelInfo = LaunchModel
// ModelInfo re-exports launcher model inventory details for callers.
type ModelInfo = LaunchModel
func (m LaunchModel) HasCapability(capability modelpkg.Capability) bool {
return slices.Contains(m.Capabilities, capability)
}
func (m LaunchModel) WithCloudLimits() LaunchModel {
if limit, ok := lookupCloudModelLimit(m.Name); ok {
if m.ContextLength <= 0 {
m.ContextLength = limit.Context
}
if m.MaxOutputTokens <= 0 {
m.MaxOutputTokens = limit.Output
}
}
return m
}
type modelInventory struct {
client *api.Client
mu sync.Mutex
loaded bool
models []LaunchModel
err error
}
func newModelInventory(client *api.Client) *modelInventory {
return &modelInventory{client: client}
}
func (i *modelInventory) Load(ctx context.Context) ([]LaunchModel, error) {
return i.load(ctx, false)
}
func (i *modelInventory) Refresh(ctx context.Context) ([]LaunchModel, error) {
return i.load(ctx, true)
}
func (i *modelInventory) load(ctx context.Context, force bool) ([]LaunchModel, error) {
if i == nil || i.client == nil {
return nil, nil
}
i.mu.Lock()
defer i.mu.Unlock()
if i.loaded && !force {
return cloneLaunchModels(i.models), i.err
}
resp, err := i.client.List(ctx)
if err != nil {
i.models = nil
i.err = err
i.loaded = true
return nil, err
}
i.models = make([]LaunchModel, 0, len(resp.Models))
for _, model := range resp.Models {
i.models = append(i.models, launchModelFromListResponse(model))
}
i.err = nil
i.loaded = true
return cloneLaunchModels(i.models), i.err
}
func (i *modelInventory) Resolve(ctx context.Context, names []string) []LaunchModel {
names = dedupeModelList(names)
if len(names) == 0 {
return nil
}
models, err := i.Load(ctx)
if err != nil {
models = nil
}
resolved, localMiss := resolveLaunchModels(names, models)
if localMiss {
if refreshed, err := i.Refresh(ctx); err == nil {
resolved, _ = resolveLaunchModels(names, refreshed)
}
}
return resolved
}
func resolveLaunchModels(names []string, models []LaunchModel) ([]LaunchModel, bool) {
resolved := make([]LaunchModel, 0, len(names))
localMiss := false
for _, name := range names {
if model, ok := findLaunchModel(models, name); ok {
resolved = append(resolved, model.WithCloudLimits())
continue
}
if !isCloudModelName(name) {
localMiss = true
}
resolved = append(resolved, fallbackLaunchModel(name))
}
return resolved, localMiss
}
func launchModelFromListResponse(model api.ListModelResponse) LaunchModel {
return LaunchModel{
Name: model.Name,
Remote: model.RemoteModel != "",
ToolCapable: slices.Contains(model.Capabilities, modelpkg.CapabilityTools),
Capabilities: append([]modelpkg.Capability(nil), model.Capabilities...),
ContextLength: model.Details.ContextLength,
EmbeddingLength: model.Details.EmbeddingLength,
Size: model.Size,
Details: model.Details,
}.WithCloudLimits()
}
func fallbackLaunchModel(name string) LaunchModel {
return LaunchModel{Name: name, Remote: isCloudModelName(name)}.WithCloudLimits()
}
func findLaunchModel(models []LaunchModel, name string) (LaunchModel, bool) {
for _, model := range models {
if launchModelMatches(model.Name, name) {
return cloneLaunchModel(model), true
}
}
return LaunchModel{}, false
}
func launchModelMatches(candidate, name string) bool {
if candidate == name {
return true
}
return strings.TrimSuffix(candidate, ":latest") == name
}
func cloneLaunchModel(model LaunchModel) LaunchModel {
model.Capabilities = append([]modelpkg.Capability(nil), model.Capabilities...)
model.Details.Families = append([]string(nil), model.Details.Families...)
return model
}
func cloneLaunchModels(models []LaunchModel) []LaunchModel {
cloned := make([]LaunchModel, len(models))
for i, model := range models {
cloned[i] = cloneLaunchModel(model)
}
return cloned
}
func launchModelNames(models []LaunchModel) []string {
names := make([]string, 0, len(models))
for _, model := range models {
if model.Name != "" {
names = append(names, model.Name)
}
}
return names
}
func launchModelsFromNames(names []string) []LaunchModel {
models := make([]LaunchModel, 0, len(names))
for _, name := range names {
if name == "" {
continue
}
models = append(models, fallbackLaunchModel(name))
}
return models
}
+80
View File
@@ -0,0 +1,80 @@
package launch
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"net/url"
"testing"
"github.com/ollama/ollama/api"
modelpkg "github.com/ollama/ollama/types/model"
)
func TestModelInventoryResolveRefreshesLocalMiss(t *testing.T) {
calls := 0
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/api/tags" {
http.NotFound(w, r)
return
}
calls++
if calls == 1 {
fmt.Fprint(w, `{"models":[]}`)
return
}
fmt.Fprint(w, `{"models":[{"name":"new-model","size":123,"details":{"context_length":65536,"embedding_length":1024},"capabilities":["vision","tools"]}]}`)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
inventory := newModelInventory(api.NewClient(u, srv.Client()))
got := inventory.Resolve(context.Background(), []string{"new-model"})
if calls != 2 {
t.Fatalf("List calls = %d, want 2", calls)
}
if len(got) != 1 {
t.Fatalf("Resolve returned %d models, want 1", len(got))
}
if got[0].Name != "new-model" {
t.Fatalf("Name = %q, want new-model", got[0].Name)
}
if got[0].ContextLength != 65_536 || got[0].EmbeddingLength != 1_024 {
t.Fatalf("metadata = context %d embedding %d, want refreshed metadata", got[0].ContextLength, got[0].EmbeddingLength)
}
if !got[0].HasCapability(modelpkg.CapabilityVision) || !got[0].ToolCapable {
t.Fatalf("capabilities = %v toolCapable=%v, want refreshed capabilities", got[0].Capabilities, got[0].ToolCapable)
}
}
func TestModelInventoryResolveDoesNotRefreshCloudMiss(t *testing.T) {
calls := 0
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/api/tags" {
http.NotFound(w, r)
return
}
calls++
fmt.Fprint(w, `{"models":[]}`)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
inventory := newModelInventory(api.NewClient(u, srv.Client()))
got := inventory.Resolve(context.Background(), []string{"glm-5.1:cloud"})
if calls != 1 {
t.Fatalf("List calls = %d, want 1", calls)
}
if len(got) != 1 {
t.Fatalf("Resolve returned %d models, want 1", len(got))
}
if got[0].Name != "glm-5.1:cloud" || !got[0].Remote {
t.Fatalf("resolved model = %#v, want cloud fallback", got[0])
}
if got[0].ContextLength <= 0 || got[0].MaxOutputTokens <= 0 {
t.Fatalf("cloud limits not applied: %#v", got[0])
}
}
+21 -13
View File
@@ -23,10 +23,10 @@ import (
)
var recommendedModels = []ModelItem{
{Name: "kimi-k2.6:cloud", Description: "State-of-the-art coding, long-horizon execution, and multimodal agent swarm capability", Recommended: true, ContextLength: 262_144, MaxOutputTokens: 262_144},
{Name: "qwen3.5:cloud", Description: "Reasoning, coding, and agentic tool use with vision", Recommended: true, ContextLength: 262_144, MaxOutputTokens: 32_768},
{Name: "glm-5.1:cloud", Description: "Reasoning and code generation", Recommended: true, ContextLength: 202_752, MaxOutputTokens: 131_072},
{Name: "minimax-m2.7:cloud", Description: "Fast, efficient coding and real-world productivity", Recommended: true, ContextLength: 204_800, MaxOutputTokens: 128_000},
{Name: "kimi-k2.6:cloud", Description: "State-of-the-art coding, long-horizon execution, and multimodal agent swarm capability", Recommended: true, Details: api.ModelDetails{ContextLength: 262_144}, MaxOutputTokens: 262_144},
{Name: "qwen3.5:cloud", Description: "Reasoning, coding, and agentic tool use with vision", Recommended: true, Details: api.ModelDetails{ContextLength: 262_144}, MaxOutputTokens: 32_768},
{Name: "glm-5.1:cloud", Description: "Reasoning and code generation", Recommended: true, Details: api.ModelDetails{ContextLength: 202_752}, MaxOutputTokens: 131_072},
{Name: "minimax-m2.7:cloud", Description: "Fast, efficient coding and real-world productivity", Recommended: true, Details: api.ModelDetails{ContextLength: 204_800}, MaxOutputTokens: 128_000},
{Name: "gemma4", Description: "Reasoning and code generation locally", Recommended: true, VRAMBytes: 12 * format.GigaByte},
{Name: "qwen3.5", Description: "Reasoning, coding, and visual understanding locally", Recommended: true, VRAMBytes: 14 * format.GigaByte},
}
@@ -115,7 +115,7 @@ func setDynamicCloudModelLimits(limits map[string]cloudModelLimit) {
func cloudModelLimitsFromRecommendations(recommendations []ModelItem) map[string]cloudModelLimit {
limits := make(map[string]cloudModelLimit, len(recommendations))
for _, rec := range recommendations {
if !isCloudModelName(rec.Name) || rec.ContextLength <= 0 || rec.MaxOutputTokens <= 0 {
if !isCloudModelName(rec.Name) || rec.Details.ContextLength <= 0 || rec.MaxOutputTokens <= 0 {
continue
}
base, stripped := modelref.StripCloudSourceTag(rec.Name)
@@ -123,7 +123,7 @@ func cloudModelLimitsFromRecommendations(recommendations []ModelItem) map[string
continue
}
limits[base] = cloudModelLimit{
Context: rec.ContextLength,
Context: rec.Details.ContextLength,
Output: rec.MaxOutputTokens,
}
}
@@ -299,18 +299,17 @@ func pullMissingModel(ctx context.Context, client *api.Client, model string) err
}
// prepareEditorIntegration persists models and applies editor-managed config files.
func prepareEditorIntegration(name string, editor Editor, models []string) error {
func prepareEditorIntegration(name string, editor Editor, models []LaunchModel) error {
if err := editor.Edit(models); err != nil {
return fmt.Errorf("setup failed: %w", err)
}
if err := config.SaveIntegration(name, models); err != nil {
if err := config.SaveIntegration(name, launchModelNames(models)); err != nil {
return fmt.Errorf("failed to save: %w", err)
}
return nil
}
func prepareManagedSingleIntegration(name string, managed ManagedSingleModel, model string, models []string) error {
models = dedupeModelList(append([]string{model}, models...))
func prepareManagedSingleIntegration(name string, managed ManagedSingleModel, model string, models []LaunchModel) error {
var err error
if withModels, ok := managed.(ManagedModelListConfigurer); ok {
err = withModels.ConfigureWithModels(model, models)
@@ -365,11 +364,11 @@ func buildModelListWithRecommendations(existing []modelInfo, recommendations []M
}
displayName := strings.TrimSuffix(m.Name, ":latest")
existingModels[displayName] = true
item := ModelItem{Name: displayName, Recommended: recommended[displayName], Description: recDesc[displayName]}
if rec, ok := recByName[displayName]; ok {
item = copyModelRecommendationFields(displayName, rec)
items = append(items, modelItemFromInventory(displayName, m, copyModelRecommendationFields(displayName, rec)))
} else {
items = append(items, modelItemFromInventory(displayName, m, ModelItem{Name: displayName, Recommended: recommended[displayName], Description: recDesc[displayName]}))
}
items = append(items, item)
}
for _, rec := range recommendations {
@@ -483,6 +482,15 @@ func copyModelRecommendationFields(name string, rec ModelItem) ModelItem {
return rec
}
func modelItemFromInventory(name string, info modelInfo, item ModelItem) ModelItem {
item.Name = name
item.ToolCapable = info.ToolCapable
item.Capabilities = slices.Clone(info.Capabilities)
item.Size = info.Size
item.Details = info.Details
return item
}
// isCloudModelName reports whether the model name has an explicit cloud source.
func isCloudModelName(name string) bool {
return modelref.HasExplicitCloudSource(name)
+83
View File
@@ -0,0 +1,83 @@
package launch
import (
"testing"
"github.com/ollama/ollama/api"
"github.com/ollama/ollama/format"
modelpkg "github.com/ollama/ollama/types/model"
)
func TestBuildModelList_UsesInventoryMetadataForInstalledModels(t *testing.T) {
existing := []modelInfo{
{
Name: "custom-tools:latest",
ToolCapable: true,
Capabilities: []modelpkg.Capability{modelpkg.CapabilityCompletion, modelpkg.CapabilityTools, modelpkg.CapabilityThinking},
Size: 7500 * format.MegaByte,
Details: api.ModelDetails{
ParameterSize: "8B",
QuantizationLevel: "Q4_K_M",
ContextLength: 131_072,
EmbeddingLength: 4096,
},
},
}
items, _, _, _ := buildModelList(existing, nil, "")
var got ModelItem
for _, item := range items {
if item.Name == "custom-tools" {
got = item
break
}
}
if got.Name == "" {
t.Fatal("custom-tools not found in items")
}
if !got.ToolCapable {
t.Fatal("expected installed model to preserve tool capability from tags metadata")
}
if got.Details.ContextLength != 131_072 {
t.Fatalf("Details.ContextLength = %d, want 131072", got.Details.ContextLength)
}
if got.Size != 7500*format.MegaByte {
t.Fatalf("Size = %d, want %d", got.Size, 7500*format.MegaByte)
}
if got.Description != "" {
t.Fatalf("Description = %q, want empty for installed model without recommendation copy", got.Description)
}
}
func TestBuildModelList_InstalledRecommendedPreservesRecommendationAndMetadata(t *testing.T) {
existing := []modelInfo{
{
Name: "qwen3.5",
ToolCapable: true,
Capabilities: []modelpkg.Capability{modelpkg.CapabilityCompletion, modelpkg.CapabilityTools, modelpkg.CapabilityVision},
Size: 14 * format.GigaByte,
Details: api.ModelDetails{ContextLength: 262_144},
},
}
items, _, _, _ := buildModelList(existing, nil, "")
var got ModelItem
for _, item := range items {
if item.Name == "qwen3.5" {
got = item
break
}
}
if got.Name == "" {
t.Fatal("qwen3.5 not found in items")
}
if !got.Recommended || !got.ToolCapable {
t.Fatalf("recommended/tool metadata = %v/%v, want true/true", got.Recommended, got.ToolCapable)
}
if got.Details.ContextLength != 262_144 {
t.Fatalf("Details.ContextLength = %d, want 262144", got.Details.ContextLength)
}
if got.Description != "Reasoning, coding, and visual understanding locally" {
t.Fatalf("Description = %q, want recommendation description", got.Description)
}
}
+17 -55
View File
@@ -1,7 +1,6 @@
package launch
import (
"context"
"encoding/json"
"fmt"
"net"
@@ -10,21 +9,15 @@ import (
"os/exec"
"path/filepath"
"runtime"
"slices"
"strings"
"time"
"github.com/ollama/ollama/api"
"github.com/ollama/ollama/cmd/internal/fileutil"
"github.com/ollama/ollama/envconfig"
"github.com/ollama/ollama/types/model"
)
const defaultGatewayPort = 18789
// Bound model capability probing so launch/config cannot hang on slow/unreachable API calls.
var openclawModelShowTimeout = 5 * time.Second
// openclawFreshInstall is set to true when ensureOpenclawInstalled performs an install
var openclawFreshInstall bool
@@ -34,7 +27,7 @@ type Openclaw struct{}
func (c *Openclaw) String() string { return "OpenClaw" }
func (c *Openclaw) Run(model string, args []string) error {
func (c *Openclaw) Run(model string, _ []LaunchModel, args []string) error {
bin, err := ensureOpenclawInstalled()
if err != nil {
return err
@@ -656,7 +649,7 @@ func (c *Openclaw) Paths() []string {
return nil
}
func (c *Openclaw) Edit(models []string) error {
func (c *Openclaw) Edit(models []LaunchModel) error {
if len(models) == 0 {
return nil
}
@@ -710,13 +703,11 @@ func (c *Openclaw) Edit(models []string) error {
}
}
client, _ := api.ClientFromEnvironment()
var newModels []any
for _, m := range models {
entry, _ := openclawModelConfig(context.Background(), client, m)
entry, _ := openclawModelConfig(m)
// Merge existing fields (user customizations)
if existing, ok := existingByID[m]; ok {
if existing, ok := existingByID[m.Name]; ok {
for k, v := range existing {
if _, isNew := entry[k]; !isNew {
entry[k] = v
@@ -744,7 +735,7 @@ func (c *Openclaw) Edit(models []string) error {
if modelConfig == nil {
modelConfig = make(map[string]any)
}
modelConfig["primary"] = "ollama/" + models[0]
modelConfig["primary"] = "ollama/" + models[0].Name
defaults["model"] = modelConfig
agents["defaults"] = defaults
config["agents"] = agents
@@ -759,7 +750,7 @@ func (c *Openclaw) Edit(models []string) error {
// Clear any per-session model overrides so the new primary takes effect
// immediately rather than being shadowed by a cached modelOverride.
clearSessionModelOverride(models[0])
clearSessionModelOverride(models[0].Name)
return nil
}
@@ -936,10 +927,10 @@ func configureOllamaWebSearch() {
// openclawModelConfig builds an OpenClaw model config entry with capability detection.
// The second return value indicates whether the model is a cloud (remote) model.
func openclawModelConfig(ctx context.Context, client *api.Client, modelID string) (map[string]any, bool) {
func openclawModelConfig(model LaunchModel) (map[string]any, bool) {
entry := map[string]any{
"id": modelID,
"name": modelID,
"id": model.Name,
"name": model.Name,
"input": []any{"text"},
"cost": map[string]any{
"input": 0,
@@ -949,53 +940,24 @@ func openclawModelConfig(ctx context.Context, client *api.Client, modelID string
},
}
if client == nil {
return entry, false
}
showCtx := ctx
if _, hasDeadline := ctx.Deadline(); !hasDeadline {
var cancel context.CancelFunc
showCtx, cancel = context.WithTimeout(ctx, openclawModelShowTimeout)
defer cancel()
}
resp, err := client.Show(showCtx, &api.ShowRequest{Model: modelID})
if err != nil {
return entry, false
}
// Set input types based on vision capability
if slices.Contains(resp.Capabilities, model.CapabilityVision) {
if model.HasCapability("vision") {
entry["input"] = []any{"text", "image"}
}
// Set reasoning based on thinking capability
if slices.Contains(resp.Capabilities, model.CapabilityThinking) {
if model.HasCapability("thinking") {
entry["reasoning"] = true
}
// Cloud models: use hardcoded limits for context/output tokens.
// Capability detection above still applies (vision, thinking).
if resp.RemoteModel != "" {
if l, ok := lookupCloudModelLimit(modelID); ok {
entry["contextWindow"] = l.Context
entry["maxTokens"] = l.Output
}
return entry, true
if model.ContextLength > 0 {
entry["contextWindow"] = model.ContextLength
}
if model.MaxOutputTokens > 0 {
entry["maxTokens"] = model.MaxOutputTokens
}
// Extract context window from ModelInfo (local models only)
for key, val := range resp.ModelInfo {
if strings.HasSuffix(key, ".context_length") {
if ctxLen, ok := val.(float64); ok && ctxLen > 0 {
entry["contextWindow"] = int(ctxLen)
}
break
}
}
return entry, false
return entry, model.Remote || isCloudModelName(model.Name)
}
func (c *Openclaw) Models() []string {
+62 -211
View File
@@ -2,12 +2,9 @@ package launch
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
@@ -16,8 +13,8 @@ import (
"testing"
"time"
"github.com/ollama/ollama/api"
"github.com/ollama/ollama/cmd/internal/fileutil"
"github.com/ollama/ollama/types/model"
)
func TestOpenclawIntegration(t *testing.T) {
@@ -78,7 +75,7 @@ func TestOpenclawRunPassthroughArgs(t *testing.T) {
defer func() { DefaultConfirmPrompt = oldConfirmPrompt }()
c := &Openclaw{}
if err := c.Run("llama3.2", []string{"gateway", "--someflag"}); err != nil {
if err := c.Run("llama3.2", nil, []string{"gateway", "--someflag"}); err != nil {
t.Fatalf("Run() error = %v", err)
}
@@ -152,7 +149,7 @@ fi
defer func() { DefaultConfirmPrompt = oldConfirmPrompt }()
c := &Openclaw{}
if err := c.Run("llama3.2", nil); err != nil {
if err := c.Run("llama3.2", nil, nil); err != nil {
t.Fatalf("Run() error = %v", err)
}
@@ -224,7 +221,7 @@ func TestOpenclawRun_SetupLaterContinuesToGatewayAndTUI(t *testing.T) {
defer func() { DefaultConfirmPrompt = oldConfirmPrompt }()
c := &Openclaw{}
if err := c.Run("llama3.2", nil); err != nil {
if err := c.Run("llama3.2", nil, nil); err != nil {
t.Fatalf("Run() error = %v", err)
}
@@ -287,7 +284,7 @@ exit 0
defer func() { DefaultConfirmPrompt = oldConfirmPrompt }()
c := &Openclaw{}
if err := c.Run("llama3.2", []string{"status"}); err != nil {
if err := c.Run("llama3.2", nil, []string{"status"}); err != nil {
t.Fatalf("Run() error = %v", err)
}
@@ -367,7 +364,7 @@ exit 0
defer func() { DefaultConfirmPrompt = oldConfirmPrompt }()
c := &Openclaw{}
if err := c.Run("llama3.2", []string{"tui"}); err != nil {
if err := c.Run("llama3.2", nil, []string{"tui"}); err != nil {
t.Fatalf("Run() error = %v", err)
}
@@ -617,7 +614,7 @@ func TestOpenclawEdit(t *testing.T) {
t.Run("fresh install", func(t *testing.T) {
cleanup()
if err := c.Edit([]string{"llama3.2"}); err != nil {
if err := c.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
assertOpenclawModelExists(t, configPath, "llama3.2")
@@ -626,7 +623,7 @@ func TestOpenclawEdit(t *testing.T) {
t.Run("multiple models - first is primary", func(t *testing.T) {
cleanup()
if err := c.Edit([]string{"llama3.2", "mistral"}); err != nil {
if err := c.Edit(testLaunchModels("llama3.2", "mistral")); err != nil {
t.Fatal(err)
}
assertOpenclawModelExists(t, configPath, "llama3.2")
@@ -638,7 +635,7 @@ func TestOpenclawEdit(t *testing.T) {
cleanup()
os.MkdirAll(configDir, 0o755)
os.WriteFile(configPath, []byte(`{"models":{"providers":{"anthropic":{"apiKey":"xxx"}}}}`), 0o644)
if err := c.Edit([]string{"llama3.2"}); err != nil {
if err := c.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
data, _ := os.ReadFile(configPath)
@@ -655,7 +652,7 @@ func TestOpenclawEdit(t *testing.T) {
cleanup()
os.MkdirAll(configDir, 0o755)
os.WriteFile(configPath, []byte(`{"theme":"dark","mcp":{"servers":{}}}`), 0o644)
if err := c.Edit([]string{"llama3.2"}); err != nil {
if err := c.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
data, _ := os.ReadFile(configPath)
@@ -671,7 +668,7 @@ func TestOpenclawEdit(t *testing.T) {
t.Run("preserve user customizations on models", func(t *testing.T) {
cleanup()
c.Edit([]string{"llama3.2"})
c.Edit(testLaunchModels("llama3.2"))
// User adds custom field
data, _ := os.ReadFile(configPath)
@@ -687,7 +684,7 @@ func TestOpenclawEdit(t *testing.T) {
os.WriteFile(configPath, configData, 0o644)
// Re-run Edit
c.Edit([]string{"llama3.2"})
c.Edit(testLaunchModels("llama3.2"))
data, _ = os.ReadFile(configPath)
json.Unmarshal(data, &cfg)
@@ -703,8 +700,8 @@ func TestOpenclawEdit(t *testing.T) {
t.Run("edit replaces models list", func(t *testing.T) {
cleanup()
c.Edit([]string{"llama3.2", "mistral"})
c.Edit([]string{"llama3.2"})
c.Edit(testLaunchModels("llama3.2", "mistral"))
c.Edit(testLaunchModels("llama3.2"))
assertOpenclawModelExists(t, configPath, "llama3.2")
assertOpenclawModelNotExists(t, configPath, "mistral")
@@ -716,7 +713,7 @@ func TestOpenclawEdit(t *testing.T) {
original := `{"existing":"data"}`
os.WriteFile(configPath, []byte(original), 0o644)
c.Edit([]string{})
c.Edit(testLaunchModels())
data, _ := os.ReadFile(configPath)
if string(data) != original {
@@ -729,7 +726,7 @@ func TestOpenclawEdit(t *testing.T) {
os.MkdirAll(configDir, 0o755)
os.WriteFile(configPath, []byte(`{corrupted`), 0o644)
if err := c.Edit([]string{"llama3.2"}); err != nil {
if err := c.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
@@ -745,7 +742,7 @@ func TestOpenclawEdit(t *testing.T) {
os.MkdirAll(configDir, 0o755)
os.WriteFile(configPath, []byte(`{"models":"not a map"}`), 0o644)
if err := c.Edit([]string{"llama3.2"}); err != nil {
if err := c.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
assertOpenclawModelExists(t, configPath, "llama3.2")
@@ -925,7 +922,7 @@ func TestOpenclawEditSchemaFields(t *testing.T) {
setTestHome(t, tmpDir)
configPath := filepath.Join(tmpDir, ".openclaw", "openclaw.json")
if err := c.Edit([]string{"llama3.2"}); err != nil {
if err := c.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
@@ -966,7 +963,7 @@ func TestOpenclawEditModelNames(t *testing.T) {
t.Run("model with colon tag", func(t *testing.T) {
cleanup()
if err := c.Edit([]string{"llama3.2:70b"}); err != nil {
if err := c.Edit(testLaunchModels("llama3.2:70b")); err != nil {
t.Fatal(err)
}
assertOpenclawModelExists(t, configPath, "llama3.2:70b")
@@ -975,7 +972,7 @@ func TestOpenclawEditModelNames(t *testing.T) {
t.Run("model with slash", func(t *testing.T) {
cleanup()
if err := c.Edit([]string{"library/model:tag"}); err != nil {
if err := c.Edit(testLaunchModels("library/model:tag")); err != nil {
t.Fatal(err)
}
assertOpenclawModelExists(t, configPath, "library/model:tag")
@@ -984,7 +981,7 @@ func TestOpenclawEditModelNames(t *testing.T) {
t.Run("model with hyphen", func(t *testing.T) {
cleanup()
if err := c.Edit([]string{"test-model"}); err != nil {
if err := c.Edit(testLaunchModels("test-model")); err != nil {
t.Fatal(err)
}
assertOpenclawModelExists(t, configPath, "test-model")
@@ -1004,7 +1001,7 @@ func TestOpenclawEditAgentsPreservation(t *testing.T) {
os.MkdirAll(configDir, 0o755)
os.WriteFile(configPath, []byte(`{"agents":{"defaults":{"model":{"primary":"old"},"temperature":0.7}}}`), 0o644)
c.Edit([]string{"llama3.2"})
c.Edit(testLaunchModels("llama3.2"))
data, _ := os.ReadFile(configPath)
var cfg map[string]any
@@ -1021,7 +1018,7 @@ func TestOpenclawEditAgentsPreservation(t *testing.T) {
os.MkdirAll(configDir, 0o755)
os.WriteFile(configPath, []byte(`{"agents":{"defaults":{},"custom-agent":{"foo":"bar"}}}`), 0o644)
c.Edit([]string{"llama3.2"})
c.Edit(testLaunchModels("llama3.2"))
data, _ := os.ReadFile(configPath)
var cfg map[string]any
@@ -1061,7 +1058,7 @@ func TestOpenclawEdit_RoundTrip(t *testing.T) {
os.MkdirAll(configDir, 0o755)
os.WriteFile(configPath, []byte(testOpenclawFixture), 0o644)
if err := c.Edit([]string{"llama3.2", "mistral"}); err != nil {
if err := c.Edit(testLaunchModels("llama3.2", "mistral")); err != nil {
t.Fatal(err)
}
@@ -1107,10 +1104,10 @@ func TestOpenclawEdit_Idempotent(t *testing.T) {
os.MkdirAll(configDir, 0o755)
os.WriteFile(configPath, []byte(testOpenclawFixture), 0o644)
c.Edit([]string{"llama3.2", "mistral"})
c.Edit(testLaunchModels("llama3.2", "mistral"))
firstData, _ := os.ReadFile(configPath)
c.Edit([]string{"llama3.2", "mistral"})
c.Edit(testLaunchModels("llama3.2", "mistral"))
secondData, _ := os.ReadFile(configPath)
if string(firstData) != string(secondData) {
@@ -1133,7 +1130,7 @@ func TestOpenclawEdit_MultipleConsecutiveEdits(t *testing.T) {
if i%2 == 0 {
models = []string{"model-x", "model-y", "model-z"}
}
if err := c.Edit(models); err != nil {
if err := c.Edit(launchModelsFromNames(models)); err != nil {
t.Fatalf("edit %d failed: %v", i, err)
}
}
@@ -1162,7 +1159,7 @@ func TestOpenclawEdit_BackupCreated(t *testing.T) {
original := fmt.Sprintf(`{"theme": "%s"}`, uniqueMarker)
os.WriteFile(configPath, []byte(original), 0o644)
if err := c.Edit([]string{"model-a"}); err != nil {
if err := c.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
@@ -1284,7 +1281,7 @@ func TestOpenclawLegacyPaths(t *testing.T) {
os.WriteFile(filepath.Join(newDir, "openclaw.json"), []byte(`{"theme":"new"}`), 0o644)
os.WriteFile(filepath.Join(legacyDir, "clawdbot.json"), []byte(`{"theme":"legacy"}`), 0o644)
if err := c.Edit([]string{"llama3.2"}); err != nil {
if err := c.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
@@ -1303,7 +1300,7 @@ func TestOpenclawLegacyPaths(t *testing.T) {
os.MkdirAll(legacyDir, 0o755)
os.WriteFile(filepath.Join(legacyDir, "clawdbot.json"), []byte(`{"theme":"dark"}`), 0o644)
if err := c.Edit([]string{"llama3.2"}); err != nil {
if err := c.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
@@ -1331,7 +1328,7 @@ func TestOpenclawEdit_CreatesDirectoryIfMissing(t *testing.T) {
t.Fatal("directory should not exist before test")
}
if err := c.Edit([]string{"model-a"}); err != nil {
if err := c.Edit(testLaunchModels("model-a")); err != nil {
t.Fatal(err)
}
@@ -2248,8 +2245,8 @@ func TestPrintOpenclawReady(t *testing.T) {
}
func TestOpenclawModelConfig(t *testing.T) {
t.Run("nil client returns base config", func(t *testing.T) {
cfg, _ := openclawModelConfig(context.Background(), nil, "llama3.2")
t.Run("minimal model returns base config", func(t *testing.T) {
cfg, _ := openclawModelConfig(fallbackLaunchModel("llama3.2"))
if cfg["id"] != "llama3.2" {
t.Errorf("id = %v, want llama3.2", cfg["id"])
@@ -2260,29 +2257,17 @@ func TestOpenclawModelConfig(t *testing.T) {
if cfg["cost"] == nil {
t.Error("cost should be set")
}
// Should not have capability fields without API
// Should not have capability fields without inventory metadata.
if _, ok := cfg["reasoning"]; ok {
t.Error("reasoning should not be set without API")
t.Error("reasoning should not be set without metadata")
}
if _, ok := cfg["contextWindow"]; ok {
t.Error("contextWindow should not be set without API")
t.Error("contextWindow should not be set without metadata")
}
})
t.Run("sets vision input when model has vision capability", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":["vision"],"model_info":{"llama.context_length":4096}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg, _ := openclawModelConfig(context.Background(), client, "llava:7b")
cfg, _ := openclawModelConfig(LaunchModel{Name: "llava:7b", Capabilities: []model.Capability{"vision"}, ContextLength: 4096})
input, ok := cfg["input"].([]any)
if !ok || len(input) != 2 {
@@ -2291,19 +2276,7 @@ func TestOpenclawModelConfig(t *testing.T) {
})
t.Run("sets text-only input when model lacks vision", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":["completion"],"model_info":{}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg, _ := openclawModelConfig(context.Background(), client, "llama3.2")
cfg, _ := openclawModelConfig(LaunchModel{Name: "llama3.2", Capabilities: []model.Capability{"completion"}})
input, ok := cfg["input"].([]any)
if !ok || len(input) != 1 {
@@ -2315,39 +2288,15 @@ func TestOpenclawModelConfig(t *testing.T) {
})
t.Run("sets reasoning when model has thinking capability", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":["thinking"],"model_info":{}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg, _ := openclawModelConfig(context.Background(), client, "qwq")
cfg, _ := openclawModelConfig(LaunchModel{Name: "qwq", Capabilities: []model.Capability{"thinking"}})
if cfg["reasoning"] != true {
t.Error("expected reasoning = true for thinking model")
}
})
t.Run("extracts context window from model info", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":[],"model_info":{"llama.context_length":131072}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg, _ := openclawModelConfig(context.Background(), client, "llama3.2")
t.Run("sets context window from inventory metadata", func(t *testing.T) {
cfg, _ := openclawModelConfig(LaunchModel{Name: "llama3.2", ContextLength: 131072})
if cfg["contextWindow"] != 131072 {
t.Errorf("contextWindow = %v, want 131072", cfg["contextWindow"])
@@ -2355,19 +2304,11 @@ func TestOpenclawModelConfig(t *testing.T) {
})
t.Run("handles all capabilities together", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":["vision","thinking"],"model_info":{"qwen3.context_length":32768}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg, _ := openclawModelConfig(context.Background(), client, "qwen3-vision")
cfg, _ := openclawModelConfig(LaunchModel{
Name: "qwen3-vision",
Capabilities: []model.Capability{"vision", "thinking"},
ContextLength: 32768,
})
input, ok := cfg["input"].([]any)
if !ok || len(input) != 2 {
@@ -2381,17 +2322,8 @@ func TestOpenclawModelConfig(t *testing.T) {
}
})
t.Run("returns base config when show fails", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound)
fmt.Fprintf(w, `{"error":"model not found"}`)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg, _ := openclawModelConfig(context.Background(), client, "missing-model")
t.Run("returns base config when metadata is unavailable", func(t *testing.T) {
cfg, _ := openclawModelConfig(fallbackLaunchModel("missing-model"))
if cfg["id"] != "missing-model" {
t.Errorf("id = %v, want missing-model", cfg["id"])
@@ -2401,62 +2333,15 @@ func TestOpenclawModelConfig(t *testing.T) {
t.Error("input should always be set")
}
if _, ok := cfg["reasoning"]; ok {
t.Error("reasoning should not be set when show fails")
t.Error("reasoning should not be set when metadata is unavailable")
}
if _, ok := cfg["contextWindow"]; ok {
t.Error("contextWindow should not be set when show fails")
}
})
t.Run("times out slow show and returns base config", func(t *testing.T) {
oldTimeout := openclawModelShowTimeout
openclawModelShowTimeout = 50 * time.Millisecond
t.Cleanup(func() { openclawModelShowTimeout = oldTimeout })
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
time.Sleep(300 * time.Millisecond)
fmt.Fprintf(w, `{"capabilities":["thinking"],"model_info":{"llama.context_length":4096}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
start := time.Now()
cfg, _ := openclawModelConfig(context.Background(), client, "slow-model")
elapsed := time.Since(start)
if elapsed >= 250*time.Millisecond {
t.Fatalf("openclawModelConfig took too long: %v", elapsed)
}
if cfg["id"] != "slow-model" {
t.Errorf("id = %v, want slow-model", cfg["id"])
}
if _, ok := cfg["reasoning"]; ok {
t.Error("reasoning should not be set on timeout")
}
if _, ok := cfg["contextWindow"]; ok {
t.Error("contextWindow should not be set on timeout")
t.Error("contextWindow should not be set when metadata is unavailable")
}
})
t.Run("skips zero context length", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":[],"model_info":{"llama.context_length":0}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg, _ := openclawModelConfig(context.Background(), client, "test-model")
cfg, _ := openclawModelConfig(LaunchModel{Name: "test-model", ContextLength: 0})
if _, ok := cfg["contextWindow"]; ok {
t.Error("contextWindow should not be set for zero value")
@@ -2464,21 +2349,7 @@ func TestOpenclawModelConfig(t *testing.T) {
})
t.Run("cloud model uses hardcoded limits", func(t *testing.T) {
// Use a model name that's in cloudModelLimits and make the server
// report it as a remote/cloud model
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":[],"model_info":{},"remote_model":"minimax-m2.7"}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg, isCloud := openclawModelConfig(context.Background(), client, "minimax-m2.7:cloud")
cfg, isCloud := openclawModelConfig(fallbackLaunchModel("minimax-m2.7:cloud"))
if !isCloud {
t.Error("expected isCloud = true for cloud model")
@@ -2492,21 +2363,11 @@ func TestOpenclawModelConfig(t *testing.T) {
})
t.Run("cloud model with vision capability gets image input", func(t *testing.T) {
// Regression test: cloud models must not skip capability detection.
// A cloud model that reports vision capability should have input: [text, image].
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":["vision"],"model_info":{},"remote_model":"qwen3-vl"}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg, isCloud := openclawModelConfig(context.Background(), client, "qwen3-vl:235b-cloud")
cfg, isCloud := openclawModelConfig(LaunchModel{
Name: "qwen3-vl:235b-cloud",
Remote: true,
Capabilities: []model.Capability{"vision"},
}.WithCloudLimits())
if !isCloud {
t.Error("expected isCloud = true for cloud vision model")
@@ -2518,21 +2379,11 @@ func TestOpenclawModelConfig(t *testing.T) {
})
t.Run("cloud model with thinking capability gets reasoning flag", func(t *testing.T) {
// Regression test: cloud models must not skip capability detection.
// A cloud model that reports thinking capability should have reasoning: true.
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":["thinking"],"model_info":{},"remote_model":"qwq-cloud"}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg, isCloud := openclawModelConfig(context.Background(), client, "qwq:cloud")
cfg, isCloud := openclawModelConfig(LaunchModel{
Name: "qwq:cloud",
Remote: true,
Capabilities: []model.Capability{"thinking"},
})
if !isCloud {
t.Error("expected isCloud = true for cloud thinking model")
+67 -21
View File
@@ -43,7 +43,7 @@ func findOpenCode() (string, bool) {
return "", false
}
func (o *OpenCode) Run(model string, args []string) error {
func (o *OpenCode) Run(model string, models []LaunchModel, args []string) error {
opencodePath, ok := findOpenCode()
if !ok {
return fmt.Errorf("opencode is not installed, install from https://opencode.ai")
@@ -54,7 +54,7 @@ func (o *OpenCode) Run(model string, args []string) error {
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Env = os.Environ()
if content := o.resolveContent(model); content != "" {
if content := o.resolveContent(model, models); content != "" {
cmd.Env = append(cmd.Env, "OPENCODE_CONFIG_CONTENT="+content)
}
return cmd.Run()
@@ -63,21 +63,57 @@ func (o *OpenCode) Run(model string, args []string) error {
// resolveContent returns the inline config to send via OPENCODE_CONFIG_CONTENT.
// Returns content built by Edit if available, otherwise builds from model.json
// with the requested model as primary (e.g. re-launch with saved config).
func (o *OpenCode) resolveContent(model string) string {
func (o *OpenCode) resolveContent(model string, models []LaunchModel) string {
if o.configContent != "" {
return o.configContent
}
models := readModelJSONModels()
if !slices.Contains(models, model) {
models = append([]string{model}, models...)
resolvedModels := resolveOpenCodeRunModels(model, models, readModelJSONModels())
if len(resolvedModels) == 0 {
return ""
}
content, err := buildInlineConfig(model, models)
content, err := buildInlineConfig(resolvedModels[0], resolvedModels)
if err != nil {
return ""
}
return content
}
func resolveOpenCodeRunModels(primary string, models []LaunchModel, stateModels []string) []LaunchModel {
if primary == "" {
return nil
}
resolved := make([]LaunchModel, 0, 1+len(models)+len(stateModels))
appendModel := func(name string) {
if name == "" || hasLaunchModel(resolved, name) {
return
}
if model, ok := findLaunchModel(models, name); ok {
resolved = append(resolved, model)
return
}
resolved = append(resolved, fallbackLaunchModel(name))
}
appendModel(primary)
for _, model := range models {
appendModel(model.Name)
}
for _, model := range stateModels {
appendModel(model)
}
return resolved
}
func hasLaunchModel(models []LaunchModel, name string) bool {
for _, model := range models {
if launchModelMatches(model.Name, name) || launchModelMatches(name, model.Name) {
return true
}
}
return false
}
func (o *OpenCode) Paths() []string {
sp, err := openCodeStatePath()
if err != nil {
@@ -100,12 +136,13 @@ func openCodeStatePath() (string, error) {
return filepath.Join(home, ".local", "state", "opencode", "model.json"), nil
}
func (o *OpenCode) Edit(modelList []string) error {
func (o *OpenCode) Edit(models []LaunchModel) error {
modelList := launchModelNames(models)
if len(modelList) == 0 {
return nil
}
content, err := buildInlineConfig(modelList[0], modelList)
content, err := buildInlineConfig(models[0], models)
if err != nil {
return err
}
@@ -172,10 +209,11 @@ func (o *OpenCode) Models() []string {
// buildInlineConfig produces the JSON string for OPENCODE_CONFIG_CONTENT.
// primary is the model to launch with, models is the full list of available models.
func buildInlineConfig(primary string, models []string) (string, error) {
if primary == "" || len(models) == 0 {
func buildInlineConfig(primary LaunchModel, models []LaunchModel) (string, error) {
if primary.Name == "" || len(models) == 0 {
return "", fmt.Errorf("buildInlineConfig: primary and models are required")
}
config := map[string]any{
"$schema": "https://opencode.ai/config.json",
"provider": map[string]any{
@@ -188,7 +226,7 @@ func buildInlineConfig(primary string, models []string) (string, error) {
"models": buildModelEntries(models),
},
},
"model": "ollama/" + primary,
"model": "ollama/" + primary.Name,
}
data, err := json.Marshal(config)
if err != nil {
@@ -228,21 +266,29 @@ func readModelJSONModels() []string {
return models
}
func buildModelEntries(modelList []string) map[string]any {
func buildModelEntries(modelList []LaunchModel) map[string]any {
models := make(map[string]any)
for _, model := range modelList {
entry := map[string]any{
"name": model,
"name": model.Name,
}
if isCloudModelName(model) {
if l, ok := lookupCloudModelLimit(model); ok {
entry["limit"] = map[string]any{
"context": l.Context,
"output": l.Output,
}
if model.HasCapability("vision") {
entry["modalities"] = map[string]any{
"input": []string{"text", "image"},
"output": []string{"text"},
}
}
models[model] = entry
if model.ContextLength > 0 || model.MaxOutputTokens > 0 {
limit := make(map[string]any)
if model.ContextLength > 0 {
limit["context"] = model.ContextLength
}
if model.MaxOutputTokens > 0 {
limit["output"] = model.MaxOutputTokens
}
entry["limit"] = limit
}
models[model.Name] = entry
}
return models
}
+108 -24
View File
@@ -7,6 +7,8 @@ import (
"path/filepath"
"runtime"
"testing"
"github.com/ollama/ollama/types/model"
)
func TestOpenCodeIntegration(t *testing.T) {
@@ -31,7 +33,7 @@ func TestOpenCodeEdit(t *testing.T) {
t.Run("builds config content with provider", func(t *testing.T) {
setTestHome(t, t.TempDir())
o := &OpenCode{}
if err := o.Edit([]string{"llama3.2"}); err != nil {
if err := o.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
@@ -65,7 +67,7 @@ func TestOpenCodeEdit(t *testing.T) {
t.Run("multiple models", func(t *testing.T) {
setTestHome(t, t.TempDir())
o := &OpenCode{}
if err := o.Edit([]string{"llama3.2", "qwen3:32b"}); err != nil {
if err := o.Edit(testLaunchModels("llama3.2", "qwen3:32b")); err != nil {
t.Fatal(err)
}
@@ -90,7 +92,7 @@ func TestOpenCodeEdit(t *testing.T) {
t.Run("empty models is no-op", func(t *testing.T) {
setTestHome(t, t.TempDir())
o := &OpenCode{}
if err := o.Edit([]string{}); err != nil {
if err := o.Edit(testLaunchModels()); err != nil {
t.Fatal(err)
}
if o.configContent != "" {
@@ -102,7 +104,7 @@ func TestOpenCodeEdit(t *testing.T) {
tmpDir := t.TempDir()
setTestHome(t, tmpDir)
o := &OpenCode{}
o.Edit([]string{"llama3.2"})
o.Edit(testLaunchModels("llama3.2"))
configDir := filepath.Join(tmpDir, ".config", "opencode")
@@ -117,7 +119,7 @@ func TestOpenCodeEdit(t *testing.T) {
t.Run("cloud model has limits", func(t *testing.T) {
setTestHome(t, t.TempDir())
o := &OpenCode{}
if err := o.Edit([]string{"glm-4.7:cloud"}); err != nil {
if err := o.Edit(testLaunchModels("glm-4.7:cloud")); err != nil {
t.Fatal(err)
}
@@ -144,7 +146,7 @@ func TestOpenCodeEdit(t *testing.T) {
t.Run("local model has no limits", func(t *testing.T) {
setTestHome(t, t.TempDir())
o := &OpenCode{}
o.Edit([]string{"llama3.2"})
o.Edit(testLaunchModels("llama3.2"))
var cfg map[string]any
json.Unmarshal([]byte(o.configContent), &cfg)
@@ -157,6 +159,43 @@ func TestOpenCodeEdit(t *testing.T) {
t.Errorf("local model should not have limit, got %v", entry["limit"])
}
})
t.Run("vision model gets image input modalities", func(t *testing.T) {
models := buildModelEntries([]LaunchModel{{Name: "gemma4:26b", Capabilities: []model.Capability{"vision"}}})
entry, _ := models["gemma4:26b"].(map[string]any)
modalities, _ := entry["modalities"].(map[string]any)
input, _ := modalities["input"].([]string)
output, _ := modalities["output"].([]string)
if len(input) != 2 || input[0] != "text" || input[1] != "image" {
t.Fatalf("modalities.input = %v, want [text image]", input)
}
if len(output) != 1 || output[0] != "text" {
t.Fatalf("modalities.output = %v, want [text]", output)
}
})
}
func TestBuildModelEntries(t *testing.T) {
t.Run("defaults to model name without capabilities", func(t *testing.T) {
models := buildModelEntries(testLaunchModels("llama3.2"))
entry, _ := models["llama3.2"].(map[string]any)
if entry["name"] != "llama3.2" {
t.Fatalf("name = %v, want llama3.2", entry["name"])
}
if _, ok := entry["modalities"]; ok {
t.Fatalf("modalities should not be set without capabilities, got %v", entry["modalities"])
}
})
t.Run("uses context and output limits from metadata", func(t *testing.T) {
models := buildModelEntries([]LaunchModel{{Name: "glm-5:cloud", ContextLength: 202_752, MaxOutputTokens: 131_072}})
entry, _ := models["glm-5:cloud"].(map[string]any)
limit, _ := entry["limit"].(map[string]any)
if limit["context"] != 202_752 || limit["output"] != 131_072 {
t.Fatalf("limit = %v, want context/output", limit)
}
})
}
func TestOpenCodeModels_ReturnsNil(t *testing.T) {
@@ -284,7 +323,7 @@ func TestOpenCodeEdit_CloudModelLimitStructure(t *testing.T) {
expected := cloudModelLimits["glm-4.7"]
if err := o.Edit([]string{"glm-4.7:cloud"}); err != nil {
if err := o.Edit(testLaunchModels("glm-4.7:cloud")); err != nil {
t.Fatal(err)
}
@@ -314,7 +353,7 @@ func TestOpenCodeEdit_SpecialCharsInModelName(t *testing.T) {
specialModel := `model-with-"quotes"`
err := o.Edit([]string{specialModel})
err := o.Edit(testLaunchModels(specialModel))
if err != nil {
t.Fatalf("Edit with special chars failed: %v", err)
}
@@ -407,7 +446,7 @@ func TestOpenCodeResolveContent(t *testing.T) {
setTestHome(t, tmpDir)
o := &OpenCode{}
if err := o.Edit([]string{"gemma4"}); err != nil {
if err := o.Edit(testLaunchModels("gemma4")); err != nil {
t.Fatal(err)
}
editContent := o.configContent
@@ -422,7 +461,7 @@ func TestOpenCodeResolveContent(t *testing.T) {
data, _ := json.MarshalIndent(state, "", " ")
os.WriteFile(filepath.Join(stateDir, "model.json"), data, 0o644)
got := o.resolveContent("gemma4")
got := o.resolveContent("gemma4", nil)
if got != editContent {
t.Errorf("resolveContent returned different content than Edit set\ngot: %s\nwant: %s", got, editContent)
}
@@ -444,7 +483,7 @@ func TestOpenCodeResolveContent(t *testing.T) {
os.WriteFile(filepath.Join(stateDir, "model.json"), data, 0o644)
o := &OpenCode{}
content := o.resolveContent("llama3.2")
content := o.resolveContent("llama3.2", nil)
if content == "" {
t.Fatal("resolveContent returned empty")
}
@@ -478,7 +517,7 @@ func TestOpenCodeResolveContent(t *testing.T) {
os.WriteFile(filepath.Join(stateDir, "model.json"), data, 0o644)
o := &OpenCode{}
content := o.resolveContent("qwen3:32b")
content := o.resolveContent("qwen3:32b", nil)
var cfg map[string]any
json.Unmarshal([]byte(content), &cfg)
@@ -502,7 +541,7 @@ func TestOpenCodeResolveContent(t *testing.T) {
os.WriteFile(filepath.Join(stateDir, "model.json"), data, 0o644)
o := &OpenCode{}
content := o.resolveContent("gemma4")
content := o.resolveContent("gemma4", nil)
var cfg map[string]any
json.Unmarshal([]byte(content), &cfg)
@@ -522,11 +561,56 @@ func TestOpenCodeResolveContent(t *testing.T) {
setTestHome(t, tmpDir)
o := &OpenCode{}
if got := o.resolveContent(""); got != "" {
if got := o.resolveContent("", nil); got != "" {
t.Errorf("resolveContent(\"\") = %q, want empty", got)
}
})
t.Run("uses run model metadata when Edit was not called", func(t *testing.T) {
tmpDir := t.TempDir()
setTestHome(t, tmpDir)
stateDir := filepath.Join(tmpDir, ".local", "state", "opencode")
os.MkdirAll(stateDir, 0o755)
state := map[string]any{
"recent": []any{
map[string]any{"providerID": "ollama", "modelID": "llama3.2"},
},
}
data, _ := json.MarshalIndent(state, "", " ")
os.WriteFile(filepath.Join(stateDir, "model.json"), data, 0o644)
o := &OpenCode{}
content := o.resolveContent("gemma4", []LaunchModel{
{
Name: "gemma4",
Capabilities: []model.Capability{model.CapabilityVision},
ContextLength: 65_536,
MaxOutputTokens: 8_192,
},
})
if content == "" {
t.Fatal("resolveContent returned empty")
}
var cfg map[string]any
json.Unmarshal([]byte(content), &cfg)
provider, _ := cfg["provider"].(map[string]any)
ollama, _ := provider["ollama"].(map[string]any)
cfgModels, _ := ollama["models"].(map[string]any)
entry, _ := cfgModels["gemma4"].(map[string]any)
limit, _ := entry["limit"].(map[string]any)
if limit["context"] != float64(65_536) || limit["output"] != float64(8_192) {
t.Fatalf("limit = %v, want context/output from launch metadata", limit)
}
if _, ok := entry["modalities"].(map[string]any); !ok {
t.Fatalf("modalities should be set from launch metadata, got %v", entry["modalities"])
}
if cfgModels["llama3.2"] == nil {
t.Fatalf("state model missing from fallback config: %v", cfgModels)
}
})
t.Run("does not mutate configContent on fallback", func(t *testing.T) {
tmpDir := t.TempDir()
setTestHome(t, tmpDir)
@@ -542,7 +626,7 @@ func TestOpenCodeResolveContent(t *testing.T) {
os.WriteFile(filepath.Join(stateDir, "model.json"), data, 0o644)
o := &OpenCode{}
_ = o.resolveContent("llama3.2")
_ = o.resolveContent("llama3.2", nil)
if o.configContent != "" {
t.Errorf("resolveContent should not mutate configContent, got %q", o.configContent)
}
@@ -551,19 +635,19 @@ func TestOpenCodeResolveContent(t *testing.T) {
func TestBuildInlineConfig(t *testing.T) {
t.Run("returns error for empty primary", func(t *testing.T) {
if _, err := buildInlineConfig("", []string{"llama3.2"}); err == nil {
if _, err := buildInlineConfig(LaunchModel{}, testLaunchModels("llama3.2")); err == nil {
t.Error("expected error for empty primary")
}
})
t.Run("returns error for empty models", func(t *testing.T) {
if _, err := buildInlineConfig("llama3.2", nil); err == nil {
if _, err := buildInlineConfig(fallbackLaunchModel("llama3.2"), nil); err == nil {
t.Error("expected error for empty models")
}
})
t.Run("primary differs from first model in list", func(t *testing.T) {
content, err := buildInlineConfig("qwen3:32b", []string{"llama3.2", "qwen3:32b"})
content, err := buildInlineConfig(fallbackLaunchModel("qwen3:32b"), testLaunchModels("llama3.2", "qwen3:32b"))
if err != nil {
t.Fatal(err)
}
@@ -592,7 +676,7 @@ func TestOpenCodeEdit_PreservesRecentEntries(t *testing.T) {
os.WriteFile(filepath.Join(stateDir, "model.json"), data, 0o644)
o := &OpenCode{}
if err := o.Edit([]string{"new-X"}); err != nil {
if err := o.Edit(testLaunchModels("new-X")); err != nil {
t.Fatal(err)
}
@@ -626,7 +710,7 @@ func TestOpenCodeEdit_PreservesRecentEntries(t *testing.T) {
os.WriteFile(filepath.Join(stateDir, "model.json"), data, 0o644)
o := &OpenCode{}
if err := o.Edit([]string{"X", "Y", "Z"}); err != nil {
if err := o.Edit(testLaunchModels("X", "Y", "Z")); err != nil {
t.Fatal(err)
}
@@ -663,7 +747,7 @@ func TestOpenCodeEdit_PreservesRecentEntries(t *testing.T) {
os.WriteFile(filepath.Join(stateDir, "model.json"), data, 0o644)
o := &OpenCode{}
if err := o.Edit([]string{"qwen3:32b"}); err != nil {
if err := o.Edit(testLaunchModels("qwen3:32b")); err != nil {
t.Fatal(err)
}
@@ -700,7 +784,7 @@ func TestOpenCodeEdit_PreservesRecentEntries(t *testing.T) {
os.WriteFile(filepath.Join(stateDir, "model.json"), data, 0o644)
o := &OpenCode{}
if err := o.Edit([]string{"llama3.2"}); err != nil {
if err := o.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
@@ -742,7 +826,7 @@ func TestOpenCodeEdit_PreservesRecentEntries(t *testing.T) {
// Add 5 new models — should cap at 10 total
o := &OpenCode{}
if err := o.Edit([]string{"new-0", "new-1", "new-2", "new-3", "new-4"}); err != nil {
if err := o.Edit(testLaunchModels("new-0", "new-1", "new-2", "new-3", "new-4")); err != nil {
t.Fatal(err)
}
@@ -763,7 +847,7 @@ func TestOpenCodeEdit_BaseURL(t *testing.T) {
setTestHome(t, tmpDir)
// Default OLLAMA_HOST
o.Edit([]string{"llama3.2"})
o.Edit(testLaunchModels("llama3.2"))
var cfg map[string]any
json.Unmarshal([]byte(o.configContent), &cfg)
+13 -44
View File
@@ -4,7 +4,6 @@ import (
"context"
"encoding/json"
"fmt"
"net/http"
"os"
"os/exec"
"path/filepath"
@@ -14,7 +13,6 @@ import (
"github.com/ollama/ollama/api"
"github.com/ollama/ollama/cmd/internal/fileutil"
"github.com/ollama/ollama/envconfig"
"github.com/ollama/ollama/types/model"
)
// Pi implements Runner and Editor for Pi (Pi Coding Agent) integration
@@ -28,7 +26,7 @@ const (
func (p *Pi) String() string { return "Pi" }
func (p *Pi) Run(model string, args []string) error {
func (p *Pi) Run(_ string, _ []LaunchModel, args []string) error {
fmt.Fprintf(os.Stderr, "\n%sPreparing Pi...%s\n", ansiGray, ansiReset)
if err := ensureNpmInstalled(); err != nil {
return err
@@ -183,7 +181,7 @@ func (p *Pi) Paths() []string {
return paths
}
func (p *Pi) Edit(models []string) error {
func (p *Pi) Edit(models []LaunchModel) error {
if len(models) == 0 {
return nil
}
@@ -225,7 +223,7 @@ func (p *Pi) Edit(models []string) error {
// Build set of selected models to track which need to be added
selectedSet := make(map[string]bool, len(models))
for _, m := range models {
selectedSet[m] = true
selectedSet[m.Name] = true
}
// Build new models list:
@@ -256,11 +254,9 @@ func (p *Pi) Edit(models []string) error {
}
// Add newly selected models that weren't already in the list
client := api.NewClient(envconfig.Host(), http.DefaultClient)
ctx := context.Background()
for _, model := range models {
if selectedSet[model] {
newModels = append(newModels, createConfig(ctx, client, model))
if selectedSet[model.Name] {
newModels = append(newModels, createConfig(model))
}
}
@@ -284,7 +280,7 @@ func (p *Pi) Edit(models []string) error {
}
settings["defaultProvider"] = "ollama"
settings["defaultModel"] = models[0]
settings["defaultModel"] = models[0].Name
settingsData, err := json.MarshalIndent(settings, "", " ")
if err != nil {
@@ -342,54 +338,27 @@ func hasContextWindow(cfg map[string]any) bool {
}
}
// createConfig builds Pi model config with capability detection
func createConfig(ctx context.Context, client *api.Client, modelID string) map[string]any {
// createConfig builds Pi model config with capability detection.
func createConfig(model LaunchModel) map[string]any {
cfg := map[string]any{
"id": modelID,
"id": model.Name,
"_launch": true,
}
if l, ok := lookupCloudModelLimit(modelID); ok {
cfg["contextWindow"] = l.Context
}
applyCloudContextFallback := func() {
if l, ok := lookupCloudModelLimit(modelID); ok {
cfg["contextWindow"] = l.Context
}
}
resp, err := client.Show(ctx, &api.ShowRequest{Model: modelID})
if err != nil {
applyCloudContextFallback()
return cfg
}
// Set input types based on vision capability
if slices.Contains(resp.Capabilities, model.CapabilityVision) {
if model.HasCapability("vision") {
cfg["input"] = []string{"text", "image"}
} else {
cfg["input"] = []string{"text"}
}
// Set reasoning based on thinking capability
if slices.Contains(resp.Capabilities, model.CapabilityThinking) {
if model.HasCapability("thinking") {
cfg["reasoning"] = true
}
// Extract context window from ModelInfo. For known cloud models, the
// pre-filled shared limit remains unless the server provides a positive value.
hasContextWindow := false
for key, val := range resp.ModelInfo {
if strings.HasSuffix(key, ".context_length") {
if ctxLen, ok := val.(float64); ok && ctxLen > 0 {
cfg["contextWindow"] = int(ctxLen)
hasContextWindow = true
}
break
}
}
if !hasContextWindow {
applyCloudContextFallback()
if model.ContextLength > 0 {
cfg["contextWindow"] = model.ContextLength
}
return cfg
+44 -153
View File
@@ -1,19 +1,16 @@
package launch
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"github.com/ollama/ollama/api"
"github.com/ollama/ollama/cmd/internal/fileutil"
"github.com/ollama/ollama/types/model"
)
@@ -138,7 +135,7 @@ exit 0
})
p := &Pi{}
if err := p.Run("ignored", []string{"--version"}); err != nil {
if err := p.Run("ignored", nil, []string{"--version"}); err != nil {
t.Fatalf("Run() error = %v", err)
}
@@ -181,7 +178,7 @@ exit 0
})
p := &Pi{}
err := p.Run("ignored", nil)
err := p.Run("ignored", nil, nil)
if err == nil || !strings.Contains(err.Error(), "pi installation cancelled") {
t.Fatalf("expected install cancellation error, got %v", err)
}
@@ -203,7 +200,7 @@ exit 0
})
p := &Pi{}
if err := p.Run("ignored", []string{"session"}); err != nil {
if err := p.Run("ignored", nil, []string{"session"}); err != nil {
t.Fatalf("Run() error = %v", err)
}
@@ -238,7 +235,7 @@ exit 0
seedNpmNoop(t, tmpDir)
p := &Pi{}
if err := p.Run("ignored", []string{"doctor"}); err != nil {
if err := p.Run("ignored", nil, []string{"doctor"}); err != nil {
t.Fatalf("Run() error = %v", err)
}
@@ -266,7 +263,7 @@ exit 0
p := &Pi{}
stderr := captureStderr(t, func() {
if err := p.Run("ignored", []string{"session"}); err != nil {
if err := p.Run("ignored", nil, []string{"session"}); err != nil {
t.Fatalf("Run() should continue after web search update failure, got %v", err)
}
})
@@ -301,7 +298,7 @@ exit 0
p := &Pi{}
stderr := captureStderr(t, func() {
if err := p.Run("ignored", []string{"session"}); err != nil {
if err := p.Run("ignored", nil, []string{"session"}); err != nil {
t.Fatalf("Run() should continue after web search install failure, got %v", err)
}
})
@@ -331,7 +328,7 @@ exit 0
p := &Pi{}
stderr := captureStderr(t, func() {
if err := p.Run("ignored", []string{"session"}); err != nil {
if err := p.Run("ignored", nil, []string{"session"}); err != nil {
t.Fatalf("Run() error = %v", err)
}
})
@@ -360,7 +357,7 @@ exit 0
seedPiScript(t, tmpDir)
p := &Pi{}
err := p.Run("ignored", []string{"session"})
err := p.Run("ignored", nil, []string{"session"})
if err == nil || !strings.Contains(err.Error(), "npm (Node.js) is required to launch pi") {
t.Fatalf("expected missing npm error, got %v", err)
}
@@ -435,7 +432,7 @@ func TestPiEdit(t *testing.T) {
}
t.Run("returns nil for empty models", func(t *testing.T) {
if err := pi.Edit([]string{}); err != nil {
if err := pi.Edit(testLaunchModels()); err != nil {
t.Errorf("Edit([]) error = %v, want nil", err)
}
})
@@ -444,7 +441,7 @@ func TestPiEdit(t *testing.T) {
cleanup()
models := []string{"llama3.2", "qwen3:8b"}
if err := pi.Edit(models); err != nil {
if err := pi.Edit(launchModelsFromNames(models)); err != nil {
t.Fatalf("Edit() error = %v", err)
}
@@ -497,7 +494,7 @@ func TestPiEdit(t *testing.T) {
}
models := []string{"new-model"}
if err := pi.Edit(models); err != nil {
if err := pi.Edit(launchModelsFromNames(models)); err != nil {
t.Fatalf("Edit() error = %v", err)
}
@@ -550,7 +547,7 @@ func TestPiEdit(t *testing.T) {
t.Fatal(err)
}
if err := pi.Edit([]string{"glm-5:cloud"}); err != nil {
if err := pi.Edit(testLaunchModels("glm-5:cloud")); err != nil {
t.Fatalf("Edit() error = %v", err)
}
@@ -595,7 +592,7 @@ func TestPiEdit(t *testing.T) {
}
newModels := []string{"new-model-1", "new-model-2"}
if err := pi.Edit(newModels); err != nil {
if err := pi.Edit(launchModelsFromNames(newModels)); err != nil {
t.Fatalf("Edit() error = %v", err)
}
@@ -646,7 +643,7 @@ func TestPiEdit(t *testing.T) {
}
newModels := []string{"keep-model", "add-model"}
if err := pi.Edit(newModels); err != nil {
if err := pi.Edit(launchModelsFromNames(newModels)); err != nil {
t.Fatalf("Edit() error = %v", err)
}
@@ -683,7 +680,7 @@ func TestPiEdit(t *testing.T) {
}
models := []string{"test-model"}
if err := pi.Edit(models); err != nil {
if err := pi.Edit(launchModelsFromNames(models)); err != nil {
t.Fatalf("Edit() should not fail with corrupt config, got %v", err)
}
@@ -732,7 +729,7 @@ func TestPiEdit(t *testing.T) {
// Add a new ollama-managed model
newModels := []string{"new-ollama-model"}
if err := pi.Edit(newModels); err != nil {
if err := pi.Edit(launchModelsFromNames(newModels)); err != nil {
t.Fatalf("Edit() error = %v", err)
}
@@ -793,7 +790,7 @@ func TestPiEdit(t *testing.T) {
}
models := []string{"llama3.2"}
if err := pi.Edit(models); err != nil {
if err := pi.Edit(launchModelsFromNames(models)); err != nil {
t.Fatalf("Edit() error = %v", err)
}
@@ -831,7 +828,7 @@ func TestPiEdit(t *testing.T) {
os.MkdirAll(configDir, 0o755)
models := []string{"qwen3:8b"}
if err := pi.Edit(models); err != nil {
if err := pi.Edit(launchModelsFromNames(models)); err != nil {
t.Fatalf("Edit() error = %v", err)
}
@@ -865,7 +862,7 @@ func TestPiEdit(t *testing.T) {
}
models := []string{"test-model"}
if err := pi.Edit(models); err != nil {
if err := pi.Edit(launchModelsFromNames(models)); err != nil {
t.Fatalf("Edit() should not fail with corrupt settings, got %v", err)
}
@@ -921,7 +918,7 @@ func TestPiEdit_CreatesDistinctBackupsForEachManagedFile(t *testing.T) {
t.Fatal(err)
}
if err := pi.Edit([]string{"llama3.2"}); err != nil {
if err := pi.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatalf("Edit() error = %v", err)
}
@@ -1087,19 +1084,7 @@ func TestIsPiOllamaModel(t *testing.T) {
func TestCreateConfig(t *testing.T) {
t.Run("sets vision input when model has vision capability", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":["vision"],"model_info":{}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg := createConfig(context.Background(), client, "llava:7b")
cfg := createConfig(LaunchModel{Name: "llava:7b", Capabilities: []model.Capability{model.CapabilityVision}})
if cfg["id"] != "llava:7b" {
t.Errorf("id = %v, want llava:7b", cfg["id"])
@@ -1114,19 +1099,7 @@ func TestCreateConfig(t *testing.T) {
})
t.Run("sets text-only input when model lacks vision", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":["completion"],"model_info":{}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg := createConfig(context.Background(), client, "llama3.2")
cfg := createConfig(LaunchModel{Name: "llama3.2", Capabilities: []model.Capability{model.CapabilityCompletion}})
input, ok := cfg["input"].([]string)
if !ok || len(input) != 1 || input[0] != "text" {
@@ -1138,39 +1111,15 @@ func TestCreateConfig(t *testing.T) {
})
t.Run("sets reasoning when model has thinking capability", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":["thinking"],"model_info":{}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg := createConfig(context.Background(), client, "qwq")
cfg := createConfig(LaunchModel{Name: "qwq", Capabilities: []model.Capability{model.CapabilityThinking}})
if cfg["reasoning"] != true {
t.Error("expected reasoning = true for thinking model")
}
})
t.Run("extracts context window from model info", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":[],"model_info":{"llama.context_length":131072}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg := createConfig(context.Background(), client, "llama3.2")
t.Run("sets context window from metadata", func(t *testing.T) {
cfg := createConfig(LaunchModel{Name: "llama3.2", ContextLength: 131072})
if cfg["contextWindow"] != 131072 {
t.Errorf("contextWindow = %v, want 131072", cfg["contextWindow"])
@@ -1178,19 +1127,11 @@ func TestCreateConfig(t *testing.T) {
})
t.Run("handles all capabilities together", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":["vision","thinking"],"model_info":{"qwen3.context_length":32768}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg := createConfig(context.Background(), client, "qwen3-vision")
cfg := createConfig(LaunchModel{
Name: "qwen3-vision",
Capabilities: []model.Capability{model.CapabilityVision, model.CapabilityThinking},
ContextLength: 32768,
})
input := cfg["input"].([]string)
if len(input) != 2 || input[0] != "text" || input[1] != "image" {
@@ -1204,17 +1145,8 @@ func TestCreateConfig(t *testing.T) {
}
})
t.Run("returns minimal config when show fails", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound)
fmt.Fprintf(w, `{"error":"model not found"}`)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg := createConfig(context.Background(), client, "missing-model")
t.Run("returns minimal config when metadata is unavailable", func(t *testing.T) {
cfg := createConfig(LaunchModel{Name: "missing-model"})
if cfg["id"] != "missing-model" {
t.Errorf("id = %v, want missing-model", cfg["id"])
@@ -1222,49 +1154,29 @@ func TestCreateConfig(t *testing.T) {
if cfg["_launch"] != true {
t.Error("expected _launch = true")
}
// Should not have capability fields
if _, ok := cfg["input"]; ok {
t.Error("input should not be set when show fails")
// Input defaults to text even when capabilities are unavailable.
input, ok := cfg["input"].([]string)
if !ok || len(input) != 1 || input[0] != "text" {
t.Errorf("input = %v, want [text]", cfg["input"])
}
if _, ok := cfg["reasoning"]; ok {
t.Error("reasoning should not be set when show fails")
t.Error("reasoning should not be set when metadata is unavailable")
}
if _, ok := cfg["contextWindow"]; ok {
t.Error("contextWindow should not be set when show fails")
t.Error("contextWindow should not be set when metadata is unavailable")
}
})
t.Run("cloud model falls back to hardcoded context when show fails", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound)
fmt.Fprintf(w, `{"error":"model not found"}`)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg := createConfig(context.Background(), client, "kimi-k2.5:cloud")
t.Run("cloud model falls back to hardcoded context", func(t *testing.T) {
cfg := createConfig(fallbackLaunchModel("kimi-k2.5:cloud"))
if cfg["contextWindow"] != 262_144 {
t.Errorf("contextWindow = %v, want 262144", cfg["contextWindow"])
}
})
t.Run("cloud model falls back to hardcoded context when show omits model info", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":[],"model_info":{}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg := createConfig(context.Background(), client, "glm-5:cloud")
t.Run("cloud model uses hardcoded context when tags omit context", func(t *testing.T) {
cfg := createConfig(fallbackLaunchModel("glm-5:cloud"))
if cfg["contextWindow"] != 202_752 {
t.Errorf("contextWindow = %v, want 202752", cfg["contextWindow"])
@@ -1272,35 +1184,14 @@ func TestCreateConfig(t *testing.T) {
})
t.Run("cloud model with dash suffix falls back to hardcoded context", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound)
fmt.Fprintf(w, `{"error":"model not found"}`)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg := createConfig(context.Background(), client, "gpt-oss:120b-cloud")
cfg := createConfig(fallbackLaunchModel("gpt-oss:120b-cloud"))
if cfg["contextWindow"] != 131_072 {
t.Errorf("contextWindow = %v, want 131072", cfg["contextWindow"])
}
})
t.Run("skips zero context length", func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/show" {
fmt.Fprintf(w, `{"capabilities":[],"model_info":{"llama.context_length":0}}`)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u, _ := url.Parse(srv.URL)
client := api.NewClient(u, srv.Client())
cfg := createConfig(context.Background(), client, "test-model")
cfg := createConfig(LaunchModel{Name: "test-model", ContextLength: 0})
if _, ok := cfg["contextWindow"]; ok {
t.Error("contextWindow should not be set for zero value")
+1 -1
View File
@@ -29,7 +29,7 @@ func (p *Poolside) args(model string, extra []string) []string {
return args
}
func (p *Poolside) Run(model string, args []string) error {
func (p *Poolside) Run(model string, _ []LaunchModel, args []string) error {
if poolsideGOOS == "windows" {
return poolsideUnsupportedError()
}
+2 -2
View File
@@ -51,7 +51,7 @@ func TestPoolsideRunSetsOllamaEnv(t *testing.T) {
t.Setenv("OLLAMA_HOST", "http://127.0.0.1:11434")
p := &Poolside{}
if err := p.Run("qwen3.5", []string{"session"}); err != nil {
if err := p.Run("qwen3.5", nil, []string{"session"}); err != nil {
t.Fatalf("Run returned error: %v", err)
}
@@ -78,7 +78,7 @@ func TestPoolsideRunWindowsUnsupported(t *testing.T) {
t.Cleanup(func() { poolsideGOOS = prev })
p := &Poolside{}
err := p.Run("kimi-k2.6:cloud", nil)
err := p.Run("kimi-k2.6:cloud", nil, nil)
if err == nil {
t.Fatal("expected Windows unsupported error")
}
+13 -1
View File
@@ -33,7 +33,7 @@ type IntegrationInfo struct {
Description string
}
var launcherIntegrationOrder = []string{"claude", "openclaw", "hermes", "opencode", "codex", "copilot", "droid", "pi", "pool"}
var launcherIntegrationOrder = []string{"claude", "codex-app", "hermes", "openclaw", "opencode", "codex", "copilot", "droid", "pi", "pool"}
var integrationSpecs = []*IntegrationSpec{
{
@@ -87,6 +87,18 @@ var integrationSpecs = []*IntegrationSpec{
Command: []string{"npm", "install", "-g", "@openai/codex"},
},
},
{
Name: "codex-app",
Runner: &CodexApp{},
Aliases: []string{"codex-desktop", "codex-gui"},
Description: "An AI agent you can delegate real work to, by OpenAI",
Install: IntegrationInstallSpec{
CheckInstalled: func() bool {
return codexAppInstalled()
},
URL: "https://developers.openai.com/codex/quickstart",
},
},
{
Name: "kimi",
Runner: &Kimi{},
+1 -1
View File
@@ -84,7 +84,7 @@ func TestEditorRunsDoNotRewriteConfig(t *testing.T) {
t.Setenv("PATH", binDir)
configPath := tt.checkPath(home)
if err := tt.runner.Run("llama3.2", nil); err != nil {
if err := tt.runner.Run("llama3.2", nil, nil); err != nil {
t.Fatalf("Run returned error: %v", err)
}
if _, err := os.Stat(configPath); !os.IsNotExist(err) {
+4
View File
@@ -41,6 +41,10 @@ func setTestHome(t *testing.T, dir string) {
setLaunchTestHome(t, dir)
}
func testLaunchModels(names ...string) []LaunchModel {
return launchModelsFromNames(names)
}
func SaveIntegration(appName string, models []string) error {
return config.SaveIntegration(appName, models)
}
+2 -2
View File
@@ -126,7 +126,7 @@ const (
minVSCodeVersion = "1.113"
)
func (v *VSCode) Run(model string, args []string) error {
func (v *VSCode) Run(model string, _ []LaunchModel, args []string) error {
v.checkVSCodeVersion()
v.checkCopilotChatVersion()
@@ -238,7 +238,7 @@ func (v *VSCode) Paths() []string {
return nil
}
func (v *VSCode) Edit(models []string) error {
func (v *VSCode) Edit(models []LaunchModel) error {
if len(models) == 0 {
return nil
}
+3 -3
View File
@@ -113,7 +113,7 @@ func TestVSCodeEdit(t *testing.T) {
os.WriteFile(clmPath, []byte(tt.setup), 0o644)
}
if err := v.Edit(tt.models); err != nil {
if err := v.Edit(launchModelsFromNames(tt.models)); err != nil {
t.Fatal(err)
}
@@ -134,7 +134,7 @@ func TestVSCodeEditCleansUpOldSettings(t *testing.T) {
os.MkdirAll(filepath.Dir(settingsPath), 0o755)
os.WriteFile(settingsPath, []byte(`{"github.copilot.chat.byok.ollamaEndpoint": "http://old:11434", "ollama.launch.configured": true, "editor.fontSize": 14}`), 0o644)
if err := v.Edit([]string{"llama3.2"}); err != nil {
if err := v.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
@@ -180,7 +180,7 @@ func TestVSCodeEdit_CreatesDistinctBackupsForManagedFiles(t *testing.T) {
t.Fatal(err)
}
if err := v.Edit([]string{"llama3.2"}); err != nil {
if err := v.Edit(testLaunchModels("llama3.2")); err != nil {
t.Fatal(err)
}
+21 -1
View File
@@ -29,6 +29,13 @@ func launcherTestState() *launch.LauncherState {
Selectable: true,
Changeable: true,
},
"codex-app": {
Name: "codex-app",
DisplayName: "Codex App",
Description: "An AI agent you can delegate real work to, by OpenAI",
Selectable: true,
Changeable: true,
},
"openclaw": {
Name: "openclaw",
DisplayName: "OpenClaw",
@@ -122,12 +129,25 @@ func expectedExpandedSequence(state *launch.LauncherState) []string {
func TestMenuRendersPinnedItemsAndMore(t *testing.T) {
state := launcherTestState()
menu := newModel(state)
wantPrefix := []string{"run", "claude", "codex-app", "hermes", "openclaw"}
if findMenuCursorByIntegration(menu.items, "codex-app") == -1 {
wantPrefix = []string{"run", "claude", "hermes", "openclaw", "opencode"}
}
if got := integrationSequence(menu.items); len(got) < len(wantPrefix) {
t.Fatalf("expected at least %d menu items, got %v", len(wantPrefix), got)
} else if diff := compareStrings(got[:len(wantPrefix)], wantPrefix); diff != "" {
t.Fatalf("unexpected primary TUI order: %s", diff)
}
view := menu.View()
for _, want := range []string{"Chat with a model", "Launch Claude Code", "Launch OpenClaw", "Launch Hermes Agent", "More..."} {
for _, want := range []string{"Chat with a model", "Launch Claude Code", "Launch Hermes Agent", "Launch OpenClaw", "More..."} {
if !strings.Contains(view, want) {
t.Fatalf("expected menu view to contain %q\n%s", want, view)
}
}
if findMenuCursorByIntegration(menu.items, "codex-app") != -1 && !strings.Contains(view, "Launch Codex App") {
t.Fatalf("expected menu view to contain Codex App\n%s", view)
}
if strings.Contains(view, "Launch Claude Desktop") {
t.Fatalf("expected hidden Claude Desktop to be absent\n%s", view)
}
+2
View File
@@ -123,11 +123,13 @@
"expanded": true,
"pages": [
"/integrations/claude-code",
"/integrations/codex-app",
"/integrations/codex",
"/integrations/copilot-cli",
"/integrations/opencode",
"/integrations/droid",
"/integrations/goose",
"/integrations/omp",
"/integrations/pi",
"/integrations/pool"
]
Binary file not shown.

After

Width:  |  Height:  |  Size: 491 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 610 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 593 KiB

+82
View File
@@ -0,0 +1,82 @@
---
title: Codex App
---
Codex App is OpenAI's desktop coding agent for macOS and Windows. Ollama configures the app to use Ollama's OpenAI-compatible endpoint, so Codex can work with local models and Ollama Cloud models in the desktop app.
<img
src="/images/codex-app-home.png"
alt="Codex App with Ollama selected"
style={{ borderRadius: "12px" }}
/>
## Install
Install the [Codex App](https://developers.openai.com/codex/quickstart/) for macOS or Windows.
<Note>Codex App support is available in Ollama v0.24.0 and newer.</Note>
## Quick setup
```shell
ollama launch codex-app
```
Once Codex App opens, start a task or open a repository as usual.
## Built-in browser
Codex App can open local servers and sites in its built-in browser. Annotate directly on the page to request changes.
<img
src="/images/codex-app-annotate.png"
alt="Codex App browser annotations"
style={{ borderRadius: "12px" }}
/>
## Review mode
Use review mode to inspect code changes, leave comments, and iterate on fixes without leaving the app.
<img
src="/images/codex-app-review.png"
alt="Codex App review comments"
style={{ borderRadius: "12px" }}
/>
### Run directly with a model
```shell
ollama launch codex-app --model kimi-k2.6:cloud
```
Use a local model by passing its model name:
```shell
ollama launch codex-app --model gemma4:31b
```
Running `ollama launch codex-app` is persistent and will have your model selected next time you open Codex.
### Restore Codex App
To switch Codex App back to the profile you were using before `ollama launch codex-app`, run:
```shell
ollama launch codex-app --restore
```
Ollama restores Codex App's settings and configs. If Codex App is open, Ollama asks before restarting it.
The Codex CLI profile managed by `ollama launch codex` is left separate from the Codex App profile.
Before overwriting Codex App config files, Ollama Launch saves backups under `~/.ollama/backup/codex-app/`. On Windows, `~` resolves to your user profile directory.
## Troubleshooting
If Codex App does not open after setup, open Codex manually once and run `ollama launch codex-app` again.
If Codex App is already running and does not switch models, allow Ollama to restart it when prompted, or quit Codex App and run `ollama launch codex-app` again.
+5 -2
View File
@@ -1,11 +1,11 @@
---
title: Codex
title: Codex CLI
---
## Install
Install the [Codex CLI](https://developers.openai.com/codex/cli/):
Install the [Codex CLI](https://developers.openai.com/codex/cli/). For the desktop app, see [Codex App](/integrations/codex-app).
```
npm install -g @openai/codex
@@ -21,6 +21,9 @@ npm install -g @openai/codex
ollama launch codex
```
When launched through `ollama launch codex`, Ollama refreshes the model catalog
and passes it to Codex for that session.
To configure without launching:
```shell
+3 -1
View File
@@ -9,11 +9,13 @@ Ollama integrates with a wide range of tools.
Coding assistants that can read, modify, and execute code in your projects.
- [Claude Code](/integrations/claude-code)
- [Codex](/integrations/codex)
- [Codex App](/integrations/codex-app)
- [Codex CLI](/integrations/codex)
- [Copilot CLI](/integrations/copilot-cli)
- [OpenCode](/integrations/opencode)
- [Droid](/integrations/droid)
- [Goose](/integrations/goose)
- [OMP](/integrations/omp)
- [Pi](/integrations/pi)
- [Pool](/integrations/pool)
+47
View File
@@ -0,0 +1,47 @@
---
title: OMP
---
OMP is an AI coding agent with IDE integration that runs in your terminal.
## Install
Install [OMP](https://omp.sh):
```bash
curl -fsSL https://omp.sh/install | sh
```
On Windows, install from PowerShell:
```powershell
irm https://omp.sh/install.ps1 | iex
```
## Usage with Ollama
OMP discovers Ollama automatically when the Ollama server is running locally.
Start Ollama if it is not already running, then pull a model:
```bash
ollama pull qwen3.5
```
Then launch OMP:
```bash
omp
```
Use `/model` in OMP and select the Ollama model, such as `ollama/qwen3.5` and set it as the default.
To launch OMP with a specific Ollama model:
```bash
omp --model ollama/qwen3.5
```
<Note>For coding agents, larger context windows work best. See [Context length](/context-length) for more information.</Note>
For more OMP model configuration options, see the [custom models documentation](https://omp.sh/docs/custom-models).
+2 -1
View File
@@ -15,6 +15,7 @@ import (
"github.com/ollama/ollama/format"
"github.com/ollama/ollama/fs/util/bufioutil"
"github.com/ollama/ollama/logutil"
"github.com/ollama/ollama/ml"
)
@@ -323,7 +324,7 @@ func keyValue[T valueTypes | arrayValueTypes](kv KV, key string, defaultValue ..
return val, true
}
slog.Debug("key with type not found", "key", key, "default", defaultValue[0])
logutil.Trace("key with type not found", "key", key, "default", defaultValue[0])
return defaultValue[0], false
}
+1 -1
View File
@@ -30,6 +30,7 @@ require (
github.com/mattn/go-runewidth v0.0.16
github.com/nlpodyssey/gopickle v0.3.0
github.com/pdevine/tensor v0.0.0-20240510204454-f88f4562727c
github.com/pelletier/go-toml/v2 v2.2.2
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c
github.com/tkrajina/typescriptify-golang-structs v0.2.0
github.com/tree-sitter/go-tree-sitter v0.25.0
@@ -95,7 +96,6 @@ require (
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.12 // indirect
+11 -7
View File
@@ -98,7 +98,8 @@ func (r *Gemma4Renderer) Render(messages []api.Message, tools []api.Tool, thinkV
toolResponsesEmitted := false
if len(message.ToolCalls) > 0 {
for k := i + 1; k < len(loopMessages) && loopMessages[k].Role == "tool"; k++ {
sb.WriteString(r.formatToolResponseBlock(r.toolResponseName(loopMessages[k], message.ToolCalls), loopMessages[k].Content))
response := r.renderToolResponseContent(loopMessages[k], &imageOffset)
sb.WriteString(r.formatToolResponseBlock(r.toolResponseName(loopMessages[k], message.ToolCalls), response))
toolResponsesEmitted = true
prevMessageType = "tool_response"
}
@@ -160,19 +161,22 @@ func stripThinking(text string) string {
// When trim is true, leading/trailing whitespace is stripped (matching the Jinja2
// template's | trim filter applied to non-model content).
func (r *Gemma4Renderer) renderContent(sb *strings.Builder, msg api.Message, imageOffset *int, trim bool) {
if len(msg.Images) > 0 && r.useImgTags {
for range msg.Images {
sb.WriteString(fmt.Sprintf("[img-%d]", *imageOffset))
*imageOffset++
}
}
content := msg.Content
if trim {
content = strings.TrimSpace(content)
}
if len(msg.Images) > 0 && r.useImgTags {
content, *imageOffset = renderContentWithImageTags(content, len(msg.Images), *imageOffset)
}
sb.WriteString(content)
}
func (r *Gemma4Renderer) renderToolResponseContent(msg api.Message, imageOffset *int) string {
var sb strings.Builder
r.renderContent(&sb, msg, imageOffset, false)
return sb.String()
}
func (r *Gemma4Renderer) previousNonToolRole(messages []api.Message, idx int) string {
for i := idx - 1; i >= 0; i-- {
if messages[i].Role != "tool" {
+7 -9
View File
@@ -13,15 +13,11 @@ type GlmOcrRenderer struct {
}
func (r *GlmOcrRenderer) renderContent(message api.Message, imageOffset int) (string, int) {
var sb strings.Builder
for range message.Images {
if r.useImgTags {
sb.WriteString(fmt.Sprintf("[img-%d]", imageOffset))
imageOffset++
}
if r.useImgTags {
return renderContentWithImageTags(message.Content, len(message.Images), imageOffset)
}
sb.WriteString(message.Content)
return sb.String(), imageOffset
return message.Content, imageOffset
}
func (r *GlmOcrRenderer) Render(messages []api.Message, tools []api.Tool, thinkValue *api.ThinkValue) (string, error) {
@@ -85,8 +81,10 @@ func (r *GlmOcrRenderer) Render(messages []api.Message, tools []api.Tool, thinkV
if i == 0 || messages[i-1].Role != "tool" {
sb.WriteString("<|observation|>")
}
content, nextOffset := r.renderContent(message, imageOffset)
imageOffset = nextOffset
sb.WriteString("\n<tool_response>\n")
sb.WriteString(message.Content)
sb.WriteString(content)
sb.WriteString("\n</tool_response>\n")
case "system":
sb.WriteString("<|system|>\n")
+3 -3
View File
@@ -25,7 +25,7 @@ func TestGlmOcrRenderer_Images(t *testing.T) {
Images: []api.ImageData{api.ImageData("img1")},
},
},
expected: "[gMASK]<sop><|user|>\n[img-0]Describe this image.<|assistant|>\n",
expected: "[gMASK]<sop><|user|>\n[img-0] Describe this image.<|assistant|>\n",
},
{
name: "use_img_tags_multiple_images",
@@ -37,7 +37,7 @@ func TestGlmOcrRenderer_Images(t *testing.T) {
Images: []api.ImageData{api.ImageData("img1"), api.ImageData("img2")},
},
},
expected: "[gMASK]<sop><|user|>\n[img-0][img-1]Describe these images.<|assistant|>\n",
expected: "[gMASK]<sop><|user|>\n[img-0][img-1] Describe these images.<|assistant|>\n",
},
{
name: "multi_turn_increments_image_offset",
@@ -58,7 +58,7 @@ func TestGlmOcrRenderer_Images(t *testing.T) {
Images: []api.ImageData{api.ImageData("img2")},
},
},
expected: "[gMASK]<sop><|user|>\n[img-0]First image<|assistant|>\n<think></think>\nProcessed.\n<|user|>\n[img-1]Second image<|assistant|>\n",
expected: "[gMASK]<sop><|user|>\n[img-0] First image<|assistant|>\n<think></think>\nProcessed.\n<|user|>\n[img-1] Second image<|assistant|>\n",
},
{
name: "default_no_img_tags",
+39
View File
@@ -0,0 +1,39 @@
package renderers
import (
"fmt"
"strings"
"unicode"
"unicode/utf8"
)
// renderContentWithImageTags preserves the legacy server-side placeholder
// semantics for explicit [img] tokens: replace placeholders in order, and
// only prepend tags for any remaining images without placeholders.
func renderContentWithImageTags(content string, imageCount int, imageOffset int) (string, int) {
if imageCount == 0 {
return content, imageOffset
}
if strings.Contains(content, "[img-") {
return content, imageOffset + imageCount
}
var prefix strings.Builder
for i := range imageCount {
imgTag := fmt.Sprintf("[img-%d]", imageOffset+i)
if strings.Contains(content, "[img]") {
content = strings.Replace(content, "[img]", imgTag, 1)
} else {
prefix.WriteString(imgTag)
}
}
if prefix.Len() > 0 && content != "" {
if r, _ := utf8.DecodeRuneInString(content); r != utf8.RuneError && !unicode.IsSpace(r) {
prefix.WriteByte(' ')
}
}
return prefix.String() + content, imageOffset + imageCount
}
+67
View File
@@ -0,0 +1,67 @@
package renderers
import "testing"
func TestRenderContentWithImageTags(t *testing.T) {
tests := []struct {
name string
content string
imageCount int
imageOffset int
want string
wantOffset int
}{
{
name: "prefixes when there are no placeholders",
content: "describe this image",
imageCount: 2,
imageOffset: 0,
want: "[img-0][img-1] describe this image",
wantOffset: 2,
},
{
name: "replaces explicit placeholders in order",
content: "compare [img] and [img]",
imageCount: 2,
imageOffset: 3,
want: "compare [img-3] and [img-4]",
wantOffset: 5,
},
{
name: "prefixes extra images after placeholders are exhausted",
content: "compare [img]",
imageCount: 2,
imageOffset: 0,
want: "[img-1] compare [img-0]",
wantOffset: 2,
},
{
name: "leaves leftover placeholders when there are fewer images",
content: "compare [img] and [img]",
imageCount: 1,
imageOffset: 0,
want: "compare [img-0] and [img]",
wantOffset: 1,
},
{
name: "preserves already-numbered placeholders",
content: "compare [img-0] and [img-1]",
imageCount: 2,
imageOffset: 0,
want: "compare [img-0] and [img-1]",
wantOffset: 2,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, gotOffset := renderContentWithImageTags(tt.content, tt.imageCount, tt.imageOffset)
if got != tt.want {
t.Fatalf("content = %q, want %q", got, tt.want)
}
if gotOffset != tt.wantOffset {
t.Fatalf("offset = %d, want %d", gotOffset, tt.wantOffset)
}
})
}
}
+11 -13
View File
@@ -3,7 +3,6 @@ package renderers
import (
"bytes"
"encoding/json"
"fmt"
"sort"
"strings"
@@ -199,19 +198,18 @@ func (r *LFM2Renderer) renderMessageContent(message api.Message, imageOffset int
return content
}
var sb strings.Builder
if r.useImgTags {
for i := range message.Images {
sb.WriteString(fmt.Sprintf("[img-%d]", imageOffset+i))
}
} else {
placeholder := lfm2ImagePlaceholder(false)
if strings.Contains(content, placeholder) {
return content
}
for range message.Images {
sb.WriteString(placeholder)
}
content, _ = renderContentWithImageTags(content, len(message.Images), imageOffset)
return content
}
var sb strings.Builder
placeholder := lfm2ImagePlaceholder(false)
if strings.Contains(content, placeholder) {
return content
}
for range message.Images {
sb.WriteString(placeholder)
}
sb.WriteString(content)
return sb.String()
+1 -1
View File
@@ -236,7 +236,7 @@ func TestLFM2Renderer_Images(t *testing.T) {
Content: "Describe this image.",
Images: []api.ImageData{api.ImageData("img1")},
},
expected: "<|startoftext|><|im_start|>user\n[img-0]Describe this image.<|im_end|>\n<|im_start|>assistant\n",
expected: "<|startoftext|><|im_start|>user\n[img-0] Describe this image.<|im_end|>\n<|im_start|>assistant\n",
},
{
name: "existing_template_image_placeholder_not_duplicated",
+5 -18
View File
@@ -79,12 +79,14 @@ func (r *Nemotron3NanoRenderer) Render(messages []api.Message, tools []api.Tool,
// Check if previous message was also a tool message
prevWasTool := i > 0 && loopMessages[i-1].Role == "tool"
nextIsTool := i+1 < len(loopMessages) && loopMessages[i+1].Role == "tool"
content := r.renderMessageContent(message, imageOffset)
imageOffset += len(message.Images)
if !prevWasTool {
sb.WriteString("<|im_start|>user\n")
}
sb.WriteString("<tool_response>\n")
sb.WriteString(message.Content)
sb.WriteString(content)
sb.WriteString("\n</tool_response>\n")
if !nextIsTool {
@@ -237,23 +239,8 @@ func (r *Nemotron3NanoRenderer) renderMessageContent(message api.Message, imageO
return content
}
if strings.Contains(content, "[img-") {
return content
}
if strings.Contains(content, "[img]") {
for i := range message.Images {
content = strings.Replace(content, "[img]", fmt.Sprintf("[img-%d]", imageOffset+i), 1)
}
return content
}
var sb strings.Builder
for i := range message.Images {
sb.WriteString(fmt.Sprintf("[img-%d]", imageOffset+i))
}
sb.WriteString(content)
return sb.String()
content, _ = renderContentWithImageTags(content, len(message.Images), imageOffset)
return content
}
func nemotron3NanoRenderContent(content any) string {
+2 -2
View File
@@ -19,7 +19,7 @@ func TestNemotron3NanoRenderer_Images(t *testing.T) {
msgs: []api.Message{
{Role: "user", Content: "Describe this image.", Images: []api.ImageData{api.ImageData("img1")}},
},
expected: "\n\n\n<|im_start|>system\n<|im_end|>\n\n<|im_start|>user\n[img-0]Describe this image.<|im_end|>\n\n<|im_start|>assistant\n<think>\n",
expected: "\n\n\n<|im_start|>system\n<|im_end|>\n\n<|im_start|>user\n[img-0] Describe this image.<|im_end|>\n\n<|im_start|>assistant\n<think>\n",
},
{
name: "generic image placeholder is rewritten",
@@ -35,7 +35,7 @@ func TestNemotron3NanoRenderer_Images(t *testing.T) {
{Role: "assistant", Content: "It shows something."},
{Role: "user", Content: "Compare these.", Images: []api.ImageData{api.ImageData("img2"), api.ImageData("img3")}},
},
expected: "\n\n\n<|im_start|>system\n<|im_end|>\n\n<|im_start|>user\n[img-0]Describe the first image.<|im_end|>\n<|im_start|>assistant\n<think></think>It shows something.<|im_end|>\n<|im_start|>user\n[img-1][img-2]Compare these.<|im_end|>\n\n<|im_start|>assistant\n<think>\n",
expected: "\n\n\n<|im_start|>system\n<|im_end|>\n\n<|im_start|>user\n[img-0] Describe the first image.<|im_end|>\n<|im_start|>assistant\n<think></think>It shows something.<|im_end|>\n<|im_start|>user\n[img-1][img-2] Compare these.<|im_end|>\n\n<|im_start|>assistant\n<think>\n",
},
}
+5 -7
View File
@@ -1,7 +1,6 @@
package renderers
import (
"fmt"
"strings"
"github.com/ollama/ollama/api"
@@ -45,15 +44,14 @@ type Qwen35Renderer struct {
}
func (r *Qwen35Renderer) renderContent(content api.Message, imageOffset int) (string, int) {
if r.useImgTags {
return renderContentWithImageTags(content.Content, len(content.Images), imageOffset)
}
// This assumes all images are at the front of the message - same assumption as ollama/ollama/runner.go
var subSb strings.Builder
for range content.Images {
if r.useImgTags {
subSb.WriteString(fmt.Sprintf("[img-%d]", imageOffset))
imageOffset++
} else {
subSb.WriteString("<|vision_start|><|image_pad|><|vision_end|>")
}
subSb.WriteString("<|vision_start|><|image_pad|><|vision_end|>")
}
// TODO: support videos
+6 -8
View File
@@ -1,7 +1,6 @@
package renderers
import (
"fmt"
"strings"
"github.com/ollama/ollama/api"
@@ -15,18 +14,17 @@ type Qwen3VLRenderer struct {
}
func (r *Qwen3VLRenderer) renderContent(content api.Message, imageOffset int) (string, int) {
if r.useImgTags {
return renderContentWithImageTags(content.Content, len(content.Images), imageOffset)
}
// This assumes all images are at the front of the message - same assumption as ollama/ollama/runner.go
var subSb strings.Builder
for range content.Images {
// TODO: (jmorganca): how to render this is different for different
// model backends, and so we should eventually parameterize this or
// only output a placeholder such as [img]
if r.useImgTags {
subSb.WriteString(fmt.Sprintf("[img-%d]", imageOffset))
imageOffset++
} else {
subSb.WriteString("<|vision_start|><|image_pad|><|vision_end|>")
}
subSb.WriteString("<|vision_start|><|image_pad|><|vision_end|>")
}
// TODO: support videos
@@ -126,7 +124,7 @@ func (r *Qwen3VLRenderer) Render(messages []api.Message, tools []api.Tool, think
if i == 0 || messages[i-1].Role != "tool" {
sb.WriteString("<|im_start|>user")
}
sb.WriteString("\n<tool_response>\n" + message.Content + "\n</tool_response>")
sb.WriteString("\n<tool_response>\n" + content + "\n</tool_response>")
if i == len(messages)-1 || messages[i+1].Role != "tool" {
sb.WriteString("<|im_end|>\n")
}
+2 -2
View File
@@ -101,7 +101,7 @@ Let me analyze this image.`,
},
useImgTags: true,
expected: `<|im_start|>user
[img-0]Describe this image.<|im_end|>
[img-0] Describe this image.<|im_end|>
<|im_start|>assistant
Let me analyze this image.`,
},
@@ -123,7 +123,7 @@ Let me analyze this image.`,
},
useImgTags: true,
expected: `<|im_start|>user
[img-0][img-1]Describe these images.<|im_end|>
[img-0][img-1] Describe these images.<|im_end|>
<|im_start|>assistant
Let me analyze this image.`,
},
+22
View File
@@ -28,6 +28,24 @@ usage() {
mkdir -p dist
# Work around MLX's v3 metallib link leaking the macOS 26 deployment target.
_relink_mlx_metallib() {
BUILD_DIR="$1"
KERNEL_DIR="$BUILD_DIR/_deps/mlx-build/mlx/backend/metal/kernels"
AIR_LIST="$BUILD_DIR/mlx-air-files.txt"
METALLIB="$KERNEL_DIR/mlx.metallib"
find "$KERNEL_DIR" -type f -name '*.air' | sort > "$AIR_LIST"
if [ ! -s "$AIR_LIST" ]; then
echo "error: could not find MLX AIR files in $KERNEL_DIR" >&2
exit 1
fi
status "Relinking MLX metallib"
rm -f "$METALLIB"
xargs xcrun -sdk macosx metallib -o "$METALLIB" < "$AIR_LIST"
}
ARCHS="arm64 amd64"
while getopts "a:h" OPTION; do
@@ -58,6 +76,7 @@ _build_darwin() {
cmake --build $BUILD_DIR --target mlx mlxc -j
cmake --install $BUILD_DIR --component CPU
cmake --install $BUILD_DIR --component MLX
cmake --install $BUILD_DIR --component MLX_VENDOR
# Override CGO flags to point to the amd64 build directory
MLX_CGO_CFLAGS="-O3 -mmacosx-version-min=14.0"
MLX_CGO_LDFLAGS="-ldl -lc++ -framework Accelerate -mmacosx-version-min=14.0"
@@ -83,7 +102,9 @@ _build_darwin() {
-DCMAKE_OSX_DEPLOYMENT_TARGET=14.0 \
-DCMAKE_INSTALL_PREFIX=$INSTALL_PREFIX
cmake --build $BUILD_DIR --target mlx mlxc --parallel
_relink_mlx_metallib $BUILD_DIR
cmake --install $BUILD_DIR --component MLX
cmake --install $BUILD_DIR --component MLX_VENDOR
# Metal 4.x build (NAX-enabled, macOS 26+)
# Only possible with Xcode 26+ SDK; skip on older toolchains.
@@ -105,6 +126,7 @@ _build_darwin() {
-DFETCHCONTENT_SOURCE_DIR_METAL_CPP=$V3_DEPS/metal_cpp-src
cmake --build $BUILD_DIR_V4 --target mlx mlxc --parallel
cmake --install $BUILD_DIR_V4 --component MLX
cmake --install $BUILD_DIR_V4 --component MLX_VENDOR
else
status "Skipping MLX Metal v4 (SDK $SDK_MAJOR < 26, need Xcode 26+)"
fi
+29 -14
View File
@@ -1,9 +1,14 @@
#!/bin/sh
#
# Mac ARM users, rosetta can be flaky, so to use a remote x86 builder
# Mac ARM users, rosetta can be flaky, so to use a remote x86 builder.
# Use the docker-container driver with the bundled buildkit GC config
# for improved cache behavior
#
# docker context create amd64 --docker host=ssh://mybuildhost
# docker buildx create --name mybuilder amd64 --platform linux/amd64
# docker buildx create --name mybuilder \
# --driver docker-container \
# --config ./buildkitd.toml.example \
# --bootstrap amd64 --platform linux/amd64
# docker buildx create --name mybuilder --append desktop-linux --platform linux/arm64
# docker buildx use mybuilder
@@ -59,18 +64,28 @@ fi
# buildx behavior changes for single vs. multiplatform
echo "Compressing linux tar bundles..."
if echo $PLATFORM | grep "," > /dev/null ; then
tar c -C ./dist/linux_arm64 --exclude cuda_jetpack5 --exclude cuda_jetpack6 . | zstd --ultra -22 -T0 >./dist/ollama-linux-arm64.tar.zst
tar c -C ./dist/linux_arm64 ./lib/ollama/cuda_jetpack5 | zstd --ultra -22 -T0 >./dist/ollama-linux-arm64-jetpack5.tar.zst
tar c -C ./dist/linux_arm64 ./lib/ollama/cuda_jetpack6 | zstd --ultra -22 -T0 >./dist/ollama-linux-arm64-jetpack6.tar.zst
tar c -C ./dist/linux_amd64 --exclude rocm --exclude 'mlx*' --exclude include . | zstd --ultra -22 -T0 >./dist/ollama-linux-amd64.tar.zst
tar c -C ./dist/linux_amd64 ./lib/ollama/rocm | zstd --ultra -22 -T0 >./dist/ollama-linux-amd64-rocm.tar.zst
tar c -C ./dist/linux_amd64 ./lib/ollama/mlx_cuda_v13 ./lib/ollama/include | zstd --ultra -22 -T0 >./dist/ollama-linux-amd64-mlx.tar.zst
tar c -C ./dist/linux_arm64 --exclude cuda_jetpack5 --exclude cuda_jetpack6 . | zstd -9 -T0 >./dist/ollama-linux-arm64.tar.zst
tar c -C ./dist/linux_arm64 ./lib/ollama/cuda_jetpack5 | zstd -9 -T0 >./dist/ollama-linux-arm64-jetpack5.tar.zst
tar c -C ./dist/linux_arm64 ./lib/ollama/cuda_jetpack6 | zstd -9 -T0 >./dist/ollama-linux-arm64-jetpack6.tar.zst
tar c -C ./dist/linux_amd64 --exclude rocm --exclude 'mlx*' . | zstd -9 -T0 >./dist/ollama-linux-amd64.tar.zst
tar c -C ./dist/linux_amd64 ./lib/ollama/rocm | zstd -9 -T0 >./dist/ollama-linux-amd64-rocm.tar.zst
( cd ./dist/linux_amd64 && tar c lib/ollama/mlx* ) | zstd -9 -T0 >./dist/ollama-linux-amd64-mlx.tar.zst
elif echo $PLATFORM | grep "arm64" > /dev/null ; then
tar c -C ./dist/ --exclude cuda_jetpack5 --exclude cuda_jetpack6 bin lib | zstd --ultra -22 -T0 >./dist/ollama-linux-arm64.tar.zst
tar c -C ./dist/ ./lib/ollama/cuda_jetpack5 | zstd --ultra -22 -T0 >./dist/ollama-linux-arm64-jetpack5.tar.zst
tar c -C ./dist/ ./lib/ollama/cuda_jetpack6 | zstd --ultra -22 -T0 >./dist/ollama-linux-arm64-jetpack6.tar.zst
tar c -C ./dist/ --exclude cuda_jetpack5 --exclude cuda_jetpack6 bin lib | zstd -9 -T0 >./dist/ollama-linux-arm64.tar.zst
tar c -C ./dist/ ./lib/ollama/cuda_jetpack5 | zstd -9 -T0 >./dist/ollama-linux-arm64-jetpack5.tar.zst
tar c -C ./dist/ ./lib/ollama/cuda_jetpack6 | zstd -9 -T0 >./dist/ollama-linux-arm64-jetpack6.tar.zst
elif echo $PLATFORM | grep "amd64" > /dev/null ; then
tar c -C ./dist/ --exclude rocm --exclude 'mlx*' --exclude include bin lib | zstd --ultra -22 -T0 >./dist/ollama-linux-amd64.tar.zst
tar c -C ./dist/ ./lib/ollama/rocm | zstd --ultra -22 -T0 >./dist/ollama-linux-amd64-rocm.tar.zst
tar c -C ./dist/ ./lib/ollama/mlx_cuda_v13 ./lib/ollama/include | zstd --ultra -22 -T0 >./dist/ollama-linux-amd64-mlx.tar.zst
tar c -C ./dist/ --exclude rocm --exclude 'mlx*' bin lib | zstd -9 -T0 >./dist/ollama-linux-amd64.tar.zst
tar c -C ./dist/ ./lib/ollama/rocm | zstd -9 -T0 >./dist/ollama-linux-amd64-rocm.tar.zst
( cd ./dist/ && tar c lib/ollama/mlx* ) | zstd -9 -T0 >./dist/ollama-linux-amd64-mlx.tar.zst
fi
# Warn if any compressed tarball exceeds GitHub's 2 GiB release-asset limit
LIMIT=2147483648
for f in ./dist/ollama-linux-*.tar.zst; do
[ -f "$f" ] || continue
size=$(stat -f%z "$f" 2>/dev/null || stat -c%s "$f")
if [ "$size" -gt "$LIMIT" ]; then
echo "WARNING: $f is $size bytes ($((size - LIMIT)) over the 2 GiB GitHub release-asset limit)" >&2
fi
done
+3 -1
View File
@@ -308,6 +308,8 @@ function mlxCuda13 {
if ($LASTEXITCODE -ne 0) { exit($LASTEXITCODE)}
& cmake --install build\mlx_cuda_v$cudaMajorVer --component "MLX" --strip
if ($LASTEXITCODE -ne 0) { exit($LASTEXITCODE)}
& cmake --install build\mlx_cuda_v$cudaMajorVer --component "MLX_VENDOR"
if ($LASTEXITCODE -ne 0) { exit($LASTEXITCODE)}
} else {
Write-Output "CUDA v$cudaMajorVer not detected, skipping MLX build"
}
@@ -430,7 +432,7 @@ function newZipJob($sourceDir, $destZip) {
Start-Job -ScriptBlock {
param($src, $dst, $use7z)
if ($use7z) {
& 7z a -tzip -mx=9 -mmt=on $dst "${src}\*"
& 7z a -tzip -mx=7 -mmt=on $dst "${src}\*"
if ($LASTEXITCODE -ne 0) { throw "7z failed with exit code $LASTEXITCODE" }
} else {
Compress-Archive -CompressionLevel Optimal -Path "${src}\*" -DestinationPath $dst -Force
+21
View File
@@ -0,0 +1,21 @@
# Suggested BuildKit GC config for ollama local development.
#
[worker.oci]
gc = true
gckeepstorage = "150GB"
[[worker.oci.gcpolicy]]
filters = ["type==source.local", "type==source.git.checkout"]
keepDuration = "48h"
maxUsedSpace = "5GB"
[[worker.oci.gcpolicy]]
filters = ["type==exec.cachemount"]
keepDuration = "168h" # 7 days
maxUsedSpace = "20GB"
[[worker.oci.gcpolicy]]
keepDuration = "720h" # 30 days
reservedSpace = "20GB"
maxUsedSpace = "150GB"
minFreeSpace = "50GB"
+2 -3
View File
@@ -271,6 +271,8 @@ func (s *Server) CreateHandler(c *gin.Context) {
}
}
s.refreshModelListCache(name)
ch <- api.ProgressResponse{Status: "success"}
}()
@@ -741,9 +743,6 @@ func setTemplate(layers []manifest.Layer, t string) ([]manifest.Layer, error) {
if _, err := template.Parse(t); err != nil {
return nil, fmt.Errorf("%w: %s", errBadTemplate, err)
}
if _, err := template.Parse(t); err != nil {
return nil, fmt.Errorf("%w: %s", errBadTemplate, err)
}
blob := strings.NewReader(t)
layer, err := manifest.NewLayer(blob, "application/vnd.ollama.image.template")
+30
View File
@@ -256,3 +256,33 @@ func TestRemoteURL_Idempotent(t *testing.T) {
})
}
}
func TestSetTemplate(t *testing.T) {
t.Setenv("OLLAMA_MODELS", t.TempDir())
t.Run("valid template", func(t *testing.T) {
layers, err := setTemplate(nil, "{{ .Prompt }}")
if err != nil {
t.Fatalf("setTemplate returned error for valid template: %v", err)
}
if len(layers) != 1 {
t.Fatalf("expected 1 layer, got %d", len(layers))
}
if got, want := layers[0].MediaType, "application/vnd.ollama.image.template"; got != want {
t.Fatalf("unexpected media type: got %q, want %q", got, want)
}
})
t.Run("invalid template", func(t *testing.T) {
_, err := setTemplate(nil, "{{ if .Prompt }}")
if err == nil {
t.Fatal("expected error for invalid template, got nil")
}
if !errors.Is(err, errBadTemplate) {
t.Fatalf("expected errBadTemplate, got %v", err)
}
})
}
+1 -1
View File
@@ -132,7 +132,7 @@ func (m *Model) Capabilities() []model.Capability {
if err != nil {
slog.Warn("model template contains errors", "error", err)
}
if slices.Contains(v, "tools") || (builtinParser != nil && builtinParser.HasToolSupport()) {
if !slices.Contains(capabilities, model.CapabilityTools) && (slices.Contains(v, "tools") || (builtinParser != nil && builtinParser.HasToolSupport())) {
capabilities = append(capabilities, model.CapabilityTools)
}
+11
View File
@@ -138,6 +138,17 @@ func TestModelCapabilities(t *testing.T) {
},
expectedCaps: []model.Capability{model.CapabilityCompletion, model.CapabilityTools},
},
{
name: "model with tools capability from config and parser",
model: Model{
Config: model.ConfigV2{
Capabilities: []string{"completion", "tools"},
Parser: "qwen3-coder",
},
Template: chatTemplate,
},
expectedCaps: []model.Capability{model.CapabilityCompletion, model.CapabilityTools},
},
{
name: "model with vision capability",
model: Model{
+5
View File
@@ -5,12 +5,14 @@ import "context"
type modelCaches struct {
recommendations *modelRecommendationsCache
show *modelShowCache
modelList *modelListCache
}
func newModelCaches() *modelCaches {
return &modelCaches{
recommendations: newModelRecommendationsCache(),
show: newModelShowCache(),
modelList: newModelListCache(),
}
}
@@ -24,4 +26,7 @@ func (c *modelCaches) Start(ctx context.Context) {
if c.show != nil {
c.show.Start(ctx)
}
if c.modelList != nil {
c.modelList.Start(ctx)
}
}
+824
View File
@@ -0,0 +1,824 @@
package server
import (
"bufio"
"cmp"
"context"
"encoding/binary"
"encoding/json"
"fmt"
"io"
"log/slog"
"os"
"slices"
"strings"
"sync"
"time"
"github.com/ollama/ollama/api"
"github.com/ollama/ollama/manifest"
"github.com/ollama/ollama/model/parsers"
ollamatemplate "github.com/ollama/ollama/template"
"github.com/ollama/ollama/thinking"
"github.com/ollama/ollama/types/model"
)
type modelListSummary struct {
Model string
Name string
RemoteModel string
RemoteHost string
Size int64
Digest string
ModifiedAt time.Time
Details api.ModelDetails
Capabilities []model.Capability
}
type modelListCacheEntry struct {
Digest string
Summary modelListSummary
}
type modelListCache struct {
mu sync.RWMutex
entries map[string]modelListCacheEntry
once sync.Once
readyOnce sync.Once
ready chan struct{}
hydrateErr error
build func(model.Name, *manifest.Manifest) (modelListSummary, error)
}
func newModelListCache() *modelListCache {
return &modelListCache{
entries: make(map[string]modelListCacheEntry),
ready: make(chan struct{}),
build: buildModelListSummary,
}
}
func (c *modelListCache) Start(ctx context.Context) {
if c == nil {
return
}
c.once.Do(func() {
slog.Debug("starting model list cache")
go func() {
err := c.hydrate(ctx)
c.markReady(err)
if err != nil {
if ctx != nil && ctx.Err() != nil {
return
}
slog.Warn("model list cache hydration failed", "error", err)
}
}()
})
}
func (c *modelListCache) hydrate(ctx context.Context) error {
start := time.Now()
manifests, err := manifest.Manifests(true)
if err != nil {
return err
}
var hydrated, failed int
for name, mf := range manifests {
if ctx != nil {
if err := ctx.Err(); err != nil {
return err
}
}
summary, err := c.build(name, mf)
if err != nil {
failed++
slog.Warn("failed to hydrate model list cache", "model", name.String(), "error", err)
continue
}
c.set(name, mf.Digest(), summary)
hydrated++
}
slog.Info("model list cache hydration complete", "models", hydrated, "failures", failed, "elapsed", time.Since(start))
return nil
}
func (c *modelListCache) markReady(err error) {
c.mu.Lock()
c.hydrateErr = err
c.mu.Unlock()
c.readyOnce.Do(func() {
close(c.ready)
})
}
func (c *modelListCache) Wait(ctx context.Context) error {
if c == nil {
return nil
}
if ctx == nil {
ctx = context.Background()
}
select {
case <-c.ready:
c.mu.RLock()
err := c.hydrateErr
c.mu.RUnlock()
return err
case <-ctx.Done():
return ctx.Err()
}
}
func (c *modelListCache) List(ctx context.Context) ([]api.ListModelResponse, error) {
if err := c.Wait(ctx); err != nil {
return nil, err
}
if err := c.syncManifests(ctx); err != nil {
return nil, err
}
c.mu.RLock()
models := make([]api.ListModelResponse, 0, len(c.entries))
for _, entry := range c.entries {
models = append(models, entry.Summary.ListModelResponse())
}
c.mu.RUnlock()
sortListModelResponses(models)
return models, nil
}
func (c *modelListCache) syncManifests(ctx context.Context) error {
manifests, err := manifest.Manifests(true)
if err != nil {
return err
}
c.mu.RLock()
current := make(map[string]string, len(c.entries))
for name, entry := range c.entries {
current[name] = entry.Digest
}
c.mu.RUnlock()
type update struct {
name model.Name
digest string
summary modelListSummary
}
seen := make(map[string]struct{}, len(manifests))
stale := make(map[string]struct{})
var updates []update
for name, mf := range manifests {
if ctx != nil {
if err := ctx.Err(); err != nil {
return err
}
}
key := name.String()
digest := mf.Digest()
seen[key] = struct{}{}
if current[key] == digest {
continue
}
summary, err := c.build(name, mf)
if err != nil {
slog.Warn("failed to refresh model list cache", "model", key, "error", err)
if _, ok := current[key]; ok {
stale[key] = struct{}{}
}
continue
}
updates = append(updates, update{name: name, digest: digest, summary: summary})
}
c.mu.Lock()
for name := range c.entries {
if _, ok := seen[name]; !ok {
delete(c.entries, name)
continue
}
if _, ok := stale[name]; ok {
delete(c.entries, name)
}
}
for _, update := range updates {
c.entries[update.name.String()] = modelListCacheEntry{
Digest: update.digest,
Summary: cloneModelListSummary(update.summary),
}
}
c.mu.Unlock()
return nil
}
func (c *modelListCache) RefreshModel(name model.Name) error {
if c == nil {
return nil
}
if !name.IsFullyQualified() {
var err error
name, err = getExistingName(name)
if err != nil {
return err
}
}
mf, err := manifest.ParseNamedManifest(name)
if err != nil {
c.DeleteModel(name)
return err
}
summary, err := c.build(name, mf)
if err != nil {
c.DeleteModel(name)
return err
}
c.set(name, mf.Digest(), summary)
return nil
}
func (c *modelListCache) DeleteModel(name model.Name) {
if c == nil {
return
}
c.mu.Lock()
delete(c.entries, name.String())
c.mu.Unlock()
}
func (c *modelListCache) Get(name model.Name) (modelListSummary, bool) {
if c == nil {
return modelListSummary{}, false
}
if !name.IsFullyQualified() {
if existing, err := getExistingName(name); err == nil {
name = existing
}
}
c.mu.RLock()
entry, ok := c.entries[name.String()]
c.mu.RUnlock()
if !ok {
return modelListSummary{}, false
}
return cloneModelListSummary(entry.Summary), true
}
func (c *modelListCache) Len() int {
if c == nil {
return 0
}
c.mu.RLock()
defer c.mu.RUnlock()
return len(c.entries)
}
func (c *modelListCache) set(name model.Name, digest string, summary modelListSummary) {
c.mu.Lock()
c.entries[name.String()] = modelListCacheEntry{
Digest: digest,
Summary: cloneModelListSummary(summary),
}
c.mu.Unlock()
}
func buildModelListSummary(name model.Name, mf *manifest.Manifest) (modelListSummary, error) {
cfg, err := readModelListConfig(mf)
if err != nil {
return modelListSummary{}, err
}
var modified time.Time
if fi := mf.FileInfo(); fi != nil {
modified = fi.ModTime()
}
summary := modelListSummary{
Model: name.DisplayShortest(),
Name: name.DisplayShortest(),
RemoteModel: cfg.RemoteModel,
RemoteHost: cfg.RemoteHost,
Size: mf.Size(),
Digest: mf.Digest(),
ModifiedAt: modified,
Details: api.ModelDetails{
Format: cfg.ModelFormat,
Family: cfg.ModelFamily,
Families: append([]string(nil), cfg.ModelFamilies...),
ParameterSize: cfg.ModelType,
QuantizationLevel: cfg.FileType,
ContextLength: cfg.ContextLen,
EmbeddingLength: cfg.EmbedLen,
},
}
modelPath, projectorCount, tmpl, err := readModelListLayers(mf, &summary)
if err != nil {
return modelListSummary{}, err
}
if cfg.RemoteHost == "" && cfg.RemoteModel == "" && modelPath != "" {
info, err := readModelListGGUF(modelPath)
if err != nil {
slog.Debug("failed to read gguf model metadata", "model", name.String(), "error", err)
} else {
summary.Capabilities = appendModelListCapabilities(summary.Capabilities, info.Capabilities...)
if summary.Details.ContextLength == 0 {
summary.Details.ContextLength = info.ContextLength
}
if summary.Details.EmbeddingLength == 0 {
summary.Details.EmbeddingLength = info.EmbeddingLength
}
}
}
for _, c := range cfg.Capabilities {
summary.Capabilities = appendModelListCapability(summary.Capabilities, model.Capability(c))
}
builtinParser := parsers.ParserForName(cfg.Parser)
if tmpl != nil {
vars, err := tmpl.Vars()
if err != nil {
slog.Warn("model template contains errors", "model", name.String(), "error", err)
}
if slices.Contains(vars, "tools") || (builtinParser != nil && builtinParser.HasToolSupport()) {
summary.Capabilities = appendModelListCapability(summary.Capabilities, model.CapabilityTools)
}
if slices.Contains(vars, "suffix") {
summary.Capabilities = appendModelListCapability(summary.Capabilities, model.CapabilityInsert)
}
openingTag, closingTag := thinking.InferTags(tmpl.Template)
hasTags := openingTag != "" && closingTag != ""
isGptoss := slices.Contains([]string{"gptoss", "gpt-oss"}, cfg.ModelFamily)
if !slices.Contains(summary.Capabilities, model.CapabilityThinking) &&
(hasTags || isGptoss || (builtinParser != nil && builtinParser.HasThinkingSupport())) {
summary.Capabilities = appendModelListCapability(summary.Capabilities, model.CapabilityThinking)
}
}
if projectorCount > 0 {
summary.Capabilities = appendModelListCapability(summary.Capabilities, model.CapabilityVision)
}
if cfg.ModelFormat == "safetensors" && isGemma4Renderer(cfg.Renderer) {
summary.Capabilities = slices.DeleteFunc(summary.Capabilities, func(c model.Capability) bool {
return c == model.CapabilityVision || c == model.CapabilityAudio
})
}
return summary, nil
}
func readModelListConfig(mf *manifest.Manifest) (model.ConfigV2, error) {
var cfg model.ConfigV2
if mf == nil || mf.Config.Digest == "" {
return cfg, nil
}
f, err := mf.Config.Open()
if err != nil {
return cfg, err
}
defer f.Close()
if err := json.NewDecoder(f).Decode(&cfg); err != nil {
return cfg, err
}
return cfg, nil
}
func readModelListLayers(mf *manifest.Manifest, summary *modelListSummary) (string, int, *ollamatemplate.Template, error) {
var modelPath string
var projectorCount int
tmpl := ollamatemplate.DefaultTemplate
for _, layer := range mf.Layers {
switch layer.MediaType {
case "application/vnd.ollama.image.model":
filename, err := manifest.BlobsPath(layer.Digest)
if err != nil {
return "", 0, nil, err
}
modelPath = filename
summary.Details.ParentModel = layer.From
case "application/vnd.ollama.image.projector":
projectorCount++
case "application/vnd.ollama.image.prompt",
"application/vnd.ollama.image.template":
filename, err := manifest.BlobsPath(layer.Digest)
if err != nil {
return "", 0, nil, err
}
bts, err := os.ReadFile(filename)
if err != nil {
return "", 0, nil, err
}
tmpl, err = ollamatemplate.Parse(string(bts))
if err != nil {
return "", 0, nil, err
}
}
}
return modelPath, projectorCount, tmpl, nil
}
type modelListGGUF struct {
Capabilities []model.Capability
ContextLength int
EmbeddingLength int
}
const (
modelListGGUFMagicLE = 0x46554747
modelListGGUFMagicBE = 0x47475546
)
const (
modelListGGUFTypeUint8 uint32 = iota
modelListGGUFTypeInt8
modelListGGUFTypeUint16
modelListGGUFTypeInt16
modelListGGUFTypeUint32
modelListGGUFTypeInt32
modelListGGUFTypeFloat32
modelListGGUFTypeBool
modelListGGUFTypeString
modelListGGUFTypeArray
modelListGGUFTypeUint64
modelListGGUFTypeInt64
modelListGGUFTypeFloat64
)
// readModelListGGUF scans only the small GGUF header values launch needs
// and stops before tokenizer arrays. Using gguf.File.KeyValue for missing keys
// can otherwise advance through large arrays just to discover absence.
func readModelListGGUF(path string) (modelListGGUF, error) {
f, err := os.Open(path)
if err != nil {
return modelListGGUF{}, err
}
defer f.Close()
r := bufio.NewReaderSize(f, 32<<10)
var magic uint32
if err := binary.Read(r, binary.LittleEndian, &magic); err != nil {
return modelListGGUF{}, err
}
var byteOrder binary.ByteOrder = binary.LittleEndian
switch magic {
case modelListGGUFMagicLE:
case modelListGGUFMagicBE:
byteOrder = binary.BigEndian
default:
return modelListGGUF{}, fmt.Errorf("invalid file magic")
}
var version uint32
if err := binary.Read(r, byteOrder, &version); err != nil {
return modelListGGUF{}, err
}
var numKV uint64
switch version {
case 1:
var header struct {
NumTensor uint32
NumKV uint32
}
if err := binary.Read(r, byteOrder, &header); err != nil {
return modelListGGUF{}, err
}
numKV = uint64(header.NumKV)
default:
var header struct {
NumTensor uint64
NumKV uint64
}
if err := binary.Read(r, byteOrder, &header); err != nil {
return modelListGGUF{}, err
}
numKV = header.NumKV
}
info := modelListGGUF{}
var architecture string
var hasPoolingType bool
for range numKV {
key, err := readModelListGGUFString(r, byteOrder, version)
if err != nil {
return modelListGGUF{}, err
}
var valueType uint32
if err := binary.Read(r, byteOrder, &valueType); err != nil {
return modelListGGUF{}, err
}
if key == "general.architecture" {
value, err := readModelListGGUFStringValue(r, byteOrder, version, valueType)
if err != nil {
return modelListGGUF{}, err
}
architecture = value
continue
}
if architecture != "" && strings.HasPrefix(key, "tokenizer.") {
break
}
if architecture != "" && strings.HasPrefix(key, architecture+".") {
switch strings.TrimPrefix(key, architecture+".") {
case "pooling_type":
hasPoolingType = true
case "vision.block_count":
info.Capabilities = appendModelListCapability(info.Capabilities, model.CapabilityVision)
case "audio.block_count":
info.Capabilities = appendModelListCapability(info.Capabilities, model.CapabilityAudio)
case "context_length":
value, err := readModelListGGUFIntValue(r, byteOrder, version, valueType)
if err != nil {
return modelListGGUF{}, err
}
info.ContextLength = value
continue
case "embedding_length":
value, err := readModelListGGUFIntValue(r, byteOrder, version, valueType)
if err != nil {
return modelListGGUF{}, err
}
info.EmbeddingLength = value
continue
}
}
if err := skipModelListGGUFValue(r, byteOrder, version, valueType); err != nil {
return modelListGGUF{}, err
}
}
if hasPoolingType {
info.Capabilities = appendModelListCapability(info.Capabilities, model.CapabilityEmbedding)
} else {
info.Capabilities = appendModelListCapability(info.Capabilities, model.CapabilityCompletion)
}
return info, nil
}
func readModelListGGUFStringValue(r io.Reader, byteOrder binary.ByteOrder, version uint32, valueType uint32) (string, error) {
if valueType != modelListGGUFTypeString {
if err := skipModelListGGUFValue(r, byteOrder, version, valueType); err != nil {
return "", err
}
return "", fmt.Errorf("unexpected gguf string type %d", valueType)
}
return readModelListGGUFString(r, byteOrder, version)
}
func readModelListGGUFIntValue(r io.Reader, byteOrder binary.ByteOrder, version uint32, valueType uint32) (int, error) {
switch valueType {
case modelListGGUFTypeUint8:
var value uint8
if err := binary.Read(r, byteOrder, &value); err != nil {
return 0, err
}
return int(value), nil
case modelListGGUFTypeInt8:
var value int8
if err := binary.Read(r, byteOrder, &value); err != nil {
return 0, err
}
return int(value), nil
case modelListGGUFTypeUint16:
var value uint16
if err := binary.Read(r, byteOrder, &value); err != nil {
return 0, err
}
return int(value), nil
case modelListGGUFTypeInt16:
var value int16
if err := binary.Read(r, byteOrder, &value); err != nil {
return 0, err
}
return int(value), nil
case modelListGGUFTypeUint32:
var value uint32
if err := binary.Read(r, byteOrder, &value); err != nil {
return 0, err
}
return int(value), nil
case modelListGGUFTypeInt32:
var value int32
if err := binary.Read(r, byteOrder, &value); err != nil {
return 0, err
}
return int(value), nil
case modelListGGUFTypeUint64:
var value uint64
if err := binary.Read(r, byteOrder, &value); err != nil {
return 0, err
}
return int(value), nil
case modelListGGUFTypeInt64:
var value int64
if err := binary.Read(r, byteOrder, &value); err != nil {
return 0, err
}
return int(value), nil
default:
if err := skipModelListGGUFValue(r, byteOrder, version, valueType); err != nil {
return 0, err
}
return 0, fmt.Errorf("unexpected gguf integer type %d", valueType)
}
}
func skipModelListGGUFValue(r io.Reader, byteOrder binary.ByteOrder, version uint32, valueType uint32) error {
switch valueType {
case modelListGGUFTypeUint8, modelListGGUFTypeInt8, modelListGGUFTypeBool:
return discardModelListGGUFBytes(r, 1)
case modelListGGUFTypeUint16, modelListGGUFTypeInt16:
return discardModelListGGUFBytes(r, 2)
case modelListGGUFTypeUint32, modelListGGUFTypeInt32, modelListGGUFTypeFloat32:
return discardModelListGGUFBytes(r, 4)
case modelListGGUFTypeUint64, modelListGGUFTypeInt64, modelListGGUFTypeFloat64:
return discardModelListGGUFBytes(r, 8)
case modelListGGUFTypeString:
return skipModelListGGUFString(r, byteOrder, version)
case modelListGGUFTypeArray:
var arrayType uint32
if err := binary.Read(r, byteOrder, &arrayType); err != nil {
return err
}
var count uint64
if err := binary.Read(r, byteOrder, &count); err != nil {
return err
}
return skipModelListGGUFArray(r, byteOrder, version, arrayType, count)
default:
return fmt.Errorf("unsupported gguf value type %d", valueType)
}
}
func skipModelListGGUFArray(r io.Reader, byteOrder binary.ByteOrder, version uint32, arrayType uint32, count uint64) error {
var size uint64
switch arrayType {
case modelListGGUFTypeUint8, modelListGGUFTypeInt8, modelListGGUFTypeBool:
size = 1
case modelListGGUFTypeUint16, modelListGGUFTypeInt16:
size = 2
case modelListGGUFTypeUint32, modelListGGUFTypeInt32, modelListGGUFTypeFloat32:
size = 4
case modelListGGUFTypeUint64, modelListGGUFTypeInt64, modelListGGUFTypeFloat64:
size = 8
case modelListGGUFTypeString:
for range count {
if err := skipModelListGGUFString(r, byteOrder, version); err != nil {
return err
}
}
return nil
default:
return fmt.Errorf("unsupported gguf array type %d", arrayType)
}
return discardModelListGGUFBytes(r, int64(count*size))
}
func readModelListGGUFString(r io.Reader, byteOrder binary.ByteOrder, version uint32) (string, error) {
var length uint64
if err := binary.Read(r, byteOrder, &length); err != nil {
return "", err
}
if length == 0 {
return "", nil
}
bts := make([]byte, length)
if _, err := io.ReadFull(r, bts); err != nil {
return "", err
}
if version == 1 && bts[len(bts)-1] == 0 {
bts = bts[:len(bts)-1]
}
return string(bts), nil
}
func skipModelListGGUFString(r io.Reader, byteOrder binary.ByteOrder, version uint32) error {
var length uint64
if err := binary.Read(r, byteOrder, &length); err != nil {
return err
}
return discardModelListGGUFBytes(r, int64(length))
}
func discardModelListGGUFBytes(r io.Reader, n int64) error {
if n <= 0 {
return nil
}
_, err := io.CopyN(io.Discard, r, n)
return err
}
func appendModelListCapabilities(capabilities []model.Capability, values ...model.Capability) []model.Capability {
for _, capability := range values {
capabilities = appendModelListCapability(capabilities, capability)
}
return capabilities
}
func appendModelListCapability(capabilities []model.Capability, capability model.Capability) []model.Capability {
if capability == "" || slices.Contains(capabilities, capability) {
return capabilities
}
return append(capabilities, capability)
}
func cloneModelListSummary(summary modelListSummary) modelListSummary {
summary.Details.Families = append([]string(nil), summary.Details.Families...)
summary.Capabilities = append([]model.Capability(nil), summary.Capabilities...)
return summary
}
func (s modelListSummary) ListModelResponse() api.ListModelResponse {
resp := api.ListModelResponse{
Model: s.Model,
Name: s.Name,
RemoteModel: s.RemoteModel,
RemoteHost: s.RemoteHost,
Size: s.Size,
Digest: s.Digest,
ModifiedAt: s.ModifiedAt,
Details: api.ModelDetails{
ParentModel: s.Details.ParentModel,
Format: s.Details.Format,
Family: s.Details.Family,
Families: append([]string(nil), s.Details.Families...),
ParameterSize: s.Details.ParameterSize,
QuantizationLevel: s.Details.QuantizationLevel,
ContextLength: s.Details.ContextLength,
EmbeddingLength: s.Details.EmbeddingLength,
},
}
resp.Capabilities = append([]model.Capability(nil), s.Capabilities...)
return resp
}
func sortListModelResponses(models []api.ListModelResponse) {
slices.SortStableFunc(models, func(i, j api.ListModelResponse) int {
// Preserve the existing /api/tags order: most recently modified first.
return cmp.Compare(j.ModifiedAt.Unix(), i.ModifiedAt.Unix())
})
}
func (s *Server) refreshModelListCache(name model.Name) {
if s == nil || s.modelCaches == nil || s.modelCaches.modelList == nil {
return
}
if err := s.modelCaches.modelList.RefreshModel(name); err != nil {
slog.Warn("failed to refresh model list cache", "model", name.String(), "error", err)
}
}
func (s *Server) deleteModelListCache(name model.Name) {
if s == nil || s.modelCaches == nil || s.modelCaches.modelList == nil {
return
}
s.modelCaches.modelList.DeleteModel(name)
}
+239
View File
@@ -0,0 +1,239 @@
package server
import (
"context"
"errors"
"net/http"
"slices"
"testing"
"github.com/gin-gonic/gin"
"github.com/ollama/ollama/api"
"github.com/ollama/ollama/manifest"
"github.com/ollama/ollama/types/model"
)
func TestModelListCacheHydratesSummary(t *testing.T) {
gin.SetMode(gin.TestMode)
setTestHome(t, t.TempDir())
createListCacheModel(t, "list-cache", map[string]any{
"test.context_length": uint32(4096),
"test.embedding_length": uint32(384),
}, "{{ .prompt }}{{ if .tools }}{{ .tools }}{{ end }}{{ if .suffix }}{{ .suffix }}{{ end }}")
cache := newModelListCache()
if err := cache.hydrate(context.Background()); err != nil {
t.Fatalf("hydrate failed: %v", err)
}
summary, ok := cache.Get(model.ParseName("list-cache"))
if !ok {
t.Fatal("list summary missing")
}
if summary.Model != "list-cache:latest" || summary.Name != "list-cache:latest" {
t.Fatalf("summary model/name = %q/%q, want list-cache:latest", summary.Model, summary.Name)
}
if summary.Digest == "" {
t.Fatal("summary digest is empty")
}
if summary.Size == 0 {
t.Fatal("summary size is zero")
}
if summary.Details.Family != "test" || summary.Details.Format != "gguf" {
t.Fatalf("summary details = %+v, want gguf/test", summary.Details)
}
if summary.Details.ContextLength != 4096 {
t.Fatalf("context length = %d, want 4096", summary.Details.ContextLength)
}
if summary.Details.EmbeddingLength != 384 {
t.Fatalf("embedding length = %d, want 384", summary.Details.EmbeddingLength)
}
for _, capability := range []model.Capability{model.CapabilityCompletion, model.CapabilityTools, model.CapabilityInsert} {
if !slices.Contains(summary.Capabilities, capability) {
t.Fatalf("capabilities = %v, want %s", summary.Capabilities, capability)
}
}
listModel := summary.ListModelResponse()
if !slices.Contains(listModel.Capabilities, model.CapabilityTools) ||
listModel.Details.ContextLength != 4096 ||
listModel.Details.EmbeddingLength != 384 {
t.Fatalf("list response = %+v, want capabilities/context/embedding", listModel)
}
}
func TestModelListCacheRefreshUpdatesEntry(t *testing.T) {
gin.SetMode(gin.TestMode)
setTestHome(t, t.TempDir())
createListCacheModel(t, "list-refresh", map[string]any{"test.context_length": uint32(1024)}, "")
cache := newModelListCache()
if err := cache.hydrate(context.Background()); err != nil {
t.Fatalf("hydrate failed: %v", err)
}
name := model.ParseName("list-refresh")
first, ok := cache.Get(name)
if !ok {
t.Fatal("list summary missing")
}
changeShowCacheManifest(t, "list-refresh")
if err := cache.RefreshModel(name); err != nil {
t.Fatalf("refresh failed: %v", err)
}
refreshed, ok := cache.Get(name)
if !ok {
t.Fatal("refreshed list summary missing")
}
if refreshed.Digest == first.Digest {
t.Fatalf("digest did not change after refresh: %s", refreshed.Digest)
}
if cache.Len() != 1 {
t.Fatalf("cache entries = %d, want 1", cache.Len())
}
}
func TestModelListCacheMutationHooks(t *testing.T) {
gin.SetMode(gin.TestMode)
setTestHome(t, t.TempDir())
cache := newModelListCache()
s := Server{modelCaches: &modelCaches{modelList: cache}}
_, digest := createBinFile(t, map[string]any{"test.context_length": uint32(2048)}, nil)
w := createRequest(t, s.CreateHandler, api.CreateRequest{
Model: "list-hooks",
Files: map[string]string{"model.gguf": digest},
Stream: &stream,
})
if w.Code != http.StatusOK {
t.Fatalf("create model status = %d, want 200: %s", w.Code, w.Body.String())
}
if _, ok := cache.Get(model.ParseName("list-hooks")); !ok {
t.Fatal("create did not refresh model list cache")
}
w = createRequest(t, s.CopyHandler, api.CopyRequest{
Source: "list-hooks",
Destination: "list-hooks-copy",
})
if w.Code != http.StatusOK {
t.Fatalf("copy model status = %d, want 200: %s", w.Code, w.Body.String())
}
if _, ok := cache.Get(model.ParseName("list-hooks-copy")); !ok {
t.Fatal("copy did not refresh model list cache")
}
w = createRequest(t, s.DeleteHandler, api.DeleteRequest{Model: "list-hooks-copy"})
if w.Code != http.StatusOK {
t.Fatalf("delete model status = %d, want 200: %s", w.Code, w.Body.String())
}
if _, ok := cache.Get(model.ParseName("list-hooks-copy")); ok {
t.Fatal("delete did not remove model list cache entry")
}
}
func TestModelListCacheSyncsManifestChanges(t *testing.T) {
gin.SetMode(gin.TestMode)
setTestHome(t, t.TempDir())
createListCacheModel(t, "list-sync-a", map[string]any{"test.context_length": uint32(1024)}, "")
cache := newModelListCache()
cache.Start(context.Background())
if err := cache.Wait(context.Background()); err != nil {
t.Fatalf("wait failed: %v", err)
}
createListCacheModel(t, "list-sync-b", map[string]any{"test.context_length": uint32(2048)}, "")
models, err := cache.List(context.Background())
if err != nil {
t.Fatalf("list failed: %v", err)
}
names := make([]string, 0, len(models))
for _, m := range models {
names = append(names, m.Name)
}
for _, want := range []string{"list-sync-a:latest", "list-sync-b:latest"} {
if !slices.Contains(names, want) {
t.Fatalf("names = %v, want %s", names, want)
}
}
var other Server
w := createRequest(t, other.DeleteHandler, api.DeleteRequest{Model: "list-sync-a"})
if w.Code != http.StatusOK {
t.Fatalf("delete model status = %d, want 200: %s", w.Code, w.Body.String())
}
models, err = cache.List(context.Background())
if err != nil {
t.Fatalf("list after delete failed: %v", err)
}
names = names[:0]
for _, m := range models {
names = append(names, m.Name)
}
if slices.Contains(names, "list-sync-a:latest") || !slices.Contains(names, "list-sync-b:latest") {
t.Fatalf("names after delete = %v, want only list-sync-b", names)
}
}
func TestModelListCacheSyncDropsStaleEntryOnRefreshFailure(t *testing.T) {
gin.SetMode(gin.TestMode)
setTestHome(t, t.TempDir())
createListCacheModel(t, "list-stale", map[string]any{"test.context_length": uint32(1024)}, "")
cache := newModelListCache()
cache.Start(context.Background())
if err := cache.Wait(context.Background()); err != nil {
t.Fatalf("wait failed: %v", err)
}
name := model.ParseName("list-stale")
if _, ok := cache.Get(name); !ok {
t.Fatal("list summary missing")
}
changeShowCacheManifest(t, "list-stale")
cache.build = func(model.Name, *manifest.Manifest) (modelListSummary, error) {
return modelListSummary{}, errors.New("refresh failed")
}
models, err := cache.List(context.Background())
if err != nil {
t.Fatalf("list failed: %v", err)
}
if len(models) != 0 {
t.Fatalf("models = %+v, want stale entry removed", models)
}
if _, ok := cache.Get(name); ok {
t.Fatal("stale entry remained in cache after refresh failure")
}
}
func createListCacheModel(t *testing.T, name string, kv map[string]any, tmpl string) {
t.Helper()
_, digest := createBinFile(t, kv, nil)
req := api.CreateRequest{
Model: name,
Files: map[string]string{"model.gguf": digest},
Stream: &stream,
}
if tmpl != "" {
req.Template = tmpl
}
var s Server
w := createRequest(t, s.CreateHandler, req)
if w.Code != http.StatusOK {
t.Fatalf("create model status = %d, want 200: %s", w.Code, w.Body.String())
}
}
+26 -39
View File
@@ -26,13 +26,15 @@ import (
The /api/show cache stores full api.ShowResponse values because callers use
more than capabilities: launch flows also need context length, embeddings
metadata, quantization details, remote metadata, and model-specific fields.
TODO(parthsareen): Consider removing show cache if /api/tags grows to cover
the remaining callers.
Local model entries are stored by canonical model name and verbose flag, with
the manifest digest recorded in the entry. The manifest digest is the freshness
boundary: if the model content changes, the digest changes, so the previous
response is replaced instead of accumulating under an old digest key. Requests
with System or Options overlays bypass the cache because those overlays mutate
the effective show response.
Local model entries are stored lazily by canonical model name and verbose flag,
with the manifest digest recorded in the entry. The manifest digest is the
freshness boundary: if the model content changes, the digest changes, so the
previous response is replaced instead of accumulating under an old digest key.
Requests with System or Options overlays bypass the cache because those overlays
mutate the effective show response.
Cloud model entries are keyed by normalized cloud base model name and verbose.
They use stale-while-revalidate behavior: a warm read returns the cached
@@ -42,10 +44,11 @@ in separate maps, so a local "qwen3.5" and an explicit "qwen3.5:cloud" cannot
collide. The cloud suffix is request routing intent; api.ShowResponse does not
carry a model-name field to reconstruct on the way out.
The cache is process-local. Startup hydration runs asynchronously from the
current local manifests and cloud tags; no show responses are written to or read
from ~/.ollama/cache/show. That keeps cache lifetime tied to the server process
and avoids snapshot freshness and invalidation cases for this iteration.
The cache is process-local. Cloud startup hydration runs asynchronously from
cloud tags, while local show responses are populated on demand. No show
responses are written to or read from ~/.ollama/cache/show. That keeps cache
lifetime tied to the server process and avoids snapshot freshness and
invalidation cases for this iteration.
*/
const (
@@ -117,9 +120,9 @@ func modelShowCacheable(req api.ShowRequest) bool {
return req.System == "" && len(req.Options) == 0
}
// Start kicks off non-blocking startup hydration. The cache remains
// process-local; warm entries appear as the background local and cloud scans
// populate the maps.
// Start kicks off non-blocking startup hydration for cloud entries. Local show
// responses stay lazy because even non-verbose show must load GGUF metadata that
// is expensive for large model stores.
func (c *modelShowCache) Start(ctx context.Context) {
c.once.Do(func() {
slog.Debug("starting model show cache")
@@ -127,34 +130,18 @@ func (c *modelShowCache) Start(ctx context.Context) {
})
}
// runStartup hydrates local and cloud caches concurrently. It is only called in
// a goroutine from Start, so manifest scans and cloud requests cannot delay the
// listener from accepting traffic.
// runStartup hydrates the cloud cache. It is only called in a goroutine from
// Start, so cloud requests cannot delay the listener from accepting traffic.
func (c *modelShowCache) runStartup(ctx context.Context) {
var wg sync.WaitGroup
wg.Add(2)
go func() {
defer wg.Done()
if err := c.hydrateLocal(ctx); err != nil && !errors.Is(err, context.Canceled) {
slog.Warn("model show local cache hydration failed", "error", err)
if err := c.hydrateCloud(ctx); err != nil {
switch {
case errors.Is(err, context.Canceled):
case errors.Is(err, errModelShowNoCloud):
slog.Debug("skipping model show cloud cache hydration because cloud is disabled")
default:
slog.Warn("model show cloud cache hydration failed", "error", err)
}
}()
go func() {
defer wg.Done()
if err := c.hydrateCloud(ctx); err != nil {
switch {
case errors.Is(err, context.Canceled):
case errors.Is(err, errModelShowNoCloud):
slog.Debug("skipping model show cloud cache hydration because cloud is disabled")
default:
slog.Warn("model show cloud cache hydration failed", "error", err)
}
}
}()
wg.Wait()
}
}
// GetLocal returns a cached local show response when the current manifest
+19
View File
@@ -148,6 +148,25 @@ func TestModelShowCacheLocalHydrationSkipsUnchangedInMemory(t *testing.T) {
}
}
func TestModelShowCacheStartupSkipsLocalHydration(t *testing.T) {
gin.SetMode(gin.TestMode)
setTestHome(t, t.TempDir())
t.Setenv("OLLAMA_NO_CLOUD", "1")
createShowCacheModel(t, "show-cache-startup", map[string]any{"test.context_length": uint32(1024)})
cache := newModelShowCache()
cache.getModelInfo = func(req api.ShowRequest) (*api.ShowResponse, error) {
t.Fatalf("startup should not hydrate local show cache, got request: %+v", req)
return nil, nil
}
cache.runStartup(context.Background())
if len(cache.local) != 0 {
t.Fatalf("local cache entries = %d, want 0", len(cache.local))
}
}
func TestModelShowCacheBypassesSystemAndOptionsOverlays(t *testing.T) {
gin.SetMode(gin.TestMode)
setTestHome(t, t.TempDir())
+10 -3
View File
@@ -75,7 +75,9 @@ func chatPrompt(ctx context.Context, m *Model, tokenize tokenizeFunc, opts *api.
slog.Debug("truncating input messages which exceed context length", "truncated", len(msgs[currMsgIdx:]))
}
for cnt, msg := range msgs[currMsgIdx:] {
renderMsgs := slices.Clone(msgs)
for cnt, msg := range renderMsgs[currMsgIdx:] {
if slices.Contains(m.Config.ModelFamilies, "mllama") && len(msg.Images) > 1 {
return "", nil, errors.New("this model only supports one image while more than one image requested")
}
@@ -101,11 +103,16 @@ func chatPrompt(ctx context.Context, m *Model, tokenize tokenizeFunc, opts *api.
prompt = strings.Replace(prompt, "[img]", imgTag, 1)
}
}
msgs[currMsgIdx+cnt].Content = prefix + prompt
if m.Config.Renderer != "" {
continue
}
renderMsgs[currMsgIdx+cnt].Content = prefix + prompt
}
// truncate any messages that do not fit into the context window
p, err := renderPrompt(m, append(system, msgs[currMsgIdx:]...), tools, think)
p, err := renderPrompt(m, append(system, renderMsgs[currMsgIdx:]...), tools, think)
if err != nil {
return "", nil, err
}
+160 -1
View File
@@ -401,11 +401,170 @@ func TestChatPromptGLMOcrRendererAddsImageTags(t *testing.T) {
t.Fatalf("len(images) = %d, want %d", got, want)
}
if !strings.Contains(prompt, "<|user|>\n[img-0][img-1]extract text") {
if !strings.Contains(prompt, "<|user|>\n[img-0][img-1] extract text") {
t.Fatalf("prompt missing glm-ocr image tags, got: %q", prompt)
}
}
func TestChatPromptRendererAddsToolImageTags(t *testing.T) {
msgs := []api.Message{
{
Role: "user",
Content: "look at this file",
Images: []api.ImageData{[]byte("img-1")},
},
{
Role: "assistant",
ToolCalls: []api.ToolCall{
{
ID: "call_read",
Function: api.ToolCallFunction{
Name: "Read",
},
},
},
},
{
Role: "tool",
Content: "attached image",
Images: []api.ImageData{[]byte("img-2")},
ToolCallID: "call_read",
},
}
tests := []struct {
name string
renderer string
wantUserTag string
wantToolContent string
}{
{
name: "gemma4",
renderer: "gemma4",
wantUserTag: "<|turn>user\n[img-0] look at this file<turn|>\n",
wantToolContent: "[img-1] attached image",
},
{
name: "qwen3-vl",
renderer: "qwen3-vl-instruct",
wantUserTag: "<|im_start|>user\n[img-0] look at this file<|im_end|>\n",
wantToolContent: "<tool_response>\n[img-1] attached image\n</tool_response>",
},
{
name: "qwen3.5",
renderer: "qwen3.5",
wantUserTag: "<|im_start|>user\n[img-0] look at this file<|im_end|>\n",
wantToolContent: "<tool_response>\n[img-1] attached image\n</tool_response>",
},
{
name: "glm-ocr",
renderer: "glm-ocr",
wantUserTag: "<|user|>\n[img-0] look at this file",
wantToolContent: "<tool_response>\n[img-1] attached image\n</tool_response>",
},
{
name: "nemotron-3-nano",
renderer: "nemotron-3-nano",
wantUserTag: "<|im_start|>user\n[img-0] look at this file<|im_end|>\n",
wantToolContent: "<tool_response>\n[img-1] attached image\n</tool_response>",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
m := Model{
Config: model.ConfigV2{Renderer: tt.renderer},
ProjectorPaths: []string{"vision"},
}
opts := api.Options{Runner: api.Runner{NumCtx: 8192}}
think := false
prompt, images, err := chatPrompt(t.Context(), &m, mockRunner{}.Tokenize, &opts, msgs, nil, &api.ThinkValue{Value: think}, true)
if err != nil {
t.Fatal(err)
}
if got, want := len(images), 2; got != want {
t.Fatalf("len(images) = %d, want %d", got, want)
}
if !strings.Contains(prompt, tt.wantUserTag) {
t.Fatalf("prompt missing user image tag, got: %q", prompt)
}
if !strings.Contains(prompt, tt.wantToolContent) {
t.Fatalf("prompt missing tool image tag, got: %q", prompt)
}
})
}
}
func TestChatPromptRendererPreservesExplicitImagePlaceholders(t *testing.T) {
msgs := []api.Message{
{
Role: "user",
Content: "compare [img] and [img]",
Images: []api.ImageData{[]byte("img-1"), []byte("img-2")},
},
}
tests := []struct {
name string
renderer string
wantSnippet string
}{
{
name: "gemma4",
renderer: "gemma4",
wantSnippet: "<|turn>user\ncompare [img-0] and [img-1]<turn|>\n",
},
{
name: "qwen3-vl",
renderer: "qwen3-vl-instruct",
wantSnippet: "<|im_start|>user\ncompare [img-0] and [img-1]<|im_end|>\n",
},
{
name: "qwen3.5",
renderer: "qwen3.5",
wantSnippet: "<|im_start|>user\ncompare [img-0] and [img-1]<|im_end|>\n",
},
{
name: "glm-ocr",
renderer: "glm-ocr",
wantSnippet: "<|user|>\ncompare [img-0] and [img-1]",
},
{
name: "nemotron-3-nano",
renderer: "nemotron-3-nano",
wantSnippet: "<|im_start|>user\ncompare [img-0] and [img-1]<|im_end|>\n",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
m := Model{
Config: model.ConfigV2{Renderer: tt.renderer},
ProjectorPaths: []string{"vision"},
}
opts := api.Options{Runner: api.Runner{NumCtx: 8192}}
think := false
prompt, images, err := chatPrompt(t.Context(), &m, mockRunner{}.Tokenize, &opts, msgs, nil, &api.ThinkValue{Value: think}, true)
if err != nil {
t.Fatal(err)
}
if got, want := len(images), 2; got != want {
t.Fatalf("len(images) = %d, want %d", got, want)
}
if !strings.Contains(prompt, tt.wantSnippet) {
t.Fatalf("prompt missing replaced placeholders, got: %q", prompt)
}
})
}
}
func TestRenderPromptResolvesDynamicGemma4Renderer(t *testing.T) {
msgs := []api.Message{{Role: "user", Content: "Hello"}}
+13 -43
View File
@@ -969,7 +969,10 @@ func (s *Server) PullHandler(c *gin.Context) {
if err := PullModel(ctx, name.DisplayShortest(), regOpts, fn); err != nil {
ch <- gin.H{"error": err.Error()}
return
}
s.refreshModelListCache(name)
}()
if req.Stream != nil && !*req.Stream {
@@ -1108,6 +1111,8 @@ func (s *Server) DeleteHandler(c *gin.Context) {
return
}
s.deleteModelListCache(n)
if err := m.RemoveLayers(); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
@@ -1431,54 +1436,17 @@ func getModelData(digest string, verbose bool) (ggml.KV, ggml.Tensors, error) {
}
func (s *Server) ListHandler(c *gin.Context) {
ms, err := manifest.Manifests(true)
if s.modelCaches == nil || s.modelCaches.modelList == nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": "model list cache unavailable"})
return
}
models, err := s.modelCaches.modelList.List(c.Request.Context())
if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
models := []api.ListModelResponse{}
for n, m := range ms {
var cf model.ConfigV2
if m.Config.Digest != "" {
f, err := m.Config.Open()
if err != nil {
slog.Warn("bad manifest filepath", "name", n, "error", err)
continue
}
defer f.Close()
if err := json.NewDecoder(f).Decode(&cf); err != nil {
slog.Warn("bad manifest config", "name", n, "error", err)
continue
}
}
// tag should never be masked
models = append(models, api.ListModelResponse{
Model: n.DisplayShortest(),
Name: n.DisplayShortest(),
RemoteModel: cf.RemoteModel,
RemoteHost: cf.RemoteHost,
Size: m.Size(),
Digest: m.Digest(),
ModifiedAt: m.FileInfo().ModTime(),
Details: api.ModelDetails{
Format: cf.ModelFormat,
Family: cf.ModelFamily,
Families: cf.ModelFamilies,
ParameterSize: cf.ModelType,
QuantizationLevel: cf.FileType,
},
})
}
slices.SortStableFunc(models, func(i, j api.ListModelResponse) int {
// most recently modified first
return cmp.Compare(j.ModifiedAt.Unix(), i.ModifiedAt.Unix())
})
c.JSON(http.StatusOK, api.ListResponse{Models: models})
}
@@ -1518,6 +1486,8 @@ func (s *Server) CopyHandler(c *gin.Context) {
c.JSON(http.StatusNotFound, gin.H{"error": fmt.Sprintf("model %q not found", r.Source)})
} else if err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
} else {
s.refreshModelListCache(dst)
}
}
+13 -1
View File
@@ -1,6 +1,7 @@
package server
import (
"context"
"encoding/json"
"net/http"
"slices"
@@ -28,7 +29,12 @@ func TestList(t *testing.T) {
"myhost/mynamespace/lips:code",
}
var s Server
s := Server{modelCaches: &modelCaches{modelList: newModelListCache()}}
s.modelCaches.modelList.Start(context.Background())
if err := s.modelCaches.modelList.Wait(context.Background()); err != nil {
t.Fatal(err)
}
for _, n := range expectNames {
_, digest := createBinFile(t, nil, nil)
@@ -63,4 +69,10 @@ func TestList(t *testing.T) {
if !slices.Equal(actualNames, expectNames) {
t.Fatalf("expected slices to be equal %v", actualNames)
}
for _, m := range resp.Models {
if !slices.Contains(m.Capabilities, "completion") {
t.Fatalf("capabilities for %q = %v, want completion", m.Name, m.Capabilities)
}
}
}
+10 -4
View File
@@ -93,6 +93,9 @@ func (t *panicTransport) RoundTrip(r *http.Request) (*http.Response, error) {
var panicOnRoundTrip = &http.Client{Transport: &panicTransport{}}
func TestRoutes(t *testing.T) {
modelsDir := t.TempDir()
t.Setenv("OLLAMA_MODELS", modelsDir)
type testCase struct {
Name string
Method string
@@ -101,6 +104,12 @@ func TestRoutes(t *testing.T) {
Expected func(t *testing.T, resp *http.Response)
}
s := &Server{modelCaches: &modelCaches{modelList: newModelListCache()}}
s.modelCaches.modelList.Start(context.Background())
if err := s.modelCaches.modelList.Wait(context.Background()); err != nil {
t.Fatal(err)
}
createTestModel := func(t *testing.T, name string) {
t.Helper()
@@ -138,6 +147,7 @@ func TestRoutes(t *testing.T) {
if err := createModel(r, modelName, baseLayers, config, fn); err != nil {
t.Fatal(err)
}
s.refreshModelListCache(modelName)
}
testCases := []testCase{
@@ -496,9 +506,6 @@ func TestRoutes(t *testing.T) {
},
}
modelsDir := t.TempDir()
t.Setenv("OLLAMA_MODELS", modelsDir)
rc := &ollama.Registry{
// This is a temporary measure to allow us to move forward,
// surfacing any code contacting ollama.com we do not intended
@@ -514,7 +521,6 @@ func TestRoutes(t *testing.T) {
HTTPClient: panicOnRoundTrip,
}
s := &Server{}
router, err := s.GenerateRoutes(rc)
if err != nil {
t.Fatalf("failed to generate routes: %v", err)
+25 -5
View File
@@ -536,7 +536,7 @@ func TestSchedGetRunnerReusesSameDigestWhenModelPathEmpty(t *testing.T) {
}
func TestSchedExpireRunner(t *testing.T) {
ctx, done := context.WithTimeout(t.Context(), 20*time.Millisecond)
ctx, done := context.WithCancel(t.Context())
defer done()
s := InitScheduler(ctx)
s.waitForRecovery = 10 * time.Millisecond
@@ -556,8 +556,11 @@ func TestSchedExpireRunner(t *testing.T) {
{Name: "output.weight", Kind: uint32(0), Offset: uint64(0), Shape: []uint64{1, 1, 1, 1}, WriterTo: bytes.NewReader(make([]byte, 32))},
})
reqCtx, cancelReq := context.WithCancel(ctx)
defer cancelReq()
req := &LlmRequest{
ctx: ctx,
ctx: reqCtx,
model: &Model{ModelPath: modelPath},
opts: api.DefaultOptions(),
successCh: make(chan *runnerRef, 1),
@@ -587,16 +590,33 @@ func TestSchedExpireRunner(t *testing.T) {
s.loadedMu.Unlock()
}
s.expireRunner(&Model{ModelPath: modelPath})
completedDone := make(chan struct{})
go func() {
defer close(completedDone)
s.processCompleted(ctx)
}()
s.finishedReqCh <- req
s.processCompleted(ctx)
s.expireRunner(&Model{ModelPath: modelPath})
cancelReq()
select {
case <-s.unloadedCh:
case <-time.After(5 * time.Second):
t.Fatal("expected model to be unloaded")
}
s.loadedMu.Lock()
if len(s.loaded) != 0 {
t.Fatalf("expected model to be unloaded")
}
s.loadedMu.Unlock()
done()
select {
case <-completedDone:
case <-time.After(5 * time.Second):
t.Fatal("expected completed loop to stop")
}
}
// TODO - add one scenario that triggers the bogus finished event with positive ref count
+38 -5
View File
@@ -243,6 +243,40 @@ func appendLayersManifestWriter(next create.ManifestWriter, extra []create.Layer
}
}
func draftMetadata(draftDir string) (*model.Draft, error) {
configPath := filepath.Join(draftDir, "config.json")
data, err := os.ReadFile(configPath)
if err != nil {
return nil, fmt.Errorf("failed to read draft config %s: %w", configPath, err)
}
var cfg struct {
Architectures []string `json:"architectures"`
ModelType string `json:"model_type"`
}
if err := json.Unmarshal(data, &cfg); err != nil {
return nil, fmt.Errorf("failed to parse draft config %s: %w", configPath, err)
}
arch := ""
if len(cfg.Architectures) > 0 {
arch = cfg.Architectures[0]
}
if arch == "" {
arch = cfg.ModelType
}
if arch == "" {
return nil, fmt.Errorf("draft architecture not found in %s", configPath)
}
return &model.Draft{
ModelFormat: "safetensors",
Architecture: arch,
TensorPrefix: "draft.",
Config: "draft/config.json",
}, nil
}
func createModelFromBaseWithDraft(opts CreateOptions, draftLayers []create.LayerInfo, progressFn func(string)) error {
progressFn(fmt.Sprintf("loading base model %s", opts.ModelDir))
baseManifest, err := imagemanifest.LoadManifest(opts.ModelDir)
@@ -487,12 +521,11 @@ func newManifestWriter(opts CreateOptions, capabilities []string, parserName, re
configData.Parser = resolveParserName(opts.Modelfile, parserName)
configData.Renderer = resolveRendererName(opts.Modelfile, rendererName)
if opts.Modelfile != nil && opts.Modelfile.Draft != "" {
configData.Draft = &model.Draft{
ModelFormat: "safetensors",
Architecture: "Gemma4AssistantForCausalLM",
TensorPrefix: "draft.",
Config: "draft/config.json",
draft, err := draftMetadata(opts.Modelfile.Draft)
if err != nil {
return err
}
configData.Draft = draft
}
configJSON, err := json.Marshal(configData)
if err != nil {
+9 -1
View File
@@ -544,10 +544,15 @@ func TestNewManifestWriter_PopulatesFileTypeFromQuantize(t *testing.T) {
func TestNewManifestWriter_PopulatesDraftMetadata(t *testing.T) {
t.Setenv("OLLAMA_MODELS", t.TempDir())
draftDir := t.TempDir()
if err := os.WriteFile(filepath.Join(draftDir, "config.json"), []byte(`{"architectures":["DFlashDraftModel"],"model_type":"qwen3"}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
opts := CreateOptions{
ModelName: "test-draft",
ModelDir: t.TempDir(),
Modelfile: &ModelfileConfig{Draft: "/tmp/assistant"},
Modelfile: &ModelfileConfig{Draft: draftDir},
}
writer := newManifestWriter(opts, []string{"completion"}, "gemma4", "gemma4")
@@ -581,6 +586,9 @@ func TestNewManifestWriter_PopulatesDraftMetadata(t *testing.T) {
if cfg.Draft.TensorPrefix != "draft." || cfg.Draft.Config != "draft/config.json" {
t.Fatalf("Draft = %#v, want draft prefix/config", cfg.Draft)
}
if cfg.Draft.Architecture != "DFlashDraftModel" {
t.Fatalf("Draft architecture = %q, want DFlashDraftModel", cfg.Draft.Architecture)
}
}
func TestSupportsThinking(t *testing.T) {
+41
View File
@@ -94,6 +94,47 @@ FetchContent_Declare(
)
FetchContent_MakeAvailable(mlx-c)
# To avoid a "long tail" when building MLX with a large set of GPU
# architectures, utilize a higher --threads (-t) setting. At high -t
# every .cu spawns concurrent cicc instances; each cicc can consume several GB
# compiling MLX's CUTLASS-using kernels. This in turn can cause OOMs.
#
# We use a pool to cover all MLX CUDA sources. Pool size is derived from total
# host RAM via a per-file memory budget.
#
# This was calibrated with `-t 6`. Higher -t may require overriding
# MLX_CUDA_RAM_MB
if(CMAKE_GENERATOR STREQUAL "Ninja")
file(GLOB_RECURSE _mlx_cu
"${mlx_SOURCE_DIR}/mlx/backend/cuda/*.cu"
"${mlx_BINARY_DIR}/mlx/backend/cuda/*.cu"
)
if(_mlx_cu)
set(MLX_CUDA_RAM_MB 22000 CACHE STRING
"Per-file memory budget (MB) for the cuda_compile JOB_POOL. Override for higher -t.")
cmake_host_system_information(RESULT _ram_mb QUERY TOTAL_PHYSICAL_MEMORY)
math(EXPR _cuda_pool "${_ram_mb} / ${MLX_CUDA_RAM_MB}")
if(_cuda_pool LESS 2)
set(_cuda_pool 2)
endif()
set_property(GLOBAL APPEND PROPERTY JOB_POOLS cuda_compile=${_cuda_pool})
list(LENGTH _mlx_cu _cu_count)
# SOURCE properties default to directory-scoped, which means a plain
# set_property(SOURCE ...) here would NOT affect the build rules
# generated for the mlx target (defined in mlx_SOURCE_DIR after
# FetchContent). TARGET_DIRECTORY mlx puts the property in the
# directory where mlx was defined, so it actually applies.
foreach(f ${_mlx_cu})
set_property(SOURCE "${f}"
TARGET_DIRECTORY mlx
PROPERTY JOB_POOL_COMPILE cuda_compile)
endforeach()
message(STATUS "MLX cuda_compile JOB_POOL: ${_cu_count} files, pool size ${_cuda_pool} (host RAM ${_ram_mb} MB / ${MLX_CUDA_RAM_MB} MB per file)")
else()
message(WARNING "MLX cuda_compile JOB_POOL: no .cu files found under mlx/backend/cuda/ - check MLX layout")
endif()
endif()
# Sync vendored headers with fetched version
file(GLOB _mlx_c_hdrs "${mlx-c_SOURCE_DIR}/mlx/c/*.h")
file(COPY ${_mlx_c_hdrs} DESTINATION "${CMAKE_SOURCE_DIR}/x/mlxrunner/mlx/include/mlx/c/")
+25 -21
View File
@@ -1,6 +1,8 @@
package cache
import (
"fmt"
"github.com/ollama/ollama/x/mlxrunner/batch"
"github.com/ollama/ollama/x/mlxrunner/mlx"
"github.com/ollama/ollama/x/models/nn"
@@ -8,6 +10,11 @@ import (
// RecurrentCache stores state for linear-recurrent layers.
//
// Conv state takes its dtype from the first Get call (the activation dtype).
// Delta state is always float32: the gated-delta recurrent accumulator runs
// for the full sequence length and needs the extra precision regardless of
// activation dtype.
//
// Conv state shape: [B, convTail, convDim]
// Delta state shape: [B, numVHeads, headVDim, headKDim]
type RecurrentCache struct {
@@ -48,33 +55,30 @@ func NewRecurrentCache(convTail, convDim, numVHeads, headVDim, headKDim int32) *
}
}
func (c *RecurrentCache) ensure(batch int, dtype mlx.DType) {
// Get returns the current conv/delta state for the SSM layer's read
// phase. On first call it lazy-initializes zero-filled state tensors
// sized from b.InputIDs and dtyped from the caller's activation dtype.
// On subsequent calls it returns the existing state; batch size and
// dtype must match the first call, since recurrent state is cumulative
// and cannot be reshaped without losing history.
func (c *RecurrentCache) Get(b *batch.Batch, dtype mlx.DType) *nn.RecurrentHistory {
batch := b.InputIDs.Dim(0)
if batch <= 0 {
batch = 1
}
needConv := c.convState == nil || !c.convState.Valid() || c.convState.DType() != dtype ||
c.convState.Dim(0) != batch || c.convState.Dim(1) != c.convTail || c.convState.Dim(2) != c.convDim
needDelta := c.deltaState == nil || !c.deltaState.Valid() || c.deltaState.DType() != dtype ||
c.deltaState.Dim(0) != batch || c.deltaState.Dim(1) != c.numVHeads || c.deltaState.Dim(2) != c.headVDim || c.deltaState.Dim(3) != c.headKDim
if !needConv && !needDelta {
return
if c.convState != nil {
if got := c.convState.Dim(0); got != batch {
panic(fmt.Sprintf("recurrent cache: batch size changed mid-sequence (have %d, got %d)", got, batch))
}
if got := c.convState.DType(); got != dtype {
panic(fmt.Sprintf("recurrent cache: conv dtype changed mid-sequence (have %v, got %v)", got, dtype))
}
return nn.NewRecurrentHistory(c.convState, c.deltaState)
}
if needConv {
c.convState = c.setState(c.convState, mlx.Zeros(dtype, batch, c.convTail, c.convDim), false)
}
if needDelta {
c.deltaState = c.setState(c.deltaState, mlx.Zeros(dtype, batch, c.numVHeads, c.headVDim, c.headKDim), false)
}
}
// Get returns the current conv/delta state for the SSM layer's read
// phase. Lazy-initializes zero-filled state tensors using b.InputIDs
// for the batch size; reallocates if the existing state's batch size
// or dtype no longer matches.
func (c *RecurrentCache) Get(b *batch.Batch, dtype mlx.DType) *nn.RecurrentHistory {
c.ensure(b.InputIDs.Dim(0), dtype)
c.convState = c.setState(nil, mlx.Zeros(dtype, batch, c.convTail, c.convDim), false)
c.deltaState = c.setState(nil, mlx.Zeros(mlx.DTypeFloat32, batch, c.numVHeads, c.headVDim, c.headKDim), false)
return nn.NewRecurrentHistory(c.convState, c.deltaState)
}
+2 -2
View File
@@ -63,8 +63,8 @@ func TestRecurrentCacheGetLazyInit(t *testing.T) {
if got := h.ConvState().DType(); got != mlx.DTypeBFloat16 {
t.Fatalf("conv state dtype = %v, want %v", got, mlx.DTypeBFloat16)
}
if got := h.DeltaState().DType(); got != mlx.DTypeBFloat16 {
t.Fatalf("delta state dtype = %v, want %v", got, mlx.DTypeBFloat16)
if got := h.DeltaState().DType(); got != mlx.DTypeFloat32 {
t.Fatalf("delta state dtype = %v, want %v", got, mlx.DTypeFloat32)
}
}
+26 -12
View File
@@ -83,7 +83,7 @@ for (int t = 0; t < T; ++t) {
for (int i = 0; i < n_per_t; ++i) {
auto s_idx = n_per_t * dk_idx + i;
o_state[s_idx] = static_cast<InT>(state[i]);
o_state[s_idx] = static_cast<StT>(state[i]);
}
`
@@ -163,7 +163,7 @@ for (int t = 0; t < T_val; ++t) {
for (int i = 0; i < n_per_t; ++i) {
auto s_idx = n_per_t * dk_idx + i;
o_state[s_idx] = static_cast<InT>(state[i]);
o_state[s_idx] = static_cast<StT>(state[i]);
}
`
@@ -262,8 +262,9 @@ func gatedDeltaKernel(q, k, v, g, beta, state *Array) (y, nextState *Array, ok b
return nil, nil, false
}
dtype := q.DType()
if k.DType() != dtype || v.DType() != dtype || g.DType() != dtype || beta.DType() != dtype || state.DType() != dtype {
inputDType := q.DType()
stateDType := state.DType()
if k.DType() != inputDType || v.DType() != inputDType || g.DType() != inputDType || beta.DType() != inputDType {
return nil, nil, false
}
@@ -277,7 +278,13 @@ func gatedDeltaKernel(q, k, v, g, beta, state *Array) (y, nextState *Array, ok b
cInT := C.CString("InT")
defer C.free(unsafe.Pointer(cInT))
if C.mlx_fast_metal_kernel_config_add_template_arg_dtype(cfg, cInT, C.mlx_dtype(dtype)) != 0 {
if C.mlx_fast_metal_kernel_config_add_template_arg_dtype(cfg, cInT, C.mlx_dtype(inputDType)) != 0 {
gatedDeltaMetalDisabled = true
return nil, nil, false
}
cStT := C.CString("StT")
defer C.free(unsafe.Pointer(cStT))
if C.mlx_fast_metal_kernel_config_add_template_arg_dtype(cfg, cStT, C.mlx_dtype(stateDType)) != 0 {
gatedDeltaMetalDisabled = true
return nil, nil, false
}
@@ -301,11 +308,11 @@ func gatedDeltaKernel(q, k, v, g, beta, state *Array) (y, nextState *Array, ok b
yShape := []C.int{C.int(B), C.int(T), C.int(Hv), C.int(Dv)}
stateShape := []C.int{C.int(B), C.int(Hv), C.int(Dv), C.int(Dk)}
if C.mlx_fast_metal_kernel_config_add_output_arg(cfg, unsafe.SliceData(yShape), C.size_t(len(yShape)), C.mlx_dtype(dtype)) != 0 {
if C.mlx_fast_metal_kernel_config_add_output_arg(cfg, unsafe.SliceData(yShape), C.size_t(len(yShape)), C.mlx_dtype(inputDType)) != 0 {
gatedDeltaMetalDisabled = true
return nil, nil, false
}
if C.mlx_fast_metal_kernel_config_add_output_arg(cfg, unsafe.SliceData(stateShape), C.size_t(len(stateShape)), C.mlx_dtype(dtype)) != 0 {
if C.mlx_fast_metal_kernel_config_add_output_arg(cfg, unsafe.SliceData(stateShape), C.size_t(len(stateShape)), C.mlx_dtype(stateDType)) != 0 {
gatedDeltaMetalDisabled = true
return nil, nil, false
}
@@ -516,8 +523,9 @@ func gatedDeltaCUDAKernelApply(q, k, v, g, beta, state *Array) (y, nextState *Ar
return nil, nil, false
}
dtype := q.DType()
if k.DType() != dtype || v.DType() != dtype || g.DType() != dtype || beta.DType() != dtype || state.DType() != dtype {
inputDType := q.DType()
stateDType := state.DType()
if k.DType() != inputDType || v.DType() != inputDType || g.DType() != inputDType || beta.DType() != inputDType {
return nil, nil, false
}
@@ -531,7 +539,13 @@ func gatedDeltaCUDAKernelApply(q, k, v, g, beta, state *Array) (y, nextState *Ar
cInT := C.CString("InT")
defer C.free(unsafe.Pointer(cInT))
if C.mlx_fast_cuda_kernel_config_add_template_arg_dtype(cfg, cInT, C.mlx_dtype(dtype)) != 0 {
if C.mlx_fast_cuda_kernel_config_add_template_arg_dtype(cfg, cInT, C.mlx_dtype(inputDType)) != 0 {
gatedDeltaCUDADisabled = true
return nil, nil, false
}
cStT := C.CString("StT")
defer C.free(unsafe.Pointer(cStT))
if C.mlx_fast_cuda_kernel_config_add_template_arg_dtype(cfg, cStT, C.mlx_dtype(stateDType)) != 0 {
gatedDeltaCUDADisabled = true
return nil, nil, false
}
@@ -555,11 +569,11 @@ func gatedDeltaCUDAKernelApply(q, k, v, g, beta, state *Array) (y, nextState *Ar
yShape := []C.int{C.int(B), C.int(T), C.int(Hv), C.int(Dv)}
stateShape := []C.int{C.int(B), C.int(Hv), C.int(Dv), C.int(Dk)}
if C.mlx_fast_cuda_kernel_config_add_output_arg(cfg, unsafe.SliceData(yShape), C.size_t(len(yShape)), C.mlx_dtype(dtype)) != 0 {
if C.mlx_fast_cuda_kernel_config_add_output_arg(cfg, unsafe.SliceData(yShape), C.size_t(len(yShape)), C.mlx_dtype(inputDType)) != 0 {
gatedDeltaCUDADisabled = true
return nil, nil, false
}
if C.mlx_fast_cuda_kernel_config_add_output_arg(cfg, unsafe.SliceData(stateShape), C.size_t(len(stateShape)), C.mlx_dtype(dtype)) != 0 {
if C.mlx_fast_cuda_kernel_config_add_output_arg(cfg, unsafe.SliceData(stateShape), C.size_t(len(stateShape)), C.mlx_dtype(stateDType)) != 0 {
gatedDeltaCUDADisabled = true
return nil, nil, false
}
+20 -2
View File
@@ -5,21 +5,39 @@ import "C"
import "unsafe"
func RandomKey(seed uint64) *Array {
out := New("RANDOM_KEY")
C.mlx_random_key(&out.ctx, C.uint64_t(seed))
return out
}
func (t *Array) Categorical(axis int) *Array {
key := New("")
return t.CategoricalWithKey(axis, nil)
}
func (t *Array) CategoricalWithKey(axis int, key *Array) *Array {
if key == nil {
key = New("")
}
out := New("")
C.mlx_random_categorical(&out.ctx, t.ctx, C.int(axis), key.ctx, DefaultStream().ctx)
return out
}
func Bernoulli(p *Array) *Array {
return BernoulliWithKey(p, nil)
}
func BernoulliWithKey(p *Array, key *Array) *Array {
dims := p.Dims()
shape := make([]C.int, len(dims))
for i, d := range dims {
shape[i] = C.int(d)
}
key := New("")
if key == nil {
key = New("")
}
out := New("BERNOULLI")
C.mlx_random_bernoulli(&out.ctx, p.ctx, unsafe.SliceData(shape), C.size_t(len(shape)), key.ctx, DefaultStream().ctx)
return out
+32 -39
View File
@@ -4,7 +4,6 @@ import (
"context"
"fmt"
"log/slog"
"math"
"os"
"strconv"
"strings"
@@ -375,9 +374,11 @@ func (r *Runner) runSampleMTPDecode(ctx context.Context, request Request, sessio
t0 = time.Now()
candidates := r.generateMTPDraftCandidates(draft, targetEmbeddings, current.Token, hidden, caches, int32(*position-1), maxDraft)
draftCount := 0
var candidateArrays []*mlx.Array
if candidates != nil {
draftCount = candidates.tokens.Dim(1)
mlx.Pin(baseLogits, candidates.tokens, candidates.logits)
candidateArrays = append([]*mlx.Array{baseLogits}, candidates.Arrays()...)
mlx.Pin(candidateArrays...)
mlx.Sweep()
}
stats.draftDuration += time.Since(t0)
@@ -391,7 +392,7 @@ func (r *Runner) runSampleMTPDecode(ctx context.Context, request Request, sessio
t0 = time.Now()
next, accepted, done, err = r.acceptSampleMTPDrafts(ctx, request, session, &dec, caches, position, baseLogits, candidates, &final, &generated, &stats)
stats.validateDuration += time.Since(t0)
mlx.Unpin(baseLogits, candidates.tokens, candidates.logits)
mlx.Unpin(candidateArrays...)
if err != nil {
return err
}
@@ -456,8 +457,15 @@ func (r *Runner) runSampleMTPDecode(ctx context.Context, request Request, sessio
type mtpDraftCandidates struct {
tokens *mlx.Array
// logits are the processed proposal scores used to sample tokens.
logits *mlx.Array
// dist is the proposal distribution used to sample each drafted token.
dist sampler.Distribution
}
func (c *mtpDraftCandidates) Arrays() []*mlx.Array {
if c == nil {
return nil
}
return append([]*mlx.Array{c.tokens}, c.dist.Arrays()...)
}
func (r *Runner) generateMTPDrafts(draft base.MTPDraftModel, target base.MTPEmbeddingModel, token *mlx.Array, hidden *mlx.Array, caches []cache.Cache, position int32, maxDraft int) *mlx.Array {
@@ -497,7 +505,7 @@ func (r *Runner) generateMTPDraftCandidates(draft base.MTPDraftModel, target bas
lastToken := mtpTokenInput(token)
lastHidden := hidden
draftTokens := make([]*mlx.Array, 0, maxDraft)
draftLogits := make([]*mlx.Array, 0, maxDraft)
draftDists := make([]sampler.Distribution, 0, maxDraft)
var prefix *mlx.Array
// Gemma4 assistant MTP is trained as "single-position" drafting:
@@ -508,13 +516,13 @@ func (r *Runner) generateMTPDraftCandidates(draft base.MTPDraftModel, target bas
inputs := tokenEmbedding.Concatenate(-1, lastHidden)
logits, projected := draft.Draft(inputs, position, caches)
stepLogits := r.lastLogitsFromLogits(logits)
stepScores := r.Sampler.SpeculativeScores(pipelineSlot, stepLogits, prefix)
nextToken := stepScores.Categorical(-1).AsType(mlx.DTypeInt32)
dist := r.Sampler.Distribution(pipelineSlot, stepLogits, prefix)
nextToken := r.Sampler.SampleDistribution(pipelineSlot, dist)
lastToken = mtpTokenInput(nextToken)
lastHidden = projected
draftTokens = append(draftTokens, lastToken)
draftLogits = append(draftLogits, stepScores.ExpandDims(1))
draftDists = append(draftDists, dist)
if prefix == nil {
prefix = lastToken
} else {
@@ -526,7 +534,7 @@ func (r *Runner) generateMTPDraftCandidates(draft base.MTPDraftModel, target bas
}
return &mtpDraftCandidates{
tokens: mlx.Concatenate(draftTokens, 1),
logits: mlx.Concatenate(draftLogits, 1),
dist: sampler.ConcatenateDistributions(draftDists),
}
}
@@ -630,12 +638,9 @@ func (r *Runner) acceptSampleMTPDrafts(ctx context.Context, request Request, ses
SeqQueryLens: []int32{int32(draftCount)},
}, specCaches)
targetScores := r.Sampler.SpeculativeScores(pipelineSlot, r.mtpValidationLogits(baseLogits, hiddenSeq), candidates.tokens)
draftScores := candidates.logits
if draftScores.NumDims() == 3 {
draftScores = draftScores.Squeeze(0)
}
acceptedMask := mtpSampleAcceptedMask(targetScores, draftScores, candidates.tokens, draftCount)
targetDist := r.Sampler.Distribution(pipelineSlot, r.mtpValidationLogits(baseLogits, hiddenSeq), candidates.tokens)
draftDist := candidates.dist
acceptedMask := r.mtpSampleAcceptedMask(targetDist.SliceRows(0, draftCount), draftDist, candidates.tokens)
mlx.Eval(candidates.tokens, acceptedMask)
draftIDs := candidates.tokens.Ints()
@@ -692,9 +697,9 @@ func (r *Runner) acceptSampleMTPDrafts(ctx context.Context, request Request, ses
var nextToken *mlx.Array
if accepted == draftCount {
nextToken = mtpSampleTokenAt(targetScores, draftCount)
nextToken = r.mtpSampleTokenAt(targetDist, draftCount)
} else {
nextToken = mtpSampleResidualToken(targetScores, draftScores, accepted)
nextToken = r.mtpSampleResidualToken(targetDist, draftDist, accepted)
}
mlx.Eval(nextToken)
nextID := int32(tokenID(nextToken))
@@ -704,32 +709,20 @@ func (r *Runner) acceptSampleMTPDrafts(ctx context.Context, request Request, ses
return sampler.Result{Token: nextToken}, accepted, false, nil
}
func mtpSampleAcceptedMask(targetScores, draftScores, draftTokens *mlx.Array, draftCount int) *mlx.Array {
targetProbs := mlx.SoftmaxAxis(targetScores.Slice(mlx.Slice(0, draftCount), mlx.Slice()), -1, true)
draftProbs := mlx.SoftmaxAxis(draftScores, -1, true)
if draftTokens.NumDims() == 2 {
draftTokens = draftTokens.Squeeze(0)
}
indices := draftTokens.ExpandDims(-1)
p := targetProbs.TakeAlongAxis(indices, -1).Squeeze(-1)
q := draftProbs.TakeAlongAxis(indices, -1).Squeeze(-1)
func (r *Runner) mtpSampleAcceptedMask(targetDist, draftDist sampler.Distribution, draftTokens *mlx.Array) *mlx.Array {
p := targetDist.Prob(draftTokens)
q := draftDist.Prob(draftTokens)
acceptP := mlx.Minimum(p.Divide(q), mlx.FromValue(float32(1)))
return mlx.Bernoulli(acceptP).AsType(mlx.DTypeInt32)
return r.Sampler.Bernoulli(pipelineSlot, acceptP).AsType(mlx.DTypeInt32)
}
func mtpSampleTokenAt(scores *mlx.Array, index int) *mlx.Array {
row := scores.Slice(mlx.Slice(index, index+1), mlx.Slice())
return mtpTokenVector(row.Categorical(-1).AsType(mlx.DTypeInt32))
func (r *Runner) mtpSampleTokenAt(dist sampler.Distribution, index int) *mlx.Array {
return mtpTokenVector(r.Sampler.SampleDistribution(pipelineSlot, dist.SliceRows(index, index+1)))
}
func mtpSampleResidualToken(targetScores, draftScores *mlx.Array, index int) *mlx.Array {
p := mlx.SoftmaxAxis(targetScores.Slice(mlx.Slice(index, index+1), mlx.Slice()), -1, true)
q := mlx.SoftmaxAxis(draftScores.Slice(mlx.Slice(index, index+1), mlx.Slice()), -1, true)
diff := p.Subtract(q)
positive := mlx.Maximum(diff, mlx.FromValue(float32(1e-20)))
logits := mlx.Log(positive)
logits = mlx.Where(diff.LessEqual(mlx.FromValue(float32(0))), mlx.FromValue(float32(math.Inf(-1))), logits)
return mtpTokenVector(logits.Categorical(-1).AsType(mlx.DTypeInt32))
func (r *Runner) mtpSampleResidualToken(targetDist, draftDist sampler.Distribution, index int) *mlx.Array {
residual := targetDist.SliceRows(index, index+1).ResidualAgainst(draftDist.SliceRows(index, index+1))
return mtpTokenVector(r.Sampler.SampleDistribution(pipelineSlot, residual))
}
func mtpTokenInput(token *mlx.Array) *mlx.Array {
Loaded 100 of 111 files, more files were not shown because too many files have changed in this diff. Show more