diff --git a/.agents/ai-coding-assistants.md b/.agents/ai-coding-assistants.md index d0d9c882c..b2ce5e27d 100644 --- a/.agents/ai-coding-assistants.md +++ b/.agents/ai-coding-assistants.md @@ -49,6 +49,40 @@ AI agents MUST NOT add `Co-Authored-By` trailers for themselves either. A human reviewer owns the contribution; the AI's involvement is recorded via `Assisted-by` (see below). +### Exception: automation operated by a maintainer + +The rule above addresses the common case, an AI assistant helping a human +contributor who then signs off. It does not fit automation that a +maintainer runs themselves, which opens pull requests with no human +submitter to sign. Applied literally there, nothing ever signs and the +DCO check blocks the pull request permanently. + +A maintainer-operated bot MUST therefore add a `Signed-off-by` trailer +naming **the maintainer who operates it**, not the bot and not the model: + +``` +Assisted-by: Codex:gpt-5 +Signed-off-by: Ettore Di Giacinto +``` + +This is not the AI certifying the DCO. The maintainer is, exactly as they +do for a commit they typed by hand: they configured the automation, they +own its output, and they take responsibility for it when they merge it. +The `Assisted-by` trailer still records that a model produced the code, so +the provenance trail is unchanged. + +The exception is narrow and does not widen the rule for anyone else: + +- It applies only to automation a LocalAI maintainer operates and whose + output that maintainer reviews before merge. +- The sign-off names a real person who accepts DCO responsibility. +- An AI assistant helping an outside contributor still MUST NOT sign off. + That contributor adds their own trailer. +- A bot MUST NOT sign off on behalf of anyone other than its operator, and + MUST NOT add a trailer for a contributor whose branch it pushes to. If + automation contributes to someone else's branch, it leaves the sign-off + to that contributor. + ## Attribution When AI tools contribute to LocalAI development, proper attribution helps diff --git a/.agents/api-endpoints-and-auth.md b/.agents/api-endpoints-and-auth.md index fffee21c9..e3a63b1e5 100644 --- a/.agents/api-endpoints-and-auth.md +++ b/.agents/api-endpoints-and-auth.md @@ -236,6 +236,58 @@ Use these HTTP status codes: If your endpoint should be tracked for usage (token counts, request counts), add the `usageMiddleware` to its middleware chain. See `core/http/middleware/usage.go` and how it's applied in `routes/openai.go`. +## Control-plane database health metrics + +In distributed mode the frontend registers three OpenTelemetry gauges over the +PostgreSQL control-plane database (`core/services/monitoring/control_plane_db.go`, +wired in `core/application/distributed.go`). They reach `/metrics` through the +same Prometheus exporter as the rest of the API metrics. + +| Metric | Meaning | Page when | +|--------|---------|-----------| +| `localai_control_plane_oldest_xmin_age` | Transactions elapsed since the oldest snapshot any backend still holds | above a few million, and rising | +| `localai_control_plane_longest_transaction_seconds` | Age of the longest open transaction | above 3600 | +| `localai_control_plane_dead_tuple_ratio` | Dead tuples per live tuple, labelled by `table`, on `backend_nodes`, `node_models` and `gallery_operations` | sustained above ~10 on a small table | + +A sustained high `localai_control_plane_oldest_xmin_age` is the one to page on. +While it grows, autovacuum can reclaim nothing anywhere in the database no +matter how often it runs, so the dead tuple ratio keeps climbing and a six-row +registry table can reach hundreds of megabytes. Tuning autovacuum does not help. +The fix is to find the transaction holding the horizon open and clear it: + +```sql +SELECT pid, state, age(backend_xmin) AS xmin_age, now() - xact_start AS xact_age, query +FROM pg_stat_activity +WHERE backend_xmin IS NOT NULL +ORDER BY age(backend_xmin) DESC; +``` + +Then `pg_terminate_backend(pid)` on the offenders, and `VACUUM (VERBOSE)` the +bloated tables once the horizon has moved. + +**A healthy-looking xmin age does not on its own prove the horizon is free.** +The gauge reads `pg_stat_activity`, which only sees live backends. Two other +things pin the very same horizon and are invisible there, so either one can hold +vacuum back while the gauge reads 0: + +```sql +SELECT gid, prepared, database, transaction FROM pg_prepared_xacts; +SELECT slot_name, active, xmin, catalog_xmin FROM pg_replication_slots; +``` + +An orphaned prepared transaction is cleared with `ROLLBACK PREPARED ''`, +and a stale slot with `pg_drop_replication_slot('')`. Check both +before concluding that a bloated table has some other cause. + +Sampling is scrape-driven behind a 30 second cache, so scrape frequency does not +translate into database load. Failed and timed-out samples cost the same interval +as successful ones, so a database that is already struggling is not retried on +every scrape. A failed sample reports the last good values rather than failing the +scrape, because these gauges matter most when the database is struggling. Before +the first successful sample the gauges are absent rather than zero, since a zero +xmin age would read as a healthy horizon: alert on `absent()` too if you need to +distinguish "healthy" from "never sampled". + ## Advertising surfaces — where to register a new capability Beyond routing and auth, LocalAI publishes its capability surface in **four independent places**. When you add an endpoint — especially one introducing a net-new capability like a new media type or a new auth-gated feature — you must update every relevant surface. These aren't optional: missing them means the endpoint works but is invisible to clients, admins, and the UI. diff --git a/.agents/ds4-backend.md b/.agents/ds4-backend.md index c1b649857..8bf1eb45f 100644 --- a/.agents/ds4-backend.md +++ b/.agents/ds4-backend.md @@ -77,6 +77,56 @@ spectrum. **Metal (Darwin) only** - it is a no-op on CUDA/CPU. Enable with budget). Gallery entries built on this: `deepseek-v4-flash-q4-ssd` (153 GB Flash on a 128 GB Mac) and `deepseek-v4-pro-q2-ssd` (433 GB Pro, experimental). +## CUDA architecture (do not build without one) + +`backend/cpp/ds4/Makefile` drives upstream's **object targets** directly +(`$(MAKE) -C ds4 ds4.o ds4_cuda.o ...`), which bypasses upstream's own guard: +its `cuda` target refuses to build unless `CUDA_ARCH` is set, and offers +`cuda-spark` (sm_121, DGX Spark / GB10) and `cuda-generic` (native) instead. +Built with no `-arch`, nvcc targets its default architecture and the kernels run +as JIT'd PTX. On GB10 that silently corrupted every prefill batch of >=128 +tokens - the model emitted text unrelated to the prompt and never closed its +thinking block, so `content` came back empty - and cost close to two orders of +magnitude of prefill throughput (4.21 t/s vs 325.70 t/s, same box, same model). +Short prompts stayed correct, which is why it went unnoticed. + +The Makefile therefore picks a gencode list from `CUDA_MAJOR_VERSION` (a build +arg the backend matrix already declares, forwarded by `Dockerfile.ds4`) and +`uname -m`, and passes it as `NVCC_ARCH_FLAGS` to the sub-make. Upstream's +`CUDA_ARCH` accepts a single value, so it cannot express the fat binary the +shipped images need; a command-line assignment beats its `:=`. An empty +`CUDA_MAJOR_VERSION` falls back to upstream's `native` for local developer +builds, and an unrecognised one is a hard error - no CI runner has a GPU, so a +silent `native` there is exactly the failure mode this guards against. + +`DS4_CUDA_HAVE_MXF4` is deliberately unset: upstream defines it only for +single-arch sm_120/sm_121 builds and guards it with a plain `#ifdef` rather than +`__CUDA_ARCH__`, so it cannot be combined with older archs. It gates an optional +MXFP4 indexer fast path whose `#ifndef` branch returns 0, so omitting it costs +speed, not correctness. + +### Verifying a build + +Check which flags a configuration resolves to, without compiling anything: + +``` +make -C backend/cpp/ds4 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=13 NATIVE=false \ + --eval='show: ; @echo [$(DS4_ARCH_MAKEVARS)]' show +``` + +Do not use `make -n` for this: the recipe is `+$(MAKE) ...`, and the `+` prefix +makes it run even under `-n`. + +Then exercise the failure mode itself against a built backend. It only appears +above one prefill batch, so the ordinary `predict` spec cannot catch it: + +``` +BACKEND_BINARY=$(pwd)/backend/cpp/ds4/package/run.sh \ +BACKEND_TEST_MODEL_FILE=/path/to/ds4flash.gguf \ +BACKEND_TEST_CAPS=health,load,predict,long_prefill \ +go test -count=1 -timeout=30m -v ./tests/e2e-backends/... +``` + ## Build matrix | Build | Where | Notes | diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 6d0e30e7a..a7ae060fb 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -5,7 +5,7 @@ This PR fixes # **Notes for Reviewers** -**[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)** +**[Signed commits](../CONTRIBUTING.md#commit-messages)** - [ ] Yes, I signed my commits. - [ ] Documentation updated (docs/content/) for user-facing changes, or not applicable diff --git a/.github/backend-matrix.yml b/.github/backend-matrix.yml index e7bada4b4..3d3b2b804 100644 --- a/.github/backend-matrix.yml +++ b/.github/backend-matrix.yml @@ -3754,6 +3754,19 @@ include: dockerfile: "./backend/Dockerfile.golang" context: "./" ubuntu-version: '2404' + - build-type: 'hipblas' + cuda-major-version: "" + cuda-minor-version: "" + platforms: 'linux/amd64' + tag-latest: 'auto' + tag-suffix: '-gpu-rocm-hipblas-stablediffusion-ggml' + runs-on: 'ubuntu-latest' + base-image: "rocm/dev-ubuntu-24.04:7.2.1" + skip-drivers: 'false' + backend: "stablediffusion-ggml" + dockerfile: "./backend/Dockerfile.golang" + context: "./" + ubuntu-version: '2404' - build-type: 'sycl_f16' cuda-major-version: "" cuda-minor-version: "" diff --git a/.github/bump_vllm_metal.sh b/.github/bump_vllm_metal.sh index d2aedf4bc..459f97fd2 100755 --- a/.github/bump_vllm_metal.sh +++ b/.github/bump_vllm_metal.sh @@ -3,9 +3,9 @@ # darwin (Apple Silicon) install path. The macOS/Metal build # (backend/python/vllm/install.sh, Darwin branch) installs vllm-metal, which is # version-locked to a specific vLLM source release. install.sh derives that vLLM -# version at build time from vllm-metal's own installer at the pinned -# tag, so there is only ONE value to bump here -- mirroring bump_vllm_wheel.sh, -# which bumps the Linux cu130 wheel pin. +# version, and the wheel asset name, at build time from the pinned tag, so there +# is only ONE value to bump here -- mirroring bump_vllm_wheel.sh, which bumps the +# Linux cu130 wheel pin. # # This deliberately tracks vllm-project/vllm-metal, NOT vllm-project/vllm: the # darwin build can only use the exact vLLM version vllm-metal supports, so it may @@ -23,15 +23,20 @@ if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then exit 1 fi -# vllm-metal ships frequent dev releases, all flagged as non-prerelease, so -# /releases/latest returns the newest one (with its cp312 wheel asset). +# vllm-metal ships frequent .dev releases, flagged as prereleases, alongside the +# stable ones. /releases/latest skips the prereleases and returns the newest +# stable tag, which is what darwin should pin: upstream deletes and re-cuts .dev +# tags, and a pin to a deleted tag 404s the whole build. LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \ "https://api.github.com/repos/$REPO/releases/latest" \ | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])") -# The coupled vLLM source version lives in vllm-metal's installer at that tag. -NEW_VLLM_VERSION=$(gh_curl \ - "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh" \ +# The coupled vLLM release lives in .github/vllm-release-tag.commit at that tag +# (since vllm-metal 0.28); releases predating that file pinned it inline in their +# own install.sh. The extractor reads both forms. +NEW_VLLM_VERSION=$( { gh_curl \ + "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/.github/vllm-release-tag.commit" \ + || gh_curl "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh"; } \ | "$(dirname "${BASH_SOURCE[0]}")/../scripts/lib/extract-vllm-metal-version.sh") if [ -z "$LATEST_TAG" ] || [ -z "$NEW_VLLM_VERSION" ]; then diff --git a/.github/workflows/bump_deps.yaml b/.github/workflows/bump_deps.yaml index 069b094a0..d86725933 100644 --- a/.github/workflows/bump_deps.yaml +++ b/.github/workflows/bump_deps.yaml @@ -166,7 +166,7 @@ jobs: push-to-fork: ci-forks/LocalAI commit-message: ':arrow_up: Update ${{ matrix.repository }}' title: 'chore: :arrow_up: Update ${{ matrix.repository }} to `${{ steps.bump.outputs.commit }}`' - branch: "update/${{ matrix.variable }}" + branch: "bump/${{ matrix.variable }}" body: ${{ steps.bump.outputs.message }} signoff: true @@ -203,7 +203,7 @@ jobs: push-to-fork: ci-forks/LocalAI commit-message: ':arrow_up: Update vllm-project/vllm cu130 wheel' title: 'chore: :arrow_up: Update vllm-project/vllm cu130 wheel to `${{ steps.bump.outputs.commit }}`' - branch: "update/VLLM_VERSION" + branch: "bump/VLLM_VERSION" body: ${{ steps.bump.outputs.message }} signoff: true @@ -241,6 +241,6 @@ jobs: push-to-fork: ci-forks/LocalAI commit-message: ':arrow_up: Update vllm-project/vllm-metal (darwin)' title: 'chore: :arrow_up: Update vllm-metal (darwin) to `${{ steps.bump.outputs.commit }}`' - branch: "update/VLLM_METAL_VERSION" + branch: "bump/VLLM_METAL_VERSION" body: ${{ steps.bump.outputs.message }} signoff: true diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 7929f6b65..74d57567f 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -80,8 +80,13 @@ jobs: coverage/coverage.out coverage/coverage.html if-no-files-found: ignore + # tmate keeps the runner busy until the 6 hour job limit, so a single + # failure costs a whole runner slot. Only open a session when someone + # asked for one by labelling the pull request `ci-debug`, and cap the + # session so a forgotten label cannot idle a runner either. - name: Setup tmate session if tests fail - if: ${{ failure() }} + if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }} + timeout-minutes: 30 uses: mxschmitt/action-tmate@v3.23 with: detached: true @@ -125,8 +130,13 @@ jobs: export PATH="/opt/homebrew/opt/make/libexec/gnubin:$PATH" PATH="$PATH:$HOME/go/bin" make protogen-go PATH="$PATH:$HOME/go/bin" BUILD_TYPE="GITHUB_CI_HAS_BROKEN_METAL" CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" make --jobs 4 --output-sync=target test + # tmate keeps the runner busy until the 6 hour job limit, so a single + # failure costs a whole runner slot. Only open a session when someone + # asked for one by labelling the pull request `ci-debug`, and cap the + # session so a forgotten label cannot idle a runner either. - name: Setup tmate session if tests fail - if: ${{ failure() }} + if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }} + timeout-minutes: 30 uses: mxschmitt/action-tmate@v3.23 with: detached: true diff --git a/.github/workflows/tests-aio.yml b/.github/workflows/tests-aio.yml index f8d3d34f0..738f96dac 100644 --- a/.github/workflows/tests-aio.yml +++ b/.github/workflows/tests-aio.yml @@ -77,8 +77,13 @@ jobs: - name: Test run: | PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio + # tmate keeps the runner busy until the 6 hour job limit, so a single + # failure costs a whole runner slot. Only open a session when someone + # asked for one by labelling the pull request `ci-debug`, and cap the + # session so a forgotten label cannot idle a runner either. - name: Setup tmate session if tests fail - if: ${{ failure() }} + if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }} + timeout-minutes: 30 uses: mxschmitt/action-tmate@v3.23 with: detached: true diff --git a/.github/workflows/tests-e2e.yml b/.github/workflows/tests-e2e.yml index 3c1cb711c..b5c84d2a8 100644 --- a/.github/workflows/tests-e2e.yml +++ b/.github/workflows/tests-e2e.yml @@ -63,8 +63,13 @@ jobs: - name: Test Backend E2E run: | PATH="$PATH:$HOME/go/bin" make build-mock-backend test-e2e + # tmate keeps the runner busy until the 6 hour job limit, so a single + # failure costs a whole runner slot. Only open a session when someone + # asked for one by labelling the pull request `ci-debug`, and cap the + # session so a forgotten label cannot idle a runner either. - name: Setup tmate session if tests fail - if: ${{ failure() }} + if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }} + timeout-minutes: 30 uses: mxschmitt/action-tmate@v3.23 with: detached: true diff --git a/.github/workflows/tests-pii-ner-e2e.yml b/.github/workflows/tests-pii-ner-e2e.yml index 800f67190..c9ab182f2 100644 --- a/.github/workflows/tests-pii-ner-e2e.yml +++ b/.github/workflows/tests-pii-ner-e2e.yml @@ -88,8 +88,13 @@ jobs: # CPU and runs the token_classify capability spec (byte-offset contract). - name: Run live PII NER backend E2E run: PATH="$PATH:$HOME/go/bin" make test-extra-backend-privacy-filter + # tmate keeps the runner busy until the 6 hour job limit, so a single + # failure costs a whole runner slot. Only open a session when someone + # asked for one by labelling the pull request `ci-debug`, and cap the + # session so a forgotten label cannot idle a runner either. - name: Setup tmate session if tests fail - if: ${{ failure() }} + if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }} + timeout-minutes: 30 uses: mxschmitt/action-tmate@v3.23 with: detached: true diff --git a/.github/workflows/tests-ui-e2e.yml b/.github/workflows/tests-ui-e2e.yml index 1a170fcf3..6d45c8e9f 100644 --- a/.github/workflows/tests-ui-e2e.yml +++ b/.github/workflows/tests-ui-e2e.yml @@ -75,8 +75,13 @@ jobs: path: core/http/react-ui/coverage/ if-no-files-found: ignore retention-days: 7 + # tmate keeps the runner busy until the 6 hour job limit, so a single + # failure costs a whole runner slot. Only open a session when someone + # asked for one by labelling the pull request `ci-debug`, and cap the + # session so a forgotten label cannot idle a runner either. - name: Setup tmate session if tests fail - if: ${{ failure() }} + if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }} + timeout-minutes: 30 uses: mxschmitt/action-tmate@v3.23 with: detached: true diff --git a/AGENTS.md b/AGENTS.md index 71302268a..abbb342df 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,7 @@ Human contributors: see [CONTRIBUTING.md](CONTRIBUTING.md) for the development w LocalAI follows the Linux kernel project's [guidelines for AI coding assistants](https://docs.kernel.org/process/coding-assistants.html). Before submitting AI-assisted code, read [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md). Key rules: -- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin. +- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin. One exception: automation a maintainer operates signs off with *that maintainer's* identity, since no other human submitter exists to certify it. See [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md). - **No `Co-Authored-By: ` trailers.** The human contributor owns the change. - **Use an `Assisted-by:` trailer** to attribute AI involvement. Format: `Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]`. - **The human submitter is responsible** for reviewing, testing, and understanding every line of generated code. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d87db37ea..655695cfc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -218,7 +218,7 @@ LocalAI follows the **same guidelines as the Linux kernel project** for AI-assis The full policy for this repository lives in [`.agents/ai-coding-assistants.md`](.agents/ai-coding-assistants.md). Summary: -- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin. +- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin. Automation operated by a maintainer is the one exception: it signs off with that maintainer's identity, because there is no other human submitter to certify it. - **AI agents MUST NOT add `Co-Authored-By` trailers** attributing themselves as co-authors. - **Attribute AI involvement with an `Assisted-by` trailer** in the commit message: diff --git a/Makefile b/Makefile index c68cd5522..cf3248c3b 100644 --- a/Makefile +++ b/Makefile @@ -34,6 +34,11 @@ TEST_FLAKES?=5 RANDOM := $(shell bash -c 'echo $$RANDOM') VERSION?=$(shell git describe --always --tags || echo "dev" ) +# fyne package only accepts numeric x[.y[.z]] app versions, so reduce git +# describe output (v4.9.0, v4.9.0-14-gabc1234, or a bare sha on untagged +# checkouts) to its numeric core; anything non-numeric falls back to 0.0.0. +# Without this the packaged launcher reports itself as version 0.0.0 (#11673). +LAUNCHER_APP_VERSION?=$(shell v=$$(echo "$(VERSION)" | sed -E 's/^v//; s/[+-].*$$//'); echo "$$v" | grep -qE '^[0-9]+(\.[0-9]+){0,2}$$' && echo "$$v" || echo "0.0.0") # go tool nm ./local-ai | grep Commit LD_FLAGS?=-s -w override LD_FLAGS += -X "github.com/mudler/LocalAI/internal.Version=$(VERSION)" @@ -388,9 +393,17 @@ test-e2e: build-mock-backend build-cloud-proxy-backend prepare-e2e run-e2e-image $(MAKE) teardown-e2e docker rmi localai-tests +# `docker stop` returns as soon as the container exits, but Docker reaps a +# `--rm` container asynchronously after that. The `docker rmi localai-tests` in +# test-e2e then loses the race against the reaper and fails on a still +# referenced image, turning a green suite red. Removing the container ourselves +# is synchronous, so the image reference is gone before we return. It also +# covers the case where nothing is running, which `docker stop` could not +# because it rejects an empty argument list. teardown-e2e: rm -rf $(TEST_DIR) || true - docker stop $$(docker ps -q --filter ancestor=localai-tests) + @CONTAINERS=$$(docker ps -aq --filter ancestor=localai-tests 2>/dev/null); \ + if [ -n "$$CONTAINERS" ]; then docker rm -f $$CONTAINERS || true; fi ######################################################## ## Integration and unit tests @@ -1622,7 +1635,7 @@ site-serve: site build-launcher-darwin: rm -rf dist/LocalAI.app cmd/launcher/LocalAI.app mkdir -p dist - cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME) + cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME) --app-version $(LAUNCHER_APP_VERSION) mv cmd/launcher/LocalAI.app dist/LocalAI.app bash contrib/macos/sign-and-notarize.sh sign dist/LocalAI.app @@ -1649,4 +1662,4 @@ release-launcher-darwin: notarize-launcher-darwin @echo "dist/LocalAI.dmg is ready" build-launcher-linux: - cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz + cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux --app-version $(LAUNCHER_APP_VERSION) && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz diff --git a/backend/Dockerfile.ds4 b/backend/Dockerfile.ds4 index 370d1eaae..5d4f02a8d 100644 --- a/backend/Dockerfile.ds4 +++ b/backend/Dockerfile.ds4 @@ -10,6 +10,7 @@ FROM ${BASE_IMAGE} AS builder ARG BUILD_TYPE ARG TARGETARCH ARG TARGETVARIANT +ARG CUDA_MAJOR_VERSION ENV BUILD_TYPE=${BUILD_TYPE} \ DEBIAN_FRONTEND=noninteractive \ @@ -35,7 +36,8 @@ RUN apt-get update && \ COPY . /LocalAI RUN --mount=type=cache,target=/root/.ccache,id=ds4-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \ - make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} NATIVE=false grpc-server package + make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} \ + CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} NATIVE=false grpc-server package FROM scratch COPY --from=builder /LocalAI/backend/cpp/ds4/package/. ./ diff --git a/backend/cpp/audio-cpp/Makefile b/backend/cpp/audio-cpp/Makefile index bee144fbc..811fc1115 100644 --- a/backend/cpp/audio-cpp/Makefile +++ b/backend/cpp/audio-cpp/Makefile @@ -9,7 +9,7 @@ # recipe is a make target (not a prepare.sh) so 'make purge && make' is a clean # rebuild and so the bump bot can see the pin. -AUDIO_CPP_VERSION?=288a2712316470847a730e55db9ac9e5062a2b03 +AUDIO_CPP_VERSION?=f6277c1695a83cf388a8282c1c1a8757cf626f18 AUDIO_CPP_REPO?=https://github.com/0xShug0/audio.cpp CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST)))) diff --git a/backend/cpp/bonsai/Makefile b/backend/cpp/bonsai/Makefile index 96171e6a9..522791d42 100644 --- a/backend/cpp/bonsai/Makefile +++ b/backend/cpp/bonsai/Makefile @@ -1,7 +1,7 @@ # Pinned to the HEAD of the `prism` branch on https://github.com/PrismML-Eng/llama.cpp. # Auto-bumped nightly by .github/workflows/bump_deps.yaml. -BONSAI_VERSION?=9ca265a57f85f2117942490f421f64a226dd9847 +BONSAI_VERSION?=312bb2a93ea2bf798333fa859614fbf913ecb9e2 LLAMA_REPO?=https://github.com/PrismML-Eng/llama.cpp CMAKE_ARGS?= @@ -41,6 +41,7 @@ define bonsai-build # and are applied by apply-patches.sh below. rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/patches $(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build purge + bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp bash $(LLAMA_CPP_DIR)/disable-tts-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp $(info $(GREEN)I bonsai build info:$(1)$(RESET)) @@ -79,6 +80,7 @@ bonsai-cpu-all: # and are applied by apply-patches.sh below. rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/patches $(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build purge + bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp bash $(LLAMA_CPP_DIR)/disable-tts-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp $(info $(GREEN)I bonsai build info:cpu-all-variants$(RESET)) diff --git a/backend/cpp/bonsai/patch-grpc-server.sh b/backend/cpp/bonsai/patch-grpc-server.sh new file mode 100644 index 000000000..aa9b23a50 --- /dev/null +++ b/backend/cpp/bonsai/patch-grpc-server.sh @@ -0,0 +1,24 @@ +#!/bin/bash +# Adapt the shared llama.cpp gRPC source to the older JSON API in Bonsai. + +set -euo pipefail + +if [[ $# -ne 1 ]]; then + echo "usage: $0 " >&2 + exit 2 +fi + +SRC=$1 +if [[ ! -f "$SRC" ]]; then + echo "grpc-server.cpp not found at $SRC" >&2 + exit 2 +fi + +if grep -q 'common_json_error' "$SRC"; then + echo "==> patching $SRC to use the Bonsai JSON exception type" + awk '{ gsub(/common_json_error/, "json::parse_error"); print }' "$SRC" > "$SRC.tmp" + mv "$SRC.tmp" "$SRC" + echo "==> Bonsai JSON exception patch OK" +else + echo "==> $SRC already uses a Bonsai-compatible JSON exception type, skipping" +fi diff --git a/backend/cpp/ds4/CMakeLists.txt b/backend/cpp/ds4/CMakeLists.txt index 0535a8a44..5783db942 100644 --- a/backend/cpp/ds4/CMakeLists.txt +++ b/backend/cpp/ds4/CMakeLists.txt @@ -84,9 +84,10 @@ elseif(DS4_GPU STREQUAL "cpu") set(DS4_OBJS "${DS4_DIR}/ds4_cpu.o") endif() -# Upstream splits distributed inference, tensor-parallel transport, the SSD -# expert cache, and layer placement into GPU-agnostic translation units. Link -# them regardless of DS4_GPU. +# Upstream splits image preprocessing, distributed inference, tensor-parallel +# transport, the SSD expert cache, and layer placement into GPU-agnostic +# translation units. Link them regardless of DS4_GPU. +list(APPEND DS4_OBJS "${DS4_DIR}/ds4_image.o") list(APPEND DS4_OBJS "${DS4_DIR}/ds4_distributed.o") list(APPEND DS4_OBJS "${DS4_DIR}/ds4_tp.o") list(APPEND DS4_OBJS "${DS4_DIR}/ds4_ssd.o") diff --git a/backend/cpp/ds4/Makefile b/backend/cpp/ds4/Makefile index 5e73c1328..455b554d8 100644 --- a/backend/cpp/ds4/Makefile +++ b/backend/cpp/ds4/Makefile @@ -1,10 +1,10 @@ # ds4 backend Makefile. # -# Upstream pin lives below as DS4_VERSION?=c1d4597a80e300b803dc642519718f2c999589da +# Upstream pin lives below as DS4_VERSION?=9ab705347c1775e7599ede7eb81a6255ec7dccb5 # (.github/bump_deps.sh) can find and update it - matches the # llama-cpp / ik-llama-cpp / turboquant convention. -DS4_VERSION?=c1d4597a80e300b803dc642519718f2c999589da +DS4_VERSION?=9ab705347c1775e7599ede7eb81a6255ec7dccb5 DS4_REPO?=https://github.com/antirez/ds4 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST)))) @@ -18,21 +18,83 @@ UNAME_S := $(shell uname -s) CMAKE_ARGS ?= -DCMAKE_BUILD_TYPE=Release -# Upstream splits distributed inference, tensor-parallel transport, the SSD -# expert cache, and layer placement into GPU-agnostic translation units. They -# are shared by every GPU mode, so append them unconditionally below. +# nvcc must be told the target architecture explicitly for a cublas build, and +# this is not a tuning knob. Upstream's Makefile leaves CUDA_ARCH empty and its +# `cuda` target REFUSES to build without one, offering `cuda-spark` +# (CUDA_ARCH=sm_121) and `cuda-generic` (CUDA_ARCH=native) instead. We drive its +# object targets directly, which bypasses that guard: nvcc then compiles with no +# -arch at all, and the kernels run as JIT'd PTX for its default architecture. +# On GB10 (sm_121) that silently produced corrupt inference output above a +# ~128-token prefill batch and ~77x slower prefill (4.21 t/s vs 325.70 t/s, +# measured on the same box with the same model). No CI runner has a GPU, so +# `native` has nothing to enumerate there. +# +# Upstream's CUDA_ARCH takes a SINGLE value (see its sm_120/sm_121 special cases +# and the `-arch=$(CUDA_ARCH)` fallback), so it cannot express the fat binary +# these images need. NVCC_ARCH_FLAGS is overridden instead: a command-line +# assignment wins over the `:=` in upstream's Makefile, and its NVCCFLAGS +# expands whatever we pass. +# +# The architecture lists are copied from backend/go/vllm-cpp/Makefile rather +# than invented, so the two CUDA images cover the same GPUs: amd64 datacenter + +# consumer, and l4t/arm64 covering Orin (87), Thor (110) and GB10 (121a). +# +# -DDS4_CUDA_HAVE_MXF4=1 is deliberately NOT set. Upstream only defines it for +# single-arch sm_120/sm_121 builds and guards the code with a plain #ifdef +# rather than __CUDA_ARCH__, so it cannot be combined with older archs in one +# fat binary. It gates an optional MXFP4 indexer fast path whose #ifndef branch +# returns 0 and falls back to the generic path, so omitting it costs some speed +# on GB10, not correctness. Revisit if upstream adds __CUDA_ARCH__ guards. +# +# An EMPTY CUDA_MAJOR_VERSION means a local developer build, not CI: fall back +# to upstream's own `native` handling, which needs a GPU present but is what a +# developer building on their own machine wants. Both variables are `?=` so an +# explicit value on the command line always wins. +UNAME_M := $(shell uname -m) +CUDA_MAJOR_VERSION ?= +ifeq ($(BUILD_TYPE),cublas) +ifeq ($(CUDA_MAJOR_VERSION),13) +ifeq ($(UNAME_M),aarch64) + DS4_NVCC_ARCH_FLAGS ?= -gencode arch=compute_87,code=sm_87 \ + -gencode arch=compute_90a,code=sm_90a \ + -gencode arch=compute_100a,code=sm_100a \ + -gencode arch=compute_110,code=sm_110 \ + -gencode arch=compute_121a,code=sm_121a +else + DS4_NVCC_ARCH_FLAGS ?= -gencode arch=compute_80,code=sm_80 \ + -gencode arch=compute_86,code=sm_86 \ + -gencode arch=compute_89,code=sm_89 \ + -gencode arch=compute_90a,code=sm_90a \ + -gencode arch=compute_100a,code=sm_100a \ + -gencode arch=compute_103a,code=sm_103a \ + -gencode arch=compute_120a,code=sm_120a \ + -gencode arch=compute_121a,code=sm_121a +endif + DS4_ARCH_MAKEVARS := NVCC_ARCH_FLAGS="$(DS4_NVCC_ARCH_FLAGS)" +else ifeq ($(CUDA_MAJOR_VERSION),) + # Local build: let upstream resolve the host GPU. + DS4_ARCH_MAKEVARS := CUDA_ARCH=native +else + $(error CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) has no architecture list here (13 does). Leave it empty for a native build, or pass DS4_NVCC_ARCH_FLAGS explicitly.) +endif +endif + +# Upstream splits image preprocessing, distributed inference, tensor-parallel +# transport, the SSD expert cache, and layer placement into GPU-agnostic +# translation units. They are shared by every GPU mode, so append them +# unconditionally below. ifeq ($(BUILD_TYPE),cublas) CMAKE_ARGS += -DDS4_GPU=cuda - DS4_OBJ_TARGET := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o \ + DS4_OBJ_TARGET := ds4.o ds4_image.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o \ cuda/mmq/ds4_ggml_stubs.o cuda/mmq/ds4_mmq.o cuda/mmq/ds4_mmq_d2r.o \ cuda/mmq/quantize.o cuda/mmq/mmid.o cuda/mmq/mmvq.o cuda/mmq/ds4_repack.o else ifeq ($(UNAME_S),Darwin) CMAKE_ARGS += -DDS4_GPU=metal - DS4_OBJ_TARGET := ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o + DS4_OBJ_TARGET := ds4.o ds4_image.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o else # CPU reference path (Linux only - macOS CPU path is broken by VM bug per ds4 README). CMAKE_ARGS += -DDS4_GPU=cpu - DS4_OBJ_TARGET := ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o + DS4_OBJ_TARGET := ds4_cpu.o ds4_image.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o endif ifneq ($(NATIVE),true) @@ -57,11 +119,11 @@ ds4: # the right per-platform compile flags (Objective-C/Metal on Darwin, nvcc on Linux+CUDA). ds4/ds4.o: ds4 ifeq ($(BUILD_TYPE),cublas) - +$(MAKE) -C ds4 $(DS4_OBJ_TARGET) + +$(MAKE) -C ds4 $(DS4_ARCH_MAKEVARS) $(DS4_OBJ_TARGET) else ifeq ($(UNAME_S),Darwin) - +$(MAKE) -C ds4 ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o + +$(MAKE) -C ds4 ds4.o ds4_image.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o else - +$(MAKE) -C ds4 ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o + +$(MAKE) -C ds4 ds4_cpu.o ds4_image.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o endif grpc-server: ds4/ds4.o diff --git a/backend/cpp/ds4/dsml_parser.cpp b/backend/cpp/ds4/dsml_parser.cpp index 6fb88a9fc..e360c580c 100644 --- a/backend/cpp/ds4/dsml_parser.cpp +++ b/backend/cpp/ds4/dsml_parser.cpp @@ -92,7 +92,8 @@ std::string json_escape(const std::string &in) { } // namespace -DsmlParser::DsmlParser() = default; +DsmlParser::DsmlParser(bool starts_in_thinking) + : state_(starts_in_thinking ? State::THINK : State::TEXT) {} bool DsmlParser::IsInDsmlStructural() const { switch (state_) { diff --git a/backend/cpp/ds4/dsml_parser.h b/backend/cpp/ds4/dsml_parser.h index c09833673..14b977af6 100644 --- a/backend/cpp/ds4/dsml_parser.h +++ b/backend/cpp/ds4/dsml_parser.h @@ -17,7 +17,9 @@ struct ParserEvent { // Streaming parser. Stateless across instances; one per Predict call. class DsmlParser { public: - DsmlParser(); + // The chat prompt may already contain the opening thinking marker, so the + // generated text can begin directly with reasoning bytes. + explicit DsmlParser(bool starts_in_thinking = false); // Feed a chunk of raw model-emitted text. Appends classified events to // `out`. May buffer the tail of `chunk` internally if it looks like a @@ -43,7 +45,7 @@ public: private: enum class State { TEXT, THINK, TOOL_CALLS, INVOKE, PARAM_VALUE }; - State state_ = State::TEXT; + State state_; std::string buf_; std::string current_tool_name_; int tool_index_ = -1; diff --git a/backend/cpp/ds4/dsml_parser_test.cpp b/backend/cpp/ds4/dsml_parser_test.cpp new file mode 100644 index 000000000..325c36252 --- /dev/null +++ b/backend/cpp/ds4/dsml_parser_test.cpp @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: MIT +// Standalone regression tests for the DSML streaming parser. +// +// The repository's backend/cpp/run-unit-tests.sh harness compiles each +// *_test.cpp as a single translation unit, so include the implementation here. + +#include "dsml_parser.cpp" + +#include +#include +#include +#include + +namespace { + +struct ParsedText { + std::string content; + std::string reasoning; +}; + +int failures = 0; + +void check_equal(const std::string &got, const std::string &want, + const char *name) { + if (got == want) return; + std::fprintf(stderr, "FAIL %s: got \"%s\", want \"%s\"\n", + name, got.c_str(), want.c_str()); + failures++; +} + +void collect_text(const std::vector &events, + ParsedText *parsed) { + for (const auto &event : events) { + if (event.type == ds4cpp::ParserEvent::CONTENT) { + parsed->content += event.text; + } else if (event.type == ds4cpp::ParserEvent::REASONING) { + parsed->reasoning += event.text; + } + } +} + +ParsedText parse_chunks(ds4cpp::DsmlParser *parser, + const std::vector &chunks) { + ParsedText parsed; + for (const auto &chunk : chunks) { + std::vector events; + parser->Feed(chunk, events); + collect_text(events, &parsed); + } + std::vector events; + parser->Flush(events); + collect_text(events, &parsed); + return parsed; +} + +template +void test_reasoning_opened_by_prompt() { + if constexpr (!std::is_constructible_v) { + std::fprintf(stderr, + "FAIL reasoning_opened_by_prompt: parser cannot start in thinking state\n"); + failures++; + } else { + Parser parser(true); + ParsedText parsed = parse_chunks( + &parser, + {"We need to calculate factorial recursively.Here is the answer."}); + check_equal(parsed.reasoning, + "We need to calculate factorial recursively.", + "reasoning_opened_by_prompt:reasoning"); + check_equal(parsed.content, "Here is the answer.", + "reasoning_opened_by_prompt:content"); + } +} + +template +Parser text_parser() { + if constexpr (std::is_constructible_v) { + return Parser(false); + } else { + return Parser(); + } +} + +void test_reasoning_disabled() { + auto parser = text_parser(); + ParsedText parsed = parse_chunks(&parser, {"Here is the answer."}); + check_equal(parsed.reasoning, "", "reasoning_disabled:reasoning"); + check_equal(parsed.content, "Here is the answer.", + "reasoning_disabled:content"); +} + +void test_explicit_think_tag() { + auto parser = text_parser(); + ParsedText parsed = parse_chunks( + &parser, {"reasoninganswer"}); + check_equal(parsed.reasoning, "reasoning", "explicit_think_tag:reasoning"); + check_equal(parsed.content, "answer", "explicit_think_tag:content"); +} + +template +void test_split_think_close_marker() { + if constexpr (!std::is_constructible_v) { + std::fprintf(stderr, + "FAIL split_think_close_marker: parser cannot start in thinking state\n"); + failures++; + } else { + Parser parser(true); + ParsedText parsed = parse_chunks( + &parser, + {"We need ", "to calculate ", "factorial", "", + "Here is ", "the answer."}); + check_equal(parsed.reasoning, "We need to calculate factorial", + "split_think_close_marker:reasoning"); + check_equal(parsed.content, "Here is the answer.", + "split_think_close_marker:content"); + } +} + +} // namespace + +int main() { + test_reasoning_opened_by_prompt(); + test_reasoning_disabled(); + test_explicit_think_tag(); + test_split_think_close_marker(); + + if (failures == 0) { + std::fprintf(stderr, "all dsml_parser checks passed\n"); + return 0; + } + std::fprintf(stderr, "%d check(s) failed\n", failures); + return 1; +} diff --git a/backend/cpp/ds4/generation_limits.h b/backend/cpp/ds4/generation_limits.h new file mode 100644 index 000000000..0985b0590 --- /dev/null +++ b/backend/cpp/ds4/generation_limits.h @@ -0,0 +1,27 @@ +// SPDX-License-Identifier: MIT +#pragma once + +#include + +namespace ds4cpp { + +inline int EffectiveGenerationLimit(int requested, int context_size, + int session_position) { + const int limit = requested > 0 ? requested : 256; + const int room = context_size - session_position; + if (room <= 1) return 0; + return std::min(limit, room - 1); +} + +inline int RemainingGenerationBudget(int effective_limit, int produced) { + if (effective_limit <= produced) return 0; + return effective_limit - produced; +} + +inline int SpeculativeAcceptedCapacity(int remaining, int draft_allowance, + int buffer_capacity) { + if (remaining <= 0 || draft_allowance < 0 || buffer_capacity <= 0) return 0; + return std::min({remaining, draft_allowance + 1, buffer_capacity}); +} + +} // namespace ds4cpp diff --git a/backend/cpp/ds4/generation_limits_test.cpp b/backend/cpp/ds4/generation_limits_test.cpp new file mode 100644 index 000000000..ee38251f6 --- /dev/null +++ b/backend/cpp/ds4/generation_limits_test.cpp @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: MIT + +#include "generation_limits.h" + +#include + +namespace { + +int failures = 0; + +void check_equal(int got, int want, const char *name) { + if (got == want) return; + std::fprintf(stderr, "FAIL %s: got %d, want %d\n", name, got, want); + failures++; +} + +// Mutation caught: treating omitted or negative max_tokens as unlimited instead +// of preserving DS4's legacy 256-token default. +void test_nonpositive_uses_legacy_default_when_space_permits() { + check_equal(ds4cpp::EffectiveGenerationLimit(0, 4096, 100), 256, + "zero max_tokens uses legacy default"); + check_equal(ds4cpp::EffectiveGenerationLimit(-1, 4096, 100), 256, + "negative max_tokens uses legacy default"); +} + +// Mutation caught: applying the legacy default without clamping it to the +// post-prefill context room and reserved slot. +void test_legacy_default_is_clamped_by_context() { + check_equal(ds4cpp::EffectiveGenerationLimit(0, 300, 100), 199, + "legacy default is context-clamped"); +} + +// Mutation caught: allowing an explicitly large request to overrun the +// post-prefill context boundary. +void test_large_positive_limit_is_clamped_to_context() { + check_equal(ds4cpp::EffectiveGenerationLimit(32768, 32768, 100), 32667, + "large positive is context-clamped"); +} + +// Mutation caught: replacing every positive request with the legacy default +// rather than preserving a smaller configured limit. +void test_smaller_positive_limit_is_preserved() { + check_equal(ds4cpp::EffectiveGenerationLimit(64, 4096, 100), 64, + "smaller positive is preserved"); +} + +// Mutation caught: consuming the final context slot instead of reserving it as +// required by DS4's generation loop. +void test_no_usable_room_returns_zero() { + check_equal(ds4cpp::EffectiveGenerationLimit(32, 100, 99), 0, + "one remaining context slot is not usable"); +} + +// Mutation caught: sending the original generation limit to a later +// speculative cycle instead of subtracting tokens already produced. +void test_remaining_budget_accounts_for_produced_tokens() { + check_equal(ds4cpp::RemainingGenerationBudget(10, 4), 6, + "remaining budget subtracts produced tokens"); + check_equal(ds4cpp::RemainingGenerationBudget(10, 12), 0, + "remaining budget never becomes negative"); +} + +// Mutation caught: giving speculative evaluation capacity beyond either the +// output budget, the draft allowance plus its first target token, or the fixed +// accepted-token buffer. +void test_speculative_capacity_obeys_all_bounds() { + check_equal(ds4cpp::SpeculativeAcceptedCapacity(3, 8, 8), 3, + "capacity respects remaining output budget"); + check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 4, 8), 5, + "capacity includes one target token beyond draft allowance"); + check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 8, 6), 6, + "capacity respects fixed buffer"); +} + +} // namespace + +int main() { + test_nonpositive_uses_legacy_default_when_space_permits(); + test_legacy_default_is_clamped_by_context(); + test_large_positive_limit_is_clamped_to_context(); + test_smaller_positive_limit_is_preserved(); + test_no_usable_room_returns_zero(); + test_remaining_budget_accounts_for_produced_tokens(); + test_speculative_capacity_obeys_all_bounds(); + + if (failures == 0) { + std::fprintf(stderr, "all generation limit checks passed\n"); + return 0; + } + std::fprintf(stderr, "%d check(s) failed\n", failures); + return 1; +} diff --git a/backend/cpp/ds4/grpc-server.cpp b/backend/cpp/ds4/grpc-server.cpp index 924da5476..d00f6ba3f 100644 --- a/backend/cpp/ds4/grpc-server.cpp +++ b/backend/cpp/ds4/grpc-server.cpp @@ -10,7 +10,9 @@ #include "dsml_parser.h" // populated in Task 12 #include "dsml_renderer.h" // populated in Task 16 +#include "generation_limits.h" #include "kv_cache.h" // populated in Task 17 +#include "request_lifecycle.h" extern "C" { #include "ds4.h" @@ -35,6 +37,7 @@ extern "C" { #include #include #include +#include #include using grpc::Server; @@ -69,6 +72,21 @@ int g_route_timeout_sec = 60; std::atomic g_server{nullptr}; +static bool server_context_cancelled(void *ud) { + return static_cast(ud)->IsCancelled(); +} + +static void set_session_cancel(void *target, ds4cpp::CancelCallback callback, + void *userdata) noexcept { + ds4_session_set_cancel(static_cast(target), callback, userdata); +} + +static bool request_should_continue(ds4cpp::RequestLifecycle *request, + ServerContext *context) { + request->ObserveContextCancellation(context->IsCancelled()); + return request->ShouldContinue(); +} + // Parse a "key:value" option string. Returns empty when no colon. static std::pair split_option(const std::string &opt) { auto colon = opt.find(':'); @@ -238,37 +256,58 @@ static bool apply_engine_option(ds4_engine_options *opt, const std::string &key, // When acting as a distributed coordinator, block until the worker route // covers all layers (ds4_session_distributed_route_ready == 1) or the timeout -// elapses. Returns an empty string on success, or an error message to return -// to the client. No-op when not distributed. +// elapses. No-op when not distributed. // // Takes the g_engine_mu lock by reference and RELEASES it during each poll // sleep. The wait can span up to g_route_timeout_sec seconds while workers // connect; holding g_engine_mu the whole time would block the Status/Health // readiness probes (they also lock g_engine_mu), making LocalAI's loader treat // a still-starting worker as hung. -static std::string wait_route_ready(std::unique_lock &lock) { - if (!g_distributed) return ""; +struct RouteWaitResult { + ds4cpp::RouteWaitDecision decision; + std::string error; +}; + +static RouteWaitResult wait_route_ready(std::unique_lock &lock, + ServerContext *context) { + if (!g_distributed) return {ds4cpp::RouteWaitDecision::Ready, ""}; char err[256] = {0}; const int deadline_polls = g_route_timeout_sec * 10; // 100ms per poll for (int i = 0; i <= deadline_polls; ++i) { int ready = ds4_session_distributed_route_ready(g_session, err, sizeof(err)); - if (ready == 1) return ""; - if (ready < 0) { - return std::string("ds4 distributed route error: ") + - (err[0] ? err : "unknown"); + switch (ds4cpp::DecideRouteWait(ready, context->IsCancelled())) { + case ds4cpp::RouteWaitDecision::Ready: + return {ds4cpp::RouteWaitDecision::Ready, ""}; + case ds4cpp::RouteWaitDecision::Error: + return {ds4cpp::RouteWaitDecision::Error, + std::string("ds4 distributed route error: ") + + (err[0] ? err : "unknown")}; + case ds4cpp::RouteWaitDecision::Cancelled: + return {ds4cpp::RouteWaitDecision::Cancelled, ""}; + case ds4cpp::RouteWaitDecision::Pending: + break; } + if (i == deadline_polls) break; // Release the lock while sleeping so Status/Health and other RPCs can // interleave during worker startup. lock.unlock(); struct timespec ts = {0, 100L * 1000L * 1000L}; // 100ms nanosleep(&ts, nullptr); lock.lock(); + if (context->IsCancelled()) { + return {ds4cpp::RouteWaitDecision::Cancelled, ""}; + } // A concurrent Free() may have torn down the engine while we slept. if (!g_engine || !g_session) { - return "ds4: model unloaded while waiting for distributed route"; + return {ds4cpp::RouteWaitDecision::Error, + "ds4: model unloaded while waiting for distributed route"}; } } - return "ds4 distributed route incomplete: workers not connected (layers uncovered)"; + if (context->IsCancelled()) { + return {ds4cpp::RouteWaitDecision::Cancelled, ""}; + } + return {ds4cpp::RouteWaitDecision::Error, + "ds4 distributed route incomplete: workers not connected (layers uncovered)"}; } static void append_token_text(ds4_engine *engine, int token, std::string &out) { @@ -341,9 +380,9 @@ static void collect_done(void *) {} struct StreamCtx { ds4_engine *engine; ServerWriter *writer; + ds4cpp::RequestLifecycle *request; ds4cpp::DsmlParser parser; int tokens; - bool aborted; // Track which tool indices we've seen TOOL_START for, so subsequent // ARGS deltas can elide the redundant id/name fields. std::vector tool_started; @@ -351,7 +390,7 @@ struct StreamCtx { static void stream_emit(void *ud, int token) { auto *s = static_cast(ud); - if (s->aborted) return; + if (!s->request->ShouldContinue()) return; if (token == ds4_token_eos(s->engine)) return; size_t len = 0; const char *text = ds4_token_text(s->engine, token, &len); @@ -401,7 +440,7 @@ static void stream_emit(void *ud, int token) { reply.set_message(chunk); reply.set_tokens(1); if (any_field) { - if (!s->writer->Write(reply)) s->aborted = true; + s->request->ObserveStreamWrite(s->writer->Write(reply)); } s->tokens++; } @@ -757,21 +796,30 @@ public: return GStatus::OK; } - GStatus Predict(ServerContext *, const backend::PredictOptions *request, + GStatus Predict(ServerContext *context, const backend::PredictOptions *request, backend::Reply *reply) override { std::unique_lock lock(g_engine_mu); if (!g_engine || !g_session) { return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded"); } if (GStatus id = check_model_identity(request); !id.ok()) return id; - if (std::string route_err = wait_route_ready(lock); !route_err.empty()) { - return GStatus(StatusCode::UNAVAILABLE, route_err); + RouteWaitResult route = wait_route_ready(lock, context); + if (route.decision == ds4cpp::RouteWaitDecision::Cancelled) { + return GStatus(StatusCode::CANCELLED, "ds4 request cancelled"); + } + if (route.decision == ds4cpp::RouteWaitDecision::Error) { + return GStatus(StatusCode::UNAVAILABLE, route.error); } ds4_tokens prompt = {}; build_prompt(g_engine, request, &prompt); - int n_predict = request->tokens() > 0 ? request->tokens() : 256; - CollectCtx collect = {g_engine, "", {}, reply, 0, {}, "", ""}; + const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request)); + const bool starts_in_thinking = think_enabled && + request->usetokenizertemplate() && request->messages_size() > 0; + CollectCtx collect = { + g_engine, "", ds4cpp::DsmlParser(starts_in_thinking), + reply, 0, {}, "", ""}; + ds4cpp::RequestLifecycle lifecycle; std::string cache_key = render_prompt_text(request); size_t cache_hit = maybe_load_cache(cache_key); (void)cache_hit; // future: skip prompt prefix if hit covers full prompt @@ -783,15 +831,27 @@ public: // Either way g_session advances so the disk KV cache picks up a // real checkpoint after the call (see maybe_save_cache below). char err[256] = {0}; - int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err)); + int rc; + { + ds4cpp::CancelCallbackScope cancel_scope( + g_session, set_session_cancel, server_context_cancelled, context); + rc = ds4_session_sync(g_session, &prompt, err, sizeof(err)); + } int prompt_len = prompt.len; ds4_tokens_free(&prompt); - if (rc == 0) { + if (rc == DS4_SESSION_SYNC_INTERRUPTED) { + lifecycle.ObserveContextCancellation(true); + } + const bool generation_started = rc == 0; + if (generation_started) { + const int n_predict = ds4cpp::EffectiveGenerationLimit( + request->tokens(), ds4_session_ctx(g_session), + ds4_session_pos(g_session)); const int eos = ds4_token_eos(g_engine); const int draft_max = ds4_engine_mtp_draft_tokens(g_engine); - const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request)); int produced = 0; while (produced < n_predict) { + if (!request_should_continue(&lifecycle, context)) break; SampleParams sp = compute_sample_params(request, collect.parser, think_enabled); int first; if (sp.temperature <= 0.0f) { @@ -806,13 +866,20 @@ public: if (draft_max > 0 && sp.temperature <= 0.0f) { constexpr int kAcceptedMax = 8; int accepted[kAcceptedMax]; - int cap = std::min(kAcceptedMax, draft_max + 1); + const int remaining = ds4cpp::RemainingGenerationBudget( + n_predict, produced); + const int cap = ds4cpp::SpeculativeAcceptedCapacity( + remaining, draft_max, kAcceptedMax); int n = ds4_session_eval_speculative_argmax( - g_session, first, draft_max, eos, + g_session, first, remaining, eos, accepted, cap, err, sizeof(err)); if (n < 0) { rc = -1; break; } bool stop = false; for (int j = 0; j < n; ++j) { + if (!request_should_continue(&lifecycle, context)) { + stop = true; + break; + } if (accepted[j] == eos) { stop = true; break; } collect_emit(&collect, accepted[j]); if (++produced >= n_predict) { stop = true; break; } @@ -821,12 +888,26 @@ public: } else { collect_emit(&collect, first); if (++produced >= n_predict) break; + if (!request_should_continue(&lifecycle, context)) break; rc = ds4_session_eval(g_session, first, err, sizeof(err)); if (rc != 0) break; } } - collect_done(&collect); } + + request_should_continue(&lifecycle, context); + ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision( + rc == DS4_SESSION_SYNC_INTERRUPTED, rc != 0, + !lifecycle.ShouldFinalize()); + if (!terminal.should_finalize) { + if (terminal.cause == ds4cpp::TerminalCause::EngineError) { + return GStatus(StatusCode::INTERNAL, + std::string("ds4 generation failed: ") + err); + } + return GStatus(StatusCode::CANCELLED, + "ds4 request cancelled"); + } + if (generation_started) collect_done(&collect); maybe_save_cache(cache_key); // Flush any buffered parser state. @@ -834,7 +915,7 @@ public: collect.parser.Flush(events); apply_events(&collect, events); - if (rc != 0) { + if (terminal.cause == ds4cpp::TerminalCause::EngineError) { return GStatus(StatusCode::INTERNAL, std::string("ds4 generation failed: ") + err); } @@ -857,21 +938,30 @@ public: return GStatus::OK; } - GStatus PredictStream(ServerContext *, const backend::PredictOptions *request, + GStatus PredictStream(ServerContext *context, const backend::PredictOptions *request, ServerWriter *writer) override { std::unique_lock lock(g_engine_mu); if (!g_engine || !g_session) { return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded"); } if (GStatus id = check_model_identity(request); !id.ok()) return id; - if (std::string route_err = wait_route_ready(lock); !route_err.empty()) { - return GStatus(StatusCode::UNAVAILABLE, route_err); + RouteWaitResult route = wait_route_ready(lock, context); + if (route.decision == ds4cpp::RouteWaitDecision::Cancelled) { + return GStatus(StatusCode::CANCELLED, "ds4 request cancelled"); + } + if (route.decision == ds4cpp::RouteWaitDecision::Error) { + return GStatus(StatusCode::UNAVAILABLE, route.error); } ds4_tokens prompt = {}; build_prompt(g_engine, request, &prompt); - int n_predict = request->tokens() > 0 ? request->tokens() : 256; - StreamCtx s = {g_engine, writer, {}, 0, false, {}}; + const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request)); + const bool starts_in_thinking = think_enabled && + request->usetokenizertemplate() && request->messages_size() > 0; + ds4cpp::RequestLifecycle lifecycle; + StreamCtx s = { + g_engine, writer, &lifecycle, + ds4cpp::DsmlParser(starts_in_thinking), 0, {}}; std::string cache_key = render_prompt_text(request); size_t cache_hit = maybe_load_cache(cache_key); (void)cache_hit; @@ -879,14 +969,26 @@ public: // Manual loop on g_session - see Predict() above for the rationale. // MTP speculative path used when ds4_engine_mtp_draft_tokens > 0. char err[256] = {0}; - int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err)); + int rc; + { + ds4cpp::CancelCallbackScope cancel_scope( + g_session, set_session_cancel, server_context_cancelled, context); + rc = ds4_session_sync(g_session, &prompt, err, sizeof(err)); + } ds4_tokens_free(&prompt); - if (rc == 0) { + if (rc == DS4_SESSION_SYNC_INTERRUPTED) { + lifecycle.ObserveContextCancellation(true); + } + const bool generation_started = rc == 0; + if (generation_started) { + const int n_predict = ds4cpp::EffectiveGenerationLimit( + request->tokens(), ds4_session_ctx(g_session), + ds4_session_pos(g_session)); const int eos = ds4_token_eos(g_engine); const int draft_max = ds4_engine_mtp_draft_tokens(g_engine); - const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request)); int produced = 0; - while (produced < n_predict && !s.aborted) { + while (produced < n_predict) { + if (!request_should_continue(&lifecycle, context)) break; SampleParams sp = compute_sample_params(request, s.parser, think_enabled); int first; if (sp.temperature <= 0.0f) { @@ -900,50 +1002,77 @@ public: if (draft_max > 0 && sp.temperature <= 0.0f) { constexpr int kAcceptedMax = 8; int accepted[kAcceptedMax]; - int cap = std::min(kAcceptedMax, draft_max + 1); + const int remaining = ds4cpp::RemainingGenerationBudget( + n_predict, produced); + const int cap = ds4cpp::SpeculativeAcceptedCapacity( + remaining, draft_max, kAcceptedMax); int n = ds4_session_eval_speculative_argmax( - g_session, first, draft_max, eos, + g_session, first, remaining, eos, accepted, cap, err, sizeof(err)); if (n < 0) { rc = -1; break; } bool stop = false; for (int j = 0; j < n; ++j) { + if (!request_should_continue(&lifecycle, context)) { + stop = true; + break; + } if (accepted[j] == eos) { stop = true; break; } stream_emit(&s, accepted[j]); - if (s.aborted) { stop = true; break; } + if (!lifecycle.ShouldContinue()) { stop = true; break; } if (++produced >= n_predict) { stop = true; break; } } if (stop) break; } else { stream_emit(&s, first); - if (s.aborted || ++produced >= n_predict) break; + if (!lifecycle.ShouldContinue() || ++produced >= n_predict) break; + if (!request_should_continue(&lifecycle, context)) break; rc = ds4_session_eval(g_session, first, err, sizeof(err)); if (rc != 0) break; } } - stream_done(&s); } - maybe_save_cache(cache_key); - // Flush parser state. - std::vector events; - s.parser.Flush(events); - if (!events.empty() && !s.aborted) { - backend::Reply reply; - auto *delta = reply.add_chat_deltas(); - for (const auto &e : events) { - if (e.type == ds4cpp::ParserEvent::CONTENT) { - delta->set_content(delta->content() + e.text); - } else if (e.type == ds4cpp::ParserEvent::REASONING) { - delta->set_reasoning_content(delta->reasoning_content() + e.text); + request_should_continue(&lifecycle, context); + ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision( + rc == DS4_SESSION_SYNC_INTERRUPTED, rc != 0, + !lifecycle.ShouldFinalize()); + terminal = ds4cpp::RunPostlude( + terminal, + [&]() { + ds4cpp::DsmlParser staged_parser = s.parser; + std::vector events; + staged_parser.Flush(events); + bool write_succeeded = true; + if (!events.empty()) { + backend::Reply reply; + auto *delta = reply.add_chat_deltas(); + for (const auto &e : events) { + if (e.type == ds4cpp::ParserEvent::CONTENT) { + delta->set_content(delta->content() + e.text); + } else if (e.type == ds4cpp::ParserEvent::REASONING) { + delta->set_reasoning_content( + delta->reasoning_content() + e.text); + } + } + write_succeeded = s.writer->Write(reply); } - } - s.writer->Write(reply); - } + lifecycle.ObserveStreamWrite(write_succeeded); + request_should_continue(&lifecycle, context); + if (!lifecycle.ShouldFinalize()) return false; + s.parser = std::move(staged_parser); + if (generation_started) stream_done(&s); + return true; + }, + [&]() { maybe_save_cache(cache_key); }); - if (rc != 0 && !s.aborted) { + if (terminal.cause == ds4cpp::TerminalCause::EngineError) { return GStatus(StatusCode::INTERNAL, std::string("ds4 generation failed: ") + err); } + if (terminal.cause == ds4cpp::TerminalCause::Cancelled) { + return GStatus(StatusCode::CANCELLED, + "ds4 request cancelled"); + } return GStatus::OK; } diff --git a/backend/cpp/ds4/request_lifecycle.h b/backend/cpp/ds4/request_lifecycle.h new file mode 100644 index 000000000..c3bc88f3c --- /dev/null +++ b/backend/cpp/ds4/request_lifecycle.h @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: MIT +#pragma once + +namespace ds4cpp { + +using CancelCallback = bool (*)(void *); +using CancelSetter = void (*)(void *, CancelCallback, void *) noexcept; + +class CancelCallbackScope { +public: + CancelCallbackScope(void *target, CancelSetter setter, + CancelCallback callback, void *userdata) noexcept + : target_(target), setter_(setter) { + setter_(target_, callback, userdata); + } + + ~CancelCallbackScope() noexcept { + setter_(target_, nullptr, nullptr); + } + + CancelCallbackScope(const CancelCallbackScope &) = delete; + CancelCallbackScope &operator=(const CancelCallbackScope &) = delete; + +private: + void *target_; + CancelSetter setter_; +}; + +enum class RouteWaitDecision { + Pending, + Ready, + Error, + Cancelled, +}; + +inline RouteWaitDecision DecideRouteWait(int route_status, bool cancelled) { + if (cancelled) return RouteWaitDecision::Cancelled; + if (route_status > 0) return RouteWaitDecision::Ready; + if (route_status < 0) return RouteWaitDecision::Error; + return RouteWaitDecision::Pending; +} + +enum class TerminalCause { + Success, + Cancelled, + EngineError, +}; + +inline TerminalCause DecideTerminalCause(bool sync_interrupted, + bool engine_error, + bool abandoned) { + if (sync_interrupted) return TerminalCause::Cancelled; + if (engine_error) return TerminalCause::EngineError; + if (abandoned) return TerminalCause::Cancelled; + return TerminalCause::Success; +} + +struct TerminalDecision { + TerminalCause cause; + bool should_finalize; +}; + +inline TerminalDecision ResolveTerminalDecision(bool sync_interrupted, + bool engine_error, + bool abandoned) { + return { + DecideTerminalCause(sync_interrupted, engine_error, abandoned), + !sync_interrupted && !abandoned, + }; +} + +template +TerminalDecision RunPostlude(TerminalDecision terminal, + Finalize transactional_finalize, + Persist persist) { + if (!terminal.should_finalize) return terminal; + if (!transactional_finalize()) { + terminal.should_finalize = false; + if (terminal.cause != TerminalCause::EngineError) { + terminal.cause = TerminalCause::Cancelled; + } + return terminal; + } + persist(); + return terminal; +} + +class RequestLifecycle { +public: + void ObserveContextCancellation(bool cancelled) { + context_cancelled_ = context_cancelled_ || cancelled; + } + + void ObserveStreamWrite(bool succeeded) { + stream_write_aborted_ = stream_write_aborted_ || !succeeded; + } + + bool ShouldContinue() const { + return !context_cancelled_ && !stream_write_aborted_; + } + + bool ShouldFinalize() const { + return ShouldContinue(); + } + +private: + bool context_cancelled_ = false; + bool stream_write_aborted_ = false; +}; + +} // namespace ds4cpp diff --git a/backend/cpp/ds4/request_lifecycle_test.cpp b/backend/cpp/ds4/request_lifecycle_test.cpp new file mode 100644 index 000000000..d6b968604 --- /dev/null +++ b/backend/cpp/ds4/request_lifecycle_test.cpp @@ -0,0 +1,414 @@ +// SPDX-License-Identifier: MIT +// Standalone regression tests for DS4 request cancellation policy. + +#include "request_lifecycle.h" + +#include + +namespace { + +int failures = 0; + +struct FakeCancelTarget { + ds4cpp::CancelCallback callback = nullptr; + void *userdata = nullptr; + int installs = 0; + int clears = 0; +}; + +struct PostludeCounts { + int finalize_attempts = 0; + int finalize_commits = 0; + int cache_persists = 0; + bool cache_followed_commit = true; +}; + +ds4cpp::TerminalDecision run_fake_postlude( + ds4cpp::TerminalDecision terminal, bool finalize_succeeds, + PostludeCounts *counts) { + return ds4cpp::RunPostlude( + terminal, + [=]() { + counts->finalize_attempts++; + if (!finalize_succeeds) return false; + counts->finalize_commits++; + return true; + }, + [=]() { + counts->cache_followed_commit = counts->finalize_commits == 1; + counts->cache_persists++; + }); +} + +bool fake_cancel(void *) { + return false; +} + +void fake_set_cancel(void *target, ds4cpp::CancelCallback callback, + void *userdata) noexcept { + auto *fake = static_cast(target); + fake->callback = callback; + fake->userdata = userdata; + if (callback) { + fake->installs++; + } else { + fake->clears++; + } +} + +void check(bool condition, const char *name) { + if (condition) return; + std::fprintf(stderr, "FAIL %s\n", name); + failures++; +} + +// Production mutation caught: treating an active request as abandoned would +// skip its parser finalization and cache save. +void test_active_request_continues_and_finalizes() { + ds4cpp::RequestLifecycle request; + + check(request.ShouldContinue(), "active:continue"); + check(request.ShouldFinalize(), "active:finalize"); +} + +// Production mutation caught: omitting the ServerContext cancellation branch +// would continue decoding and finalize a partial response. +void test_context_cancellation_stops_without_finalizing() { + ds4cpp::RequestLifecycle request; + + request.ObserveContextCancellation(true); + + check(!request.ShouldContinue(), "context_cancelled:stop"); + check(!request.ShouldFinalize(), "context_cancelled:no_finalize"); +} + +// Production mutation caught: ignoring ServerWriter::Write failure would keep +// streaming and finalize a response whose client has gone away. +void test_stream_write_abort_stops_without_finalizing() { + ds4cpp::RequestLifecycle request; + + request.ObserveStreamWrite(false); + + check(!request.ShouldContinue(), "write_abort:stop"); + check(!request.ShouldFinalize(), "write_abort:no_finalize"); +} + +// Production mutation caught: combining cancellation and write failure with +// AND would fail to stop when either signal occurs on its own. +void test_cancellation_and_write_abort_are_independent_or_conditions() { + ds4cpp::RequestLifecycle cancelled; + cancelled.ObserveContextCancellation(true); + cancelled.ObserveStreamWrite(true); + + ds4cpp::RequestLifecycle write_aborted; + write_aborted.ObserveContextCancellation(false); + write_aborted.ObserveStreamWrite(false); + + check(!cancelled.ShouldContinue(), "or:context_only"); + check(!write_aborted.ShouldContinue(), "or:write_only"); +} + +// Production mutation caught: treating an incomplete distributed route as an +// error would return before workers have time to connect. +void test_route_wait_pending() { + check(ds4cpp::DecideRouteWait(0, false) == + ds4cpp::RouteWaitDecision::Pending, + "route_wait:pending"); +} + +// Production mutation caught: failing to recognize a complete route would +// keep a ready inference request in the polling loop. +void test_route_wait_ready() { + check(ds4cpp::DecideRouteWait(1, false) == + ds4cpp::RouteWaitDecision::Ready, + "route_wait:ready"); +} + +// Production mutation caught: ignoring a route probe error would poll until a +// misleading timeout instead of returning UNAVAILABLE promptly. +void test_route_wait_error() { + check(ds4cpp::DecideRouteWait(-1, false) == + ds4cpp::RouteWaitDecision::Error, + "route_wait:error"); +} + +// Production mutation caught: omitting cancellation from route waiting would +// leave an abandoned request blocked until the distributed timeout. +void test_route_wait_cancellation() { + check(ds4cpp::DecideRouteWait(0, true) == + ds4cpp::RouteWaitDecision::Cancelled, + "route_wait:cancelled"); +} + +// Production mutation caught: checking route errors before cancellation would +// report UNAVAILABLE for a request the client already abandoned. +void test_route_wait_cancellation_precedes_error() { + check(ds4cpp::DecideRouteWait(-1, true) == + ds4cpp::RouteWaitDecision::Cancelled, + "route_wait:cancellation_precedence"); +} + +// Production mutation caught: classifying a successful active request as a +// terminal failure would suppress its normal response finalization. +void test_terminal_success() { + check(ds4cpp::DecideTerminalCause(false, false, false) == + ds4cpp::TerminalCause::Success, + "terminal:success"); +} + +// Production mutation caught: treating DS4's cooperative sync interruption +// as an ordinary engine error would return INTERNAL instead of CANCELLED. +void test_terminal_sync_interruption_is_cancelled() { + check(ds4cpp::DecideTerminalCause(true, true, true) == + ds4cpp::TerminalCause::Cancelled, + "terminal:sync_interrupted"); +} + +// Production mutation caught: treating every nonzero engine result as client +// abandonment would hide genuine DS4 failures behind CANCELLED. +void test_terminal_engine_error() { + check(ds4cpp::DecideTerminalCause(false, true, false) == + ds4cpp::TerminalCause::EngineError, + "terminal:engine_error"); +} + +// Production mutation caught: ignoring an rc==0 context cancellation would +// finalize and cache an abandoned request. +void test_terminal_context_abandonment() { + ds4cpp::RequestLifecycle request; + request.ObserveContextCancellation(true); + + check(ds4cpp::DecideTerminalCause( + false, false, !request.ShouldFinalize()) == + ds4cpp::TerminalCause::Cancelled, + "terminal:context_abandonment"); +} + +// Production mutation caught: ignoring an rc==0 stream write failure would +// finalize and cache an abandoned streaming request. +void test_terminal_write_abandonment() { + ds4cpp::RequestLifecycle request; + request.ObserveStreamWrite(false); + + check(ds4cpp::DecideTerminalCause( + false, false, !request.ShouldFinalize()) == + ds4cpp::TerminalCause::Cancelled, + "terminal:write_abandonment"); +} + +// Production mutation caught: checking late cancellation or write failure +// before a determined ordinary DS4 error would replace INTERNAL with CANCELLED. +void test_terminal_engine_error_precedes_late_abandonment() { + ds4cpp::RequestLifecycle cancelled; + cancelled.ObserveContextCancellation(true); + ds4cpp::RequestLifecycle write_aborted; + write_aborted.ObserveStreamWrite(false); + + check(ds4cpp::DecideTerminalCause( + false, true, !cancelled.ShouldFinalize()) == + ds4cpp::TerminalCause::EngineError, + "terminal:engine_error_precedes_cancellation"); + check(ds4cpp::DecideTerminalCause( + false, true, !write_aborted.ShouldFinalize()) == + ds4cpp::TerminalCause::EngineError, + "terminal:engine_error_precedes_write_abort"); +} + +// Production mutation caught: using status precedence alone to gate side +// effects would finalize and persist an engine-error request abandoned later. +void test_abandoned_engine_error_keeps_internal_without_finalizing() { + ds4cpp::RequestLifecycle request; + request.ObserveContextCancellation(true); + + ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision( + false, true, !request.ShouldFinalize()); + + check(terminal.cause == ds4cpp::TerminalCause::EngineError, + "terminal_decision:abandoned_engine_error_status"); + check(!terminal.should_finalize, + "terminal_decision:abandoned_engine_error_no_finalize"); +} + +// Production mutation caught: suppressing side effects for every engine error +// would change the existing finalization and cache behavior of active failures. +void test_active_engine_error_still_finalizes() { + ds4cpp::RequestLifecycle request; + + ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision( + false, true, !request.ShouldFinalize()); + + check(terminal.cause == ds4cpp::TerminalCause::EngineError, + "terminal_decision:active_engine_error_status"); + check(terminal.should_finalize, + "terminal_decision:active_engine_error_finalize"); +} + +// Production mutation caught: persisting before committed finalization would +// cache a state whose final buffered stream reply was never completed. +void test_postlude_active_success_commits_then_persists() { + PostludeCounts counts; + + ds4cpp::TerminalDecision terminal = run_fake_postlude( + {ds4cpp::TerminalCause::Success, true}, true, &counts); + + check(terminal.cause == ds4cpp::TerminalCause::Success, + "postlude:success_outcome"); + check(terminal.should_finalize, "postlude:success_committed"); + check(counts.finalize_attempts == 1, "postlude:success_attempts"); + check(counts.finalize_commits == 1, "postlude:success_commits"); + check(counts.cache_persists == 1, "postlude:success_cache"); + check(counts.cache_followed_commit, "postlude:success_cache_order"); +} + +// Production mutation caught: starting the postlude for an already-cancelled +// request would flush buffered parser state or persist an abandoned session. +void test_postlude_cancellation_skips_all_side_effects() { + PostludeCounts counts; + + ds4cpp::TerminalDecision terminal = run_fake_postlude( + {ds4cpp::TerminalCause::Cancelled, false}, true, &counts); + + check(terminal.cause == ds4cpp::TerminalCause::Cancelled, + "postlude:cancelled_outcome"); + check(counts.finalize_attempts == 0, "postlude:cancelled_attempts"); + check(counts.finalize_commits == 0, "postlude:cancelled_commits"); + check(counts.cache_persists == 0, "postlude:cancelled_cache"); +} + +// Production mutation caught: committing the live parser or cache after a +// failed final Write would publish an abandoned streaming postlude. +void test_postlude_finalize_failure_cancels_without_commit_or_cache() { + PostludeCounts counts; + + ds4cpp::TerminalDecision terminal = run_fake_postlude( + {ds4cpp::TerminalCause::Success, true}, false, &counts); + + check(terminal.cause == ds4cpp::TerminalCause::Cancelled, + "postlude:write_failure_outcome"); + check(!terminal.should_finalize, "postlude:write_failure_not_committed"); + check(counts.finalize_attempts == 1, "postlude:write_failure_attempts"); + check(counts.finalize_commits == 0, "postlude:write_failure_commits"); + check(counts.cache_persists == 0, "postlude:write_failure_cache"); +} + +// Production mutation caught: skipping the postlude for every engine error +// would change active internal-error finalization and cache behavior. +void test_postlude_active_engine_error_finalizes_and_persists() { + PostludeCounts counts; + + ds4cpp::TerminalDecision terminal = run_fake_postlude( + {ds4cpp::TerminalCause::EngineError, true}, true, &counts); + + check(terminal.cause == ds4cpp::TerminalCause::EngineError, + "postlude:engine_error_outcome"); + check(counts.finalize_attempts == 1, "postlude:engine_error_attempts"); + check(counts.finalize_commits == 1, "postlude:engine_error_commits"); + check(counts.cache_persists == 1, "postlude:engine_error_cache"); + check(counts.cache_followed_commit, "postlude:engine_error_cache_order"); +} + +// Production mutation caught: replacing every failed transactional finalize +// with cancellation would hide an already-determined engine error. +void test_postlude_engine_error_finalize_failure_preserves_internal() { + PostludeCounts counts; + + ds4cpp::TerminalDecision terminal = run_fake_postlude( + {ds4cpp::TerminalCause::EngineError, true}, false, &counts); + + check(terminal.cause == ds4cpp::TerminalCause::EngineError, + "postlude:engine_error_write_failure_outcome"); + check(!terminal.should_finalize, + "postlude:engine_error_write_failure_not_committed"); + check(counts.finalize_attempts == 1, + "postlude:engine_error_write_failure_attempts"); + check(counts.finalize_commits == 0, + "postlude:engine_error_write_failure_commits"); + check(counts.cache_persists == 0, + "postlude:engine_error_write_failure_cache"); +} + +// Production mutation caught: status precedence must not grant side-effect +// permission to an engine-error request that was also abandoned. +void test_postlude_abandoned_engine_error_skips_all_side_effects() { + PostludeCounts counts; + + ds4cpp::TerminalDecision terminal = run_fake_postlude( + {ds4cpp::TerminalCause::EngineError, false}, true, &counts); + + check(terminal.cause == ds4cpp::TerminalCause::EngineError, + "postlude:abandoned_engine_error_outcome"); + check(counts.finalize_attempts == 0, + "postlude:abandoned_engine_error_attempts"); + check(counts.finalize_commits == 0, + "postlude:abandoned_engine_error_commits"); + check(counts.cache_persists == 0, + "postlude:abandoned_engine_error_cache"); +} + +// Production mutation caught: failing to install the request callback would +// make DS4 prompt synchronization unable to observe client cancellation. +void test_cancel_callback_scope_installs_callback() { + FakeCancelTarget target; + int request_context = 42; + + { + ds4cpp::CancelCallbackScope scope( + &target, fake_set_cancel, fake_cancel, &request_context); + check(target.callback == fake_cancel, "cancel_scope:callback_installed"); + check(target.userdata == &request_context, "cancel_scope:userdata_installed"); + check(target.installs == 1, "cancel_scope:installed_once"); + } +} + +// Production mutation caught: failing to clear the callback at every scope +// exit would leave DS4 pointing at a destroyed stack-owned ServerContext. +void test_cancel_callback_scope_clears_callback() { + FakeCancelTarget target; + int request_context = 42; + + { + ds4cpp::CancelCallbackScope scope( + &target, fake_set_cancel, fake_cancel, &request_context); + } + + check(target.callback == nullptr, "cancel_scope:callback_cleared"); + check(target.userdata == nullptr, "cancel_scope:userdata_cleared"); + check(target.clears == 1, "cancel_scope:cleared_once"); +} + +} // namespace + +int main() { + test_active_request_continues_and_finalizes(); + test_context_cancellation_stops_without_finalizing(); + test_stream_write_abort_stops_without_finalizing(); + test_cancellation_and_write_abort_are_independent_or_conditions(); + test_route_wait_pending(); + test_route_wait_ready(); + test_route_wait_error(); + test_route_wait_cancellation(); + test_route_wait_cancellation_precedes_error(); + test_terminal_success(); + test_terminal_sync_interruption_is_cancelled(); + test_terminal_engine_error(); + test_terminal_context_abandonment(); + test_terminal_write_abandonment(); + test_terminal_engine_error_precedes_late_abandonment(); + test_abandoned_engine_error_keeps_internal_without_finalizing(); + test_active_engine_error_still_finalizes(); + test_postlude_active_success_commits_then_persists(); + test_postlude_cancellation_skips_all_side_effects(); + test_postlude_finalize_failure_cancels_without_commit_or_cache(); + test_postlude_active_engine_error_finalizes_and_persists(); + test_postlude_engine_error_finalize_failure_preserves_internal(); + test_postlude_abandoned_engine_error_skips_all_side_effects(); + test_cancel_callback_scope_installs_callback(); + test_cancel_callback_scope_clears_callback(); + + if (failures == 0) { + std::fprintf(stderr, "all request_lifecycle checks passed\n"); + return 0; + } + std::fprintf(stderr, "%d check(s) failed\n", failures); + return 1; +} diff --git a/backend/cpp/ik-llama-cpp/Makefile b/backend/cpp/ik-llama-cpp/Makefile index 847ddbb20..1d0304874 100644 --- a/backend/cpp/ik-llama-cpp/Makefile +++ b/backend/cpp/ik-llama-cpp/Makefile @@ -1,5 +1,5 @@ -IK_LLAMA_VERSION?=8337e4cd3861406fc04e0854b1409cd1b027fbc9 +IK_LLAMA_VERSION?=fe215a8ccdce6b844d2a3a3bbde08ae76a6284bf LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp CMAKE_ARGS?= diff --git a/backend/cpp/llama-cpp/Makefile b/backend/cpp/llama-cpp/Makefile index 41861f5f2..3f158f49a 100644 --- a/backend/cpp/llama-cpp/Makefile +++ b/backend/cpp/llama-cpp/Makefile @@ -1,5 +1,5 @@ -LLAMA_VERSION?=d59d455fd8ea09e5a2e87ce2a9d668267ffb5ccd +LLAMA_VERSION?=465e49b9cea78a68b9c244ffb48d0ee24a82873d LLAMA_REPO?=https://github.com/ggerganov/llama.cpp CMAKE_ARGS?= diff --git a/backend/cpp/llama-cpp/grpc-server.cpp b/backend/cpp/llama-cpp/grpc-server.cpp index 171ae0483..95345aaf1 100644 --- a/backend/cpp/llama-cpp/grpc-server.cpp +++ b/backend/cpp/llama-cpp/grpc-server.cpp @@ -88,6 +88,12 @@ using grpc::ServerBuilder; using grpc::ServerContext; using grpc::Status; +#if LOCALAI_HAS_MTMD_INIT_OPT +#define LOCALAI_MTMD_INIT_OPT_ARG(value) , value +#else +#define LOCALAI_MTMD_INIT_OPT_ARG(value) +#endif + // gRPC bearer token auth for distributed mode. // Reads LOCALAI_GRPC_AUTH_TOKEN from the environment. When set, rejects // requests without a matching "authorization: Bearer " metadata header. @@ -294,7 +300,7 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const } else { SRV_WRN("[TOOLS DEBUG] parse_options: Parsed tools JSON is not an array: %s\n", tools_json.dump().c_str()); } - } catch (const json::parse_error& e) { + } catch (const common_json_error& e) { SRV_WRN("Failed to parse tools JSON from proto: %s\n", e.what()); SRV_WRN("[TOOLS DEBUG] parse_options: Tools string that failed to parse: %s\n", predict->tools().c_str()); } @@ -324,7 +330,7 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const SRV_DBG("[TOOLS DEBUG] Received tool_choice object from Go layer: %s\n", tool_choice_json.dump().c_str()); } SRV_INF("Extracted tool_choice from proto: %s\n", predict->toolchoice().c_str()); - } catch (const json::parse_error& e) { + } catch (const common_json_error& e) { // If parsing fails, treat as string data["tool_choice"] = predict->toolchoice(); SRV_INF("Extracted tool_choice as string: %s\n", predict->toolchoice().c_str()); @@ -353,7 +359,7 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const // Add to data - llama.cpp server expects it as an object (map) data["logit_bias"] = logit_bias_json; SRV_INF("Using logit_bias: %s\n", predict->logitbias().c_str()); - } catch (const json::parse_error& e) { + } catch (const common_json_error& e) { SRV_ERR("Failed to parse logit_bias JSON from proto: %s\n", e.what()); } } @@ -398,7 +404,10 @@ json parse_options(bool streaming, const backend::PredictOptions* predict, const }); } - data["stop"] = predict->stopprompts(); + data["stop"] = json::array(); + for (const auto & stop : predict->stopprompts()) { + data["stop"].push_back(stop); + } // data["n_probs"] = predict->nprobs(); //TODO: images, @@ -1116,14 +1125,16 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt try { int n = std::stoi(optval_str); if (n < 0) n = 0; - // Keep override-name storage alive for the lifetime of the params struct - // (mirrors upstream arg.cpp behavior with a function-local static). +#if LOCALAI_HAS_N_CPU_FFN_HELPER + llm_add_n_cpu_ffn_overrides(n, LLM_FFN_EXPS_REGEX, params.speculative.draft.tensor_buft_overrides); +#else static std::list buft_overrides_draft; for (int i = 0; i < n; ++i) { buft_overrides_draft.push_back(llm_ffn_exps_block_regex(i)); params.speculative.draft.tensor_buft_overrides.push_back( {buft_overrides_draft.back().c_str(), ggml_backend_cpu_buffer_type()}); } +#endif } catch (...) {} } @@ -1141,14 +1152,16 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt try { int n = std::stoi(optval_str); if (n < 0) n = 0; - // Keep override-name storage alive for the lifetime of the - // params struct (mirrors upstream arg.cpp's function-local static). +#if LOCALAI_HAS_N_CPU_FFN_HELPER + llm_add_n_cpu_ffn_overrides(n, LLM_FFN_EXPS_REGEX, params.tensor_buft_overrides); +#else static std::list buft_overrides_main; for (int i = 0; i < n; ++i) { buft_overrides_main.push_back(llm_ffn_exps_block_regex(i)); params.tensor_buft_overrides.push_back( {buft_overrides_main.back().c_str(), ggml_backend_cpu_buffer_type()}); } +#endif } catch (...) {} } @@ -1795,7 +1808,7 @@ public: for (int j = 0; j < request->audios_size(); j++) rin.audios.push_back(request->audios(j)); for (int j = 0; j < request->videos_size(); j++) rin.videos.push_back(request->videos(j)); } - messages_json.push_back(llama_grpc::build_reconstructed_message(rin)); + messages_json.push_back(json::parse(llama_grpc::build_reconstructed_message(rin).dump())); } // Final safety check: Ensure no message has null content (Jinja templates require strings) @@ -1988,7 +2001,7 @@ public: if (!body_json.contains("chat_template_kwargs")) { body_json["chat_template_kwargs"] = json::object(); } - for (auto& el : ctk.items()) { + for (auto el : ctk.items()) { body_json["chat_template_kwargs"][el.key()] = el.value(); } } @@ -2074,30 +2087,27 @@ public: // If not using chat templates, extract files from image_data/audio_data fields // (If using chat templates, files were already extracted by oaicompat_chat_params_parse) if (!request->usetokenizertemplate() || request->messages_size() == 0 || ctx_server.impl->chat_params.tmpls == nullptr) { - const auto &images_data = data.find("image_data"); - if (images_data != data.end() && images_data->is_array()) + if (data.contains("image_data") && data.at("image_data").is_array()) { - for (const auto &img : *images_data) + for (const auto &img : data.at("image_data")) { auto decoded_data = base64_decode(img["data"].get()); files.push_back(decoded_data); } } - const auto &audio_data = data.find("audio_data"); - if (audio_data != data.end() && audio_data->is_array()) + if (data.contains("audio_data") && data.at("audio_data").is_array()) { - for (const auto &audio : *audio_data) + for (const auto &audio : data.at("audio_data")) { auto decoded_data = base64_decode(audio["data"].get()); files.push_back(decoded_data); } } - const auto &video_data = data.find("video_data"); - if (video_data != data.end() && video_data->is_array()) + if (data.contains("video_data") && data.at("video_data").is_array()) { - for (const auto &video : *video_data) + for (const auto &video : data.at("video_data")) { auto decoded_data = base64_decode(video["data"].get()); files.push_back(decoded_data); @@ -2111,10 +2121,10 @@ public: std::vector inputs; if (has_mtmd) { // multimodal - inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files)); + inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt))); } else { // Everything else, including multimodal completions. - inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true); + inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt)); } tasks.reserve(inputs.size()); @@ -2370,7 +2380,7 @@ public: for (int j = 0; j < request->audios_size(); j++) rin.audios.push_back(request->audios(j)); for (int j = 0; j < request->videos_size(); j++) rin.videos.push_back(request->videos(j)); } - messages_json.push_back(llama_grpc::build_reconstructed_message(rin)); + messages_json.push_back(json::parse(llama_grpc::build_reconstructed_message(rin).dump())); } // Final safety check: Ensure no message has null content (Jinja templates require strings) @@ -2563,7 +2573,7 @@ public: if (!body_json.contains("chat_template_kwargs")) { body_json["chat_template_kwargs"] = json::object(); } - for (auto& el : ctk.items()) { + for (auto el : ctk.items()) { body_json["chat_template_kwargs"][el.key()] = el.value(); } } @@ -2649,11 +2659,10 @@ public: // If not using chat templates, extract files from image_data/audio_data fields // (If using chat templates, files were already extracted by oaicompat_chat_params_parse) if (!request->usetokenizertemplate() || request->messages_size() == 0 || ctx_server.impl->chat_params.tmpls == nullptr) { - const auto &images_data = data.find("image_data"); - if (images_data != data.end() && images_data->is_array()) + if (data.contains("image_data") && data.at("image_data").is_array()) { - std::cout << "[PREDICT] Processing " << images_data->size() << " images" << std::endl; - for (const auto &img : *images_data) + std::cout << "[PREDICT] Processing " << data.at("image_data").size() << " images" << std::endl; + for (const auto &img : data.at("image_data")) { std::cout << "[PREDICT] Processing image" << std::endl; auto decoded_data = base64_decode(img["data"].get()); @@ -2661,20 +2670,18 @@ public: } } - const auto &audio_data = data.find("audio_data"); - if (audio_data != data.end() && audio_data->is_array()) + if (data.contains("audio_data") && data.at("audio_data").is_array()) { - for (const auto &audio : *audio_data) + for (const auto &audio : data.at("audio_data")) { auto decoded_data = base64_decode(audio["data"].get()); files.push_back(decoded_data); } } - const auto &video_data = data.find("video_data"); - if (video_data != data.end() && video_data->is_array()) + if (data.contains("video_data") && data.at("video_data").is_array()) { - for (const auto &video : *video_data) + for (const auto &video : data.at("video_data")) { auto decoded_data = base64_decode(video["data"].get()); files.push_back(decoded_data); @@ -2689,10 +2696,10 @@ public: std::vector inputs; if (has_mtmd) { // multimodal - inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files)); + inputs.push_back(process_mtmd_prompt(ctx_server.impl->mctx, prompt_str, files LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt))); } else { // Everything else, including multimodal completions. - inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true); + inputs = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt_str, true, true LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt)); } tasks.reserve(inputs.size()); @@ -2879,7 +2886,7 @@ public: json prompt = body.at("embeddings"); - auto tokenized_prompts = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt, true, true); + auto tokenized_prompts = tokenize_input_prompts(ctx_server.impl->vocab, ctx_server.impl->mctx, prompt, true, true LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt)); for (const auto & tokens : tokenized_prompts) { // this check is necessary for models that do not add BOS token to the input if (tokens.empty()) { @@ -2984,7 +2991,7 @@ public: tasks.reserve(documents.size()); for (size_t i = 0; i < documents.size(); i++) { - auto tmp = format_prompt_rerank(ctx_server.impl->model_tgt, ctx_server.impl->vocab, ctx_server.impl->mctx, request->query(), documents[i]); + auto tmp = format_prompt_rerank(ctx_server.impl->model_tgt, ctx_server.impl->vocab, ctx_server.impl->mctx, request->query(), documents[i] LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt)); server_task task = server_task(SERVER_TASK_TYPE_RERANK); task.id = rd.queue_tasks.get_new_id(); task.index = i; @@ -3005,7 +3012,7 @@ public: } // Collect responses - json responses = json::array(); + std::vector responses; for (auto & res : all_results.results) { GGML_ASSERT(dynamic_cast(res.get()) != nullptr); responses.push_back(res->to_json()); @@ -3018,7 +3025,7 @@ public: // Crop results by request.top_n if specified int top_n = request->top_n(); if (top_n > 0 && top_n < static_cast(responses.size())) { - responses = json(responses.begin(), responses.begin() + top_n); + responses.resize(top_n); } // Set usage information backend::Usage* usage = rerankResult->mutable_usage(); @@ -3065,7 +3072,7 @@ public: return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, opts.error); } - auto wrapper = mtmd_helper_bitmap_init_from_file(ctx_server.impl->mctx, opts.voice_path.c_str(), false); + auto wrapper = mtmd_helper_bitmap_init_from_file(ctx_server.impl->mctx, opts.voice_path.c_str(), false LOCALAI_MTMD_INIT_OPT_ARG(ctx_server.impl->init_opt)); if (!wrapper.bitmap) { return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "failed to read speaker reference audio: " + opts.voice_path); diff --git a/backend/cpp/llama-cpp/message_content.h b/backend/cpp/llama-cpp/message_content.h index 4c7317ecd..0b70c8b56 100644 --- a/backend/cpp/llama-cpp/message_content.h +++ b/backend/cpp/llama-cpp/message_content.h @@ -52,14 +52,15 @@ inline nlohmann::ordered_json normalize_message_content(const std::string& role, // (#7528). A multimodal user message legitimately carries a typed-part array // ({type:text}, {type:image_url}, ...), which must be left intact. Shared by the // streaming and non-streaming paths so this invariant cannot drift between them. -inline void normalize_template_message(nlohmann::ordered_json& msg) { +template +inline void normalize_template_message(Json& msg) { if (!msg.contains("content")) { msg["content"] = ""; // templates expect the field to exist return; } - nlohmann::ordered_json& content = msg["content"]; + auto& content = msg["content"]; const std::string role = (msg.contains("role") && msg["role"].is_string()) - ? msg["role"].get() + ? msg["role"].template get() : std::string(); if (content.is_null()) { content = ""; // #7324: null would crash content[:N] slicing diff --git a/backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch b/backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch index f056d47ce..253e8da5f 100644 --- a/backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch +++ b/backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch @@ -6,10 +6,9 @@ Subject: [PATCH 1/2] score-patch --- common/common.cpp | 6 +- common/common.h | 3 + - tools/CMakeLists.txt | 1 + tools/server/server-context.cpp | 358 +++++++++++++++++++++++++++++++- tools/server/server-task.h | 47 +++++ - 5 files changed, 406 insertions(+), 9 deletions(-) + 4 files changed, 405 insertions(+), 9 deletions(-) diff --git a/common/common.cpp b/common/common.cpp index 2e3f14c..0cec0dc 100644 @@ -42,15 +41,6 @@ index 878534d..4001df2 100644 int32_t n_sequences = 1; // number of sequences to decode int32_t n_outputs_max = 0; // max outputs in a batch (0 = n_batch) int32_t n_outputs_max_per_seq = 1; // max outputs per sequence -diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt -index 780df32..1d2fe8f 100644 ---- a/tools/CMakeLists.txt -+++ b/tools/CMakeLists.txt -@@ -41,3 +41,4 @@ else() - add_subdirectory(fit-params) - add_subdirectory(results) - endif() -+add_subdirectory(grpc-server) diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp index 3b5f6a1..d0e18e6 100644 --- a/tools/server/server-context.cpp diff --git a/backend/cpp/llama-cpp/patches/0002-add-server-task-type-tts.patch b/backend/cpp/llama-cpp/patches/0002-add-server-task-type-tts.patch index 6681b04e8..6789ffc0d 100644 --- a/backend/cpp/llama-cpp/patches/0002-add-server-task-type-tts.patch +++ b/backend/cpp/llama-cpp/patches/0002-add-server-task-type-tts.patch @@ -659,7 +659,7 @@ index 9069463fe..b7fa1e534 100644 + } + + if (speaker_ref_len > 0) { -+ auto wrapper = mtmd_helper_bitmap_init_from_buf(ctx_server.mctx, speaker_ref_data, speaker_ref_len, false); ++ auto wrapper = mtmd_helper_bitmap_init_from_buf(ctx_server.mctx, speaker_ref_data, speaker_ref_len, false, ctx_server.init_opt); + if (!wrapper.bitmap) { + res->error(format_error_response("failed to decode \"speaker_ref\"", ERROR_TYPE_INVALID_REQUEST)); + return res; diff --git a/backend/cpp/llama-cpp/prepare.sh b/backend/cpp/llama-cpp/prepare.sh index e658a940a..c76d15bb7 100644 --- a/backend/cpp/llama-cpp/prepare.sh +++ b/backend/cpp/llama-cpp/prepare.sh @@ -15,6 +15,30 @@ if [ -d "patches" ]; then done fi +## Apple RDMA link fixup. + +## ggml-rpc hands Apple's librdma to the linker with +## target_link_options(ggml-rpc PRIVATE "LINKER:-weak_library,..."). Link options are not +## a usage requirement of a static library, so in our BUILD_SHARED_LIBS=OFF build the flag +## dies with libggml-rpc.a and every ibv_* symbol transport-apple.cpp reaches for comes out +## undefined when grpc-server and ggml-rpc-server link. Re-declare the same weak link as +## INTERFACE so it travels to whoever links the static library. +## +## Guarded on the marker so a second prepare.sh over the same checkout is a no-op, and on +## GGML_RPC_RDMA_APPLE so forks that branched before the Apple RDMA transport (turboquant, +## bonsai) are left alone. +RPC_CMAKE=llama.cpp/ggml/src/ggml-rpc/CMakeLists.txt +if [ -f "$RPC_CMAKE" ] && grep -q "GGML_RPC_RDMA_APPLE" "$RPC_CMAKE" && ! grep -q "LOCALAI_RDMA_IFACE" "$RPC_CMAKE"; then + echo "==> ggml-rpc carries the Apple RDMA transport, re-declaring its weak librdma link as INTERFACE" + cat >> "$RPC_CMAKE" <<'EOF' + +# LOCALAI_RDMA_IFACE: added by backend/cpp/llama-cpp/prepare.sh +if (GGML_RPC_RDMA AND APPLE AND NOT BUILD_SHARED_LIBS) + target_link_options(ggml-rpc INTERFACE "LINKER:-weak_library,${RDMA_LIB}") +endif() +EOF +fi + for file in $(ls llama.cpp/tools/server/); do cp -rfv llama.cpp/tools/server/$file llama.cpp/tools/grpc-server/ done @@ -61,11 +85,23 @@ if grep -q "server_metrics metrics;" llama.cpp/tools/server/server-task.h; then else HAS_SERVER_METRICS=0 fi +if grep -q "mtmd_helper_init_opt" llama.cpp/tools/mtmd/mtmd-helper.h; then + HAS_MTMD_INIT_OPT=1 +else + HAS_MTMD_INIT_OPT=0 +fi +if grep -q "llm_add_n_cpu_ffn_overrides" llama.cpp/common/common.h; then + HAS_N_CPU_FFN_HELPER=1 +else + HAS_N_CPU_FFN_HELPER=0 +fi cat > llama.cpp/tools/grpc-server/llama_compat.h < LOCALAI_TURBOQUANT_NO_CHECKPOINT_MIN_STEP define OK" fi +# 3. The shared source follows current upstream and catches common_json_error. +# TurboQuant still exposes nlohmann::json directly, so its equivalent parse +# failures use json::parse_error instead. +if grep -q 'common_json_error' "$SRC"; then + echo "==> patching $SRC to use the TurboQuant JSON exception type" + awk '{ gsub(/common_json_error/, "json::parse_error"); print }' "$SRC" > "$SRC.tmp" + mv "$SRC.tmp" "$SRC" + echo "==> TurboQuant JSON exception patch OK" +else + echo "==> $SRC already uses a TurboQuant-compatible JSON exception type, skipping" +fi + echo "==> all patches applied" diff --git a/backend/go/crispasr/Makefile b/backend/go/crispasr/Makefile index b87b395a7..e01dbef0a 100644 --- a/backend/go/crispasr/Makefile +++ b/backend/go/crispasr/Makefile @@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1) # CrispASR version (release tag) CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR -CRISPASR_VERSION?=ae4474dd8306384a0e697183d863dfc52e69a2fb +CRISPASR_VERSION?=c6d4265ae2ee2b8931b09d7d25d5c65c75c36a41 SO_TARGET?=libgocrispasr.so CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF diff --git a/backend/go/depth-anything-cpp/Makefile b/backend/go/depth-anything-cpp/Makefile index 79a1598e2..267955618 100644 --- a/backend/go/depth-anything-cpp/Makefile +++ b/backend/go/depth-anything-cpp/Makefile @@ -14,7 +14,7 @@ JOBS?=$(shell nproc --ignore=1) # It is kept alive by the upstream tag da2-support (survives a squash-merge); # repoint to the master merge commit once mudler/depth-anything.cpp PR #1 lands. DEPTHANYTHING_REPO?=https://github.com/mudler/depth-anything.cpp.git -DEPTHANYTHING_VERSION?=54abd5c0abfd1f394e01cb3c38f2e3af4daedf85 +DEPTHANYTHING_VERSION?=02ba082274e001a63e50de5a1eb0ccc50c6af4b1 ifeq ($(NATIVE),false) CMAKE_ARGS+=-DGGML_NATIVE=OFF diff --git a/backend/go/nemo-speech-cpp/Makefile b/backend/go/nemo-speech-cpp/Makefile index 1120267f1..1496b4244 100644 --- a/backend/go/nemo-speech-cpp/Makefile +++ b/backend/go/nemo-speech-cpp/Makefile @@ -12,7 +12,7 @@ # runs 'make -C backend/go/$(BACKEND) build' and then copies package/), so it # has to produce the binary and the package, not just the shared libraries. -NEMO_SPEECH_VERSION?=4f9676226f667d14608487df744f375db87127f8 +NEMO_SPEECH_VERSION?=ffa38cb2408f1e832a36d46fef5e3e1e80d07e6c NEMO_SPEECH_REPO?=https://github.com/NVIDIA/NeMo-Speech.cpp GOCMD?=go diff --git a/backend/go/omnivoice-cpp/Makefile b/backend/go/omnivoice-cpp/Makefile index 4cb089dab..92810a5ba 100644 --- a/backend/go/omnivoice-cpp/Makefile +++ b/backend/go/omnivoice-cpp/Makefile @@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1) # omnivoice.cpp version OMNIVOICE_REPO?=https://github.com/ServeurpersoCom/omnivoice.cpp -OMNIVOICE_VERSION?=4f33af825d66e6ef1cb185e87b4589cacf747291 +OMNIVOICE_VERSION?=040c8b344d8c670ce1475194751d119b5ef82c78 SO_TARGET?=libgomnivoicecpp.so CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF diff --git a/backend/go/stablediffusion-ggml/Makefile b/backend/go/stablediffusion-ggml/Makefile index 1c1552074..e39a1d09c 100644 --- a/backend/go/stablediffusion-ggml/Makefile +++ b/backend/go/stablediffusion-ggml/Makefile @@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1) # stablediffusion.cpp (ggml) STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp -STABLEDIFFUSION_GGML_VERSION?=97d2990807fe6d558e395f8764198d7c7e7b411c +STABLEDIFFUSION_GGML_VERSION?=d8fb10c02977c8ca999f3fb4e02df9ecf10f7ba6 CMAKE_ARGS+=-DGGML_MAX_NAME=128 @@ -38,8 +38,11 @@ else ifeq ($(BUILD_TYPE),hipblas) ROCM_PATH ?= /opt/rocm export CXX=$(ROCM_HOME)/llvm/bin/clang++ export CC=$(ROCM_HOME)/llvm/bin/clang - AMDGPU_TARGETS?=gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1200,gfx1201 - CMAKE_ARGS+=-DSD_HIPBLAS=ON -DGGML_HIPBLAS=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS) + AMDGPU_TARGETS?=gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201 + # SD_HIPBLAS turns on ggml's HIP backend itself; GGML_HIPBLAS is the name ggml + # used before it was renamed to GGML_HIP, so passing it here only produced an + # unused-variable warning. + CMAKE_ARGS+=-DSD_HIPBLAS=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS) else ifeq ($(BUILD_TYPE),vulkan) CMAKE_ARGS+=-DSD_VULKAN=ON -DGGML_VULKAN=ON else ifeq ($(BUILD_TYPE),metal) diff --git a/backend/go/stablediffusion-ggml/cpp/gosd.cpp b/backend/go/stablediffusion-ggml/cpp/gosd.cpp index 12cc4a83e..7722e8d06 100644 --- a/backend/go/stablediffusion-ggml/cpp/gosd.cpp +++ b/backend/go/stablediffusion-ggml/cpp/gosd.cpp @@ -401,7 +401,6 @@ int load_model(const char *model, char *model_path, char* options[], int threads const char *params_backend_arg = ""; const char *rpc_servers_arg = ""; const char *max_vram_arg = ""; - bool stream_layers = false; int n_threads = threads; enum sd_type_t wtype = SD_TYPE_COUNT; @@ -510,7 +509,10 @@ int load_model(const char *model, char *model_path, char* options[], int threads if (!strcmp(optname, "params_backend")) params_backend_arg = strdup(optval); if (!strcmp(optname, "rpc_servers")) rpc_servers_arg = strdup(optval); if (!strcmp(optname, "max_vram")) max_vram_arg = strdup(optval); - if (!strcmp(optname, "stream_layers")) stream_layers = (strcmp(optval, "true") == 0 || strcmp(optval, "1") == 0); + if (!strcmp(optname, "stream_layers")) { + // Retained as a no-op for existing configurations. Upstream now + // selects segmented weight streaming automatically. + } // vae_decode_only is still accepted for backwards compatibility with // existing gallery configs, but upstream dropped the option (the model @@ -650,11 +652,9 @@ int load_model(const char *model, char *model_path, char* options[], int threads ctx_params.rpc_servers = env_rpc_servers; } } - // max_vram: GiB budget or per-backend spec for graph-cut segmented param - // offload ("0" = disabled, "-1" = auto). stream_layers only has effect when - // max_vram is set. + // max_vram is an optional GiB budget or per-backend spec for automatic + // graph-cut execution. A zero value uses the live free-VRAM budget. if (strlen(max_vram_arg) > 0) ctx_params.max_vram = max_vram_arg; - ctx_params.stream_layers = stream_layers; ctx_params.diffusion_flash_attn = diffusion_flash_attn; ctx_params.tae_preview_only = tae_preview_only; ctx_params.diffusion_conv_direct = diffusion_conv_direct; @@ -1438,4 +1438,3 @@ int unload() { free_sd_ctx(sd_c); return 0; } - diff --git a/backend/go/vllm-cpp/Makefile b/backend/go/vllm-cpp/Makefile index 754ce5560..598d814c3 100644 --- a/backend/go/vllm-cpp/Makefile +++ b/backend/go/vllm-cpp/Makefile @@ -11,7 +11,7 @@ JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || e # vllm.cpp version VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp -VLLM_CPP_VERSION?=438305e1577768ec0f75729456a4c8b9f425e2ee +VLLM_CPP_VERSION?=6bf3abb580982f4fd2e4525ef37802ee0ce28981 # MLX GEMM provider (darwin/metal only; see the metal branch below for why). # Consumed as the prebuilt pip wheel: building MLX from source needs `xcrun diff --git a/backend/go/vllm-cpp/govllmcpp.go b/backend/go/vllm-cpp/govllmcpp.go index b880105af..20e587a6b 100644 --- a/backend/go/vllm-cpp/govllmcpp.go +++ b/backend/go/vllm-cpp/govllmcpp.go @@ -1,6 +1,6 @@ package main -// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v21). +// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v23). // // The structs below are hand-mirrored PODs of the C declarations, with // explicit padding so the Go layout matches the C layout on linux/darwin @@ -21,7 +21,7 @@ import ( // the header of the VLLM_CPP_VERSION pinned in the Makefile: the build checks // the two against each other, because a mismatch is only caught at runtime by // registerLib, where it takes the backend down on every load (issue #11379). -const abiVersion = 21 +const abiVersion = 23 // The ABI's tri-state toggles (enable_prefix_caching ABI v7, // enable_jump_forward ABI v10) share one encoding: 0 is NOT "off", it is @@ -83,6 +83,7 @@ type cModelParams struct { LanguageModelOnly int32 // 0 = multimodal inputs enabled (ABI v19) _ [4]byte LimitMMPerPrompt uintptr // const char* JSON; NULL = default limits (ABI v19) + MMProjPath uintptr // const char*; NULL = no GGUF projector (ABI v22) } // cSamplingParams mirrors vllm_sampling_params (structured fields included). diff --git a/backend/go/vllm-cpp/options.go b/backend/go/vllm-cpp/options.go index 86717e7ac..b75e5a7fc 100644 --- a/backend/go/vllm-cpp/options.go +++ b/backend/go/vllm-cpp/options.go @@ -128,9 +128,40 @@ func parseOptions(opts *pb.ModelOptions) loadOptions { lo := loadOptions{} applyOptionsList(&lo, opts.GetOptions()) applyEngineArgs(&lo, opts.GetEngineArgs()) + applyDraftModelOption(&lo, opts.GetOptions()) return lo } +// applyDraftModelOption binds a managed companion snapshot after engine_args +// has supplied the speculative document. Companion paths do not exist until +// LocalAI materializes the artifact, so they must replace the gallery's static +// repository reference without disturbing the method or token budget. +func applyDraftModelOption(lo *loadOptions, options []string) { + if strings.TrimSpace(lo.speculativeConfig) == "" { + return + } + var draftModel string + for _, option := range options { + key, value, found := strings.Cut(option, ":") + if found && strings.TrimSpace(key) == "draft_model" { + draftModel = strings.TrimSpace(value) + } + } + if draftModel == "" { + return + } + + var spec map[string]any + if err := json.Unmarshal([]byte(lo.speculativeConfig), &spec); err != nil { + return + } + spec["model"] = draftModel + encoded, err := json.Marshal(spec) + if err == nil { + lo.speculativeConfig = string(encoded) + } +} + // applyOptionsList reads the legacy free-form "key:value" list. strings.Cut // splits on the FIRST colon only, so a JSON object value survives intact. func applyOptionsList(lo *loadOptions, options []string) { diff --git a/backend/go/vllm-cpp/options_test.go b/backend/go/vllm-cpp/options_test.go new file mode 100644 index 000000000..11b222b4a --- /dev/null +++ b/backend/go/vllm-cpp/options_test.go @@ -0,0 +1,38 @@ +package main + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + pb "github.com/mudler/LocalAI/pkg/grpc/proto" +) + +var _ = Describe("managed DFlash companion options", func() { + It("replaces only the draft model in an existing speculative configuration", func() { + managedPath := ".artifacts/huggingface/0123456789abcdef/snapshot" + lo := parseOptions(&pb.ModelOptions{ + Options: []string{"draft_model:" + managedPath}, + EngineArgs: `{ + "speculative_config": { + "method": "dflash", + "model": "Mia-AiLab/Qwen3.8-27B-DFlash2-EXL3-5.0bpw", + "num_speculative_tokens": 7 + } + }`, + }) + + Expect(lo.speculativeConfig).To(MatchJSON(`{ + "method": "dflash", + "model": ".artifacts/huggingface/0123456789abcdef/snapshot", + "num_speculative_tokens": 7 + }`)) + }) + + It("ignores a draft companion when speculative decoding is not configured", func() { + lo := parseOptions(&pb.ModelOptions{ + Options: []string{"draft_model:.artifacts/huggingface/0123456789abcdef/snapshot"}, + }) + + Expect(lo.speculativeConfig).To(BeEmpty()) + }) +}) diff --git a/backend/go/vllm-cpp/vllmcpp_test.go b/backend/go/vllm-cpp/vllmcpp_test.go index 681ca4d4e..55f6e211a 100644 --- a/backend/go/vllm-cpp/vllmcpp_test.go +++ b/backend/go/vllm-cpp/vllmcpp_test.go @@ -16,7 +16,7 @@ func TestVllmCpp(t *testing.T) { RunSpecs(t, "vllm-cpp suite") } -// The Go POD mirrors must match the C struct layout of vllm.h (ABI v21) +// The Go POD mirrors must match the C struct layout of vllm.h (ABI v23) // byte-for-byte: these offsets are the C offsets on LP64 (linux/darwin // amd64+arm64). A failure here means govllmcpp.go drifted from vllm.h. var _ = Describe("C ABI struct mirrors", func() { @@ -24,7 +24,7 @@ var _ = Describe("C ABI struct mirrors", func() { // VLLM_ABI_VERSION in the vllm.h of VLLM_CPP_VERSION (Makefile). // Moving the pin past this without growing the mirrors below ships a // backend that refuses every load at startup (issue #11379). - Expect(abiVersion).To(Equal(21)) + Expect(abiVersion).To(Equal(23)) }) It("cModelParams matches vllm_model_params", func() { @@ -51,7 +51,8 @@ var _ = Describe("C ABI struct mirrors", func() { Expect(unsafe.Offsetof(p.KVCacheMemoryBytes)).To(Equal(uintptr(104))) Expect(unsafe.Offsetof(p.LanguageModelOnly)).To(Equal(uintptr(112))) Expect(unsafe.Offsetof(p.LimitMMPerPrompt)).To(Equal(uintptr(120))) - Expect(unsafe.Sizeof(p)).To(Equal(uintptr(128))) + Expect(unsafe.Offsetof(p.MMProjPath)).To(Equal(uintptr(128))) + Expect(unsafe.Sizeof(p)).To(Equal(uintptr(136))) }) It("cSamplingParams matches vllm_sampling_params (ABI v8)", func() { diff --git a/backend/go/whisper/Makefile b/backend/go/whisper/Makefile index 214fba777..bd789a915 100644 --- a/backend/go/whisper/Makefile +++ b/backend/go/whisper/Makefile @@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1) # whisper.cpp version WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp -WHISPER_CPP_VERSION?=233fe1fc9b48a09e361d3594520838ca266537fe +WHISPER_CPP_VERSION?=52a939a2a762224e255d366c1182b2af4dd1a032 SO_TARGET?=libgowhisper.so CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF diff --git a/backend/index.yaml b/backend/index.yaml index 3420d8549..6e3519f66 100644 --- a/backend/index.yaml +++ b/backend/index.yaml @@ -510,7 +510,7 @@ default: "cpu-stablediffusion-ggml" nvidia: "cuda12-stablediffusion-ggml" intel: "intel-sycl-f16-stablediffusion-ggml" - # amd: "rocm-stablediffusion-ggml" + amd: "rocm-stablediffusion-ggml" vulkan: "vulkan-stablediffusion-ggml" nvidia-l4t: "nvidia-l4t-arm64-stablediffusion-ggml" metal: "metal-stablediffusion-ggml" @@ -2109,7 +2109,7 @@ default: "cpu-stablediffusion-ggml-development" nvidia: "cuda12-stablediffusion-ggml-development" intel: "intel-sycl-f16-stablediffusion-ggml-development" - # amd: "rocm-stablediffusion-ggml-development" + amd: "rocm-stablediffusion-ggml-development" vulkan: "vulkan-stablediffusion-ggml-development" nvidia-l4t: "nvidia-l4t-arm64-stablediffusion-ggml-development" metal: "metal-stablediffusion-ggml-development" @@ -3904,6 +3904,11 @@ uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-stablediffusion-ggml" mirrors: - localai/localai-backends:latest-gpu-nvidia-cuda-12-stablediffusion-ggml +- !!merge <<: *stablediffusionggml + name: "rocm-stablediffusion-ggml" + uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-stablediffusion-ggml" + mirrors: + - localai/localai-backends:latest-gpu-rocm-hipblas-stablediffusion-ggml - !!merge <<: *stablediffusionggml name: "intel-sycl-f32-stablediffusion-ggml" uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-stablediffusion-ggml" @@ -3917,6 +3922,11 @@ uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-stablediffusion-ggml" mirrors: - localai/localai-backends:master-gpu-nvidia-cuda-12-stablediffusion-ggml +- !!merge <<: *stablediffusionggml + name: "rocm-stablediffusion-ggml-development" + uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-stablediffusion-ggml" + mirrors: + - localai/localai-backends:master-gpu-rocm-hipblas-stablediffusion-ggml - !!merge <<: *stablediffusionggml name: "intel-sycl-f32-stablediffusion-ggml-development" uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-stablediffusion-ggml" diff --git a/backend/python/common/python_utils.py b/backend/python/common/python_utils.py index c89813e2c..88ec0a530 100644 --- a/backend/python/common/python_utils.py +++ b/backend/python/common/python_utils.py @@ -37,6 +37,46 @@ def parse_options(options_list): return opts +def attach_media_parts(messages_dicts, n_images=0, n_videos=0): + """Rebuild the last user message as content *parts* carrying media markers. + + Backends that let the tokenizer do the templating hand plain string content + to ``apply_chat_template``, but a chat template only emits the model's own + media tokens (``<|vision_start|><|image_pad|><|vision_end|>`` for the + Qwen-VL family, and the equivalents elsewhere) when the content is a list + of parts. Without those markers the engine's multimodal processor finds + nothing to substitute and silently discards the pixels, even though they + were forwarded correctly out of band. + + Returns a new list whose last user message has + ``[{"type": "image"} * n_images, {"type": "video"} * n_videos, text]`` as + its content, or ``None`` when there is nothing to attach - no media, no + user turn, or content that is already a list of parts - so the caller can + keep using the original string-content list. + """ + if not n_images and not n_videos: + return None + idx = next( + ( + i + for i in reversed(range(len(messages_dicts))) + if messages_dicts[i].get("role") == "user" + ), + None, + ) + if idx is None: + return None + text = messages_dicts[idx].get("content") or "" + if not isinstance(text, str): + return None + parts = [{"type": "image"}] * n_images + [{"type": "video"}] * n_videos + if text: + parts.append({"type": "text", "text": text}) + patched = list(messages_dicts) + patched[idx] = dict(patched[idx], content=parts) + return patched + + def messages_to_dicts(proto_messages): """Convert proto ``Message`` objects to dicts suitable for ``apply_chat_template``. diff --git a/backend/python/common/python_utils_test.py b/backend/python/common/python_utils_test.py index c395ce92d..d12bac9c5 100644 --- a/backend/python/common/python_utils_test.py +++ b/backend/python/common/python_utils_test.py @@ -14,7 +14,7 @@ import json import types import unittest -from python_utils import messages_to_dicts, parse_options +from python_utils import attach_media_parts, messages_to_dicts, parse_options def _msg(**fields): @@ -118,5 +118,63 @@ class TestMessagesToDicts(unittest.TestCase): self.assertNotIn("tool_calls", out[0]) +class TestAttachMediaParts(unittest.TestCase): + def test_image_marker_added_to_last_user_turn(self): + messages = [ + {"role": "system", "content": "be brief"}, + {"role": "user", "content": "first"}, + {"role": "assistant", "content": "ok"}, + {"role": "user", "content": "how high is the water?"}, + ] + out = attach_media_parts(messages, n_images=1) + self.assertEqual( + out[3]["content"], + [{"type": "image"}, {"type": "text", "text": "how high is the water?"}], + ) + # Earlier turns and the input list itself are untouched. + self.assertEqual(out[:3], messages[:3]) + self.assertEqual(messages[3]["content"], "how high is the water?") + + def test_counts_and_order_images_then_videos(self): + out = attach_media_parts( + [{"role": "user", "content": "describe"}], n_images=2, n_videos=1 + ) + self.assertEqual( + out[0]["content"], + [ + {"type": "image"}, + {"type": "image"}, + {"type": "video"}, + {"type": "text", "text": "describe"}, + ], + ) + + def test_empty_text_yields_media_only_parts(self): + out = attach_media_parts([{"role": "user", "content": ""}], n_images=1) + self.assertEqual(out[0]["content"], [{"type": "image"}]) + + def test_other_message_keys_are_preserved(self): + out = attach_media_parts( + [{"role": "user", "content": "hi", "name": "bob"}], n_images=1 + ) + self.assertEqual(out[0]["name"], "bob") + + def test_no_media_is_a_no_op(self): + self.assertIsNone(attach_media_parts([{"role": "user", "content": "hi"}])) + + def test_no_user_turn_is_a_no_op(self): + self.assertIsNone( + attach_media_parts([{"role": "system", "content": "hi"}], n_images=1) + ) + + def test_content_already_parts_is_a_no_op(self): + self.assertIsNone( + attach_media_parts( + [{"role": "user", "content": [{"type": "text", "text": "hi"}]}], + n_images=1, + ) + ) + + if __name__ == "__main__": unittest.main() diff --git a/backend/python/common/template/requirements-intel.txt b/backend/python/common/template/requirements-intel.txt index cfe9fd3fa..c17f22c30 100644 --- a/backend/python/common/template/requirements-intel.txt +++ b/backend/python/common/template/requirements-intel.txt @@ -1,4 +1,4 @@ --extra-index-url https://download.pytorch.org/whl/xpu -torch==2.13.0+xpu +torch==2.14.0+xpu oneccl_bind_pt==2.8.0+xpu optimum[openvino] \ No newline at end of file diff --git a/backend/python/common/template/requirements.txt b/backend/python/common/template/requirements.txt index 1795a9694..205cdbb75 100644 --- a/backend/python/common/template/requirements.txt +++ b/backend/python/common/template/requirements.txt @@ -1,3 +1,3 @@ -grpcio==1.82.1 +grpcio==1.83.1 protobuf grpcio-tools \ No newline at end of file diff --git a/backend/python/coqui/requirements.txt b/backend/python/coqui/requirements.txt index db72b4b98..305f95f12 100644 --- a/backend/python/coqui/requirements.txt +++ b/backend/python/coqui/requirements.txt @@ -1,4 +1,4 @@ -grpcio==1.83.0 +grpcio==1.83.1 protobuf certifi packaging==26.3 \ No newline at end of file diff --git a/backend/python/diffusers/backend.py b/backend/python/diffusers/backend.py index 539ce5444..f32187a25 100755 --- a/backend/python/diffusers/backend.py +++ b/backend/python/diffusers/backend.py @@ -800,12 +800,12 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): image = image.resize((1024, 576)) generator = torch.manual_seed(request.seed) - frames = self.pipe(image, guidance_scale=self.cfg_scale, decode_chunk_size=CHUNK_SIZE, generator=generator).frames[0] + frames = self.pipe(image=image, guidance_scale=self.cfg_scale, decode_chunk_size=CHUNK_SIZE, generator=generator).frames[0] export_to_video(frames, request.dst, fps=FPS) return backend_pb2.Result(message="Media generated successfully", success=True) if self.txt2vid: - video_frames = self.pipe(prompt, guidance_scale=self.cfg_scale, num_inference_steps=steps, num_frames=int(FRAMES)).frames + video_frames = self.pipe(prompt=prompt, guidance_scale=self.cfg_scale, num_inference_steps=steps, num_frames=int(FRAMES)).frames export_to_video(video_frames, request.dst) return backend_pb2.Result(message="Media generated successfully", success=True) @@ -868,7 +868,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): else: # pass the kwargs dictionary to the self.pipe method image = self.pipe( - prompt, + prompt=prompt, guidance_scale=self.cfg_scale, **kwargs ).images[0] diff --git a/backend/python/diffusers/test.py b/backend/python/diffusers/test.py index eff293ee6..2922f2a03 100644 --- a/backend/python/diffusers/test.py +++ b/backend/python/diffusers/test.py @@ -373,3 +373,55 @@ class TestGenerateImageOptionsKwargsMerge(unittest.TestCase): finally: os.unlink(src_file.name) os.unlink(dst_file.name) + + def test_text_to_image_prompt_is_passed_by_keyword(self): + """Test compatibility with pipelines that take image before prompt.""" + import os + import tempfile + + from PIL import Image + + from backend import BackendServicer + + class Flux2CompatiblePipeline: + """Model the FLUX.2 call signature: image is before prompt.""" + + def __call__(self, image=None, prompt=None, **kwargs): + if prompt is None: + raise ValueError("prompt was not passed by keyword") + self.prompt = prompt + self.kwargs = kwargs + return MagicMock(images=[Image.new("RGB", (4, 4))]) + + pipeline = Flux2CompatiblePipeline() + svc = BackendServicer.__new__(BackendServicer) + svc.pipe = pipeline + svc.cfg_scale = 7.5 + svc.controlnet = None + svc.img2vid = False + svc.txt2vid = False + svc.clip_skip = 0 + svc.PipelineType = "Flux2KleinPipeline" + svc.options = {} + + with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as dst_file: + dst_path = dst_file.name + + try: + request = MagicMock() + request.positive_prompt = "a red apple on a wooden table" + request.negative_prompt = "" + request.step = 4 + request.seed = 0 + request.width = 0 + request.height = 0 + request.src = "" + request.ref_images = [] + request.dst = dst_path + + svc.GenerateImage(request, context=None) + + self.assertEqual(pipeline.prompt, request.positive_prompt) + self.assertEqual(pipeline.kwargs["num_inference_steps"], 4) + finally: + os.unlink(dst_path) diff --git a/backend/python/longcat-video/README.md b/backend/python/longcat-video/README.md index 821de0130..e50a40f51 100644 --- a/backend/python/longcat-video/README.md +++ b/backend/python/longcat-video/README.md @@ -11,7 +11,7 @@ RPC. It supports: systems such as NVIDIA DGX Spark. Install the `longcat-video` or `longcat-video-avatar-1.5` recipe from the -LocalAI Model Gallery. See the [LongCat user guide](../../../docs/content/features/longcat-video.md) +LocalAI Model Gallery. LongCat video backend for Studio and API examples, hardware requirements, and manual configuration. The upstream source is pinned in `Makefile` and patched at build time. The diff --git a/backend/python/qwen-asr/backend.py b/backend/python/qwen-asr/backend.py index ea5e877f2..a559ff5e6 100644 --- a/backend/python/qwen-asr/backend.py +++ b/backend/python/qwen-asr/backend.py @@ -18,6 +18,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common')) sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common')) from grpc_auth import get_auth_interceptors from model_utils import resolve_model_reference +from device_utils import device_map_for, select_device @@ -95,13 +96,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): return backend_pb2.Reply(message=bytes("OK", 'utf-8')) def LoadModel(self, request, context): - if torch.cuda.is_available(): - device = "cuda" - else: - device = "cpu" - mps_available = hasattr(torch.backends, "mps") and torch.backends.mps.is_available() - if mps_available: - device = "mps" + device = select_device(torch) if not torch.cuda.is_available() and request.CUDA: return backend_pb2.Result(success=False, message="CUDA is not available") @@ -123,7 +118,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): model_path, local_only = resolve_model_reference( request, "Qwen/Qwen3-ASR-1.7B" ) - default_dtype = torch.bfloat16 if self.device == "cuda" else torch.float32 + default_dtype = torch.bfloat16 if self.device in ("cuda", "xpu") else torch.float32 load_dtype = default_dtype if "torch_dtype" in self.options: d = str(self.options["torch_dtype"]).lower() @@ -145,12 +140,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): if attn_implementation is not None and isinstance(attn_implementation, str): attn_implementation = attn_implementation.strip() or None - if self.device == "mps": - device_map = None - elif self.device == "cuda": - device_map = "cuda:0" - else: - device_map = "cpu" + device_map = device_map_for(self.device) load_kwargs = dict( dtype=load_dtype, @@ -423,4 +413,4 @@ if __name__ == "__main__": parser = argparse.ArgumentParser(description="Run the gRPC server.") parser.add_argument("--addr", default="localhost:50051", help="The address to bind the server to.") args = parser.parse_args() - serve(args.addr) \ No newline at end of file + serve(args.addr) diff --git a/backend/python/qwen-asr/device_utils.py b/backend/python/qwen-asr/device_utils.py new file mode 100644 index 000000000..0e1cb8005 --- /dev/null +++ b/backend/python/qwen-asr/device_utils.py @@ -0,0 +1,18 @@ +def select_device(torch_module): + mps = getattr(getattr(torch_module, "backends", None), "mps", None) + if mps is not None and mps.is_available(): + return "mps" + if torch_module.cuda.is_available(): + return "cuda" + xpu = getattr(torch_module, "xpu", None) + if xpu is not None and xpu.is_available(): + return "xpu" + return "cpu" + + +def device_map_for(device): + if device == "mps": + return None + if device in ("cuda", "xpu"): + return f"{device}:0" + return "cpu" diff --git a/backend/python/qwen-asr/device_utils_test.py b/backend/python/qwen-asr/device_utils_test.py new file mode 100644 index 000000000..fde07f32b --- /dev/null +++ b/backend/python/qwen-asr/device_utils_test.py @@ -0,0 +1,58 @@ +import unittest + +from device_utils import device_map_for, select_device + + +class Availability: + def __init__(self, available): + self._available = available + + def is_available(self): + return self._available + + +class TorchStub: + def __init__(self, *, cuda=False, mps=False, xpu=False): + self.cuda = Availability(cuda) + self.backends = type("Backends", (), {"mps": Availability(mps)})() + self.xpu = Availability(xpu) + + +class SelectDeviceTest(unittest.TestCase): + def test_preserves_cuda_selection(self): + torch_module = TorchStub(cuda=True) + + self.assertEqual(select_device(torch_module), "cuda") + + def test_preserves_mps_selection(self): + torch_module = TorchStub(mps=True) + + self.assertEqual(select_device(torch_module), "mps") + + def test_selects_xpu_when_intel_gpu_is_available(self): + torch_module = TorchStub(xpu=True) + + self.assertEqual(select_device(torch_module), "xpu") + + def test_falls_back_to_cpu(self): + torch_module = TorchStub() + + self.assertEqual(select_device(torch_module), "cpu") + + +class DeviceMapTest(unittest.TestCase): + def test_preserves_cuda_model_placement(self): + self.assertEqual(device_map_for("cuda"), "cuda:0") + + def test_preserves_mps_model_placement(self): + self.assertIsNone(device_map_for("mps")) + + def test_places_the_model_on_the_first_xpu(self): + self.assertEqual(device_map_for("xpu"), "xpu:0") + + def test_preserves_cpu_model_placement(self): + self.assertEqual(device_map_for("cpu"), "cpu") + + +if __name__ == "__main__": + unittest.main() diff --git a/backend/python/rerankers/requirements.txt b/backend/python/rerankers/requirements.txt index 49943543f..5f1e5f0ae 100644 --- a/backend/python/rerankers/requirements.txt +++ b/backend/python/rerankers/requirements.txt @@ -1,3 +1,3 @@ -grpcio==1.82.1 +grpcio==1.83.1 protobuf certifi \ No newline at end of file diff --git a/backend/python/sglang/backend.py b/backend/python/sglang/backend.py index c28b59a0e..9c6848dc9 100644 --- a/backend/python/sglang/backend.py +++ b/backend/python/sglang/backend.py @@ -40,6 +40,7 @@ import grpc sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common')) sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common')) +from python_utils import attach_media_parts from grpc_auth import get_auth_interceptors from model_utils import resolve_model_reference @@ -90,6 +91,14 @@ except Exception: _ONE_DAY_IN_SECONDS = 60 * 60 * 24 + +# proto3 has no field presence, so an explicit 0 is indistinguishable from +# "unset" and the zero-filter below would drop it. These two fields have a +# meaningful zero a caller can actually intend: temperature 0 is greedy +# decoding, and 0 is a valid seed. Silently substituting a default for either +# turns a reproducible request into a random one. +_EXPLICIT_ZERO_FIELDS = ("Temperature", "Seed") + MAX_WORKERS = int(os.environ.get('PYTHON_GRPC_MAX_WORKERS', '1')) @@ -323,7 +332,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): if not hasattr(request, proto_field): continue value = getattr(request, proto_field) - if value in (None, 0, 0.0, [], False, ""): + if proto_field not in _EXPLICIT_ZERO_FIELDS and value in (None, 0, 0.0, [], False, ""): continue # repeated fields come back as RepeatedScalarContainer — convert if hasattr(value, "__iter__") and not isinstance(value, (str, bytes)): @@ -363,8 +372,27 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): template_kwargs["tools"] = json.loads(request.Tools) except json.JSONDecodeError: pass - if request.Metadata.get("enable_thinking", "").lower() == "true": - template_kwargs["enable_thinking"] = True + _thinking = request.Metadata.get("enable_thinking", "").lower() + if _thinking in ("true", "false"): + template_kwargs["enable_thinking"] = (_thinking == "true") + + # sglang locates the attached images/videos by scanning the rendered + # prompt for the model's own media token, so the template has to be + # given content *parts* - string content renders a prompt with no + # placeholder and the media are dropped without a word (#11621). + media_dicts = attach_media_parts( + messages_dicts, len(request.Images), len(request.Videos) + ) + if media_dicts is not None: + try: + return self.tokenizer.apply_chat_template(media_dicts, **template_kwargs) + except Exception as e: + # A text-only template cannot iterate content parts; fall + # through to the text-only prompt instead of failing. + print( + f"chat template rejected multimodal content parts: {e!r}", + file=sys.stderr, + ) try: return self.tokenizer.apply_chat_template(messages_dicts, **template_kwargs) @@ -373,10 +401,67 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): messages_dicts, tokenize=False, add_generation_prompt=True, ) - def _make_parsers(self, request): + def _new_reasoning_parser(self, stream_reasoning: bool, prompt: str = "", + grammar_constrained: bool = False): + """Build a ReasoningParser for one request, or None. + + Reasoning templates come in two flavours. Some let the model emit the + opening tag, others put it into the *prompt* — Qwen3's template appends + ```` when thinking is on, so the completion starts straight in + the reasoning block and only the closing ```` ever shows up. + sglang's detector keys off the opening tag, so in that second case it + classifies the whole completion as normal content and + ``reasoning_content`` stays empty. + + sglang's own OpenAI server covers this with + ``template_manager.force_reasoning``; this backend has no template + manager, so it derives the same signal from the rendered prompt. + ``force_reasoning`` is only passed when we mean True, leaving detector + defaults (e.g. DeepSeek-R1's built-in True) untouched. + + ``grammar_constrained`` suppresses the prefill heuristic. A structured + decoding constraint applies from the first token, so the model cannot + emit the closing tag even though the template opened the block: the + whole completion is schema output and belongs in ``content``. Forcing + there files the answer as reasoning and leaves content empty. sglang's + own server keeps the two apart for the same reason — its grammar + backend owns the reasoning prefix when a reasoning parser is set. + """ + if grammar_constrained: + prompt = "" + + if not (HAS_REASONING_PARSERS and self.reasoning_parser_name): + return None + + kwargs = { + "model_type": self.reasoning_parser_name, + "stream_reasoning": stream_reasoning, + } + try: + parser = ReasoningParser(**kwargs) + except Exception as e: + print(f"ReasoningParser init failed: {e!r}", file=sys.stderr) + return None + + start = getattr(getattr(parser, "detector", None), "think_start_token", None) + if start and prompt and prompt.rstrip().endswith(start): + try: + parser = ReasoningParser(force_reasoning=True, **kwargs) + except TypeError: + # sglang without the force_reasoning kwarg: keep the default + # parser rather than failing the request. + pass + except Exception as e: + print( + f"ReasoningParser(force_reasoning=True) failed: {e!r}", + file=sys.stderr, + ) + + return parser + + def _make_parsers(self, request, prompt: str = ""): """Construct fresh per-request parser instances (stateful).""" tool_parser = None - reasoning_parser = None if HAS_TOOL_PARSERS and self.tool_parser_name and request.Tools: try: @@ -388,14 +473,9 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): except Exception as e: print(f"FunctionCallParser init failed: {e!r}", file=sys.stderr) - if HAS_REASONING_PARSERS and self.reasoning_parser_name: - try: - reasoning_parser = ReasoningParser( - model_type=self.reasoning_parser_name, - stream_reasoning=True, - ) - except Exception as e: - print(f"ReasoningParser init failed: {e!r}", file=sys.stderr) + reasoning_parser = self._new_reasoning_parser( + True, prompt, bool(getattr(request, "Grammar", "")), + ) return tool_parser, reasoning_parser @@ -403,7 +483,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): sampling_params = self._build_sampling_params(request) prompt = self._build_prompt(request) - tool_parser, reasoning_parser = self._make_parsers(request) + tool_parser, reasoning_parser = self._make_parsers(request, prompt) image_data = list(request.Images) if request.Images else None video_data = list(request.Videos) if request.Videos else None @@ -499,15 +579,9 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): final_tool_calls: List[backend_pb2.ToolCallDelta] = [] if not streaming: - final_reasoning_parser = None - if HAS_REASONING_PARSERS and self.reasoning_parser_name: - try: - final_reasoning_parser = ReasoningParser( - model_type=self.reasoning_parser_name, - stream_reasoning=False, - ) - except Exception: - final_reasoning_parser = None + final_reasoning_parser = self._new_reasoning_parser( + False, prompt, bool(getattr(request, "Grammar", "")), + ) if final_reasoning_parser is not None: try: diff --git a/backend/python/sglang/test.py b/backend/python/sglang/test.py index 92688f444..4cd559b09 100644 --- a/backend/python/sglang/test.py +++ b/backend/python/sglang/test.py @@ -96,6 +96,126 @@ class TestSglangHelpers(unittest.TestCase): servicer._apply_engine_args({}, "[1,2,3]") self.assertIn("must be a JSON object", str(ctx.exception)) + def test_build_prompt_forwards_enable_thinking(self): + from types import SimpleNamespace + + class Tok: + def __init__(self): + self.kwargs = None + + def apply_chat_template(self, messages, **kwargs): + self.kwargs = kwargs + return "PROMPT" + + def kwargs_for(metadata): + servicer = self._servicer() + tok = Tok() + servicer.tokenizer = tok + msg = SimpleNamespace( + role="user", content="hi", name="", + tool_call_id="", reasoning_content="", tool_calls="", + ) + req = SimpleNamespace( + Prompt="", UseTokenizerTemplate=True, + Messages=[msg], Tools="", Metadata=metadata, + ) + self.assertEqual(servicer._build_prompt(req), "PROMPT") + return tok.kwargs + + self.assertIs(kwargs_for({"enable_thinking": "true"})["enable_thinking"], True) + # "false" used to be dropped, so Qwen3 kept thinking on + self.assertIs(kwargs_for({"enable_thinking": "false"})["enable_thinking"], False) + self.assertNotIn("enable_thinking", kwargs_for({})) + self.assertIs(kwargs_for({"enable_thinking": "FALSE"})["enable_thinking"], False) + + def test_reasoning_parser_forced_when_template_prefills_think_tag(self): + """Qwen3's template puts ```` in the prompt, so the completion + never contains it. Without force_reasoning the detector treats the whole + completion as normal text and reasoning_content stays empty.""" + servicer = self._servicer() + servicer.reasoning_parser_name = "qwen3" + + # What the model actually emits when the prompt ends in "". + completion = "adding two and two4" + + forced = servicer._new_reasoning_parser(False, prompt="user: hi\n\n") + reasoning, content = forced.parse_non_stream(completion) + self.assertEqual(reasoning, "adding two and two") + self.assertEqual(content, "4") + + # No prefilled tag in the prompt: detector default, unchanged behaviour. + unforced = servicer._new_reasoning_parser(False, prompt="user: hi\n") + reasoning, content = unforced.parse_non_stream(completion) + self.assertFalse(reasoning) + self.assertEqual(content, completion) + + def test_reasoning_parser_not_forced_when_thinking_is_off(self): + """Thinking off means no ```` in the prompt either, so the answer + must not be swallowed into reasoning_content.""" + servicer = self._servicer() + servicer.reasoning_parser_name = "qwen3" + + parser = servicer._new_reasoning_parser(False, prompt="user: primes?\n") + reasoning, content = parser.parse_non_stream("2,3,5,7,11") + self.assertFalse(reasoning) + self.assertEqual(content, "2,3,5,7,11") + + def test_grammar_constrained_output_is_not_forced_into_reasoning(self): + """Structured decoding applies from the first token, so the model cannot + emit the closing tag even though the template opened the block. The whole + completion is schema output and must stay in content.""" + servicer = self._servicer() + servicer.reasoning_parser_name = "qwen3" + + schema_out = '{"findings": [{"line": 42, "issue": "off-by-one"}]}' + parser = servicer._new_reasoning_parser( + False, prompt="audit this\n\n", grammar_constrained=True, + ) + reasoning, content = parser.parse_non_stream(schema_out) + self.assertFalse(reasoning) + self.assertEqual(content, schema_out) + + def test_reasoning_parser_absent_without_configured_parser(self): + servicer = self._servicer() + servicer.reasoning_parser_name = None + self.assertIsNone(servicer._new_reasoning_parser(False, prompt="")) + + def test_explicit_zero_temperature_and_seed_are_preserved(self): + """Temperature=0 is greedy decoding and 0 is a valid seed — neither is + an unset value. A dropped seed turns a reproducible request random.""" + from types import SimpleNamespace + + servicer = self._servicer() + import sys as _sys + _SEED_KEY_FOR_TEST = _sys.modules["backend"]._SEED_KEY + request = SimpleNamespace( + Temperature=0, + N=0, + PresencePenalty=0, + FrequencyPenalty=0, + RepetitionPenalty=0, + TopP=0, + TopK=0, + MinP=0, + Seed=0, + StopPrompts=[], + StopTokenIds=[], + IgnoreEOS=False, + Tokens=0, + MinTokens=0, + SkipSpecialTokens=False, + Grammar="", + ) + + params = servicer._build_sampling_params(request) + self.assertEqual(params["temperature"], 0) + self.assertEqual(params[_SEED_KEY_FOR_TEST], 0) + # Other protobuf-default scalar fields must remain filtered. top_k=0 in + # particular is not a value sglang accepts (-1 disables it), so it must + # keep falling through to the engine default. + self.assertNotIn("top_p", params) + self.assertNotIn("top_k", params) + if __name__ == "__main__": unittest.main() diff --git a/backend/python/vllm/backend.py b/backend/python/vllm/backend.py index 8fd3c2dc1..de9af3798 100644 --- a/backend/python/vllm/backend.py +++ b/backend/python/vllm/backend.py @@ -20,6 +20,7 @@ import backend_pb2_grpc import grpc sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common')) sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common')) +from python_utils import attach_media_parts from grpc_auth import get_auth_interceptors from model_utils import resolve_model_reference from vllm_utils import apply_options_to_engine_args, normalize_option_key @@ -60,6 +61,12 @@ except ImportError: _ONE_DAY_IN_SECONDS = 60 * 60 * 24 +# proto3 has no field presence, so an explicit 0 is indistinguishable from +# "unset". These two fields have a meaningful zero a caller can intend: +# temperature 0 is greedy decoding, and 0 is a valid seed. +_EXPLICIT_ZERO_FIELDS = ("Temperature", "Seed") + + # If MAX_WORKERS are specified in the environment use it, otherwise default to 1 MAX_WORKERS = int(os.environ.get('PYTHON_GRPC_MAX_WORKERS', '1')) @@ -523,9 +530,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): context.set_details(str(e)) return backend_pb2.ScoreResponse() - async def _predict(self, request, context, streaming=False): - # Build the sampling parameters - # NOTE: this must stay in sync with the vllm backend + def _build_sampling_params(self, request): request_to_sampling_params = { "N": "n", "PresencePenalty": "presence_penalty", @@ -555,9 +560,84 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): for request_field, param_field in request_to_sampling_params.items(): if hasattr(request, request_field): value = getattr(request, request_field) - if value not in (None, 0, [], False): + # See _EXPLICIT_ZERO_FIELDS: temperature 0 is greedy decoding + # and 0 is a valid seed, so neither may be filtered out. + if request_field in _EXPLICIT_ZERO_FIELDS or value not in (None, 0, [], False): setattr(sampling_params, param_field, value) + return sampling_params + + def _new_reasoning_parser(self, chat_template_kwargs): + """Build the reasoning parser, telling it whether thinking is on. + + vLLM's newer parser engines decide their *initial state* from + ``chat_template_kwargs``: ``Qwen3Parser`` reads + ``chat_template_kwargs["enable_thinking"]`` and defaults to ``True``, + starting in the REASONING state. Constructed without it, a completion + produced with thinking disabled is classified as reasoning end to end, + and the answer is reported in both ``reasoning_content`` and + ``content``. + + vLLM's own OpenAI server forwards the request's chat template kwargs + here; this backend renders the template itself, so it forwards the + same dict. Older parsers do not accept the argument — fall back to the + plain constructor for those. + """ + try: + return self.reasoning_parser_cls( + self.tokenizer, chat_template_kwargs=chat_template_kwargs or {}, + ) + except TypeError: + return self.reasoning_parser_cls(self.tokenizer) + + @staticmethod + def _split_reasoning(rp, generated_text, prompt, reasoning, content): + """Decide what the reasoning parser's output actually means. + + Covers the *older* parser shape, which has no initial state to set: + ``BaseThinkingReasoningParser.extract_reasoning`` documents its own + fallback — "For models that may not generate start token, assume the + reasoning content is always at the start." When no end token is + present it returns *everything* as reasoning and ``None`` as content, + which is right for a truncated reasoning run and wrong for a + completion that never contained reasoning at all. + + Taking ``None`` content to mean "keep the raw text" then duplicates + the answer into both fields. + + The prompt says which case it is. A template with thinking on leaves + the reasoning block open (the prompt ends with the start token); with + thinking off it closes the block in the prompt, so the completion is + plain content. Parsers that expose no token pair (the engine-based + adapters, which take the ``chat_template_kwargs`` route above) keep + the parser's verdict unchanged. + """ + start = getattr(rp, "start_token", None) + end = getattr(rp, "end_token", None) + + if end and end in generated_text: + # The parser split on the end token. Empty content here means the + # model stopped right after it, not that parsing failed. + return reasoning or "", content or "" + + if not start: + # Unknown token layout — keep the previous behaviour rather than + # guess. + return reasoning or "", content if content is not None else generated_text + + if not (start in generated_text or (prompt or "").rstrip().endswith(start)): + # No end token and the block was never open: the "reasoning starts + # at the beginning" fallback does not apply to this completion. + return "", generated_text + + # Block was open and the end token never arrived — reasoning ran out of + # budget. It is all reasoning, and there is no answer to report. + return reasoning or "", content or "" + + async def _predict(self, request, context, streaming=False): + # Build the sampling parameters + sampling_params = self._build_sampling_params(request) + # Structured-output decoding: use Grammar field to pass JSON schema or BNF if HAS_GUIDED_DECODING and request.Grammar: try: @@ -568,6 +648,9 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): # Extract image paths and process images prompt = request.Prompt + # Kept in scope: the reasoning parser needs to know which chat + # template kwargs produced this prompt. + template_kwargs = {} image_paths = request.Images image_data = [self.load_image(img_path) for img_path in image_paths] @@ -578,7 +661,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): # If tokenizer template is enabled and messages are provided instead of prompt, apply the tokenizer template if not request.Prompt and request.UseTokenizerTemplate and request.Messages: messages_dicts = self._messages_to_dicts(request.Messages) - template_kwargs = {"tokenize": False, "add_generation_prompt": True} + template_kwargs.update({"tokenize": False, "add_generation_prompt": True}) # Pass tools for tool calling if request.Tools: @@ -587,17 +670,37 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): except json.JSONDecodeError: pass - # Enable thinking mode if requested - if request.Metadata.get("enable_thinking", "").lower() == "true": - template_kwargs["enable_thinking"] = True + _thinking = request.Metadata.get("enable_thinking", "").lower() + if _thinking in ("true", "false"): + template_kwargs["enable_thinking"] = (_thinking == "true") - try: - prompt = self.tokenizer.apply_chat_template(messages_dicts, **template_kwargs) - except TypeError: - # Some tokenizers don't support tools/enable_thinking kwargs — retry without them - prompt = self.tokenizer.apply_chat_template( - messages_dicts, tokenize=False, add_generation_prompt=True - ) + # vLLM substitutes multi_modal_data into the model's own media + # token, so the template has to be given content *parts* - string + # content renders a prompt with no placeholder and the media are + # dropped without a word (#11621). + prompt = None + media_dicts = attach_media_parts( + messages_dicts, len(image_data), len(video_data) + ) + if media_dicts is not None: + try: + prompt = self.tokenizer.apply_chat_template(media_dicts, **template_kwargs) + except Exception as e: + # A text-only template cannot iterate content parts; fall + # through to the text-only prompt instead of failing. + print( + f"chat template rejected multimodal content parts: {e!r}", + file=sys.stderr, + ) + + if prompt is None: + try: + prompt = self.tokenizer.apply_chat_template(messages_dicts, **template_kwargs) + except TypeError: + # Some tokenizers don't support tools/enable_thinking kwargs — retry without them + prompt = self.tokenizer.apply_chat_template( + messages_dicts, tokenize=False, add_generation_prompt=True + ) # Generate text using the LLM engine request_id = random_uuid() @@ -753,10 +856,11 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): if self.reasoning_parser_cls: try: - rp = self.reasoning_parser_cls(self.tokenizer) + rp = self._new_reasoning_parser(template_kwargs) r, c = rp.extract_reasoning(generated_text, request=None) - reasoning_content = r or "" - content = c if c is not None else generated_text + reasoning_content, content = self._split_reasoning( + rp, generated_text, prompt, r, c, + ) except Exception as e: print(f"Reasoning parser error: {e}", file=sys.stderr) diff --git a/backend/python/vllm/install.sh b/backend/python/vllm/install.sh index 68d5ba257..a4124977c 100755 --- a/backend/python/vllm/install.sh +++ b/backend/python/vllm/install.sh @@ -119,14 +119,18 @@ if [ "$(uname -s)" = "Darwin" ]; then # can rewrite it. Darwin therefore follows vllm-metal and can lag the Linux # vllm pin (requirements-cublas13-after.txt, bumped independently against # vllm/vllm) until vllm-metal supports a newer vLLM. - VLLM_METAL_VERSION="v0.3.0.dev20260818075955" + VLLM_METAL_VERSION="v0.28.0" # The coupled vLLM source version is whatever this vllm-metal release builds - # against. Derive it from - # the PINNED tag rather than hardcoding a second value that could drift. The - # tag is immutable, so this stays reproducible across rebuilds. - VLLM_VERSION=$(curl -fsSL "https://raw.githubusercontent.com/vllm-project/vllm-metal/${VLLM_METAL_VERSION}/install.sh" \ - | "$backend_dir/../../../scripts/lib/extract-vllm-metal-version.sh") + # against. Derive it from the PINNED tag rather than hardcoding a second value + # that could drift. The tag is immutable, so this stays reproducible across + # rebuilds. Since vllm-metal 0.28 the coupling is declared in + # .github/vllm-release-tag.commit; older releases pinned it inline in their + # own install.sh, so fall back to that. The extractor reads both forms. + _vllm_metal_raw="https://raw.githubusercontent.com/vllm-project/vllm-metal/${VLLM_METAL_VERSION}" + VLLM_VERSION=$( { curl -fsSL "${_vllm_metal_raw}/.github/vllm-release-tag.commit" \ + || curl -fsSL "${_vllm_metal_raw}/install.sh"; } \ + | "$backend_dir/../../../scripts/lib/extract-vllm-metal-version.sh" || true) if [ -z "${VLLM_VERSION}" ]; then echo "ERROR: could not derive the vLLM version from vllm-metal ${VLLM_METAL_VERSION}" >&2 exit 1 @@ -153,10 +157,18 @@ if [ "$(uname -s)" = "Darwin" ]; then # 2) Install the prebuilt vllm-metal wheel for the PINNED release. It pulls # mlx / mlx-metal as deps and registers the `metal` platform plugin that # backend.py resolves to at engine-init time. Build the release-asset URL - # deterministically (tag + the cp312/arm64 wheel name) rather than querying - # api.github.com, whose unauthenticated rate limit (60/hr per IP) 403s on - # shared CI runners. The wheel version is the tag without its leading 'v'. - _metal_wheel="vllm_metal-${VLLM_METAL_VERSION#v}-cp312-cp312-macosx_11_0_arm64.whl" + # from the release's OWN asset listing rather than composing it from a + # hardcoded platform tag: upstream raised its macOS deployment target + # (macosx_11_0 -> macosx_15_0) and every composed URL started to 404. + # expanded_assets is the plain release page, not api.github.com, whose + # unauthenticated rate limit (60/hr per IP) 403s on shared CI runners. + # The wheel version is the tag without its leading 'v'. + _metal_wheel=$(curl -fsSL "https://github.com/vllm-project/vllm-metal/releases/expanded_assets/${VLLM_METAL_VERSION}" \ + | grep -oE "vllm_metal-${VLLM_METAL_VERSION#v}-cp312-cp312-[A-Za-z0-9_]+\.whl" | head -1 || true) + if [ -z "${_metal_wheel}" ]; then + echo "ERROR: no cp312 wheel asset on vllm-metal release ${VLLM_METAL_VERSION}" >&2 + exit 1 + fi _metal_wheel_url="https://github.com/vllm-project/vllm-metal/releases/download/${VLLM_METAL_VERSION}/${_metal_wheel}" echo "Installing vllm-metal wheel: ${_metal_wheel_url}" uv pip install "${_metal_wheel_url}" diff --git a/backend/python/vllm/requirements-cublas13-after.txt b/backend/python/vllm/requirements-cublas13-after.txt index 34f4f50ff..b519b73fe 100644 --- a/backend/python/vllm/requirements-cublas13-after.txt +++ b/backend/python/vllm/requirements-cublas13-after.txt @@ -3,8 +3,8 @@ # on a cu130 host. Pull the cu130-flavoured wheel from vLLM's per-tag index # instead — the cublas13 case in install.sh adds --index-strategy=unsafe-best-match # so uv consults this index alongside PyPI. ---extra-index-url https://wheels.vllm.ai/0.27.1/cu130 +--extra-index-url https://wheels.vllm.ai/0.28.0/cu130 # VERSION COUPLING: darwin/Apple-Silicon builds use vllm-metal (see install.sh), # which pins this exact vLLM version. Bumping vllm here means coordinating with a # vllm-metal release that supports the new version, or macOS/Metal builds break. -vllm==0.27.1 +vllm==0.28.0 diff --git a/backend/python/vllm/requirements-l4t13-after.txt b/backend/python/vllm/requirements-l4t13-after.txt index 14fb46634..68860db2b 100644 --- a/backend/python/vllm/requirements-l4t13-after.txt +++ b/backend/python/vllm/requirements-l4t13-after.txt @@ -9,4 +9,4 @@ # memory architecture crash deterministically with an empty "Engine core init # failed" set (mudler/LocalAI#10722). Leaving this unpinned let the L4T image # drift onto whatever wheel was latest at build time. -vllm==0.26.0 +vllm==0.28.0 diff --git a/backend/python/vllm/requirements.txt b/backend/python/vllm/requirements.txt index 889f21d0d..ba7147632 100644 --- a/backend/python/vllm/requirements.txt +++ b/backend/python/vllm/requirements.txt @@ -1,4 +1,4 @@ -grpcio==1.83.0 +grpcio==1.83.1 protobuf certifi setuptools diff --git a/backend/python/vllm/test.py b/backend/python/vllm/test.py index d00595f01..27a846f09 100644 --- a/backend/python/vllm/test.py +++ b/backend/python/vllm/test.py @@ -121,6 +121,21 @@ class TestBackendServicer(unittest.TestCase): finally: self.tearDown() + def test_explicit_zero_temperature_and_seed_are_preserved(self): + """Temperature=0 is greedy decoding and 0 is a valid seed — neither is + an unset value. A dropped seed turns a reproducible request random.""" + import sys, os + sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + from backend import BackendServicer + + servicer = BackendServicer() + request = backend_pb2.PredictOptions(Prompt="hello", Temperature=0, Seed=0) + sampling_params = servicer._build_sampling_params(request) + self.assertEqual(sampling_params.temperature, 0) + self.assertEqual(sampling_params.seed, 0) + # Other protobuf-default scalar fields must remain filtered. + self.assertEqual(sampling_params.top_p, 0.9) + def test_messages_to_dicts(self): """ @@ -536,3 +551,116 @@ class TestStreamingToolParser(unittest.TestCase): intermediate, ["Hello ", "world", "!"], f"plain streaming changed; got {intermediate!r}", ) + + +class TestReasoningSplit(unittest.TestCase): + """Server-less tests for BackendServicer._split_reasoning. + + vLLM's BaseThinkingReasoningParser returns the whole completion as + reasoning and None as content whenever the end token is missing. Taken + literally that duplicates a thinking-disabled answer into both fields. + """ + + class _Parser: + start_token = "" + end_token = "" + + def _split(self, generated, prompt, reasoning, content): + import sys, os + sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + from backend import BackendServicer + return BackendServicer._split_reasoning( + self._Parser(), generated, prompt, reasoning, content, + ) + + def test_thinking_off_is_not_duplicated_into_reasoning(self): + """No tags anywhere: the answer is content, and only content.""" + r, c = self._split( + "391", "user: 17*23?\n\n\n\n\n", + reasoning="391", content=None, + ) + self.assertEqual(r, "") + self.assertEqual(c, "391") + + def test_prefilled_start_tag_keeps_truncated_reasoning(self): + """Prompt left the block open and the end token never arrived + (budget exhausted): that really is all reasoning.""" + r, c = self._split( + "thinking and thinking", "user: hi\n\n", + reasoning="thinking and thinking", content=None, + ) + self.assertEqual(r, "thinking and thinking") + self.assertEqual(c, "") + + def test_end_token_present_keeps_parser_split(self): + r, c = self._split( + "adding two and two4", "user: hi\n\n", + reasoning="adding two and two", content="4", + ) + self.assertEqual(r, "adding two and two") + self.assertEqual(c, "4") + + def test_stop_right_after_end_token_yields_empty_content(self): + """Content must not fall back to the raw text — that would put the + reasoning into the answer.""" + r, c = self._split( + "reasoned", "user: hi\n\n", + reasoning="reasoned", content=None, + ) + self.assertEqual(r, "reasoned") + self.assertEqual(c, "") + + def test_unknown_token_layout_keeps_previous_behaviour(self): + class _Bare: + pass + import sys, os + sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + from backend import BackendServicer + r, c = BackendServicer._split_reasoning( + _Bare(), "raw", "prompt", "raw", None, + ) + self.assertEqual(r, "raw") + self.assertEqual(c, "raw") + + +class TestReasoningParserConstruction(unittest.TestCase): + """The parser must learn whether thinking was on for this request. + + vLLM's engine-based parsers (Qwen3Parser and friends) read + chat_template_kwargs["enable_thinking"] and default to True, so a parser + built without it treats a thinking-disabled completion as pure reasoning. + """ + + def _servicer(self): + import sys, os + sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + from backend import BackendServicer + s = BackendServicer() + s.tokenizer = object() + return s + + def test_chat_template_kwargs_are_forwarded(self): + seen = {} + + class _Parser: + def __init__(self, tokenizer, **kwargs): + seen.update(kwargs) + + s = self._servicer() + s.reasoning_parser_cls = _Parser + s._new_reasoning_parser({"enable_thinking": False}) + self.assertEqual( + seen.get("chat_template_kwargs"), {"enable_thinking": False}, + ) + + def test_parser_without_the_kwarg_still_builds(self): + """Older parsers take only the tokenizer — must not break them.""" + class _Old: + def __init__(self, tokenizer): + self.tokenizer = tokenizer + + s = self._servicer() + s.reasoning_parser_cls = _Old + self.assertIsInstance( + s._new_reasoning_parser({"enable_thinking": False}), _Old, + ) diff --git a/backend/python/whisperx/backend.py b/backend/python/whisperx/backend.py index 7318e10b7..dc6202286 100644 --- a/backend/python/whisperx/backend.py +++ b/backend/python/whisperx/backend.py @@ -16,6 +16,7 @@ import grpc sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common')) sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common')) from grpc_auth import get_auth_interceptors +from transcript_utils import require_diarization_token, seconds_to_nanoseconds @@ -81,6 +82,11 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): import whisperx from whisperx.diarize import DiarizationPipeline + try: + require_diarization_token(request.diarize, self.hf_token) + except ValueError as err: + context.abort(grpc.StatusCode.FAILED_PRECONDITION, str(err)) + resultSegments = [] text = "" try: @@ -117,8 +123,8 @@ class BackendServicer(backend_pb2_grpc.BackendServicer): # Build result segments for idx, seg in enumerate(transcript["segments"]): seg_text = seg.get("text", "") - start = int(seg.get("start", 0)) - end = int(seg.get("end", 0)) + start = seconds_to_nanoseconds(seg.get("start", 0)) + end = seconds_to_nanoseconds(seg.get("end", 0)) speaker = seg.get("speaker", "") resultSegments.append(backend_pb2.TranscriptSegment( diff --git a/backend/python/whisperx/test_transcript_utils.py b/backend/python/whisperx/test_transcript_utils.py new file mode 100644 index 000000000..debe2ea6e --- /dev/null +++ b/backend/python/whisperx/test_transcript_utils.py @@ -0,0 +1,25 @@ +import unittest + +import transcript_utils + + +class TestTranscriptUtils(unittest.TestCase): + def test_diarization_requires_hugging_face_token(self): + with self.assertRaisesRegex( + ValueError, + "HF_TOKEN is required for WhisperX diarization", + ): + transcript_utils.require_diarization_token(True, None) + + def test_diarization_does_not_require_token_when_disabled(self): + transcript_utils.require_diarization_token(False, None) + + def test_seconds_are_serialized_as_nanoseconds(self): + self.assertEqual( + transcript_utils.seconds_to_nanoseconds(3.25), + 3_250_000_000, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/backend/python/whisperx/transcript_utils.py b/backend/python/whisperx/transcript_utils.py new file mode 100644 index 000000000..a8ac57510 --- /dev/null +++ b/backend/python/whisperx/transcript_utils.py @@ -0,0 +1,12 @@ +"""Helpers for WhisperX transcript responses.""" + + +def require_diarization_token(diarize, token): + """Reject diarization when WhisperX cannot load its gated pipeline.""" + if diarize and not token: + raise ValueError("HF_TOKEN is required for WhisperX diarization") + + +def seconds_to_nanoseconds(seconds): + """Convert WhisperX timestamps to the duration unit used by LocalAI.""" + return int(seconds * 1_000_000_000) diff --git a/cmd/launcher/internal/launcher.go b/cmd/launcher/internal/launcher.go index aa5f7d6f4..63478d4f4 100644 --- a/cmd/launcher/internal/launcher.go +++ b/cmd/launcher/internal/launcher.go @@ -24,10 +24,17 @@ import ( // Config represents the launcher configuration type Config struct { - ModelsPath string `json:"models_path"` - BackendsPath string `json:"backends_path"` - Address string `json:"address"` - AutoStart bool `json:"auto_start"` + ModelsPath string `json:"models_path"` + BackendsPath string `json:"backends_path"` + Address string `json:"address"` + // AutoStart controls whether the launcher starts the LocalAI server as + // soon as the launcher itself opens (and right after a fresh install). + // Unset means enabled: launching the app must yield a serving endpoint, + // which is what the quickstart docs promise. The JSON key is deliberately + // not the legacy "auto_start": that field was never honored nor exposed + // in any UI, so every existing launcher.json carries an unintentional + // false that would keep auto-start permanently off (#11673). + AutoStart *bool `json:"auto_start_server"` StartOnBoot bool `json:"start_on_boot"` LogLevel string `json:"log_level"` EnvironmentVars map[string]string `json:"environment_vars"` @@ -122,9 +129,6 @@ func (l *Launcher) Initialize() error { log.Printf("Warning: failed to cleanup partial downloads: %v", err) } - if l.config.StartOnBoot { - l.StartLocalAI() - } // Set default paths if not configured (only if not already loaded from config) if l.config.ModelsPath == "" { homeDir, _ := os.UserHomeDir() @@ -156,6 +160,12 @@ func (l *Launcher) Initialize() error { log.Printf("Setting default ShowWelcome: true") } + if l.config.AutoStart == nil { + enabled := true + l.config.AutoStart = &enabled + log.Printf("Setting default AutoStart: true") + } + // Create directories os.MkdirAll(l.config.ModelsPath, 0755) os.MkdirAll(l.config.BackendsPath, 0755) @@ -177,6 +187,11 @@ func (l *Launcher) Initialize() error { l.showDownloadLocalAIDialog() } }) + } else if l.ShouldAutoStartServer() { + // The launcher is a tray-only app: without this the user launches it, + // sees no window and no server, and concludes it does nothing (#11673). + log.Printf("Auto-starting LocalAI server") + l.autoStartServer() } // Check for updates periodically @@ -185,6 +200,35 @@ func (l *Launcher) Initialize() error { return nil } +// ShouldAutoStartServer reports whether the launcher should start the server +// without user interaction: at launcher startup and right after a fresh +// install. Defaults to enabled; StartOnBoot forces a start even when +// auto-start was explicitly disabled, preserving its historical behavior. +func (l *Launcher) ShouldAutoStartServer() bool { + if l.config == nil { + return false + } + if l.config.StartOnBoot { + return true + } + return l.config.AutoStart == nil || *l.config.AutoStart +} + +// autoStartServer starts LocalAI in the background and surfaces failures +// through the systray error dialog: during an auto-start there is no visible +// window for a regular error dialog to attach to. +func (l *Launcher) autoStartServer() { + go func() { + if err := l.StartLocalAI(); err != nil { + log.Printf("Failed to auto-start LocalAI: %v", err) + l.updateStatus(fmt.Sprintf("Failed to start LocalAI: %v", err)) + if l.systray != nil { + l.systray.showStartupErrorDialog(err) + } + } + }() +} + // StartLocalAI starts the LocalAI server func (l *Launcher) StartLocalAI() error { if l.isRunning { @@ -644,14 +688,22 @@ func (l *Launcher) showDownloadError(title, message string) { // after a fresh install (no LocalAI binary present yet). func (l *Launcher) showDownloadProgress(version, title string) { l.showDownloadProgressWindow(version, title, func(win fyne.Window) { - dialog.ShowConfirm("Installation Complete", - "LocalAI has been downloaded and installed successfully. You can now start LocalAI from the launcher.", + message := "LocalAI has been downloaded and installed successfully. You can now start LocalAI from the launcher." + if l.ShouldAutoStartServer() { + message = "LocalAI has been downloaded and installed successfully. It will start now: manage it and open the WebUI from the system tray icon." + } + dialog.ShowConfirm("Installation Complete", message, func(bool) { win.Close() l.updateStatus("LocalAI installed successfully") if l.systray != nil { l.systray.recreateMenu() } + // A fresh install should end with a running server, not with + // the user hunting for a start button in the tray (#11673). + if l.ShouldAutoStartServer() && !l.isRunning { + l.autoStartServer() + } }, win) }) } diff --git a/cmd/launcher/internal/launcher_test.go b/cmd/launcher/internal/launcher_test.go index 4786193ff..a73dcaaf5 100644 --- a/cmd/launcher/internal/launcher_test.go +++ b/cmd/launcher/internal/launcher_test.go @@ -1,6 +1,7 @@ package launcher_test import ( + "encoding/json" "os" "path/filepath" "strings" @@ -55,7 +56,8 @@ var _ = Describe("Launcher", func() { Expect(err).ToNot(HaveOccurred()) config := launcherInstance.GetConfig() - Expect(config.ShowWelcome).To(BeTrue()) + Expect(config.ShowWelcome).ToNot(BeNil()) + Expect(*config.ShowWelcome).To(BeTrue()) Expect(config.Address).To(Equal("127.0.0.1:8080")) Expect(config.LogLevel).To(Equal("info")) }) @@ -177,13 +179,53 @@ var _ = Describe("Launcher", func() { assertFlagValue("--generated-content-path", filepath.Join(dataPath, "generated")) assertFlagValue("--upload-path", filepath.Join(dataPath, "uploads")) - // The bug was the server resolving these to shared /tmp paths. + // The bug was the server resolving these to its shared /tmp + // defaults. Only reject those specific paths: on Linux the test's + // own temp directory legitimately lives under /tmp. for _, a := range args { - Expect(a).ToNot(HavePrefix("/tmp/"), "run args must not reference shared /tmp paths, got %s", a) + Expect(a).ToNot(HavePrefix("/tmp/generated"), "run args must not reference the shared /tmp generated-content default, got %s", a) + Expect(a).ToNot(HavePrefix("/tmp/upload"), "run args must not reference the shared /tmp upload default, got %s", a) } }) }) + // Regression for "Mac dmg launcher launches nothing" (issue #11673): the + // launcher created empty log files and served nothing because nothing ever + // started the server unless the unrelated "start on system boot" option was + // enabled. Launching the app must yield a serving endpoint by default. + Describe("ShouldAutoStartServer", func() { + It("should auto-start by default when nothing is configured", func() { + Expect(launcherInstance.ShouldAutoStartServer()).To(BeTrue()) + }) + + It("should respect an explicit opt-out", func() { + config := launcherInstance.GetConfig() + err := json.Unmarshal([]byte(`{"auto_start_server": false}`), config) + Expect(err).ToNot(HaveOccurred()) + + Expect(launcherInstance.ShouldAutoStartServer()).To(BeFalse()) + }) + + It("should still auto-start when StartOnBoot is set even if auto-start is off", func() { + config := launcherInstance.GetConfig() + err := json.Unmarshal([]byte(`{"auto_start_server": false, "start_on_boot": true}`), config) + Expect(err).ToNot(HaveOccurred()) + + Expect(launcherInstance.ShouldAutoStartServer()).To(BeTrue()) + }) + + It("should ignore the legacy auto_start key older launchers persisted as false", func() { + // Old launchers marshaled the never-honored AutoStart field as + // "auto_start": false into every launcher.json. That stale value + // carries no user intent and must not disable auto-start. + config := launcherInstance.GetConfig() + err := json.Unmarshal([]byte(`{"auto_start": false}`), config) + Expect(err).ToNot(HaveOccurred()) + + Expect(launcherInstance.ShouldAutoStartServer()).To(BeTrue()) + }) + }) + Describe("Logs", func() { It("should return empty logs initially", func() { logs := launcherInstance.GetLogs() @@ -210,13 +252,38 @@ var _ = Describe("Launcher", func() { }) }) +// Regression for the welcome window suppressing itself (part of issue +// #11673): the "don't show this welcome window again" checkbox was +// initialized with the ShowWelcome value itself, so on the very first +// showing it came up checked AND its change callback persisted +// ShowWelcome=false, hiding the welcome window forever. +var _ = Describe("WelcomeDontShowAgainChecked", func() { + It("should be unchecked when the welcome window is enabled", func() { + show := true + config := &launcher.Config{ShowWelcome: &show} + Expect(launcher.WelcomeDontShowAgainChecked(config)).To(BeFalse()) + }) + + It("should be checked when the user opted out", func() { + show := false + config := &launcher.Config{ShowWelcome: &show} + Expect(launcher.WelcomeDontShowAgainChecked(config)).To(BeTrue()) + }) + + It("should be unchecked when the preference is unset", func() { + Expect(launcher.WelcomeDontShowAgainChecked(&launcher.Config{})).To(BeFalse()) + Expect(launcher.WelcomeDontShowAgainChecked(nil)).To(BeFalse()) + }) +}) + var _ = Describe("Config", func() { It("should have proper JSON tags", func() { + autoStart := true config := &launcher.Config{ ModelsPath: "/test/models", BackendsPath: "/test/backends", Address: ":8080", - AutoStart: true, + AutoStart: &autoStart, LogLevel: "info", EnvironmentVars: map[string]string{"TEST": "value"}, } @@ -224,7 +291,7 @@ var _ = Describe("Config", func() { Expect(config.ModelsPath).To(Equal("/test/models")) Expect(config.BackendsPath).To(Equal("/test/backends")) Expect(config.Address).To(Equal(":8080")) - Expect(config.AutoStart).To(BeTrue()) + Expect(*config.AutoStart).To(BeTrue()) Expect(config.LogLevel).To(Equal("info")) Expect(config.EnvironmentVars).To(HaveKeyWithValue("TEST", "value")) }) diff --git a/cmd/launcher/internal/ui.go b/cmd/launcher/internal/ui.go index a3f19a146..bf4f73a9a 100644 --- a/cmd/launcher/internal/ui.go +++ b/cmd/launcher/internal/ui.go @@ -34,6 +34,7 @@ type LauncherUI struct { backendsPathEntry *widget.Entry addressEntry *widget.Entry logLevelSelect *widget.Select + autoStartCheck *widget.Check startOnBootCheck *widget.Check // Environment Variables @@ -75,6 +76,7 @@ func NewLauncherUI() *LauncherUI { backendsPathEntry: widget.NewEntry(), addressEntry: widget.NewEntry(), logLevelSelect: widget.NewSelect([]string{"error", "warn", "info", "debug", "trace"}, nil), + autoStartCheck: widget.NewCheck("Start LocalAI when the launcher opens", nil), startOnBootCheck: widget.NewCheck("Start LocalAI on system boot", nil), logText: widget.NewMultiLineEntry(), progressBar: widget.NewProgressBar(), @@ -117,6 +119,7 @@ func (ui *LauncherUI) createConfigTab() *fyne.Container { widget.NewLabel("Log Level:"), ui.logLevelSelect, ), + ui.autoStartCheck, ui.startOnBootCheck, )) @@ -401,6 +404,8 @@ func (ui *LauncherUI) saveConfiguration() { config.BackendsPath = ui.backendsPathEntry.Text config.Address = ui.addressEntry.Text config.LogLevel = ui.logLevelSelect.Selected + autoStart := ui.autoStartCheck.Checked + config.AutoStart = &autoStart config.StartOnBoot = ui.startOnBootCheck.Checked // Ensure environment variables are included in the configuration @@ -583,6 +588,7 @@ func (ui *LauncherUI) LoadConfiguration() { ui.backendsPathEntry.SetText(config.BackendsPath) ui.addressEntry.SetText(config.Address) ui.logLevelSelect.SetSelected(config.LogLevel) + ui.autoStartCheck.SetChecked(config.AutoStart == nil || *config.AutoStart) ui.startOnBootCheck.SetChecked(config.StartOnBoot) // Load environment variables @@ -616,6 +622,14 @@ func (ui *LauncherUI) UpdateRunningState(isRunning bool) { }) } +// WelcomeDontShowAgainChecked reports the initial state of the welcome +// window's "don't show this welcome window again" checkbox for the given +// config: checked only when the user has already opted out of the welcome +// window. +func WelcomeDontShowAgainChecked(config *Config) bool { + return config != nil && config.ShowWelcome != nil && !*config.ShowWelcome +} + // ShowWelcomeWindow displays the welcome window with helpful information func (ui *LauncherUI) ShowWelcomeWindow() { if ui.launcher == nil || ui.launcher.window == nil { @@ -677,19 +691,20 @@ Getting Started: ui.openURL("https://discord.gg/XgwjKptP7Z") }) - // Checkbox to disable welcome window - dontShowAgainCheck := widget.NewCheck("Don't show this welcome window again", func(checked bool) { + // Checkbox to disable welcome window. The initial state is applied + // BEFORE the change callback is attached: SetChecked fires OnChanged, + // and letting the initialization itself persist a ShowWelcome flip is + // exactly the bug that suppressed this window forever after its first + // showing (#11673). + dontShowAgainCheck := widget.NewCheck("Don't show this welcome window again", nil) + dontShowAgainCheck.SetChecked(WelcomeDontShowAgainChecked(ui.launcher.GetConfig())) + dontShowAgainCheck.OnChanged = func(checked bool) { if ui.launcher != nil { config := ui.launcher.GetConfig() v := !checked config.ShowWelcome = &v ui.launcher.SetConfig(config) } - }) - - config := ui.launcher.GetConfig() - if config.ShowWelcome != nil { - dontShowAgainCheck.SetChecked(*config.ShowWelcome) } // Close button diff --git a/core/application/distributed.go b/core/application/distributed.go index 8389c5c9f..1fceee131 100644 --- a/core/application/distributed.go +++ b/core/application/distributed.go @@ -15,6 +15,7 @@ import ( "github.com/mudler/LocalAI/core/services/distributed" "github.com/mudler/LocalAI/core/services/jobs" "github.com/mudler/LocalAI/core/services/messaging" + "github.com/mudler/LocalAI/core/services/monitoring" "github.com/mudler/LocalAI/core/services/nodes" "github.com/mudler/LocalAI/core/services/nodes/prefixcache" "github.com/mudler/LocalAI/core/services/storage" @@ -162,6 +163,26 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade } xlog.Info("Node registry initialized") + // Bound durable heartbeat writes: a beat that only carries a fresher + // timestamp is what turned backend_nodes into a 460 MB six-row table. + registry.SetHeartbeatCheckpoint(cfg.Distributed.NodeHeartbeatCheckpointOrDefault()) + + // Measure the vacuum horizon. The 42 days it stayed open went unnoticed + // because no gauge reported it until models started failing to load. + if err := monitoring.RegisterControlPlaneDBMetrics(authDB, 30*time.Second); err != nil { + // Metrics are diagnostic; a failure here must not stop the frontend. + xlog.Warn("Control-plane database metrics unavailable", "error", err) + } + + // Let scheduling rules be keyed by a model alias. The registry resolves a + // rule's name through the config loader to find the model it governs, so an + // operator can pin placement to a stable name like "production" and have it + // follow the alias when the alias is repointed. Wired before the seed below + // and before the reconciler starts, so the first tick already resolves. + if configLoader != nil { + registry.SetAliasResolver(configLoader) + } + // Seed declarative per-model scheduling config (LOCALAI_MODEL_SCHEDULING / // LOCALAI_MODEL_SCHEDULING_CONFIG). Authoritative: overwrites matching models // on every boot. Runs before the reconciler starts so the first tick already diff --git a/core/cli/run.go b/core/cli/run.go index 6b9b3e4dc..6bf377682 100644 --- a/core/cli/run.go +++ b/core/cli/run.go @@ -182,6 +182,8 @@ type RunCMD struct { BackendUpgradeTimeout string `env:"LOCALAI_NATS_BACKEND_UPGRADE_TIMEOUT" help:"NATS round-trip timeout for backend.upgrade requests (default 15m)." group:"distributed"` ModelLoadTimeout string `env:"LOCALAI_NATS_MODEL_LOAD_TIMEOUT" help:"Fixed gRPC deadline for the remote LoadModel call sent to a worker node once its backend is installed and model files are staged. Unset (the default), the deadline is derived from the checkpoint size instead: 5m plus 20s per GiB, capped at 6h, so multi-tens-of-GB diffusion/video checkpoints get the minutes they need without a fixed cliff. Set this only to pin a specific budget; the value is used verbatim, including when it is shorter than the derived one." group:"distributed"` ModelLoadWait string `env:"LOCALAI_MODEL_LOAD_WAIT" help:"How long an inference request waits for a model that is still cold-loading onto a worker before it is answered with 503, a Retry-After header and live staging progress (default 60s). The request is served the moment the model becomes ready, so a model already most of the way staged needs no client retry. Set to 0 to wait as long as the load takes — only safe when no ingress or load balancer with an idle timeout sits in front." group:"distributed"` + StaleNodeThreshold string `env:"LOCALAI_STALE_NODE_THRESHOLD" help:"How long a worker node may go without a durable heartbeat before the health monitor marks it offline (default 5m). Because a beat that only carries a fresher timestamp is held back by --node-heartbeat-checkpoint, this must stay comfortably wider than that interval; raise both together. Dead-node detection through the per-model gRPC health check and through request-time failure is unaffected by this knob." group:"distributed"` + NodeHeartbeatCheckpoint string `env:"LOCALAI_NODE_HEARTBEAT_CHECKPOINT" help:"Minimum gap between durable heartbeat writes for a worker node (default 60s). A beat that only carries a fresher timestamp is dropped until this interval elapses; every field is compared against the value last written, so a node's first beat, a changed total VRAM/total disk/GPU vendor, and a free VRAM/RAM/disk reading that has moved more than 256 MiB from the written value all still write immediately, and a node that is not active is never suppressed. Set below the worker heartbeat interval to write on every beat." group:"distributed"` NatsAccountSeed string `env:"LOCALAI_NATS_ACCOUNT_SEED" help:"NATS account signing seed (SU...) used to mint per-node worker JWTs at registration" group:"distributed"` NatsServiceJWT string `env:"LOCALAI_NATS_SERVICE_JWT" help:"NATS user JWT for the frontend (and agent workers) to publish control-plane messages" group:"distributed"` NatsServiceSeed string `env:"LOCALAI_NATS_SERVICE_SEED" help:"NATS user signing seed (SU...) paired with LOCALAI_NATS_SERVICE_JWT" group:"distributed"` @@ -397,6 +399,20 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error { } opts = append(opts, config.WithModelLoadWait(d)) } + if r.StaleNodeThreshold != "" { + d, err := parseDistributedDuration("LOCALAI_STALE_NODE_THRESHOLD", r.StaleNodeThreshold) + if err != nil { + return err + } + opts = append(opts, config.WithStaleNodeThreshold(d)) + } + if r.NodeHeartbeatCheckpoint != "" { + d, err := parseDistributedDuration("LOCALAI_NODE_HEARTBEAT_CHECKPOINT", r.NodeHeartbeatCheckpoint) + if err != nil { + return err + } + opts = append(opts, config.WithNodeHeartbeatCheckpoint(d)) + } if r.RegistrationToken != "" { opts = append(opts, config.WithRegistrationToken(r.RegistrationToken)) } diff --git a/core/cli/worker/labels.go b/core/cli/worker/labels.go index 1f5142e7a..92f484557 100644 --- a/core/cli/worker/labels.go +++ b/core/cli/worker/labels.go @@ -10,7 +10,7 @@ func ParseNodeLabels(input string) map[string]string { if input == "" { return labels } - for _, pair := range strings.Split(input, ",") { + for pair := range strings.SplitSeq(input, ",") { pair = strings.TrimSpace(pair) if k, v, ok := strings.Cut(pair, "="); ok { labels[strings.TrimSpace(k)] = strings.TrimSpace(v) diff --git a/core/config/distributed_config.go b/core/config/distributed_config.go index 5a48a84e9..ef7e01bfd 100644 --- a/core/config/distributed_config.go +++ b/core/config/distributed_config.go @@ -57,12 +57,13 @@ type DistributedConfig struct { StorageSecretKey string // --storage-secret-key / LOCALAI_STORAGE_SECRET_KEY // Timeout configuration (all have sensible defaults — zero means use default) - MCPToolTimeout time.Duration // MCP tool execution timeout (default 360s) - MCPDiscoveryTimeout time.Duration // MCP discovery timeout (default 60s) - WorkerWaitTimeout time.Duration // Max wait for healthy worker at startup (default 5m) - DrainTimeout time.Duration // Time to wait for in-flight requests during drain (default 30s) - HealthCheckInterval time.Duration // Health monitor check interval (default 15s) - StaleNodeThreshold time.Duration // Time before a node is considered stale (default 60s) + MCPToolTimeout time.Duration // MCP tool execution timeout (default 360s) + MCPDiscoveryTimeout time.Duration // MCP discovery timeout (default 60s) + WorkerWaitTimeout time.Duration // Max wait for healthy worker at startup (default 5m) + DrainTimeout time.Duration // Time to wait for in-flight requests during drain (default 30s) + HealthCheckInterval time.Duration // Health monitor check interval (default 15s) + StaleNodeThreshold time.Duration // Time before a node is considered stale (default 5m) + NodeHeartbeatCheckpoint time.Duration // Minimum gap between durable heartbeat writes (default 60s, 0 = every beat) // DisablePerModelHealthCheck turns off the health monitor's per-model // gRPC probe. When enabled (the default), the monitor pings each model's // gRPC address and removes stale node_models rows whose backend has @@ -165,16 +166,17 @@ func (c DistributedConfig) Validate() error { c.NatsAuthConfig().WarnIfInsecure(true) // Check for negative durations for name, d := range map[string]time.Duration{ - FlagMCPToolTimeout: c.MCPToolTimeout, - FlagMCPDiscoveryTimeout: c.MCPDiscoveryTimeout, - FlagWorkerWaitTimeout: c.WorkerWaitTimeout, - FlagDrainTimeout: c.DrainTimeout, - FlagHealthCheckInterval: c.HealthCheckInterval, - FlagStaleNodeThreshold: c.StaleNodeThreshold, - FlagMCPCIJobTimeout: c.MCPCIJobTimeout, - FlagBackendInstallTimeout: c.BackendInstallTimeout, - FlagBackendUpgradeTimeout: c.BackendUpgradeTimeout, - FlagModelLoadTimeout: c.ModelLoadTimeout, + FlagMCPToolTimeout: c.MCPToolTimeout, + FlagMCPDiscoveryTimeout: c.MCPDiscoveryTimeout, + FlagWorkerWaitTimeout: c.WorkerWaitTimeout, + FlagDrainTimeout: c.DrainTimeout, + FlagHealthCheckInterval: c.HealthCheckInterval, + FlagStaleNodeThreshold: c.StaleNodeThreshold, + FlagNodeHeartbeatCheckpoint: c.NodeHeartbeatCheckpoint, + FlagMCPCIJobTimeout: c.MCPCIJobTimeout, + FlagBackendInstallTimeout: c.BackendInstallTimeout, + FlagBackendUpgradeTimeout: c.BackendUpgradeTimeout, + FlagModelLoadTimeout: c.ModelLoadTimeout, } { if d < 0 { return fmt.Errorf("%s must not be negative", name) @@ -337,6 +339,27 @@ func WithModelLoadWait(d time.Duration) AppOption { } } +// WithStaleNodeThreshold sets how long a node may go without a durable +// heartbeat before the health monitor marks it offline. It has to be raised +// alongside WithNodeHeartbeatCheckpoint: a checkpoint interval wider than this +// threshold makes every healthy node look dead the moment its beats start +// being suppressed. +func WithStaleNodeThreshold(d time.Duration) AppOption { + return func(o *ApplicationConfig) { + o.Distributed.StaleNodeThreshold = d + } +} + +// WithNodeHeartbeatCheckpoint bounds durable heartbeat writes. A zero d is +// deliberately not special-cased into "unbounded": NodeHeartbeatCheckpointOrDefault +// reads zero as unset, and an operator who wants a write per beat sets a value +// below the worker's heartbeat interval instead. +func WithNodeHeartbeatCheckpoint(d time.Duration) AppOption { + return func(o *ApplicationConfig) { + o.Distributed.NodeHeartbeatCheckpoint = d + } +} + var EnableAutoApproveNodes = func(o *ApplicationConfig) { o.Distributed.AutoApproveNodes = true } @@ -391,17 +414,18 @@ func WithModelSchedulingConfigPath(path string) AppOption { // them as constants prevents the string from drifting from the actual // flag a future rename would produce. const ( - FlagMCPToolTimeout = "mcp-tool-timeout" - FlagMCPDiscoveryTimeout = "mcp-discovery-timeout" - FlagWorkerWaitTimeout = "worker-wait-timeout" - FlagDrainTimeout = "drain-timeout" - FlagHealthCheckInterval = "health-check-interval" - FlagStaleNodeThreshold = "stale-node-threshold" - FlagMCPCIJobTimeout = "mcp-ci-job-timeout" - FlagBackendInstallTimeout = "backend-install-timeout" - FlagBackendUpgradeTimeout = "backend-upgrade-timeout" - FlagModelLoadTimeout = "model-load-timeout" - FlagModelLoadWait = "model-load-wait" + FlagMCPToolTimeout = "mcp-tool-timeout" + FlagMCPDiscoveryTimeout = "mcp-discovery-timeout" + FlagWorkerWaitTimeout = "worker-wait-timeout" + FlagDrainTimeout = "drain-timeout" + FlagHealthCheckInterval = "health-check-interval" + FlagStaleNodeThreshold = "stale-node-threshold" + FlagNodeHeartbeatCheckpoint = "node-heartbeat-checkpoint" + FlagMCPCIJobTimeout = "mcp-ci-job-timeout" + FlagBackendInstallTimeout = "backend-install-timeout" + FlagBackendUpgradeTimeout = "backend-upgrade-timeout" + FlagModelLoadTimeout = "model-load-timeout" + FlagModelLoadWait = "model-load-wait" // FlagDiskHeadroomCheck names the disk-headroom toggle. It is quoted in // the warning the check emits while disabled, so the operator reading a // log line knows exactly which knob produced it. @@ -410,16 +434,22 @@ const ( // Defaults for distributed timeouts. const ( - DefaultMCPToolTimeout = 360 * time.Second - DefaultMCPDiscoveryTimeout = 60 * time.Second - DefaultWorkerWaitTimeout = 5 * time.Minute - DefaultDrainTimeout = 30 * time.Second - DefaultHealthCheckInterval = 15 * time.Second - DefaultStaleNodeThreshold = 60 * time.Second - DefaultMCPCIJobTimeout = 10 * time.Minute - DefaultBackendInstallTimeout = 15 * time.Minute - DefaultBackendUpgradeTimeout = 15 * time.Minute - DefaultModelLoadTimeout = 5 * time.Minute + DefaultMCPToolTimeout = 360 * time.Second + DefaultMCPDiscoveryTimeout = 60 * time.Second + DefaultWorkerWaitTimeout = 5 * time.Minute + DefaultDrainTimeout = 30 * time.Second + DefaultHealthCheckInterval = 15 * time.Second + // A beat that only refreshes the timestamp is now dropped until the + // checkpoint interval elapses, so the persisted column is up to one + // interval stale by design. The threshold covers that plus jitter. + // A genuinely dead node is still caught sooner by the per-model gRPC + // health check and by request-time failure, neither of which reads this. + DefaultStaleNodeThreshold = 5 * time.Minute + DefaultNodeHeartbeatCheckpoint = 60 * time.Second + DefaultMCPCIJobTimeout = 10 * time.Minute + DefaultBackendInstallTimeout = 15 * time.Minute + DefaultBackendUpgradeTimeout = 15 * time.Minute + DefaultModelLoadTimeout = 5 * time.Minute // DefaultModelLoadWait is how long a request waits for a cold-loading model // before it is answered with 503 and live progress. Chosen to sit under the // idle timeout of typical ingress/LB defaults, so the answer comes from @@ -519,6 +549,14 @@ func (c DistributedConfig) StaleNodeThresholdOrDefault() time.Duration { return cmp.Or(c.StaleNodeThreshold, DefaultStaleNodeThreshold) } +// NodeHeartbeatCheckpointOrDefault returns the configured interval or the +// default. A configured zero is indistinguishable from unset here, which is +// intentional: cmp.Or falls back to the default, and an operator who wants a +// write per beat sets a value below the heartbeat interval instead. +func (c DistributedConfig) NodeHeartbeatCheckpointOrDefault() time.Duration { + return cmp.Or(c.NodeHeartbeatCheckpoint, DefaultNodeHeartbeatCheckpoint) +} + // MCPCIJobTimeoutOrDefault returns the configured MCP CI job timeout or the default. func (c DistributedConfig) MCPCIJobTimeoutOrDefault() time.Duration { return cmp.Or(c.MCPCIJobTimeout, DefaultMCPCIJobTimeout) diff --git a/core/config/distributed_config_test.go b/core/config/distributed_config_test.go index ec7fbe8dc..dc98c09bb 100644 --- a/core/config/distributed_config_test.go +++ b/core/config/distributed_config_test.go @@ -47,6 +47,27 @@ var _ = Describe("DistributedConfig backend NATS timeouts", func() { }) }) +// Heartbeat checkpointing makes last_heartbeat up to one checkpoint interval +// stale by design, which is why the threshold defaults to 5 minutes. An +// operator who widens the checkpoint has to widen this to match, so it has to +// be reachable from the CLI rather than being a compile-time constant. +var _ = Describe("DistributedConfig stale node threshold", func() { + It("defaults to 5 minutes, wide enough to cover a suppressed beat", func() { + Expect(config.DistributedConfig{}.StaleNodeThresholdOrDefault()). + To(Equal(5 * time.Minute)) + Expect(config.DefaultStaleNodeThreshold). + To(BeNumerically(">", config.DefaultNodeHeartbeatCheckpoint), + "a threshold at or below the checkpoint interval marks healthy, "+ + "beating nodes offline every cycle") + }) + + It("is configurable, so a widened checkpoint can be matched", func() { + o := config.NewApplicationConfig(config.WithStaleNodeThreshold(20 * time.Minute)) + Expect(o.Distributed.StaleNodeThreshold).To(Equal(20 * time.Minute)) + Expect(o.Distributed.StaleNodeThresholdOrDefault()).To(Equal(20 * time.Minute)) + }) +}) + var _ = Describe("DistributedConfig flag-name constants", func() { // Pin the kebab-case strings so a rename of the Go field name (or a // CLI flag naming convention change) forces the constant to update, @@ -62,6 +83,7 @@ var _ = Describe("DistributedConfig flag-name constants", func() { Entry("drain timeout", config.FlagDrainTimeout, "drain-timeout"), Entry("health check interval", config.FlagHealthCheckInterval, "health-check-interval"), Entry("stale node threshold", config.FlagStaleNodeThreshold, "stale-node-threshold"), + Entry("node heartbeat checkpoint", config.FlagNodeHeartbeatCheckpoint, "node-heartbeat-checkpoint"), Entry("MCP CI job timeout", config.FlagMCPCIJobTimeout, "mcp-ci-job-timeout"), Entry("backend install timeout", config.FlagBackendInstallTimeout, "backend-install-timeout"), Entry("backend upgrade timeout", config.FlagBackendUpgradeTimeout, "backend-upgrade-timeout"), diff --git a/core/config/inference_defaults.json b/core/config/inference_defaults.json index 8fe888de1..42c463df8 100644 --- a/core/config/inference_defaults.json +++ b/core/config/inference_defaults.json @@ -1,6 +1,8 @@ { "_comment": "Auto-generated from unsloth inference_defaults.json. DO NOT EDIT. Run go generate ./core/config/ to update.", "families": { + "moss-tts-local-transformer-v1.5": {"min_p":0,"repeat_penalty":1,"temperature":1.7,"top_k":25,"top_p":0.8}, + "moss-tts-nano": {"min_p":0,"repeat_penalty":1,"temperature":1.7,"top_k":25,"top_p":0.8}, "qwen3.8": {"min_p":0,"presence_penalty":1.5,"repeat_penalty":1,"temperature":0.7,"top_k":20,"top_p":0.8}, "qwen3.6": {"min_p":0,"presence_penalty":1.5,"repeat_penalty":1,"temperature":0.7,"top_k":20,"top_p":0.8}, "qwen3.5": {"min_p":0,"presence_penalty":1.5,"repeat_penalty":1,"temperature":0.7,"top_k":20,"top_p":0.8}, @@ -60,5 +62,5 @@ "grok": {"min_p":0.01,"repeat_penalty":1,"temperature":1,"top_k":-1,"top_p":0.95}, "mimo": {"min_p":0.01,"repeat_penalty":1,"temperature":0.7,"top_k":-1,"top_p":0.95} }, - "patterns": ["qwen3.8","qwen3.6","qwen3.5","qwen3-coder","qwen3-next","qwen3-vl","qwen3","qwen2.5-coder","qwen2.5-vl","qwen2.5-omni","qwen2.5-math","qwen2.5","qwen2-vl","qwen2","qwq","gemma-4","gemma-3n","gemma-3","medgemma","gemma-2","muse-glimmer","llama-4","llama-3.3","llama-3.2","llama-3.1","llama-3","phi-4","phi-3","mistral-nemo","mistral-small","mistral-large","magistral","ministral","devstral","pixtral","deepseek-v4","deepseek-r1","deepseek-v3","deepseek-ocr","glm-5","glm-4","nemotron","minimax-m2.7","minimax-m2.5","minimax","gpt-oss","granite-4","kimi-k3","kimi-k2","kimi","lfm2","smollm","olmo","falcon","ernie","seed","grok","mimo"] + "patterns": ["moss-tts-local-transformer-v1.5","moss-tts-nano","qwen3.8","qwen3.6","qwen3.5","qwen3-coder","qwen3-next","qwen3-vl","qwen3","qwen2.5-coder","qwen2.5-vl","qwen2.5-omni","qwen2.5-math","qwen2.5","qwen2-vl","qwen2","qwq","gemma-4","gemma-3n","gemma-3","medgemma","gemma-2","muse-glimmer","llama-4","llama-3.3","llama-3.2","llama-3.1","llama-3","phi-4","phi-3","mistral-nemo","mistral-small","mistral-large","magistral","ministral","devstral","pixtral","deepseek-v4","deepseek-r1","deepseek-v3","deepseek-ocr","glm-5","glm-4","nemotron","minimax-m2.7","minimax-m2.5","minimax","gpt-oss","granite-4","kimi-k3","kimi-k2","kimi","lfm2","smollm","olmo","falcon","ernie","seed","grok","mimo"] } diff --git a/core/config/model_config_loader.go b/core/config/model_config_loader.go index b91449ff0..9120cc268 100644 --- a/core/config/model_config_loader.go +++ b/core/config/model_config_loader.go @@ -441,6 +441,26 @@ func (bcl *ModelConfigLoader) ResolveAlias(cfg *ModelConfig) (*ModelConfig, bool return &target, true, nil } +// ResolveAliasName maps a model name to the name of the model that actually +// serves it: an alias resolves to its target, anything else resolves to +// itself. The second return reports whether name was an alias. +// +// Unlike ResolveAlias this never errors. A name with no config (a rule may be +// authored before the model is installed), a dangling alias, and a chained +// alias all resolve to themselves, so callers keep a usable name that simply +// has no model behind it rather than silently governing a different model. +func (bcl *ModelConfigLoader) ResolveAliasName(name string) (string, bool) { + cfg, exists := bcl.GetModelConfig(name) + if !exists || !cfg.IsAlias() { + return name, false + } + target, exists := bcl.GetModelConfig(cfg.Alias) + if !exists || target.IsAlias() { + return name, true + } + return target.Name, true +} + // ValidateAliasTarget checks an alias config's target at create/swap time: // the target must exist, must not be an alias, and must not be disabled. // Returns nil for non-alias configs. diff --git a/core/config/model_config_loader_test.go b/core/config/model_config_loader_test.go index 87807deec..d654226ef 100644 --- a/core/config/model_config_loader_test.go +++ b/core/config/model_config_loader_test.go @@ -314,3 +314,57 @@ var _ = Describe("ModelConfigLoader alias resolution", func() { Expect(loader.ValidateAliasTarget(&bad)).To(MatchError(ContainSubstring("itself an alias"))) }) }) + +var _ = Describe("ModelConfigLoader ResolveAliasName", func() { + var loader *ModelConfigLoader + + BeforeEach(func() { + loader = NewModelConfigLoader("") + loader.configs["real"] = ModelConfig{Name: "real", Backend: "llama-cpp"} + loader.configs["production"] = ModelConfig{Name: "production", Alias: "real"} + loader.configs["chain"] = ModelConfig{Name: "chain", Alias: "production"} + loader.configs["dangling"] = ModelConfig{Name: "dangling", Alias: "nope"} + }) + + It("maps an alias name to the model that actually serves it", func() { + target, isAlias := loader.ResolveAliasName("production") + Expect(isAlias).To(BeTrue()) + Expect(target).To(Equal("real")) + }) + + It("maps a real model name to itself", func() { + target, isAlias := loader.ResolveAliasName("real") + Expect(isAlias).To(BeFalse()) + Expect(target).To(Equal("real")) + }) + + // A rule may be authored for a model that is not installed yet (pre-staging + // placement before standing up a node), so an unknown name must resolve to + // itself rather than to the empty string. + It("maps an unknown name to itself", func() { + target, isAlias := loader.ResolveAliasName("not-installed-yet") + Expect(isAlias).To(BeFalse()) + Expect(target).To(Equal("not-installed-yet")) + }) + + // A broken alias has no model behind it. Resolving to itself keeps the + // caller on a name that simply has no replicas, instead of silently + // governing some other model. + It("maps a dangling alias to itself", func() { + target, isAlias := loader.ResolveAliasName("dangling") + Expect(isAlias).To(BeTrue()) + Expect(target).To(Equal("dangling")) + }) + + It("maps a chained alias to itself rather than following the chain", func() { + target, isAlias := loader.ResolveAliasName("chain") + Expect(isAlias).To(BeTrue()) + Expect(target).To(Equal("chain")) + }) + + It("maps the empty name to itself", func() { + target, isAlias := loader.ResolveAliasName("") + Expect(isAlias).To(BeFalse()) + Expect(target).To(BeEmpty()) + }) +}) diff --git a/core/gallery/cluster_env_test.go b/core/gallery/cluster_env_test.go new file mode 100644 index 000000000..4a411277e --- /dev/null +++ b/core/gallery/cluster_env_test.go @@ -0,0 +1,94 @@ +package gallery_test + +import ( + "context" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/gallery" + "github.com/mudler/LocalAI/pkg/system" +) + +// On a distributed controller the GPUs live on the workers, so a variant +// picker sized against the controller tells admins a cluster of A100s can only +// run the smallest CPU build. +var _ = Describe("ClusterResolveEnv", func() { + gib := func(n uint64) uint64 { return n * 1024 * 1024 * 1024 } + + // The controller as Argus actually runs it: no GPU at all. + var controller *system.SystemState + + BeforeEach(func() { + controller = system.NewCapabilityState("default") + }) + + It("sizes models against the cluster reading rather than the controller", func() { + env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13"}) + + Expect(env.AvailableMemory).To(Equal(gib(80))) + }) + + It("accepts a CUDA backend that only the workers can run", func() { + env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13"}) + + Expect(env.BackendCompatible).ToNot(BeNil()) + // A name carrying the cuda token is what the controller rejects today; + // a bare engine name like "vllm" passes on any host and would prove + // nothing about the union. + Expect(env.BackendCompatible("cuda-13-vllm")).To(BeTrue()) + Expect(env.BackendCompatible("llama-cpp")).To(BeTrue()) + }) + + // The union must stay a filter, not an open door: a Linux NVIDIA fleet + // still cannot run an Apple-only build. + It("still rejects a backend no node in the cluster can run", func() { + env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13"}) + + Expect(env.BackendCompatible("mlx")).To(BeFalse()) + }) + + It("accepts a backend that any one node in a mixed fleet can run", func() { + env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13", "metal"}) + + Expect(env.BackendCompatible("mlx")).To(BeTrue()) + Expect(env.BackendCompatible("cuda-13-vllm")).To(BeTrue()) + }) + + // Ranking has to follow the hardware too, or a cluster of NVIDIA workers + // gets offered the GGUF build over the vLLM one it should prefer. + It("ranks engines by the workers' hardware, not the controller's", func() { + env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13"}) + + Expect(env.EnginePreference).To(Equal(system.NewCapabilityState("nvidia-cuda-13").EnginePreferenceTokens())) + }) + + // Every degradation path lands here, so it must be indistinguishable from + // the single-node behavior that shipped before any of this existed. + It("falls back to the host description when the cluster reports nothing", func() { + host := gallery.HostResolveEnv(context.Background(), controller) + env := gallery.ClusterResolveEnv(context.Background(), controller, 0, nil) + + Expect(env.AvailableMemory).To(Equal(host.AvailableMemory)) + Expect(env.EnginePreference).To(Equal(host.EnginePreference)) + Expect(env.BackendCompatible("cuda-13-vllm")).To(Equal(host.BackendCompatible("cuda-13-vllm"))) + Expect(env.BackendCompatible("mlx")).To(Equal(host.BackendCompatible("mlx"))) + }) + + // A cluster that reports capabilities but no usable memory reading should + // still gain the hardware view; only the size question falls back. + It("keeps the host memory when only the memory reading is missing", func() { + host := gallery.HostResolveEnv(context.Background(), controller) + env := gallery.ClusterResolveEnv(context.Background(), controller, 0, []string{"nvidia-cuda-13"}) + + Expect(env.AvailableMemory).To(Equal(host.AvailableMemory)) + Expect(env.BackendCompatible("cuda-13-vllm")).To(BeTrue()) + }) + + It("keeps the probe wired so variant sizes are still measured", func() { + env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13"}) + + Expect(env.ProbeMemory).ToNot(BeNil()) + Expect(env.ServingFeaturePreference).To(Equal(system.ServingFeaturePreferenceTokens())) + }) +}) diff --git a/core/gallery/empty_base_install_test.go b/core/gallery/empty_base_install_test.go index 6b104e233..b2a15c1ea 100644 --- a/core/gallery/empty_base_install_test.go +++ b/core/gallery/empty_base_install_test.go @@ -236,8 +236,15 @@ var _ = Describe("InstallModelFromGallery with an empty base config", func() { Expect(install(e.Name, gallery.GalleryModel{})).To(Succeed()) cfg := installedConfig(e.Name) Expect(cfg["name"]).To(Equal(e.Name)) - // The catalog's own overrides, verbatim, laid over the empty base. - Expect(cfg["parameters"]).To(Equal(e.Overrides["parameters"])) + // The catalog's own overrides, laid over the empty base. parameters is + // checked key by key rather than as a whole map: the install also merges + // the model family's inference defaults into it, and what matters here is + // that the authored keys survive that. + authored, ok := e.Overrides["parameters"].(map[string]any) + Expect(ok).To(BeTrue()) + for key, want := range authored { + Expect(cfg["parameters"]).To(HaveKeyWithValue(key, want)) + } Expect(cfg["known_usecases"]).To(Equal(e.Overrides["known_usecases"])) }) }) diff --git a/core/gallery/exl3_gallery_test.go b/core/gallery/exl3_gallery_test.go new file mode 100644 index 000000000..5c55be57e --- /dev/null +++ b/core/gallery/exl3_gallery_test.go @@ -0,0 +1,101 @@ +package gallery_test + +import ( + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/gallery" + "gopkg.in/yaml.v3" +) + +func remarshal(value any, target any) error { + data, err := yaml.Marshal(value) + if err != nil { + return err + } + return yaml.Unmarshal(data, target) +} + +var _ = Describe("EXL3 gallery entries", func() { + It("pins the four full repositories and configures the Qwen DFlash companion", func() { + entries, err := gallery.ReadConfigFile[[]gallery.GalleryModel](filepath.Join("..", "..", "gallery", "index.yaml")) + Expect(err).ToNot(HaveOccurred()) + + byName := make(map[string]gallery.GalleryModel, len(*entries)) + for _, entry := range *entries { + byName[entry.Name] = entry + } + + expected := map[string]struct { + repo string + revision string + }{ + "qwen3.8-27b-exl3-vllm-cpp": { + repo: "Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw", revision: "19441ac874c4018295da848e250f23511361cda4", + }, + "qwen3.8-27b-dflash2-exl3-vllm-cpp": { + repo: "Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw", revision: "19441ac874c4018295da848e250f23511361cda4", + }, + "deepseek-v4-flash-spark-exl3-vllm-cpp": { + repo: "0xSero/deepseek-v4-flash-0731-spark", revision: "ce5ff0f1efb2e184aafc759d281bfae47d3a359c", + }, + "deepseek-v4-flash-exl3-3bpw-vllm-cpp": { + repo: "0xSero/DeepSeek-V4-Flash-0731-EXL3-3.0bpw", revision: "e0bf84ac76a5100e8790c22ad10b70b1e2d06d71", + }, + } + + for name, want := range expected { + entry, found := byName[name] + Expect(found).To(BeTrue(), "missing gallery entry %q", name) + Expect(entry.Tags).To(ContainElements("vllm-cpp", "exl3", "gpu", "cuda"), name) + Expect(entry.Overrides).To(HaveKeyWithValue("backend", "vllm-cpp"), name) + cfg := config.ModelConfig{} + Expect(remarshal(entry.Overrides, &cfg)).To(Succeed(), name) + Expect(cfg.Artifacts).ToNot(BeEmpty(), name) + Expect(cfg.Artifacts[0].Source.Repo).To(Equal(want.repo), name) + Expect(cfg.Artifacts[0].Source.Revision).To(Equal(want.revision), name) + } + + plain := byName["qwen3.8-27b-exl3-vllm-cpp"] + Expect(plain.Tags).ToNot(ContainElement("dflash")) + dflashTags := 0 + for name := range expected { + if contains(byName[name].Tags, "dflash") { + dflashTags++ + } + } + Expect(dflashTags).To(Equal(1)) + + dflash := byName["qwen3.8-27b-dflash2-exl3-vllm-cpp"] + Expect(dflash.Tags).To(ContainElement("dflash")) + Expect(dflash.Variants).To(ConsistOf(gallery.Variant{Model: "qwen3.8-27b-exl3-vllm-cpp"})) + cfg := config.ModelConfig{} + Expect(remarshal(dflash.Overrides, &cfg)).To(Succeed()) + Expect(cfg.ContextSize).To(HaveValue(Equal(8192))) + Expect(cfg.EngineArgs).To(HaveKeyWithValue("num_blocks", 2048)) + Expect(cfg.EngineArgs).To(HaveKeyWithValue("max_num_seqs", 8)) + Expect(cfg.EngineArgs).To(HaveKeyWithValue("max_num_batched_tokens", 16384)) + Expect(cfg.EngineArgs).To(HaveKeyWithValue("enable_prefix_caching", false)) + spec, ok := cfg.EngineArgs["speculative_config"].(map[string]any) + Expect(ok).To(BeTrue()) + Expect(spec).To(HaveKeyWithValue("method", "dflash")) + Expect(spec).To(HaveKeyWithValue("num_speculative_tokens", 7)) + Expect(cfg.Artifacts).To(HaveLen(2)) + Expect(cfg.Artifacts[1].Name).To(Equal("draft_model")) + Expect(cfg.Artifacts[1].Target).To(Equal("companion")) + Expect(cfg.Artifacts[1].Source.Repo).To(Equal("Mia-AiLab/Qwen3.8-27B-DFlash2-EXL3-5.0bpw")) + Expect(cfg.Artifacts[1].Source.Revision).To(Equal("4f0436269bca761b071f05319e8e04a87cc633f9")) + }) +}) + +func contains(values []string, target string) bool { + for _, value := range values { + if value == target { + return true + } + } + return false +} diff --git a/core/gallery/inference_defaults_install_test.go b/core/gallery/inference_defaults_install_test.go new file mode 100644 index 000000000..11f46bc42 --- /dev/null +++ b/core/gallery/inference_defaults_install_test.go @@ -0,0 +1,189 @@ +package gallery_test + +import ( + "context" + "fmt" + "maps" + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gopkg.in/yaml.v3" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/gallery" + "github.com/mudler/LocalAI/pkg/modelartifacts" + "github.com/mudler/LocalAI/pkg/system" +) + +// The recommended sampling parameters for a model family are applied at install +// and persisted into the model YAML. Persisting them is only worth anything if +// they are written where the loader reads them back: PredictionOptions is nested +// under "parameters" in ModelConfig, so a top level "temperature" key parses +// without error and is then ignored for the life of the model. +// +// The expected values are read from the family table rather than written out +// here, so that retuning a family stays a one file change. +// +// Nothing here reaches the network. +var _ = Describe("Inference defaults persisted at install", func() { + var tempdir string + var galleries []config.Gallery + var systemState *system.SystemState + // The gallery listing is cached on the name and URL pair, so every spec + // needs a gallery of its own or it reads the previous spec's catalog. + galleryRevision := 0 + + // The name has to contain a pattern from inference_defaults.json, otherwise + // no defaults are applied and every assertion below passes vacuously. + const modelName = "qwen3.5-install-defaults" + + newGallery := func(entries ...gallery.GalleryModel) { + out, err := yaml.Marshal(entries) + Expect(err).ToNot(HaveOccurred()) + name := fmt.Sprintf("inference-defaults-%d", galleryRevision) + galleryRevision++ + galleryPath := filepath.Join(tempdir, name+".yaml") + Expect(os.WriteFile(galleryPath, out, 0600)).To(Succeed()) + galleries = []config.Gallery{{Name: name, URL: "file://" + galleryPath}} + } + + install := func(name string) error { + return gallery.InstallModelFromGallery( + context.TODO(), galleries, []config.Gallery{}, systemState, nil, + name, gallery.GalleryModel{}, func(string, string, string, float64) {}, false, false, false) + } + + installedConfig := func(name string) map[string]any { + dat, err := os.ReadFile(filepath.Join(tempdir, name+".yaml")) + Expect(err).ToNot(HaveOccurred()) + content := map[string]any{} + Expect(yaml.Unmarshal(dat, &content)).To(Succeed()) + return content + } + + // Seeding the weights keeps the install off the network: the downloader + // treats an already-present destination with no declared sha256 as fetched. + // extra goes into parameters:, so a spec can pin a value the defaults would + // otherwise supply. + seedGallery := func(extra map[string]any) { + Expect(os.WriteFile(filepath.Join(tempdir, "weights.gguf"), []byte("weights"), 0600)).To(Succeed()) + + params := map[string]any{"model": "weights.gguf"} + maps.Copy(params, extra) + + e := gallery.GalleryModel{Overrides: map[string]any{ + "backend": "llama-cpp", + "parameters": params, + }} + e.Name = modelName + e.AdditionalFiles = []gallery.File{{Filename: "weights.gguf", URI: "https://example.com/weights.gguf"}} + newGallery(e) + } + + // Guards the fixture itself. If the name stops matching a family the specs + // below would still pass while asserting nothing at all. + expectedFamily := func() map[string]float64 { + family := config.MatchModelFamily(modelName) + Expect(family).ToNot(BeEmpty(), "fixture name no longer matches a family in inference_defaults.json") + return family + } + + BeforeEach(func() { + var err error + tempdir, err = os.MkdirTemp("", "inference-defaults-install") + Expect(err).ToNot(HaveOccurred()) + DeferCleanup(func() { Expect(os.RemoveAll(tempdir)).To(Succeed()) }) + + systemState, err = system.GetSystemState(system.WithModelPath(tempdir)) + Expect(err).ToNot(HaveOccurred()) + }) + + It("writes them under parameters, where the loader reads them back", func() { + family := expectedFamily() + seedGallery(nil) + + Expect(install(modelName)).To(Succeed()) + + params, ok := installedConfig(modelName)["parameters"].(map[string]any) + Expect(ok).To(BeTrue(), "parameters should be a map") + + for key, want := range family { + Expect(params).To(HaveKey(key)) + Expect(params[key]).To(BeNumerically("==", want), "parameters.%s", key) + } + }) + + It("does not leave them at the top level, where they are ignored", func() { + family := expectedFamily() + seedGallery(nil) + + Expect(install(modelName)).To(Succeed()) + + cfg := installedConfig(modelName) + for key := range family { + Expect(cfg).ToNot(HaveKey(key), "%s at the top level is never read", key) + } + }) + + It("leaves a value the entry already sets alone", func() { + family := expectedFamily() + Expect(family).To(HaveKey("temperature")) + Expect(family["temperature"]).ToNot(BeNumerically("==", 0.05), "pick a value the family does not use") + + seedGallery(map[string]any{"temperature": 0.05}) + + Expect(install(modelName)).To(Succeed()) + + params, ok := installedConfig(modelName)["parameters"].(map[string]any) + Expect(ok).To(BeTrue(), "parameters should be a map") + Expect(params["temperature"]).To(BeNumerically("==", 0.05)) + }) + + // An entry that binds a primary artifact carries no files: of its own, so it + // takes the other branch of the install and none of the specs above reach it. + // It is also the one branch that already re-marshalled, which is why the + // defaults did land on disk there, at the top level where nothing reads them. + It("writes them under parameters on the artifact binding path too", func() { + family := expectedFamily() + + definition := &gallery.ModelConfig{ConfigFile: ` +backend: transformers +artifacts: + - name: model + target: model + source: + type: huggingface + repo: owner/repo +parameters: + model: owner/repo +`} + // Standing in for the materializer keeps the install off the network. + materializer := &fakeArtifactMaterializer{result: modelartifacts.Result{ + Spec: modelartifacts.Spec{ + Name: "model", Target: "model", + Source: modelartifacts.Source{Type: "huggingface", Repo: "owner/repo", Revision: "main"}, + Resolved: &modelartifacts.Resolved{ + Endpoint: "https://huggingface.co", + Revision: "0123456789abcdef0123456789abcdef01234567", + CacheKey: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + }, + }, + RelativePath: ".artifacts/huggingface/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef/snapshot", + }} + + _, err := gallery.InstallModel(context.TODO(), systemState, modelName, definition, nil, nil, false, + gallery.WithArtifactMaterializer(materializer)) + Expect(err).ToNot(HaveOccurred()) + + cfg := installedConfig(modelName) + params, ok := cfg["parameters"].(map[string]any) + Expect(ok).To(BeTrue(), "parameters should be a map") + for key, want := range family { + Expect(params).To(HaveKey(key)) + Expect(params[key]).To(BeNumerically("==", want), "parameters.%s", key) + Expect(cfg).ToNot(HaveKey(key), "%s at the top level is never read", key) + } + }) +}) diff --git a/core/gallery/models.go b/core/gallery/models.go index f3184648b..c787dcde4 100644 --- a/core/gallery/models.go +++ b/core/gallery/models.go @@ -622,35 +622,51 @@ func InstallModel(ctx context.Context, systemState *system.SystemState, nameOver lconfig.ApplyInferenceDefaults(&modelConfig, name, modelConfig.Model) // Merge inference defaults into configMap so they are persisted without losing unknown fields. - if modelConfig.Temperature != nil { - if _, exists := configMap["temperature"]; !exists { - configMap["temperature"] = *modelConfig.Temperature + // They belong under "parameters": ModelConfig embeds PredictionOptions with + // that yaml key, so a top level "temperature" parses without error and is + // then ignored for the life of the model. + params, mergeable := configMap["parameters"].(map[string]any) + if configMap["parameters"] == nil { + params, mergeable = map[string]any{}, true + } + if mergeable { + // An entry that sets one of these keeps its own value. ApplyInferenceDefaults + // already skipped those fields; this keeps the write side symmetric. + setDefault := func(key string, value any) { + if _, exists := params[key]; !exists { + params[key] = value + } + } + if modelConfig.Temperature != nil { + setDefault("temperature", *modelConfig.Temperature) + } + if modelConfig.TopP != nil { + setDefault("top_p", *modelConfig.TopP) + } + if modelConfig.TopK != nil { + setDefault("top_k", *modelConfig.TopK) + } + if modelConfig.MinP != nil { + setDefault("min_p", *modelConfig.MinP) + } + if modelConfig.RepeatPenalty != 0 { + setDefault("repeat_penalty", modelConfig.RepeatPenalty) + } + if modelConfig.PresencePenalty != 0 { + setDefault("presence_penalty", modelConfig.PresencePenalty) + } + if len(params) > 0 { + configMap["parameters"] = params } } - if modelConfig.TopP != nil { - if _, exists := configMap["top_p"]; !exists { - configMap["top_p"] = *modelConfig.TopP - } - } - if modelConfig.TopK != nil { - if _, exists := configMap["top_k"]; !exists { - configMap["top_k"] = *modelConfig.TopK - } - } - if modelConfig.MinP != nil { - if _, exists := configMap["min_p"]; !exists { - configMap["min_p"] = *modelConfig.MinP - } - } - if modelConfig.RepeatPenalty != 0 { - if _, exists := configMap["repeat_penalty"]; !exists { - configMap["repeat_penalty"] = modelConfig.RepeatPenalty - } - } - if modelConfig.PresencePenalty != 0 { - if _, exists := configMap["presence_penalty"]; !exists { - configMap["presence_penalty"] = modelConfig.PresencePenalty - } + + // The marshal above predates this merge, and the only other re-marshal is + // behind the artifact binding below, which an entry carrying files: never + // reaches. Without this the defaults are computed and then dropped on the + // way to disk. + updatedConfigYAML, err = yaml.Marshal(configMap) + if err != nil { + return nil, fmt.Errorf("failed to marshal config with inference defaults: %v", err) } if valid, err := modelConfig.Validate(); !valid { @@ -938,3 +954,73 @@ func SafetyScanGalleryModel(galleryModel *GalleryModel) error { } return nil } + +// ClusterResolveEnv describes a CLUSTER to variant selection, where +// HostResolveEnv describes one machine. +// +// It exists because a distributed controller is the wrong machine to ask. The +// controller is typically a GPU-less pod while every model actually runs on a +// worker, so a picker sized against it reports that a fleet of A100s can only +// run the smallest CPU build, and auto-selection then installs exactly that. +// +// availableMemory is the largest single healthy node's budget, and capabilities +// are the capability strings present in the cluster. Either may be empty: a +// zero memory reading keeps the host's own figure and an empty capability list +// keeps the host's own hardware verdict, so every degradation path lands back +// on the single-node behavior rather than on a cluster described as having +// nothing. +func ClusterResolveEnv(ctx context.Context, systemState *system.SystemState, availableMemory uint64, capabilities []string) ResolveEnv { + env := HostResolveEnv(ctx, systemState) + + if availableMemory > 0 { + env.AvailableMemory = availableMemory + } + if len(capabilities) == 0 { + return env + } + + // One state pinned per capability, mirroring AvailableBackendsForCapabilities: + // the controller's own detection must not leak into a worker's verdict, and + // a forced capability on the controller image must not either. + nodeStates := make([]*system.SystemState, 0, len(capabilities)) + for _, capability := range capabilities { + nodeStates = append(nodeStates, system.NewCapabilityState(capability, + system.WithBackendPath(systemState.Backend.BackendsPath))) + } + + hostCompatible := env.BackendCompatible + // A union, because a variant only has to run SOMEWHERE. The controller + // stays in the union so a cluster whose workers all went offline still + // describes itself the way it did before distributed mode existed. + env.BackendCompatible = func(backend string) bool { + if hostCompatible != nil && hostCompatible(backend) { + return true + } + for _, nodeState := range nodeStates { + if nodeState.IsBackendCompatible(backend, "") { + return true + } + } + return false + } + + // Ranking follows the same hardware as the filter. Left on the controller's + // tokens, an NVIDIA fleet would be offered the GGUF build over the vLLM one + // even though nothing filtered the vLLM build out. + seen := make(map[string]struct{}) + preference := make([]string, 0, len(nodeStates)) + for _, nodeState := range nodeStates { + for _, token := range nodeState.EnginePreferenceTokens() { + if _, dup := seen[token]; dup { + continue + } + seen[token] = struct{}{} + preference = append(preference, token) + } + } + if len(preference) > 0 { + env.EnginePreference = preference + } + + return env +} diff --git a/core/gallery/variants_lint_test.go b/core/gallery/variants_lint_test.go index c0cbad804..370cec8cc 100644 --- a/core/gallery/variants_lint_test.go +++ b/core/gallery/variants_lint_test.go @@ -540,6 +540,29 @@ var _ = Describe("gallery/index.yaml Higgs Audio entry", func() { }) }) +var _ = Describe("gallery/index.yaml qwythos-9b-claude-mythos-5-1m mmproj", func() { + It("points at the published F16 mmproj artifact", func() { + entries, err := loadGalleryIndex() + Expect(err).ToNot(HaveOccurred()) + + models := make([]*gallery.GalleryModel, 0, len(entries)) + for i := range entries { + models = append(models, &entries[i]) + } + entry := gallery.FindGalleryElement(models, "qwythos-9b-claude-mythos-5-1m") + Expect(entry).ToNot(BeNil()) + Expect(entry.Overrides).To(HaveKeyWithValue( + "mmproj", + "llama-cpp/mmproj/Qwythos-9B-Claude-Mythos-5-1M-GGUF/mmproj-Qwythos-9B-Claude-Mythos-5-1M-F16.gguf", + )) + Expect(entry.AdditionalFiles).To(ContainElement(gallery.File{ + Filename: "llama-cpp/mmproj/Qwythos-9B-Claude-Mythos-5-1M-GGUF/mmproj-Qwythos-9B-Claude-Mythos-5-1M-F16.gguf", + SHA256: "f977efc337a2ac2ba183eea0c73e25b75fc240d56c05ed4d9b56ab451f64c82c", + URI: "https://huggingface.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF/resolve/main/mmproj-Qwythos-9B-Claude-Mythos-5-1M-F16.gguf", + })) + }) +}) + // The lint rules above check the catalog as text. This drives the real // resolution path for the entry a user actually clicked and failed to install, // so the fix is proven at the layer that broke and not only at the layer that diff --git a/core/http/endpoints/localai/backend.go b/core/http/endpoints/localai/backend.go index 4083333c9..d8a5612a1 100644 --- a/core/http/endpoints/localai/backend.go +++ b/core/http/endpoints/localai/backend.go @@ -356,7 +356,7 @@ func (mgs *BackendEndpointService) UpgradeBackendEndpoint() echo.HandlerFunc { // local system state is the only thing worth filtering against. type ClusterCapabilityProvider func(ctx context.Context) ([]string, error) -// resolveClusterCapabilities reads the capabilities present in the cluster, +// ResolveClusterCapabilities reads the capabilities present in the cluster, // degrading to the local-only listing on error. // // Every capability-filtered discovery endpoint shares this: on a distributed @@ -364,7 +364,7 @@ type ClusterCapabilityProvider func(ctx context.Context) ([]string, error) // (usually GPU-less) host hides GPU-only backends the cluster can actually // run. A registry hiccup must never blank the catalog, so a failure falls back // to the pre-existing local-only behavior rather than erroring the request. -func resolveClusterCapabilities(ctx context.Context, provider ClusterCapabilityProvider) []string { +func ResolveClusterCapabilities(ctx context.Context, provider ClusterCapabilityProvider) []string { if provider == nil { return nil } @@ -423,7 +423,7 @@ func installedInCluster(backend *gallery.GalleryBackend, clusterInstalled map[st // @Router /backends/available [get] func (mgs *BackendEndpointService) ListAvailableBackendsEndpoint(systemState *system.SystemState, clusterCapabilities ClusterCapabilityProvider, clusterInstalled ClusterInstalledProvider) echo.HandlerFunc { return func(c echo.Context) error { - capabilities := resolveClusterCapabilities(c.Request().Context(), clusterCapabilities) + capabilities := ResolveClusterCapabilities(c.Request().Context(), clusterCapabilities) backends, err := gallery.AvailableBackendsForCapabilities(mgs.galleries, systemState, capabilities) if err != nil { diff --git a/core/http/endpoints/localai/face_register.go b/core/http/endpoints/localai/face_register.go index fbeb29e0c..9cd40b456 100644 --- a/core/http/endpoints/localai/face_register.go +++ b/core/http/endpoints/localai/face_register.go @@ -1,6 +1,7 @@ package localai import ( + "errors" "net/http" "github.com/labstack/echo/v4" @@ -33,22 +34,31 @@ func FaceRegisterEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, a return echo.NewHTTPError(http.StatusBadRequest, "name is required") } - img, err := decodeImageInput(input.Img) - if err != nil { - return err + if (input.Img == "") == (len(input.Embedding) == 0) { + return echo.NewHTTPError(http.StatusBadRequest, "provide exactly one of img or embedding") } - - xlog.Debug("FaceRegister", "model", cfg.Name, "name", input.Name) - embedding, err := backend.FaceEmbed(c.Request().Context(), img, ml, appConfig, *cfg) - if err != nil { - return mapBackendError(err) + embedding := input.Embedding + if len(embedding) == 0 { + img, err := decodeImageInput(input.Img) + if err != nil { + return err + } + xlog.Debug("FaceRegister", "model", cfg.Name, "name", input.Name) + embedding, err = backend.FaceEmbed(c.Request().Context(), img, ml, appConfig, *cfg) + if err != nil { + return mapBackendError(err) + } } stored, err := registry.Register(c.Request().Context(), embedding, facerecognition.Metadata{ - Name: input.Name, - Labels: input.Labels, + Name: input.Name, + RegisteredAt: input.RegisteredAt, + Labels: input.Labels, }) if err != nil { + if errors.Is(err, facerecognition.ErrInvalidEmbedding) || errors.Is(err, facerecognition.ErrDimensionMismatch) { + return echo.NewHTTPError(http.StatusBadRequest, err.Error()) + } return err } return c.JSON(http.StatusOK, schema.FaceRegisterResponse{ diff --git a/core/http/endpoints/localai/face_register_test.go b/core/http/endpoints/localai/face_register_test.go new file mode 100644 index 000000000..16970345a --- /dev/null +++ b/core/http/endpoints/localai/face_register_test.go @@ -0,0 +1,80 @@ +// SPDX-License-Identifier: MIT + +package localai_test + +import ( + "context" + "net/http" + "net/http/httptest" + "time" + + "github.com/labstack/echo/v4" + "github.com/mudler/LocalAI/core/config" + . "github.com/mudler/LocalAI/core/http/endpoints/localai" + "github.com/mudler/LocalAI/core/http/middleware" + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/core/services/facerecognition" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +type registrationRecorder struct { + facerecognition.Registry + vector []float32 + meta facerecognition.Metadata + err error +} + +func (r *registrationRecorder) Register(_ context.Context, v []float32, m facerecognition.Metadata) (facerecognition.Metadata, error) { + r.vector = v + r.meta = m + m.ID = "saved-id" + return m, r.err +} + +var _ = Describe("Face registration replay", func() { + var reg *registrationRecorder + call := func(in schema.FaceRegisterRequest) (*httptest.ResponseRecorder, error) { + e := echo.New() + rec := httptest.NewRecorder() + c := e.NewContext(httptest.NewRequest(http.MethodPost, "/v1/face/register", nil), rec) + c.Set(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST, &in) + c.Set(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG, &config.ModelConfig{}) + // No model loader: replay must not call the embedding backend. + err := FaceRegisterEndpoint(nil, nil, nil, reg)(c) + return rec, err + } + BeforeEach(func() { reg = ®istrationRecorder{} }) + It("accepts the saved vector and timestamp without running inference", func() { + at := time.Now().UTC() + in := schema.FaceRegisterRequest{Name: "Alice", Embedding: []float32{1, 0}, RegisteredAt: at, Labels: map[string]string{"client_id": "alice"}} + in.Model = "faces" + rec, err := call(in) + Expect(err).NotTo(HaveOccurred()) + Expect(rec.Code).To(Equal(http.StatusOK)) + Expect(reg.vector).To(Equal(in.Embedding)) + Expect(reg.meta.RegisteredAt).To(Equal(at)) + Expect(reg.meta.Labels).To(Equal(in.Labels)) + Expect(rec.Body.String()).To(ContainSubstring("saved-id")) + }) + It("rejects ambiguous and missing inputs before inference", func() { + for _, in := range []schema.FaceRegisterRequest{ + {Name: "Alice"}, + {Name: "Alice", Img: "image", Embedding: []float32{1, 0}}, + } { + in.Model = "faces" + _, err := call(in) + Expect(err).To(HaveOccurred()) + Expect(err.(*echo.HTTPError).Code).To(Equal(http.StatusBadRequest)) + Expect(reg.vector).To(BeNil()) + } + }) + It("reports invalid vectors as a client error", func() { + reg.err = facerecognition.ErrInvalidEmbedding + in := schema.FaceRegisterRequest{Name: "Alice", Embedding: []float32{0, 0}} + in.Model = "faces" + _, err := call(in) + Expect(err).To(HaveOccurred()) + Expect(err.(*echo.HTTPError).Code).To(Equal(http.StatusBadRequest)) + }) +}) diff --git a/core/http/endpoints/localai/finetune.go b/core/http/endpoints/localai/finetune.go index 4948b65fc..2ddf1bcff 100644 --- a/core/http/endpoints/localai/finetune.go +++ b/core/http/endpoints/localai/finetune.go @@ -276,7 +276,7 @@ func DownloadExportedModelEndpoint(ftService *finetune.FineTuneService) echo.Han // ListFineTuneBackendsEndpoint returns installed backends tagged with "fine-tuning". func ListFineTuneBackendsEndpoint(appConfig *config.ApplicationConfig, clusterCapabilities ClusterCapabilityProvider, clusterInstalled ClusterInstalledProvider) echo.HandlerFunc { return func(c echo.Context) error { - capabilities := resolveClusterCapabilities(c.Request().Context(), clusterCapabilities) + capabilities := ResolveClusterCapabilities(c.Request().Context(), clusterCapabilities) installed := resolveClusterInstalled(c.Request().Context(), clusterInstalled) backends, err := gallery.AvailableBackendsForCapabilities(appConfig.BackendGalleries, appConfig.SystemState, capabilities) if err != nil { diff --git a/core/http/endpoints/localai/nodes.go b/core/http/endpoints/localai/nodes.go index bc26baf49..bbae523b1 100644 --- a/core/http/endpoints/localai/nodes.go +++ b/core/http/endpoints/localai/nodes.go @@ -1218,6 +1218,20 @@ func SetSchedulingEndpoint(registry *nodes.NodeRegistry) echo.HandlerFunc { return c.JSON(http.StatusBadRequest, nodeError(http.StatusBadRequest, err.Error())) } + // A rule may be keyed by an alias, in which case it governs whatever + // that alias currently points at. Reject an alias that resolves to + // nothing, and reject a second rule for a model some other rule already + // governs, so the operator hears about the clash instead of silently + // writing a rule that never takes effect. + target, err := registry.ValidateSchedulingTarget(ctx, req.ModelName) + if err != nil { + status := http.StatusBadRequest + if errors.Is(err, nodes.ErrSchedulingConflict) { + status = http.StatusConflict + } + return c.JSON(status, nodeError(status, err.Error())) + } + // Serialize node selector to JSON var selectorJSON string if len(req.NodeSelector) > 0 { @@ -1230,6 +1244,7 @@ func SetSchedulingEndpoint(registry *nodes.NodeRegistry) echo.HandlerFunc { config := &nodes.ModelSchedulingConfig{ ModelName: req.ModelName, + TargetModel: target, NodeSelector: selectorJSON, MinReplicas: req.MinReplicas, MaxReplicas: req.MaxReplicas, diff --git a/core/http/endpoints/localai/nodes_scheduling_alias_test.go b/core/http/endpoints/localai/nodes_scheduling_alias_test.go new file mode 100644 index 000000000..35065f614 --- /dev/null +++ b/core/http/endpoints/localai/nodes_scheduling_alias_test.go @@ -0,0 +1,125 @@ +package localai + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + + "github.com/labstack/echo/v4" + "github.com/mudler/LocalAI/core/services/nodes" + "github.com/mudler/LocalAI/core/services/testutil" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// aliasResolverStub maps alias names to targets in place of a config loader. +type aliasResolverStub struct{ aliases map[string]string } + +func (s *aliasResolverStub) ResolveAliasName(name string) (string, bool) { + target, ok := s.aliases[name] + if !ok { + return name, false + } + return target, true +} + +var _ = Describe("Scheduling endpoints with model aliases", func() { + var ( + registry *nodes.NodeRegistry + resolver *aliasResolverStub + ) + + BeforeEach(func() { + db := testutil.SetupTestDB() + var err error + registry, err = nodes.NewNodeRegistry(db) + Expect(err).ToNot(HaveOccurred()) + resolver = &aliasResolverStub{aliases: map[string]string{"production": "qwen3"}} + registry.SetAliasResolver(resolver) + }) + + post := func(body string) *httptest.ResponseRecorder { + e := echo.New() + req := httptest.NewRequest(http.MethodPost, "/", strings.NewReader(body)) + req.Header.Set(echo.HeaderContentType, echo.MIMEApplicationJSON) + rec := httptest.NewRecorder() + c := e.NewContext(req, rec) + ExpectWithOffset(1, SetSchedulingEndpoint(registry)(c)).To(Succeed()) + return rec + } + + It("accepts a rule keyed by an alias and reports the model it governs", func() { + rec := post(`{"model_name":"production","min_replicas":2,"node_selector":{"tier":"gpu"}}`) + Expect(rec.Code).To(Equal(http.StatusOK)) + + var resp map[string]any + Expect(json.Unmarshal(rec.Body.Bytes(), &resp)).To(Succeed()) + Expect(resp["model_name"]).To(Equal("production")) + Expect(resp["target_model"]).To(Equal("qwen3")) + }) + + It("rejects a second rule for a model an alias rule already governs", func() { + Expect(post(`{"model_name":"production","min_replicas":2}`).Code).To(Equal(http.StatusOK)) + + rec := post(`{"model_name":"qwen3","min_replicas":1}`) + Expect(rec.Code).To(Equal(http.StatusConflict)) + Expect(rec.Body.String()).To(ContainSubstring("production")) + }) + + It("rejects an alias rule for a model that already has its own rule", func() { + Expect(post(`{"model_name":"qwen3","min_replicas":1}`).Code).To(Equal(http.StatusOK)) + + rec := post(`{"model_name":"production","min_replicas":2}`) + Expect(rec.Code).To(Equal(http.StatusConflict)) + Expect(rec.Body.String()).To(ContainSubstring("qwen3")) + }) + + It("still allows editing a rule in place", func() { + Expect(post(`{"model_name":"production","min_replicas":2}`).Code).To(Equal(http.StatusOK)) + + rec := post(`{"model_name":"production","min_replicas":4}`) + Expect(rec.Code).To(Equal(http.StatusOK)) + + stored, err := registry.GetModelScheduling(context.Background(), "production") + Expect(err).ToNot(HaveOccurred()) + Expect(stored.MinReplicas).To(Equal(4)) + }) + + It("rejects a rule keyed by an alias that does not resolve", func() { + resolver.aliases["orphan"] = "orphan" + + rec := post(`{"model_name":"orphan","min_replicas":1}`) + Expect(rec.Code).To(Equal(http.StatusBadRequest)) + Expect(rec.Body.String()).To(ContainSubstring("does not resolve")) + }) + + It("still accepts a rule for a model that is not installed yet", func() { + rec := post(`{"model_name":"not-installed-yet","min_replicas":1}`) + Expect(rec.Code).To(Equal(http.StatusOK)) + }) + + It("labels a rule that another rule shadows when listing", func() { + // A seed file or a repointed alias can leave two rules on one model, + // which the write path above rejects but cannot retract. + Expect(registry.SetModelScheduling(context.Background(), &nodes.ModelSchedulingConfig{ModelName: "production", MinReplicas: 2})).To(Succeed()) + Expect(registry.SetModelScheduling(context.Background(), &nodes.ModelSchedulingConfig{ModelName: "qwen3", MinReplicas: 1})).To(Succeed()) + + e := echo.New() + req := httptest.NewRequest(http.MethodGet, "/", nil) + rec := httptest.NewRecorder() + c := e.NewContext(req, rec) + Expect(ListSchedulingEndpoint(registry)(c)).To(Succeed()) + + var listed []map[string]any + Expect(json.Unmarshal(rec.Body.Bytes(), &listed)).To(Succeed()) + byName := map[string]map[string]any{} + for _, item := range listed { + byName[item["model_name"].(string)] = item + } + Expect(byName["qwen3"]["shadowed"]).To(BeNil()) + Expect(byName["production"]["shadowed"]).To(Equal(true)) + }) +}) diff --git a/core/http/endpoints/localai/quantization.go b/core/http/endpoints/localai/quantization.go index 175bf9740..ae6c91739 100644 --- a/core/http/endpoints/localai/quantization.go +++ b/core/http/endpoints/localai/quantization.go @@ -195,7 +195,7 @@ func DownloadQuantizedModelEndpoint(qService *quantization.QuantizationService) // ListQuantizationBackendsEndpoint returns installed backends tagged with "quantization". func ListQuantizationBackendsEndpoint(appConfig *config.ApplicationConfig, clusterCapabilities ClusterCapabilityProvider, clusterInstalled ClusterInstalledProvider) echo.HandlerFunc { return func(c echo.Context) error { - capabilities := resolveClusterCapabilities(c.Request().Context(), clusterCapabilities) + capabilities := ResolveClusterCapabilities(c.Request().Context(), clusterCapabilities) installed := resolveClusterInstalled(c.Request().Context(), clusterInstalled) backends, err := gallery.AvailableBackendsForCapabilities(appConfig.BackendGalleries, appConfig.SystemState, capabilities) if err != nil { diff --git a/core/http/endpoints/ollama/models_test.go b/core/http/endpoints/ollama/models_test.go index b13cf59a0..c4d0d6b5e 100644 --- a/core/http/endpoints/ollama/models_test.go +++ b/core/http/endpoints/ollama/models_test.go @@ -146,6 +146,20 @@ parameters: Expect(resp.Details.Format).To(Equal("gguf")) Expect(resp.Details.Families).ToNot(BeEmpty()) }) + + It("looks up the model when the Ollama :latest tag is included", func() { + writeConfig("chat", ` +name: chat +backend: llama-cpp +template: + chat: "{{ .Input }}" +parameters: + model: Llama-3-8B-Q4_K_M.gguf +`) + resp := callShow("chat:latest") + Expect(resp.Details.Format).To(Equal("gguf")) + Expect(resp.Capabilities).To(ContainElement("completion")) + }) }) Describe("ListModelsEndpoint", func() { diff --git a/core/http/endpoints/openai/chat.go b/core/http/endpoints/openai/chat.go index fcec18932..72fe59b0e 100644 --- a/core/http/endpoints/openai/chat.go +++ b/core/http/endpoints/openai/chat.go @@ -4,6 +4,7 @@ import ( "encoding/json" "fmt" "net/http" + "strings" "time" "github.com/google/uuid" @@ -24,18 +25,47 @@ import ( "github.com/mudler/xlog" ) +// messageText returns the textual content of a message, preferring the +// middleware-populated StringContent and falling back to a string Content. +func messageText(m schema.Message) string { + if m.StringContent != "" { + return m.StringContent + } + if s, ok := m.Content.(string); ok { + return s + } + return "" +} + // hasSystemMessage reports whether the message slice already contains a -// system-role message — used to avoid clobbering a caller-supplied system -// prompt when the LocalAI Assistant modality is on. +// non-empty system-role message — used to avoid clobbering a caller-supplied +// system prompt when the LocalAI Assistant modality is on. Empty / whitespace +// system turns (historically sent by the web Chat UI) are ignored so they do +// not suppress the model config system_prompt. func hasSystemMessage(messages []schema.Message) bool { for _, m := range messages { - if m.Role == "system" { + if m.Role == "system" && strings.TrimSpace(messageText(m)) != "" { return true } } return false } +// stripEmptySystemMessages drops system-role messages whose content is empty +// or whitespace-only. An explicit blank system turn would otherwise satisfy +// tokenizer chat templates' `messages[0].role == "system"` check and suppress +// both the model's configured system_prompt and any template default. +func stripEmptySystemMessages(messages []schema.Message) []schema.Message { + out := messages[:0:0] + for _, m := range messages { + if m.Role == "system" && strings.TrimSpace(messageText(m)) == "" { + continue + } + out = append(out, m) + } + return out +} + // mergeToolCallDeltas merges streaming tool call deltas into complete tool calls. // In SSE streaming, a single tool call arrives as multiple chunks sharing the same Index: // the first chunk carries the ID, Type, and Name; subsequent chunks append to Arguments. @@ -149,6 +179,18 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator xlog.Debug("Chat endpoint configuration read", "config", config) + // Drop blank system turns from the web UI (and similar clients) so they + // cannot suppress the model YAML system_prompt / tokenizer defaults. + input.Messages = stripEmptySystemMessages(input.Messages) + + // Tokenizer-template models pass messages through to the backend as-is, + // so apply the configured system_prompt when the request did not supply + // one. Go-template models already receive SystemPrompt via PromptTemplateData. + if config.TemplateConfig.UseTokenizerTemplate && config.SystemPrompt != "" && !hasSystemMessage(input.Messages) { + prompt := config.SystemPrompt + input.Messages = append([]schema.Message{{Role: "system", Content: prompt, StringContent: prompt}}, input.Messages...) + } + // Cloud-proxy bail. Bypasses the local pipeline (templating, // MCP injection, gRPC backend) and forwards via the cloud- // proxy backend, which does the outbound HTTP. Request-side PII diff --git a/core/http/endpoints/openai/chat_test.go b/core/http/endpoints/openai/chat_test.go index ffb3086fb..0ef0991dd 100644 --- a/core/http/endpoints/openai/chat_test.go +++ b/core/http/endpoints/openai/chat_test.go @@ -357,3 +357,47 @@ var _ = Describe("mergeToolCallDeltas", func() { }) }) }) + +var _ = Describe("system message helpers", func() { + Describe("hasSystemMessage", func() { + It("ignores empty and whitespace-only system turns", func() { + Expect(hasSystemMessage([]schema.Message{ + {Role: "system", Content: "", StringContent: ""}, + {Role: "user", Content: "hi", StringContent: "hi"}, + })).To(BeFalse()) + Expect(hasSystemMessage([]schema.Message{ + {Role: "system", Content: " ", StringContent: " "}, + })).To(BeFalse()) + }) + + It("detects a real system prompt", func() { + Expect(hasSystemMessage([]schema.Message{ + {Role: "system", Content: "You are helpful.", StringContent: "You are helpful."}, + {Role: "user", Content: "hi", StringContent: "hi"}, + })).To(BeTrue()) + }) + }) + + Describe("stripEmptySystemMessages", func() { + It("removes blank system turns and keeps the rest", func() { + in := []schema.Message{ + {Role: "system", Content: "", StringContent: ""}, + {Role: "system", Content: " ", StringContent: " "}, + {Role: "user", Content: "Explain how this works", StringContent: "Explain how this works"}, + } + out := stripEmptySystemMessages(in) + Expect(out).To(HaveLen(1)) + Expect(out[0].Role).To(Equal("user")) + }) + + It("keeps a non-empty system turn", func() { + in := []schema.Message{ + {Role: "system", Content: "You are LocalAI.", StringContent: "You are LocalAI."}, + {Role: "user", Content: "hi", StringContent: "hi"}, + } + out := stripEmptySystemMessages(in) + Expect(out).To(HaveLen(2)) + Expect(out[0].StringContent).To(Equal("You are LocalAI.")) + }) + }) +}) diff --git a/core/http/endpoints/openai/realtime_webrtc.go b/core/http/endpoints/openai/realtime_webrtc.go index 4f13862c8..eca3aa2c6 100644 --- a/core/http/endpoints/openai/realtime_webrtc.go +++ b/core/http/endpoints/openai/realtime_webrtc.go @@ -1,6 +1,9 @@ package openai import ( + "encoding/json" + "io" + "mime" "net/http" "time" @@ -27,6 +30,61 @@ type RealtimeCallResponse struct { SessionID string `json:"session_id"` } +func decodeRealtimeCallRequest(c echo.Context) (RealtimeCallRequest, bool, error) { + var req RealtimeCallRequest + mediaType := "" + contentType := c.Request().Header.Get(echo.HeaderContentType) + if contentType != "" { + var err error + mediaType, _, err = mime.ParseMediaType(contentType) + if err != nil { + return req, false, err + } + } + + switch mediaType { + case echo.MIMEMultipartForm: + if err := c.Request().ParseMultipartForm(32 << 20); err != nil { + return req, true, err + } + req.SDP = c.FormValue("sdp") + var session struct { + Model string `json:"model"` + LocalAIAssistant bool `json:"localai_assistant,omitempty"` + } + if err := json.Unmarshal([]byte(c.FormValue("session")), &session); err != nil { + return req, true, err + } + req.Model = session.Model + req.LocalAIAssistant = session.LocalAIAssistant + return req, true, nil + case "application/sdp": + sdp, err := readRealtimeSDP(c.Request().Body) + req.SDP = sdp + req.Model = c.QueryParam("model") + return req, true, err + default: + err := c.Bind(&req) + return req, false, err + } +} + +func readRealtimeSDP(body io.Reader) (string, error) { + data, err := io.ReadAll(body) + return string(data), err +} + +func writeRealtimeCallResponse(c echo.Context, plainSDPResponse bool, sdp, sessionID string) error { + if plainSDPResponse { + return c.Blob(http.StatusCreated, "application/sdp", []byte(sdp)) + } + + return c.JSON(http.StatusCreated, RealtimeCallResponse{ + SDP: sdp, + SessionID: sessionID, + }) +} + // RealtimeCalls handles POST /v1/realtime/calls for WebRTC signaling. func RealtimeCalls(application *application.Application) echo.HandlerFunc { se, settingEngineErr := webRTCSettingEngine(application.ApplicationConfig()) @@ -38,8 +96,8 @@ func RealtimeCalls(application *application.Application) echo.HandlerFunc { if settingEngineErr != nil { return c.JSON(http.StatusInternalServerError, map[string]string{"error": settingEngineErr.Error()}) } - var req RealtimeCallRequest - if err := c.Bind(&req); err != nil { + req, plainSDPResponse, err := decodeRealtimeCallRequest(c) + if err != nil { return c.JSON(http.StatusBadRequest, map[string]string{"error": "invalid request body"}) } if req.SDP == "" { @@ -189,10 +247,7 @@ func RealtimeCalls(application *application.Application) echo.HandlerFunc { runRealtimeSession(application, transport, req.Model, evaluator, opts) }() - return c.JSON(http.StatusCreated, RealtimeCallResponse{ - SDP: localDesc.SDP, - SessionID: sessionID, - }) + return writeRealtimeCallResponse(c, plainSDPResponse, localDesc.SDP, sessionID) } } diff --git a/core/http/endpoints/openai/realtime_webrtc_request_test.go b/core/http/endpoints/openai/realtime_webrtc_request_test.go new file mode 100644 index 000000000..187555329 --- /dev/null +++ b/core/http/endpoints/openai/realtime_webrtc_request_test.go @@ -0,0 +1,91 @@ +package openai + +import ( + "bytes" + "mime/multipart" + "net/http" + "net/http/httptest" + "net/textproto" + + "github.com/labstack/echo/v4" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("decodeRealtimeCallRequest", func() { + It("decodes the legacy JSON request", func() { + request := httptest.NewRequest(http.MethodPost, "/v1/realtime/calls", bytes.NewBufferString(`{"sdp":"offer","model":"voice","localai_assistant":true}`)) + request.Header.Set(echo.HeaderContentType, echo.MIMEApplicationJSON) + + req, plainSDPResponse, err := decodeRealtimeCallRequest(echo.New().NewContext(request, httptest.NewRecorder())) + + Expect(err).NotTo(HaveOccurred()) + Expect(req).To(Equal(RealtimeCallRequest{SDP: "offer", Model: "voice", LocalAIAssistant: true})) + Expect(plainSDPResponse).To(BeFalse()) + }) + + It("decodes the OpenAI multipart request", func() { + var body bytes.Buffer + writer := multipart.NewWriter(&body) + sdpHeader := make(textproto.MIMEHeader) + sdpHeader.Set("Content-Disposition", `form-data; name="sdp"`) + sdpHeader.Set("Content-Type", "application/sdp") + sdpPart, err := writer.CreatePart(sdpHeader) + Expect(err).NotTo(HaveOccurred()) + _, err = sdpPart.Write([]byte("offer")) + Expect(err).NotTo(HaveOccurred()) + sessionHeader := make(textproto.MIMEHeader) + sessionHeader.Set("Content-Disposition", `form-data; name="session"`) + sessionHeader.Set("Content-Type", echo.MIMEApplicationJSON) + sessionPart, err := writer.CreatePart(sessionHeader) + Expect(err).NotTo(HaveOccurred()) + _, err = sessionPart.Write([]byte(`{"type":"realtime","model":"voice","localai_assistant":true}`)) + Expect(err).NotTo(HaveOccurred()) + Expect(writer.Close()).To(Succeed()) + + request := httptest.NewRequest(http.MethodPost, "/v1/realtime/calls", &body) + request.Header.Set(echo.HeaderContentType, writer.FormDataContentType()) + req, plainSDPResponse, err := decodeRealtimeCallRequest(echo.New().NewContext(request, httptest.NewRecorder())) + + Expect(err).NotTo(HaveOccurred()) + Expect(req).To(Equal(RealtimeCallRequest{SDP: "offer", Model: "voice", LocalAIAssistant: true})) + Expect(plainSDPResponse).To(BeTrue()) + }) + + It("decodes a raw SDP request with the model query parameter", func() { + request := httptest.NewRequest(http.MethodPost, "/v1/realtime/calls?model=voice", bytes.NewBufferString("offer")) + request.Header.Set(echo.HeaderContentType, "application/sdp") + + req, plainSDPResponse, err := decodeRealtimeCallRequest(echo.New().NewContext(request, httptest.NewRecorder())) + + Expect(err).NotTo(HaveOccurred()) + Expect(req).To(Equal(RealtimeCallRequest{SDP: "offer", Model: "voice"})) + Expect(plainSDPResponse).To(BeTrue()) + }) +}) + +var _ = Describe("writeRealtimeCallResponse", func() { + It("writes the bare SDP answer for OpenAI request formats", func() { + response := httptest.NewRecorder() + request := httptest.NewRequest(http.MethodPost, "/v1/realtime/calls", nil) + context := echo.New().NewContext(request, response) + + Expect(writeRealtimeCallResponse(context, true, "answer", "session-id")).To(Succeed()) + + Expect(response.Code).To(Equal(http.StatusCreated)) + Expect(response.Header().Get(echo.HeaderContentType)).To(Equal("application/sdp")) + Expect(response.Body.String()).To(Equal("answer")) + }) + + It("preserves the JSON response for legacy requests", func() { + response := httptest.NewRecorder() + request := httptest.NewRequest(http.MethodPost, "/v1/realtime/calls", nil) + context := echo.New().NewContext(request, response) + + Expect(writeRealtimeCallResponse(context, false, "answer", "session-id")).To(Succeed()) + + Expect(response.Code).To(Equal(http.StatusCreated)) + Expect(response.Header().Get(echo.HeaderContentType)).To(Equal(echo.MIMEApplicationJSON)) + Expect(response.Body.String()).To(MatchJSON(`{"sdp":"answer","session_id":"session-id"}`)) + }) +}) diff --git a/core/http/middleware/request.go b/core/http/middleware/request.go index 1599ef05c..080a0b73c 100644 --- a/core/http/middleware/request.go +++ b/core/http/middleware/request.go @@ -141,6 +141,12 @@ func (re *RequestExtractor) SetModelAndConfig(initializer func() schema.LocalAIR } modelName := input.ModelName(nil) + // Ollama-compat /api/tags appends ":latest" to untagged names. + // Strip it for lookup so the listed name works on /api/chat, + // /v1/chat/completions, and the other model-bearing endpoints. + if strings.HasSuffix(modelName, ":latest") { + modelName = strings.TrimSuffix(modelName, ":latest") + } cfg, err := re.modelConfigLoader.LoadModelConfigFileByNameDefaultOptions(modelName, re.applicationConfig) if err != nil { diff --git a/core/http/middleware/request_test.go b/core/http/middleware/request_test.go index 1b00c7f02..afaf8d9c8 100644 --- a/core/http/middleware/request_test.go +++ b/core/http/middleware/request_test.go @@ -82,6 +82,13 @@ var _ = Describe("SetModelAndConfig middleware", func() { Expect(resp.Error.Message).To(ContainSubstring("not found")) Expect(resp.Error.Type).To(Equal("invalid_request_error")) }) + + It("still 404s when :latest is appended to an unknown model", func() { + rec := postJSON(app, "/v1/chat/completions", + `{"model":"nonexistent-model:latest","messages":[{"role":"user","content":"hi"}]}`) + + Expect(rec.Code).To(Equal(http.StatusNotFound)) + }) }) Context("when the model exists as a config file", func() { @@ -97,6 +104,13 @@ var _ = Describe("SetModelAndConfig middleware", func() { Expect(rec.Code).To(Equal(http.StatusOK)) }) + + It("accepts the Ollama :latest tag that /api/tags appends", func() { + rec := postJSON(app, "/v1/chat/completions", + `{"model":"test-model:latest","messages":[{"role":"user","content":"hi"}]}`) + + Expect(rec.Code).To(Equal(http.StatusOK)) + }) }) Context("when the model exists as a pre-loaded config", func() { diff --git a/core/http/react-ui/bun.lock b/core/http/react-ui/bun.lock index f0b034a69..4d9ce9572 100644 --- a/core/http/react-ui/bun.lock +++ b/core/http/react-ui/bun.lock @@ -53,6 +53,7 @@ "overrides": { "hono": "4.12.34", "ip-address": "10.3.1", + "path-to-regexp": "^8.4.0", }, "packages": { "@babel/code-frame": ["@babel/code-frame@7.29.0", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.28.5", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw=="], @@ -807,7 +808,7 @@ "path-scurry": ["path-scurry@2.0.2", "", { "dependencies": { "lru-cache": "^11.0.0", "minipass": "^7.1.2" } }, "sha512-3O/iVVsJAPsOnpwWIeD+d6z/7PmqApyQePUtCndjatj/9I5LylHvt5qluFaBT3I5h3r1ejfR056c+FCv+NnNXg=="], - "path-to-regexp": ["path-to-regexp@8.3.0", "", {}, "sha512-7jdwVIRtsP8MYpdXSwOS0YdD0Du+qOoF/AEPIt88PcCFrZCzx41oxku1jD88hZBwbNUIEfpqvuhjFaMAqMTWnA=="], + "path-to-regexp": ["path-to-regexp@8.4.0", "", {}, "sha512-PuseHIvAnz3bjrM2rGJtSgo1zjgxapTLZ7x2pjhzWwlp4SJQgK3f3iZIQwkpEnBaKz6seKBADpM4B4ySkuYypg=="], "picocolors": ["picocolors@1.1.1", "", {}, "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA=="], diff --git a/core/http/react-ui/e2e/collections.spec.js b/core/http/react-ui/e2e/collections.spec.js index 4fa4168dd..f3f4bd8c2 100644 --- a/core/http/react-ui/e2e/collections.spec.js +++ b/core/http/react-ui/e2e/collections.spec.js @@ -19,4 +19,33 @@ test.describe('Collections page', () => { await input.fill('my-kb') await expect(input).toHaveValue('my-kb') }) + + test('posts the source update interval as a JSON number', async ({ page }) => { + const collectionName = 'interval-regression' + const collectionPath = encodeURIComponent(collectionName) + let postedBody + + await page.route(`**/api/agents/collections/${collectionPath}/entries`, route => + route.fulfill({ contentType: 'application/json', body: JSON.stringify({ entries: [] }) })) + await page.route(`**/api/agents/collections/${collectionPath}/sources`, async route => { + if (route.request().method() === 'POST') { + postedBody = route.request().postDataJSON() + await route.fulfill({ contentType: 'application/json', body: JSON.stringify({ status: 'ok' }) }) + } else { + await route.fulfill({ contentType: 'application/json', body: JSON.stringify({ sources: [] }) }) + } + }) + + await page.goto(`/app/collections/${collectionPath}`) + await page.getByRole('button', { name: 'Sources' }).click() + await page.locator('#source-url').fill('https://example.com/feed') + await page.locator('#source-interval').fill('3600') + await page.getByRole('button', { name: 'Add Source' }).click() + + await expect.poll(() => postedBody).toEqual({ + url: 'https://example.com/feed', + update_interval: 3600, + }) + expect(typeof postedBody.update_interval).toBe('number') + }) }) diff --git a/core/http/react-ui/e2e/models-cluster-fit.spec.js b/core/http/react-ui/e2e/models-cluster-fit.spec.js new file mode 100644 index 000000000..9c8916ca4 --- /dev/null +++ b/core/http/react-ui/e2e/models-cluster-fit.spec.js @@ -0,0 +1,120 @@ +import { test, expect } from "./coverage-fixtures.js"; + +// On a distributed controller the models run on the workers, so every "will +// this fit" answer on this page is about their hardware. The controller is +// usually a GPU-less pod: sized against it, a cluster of A100s is told it can +// only run the smallest CPU build. + +const GB = 1024 * 1024 * 1024; + +const MODELS = [ + { name: "big-gpu-model", description: "Needs a real GPU", backend: "vllm", installed: false, tags: ["chat"] }, +]; + +// 40GB: far past the controller's 8GB of RAM, comfortably inside one 80GB card. +const ESTIMATES = { + "big-gpu-model": { + sizeBytes: 40 * GB, + sizeDisplay: "40.0 GB", + estimates: { 8192: { vramBytes: 40 * GB, vramDisplay: "40.0 GB" } }, + }, +}; + +// The controller as Argus actually runs it: 8GB of system RAM, no GPU. +const CONTROLLER_ONLY = { + type: "ram", + available: true, + gpus: [], + aggregate: { total_memory: 8 * GB, used_memory: 2 * GB, free_memory: 6 * GB, gpu_count: 0 }, +}; + +const WITH_CLUSTER = { + ...CONTROLLER_ONLY, + cluster: { + enabled: true, + node_id: "n-1", + node_name: "dgx-01", + total_memory: 80 * GB, + is_gpu: true, + node_count: 4, + }, +}; + +async function mockModels(page, resources) { + await page.route("**/api/models*", (route) => + route.fulfill({ + contentType: "application/json", + body: JSON.stringify({ + models: MODELS, + allBackends: ["vllm"], + allTags: ["chat"], + availableModels: MODELS.length, + installedModels: 3, + totalPages: 1, + currentPage: 1, + }), + }), + ); + await page.route("**/api/models/estimate/*", (route) => { + const name = decodeURIComponent(new URL(route.request().url()).pathname.split("/").pop()); + return route.fulfill({ contentType: "application/json", body: JSON.stringify(ESTIMATES[name] || {}) }); + }); + await page.route("**/api/resources", (route) => + route.fulfill({ contentType: "application/json", body: JSON.stringify(resources) }), + ); +} + +const railItems = (page) => page.locator('[data-testid="discover-rail-item"]'); +const railItem = (page, name) => page.locator(`[data-entity="${name}"]`); +const railReady = (page) => expect(railItems(page).first()).toBeVisible({ timeout: 20_000 }); +const PANE = '[data-testid="discover-pane"]'; + +test.describe("Models gallery - cluster-aware fit", () => { + test("a model that only a worker can hold is not called too large", async ({ page }) => { + await mockModels(page, WITH_CLUSTER); + await page.goto("/app/models"); + + await railReady(page); + + // The whole defect in one assertion: 40GB against a 4-node cluster whose + // largest card holds 80GB. + await expect(railItem(page, "big-gpu-model")).toContainText("fits", { timeout: 20_000 }); + await expect(railItem(page, "big-gpu-model")).not.toContainText("too large"); + }); + + test("the fit verdict names the node it belongs to", async ({ page }) => { + await mockModels(page, WITH_CLUSTER); + await page.goto("/app/models"); + + await railReady(page); + await railItem(page, "big-gpu-model").click(); + // Wait for the detail itself: until it renders, the pane still holds the + // zero-state hero, which names the node for its own reasons. + await expect(page.locator(PANE).getByText("40.0 GB")).toBeVisible({ timeout: 20_000 }); + + // The headroom this model has is headroom SOMEWHERE, and the stat says + // where rather than leaving it to read as this machine's. + await expect(page.locator(PANE)).toContainText(/headroom on dgx-01/i); + }); + + test("the host summary describes the cluster, not the controller", async ({ page }) => { + await mockModels(page, WITH_CLUSTER); + await page.goto("/app/models"); + + await railReady(page); + // 80 GB is the cluster's best node; 8 GB is this pod's own RAM and must + // not be what the page advertises. + await expect(page.locator(".zero-pane__title")).toContainText("80 GB"); + await expect(page.locator(".zero-pane__title")).not.toContainText("8.00 GB"); + }); + + // Single-node behavior is the fallback every degradation path lands on, so + // it has to stay exactly as it was. + test("without a cluster the verdict is still the local host's", async ({ page }) => { + await mockModels(page, CONTROLLER_ONLY); + await page.goto("/app/models"); + + await railReady(page); + await expect(railItem(page, "big-gpu-model")).toContainText("too large", { timeout: 20_000 }); + }); +}); diff --git a/core/http/react-ui/e2e/scheduling.spec.js b/core/http/react-ui/e2e/scheduling.spec.js index 4d11d8b26..79d781a3f 100644 --- a/core/http/react-ui/e2e/scheduling.spec.js +++ b/core/http/react-ui/e2e/scheduling.spec.js @@ -36,35 +36,80 @@ async function mockScheduling(page, { rules = [rule], nodeList = nodes } = {}) { } test.describe('Scheduling page', () => { - test('groups node labels, collapses the reference, filters forgivingly, and expands results', async ({ page }) => { + // Node labels are only ever needed while writing a rule's node selector, so + // they live in that field rather than in a card standing open above the + // rules whether or not anyone is writing one. + test('keeps no standing label browser on the page', async ({ page }) => { await mockScheduling(page) await page.goto('/app/scheduling') + await expect(page.getByText('llama-3.3')).toBeVisible() - const reference = page.getByTestId('node-label-reference') - await expect(reference.getByText('Falcon GPU')).toBeVisible() - await expect(reference.getByText('No labels')).toBeVisible() - await expect(reference.locator('.scheduling-node-card')).toHaveCount(5) - await expect(reference.getByText('5 of 27 nodes')).toBeVisible() + await expect(page.getByTestId('node-label-reference')).toHaveCount(0) + await expect(page.getByRole('button', { name: /node labels/i })).toHaveCount(0) + await expect(page.locator('.scheduling-node-card')).toHaveCount(0) + // Falcon GPU is a node name, and nothing on this page has a reason to + // enumerate node names until a selector is being filled. + await expect(page.getByText('Falcon GPU')).toHaveCount(0) + }) - const toggle = page.getByRole('button', { name: /node labels/i }) - await expect(toggle).toHaveAttribute('aria-expanded', 'true') - await toggle.click() - await expect(toggle).toHaveAttribute('aria-expanded', 'false') - await expect(reference.getByRole('searchbox')).toBeHidden() - await toggle.click() + test('suggests the cluster\'s own label keys and values as the selector is typed', async ({ page }) => { + await mockScheduling(page) + await page.goto('/app/scheduling') + await page.getByRole('button', { name: 'Add Scheduling Rule' }).click() - await reference.getByRole('searchbox').fill('GPU.VENDOR=nvi') - await expect(reference.locator('.scheduling-node-card')).toHaveCount(1) - await expect(reference.getByText('Falcon GPU')).toBeVisible() + const keyInput = page.getByRole('combobox', { name: 'Selector key' }) + await keyInput.click() + const suggestions = page.getByTestId('label-suggestions') + // Every key the cluster reports, before a single character is typed. + await expect(suggestions.getByRole('option', { name: 'gpu.vendor' })).toBeVisible() + await expect(suggestions.getByRole('option', { name: 'zone' })).toBeVisible() - await reference.getByRole('searchbox').fill('flcn') - await expect(reference.locator('.scheduling-node-card')).toHaveCount(1) - await expect(reference.getByText('Falcon GPU')).toBeVisible() + await keyInput.fill('vend') + await expect(suggestions.getByRole('option')).toHaveCount(1) + await suggestions.getByRole('option', { name: 'gpu.vendor' }).click() + await expect(keyInput).toHaveValue('gpu.vendor') - await reference.getByRole('searchbox').fill('') - await reference.getByRole('button', { name: 'Show 20 more nodes' }).click() - await expect(reference.locator('.scheduling-node-card')).toHaveCount(25) - await expect(reference.getByText('25 of 27 nodes')).toBeVisible() + // Values are scoped to the key being filled, so a selector cannot be built + // out of a pair no node matches. + const valueInput = page.getByRole('combobox', { name: 'Selector value' }) + await valueInput.click() + await expect(suggestions.getByRole('option', { name: 'NVIDIA' })).toBeVisible() + await expect(suggestions.getByRole('option', { name: 'amd' })).toBeVisible() + await expect(suggestions.getByRole('option', { name: 'east' })).toHaveCount(0) + + await valueInput.fill('nvi') + await suggestions.getByRole('option', { name: 'NVIDIA' }).click() + await expect(valueInput).toHaveValue('NVIDIA') + }) + + test('picks a suggestion from the keyboard', async ({ page }) => { + await mockScheduling(page) + await page.goto('/app/scheduling') + await page.getByRole('button', { name: 'Add Scheduling Rule' }).click() + + const keyInput = page.getByRole('combobox', { name: 'Selector key' }) + await keyInput.fill('zon') + await keyInput.press('ArrowDown') + await keyInput.press('Enter') + await expect(keyInput).toHaveValue('zone') + // Enter picked the suggestion rather than committing the chip, so the + // half-built pair is still in the inputs. + await expect(page.getByLabel('Node selector').getByText('zone=', { exact: true })).toHaveCount(0) + }) + + // The cluster's vocabulary is a suggestion, never a constraint: an admin + // labelling nodes for a rule they are about to write must still be able to + // type a key no node reports yet. + test('still accepts a label the cluster has never reported', async ({ page }) => { + await mockScheduling(page) + await page.goto('/app/scheduling') + await page.getByRole('button', { name: 'Add Scheduling Rule' }).click() + + await page.getByRole('combobox', { name: 'Selector key' }).fill('tenant') + await page.getByRole('combobox', { name: 'Selector value' }).fill('acme') + await page.getByRole('button', { name: 'Add selector' }).click() + + await expect(page.getByLabel('Node selector').getByText('tenant=acme', { exact: true })).toBeVisible() }) test('edits all fields with a locked model and preserves values after a failed save', async ({ page }) => { @@ -113,42 +158,98 @@ test.describe('Scheduling page', () => { await expect(page.getByRole('combobox', { name: '' }).first()).toBeEnabled() }) - test('shows node loading, empty, no-match, and retry states independently from rules', async ({ page }) => { - let attempts = 0 + // The roster feeds suggestions and nothing else now, so failing to load it + // must cost the admin nothing but the hints. + test('leaves the selector fully usable when the node roster fails to load', async ({ page }) => { await page.route('**/api/nodes/scheduling', route => route.fulfill({ status: 200, contentType: 'application/json', body: JSON.stringify([rule]) })) - await page.route('**/api/nodes', async route => { - attempts++ - if (attempts === 1) { - await new Promise(resolve => setTimeout(resolve, 250)) - await route.fulfill({ status: 500, body: 'failed' }) - } else { - await route.fulfill({ status: 200, contentType: 'application/json', body: '[]' }) - } - }) + await page.route('**/api/nodes', route => route.fulfill({ status: 500, body: 'failed' })) await page.goto('/app/scheduling') - await expect(page.getByText('Loading node labels…')).toBeVisible() - await expect(page.getByText('llama-3.3')).toBeVisible() - await expect(page.getByText('Could not load node labels.')).toBeVisible() - await page.getByRole('button', { name: 'Retry loading node labels' }).click() - await expect(page.getByText('No nodes are available yet.')).toBeVisible() - await page.unroute('**/api/nodes') - await page.route('**/api/nodes', route => route.fulfill({ status: 200, contentType: 'application/json', body: JSON.stringify(nodes) })) - await page.reload() - await page.getByRole('searchbox', { name: 'Search node labels' }).fill('not-a-real-label') - await expect(page.getByText('No nodes match your search.')).toBeVisible() + // The rules still render: the roster is not on their path. + await expect(page.getByText('llama-3.3')).toBeVisible() + + await page.getByRole('button', { name: 'Add Scheduling Rule' }).click() + await page.getByRole('combobox', { name: 'Selector key' }).fill('gpu.vendor') + await page.getByRole('combobox', { name: 'Selector value' }).fill('nvidia') + await page.getByRole('button', { name: 'Add selector' }).click() + + await expect(page.getByLabel('Node selector').getByText('gpu.vendor=nvidia', { exact: true })).toBeVisible() }) - test('uses one node column and accessible rule actions on a narrow viewport', async ({ page }) => { + // A rule may be keyed by an alias, in which case it governs whichever model + // the alias points at. The page has to say which model that is, because the + // rule's own name no longer tells you. + test.describe('rules keyed by a model alias', () => { + const aliasRule = { + model_name: 'production', + target_model: 'llama-3.3', + model_is_alias: true, + node_selector: { tier: 'gpu' }, + min_replicas: 2, + max_replicas: 4, + } + + async function mockAliases(page, aliases = [{ name: 'production', target: 'llama-3.3' }]) { + await page.route('**/api/aliases', route => route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify(aliases), + })) + await page.route('**/api/models/capabilities', route => route.fulfill({ + status: 200, + contentType: 'application/json', + body: JSON.stringify({ object: 'list', data: [{ id: 'llama-3.3' }, { id: 'production' }] }), + })) + } + + test('names the model an alias rule governs', async ({ page }) => { + await mockScheduling(page, { rules: [aliasRule] }) + await mockAliases(page) + await page.goto('/app/scheduling') + + await expect(page.getByText('production')).toBeVisible() + await expect(page.locator('.scheduling-rule-target')).toHaveText(/llama-3\.3/) + }) + + test('marks a rule another rule already governs as shadowed', async ({ page }) => { + await mockScheduling(page, { rules: [{ ...aliasRule, shadowed: true }, rule] }) + await mockAliases(page) + await page.goto('/app/scheduling') + + await expect(page.locator('.scheduling-rule-shadowed')).toHaveCount(1) + await expect(page.locator('.scheduling-rule-shadowed')).toContainText('Shadowed') + }) + + test('flags an alias rule that no longer resolves', async ({ page }) => { + await mockScheduling(page, { + rules: [{ model_name: 'orphan', target_model: 'orphan', model_is_alias: true, min_replicas: 1 }], + }) + await mockAliases(page, []) + await page.goto('/app/scheduling') + + await expect(page.locator('.scheduling-rule-target--broken')).toBeVisible() + }) + + test('offers aliases in the model picker, tagged with their target', async ({ page }) => { + await mockScheduling(page) + await mockAliases(page) + await page.goto('/app/scheduling') + await page.getByRole('button', { name: 'Add Scheduling Rule' }).click() + + const picker = page.locator('.searchable-model-select input') + await picker.click() + await expect(page.locator('.sms-hint')).toHaveText('alias of llama-3.3') + + await page.getByRole('option', { name: /production/ }).click() + await expect(page.getByText(/production is an alias for llama-3\.3/)).toBeVisible() + }) + }) + + test('keeps rule actions reachable on a narrow viewport', async ({ page }) => { await page.setViewportSize({ width: 390, height: 844 }) await mockScheduling(page, { nodeList: nodes.slice(0, 2) }) await page.goto('/app/scheduling') - const cards = page.locator('.scheduling-node-card') - const first = await cards.nth(0).boundingBox() - const second = await cards.nth(1).boundingBox() - expect(second.y).toBeGreaterThan(first.y + first.height - 1) - const actions = page.locator('.scheduling-rule-actions') await expect(actions.getByRole('button', { name: 'Edit llama-3.3' })).toBeVisible() await expect(actions.getByRole('button', { name: 'Delete llama-3.3' })).toBeVisible() diff --git a/core/http/react-ui/e2e/traces-pagination.spec.js b/core/http/react-ui/e2e/traces-pagination.spec.js index c6d7b89a1..a7e56afd4 100644 --- a/core/http/react-ui/e2e/traces-pagination.spec.js +++ b/core/http/react-ui/e2e/traces-pagination.spec.js @@ -79,4 +79,30 @@ test.describe('Traces - bounded list and on-demand detail', () => { await expect(page.locator('text=hello from the response body')).toBeVisible() await expect(page.locator('text=203.0.113.9').first()).toBeVisible() }) + + test('keeps the expanded trace open when a refresh prepends a new row', async ({ page }) => { + await page.locator('tr', { hasText: '/v1/chat/completions' }).first().click() + await expect(page.locator('text=hello from the request body')).toBeVisible() + + await page.route('**/api/traces?*', (route) => { + route.fulfill({ + contentType: 'application/json', + headers: { 'X-Total-Count': '843' }, + body: JSON.stringify([ + { + id: '8', + request: { method: 'GET', path: '/v1/models', body: null }, + response: { status: 200, body: null }, + }, + ...LIST_BODY, + ]), + }) + }) + + await page.getByRole('button', { name: 'Refresh' }).click() + + await expect(page.locator('text=hello from the request body')).toBeVisible() + const originalRow = page.locator('tr', { hasText: '/v1/chat/completions' }).first() + await expect(originalRow.locator('i.fa-chevron-down')).toBeVisible() + }) }) diff --git a/core/http/react-ui/inline-style-baseline.txt b/core/http/react-ui/inline-style-baseline.txt index 08f851b6e..a08796291 100644 --- a/core/http/react-ui/inline-style-baseline.txt +++ b/core/http/react-ui/inline-style-baseline.txt @@ -1 +1 @@ -519 +514 diff --git a/core/http/react-ui/package-lock.json b/core/http/react-ui/package-lock.json index ccf8b576a..7ac92af0c 100644 --- a/core/http/react-ui/package-lock.json +++ b/core/http/react-ui/package-lock.json @@ -32,7 +32,8 @@ "react": "^19.1.0", "react-dom": "^19.1.0", "react-i18next": "^17.0.6", - "react-router-dom": "^7.18.1", + "react-router": "8.3.1", + "react-router-dom": "7.18.2", "yaml": "^2.8.3" }, "devDependencies": { @@ -648,27 +649,43 @@ } }, "node_modules/@humanfs/core": { - "version": "0.19.1", - "resolved": "https://registry.npmjs.org/@humanfs/core/-/core-0.19.1.tgz", - "integrity": "sha512-5DyQ4+1JEUzejeK1JGICcideyfUbGixgS9jNgex5nqkW+cY7WZhxBigmieN5Qnw9ZosSNVC9KQKyb+GUaGyKUA==", + "version": "0.19.2", + "resolved": "https://registry.npmjs.org/@humanfs/core/-/core-0.19.2.tgz", + "integrity": "sha512-UhXNm+CFMWcbChXywFwkmhqjs3PRCmcSa/hfBgLIb7oQ5HNb1wS0icWsGtSAUNgefHeI+eBrA8I1fxmbHsGdvA==", "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@humanfs/types": "^0.15.0" + }, "engines": { "node": ">=18.18.0" } }, "node_modules/@humanfs/node": { - "version": "0.16.7", - "resolved": "https://registry.npmjs.org/@humanfs/node/-/node-0.16.7.tgz", - "integrity": "sha512-/zUx+yOsIrG4Y43Eh2peDeKCxlRt/gET6aHfaKpuq267qXdYDFViVHfMaLyygZOnl0kGWxFIgsBy8QFuTLUXEQ==", + "version": "0.16.8", + "resolved": "https://registry.npmjs.org/@humanfs/node/-/node-0.16.8.tgz", + "integrity": "sha512-gE1eQNZ3R++kTzFUpdGlpmy8kDZD/MLyHqDwqjkVQI0JMdI1D51sy1H958PNXYkM2rAac7e5/CnIKZrHtPh3BQ==", "dev": true, + "license": "Apache-2.0", "dependencies": { - "@humanfs/core": "^0.19.1", + "@humanfs/core": "^0.19.2", + "@humanfs/types": "^0.15.0", "@humanwhocodes/retry": "^0.4.0" }, "engines": { "node": ">=18.18.0" } }, + "node_modules/@humanfs/types": { + "version": "0.15.0", + "resolved": "https://registry.npmjs.org/@humanfs/types/-/types-0.15.0.tgz", + "integrity": "sha512-ZZ1w0aoQkwuUuC7Yf+7sdeaNfqQiiLcSRbfI08oAxqLtpXQr9AIVX7Ay7HLDuiLYAaFPu8oBYNq/QIi9URHJ3Q==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=18.18.0" + } + }, "node_modules/@humanwhocodes/module-importer": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/@humanwhocodes/module-importer/-/module-importer-1.0.1.tgz", @@ -1639,9 +1656,9 @@ ] }, "node_modules/baseline-browser-mapping": { - "version": "2.10.38", - "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.10.38.tgz", - "integrity": "sha512-31/02mVB4yuQU6adKk5SlY6m+mxDwUq5KZkyYgnLrrKl7TEm1+3PyDtDBz2kOv/wxZz41GHsvV1A/u6RmiyBvw==", + "version": "2.11.20", + "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.20.tgz", + "integrity": "sha512-H0ulySigv6icDJ1F7SjtdCD6PrhTpdYCmP0CactWy1+ekh0AFd0o1Wn5T8b+hnTmdBx19u9yhL6wvCylXMY7zw==", "dev": true, "license": "Apache-2.0", "bin": { @@ -1810,9 +1827,9 @@ } }, "node_modules/browserslist": { - "version": "4.28.2", - "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.2.tgz", - "integrity": "sha512-48xSriZYYg+8qXna9kwqjIVzuQxi+KYWp2+5nCYnYKPTr0LvD89Jqk2Or5ogxz0NUMfIjhh2lIUX/LyX9B4oIg==", + "version": "4.28.8", + "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.8.tgz", + "integrity": "sha512-V2NpofLblG64mfOtSgDhOJESZEGogzDMBv/q+W6oc4LXWP/q75eOXoOaaOu1EOadB9U4Bwx/e0yzbvwKH8zalA==", "dev": true, "funding": [ { @@ -1830,11 +1847,11 @@ ], "license": "MIT", "dependencies": { - "baseline-browser-mapping": "^2.10.12", - "caniuse-lite": "^1.0.30001782", - "electron-to-chromium": "^1.5.328", - "node-releases": "^2.0.36", - "update-browserslist-db": "^1.2.3" + "baseline-browser-mapping": "^2.11.12", + "caniuse-lite": "^1.0.30001809", + "electron-to-chromium": "^1.5.402", + "node-releases": "^2.0.53", + "update-browserslist-db": "^1.3.0" }, "bin": { "browserslist": "cli.js" @@ -1940,9 +1957,9 @@ } }, "node_modules/caniuse-lite": { - "version": "1.0.30001799", - "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001799.tgz", - "integrity": "sha512-hG1bReV+OUU+MOqK4t/ZWI0tZOyz3rqS9XuhOUz1cIcbwBKjOyJEJuw9ER5JuNyqxNk8u/JUVbGibBOL1yrjFw==", + "version": "1.0.30001810", + "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001810.tgz", + "integrity": "sha512-TITQPUkaz+aVk5GL6NhOdwk1aEaNTSDPsGFWrTuhKGtjTF70jL/Oht2W4c6rXUe5fu7Ie19VIahAXHIIiWWNeg==", "dev": true, "funding": [ { @@ -2197,6 +2214,12 @@ "url": "https://opencollective.com/express" } }, + "node_modules/cookie-es": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/cookie-es/-/cookie-es-3.1.1.tgz", + "integrity": "sha512-UaXxwISYJPTr9hwQxMFYZ7kNhSXboMXP+Z3TRX6f1/NyaGPfuNUZOWP1pUEb75B2HjfklIYLVRfWiFZJyC6Npg==", + "license": "MIT" + }, "node_modules/cookie-signature": { "version": "1.2.2", "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.2.2.tgz", @@ -2437,9 +2460,9 @@ "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==" }, "node_modules/electron-to-chromium": { - "version": "1.5.375", - "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.375.tgz", - "integrity": "sha512-ZWP5eB4BVPW/ZYo9252hQZHZ5XavtsTgpbhcmMmRwymavC5AsLWQWBPaKMeNd2LW0KGby5HPXvj7+sr4ta5j/Q==", + "version": "1.5.420", + "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.420.tgz", + "integrity": "sha512-2yD6XreGusOfNV+dUcvipJEXc3n/n7fgr7996aszTG+YY5E4mqM4tOq/3uhP129cazL9YHbVWSpc79ePotWtPA==", "dev": true, "license": "ISC" }, @@ -2847,6 +2870,15 @@ "express": ">= 4.11" } }, + "node_modules/express-rate-limit/node_modules/ip-address": { + "version": "10.3.1", + "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.3.1.tgz", + "integrity": "sha512-1e9d3kb97NHJTIJDZW9rKqW2h6+dFa50Dy0fpPSMQp2ADje5gvKsXmdiK6dwY5t76TaTt5+P5N1Y/LoToIxP6g==", + "license": "MIT", + "engines": { + "node": ">= 12" + } + }, "node_modules/express/node_modules/cookie": { "version": "0.7.2", "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz", @@ -2879,9 +2911,9 @@ "dev": true }, "node_modules/fast-uri": { - "version": "3.1.5", - "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.5.tgz", - "integrity": "sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw==", + "version": "3.1.7", + "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.7.tgz", + "integrity": "sha512-dOvZVzjdZdz7phd9v6jCbwxrBW3fK6n8Rc0CtdmM4bumzMnxywBYhuph6J819RRw/ku+rLbelwfMunktuzVVHg==", "funding": [ { "type": "github", @@ -4195,15 +4227,6 @@ "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==" }, - "node_modules/ip-address": { - "version": "10.4.0", - "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.4.0.tgz", - "integrity": "sha512-oSK96Grm3aP6OrS263xVxbNDGVL7rzBtYdpGqlDG8iQdoenDoTs/nkki+DflYbAEE8Xl6o5YxhxlrKvI3nqKXQ==", - "license": "MIT", - "engines": { - "node": ">= 12" - } - }, "node_modules/ipaddr.js": { "version": "1.9.1", "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz", @@ -5186,9 +5209,9 @@ } }, "node_modules/node-releases": { - "version": "2.0.48", - "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.48.tgz", - "integrity": "sha512-1uz8041X6LoI6ZSdZacM9lVY28vuzDlSKitnpbSNK0RfKoIJkX29NBPVEFXhnuSuEOA9Ww0xnPJ+ILWbGAv8DA==", + "version": "2.0.54", + "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.54.tgz", + "integrity": "sha512-YHs7BmmcsdAI5Ozuf8JZo6PT0mv2GIWC9vMfvUC3dp65M8hn7Ux8CPL+2oBI7juNuj9d0ndhTcznq2ODBps9cQ==", "dev": true, "license": "MIT", "engines": { @@ -5950,9 +5973,9 @@ } }, "node_modules/qs": { - "version": "6.15.3", - "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.3.tgz", - "integrity": "sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A==", + "version": "6.16.0", + "resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz", + "integrity": "sha512-h6fhOIaRrID2CbEY2fqs+7t+UXZo+MLAnU5gRIq85uFtdiUPCdsApMlHhXogKVM4HM2DVbIjGNTTYH2OcmP1vA==", "license": "BSD-3-Clause", "dependencies": { "es-define-property": "^1.0.1", @@ -6060,22 +6083,24 @@ } }, "node_modules/react": { - "version": "19.2.4", - "resolved": "https://registry.npmjs.org/react/-/react-19.2.4.tgz", - "integrity": "sha512-9nfp2hYpCwOjAN+8TZFGhtWEwgvWHXqESH8qT89AT/lWklpLON22Lc8pEtnpsZz7VmawabSU0gCjnj8aC0euHQ==", + "version": "19.2.8", + "resolved": "https://registry.npmjs.org/react/-/react-19.2.8.tgz", + "integrity": "sha512-PWaYA1L/q9u2u7xYQi+Y3L3Yfnie7XyLeaJICV1MGD6LprsBxcAqGjYyr0eY3p+QdsA+x/Irkt4Qif8D63+Sbw==", + "license": "MIT", "engines": { "node": ">=0.10.0" } }, "node_modules/react-dom": { - "version": "19.2.4", - "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.4.tgz", - "integrity": "sha512-AXJdLo8kgMbimY95O2aKQqsz2iWi9jMgKJhRBAxECE4IFxfcazB2LmzloIoibJI3C12IlY20+KFaLv+71bUJeQ==", + "version": "19.2.8", + "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.8.tgz", + "integrity": "sha512-rVprimfGBG3DR+Tq0IQG2DT5PxKth1WIGDmj5yPmlzr4YBe7uyE+Du4oVqTDXZSHGGGXRtTJEGSSePyQCMBglQ==", + "license": "MIT", "dependencies": { "scheduler": "^0.27.0" }, "peerDependencies": { - "react": "^19.2.4" + "react": "^19.2.8" } }, "node_modules/react-i18next": { @@ -6105,9 +6130,46 @@ } }, "node_modules/react-router": { - "version": "7.18.1", - "resolved": "https://registry.npmjs.org/react-router/-/react-router-7.18.1.tgz", - "integrity": "sha512-GDLgg3i3uM0aeJO3Fm+TCS+sDQ7gu12T6x0qdTEzcwqEfleci7JwugVNIF3U//0FWKnJT7ptG+20B2jfDqnZAg==", + "version": "8.3.1", + "resolved": "https://registry.npmjs.org/react-router/-/react-router-8.3.1.tgz", + "integrity": "sha512-TEOpiO2g0TJHEOJeRVv4amUFun9v1npCKszvcquNvzETUtJ8udV86ah5eFoHT7g26bsBvT6EiIhqulR8eDF++A==", + "license": "MIT", + "dependencies": { + "cookie-es": "^3.1.1" + }, + "engines": { + "node": ">=22.22.0" + }, + "peerDependencies": { + "react": ">=19.2.7", + "react-dom": ">=19.2.7" + }, + "peerDependenciesMeta": { + "react-dom": { + "optional": true + } + } + }, + "node_modules/react-router-dom": { + "version": "7.18.2", + "resolved": "https://registry.npmjs.org/react-router-dom/-/react-router-dom-7.18.2.tgz", + "integrity": "sha512-AIKJ/jgGlFb3EbfCXk5Gzshiwt+l3mqbCrNjmEWMMjqQxNJ3svBa6bgzFyCC2Sw3RA0VWF1kg3uQf2OFhxb8hw==", + "license": "MIT", + "dependencies": { + "react-router": "7.18.2" + }, + "engines": { + "node": ">=20.0.0" + }, + "peerDependencies": { + "react": ">=18", + "react-dom": ">=18" + } + }, + "node_modules/react-router-dom/node_modules/react-router": { + "version": "7.18.2", + "resolved": "https://registry.npmjs.org/react-router/-/react-router-7.18.2.tgz", + "integrity": "sha512-aUVMjFm3GAPTTZL7oYr5E7ETiqfQCHRLH+B+5afnICvf0r7kkK4eR6SMuwbSTJw/7t+12khT/Kahij49fqOCIg==", "license": "MIT", "dependencies": { "cookie": "^1.0.1", @@ -6126,22 +6188,6 @@ } } }, - "node_modules/react-router-dom": { - "version": "7.18.1", - "resolved": "https://registry.npmjs.org/react-router-dom/-/react-router-dom-7.18.1.tgz", - "integrity": "sha512-KaZh+X/6UtEp28x51AUYZDMg9NGoz2ja3dNHa+ta/tk40vCzKhQ/RypCWBMLbmDr6//E24Vv5uPsrqXFozdkAg==", - "license": "MIT", - "dependencies": { - "react-router": "7.18.1" - }, - "engines": { - "node": ">=20.0.0" - }, - "peerDependencies": { - "react": ">=18", - "react-dom": ">=18" - } - }, "node_modules/readable-stream": { "version": "2.3.8", "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.8.tgz", @@ -7165,9 +7211,9 @@ } }, "node_modules/update-browserslist-db": { - "version": "1.2.3", - "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.2.3.tgz", - "integrity": "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w==", + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.3.2.tgz", + "integrity": "sha512-UQ+MSxlhRm1bzjhU+DcuXfjFO1FzNtqhK5+9Yvlp90ItDLk5vT932A0rFu619nf7RVS+Y/VeaUW1jaRDqZ8VJw==", "dev": true, "funding": [ { diff --git a/core/http/react-ui/package.json b/core/http/react-ui/package.json index fc3832680..6d3861a79 100644 --- a/core/http/react-ui/package.json +++ b/core/http/react-ui/package.json @@ -20,7 +20,8 @@ }, "overrides": { "hono": "4.12.34", - "ip-address": "10.3.1" + "ip-address": "10.3.1", + "path-to-regexp": "^8.4.0" }, "dependencies": { "@codemirror/autocomplete": "^6.18.6", @@ -47,7 +48,7 @@ "react": "^19.1.0", "react-dom": "^19.1.0", "react-i18next": "^17.0.6", - "react-router": "7.18.2", + "react-router": "8.3.1", "react-router-dom": "7.18.2", "yaml": "^2.8.3" }, diff --git a/core/http/react-ui/public/locales/de/models.json b/core/http/react-ui/public/locales/de/models.json index 21e29f782..779e0e7f6 100644 --- a/core/http/react-ui/public/locales/de/models.json +++ b/core/http/react-ui/public/locales/de/models.json @@ -142,7 +142,8 @@ "sha256": "SHA256", "backToAll": "Alle Modelle", "vramAt": "VRAM bei {{context}}", - "headroom": "Spielraum" + "headroom": "Spielraum", + "headroomOn": "Spielraum auf {{node}}" }, "empty": { "title": "Keine Modelle gefunden", @@ -216,6 +217,8 @@ "browsing": "Durchsuchen", "pickHint": "Wähle ein Modell, um die Details zu sehen.", "heroWithRam": "{{ram}} Systemspeicher, {{count}} Modelle in der Galerie.", + "heroWithCluster": "{{vram}} auf {{node}}, dem größten von {{nodes}} Knoten, {{count}} Modelle in der Galerie.", + "heroWithNode": "{{vram}} auf {{node}}, {{count}} Modelle in der Galerie.", "byUseCase": "Oder mit einem Anwendungsfall starten", "pickText": "Chat, Reasoning, Embeddings", "pickVision": "Bilder und Dokumente lesen", diff --git a/core/http/react-ui/public/locales/en/chat.json b/core/http/react-ui/public/locales/en/chat.json index f004cee56..f1c058790 100644 --- a/core/http/react-ui/public/locales/en/chat.json +++ b/core/http/react-ui/public/locales/en/chat.json @@ -33,6 +33,8 @@ "title": "Chat Settings", "manageMode": "Manage mode", "manageModeDesc": "Let this chat install models, switch backends, and edit configs by talking to LocalAI.", + "focusMode": "Focus mode", + "focusModeDesc": "Collapse the sidebar and slim the header while a conversation is active. Esc restores them temporarily; turn this off to keep the full layout.", "systemPrompt": "System Prompt", "systemPromptPlaceholder": "You are a helpful assistant...", "temperature": "Temperature", diff --git a/core/http/react-ui/public/locales/en/models.json b/core/http/react-ui/public/locales/en/models.json index fdc2e397f..ebadd3b3d 100644 --- a/core/http/react-ui/public/locales/en/models.json +++ b/core/http/react-ui/public/locales/en/models.json @@ -152,7 +152,8 @@ "sha256": "SHA256", "backToAll": "All models", "vramAt": "VRAM at {{context}}", - "headroom": "Headroom" + "headroom": "Headroom", + "headroomOn": "Headroom on {{node}}" }, "empty": { "title": "No models found", @@ -232,6 +233,8 @@ "browsing": "Browsing", "pickHint": "Select a model to see its detail.", "heroWithRam": "{{ram}} of system memory, {{count}} models in the gallery.", + "heroWithCluster": "{{vram}} on {{node}}, the largest of {{nodes}} nodes, {{count}} models in the gallery.", + "heroWithNode": "{{vram}} on {{node}}, {{count}} models in the gallery.", "byUseCase": "Or start with a use case", "pickText": "Chat, reasoning, embeddings", "pickVision": "Read images and documents", diff --git a/core/http/react-ui/public/locales/es/models.json b/core/http/react-ui/public/locales/es/models.json index 3d5cdd8df..d833fba55 100644 --- a/core/http/react-ui/public/locales/es/models.json +++ b/core/http/react-ui/public/locales/es/models.json @@ -142,7 +142,8 @@ "sha256": "SHA256", "backToAll": "Todos los modelos", "vramAt": "VRAM a {{context}}", - "headroom": "Margen" + "headroom": "Margen", + "headroomOn": "Margen en {{node}}" }, "empty": { "title": "No se encontraron modelos", @@ -216,6 +217,8 @@ "browsing": "Explorando", "pickHint": "Selecciona un modelo para ver su detalle.", "heroWithRam": "{{ram}} de memoria del sistema, {{count}} modelos en la galería.", + "heroWithCluster": "{{vram}} en {{node}}, el mayor de {{nodes}} nodos, {{count}} modelos en la galería.", + "heroWithNode": "{{vram}} en {{node}}, {{count}} modelos en la galería.", "byUseCase": "O empieza por un caso de uso", "pickText": "Chat, razonamiento, embeddings", "pickVision": "Leer imágenes y documentos", diff --git a/core/http/react-ui/public/locales/id/models.json b/core/http/react-ui/public/locales/id/models.json index 27a42af64..9088647b9 100644 --- a/core/http/react-ui/public/locales/id/models.json +++ b/core/http/react-ui/public/locales/id/models.json @@ -149,7 +149,8 @@ "sha256": "SHA256", "backToAll": "Semua model", "vramAt": "VRAM pada {{context}}", - "headroom": "Sisa ruang" + "headroom": "Sisa ruang", + "headroomOn": "Sisa ruang di {{node}}" }, "empty": { "title": "Model tidak ditemukan", @@ -229,6 +230,8 @@ "browsing": "Menjelajah", "pickHint": "Pilih model untuk melihat detailnya.", "heroWithRam": "Memori sistem {{ram}}, {{count}} model di galeri.", + "heroWithCluster": "{{vram}} di {{node}}, terbesar dari {{nodes}} node, {{count}} model di galeri.", + "heroWithNode": "{{vram}} di {{node}}, {{count}} model di galeri.", "byUseCase": "Atau mulai dari kasus penggunaan", "pickText": "Obrolan, penalaran, embedding", "pickVision": "Membaca gambar dan dokumen", diff --git a/core/http/react-ui/public/locales/it/models.json b/core/http/react-ui/public/locales/it/models.json index 80bd16284..cbc06d22c 100644 --- a/core/http/react-ui/public/locales/it/models.json +++ b/core/http/react-ui/public/locales/it/models.json @@ -142,7 +142,8 @@ "sha256": "SHA256", "backToAll": "Tutti i modelli", "vramAt": "VRAM a {{context}}", - "headroom": "Margine" + "headroom": "Margine", + "headroomOn": "Margine su {{node}}" }, "empty": { "title": "Nessun modello trovato", @@ -216,6 +217,8 @@ "browsing": "Esplorazione", "pickHint": "Seleziona un modello per vederne i dettagli.", "heroWithRam": "{{ram}} di memoria di sistema, {{count}} modelli nella galleria.", + "heroWithCluster": "{{vram}} su {{node}}, il più grande di {{nodes}} nodi, {{count}} modelli nella galleria.", + "heroWithNode": "{{vram}} su {{node}}, {{count}} modelli nella galleria.", "byUseCase": "Oppure parti da un caso d’uso", "pickText": "Chat, ragionamento, embedding", "pickVision": "Leggere immagini e documenti", diff --git a/core/http/react-ui/public/locales/ko/models.json b/core/http/react-ui/public/locales/ko/models.json index b4f1a4656..8ed38bf0f 100644 --- a/core/http/react-ui/public/locales/ko/models.json +++ b/core/http/react-ui/public/locales/ko/models.json @@ -148,7 +148,8 @@ "sha256": "SHA256", "backToAll": "모든 모델", "vramAt": "{{context}}에서의 VRAM", - "headroom": "여유 공간" + "headroom": "여유 공간", + "headroomOn": "{{node}}의 여유 공간" }, "empty": { "title": "모델을 찾을 수 없습니다", @@ -200,6 +201,8 @@ "browsing": "둘러보기", "pickHint": "모델을 선택하면 상세 정보가 표시됩니다.", "heroWithRam": "시스템 메모리 {{ram}}, 갤러리에 모델 {{count}}개.", + "heroWithCluster": "{{nodes}}개 노드 중 가장 큰 {{node}}에 {{vram}}, 갤러리에 모델 {{count}}개.", + "heroWithNode": "{{node}}에 {{vram}}, 갤러리에 모델 {{count}}개.", "byUseCase": "또는 용도로 시작하기", "pickText": "채팅, 추론, 임베딩", "pickVision": "이미지와 문서 읽기", diff --git a/core/http/react-ui/public/locales/pt-BR/models.json b/core/http/react-ui/public/locales/pt-BR/models.json index 9bcbdcffd..c354e89ae 100644 --- a/core/http/react-ui/public/locales/pt-BR/models.json +++ b/core/http/react-ui/public/locales/pt-BR/models.json @@ -152,7 +152,8 @@ "sha256": "SHA256", "backToAll": "Todos os modelos", "vramAt": "VRAM em {{context}}", - "headroom": "Margem de sobra" + "headroom": "Margem de sobra", + "headroomOn": "Folga em {{node}}" }, "empty": { "title": "Nenhum modelo encontrado", @@ -232,6 +233,8 @@ "browsing": "Explorando", "pickHint": "Selecione um modelo para ver seus detalhes.", "heroWithRam": "{{ram}} de memória do sistema, {{count}} modelos na galeria.", + "heroWithCluster": "{{vram}} em {{node}}, o maior de {{nodes}} nós, {{count}} modelos na galeria.", + "heroWithNode": "{{vram}} em {{node}}, {{count}} modelos na galeria.", "byUseCase": "Ou comece por um caso de uso", "pickText": "Chat, raciocínio, embeddings", "pickVision": "Leia imagens e documentos", diff --git a/core/http/react-ui/public/locales/zh-CN/models.json b/core/http/react-ui/public/locales/zh-CN/models.json index 181467b6a..667009901 100644 --- a/core/http/react-ui/public/locales/zh-CN/models.json +++ b/core/http/react-ui/public/locales/zh-CN/models.json @@ -142,7 +142,8 @@ "sha256": "SHA256", "backToAll": "全部模型", "vramAt": "{{context}} 时显存", - "headroom": "剩余显存" + "headroom": "剩余显存", + "headroomOn": "{{node}} 上的余量" }, "empty": { "title": "未找到模型", @@ -216,6 +217,8 @@ "browsing": "浏览中", "pickHint": "选择一个模型以查看详情。", "heroWithRam": "{{ram}} 系统内存，图库中有 {{count}} 个模型。", + "heroWithCluster": "{{node}} 上 {{vram}}，为 {{nodes}} 个节点中最大，图库中有 {{count}} 个模型。", + "heroWithNode": "{{node}} 上 {{vram}}，图库中有 {{count}} 个模型。", "byUseCase": "或从用途开始", "pickText": "对话、推理、向量", "pickVision": "读取图像与文档", diff --git a/core/http/react-ui/src/App.css b/core/http/react-ui/src/App.css index c7a2553db..679e16522 100644 --- a/core/http/react-ui/src/App.css +++ b/core/http/react-ui/src/App.css @@ -2697,116 +2697,82 @@ select.input { cursor: not-allowed; } -.scheduling-node-reference { - margin-bottom: var(--spacing-md); - overflow: hidden; +/* Key-value chip builder (node selectors, node labels) */ +.kvchips__chips { + display: flex; + flex-wrap: wrap; + gap: 4px; + margin-bottom: var(--spacing-xs); } -.scheduling-node-reference__toggle { +.kvchips__chip { align-items: center; - background: transparent; - border: 0; + background: var(--color-bg-tertiary); + border: 1px solid var(--color-border-subtle); + border-radius: var(--radius-sm); + display: inline-flex; + font-family: var(--font-mono); + font-size: 0.75rem; + gap: 4px; + padding: 2px 8px; +} + +.kvchips__chip-remove { + background: none; + border: none; + color: var(--color-text-muted); + cursor: pointer; + font-size: 0.625rem; + padding: 0; +} + +.kvchips__row { + align-items: stretch; + display: flex; + gap: var(--spacing-xs); + position: relative; +} + +.kvchips__add { + min-height: 36px; +} + +/* Anchored to the input row so the list covers what follows the field rather + than pushing the rest of the form down as the user types. */ +.kvchips__suggestions { + background: var(--color-bg-secondary); + border: 1px solid var(--color-border); + border-radius: var(--radius-sm); + box-shadow: var(--shadow-md); + left: 0; + list-style: none; + margin: 4px 0 0; + max-height: 220px; + overflow-y: auto; + padding: 4px; + position: absolute; + right: 0; + top: 100%; + z-index: 20; +} + +.kvchips__suggestion { + background: none; + border: none; + border-radius: var(--radius-sm); color: var(--color-text-primary); cursor: pointer; - display: flex; - font: inherit; - font-weight: var(--font-weight-semibold); - justify-content: space-between; - padding: var(--spacing-md); + display: block; + font-family: var(--font-mono); + font-size: 0.8125rem; + padding: var(--spacing-xs) var(--spacing-sm); text-align: left; width: 100%; } -.scheduling-node-reference__toggle:focus-visible { - outline: 2px solid var(--color-primary); - outline-offset: -2px; -} - -.scheduling-node-reference__content { - border-top: 1px solid var(--color-border-subtle); - padding: var(--spacing-md); -} - -.scheduling-node-reference__content > .text-note { - margin: 0 0 var(--spacing-sm); -} - -.scheduling-node-toolbar { - align-items: center; - display: flex; - gap: var(--spacing-md); - margin-bottom: var(--spacing-md); -} - -.scheduling-node-toolbar .input { - flex: 1; -} - -.scheduling-node-toolbar .text-meta { - flex: none; -} - -.scheduling-node-grid { - display: grid; - gap: var(--spacing-sm); - grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); -} - -.scheduling-node-card { +.kvchips__suggestion:hover, +.kvchips__suggestion--active { background: var(--color-bg-tertiary); - border: 1px solid var(--color-border-subtle); - border-radius: var(--radius-md); - min-width: 0; - padding: var(--spacing-sm); -} - -.scheduling-node-card__header { - align-items: center; - display: flex; - gap: var(--spacing-sm); - justify-content: space-between; - margin-bottom: var(--spacing-xs); -} - -.scheduling-node-status { - color: var(--color-text-muted); - font-size: 0.75rem; - text-transform: capitalize; -} - -.scheduling-node-status--online, -.scheduling-node-status--ready, -.scheduling-node-status--healthy { - color: var(--color-success); -} - -.scheduling-node-labels { - display: flex; - flex-wrap: wrap; - gap: 4px; -} - -.scheduling-node-label { - border: 1px solid var(--color-border-subtle); - border-radius: var(--radius-sm); - font-family: var(--font-mono); - font-size: 0.75rem; - overflow-wrap: anywhere; - padding: 2px 6px; -} - -.scheduling-node-message { - align-items: center; - color: var(--color-text-muted); - display: flex; - gap: var(--spacing-sm); - justify-content: center; - margin: var(--spacing-md) 0; - text-align: center; -} - -.scheduling-show-more { - margin-top: var(--spacing-md); } .scheduling-rule-actions { @@ -2815,17 +2781,32 @@ select.input { justify-content: flex-end; } +/* Second line of a rule's Model cell: the model an alias-keyed rule currently + governs. The cell itself is bold, so the weight is reset here rather than + inherited. */ +.scheduling-rule-target { + font-weight: 400; +} + +.scheduling-rule-target--broken { + font-weight: 400; + color: var(--color-warning); +} + +/* Status pill for a rule another rule already governs, so it has no effect. + Mirrors the unsatisfiable pill's shape. */ +.scheduling-rule-shadowed { + display: inline-block; + font-size: var(--text-xs); + padding: 2px 8px; + border-radius: var(--radius-sm); + font-weight: 600; + background: var(--color-bg-tertiary); + border: 1px solid var(--color-warning); + color: var(--color-warning); +} + @media (max-width: 640px) { - .scheduling-node-grid { - grid-template-columns: minmax(0, 1fr); - } - - .scheduling-node-toolbar { - align-items: stretch; - flex-direction: column; - gap: var(--spacing-xs); - } - .scheduling-rule-actions { width: 100%; } diff --git a/core/http/react-ui/src/components/SearchableModelSelect.jsx b/core/http/react-ui/src/components/SearchableModelSelect.jsx index 3d920fa4d..f63902956 100644 --- a/core/http/react-ui/src/components/SearchableModelSelect.jsx +++ b/core/http/react-ui/src/components/SearchableModelSelect.jsx @@ -7,7 +7,10 @@ import { useModels } from '../hooks/useModels' // query isn't treated as a chosen value. After a commit the field is cleared, // matching the add-and-clear flow. Default false keeps the as-you-type // behaviour single-value editors rely on. -export default function SearchableModelSelect({ value, onChange, capability, placeholder = 'Type or select a model...', style, commitOnly = false }) { +// hints: optional { [modelId]: string } shown as muted text beside an entry and +// searchable along with the name. Used to mark aliases with the model they +// point at, so a picker that lists both can tell them apart. +export default function SearchableModelSelect({ value, onChange, capability, placeholder = 'Type or select a model...', style, commitOnly = false, hints = {} }) { const { models, loading } = useModels(capability) const [query, setQuery] = useState('') const [open, setOpen] = useState(false) @@ -29,8 +32,10 @@ export default function SearchableModelSelect({ value, onChange, capability, pla return () => document.removeEventListener('mousedown', handler) }, []) + const needle = query.toLowerCase() const filtered = models.filter(m => - m.id.toLowerCase().includes(query.toLowerCase()) + m.id.toLowerCase().includes(needle) || + (hints[m.id] || '').toLowerCase().includes(needle) ) // Which item Enter will select — matches SearchableSelect behavior @@ -126,6 +131,11 @@ export default function SearchableModelSelect({ value, onChange, capability, pla color: var(--color-primary); font-weight: 600; } + .sms-hint { + color: var(--color-text-muted); + font-size: 0.75rem; + flex-shrink: 0; + } .sms-empty { padding: 8px 10px; font-size: 0.8125rem; @@ -172,6 +182,9 @@ export default function SearchableModelSelect({ value, onChange, capability, pla }} > {m.id} + {hints[m.id] && ( + {hints[m.id]} + )} {isEnterTarget && ( ↵ )} diff --git a/core/http/react-ui/src/components/nodes/KeyValueChips.jsx b/core/http/react-ui/src/components/nodes/KeyValueChips.jsx index 8dcc926c2..1d6baba45 100644 --- a/core/http/react-ui/src/components/nodes/KeyValueChips.jsx +++ b/core/http/react-ui/src/components/nodes/KeyValueChips.jsx @@ -1,4 +1,6 @@ -import { useState } from 'react' +import { useState, useRef, useEffect } from 'react' + +import { suggestKeys, suggestValues } from '../../utils/nodeLabelSuggestions' /** * Controlled chip-builder for { key: value } maps. Replaces the prior @@ -11,6 +13,14 @@ import { useState } from 'react' * labels editor). The component just renders chips and a key/value input * row. * + * With `suggestions` it also completes what the user types against the + * vocabulary the cluster actually uses. That is where label discovery lives on + * the scheduling page: labels only matter while a selector is being written, + * so browsing them belongs in the field rather than in a card standing open + * above the rules. The suggestions are never a constraint - a key no node + * reports yet still commits as typed, which is the workflow of writing a rule + * before labelling the nodes for it. + * * Props: * pairs - current map of key -> value * onAdd(k,v) - called when the user adds a pair (parent handles dedup @@ -18,45 +28,119 @@ import { useState } from 'react' * onRemove(k) - called when a chip's × is clicked * placeholderKey, placeholderValue - input hints * ariaLabel - accessible name for the section + * ariaLabelKey, ariaLabelValue - accessible names for the two inputs + * addLabel - accessible name for the commit button + * suggestions - label index from utils/nodeLabelSuggestions; omit for none */ -export default function KeyValueChips({ pairs, onAdd, onRemove, placeholderKey = 'key', placeholderValue = 'value', ariaLabel }) { +export default function KeyValueChips({ + pairs, onAdd, onRemove, + placeholderKey = 'key', placeholderValue = 'value', + ariaLabel, ariaLabelKey = 'Key', ariaLabelValue = 'Value', + addLabel = 'Add', suggestions, +}) { const [k, setK] = useState('') const [v, setV] = useState('') + // Which input owns the open list, and which of its options is armed for + // Enter. -1 means the user is typing free text and Enter should commit the + // pair rather than pick anything. + const [openField, setOpenField] = useState(null) + const [active, setActive] = useState(-1) + const rowRef = useRef(null) + + const entries = pairs ? Object.entries(pairs) : [] + + const options = !suggestions || !openField + ? [] + : openField === 'key' + ? suggestKeys(suggestions, k, entries.map(([key]) => key)) + : suggestValues(suggestions, k.trim(), v) + + // A click anywhere else is a dismissal. Without this the list survives the + // user moving on to the rest of the form and covers it. + useEffect(() => { + if (!openField) return undefined + const onDocumentPointerDown = (event) => { + if (!rowRef.current?.contains(event.target)) setOpenField(null) + } + document.addEventListener('mousedown', onDocumentPointerDown) + return () => document.removeEventListener('mousedown', onDocumentPointerDown) + }, [openField]) const add = () => { const key = k.trim() if (!key) return onAdd(key, v.trim()) setK(''); setV('') - } - const onKeyDown = (e) => { - if (e.key === 'Enter') { e.preventDefault(); add() } + setOpenField(null); setActive(-1) } - const entries = pairs ? Object.entries(pairs) : [] + const pick = (field, option) => { + if (field === 'key') setK(option) + else setV(option) + setOpenField(null) + setActive(-1) + } + + const onKeyDown = (field) => (e) => { + const open = openField === field && options.length > 0 + if (e.key === 'ArrowDown' && open) { + e.preventDefault() + setActive(current => (current + 1) % options.length) + return + } + if (e.key === 'ArrowUp' && open) { + e.preventDefault() + setActive(current => (current <= 0 ? options.length - 1 : current - 1)) + return + } + if (e.key === 'Escape' && openField) { + e.preventDefault() + setOpenField(null) + setActive(-1) + return + } + if (e.key === 'Enter') { + e.preventDefault() + // Enter completes the suggestion the user armed, and commits the pair + // otherwise. Committing a half-typed key because a list happened to be + // open is the error this ordering avoids. + if (open && active >= 0) pick(field, options[active]) + else add() + } + } + + const listId = 'kvchips-suggestions' + const inputProps = (field, value, setValue, placeholder, label) => ({ + className: 'input flex-1', + type: 'text', + role: suggestions ? 'combobox' : undefined, + 'aria-expanded': suggestions ? openField === field : undefined, + 'aria-controls': suggestions && openField === field ? listId : undefined, + 'aria-autocomplete': suggestions ? 'list' : undefined, + 'aria-label': label, + placeholder, + value, + onChange: (e) => { + setValue(e.target.value) + if (suggestions) { setOpenField(field); setActive(-1) } + }, + onFocus: () => { if (suggestions) { setOpenField(field); setActive(-1) } }, + onKeyDown: onKeyDown(field), + }) + return (
{entries.length > 0 && ( -
+
{entries.map(([key, val]) => ( - + {key}={val} @@ -64,32 +148,35 @@ export default function KeyValueChips({ pairs, onAdd, onRemove, placeholderKey = ))}
)} -
- setK(e.target.value)} - onKeyDown={onKeyDown} - /> - setV(e.target.value)} - onKeyDown={onKeyDown} - /> +
+ + + {options.length > 0 && ( +
    + {options.map((option, index) => ( +
  • + +
    • + ))} +
+ )}
) diff --git a/core/http/react-ui/src/hooks/useChat.js b/core/http/react-ui/src/hooks/useChat.js index 567cf7f55..16f99f649 100644 --- a/core/http/react-ui/src/hooks/useChat.js +++ b/core/http/react-ui/src/hooks/useChat.js @@ -1,6 +1,7 @@ import { useState, useCallback, useRef } from 'react' import { API_CONFIG } from '../utils/config' import { apiUrl } from '../utils/basePath' +import { effectiveSystemPrompt } from '../utils/systemPrompt' import { useDebouncedEffect } from './useDebounce' const thinkingTagRegex = /([\s\S]*?)<\/thinking>|([\s\S]*?)<\/think>|<\|channel>thought([\s\S]*?)/g @@ -348,8 +349,12 @@ export function useChat(initialModel = '') { // Build messages array for API const chat = chats.find(c => c.id === chatId) const messages = [] - if (chat?.systemPrompt) { - messages.push({ role: 'system', content: chat.systemPrompt }) + // Omit empty/whitespace system prompts so the model YAML system_prompt + // (and tokenizer chat-template defaults) are not suppressed by a blank + // system turn from Chat Settings. + const systemPrompt = effectiveSystemPrompt(chat?.systemPrompt) + if (systemPrompt) { + messages.push({ role: 'system', content: systemPrompt }) } // Filter out thinking/reasoning/tool_call/tool_result messages. // options.baseHistory lets callers (e.g. mid-conversation retry) pass the @@ -358,6 +363,7 @@ export function useChat(initialModel = '') { const baseHistory = options.baseHistory || chat?.history || [] const historyForApi = baseHistory.filter(m => m.role !== 'thinking' && m.role !== 'reasoning' && m.role !== 'tool_call' && m.role !== 'tool_result' + && !(m.role === 'system' && !effectiveSystemPrompt(typeof m.content === 'string' ? m.content : '')) ) messages.push(...historyForApi, { role: 'user', content: messageContent }) diff --git a/core/http/react-ui/src/hooks/useRecommendedModels.js b/core/http/react-ui/src/hooks/useRecommendedModels.js index ca6090177..c970c7625 100644 --- a/core/http/react-ui/src/hooks/useRecommendedModels.js +++ b/core/http/react-ui/src/hooks/useRecommendedModels.js @@ -1,6 +1,7 @@ import { useState, useEffect } from 'react' import { modelsApi } from '../utils/api' import { useResources } from './useResources' +import { modelBudget } from '../utils/modelBudget' // Data-driven "recommended for your hardware" model picks. The gallery exposes // no popularity/download signal and the list response carries no size, so we: @@ -21,13 +22,21 @@ const DEFAULT_CTX = 4096 export const isNvfp4Name = (name) => /nvfp4/i.test(name || '') export function hasNvidiaGpu(resources) { + // A distributed controller has no GPUs of its own, so the question is + // whether any worker does. The registry reports the cluster's best node, + // and it is that node these picks have to run on. + if (resources?.cluster?.enabled) return !!resources.cluster.is_gpu return Array.isArray(resources?.gpus) && resources.gpus.some(g => (g?.vendor || '').toLowerCase() === 'nvidia') } export function recommendTier(resources) { - const isGpu = resources?.type === 'gpu' - const vram = resources?.aggregate?.total_memory || 0 + // Same reading the models page sizes against: the cluster's largest node in + // distributed mode, the local host otherwise. Ranked against the controller, + // a fleet of A100s was recommended the models a GPU-less pod could run. + const budget = modelBudget(resources) + const isGpu = budget.scope === 'cluster' ? budget.hasGpu : resources?.type === 'gpu' + const vram = budget.totalMemory if (!isGpu || vram <= 0) return { id: 'cpu', vram: 0 } if (vram < 8 * GB) return { id: 'gpu-small', vram } if (vram < 24 * GB) return { id: 'gpu-mid', vram } diff --git a/core/http/react-ui/src/pages/Chat.jsx b/core/http/react-ui/src/pages/Chat.jsx index 5866a2514..d799efba0 100644 --- a/core/http/react-ui/src/pages/Chat.jsx +++ b/core/http/react-ui/src/pages/Chat.jsx @@ -21,6 +21,8 @@ import { useOperations } from '../hooks/useOperations' import { relativeTime } from '../utils/format' import { copyToClipboard } from '../utils/clipboard' +const FOCUS_MODE_KEY = 'localai_chat_focus_mode' + function getLastMessagePreview(chat) { if (!chat.history || chat.history.length === 0) return '' for (let i = chat.history.length - 1; i >= 0; i--) { @@ -405,12 +407,20 @@ export default function Chat() { // Focus mode: once a conversation has at least one message we slim the // surrounding chrome (collapse the global app rail, fade non-essential // header items). Esc gives the user back the full chrome for the rest of - // this session. + // this session. The settings drawer offers a persistent opt-out. const isInConversation = (activeChat?.history?.length || 0) > 0 const [focusOverride, setFocusOverride] = useState(false) - const focusActive = isInConversation && !focusOverride + const [focusModeEnabled, setFocusModeEnabled] = useState(() => { + try { return localStorage.getItem(FOCUS_MODE_KEY) !== 'false' } catch (_) { return true } + }) + const focusActive = focusModeEnabled && isInConversation && !focusOverride const prevAppCollapseRef = useRef(null) + const toggleFocusMode = (next) => { + setFocusModeEnabled(next) + try { localStorage.setItem(FOCUS_MODE_KEY, String(next)) } catch (_) {} + } + const artifacts = useMemo( () => canvasMode ? extractCodeArtifacts(activeChat?.history, 'role', 'assistant') : [], [activeChat?.history, canvasMode] @@ -1110,6 +1120,20 @@ export default function Chat() { />
)} +
+
+ + {t('settings.focusMode')} + + + {t('settings.focusModeDesc')} + +
+ +