* chore(deps): bump the pip group across 6 directories with 1 update
Bumps the pip group with 1 update in the /backend/python/ace-step directory: torch.
Bumps the pip group with 1 update in the /backend/python/llama-cpp-quantization directory: torch.
Bumps the pip group with 1 update in the /backend/python/longcat-video directory: torch.
Bumps the pip group with 1 update in the /backend/python/sglang directory: torch.
Bumps the pip group with 1 update in the /backend/python/trl directory: torch.
Bumps the pip group with 1 update in the /backend/python/vllm-omni directory: torch.
Updates `torch` from 2.10.0+rocm7.0 to 2.12.1+cpu
Updates `torch` from 2.10.0 to 2.12.1+cpu
Updates `torch` from 2.12.1 to 2.12.1+cu130
Updates `torch` from 2.9.0 to 2.12.1+cpu
Updates `torch` from 2.10.0 to 2.12.1+cpu
Updates `torch` from 2.7.0 to 2.12.1+cu130
---
updated-dependencies:
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cu130
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cu130
dependency-type: direct:production
dependency-group: pip
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(deps): preserve platform-specific torch requirements
Keep the 2.12.1 CPU bump only where uv resolves it cleanly, and restore ROCm, CUDA, MPS, and unrelated transformers constraints that Dependabot rewrote to incompatible wheel variants.
Assisted-by: Codex:gpt-5 [uv]
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Every vibevoice requirements file pulled diffusers straight from
git+https://github.com/huggingface/diffusers. That branch now reports
itself as 0.40.0.dev0 and requires huggingface-hub>=1.23.0,<2.0, while
transformers>=4.51.3,<5.0.0 (which upstream VibeVoice mandates) still
caps huggingface-hub at <1.0. Because a git URL offers the resolver
exactly one candidate version, uv has nothing to backtrack to and the
install fails outright:
Because only diffusers==0.40.0.dev0 is available and diffusers==0.40.0.dev0
depends on huggingface-hub>=1.23.0,<2.0 [...] we can conclude that your
requirements are unsatisfiable.
This broke the vibevoice build on every variant - cpu (amd64/arm64),
cuda 12/13, l4t 12/13, intel and rocm, plus the darwin metal job - and
has been failing the weekly full-matrix rebuild for three weeks. It is
not caught by master pushes because backend builds are path-filtered
there, so it only surfaces on the Sunday cron and on release tags.
Use the PyPI package instead. That is what upstream VibeVoice declares
in its own pyproject.toml, and what every other LocalAI backend already
does - vibevoice was the only one tracking the git branch. With a real
release series available the resolver settles on diffusers 0.39.0 with
huggingface-hub 0.36.2 and transformers 4.57.6, and it can keep
backtracking on its own if upstream shifts again.
Verified with uv pip compile against cpu, cublas12, cublas13, hipblas,
intel, mps and l4t13: all resolve to that same coherent set. l4t12 only
resolves on aarch64, since its Jetson index ships no x86_64 torch wheel.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Bash] [uv]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Adds inference parameter defaults for the minimax-m3 model family and
includes a vendored patch of upstream llama.cpp PR #24523 to recognize
the minimax-m3 architecture. Once the upstream PR merges, the patch can
be removed and LLAMA_VERSION bumped normally.
Changes:
- backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch:
vendored patch from ggml-org/llama.cpp#24523 (Preliminary MiniMax-M3
support). Applied by prepare.sh during the build; keeps the pinned
LLAMA_VERSION pointing at the latest upstream tag.
- core/config/inference_defaults.json: add minimax-m3 family entry
(temperature=1.0, top_p=0.95, top_k=40, min_p=0.01,
repeat_penalty=1.0, matching the existing minimax defaults) and
register it in the patterns list before the shorter minimax-m2.7
entry for correct longest-match-first ordering.
Upstream: depends on ggml-org/llama.cpp#24523
Closes: https://github.com/mudler/LocalAI/issues/10820
Signed-off-by: Nandana Dileep <110280757+nandanadileep@users.noreply.github.com>
The worker's gRPC port allocator grew monotonically with no upper bound:
nextPort started at the base port and incremented whenever freePorts was
empty, and nothing checked 65535. Past that it handed out integers that
cannot be bound, surfacing as an opaque "backend won't start".
#10961 estimated this needed ~15,000 concurrent-peak allocations, i.e.
effectively unreachable. It is not, because of a second defect: the
"process died unexpectedly" branch in startBackend deleted the process
map entry without releasing its port at all. That port was leaked, never
quarantined and never reused. A crash-looping backend leaks one port per
restart, so a backend dying every 30s walks 50051 to 65535 in about five
days. The leak, not concurrent peak, is the realistic route to exhaustion.
Fixing the leak alone would have been wrong. Releasing that port makes it
re-bindable, and the death path is the one teardown path with no
request/reply to carry StoppedProcessKeys back to the controller (#10952's
eager row removal), so a stale NodeModel row could then resolve to a live
listener belonging to a different backend. probeHealth verifies liveness,
not identity, so the request is silently misrouted. The 15s port
quarantine does not cover this: the only reaper is the per-model health
check at ~45s, and it can be disabled outright. The residual was masked
only because the port was never rebound.
So both are fixed together:
- The allocator takes an explicit [basePort, LOCALAI_GRPC_MAX_PORT] range
and returns ErrNoFreePort naming the range, the live backend count, the
quarantined count, and the knob to raise. Exhaustion is now diagnosable
instead of surfacing as an unbindable port.
- Released ports carry per-key affinity: a port is offered back to the
process key that last held it before any other key. Process keys
(modelID#replica) and NodeModel rows (nodeID, modelName, replicaIndex)
are isomorphic, so a port that can only be re-bound by its previous
owner can only ever be named by that owner's row, which that key's
re-registration overwrites. Misrouting to a different model becomes
impossible by construction rather than by racing the quarantine timer.
Affinity is a preference, not a reservation: under range pressure an owned
port is stolen with a warning, because a guaranteed outage is worse than a
rare misroute window on a port long out of quarantine. Claiming a port
evicts its previous owner's entry, keeping ownership injective over ports
so the affinity map can never exceed the range width regardless of how
many distinct model keys the worker sees.
Ownership also expires. It is only load-bearing while a controller row
could still name the port, which the per-model reaper bounds at roughly
45s, so it lapses after five minutes and the port becomes ordinary free
space again. Holding it indefinitely would have made every distinct model
the worker ever served consume a port permanently: every release path is
keyed, so nothing would ever be unowned, the allocator would climb to the
end of its range on distinct-key count rather than concurrency, stealing
would become routine, and the steal warning would tell operators to widen
a range that was not the constraint. With expiry, reaching the steal
branch means the worker is genuinely out of concurrent capacity, so that
advice is correct when it appears.
Closes#10961Closes#10952
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(distributed): backend discovery hid worker-installed backends
Backend discovery endpoints filter on installed-state, which on a
distributed controller derives from the controller's own filesystem. A
backend lives on the worker node that runs it, so every backend an admin
installed on a GPU worker read as "not installed" and vanished from the
listing. #10947 fixed the sibling capability filter on the same endpoints,
so a fine-tuning-capable GPU worker now made the backend listable while
the installed-state filter still dropped it: the dropdown stayed empty.
The controller cannot derive this locally, but it already aggregates the
per-node view that GET /backends renders, so discovery reuses the active
BackendManager rather than growing a second path. Three surfaces shared the
root cause and route through the same helper now:
- GET /backends/available (Installed is now cluster-wide)
- GET /api/fine-tuning/backends
- GET /api/quantization/backends
The response stays a boolean rather than an installed-on-N-of-M count:
per-node install state is already served by GET /backends nodes[], and
per-node control by POST /api/nodes/:id/backends/install, so a summary is
all these dropdowns need.
A nil provider (single-node) leaves the local filesystem as the only source
and reproduces today's listing exactly, and a registry error degrades to
that same listing instead of blanking the catalog.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(worker): reap deleted backends and stop models that live on a worker
Three related backend-lifecycle defects, all reachable from the same
production incident on a Jetson/Thor worker: a deleted backend's gRPC
process survived ~40 minutes with its directory removed from disk, a later
model load was routed to that orphan and failed with a certifi path pointing
into the deleted directory, and the admin could not stop the model because
the frontend reported it as not loaded.
1. backend.delete orphaned the process it claimed to delete
------------------------------------------------------------
s.processes is keyed by `modelID#replicaIndex` (buildProcessKey), so the
backend name never appeared in a key and was recorded nowhere on the
process. backend.delete resolved its target via isRunning/stopBackend, whose
prefix path only matches a bare *modelID* - a delete keyed on a backend name
resolved to zero keys, the stop silently no-op'd, and the files were removed
out from under a live process.
The install fast path then handed that orphan back out: it returns any live
process for the (model, replica) slot without checking which backend started
it, so a reinstalled variant inherited the deleted backend's port.
- Record backendName on backendProcess, threaded installBackend ->
startBackend.
- Add resolveProcessKeysForBackend, matching the recorded name and resolving
alias <-> concrete via ListSystemBackends *before* DeleteBackendFromSystem
erases the metadata that carries the alias. Alias resolution failure
degrades to name-only matching so a delete never fails on it.
- backend.stop goes through resolveStopTargets, which accepts a backend
name, a model name, or an exact modelID#replica key. Its payload field is
named "backend" but is published with all three meanings: the admin UI
sends a backend name, UnloadRemoteModel sends a model name, and the
router's abandoned-load reap (#10948) sends an exact replica key.
Narrowing it to backend names alone would strand the latter two.
backend.delete stays strict - its identifier is unambiguously a backend.
- Gate the install fast path on processMatchesBackend so a slot held by a
different backend is restarted rather than reused. Processes with no
recorded name (pre-upgrade) are accepted, so rollout does not restart
every running backend.
- stopBackendExact reports a real stop failure - the process still being
alive afterwards, which is precisely what finishBackendStop already
detects to keep the entry and its port reserved - and backend.delete no
longer replies success when it knew about a process and could not kill it.
"No process was running" stays a success but is logged, so the orphan case
is visible rather than silent.
2. /backend/shutdown reported a running model as missing
---------------------------------------------------------
ModelLoader.deleteProcess short-circuits on a miss in this replica's
in-memory store. In distributed mode the authoritative record of "is this
model loaded" is the shared node registry: a frontend replica that never
served the model itself (load balancer picked a peer, or the replica
restarted) has no local entry. The remote unload path that pkg/model
documents ("when ShutdownModel is called for a model with no local process,
UnloadRemoteModel is called") sat behind that short-circuit, unreachable in
exactly the case it exists for. #10865 reworked this function but kept the
short-circuit at the top, so the gap survived that refactor.
- deleteProcess consults the remote unloader on a local-store miss, via a
shared unloadRemote helper so this branch and the existing
no-local-process branch both prefer #10865's RemoteModelContextUnloader,
preserving force propagation across the distributed boundary.
- UnloadRemoteModelContext reports ErrRemoteModelNotLoaded when no node has
the model; it previously returned nil, making a no-op stop
indistinguishable from a real one. The converse case (nodes have it, none
could be stopped) already errors since #10865 joined the per-node
failures, so that half of the original fix was dropped as redundant.
- Only when the model is absent locally AND cluster-wide does the endpoint
report not-found, now 404 naming both scopes rather than a bare 500.
- modelNotFoundErr becomes the exported ErrModelNotFound so the HTTP layer
can map it without string matching; watchdog's identity comparison becomes
errors.Is.
3. Coverage for the bounded Free() that #10865 shipped untested
----------------------------------------------------------------
The original branch also bounded the pre-stop Free(), but #10865 landed that
fix first (workerBackendFreeTimeout, applied in both stopBackendExact and
handleModelUnload). That production change is therefore DROPPED here as
superseded - master's version is strictly better, since it also releases the
supervisor mutex across the call and keeps the port reserved until
termination completes.
What #10865 did not ship is a test, and the bound is load-bearing: the
router-side reap in #10948 sends backend.stop for an abandoned load, and
against a wedged backend an unbounded Free would swallow that stop before it
reached the process. Nothing failed if the bound regressed.
The spec stands up a real gRPC backend server whose Free handler never
returns - what a Python backend looks like when its single worker thread
(PYTHON_GRPC_MAX_WORKERS=1 on 37 backends) is occupied by a stuck LoadModel.
A stub socket is not sufficient and was tried first: without a completed
HTTP/2 handshake, gRPC's own ~20s connect timeout ends the call, so that
version passed against the very bug it targets. With the connection READY,
only the caller's deadline can end it, so the spec hangs to its 60s limit if
the timeout is removed and passes with it.
Its fixture process is deliberately never started. go-processmanager v0.1.1
writes Process.pid from readPID() without synchronization, so a live process
races its own monitor goroutine under -race - reproducible with a bare
Run()+Stop() and unrelated to this spec. Since
scripts/model-lifecycle-conformance.sh runs this package with -race and is
fail-closed, starting one would turn that gate red on an upstream defect. An
unstarted process still proves the point: the stop is reached and the slot
released, which is exactly what an unbounded Free prevents.
Verified: make lint (new-from-merge-base origin/master) reports 0 issues;
scripts/model-lifecycle-conformance.sh passes all three stages including the
FizzBee liveness check (1458 states, IsLive: true).
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): keep remote unload idempotent, ask presence separately
2035a4d25 made UnloadRemoteModel return ErrRemoteModelNotLoaded when no node
holds the model, so ShutdownModel could answer 404 instead of a misleading
500. That narrowed a shared adapter contract to serve one caller and broke
the documented idempotent-unload guarantee, which CI caught on PR #10956:
[FAIL] Node Backend Lifecycle (NATS-driven) > NATS backend.stop events
should be no-op for models not on any node [Distributed]
Expected success, but got: model not loaded on any node
The spec name states the contract outright. The matching unit assertion was
updated in that commit; this e2e one was missed because it lives under
tests/e2e/ with no build tags and does not run in package-scoped test runs.
Caller audit - who breaks when an idempotent unload becomes an error:
- pkg/model/watchdog.go:902 (LRU memory reclaimer) is the serious one. It
untracks a model ONLY when shutdown returns nil or ErrModelNotFound. A new
error type means the model is never untracked, so the reclaimer keeps
re-selecting the same entry and never reclaims - a live wedge whenever a
local store entry outlives the remote model.
- core/services/galleryop/managers_local.go:43 (DeleteModel) would warn on
every deletion of an already-unloaded model.
- core/services/modeladmin/{state,config,remote_sync}.go stop instances
best-effort against models that are frequently not loaded.
- deleteProcess itself: the no-local-process branch returns the unload result
directly, so a stale local entry for a model no longer on any node turned a
previously-successful cleanup into a failure.
Only ShutdownModel wants the distinction, and only on the local-store-miss
path. So the distinction moves to the caller instead of the contract:
- UnloadRemoteModel/UnloadRemoteModelContext return nil again when no node
has the model, and ErrRemoteModelNotLoaded is removed.
- New optional RemoteModelPresenceChecker (HasRemoteModel) answers the
question directly. deleteProcess consults it BEFORE unloading, because an
idempotent unload cannot report afterwards whether anything was stopped.
Absent locally AND cluster-wide is the only case that reports 404.
- A failed registry lookup is surfaced rather than reported as absence: an
unreachable registry is not evidence a model is gone, and answering a
confident 404 off a failed lookup is how an operator gets told a running
model does not exist.
- Unloaders that predate the extension keep working - deleteProcess attempts
the unload rather than refusing it - and compile-time assertions in the
nodes package now pin all three optional interfaces, since both are
consumed by runtime type assertion where drift degrades behavior silently
instead of failing the build.
The contract is now pinned at both levels that disagreed, each spec pointing
at the other: "with no nodes returns nil" in unloader_test.go and "should be
no-op for models not on any node" in node_lifecycle_test.go.
Verified: full distributed e2e suite 233 passed / 0 failed (the suite that
failed 232/1 in CI); pkg/model and core/services/nodes green; make lint
new-from-merge-base reports 0 issues.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): drop replica rows when a worker stops a backend
A worker returns a stopped backend's gRPC port to its allocator as soon as
the process is confirmed dead, and hands it to the next backend that starts.
The controller's NodeModel row for the old address survives, and both
SmartRouter.probeHealth and the HealthMonitor per-model probe verify
liveness, not identity, so once an unrelated backend binds the recycled port
the stale row passes every check and the request is served by the wrong
backend instead of failing.
backend.delete is newly able to trigger this: before #10956 a delete never
actually stopped a process, so it never recycled a port. backend.upgrade has
the identical gap and always did — upgradeBackend force-stops every process
using the binary and starts none back up, while
DistributedBackendManager.UpgradeBackend never removes rows. model.unload is
the one path that gets this right today: it calls RemoveAllNodeModelReplicas
straight after StopBackend.
Report the process keys the worker terminated on the delete and upgrade
replies, and drop the matching rows in RemoteUnloaderAdapter, which already
holds a ModelLocator with RemoveNodeModel. All three call sites funnel
through that adapter, so no new interface, DB migration, or proto change is
needed. A key is reported only once its process is confirmed gone, so the
list stays trustworthy on the partial-failure replies too.
Old workers never populate the new fields. ReportsStoppedProcesses tells
"stopped nothing" apart from "does not report", so an old worker's silence
falls back to the pre-existing probe-based staleness recovery instead of
being mistaken for a completed cleanup.
Quarantine released ports for a short window as an interlock covering the
NATS round-trip between the worker freeing the port and the controller
dropping the row. It is deliberately not derived from HealthCheckInterval:
that cadence is operator-tunable and the per-model reaper can be disabled
outright, so coupling a worker-local constant to it would be silently wrong
on some clusters. Eager row removal is the fix; the delay only closes the
handoff gap.
Identity verification in probeHealth was considered and rejected: Health and
Status carry no backend identity, so it needs a proto change plus an
implementation in 36 Python and 4 C++ Health servicers, it is fail-open for
any backend not yet rebuilt, and the probeCache short-circuit means it would
not even execute during the 30s window where the misroute happens.
Fixes#10952
Refs #10954, #10956
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(deps): bump go-processmanager, assert real backend termination
go-processmanager wrote Process.PID from readPID() with no synchronization
while its own monitor goroutine cleared the same field on exit, so a bare
Run()+Stop() tripped the race detector without any concurrent access from
the caller. LocalAI hit this on every backend stop.
Upstream fixed it in a94e2b7 by guarding PID with a mutex and adding
CurrentPID() as a race-safe accessor. The exported field was kept to avoid
a breaking change but is now deprecated: a direct read still races the
monitor. No tag carries the fix yet, so pin the pseudo-version.
GetGRPCPID reads through CurrentPID() instead of the field. The accessor
returns the same string under an RLock, so the empty-PID and strconv error
paths are unchanged; it is the only direct field read in the tree.
With the race gone, the Free-timeout spec no longer has to leave its
fixture process unstarted. It now runs a real child and asserts the child
genuinely exits, which is exactly what the earlier workaround gave up: the
spec could show the stop was reached and the slot released, but not that
SIGTERM ever landed. Termination is observed through Done(), which closes
only once the library has waited on the child. The pidfile-based liveness
helpers cannot serve here, because Stop() deletes the pidfile while
releasing the handle and so reports "not alive" even if no signal was sent.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update CrispStrobe/CrispASR
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(crispasr): initialize only declared submodules
The latest upstream commit contains an undeclared CrispASR gitlink that makes a blanket recursive submodule update fail. Limit initialization to the two submodules used by the backend build.
Assisted-by: Codex:gpt-5 [Codex]
* fix(crispasr): resolve vendored WebRTC from project root
CrispASR now builds a vendored WebRTC VAD, but its include paths assume CrispASR is the top-level CMake project. Extend the existing embedded-project rewrite to the shared third_party root.
Assisted-by: Codex:gpt-5 [Codex]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(gpu): detect GPUs via sysfs when no pci.ids database is present
ghw.GPU() calls pci.New() before it reads /sys/class/drm and fails
outright when it cannot find a pci.ids database file. jaypipes/pcidb
embeds no database and has network fetch disabled by default, so on an
image that ships no pci.ids, GPU enumeration returns an error and every
detection path downstream goes dark.
The Dockerfile installs pciutils only in the vulkan and cublas branches,
so the Intel image had no pci.ids. A correctly passed-through Arc A310
was reported as "No GPU detected" with zero VRAM even though clinfo and
sycl-ls both enumerated it inside the same container. NVIDIA and AMD
images were shielded by their nvidia-smi / rocm-smi binary fallbacks;
Intel has no equivalent, leaving it fully exposed.
Read PCI vendor IDs directly from /sys/class/drm/card*/device/vendor,
which needs no database, and consult that from DetectGPUVendor. The
same scan replaces the ghw-only guard in getIntelGPUMemory, which is
what had been blocking the working clinfo path and keeping VRAM at
zero. Install hwdata in the base image stage as well, so ghw stops
failing for every image variant rather than only Intel.
Also apply the documented NVIDIA > AMD > Intel priority to the ghw
path, which previously returned whichever card DRM enumerated first
and so reported "intel" on a machine with an Intel iGPU at card0 and
an NVIDIA dGPU at card1.
HasGPU() carried the same blindness plus one of its own: it matched
the requested vendor against ghw's card description with a
case-sensitive Contains, so "nvidia" never matched the pci.ids
spelling "NVIDIA Corporation". It only worked because that same
description embeds the lowercase kernel driver name ("nvidia",
"amdgpu"), and it returned false outright whenever ghw errored. Route
it through the shared vendor lookup so it matches case-insensitively
and falls back to sysfs. It feeds the GPU option and NGPULayers
defaults in core/config/gguf.go.
Fixes#10941
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(gpu): key vendor detection off the numeric PCI ID in both paths
The ghw and sysfs legs were identifying vendors by different means: ghw
by substring-matching the pci.ids vendor name, sysfs by the numeric PCI
vendor ID. ghw already exposes that same numeric ID via
DeviceInfo.Vendor.ID, read from the kernel's modalias rather than from
the database, so the name matching was both a duplicate mechanism and
the weaker of the two.
It is weaker because a card absent from an outdated pci.ids gets
Name: "unknown" while its ID is still correct. Detection then failed
even though ghw had enumerated the card successfully. Verified in a
container with a vendor-less pci.ids and an Arc's modalias: before,
DetectGPUVendor returned ""; after, "intel".
Both legs now resolve through the same pciVendorIDs table and share the
hex parsing, with the vendor name kept only as a fallback for devices
exposing no parseable ID.
ghwHasVendor is deliberately not a priority pick, unlike vendorFromGHW:
HasGPU("intel") must stay true on a hybrid-graphics host whose discrete
NVIDIA card outranks the integrated Intel one.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gpu): silence the gosec G304 on the sysfs attribute read
gosec flags os.ReadFile with a non-literal path. The path here is the
DRM root (a package constant in production, a temp dir under test)
joined with a ReadDir entry name and a fixed attribute filename, so no
external input reaches it.
gosec's suggested autofix, os.Root, cannot be used: /sys/class/drm/cardN
is a symlink into the PCI device tree, and os.Root refuses to traverse
it ("path escapes from parent"), which would disable the whole scan.
Assisted-by: Claude:claude-opus-4-8 gosec golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
request.language usually carries an ISO 639-1 code (e.g. "de"), which
OpenAI-compatible clients such as Home Assistant / wyoming_openai send,
but qwen_asr.validate_language() only accepts full English names
("German") and raises ValueError otherwise. Normalize the requested
language: accept full names case-insensitively, translate ISO codes
(with optional region suffix like "de-DE") to the expected name, and
pass anything unrecognised through so qwen_asr still reports it clearly.
Fixes#10958
Signed-off-by: Tai An <antai12232931@outlook.com>
llama.cpp picks a random per-process media marker (ggml-org/llama.cpp#21962),
so LocalAI renders the prompt with a "<__media__>" sentinel and swaps in the
backend's real marker after probing ModelMetadata.
That probe was gated on an exact match against "llama-cpp", the gallery's meta
backend name. A model config pinning a concrete build ("vulkan-llama-cpp",
"cuda12-llama-cpp", "rocm-llama-cpp", ... and their -development counterparts)
runs the same llama.cpp gRPC server but skipped the probe, so MediaMarker
stayed empty, no substitution happened, and the prompt reached mtmd still
carrying the sentinel. mtmd_tokenize then counted zero markers against one
bitmap and every image request failed with "Failed to tokenize prompt".
The same early return also skipped thinking-mode detection and tool-format
marker extraction, so a pinned variant silently lost reasoning and native
tool-call parsing too.
Add IsLlamaCppBackend, which recognises the whole variant family (plus the
empty auto-detect name, which resolves to llama.cpp) while excluding
ik-llama.cpp, a separate engine that merely shares the suffix.
Fixes#10945
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): make the remote LoadModel deadline configurable
The router hardcoded a 5 minute gRPC deadline for the remote LoadModel
call. Staging finishes before the timer starts, so those five minutes
cover only the worker backend's own checkpoint load and pipeline init.
A cold load of meituan-longcat/LongCat-Video-Avatar-1.5 (~83 GB) on an
ARM64 Thor worker fails at exactly 302s with DeadlineExceeded while the
backend process is still making progress (CPU time accumulating, RSS
moving as weights are mapped), so the load was cut short rather than
wedged.
Add LOCALAI_NATS_MODEL_LOAD_TIMEOUT / --model-load-timeout mirroring the
existing backend-install timeout knob, defaulting to 5m so unset
clusters keep today's behaviour.
The cold-load hold ceiling (which bounds how long one load may hold the
per-model advisory lock) was derived from the install timeout alone, so
raising the load deadline past it would have been silently clipped.
Derive it from both budgets via ModelLoadCeilingFor:
max(install + load + 5m staging margin, 25m)
With the defaults that is 15m + 5m + 5m = 25m, identical to the previous
constant, and the 25m floor means shrinking either budget can never
tighten the ceiling below what clusters relied on before.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): reap the abandoned replica when a remote load times out
The gRPC deadline on the remote LoadModel call only cancels the client
side. A backend blocked in a synchronous weight load never observes its
cancelled handler context, so when scheduleAndLoad gave up it left the
worker loading with nobody waiting for the result.
Observed on an ARM64 Thor worker loading LongCat-Video-Avatar-1.5: the
client returned DeadlineExceeded at 302s, and the backend process was
still alive 30 minutes later having pulled ~57GB from HuggingFace. Every
retry stacked another multi-GB loader on the worker; they had to be
reaped by hand via POST /api/nodes/:id/models/unload.
Send backend.stop for the exact `modelID#replicaIndex` process key we
just abandoned. The exact key matters: a bare model ID stops every
replica on that node, including healthy ones serving traffic.
Only a deadline or cancellation triggers the reap. Any other LoadModel
failure is the backend answering, which means its handler returned and
the process is idle - stopping it there would discard a warm process and
its downloaded weights. The reap is best-effort and never replaces the
load error the caller is waiting on.
The `modelID#replicaIndex` format was already hand-rolled in two places
(the worker's buildProcessKey and pkg/model's log store). Rather than add
a third, export model.BackendProcessKey from pkg/model, the lowest common
dependency of both sides.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 golangci-lint
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(model-artifacts): materialize longcat-video checkpoints on the controller
longcat-video loads a checkpoint directory: its backend.py takes
request.ModelFile when os.path.isdir(request.ModelFile) and otherwise
falls back to snapshot_download. That places it in the same class as
transformers/vllm/diffusers/sglang, but the allow-list added in #10910
did not enumerate it, so PrimaryArtifactSpec returned no managed
artifact for a bare HuggingFace repo id.
The consequence in distributed mode: nothing was acquired on the
controller, ModelFileName fell through to the raw repo id, and staging
skipped the resulting phantom /models/<owner>/<repo> path. The worker
received a blank ModelFile, fell back to request.Model, and downloaded
~83GB from HuggingFace inside the remote LoadModel deadline - so the
load could only ever fail with DeadlineExceeded while an abandoned
backend process kept downloading.
Note this materializes the full repository. The backend restricts its
own snapshot_download with allow_patterns, and the avatar repo ships
both base_model/ and base_model_int8/ where only one is ever loaded;
inferred specs have no way to carry patterns today. Tracked separately.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): warn when staging skips a non-existent model path
stageModelFiles logs "Staging model files for remote node" up front, then
silently drops any path field that does not exist on the controller. The
skip itself is legitimate and must stay: a backend outside
managedArtifactBackends that takes a bare HuggingFace repo id gets an
optimistically constructed path (ModelFileName falls through to the raw
model reference) that was never materialized, and sources its own weights
on the worker. Erroring would break those configs.
But at debug level the operator is left with a reassuring staging line and
no trace of the skip, so a genuine controller-side acquisition gap is
indistinguishable from a healthy pass-through - it surfaces much later as
a remote LoadModel timeout, on a worker that is quietly downloading tens
of gigabytes. Raise the skip to warn and name the field, path, node and
tracking key. Behavior is unchanged.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): allow a config to declare companion artifacts
A composed pipeline needs more than one HuggingFace snapshot.
LongCat-Video-Avatar-1.5 loads its own transformer but takes the
tokenizer, text encoder and VAE from the separate LongCat-Video base
repo, so a single-artifact config cannot express it and the backend is
left to fetch the second repo itself at load time.
Widen the artifact model to target: model plus any number of named
target: companion entries. Normalize accepts the new target and
constrains a companion name to [a-z0-9][a-z0-9_-]{0,63} because that
name is the option key the backend later receives; a companion may not
claim primary_file, which only means anything for a load target.
ModelConfig.Validate requires exactly one primary and requires it first,
since Artifacts[0] is what ModelFileName, size estimation and staging all
resolve from.
Both acquisition paths now loop instead of touching index 0 alone:
preloadOne for an already-installed config, bindPrimaryArtifact for a
gallery install. Failure policy differs by provenance. An inferred
primary keeps its warn-and-fall-back, because the legacy download path
still exists for it. Companions are explicit by construction, so they are
all-or-nothing: a config naming one is asserting the backend needs it,
and failing at the acquisition boundary is far more legible than a
missing-weights error surfacing later inside the backend.
The cache key is deliberately unchanged. It hashes source identity only,
never name or target, so every already-installed managed model still hits
its existing snapshot instead of silently re-downloading. Two specs pin
that: one proving a companion and a primary with identical sources agree
on the key, and one pinning the digest of a known primary outright.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): hand resolved companion snapshots to the backend
A materialized companion is useless until the backend can find it, and
its location is a content-addressed cache key that does not exist until
the artifact resolves. A static gallery override cannot carry that, and
persisting it into the config YAML would rot the moment a re-resolve
produced a new key.
Synthesize it instead at load time: each resolved companion becomes
"<artifact name>:<snapshot path>" in ModelOptions.Options, reusing the
key:value convention backends already parse for options like
attention_backend. The value stays relative to the models directory so a
remote worker can resolve it under its own ModelPath once staging has
rewritten the model root. An option the author set explicitly always
wins, so pinning a companion to a local checkout still beats the managed
snapshot.
longcat-video resolves base_model through ModelPath, the same convention
qwen-tts, voxcpm, outetts and ace-step already use for companion assets.
Its sibling-directory heuristic is deleted: it looked for a LongCat-Video
directory next to the model, which cannot exist under the content
addressed .artifacts/huggingface/<key>/snapshot layout, so it was dead
code the moment the model became managed.
The gallery entry declares both repositories and restricts each with
allow_patterns. The avatar repo ships base_model/ and base_model_int8/
and only ever loads one, so fetching the whole repo would roughly double
the download. The patterns match the entry's own options (use_distill
true, use_int8 default false); enabling use_int8 here also requires
adding base_model_int8/**, which is called out in the entry.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stage managed artifact trees from the models root
Staging anchored the worker's models directory on the primary snapshot
whenever a model was managed, so a companion snapshot could not reach the
worker at all.
frontendModelsDir was derived by stripping the Model relative path off
the end of ModelFile. For a managed artifact nothing matches: ModelFile
is .artifacts/huggingface/<key>/snapshot while Model stays a bare
HuggingFace repo id, so the strip was a no-op and the "models directory"
came out as the snapshot itself. Two consequences, both silent. Staging
keys lost the .artifacts/huggingface/<key>/snapshot prefix, so two
snapshots of one model were indistinguishable on the worker. And a
companion, which lives in a sibling snapshot directory outside the
primary, fell outside that directory entirely: StagingKeyMapper.Key
collapsed its files to bare basenames and resolveOptionPath could not
resolve the relative option at all, so it was skipped without a word.
Derive the models root from the artifact tree instead when the path runs
through it, and compute the worker's ModelPath from the file's path
relative to that root rather than from the Model field. The legacy layout
is unaffected: where Model really is the relative path, the new
derivation reduces to the old one, which a regression spec pins.
This deliberately changes an invariant that router_dirstage_test.go
pinned: for a managed primary, ModelFile and ModelPath were both the
snapshot directory, and staging keys were relative to it. Now ModelFile
is the snapshot, ModelPath is the models root above it, and keys keep the
full relative path. That spec is updated rather than accommodated, with
the reasoning recorded inline, because the old invariant is exactly what
made a sibling companion unreachable.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(ci): build Bonsai backend images
Register the Bonsai C++ source path with the backend matrix filter so changes select its image jobs. Also make shared llama.cpp changes rebuild the Bonsai and Turboquant fork images in the actual matrix, not only their test flags.\n\nAssisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(backends): list backends runnable on worker nodes in distributed mode
GET /backends/available filtered the gallery against the system state of
the host serving the request. In a distributed deployment that host is the
controller, which typically has no GPU, while the GPUs live on worker
nodes. Any meta backend whose capabilities map lacks a "default" (or "cpu")
key was therefore dropped from the listing entirely — longcat-video,
vllm-omni, ltx-video, parakeet, edgetam and qwentts were invisible in the
UI even though installing them by name on a GPU worker worked fine.
Workers now report their own meta-backend capability at registration and
the controller persists it on the node row. The controller cannot derive
it: OS-dependent capabilities (metal, darwin-x86, nvidia-l4t) and the CUDA
runtime refinements are only observable on the worker. Nodes registered
before this field existed fall back to a coarse capability derived from
their GPU vendor and VRAM.
Backend discovery then evaluates compatibility as the union over healthy
backend nodes, so a backend runnable on any node is offered while one no
node can run stays hidden. Each remote capability is evaluated through a
capability-pinned system state, otherwise a forced capability on the
controller image (LOCALAI_FORCE_META_BACKEND_CAPABILITY or
/run/localai/capability) would silently override every worker's verdict.
With no registered nodes the listing is byte-for-byte what it was, so
single-node deployments are unaffected.
Also fixes the same-root-cause misclassification in /api/operations, which
used the capability-filtered listing to decide whether an operation was a
backend or a model install. A GPU-only backend installing on a worker is
still a backend operation on the controller, so that lookup is now
unfiltered.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(backends): union worker capabilities in backend discovery
Implementation for the specs added in the previous commit, plus the two
remaining discovery endpoints.
Capability-filtered backend discovery evaluated compatibility against the
system state of the host serving the request. In a distributed deployment
that host is the controller, which typically has no GPU, while the GPUs
live on worker nodes. Any meta backend whose capabilities map lacks a
"default" (or "cpu") key was dropped entirely — longcat-video, vllm-omni,
ltx-video, parakeet, edgetam and qwentts were invisible in the UI even
though installing them by name on a GPU worker worked fine.
Workers now report their own meta-backend capability at registration and
the controller persists it on the node row. The controller cannot derive
it: OS-dependent capabilities (metal, darwin-x86, nvidia-l4t) and the CUDA
runtime refinements are only observable on the worker. Nodes registered
before this field existed fall back to a coarse capability derived from
their GPU vendor and VRAM.
Discovery then evaluates compatibility as the union over healthy backend
nodes, so a backend runnable on any node is offered while one no node can
run stays hidden. Each remote capability is evaluated through a
capability-pinned system state, otherwise a forced capability on the
controller image (LOCALAI_FORCE_META_BACKEND_CAPABILITY or
/run/localai/capability) would silently override every worker's verdict.
With no registered nodes the listing is byte-for-byte what it was, so
single-node deployments are unaffected.
Four surfaces shared this root cause and are all routed through the same
helper now:
- GET /backends/available
- GET /api/fine-tuning/backends
- GET /api/quantization/backends
- /api/operations backend-vs-model classification, which additionally
had no reason to filter by capability at all: a GPU-only backend
installing on a worker is still a backend operation on the
controller, so that lookup is now unfiltered.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
cuDNN 9 is a dispatcher (libcudnn.so.9) plus seven sublibraries the dispatcher
dlopen()s by bare soname. Only the dispatcher is ever a DT_NEEDED, so ldd finds
it and never the seven. The allowlist force-copied three of them
(libcudnn.so*, libcudnn_ops.so*, libcudnn_cnn.so*) into every CUDA backend,
which is wrong in both directions at once: too few libraries for a backend that
uses cuDNN, and too many for one that does not.
On an L4T fleet, ten of the eleven backends carrying cuDNN were in a broken end
state; the one that was correct was correct by accident, being BUILD_TYPE=cpu
so package_cuda_libs never ran for it.
longcat-video bundled 4 of 8 at 9.24.0 over a complete pip set at 9.20.0.48
in its venv. libbackend.sh puts lib/ on LD_LIBRARY_PATH, searched before
DT_RUNPATH, so the bundle won and the rest still came from the venv:
CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH.
Nine others bundled 3 of 8 and had no venv cuDNN. None bundled
libcudnn_graph, which libcudnn_cnn has a hard DT_NEEDED on, so it resolved
out of the runtime image and the process ran bundled 9.22.0 against system
9.23.2.
Five of those nine - llama-cpp, whisper, rfdetr-cpp, sam3-cpp,
stablediffusion-ggml - do not reference cuDNN at all. ggml goes through cuBLAS.
They were carrying ~57 MB of cuDNN with no consumer, and completing the family
for them would have taken that to ~576 MB for nothing.
Sizes overall: backends with no cuDNN consumer shed ~57 MB each (seven
instances on the fleet measured, plus longcat's ~60 MB), while the ones that
genuinely use cuDNN grow from ~57 MB to ~576 MB, because the five missing
sublibraries are ~517 MB, dominated by libcudnn_engines_precompiled. Net on
that fleet is an increase of roughly 570 MB. That growth is the bug being paid
off, not a regression: those backends only work today by silently borrowing the
missing five from the runtime image. Whether the engines set can be trimmed is
an open question, not addressed here.
So bundle per backend, by what that backend actually needs:
- venv has a complete pip cuDNN -> bundle nothing; $ORIGIN resolves the pip
set, which is the one its torch was built against (longcat-video)
- venv has no pip cuDNN -> bundle the complete family. Stays
conservative rather than detecting consumers: for a Python backend they sit
inside the venv (torch, ctranslate2, onnxruntime) where the sweep does not
look (vllm)
- no venv, nothing references cuDNN -> bundle nothing (llama-cpp, whisper,
rfdetr-cpp, sam3-cpp, stablediffusion-ggml)
- no venv, something references it -> bundle the complete family
(face-detect, voice-detect)
The no-venv case needs no new machinery. Go backends stage their own shared
object into package/lib, which IS the target dir, so sweep_transitive_deps
already pulls the dispatcher when it is a genuine dependency - that is exactly
how libcudnn_graph reached longcat. cuDNN simply comes off the force-copy list,
and complete_cudnn_family fills in the seven dlopen'd sublibraries around
whatever the sweep found. Detection is a string scan rather than ldd, so a
consumer that only dlopen()s cuDNN is seen too; over-matching costs an unused
library, under-matching costs a backend that cannot load.
Keeping bundled and pip versions in agreement instead is not viable: nothing
here pins nvidia-cudnn (zero occurrences), torch is unpinned for l4t13 except
longcat-video, and the fleet already runs five concurrent cuDNN versions -
9.19.0.56, 9.20.0.48, 9.22.0, 9.23.2, 9.24.0.
verify_cudnn_bundle asserts the end state: exactly one complete cuDNN visible to
whoever needs one - never both, never partial, and never zero for a backend that
references it. Zero is correct and common otherwise. It deliberately does not
accept the build image's system cuDNN as completing a partial bundle, which is
the shape that had been shipping silently; the build image is not the runtime
image. A version check alone would have missed longcat too, whose four bundled
libs were all 9.24.0 and mutually consistent.
Match per family for the other components for the same dlopen reason: TensorRT
(libnvinfer_plugin, libnvinfer_builder_resource), cuBLAS, cuFFT, cuSPARSE,
cuSOLVER, nvRTC. Exclusions bind inside copy_lib so they cover the sweep.
The packaging scripts' shell tests ran nowhere in CI. Add make
test-build-scripts and a lint workflow job so they gate every PR.
Fixes#10905
Assisted-by: Claude:claude-opus-4-8 golangci-lint shellcheck
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Pin vLLM protogen to grpcio-tools 1.78.0 so its generated code remains importable by protobuf 6.33.x, and remove stale generated artifacts before regeneration.
Closes#10940
Assisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Avoid holding the global loader lock across backend lifecycle waits and propagate forced shutdown through distributed workers. Track parallel requests with in-flight counters and reserve worker ports until process termination.
Add focused race tests and an authoritative FizzBee lifecycle model with a fail-closed conformance target.
Assisted-by: Codex:GPT-5 [FizzBee] [Ginkgo]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
The Vite build emitted path-absolute asset URLs (base: '/'). index.html
entry scripts and the favicon were rewritten to include the reverse-proxy
prefix in serveIndex, but two reference kinds are not in index.html and so
bypassed that rewrite:
- CSS `url()` font references (e.g. Font Awesome .woff2), which the browser
resolves relative to the stylesheet and which `<base href>` never affects
- lazily-imported route chunks, whose preload base came from the absolute
Vite base
Under a subpath mount (X-Forwarded-Prefix: /llm/) both were fetched from the
origin root, 404ing — missing-glyph "tofu" icons and broken lazy-loaded pages.
Switch Vite to a relative base ('./') so every generated URL resolves against
the file that references it: CSS fonts and route chunks now load from
`/llm/assets/...`, and index.html's now-relative entry refs resolve via the
`<base href>` serveIndex already injects on every response. Root deployments
are unaffected. The existing path-absolute rewrite in app.go still covers the
public `/favicon.svg`.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The weekly "Dependabot Updates" pip job for /backend/python/diffusers has been
failing with `private_source_authentication_failure` against the Jetson pip
index (https://pypi.jetson-ai-lab.io/jp6/cu129/), referenced by that backend's
requirements-l4t12.txt. diffusers is the only dependabot-configured pip
directory that pulls from that private index, so it is the only update job that
fails; the other backends update cleanly.
torch and transformers are deliberately pinned in this backend for
reproducibility (see backend/python/diffusers/requirements-*.txt and #9979), so
we do not want dependabot bumping them anyway. Ignoring both dependencies for
this directory stops dependabot from resolving them against the unreachable
Jetson index and keeps the weekly update job green, without removing update
coverage for the rest of the backend's dependencies.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The managed-artifact materializer stages a HuggingFace snapshot into a
directory (.artifacts/huggingface/<key>/snapshot/). That is the right load
target for directory-consuming backends (transformers, vLLM, diffusers, ...),
but PrimaryArtifactSpec inferred a managed artifact from ANY HuggingFace-shaped
model reference regardless of backend. A single-file backend such as llama.cpp
or whisper was therefore handed the snapshot directory instead of the weight
file and failed to load it.
The /import-model importer already guards this with a backend allow-list
(managedArtifactBackends), but the loader-side inference did not. Move the
allow-list into core/config as IsManagedArtifactBackend and apply it in
PrimaryArtifactSpec: only directory-consuming backends may have an artifact
inferred from a bare reference; every other backend stays on the legacy
download-to-file path. An explicit artifacts: block still bypasses the gate,
where single-file snapshot resolution handles the load path.
The importer now shares the same predicate, so both paths agree on which
backends auto-materialize.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(model-artifacts): load single-file HF snapshots from the file, not the dir
The managed Hugging Face artifact materializer (#10825) always pointed
backends at the snapshot *directory*
(.artifacts/huggingface/<key>/snapshot). For a single-file model
reference such as huggingface://nomic-ai/nomic-embed-text-v1.5-GGUF/nomic-embed-text-v1.5.f16.gguf,
the GGUF lives *inside* that directory, so llama.cpp was handed a
directory and failed with "gguf_init_from_reader: failed to read magic".
This has kept the tests-aio job red on master since the feature merged
(the embeddings e2e tests could not load text-embedding-ada-002).
Record the single file of a one-file snapshot as Resolved.PrimaryFile and
have ModelFileName() resolve to snapshot/<PrimaryFile> when it is set.
Multi-file snapshots (e.g. transformers repos consumed as a directory)
keep pointing at the snapshot directory. PrimaryFile is derived from the
resolved contents and is deliberately excluded from the artifact cache
key. estimateModelSizeBytes now derives the snapshot directory from the
cache key instead of ModelFileName(), so its manifest lookup is unaffected
by the file-vs-directory resolution.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
