* ⬆️ Update CrispStrobe/CrispASR
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(crispasr): initialize only declared submodules
The latest upstream commit contains an undeclared CrispASR gitlink that makes a blanket recursive submodule update fail. Limit initialization to the two submodules used by the backend build.
Assisted-by: Codex:gpt-5 [Codex]
* fix(crispasr): resolve vendored WebRTC from project root
CrispASR now builds a vendored WebRTC VAD, but its include paths assume CrispASR is the top-level CMake project. Extend the existing embedded-project rewrite to the shared third_party root.
Assisted-by: Codex:gpt-5 [Codex]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(gpu): detect GPUs via sysfs when no pci.ids database is present
ghw.GPU() calls pci.New() before it reads /sys/class/drm and fails
outright when it cannot find a pci.ids database file. jaypipes/pcidb
embeds no database and has network fetch disabled by default, so on an
image that ships no pci.ids, GPU enumeration returns an error and every
detection path downstream goes dark.
The Dockerfile installs pciutils only in the vulkan and cublas branches,
so the Intel image had no pci.ids. A correctly passed-through Arc A310
was reported as "No GPU detected" with zero VRAM even though clinfo and
sycl-ls both enumerated it inside the same container. NVIDIA and AMD
images were shielded by their nvidia-smi / rocm-smi binary fallbacks;
Intel has no equivalent, leaving it fully exposed.
Read PCI vendor IDs directly from /sys/class/drm/card*/device/vendor,
which needs no database, and consult that from DetectGPUVendor. The
same scan replaces the ghw-only guard in getIntelGPUMemory, which is
what had been blocking the working clinfo path and keeping VRAM at
zero. Install hwdata in the base image stage as well, so ghw stops
failing for every image variant rather than only Intel.
Also apply the documented NVIDIA > AMD > Intel priority to the ghw
path, which previously returned whichever card DRM enumerated first
and so reported "intel" on a machine with an Intel iGPU at card0 and
an NVIDIA dGPU at card1.
HasGPU() carried the same blindness plus one of its own: it matched
the requested vendor against ghw's card description with a
case-sensitive Contains, so "nvidia" never matched the pci.ids
spelling "NVIDIA Corporation". It only worked because that same
description embeds the lowercase kernel driver name ("nvidia",
"amdgpu"), and it returned false outright whenever ghw errored. Route
it through the shared vendor lookup so it matches case-insensitively
and falls back to sysfs. It feeds the GPU option and NGPULayers
defaults in core/config/gguf.go.
Fixes#10941
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(gpu): key vendor detection off the numeric PCI ID in both paths
The ghw and sysfs legs were identifying vendors by different means: ghw
by substring-matching the pci.ids vendor name, sysfs by the numeric PCI
vendor ID. ghw already exposes that same numeric ID via
DeviceInfo.Vendor.ID, read from the kernel's modalias rather than from
the database, so the name matching was both a duplicate mechanism and
the weaker of the two.
It is weaker because a card absent from an outdated pci.ids gets
Name: "unknown" while its ID is still correct. Detection then failed
even though ghw had enumerated the card successfully. Verified in a
container with a vendor-less pci.ids and an Arc's modalias: before,
DetectGPUVendor returned ""; after, "intel".
Both legs now resolve through the same pciVendorIDs table and share the
hex parsing, with the vendor name kept only as a fallback for devices
exposing no parseable ID.
ghwHasVendor is deliberately not a priority pick, unlike vendorFromGHW:
HasGPU("intel") must stay true on a hybrid-graphics host whose discrete
NVIDIA card outranks the integrated Intel one.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gpu): silence the gosec G304 on the sysfs attribute read
gosec flags os.ReadFile with a non-literal path. The path here is the
DRM root (a package constant in production, a temp dir under test)
joined with a ReadDir entry name and a fixed attribute filename, so no
external input reaches it.
gosec's suggested autofix, os.Root, cannot be used: /sys/class/drm/cardN
is a symlink into the PCI device tree, and os.Root refuses to traverse
it ("path escapes from parent"), which would disable the whole scan.
Assisted-by: Claude:claude-opus-4-8 gosec golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
request.language usually carries an ISO 639-1 code (e.g. "de"), which
OpenAI-compatible clients such as Home Assistant / wyoming_openai send,
but qwen_asr.validate_language() only accepts full English names
("German") and raises ValueError otherwise. Normalize the requested
language: accept full names case-insensitively, translate ISO codes
(with optional region suffix like "de-DE") to the expected name, and
pass anything unrecognised through so qwen_asr still reports it clearly.
Fixes#10958
Signed-off-by: Tai An <antai12232931@outlook.com>
llama.cpp picks a random per-process media marker (ggml-org/llama.cpp#21962),
so LocalAI renders the prompt with a "<__media__>" sentinel and swaps in the
backend's real marker after probing ModelMetadata.
That probe was gated on an exact match against "llama-cpp", the gallery's meta
backend name. A model config pinning a concrete build ("vulkan-llama-cpp",
"cuda12-llama-cpp", "rocm-llama-cpp", ... and their -development counterparts)
runs the same llama.cpp gRPC server but skipped the probe, so MediaMarker
stayed empty, no substitution happened, and the prompt reached mtmd still
carrying the sentinel. mtmd_tokenize then counted zero markers against one
bitmap and every image request failed with "Failed to tokenize prompt".
The same early return also skipped thinking-mode detection and tool-format
marker extraction, so a pinned variant silently lost reasoning and native
tool-call parsing too.
Add IsLlamaCppBackend, which recognises the whole variant family (plus the
empty auto-detect name, which resolves to llama.cpp) while excluding
ik-llama.cpp, a separate engine that merely shares the suffix.
Fixes#10945
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): make the remote LoadModel deadline configurable
The router hardcoded a 5 minute gRPC deadline for the remote LoadModel
call. Staging finishes before the timer starts, so those five minutes
cover only the worker backend's own checkpoint load and pipeline init.
A cold load of meituan-longcat/LongCat-Video-Avatar-1.5 (~83 GB) on an
ARM64 Thor worker fails at exactly 302s with DeadlineExceeded while the
backend process is still making progress (CPU time accumulating, RSS
moving as weights are mapped), so the load was cut short rather than
wedged.
Add LOCALAI_NATS_MODEL_LOAD_TIMEOUT / --model-load-timeout mirroring the
existing backend-install timeout knob, defaulting to 5m so unset
clusters keep today's behaviour.
The cold-load hold ceiling (which bounds how long one load may hold the
per-model advisory lock) was derived from the install timeout alone, so
raising the load deadline past it would have been silently clipped.
Derive it from both budgets via ModelLoadCeilingFor:
max(install + load + 5m staging margin, 25m)
With the defaults that is 15m + 5m + 5m = 25m, identical to the previous
constant, and the 25m floor means shrinking either budget can never
tighten the ceiling below what clusters relied on before.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): reap the abandoned replica when a remote load times out
The gRPC deadline on the remote LoadModel call only cancels the client
side. A backend blocked in a synchronous weight load never observes its
cancelled handler context, so when scheduleAndLoad gave up it left the
worker loading with nobody waiting for the result.
Observed on an ARM64 Thor worker loading LongCat-Video-Avatar-1.5: the
client returned DeadlineExceeded at 302s, and the backend process was
still alive 30 minutes later having pulled ~57GB from HuggingFace. Every
retry stacked another multi-GB loader on the worker; they had to be
reaped by hand via POST /api/nodes/:id/models/unload.
Send backend.stop for the exact `modelID#replicaIndex` process key we
just abandoned. The exact key matters: a bare model ID stops every
replica on that node, including healthy ones serving traffic.
Only a deadline or cancellation triggers the reap. Any other LoadModel
failure is the backend answering, which means its handler returned and
the process is idle - stopping it there would discard a warm process and
its downloaded weights. The reap is best-effort and never replaces the
load error the caller is waiting on.
The `modelID#replicaIndex` format was already hand-rolled in two places
(the worker's buildProcessKey and pkg/model's log store). Rather than add
a third, export model.BackendProcessKey from pkg/model, the lowest common
dependency of both sides.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 golangci-lint
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(model-artifacts): materialize longcat-video checkpoints on the controller
longcat-video loads a checkpoint directory: its backend.py takes
request.ModelFile when os.path.isdir(request.ModelFile) and otherwise
falls back to snapshot_download. That places it in the same class as
transformers/vllm/diffusers/sglang, but the allow-list added in #10910
did not enumerate it, so PrimaryArtifactSpec returned no managed
artifact for a bare HuggingFace repo id.
The consequence in distributed mode: nothing was acquired on the
controller, ModelFileName fell through to the raw repo id, and staging
skipped the resulting phantom /models/<owner>/<repo> path. The worker
received a blank ModelFile, fell back to request.Model, and downloaded
~83GB from HuggingFace inside the remote LoadModel deadline - so the
load could only ever fail with DeadlineExceeded while an abandoned
backend process kept downloading.
Note this materializes the full repository. The backend restricts its
own snapshot_download with allow_patterns, and the avatar repo ships
both base_model/ and base_model_int8/ where only one is ever loaded;
inferred specs have no way to carry patterns today. Tracked separately.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): warn when staging skips a non-existent model path
stageModelFiles logs "Staging model files for remote node" up front, then
silently drops any path field that does not exist on the controller. The
skip itself is legitimate and must stay: a backend outside
managedArtifactBackends that takes a bare HuggingFace repo id gets an
optimistically constructed path (ModelFileName falls through to the raw
model reference) that was never materialized, and sources its own weights
on the worker. Erroring would break those configs.
But at debug level the operator is left with a reassuring staging line and
no trace of the skip, so a genuine controller-side acquisition gap is
indistinguishable from a healthy pass-through - it surfaces much later as
a remote LoadModel timeout, on a worker that is quietly downloading tens
of gigabytes. Raise the skip to warn and name the field, path, node and
tracking key. Behavior is unchanged.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): allow a config to declare companion artifacts
A composed pipeline needs more than one HuggingFace snapshot.
LongCat-Video-Avatar-1.5 loads its own transformer but takes the
tokenizer, text encoder and VAE from the separate LongCat-Video base
repo, so a single-artifact config cannot express it and the backend is
left to fetch the second repo itself at load time.
Widen the artifact model to target: model plus any number of named
target: companion entries. Normalize accepts the new target and
constrains a companion name to [a-z0-9][a-z0-9_-]{0,63} because that
name is the option key the backend later receives; a companion may not
claim primary_file, which only means anything for a load target.
ModelConfig.Validate requires exactly one primary and requires it first,
since Artifacts[0] is what ModelFileName, size estimation and staging all
resolve from.
Both acquisition paths now loop instead of touching index 0 alone:
preloadOne for an already-installed config, bindPrimaryArtifact for a
gallery install. Failure policy differs by provenance. An inferred
primary keeps its warn-and-fall-back, because the legacy download path
still exists for it. Companions are explicit by construction, so they are
all-or-nothing: a config naming one is asserting the backend needs it,
and failing at the acquisition boundary is far more legible than a
missing-weights error surfacing later inside the backend.
The cache key is deliberately unchanged. It hashes source identity only,
never name or target, so every already-installed managed model still hits
its existing snapshot instead of silently re-downloading. Two specs pin
that: one proving a companion and a primary with identical sources agree
on the key, and one pinning the digest of a known primary outright.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): hand resolved companion snapshots to the backend
A materialized companion is useless until the backend can find it, and
its location is a content-addressed cache key that does not exist until
the artifact resolves. A static gallery override cannot carry that, and
persisting it into the config YAML would rot the moment a re-resolve
produced a new key.
Synthesize it instead at load time: each resolved companion becomes
"<artifact name>:<snapshot path>" in ModelOptions.Options, reusing the
key:value convention backends already parse for options like
attention_backend. The value stays relative to the models directory so a
remote worker can resolve it under its own ModelPath once staging has
rewritten the model root. An option the author set explicitly always
wins, so pinning a companion to a local checkout still beats the managed
snapshot.
longcat-video resolves base_model through ModelPath, the same convention
qwen-tts, voxcpm, outetts and ace-step already use for companion assets.
Its sibling-directory heuristic is deleted: it looked for a LongCat-Video
directory next to the model, which cannot exist under the content
addressed .artifacts/huggingface/<key>/snapshot layout, so it was dead
code the moment the model became managed.
The gallery entry declares both repositories and restricts each with
allow_patterns. The avatar repo ships base_model/ and base_model_int8/
and only ever loads one, so fetching the whole repo would roughly double
the download. The patterns match the entry's own options (use_distill
true, use_int8 default false); enabling use_int8 here also requires
adding base_model_int8/**, which is called out in the entry.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stage managed artifact trees from the models root
Staging anchored the worker's models directory on the primary snapshot
whenever a model was managed, so a companion snapshot could not reach the
worker at all.
frontendModelsDir was derived by stripping the Model relative path off
the end of ModelFile. For a managed artifact nothing matches: ModelFile
is .artifacts/huggingface/<key>/snapshot while Model stays a bare
HuggingFace repo id, so the strip was a no-op and the "models directory"
came out as the snapshot itself. Two consequences, both silent. Staging
keys lost the .artifacts/huggingface/<key>/snapshot prefix, so two
snapshots of one model were indistinguishable on the worker. And a
companion, which lives in a sibling snapshot directory outside the
primary, fell outside that directory entirely: StagingKeyMapper.Key
collapsed its files to bare basenames and resolveOptionPath could not
resolve the relative option at all, so it was skipped without a word.
Derive the models root from the artifact tree instead when the path runs
through it, and compute the worker's ModelPath from the file's path
relative to that root rather than from the Model field. The legacy layout
is unaffected: where Model really is the relative path, the new
derivation reduces to the old one, which a regression spec pins.
This deliberately changes an invariant that router_dirstage_test.go
pinned: for a managed primary, ModelFile and ModelPath were both the
snapshot directory, and staging keys were relative to it. Now ModelFile
is the snapshot, ModelPath is the models root above it, and keys keep the
full relative path. That spec is updated rather than accommodated, with
the reasoning recorded inline, because the old invariant is exactly what
made a sibling companion unreachable.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(ci): build Bonsai backend images
Register the Bonsai C++ source path with the backend matrix filter so changes select its image jobs. Also make shared llama.cpp changes rebuild the Bonsai and Turboquant fork images in the actual matrix, not only their test flags.\n\nAssisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(backends): list backends runnable on worker nodes in distributed mode
GET /backends/available filtered the gallery against the system state of
the host serving the request. In a distributed deployment that host is the
controller, which typically has no GPU, while the GPUs live on worker
nodes. Any meta backend whose capabilities map lacks a "default" (or "cpu")
key was therefore dropped from the listing entirely — longcat-video,
vllm-omni, ltx-video, parakeet, edgetam and qwentts were invisible in the
UI even though installing them by name on a GPU worker worked fine.
Workers now report their own meta-backend capability at registration and
the controller persists it on the node row. The controller cannot derive
it: OS-dependent capabilities (metal, darwin-x86, nvidia-l4t) and the CUDA
runtime refinements are only observable on the worker. Nodes registered
before this field existed fall back to a coarse capability derived from
their GPU vendor and VRAM.
Backend discovery then evaluates compatibility as the union over healthy
backend nodes, so a backend runnable on any node is offered while one no
node can run stays hidden. Each remote capability is evaluated through a
capability-pinned system state, otherwise a forced capability on the
controller image (LOCALAI_FORCE_META_BACKEND_CAPABILITY or
/run/localai/capability) would silently override every worker's verdict.
With no registered nodes the listing is byte-for-byte what it was, so
single-node deployments are unaffected.
Also fixes the same-root-cause misclassification in /api/operations, which
used the capability-filtered listing to decide whether an operation was a
backend or a model install. A GPU-only backend installing on a worker is
still a backend operation on the controller, so that lookup is now
unfiltered.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(backends): union worker capabilities in backend discovery
Implementation for the specs added in the previous commit, plus the two
remaining discovery endpoints.
Capability-filtered backend discovery evaluated compatibility against the
system state of the host serving the request. In a distributed deployment
that host is the controller, which typically has no GPU, while the GPUs
live on worker nodes. Any meta backend whose capabilities map lacks a
"default" (or "cpu") key was dropped entirely — longcat-video, vllm-omni,
ltx-video, parakeet, edgetam and qwentts were invisible in the UI even
though installing them by name on a GPU worker worked fine.
Workers now report their own meta-backend capability at registration and
the controller persists it on the node row. The controller cannot derive
it: OS-dependent capabilities (metal, darwin-x86, nvidia-l4t) and the CUDA
runtime refinements are only observable on the worker. Nodes registered
before this field existed fall back to a coarse capability derived from
their GPU vendor and VRAM.
Discovery then evaluates compatibility as the union over healthy backend
nodes, so a backend runnable on any node is offered while one no node can
run stays hidden. Each remote capability is evaluated through a
capability-pinned system state, otherwise a forced capability on the
controller image (LOCALAI_FORCE_META_BACKEND_CAPABILITY or
/run/localai/capability) would silently override every worker's verdict.
With no registered nodes the listing is byte-for-byte what it was, so
single-node deployments are unaffected.
Four surfaces shared this root cause and are all routed through the same
helper now:
- GET /backends/available
- GET /api/fine-tuning/backends
- GET /api/quantization/backends
- /api/operations backend-vs-model classification, which additionally
had no reason to filter by capability at all: a GPU-only backend
installing on a worker is still a backend operation on the
controller, so that lookup is now unfiltered.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
cuDNN 9 is a dispatcher (libcudnn.so.9) plus seven sublibraries the dispatcher
dlopen()s by bare soname. Only the dispatcher is ever a DT_NEEDED, so ldd finds
it and never the seven. The allowlist force-copied three of them
(libcudnn.so*, libcudnn_ops.so*, libcudnn_cnn.so*) into every CUDA backend,
which is wrong in both directions at once: too few libraries for a backend that
uses cuDNN, and too many for one that does not.
On an L4T fleet, ten of the eleven backends carrying cuDNN were in a broken end
state; the one that was correct was correct by accident, being BUILD_TYPE=cpu
so package_cuda_libs never ran for it.
longcat-video bundled 4 of 8 at 9.24.0 over a complete pip set at 9.20.0.48
in its venv. libbackend.sh puts lib/ on LD_LIBRARY_PATH, searched before
DT_RUNPATH, so the bundle won and the rest still came from the venv:
CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH.
Nine others bundled 3 of 8 and had no venv cuDNN. None bundled
libcudnn_graph, which libcudnn_cnn has a hard DT_NEEDED on, so it resolved
out of the runtime image and the process ran bundled 9.22.0 against system
9.23.2.
Five of those nine - llama-cpp, whisper, rfdetr-cpp, sam3-cpp,
stablediffusion-ggml - do not reference cuDNN at all. ggml goes through cuBLAS.
They were carrying ~57 MB of cuDNN with no consumer, and completing the family
for them would have taken that to ~576 MB for nothing.
Sizes overall: backends with no cuDNN consumer shed ~57 MB each (seven
instances on the fleet measured, plus longcat's ~60 MB), while the ones that
genuinely use cuDNN grow from ~57 MB to ~576 MB, because the five missing
sublibraries are ~517 MB, dominated by libcudnn_engines_precompiled. Net on
that fleet is an increase of roughly 570 MB. That growth is the bug being paid
off, not a regression: those backends only work today by silently borrowing the
missing five from the runtime image. Whether the engines set can be trimmed is
an open question, not addressed here.
So bundle per backend, by what that backend actually needs:
- venv has a complete pip cuDNN -> bundle nothing; $ORIGIN resolves the pip
set, which is the one its torch was built against (longcat-video)
- venv has no pip cuDNN -> bundle the complete family. Stays
conservative rather than detecting consumers: for a Python backend they sit
inside the venv (torch, ctranslate2, onnxruntime) where the sweep does not
look (vllm)
- no venv, nothing references cuDNN -> bundle nothing (llama-cpp, whisper,
rfdetr-cpp, sam3-cpp, stablediffusion-ggml)
- no venv, something references it -> bundle the complete family
(face-detect, voice-detect)
The no-venv case needs no new machinery. Go backends stage their own shared
object into package/lib, which IS the target dir, so sweep_transitive_deps
already pulls the dispatcher when it is a genuine dependency - that is exactly
how libcudnn_graph reached longcat. cuDNN simply comes off the force-copy list,
and complete_cudnn_family fills in the seven dlopen'd sublibraries around
whatever the sweep found. Detection is a string scan rather than ldd, so a
consumer that only dlopen()s cuDNN is seen too; over-matching costs an unused
library, under-matching costs a backend that cannot load.
Keeping bundled and pip versions in agreement instead is not viable: nothing
here pins nvidia-cudnn (zero occurrences), torch is unpinned for l4t13 except
longcat-video, and the fleet already runs five concurrent cuDNN versions -
9.19.0.56, 9.20.0.48, 9.22.0, 9.23.2, 9.24.0.
verify_cudnn_bundle asserts the end state: exactly one complete cuDNN visible to
whoever needs one - never both, never partial, and never zero for a backend that
references it. Zero is correct and common otherwise. It deliberately does not
accept the build image's system cuDNN as completing a partial bundle, which is
the shape that had been shipping silently; the build image is not the runtime
image. A version check alone would have missed longcat too, whose four bundled
libs were all 9.24.0 and mutually consistent.
Match per family for the other components for the same dlopen reason: TensorRT
(libnvinfer_plugin, libnvinfer_builder_resource), cuBLAS, cuFFT, cuSPARSE,
cuSOLVER, nvRTC. Exclusions bind inside copy_lib so they cover the sweep.
The packaging scripts' shell tests ran nowhere in CI. Add make
test-build-scripts and a lint workflow job so they gate every PR.
Fixes#10905
Assisted-by: Claude:claude-opus-4-8 golangci-lint shellcheck
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Pin vLLM protogen to grpcio-tools 1.78.0 so its generated code remains importable by protobuf 6.33.x, and remove stale generated artifacts before regeneration.
Closes#10940
Assisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Avoid holding the global loader lock across backend lifecycle waits and propagate forced shutdown through distributed workers. Track parallel requests with in-flight counters and reserve worker ports until process termination.
Add focused race tests and an authoritative FizzBee lifecycle model with a fail-closed conformance target.
Assisted-by: Codex:GPT-5 [FizzBee] [Ginkgo]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
The Vite build emitted path-absolute asset URLs (base: '/'). index.html
entry scripts and the favicon were rewritten to include the reverse-proxy
prefix in serveIndex, but two reference kinds are not in index.html and so
bypassed that rewrite:
- CSS `url()` font references (e.g. Font Awesome .woff2), which the browser
resolves relative to the stylesheet and which `<base href>` never affects
- lazily-imported route chunks, whose preload base came from the absolute
Vite base
Under a subpath mount (X-Forwarded-Prefix: /llm/) both were fetched from the
origin root, 404ing — missing-glyph "tofu" icons and broken lazy-loaded pages.
Switch Vite to a relative base ('./') so every generated URL resolves against
the file that references it: CSS fonts and route chunks now load from
`/llm/assets/...`, and index.html's now-relative entry refs resolve via the
`<base href>` serveIndex already injects on every response. Root deployments
are unaffected. The existing path-absolute rewrite in app.go still covers the
public `/favicon.svg`.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The weekly "Dependabot Updates" pip job for /backend/python/diffusers has been
failing with `private_source_authentication_failure` against the Jetson pip
index (https://pypi.jetson-ai-lab.io/jp6/cu129/), referenced by that backend's
requirements-l4t12.txt. diffusers is the only dependabot-configured pip
directory that pulls from that private index, so it is the only update job that
fails; the other backends update cleanly.
torch and transformers are deliberately pinned in this backend for
reproducibility (see backend/python/diffusers/requirements-*.txt and #9979), so
we do not want dependabot bumping them anyway. Ignoring both dependencies for
this directory stops dependabot from resolving them against the unreachable
Jetson index and keeps the weekly update job green, without removing update
coverage for the rest of the backend's dependencies.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The managed-artifact materializer stages a HuggingFace snapshot into a
directory (.artifacts/huggingface/<key>/snapshot/). That is the right load
target for directory-consuming backends (transformers, vLLM, diffusers, ...),
but PrimaryArtifactSpec inferred a managed artifact from ANY HuggingFace-shaped
model reference regardless of backend. A single-file backend such as llama.cpp
or whisper was therefore handed the snapshot directory instead of the weight
file and failed to load it.
The /import-model importer already guards this with a backend allow-list
(managedArtifactBackends), but the loader-side inference did not. Move the
allow-list into core/config as IsManagedArtifactBackend and apply it in
PrimaryArtifactSpec: only directory-consuming backends may have an artifact
inferred from a bare reference; every other backend stays on the legacy
download-to-file path. An explicit artifacts: block still bypasses the gate,
where single-file snapshot resolution handles the load path.
The importer now shares the same predicate, so both paths agree on which
backends auto-materialize.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(model-artifacts): load single-file HF snapshots from the file, not the dir
The managed Hugging Face artifact materializer (#10825) always pointed
backends at the snapshot *directory*
(.artifacts/huggingface/<key>/snapshot). For a single-file model
reference such as huggingface://nomic-ai/nomic-embed-text-v1.5-GGUF/nomic-embed-text-v1.5.f16.gguf,
the GGUF lives *inside* that directory, so llama.cpp was handed a
directory and failed with "gguf_init_from_reader: failed to read magic".
This has kept the tests-aio job red on master since the feature merged
(the embeddings e2e tests could not load text-embedding-ada-002).
Record the single file of a one-file snapshot as Resolved.PrimaryFile and
have ModelFileName() resolve to snapshot/<PrimaryFile> when it is set.
Multi-file snapshots (e.g. transformers repos consumed as a directory)
keep pointing at the snapshot directory. PrimaryFile is derived from the
resolved contents and is deliberately excluded from the artifact cache
key. estimateModelSizeBytes now derives the snapshot directory from the
cache key instead of ModelFileName(), so its manifest lookup is unaffected
by the file-vs-directory resolution.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The artifact progress bridge mapped every PhaseVerifying event to a flat
95%. The materializer emits PhaseVerifying once per file (from each file's
AfterDownload hook) and downloads run sequentially, so the first small file
to finish pinned the bar at 95% - and, because progress is monotonic, it
stayed at 95% for the entire remaining download (e.g. a 70GB checkpoint
reporting 95% at 410MB / 69.7GB).
Track per-file verify proportionally to the running aggregate bytes, the
same way downloading does. CurrentBytes already reflects "completed files +
this file", so the percentage advances honestly. The flat 95%/99% is now
reserved for the genuinely once-per-install Committing/Persisting phases.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The Operate → Traces "API Traces" panel already recorded who made each
request (user_id/user_name) but never showed it, and did not capture the
caller's network identity at all. Operators asked to see the requesting
user (#10886) and the client IP + user agent (#10887) so a trace can be
attributed to who/what issued it.
Backend: add ClientIP and UserAgent to APIExchange and populate them from
echo's c.RealIP() (honours X-Forwarded-For / X-Real-IP behind a trusted
proxy) and the request's User-Agent header. Both are omitempty and the
/api/traces swagger response is map[string]any, so this is additive.
UI: add a sortable "User" column to the API traces table and a metadata
block (User / Client IP / User Agent) at the top of the expanded row
detail. Fields render only when present, so older buffered traces and
unauthenticated/local requests degrade cleanly.
Adds an e2e spec covering the new column value and the expanded metadata.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The Operate -> Traces table rendered the request time with the time of day
only, so entries that span more than one day were ambiguous. Add a
formatDateTime helper (localized date + existing time-with-millis) and use it
for the Traces "Time" column, keeping the cell on a single line. The shared
formatTimestamp used by the log views is unchanged.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
