Add CachyLLaMA as a GGUF-compatible llama.cpp fork backend with CPU and Vulkan builds on Linux and Metal on Apple silicon. Expose it through model import, document persistent SSD cache options, and wire CI path filtering and dependency updates.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(models): fallback to application config default context size (#12202)
Honor appConfig.ContextSize in /v1/models/capabilities when model context_size is unset.
* docs(models): explain context size fallback
Describe the application default used by capability discovery and
preserve the distinction between total context and per-request limits.
Assisted-by: Codex:GPT-6
* fix(models): apply the default context size only when context_size is unset
The request path applies the application default context size only
when a model leaves context_size unset. An explicit 0 or -1 falls
through to the backend fallback. The capabilities endpoint now does
the same, so it reports the value the backend uses.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(functions): honor function_arguments_key when building the tool grammar
All four call sites of `Functions.ToJSONStructure(name, args string)` pass
`FunctionsConfig.FunctionNameKey` as *both* arguments, so
`FunctionArgumentsKey` never reaches the grammar generator.
`ToJSONStructure` writes the two properties into the same map:
property[nameKey] = FunctionName{Const: function.Name}
property[argsKey] = Argument{...}
When `nameKey == argsKey` the second assignment overwrites the first, so a
model configured with `function_name_key` gets a grammar carrying only the
arguments object -- the `{"const": "<function name>"}` constraint is gone and
the grammar can no longer express which function was called.
With `function_name_key: function`, the generated property set collapses from
{"function": {"const": "get_weather"}, "arguments": {...}}
to
{"function": {"type": "object", "properties": {...}}}
Setting only `function_arguments_key` is equally broken in the other
direction: the grammar keeps emitting `arguments` while `ParseFunctionCall`
(pkg/functions/parse.go) looks up the configured key, so the parsed call comes
back with its arguments empty.
The default configuration is unaffected -- with both keys empty
`ToJSONStructure` falls back to `name`/`arguments` for both parameters, which
is why this went unnoticed.
The existing `ToJSONStructure()` unit test already calls the helper with two
distinct keys, so only the call sites were wrong. Extend that test with a case
that keeps both custom keys distinct and asserts the two properties survive.
Signed-off-by: Anai-Guo <antai12232931@outlook.com>
* test(functions): cover configured grammar keys
Route grammar construction through FunctionsConfig so the regression test
covers the key wiring used by every endpoint.
Assisted-by: Codex:gpt-5
* chore: empty commit to trigger workflow approval
Signed-off-by: Tai An <antai12232931@outlook.com>
---------
Signed-off-by: Anai-Guo <antai12232931@outlook.com>
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
fix(router): re-seed the knn corpus index when the vector store comes back empty
The corpus manager records a store as synced by file fingerprint and
embedding fingerprint. The local-store backend behind it is an in-memory
gRPC process the model loader may evict (active-backend cap, memory
pressure) or the idle watchdog may kill, and relaunch on the next
request — empty. The file is unchanged, so EnsureLoaded returned early
and the router went blind: every probe fell back with similarity 0 while
corpus/stats kept reporting the full count.
Measured on a production router (LOCALAI_MAX_ACTIVE_BACKENDS=6, four
resident models + two router stores): loading any further backend
evicted a store, and the idle watchdog killed both after 15 minutes;
/stores/find returned 0 hits against a 100-line corpus file whose stored
vectors matched fresh embeddings with cosine 1.000.
Two parts, because the knn classifier is built once and cached
(GetOrBuildClassifier), so the sync at build time is otherwise the only
one for the process lifetime:
- corpus.Manager remembers one vector it inserted (probe) and, on the
synced path, asks the live index for it. A miss means the index was
relaunched — fall through and re-seed from the file (no re-embedding).
- The router middleware wraps the knn classifier's store so every
lookup runs EnsureLoaded first; the loader gets the raw store, so its
probe never re-enters the wrapper. A sync error fails the lookup
closed, like the build-time load.
Specs: corpus package (relaunched empty store is re-seeded under an
unchanged file), middleware (relaunched index behind the cached
classifier is re-seeded instead of falling back; the spec is red without
the wrapper). The test fake now answers Search for inserted vectors.
Folds in the maintainer's follow-up (router-corpus-reseed-after-store-relaunch): reviewed and accepted.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* feat: return 429 when backends are saturated
When backends are at capacity (per-model max_concurrent or the
process-wide --max-concurrent-backend-requests ceiling), the response
was 503. The OpenAI SDK, litellm, and most agent harnesses key on 429
for rate-limit backoff and treat 503 as a hard error.
Both saturation paths now return 429 with the existing Retry-After
header and type: "rate_limit_error" in the JSON body. The per-model
admission middleware keeps admission_rejected as the code field so
existing alerts that match on it still fire.
Non-saturation 503s are unchanged: model cold-loading (with progress
body), model-load failure cooldown, PII detector fail-closed, and
classifier unavailable. These mean "not ready" rather than "busy".
Assisted-by: AGENT:regolo/glm5.2 [TOOL]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix: return 503 when scheduler has no available nodes
When the scheduler cannot find any healthy node to serve a model —
all nodes are full and eviction cannot free a slot, or a node_selector
excludes every candidate — the error fell through to 500. A 500 tells
clients something is broken when the condition is transient and
retryable.
The router now wraps these errors with a new ErrNoAvailableNodes
sentinel. The HTTP error handler maps it to 503 via applyNoAvailableNodes,
following the same pattern as applyBackendAdmission (429). Unrelated
scheduler errors (DB timeouts, registry lookups) still return 500.
Three return sites are wrapped:
- resolveSelectorCandidates: selector matches zero healthy nodes
- scheduleNewModel eviction-busy: all models have in-flight requests
- scheduleNewModel eviction-failed: eviction itself errored
The existing scheduleAndLoad wrapper ("no available nodes: %w") preserves
the sentinel through the chain via errors.Is, as does ModelRouterAdapter.
Assisted-by: AGENT:regolo/glm5.2 [TOOL]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(http): use Ginkgo for admission tests
Replace forbidden testing.T calls with Ginkgo and Gomega so the lint
check accepts the admission handler tests.
Assisted-by: Codex:GPT-6 forbidigo
* fix(middleware): show the recorded status for admission rejections
The admission audit row now records 429, but the Middleware page still
printed a hard-coded 503. Read the status from the event, and update
the two package comments that still said 503.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Hardcoding size/size_vram as 0 made Ollama clients treat loaded models as
free. Prefer ModelFileName+ModelPath Stat when available, and omit size_vram
(and size) when the value is unknown instead of emitting literal zeros.
Resolve each listed model by its stored ID so tagged variants use their
own weights.
Fixes#11969
Signed-off-by: lei_lei <imleilei123@gmail.com>
* feat(system): report per-model DRM VRAM
Expose optional resident device memory for local backend process trees.
Deduplicate DRM clients and omit unsupported or incomplete readings.
Document accounting limits and preserve a measured zero in JSON.
Closes#11970.
Assisted-by: Codex:gpt-6
* fix(system): document trusted procfs reads
Scope G304 annotations to paths built from the fixed procfs root,
integer process IDs, and kernel directory entries. These reads accept
no user-controlled path components.
Assisted-by: Codex:GPT-6 gosec
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Keep each message and reasoning item at its announced output index.
Include the answer in completed responses with reasoning or fallback
function calls, and retain reasoning supplied through backend deltas.
Add regression coverage for stream indices, final output, plain text,
and automatic tool parsing.
Assisted-by: Codex:GPT-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Partial JSON parsing heals a name-only chunk into a tool call. The
stream emits that call with empty arguments and skips later chunks.
Require complete JSON before emitting terminal tool-call events.
Preserve complete calls before an unfinished trailing call, and count
only actual tool calls. Add split-chunk regression tests and docs.
Refs #11635. The non-streaming report remains unconfirmed.
Assisted-by: Codex:GPT-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(vllm-cpp): unify decision pipeline through Score RPC with vllm_decide ABI v29
Replace the model-specific SystemOne gRPC approach with a generic Score
RPC extension. The pre-existing Score RPC (previously unused by any
backend) now carries question_type and response_json fields:
- question_type="systemone" routes kev/laya decision-pipeline requests
through the unified vllm_decide C ABI (v29), returning the full
response JSON in response_json.
- question_type empty routes cua-s1-forms candidate scoring through the
same vllm_decide ABI, returning CandidateScore probabilities.
The vllm-cpp backend's Score() method calls vllm_decide and dispatches
by architecture internally. The /v1/systemone HTTP endpoint checks
whether the model's backend supports Score; if so, it forwards the raw
request JSON and returns the backend response as-is. Other backends
fall through to the existing NER-based path.
This mirrors the vllm.cpp C ABI refactor (PR #3301) that replaced
vllm_systemone + vllm_score with a single vllm_decide function. The
purego bindings bump abiVersion from 27 to 29 and resolve vllm_decide
and vllm_decide_free symbols.
Also fixes validModelPath to accept cua-s1-forms.json and
rl_agent_config.json alongside config.json, matching the engine's
model_loader.cpp config-filename ordering.
AI-Assisted: true
Assisted-by: Maki:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore: ⬆️ Update mudler/vllm.cpp to e28ec46c6 (fix macOS -Werror build)
Bumps vllm.cpp to e28ec46c6 which fixes a -Wnull-conversion error in
qwen3_5.cpp:12483 that broke the macOS Metal CI build under -Werror.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): verification follow-ups for oci:// galleries
Follow-ups from the post-merge review of #12238 and #12239.
Only a policy decision is a refusal now. cosignverify wraps
ErrPolicyRejected around a failed signature check, an identity or
source-repository mismatch, a not_before cutoff and a missing or
unparseable bundle. A TUF, registry or network failure during
verification, or a timeout, is an outage: the gallery falls back to the
copy verified under the current policy, as it does when the registry is
down.
An oci:// gallery with a verification block, or any oci:// gallery under
strict integrity, is no longer answered by an https://, github: or
file:// mirror. Such a mirror is ignored with a warning, because nothing
can check its signature. The index of an HTTP gallery, whose policy only
covers its backend images, is cached under the URL-only name again, so no
unchecked body is stored under a policy-keyed name.
The in-memory index cache key now includes the policy. After a runtime
policy change the index is fetched again, and entries with a relative url
install again.
The registry digest lookups after install and upgrade, and in the
upgrade check, run only for real registry references (new
URI.LooksLikeRegistryOCI), not for ollama:// or ocifile://.
The refusal message names strict integrity when that is the cause, and
the gallery name is no longer repeated.
Specs pin the URL-only cache name for galleries without a policy, a fixed
key for a fixed policy, and that every GalleryVerification field changes
the key. The docs describe refusal, outage, mirrors and strict integrity.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): reset listings on gallery changes, classify referrer outages
Review follow-ups for this PR.
The React UI lists from AvailableGalleryModelsCached, which is keyed by
nothing. A gallery change through the settings API or a
runtime_settings.json edit now drops that listing when the model or
backend gallery configuration differs. Before, the UI kept the old list,
with local paths into the old policy's tree, until the next background
refresh, or for good when the new policy refused the gallery.
In cosignverify, a referrer the registry fails to serve now makes the
lookup an outage whatever other referrers failed and in any order, since
the unread one may be the valid signature. An invalid policy (Validate in
NewVerifier, an unparseable not_before) is ErrPolicyRejected, because no
fetch can make it usable.
The docs say that only an oci:// gallery with a verification block skips
non-OCI mirrors, and list an unusable policy as a refusal.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify
Wire the vllm-cpp backend to the C ABI NER surface (vllm_gliner_ner,
ABI v27) so LocalAI can serve zero-shot named entity recognition through
the existing TokenClassify gRPC method.
backend.go: TokenClassify method on *VllmCpp calls vllm_gliner_ner with
the text and labels, copies the C-owned entity array into protobuf
TokenClassifyEntity messages, and frees the result.
govllmcpp.go: cNerEntity and cNerResult Go POD mirrors matching the C
structs; vllmGlinerNer and vllmNerResultFree purego bindings; abiVersion
bumped 26 -> 27.
options.go: ner_labels, ner_threshold, ner_max_width parsed from
engine_args.
pkg/grpc: ClassifyModel interface and TokenClassify server handler
(follows the Embedding locking pattern).
core/config: vllm-cpp backend declares MethodTokenClassify and
UsecaseTokenClassify.
docs/content/features/vllm-cpp.md: NER section documenting the
engine_args keys and the host-forward contract.
Assisted-by: MAKI:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): correct NER pointer lint directive
Use the govet directive for the C-owned NER array, matching the other
purego pointer conversions. The array remains valid until its deferred
free; the misspelled directive caused CI to flag this conversion.
Assisted-by: Codex:gpt-6 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): add kev-compatible SystemOne API endpoints
Add POST /v1/systemone, /v1/systemone/permute, and
/v1/systemone/separate to LocalAI, mirroring the kev project's
structured-extraction API. Each endpoint runs zero-shot NER over the
rendered state text and builds kev-compatible answers for three question
types: noul (binary entity presence), choice (pick one option), and
score (pick one level).
The TokenClassifyRequest proto gains a `repeated string labels` field so
each question can supply its own labels at inference time, and
TokenClassifier gains TokenClassifyWithLabels for per-call label
selection. The vllm-cpp backend uses request labels when non-empty,
falling back to configured ner_labels then the built-in defaults.
Helpers (renderState, softmax, choiceConfidence, scoreConfidence, r2)
are ported from kev/api.py and mirrored in vllm.cpp's api_server.cpp so
both servers produce the same answer shape.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): suppress gosec G404 on seeded permutation RNG
The SystemOne permute endpoint uses math/rand with a caller-supplied
seed for reproducible option permutations, matching kev's random.seed.
gosec flags this as G404 (weak RNG). Add #nosec with a comment naming
the intent: this is reproducibility, not cryptography.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(vllm-cpp): bump vllm.cpp pin to GLiNER2.5 merge commit
Advance VLLM_CPP_VERSION from f3cd97e to 5058268d, the commit that
landed GLiNER2.5 zero-shot NER support (PR #3224) in vllm.cpp. This
brings the DeBERTa v2 encoder, GLiNER2 boundary head, C ABI NER
functions, and server endpoints into the LocalAI vllm-cpp backend.
The ABI version (27) and Go struct mirrors already match.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): use instruction text as NER label in SystemOne handler
The SystemOne handler was passing question IDs as NER labels for noul
questions and bare key names for choice questions, so the model never
matched any entities. Port the label mapping from vllm.cpp's
ParseSystemOneBody:
- noul: use the rendered instructions field (with instr alias) as the
NER label, not the question ID
- choice: use optionText(name, desc) — "name: description" or "name"
when the description is null/empty — not the bare key
- score: already correct (rendered criteria text)
- permute: shuffle indices and build parallel key/label arrays so the
NER call uses the optionText labels while the response is keyed by
the original option names
Also add the instructions field to the SystemOneQuestion schema struct
(accepted alongside the instr backward-compat alias).
Verified end-to-end against the real GLiNER2.5 model: noul questions
now find "Apple Inc. is" (organization, 0.999) and "Tim Cook is"
(person, 0.852) where they previously returned zero entities.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(capabilities): report the per-request context with split KV slots
With parallel slots and kv_unified:false, llama.cpp gives each slot
n_ctx/n_parallel, padded up to a multiple of 256. /models/capabilities
still reported the full n_ctx. A client that budgets a request against
context_size then overflows at a fraction of it.
EffectiveRequestContextSize returns the per-slot size in that case and
the full context otherwise. With the unified KV cache, the grpc-server
default, one request may use all of n_ctx. The capabilities endpoint
and the router's prompt trimmer now use it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(openai): return an HTTP error when a stream fails before any chunk
A streamed chat request set the SSE headers, then waited for the
backend. When the backend failed before the first token, LocalAI sent
a 200 with a `data: {"error":...}` chunk and [DONE]. Clients that do
not parse error chunks saw an empty reply. cogito's LocalAI client was
one of them: nib users got "streaming decision produced no content"
instead of the context overflow that caused it.
Nothing has been written at that point, so the handler now returns the
error as a normal HTTP response. A failure after the first chunk keeps
the in-stream error chunk.
A prompt that exceeds the context is now a 400 on both paths, as in
the OpenAI API and llama-server, and no longer a 500. The message is
kept whole, because clients read the token counts from it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(e2e): check the error from closing the response body
golangci-lint's errcheck flags the unchecked resp.Body.Close in the
new pre-stream error helper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat(kimodocpp): add API and backend request observability
Capture animation requests, phase timings, output metadata, and failures in traces. Record correct API error statuses and cover completed, running, failed, and disabled tracing.
Assisted-by: Codex:GPT-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Track successful header authentication before allowing cross-site requests
to bypass CSRF checks. Arbitrary headers on unauthenticated servers and
cookie-authenticated requests no longer grant an exemption.
Share the production CSRF middleware with multipart tests, add regression
coverage for credential sources, and document the exemption behavior.
Assisted-by: Codex:gpt-6 golangci-lint
Signed-off-by: Richard Palethorpe <io@richiejp.com>
On a single-node install nothing in Operate listed the models loaded on
this machine or let an admin stop one. The System page that did was
retired in #11548, and its replacements (the Nodes workbench) only work
in distributed mode. The Nodes page also mis-detected single-node mode:
the cluster routes are not registered there, so /api/nodes answers 404,
but only 503 was treated as "distributed off", which sent every
single-node install to the empty worker-registration card. The rail hid
the entry anyway.
Nodes route on a single node becomes "This machine":
- the Nodes page's VRAM / RAM / CPU / models-disk gauges, fed from this
host by mapping /api/resources onto the worker heartbeat fields
- a memory bar splitting host RAM by running model
- a running-models table (backend, RSS, CPU share, uptime, PID) with
search, sorting, logs and a confirmed Stop
- the distributed setup behind an "Add machines" button
The Operate overview gains a "Running now" preview (heaviest five, with
Stop) on single node and a pointer to Nodes > Running models on a
cluster. The rail shows "This machine" in Runtime with a running count.
Backend, additive only:
- /system: each loaded model carries a `process` block (pid, rss_bytes,
memory_percent, cpu_percent, started_at). A sampler keeps one gopsutil
handle per PID so CPU is the share since the previous poll rather than
the lifetime average; it is omitted on the first reading.
- /api/resources: host `cpu` and models-path `disk`, the same readings
workers send in their heartbeat.
Also fixes the fleet tables widening the page on phones: the headers'
absolutely positioned sr-only labels escaped the scroll wrapper.
Assisted-by: Claude:claude-opus-5 [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
An alias config is a pure redirect with no backend of its own, so the
capabilities listing described it from its stub: no capabilities, no
modalities, and the default 4096 context_size. Clients that size their
context budget from this endpoint (nib, for one) then compacted every
turn against a model that really serves 100k.
Resolve the alias and report the target's capabilities, modalities and
context_size under the alias's id. A dangling or chained alias now
reports no enrichment instead of defaults no model runs with.
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Report input tokens and frame-step output units in response metadata and
record them through the existing usage accounting pipeline. Preserve the
accounting rule and model-specific dimensions as JSON without extending
the gRPC schema for each modality.
Expose animation usage only under metadata.usage, validate counts before
recording, and document the response contract and loaded-model location.
Add coverage for transport, defaults, failures, persistence, and recording
requests once with statistics enabled or disabled.
Assisted-by: Codex:GPT-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
The global ::selection used --color-primary-light (14% primary), which
composites to about 1.1:1 against the dark page ground — selected text
was nearly indistinguishable from unselected. Give selection its own
token in both palettes and align the CodeMirror themes with the same
strengths.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
Expose a negative_prompt string parameter on the image endpoints,
matching Stable Diffusion WebUI / vLLM-Omni conventions. When both the
negative_prompt parameter and a '|'-suffixed negative prompt in the
main prompt are present, they are joined with a comma so callers can
keep a global negative prompt in negative_prompt and add per-image
negative tags after '|'.
Assisted-by: Pi: DeepSeek V4 Pro
Signed-off-by: Fedor Zuev <Fedor.Zuev@gmail.com>
* fix(vulkan): preserve host ICD discovery for packaged backends
Add bundled Mesa manifests through VK_ADD_DRIVER_FILES instead of replacing the system driver list. Merge inherited and model-specific additive paths while preserving explicit operator overrides, with regression coverage.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(3d): add Kimodo CPU and Vulkan animation backend
Introduce a distinct animation capability and model-described 3D operations, with a typed /3d/animate API, RPC transport, distributed media staging, permissions, and tracing.
Add a persistent kimodo.cpp adapter, skeleton GLB export, CPU/Vulkan packages, model and backend galleries, importer support, CI builds, and documentation. Adapt Studio inputs to each model and provide real-time skeleton playback, seeking, and history.
Cover backend validation, packaging, API behavior, importer inventories, distributed staging, and Studio workflows. Validate real-model CPU/Vulkan generation and deploy the integration to the local QA instance.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(kimodocpp): adopt monolithic encoders and resident inference
Update upstream for resident weights, packed execution paths, and cached motion graphs. Default to all 32 text layers while retaining configurable streaming and legacy bundle support.
Use monolithic Q8_0 encoders by default and offer all six published quantizations through the gallery and importer. Refresh pinned hashes, tests, and documentation; remove the obsolete thread patch and ensure cached source checkouts follow the upstream pin.
Validated CPU and Vulkan generation, lower-bit streaming, gallery/importer suites, packaging, lint, and cold/warm Studio generation on localai-dev.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ui): restore node operation controls
The node restructure hid backend logs and split related controls
across inconsistent layouts.
Restore contextual log actions and align the detail page with the
fleet dashboard. Make multi-node selection clear and accessible.
Assisted-by: Codex:gpt-5 Playwright ESLint
* feat: expose context_size in /v1/models/capabilities
The /v1/models/capabilities endpoint already enriches the OpenAI models
listing with capabilities and modalities. Add context_size to that
enrichment so clients (e.g. nib) can size their context budget for
auto-compaction and pruning without guessing.
The value comes from backend.EffectiveContextSize, which resolves the
configured context_size or falls back to DefaultContextSize (4096) —
the same number the backend itself runs with. It is omitempty so a
loose file with no config surfaces 0 (unknown) rather than a misleading
default, and existing clients that don't know the field see no change.
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The node detail redesign moved backend operations into an action menu.
Four existing specs still search for the removed direct buttons, so the
UI E2E workflow fails consistently on master.
Open the backend action menu before checking or activating its items.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The node restructure hid backend logs and split related controls
across inconsistent layouts.
Restore contextual log actions and align the detail page with the
fleet dashboard. Make multi-node selection clear and accessible.
Assisted-by: Codex:gpt-5 Playwright ESLint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Do not seed streaming reasoning state when the latest prompt thinking marker is already followed by its matching closing marker. This keeps direct Gemma 4 output in content when its template disables thinking with a preclosed channel.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Adds FunASR/SenseVoice as a Python backend for speech-to-text with
support for CPU, CUDA 12/13, ROCm, Intel SYCL, L4T, and Apple MPS.
Co-authored-by: xingyifeng <xingyifeng@users.noreply.github.com>
Agent Status replaced the chat route and unmounted its EventSource. Any response still in flight could then disappear from the conversation.\n\nOpen status in a separate tab so the chat keeps its live connection until the response completes.\n\nAssisted-by: Codex:gpt-5 [eslint]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The API mock also matched navigation to /app/tts and returned a WAV download instead of the React page. Let non-POST requests reach the test server.
Assisted-by: Codex:gpt-5 [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
TTS instructions add a second textarea to the page. Target the speech input by its placeholder so the history test does not depend on the page having one textarea.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Let studio users guide speech delivery for backends that support request instructions. Blank guidance stays out of requests and media history.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Clients cannot discover the named voices that an installed TTS model accepts without consulting backend-specific documentation. Expose voice metadata through the audio API and let custom model configs declare their own catalog.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Route prefix-cache replica selection through composable filters, weighted scorers, and a replaceable picker. Preserve the existing load guard and deterministic selection policy while exposing per-model scorer weights through scheduling configuration.
Assisted-by: Codex:gpt-5 [go-vet]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(auth): bypass API-key auth for CORS preflight (OPTIONS) requests
When API-key auth is enabled, a browser making a cross-origin API call
first sends an OPTIONS CORS preflight, which cannot carry credentials by
HTTP spec. The auth middleware is registered (app.go:324) before the CORS
middleware (app.go:337-347), so the preflight hit auth first and returned
401 before the CORS middleware could answer it, blocking the actual call.
Bypass auth for OPTIONS so the request reaches the CORS middleware, which
answers the preflight with 200 + headers. Real API requests (GET/POST/etc.)
still require auth. Regression test added (red on master, green on branch).
Refs #4576
Signed-off-by: supermario_leo <leo.stack@outlook.com>
* fix(auth): exempt CORS preflights via publicRouteRegistry instead of middleware bypass
Route the global OPTIONS exemption through publicRouteRegistry
(OPTIONS on every path, replacing the OPTIONS-under-/api/auth/ rule
it subsumes) instead of a hardcoded method check inside Middleware,
so "which requests skip auth" has one mechanism. Preflights now flow
through the same authenticate-then-public-rules path as other public
routes, which also lets a credentialed OPTIONS request keep its user
context.
Update the route-coverage allowlist and the near-prefix lookalike
table for the new semantics (OPTIONS is public on every path by
design; near-prefix privacy stays pinned by the non-OPTIONS entries),
and fix the authentication docs' exempt-route enumeration, which
still described OPTIONS as an /api/auth/-only exemption.
Signed-off-by: supermario_leo <leo.stack@outlook.com>
---------
Signed-off-by: supermario_leo <leo.stack@outlook.com>
fix(ui): preserve decoded route parameters
React Router already decodes dynamic path segments before exposing them through useParams. Decoding those values again crashes pages for names containing a literal percent sign and mutates escape-like substrings.
Use route parameters as-is, encode the model editor API path at the outbound boundary, and cover all affected pages with Playwright.
Fixes#11882
Assisted-by: Codex:gpt-5 eslint playwright
Signed-off-by: QiuLG <l237455523@outlook.com>
When no sampled candidate fits GPU memory, ranking falls back to the
oversized pool and labels its first model Best fit. Keep GPU picks within
the existing 95% budget and hide the section when no candidate qualifies.
Remove static GPU starter picks so Home cannot reintroduce the same error.
Add browser regressions for both sections and document the empty result.
CPU fallback behavior stays unchanged.
Assisted-by: Codex:gpt-6 [Codex]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(chat): template.system_messages_after_first — merge or forward late system turns
Tokenizer chat templates such as Qwen3.8 / Qwen3.8-Flash-Next raise
'System message must be at the beginning' for system-role messages that
appear after the leading system block, while agent frameworks (cogito tool
selection and adjustment prompts) legitimately append system instructions
mid-conversation. Every such request failed with a 500 (48 errors in one
10-task agent run).
New per-model option template.system_messages_after_first:
merge fold late system turns into the leading system message
user forward them as user-role turns at their original position
Default (unset) keeps the current pass-through behaviour.
Fixes#11876
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* docs(model-config): document template.system_messages_after_first
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* fix(config/meta): register template.system_messages_after_first in the field registry
TestAllFieldsHaveRegistryEntries requires every model-config field to have
a registry entry. Adds the entry (templates section, select component) and
the option list for the new field so the coverage gate passes.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
---------
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* fix(realtime): support session voice profile switching
Keep the active resolved voice binding on the realtime session so updates can atomically replace model, voice, and profile parameters while releasing leases at the correct lifecycle boundaries.
Assisted-by: Codex:gpt-5
* docs(realtime): explain voice profile switching
Document the session.update payload for selecting a Voice Library URI and clarify precedence when changing the model in the same event.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): resolve saved voice profiles
Realtime pipelines now validate saved voices against the selected TTS model and retain leased audio until session teardown. Each synthesis request receives its own transcript parameter map.
Assisted-by: Codex:GPT-5
* docs(tts): document realtime voice defaults
Show how a realtime pipeline selects a saved Voice Library profile at session start. Clarify which session voice updates remain supported.
Assisted-by: Codex:GPT-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Realtime turns could invoke the LLM and TTS even when speech transcription returned only whitespace. This let ambient noise produce unsolicited assistant output and polluted conversation history with an empty user turn.
Require non-blank transcript text before automatic response generation while preserving the completed transcription event.
Assisted-by: Codex:gpt-5 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Accept original embeddings and timestamps so clients can restore faces
when the in-memory store restarts. Derive stable IDs from exact vectors
to make registration retries preserve identity without duplicate entries.
Assisted-by: Codex:GPT-6 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
