mirror/LocalAI
1
0 Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-09-12 22:33:54 -04:00
Code Issues Packages Projects Releases Wiki Activity
7,674 Commits 187 Branches 177 Tags
bebd812e7dbf520cfcc620135cb106d1cca1ec8f
Commit Graph
1085 Commits
Author SHA1 Message Date
Ettore Di Giacinto bebd812e7d fix(distributed): stop flapping agent nodes on backend listing 
Only backend workers subscribe to backend.list. ListBackends asked every
node that was not pending, offline or draining, so an agent worker could
only answer "no responders", which the error handling reads as a node
that has gone away. Every poll of the backends view therefore marked
each agent node unhealthy, and its next heartbeat marked it healthy
again.

While unhealthy the node is not schedulable, so this also cost agent
capacity for as long as each flap lasted.

Skip non-backend workers, as the backend-op fan-out already does for the
same reason. A backend worker that does not answer is still marked
unhealthy: that one really is gone.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
 2026-08-24 12:48:23 +00:00
Ettore Di Giacinto df1a40f9c0 fix(distributed): hash the config as persisted, not as defaulted 
The revision was computed after SetDefaults, which folds in things that
are not persisted configuration: the GGUF guess, the hardware defaults,
and app-level options such as threads.

The GGUF guess is the damaging one. It parses the model file to fill in
values like context size, and when that parse fails it falls back to a
different default. Whether a multi-gigabyte file on network storage
parses at a given moment is not a property of the configuration, so one
unchanged YAML produced two different revisions depending on when it was
read. The controller rejected every request carrying the other one, and
the model stayed unroutable until the stored value happened to match
again. This is why it never reproduced against a model directory with no
weights in it: the guess is skipped there and both values agree.

The app-level defaults are the same class of bug with a slower fuse:
changing threads in the settings UI changed every model's revision and
made every model unroutable.

The revision is now stamped when the file is parsed, before any defaults
are applied, so it is a function of the file alone.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
 2026-08-24 11:45:32 +00:00
Ettore Di Giacinto 505a6d040b fix(distributed): publish the revision a request actually carries 
Two code paths computed a model's revision. Inference resolves the
config through the loader, which applies SetDefaults a second time.
Everything that publishes a revision hashed the stored config instead,
with SetDefaults applied once.

SetDefaults is not idempotent for every model: it re-runs the GGUF guess
and the hardware defaults, both of which read state the stored config
does not carry. Where the two disagree, a publisher wrote a revision no
request would ever carry, and the model became unroutable the moment it
was published. On this cluster the startup resync republished one such
value and every request for that model was then rejected against it.

The publishers now resolve the revision through the loader, exactly as a
request does, so there is one definition rather than two that agree only
when SetDefaults happens to be idempotent. This covers the startup
resync, a saved config edit, and enabling or disabling a model.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
 2026-08-24 09:05:52 +00:00
localai-org-maint-botandlocalai-org-maint-bot a8bc64cd09 fix(ci): bound Discord release summaries (#11695) 
* fix(ci): bound Discord release summaries

The release model can return more than Discord's 2,000-character
message limit. Discord then rejects the entire release notification.

Ask the model for a smaller response and truncate extracted content to
1,800 characters before the notification step. The smaller bound leaves
room below Discord's hard limit when model output varies.

Assisted-by: Codex:gpt-5

* fix(tests): implement node liveness stub

NodeCommandSender now requires PingNode. The endpoint test stub must
implement it before the package can compile.

Assisted-by: Codex:gpt-5 [Codex]

---------

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-24 09:32:26 +02:00
Ettore Di Giacinto 5c9d8190d9 fix(distributed): resync revisions after the configs are loaded 
The resync added in 3953448f6 ran before LoadModelConfigsFromPath, so it
read an empty loader, reconciled nothing and reported success. The
symptom was a stored revision that stayed stale across restarts while
the log showed no complaint, which is exactly what the resync was meant
to prevent.

Move the call after the configs are loaded, and refuse to treat an empty
loader as a clean run: reconciling zero models is indistinguishable from
reconciling correctly, and that is what hid the mis-ordered call.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
 2026-08-24 06:40:30 +00:00
Ettore Di Giacinto 3953448f60 fix(distributed): resync stored config revisions at startup 
The controller pins a model's replicas to a stored revision and rejects
any request carrying a different one. Nothing ever re-derived that value
from the configuration on disk: it moved only on an edit, a gallery
install, or a peer's change broadcast. An inference request may only
establish a revision, never replace one.

So any other way for the two to diverge left the model permanently
unroutable. A configuration edited while a frontend was down lands
there, and so does a change in what the revision is computed over: an
upgrade that alters the hashed form leaves every stored revision
describing a configuration that no longer exists. The only recovery was
deleting the row by hand, which is not something a cluster should need.

Each frontend now reconciles the stored revisions against the loaded
configurations at startup and republishes the ones that disagree. Only
those: republishing quarantines every replica loaded under the old
revision, so doing it for a model that did not drift would unload a
healthy replica for nothing. A model with no stored revision has never
been served and is left for its first request to establish.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
 2026-08-23 22:17:59 +00:00
Ettore Di Giacinto e6269e3cdd fix(distributed): reclaim replica slots held by abandoned loads 
A replica row in staging or loading holds its slot, because slot
allocation counts every state except unloading. Nothing ever reclaimed
such a row: every reconciler pass and the router's eviction query filter
state = "loaded", and the per-model probe skips rows without an address,
which is exactly what a row that never finished loading has.

So a worker that dropped out mid-transfer left a row that pinned the
only replica slot for that model on that node. Scheduling then found no
free slot and eviction found nothing it was allowed to evict, and the
request failed with "no replica slot on <node> and eviction failed: all
models busy". The state persisted until an operator intervened.

The reconciler now reclaims a row stuck before serving when no load job
is driving it. Ownership is decided by the job's LastProgress heartbeat,
not by elapsed time: staging a large checkpoint legitimately runs for a
long while without touching the replica row, so a deadline would either
be a model-size cliff or reclaim a healthy transfer. That heartbeat is
the same signal job takeover already trusts. Any error reading the job
leaves the slot held, because holding one for another pass costs a
scheduling opportunity while a wrong reclaim restarts a multi-gigabyte
transfer.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
 2026-08-23 21:07:29 +00:00
Ettore Di Giacinto c541dbeef4 fix(distributed): check a node answers before scheduling onto it 
A node's status comes from its HTTP heartbeat. Backend installs travel
over NATS. The two are independent, so a worker that dies stops
answering on the bus at once but stays healthy in the database until its
heartbeat ages out. Inside that window the scheduler picked a node it
could not reach, and the request failed with "no responders available"
rather than moving to a node that was up.

The scheduler now probes the node it selected and, when nothing answers,
marks it unhealthy and selects again. The demotion is what makes the
retry terminate: the next selection reads only healthy nodes. It also
tells the other frontends what this one learned, so the cluster does not
rediscover a dead worker one failed request at a time.

Only nats.ErrNoResponders counts as absent. A worker that answers slowly
stays eligible, because dropping it would cost capacity that is really
there. The probe reuses the models.running subject: a new subject would
go unanswered by workers that have not been upgraded, and every one of
them would then look dead.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
 2026-08-23 20:44:43 +00:00
Ettore Di Giacinto cee87d1608 fix(distributed): expire staged request files on the worker 
A request that carries a file stages it to the worker, which writes it
under its staging directory. Nothing removed it afterwards. The frontend
expires ephemeral keys from object storage, but that sweep never covered
a worker's local disk, so every image, audio clip and video a worker
ever served stayed on it.

One worker had accumulated 175 request directories over three months.
The volume reached 100 percent, and from that point every backend start
failed because the process manager could not create a state directory.

The worker now sweeps its ephemeral staging directory on a timer and
once at startup, so files left by a crash are reclaimed too. Staged
model files live beside that directory and are not touched.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
 2026-08-23 20:20:02 +00:00
Ettore Di Giacinto 4bad644498 fix(distributed): name both revisions in the stale error 
"stale model config revision" reported only that two hashes differed.
It named neither, so an operator could not tell an edited configuration
from a revision that is not reproducible for one unchanged file, and the
failing value appears in no table.

The error now carries the revision the request brought and the one the
controller holds. It still wraps ErrStaleModelConfigRevision, so callers
that classify the error keep working.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
 2026-08-23 19:53:42 +00:00
Ettore Di Giacinto f3fabe8c5c fix(distributed): order derived usecases deterministically 
syncKnownUsecasesFromString rebuilds KnownUsecaseStrings by ranging
GetAllModelConfigUsecases, which is a map. Go randomizes that order per
call, and the field is part of the serialized config, so one unchanged
YAML hashed to a different config revision on every load.

A model that derives a single usecase hid the problem. One that derives
several, such as a chat model with an mmproj, alternated between as many
revisions as there are orderings. The router treats a revision it did
not establish as a config change, so requests failed with "stale model
config revision" until the stored value happened to match again.

Sorting the list makes the revision a function of the file alone.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
 2026-08-23 17:20:20 +00:00
Ettore Di Giacinto 04735cd1f6 fix(distributed): stamp config revision at load time 
The request middleware merges the caller's prediction parameters into
its copy of the model config. core/backend.ModelOptions then hashed
that copy, so the revision identified the request body rather than the
persisted configuration.

EstablishModelConfigRevision stores the first revision it sees and
requires an exact match afterwards. The first request after a restart
therefore pinned the model to its own temperature, top_p and stop
values, and every later request that sent different ones failed with
"stale model config revision". No config edit was involved.

The loader now stamps the revision when it materializes a config,
before any request override reaches it, and ModelOptions reads that
stamp. Model administration keeps hashing the same persisted config, so
both paths agree on one revision per configuration.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
 2026-08-23 14:35:44 +00:00
localai-org-maint-botandlocalai-org-maint-bot 8f56e4e042 fix(vram): persist remote probe metadata (#11487) 
* fix(vram): persist remote probe metadata

The startup warmer repeated remote size and GGUF metadata probes after every restart because both caches lived only in memory. Store successful HTTP probes for 24 hours so frequent restarts reuse the prior results.

Bound the cache, reject invalid records, and purge it when gallery data changes. Local model files continue to bypass persistence.

Assisted-by: Codex:gpt-5

* fix(vram): check temporary file cleanup

The lint gate rejects the unchecked cleanup call in the persistent cache writer.

Assisted-by: Codex:gpt-5.6 [golangci-lint]

* fix(vram): make persistent cache optional

Remote metadata probes can transfer enough data that operators need
control over disk reuse and startup warming. Gallery autoload now gates
both behaviors, and the runtime setting applies changes immediately.

Assisted-by: Codex:gpt-5

* fix(ui): expose gallery startup pre-warm

The existing gallery autoload setting also gates the startup metadata warmer. Name both effects in Settings so operators can find the requested boot control.

Assisted-by: Codex:gpt-5

---------

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-23 08:55:13 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto 82c191afad fix(distributed): keep model replicas config-consistent (#11664) 
* docs: design configurable copy buffering

Document the context-aware copy buffer option and its validation plan.

Assisted-by: Codex:gpt-5

* docs: design durable distributed staging operations

Assisted-by: Codex:gpt-5

* docs: design distributed model config revisions

Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]

* feat(config): add stable model revisions

Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons.

Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]

* feat(worker): acknowledge exact model stops

Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]

* feat(nodes): track model config revisions

Assisted-by: Codex:GPT-5 [apply_patch]

* fix(distributed): retry quarantined model cleanup

Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work.

Assisted-by: Codex:gpt-5

* fix(distributed): bind loads to config revisions

Assisted-by: Codex: GPT-5 [OpenAI Codex]

* fix(modeladmin): apply config revisions consistently

Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes.

Assisted-by: Codex: GPT-5 [OpenAI Codex]

* feat(distributed): expose model config revision state

Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options.

Assisted-by: Codex:GPT-5 [OpenAI Codex]

* test(distributed): cover model revision convergence

Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness.

Assisted-by: Codex:gpt-5

* fix(distributed): pass config revision CI checks

Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files.

Assisted-by: Codex: GPT-5 [OpenAI Codex]

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-08-22 22:44:03 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto 9d92139de4 feat(ui): edit scheduling rules in place (#11667) 
* docs(ui): design scheduling rule editing

Document the approved in-place rule editing flow and scalable node-label reference for the scheduling view.

Assisted-by: Codex:gpt-5

* feat(ui): improve scheduling rule management

Add scalable node-label discovery and editable scheduling rules with responsive, accessible controls.

Assisted-by: Codex:gpt-5

* chore(ui): ratchet inline style baseline

Record the static inline style removed by the scheduling view enhancement.

Assisted-by: Codex:gpt-5

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-08-22 00:51:48 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto a0252ad6a1 fix(distributed): keep staging operations stable (#11663) 
* docs: design configurable copy buffering

Document the context-aware copy buffer option and its validation plan.

Assisted-by: Codex:gpt-5

* docs: design durable distributed staging operations

Assisted-by: Codex:gpt-5

* fix(distributed): merge durable staging operations

Use active model load jobs as the durable operations baseline and overlay replica-local staging progress without duplication. Preserve tracker-only operations when the registry cannot be read.

Assisted-by: Codex:gpt-5

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-08-21 20:08:47 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto 5429f569e0 fix(progress): stop status updates throttling downloads (#11661) 
* feat(progress): aggregate and coalesce gallery downloads

Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ui): show rolling transfer speed

Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(ui): preserve legacy import byte labels

Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-08-21 18:54:28 +02:00
github-actions[bot]andmudler 7f2c599f4a chore: bump inference defaults from unsloth (#11654) 
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
 2026-08-21 08:46:35 +02:00
Anupam Mediratta ff6043b811 fix: upgrade react-router to 7.18.2, 8.3.0 (GHSA-qwww-vcr4-c8h2) (#11644) 
* fix: GHSA-qwww-vcr4-c8h2 security vulnerability

Automated dependency upgrade by OrbisAI Security

Signed-off-by: anupamme <mediratta@gmail.com>

* fix: upgrade react-router-dom to 7.18.2 to fully remediate GHSA-qwww-vcr4-c8h2

The prior fix pinned react-router@7.18.2 directly but left react-router-dom
at ^7.18.1, which bun resolved to 7.18.1. That package bundles its own
react-router@7.18.1 sub-dep, leaving the vulnerable version in bun.lock
via the react-router-dom/react-router scoped resolution. Pinning
react-router-dom to 7.18.2 and regenerating the lockfile removes all
7.18.1 resolutions.

Assisted-by: Claude Code:claude-sonnet-4-6
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>

---------

Signed-off-by: anupamme <mediratta@gmail.com>
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
 2026-08-21 08:41:23 +02:00
Anupam Mediratta 9ba4bbf9bb fix: upgrade ip-address to 10.3.1 (CVE-2026-69192) (#11632) 
fix: CVE-2026-69192 security vulnerability

Automated dependency upgrade by OrbisAI Security

Signed-off-by: anupamme <mediratta@gmail.com>
 2026-08-20 20:04:11 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto 0ab632b6bd fix(auth): protect HTTP routes by default (#11602) 
* fix(auth): default to protected HTTP routes

Use a method-aware registry for the small anonymous bootstrap surface.
Unknown routes now require credentials instead of inheriting fail-open
path classification.

Keep node self-service routes behind their registration-token middleware.
Global auth no longer rejects valid worker credentials first.

Assisted-by: Codex:gpt-5

* docs(auth): document public HTTP surface

Assisted-by: Codex:gpt-5

* test(auth): align route coverage with default denial

Assisted-by: Codex:gpt-5

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-08-19 20:49:19 +02:00
Ettore Di Giacinto 2383726d6d Revert "chore(tests): Avoid network, sleep and more during tests" (#11601) 
Revert "chore(tests): Avoid network, sleep and more during tests (#11050)"

This reverts commit cb3bf7af3f.
 2026-08-19 16:39:39 +02:00
Richard Palethorpeandlocalai-org-maint-bot cb3bf7af3f chore(tests): Avoid network, sleep and more during tests (#11050) 
* test: make coverage failures observable

Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: parallelize coverage without remote fixtures

Assisted-by: Codex:gpt-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: add offline resource infrastructure

Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: enforce offline resource replay

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: harden offline resource refresh

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: expose slow coverage waits

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: eliminate avoidable wall-clock waits

Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: remove repeated fixture startup waits

Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: fix offline resource CI portability

Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: cache Go modules before offline tests

Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: drop the static network lint in favour of real isolation

The offline test suite already prevents tests from reaching the network
twice over: run-test-linux-offline.sh puts the test process in a cgroup
and REJECTs egress outside the private ranges, and HardenedTransport
installs testnetwork.LocalGuard to refuse dials that resolve to a public
address. Both fail the test with a precise error at the moment of the
dial.

test-network-lint.sh added neither. Its diff stage defaulted to a HEAD
base, so on a clean checkout it compared the tree against itself and
inspected nothing; the branch's own commits were never examined. It only
produced output when an earlier job step dirtied the tree, and then it
matched a bare https?:// against whatever changed. make react-ui runs
npm install rather than npm ci, so CI rewrote
core/http/react-ui/package-lock.json and the lint reported an npm
registry URL as forbidden test network access:

  +      "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz",

Its fingerprint stage was self-defeating in a quieter way: hashing the
whole tree's network-mechanism inventory meant every rebase onto a master
that touched any _test.go needed a manual baseline bump, so the check
mostly caught its own staleness.

Remove the script, its make target and the two prerequisite edges, along
with the test-network: fixture markers that existed only to suppress it.
The isolation itself is untouched.

Assisted-by: Claude:claude-opus-5 [go vet]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: keep hidden files in the offline test bundle artifact

Cherry-picked from 15a37b0ac on the remote branch. The offline bundle lives
under .cache/, which actions/upload-artifact skips by default, so the Linux
job packed an artifact missing the very file the next step restores.

The other half of 15a37b0ac moved test-network-lint out of the `test` and
`test-coverage` prerequisite lists into a recipe line, so parallel make could
not fingerprint the tree while generated fixtures were still changing. That
is dropped: the preceding commit removes the lint entirely, and the race it
worked around is one more reason a whole-tree fingerprint was the wrong
mechanism.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* refactor: share bounded exponential backoff

Use overflow-safe saturating arithmetic for retry delays across model import polling, downloads, registration, node operations, and model loading. Keep model import status checks responsive initially while capping their interval at 500ms.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: mirror Jetson Python wheels

Keep the CUDA aarch64 wheel subset in GHCR and serve it as a local PEP 503 index during L4T backend builds, preserving last-known-good packages through upstream outages.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* docs(agents): index the Jetson wheels mirror

Mention the GHCR-hosted L4T wheel mirror in the CI caching guide summary so maintainers can find its outage and cache documentation.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: add defensive build network proxy

Record build destinations and byte counts, retry observable idempotent HTTP downloads, and isolate explorer database tests that race under coverage.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(kokoros): implement updated backend trait

Return unimplemented for image upscaling, matching the backend's other unsupported modalities after the protobuf API update.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): clear recovered proxy errors

Do not mark a request failed when a later safe retry succeeds.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: require HTTPS build interception

Inject a short-lived proxy CA into BuildKit and Dockerfile RUN steps, reject plain HTTP and opaque tunnels, and retain method/status/byte telemetry for verified HTTPS traffic.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): preserve system trust in unproxied builds

Mount the generated interception CA at a dedicated secret path and add it to the trust bundle only in proxy-aware dependency stages. This prevents optional secret mounts from masking the system CA bundle in ordinary backend test builds.

Install the requested Go toolchain before starting the proxy and satisfy cleanup error checks found by CI lint.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): persist build proxy trust

Install the generated proxy CA through the system-managed local certificate directory so ca-certificates upgrades retain it. Avoid turning canceled matrix jobs into proxy cleanup failures.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): trust proxy in nested build scripts

Install the build proxy CA before nested source fetches, route the DS4 package setup through the HTTPS mirror helper, and avoid repeated OCI setup in gallery behavior tests.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): use HTTPS apt sources for Bonsai

Rewrite ARM64 package sources before installing GCC and check gallery fixture cleanup errors so the optimized tests satisfy errcheck.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(privacy-filter): trust build proxy CA

Install the mounted build proxy certificate before privacy-filter's make target fetches its HTTPS sources, for both source and prebuilt builder paths.\n\nAssisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>

* test: fail on hidden offline egress

Count cgroup-scoped firewall rejects and fail the offline test harness with bounded aggregate diagnostics. Inject the gen-audio GGUF probe so fixture-backed importer tests do not attempt real network access.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): preserve system CA trust

Build a combined runner certificate bundle instead of replacing public roots with the generated proxy CA. Centralize additive container installation in the shared proxy CA helper.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-19 10:59:31 +02:00
localai-org-maint-botandlocalai-org-maint-bot 4cad809003 fix(ci): test stale chunks in split bundle (#11595) 
The V8 coverage build inlines every dynamic import, so the stale chunk tests cannot intercept a page chunk. Run those tests against the normal code-split bundle and exclude them from the inlined coverage pass.

Assisted-by: Codex:gpt-5

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-19 08:45:59 +02:00
Ettore Di Giacinto b806b1fec3 fix(ui): reload once when a page chunk 404s 
A deploy replaces the whole content-hashed asset set at once. A tab
holding an older index.html, or one whose request lands on a replica
that the rollout has not swapped yet, asks for a page chunk the server
no longer has. The dynamic import rejects and React Router's default
error boundary replaces the app with "Unexpected Application Error!"
until someone reloads by hand.

The router now reloads the page itself when a chunk fails to load.
index.html is served no-cache, so the reload lands on a self-consistent
asset set. A timestamp in sessionStorage bounds this to one reload per
10 seconds, so a chunk that is genuinely gone reaches the error boundary
instead of looping forever.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
 2026-08-18 19:51:15 +00:00
localai-org-maint-botandlocalai-org-maint-bot 0761bd02c7 feat(chat): add end-to-end context compression (#11556) 
* feat(config): add context compression policy

Define the opt-in model configuration contract before the chat middleware consumes it. Document each policy field so later request handling does not invent a second schema.\n\nRefs #9534\n\nAssisted-by: Codex:gpt-5

* fix(config): register compression fields

The model editor metadata gate rejects new config fields without descriptions and suitable controls. Register the compression policy so operators can edit its six fields safely.

Assisted-by: Codex:gpt-5 [monitoring-prs]

* feat(chat): compress long contexts

Long conversations currently fail once they reach the model context window. The opt-in policy now summarizes complete older turns before primary inference and preserves the newest tool chains.

Both OpenAI and MCP chat routes share the same transformation. Usage metadata and metrics expose each compression event.

Refs #9534

Assisted-by: Codex:gpt-5

---------

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-18 11:31:03 +00:00
Nicholas Ciechanowski cf93c04878 fix: tts text wrap (#11576) 
Signed-off-by: Nicholas Ciechanowski <nicholas@ciech.anow.ski>
 2026-08-18 12:57:55 +02:00
Richard Palethorpe d10374f849 feat(router): make KNN a first-class classifier with a persisted, curated corpus (#10652) 
* feat(router): make KNN a first-class classifier with a persisted, curated corpus

Add `classifier: knn` — similarity-weighted voting over labelled
example prompts. Unlike score/colbert it needs no classifier model:
label knowledge lives in a corpus seeded and curated through the
admin API, so routing decisions are deterministic, auditable, and
grounded in graded experience rather than a model's opinion.

Epistemic gate: corpus entries below knn.similarity_threshold cannot
vote; when none clears it the classifier activates no labels and the
router uses the fallback — a prompt unlike all labelled experience is
treated as undecidable, not guessed. Decisions record
nearest_similarity (also on fallback rows) so admins can see how far
the nearest labelled experience was; the Routing tab explains
out-of-corpus fallbacks and shows per-label corpus counts.

Persistence: one JSONL file per router under
<data path>/router-corpus (text, labels, vector, embedder
fingerprint). The file is the source of truth; the local-store index
is rebuilt from it at classifier build time and stays a pure
in-memory index. Entries recorded under a different embedding model
re-embed on load. Also corrects the docs' false claim that
local-store collections persist — the embedding cache never survived
restarts (and still doesn't); the corpus does.

Corpus input is API-only by design (entries may contain example user
content): POST /api/router/{name}/corpus seeds (labels validated
against declared policies, embedded server-side, indexed
immediately), GET .../corpus/stats inspects — label counts only,
entry texts are never returned by any surface — DELETE .../corpus
wipes. Admin-gated like the sibling router endpoints, and exposed as
MCP tools (seed_router_corpus / get_router_corpus_stats /
clear_router_corpus) in both the httpapi and inproc clients with
coverage-test route mappings.

Plumbing: VectorStore gains SearchK (top-K was hardcoded to 1);
local-store gets InsertBatch/Delete as optional fast paths;
RouterConfig gains a knn block (embedding_model, k,
similarity_threshold, vote_threshold, store_name) with meta-registry
fields; the classifier dropdown now offers knn and the
previously-missing colbert; embedding_cache is ignored (with a
warning) for knn — it IS an embedding-KNN lookup; the stale
/api/instructions intelligent-routing entry is rewritten (it
described a classifier that no longer exists); swagger regenerated.

Tests: KNN vote/gate specs with hand-computed vote shares, corpus
manager suite (restart reload without re-embedding, fingerprint
re-embed, dedupe, hostile store names), middleware specs (corpus
routing, gate fallback, config validation, cache-wrap refusal),
corpus endpoint specs pinning the texts-never-returned contract, MCP
catalog + route-mapping gates, and a Playwright spec for corpus
stats and the out-of-corpus decision detail.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(router): name consulted corpus neighbours in knn decisions

Every knn decision (decision log rows and the /api/router/decide
response) now carries neighbors: the K retrieved corpus entries by
descending similarity - including ones below the epistemic gate, which
is what makes fallback decisions diagnosable - each as {id, similarity,
labels}. The id is the entry's content hash (first 8 bytes of the
SHA-256 of its text, hex): stable across reseeds and re-embeds, and
text-free, so an external platform that seeded the corpus can recompute
text->id on its own copy and bucket decisions by corpus region (per-
region reliability accounting) without corpus text ever leaving the
server. A corrupt index payload surfaces as an id-less neighbour at a
real similarity instead of disappearing.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* refactor(router): deduplicate knn plumbing and cut corpus hot-path waste

Post-review cleanup of the knn-first-class-router branch; no behaviour
changes on the API surface.

Reuse/altitude:
- RouterKNNConfig.ResolvedStoreName is now the single source of the
  router-corpus-<name> default (was hand-derived in four files).
- corpus.ResolveKNNRouter + corpus.Seed carry the shared model
  resolution and seed validation; the REST endpoints and the assistant
  MCP client are thin transport adapters over them, with sentinel
  errors mapped to HTTP statuses at the echo boundary.
- middleware.NewClassifierDeps assembles the classifier dependency set
  once for all five entry points (OpenAI, Anthropic, realtime, decide,
  corpus) instead of five hand-copied literals.
- router.AllClassifiers feeds both the status endpoint and the
  unknown-classifier error, ending the classifier-list drift.
- Per-classifier requirements moved out of validateRouterPolicies into
  their buildClassifier arms; the knn arm owns its embedding_cache
  opt-out instead of a name-check in the shared wrap tail.
- adminOnly replaces four inline copies of the admin gate in the
  middleware routes.
- localVectorStore.Search delegates to SearchK (identical traces).

Efficiency:
- Manager.Add embeds outside the manager mutex and appends to the
  JSONL file (O(new) instead of O(corpus) rewrite); a torn tail from a
  crash mid-append is tolerated on read and repaired on next write.
- Stats memoises per store keyed on the file's stat fingerprint and no
  longer takes the manager mutex, so the 5s status poll stops parsing
  vector-laden JSONL and stops blocking behind seeds.
- KNN Classify decodes each neighbour payload once (was twice) and
  builds refs and votes in a single pass with one fallback return.
- Corpus file writes fsync before rename/close.
- The corpus manager is built eagerly in newApplication (sync.Once
  dropped); test helper dead branch removed.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(router): bind knn corpus vectors to an embedder fingerprint and fail closed on mismatch

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* chore(mcp): align corpus tool prompts and the mutating-tool safety list

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(proto,backend): report embedding shape from the llama-cpp backend

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(embeddings): Go-side pooling — mean/last/decayed_mean with half-life

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(embeddings): accept chat messages[] and per-request pooling on /v1/embeddings

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* chore(middleware): name the failing fields when post-merge validation 400s

An intermittent post-merge validation failure surfaced as an opaque 400
during integration (pooling scheme mismatch that no client had sent).
Log the model, the request's pooling override, and the merged config's
pooling fields at the failure point so the next occurrence identifies
whether the request or the stored config carried the bad value.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(embeddings): scheme override must not inherit the config's half-life

A model config defaulting to decayed_mean pooling carries
pooling_half_life_tokens; a request overriding the scheme to mean/last
without its own half-life inherited that value, and post-merge
validation rejected the pair the server itself had assembled. Zero the
inherited half-life when the overridden scheme is not decayed_mean; a
request that explicitly pairs a half-life with a non-decayed scheme
still 400s.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix embedding pooling validation and router bounds

Declare backend embedding layouts and reject incompatible pooling modes. Reset local-store dimensions after a full clear, validate KNN thresholds, and add real backend and store integration coverage.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: run local-store integration tests

Build and install the local-store backend in the Linux test job, then run the existing store integration suite so new specs are discovered automatically.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
 2026-08-18 09:37:43 +02:00
localai-org-maint-botandlocalai-org-maint-bot a7bce6a128 fix(audio): reject incompatible transform streams (#11565) 
The transform WebSocket accepted any model and opened its frame-based RPC. Any-to-any models use a different stream contract, so liquid-audio failed with an unimplemented RPC after the handshake.

Reject incompatible model use cases before loading the backend. Direct realtime-audio callers to the OpenAI Realtime API.

Assisted-by: Codex:gpt-5

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-18 09:03:16 +02:00
Richard Palethorpe 799cc9f211 feat: bound global admission and expose running backend traces (#11560) 
feat: bound backend admission and expose running traces

Add process-wide backend execution admission without blocking UI or administrative HTTP work. Represent backend operations while they are in flight, surface running traces with immediate log links, and tie streaming admission leases to the gRPC receive lifecycle.

Assisted-by: OpenAI Codex: GPT-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>
 2026-08-18 08:56:59 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto 0aaff91ebd feat(ui): unify model and backend lifecycle (#11548) 
* feat(ui): add installed model lifecycle

Models now owns catalog exploration and installed runtime controls under one canonical route. URL-owned state keeps lifecycle context recoverable through links and browser history.

Assisted-by: Codex:gpt-5 Playwright

* feat(ui): add installed backend lifecycle

Backends split discovery from backend-binary management. The canonical
page now keeps both lifecycle views under one URL-backed shell while it
preserves target-node placement.

Assisted-by: Codex:gpt-5 Playwright

* fix(ui): repair lifecycle state updates

Installed models lost distributed refreshes and kept a deleted selection. Backend searches also stopped tracking URL changes, while batch upgrades stopped after their first error.

Preserve background refreshes and finish each requested batch action. Drive catalog results from URL-backed state without losing full metadata.

Assisted-by: Codex:gpt-5 [Playwright]

* feat(ui): make resource pages canonical

Replace Host navigation with canonical Models and Backends lifecycle routes, preserve legacy management URLs, and surface shared host capacity on the Operate overview.

Assisted-by: Codex:gpt-5 [Playwright]

* feat(ui): complete canonical resource lifecycle

Finish the responsive list-to-detail behavior, remove the retired Host implementation, and keep Explore focused on discovery while Installed owns destructive actions. Update regression coverage, localization, documentation, and development binding for the canonical resource pages.

Assisted-by: Codex:gpt-5 [Playwright]

* docs(ui): record the UI design context

Record the approved users, brand character, and design principles so
future interface work uses the same product direction. Index the context
from the repository's agent instructions.

Assisted-by: Codex:gpt-5

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-08-16 11:57:31 +02:00
mudler's LocalAI [bot]Ettore Di Giacintolocalai-org-maint-bot
 		88edd7fc7f fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock (#11514) 
* fix(advisorylock): set statement_timeout alongside lock_timeout

WithLockCtx already overrides a deployment-wide lock_timeout on its
dedicated connection so a blocking pg_advisory_lock() waits its turn
instead of failing with 55P03. statement_timeout aborts that exact same
statement independently, with SQLSTATE 57014, and was not overridden.

Production roles commonly carry statement_timeout=60s. Any guarded
section longer than that (a cold model load stages for tens of minutes)
therefore killed every concurrent waiter:

  advisorylock: acquiring lock 9003261067483446873: ERROR: canceling
  statement due to statement timeout (SQLSTATE 57014)

Derive it from the same context budget as lock_timeout, with a matching
RESET so the pooled connection is returned clean.

Assisted-by: Claude Opus 5 [claude-code]

* feat(distributed): add ModelLoadJob, the durable cold-load record

A cold load in distributed mode is a long-running background job, but it
was modelled as a synchronous side effect of an inference request: the
whole of it (backend install, multi-GB staging, checkpoint load) ran
inside the per-model advisory lock. Loading a 35.7 GB GGUF held that lock
for ~20 minutes, so every concurrent request for the same model blocked
on pg_advisory_lock and died at the role's 60s statement_timeout.

Introduce the row that lets the lock shrink to a decision. Exactly one
ModelLoadJob may be active per tracking key; that uniqueness — not the
lifetime of a lock — is what de-duplicates concurrent loaders across
replicas. ClaimLoadJob does its read-then-write under the advisory lock
and nothing else: no network, file or gRPC I/O inside the guarded
section, so a claim costs milliseconds no matter how long the resulting
load takes.

LastProgress is a heartbeat rather than a byte counter. A checkpoint load
legitimately moves zero bytes for many minutes, so a reaper keyed on byte
movement would reclaim a healthy job mid-load; byte progress stays the
concern of load_deadline.go. A job whose heartbeat stops for longer than
the orphan window is reclaimable, so a replica killed mid-load cannot
wedge a model permanently.

Failed jobs keep their row for a short grace so an immediately-following
request reports the real cause instead of silently starting a fresh load
of a model that just failed.

No caller yet — the router moves onto this in the next commit.

Assisted-by: Claude Opus 5 [claude-code]

* refactor(distributed): run cold loads as jobs, outside the advisory lock

Route wrapped the entire cold load — node selection, backend install,
multi-GB staging and the remote LoadModel — in the per-model advisory
lock. The lock's job is to de-duplicate concurrent loaders, a decision
that takes milliseconds; holding it for the tens of minutes the resulting
work takes is what turned a dedup mechanism into a cluster-wide outage
for that model.

Split it into a claim and a run. The claim is the only thing left inside
the lock. The run is a background job owned by the claiming replica and
bounded by the same progress-extended deadline as before; every other
request for that model — local or on another replica — attaches as a
waiter and is served the moment the model is ready, with no duplicate
load and no lock contention.

Waiters share one broadcast rather than an ordered queue: they all want
the identical outcome, so ordering them would add fairness machinery that
changes no result. The local channel wakes same-replica waiters instantly
and a 2s DB poll is the authority, because a waiter on another replica
has no channel to close. On wake a waiter re-runs the warm path rather
than trusting the signal — the model may have been evicted in between.

A waiter whose client disconnects returns immediately and the job keeps
running; it belongs to the job record, not to the request. A failure is
recorded on the row so every waiter reports the real cause, and the row
survives briefly so the next request does not read "no job" as "not
loading" and start a duplicate load of a model that just failed.

The runner heartbeats the row on a fixed interval whether or not bytes
are moving, which is what keeps a legitimately silent checkpoint load
from being reclaimed as an orphan. Phase (installing/staging/loading) and
placement ride to the heartbeat on the context, the same seam
load_deadline.go already uses, so single-host paths are untouched.

Non-distributed mode (no DB) keeps the inline load exactly as it was.

Assisted-by: Claude Opus 5 [claude-code]

* feat(distributed): bound the wait for a loading model and answer with progress

A request whose model is cold-loading now attaches to the running job and
is served the moment the model is ready. That wait has to be bounded: a
held HTTP request cannot survive real infrastructure, and an ingress or LB
idle timeout kills a twenty-minute request regardless of what LocalAI
does.

New LOCALAI_MODEL_LOAD_WAIT (default 60s) bounds the CALLER, never the
load — the job keeps running either way. On expiry the request gets 503
with Retry-After and a structured body naming the model, the node, the
phase, byte progress and an ETA. The `error` envelope keeps OpenAI
clients working; `loading` is additive so they ignore it.

The ETA comes from the job's own observed rate and is omitted rather than
guessed until enough bytes have moved for that rate to mean anything: a
confidently wrong ETA on a twenty-minute wait is worse than none.
Retry-After is that ETA when known, clamped to [5s, 300s], and the wait
budget otherwise.

LOCALAI_MODEL_LOAD_WAIT=0 waits unbounded, for deployments with no proxy
in front. Zero in the config struct still means "unset, use the default",
so the CLI records the operator's zero as ModelLoadWaitUnbounded rather
than losing the distinction.

The distributed branch of ModelLoader.loadModel wrapped the router's
error with %s, which flattened it to a string. Use %w: the typed error is
what the HTTP layer keys the 503 off.

Assisted-by: Claude Opus 5 [claude-code]

* feat(api): add GET /api/models/{id}/load-status

A client that receives 503 while a model stages onto a worker needs
somewhere to poll. This returns the same `loading` object the 503 carries
— phase, node, byte progress and ETA — or 404 when no load is running.

Read-only and observability-shaped, so it is deliberately neither
admin-gated nor feature-gated: it explains a 503 the caller just
received, and hiding that behind a per-modality feature would make the
explanation for a failed image request depend on chat permissions. It
also gets no MCP tool, since there is nothing here an admin would manage
conversationally.

Registered on the surfaces from .agents/api-endpoints-and-auth.md: the
swagger block (existing `models` tag, so /api/instructions needs no new
area), the endpoint discovery maps in RegisterLocalAIRoutes, regenerated
swagger, and the distributed-mode docs page. No FLAG_* usecase is
involved, so capabilities.js is unchanged.

Assisted-by: Claude Opus 5 [claude-code]

* feat(ui): show cold-load progress in Chat and retry when the model is ready

A chat request for a model that is still staging onto a worker now gets a
503 carrying live progress instead of an error. Render it: the composer
shows the phase (installing / staging / loading), the node, the percent
and the ETA, then polls load-status and re-sends the request the moment
the model is ready.

Reuses the staging progress idiom the page already had rather than
inventing a second one — the two sources are folded into one
loadProgress, with the load job winning because it is authoritative
across frontend replicas and knows the phase, where the staging operation
only knows about a byte transfer this replica happens to be performing.

Waiting is bounded (three send attempts, ~30 min of polling each), so a
load that never finishes still surfaces as an error rather than as a
spinner nobody questions. An aborted generation stops the polling too.

Assisted-by: Claude Opus 5 [claude-code]

* fix(distributed): check warm-path cleanup errors

The router moved legacy cleanup calls onto newly linted lines. Report
cleanup failures while preserving the fallback to a cold load.

Assisted-by: Codex:gpt-5 [golangci-lint]

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-15 13:20:11 +02:00
localai-org-maint-botandlocalai-org-maint-bot 95653f221e fix(ui): keep agent import action visible (#11488) 
* fix(ui): keep agent import action visible

The header hid its full import label after the agent list became non-empty. Hide only the nested file input so users can import more agents.

Assisted-by: Codex:gpt-5

* test(ui): match the agent import label

The Agents page renders the action as Import.

The test searched for Import Agent, so it failed before checking visibility.

Mock the observables request to remove backend timing from the fixture.

Assisted-by: Codex:gpt-5

---------

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-14 15:06:51 +02:00
Richard Palethorpe 5c63969760 fix: Show MCP connection errors in the UI (#11495) 
* fix(mcp): surface configured server failures

Keep model-configured MCP servers visible when discovery or connection setup fails, propagate status through distributed discovery, and let the Chat UI show actionable errors while retrying unavailable servers.

Add model-editor metadata for remote and stdio configuration and document the expected format, deployment networking boundary, and alternate MCP scopes.

Assisted-by: Codex:gpt-5 Ordino golangci-lint
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* build(compose): match CUDA development image

Configure the API image with the cublas, CUDA 13, auth-tagged build settings used by the local development Makefile invocation, including the 24-way Docker build.

Assisted-by: Codex:gpt-5 Ordino
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* revert: keep host build settings out of compose

The CUDA development deployment is managed from ~/docker/localai, not the repository example Compose file. Restore the generic example and keep machine-specific build settings in the host deployment.

Assisted-by: Codex:gpt-5 Ordino
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(docker): exclude local agent artifacts

Keep Claude worktrees and locally installed verification tools out of the Docker build context. These host-only directories added roughly 1.9 GB to every root image build.

Assisted-by: Codex:gpt-5 Ordino
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
 2026-08-13 22:25:58 +02:00
Dedy F. Setyawan 9fd7ea7e93 i18n(id): translate admin, media, and nav UI strings to Indonesian (#11493) 
Signed-off-by: Dedy F. Setyawan <dedyfajars@gmail.com>
 2026-08-13 16:27:45 +02:00
localai-org-maint-botandlocalai-org-maint-bot 0647939953 fix(gallery): repair DeepSeek V4 fallback (#11480) 
The DeepSeek V4 Flash base entry points at a Hugging Face repository page instead of a GGUF object. When variant probing cannot rank a concrete build, the base fallback downloads no usable model weights.

Use the validated IQ2XXS object and checksum already shipped by the q2 variant. Pin that payload in the gallery resolution test.

Assisted-by: Codex:gpt-5.6

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-12 22:42:55 +02:00
github-actions[bot]andmudler b7a0646587 chore: bump inference defaults from unsloth (#11270) 
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
 2026-08-12 09:12:12 +02:00
mudler's LocalAI [bot]mudlerlocalai-org-maint-bot
 		3ec466b61b chore: ⬆️ Update ikawrakow/ik_llama.cpp to 26ceed9d4091a1696cf50e2ed87e5767d5811d81 (#11475) 
* ⬆️ Update ikawrakow/ik_llama.cpp

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix(gallery): align Higgs Audio checksum test

The validated gallery checksum changed in #11459, but its dedicated regression assertion kept the previous value and now fails the master test suite on Linux and macOS.

Assisted-by: Codex:gpt-5.6

---------

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-12 08:46:16 +02:00
Dimitris Karakasilis 3636fcbd38 fix(realtime): keep the ICE interface allow-list working with a fixed UDP port (#11466) 
LOCALAI_WEBRTC_ICE_INTERFACES was silently ignored whenever
LOCALAI_WEBRTC_UDP_PORT was set. Every interface was gathered regardless
of the allow-list, so a browser was handed the docker0/veth addresses the
setting exists to suppress, and the connection established on a good pair
and then dropped when consent checks failed on the unreachable ones.

Two things combine to cause it. A mux built over a wildcard socket makes
pion derive its host candidates by enumerating interfaces itself, with a
nil filter and loopback included. Independently, the muxed gathering path
in pion/ice never consults SetInterfaceFilter, so setting it has no effect
there either.

Bind one socket per admitted interface address via NewMultiUDPMuxFromPort,
which takes the filter, instead of one wildcard socket. All the sockets
share the same port, so the firewall requirement is still a single rule.
Networks are pinned to UDP4 to match the socket family this replaces.

An allow-list that matches no address on the host now reports the
misconfiguration rather than binding nothing and leaving signaling to
succeed while no candidate is ever advertised.

Two tests: one asserts an unmatched allow-list is an error, and one gathers
against a real peer connection and asserts no address outside the allowed
interface appears (skipped on single-interface hosts).

Assisted-by: Claude:claude-opus-5 go vet gofmt

Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
 2026-08-11 18:37:49 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto 45cb3983ee fix(ui): unmerge the class strings that left buttons in browser chrome (#11462) 
Eight header controls across seven pages had two or three elements' classes
collapsed into one string. The wrapper or the icon ended up wearing the
button classes, and the buttons themselves were left with no class at all,
so they rendered in the browser's own chrome. Reported on Agent Jobs; the
grep found the rest.

`fas` does not draw anything by itself: it sets
`font-family: "Font Awesome 6 Free"` and weight 900 on whatever carries it,
and the `fa-*` class supplies the glyph via ::before. So

    <button className="btn btn-primary fas fa-plus">

renders its own label "New Task" in the icon font, and

    <div className="hstack btn btn-primary btn-sm fas fa-edit btn-secondary fa-arrow-left">
      <button>Edit</button>
      <button>Back</button>
    </div>

styles the flex wrapper as a button that is both primary and secondary,
points two glyphs at one ::before, and leaves both real buttons bare.

Fixed, all of them keeping the correct `<i>` child they already had:

  - AgentJobs, AgentTaskDetails (x2), AgentCreate - icon classes off the
    button.
  - AgentTaskDetails, AgentJobDetails - wrapper back to plain `hstack`, and
    the two buttons inside each get the variants the wrapper had been
    holding. Back is secondary and leads, Edit/Cancel is the emphatic one
    and trails, matching every other detail header.
  - VoiceLibrary, VoiceProfileCreate - the title `<i>` had swallowed the
    action link's classes, so "Create voice" and "Back to library" were
    unstyled anchors. Back was also drawing a "+" because it had inherited
    fa-plus while its own fa-arrow-left sat up in the title.
  - P2P - a stray fa-circle-info on the title icon.

The ninth instance was ImportModel, where this class of bug was first
found. #11461 rewrote that file and landed first, so nothing is left to fix
there.

Guarded by e2e/class-hygiene.spec.js, which reads the source rather than
walking routes: several of these pages need agent or voice data before they
render a header, so a route walk would skip exactly the pages that had the
bug. It fails on an icon-font class outside an `<i>`/`<span>`, on two glyphs
or two button variants on one element, and on a layout wrapper that is also
a button. Font Awesome modifiers (fa-spin, fa-fw, sizes) are excluded, so
the `fa-spinner fa-spin` idiom stays legal.

e2e: 428 passed.


Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] [Playwright]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-08-11 12:07:42 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto 7a7fb00730 feat(ui): rebuild the import form on the restyled design language (#11461) 
The import page took the new palette in #11305 but kept its old layout, so
it stayed a 760px column with the primary action detached from the form it
submits. Two of the problems were outright bugs.

The Import button carried no className at all, so the page's single most
important control fell through to the user-agent button: system chrome,
wrong radius, no design-system focus ring. The YAML button carried
`fas fa-save fa-upload`, which sets Font Awesome as the button's own font
family (its label text inherits it) and points two glyph classes at one
::before.

On the layout: `page--narrow` is documented for "forms / single-record edit
views", and in Advanced mode this page held a URI field, a six-section
format guide, ten modality chips, nine preference fields, a key-value
repeater and a YAML editor at `calc(100vh - 400px)`. The width was the
symptom; one column was the disease.

  - `page--medium` with a work column and a format reference beside it.
    The reference answers the only question a first-time admin has and used
    to sit behind a chevron, closed by default. Below 1024px it becomes a
    disclosure rather than disappearing.
  - The source field is the hero: monospace, because it holds something you
    paste, and it carries its own Import button. That removes the hidden
    aria-hidden submit button that existed only because the real action sat
    outside the form.
  - Simple and Advanced are gone. They were ~80% the same surface, and the
    overlap cost a mode switch, a localStorage key and a three-button
    Keep/Discard/Cancel dialog whose only job was protecting state that
    switching modes would hide. One form with a collapsible options panel
    hides nothing, so none of it is needed. What genuinely differs is the
    kind of input, which is now the two tabs: a source, or YAML.
  - The size/VRAM estimate reports under the field that produced it instead
    of as a banner above the page header, and an import in flight gets the
    progress, phase and byte counts the poller already returned and the old
    status card threw away.
  - ModalityChips resolves its labels through the same `modality.*` keys as
    the dropdown it filters. It hardcoded English shorthand, so one modality
    carried two names on one screen ("Speech" on the chip, "Speech
    recognition" on the group it scrolled to) and seven locales had neither.
    Its inline styles and its pill radius move onto the design system.
  - Three inline styles go, including both conditional-padding hacks; the
    only one left is the progress bar's runtime width. Baseline 538 -> 535.

Docs updated in the same change: the WebUI section described a Simple and an
Advanced mode and told the reader to "Toggle to Advanced Mode".

e2e: 426 passed. The mode-switch suite is replaced by one covering the tabs
and the disclosure, and a new layout suite pins the width, the styled
primary action, the absence of an icon-font button, the reference column at
both widths, and the estimate's position.


Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] [Playwright]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-08-11 12:00:59 +02:00
localai-org-maint-botandlocalai-org-maint-bot 16193e1982 feat(gallery): add Higgs Audio v3 TTS (#11456) 
Expose the existing audio.cpp Higgs support as an installable Q8 gallery model and document voice cloning and licensing constraints.

Assisted-by: Codex:gpt-5

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-11 09:52:38 +02:00
localai-org-maint-botandlocalai-org-maint-bot f7db51bdf5 feat(realtime): add shared WebRTC UDP port (#11436) 
* feat(realtime): add shared WebRTC UDP port

Allow realtime WebRTC peer connections to reuse one configurable UDP mux, and surface listener bind failures through signaling.

Assisted-by: Codex:gpt-5

* test(realtime): keep UDP mux alive during bind check

The returned SettingEngine owns the UDP listener. Retain it through the duplicate-bind assertion so macOS cannot finalize the listener early and make the exclusivity check spuriously pass.

Assisted-by: Codex:gpt-5 [systematic-debugging]

* test(realtime): use IPv4 for UDP mux checks

Match the socket family used by the WebRTC UDP mux so macOS does not allocate an IPv6 probe that can coexist with the IPv4 listener.\n\nAssisted-by: Codex:gpt-5

---------

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-10 17:57:58 +02:00
dependabot[bot] 6cf2e97868 chore(deps): bump dompurify from 3.4.12 to 3.4.13 in /core/http/react-ui in the npm_and_yarn group across 1 directory (#11425) 
chore(deps): bump dompurify

Bumps the npm_and_yarn group with 1 update in the /core/http/react-ui directory: [dompurify](https://github.com/cure53/DOMPurify).


Updates `dompurify` from 3.4.12 to 3.4.13
- [Release notes](https://github.com/cure53/DOMPurify/releases)
- [Commits](https://github.com/cure53/DOMPurify/compare/3.4.12...3.4.13)

---
updated-dependencies:
- dependency-name: dompurify
  dependency-version: 3.4.13
  dependency-type: direct:production
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-10 10:21:41 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto 7b9167eaad feat(llama-cpp): serve Qwen3-TTS through the llama.cpp backend (#11392) 
* fix(config): do not read a TTS speaker-encoder mmproj as vision support

Qwen3-TTS on llama-cpp ships an mmproj holding the speaker encoder and
code predictor. VisionSupported() treated any non-empty MMProj as proof
of image input, so every such model would be advertised as vision-capable.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): add TTS request option parsing helper

Validates text and speaker reference presence and strictly parses the
top_k / top_p per-request params, in a header with no llama.cpp or gRPC
dependencies so the standalone C++ unit test gate picks it up.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(llama-cpp): range-check the TTS top_k and top_p request params

Format validation alone let NaN, infinity and out-of-range values through.
The consumer copies both values into the audio generation input
unconditionally and only guards its separate sampler assignment with
"> 0", a test NaN also fails, so a NaN reached llama.cpp with the guard
never firing. top_k must now be >= 0 and top_p must fall within 0.0 to 1.0
inclusive, with the bound written as a negated in-range test so NaN is
rejected rather than silently accepted.

Also cover the two checks the suite could not previously kill: the
whole-string check in the float parser and the int32 range check.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(llama-cpp): bump pin to f9e832c10 and carry the TTS server task

Picks up ggml-org/llama.cpp#26254 (Qwen3-TTS via mtmd) and #26536 (the
short-input audio chunk fix). Adds 0002-add-server-task-type-tts.patch,
the server-side half of the still-draft #26603, so TTS runs through the
slot scheduler instead of racing it. Remove that patch when #26603 merges.

The patch is rebased on top of the score patch: its tokenize-switch hunk
collided with the SERVER_TASK_TYPE_SCORE case, and its lone SRV_WRN call
passes no variadic argument, which the macro cannot expand. The score
patch itself needed no refresh.

Also fixes fallout from the bump in grpc-server.cpp: upstream dropped the
per-slot n_ctx argument from server_schema::eval_llama_cmpl_schema. Only
the schema branch loses it, since forks predating the server-schema split
still expect the old argument list.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): implement the TTS and TTSStream RPCs

Both were declared in backend.proto but unimplemented. They now submit a
SERVER_TASK_TYPE_TTS task and drain the response reader, the same shape
PredictStream uses.

The streaming path emits a leading sample_rate message and then raw PCM,
because ModelTTSStream builds the WAV header itself; the non-streaming
path emits a complete WAV to the requested dst.

The streamed samples are converted from the pipeline's float32 to signed
16-bit first. MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM hands back floats, while
the header ModelTTSStream writes announces 16-bit samples, so shipping
the floats verbatim would decode as noise.

prepare.sh and CMakeLists.txt now stage tts_request_options.h alongside
the other grpc-server helpers, and register its standalone test with
ctest the way passthrough_options_test is registered.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(llama-cpp): mask non-codec tokens for Qwen3-TTS generation

The Qwen3-TTS gen-audio pipeline maps a sampled backbone token to a
codebook row with an unchecked subtraction, in mtmd-helper-gen.cpp:

    inp.code0 = sampled - codec_0;

For ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF the vocab is 155008 tokens,
<|codec_0|> is 151936 and the codec codes end at 153983. The model's own
tokenizer.ggml.suppress_tokens holds 1023 ids covering 153984..155007,
every special above the codec range except <|codec_eos_token|> (154086)
which stays reachable as the stop token. Nothing masks the text range
0..151935, so the backbone can sample a text token at any step, the
subtraction goes negative, and ggml_compute_forward_get_rows aborts the
whole backend process on GGML_ASSERT(i01 >= 0 && i01 < ne01).

Complete the mask upstream started: bias every token below <|codec_0|>
to -INFINITY for TTS tasks so only codec codes and the codec EOS remain
reachable. The biases are appended to task.params.sampling.logit_bias,
which common_sampler_init already merges with the model's suppress
tokens into one llama_sampler_init_logit_bias, so no sampler is added to
the chain. Measured cost is 0.082 ms per sampled token and 1.16 MB, set
against a forward pass in the multi-millisecond range.

It lands in launch_slot_with_task rather than in a route handler so that
llama.cpp's own POST /tts and LocalAI's TTS/TTSStream RPCs are both
covered, and <|codec_0|> is resolved from the vocab rather than
hardcoded so a model without it is left alone.

This is reproducible with upstream's own llama-tts and no LocalAI code
loaded, aborting at frame 55 on Q4_K_M and frame 71 on Q8_0, so it is
neither a quantization artifact nor an artifact of the gRPC adapter.
Two further defects in the same draft pipeline still prevent end-to-end
audio; they are independent of this one and are recorded in the task
report for an upstream bug report.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(llama-cpp): bump pin to 9de0fcf2b and drop the TTS codec mask

Upstream fixed the Qwen3-TTS abort in ggml-org/llama.cpp c8e03ce81
("mtmd/ggml: add ggml_build_forward_order", #26649), landed one hour
after the previous pin. ggml_build_forward_expand marks a tensor and all
its ancestors for compute, so using it as a pure ordering hint defeated
ggml_build_forward_select and made GEN_WAV calls execute the GEN_CODE
branch against a stale inp_code0, hitting the get_rows bound assert in
ggml_compute_forward_get_rows.

That single defect accounts for every abort seen on this model, so
0003-mask-non-codec-tokens-for-tts.patch is removed rather than rebased.
The mask changed the observed behavior, but it was perturbing a graph
ordering bug rather than fixing a sampling one: at the new pin the whole
path works without it. Keeping it would have meant carrying a 152k-entry
logit bias, and rebasing it on every pin bump, for no benefit.

Verified at 9de0fcf2b with only 0001 and 0002 applied, which both apply
clean with no fuzz and needed no rebase:

  non-streaming  HTTP 200, 410924 bytes, 8.56 s
                 RIFF (little-endian) data, WAVE audio, Microsoft PCM,
                 16 bit, mono 24000 Hz
  streaming      HTTP 200, 560684 bytes, 11.68 s, exactly one RIFF at
                 byte 0, same format, which also exercises the
                 float32-to-s16 conversion at runtime for the first time

Pristine unpatched llama-tts at the same pin now also completes, 130
frames to a valid WAV, where it aborted at frame 55 before.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(llama-cpp): clear the TTS slot sequence between requests

Only the first TTS request in a backend process succeeded. Every later
one failed instantly, in about 0.13 s, with "TTS prompt processing
failed" from step_prompt, regardless of streaming or non-streaming and
regardless of the text. With LOCALAI_SINGLE_ACTIVE_BACKEND=true the
process is kept alive between requests, so a deployment would have
served exactly one utterance per backend start.

The cause is missing KV hygiene, not anything in the gRPC adapter. TTS
slots never enter the shared batch: pre_decode() returns early for them
and process_tts_slots() drives them instead, so they skip the
prompt-cache bookkeeping that clears a slot's sequence between requests.
Nothing in the gen-audio path makes up for it: mtmd_helper_gen_audio_reset
only clears host-side buffers, and the pipeline always decodes from
position 0 into the sequence identified by slot.id. So the second task
on a slot writes positions 0..N over the first task's tokens and
llama_decode fails.

Fix is one call to slot.prompt_clear(), the same helper the normal path
uses, in the SERVER_TASK_TYPE_TTS branch of launch_slot_with_task before
set_input. It goes into 0002 rather than a new patch file because it is
a defect in the code that patch introduces, and the header now records
it as ours so we know whether it still needs carrying if #26603 merges
without it.

Verified in one backend process, different text on every request:
three consecutive non-streaming requests, three consecutive streaming
requests, and an interleaved non-streaming, streaming, non-streaming,
streaming run. All ten returned HTTP 200 with
RIFF ... WAVE audio, Microsoft PCM, 16 bit, mono 24000 Hz, the streamed
ones carrying exactly one RIFF header at byte 0, and every output
measured as real speech rather than silence or a truncated fragment.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): expose max_frames for TTS requests

The Qwen3-TTS backbone does not always emit <|codec_eos_token|>, and
when it does not, generation runs to upstream's 512-frame n_predict
default. At the model's 12.5 Hz frame rate that is 40.96 s of audio,
which a short input can trigger: one request in this session produced
40.96 s for a ten-word sentence. prepareTTSTask hardcoded n_predict to
-1, so callers had no way to bound it.

Add a max_frames key alongside top_k and top_p, parsed with the same
strict whole-string parsing so a typo is an error rather than a silently
truncated value, and rejected with a field-naming message when negative.
0 keeps the existing sentinel convention and means unset, so a request
that omits it behaves exactly as before.

Named max_frames rather than n_predict because frames are what the
parameter means at a TTS endpoint: one frame is 0.08 s of audio.

The 512-frame default is deliberately unchanged. Lowering it would
truncate legitimately long inputs, which is a worse failure than an
occasionally overlong one.

Verified end to end on one text of thirty words:

  max_frames=25    HTTP 200,  96044 bytes,  2.00 s, exactly 25 frames
  max_frames=50    HTTP 200, 192044 bytes,  4.00 s, exactly 50 frames
  no max_frames    HTTP 200, 572204 bytes, 11.92 s, stopped at its own
                   codec EOS after 149 frames, unchanged behavior

  max_frames=-1    InvalidArgument "max_frames must be >= 0, got \"-1\""
  max_frames=many  InvalidArgument "max_frames must be an integer, got \"many\""

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(llama-cpp): send the TTS sample rate up front, and tidy three review items

Four items from the Task 4 review.

Streaming first-byte latency. TTSStream sent the sample-rate reply only
once the first audio result arrived, and a chunk needs a whole 72-frame
window, roughly 5.8 s of audio and far longer in wall time on CPU. The
Go side blocks on that reply before it can emit the WAV header, so a
streaming client sat at zero bytes for the whole stretch. The rate is a
property of the loaded model and is available synchronously from
mtmd_gen_audio_get_info, so it now goes out immediately after post_task
and the rate_sent bookkeeping is gone. Measured on a warm model, first
byte drops from 30.48 s to 0.014 s, and the output is still a valid WAV
with exactly one RIFF header at byte 0.

Unchecked close. The non-streaming path ignored ofstream::close(), so a
failure that only surfaces on flush was reported as success while
leaving a truncated file at dst. It now returns INTERNAL like the other
write failures.

Wrong comment on set_lang. gen_audio::inp::get() already maps a stored
blank to nullptr, so our guard is behavior-preserving, not
behavior-fixing. The comment claimed otherwise; the code was right.

Repetition penalty. penalty_last_n = -1 is inert at this pin, because
llama_sampler_init_penalties clamps it with std::max(penalty_last_n, 0)
and then builds a disabled sampler, so the 1.05 penalty never applies.
Upstream's README attributes looping to a missing repeat_penalty, so it
was worth testing as a root-cause fix for the model running to the frame
cap. Dropping the line lets the sampling default of 64 apply, which was
confirmed in the sampler chain trace as penalty_last_n = 64 with
repeat_penalty = 1.050. Over 15 uncapped short requests each way it did
not help: 0 of 15 ran to the cap with the penalty inert, 1 of 15 with it
active. Both lines are therefore kept for parity with upstream's draft,
and a comment now records that the pair is inert and why, so the next
reader does not believe a penalty is applied. max_frames remains the way
to bound output.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* build(llama-cpp): let unpatched forks opt out of the TTS task

turboquant and bonsai copy grpc-server.cpp into llama.cpp forks that do
not carry our patches. disable-tts-task.sh injects the same kind of
preprocessor switch disable-score-task.sh already uses, so those builds
answer UNIMPLEMENTED rather than failing to compile.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(config): keep a TTS speaker-encoder projector out of vision detection

Task 1 exempted a declared-TTS model's mmproj from VisionSupported, but the
first real gallery entry with an mmproj still came back vision-capable through
two paths the earlier fix did not close.

GuessUsecases has no FLAG_VISION branch, so it falls through to true for any
chat-ish model. That is not just a wrong answer at the call site:
syncKnownUsecasesFromString rewrites KnownUsecaseStrings from HasUsecases, and
the loader calls it more than once per config file, so the guessed FLAG_VISION
is written out and parsed back into KnownUsecases as if the operator had
declared it. Give GuessUsecases a FLAG_VISION branch that defers to the same
explicit signals VisionSupported uses.

Second, llama.cpp builds an mtmd context for the speaker-encoder projector and
reports its media marker on the first chat probe, which resurrected vision
after the model had been used once. Apply the same declared-TTS exemption to
MediaMarker that the mmproj check already had.

Verified against the qwen3-tts-llamacpp-q4 gallery entry: no vision capability
and no image input modality, before load, after a TTS request, and after a chat
probe.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(gallery): add Qwen3-TTS entries for the llama-cpp backend

Two entries over upstream's own GGUF conversion, Q8_0 and Q4_K_M, each
pairing a backbone with the Q8_0 projector. Named to sit alongside the
existing qwen3-tts-cpp entries rather than replace them.

Also tags the llama-cpp backend text-to-speech / TTS so the backend browser
surfaces the capability.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: cover Qwen3-TTS on the llama-cpp backend

Adds the gallery variants, the two-file mmproj configuration, the
required voice reference, and the language and sampling knobs. Also
corrects the streaming-support list, which named only voxcpm.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(config): register llama-cpp as a TTS and voice-cloning backend

The branch taught the llama-cpp backend to serve Qwen3-TTS and shipped two
gallery entries for it, but never told the capability table. llama-cpp still
declared only the text RPCs and usecases, so:

- VoiceCloningForModel returned nil at the capability check, before it ever
  reached the model's own tts.voice_cloning override, and /tts answered 400
  "selected model does not support reference-audio voice cloning" for any
  localai://voice-profiles/... voice. No model YAML could opt back in.
- GET /api/backends/usecases did not list tts for llama-cpp, so the gallery
  greyed out the TTS filter for the entries this branch adds.
- The React TTS page saw voice_cloning: null and kept both models out of the
  Voice Library.

Add the TTS RPCs and usecase, and the reference-audio contract.

The contract needs narrowing, because the per-backend switch in
VoiceCloningForModel ends in a permissive default: an unnarrowed entry would
have advertised reference-audio cloning on every GGUF chat model in the
gallery. Narrow on the declared TTS usecase rather than the model name. The
TTS checkpoints are the only llama-cpp models carrying known_usecases: [tts];
name matching would have to guess at third-party repacks, and "base", the
substring the neighbouring Qwen and vLLM cases key on, is a routine word in
text-model names. The check reads the declared bit directly instead of going
through HasUsecases, which falls through to GuessUsecases and would hand the
decision to a heuristic that never had a llama.cpp TTS model in mind.

DefaultUsecases stays [chat]: a bare GGUF served by llama.cpp is a chat model,
and both the gallery filter and the importer read that field.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(gallery): declare what nemotron-3-nano-omni actually accepts

The entry is backend: vllm-omni with known_usecases: [chat, completion], no
mmproj and no media marker, so it used to report vision only through the
blanket GuessUsecases fallthrough that the vision branch in this branch
removed. Nemotron 3 Nano Omni is a multimodal understanding model: image,
video and audio in, text out. Declaring that is what the sibling
vllm-omni-qwen3-omni-30b already does.

known_usecases gains vision only. FLAG_VIDEO is video GENERATION, an output
modality, and this model generates none; video and audio input belong in
known_input_modalities, which is where AudioInputSupported and
VideoInputSupported read them from.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(importers): import a Qwen3-TTS GGUF repo as TTS, not chat

The llama-cpp importer hardcodes known_usecases: [chat] and assigns any
mmproj-matching file as a vision projector, so ggml-org/Qwen3-TTS-12Hz-1.7B-
Base-GGUF imported as a chat model with vision. Both fields were wrong, and
the model was unreachable from /tts and from the Voice Library.

Filenames cannot fix this. A Qwen3-TTS repo has the exact shape of a vision
repo, one backbone GGUF plus one mmproj-*.gguf, so the projector's own header
is the only honest signal: mtmd writes clip.has_gen_audio_encoder for the
projectors it can drive as a speech pipeline and refuses to build one without
it. Probe the selected mmproj for that flag, reusing the range-fetch the MTP
detection already does, and declare tts when it is set. The mmproj assignment
then stops reading as vision on its own, since a declared-TTS model already
exempts its projector from vision detection.

The probe is best-effort like the MTP one: a network blip leaves the chat
default in place rather than failing the import.

Verified against the real artifacts on disk: the Qwen3-TTS projector reports
gen-audio, its backbone does not.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(llama-cpp): stop non-TTS models crashing on the new pin

Two regressions, both hit every ordinary llama-cpp model and neither was
caught locally because every test on this branch loaded a TTS model.

The first is a null dereference. server_slot::tts_ctx::reset() called
mtmd_helper_gen_audio_reset() unconditionally, but the gen-audio pipeline
is only allocated for models carrying a gen-audio mmproj, and upstream's
implementation reads ctx->pipeline before null-checking anything. Since
server_slot::reset() runs during slot initialization for every model, any
non-TTS model segfaulted the backend the moment it loaded. Guard the call
on the is_supported() predicate already defined beside it, and keep the
plain field resets unconditional.

The second is unrelated to TTS and came in with the pin bump.
PredictOptions.Penalty is a bare proto float, so a caller that names no
repetition penalty sends 0 rather than omitting the field. Since
9de0fcf2b, common_sampler_init() rejects a non-positive penalty_repeat
outright because it would divide logits by zero, turning every such
request into "Failed to initialize samplers". Treat 0 as unset and leave
llama.cpp's own neutral default in place.

Verified with the same suite CI runs, which is what caught both:
tests/e2e-backends passes 6 of 6 including the load and predict specs
that were red. Qwen3-TTS still synthesises on both paths, 24 kHz mono
16-bit WAV with exactly one RIFF header on the streamed output.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-08-10 10:18:47 +02:00
localai-org-maint-botandlocalai-org-maint-bot daa8d2adbd fix(gallery): identify invalid preload JSON (#11434) 
Wrap PRELOAD_MODELS decoding failures with the setting name and expected top-level shape so startup errors point directly to the invalid configuration. Document the required array format and cover scalar and empty-array inputs.

Assisted-by: Codex:gpt-5

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-10 00:18:01 +02:00
localai-org-maint-botandlocalai-org-maint-bot 06ff56e674 feat(pii): restore request-scoped pseudonyms (#11272) 
* feat(pii): restore request-scoped pseudonyms

Replace masked request values with unique per-request tokens when response restoration is enabled, then restore them across JSON and SSE write boundaries. Document the opt-in model setting and expose it in config metadata.\n\nAssisted-by: Codex:gpt-5

* fix(pii): wrap reversible redaction tokens

Use configurable token delimiters to avoid restoring ordinary model text that happens to match an internal identifier. Rename the option and document the confidentiality tradeoff.

Assisted-by: Codex:gpt-5

---------

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-08-09 22:37:13 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto a0f50b2af2 feat(vllm-cpp): serve MiniMax-H3 video+audio generation (#11424) 
* feat(vllm-cpp): serve MiniMax-H3 video+audio generation

vllm.cpp's C ABI grew a video slice (ABI v12): a second engine handle
loaded from the MiniMax-H3 checkpoint SET, one blocking generate, and a
composed ffmpeg argv the caller execs. This wires that into LocalAI's
existing /video endpoint, so `vllm-cpp` now serves both text and video
and a clip comes back as an MP4 with a real audio track rather than a
silent render.

The video engine is a separate handle rather than a mode of the text
one because H3 is not a model directory: the DiT, the text encoder and
two VAEs are separate artifacts, and vllm.cpp has the two loaders refuse
each other's checkpoints. `Load` takes the video branch when the config
declares any of the video options; `parameters.model` is the DiT and the
rest of the set is named in `options:`.

Three details are worth calling out because getting them wrong is
expensive:

- The partition is DECLARED, not detected. The community quantisations
  strip the release metadata and the FL2VA and Ref2VA DiTs are
  byte-structurally identical, so the engine refuses to generate until
  it is told which it has. Worse, a mismatch does not fail cleanly: a
  reference passed to an FL2VA DiT renders for hours and returns a
  coloured lattice over the frame. The backend refuses that combination
  up front instead.
- ffmpeg comes from the host. libvllm writes frames plus a WAV and
  composes the mux argv, then spawns nothing - that process boundary is
  upstream's decision. The backend execs it, the same arrangement
  vibevoice-cpp uses for transcoding, and ffmpeg also converts a
  start_image upload into the binary PPM at the exact output canvas the
  engine requires.
- It is slow. Roughly 176 s per denoise step at the default 1344x768
  canvas on a 20-SM device, so the 50-step default is a multi-hour job.
  Nothing on this path imposes a deadline.

The /video endpoint no longer forces 512x512 when the request omits the
geometry. Every video backend already supplies its own default for a
zero (512x512 for stablediffusion-ggml, 1280x720 for diffusers, 832x480
for longcat-video, 1344x768 for H3), so the hardcoded value only ever
overrode the model's trained canvas with one three of the four were
never trained at.

Moving the engine pin from ABI v10 to v16 also grows the text
vllm_model_params mirror by the v14 device field and the v16 KV-sizing
knobs. LocalAI sets none of them - 0 is the pre-v14 engine byte for byte
- but the struct SIZE is part of the layout contract, so leaving them
out would have vllm_engine_load read past the allocation.

Gallery: `minimax-h3-fl2va-q4` installs the Q4_K_M FL2VA set (~40 GB
across five weight files plus the two VAE configs that carry the latent
statistics).

Assisted-by: Claude:claude-opus-5 golangci-lint yamllint go-vet

* fix(vllm-cpp): unbreak the Darwin build at the new engine pin

src/capi/vllm_c.cpp opens one `extern "C" {` for the whole ABI surface,
so file-local helpers declared inside it inherit C linkage. The video
slice added one that returns std::string, which Apple Clang reports as
-Wreturn-type-c-linkage and vllm.cpp's target-local -Werror turns into a
build failure. GCC and upstream Clang do not diagnose it, so only the
metal-darwin-arm64 job saw it.

Suppress it the same way this Makefile already suppresses Apple Clang's
-Wgnu-folding-constant on the Metal build. The helper is never called
across the boundary so the warning describes no hazard here, but it is a
real upstream wart: the fix belongs in vllm.cpp, hoisting the helper
above the extern "C" block, and this flag should go when a pin carrying
that fix lands.

Assisted-by: Claude:claude-opus-5

* fix(vllm-cpp): patch the engine clone instead of the warning flag

The -Wno-return-type-c-linkage added in the previous commit does nothing.
vllm_cpp_set_warnings adds `-Wall -Wextra -Werror` as PRIVATE target
options, so they land after anything CMAKE_CXX_FLAGS contributes, and
-Wall re-enables the -Wreturn-type group that -Wreturn-type-c-linkage
belongs to. The darwin job failed again on the same line, which is the
evidence: a consumer cannot wave this off from outside the engine.

Position is the only fix, so carry it as a patch against the pinned SHA,
the way longcat-video patches its own upstream. It hoists the helper
above the `extern "C" {` that gives it C linkage; it is file-local and
never called across the boundary, so nothing else moves.

`git apply` is unguarded on purpose: a patch that stops applying must
fail the clone loudly, because the alternative is a pin that silently
ships without a fix it is documented to carry. The patch header names
what retires it - a pin carrying the fix upstream, where it belongs.

Verified by applying the patch with `git apply` to the exact blob at the
pinned SHA and diffing the result against the intended file.

Assisted-by: Claude:claude-opus-5

* chore(vllm-cpp): bump the engine pin to ABI v17 and drop the vendored OrEmpty patch

The OrEmpty linkage fix this backend carried as patches/0001-* landed upstream
(mudler/vllm.cpp#195, 7534da65), so the patch has done its job. It is deleted
rather than left in place: the Makefile applies patches/*.patch unguarded and
documents that "a patch that no longer applies must FAIL the clone", so keeping
it against fixed source would break the build the moment the pin moved. Bumping
the pin and deleting the patch therefore have to be the SAME change.

Pin f921062b -> 776c56f1 (current vllm.cpp main).

That range also carries the engine's ABI v17 (vllm_server_main: the OpenAI server
published on the public surface). registerLib compares the library's
vllm_abi_version against `abiVersion` for EXACT equality, so the constant moves
16 -> 17 in the same commit or every load fails with an ABI mismatch.

The bump is safe for the layout assertions in video_test.go: diffing include/vllm.h
across the two pins shows zero struct-field changes -- v17 adds one function
declaration, the version macro and a doc comment, nothing else -- so every
unsafe.Offsetof in the video params test still holds.

Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

* chore(vllm-cpp): re-pin to pick up the VLLM_CPP_SERVER=OFF link fix

The previous pin carried vllm.cpp's ABI v17 (vllm_server_main) but not the guard
that makes it link when the server is compiled out. This backend builds libvllm
with VLLM_CPP_SERVER off, so the darwin lane failed at the dylib link with
vllm::entrypoints::openai::VllmServerMain undefined.

Fixed upstream in mudler/vllm.cpp#202: the C entry point is now guarded, so the
symbol is still exported (ABI v17 stays resolvable for dlopen) while the
no-server arm reports the missing capability instead of dragging in a translation
unit that was never compiled.

Verified upstream in BOTH arms before re-pinning: SERVER=ON builds and runs, and
SERVER=OFF configures, links, produces libvllm.so, and `nm -D` shows
vllm_server_main exported next to vllm_video_generate and vllm_transcribe.

Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-08-09 22:34:51 +02:00
Matheus C. França f31c3bbf1b feat(i18n): add pt-BR translation (#11427) 
Adds a complete Brazilian Portuguese (pt-BR) translation for the
LocalAI WebUI across 14 namespaces with full key parity against the
English locale, including modelEditor.json. Registers pt-BR in
SUPPORTED_LANGUAGES with the code 'pt-BR', name 'Português (Brasil)'
and flag 'BR'. Brand/model/product names and technical identifiers are
kept untranslated, matching the existing locale conventions.

Assisted-by: opencode:deepseek-v4-flash-free python3

Signed-off-by: Matheus C. França <matheus-catarino@hotmail.com>
 2026-08-09 22:34:19 +02:00
Adiraandlocalai-org-maint-bot ab52813342 feat(modelartifacts): support bounded parallel Hugging Face file downloads (#11162) 
* feat(modelartifacts): support bounded parallel Hugging Face file downloads

Closes #11114.

Snapshot materialization fetched every file through the sequential
executor in DownloadFilesWithContext, so a repository split into many
shards spent most of its wall clock in per-file request latency rather
than moving bytes.

Add DownloadFilesWithConcurrency, an errgroup with SetLimit, and keep
DownloadFilesWithContext as a wrapper that passes a limit of 1. That
leaves the two non-artifact callers (core/gallery and the model config
loader) on exactly the path they had: tasks still run in slice order,
and the first failure still returns before any later task starts.

Only whole files run in parallel. A single file is never split, so the
.partial resume machinery and the per-file SHA check in
downloadTaskWithRetry are untouched.

Two details the parallel path forced:

- completedBytes becomes an atomic.Int64. Several AfterDownload hooks
  add to it while other files' progress callbacks read it; without this
  the race detector reports three races on the new specs.
- The caller's status callback is serialized. The sequential path gave
  it an implicit guarantee of never being entered twice at once, and it
  belongs to the caller, so the executor keeps that promise rather than
  pushing locking onto every caller. AfterDownload is deliberately not
  serialized -- it does the verify-and-promote work that parallelism
  exists to overlap.

Manifest order needed no work: each hook already writes its own
manifest.Files slot by snapshot index, so entries stay in snapshot
order whatever the completion order. A spec now pins that.

The default is 1, unchanged behaviour. A shared models volume is often
the bottleneck rather than the link, so raising it is a deployment
decision; --artifact-download-concurrency and
LOCALAI_ARTIFACT_DOWNLOAD_CONCURRENCY expose it on both `run` and
`models install`.

Not done here, per the issue: no chunk-level parallelism within a single
file, and no throughput measurements across concurrency 1/2/4/8 -- that
needs a representative sharded repo and a real link.

Assisted-by: Claude:claude-opus-5 go-test gofmt
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>

* feat(modelartifacts): expose download concurrency in settings

Follow-up to review feedback on #11162:

- The CLI flag and docs no longer describe the limit as Hugging Face
  specific. It applies to any artifact source, as @mudler pointed out.
- artifact_download_concurrency is now a persisted runtime setting and
  is editable from the WebUI, so it can be changed without a restart.

The manager's limit becomes an atomic.Int64 behind
SetDownloadConcurrency, because a live runtime setting can be updated
while a materialization is already in flight. Injected materializers
stay compatible through an optional setter interface, so a manager that
does not implement it is simply left alone.

Verified before taking this on: go build, go vet and go test -race all
pass for pkg/modelartifacts, pkg/downloader and core/config. The React
UI builds with vite, artifact_download_concurrency is present in the
built Settings chunk, and eslint reports the same 8 pre-existing
warnings on Settings.jsx as it does without the change.

Implementation contributed by localai-org-maint-bot on the review
thread; reviewed, verified and signed off by me.

Assisted-by: Codex:gpt-5
Assisted-by: Claude:claude-opus-5 go-test vite eslint
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>

---------

Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-08-07 18:00:45 +02:00