mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-13 06:45:26 -04:00
f0fa4a7b1fdd67e3bb0fa951a91af21b21820122
743
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 localai-org-maint-botandlocalai-org-maint-bot
|
893a45141c |
fix(realtime): accept GA WebRTC signaling (#11778)
OpenAI GA clients send multipart or raw SDP requests. They expect a bare SDP answer. LocalAI only accepted its legacy JSON envelope, so signaling failed before media setup. Keep the JSON contract for existing clients. Accept both GA request shapes and choose the matching response format. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
80e3240f2d |
feat(distributed): key scheduling rules by a model alias (#11771)
Node placement and replica rules could only name a model, so an operator who pinned "llama3" to the GPU tier had to rewrite the rule whenever a different model took over that job. An alias already gives a stable name for whichever model serves it, and a rule on that name makes it a deployment slot: repoint the alias and the placement follows. A rule keeps the name the operator chose. Reads resolve that name through the config loader to the model the rule governs, so the reconciler counts, schedules and trims replicas of the target, and the router finds an alias-keyed rule from the target it is already routing. An alias that resolves to nothing governs nothing loadable, so the reconciler skips it and the write paths refuse it. A replica is shared by every name that resolves to it, so only one rule can decide where it runs. The REST and MCP write paths reject a rule whose target another rule already governs. A pair that arrives some other way, such as a seed file or an alias repointed onto a model that already has a rule, resolves in favour of the rule named after the model itself and then the oldest, and the rest are listed as shadowed. The eviction guard is the exception: it matches rules to replicas in raw SQL inside a locking transaction and cannot resolve an alias. It reads a stored target that the reconciler refreshes each tick, and falls back to the rule's own name when that target is empty. Assisted-by: Claude:claude-opus-5 golangci-lint eslint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
29899cd1e0 |
fix(ui): size model fit against the cluster and move node labels into the selector (#11765)
* fix(ui): move node labels into the scheduling selector field The scheduling page kept a node-label browser open above the rules whether or not anyone was writing one, while the field that actually needs labels, the rule's node selector, was two bare text inputs with no hint of what the cluster reports. The browser is gone. The selector's key input now completes against the label keys the cluster uses, and the value input offers only the values that key takes. The roster already loads for the page, so the suggestions cost no request, and a roster that fails to load costs the admin the hints and nothing else. Suggestions stay suggestions: a key no node reports yet still commits as typed, which is how an admin writes a rule before labelling the nodes for it. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): size model fit against the cluster, not the frontend The models page asked the frontend how much memory a model may occupy. In distributed mode the frontend is usually a GPU-less pod while every model runs on a worker, so a fleet of GPU nodes was told it could only run the smallest CPU build. The variant picker's fits flag and its auto-selection came from the same place, as did the hardware recommendations. The registry now reports the largest single healthy backend node. The largest node, not the fleet total: a model loads into one node, so four 16GB workers are not a home for a 40GB model. An operator-set VRAM budget caps a node's contribution, because the scheduler refuses a load above that ceiling anyway, and a GPU node beats a CPU node holding more system RAM. GET /api/resources and GET /api/models carry this as an additional cluster object. Their aggregate and ram fields keep reporting the frontend's own hardware, which is what the resource monitor shows. Variant selection judges backends against the union of the capabilities present in the cluster, the way backend discovery already did. Every path degrades to the local host: no cluster object in single-node mode, and none when the registry cannot be read, so a hiccup narrows the answer back to single-node behaviour rather than marking the whole catalog too large. The verdicts now name the node they belong to, since a model fits somewhere or nowhere. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Plamen K. Kosseff
|
e58dabf75f |
feat(ui): add 'Focus mode' option in chat settings to persistently toggle the sidebar auto-collapse (#11750)
Assisted-by: Claude:claude-fable-5 Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com> |
||
|
lei_lei
|
f28e8b24e6 |
fix(ollama): accept :latest tag on model lookup (#11732)
/api/tags appends :latest to untagged names, but chat and the other model endpoints looked the tagged name up as-is and 404'd. Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
1dc3aeef87 |
fix(distributed): resolve config revisions through one entry point
A model's revision is published by administration and checked against on every inference request. Those were computed by separate code: the request path resolves through the loader, while each publisher hashed whatever ModelConfig it happened to hold. By then SetDefaults had folded in the GGUF guess and app-level options, so the published value was one no request would ever carry and the model became unroutable until the row was deleted by hand. Fixing the publishers one at a time did not hold. Three rounds each found another: the startup resync, then a saved edit and a toggle, then a rename and the peer-change path. ModelConfigLoader.RevisionFor is now the only way to obtain a revision, and the raw hash is unexported, so a caller outside this package cannot hash a config it holds. A publisher and a request agree by construction rather than by two implementations happening to match. The request path no longer falls back to hashing its merged config either: an unstamped config is routed without a revision rather than with a wrong one. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
df1a40f9c0 |
fix(distributed): hash the config as persisted, not as defaulted
The revision was computed after SetDefaults, which folds in things that are not persisted configuration: the GGUF guess, the hardware defaults, and app-level options such as threads. The GGUF guess is the damaging one. It parses the model file to fill in values like context size, and when that parse fails it falls back to a different default. Whether a multi-gigabyte file on network storage parses at a given moment is not a property of the configuration, so one unchanged YAML produced two different revisions depending on when it was read. The controller rejected every request carrying the other one, and the model stayed unroutable until the stored value happened to match again. This is why it never reproduced against a model directory with no weights in it: the guess is skipped there and both values agree. The app-level defaults are the same class of bug with a slower fuse: changing threads in the settings UI changed every model's revision and made every model unroutable. The revision is now stamped when the file is parsed, before any defaults are applied, so it is a function of the file alone. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
a8bc64cd09 |
fix(ci): bound Discord release summaries (#11695)
* fix(ci): bound Discord release summaries The release model can return more than Discord's 2,000-character message limit. Discord then rejects the entire release notification. Ask the model for a smaller response and truncate extracted content to 1,800 characters before the notification step. The smaller bound leaves room below Discord's hard limit when model output varies. Assisted-by: Codex:gpt-5 * fix(tests): implement node liveness stub NodeCommandSender now requires PingNode. The endpoint test stub must implement it before the package can compile. Assisted-by: Codex:gpt-5 [Codex] --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
f3fabe8c5c |
fix(distributed): order derived usecases deterministically
syncKnownUsecasesFromString rebuilds KnownUsecaseStrings by ranging GetAllModelConfigUsecases, which is a map. Go randomizes that order per call, and the field is part of the serialized config, so one unchanged YAML hashed to a different config revision on every load. A model that derives a single usecase hid the problem. One that derives several, such as a chat model with an mmproj, alternated between as many revisions as there are orderings. The router treats a revision it did not establish as a config change, so requests failed with "stale model config revision" until the stored value happened to match again. Sorting the list makes the revision a function of the file alone. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
04735cd1f6 |
fix(distributed): stamp config revision at load time
The request middleware merges the caller's prediction parameters into its copy of the model config. core/backend.ModelOptions then hashed that copy, so the revision identified the request body rather than the persisted configuration. EstablishModelConfigRevision stores the first revision it sees and requires an exact match afterwards. The first request after a restart therefore pinned the model to its own temperature, top_p and stop values, and every later request that sent different ones failed with "stale model config revision". No config edit was involved. The loader now stamps the revision when it materializes a config, before any request override reaches it, and ModelOptions reads that stamp. Model administration keeps hashing the same persisted config, so both paths agree on one revision per configuration. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
8f56e4e042 |
fix(vram): persist remote probe metadata (#11487)
* fix(vram): persist remote probe metadata The startup warmer repeated remote size and GGUF metadata probes after every restart because both caches lived only in memory. Store successful HTTP probes for 24 hours so frequent restarts reuse the prior results. Bound the cache, reject invalid records, and purge it when gallery data changes. Local model files continue to bypass persistence. Assisted-by: Codex:gpt-5 * fix(vram): check temporary file cleanup The lint gate rejects the unchecked cleanup call in the persistent cache writer. Assisted-by: Codex:gpt-5.6 [golangci-lint] * fix(vram): make persistent cache optional Remote metadata probes can transfer enough data that operators need control over disk reuse and startup warming. Gallery autoload now gates both behaviors, and the runtime setting applies changes immediately. Assisted-by: Codex:gpt-5 * fix(ui): expose gallery startup pre-warm The existing gallery autoload setting also gates the startup metadata warmer. Name both effects in Settings so operators can find the requested boot control. Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
82c191afad |
fix(distributed): keep model replicas config-consistent (#11664)
* docs: design configurable copy buffering Document the context-aware copy buffer option and its validation plan. Assisted-by: Codex:gpt-5 * docs: design durable distributed staging operations Assisted-by: Codex:gpt-5 * docs: design distributed model config revisions Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(config): add stable model revisions Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(worker): acknowledge exact model stops Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(nodes): track model config revisions Assisted-by: Codex:GPT-5 [apply_patch] * fix(distributed): retry quarantined model cleanup Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work. Assisted-by: Codex:gpt-5 * fix(distributed): bind loads to config revisions Assisted-by: Codex: GPT-5 [OpenAI Codex] * fix(modeladmin): apply config revisions consistently Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes. Assisted-by: Codex: GPT-5 [OpenAI Codex] * feat(distributed): expose model config revision state Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options. Assisted-by: Codex:GPT-5 [OpenAI Codex] * test(distributed): cover model revision convergence Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness. Assisted-by: Codex:gpt-5 * fix(distributed): pass config revision CI checks Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files. Assisted-by: Codex: GPT-5 [OpenAI Codex] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
9d92139de4 |
feat(ui): edit scheduling rules in place (#11667)
* docs(ui): design scheduling rule editing Document the approved in-place rule editing flow and scalable node-label reference for the scheduling view. Assisted-by: Codex:gpt-5 * feat(ui): improve scheduling rule management Add scalable node-label discovery and editable scheduling rules with responsive, accessible controls. Assisted-by: Codex:gpt-5 * chore(ui): ratchet inline style baseline Record the static inline style removed by the scheduling view enhancement. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
a0252ad6a1 |
fix(distributed): keep staging operations stable (#11663)
* docs: design configurable copy buffering Document the context-aware copy buffer option and its validation plan. Assisted-by: Codex:gpt-5 * docs: design durable distributed staging operations Assisted-by: Codex:gpt-5 * fix(distributed): merge durable staging operations Use active model load jobs as the durable operations baseline and overlay replica-local staging progress without duplication. Preserve tracker-only operations when the registry cannot be read. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
5429f569e0 |
fix(progress): stop status updates throttling downloads (#11661)
* feat(progress): aggregate and coalesce gallery downloads Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show rolling transfer speed Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): preserve legacy import byte labels Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Anupam Mediratta
|
ff6043b811 |
fix: upgrade react-router to 7.18.2, 8.3.0 (GHSA-qwww-vcr4-c8h2) (#11644)
* fix: GHSA-qwww-vcr4-c8h2 security vulnerability Automated dependency upgrade by OrbisAI Security Signed-off-by: anupamme <mediratta@gmail.com> * fix: upgrade react-router-dom to 7.18.2 to fully remediate GHSA-qwww-vcr4-c8h2 The prior fix pinned react-router@7.18.2 directly but left react-router-dom at ^7.18.1, which bun resolved to 7.18.1. That package bundles its own react-router@7.18.1 sub-dep, leaving the vulnerable version in bun.lock via the react-router-dom/react-router scoped resolution. Pinning react-router-dom to 7.18.2 and regenerating the lockfile removes all 7.18.1 resolutions. Assisted-by: Claude Code:claude-sonnet-4-6 Signed-off-by: Anupam Mediratta <mediratta@gmail.com> --------- Signed-off-by: anupamme <mediratta@gmail.com> Signed-off-by: Anupam Mediratta <mediratta@gmail.com> |
||
|
Anupam Mediratta
|
9ba4bbf9bb |
fix: upgrade ip-address to 10.3.1 (CVE-2026-69192) (#11632)
fix: CVE-2026-69192 security vulnerability Automated dependency upgrade by OrbisAI Security Signed-off-by: anupamme <mediratta@gmail.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
0ab632b6bd |
fix(auth): protect HTTP routes by default (#11602)
* fix(auth): default to protected HTTP routes Use a method-aware registry for the small anonymous bootstrap surface. Unknown routes now require credentials instead of inheriting fail-open path classification. Keep node self-service routes behind their registration-token middleware. Global auth no longer rejects valid worker credentials first. Assisted-by: Codex:gpt-5 * docs(auth): document public HTTP surface Assisted-by: Codex:gpt-5 * test(auth): align route coverage with default denial Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
2383726d6d |
Revert "chore(tests): Avoid network, sleep and more during tests" (#11601)
Revert "chore(tests): Avoid network, sleep and more during tests (#11050)"
This reverts commit
|
||
|
Richard Palethorpeandlocalai-org-maint-bot
|
cb3bf7af3f |
chore(tests): Avoid network, sleep and more during tests (#11050)
* test: make coverage failures observable Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: parallelize coverage without remote fixtures Assisted-by: Codex:gpt-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: add offline resource infrastructure Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: enforce offline resource replay Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: harden offline resource refresh Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: expose slow coverage waits Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: eliminate avoidable wall-clock waits Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: remove repeated fixture startup waits Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: fix offline resource CI portability Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: cache Go modules before offline tests Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: drop the static network lint in favour of real isolation The offline test suite already prevents tests from reaching the network twice over: run-test-linux-offline.sh puts the test process in a cgroup and REJECTs egress outside the private ranges, and HardenedTransport installs testnetwork.LocalGuard to refuse dials that resolve to a public address. Both fail the test with a precise error at the moment of the dial. test-network-lint.sh added neither. Its diff stage defaulted to a HEAD base, so on a clean checkout it compared the tree against itself and inspected nothing; the branch's own commits were never examined. It only produced output when an earlier job step dirtied the tree, and then it matched a bare https?:// against whatever changed. make react-ui runs npm install rather than npm ci, so CI rewrote core/http/react-ui/package-lock.json and the lint reported an npm registry URL as forbidden test network access: + "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz", Its fingerprint stage was self-defeating in a quieter way: hashing the whole tree's network-mechanism inventory meant every rebase onto a master that touched any _test.go needed a manual baseline bump, so the check mostly caught its own staleness. Remove the script, its make target and the two prerequisite edges, along with the test-network: fixture markers that existed only to suppress it. The isolation itself is untouched. Assisted-by: Claude:claude-opus-5 [go vet] Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: keep hidden files in the offline test bundle artifact Cherry-picked from |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
4cad809003 |
fix(ci): test stale chunks in split bundle (#11595)
The V8 coverage build inlines every dynamic import, so the stale chunk tests cannot intercept a page chunk. Run those tests against the normal code-split bundle and exclude them from the inlined coverage pass. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
b806b1fec3 |
fix(ui): reload once when a page chunk 404s
A deploy replaces the whole content-hashed asset set at once. A tab holding an older index.html, or one whose request lands on a replica that the rollout has not swapped yet, asks for a page chunk the server no longer has. The dynamic import rejects and React Router's default error boundary replaces the app with "Unexpected Application Error!" until someone reloads by hand. The router now reloads the page itself when a chunk fails to load. index.html is served no-cache, so the reload lands on a self-consistent asset set. A timestamp in sessionStorage bounds this to one reload per 10 seconds, so a chunk that is genuinely gone reaches the error boundary instead of looping forever. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
0761bd02c7 |
feat(chat): add end-to-end context compression (#11556)
* feat(config): add context compression policy Define the opt-in model configuration contract before the chat middleware consumes it. Document each policy field so later request handling does not invent a second schema.\n\nRefs #9534\n\nAssisted-by: Codex:gpt-5 * fix(config): register compression fields The model editor metadata gate rejects new config fields without descriptions and suitable controls. Register the compression policy so operators can edit its six fields safely. Assisted-by: Codex:gpt-5 [monitoring-prs] * feat(chat): compress long contexts Long conversations currently fail once they reach the model context window. The opt-in policy now summarizes complete older turns before primary inference and preserves the newest tool chains. Both OpenAI and MCP chat routes share the same transformation. Usage metadata and metrics expose each compression event. Refs #9534 Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Nicholas Ciechanowski
|
cf93c04878 |
fix: tts text wrap (#11576)
Signed-off-by: Nicholas Ciechanowski <nicholas@ciech.anow.ski> |
||
|
Richard Palethorpe
|
d10374f849 |
feat(router): make KNN a first-class classifier with a persisted, curated corpus (#10652)
* feat(router): make KNN a first-class classifier with a persisted, curated corpus
Add `classifier: knn` — similarity-weighted voting over labelled
example prompts. Unlike score/colbert it needs no classifier model:
label knowledge lives in a corpus seeded and curated through the
admin API, so routing decisions are deterministic, auditable, and
grounded in graded experience rather than a model's opinion.
Epistemic gate: corpus entries below knn.similarity_threshold cannot
vote; when none clears it the classifier activates no labels and the
router uses the fallback — a prompt unlike all labelled experience is
treated as undecidable, not guessed. Decisions record
nearest_similarity (also on fallback rows) so admins can see how far
the nearest labelled experience was; the Routing tab explains
out-of-corpus fallbacks and shows per-label corpus counts.
Persistence: one JSONL file per router under
<data path>/router-corpus (text, labels, vector, embedder
fingerprint). The file is the source of truth; the local-store index
is rebuilt from it at classifier build time and stays a pure
in-memory index. Entries recorded under a different embedding model
re-embed on load. Also corrects the docs' false claim that
local-store collections persist — the embedding cache never survived
restarts (and still doesn't); the corpus does.
Corpus input is API-only by design (entries may contain example user
content): POST /api/router/{name}/corpus seeds (labels validated
against declared policies, embedded server-side, indexed
immediately), GET .../corpus/stats inspects — label counts only,
entry texts are never returned by any surface — DELETE .../corpus
wipes. Admin-gated like the sibling router endpoints, and exposed as
MCP tools (seed_router_corpus / get_router_corpus_stats /
clear_router_corpus) in both the httpapi and inproc clients with
coverage-test route mappings.
Plumbing: VectorStore gains SearchK (top-K was hardcoded to 1);
local-store gets InsertBatch/Delete as optional fast paths;
RouterConfig gains a knn block (embedding_model, k,
similarity_threshold, vote_threshold, store_name) with meta-registry
fields; the classifier dropdown now offers knn and the
previously-missing colbert; embedding_cache is ignored (with a
warning) for knn — it IS an embedding-KNN lookup; the stale
/api/instructions intelligent-routing entry is rewritten (it
described a classifier that no longer exists); swagger regenerated.
Tests: KNN vote/gate specs with hand-computed vote shares, corpus
manager suite (restart reload without re-embedding, fingerprint
re-embed, dedupe, hostile store names), middleware specs (corpus
routing, gate fallback, config validation, cache-wrap refusal),
corpus endpoint specs pinning the texts-never-returned contract, MCP
catalog + route-mapping gates, and a Playwright spec for corpus
stats and the out-of-corpus decision detail.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): name consulted corpus neighbours in knn decisions
Every knn decision (decision log rows and the /api/router/decide
response) now carries neighbors: the K retrieved corpus entries by
descending similarity - including ones below the epistemic gate, which
is what makes fallback decisions diagnosable - each as {id, similarity,
labels}. The id is the entry's content hash (first 8 bytes of the
SHA-256 of its text, hex): stable across reseeds and re-embeds, and
text-free, so an external platform that seeded the corpus can recompute
text->id on its own copy and bucket decisions by corpus region (per-
region reliability accounting) without corpus text ever leaving the
server. A corrupt index payload surfaces as an id-less neighbour at a
real similarity instead of disappearing.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* refactor(router): deduplicate knn plumbing and cut corpus hot-path waste
Post-review cleanup of the knn-first-class-router branch; no behaviour
changes on the API surface.
Reuse/altitude:
- RouterKNNConfig.ResolvedStoreName is now the single source of the
router-corpus-<name> default (was hand-derived in four files).
- corpus.ResolveKNNRouter + corpus.Seed carry the shared model
resolution and seed validation; the REST endpoints and the assistant
MCP client are thin transport adapters over them, with sentinel
errors mapped to HTTP statuses at the echo boundary.
- middleware.NewClassifierDeps assembles the classifier dependency set
once for all five entry points (OpenAI, Anthropic, realtime, decide,
corpus) instead of five hand-copied literals.
- router.AllClassifiers feeds both the status endpoint and the
unknown-classifier error, ending the classifier-list drift.
- Per-classifier requirements moved out of validateRouterPolicies into
their buildClassifier arms; the knn arm owns its embedding_cache
opt-out instead of a name-check in the shared wrap tail.
- adminOnly replaces four inline copies of the admin gate in the
middleware routes.
- localVectorStore.Search delegates to SearchK (identical traces).
Efficiency:
- Manager.Add embeds outside the manager mutex and appends to the
JSONL file (O(new) instead of O(corpus) rewrite); a torn tail from a
crash mid-append is tolerated on read and repaired on next write.
- Stats memoises per store keyed on the file's stat fingerprint and no
longer takes the manager mutex, so the 5s status poll stops parsing
vector-laden JSONL and stops blocking behind seeds.
- KNN Classify decodes each neighbour payload once (was twice) and
builds refs and votes in a single pass with one fallback return.
- Corpus file writes fsync before rename/close.
- The corpus manager is built eagerly in newApplication (sync.Once
dropped); test helper dead branch removed.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): bind knn corpus vectors to an embedder fingerprint and fail closed on mismatch
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(mcp): align corpus tool prompts and the mutating-tool safety list
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(proto,backend): report embedding shape from the llama-cpp backend
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): Go-side pooling — mean/last/decayed_mean with half-life
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): accept chat messages[] and per-request pooling on /v1/embeddings
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(middleware): name the failing fields when post-merge validation 400s
An intermittent post-merge validation failure surfaced as an opaque 400
during integration (pooling scheme mismatch that no client had sent).
Log the model, the request's pooling override, and the merged config's
pooling fields at the failure point so the next occurrence identifies
whether the request or the stored config carried the bad value.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(embeddings): scheme override must not inherit the config's half-life
A model config defaulting to decayed_mean pooling carries
pooling_half_life_tokens; a request overriding the scheme to mean/last
without its own half-life inherited that value, and post-merge
validation rejected the pair the server itself had assembled. Zero the
inherited half-life when the overridden scheme is not decayed_mean; a
request that explicitly pairs a half-life with a non-decayed scheme
still 400s.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix embedding pooling validation and router bounds
Declare backend embedding layouts and reject incompatible pooling modes. Reset local-store dimensions after a full clear, validate KNN thresholds, and add real backend and store integration coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: run local-store integration tests
Build and install the local-store backend in the Linux test job, then run the existing store integration suite so new specs are discovered automatically.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
|
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
a7bce6a128 |
fix(audio): reject incompatible transform streams (#11565)
The transform WebSocket accepted any model and opened its frame-based RPC. Any-to-any models use a different stream contract, so liquid-audio failed with an unimplemented RPC after the handshake. Reject incompatible model use cases before loading the backend. Direct realtime-audio callers to the OpenAI Realtime API. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Richard Palethorpe
|
799cc9f211 |
feat: bound global admission and expose running backend traces (#11560)
feat: bound backend admission and expose running traces Add process-wide backend execution admission without blocking UI or administrative HTTP work. Represent backend operations while they are in flight, surface running traces with immediate log links, and tie streaming admission leases to the gRPC receive lifecycle. Assisted-by: OpenAI Codex: GPT-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
0aaff91ebd |
feat(ui): unify model and backend lifecycle (#11548)
* feat(ui): add installed model lifecycle Models now owns catalog exploration and installed runtime controls under one canonical route. URL-owned state keeps lifecycle context recoverable through links and browser history. Assisted-by: Codex:gpt-5 Playwright * feat(ui): add installed backend lifecycle Backends split discovery from backend-binary management. The canonical page now keeps both lifecycle views under one URL-backed shell while it preserves target-node placement. Assisted-by: Codex:gpt-5 Playwright * fix(ui): repair lifecycle state updates Installed models lost distributed refreshes and kept a deleted selection. Backend searches also stopped tracking URL changes, while batch upgrades stopped after their first error. Preserve background refreshes and finish each requested batch action. Drive catalog results from URL-backed state without losing full metadata. Assisted-by: Codex:gpt-5 [Playwright] * feat(ui): make resource pages canonical Replace Host navigation with canonical Models and Backends lifecycle routes, preserve legacy management URLs, and surface shared host capacity on the Operate overview. Assisted-by: Codex:gpt-5 [Playwright] * feat(ui): complete canonical resource lifecycle Finish the responsive list-to-detail behavior, remove the retired Host implementation, and keep Explore focused on discovery while Installed owns destructive actions. Update regression coverage, localization, documentation, and development binding for the canonical resource pages. Assisted-by: Codex:gpt-5 [Playwright] * docs(ui): record the UI design context Record the approved users, brand character, and design principles so future interface work uses the same product direction. Index the context from the repository's agent instructions. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
88edd7fc7f |
fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock (#11514)
* fix(advisorylock): set statement_timeout alongside lock_timeout
WithLockCtx already overrides a deployment-wide lock_timeout on its
dedicated connection so a blocking pg_advisory_lock() waits its turn
instead of failing with 55P03. statement_timeout aborts that exact same
statement independently, with SQLSTATE 57014, and was not overridden.
Production roles commonly carry statement_timeout=60s. Any guarded
section longer than that (a cold model load stages for tens of minutes)
therefore killed every concurrent waiter:
advisorylock: acquiring lock 9003261067483446873: ERROR: canceling
statement due to statement timeout (SQLSTATE 57014)
Derive it from the same context budget as lock_timeout, with a matching
RESET so the pooled connection is returned clean.
Assisted-by: Claude Opus 5 [claude-code]
* feat(distributed): add ModelLoadJob, the durable cold-load record
A cold load in distributed mode is a long-running background job, but it
was modelled as a synchronous side effect of an inference request: the
whole of it (backend install, multi-GB staging, checkpoint load) ran
inside the per-model advisory lock. Loading a 35.7 GB GGUF held that lock
for ~20 minutes, so every concurrent request for the same model blocked
on pg_advisory_lock and died at the role's 60s statement_timeout.
Introduce the row that lets the lock shrink to a decision. Exactly one
ModelLoadJob may be active per tracking key; that uniqueness — not the
lifetime of a lock — is what de-duplicates concurrent loaders across
replicas. ClaimLoadJob does its read-then-write under the advisory lock
and nothing else: no network, file or gRPC I/O inside the guarded
section, so a claim costs milliseconds no matter how long the resulting
load takes.
LastProgress is a heartbeat rather than a byte counter. A checkpoint load
legitimately moves zero bytes for many minutes, so a reaper keyed on byte
movement would reclaim a healthy job mid-load; byte progress stays the
concern of load_deadline.go. A job whose heartbeat stops for longer than
the orphan window is reclaimable, so a replica killed mid-load cannot
wedge a model permanently.
Failed jobs keep their row for a short grace so an immediately-following
request reports the real cause instead of silently starting a fresh load
of a model that just failed.
No caller yet — the router moves onto this in the next commit.
Assisted-by: Claude Opus 5 [claude-code]
* refactor(distributed): run cold loads as jobs, outside the advisory lock
Route wrapped the entire cold load — node selection, backend install,
multi-GB staging and the remote LoadModel — in the per-model advisory
lock. The lock's job is to de-duplicate concurrent loaders, a decision
that takes milliseconds; holding it for the tens of minutes the resulting
work takes is what turned a dedup mechanism into a cluster-wide outage
for that model.
Split it into a claim and a run. The claim is the only thing left inside
the lock. The run is a background job owned by the claiming replica and
bounded by the same progress-extended deadline as before; every other
request for that model — local or on another replica — attaches as a
waiter and is served the moment the model is ready, with no duplicate
load and no lock contention.
Waiters share one broadcast rather than an ordered queue: they all want
the identical outcome, so ordering them would add fairness machinery that
changes no result. The local channel wakes same-replica waiters instantly
and a 2s DB poll is the authority, because a waiter on another replica
has no channel to close. On wake a waiter re-runs the warm path rather
than trusting the signal — the model may have been evicted in between.
A waiter whose client disconnects returns immediately and the job keeps
running; it belongs to the job record, not to the request. A failure is
recorded on the row so every waiter reports the real cause, and the row
survives briefly so the next request does not read "no job" as "not
loading" and start a duplicate load of a model that just failed.
The runner heartbeats the row on a fixed interval whether or not bytes
are moving, which is what keeps a legitimately silent checkpoint load
from being reclaimed as an orphan. Phase (installing/staging/loading) and
placement ride to the heartbeat on the context, the same seam
load_deadline.go already uses, so single-host paths are untouched.
Non-distributed mode (no DB) keeps the inline load exactly as it was.
Assisted-by: Claude Opus 5 [claude-code]
* feat(distributed): bound the wait for a loading model and answer with progress
A request whose model is cold-loading now attaches to the running job and
is served the moment the model is ready. That wait has to be bounded: a
held HTTP request cannot survive real infrastructure, and an ingress or LB
idle timeout kills a twenty-minute request regardless of what LocalAI
does.
New LOCALAI_MODEL_LOAD_WAIT (default 60s) bounds the CALLER, never the
load — the job keeps running either way. On expiry the request gets 503
with Retry-After and a structured body naming the model, the node, the
phase, byte progress and an ETA. The `error` envelope keeps OpenAI
clients working; `loading` is additive so they ignore it.
The ETA comes from the job's own observed rate and is omitted rather than
guessed until enough bytes have moved for that rate to mean anything: a
confidently wrong ETA on a twenty-minute wait is worse than none.
Retry-After is that ETA when known, clamped to [5s, 300s], and the wait
budget otherwise.
LOCALAI_MODEL_LOAD_WAIT=0 waits unbounded, for deployments with no proxy
in front. Zero in the config struct still means "unset, use the default",
so the CLI records the operator's zero as ModelLoadWaitUnbounded rather
than losing the distinction.
The distributed branch of ModelLoader.loadModel wrapped the router's
error with %s, which flattened it to a string. Use %w: the typed error is
what the HTTP layer keys the 503 off.
Assisted-by: Claude Opus 5 [claude-code]
* feat(api): add GET /api/models/{id}/load-status
A client that receives 503 while a model stages onto a worker needs
somewhere to poll. This returns the same `loading` object the 503 carries
— phase, node, byte progress and ETA — or 404 when no load is running.
Read-only and observability-shaped, so it is deliberately neither
admin-gated nor feature-gated: it explains a 503 the caller just
received, and hiding that behind a per-modality feature would make the
explanation for a failed image request depend on chat permissions. It
also gets no MCP tool, since there is nothing here an admin would manage
conversationally.
Registered on the surfaces from .agents/api-endpoints-and-auth.md: the
swagger block (existing `models` tag, so /api/instructions needs no new
area), the endpoint discovery maps in RegisterLocalAIRoutes, regenerated
swagger, and the distributed-mode docs page. No FLAG_* usecase is
involved, so capabilities.js is unchanged.
Assisted-by: Claude Opus 5 [claude-code]
* feat(ui): show cold-load progress in Chat and retry when the model is ready
A chat request for a model that is still staging onto a worker now gets a
503 carrying live progress instead of an error. Render it: the composer
shows the phase (installing / staging / loading), the node, the percent
and the ETA, then polls load-status and re-sends the request the moment
the model is ready.
Reuses the staging progress idiom the page already had rather than
inventing a second one — the two sources are folded into one
loadProgress, with the load job winning because it is authoritative
across frontend replicas and knows the phase, where the staging operation
only knows about a byte transfer this replica happens to be performing.
Waiting is bounded (three send attempts, ~30 min of polling each), so a
load that never finishes still surfaces as an error rather than as a
spinner nobody questions. An aborted generation stops the polling too.
Assisted-by: Claude Opus 5 [claude-code]
* fix(distributed): check warm-path cleanup errors
The router moved legacy cleanup calls onto newly linted lines. Report
cleanup failures while preserving the fallback to a cold load.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
|
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
95653f221e |
fix(ui): keep agent import action visible (#11488)
* fix(ui): keep agent import action visible The header hid its full import label after the agent list became non-empty. Hide only the nested file input so users can import more agents. Assisted-by: Codex:gpt-5 * test(ui): match the agent import label The Agents page renders the action as Import. The test searched for Import Agent, so it failed before checking visibility. Mock the observables request to remove backend timing from the fixture. Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Richard Palethorpe
|
5c63969760 |
fix: Show MCP connection errors in the UI (#11495)
* fix(mcp): surface configured server failures Keep model-configured MCP servers visible when discovery or connection setup fails, propagate status through distributed discovery, and let the Chat UI show actionable errors while retrying unavailable servers. Add model-editor metadata for remote and stdio configuration and document the expected format, deployment networking boundary, and alternate MCP scopes. Assisted-by: Codex:gpt-5 Ordino golangci-lint Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(compose): match CUDA development image Configure the API image with the cublas, CUDA 13, auth-tagged build settings used by the local development Makefile invocation, including the 24-way Docker build. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> * revert: keep host build settings out of compose The CUDA development deployment is managed from ~/docker/localai, not the repository example Compose file. Restore the generic example and keep machine-specific build settings in the host deployment. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(docker): exclude local agent artifacts Keep Claude worktrees and locally installed verification tools out of the Docker build context. These host-only directories added roughly 1.9 GB to every root image build. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Dedy F. Setyawan
|
9fd7ea7e93 |
i18n(id): translate admin, media, and nav UI strings to Indonesian (#11493)
Signed-off-by: Dedy F. Setyawan <dedyfajars@gmail.com> |
||
|
Dimitris Karakasilis
|
3636fcbd38 |
fix(realtime): keep the ICE interface allow-list working with a fixed UDP port (#11466)
LOCALAI_WEBRTC_ICE_INTERFACES was silently ignored whenever LOCALAI_WEBRTC_UDP_PORT was set. Every interface was gathered regardless of the allow-list, so a browser was handed the docker0/veth addresses the setting exists to suppress, and the connection established on a good pair and then dropped when consent checks failed on the unreachable ones. Two things combine to cause it. A mux built over a wildcard socket makes pion derive its host candidates by enumerating interfaces itself, with a nil filter and loopback included. Independently, the muxed gathering path in pion/ice never consults SetInterfaceFilter, so setting it has no effect there either. Bind one socket per admitted interface address via NewMultiUDPMuxFromPort, which takes the filter, instead of one wildcard socket. All the sockets share the same port, so the firewall requirement is still a single rule. Networks are pinned to UDP4 to match the socket family this replaces. An allow-list that matches no address on the host now reports the misconfiguration rather than binding nothing and leaving signaling to succeed while no candidate is ever advertised. Two tests: one asserts an unmatched allow-list is an error, and one gathers against a real peer connection and asserts no address outside the allowed interface appears (skipped on single-interface hosts). Assisted-by: Claude:claude-opus-5 go vet gofmt Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
45cb3983ee |
fix(ui): unmerge the class strings that left buttons in browser chrome (#11462)
Eight header controls across seven pages had two or three elements' classes
collapsed into one string. The wrapper or the icon ended up wearing the
button classes, and the buttons themselves were left with no class at all,
so they rendered in the browser's own chrome. Reported on Agent Jobs; the
grep found the rest.
`fas` does not draw anything by itself: it sets
`font-family: "Font Awesome 6 Free"` and weight 900 on whatever carries it,
and the `fa-*` class supplies the glyph via ::before. So
<button className="btn btn-primary fas fa-plus">
renders its own label "New Task" in the icon font, and
<div className="hstack btn btn-primary btn-sm fas fa-edit btn-secondary fa-arrow-left">
<button>Edit</button>
<button>Back</button>
</div>
styles the flex wrapper as a button that is both primary and secondary,
points two glyphs at one ::before, and leaves both real buttons bare.
Fixed, all of them keeping the correct `<i>` child they already had:
- AgentJobs, AgentTaskDetails (x2), AgentCreate - icon classes off the
button.
- AgentTaskDetails, AgentJobDetails - wrapper back to plain `hstack`, and
the two buttons inside each get the variants the wrapper had been
holding. Back is secondary and leads, Edit/Cancel is the emphatic one
and trails, matching every other detail header.
- VoiceLibrary, VoiceProfileCreate - the title `<i>` had swallowed the
action link's classes, so "Create voice" and "Back to library" were
unstyled anchors. Back was also drawing a "+" because it had inherited
fa-plus while its own fa-arrow-left sat up in the title.
- P2P - a stray fa-circle-info on the title icon.
The ninth instance was ImportModel, where this class of bug was first
found. #11461 rewrote that file and landed first, so nothing is left to fix
there.
Guarded by e2e/class-hygiene.spec.js, which reads the source rather than
walking routes: several of these pages need agent or voice data before they
render a header, so a route walk would skip exactly the pages that had the
bug. It fails on an icon-font class outside an `<i>`/`<span>`, on two glyphs
or two button variants on one element, and on a layout wrapper that is also
a button. Font Awesome modifiers (fa-spin, fa-fw, sizes) are excluded, so
the `fa-spinner fa-spin` idiom stays legal.
e2e: 428 passed.
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
7a7fb00730 |
feat(ui): rebuild the import form on the restyled design language (#11461)
The import page took the new palette in #11305 but kept its old layout, so it stayed a 760px column with the primary action detached from the form it submits. Two of the problems were outright bugs. The Import button carried no className at all, so the page's single most important control fell through to the user-agent button: system chrome, wrong radius, no design-system focus ring. The YAML button carried `fas fa-save fa-upload`, which sets Font Awesome as the button's own font family (its label text inherits it) and points two glyph classes at one ::before. On the layout: `page--narrow` is documented for "forms / single-record edit views", and in Advanced mode this page held a URI field, a six-section format guide, ten modality chips, nine preference fields, a key-value repeater and a YAML editor at `calc(100vh - 400px)`. The width was the symptom; one column was the disease. - `page--medium` with a work column and a format reference beside it. The reference answers the only question a first-time admin has and used to sit behind a chevron, closed by default. Below 1024px it becomes a disclosure rather than disappearing. - The source field is the hero: monospace, because it holds something you paste, and it carries its own Import button. That removes the hidden aria-hidden submit button that existed only because the real action sat outside the form. - Simple and Advanced are gone. They were ~80% the same surface, and the overlap cost a mode switch, a localStorage key and a three-button Keep/Discard/Cancel dialog whose only job was protecting state that switching modes would hide. One form with a collapsible options panel hides nothing, so none of it is needed. What genuinely differs is the kind of input, which is now the two tabs: a source, or YAML. - The size/VRAM estimate reports under the field that produced it instead of as a banner above the page header, and an import in flight gets the progress, phase and byte counts the poller already returned and the old status card threw away. - ModalityChips resolves its labels through the same `modality.*` keys as the dropdown it filters. It hardcoded English shorthand, so one modality carried two names on one screen ("Speech" on the chip, "Speech recognition" on the group it scrolled to) and seven locales had neither. Its inline styles and its pill radius move onto the design system. - Three inline styles go, including both conditional-padding hacks; the only one left is the progress bar's runtime width. Baseline 538 -> 535. Docs updated in the same change: the WebUI section described a Simple and an Advanced mode and told the reader to "Toggle to Advanced Mode". e2e: 426 passed. The mode-switch suite is replaced by one covering the tabs and the disclosure, and a new layout suite pins the width, the styled primary action, the absence of an icon-font button, the reference column at both widths, and the estimate's position. Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] [Playwright] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
f7db51bdf5 |
feat(realtime): add shared WebRTC UDP port (#11436)
* feat(realtime): add shared WebRTC UDP port Allow realtime WebRTC peer connections to reuse one configurable UDP mux, and surface listener bind failures through signaling. Assisted-by: Codex:gpt-5 * test(realtime): keep UDP mux alive during bind check The returned SettingEngine owns the UDP listener. Retain it through the duplicate-bind assertion so macOS cannot finalize the listener early and make the exclusivity check spuriously pass. Assisted-by: Codex:gpt-5 [systematic-debugging] * test(realtime): use IPv4 for UDP mux checks Match the socket family used by the WebRTC UDP mux so macOS does not allocate an IPv6 probe that can coexist with the IPv4 listener.\n\nAssisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
dependabot[bot]
|
6cf2e97868 |
chore(deps): bump dompurify from 3.4.12 to 3.4.13 in /core/http/react-ui in the npm_and_yarn group across 1 directory (#11425)
chore(deps): bump dompurify Bumps the npm_and_yarn group with 1 update in the /core/http/react-ui directory: [dompurify](https://github.com/cure53/DOMPurify). Updates `dompurify` from 3.4.12 to 3.4.13 - [Release notes](https://github.com/cure53/DOMPurify/releases) - [Commits](https://github.com/cure53/DOMPurify/compare/3.4.12...3.4.13) --- updated-dependencies: - dependency-name: dompurify dependency-version: 3.4.13 dependency-type: direct:production dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
7b9167eaad |
feat(llama-cpp): serve Qwen3-TTS through the llama.cpp backend (#11392)
* fix(config): do not read a TTS speaker-encoder mmproj as vision support Qwen3-TTS on llama-cpp ships an mmproj holding the speaker encoder and code predictor. VisionSupported() treated any non-empty MMProj as proof of image input, so every such model would be advertised as vision-capable. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): add TTS request option parsing helper Validates text and speaker reference presence and strictly parses the top_k / top_p per-request params, in a header with no llama.cpp or gRPC dependencies so the standalone C++ unit test gate picks it up. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): range-check the TTS top_k and top_p request params Format validation alone let NaN, infinity and out-of-range values through. The consumer copies both values into the audio generation input unconditionally and only guards its separate sampler assignment with "> 0", a test NaN also fails, so a NaN reached llama.cpp with the guard never firing. top_k must now be >= 0 and top_p must fall within 0.0 to 1.0 inclusive, with the bound written as a negated in-range test so NaN is rejected rather than silently accepted. Also cover the two checks the suite could not previously kill: the whole-string check in the float parser and the int32 range check. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump pin to f9e832c10 and carry the TTS server task Picks up ggml-org/llama.cpp#26254 (Qwen3-TTS via mtmd) and #26536 (the short-input audio chunk fix). Adds 0002-add-server-task-type-tts.patch, the server-side half of the still-draft #26603, so TTS runs through the slot scheduler instead of racing it. Remove that patch when #26603 merges. The patch is rebased on top of the score patch: its tokenize-switch hunk collided with the SERVER_TASK_TYPE_SCORE case, and its lone SRV_WRN call passes no variadic argument, which the macro cannot expand. The score patch itself needed no refresh. Also fixes fallout from the bump in grpc-server.cpp: upstream dropped the per-slot n_ctx argument from server_schema::eval_llama_cmpl_schema. Only the schema branch loses it, since forks predating the server-schema split still expect the old argument list. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): implement the TTS and TTSStream RPCs Both were declared in backend.proto but unimplemented. They now submit a SERVER_TASK_TYPE_TTS task and drain the response reader, the same shape PredictStream uses. The streaming path emits a leading sample_rate message and then raw PCM, because ModelTTSStream builds the WAV header itself; the non-streaming path emits a complete WAV to the requested dst. The streamed samples are converted from the pipeline's float32 to signed 16-bit first. MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM hands back floats, while the header ModelTTSStream writes announces 16-bit samples, so shipping the floats verbatim would decode as noise. prepare.sh and CMakeLists.txt now stage tts_request_options.h alongside the other grpc-server helpers, and register its standalone test with ctest the way passthrough_options_test is registered. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): mask non-codec tokens for Qwen3-TTS generation The Qwen3-TTS gen-audio pipeline maps a sampled backbone token to a codebook row with an unchecked subtraction, in mtmd-helper-gen.cpp: inp.code0 = sampled - codec_0; For ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF the vocab is 155008 tokens, <|codec_0|> is 151936 and the codec codes end at 153983. The model's own tokenizer.ggml.suppress_tokens holds 1023 ids covering 153984..155007, every special above the codec range except <|codec_eos_token|> (154086) which stays reachable as the stop token. Nothing masks the text range 0..151935, so the backbone can sample a text token at any step, the subtraction goes negative, and ggml_compute_forward_get_rows aborts the whole backend process on GGML_ASSERT(i01 >= 0 && i01 < ne01). Complete the mask upstream started: bias every token below <|codec_0|> to -INFINITY for TTS tasks so only codec codes and the codec EOS remain reachable. The biases are appended to task.params.sampling.logit_bias, which common_sampler_init already merges with the model's suppress tokens into one llama_sampler_init_logit_bias, so no sampler is added to the chain. Measured cost is 0.082 ms per sampled token and 1.16 MB, set against a forward pass in the multi-millisecond range. It lands in launch_slot_with_task rather than in a route handler so that llama.cpp's own POST /tts and LocalAI's TTS/TTSStream RPCs are both covered, and <|codec_0|> is resolved from the vocab rather than hardcoded so a model without it is left alone. This is reproducible with upstream's own llama-tts and no LocalAI code loaded, aborting at frame 55 on Q4_K_M and frame 71 on Q8_0, so it is neither a quantization artifact nor an artifact of the gRPC adapter. Two further defects in the same draft pipeline still prevent end-to-end audio; they are independent of this one and are recorded in the task report for an upstream bug report. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump pin to 9de0fcf2b and drop the TTS codec mask Upstream fixed the Qwen3-TTS abort in ggml-org/llama.cpp c8e03ce81 ("mtmd/ggml: add ggml_build_forward_order", #26649), landed one hour after the previous pin. ggml_build_forward_expand marks a tensor and all its ancestors for compute, so using it as a pure ordering hint defeated ggml_build_forward_select and made GEN_WAV calls execute the GEN_CODE branch against a stale inp_code0, hitting the get_rows bound assert in ggml_compute_forward_get_rows. That single defect accounts for every abort seen on this model, so 0003-mask-non-codec-tokens-for-tts.patch is removed rather than rebased. The mask changed the observed behavior, but it was perturbing a graph ordering bug rather than fixing a sampling one: at the new pin the whole path works without it. Keeping it would have meant carrying a 152k-entry logit bias, and rebasing it on every pin bump, for no benefit. Verified at 9de0fcf2b with only 0001 and 0002 applied, which both apply clean with no fuzz and needed no rebase: non-streaming HTTP 200, 410924 bytes, 8.56 s RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit, mono 24000 Hz streaming HTTP 200, 560684 bytes, 11.68 s, exactly one RIFF at byte 0, same format, which also exercises the float32-to-s16 conversion at runtime for the first time Pristine unpatched llama-tts at the same pin now also completes, 130 frames to a valid WAV, where it aborted at frame 55 before. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): clear the TTS slot sequence between requests Only the first TTS request in a backend process succeeded. Every later one failed instantly, in about 0.13 s, with "TTS prompt processing failed" from step_prompt, regardless of streaming or non-streaming and regardless of the text. With LOCALAI_SINGLE_ACTIVE_BACKEND=true the process is kept alive between requests, so a deployment would have served exactly one utterance per backend start. The cause is missing KV hygiene, not anything in the gRPC adapter. TTS slots never enter the shared batch: pre_decode() returns early for them and process_tts_slots() drives them instead, so they skip the prompt-cache bookkeeping that clears a slot's sequence between requests. Nothing in the gen-audio path makes up for it: mtmd_helper_gen_audio_reset only clears host-side buffers, and the pipeline always decodes from position 0 into the sequence identified by slot.id. So the second task on a slot writes positions 0..N over the first task's tokens and llama_decode fails. Fix is one call to slot.prompt_clear(), the same helper the normal path uses, in the SERVER_TASK_TYPE_TTS branch of launch_slot_with_task before set_input. It goes into 0002 rather than a new patch file because it is a defect in the code that patch introduces, and the header now records it as ours so we know whether it still needs carrying if #26603 merges without it. Verified in one backend process, different text on every request: three consecutive non-streaming requests, three consecutive streaming requests, and an interleaved non-streaming, streaming, non-streaming, streaming run. All ten returned HTTP 200 with RIFF ... WAVE audio, Microsoft PCM, 16 bit, mono 24000 Hz, the streamed ones carrying exactly one RIFF header at byte 0, and every output measured as real speech rather than silence or a truncated fragment. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): expose max_frames for TTS requests The Qwen3-TTS backbone does not always emit <|codec_eos_token|>, and when it does not, generation runs to upstream's 512-frame n_predict default. At the model's 12.5 Hz frame rate that is 40.96 s of audio, which a short input can trigger: one request in this session produced 40.96 s for a ten-word sentence. prepareTTSTask hardcoded n_predict to -1, so callers had no way to bound it. Add a max_frames key alongside top_k and top_p, parsed with the same strict whole-string parsing so a typo is an error rather than a silently truncated value, and rejected with a field-naming message when negative. 0 keeps the existing sentinel convention and means unset, so a request that omits it behaves exactly as before. Named max_frames rather than n_predict because frames are what the parameter means at a TTS endpoint: one frame is 0.08 s of audio. The 512-frame default is deliberately unchanged. Lowering it would truncate legitimately long inputs, which is a worse failure than an occasionally overlong one. Verified end to end on one text of thirty words: max_frames=25 HTTP 200, 96044 bytes, 2.00 s, exactly 25 frames max_frames=50 HTTP 200, 192044 bytes, 4.00 s, exactly 50 frames no max_frames HTTP 200, 572204 bytes, 11.92 s, stopped at its own codec EOS after 149 frames, unchanged behavior max_frames=-1 InvalidArgument "max_frames must be >= 0, got \"-1\"" max_frames=many InvalidArgument "max_frames must be an integer, got \"many\"" Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): send the TTS sample rate up front, and tidy three review items Four items from the Task 4 review. Streaming first-byte latency. TTSStream sent the sample-rate reply only once the first audio result arrived, and a chunk needs a whole 72-frame window, roughly 5.8 s of audio and far longer in wall time on CPU. The Go side blocks on that reply before it can emit the WAV header, so a streaming client sat at zero bytes for the whole stretch. The rate is a property of the loaded model and is available synchronously from mtmd_gen_audio_get_info, so it now goes out immediately after post_task and the rate_sent bookkeeping is gone. Measured on a warm model, first byte drops from 30.48 s to 0.014 s, and the output is still a valid WAV with exactly one RIFF header at byte 0. Unchecked close. The non-streaming path ignored ofstream::close(), so a failure that only surfaces on flush was reported as success while leaving a truncated file at dst. It now returns INTERNAL like the other write failures. Wrong comment on set_lang. gen_audio::inp::get() already maps a stored blank to nullptr, so our guard is behavior-preserving, not behavior-fixing. The comment claimed otherwise; the code was right. Repetition penalty. penalty_last_n = -1 is inert at this pin, because llama_sampler_init_penalties clamps it with std::max(penalty_last_n, 0) and then builds a disabled sampler, so the 1.05 penalty never applies. Upstream's README attributes looping to a missing repeat_penalty, so it was worth testing as a root-cause fix for the model running to the frame cap. Dropping the line lets the sampling default of 64 apply, which was confirmed in the sampler chain trace as penalty_last_n = 64 with repeat_penalty = 1.050. Over 15 uncapped short requests each way it did not help: 0 of 15 ran to the cap with the penalty inert, 1 of 15 with it active. Both lines are therefore kept for parity with upstream's draft, and a comment now records that the pair is inert and why, so the next reader does not believe a penalty is applied. max_frames remains the way to bound output. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(llama-cpp): let unpatched forks opt out of the TTS task turboquant and bonsai copy grpc-server.cpp into llama.cpp forks that do not carry our patches. disable-tts-task.sh injects the same kind of preprocessor switch disable-score-task.sh already uses, so those builds answer UNIMPLEMENTED rather than failing to compile. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): keep a TTS speaker-encoder projector out of vision detection Task 1 exempted a declared-TTS model's mmproj from VisionSupported, but the first real gallery entry with an mmproj still came back vision-capable through two paths the earlier fix did not close. GuessUsecases has no FLAG_VISION branch, so it falls through to true for any chat-ish model. That is not just a wrong answer at the call site: syncKnownUsecasesFromString rewrites KnownUsecaseStrings from HasUsecases, and the loader calls it more than once per config file, so the guessed FLAG_VISION is written out and parsed back into KnownUsecases as if the operator had declared it. Give GuessUsecases a FLAG_VISION branch that defers to the same explicit signals VisionSupported uses. Second, llama.cpp builds an mtmd context for the speaker-encoder projector and reports its media marker on the first chat probe, which resurrected vision after the model had been used once. Apply the same declared-TTS exemption to MediaMarker that the mmproj check already had. Verified against the qwen3-tts-llamacpp-q4 gallery entry: no vision capability and no image input modality, before load, after a TTS request, and after a chat probe. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add Qwen3-TTS entries for the llama-cpp backend Two entries over upstream's own GGUF conversion, Q8_0 and Q4_K_M, each pairing a backbone with the Q8_0 projector. Named to sit alongside the existing qwen3-tts-cpp entries rather than replace them. Also tags the llama-cpp backend text-to-speech / TTS so the backend browser surfaces the capability. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: cover Qwen3-TTS on the llama-cpp backend Adds the gallery variants, the two-file mmproj configuration, the required voice reference, and the language and sampling knobs. Also corrects the streaming-support list, which named only voxcpm. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): register llama-cpp as a TTS and voice-cloning backend The branch taught the llama-cpp backend to serve Qwen3-TTS and shipped two gallery entries for it, but never told the capability table. llama-cpp still declared only the text RPCs and usecases, so: - VoiceCloningForModel returned nil at the capability check, before it ever reached the model's own tts.voice_cloning override, and /tts answered 400 "selected model does not support reference-audio voice cloning" for any localai://voice-profiles/... voice. No model YAML could opt back in. - GET /api/backends/usecases did not list tts for llama-cpp, so the gallery greyed out the TTS filter for the entries this branch adds. - The React TTS page saw voice_cloning: null and kept both models out of the Voice Library. Add the TTS RPCs and usecase, and the reference-audio contract. The contract needs narrowing, because the per-backend switch in VoiceCloningForModel ends in a permissive default: an unnarrowed entry would have advertised reference-audio cloning on every GGUF chat model in the gallery. Narrow on the declared TTS usecase rather than the model name. The TTS checkpoints are the only llama-cpp models carrying known_usecases: [tts]; name matching would have to guess at third-party repacks, and "base", the substring the neighbouring Qwen and vLLM cases key on, is a routine word in text-model names. The check reads the declared bit directly instead of going through HasUsecases, which falls through to GuessUsecases and would hand the decision to a heuristic that never had a llama.cpp TTS model in mind. DefaultUsecases stays [chat]: a bare GGUF served by llama.cpp is a chat model, and both the gallery filter and the importer read that field. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): declare what nemotron-3-nano-omni actually accepts The entry is backend: vllm-omni with known_usecases: [chat, completion], no mmproj and no media marker, so it used to report vision only through the blanket GuessUsecases fallthrough that the vision branch in this branch removed. Nemotron 3 Nano Omni is a multimodal understanding model: image, video and audio in, text out. Declaring that is what the sibling vllm-omni-qwen3-omni-30b already does. known_usecases gains vision only. FLAG_VIDEO is video GENERATION, an output modality, and this model generates none; video and audio input belong in known_input_modalities, which is where AudioInputSupported and VideoInputSupported read them from. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(importers): import a Qwen3-TTS GGUF repo as TTS, not chat The llama-cpp importer hardcodes known_usecases: [chat] and assigns any mmproj-matching file as a vision projector, so ggml-org/Qwen3-TTS-12Hz-1.7B- Base-GGUF imported as a chat model with vision. Both fields were wrong, and the model was unreachable from /tts and from the Voice Library. Filenames cannot fix this. A Qwen3-TTS repo has the exact shape of a vision repo, one backbone GGUF plus one mmproj-*.gguf, so the projector's own header is the only honest signal: mtmd writes clip.has_gen_audio_encoder for the projectors it can drive as a speech pipeline and refuses to build one without it. Probe the selected mmproj for that flag, reusing the range-fetch the MTP detection already does, and declare tts when it is set. The mmproj assignment then stops reading as vision on its own, since a declared-TTS model already exempts its projector from vision detection. The probe is best-effort like the MTP one: a network blip leaves the chat default in place rather than failing the import. Verified against the real artifacts on disk: the Qwen3-TTS projector reports gen-audio, its backbone does not. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): stop non-TTS models crashing on the new pin Two regressions, both hit every ordinary llama-cpp model and neither was caught locally because every test on this branch loaded a TTS model. The first is a null dereference. server_slot::tts_ctx::reset() called mtmd_helper_gen_audio_reset() unconditionally, but the gen-audio pipeline is only allocated for models carrying a gen-audio mmproj, and upstream's implementation reads ctx->pipeline before null-checking anything. Since server_slot::reset() runs during slot initialization for every model, any non-TTS model segfaulted the backend the moment it loaded. Guard the call on the is_supported() predicate already defined beside it, and keep the plain field resets unconditional. The second is unrelated to TTS and came in with the pin bump. PredictOptions.Penalty is a bare proto float, so a caller that names no repetition penalty sends 0 rather than omitting the field. Since 9de0fcf2b, common_sampler_init() rejects a non-positive penalty_repeat outright because it would divide logits by zero, turning every such request into "Failed to initialize samplers". Treat 0 as unset and leave llama.cpp's own neutral default in place. Verified with the same suite CI runs, which is what caught both: tests/e2e-backends passes 6 of 6 including the load and predict specs that were red. Qwen3-TTS still synthesises on both paths, 24 kHz mono 16-bit WAV with exactly one RIFF header on the streamed output. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
a0f50b2af2 |
feat(vllm-cpp): serve MiniMax-H3 video+audio generation (#11424)
* feat(vllm-cpp): serve MiniMax-H3 video+audio generation
vllm.cpp's C ABI grew a video slice (ABI v12): a second engine handle
loaded from the MiniMax-H3 checkpoint SET, one blocking generate, and a
composed ffmpeg argv the caller execs. This wires that into LocalAI's
existing /video endpoint, so `vllm-cpp` now serves both text and video
and a clip comes back as an MP4 with a real audio track rather than a
silent render.
The video engine is a separate handle rather than a mode of the text
one because H3 is not a model directory: the DiT, the text encoder and
two VAEs are separate artifacts, and vllm.cpp has the two loaders refuse
each other's checkpoints. `Load` takes the video branch when the config
declares any of the video options; `parameters.model` is the DiT and the
rest of the set is named in `options:`.
Three details are worth calling out because getting them wrong is
expensive:
- The partition is DECLARED, not detected. The community quantisations
strip the release metadata and the FL2VA and Ref2VA DiTs are
byte-structurally identical, so the engine refuses to generate until
it is told which it has. Worse, a mismatch does not fail cleanly: a
reference passed to an FL2VA DiT renders for hours and returns a
coloured lattice over the frame. The backend refuses that combination
up front instead.
- ffmpeg comes from the host. libvllm writes frames plus a WAV and
composes the mux argv, then spawns nothing - that process boundary is
upstream's decision. The backend execs it, the same arrangement
vibevoice-cpp uses for transcoding, and ffmpeg also converts a
start_image upload into the binary PPM at the exact output canvas the
engine requires.
- It is slow. Roughly 176 s per denoise step at the default 1344x768
canvas on a 20-SM device, so the 50-step default is a multi-hour job.
Nothing on this path imposes a deadline.
The /video endpoint no longer forces 512x512 when the request omits the
geometry. Every video backend already supplies its own default for a
zero (512x512 for stablediffusion-ggml, 1280x720 for diffusers, 832x480
for longcat-video, 1344x768 for H3), so the hardcoded value only ever
overrode the model's trained canvas with one three of the four were
never trained at.
Moving the engine pin from ABI v10 to v16 also grows the text
vllm_model_params mirror by the v14 device field and the v16 KV-sizing
knobs. LocalAI sets none of them - 0 is the pre-v14 engine byte for byte
- but the struct SIZE is part of the layout contract, so leaving them
out would have vllm_engine_load read past the allocation.
Gallery: `minimax-h3-fl2va-q4` installs the Q4_K_M FL2VA set (~40 GB
across five weight files plus the two VAE configs that carry the latent
statistics).
Assisted-by: Claude:claude-opus-5 golangci-lint yamllint go-vet
* fix(vllm-cpp): unbreak the Darwin build at the new engine pin
src/capi/vllm_c.cpp opens one `extern "C" {` for the whole ABI surface,
so file-local helpers declared inside it inherit C linkage. The video
slice added one that returns std::string, which Apple Clang reports as
-Wreturn-type-c-linkage and vllm.cpp's target-local -Werror turns into a
build failure. GCC and upstream Clang do not diagnose it, so only the
metal-darwin-arm64 job saw it.
Suppress it the same way this Makefile already suppresses Apple Clang's
-Wgnu-folding-constant on the Metal build. The helper is never called
across the boundary so the warning describes no hazard here, but it is a
real upstream wart: the fix belongs in vllm.cpp, hoisting the helper
above the extern "C" block, and this flag should go when a pin carrying
that fix lands.
Assisted-by: Claude:claude-opus-5
* fix(vllm-cpp): patch the engine clone instead of the warning flag
The -Wno-return-type-c-linkage added in the previous commit does nothing.
vllm_cpp_set_warnings adds `-Wall -Wextra -Werror` as PRIVATE target
options, so they land after anything CMAKE_CXX_FLAGS contributes, and
-Wall re-enables the -Wreturn-type group that -Wreturn-type-c-linkage
belongs to. The darwin job failed again on the same line, which is the
evidence: a consumer cannot wave this off from outside the engine.
Position is the only fix, so carry it as a patch against the pinned SHA,
the way longcat-video patches its own upstream. It hoists the helper
above the `extern "C" {` that gives it C linkage; it is file-local and
never called across the boundary, so nothing else moves.
`git apply` is unguarded on purpose: a patch that stops applying must
fail the clone loudly, because the alternative is a pin that silently
ships without a fix it is documented to carry. The patch header names
what retires it - a pin carrying the fix upstream, where it belongs.
Verified by applying the patch with `git apply` to the exact blob at the
pinned SHA and diffing the result against the intended file.
Assisted-by: Claude:claude-opus-5
* chore(vllm-cpp): bump the engine pin to ABI v17 and drop the vendored OrEmpty patch
The OrEmpty linkage fix this backend carried as patches/0001-* landed upstream
(mudler/vllm.cpp#195, 7534da65), so the patch has done its job. It is deleted
rather than left in place: the Makefile applies patches/*.patch unguarded and
documents that "a patch that no longer applies must FAIL the clone", so keeping
it against fixed source would break the build the moment the pin moved. Bumping
the pin and deleting the patch therefore have to be the SAME change.
Pin f921062b -> 776c56f1 (current vllm.cpp main).
That range also carries the engine's ABI v17 (vllm_server_main: the OpenAI server
published on the public surface). registerLib compares the library's
vllm_abi_version against `abiVersion` for EXACT equality, so the constant moves
16 -> 17 in the same commit or every load fails with an ABI mismatch.
The bump is safe for the layout assertions in video_test.go: diffing include/vllm.h
across the two pins shows zero struct-field changes -- v17 adds one function
declaration, the version macro and a doc comment, nothing else -- so every
unsafe.Offsetof in the video params test still holds.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
* chore(vllm-cpp): re-pin to pick up the VLLM_CPP_SERVER=OFF link fix
The previous pin carried vllm.cpp's ABI v17 (vllm_server_main) but not the guard
that makes it link when the server is compiled out. This backend builds libvllm
with VLLM_CPP_SERVER off, so the darwin lane failed at the dylib link with
vllm::entrypoints::openai::VllmServerMain undefined.
Fixed upstream in mudler/vllm.cpp#202: the C entry point is now guarded, so the
symbol is still exported (ABI v17 stays resolvable for dlopen) while the
no-server arm reports the missing capability instead of dragging in a translation
unit that was never compiled.
Verified upstream in BOTH arms before re-pinning: SERVER=ON builds and runs, and
SERVER=OFF configures, links, produces libvllm.so, and `nm -D` shows
vllm_server_main exported next to vllm_video_generate and vllm_transcribe.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Matheus C. França
|
f31c3bbf1b |
feat(i18n): add pt-BR translation (#11427)
Adds a complete Brazilian Portuguese (pt-BR) translation for the LocalAI WebUI across 14 namespaces with full key parity against the English locale, including modelEditor.json. Registers pt-BR in SUPPORTED_LANGUAGES with the code 'pt-BR', name 'Português (Brasil)' and flag 'BR'. Brand/model/product names and technical identifiers are kept untranslated, matching the existing locale conventions. Assisted-by: opencode:deepseek-v4-flash-free python3 Signed-off-by: Matheus C. França <matheus-catarino@hotmail.com> |
||
|
Adiraandlocalai-org-maint-bot
|
ab52813342 |
feat(modelartifacts): support bounded parallel Hugging Face file downloads (#11162)
* feat(modelartifacts): support bounded parallel Hugging Face file downloads Closes #11114. Snapshot materialization fetched every file through the sequential executor in DownloadFilesWithContext, so a repository split into many shards spent most of its wall clock in per-file request latency rather than moving bytes. Add DownloadFilesWithConcurrency, an errgroup with SetLimit, and keep DownloadFilesWithContext as a wrapper that passes a limit of 1. That leaves the two non-artifact callers (core/gallery and the model config loader) on exactly the path they had: tasks still run in slice order, and the first failure still returns before any later task starts. Only whole files run in parallel. A single file is never split, so the .partial resume machinery and the per-file SHA check in downloadTaskWithRetry are untouched. Two details the parallel path forced: - completedBytes becomes an atomic.Int64. Several AfterDownload hooks add to it while other files' progress callbacks read it; without this the race detector reports three races on the new specs. - The caller's status callback is serialized. The sequential path gave it an implicit guarantee of never being entered twice at once, and it belongs to the caller, so the executor keeps that promise rather than pushing locking onto every caller. AfterDownload is deliberately not serialized -- it does the verify-and-promote work that parallelism exists to overlap. Manifest order needed no work: each hook already writes its own manifest.Files slot by snapshot index, so entries stay in snapshot order whatever the completion order. A spec now pins that. The default is 1, unchanged behaviour. A shared models volume is often the bottleneck rather than the link, so raising it is a deployment decision; --artifact-download-concurrency and LOCALAI_ARTIFACT_DOWNLOAD_CONCURRENCY expose it on both `run` and `models install`. Not done here, per the issue: no chunk-level parallelism within a single file, and no throughput measurements across concurrency 1/2/4/8 -- that needs a representative sharded repo and a real link. Assisted-by: Claude:claude-opus-5 go-test gofmt Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> * feat(modelartifacts): expose download concurrency in settings Follow-up to review feedback on #11162: - The CLI flag and docs no longer describe the limit as Hugging Face specific. It applies to any artifact source, as @mudler pointed out. - artifact_download_concurrency is now a persisted runtime setting and is editable from the WebUI, so it can be changed without a restart. The manager's limit becomes an atomic.Int64 behind SetDownloadConcurrency, because a live runtime setting can be updated while a materialization is already in flight. Injected materializers stay compatible through an optional setter interface, so a manager that does not implement it is simply left alone. Verified before taking this on: go build, go vet and go test -race all pass for pkg/modelartifacts, pkg/downloader and core/config. The React UI builds with vite, artifact_download_concurrency is present in the built Settings chunk, and eslint reports the same 8 pre-existing warnings on Settings.jsx as it does without the change. Implementation contributed by localai-org-maint-bot on the review thread; reviewed, verified and signed off by me. Assisted-by: Codex:gpt-5 Assisted-by: Claude:claude-opus-5 go-test vite eslint Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> --------- Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
9a6156d808 |
feat(nemo-speech-cpp): add the NVIDIA NeMo-Speech.cpp backend (#11406)
* feat(nemo-speech-cpp): scaffold the backend and upstream build
Adds the backend skeleton and the NeMo-Speech.cpp build, pinned at
2e12e2def8a98ed06666f7ee3ca94e7193e04be4. The Go side is deliberately a stub:
it dlopens the runtime and starts the gRPC server, later work fills in the
symbol table and the model logic.
Three details of the upstream layout differ from what the plan assumed, and the
build reflects the real tree:
* The TTS C ABI ships as libnemo_speech_tts, not libnemo_speech_tts_c. Upstream
compiles c_api.cpp straight into the implementation library and only aliases
the nemo_speech_tts_c CMake target, so no _c object exists on disk. ASR and
NMT do build a real _c shim.
* Shared objects land in build/bin, since upstream points
CMAKE_LIBRARY_OUTPUT_DIRECTORY at ${CMAKE_BINARY_DIR}/bin.
* The ASR and NMT _c shims carry a DT_NEEDED on libnemo_speech_asr and
libnemo_speech_nmt, so those are staged and packaged alongside them.
Otherwise dlopen fails at startup.
The ggml patch step uses an order-only prerequisite. cmake writes into the
checkout and bumps its mtime past the sentinel, which would otherwise re-run
git apply over an already-patched tree and break every incremental build.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): make 'build' produce the package and bundle the ITN stack
Addresses the review of the scaffold commit.
backend/Dockerfile.golang runs 'make -C backend/go/$(BACKEND) build' and then
copies package/ into the final image, so 'build' has to end with a populated
package/. It only staged shared objects, which would have shipped an image with
no binary and no libraries at all. The old staging recipe is now stage-libs and
the chain is stage-libs, nemo-speech-cpp-grpc, package, build, matching every
sibling Go backend.
Text normalization was packaged incorrectly. nemo_speech_text_normalization is
STATIC but links sparrowhawk, fstfar and fst PUBLIC, so they land as DT_NEEDED
on libnemo_speech_asr.so, and they live in a project-local prefix that nothing
else provides. WITH_NORM stays ON by default on Linux, since normalization is a
wanted feature. Instead stage_libs now copies .deps/itn/lib when WITH_NORM=ON,
and package.sh bundles it.
Staging that prefix is still not enough on its own: Sparrowhawk drags in
protobuf, re2 and absl, which neither build_itn_deps.sh nor
package-system-libs.sh provides. Rather than hard-code another hand-maintained
list, package.sh now walks the DT_NEEDED entries of everything staged and copies
whatever is unresolved, skipping the core set and the GPU set that the shared
scripts already own. It fails at package time, not at first dlopen, when
something cannot be resolved. On a WITH_NORM=OFF build the closure is already
complete and it copies nothing.
Restore CGO_ENABLED=0 on the Go build to match whisper, parakeet-cpp and
omnivoice-cpp. Note that purego reaches dlopen through fakecgo, so the binary is
dynamically linked either way; what the flag changes is the NEEDED set, and
lib/ld.so routing in run.sh exists precisely because the binary is not static.
Replace the hand-rolled .patched sentinel with upstream's
scripts/apply-ggml-patches.sh. It applies the series in filename order, exits
non-zero when a patch does not apply, and detects "already applied" by comparing
the full-series tree hash rather than an mtime, so it is safe to run every time
and there is no sentinel left to go stale or to wedge the build when deleted. It
is wired as an order-only prerequisite so running it does not force a relink.
Also: correct the package.sh header, which claimed three shared objects when
there are five and none of the TTS ones carry a _c suffix; give 'make test' the
LD_LIBRARY_PATH the dlopen tests will need; document that a NEMO_SPEECH_VERSION
bump needs 'make purge'; and extend 'clean' to remove package/ and the ITN
libraries.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): make the closure guard fail closed and give CI its toolchain
Addresses the second review round.
The dependency-closure guard failed open. Its glob expands once per pass, so
each pass advanced the closure by exactly one level, and the fixed count of five
passes then fell out of the loop without checking whether anything remained. An
eight-deep chain packaged six libraries, exited zero and reported success. That
is the case the guard was written for: asr to sparrowhawk to protobuf to absl
already runs several levels deep, so a WITH_NORM build could ship missing its
deepest libraries and fail at first dlopen. The loop now runs until the staged
set stops growing, and exhausting the bound is a hard error rather than a silent
exit.
For the same reason, a build image with neither readelf nor objdump no longer
warns and skips. It cannot show the package is complete, so it refuses to ship
it. The guard is entered only when there is something to check, so an empty
package cannot trip the new error.
Dockerfile.golang installed ninja-build only in the Vulkan branch while this
Makefile runs cmake -G Ninja unconditionally, so the CPU, cuBLAS and L4T images
could not configure at all. ninja-build moves to the common apt list; it does
not change CMake's default generator, so it is inert for the other backends.
gcc-12 was nowhere in the tree, yet WITH_NORM defaults ON and
build_itn_deps.sh needs it, so the committed default was unbuildable in CI.
Install it, with the protobuf, absl, re2 and autotools that Sparrowhawk and
OpenFST need, gated on BACKEND so the other Go images do not carry it. The list
follows upstream's own docker/Dockerfile, trimmed of the gRPC, portaudio and
python entries a BUILD_GRPC=OFF build does not use. Text normalization stays ON:
downgrading it silently would ship a backend advertising a feature it lacks.
Also: make test depend on stage-libs, so LD_LIBRARY_PATH is not an empty
directory on a clean tree, and add an engine target so Dockerfile.golang's
cacheable prebuild layer is not skipped and a CUDA build stops recompiling all
of upstream on every Go-side change.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): pin protoc for ITN and make the norm stack its own target
Addresses the third review round.
Dockerfile.golang installs protoc 27.1 into /usr/local/bin, ahead of /usr/bin,
while libprotobuf-dev is the distro's 3.21 on noble and 3.12 on jammy.
Sparrowhawk resolves protoc from PATH at make time (configure.ac uses
AC_CHECK_PROG, so PROTOC substitutes to the bare word, and src/proto/Makefile.am
invokes it) and commits no pregenerated stubs, so the rule always runs. Code
generated by 27.1 includes google/protobuf/runtime_version.h and a
PROTOBUF_VERSION guard the older headers lack, so the WITH_NORM build could not
complete. Pin PROTOC to the apt one for that step; configure documents that a
pre-set value wins. The apt protoc and libprotobuf-dev come from one source
package at one version, which is the property that makes this correct.
The text-normalization stack is now a target keyed on a file build_itn_deps.sh
actually produces, rather than a side effect of the runtime library rule. As a
side effect make could not see whether it existed, so once the library was up to
date the script could never run again: a tree built WITH_NORM=OFF could not move
to ON, and make test hard-failed with no escape but a full 345 MB clean. It is
now built on demand and reachable on its own as 'make itn'. Staging keys on the
prefix existing rather than on WITH_NORM, so it stages what the tree actually
built, and package.sh's closure guard remains the backstop.
An already-configured build tree also now wins over the platform default, so a
tree built WITH_NORM=OFF is not silently reconfigured to ON by a bare make test,
which is what demanded gcc-12 from developers who chose not to have it. An
explicit WITH_NORM= on the command line still overrides both, and the ITN rule
preflights for gcc-12 with an error that names the alternative.
Move ninja-build out of the shared apt layer into the existing BACKEND-gated
block. Dockerfile.golang serves 225 matrix entries and only this backend
configures with -G Ninja, so the common list is byte-identical to master again
and no other image loses its cache.
Drop libabsl-dev and correct the comment that justified it. No base image here
ships protobuf 25, so nothing needs the absl split, and the cmake glob looks in
/usr/lib rather than the multiarch directory Ubuntu actually uses, so the
package could never have contributed anything.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): move the backend apt gate below the expensive layers
Addresses the fourth review round.
The nemo-speech-cpp apt block sat immediately after the shared apt layer, above
the Vulkan SDK build, the CUDA and ROCm installs, the Go toolchain and the
protoc download. Docker keys each layer on its parent, so inserting a step there
re-keys everything below it: a byte-identical shared layer is not enough, and
merging as it stood would have forced all of those to re-execute once for every
Go backend image. Move it down beside the existing opus, crispasr and
sherpa-onnx gates, which sit after those layers for the same reason.
Checked the ordering both ways before moving. Nothing between the two positions
uses these packages: the Vulkan and opus blocks install their own ninja and
pkg-config, go install protoc-gen-go needs the Go toolchain rather than protoc,
and the protoc 27.1 step is a release-binary download that needs neither
protobuf-compiler nor libprotobuf-dev. Nothing in the block needs anything those
layers provide; it uses only apt, and the mirror rewrite from the first RUN
persists in the image. It also runs no update-alternatives, so the default
compiler stays untouched for later layers. The diff against master is now a
single additive hunk with no shared layer touched.
Also preflight ITN_PROTOC. configure gates a preset PROTOC on test -n alone, so
a path that does not exist is accepted and the error surfaces much later as a
bare "No such file or directory" from inside make -C src/proto. The pin
introduced that failure on a box whose only protoc is in /usr/local/bin, which
worked before. Check it alongside the gcc-12 check and name the ITN_PROTOC=
override in the message.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): parse model options
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): guard the empty option value and warn on a bad gpu index
An empty value like "vad_model:" must stay empty, since callers read the
empty string as "unset". That branch of resolve() had no spec: dropping the
guard left every spec green while parseOptions started returning the models
directory itself. Add the spec that fails without the guard.
A known key with an unparseable value is a typo, not a config from a newer
backend, and "gpu:banna" failed expensively: the model loaded, produced
correct output, and ran on CPU with no signal anywhere. Log it. Unknown keys
stay silently ignored, which is what keeps configs forward compatible.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): detect model family and discover TTS assets
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): bind the C ABI with layout assertions
purego binds by name at runtime and the config structs are passed by pointer,
so both a renamed symbol and a mismatched struct layout would otherwise survive
a green build. registerSymbols names the failing symbol, and the layout specs
compare each Go mirror against the size the library reports for itself, against
the offsets a C compiler produces for the installed headers, and against the
default values upstream writes into the structs it returns.
Two of the bindings differ from the plan because the headers do. The plan's
nemo_speech_diar_segments signature omits the segmentation-config pointer that
diar.h declares as the second parameter, which would have shifted the output
buffer, the capacity and the count pointer one position each. And
nemo_speech_diar_stream_push_f32 was missing from the symbol table although
standalone diarization cannot work without it.
Also close the two panic and equality gaps left in family.go: ValueString panics
on a mistyped general.architecture, and the self-codec guard compared a Cleaned
candidate path against an uncleaned one, so a doubled separator let the primary
GGUF be selected as its own codec.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(nemo-speech-cpp): run the ABI specs in CI and refuse to skip them
The layout assertions were inert. TEST_PATHS does not cover this backend and
the per-backend list in test-extra had no entry for it, so nothing invoked the
package's tests. Add it next to depth-anything-cpp, supertonic and vllm-cpp,
the group whose own test target carries its build prerequisites; stage-libs
already pulls the native build chain, so no prepare-test-extra entry is needed.
The skip guard was also loader-inconsistent: librariesPresent stats bare
filenames relative to the working directory while openLibraries resolves them
through the loader search path, so any invocation other than make test skipped
every library-backed spec and still reported green. NEMO_SPEECH_REQUIRE_LIBS=1
turns that into a failure naming the directory and the remedy, and the Makefile
test target sets it. Unset, the plain skip survives so a developer without a
build can still run the pure-Go layer specs.
Trim the default-value fingerprint from roughly forty assertions to eight. It
was pinning tunables such as threads and flush_partial_chunk, so a legitimate
pin bump would have failed with a message reading like a layout error. What
survives is only header-documented contract: the lone non-zero max_alternatives,
the run of -1 sentinels and the zero that witnesses where it stops. Verified the
narrowed spec still catches a mirror and offset table corrupted in lockstep,
which is the one class only this layer sees.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): select the family at load and gate RPCs on it
Load sniffs the GGUF architecture, maps it to a family and dispatches to
the family's loader. requireFamily gates every other RPC, returning
Unimplemented naming both the loaded and the wanted family so a
misconfigured model YAML produces a message a user can act on.
The family is committed only once its loader has succeeded. A load that
fails part way through would otherwise leave the gate open on a handle
that was never created.
cstr uses runtime.Pinner rather than an ordinary Go allocation. The
address crosses the ABI as a uintptr, which the collector does not
trace, so incidental reachability through the release closure is not a
guarantee: a caller discarding that closure could have the bytes
collected before the create call reads them. Pinning is the sanctioned
mechanism, makes the release function do real work, and turns a dropped
release into a loud leaked-Pinner panic instead of silent corruption.
Free overrides the base no-op to destroy the handle and reset the
family. Every family owns C memory only its own destroy entry point can
release, so without this an unloaded model leaks an acoustic model.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): pin the load ordering, close the engineMu race
Three review items, plus a defect the race detector turned up.
The spec covering "no family selected after a failed load" wrote junk to
a .gguf, so Load returned at ggufArchitecture before a family was ever
chosen and the assertion was vacuous. Generalised the GGUF test helper
to take a string architecture, and added a spec that loads a magpietts
GGUF with no sibling codec, so familyFor succeeds and discoverTTSAssets
then fails. It self-guards on ggufArchitecture so it cannot degrade back
into the earlier path.
requireFamily read n.fam unlocked while Free wrote it under engineMu,
which the race detector confirms is a real race. pkg/grpc/server.go
calls Free without the backend lock every other RPC holds, so teardown
can land mid-request. withEngine now takes the lock, checks the family
and runs the body under one acquisition; two would leave a window for
Free to destroy the handle between check and use. The locking protocol
is stated in both directions for the RPCs still to be written.
Running -race also enables checkptr, which aborts on cstr's pointer
being read back by goString: converting a uintptr to a pointer is fatal
whenever the address lands in a Go allocation, so a pinned Go buffer can
never be dereferenced from Go. The pointer is for C alone. Both helpers
now document the one-way contract, and goString is tested against a real
C-owned string by rebinding the version symbol to return a raw char*.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement offline transcription
Create the ASR recognizer in loadASR and serve AudioTranscription.
Segment times are int64 nanoseconds, not seconds: the proto field is an
int64 that core/backend reads straight into a time.Duration, while the
runtime reports word offsets in milliseconds. Words are grouped into one
segment per consecutive speaker run, with the 1-based speaker tag carried
through and 0 (untagged) left unlabelled.
The whole RPC body runs inside withEngine so the family check and the C
calls happen under one acquisition of engineMu. Free runs without the
backend lock, so checking the family and then relocking would let a
teardown destroy the handle in the gap. The audio decode is inside the
closure too, which costs nothing: base.SingleThread already serialises
this backend's RPCs.
recognizeF32 guards zero-length PCM. &pcm[0] panics on an empty slice, so
Go never reaches the C side's own "empty audio" rejection, and a silent
clip or a truncated upload is ordinary input.
pkg/utils has no WAV decode helper, only the ffmpeg normalisation, so
audio.go pairs AudioToWav with go-audio the way parakeet-cpp does. It
returns the sample rate rather than a duration, since the C API resamples
off that number.
Also closes the write-side half of the race Task 5 fixed on the read
side: Load now holds engineMu across the family switch and the n.fam
commit, matching Free. The loaders still must not take it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement streaming and live transcription
AudioTranscriptionStream drives a whole clip through the cache-aware
streaming API in 100 ms pushes, emitting each finalized utterance as a
delta and closing with the assembled result. AudioTranscriptionLive
serves the bidirectional RPC over the same session: config first, a ready
ack, deltas with word timings as utterances land, and a terminal result
when the caller closes its send side.
Both wrap their body in withEngine, so a stream holds engineMu for its
whole life and Free waits on it rather than destroying the recognizer
underneath a half-finished stream. That makes the way out load-bearing:
the file loop honours the request context between pushes, and the live
loop ends when the host closes the request channel, so a disconnected
client cannot pin the model against unload.
Only finals become deltas. The runtime applies punctuation and inverse
text normalization on finals only, so a final rewrites the utterance
rather than extending its interim, and delta on the wire is
newly-finalized text that consumers concatenate. Forwarding interims
would duplicate and mispunctuate every utterance.
The four streaming entry points sit behind an asrSession interface. No
NeMo GGUF is small enough to keep in the tree, so without that seam the
need-more-audio drain would have no test at all: nemo_speech_asr_stream_next
reports OK with a NULL handle when it wants more audio, which is a pause
rather than an end, and reading it either way round drops results or
spins forever.
Also folds in three items from the offline transcription review:
- empty audio is now refused before anything crosses the ABI, not
inside recognizeF32. The added integration spec caught the old
ordering panicking on an unbound entry point instead of failing;
- an undecodable sample rate is an error rather than 0, which this
runtime reads as "already at the model rate" and would have made a
wrong rate silently pitch-shift the audio;
- AudioTranscription guards its result pointer instead of relying on
an unstated invariant.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): make live deltas concatenate and fill segment words
runLive wrote the inter-utterance separator into the accumulated
transcript but emitted the delta without it, so a two-utterance turn sent
"one." and "two." while the terminal result read "one. two.". The live
consumer is the one that really concatenates: the realtime semantic-VAD
path joins the accumulated deltas with the empty string and clears them
only at a turn reset, never at an endpoint, so the running caption read
"one.two.". The separator now goes into the delta, as it already did on
the file path, and the terminal text is the verbatim concatenation rather
than a trimmed rebuild.
TranscriptSegment.Words was never populated, so a request asking for
timestamp_granularities ["word"] came back with no words at all even
though the timings were decoded. wordsToSegments now attaches them,
gated on the granularity the same way parakeet-cpp gates it, so a
transcript that did not ask for word timestamps does not pay for them.
Also: the final that comes back from the tail flush no longer claims an
end-of-utterance. It is the end of the stream, not a user yielding the
turn, and eou is what the realtime turn detector acts on.
The comment explaining why interims are suppressed led with the runtime's
postprocessing. The wire contract is the stronger reason and now comes
first: consumers concatenate deltas, so forwarding a growing hypothesis
assembles to "hehellhelloHello.". The postprocessing only explains why no
diffing trick would rescue them. It is also ITN and strip_formatting
rather than punctuation, which is off by default here.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement standalone diarization
loadDiarizer creates the Sortformer diarizer and Diarize serves the RPC
over a diarization stream: decode, chunked push, finish, then the
count-then-fill segments protocol.
nemo_speech_diar_segment carries start_time and end_time in SECONDS
already, not frame indices, so no conversion happens on the way to
DiarizeSegment.start/end and the model's seconds-per-frame is not
involved at all. The speaker label is the runtime's 1-based tag as a
decimal string, matching what wordsToSegments emits on the ASR path, so
the same speaker reads the same way whether a caller diarized a file or
transcribed it.
The six frame-geometry overrides are written as -1 rather than left
zero. c_api.cpp applies left_context_frames when it is >= 0 while every
other override needs > 0, so a zeroed config would silently pin the left
context to zero and change the model's streaming geometry.
nemo_speech_diar_segments writes *count before it rejects a buffer that
is too small, so a rejected fill still reports the size to retry with.
collectSegments uses that rather than truncating, bounded at four
attempts because the RPC holds engineMu for its whole body and an
unbounded retry would block an unload behind it.
Two DiarizeRequest knobs map onto the segmentation config, and the
proto and header names cross over: min_duration_on is the C
min_duration_sec and min_duration_off is the C min_gap_sec. Six fields
have no equivalent in this pipeline and are logged rather than dropped
in silence: num_speakers, min_speakers and max_speakers (Sortformer's
capacity is fixed by the checkpoint), clustering_threshold (there is no
clustering stage), include_text (no ASR here) and threads.
The empty-PCM guard fires before the stream is opened, so a silent clip
never reaches a purego entry point that would dereference &pcm[0].
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): pin the diarizer geometry sentinels and cap the segment buffer
The six frame-geometry overrides were written as -1 with nothing
asserting it. c_api.cpp applies left_context_frames at >= 0 while the
other five need > 0, so a dropped sentinel there pins the model's left
context to zero, and the struct keeps exactly the same shape, which is
all the layout assertions can see. Extracting diarModelConfig makes the
values assertable: five specs now pin all six frame fields, the device
index, the declared size and the NULL preset, each frame field on its
own line so a missing sentinel names itself.
distinctSpeakers had a spec with three segments over three distinct
labels, which len(segs) satisfies just as well as the real thing. Four
segments over three labels makes it a spec that can fail.
collectSegments sized its buffer straight from a count the C side
reported, and make() panics rather than erroring on a length it cannot
satisfy, so an uninitialised size_t coming back across the ABI killed
the backend process instead of failing one request. A ceiling of 2^22
segments, upwards of 93 hours of audio at one 80 ms frame each, turns
that into a diagnosable error.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement TTS and streaming TTS
The PCM callback is compiled once per process behind a sync.Once, not once per
request and not once per load. purego.NewCallback writes into a fixed table of
2000 entries (purego/syscall_sysv.go) and never releases one, so a per-request
callback panics the backend process on the 2001st synthesis, and a per-load one
reaches the same ceiling on a server that swaps models. Synthesis is routed
through that single callback plus a user_data id: engineMu is per-model, one
process holds several models, so a single current-sink pointer would be
overwritten by two TTS models synthesizing at once.
Deviations from the brief, all verified against the real headers and proto:
- TTS is TTS(*pb.TTSRequest) error and TTSStream is
TTSStream(*pb.TTSRequest, chan []byte) error, per pkg/grpc/interface.go.
The brief's context/pb.Result and server-stream forms do not implement the
interface. The channel is closed on every path, including the family
rejection, because pkg/grpc/server.go blocks on its drain goroutine and an
unclosed channel hangs the RPC with the backend lock held.
- The callback takes unsafe.Pointer, not uintptr. Converting a uintptr
parameter back to a pointer is a checkptr violation that aborts under
-race.
- resolveSpeaker refuses to turn a negative number into a speaker index. -1
is the C API's "use the default" sentinel, so the brief's rule would have
made a request naming an invalid voice synthesize in the default voice
instead of being rejected.
temperature and cfg_scale each write their override flag as well:
magpietts/runtime.cpp reads the float only when the flag is set, so a
temperature without it is silently discarded.
Also folds in Task 8's review finding on asr.go: the six bare -1 sentinels in
loadASR move to an asrDiarConfig builder reusing diarGeometryDefault, with
specs. src/asr/c_api.cpp applies left_context_frames at >= 0, so a dropped
sentinel pins the model geometry to 0 and no layout assertion can see it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): surface NMT translation through Predict
nemo_speech_nmt_translate takes explicit source and target languages and has no
free-form generation or token-callback entry point, so there is no prompt in the
LLM sense. The pair comes from the source_language / target_language model
options, with an optional leading [src->tgt] directive as the only per-request
override, and PredictStream emits the whole translation as a single chunk
because the C API has nothing finer to give it.
Both RPCs wrap their body in withEngine so the family check and the C calls that
trust the handle share one acquisition of engineMu. PredictStream closes its
channel on every path, including the family rejection: this is the legacy
streaming contract, and pkg/grpc/server.go blocks on a drain goroutine that only
finishes when the channel closes, so leaving it open hangs the RPC rather than
failing it.
nmtTranslatorConfig is extracted so its four adjacent pointer fields can be
asserted against distinct sentinels. Transposing two of them changes neither the
struct size nor any field offset, so the layout assertions cannot see it.
Also removes goString, which had no production caller: every string-returning
symbol in abi.go is bound with a Go string return that purego converts itself.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(nemo-speech-cpp): pin the three-segment pair tag in an NMT directive
The directive regex allowed an unbounded run of two-letter segments per side, but
nothing tested it: narrowing that run back to a single optional segment left every
spec green. resolve_tag accepts a ready pair tag in one field with the other empty
(src/nmt/langpairs.cc), and those tags run to three segments (en-zh-cn, pt-br-en),
so a shorter pattern does not mis-split the tag, it fails to match the directive at
all and the whole bracket is handed to the model as text to translate.
The justification on the regex was also wrong and is corrected: pt-br and zh-cn are
two segments and parse either way. It is the single-field form that needs the run.
Renames the NMT handle to n.nmt so it stops sharing a name with the translator
interface, following n.synth, which is shortened for the same reason.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): register the backend and give its specs a CI job
Registers nemo-speech-cpp across every surface .agents/adding-backends.md
requires, and adds the CI job its unit suite never had.
backend/index.yaml gets the meta backend (capabilities map, no uri), a
development meta and 12 image entries. No amd and no intel capability keys:
upstream NeMo-Speech.cpp builds ggml with CUDA, Vulkan or Metal only, and
SystemState.Capability falls back to "default", so those hosts get the CPU
build rather than a tag that does not exist. The nvidia-cuda-* and
nvidia-l4t-cuda-* keys are present because getSystemCapabilities() refines an
NVIDIA host to them whenever the CUDA directory exists; without them every
modern CUDA host and Jetson would miss the map and quietly run on CPU.
.github/backend-matrix.yml gets 7 include rows and 1 includeDarwin row. No
hipblas and no sycl rows, for the same upstream reason. cpu and vulkan are
per-arch pairs sharing a tag-suffix so backend-merge-jobs builds a multi-arch
manifest: an ARM host with no NVIDIA GPU reports "default" and the Jetson image
does not cover it.
The CI job is the substantive part. make test-extra is dead on master, because
prepare-test-extra depends on a protogen-python target that does not exist and
no workflow invokes it anyway, so the entry added earlier in this series ran
nowhere. abi_test.go asserts the size and field offsets of every Go mirror
struct against the C ABI it is dlopened into, and those assertions are the only
defence against silent memory corruption after a purego symbol rename or an
upstream header change. tests-nemo-speech-cpp in test-extra.yml now executes
them on pull_request and on master, gated on the backend's own path filter.
The recipe sets NEMO_SPEECH_REQUIRE_LIBS=1, so a missing library fails rather
than skips. WITH_NORM=OFF skips the OpenFST leg and costs no coverage: nothing
in the four C ABI headers is conditional on it, so the layouts are identical.
Also registers the upstream pin with the bump bot, which the backend Makefile
already claimed but was never wired up, and adds the BackendCapabilities entry
so a hand-written model config gets a real usecase surface. PossibleUsecases is
the union of the four families and DefaultUsecases is transcript alone, the
audio-cpp pattern. No VoiceCloning key: MagpieTTS synthesizes from baked
speaker ids, not a reference clip.
No gallery entries: publishing converted GGUFs is a follow-up.
ModelIdentity needs no work in this backend. main.go serves through
grpc.StartServer, so every RPC lands on pkg/grpc's shared server wrapper first,
and checkModelIdentity is the first statement of all seven handlers this
backend implements. A second check inside NemoSpeech would be unreachable and
would risk diverging from the cross-language sentinel the router matches on.
AudioTranscriptionLive stays unguarded because TranscriptLiveRequest carries no
ModelIdentity field at all, which is a proto-level gap affecting every backend
and needs its own change.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): build the CUDA-13 Jetson image the l4t-cuda-13 key needs
The nvidia-l4t-cuda-13 capability pointed at nvidia-l4t-arm64-nemo-speech-cpp,
which is built on nvcr.io/nvidia/l4t-jetpack:r36.4.0 and therefore links ggml
against CUDA 12. A Jetson whose CUDA 13 runtime is present reports that
capability and would have pulled an image with no libcudart.so.12 to dlopen,
failing hard at load. That is worse than omitting the key: with no key
Capability() falls back to "default" and the host gets a working CPU build.
Fixed the way parakeet-cpp and moss-transcribe-cpp already do it, by shipping
the second L4T image rather than dropping the key. Nothing prevents building it
here: those peers use plain ubuntu:24.04 on ubuntu-24.04-arm with the same
Dockerfile.golang as this backend's other rows, and every package in the
nemo-speech-cpp apt gate exists on noble arm64.
Adds the -nvidia-l4t-cuda-13-arm64-nemo-speech-cpp matrix row and its two index
entries, repoints the key on both metas, and rewrites the capability-map comment,
which had the reasoning backwards.
Also adds the documentary inferBackendPath branch, matching all six sibling
*-cpp Go backends. Behaviour is unchanged; the generic golang fallthrough
already resolved this backend correctly.
The previous commit message said "all seven handlers" of the shared gRPC
wrapper. There are eight RPC entry points: seven are guarded by
checkModelIdentity and AudioTranscriptionLive is the unguarded eighth, which
that message already called out separately. Wording only.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(nemo-speech-cpp): document the backend and list it in the importer
Adds docs/content/features/nemo-speech-cpp.md, alongside the audio.cpp page
that is its closest sibling, and cross-links it from the speech-to-text,
diarization, text-to-speech, backend-type and compatibility-table pages so the
backend is reachable from every surface that lists its modalities.
The page covers the architecture-to-family table, every option key with a model
YAML per family, the translation prefix directive, the acceleration matrix, and
the four limitations this backend ships with: Linux-only inverse text
normalization, suppressed interim streaming results, the library's default
translation context and generation limits, and the absence of gallery entries.
knownPrefOnlyBackends gains the backend so it appears in the /import-model
dropdown. It stays preference-only and AutoDetect=false: general.architecture
lives inside the GGUF where no remote-repo probe can read it, and a translation
model carries an ordinary LLM architecture with no NeMo-specific marker.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(nemo-speech-cpp): correct the translation limits, the macOS gap and the TTS conversion
Three factual errors found in review, all of them the kind a user would act on.
The translation limits were described backwards. Input longer than the 1024-token
context is rejected, not truncated: translator.cpp throws "nmt: prompt too long
(N tokens) for context 1024", which reaches the caller as a failed request. What
is silently cut is the output, by the max_new_tokens loop at 256. The bullet now
separates the two and says which one fails quietly.
The macOS gap covers TTS text normalization as well. Both directions sit behind
the single NEMO_SPEECH_WITH_NORM flag, which the Makefile forces off on Darwin,
so tn_dir is as inert there as itn_dir. Neither fails the load: both warn and
carry on. pnc_model really is unaffected, since punctuation is compiled in
unconditionally. The tn_dir row in the option reference gained the caveat the
itn_dir row already had.
The TTS conversion procedure produced a model that could not load. It converted
MagpieTTS and stopped, leaving no NanoCodec, which the same page lists as
required; following it gave "no NanoCodec GGUF found next to ...". Both halves
are now there, each with the download that feeds it, so the block runs top to
bottom on a clean machine.
Also: any negative gpu value pins TTS to the CPU, not only -1, and FLAG_CHAT
additionally surfaces the model in the web UI chat picker.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): map every C status, not just the NMT one
INVALID_ARGUMENT was translated to codes.InvalidArgument at exactly one of
sixteen C call sites. Everywhere else a non-zero status collapsed to
codes.Internal, so the same backend answered an unsupported language pair with
HTTP 400 and an unknown TTS voice, which is the same class of caller mistake
against the same process, with HTTP 500. Status 4 is CANCELLED on the ASR and
TTS surfaces and was reported as a backend failure rather than as the consumer
having stopped listening.
asr.h, tts.h and nmt.h each declare their own status enum and diar.h reuses the
ASR one; the values they share agree, and the single divergence is that NMT
declares no CANCELLED because nemo_speech_nmt_translate has no callback for a
consumer to stop with. That is an absence, not a disagreement, so one table
serves all three. status.go carries it, with the header line numbers and a note
that a pin bump has to recheck it: purego binds by name and the status crosses
as a bare int32, so nothing in the build or the linker can see a drift.
New specs cover the whole enum, unknown values, and one real INVALID_ARGUMENT
per family driven through the shared objects rather than through the Go mapping
asserting against itself.
Also add UsecaseChat to this backend's capability entry, which the docs already
told operators to set for translation models. chat is a gallery filter key and
completion is not, so GET /api/backends/usecases would have greyed the Chat
filter out and hidden a Riva-Translate gallery entry from the one filter that
fits it. The flag gates no endpoint; it makes the model eligible as the default
chat model and puts it in the web UI chat picker, both of which Predict and
PredictStream already serve.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): audit the gosec unsafe and file-inclusion sites
gosec flags 13 alerts on this backend: one G304 and twelve G103. Each was
checked individually rather than blanket-suppressed, and each annotation
states what makes that particular site safe.
The G304 at audio.go is a false positive. The opened path is
filepath.Join of a directory the function just created with os.MkdirTemp
and a constant basename; the request-controlled path is the input to
AudioToWav and never reaches the open.
The twelve G103 sites are the package's three established shapes, and
every one was verified against them: cstr and pinPtr take the address of
something pinned on the line above and return it one-way (nothing in the
package converts either result back, which is what keeps checkptr out of
it under -race), and each *Create hands C a stack-local POD config whose
uintptr members are cstr allocations or pinPtr addresses held by a pinner
the loader unpins only after the call. The two slice-building sites are
bounded by construction: DiarSegments is handed exactly len(buf) with the
buffer sized under maxDiarSegments and a reported count larger than it
rejected rather than sliced to, and the TTS callback copies out a slice
whose length is the length the runtime declared for that buffer.
Separately, sampleRateOf gets a real fix rather than an annotation.
go-audio reads the WAV header's sample rate from an unsigned 32-bit field
into an int, so a header claiming more than 2^31-1 passed the "> 0" test
and then narrowed to a NEGATIVE rate, which the runtime would take as a
resampling ratio. AudioToWav cannot produce one today, but that is a
property of another package and this function exists precisely because
the rate is read back rather than assumed, so the bound is enforced here
and pinned by a spec.
The four remaining integer narrowings are annotated with the bound that
makes each safe: the WAV payload length is already checked against
maxWAVDataBytes, the speaker count is bounded by maxDiarSegments, and the
two segment ids are the proto's own int32 wire type.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): skip the CUDA-only ggml patch series on darwin
The macOS backend build died in patch-ggml:
scripts/apply-ggml-patches.sh: line 56: mapfile: command not found
make[1]: *** [patch-ggml] Error 127
mapfile is a bash 4 builtin (and its -d flag needs 4.4). macOS ships bash
3.2.57 as /bin/bash and GitHub's runner images add no newer one, so the
bare `bash` the recipe resolves from PATH cannot run upstream's script.
Rather than hunt for a capable bash that the runner does not have, drop
the step where it does nothing. ggml-patches/ is a CUDA series: every
kernel it adds is under src/ggml-cuda/, and its whole footprint outside
that directory is an op enum plus prototype in include/ggml.h, the
constructor and a name-table entry in src/ggml.c, and two ggml-cpu lines
that make the CUDA-only op report unsupported and abort. Nothing it
touches is compiled into a Metal kernel or changes a CPU one.
The project's own references to patch-only ggml symbols sit behind
NEMO_SPEECH_FUSED_RELPOS_ATTN and NEMO_SPEECH_FASTCONFORMER_CUDA_FUSIONS,
which cmake already forces OFF without GGML_CUDA, or behind
NEMO_SPEECH_GGML_PATCHED itself, which guards a GGML_TENSOR_FLAG_Q8_PLANAR
write that a non-CUDA buffer throws before reaching. So passing
NEMO_SPEECH_GGML_PATCHED=OFF costs the Metal build nothing, and it is
required once the series is skipped: that flag is what stops the ASR
sources referencing a tensor flag stock ggml does not define.
This is upstream's own Metal configuration. Its metal-* and vulkan-*
CMake presets inherit the cpu-* ones, which set NEMO_SPEECH_GGML_PATCHED
to OFF; docker/Dockerfile and scripts/windows/build.ps1 do the same for
their non-CUDA targets. LocalAI's Makefile never passed the flag at all
and so inherited the CUDA default everywhere.
Linux is untouched and keeps applying the series, including its
idempotency and its hard failure on a patch that does not apply. The gate
is the same uname test the WITH_NORM block above already uses, and both
branches keep the order-only clone prerequisite, which on a WITH_NORM=OFF
tree is the only thing that pulls sources/ in.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): restore std::binary_function for MeCab on libc++
NEMO_SPEECH_TTS_WITH_JA=ON compiles Open JTalk's bundled MeCab, and
mecab/src/dictionary.cpp derives a comparator from std::binary_function,
which C++17 removed. libstdc++ still ships it as deprecated-but-present
under -std=gnu++17, so Linux never notices. libc++ compiles it out and
the macOS arm64 build dies with "no template named 'binary_function' in
namespace 'std'".
This is ours, not an upstream regression: upstream defaults both
NEMO_SPEECH_TTS_WITH_JA and NEMO_SPEECH_TTS_WITH_ZH to OFF and the OSS
drop carries no CI at all, so that target is never built there. Upstream
does already carry the equivalent workaround for MSVC's STL
(_HAS_AUTO_PTR_ETC plus /FIfunctional) but has no libc++ branch.
libc++ gates the two templates on
_LIBCPP_ENABLE_CXX17_REMOVED_UNARY_BINARY_FUNCTION, and has since LLVM
16, older than any clang Xcode still ships. The name is the whole
problem: _LIBCPP_ENABLE_CXX17_REMOVED_BINDERS covers bind1st, bind2nd,
ptr_fun and mem_fun and not unary_function or binary_function, and the
umbrella _LIBCPP_ENABLE_CXX17_REMOVED_FEATURES no longer exists in
libcxx at all. A wrong name preprocesses fine and fixes nothing.
Applied through CMAKE_CXX_FLAGS rather than to the one target, because
the tokenizer CMakeLists is upstream's and sources/ is a pinned
checkout. Project-wide is also the safer scope: the macro decides
whether libc++'s internal __binary_function alias resolves to
std::binary_function or to __binary_function_keep_layout_base, a base
class of std::less and friends, so defining it for a subset of
translation units would give those class templates two spellings in one
binary. Both bases are empty and, at C++17, carry identical members, so
the define changes no layout and no ABI.
Darwin only. On Linux the branch is unreachable and the macro is not a
name libstdc++ knows, so it would be inert even if taken; a Linux
configure with the flag forced on puts it on all 23 C++ TUs of
nemo_speech_openjtalk_frontend including dictionary.cpp at -std=gnu++17,
and on none of the 16 C TUs.
Mandarin needs nothing: cppjieba v5.6.7 and limonp have no removed C++17
constructs left (limonp replaced std::not1 and std::bind2nd with
lambdas) and cppjieba's own CI builds macos-14 and macos-latest at C++11
through C++20.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): repair OpenFST's FstImpl::operator= for gcc-14
The first WITH_NORM=ON build failed compiling fst_normalizer.cpp against the
installed OpenFST 1.8.3 headers:
fst.h:690:59: error: no match for 'operator=' (operand types are
'std::unique_ptr<fst::SymbolTable, ...>' and 'fst::SymbolTable*')
FstImpl's copy-assignment operator assigns the raw pointer returned by
SymbolTable::Copy() straight to a std::unique_ptr member. No C++ standard
allows that, so the line is ill-formed everywhere; it survived because nothing
instantiates FstImpl::operator= and gcc up to 13 only checks a template
member's body when it is instantiated. gcc 14 resolves non-dependent operator
expressions at template definition time, so it rejects the line in any
translation unit that includes <fst/fst.h>. The CI diagnostic confirms the
phase: it reads "In member function", not "In instantiation of", and carries
no instantiation backtrace.
That is why this surfaces only here. build_itn_deps.sh compiles OpenFST with
gcc-12 and upstream's own images build the runtime with gcc-13, so neither
compiler reaches the check; backend/Dockerfile.golang installs gcc-14 and
promotes it with update-alternatives, and fst_normalizer.cpp is the one
translation unit in this backend that includes OpenFST.
Fix it in the installed ITN prefix, which is the only copy the cmake build
compiles against, using the same .reset() spelling FstImpl::SetInputSymbols
already uses for the identical operation. libfst.so is linked before this runs
and cannot contain the function, since no compiler could ever have emitted it,
so there is no ABI or ODR consequence. The rule is guarded on both sides so a
pin bump to a fixed OpenFST fails loudly rather than silently no-opping.
Verified with a real gcc 14.2: the CI error reproduces byte for byte from a
file whose entire content is '#include <fst/fst.h>', and gcc 14 reports
exactly two errors over the whole OpenFST include closure this backend uses,
both of them these two lines. After the patch that closure compiles clean
under gcc-14 with the target's own flags. The step is reachable only under
WITH_NORM=ON, so 'make -n stage-libs WITH_NORM=OFF' mentions neither it nor
the ITN build, and darwin, which defaults WITH_NORM to OFF, never evaluates it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): install cmake 3.31 on bases that ship less than 3.26
The JetPack r36.4.0 row dies on the first line of NeMo-Speech.cpp's
CMakeLists.txt:
CMake Error at CMakeLists.txt:3 (cmake_minimum_required):
-- Configuring incomplete, errors occurred!
Upstream opens with cmake_minimum_required(VERSION 3.26). That base image is
Ubuntu 22.04 jammy, whose apt cmake is 3.22.1, so configure aborts before it
reads a single one of the backend's -D flags. Every other Linux row in this
block is noble, which ships 3.28 and clears the bar, so the failure is one
base image wide rather than a code problem. Everything before it on that row
had already worked, including the OpenFST and Sparrowhawk ITN build.
No other Go backend needs this. parakeet-cpp and moss-transcribe-cpp share
the same JetPack base and both declare cmake_minimum_required(VERSION 3.18),
and nothing in the repo installs a cmake newer than the distro's, so there is
no existing pattern to reuse. Nothing depends on jammy's cmake staying 3.22
either: build_itn_deps.sh never invokes cmake at all, since OpenFST and
Sparrowhawk are autotools builds.
Kitware's release tarball rather than their APT repo or pip. The tarball is a
pinned URL with a published checksum, so an upstream release cannot change
what lands here. The APT repo does carry jammy arm64, but it serves a moving
latest that today is CMake 4.4, and 4.x drops compatibility with
cmake_minimum_required below 3.5, which vendored third_party subprojects
still declare; pinning it there would mean tracking Kitware's Debian revision
string instead of an upstream version. pip would drag a Python toolchain into
a backend that has none. 3.31.12 is the last 3.x release, so it clears 3.26
while keeping the CMake 3 policy surface, and it stays close to the 3.28 the
green noble rows already use. The binaries need only glibc 2.17 and carry no
libstdc++ DT_NEEDED, well under jammy's 2.35. doc/, man/, ccmake and cmake-gui
are not extracted; the final image is FROM scratch, but there is no reason to
page 100 MB of Qt GUI and docs through the CI cache.
Gated on the installed cmake actually being older than 3.26, so the rows that
already build green keep configuring with exactly the cmake they use today,
and folded into the existing ${BACKEND} block rather than added as a new
instruction, so no other Go backend image gains a layer and nothing above the
Vulkan SDK, CUDA, Go and protoc layers moves.
The symlink lands in /usr/local/bin and shadows apt's cmake. Unlike the protoc
shadowing that broke Sparrowhawk earlier in this series that is inert: protoc
has to agree with the libprotobuf headers it generates against, whereas cmake
links nothing into the product and has no ABI relationship with anything in
the image, and it resolves the symlink back to /opt to find its own Modules/
tree, so a 3.31 binary can never read 3.22's modules.
The version test avoids $(...) deliberately. BuildKit delivers a RUN heredoc
through an outer shell with an unquoted delimiter, so a command substitution
runs there, too early, in a container where the files it reads do not exist
yet, and its empty output is pasted into the script; the first draft took the
install branch on every row because of it.
Verified by building the block against nvcr.io/nvidia/l4t-jetpack:r36.4.0
arm64 under qemu, the row's actual base image: cmake 3.22.1 detected, tarball
checksum verified, 3.31.12 installed, and a cmake_minimum_required(VERSION
3.26) project configures with -G Ninja and builds, with CMAKE_ROOT resolving
to /opt/cmake/share/cmake-3.31. Same on ubuntu:22.04 amd64 and arm64.
ubuntu:24.04 skips the install, gains no /opt/cmake and still configures on
/usr/share/cmake-3.28. The NeMo-Speech.cpp compile itself on JetPack CUDA 12
is not reproducible here and remains for CI.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
a77780ad14 |
feat(gallery): fall back to mirrors and a cached index when the primary source fails (#11389)
* feat(version): include OS and arch in the outbound User-Agent
Registries and galleries already receive LocalAI/<version>; adding the
platform follows ordinary client convention and discloses nothing a
registry cannot infer from the manifest it is asked for.
Updates the User-Agent note in docs/content/getting-started/models.md,
which documented the old format.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(downloader): identify LocalAI on outbound requests
pkg/oci has always sent a User-Agent; the downloader sent none, so gallery
reads, model-file downloads, resume probes, content-length probes and the
HuggingFace safety scan all went out as a bare Go HTTP client, unattributable
to LocalAI by the hosts serving them.
HuggingFaceScan moves off the client's Get shorthand to an explicit request
for the same reason — the shorthand gives no place to hang a header.
Extends the User-Agent note in docs/content/getting-started/models.md, which
claimed the header was sent only to Ollama and OCI registries.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(gallery): add a mirrors list to gallery configuration
Mirrors are an availability fallback, tried in order only after the primary
URL fails. omitempty keeps existing configurations byte-identical.
The slice makes config.Gallery non-comparable with ==, which broke the two
slices.Equal callers in the runtime settings registry. Replace them with an
explicit Gallery.Equal / GalleriesEqual so a gallery list that differs from
the baseline only by its mirrors still counts as env/CLI-set. Equal compares
the Verification block by value; == compared it by pointer identity, which
called two structurally identical policies different.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(downloader): treat an HTTP error status as a failed read
ReadWithCallback handed the response body to its callback whatever the
status was, so a 404 page or a 502 from a CDN arrived as if it were a
gallery index or a model config: it parsed to nothing, got cached for an
hour, and no caller could tell the source had been down. DownloadFile has
always checked the status; this path never did.
Mirror fallback depends on it — a source that answers with an error page
has to count as unreachable, or the next candidate is never tried.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(gallery): fall back to mirrors when the primary source fails
Candidates are tried primary-first with a bounded timeout each, and a
source that just failed is skipped for a cooldown so a dead host is not
re-dialled on every listing. When every candidate is in cooldown they are
all tried anyway: refusing to serve a gallery we might be able to reach is
worse than one slow request.
The one-hour index cache is untouched and stays keyed on the gallery's own
identity, so a mirror-served fetch fills the entry the primary would have.
No SSRF validation is applied to the candidates. validateGalleryConfigURL
guards GetGalleryConfigFromURL because that URL arrives in a request body;
mirrors come from the operator's gallery configuration, the same place the
primary has always come from, and the index fetch has never validated the
primary. Validating mirrors while the primary goes unchecked would buy
nothing and would break the deployment mirrors exist for — an index served
from a host on the LAN.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(gallery): loosen the mirror fetch timeout and stop blaming the caller
The downloader only ever bounded response headers, never the body, so the
per-attempt deadline added with mirror fallback was the first whole-transfer
timeout this path has had. At 30s the default 2.2 MB index demanded ~75 KB/s
sustained: a rural-DSL, mobile or satellite user who used to wait 60s and
succeed would now fail, and then eat a 10-minute cooldown on a source that
was perfectly healthy. Raised to 120s (~19 KB/s), which no link that could
go on to download a model will miss, and made it a var so a test can shorten
it and prove a hanging candidate is actually abandoned.
Caller cancellation is no longer recorded as a failure of the source.
Unreachable today since getGalleryElements passes context.Background(), but
once a request context is wired through, a browser disconnect would have
blackholed every candidate for ten minutes over something the sources had
no part in.
Also document that mirrors do not cover a .ref gallery URL: the reference is
resolved before mirrors are considered, so a .ref that cannot be fetched
fails the gallery outright. Routing .ref resolution through the candidate
list needs a per-candidate resolve-and-fetch and a decision about cache
identity, which is more than this change should carry.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(gallery): serve the last known good index when everything is offline
A successful fetch is cached alongside the models directory and served when
no source is reachable, so an offline or airgapped machine can still list
its gallery. Entries may be stale in that state, and the fallback is logged.
The copy is deliberately kept out of the models directory, where a <name>.yaml
file is read as an installed model's configuration, and is named after a digest
of the gallery URL so the model and backend galleries cannot collide. Writing
it is best effort: a read-only or full disk must not fail a fetch that
otherwise succeeded.
Also corrects the mirror scheme list in the docs: the HuggingFace prefixes are
huggingface://, hf:// and hf.co/, not huggingface:.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(gallery): only cache a response that is really a gallery index
The last known good copy was written on any 2xx, before anything looked
at the bytes: the parse only happens later, in getGalleryElements. A
captive portal, a corporate proxy or a CDN error page all answer HTTP 200
with HTML, so any of them could overwrite a good copy. The listing fails
then and there, and the next offline start — the one case this cache
exists for — serves the interception page instead of the gallery it
already had.
Probe the body before persisting it: unmarshal into a []any and keep the
older copy unless the result is a non-empty sequence. An empty document
is rejected too. It parses fine, so a parse-only check would still let a
blank response replace a populated index with one that lists nothing,
which from the user's side is the same outage; and an empty index is
worth nothing offline, so there is no case where caching it beats keeping
what came before. The live body is still returned to the caller — the
probe gates persistence only, and getGalleryElements remains the thing
that reports a real parse failure.
Also in this pass:
- The empty-basePath guard only caught exact "". galleryCachePath(".")
and galleryCachePath("models") still resolved the cache sibling against
the process working directory, which is what the guard was written to
prevent. Reject any non-absolute base.
- The docs claimed the offline cache "applies to every gallery, with or
without mirrors". Not true for a .ref URL: the reference is resolved
before the cache is consulted, so a .ref gallery fails offline even
after a successful earlier fetch, and the cache file it writes can
never be read. Extend the .ref warning and qualify the sentence.
- pkg/oci's UserAgent comment never mentioned the platform component
added earlier on this branch.
- resetGalleryFailures and expireGalleryFailure had no non-test callers;
move them into the test file.
- The all-candidates-failed error reported len(attempt), so a three
mirror gallery with two sources in cooldown said "all 1 source(s)
failed" — which reads as a misconfiguration. Report how many were
configured and how many were skipped.
- Give the package's tests their own TMPDIR. The cache is a sibling of
the models directory, which is right in production, but specs that
build a models directory directly under /tmp made the sibling resolve
to /tmp/cache and left it behind after every run.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(gallery): convert the new tests to Ginkgo and clear the lint gate
.agents/coding-style.md requires Ginkgo v2 + Gomega for every Go test and
has forbidigo enforce it; the stdlib-style tests still in the tree are tech
debt, not a pattern. Every test file this branch added was written in the
forbidden style, which is what turned CI red.
Convert all five of them. internal had no suite bootstrap, so add one;
core/config, core/gallery and pkg/downloader already have theirs and are
reused, so no package mixes styles. pkg/downloader/useragent_test.go and
read_status_test.go were not in CI's forbidigo list but used the same
forbidden calls, so they are converted too.
The one conversion with a trap in it is core/gallery. Go's t.TempDir()
yields $TMPDIR/<TestName>NNNN/001, so the gallery cache — a sibling of the
models directory — was isolated per test. GinkgoT().TempDir() yields a flat
$TMPDIR/ginkgoNNNN, which would put every spec's cache in one shared
directory and break the specs that count files in it. tempModelsDir()
restores the original isolation.
Also make the deliberate cleanup-path ignores explicit with `_ =`, drop the
gallery cache directory to 0750 (nothing outside the server's own user and
group reads it), and justify the cache read with a #nosec G304 comment in
the form already used elsewhere in the tree: the path is a hex sha256 under
a fixed directory with a non-absolute base already rejected, so no
caller-supplied text reaches it.
Re-ran the mutations these specs were verified against — dropping the
platform suffix from UserAgent, making Gallery.Equal ignore Mirrors and
ignore Name, removing persistGalleryIndex's validity probe, removing the
!filepath.IsAbs guard, not skipping a cooled-down candidate, and dropping
the per-attempt timeout. All seven still fail the converted specs.
Assisted-by: Claude:claude-opus-5 [go vet] [go test] [golangci-lint] [gosec]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Richard Palethorpe
|
5ac445e1d4 |
fix(react-ui): restore 3D Studio results and history (#11393)
* fix(react-ui): restore 3D Studio results and history Keep large conditioning-image payloads out of the rendered request panel so the generated viewer can mount reliably. Accept clipboard images and synchronize 3D history consumers so new results appear in Studio without a reload. Cover clipboard input, bounded request rendering, result display, and cross-view history synchronization with Playwright. Assisted-by: Codex:gpt-5 Playwright * perf(react-ui): idle the 3D viewport when still Limit auto-rotate rendering to 30 FPS and stop scheduling frames when rotation is disabled. Resize, view controls, and pointer input invalidate the still frame on demand. Assisted-by: Codex:gpt-5 Playwright |
||
|
Nandana Dileep
|
147a5ee783 |
fix(react-ui): stop traces page crash when switching trace tabs (#11387)
Switching from Backend Traces back to API Traces crashed the page with "can't access property status, e.response is undefined" (#11376). The API table briefly renders the previous tab's backend rows while the refetch effect is still pending, and those rows carry no `response` envelope. The status column dereferenced it unguarded. Render a neutral placeholder instead of throwing, and cover the tab-switch scenario with a regression spec. Assisted-by: opencode:big-pickle Signed-off-by: Nandana Dileep <110280757+nandanadileep@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
9f62401fca |
feat(traces): show in-flight API requests (#11368)
Register JSON API exchanges before their handlers run so the traces dashboard can surface active work. Replace the live entry with the completed persisted record under the same ID, and clean it up if a handler panics. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
dependabot[bot]
|
08598a8611 |
chore(deps): bump the npm_and_yarn group across 1 directory with 5 updates (#11341)
Bumps the npm_and_yarn group with 5 updates in the /core/http/react-ui directory: | Package | From | To | | --- | --- | --- | | [hono](https://github.com/honojs/hono) | `4.12.25` | `4.12.34` | | [@hono/node-server](https://github.com/honojs/node-server) | `1.19.14` | `2.1.0` | | [fast-uri](https://github.com/fastify/fast-uri) | `3.1.4` | `3.1.5` | | [ip-address](https://github.com/beaugunderson/ip-address) | `10.2.0` | `10.4.0` | | [undici](https://github.com/nodejs/undici) | `7.28.0` | `7.29.0` | Updates `hono` from 4.12.25 to 4.12.34 - [Release notes](https://github.com/honojs/hono/releases) - [Commits](https://github.com/honojs/hono/compare/v4.12.25...v4.12.34) Updates `@hono/node-server` from 1.19.14 to 2.1.0 - [Release notes](https://github.com/honojs/node-server/releases) - [Commits](https://github.com/honojs/node-server/compare/v1.19.14...v2.1.0) Updates `fast-uri` from 3.1.4 to 3.1.5 - [Release notes](https://github.com/fastify/fast-uri/releases) - [Commits](https://github.com/fastify/fast-uri/compare/v3.1.4...v3.1.5) Updates `ip-address` from 10.2.0 to 10.4.0 - [Release notes](https://github.com/beaugunderson/ip-address/releases) - [Commits](https://github.com/beaugunderson/ip-address/compare/v10.2.0...v10.4.0) Updates `undici` from 7.28.0 to 7.29.0 - [Release notes](https://github.com/nodejs/undici/releases) - [Commits](https://github.com/nodejs/undici/compare/v7.28.0...v7.29.0) --- updated-dependencies: - dependency-name: hono dependency-version: 4.12.34 dependency-type: direct:production dependency-group: npm_and_yarn - dependency-name: "@hono/node-server" dependency-version: 2.1.0 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: fast-uri dependency-version: 3.1.5 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: ip-address dependency-version: 10.4.0 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: undici dependency-version: 7.29.0 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
93fe086798 |
chore(deps): bump the npm_and_yarn group across 1 directory with 2 updates (#11338)
Bumps the npm_and_yarn group with 2 updates in the /core/http/react-ui directory: [@hono/node-server](https://github.com/honojs/node-server) and [brace-expansion](https://github.com/juliangruber/brace-expansion). Updates `@hono/node-server` from 1.19.14 to 2.0.12 - [Release notes](https://github.com/honojs/node-server/releases) - [Commits](https://github.com/honojs/node-server/compare/v1.19.14...v2.0.12) Updates `brace-expansion` from 1.1.12 to 1.1.18 - [Release notes](https://github.com/juliangruber/brace-expansion/releases) - [Commits](https://github.com/juliangruber/brace-expansion/compare/v1.1.12...v1.1.18) --- updated-dependencies: - dependency-name: "@hono/node-server" dependency-version: 2.0.12 dependency-type: indirect dependency-group: npm_and_yarn - dependency-name: brace-expansion dependency-version: 1.1.18 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
e2311045d3 |
fix(mcp): drop the duplicated scheduling methods on stubClient (#11323)
master does not compile:
vet: core/http/endpoints/mcp/localai_assistant_test.go:157:19:
method stubClient.ListScheduling already declared at
core/http/endpoints/mcp/localai_assistant_test.go:87:19
Two fixes for the same breakage landed. The four Scheduling methods were
already present at lines 87-99, in interface order after ListNodes, by
the time #11318 merged; #11318 appended its own copy after
GetRouterDecisions. The two blocks sit in different parts of the file, so
git merged both without a conflict and nothing flagged it.
Remove the appended copy and keep the one in interface order. Pure
deletion, no behaviour change.
Verified: go vet clean on ./core/http/endpoints/mcp/, and
go test ./core/http/endpoints/mcp/ passes.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
95bd59d78e |
fix(mcp): teach the assistant test stub the scheduling methods (#11318)
#11228 added ListScheduling, GetScheduling, SetScheduling and DeleteScheduling to localaitools.LocalAIClient but did not update stubClient, the hand-written test double in the mcp endpoints package. The package therefore fails to typecheck, which takes out both lint and tests on master: cannot use stubClient{} as localaitools.LocalAIClient value in argument to h.Initialize: stubClient does not implement localaitools.LocalAIClient (missing method DeleteScheduling) Red on |