* fix(gallery): repoint qwen3-4b/qwen3.5-9b dflash drafters at post-rename GGUFs
The drafters both entries referenced were converted from the pre-merge DFlash
PR branch and carry dflash.target_layer_ids. llama.cpp reads dflash.target_layers
and refuses the load. The stored values are offset by +1 relative to the HF-side
field, so the files cannot be repaired by renaming the key and must be replaced.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ci): add apexentries HuggingFace client
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ci(apexentries): build the HF client via pkg/httpclient
The apexentries HuggingFace client was constructed as a raw
&http.Client{Timeout: 60s}. The repo convention (documented in
.golangci.yml, which cannot express this as a forbidigo pattern) is that
all outbound HTTP goes through pkg/httpclient, which refuses redirects by
default and sets a TLS 1.2 floor. The std client follows redirects and
forwards custom credential headers to the redirect target on a cross-host
hop (GHSA-3mj3-57v2-4636). Only a User-Agent is sent today, but this
calls an external API and an HF_TOKEN header added later would leak.
Switch to httpclient.NewWithTimeout, preserving the 60 second timeout.
No behaviour change for the current header set.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ci): discover APEX tiers by filename suffix
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ci): resolve unsloth counterparts and sharded quants
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ci): render APEX child entries with the dflash/mtp tag rule
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ci(apexentries): set backend, known_usecases and cross-repo drafters
RenderChild left three gaps against the hand-written gallery entries.
The generated entries reference gallery/virtual.yaml, which supplies no
backend, so every generated entry named no engine at all. All comparable
hand-written entries set backend: llama-cpp in overrides; do the same.
Set known_usecases to [chat] alongside it: LocalAI falls back to the
backend defaults when it is absent, so this is convention rather than
breakage, but generated entries should not read differently from their
neighbours.
The drafter was also assumed to live in the repo publishing the weights.
Speculative pairings routinely cross repos, and a drafter URI built from
the weights repo 404s at install time. Add ChildInput.DraftRepo, used for
both the drafter URI and its local path, falling back to Repo when empty
so pairings that do ship the drafter alongside the weights are unchanged.
The dflash/mtp tagging rule is untouched: the tag still follows SpecType
and nothing else.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ci): dedupe generated entries against the existing gallery
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* apexentries: canonicalize HF URIs and dedup the generated batch
Merge exists to stop a second gallery entry being added for weights the
gallery already ships, but two gaps let duplicates through on a bulk run.
The URI key was compared as an exact string while render.go only ever emits
https://huggingface.co/{repo}/resolve/main/{file} and the gallery records
1038 of its URIs in huggingface://{repo}/{file} shorthand. A generated
unsloth rung whose weights are already shipped in shorthand was therefore
not recognised. canonicalURI reduces both spellings to one key and is
applied on both sides, taking care that the repo is exactly the first two
path segments so sharded quants in a subdirectory still match. A URI in
neither form is returned untouched so other hosts dedup on their literal
string.
Merge also never accounted for entries it had just accepted, so two
generated entries sharing a name or a primary URI both landed in add.
Several APEX repos share one base model and resolve to the same unsloth
counterpart, so the identical rungs are generated twice under the same
name. Batch state is tracked locally rather than written back into the
caller's ExistingIndex, which a caller may reasonably reuse.
Name is still checked before URI: a name collision must block the add
regardless of the weights.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ci): verify variant and tagging invariants in the gallery index
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ci): scope the apex-entries verifier to what it can actually judge
The verifier reported 60 problems against the real gallery, 57 of which were
llama.cpp assumptions meeting entries from other backends. A gate that is wrong
57 times out of 60 cannot gate anything.
- The weight-count check catches a quant label collision in llama-cpp quant
discovery, so it now runs only for overrides.backend: llama-cpp. Entries with
no declared backend are skipped because their weights are declared in the
referenced url: template, which the verifier never reads.
- The dflash/mtp tag check now implements the per-backend table in
.agents/adding-gallery-models.md instead of assuming llama.cpp's spec_type:
vocabulary. ds4 declares mtp_path:/mtp_draft:; sglang declares
speculative_algorithm: in a file this verifier cannot follow, so sglang
entries are not judged in either direction. The check stays bidirectional
within the backends it does judge.
- sha256 is now required on .gguf files only, since every non-GGUF asset in the
index belongs to a hand-curated entry outside this generator's scope.
Against the current gallery this leaves exactly the three genuine problems:
two entries setting spec_type:draft-mtp without the mtp tag, and one entry
whose overrides.mmproj names a file it does not download.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ci(apexentries): anchor quant matching and invert the sha256 rule
UnaccountedQuants matched files to wanted quants with strings.Contains, which
reproduces the substring collision it was written to warn about: Q8_0 is a
substring of UD-Q8_0, so a repo publishing only UD-Q8_0 was reported as
publishing an unbuilt Q8_0. Subdirectory-sharded UD quants are the normal
unsloth layout for large repos, so this fired on realistic input.
Match on the quant label as an anchored token instead, the way
DiscoverUnslothQuants does, so the diagnostic and the discovery it audits
cannot disagree about what a file is. Root-level shards, the layout the
diagnostic mainly exists to catch, stay detected.
The sha256 requirement was scoped to .gguf, which exempted seven real model
weights: wan_2.1_vae.safetensors and clip_vision_h.safetensors across the
wan-2.1-*-ggml entries, both load-bearing weights named by gallery/wan-ggml.yaml.
Invert the rule so a checksum is required on everything except metadata
extensions, which keeps a future weight format covered by default rather than
silently exempt.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ci): wire the apexentries command
Adds the generation path to the apexentries command: list the mudler APEX
repos, discover each one's quality ladder and its unsloth counterpart's quant
rungs from the filenames actually published, render a child entry per build
plus a family parent carrying the variants list, dedup against the gallery,
and write the additions to -out or append them with -apply.
Discovery shortfalls are reported at discovery time rather than left to the
verifier. A quant or a tier that discovery drops leaves no trace in a finished
gallery file, and because an empty imatrix ladder falls back to the plain one,
a repo whose imatrix filenames all fail to match downgrades the whole family
silently instead of erroring.
Merge's single reused map is split into two reported categories. A URI match
means the gallery already ships exactly these weights and referencing the
existing entry is correct; a name collision means an unrelated entry owns the
name and referencing it would substitute a different build.
Multimodal children now declare known_usecases [chat, vision]. An explicit
known_usecases suppresses the backend-default fallback, so a chat-only entry
carrying an mmproj never matches the vision or multimodal gallery filters.
.github/ci is invisible to go list ./..., so a workflow names both generator
packages explicitly and their specs finally run on pull requests.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ci): gather APEX builds under the base model entry
The hub for a family is the BASE model entry, never a generated *-apex
parent. Somebody looking for qwen3.6-35b-a3b has to find every build of
those weights under that one name, so a competing qwen3.6-35b-a3b-apex
hub would split the family and leave half of it invisible.
When the gallery already ships the base entry, a variants block is
spliced into it textually, leaving its description, icon, tags,
overrides and files untouched. Only a family whose base model the
gallery does not ship gets a new hub, still named for the base model and
carrying one of the discovered builds as its own payload so it declares
a backend the verifier can judge.
The line editing is factored into .github/ci/galleryedit, shared with
the variantproposals job, so the two cannot drift apart on where a
variants block belongs.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(apexentries): treat an unreadable optional counterpart repo as absent
HuggingFace answers 401 Unauthorized, not 404, for a repository that does
not exist when the request carries no credentials. FetchRepoFiles treated
only 404 as absence, so probing for the OPTIONAL unsloth counterpart hard
failed for every family that legitimately has none: 27 of the 45 APEX
families are community merges that will never have an unsloth build, and a
full run failed all of them.
Split the fetch so the two call sites can apply different policies to the
same response. The APEX repo itself stays strict: a 401 or 403 on a repo
the run requires is a real failure and still errors. Only the optional
probe tolerates it, because without a token 401 cannot be told apart from
absence.
That collapse is lossy in one direction, since a private or gated repo also
answers 401, so the skipped candidates are named in the run summary
alongside the other silent-shortfall counters instead of being dropped in
silence.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ci(apexentries): report full-precision sources as a known exclusion
The 45 APEX repos publish their unquantized F16 sources next to the
imatrix ladder, flat or sharded. Discovery correctly emits nothing for
them, but they were landing in the unclassified total, leaving a
permanent baseline of 24 benign lines on every run.
That baseline is what the unclassified check exists to prevent: a
standing count of known-benign files is exactly what hides the one file
that ever genuinely matters. Count full-precision sources separately and
give them their own summary line, so unclassified returns to 0 and stays
loud when something really is an unknown shape.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(apexentries): namespace local paths by owner and enable MTP builds
localPath namespaced downloads by the repo basename alone, so two repos
publishing the same filename under different owners collapsed to one local
path. LiquidAI/LFM2.5-8B-A1B-GGUF and unsloth/LFM2.5-8B-A1B-GGUF collided that
way, and both were offered from the same hub, so installing the second either
overwrote the first model's weights or was skipped as already present while
recording a sha256 that did not match the bytes on disk. The owner is now its
own path segment: owner/repo is globally unique on HuggingFace and neither half
can contain a separator, so uniqueness holds by construction.
Verify gains a check for the whole class, that no local filename may map to two
different upstream URIs. It surfaces seven pre-existing collisions in the
gallery, which are left alone here.
Entries built from the *-APEX-MTP-GGUF repos now configure MTP rather than
shipping the heads inert, matching the pattern the hand-written MTP entries
already use: spec_type:draft-mtp with spec_n_max and spec_p_min, tagged mtp, and
no draft_model because the heads live in the weights. RenderChild no longer
requires a separate drafter file before it will configure a spec type, while the
cross-repo drafter path is unchanged.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add the APEX GGUF families as variant ladders
Adds the imatrix quality ladder from each mudler/*-APEX-GGUF repo, a fixed
subset of unsloth quant rungs where a counterpart repo exists, and the MTP
builds, then attaches them to the base model entry so one entry offers every
build of the same weights and LocalAI picks the one that fits the hardware.
Ten existing base model entries gain a variants list; twenty-seven families that
the gallery had no base entry for get one. Builds are discovered from the
filenames each repo actually publishes rather than derived from its name, since
six repos ship a stem that differs from their repo name. Every file carries a
sha256 taken from the HuggingFace API.
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(http): make /readyz reflect startup readiness instead of always 200
/readyz was registered as a static handler returning 200 unconditionally,
so it carried no information: it was green whenever it could be reached at
all. Readiness could not distinguish "serving" from "still starting", and
any future change that started the HTTP listener earlier would silently
turn the probe into a lie.
Track startup completion on the Application (atomic flag, flipped at the
very end of New() on the success path only) and have the readiness handler
consult it per request, returning 503 with a small JSON body while startup
is in progress. A nil readiness source fails open so embedders keep the
historical behaviour.
/healthz is deliberately left readiness-independent. Liveness and readiness
answer different questions, and failing liveness during a long preload makes
an orchestrator restart the pod mid-download so the preload never finishes.
This matters because since #10949 the startup preload materializes
HuggingFace artifacts for managed backends: tens of GB for a large model
(31 GB observed on a live cluster). Both probes stay in quietPaths and stay
exempt from auth.
Note the listener is still started only after New() returns, so today the
not-ready state is not observable over HTTP. Moving the listener earlier is
a separate, deliberate decision and is not made here.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(gitignore): anchor the mock-backend pattern so its source dir is traversable
The bare `mock-backend` pattern matched the *directory*
tests/e2e/mock-backend/, not just the binary built into it. Git will not
descend into an ignored directory even for tracked files, so
`git add tests/e2e/mock-backend/main.go` required -f. This was hit while
working on #10970.
Anchor it to the artifact's full path. The built binary stays ignored (it is
also covered by tests/e2e/mock-backend/.gitignore) while the source directory
becomes traversable again.
Verified with `git check-ignore -v`: a new source file under
tests/e2e/mock-backend/ is no longer ignored, and the binary produced by
`make build-mock-backend` still is.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(coverage): raise the coverage ratchet from 48.5% to 54.2%
The committed baseline had drifted well below reality: it still read 48.5%
while a full instrumented run measures 54.2%. A stale-low baseline makes the
gate meaningless — coverage could regress by more than 5 percentage points
and still pass.
Raising a ratchet is a deliberate act, not something to fold into an
unrelated fix, so it gets its own commit. The headroom was earned by tests
landed in #10946, #10947, #10948, #10949, #10956, #10967, #10968, #10970 and
#10975.
Measured with `make test-coverage` on this branch (the same instrumented run
`make test-coverage-baseline` uses: ginkgo over ./pkg and ./core plus the
in-process tests/e2e suite, --covermode=atomic, --coverpkg over core/... and
pkg/..., generated protobuf excluded). The run completed with exit 0 and zero
spec failures; the total was then written with the exact command the
test-coverage-baseline target uses:
go tool cover -func=coverage/coverage.out \
| awk '/^total:/{gsub(/%/,"",$NF); print $NF}' > coverage-baseline.txt
Verified afterwards with scripts/coverage-check.sh, which reports OK.
Note the measured figure includes the readiness specs added earlier on this
branch, so it is a demonstrated floor rather than an estimate.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): EOU-driven semantic_vad turn detection
Add a `semantic_vad` turn-detection mode to the realtime API that feeds
the transcription model live and decides "the user finished speaking"
from the `<EOU>` end-of-utterance token rather than from silence alone.
When EOU fires the turn commits immediately (~0.3s); otherwise it falls
back to an eagerness-scaled silence threshold (low/med/high = 8/4/2s).
Plumbing, bottom to top:
- proto: `AudioTranscriptionLive` bidirectional RPC (config-first oneof,
mono float PCM @16k, ready-ack / Unimplemented degrade signal) plus
`TranscriptResult.eou` for the unary retranscribe gate.
- pkg/grpc: client/server/base/embed scaffolding for the bidi stream,
modeled on AudioTransformStream; release stream conns on terminal Recv.
- parakeet-cpp: live transcription RPC with per-C-call engine locking
(one live stream per turn, finalize+free at commit); bump parakeet.cpp
to ABI v5 — incremental StreamingMel (no more quadratic per-feed mel
recompute that delayed EOU on long turns) and the <EOU>/<EOB> split;
strip the literal <EOU>/<EOB> from offline text and set Eou.
- core/backend: LiveTranscriptionSession wrapper + pipeline
`turn_detection:` config block (type/eagerness/retranscribe).
- realtime: semantic_vad integration — live input captions streamed as
transcription deltas while the user speaks, EOU-immediate commit with
eagerness fallback, optional retranscribe gate (batch re-decode must
also end in <EOU> to confirm), clause synthesis off the LLM token
callback, and per-turn live-transcription / model_load telemetry.
- UI: show the realtime pipeline components as a vertical list.
Docs and tests included; opt-in via the pipeline YAML or per-session
`session.update`. Non-streaming STT backends degrade to silence-only.
Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Write] [Bash]
Assisted-by: Claude Code:claude-fable-5 [Read] [Edit] [Bash]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(realtime): explicit formally-verified state machines + parakeet streaming driver
The realtime API had several implicit state machines whose state was inferred
from scattered booleans, channels, and five separate mutexes, leaving
illegal/inconsistent states reachable. Make them explicit and keep the
implementation in step with a formal design; rework the parakeet streaming
backend along the same lines.
Realtime state machines (M1-M5). Each is a sealed sum-type State/Event/Effect
with a total, pure Next(state,event)->(state,[]effect) behind a single-writer
Coordinator:
M1 conncoord connection lifecycle: VAD toggle + once-only teardown
(replaces vadServerStarted + a `done` channel closed from
two sites).
M2 turncoord turn detection: collapses speechStarted and the live-stream
"turn open" flag into one state, so discardTurn can no longer
desync them and suppress the next onset.
M3 respcoord response coordination: serializes the dual-writer
start/cancel so at most one response is live; one
response.done per response.create.
M4 compactcoord conversation compaction: single-flight (replaces the
`compacting atomic.Bool` CAS).
M5 ttscoord TTS pipeline: open->closing->closed, idempotent wait(),
rejects enqueue-after-close (was a silent drop).
The Coordinator/Sink/Next plumbing — only the sealed types and Next differed
per machine — is extracted once into core/http/endpoints/openai/coordinator as
a generic Coordinator[S,E,F]; each machine keeps its public API via type
aliases, so no sink, call-site, or test moved.
Hierarchy. session_lifecycle.fizz models M1 as the parent region with its
children (M2/M3/M4) as one statechart and asserts ChildrenDieWithParent (conn
torn => all children terminal, none start after teardown). respcoord and
compactcoord gain an absorbing Terminated state + Shutdown event; conncoord's
teardown drives the children terminal. This closes a compaction teardown gap: a
fire-and-forget compaction could outlive a torn session — compactionSink now
takes a session-scoped cancellable context + WaitGroup and joins the in-flight
summarize+evict on shutdown.
Formal verification. formal-verification/ holds one authoritative FizzBee spec
per machine plus the composition spec, each with an always-assertion and a
documented one-line edit that makes the checker fail (verified non-vacuous).
scripts/realtime-conformance.sh is fail-closed: all Go conformance suites under
-race AND a model-check of every .fizz spec; a missing FizzBee is a hard error
(only the loud REALTIME_CONFORMANCE_SKIP_FIZZBEE=1 bypasses it, never in CI).
FizzBee is pinned by sha256 and installed via scripts/install-fizzbee.sh into
.tools/ (gitignored). Wired as make test-realtime-conformance, a CI workflow,
and a pre-commit path filter. Go conformance tests are Ginkgo/Gomega (per the
repo's forbidigo lint): transition tables + fixed-seed property walks +
concurrent/-race specs, no rapid dependency. Design map:
docs/design/realtime-state-machines.md.
Parakeet streaming backend. The same treatment applied to the parakeet-cpp
streaming paths:
- AudioTranscriptionStream returns codes.Unimplemented for non-streaming models
instead of decoding offline and emitting it as one delta + final. A client
that asked for streaming learns the model cannot stream rather than receiving
a batch result shaped like a stream. New grpcerrors.StreamTranscriptionUnsupported
carries that signal; the HTTP /v1/audio/transcriptions stream path surfaces it
as an SSE error event. Mirrors AudioTranscriptionLive, which already did this.
- utteranceBoundary (boundary.go): a single definition of the end-of-utterance
latch, replacing three open-coded finalEou toggles. Modelled as a two-valued
type so illegal states are unrepresentable.
- Shared decode driver (driver.go): streamFeedResult (one per-feed event) +
feedChunk (hides the ABI v4 JSON vs text-only split) + feedSlices + flushTail.
The feed loop is written once.
- AudioTranscriptionLive becomes a bidi adapter: it streams the per-feed
{delta,eou,eob,words} the realtime turn detector consumes and a terminal
FinalResult carrying only Text. Segments/duration/eou are offline-only and no
longer produced (nor read) on the live path; liveTraceState drops the terminal
eou and keeps the per-feed eou_events count.
- AudioTranscriptionStream + streamJSON merge into one driver-based function;
streamSegmenter is generalized to the unified event with a text-only fallback
that preserves the legacy (no-words) library's per-utterance segmentation.
Verified: build/vet/gofumpt clean, golangci-lint 0 issues, all coordinator and
parakeet packages under -race, the fail-closed conformance gate green, and
make test-realtime (12 e2e WS+WebRTC).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Produce a Gatekeeper-clean macOS distribution with no user workaround:
- Launcher DMG + the LocalAI.app inside it are built via fyne, codesigned
with the Developer ID under the hardened runtime, then the DMG is signed,
notarized (notarytool) and stapled. Replaces macos-dmg-creator (which had
no signing hook) with fyne package + hdiutil so we control the .app before
packaging.
- The bare local-ai darwin server binary is signed + notarized via
GoReleaser's native notarize block (quill backend, runs on Linux).
- All signing is gated on secrets being present, so forks/PRs/local builds
stay unsigned and green (contrib/macos/sign-and-notarize.sh no-ops).
- Add hardened-runtime entitlements and FyneApp.toml for deterministic
packaging; update macOS install docs to drop the quarantine workaround.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
- Strict monotonic Go coverage gate (make test-coverage-check, 45% baseline)
run in CI; fixes ginkgo dropping all-but-one coverprofile across multiple
recursive roots, builds with -tags auth, and folds in the in-process
tests/e2e suite via --coverpkg.
- React UI e2e coverage (make test-ui-coverage: vite-plugin-istanbul + nyc,
nix-provided Chromium) plus e2e specs for 6 previously-untested pages, and a
UI coverage gate (make test-ui-coverage-check) with a small tolerance since
e2e line coverage jitters ~0.5pp run-to-run.
- pre-commit hook: lint + coverage on Go changes, Playwright e2e + UI coverage
gate on react-ui changes; install with make install-hooks.
- New Go handler tests (settings, branding), hermetic base64 download test.
- fix(ui): model editor reads vram_display (snake_case), so the VRAM estimate
renders again; covered by a regression test.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Add a routing middleware stack and a cloud-proxy backend.
* cloud-proxy: a Go gRPC backend that forwards OpenAI- and
Anthropic-shaped chat requests to upstream providers, with an
optional translate mode (OpenAI request -> Anthropic /v1/messages
-> OpenAI response) and full tool-calling support.
* routing: admission control, content-aware model routing
(embedding cache + classifier + rerank + Arch-Router score),
PII detection/redaction (regex + NER) with streaming filter and
OpenAI/Anthropic adapters, and a per-user/per-key billing recorder
backed by GORM or in-memory storage.
* middleware: UsageMiddleware records usage via the billing recorder,
plus admission, route-model, usage-stamp and trace middlewares.
* observability: BackendTrace ring buffer stores full request bodies
(capped), MITM proxy emits structured trace events, and router
classifier decisions surface at /api/router/decide.
* gallery: Arch-Router-1.5B (Q4_K_M and Q8_0).
* UI: cloud-proxy model-editor fields, classifier system-prompt and
score-normalization config, and a Traces page rendering request
bodies.
Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore: ignore local .worktrees directory
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(openai): stream usage non-zero when tools are enabled
The streaming chat-completions worker for tool-bearing requests
(processTools in core/http/endpoints/openai/chat.go) never forwarded the
cumulative TokenUsage from ComputeChoices to the chunks it placed on the
responses channel. The outer streaming loop's running usage tracker
therefore stayed at the zero value, and the include_usage trailer
reported {prompt_tokens:0, completion_tokens:0, total_tokens:0} whenever
the request carried a `tools` array. Without tools, the alternative
`process` path stamps Usage on every chunk, so that path was unaffected.
Forward the final TokenUsage via a usage-only sentinel chunk (empty
Choices, populated Usage) emitted right before close(responses). The
outer loop's per-chunk Usage capture moves above the empty-Choices skip
so the sentinel updates the tracker without ever reaching the wire,
keeping the existing OpenAI spec contract (intermediate chunks carry no
`usage` field, and the deferred-final-chunk helpers remain Usage-free
per the regression test for issue #8546).
Adds streamUsageFromTokenUsage, usageSentinelChunk, and
applyChunkToUsage helpers with focused Ginkgo coverage plus a flow-level
test that mirrors the outer-loop sequence.
Fixes#9927
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4-7 [Claude Code]
* refactor(openai): return final TokenUsage from stream workers
Replace the usage-only sentinel SSE chunk introduced in the previous
commit with a plain return value. The streaming workers process and
processTools (now extracted as package-level processStream and
processStreamWithTools) return (backend.TokenUsage, error); the outer
ChatEndpoint loop reads the cumulative counts off the existing `ended`
channel (now carrying streamWorkerResult{usage, err}) and builds the
include_usage trailer from a normal Go value after the LOOP exits.
This drops the empty-Choices "skip but capture Usage" rule from the
outer loop and removes the usageSentinelChunk / applyChunkToUsage
helpers entirely. The SSE responses channel is back to a single
purpose: wire chunks only.
processStream and processStreamWithTools move into chat_stream_workers.go
so they can be exercised directly from tests. The chat_stream_usage_test.go
suite now drives the workers with a mocked backend.ModelInferenceFunc
and asserts on the returned TokenUsage. The regression coverage for
issue #9927 is therefore behavioral: reverting the fix (discarding
ComputeChoices' usage return) makes the assertions fail with concrete
count mismatches.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4-7 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
AIO images are behind, and takes effort to maintain these. Wizard and
installation of models have been semplified massively, so AIO images
lost their purpose.
This allows us to be more laser focused on main images and reliefes
stress from CI.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(realtime): WebRTC support
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(tracing): Show full LLM opts and deltas
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* WIP response format implementation for audio transcriptions
(cherry picked from commit e271dd764bbc13846accf3beb8b6522153aa276f)
Signed-off-by: Andres Smith <andressmithdev@pm.me>
* Rework transcript response_format and add more formats
(cherry picked from commit 6a93a8f63e2ee5726bca2980b0c9cf4ef8b7aeb8)
Signed-off-by: Andres Smith <andressmithdev@pm.me>
* Add test and replace go-openai package with official openai go client
(cherry picked from commit f25d1a04e46526429c89db4c739e1e65942ca893)
Signed-off-by: Andres Smith <andressmithdev@pm.me>
* Fix faster-whisper backend and refactor transcription formatting to also work on CLI
Signed-off-by: Andres Smith <andressmithdev@pm.me>
(cherry picked from commit 69a93977d5e113eb7172bd85a0f918592d3d2168)
Signed-off-by: Andres Smith <andressmithdev@pm.me>
---------
Signed-off-by: Andres Smith <andressmithdev@pm.me>
Co-authored-by: nanoandrew4 <nanoandrew4@gmail.com>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
* Add launcher (WIP)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Update gomod
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Cleanup, focus on systray
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Separate launcher from main
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add a way to identify the binary version
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Implement save config, and start on boot
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Small fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Save installed version as metadata
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Stop LocalAI on quit
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fix goreleaser
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Check first if binary is there
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* do not show version if we don't have it
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Try to build on CI
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* use fyne package
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add to release
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fyne.Do
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* show WEBUI button only if LocalAI is started
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Default to localhost
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* CI
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Show rel notes
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Update logo
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Small improvements and fix tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Try to fix e2e tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat: split remaining backends and drop embedded backends
- Drop silero-vad, huggingface, and stores backend from embedded
binaries
- Refactor Makefile and Dockerfile to avoid building grpc backends
- Drop golang code that was used to embed backends
- Simplify building by using goreleaser
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(gallery): be specific with llama-cpp backend templates
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(docs): update
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(ci): minor fixes
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore: drop all ffmpeg references
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix: run protogen-go
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Always enable p2p mode
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Update gorelease file
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(stores): do not always load
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fix linting issues
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Simplify
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Mac OS fixup
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Build llama.cpp separately
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* WIP
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* WIP
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* WIP
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Start to try to attach some tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add git and small fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix: correctly autoload external backends
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Try to run AIO tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Slightly update the Makefile helps
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Adapt auto-bumper
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Try to run linux test
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add llama-cpp into build pipelines
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add default capability (for cpu)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Drop llama-cpp specific logic from the backend loader
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* drop grpc install in ci for tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Pass by backends path for tests
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Build protogen at start
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(tests): set backends path consistently
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Correctly configure the backends path
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Try to build for darwin
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* WIP
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Compile for metal on arm64/darwin
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Try to run build off from cross-arch
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add to the backend index nvidia-l4t and cpu's llama-cpp backends
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Build also darwin-x86 for llama-cpp
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Disable arm64 builds temporary
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Test backend build on PR
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fixup build backend reusable workflow
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* pass by skip drivers
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Use crane
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Skip drivers
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* x86 darwin
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Add packaging step for llama.cpp
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Fix leftover from bark-cpp extraction
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Try to fix hipblas build
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(bark-cpp): add new bark.cpp backend
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* build on linux only for now
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* track bark.cpp in CI bumps
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Drop old entries from bumper
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* No need to test rwkv specifically, now part of llama.cpp
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* stash initial fixes, attempt to open branch inside container
Signed-off-by: Dave Lee <dave@gray101.com>
* add yq, from inside DC
Signed-off-by: Dave Lee <dave@gray101.com>
* stash progress, rebuild container
Signed-off-by: Dave Lee <dave@gray101.com>
* snap
Signed-off-by: Dave Lee <dave@gray101.com>
* split builder into builder-sd, will speed up devcontainer build times and potentially help caching in other situations.
Signed-off-by: Dave Lee <dave@gray101.com>
* fix yq
Signed-off-by: Dave Lee <dave@gray101.com>
* fix paths
Signed-off-by: Dave Lee <dave@gray101.com>
* fix paths - new folder to bypass the .dockerignore which _should_ exclude the other files
Signed-off-by: Dave Lee <dave@gray101.com>
* fix
Signed-off-by: Dave Lee <dave@gray101.com>
* fix ]
Signed-off-by: Dave Lee <dave@gray101.com>
---------
Signed-off-by: Dave Lee <dave@gray101.com>
* WIP
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* gen a static page instead (we force DNS redirects to it)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): install models from CLI, unify install
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Uniform graphic of model page
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Makefile: update targets
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Slightly enhance gallery view
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* models(gallery): add mistral-0.3 and command-r, update functions
Add also disable_parallel_new_lines to disable newlines in the JSON
output when forcing parallel tools. Some models (like mistral) might be
very sensible to that when being used for function calling.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* models(gallery): add aya-23-8b
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* core 1
* api/openai/files fix
* core 2 - core/config
* move over core api.go and tests to the start of core/http
* move over localai specific endpoints to core/http, begin the service/endpoint split there
* refactor big chunk on the plane
* refactor chunk 2 on plane, next step: port and modify changes to request.go
* easy fixes for request.go, major changes not done yet
* lintfix
* json tag lintfix?
* gitignore and .keep files
* strange fix attempt: rename the config dir?
* refactor: rename llama-stable to llama-ggml
* Makefile: get sources in sources/
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixup path
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixup sources
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixups sd
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* update SD
* fixup
* fixup: create piper libdir also when not built
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix make target on linux test
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
