PTQ1_0 is a Prism-private GGUF type (GGML_TYPE_PTQ1_0 = 143 in the
PrismML llama.cpp fork), so stock llama-cpp cannot load it. Switch to
the bonsai backend like the existing ternary-bonsai-27b entries, and
replace the scraped Qwen3.8 description and icon.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Older Go linkers stamp pure-Go hosts with SDK metadata that disables
modern Metal APIs. Select Go 1.27 for Darwin builds and document the
backend rebuild requirement.
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The link text still said chatbot-ui, but it now pointed at the examples
repository root. Link the configurations directory, which holds the
example model config files, and describe it as such.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
The entry enables spec_type:draft-mtp, so variant ranking needs the mtp
tag. Replace the scraped model-card description, set the Swift Open
License v1.0 and link the base model repo.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
* ⬆️ Update TheTom/llama-cpp-turboquant
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(turboquant): drop the upstreamed D512 patch
Upstream c0e227c guards D512 declarations, dispatch, and instances with
GGML_USE_HIP. This prevents the CUDA shared-memory overflow that our
patch addressed. The old patch now rejects the guarded source.
Remove the obsolete patch for the pinned a3d5603d revision. The remaining
patch series applies successfully, and the build-target test passes.
Assisted-by: Codex:gpt-6
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Picks up mudler/LocalAGI 3ce0a08 "fix(mcp): re-dial an MCP session the
server has dropped". Agents open their MCP sessions once, when they are
created; when the MCP server restarts it forgets them and the go-sdk
client does not reconnect by itself, so the agent kept a dead session -
or, behind a server that revives unknown session IDs, a stale tool
list - until LocalAI restarted.
Only go.mod/go.sum change; core/services/agentpool builds and vets
against the new version.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
Code scanning reports alerts on every line of a touched file. Mark the
gRPC auth env var name and the mock backend's staged-path reads as
reviewed, and log the error when evicting a model after its connection
fails instead of dropping it.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add IQ4_XS and Q4_K_M GGUF builds with a BF16 vision projector.
Pin verified artifacts and document installation and variant selection.
Assisted-by: Codex:gpt-6
Review asked for constants instead of the stage literals ("vad",
"transcription", "llm", "tts", "sound_detection") passed to resolveStage,
stageCall and isChainStage, so the uses can be cross-checked. Add
PipelineStage* constants next to the Pipeline type in core/config: the
names match its yaml keys, and core/backend (preload roles) and the openai
realtime endpoint both need them.
Use them in realtime_model.go (stage routing and preload roles),
realtime.go (the voice_recognition preload role) and core/backend
preload.go. model_failover events take their stage from the stageChains
keys, so they now carry the constants too. The failover tests use the
constants for inputs and keep literal wire values in their event
assertions.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review asked for constants instead of the "cloud-proxy" and
"localai-proxy" literals. Add CloudProxyBackend and LocalAIProxyBackend next
to the other backend-name constants in pkg/model (WhisperBackend,
TransformersBackend, ...), which core/config already imports, and use them
in every production check: the proxy options builder, IsRemoteProxy,
IsCloudProxyBackendPassthrough, the PII defaults, the localai-proxy backend
hook and loader warning, and the PII middleware metadata in the routes and
the in-process MCP client.
The proxy options builder now calls IsRemoteProxy() instead of repeating
the two-backend check, so the set of proxy backends is defined once.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Code scanning flagged seven issues in the new backend:
- G115 text.go: tool-call indexes and tokenize lengths come from the
upstream server as int and were cast straight to int32. Add clampInt32 so
an absurd upstream value saturates instead of wrapping.
- G115 live.go: the int16 -> uint16 cast in PCM16 encoding is a deliberate
two's-complement reinterpretation of an already clamped sample; mark it
with #nosec and say so.
- G304 proxy.go, media.go, client.go: api_key_file comes from the model
config, and the media input and output paths are files core staged or
chose for the call. None are caller-supplied. Clean the paths and add
#nosec with that reason, as core/gallery and the sound classification
endpoint already do.
- G306 media.go: write generated media 0o600. Core runs as the same user
and serves the file itself.
gosec reports 0 issues for backend/go/localai-proxy and
core/services/failover. The G104 once reported for failover/prober.go is
no longer present.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
TranscriptEndpoint, DiarizationEndpoint and SoundClassificationEndpoint
returned the raw c.FormFile("file") error. Echo turns a non-HTTPError into
a 500, so a request with no multipart boundary or no file field looked like
a server fault. This became visible in tests/e2e once the suite registered
a transcription-capable model (lp-transcription) at runtime: the
default-model middleware then resolves a model, the request reaches the
handler, and "should return mocked transcription" got a 500 in some spec
orders (seed 1790493709).
Read the upload through a small uploadedFile helper that maps any
FormFile failure to 400 with the field name and parser reason. Server-side
failures after that (temp dir, file create, copy) stay 500. The image and
LocalAI upload endpoints already return 400 here.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
errcheck flagged tests/e2e/realtime_ws_test.go:307 as new on this branch
(the localai-proxy realtime failover spec) once the e2e lint scope widened.
Same house style the maintainer bot used for the other close results in
this file (19f6a18b8): wrap in a deferred closure so the error is explicitly
discarded rather than silently ignored.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
19f6a18b8 moved credential-env resolution behind an explicit lookup
(ApplicationConfig.ProxyAPIKeyEnvLookup / config.WithProxyAPIKeyEnvLookup),
wired only at the CLI boundary (core/cli/run.go). The e2e suite builds its
Application in-process without that option, so the failover prober could
never resolve a remote target's api_key_env, remote liveness never passed,
and "fails over ... and fails back" hung waiting for chain-remote to
recover. Pass config.WithProxyAPIKeyEnvLookup(os.Getenv) there too, same as
the CLI. worker/federated commands don't serve proxy/failover configs and
tests/e2e-ui never sets api_key_env, so neither needs the lookup.
Also make the misconfiguration itself easier to diagnose: the prober now
logs a one-time xlog.Warn per api_key_env when a remote target sets it but
no lookup is configured, instead of only surfacing it as a per-probe
"is unset" error indistinguishable from a genuinely empty env var.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the localai-proxy known limits (no grammar or media forwarding,
TTS streams that end cleanly after an upstream failure, the /v1 path in
upstream_url), state that the Unimplemented skip covers the APIs that
answer HTTP 501, and describe a NATS-partitioned leader and pin
re-sync in distributed mode.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The failover prober gate calls TryAcquire with the application context,
which never ends. A database that stopped answering blocked the
scheduler goroutine for good, with the lock's mutex held. Bound the
session open and the lock query with the same timeout as Verify.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Every frontend recorded localai_failover_switches_total for the switches
it adopted from the leader, so a cluster of N frontends counted each
switch N times. Only the leader, which decides the switch, records it.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Pin and Unpin apply the change locally first for read-your-writes. When
the shared write then failed, the local pin stayed, so this frontend
served a target the others did not. Restore the previous pin on error,
unless a newer change arrived meanwhile.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A NATS reconnect re-hydrates the pins map from the DB without OnApply, so
a frontend that missed an unpin kept serving the old pin and flip-flopped
with the leader's republish. Every frontend now reconciles the manager's
pins with the shared set every ten ticks and after a reconnect, and the
pins map re-reads the DB every 30 s to repair a delta dropped without a
reconnect.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The failover prober cuts upstream_url at /v1, but Load kept the path, so
an upstream_url ending in /v1 probed healthy while every request went to
/v1/v1/... and got a 404, which never trips the target. Cut the path at
/v1 in Load too, with a warning.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
ResourceExhausted is retried and trips the target, which is right for a
rate limit or an out-of-memory backend. A payload over the gRPC message
cap is ResourceExhausted too, but every target rejects it the same way,
so it tripped the whole chain. Classify it as a request error.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The non-OpenAI endpoints (depth, detection, face_*, voice_*, images,
video, 3d) map a backend's gRPC Unimplemented to an echo 501 without the
gRPC status. The retry loop did not see a capability gap, and IsRetryable
is false for 501, so the client got 501 and the next target was never
tried. Treat a returned or written 501 as a capability gap.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The failover chain editor leaves the warm toggle enabled on every row
(the model list has no backend field to gate on) and relies on the
server warning instead, so the docs describing it as disabled for
remote targets were wrong. Separately, syncstate's hydrate() returns
early with no Store or Loader, so a Reconcile tick is a no-op rather
than one that empties the map — correct that claim everywhere it was
repeated (contributor guide, distsync comment, design spec).
No behavior change.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the distributed-aware state contributor rule: any feature that
keeps runtime state must choose shared (syncstate), single-runner
(advisorylock), stateless, or documented per-instance behaviour, so it
behaves correctly across multiple frontends instead of diverging
silently. Also sweeps the failover/localai-proxy docs for gaps found
along the way: the UI (chain editor field, health strip, overview
page, chain badge), the 429->ResourceExhausted trip and 501->skip
mappings, and a spec correction for the live-transcription bridge's
actual close behavior.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Editing a failover chain now shows its state, the target serving it,
and a per-target health table under the editor header. The strip reads
GET /api/failover and follows /api/failover/events, with a 15 s re-list
to cover SSE reconnect gaps. Admins can pin a target or unpin the chain
after a confirmation.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The e2e suite now registers the localai-proxy binary and points proxy
models back at the test server itself, so a request leaves LocalAI
through the backend, returns over REST and is answered by a mock model.
Chat, embeddings, TTS and transcription through the proxy return the
upstream model's answer; a chain whose proxy target's upstream model
fails to load serves from the local target; and a realtime pipeline
whose LLM stage is a chain on a remote target completes a turn, then
switches to the local target with a localai.model.failover trip event
when a gate in front of the upstream starts answering 503.
The docs describe the localai-proxy backend next to cloud-proxy and add
a per-stage remote LocalAI example to the failover page.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A caller that gives up before the ready ack now ends the call with Canceled
and closes the upstream socket, and setup has a 3 minute default bound when
request_timeout_seconds is unset, so a hung upstream cannot hold the call
or block failover. A closing session now waits only for a turn the upstream
committed (or is still speaking), not for turns it discarded. Audio held
before the ready ack is capped at 5 s, and NaN samples become silence.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
AudioTranscriptionLive opens <upstream>/v1/realtime?model=<realtime_pipeline>
as a transcription session with server VAD, forwards PCM as base64 PCM16
appends, and maps transcription deltas and completions to Delta/Eou. Closing
the send side waits briefly for an in-flight utterance, then sends the final
text. An upstream error, failed transcription or disconnect ends the stream
with Unavailable so failover reopens on the next target.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Detect/Depth/FaceVerify/FaceAnalyze forwarded the bare base64 core hands the
backend, but the upstream's own REST handlers only accept a URL or a
data:...;base64, string, so every real call 400'd. Wrap the payload as a
data URI (sniffing its MIME type) before sending it.
Depth requests for exports/dst now return Unimplemented: those files are
written to the upstream's own local disk and are unreachable from here, so
failover should move to a local target instead.
Generation replies that hand back a URL are now re-fetched by path only,
checked against the upstream's known generated-content prefixes, instead of
stripping the configured base as a literal string prefix — the old approach
broke (or silently trusted an arbitrary host) the moment the upstream
advertised a different base via LOCALAI_BASE_URL, a reverse proxy, or
X-Forwarded-Host.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The proxy now forwards TTS, streaming TTS, sound generation,
transcription (plain and streaming), diarization, VAD, sound
detection and audio transforms to the upstream LocalAI.
Streaming TTS passes the upstream WAV bytes through unchanged. A
streaming transcription that stops before its final frame, or sends
an error frame, fails with Unavailable instead of ending as a short
success. Transcription always sends diarize, because the upstream
treats a missing field as true. Audio transforms also download the
separation stems the upstream names and write them beside Dst.
Sound generation from a source clip returns Unimplemented, because
the REST endpoint has no field for the clip.
The multipart helper now takes repeated fields and several files.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Treating ResourceExhausted as a capability gap skipped the target
without counting a failure, so a target that stays rate limited or out
of memory kept its traffic. It is now an ordinary retryable failure:
the request moves to the next target and the exhausted one trips.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Rerank no longer sends top_n 0, which the upstream rejects. A
mid-stream upstream error frame now fails the call instead of ending
it as a short success. Temperature 0 is forwarded. An upstream 429
becomes ResourceExhausted, which failover skips like Unimplemented.
A localai-proxy config sends its own name upstream when upstream_model
is unset, and a chat proxy defaults to the tokenizer template so chat
reaches the upstream as messages.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Ettore Di Giacinto