Add official Q4_K_M and Q8_0 builds for memory-writing workflows.
Use the embedded template with no-think mode and pinned downloads.
Document installation and variant selection.
Assisted-by: Codex:gpt-6
(cherry picked from commit a1259642c5, mudler/LocalAI#12851)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Offer the official Q4_K_M and Q8_0 builds with their vision projector.
Pin artifact revisions and document installation and variant selection.
Assisted-by: Codex:gpt-6
(cherry picked from commit 9f98baa8e5, mudler/LocalAI#12833)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add IQ4_XS and Q8_0 MAX builds with pinned downloads and linked variants.
Use the embedded chat template and document installation.
Assisted-by: Codex:gpt-6
(cherry picked from commit 94b54e009a, mudler/LocalAI#12831)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add three GGUF quantizations with a shared vision projector and pinned
artifact URLs. Document installation and the default context size.
Assisted-by: Codex:gpt-6
(cherry picked from commit c8df4d6fef, mudler/LocalAI#12823)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the publisher Q4_K_M and Q8_0 builds with pinned downloads and
verified SHA256 hashes. Document installation and variant selection.
Assisted-by: Codex:gpt-6
(cherry picked from commit 64d0b3b094, mudler/LocalAI#12821)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Use the published mmproj filename for both uncensored Q4 and Q8 entries.
The old filename returns 404, preventing model installation.
Preserve the local projector path and existing checksum.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
(cherry picked from commit 80b0708da4, mudler/LocalAI#12820)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add IQ4_XS and Q6_K_P builds with pinned weights and linked variants.
Document installation and the text-only, ordinary-decoding defaults.
Assisted-by: Codex:gpt-6
(cherry picked from commit 1c5eea9580, mudler/LocalAI#12819)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add pinned UD-Q4_K_XL and Q8_0 builds with the vision projector and installation documentation.
Assisted-by: Codex:gpt-6
(cherry picked from commit 31978a28ef, mudler/LocalAI#12809)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Offer Unsloth dynamic Q4 and Q6 builds alongside the official GGUFs.
Pin their revisions and verify hashes through two Hugging Face sources.
Assisted-by: Codex:gpt-6
(cherry picked from commit ab8d685f01, mudler/LocalAI#12774)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add PQ2_0, PTQ1_0, and Q2_0 builds with the Q8_0 vision projector.
Pin files to the publisher revision and document variant selection.
Assisted-by: Codex:gpt-6
(cherry picked from commit 7270bdc508, mudler/LocalAI#12773)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add three GGUF packings with the Q8_0 vision projector and document
installation. Pin asset revisions and verify hashes against HF metadata
and download headers.
Assisted-by: Codex:gpt-6
(cherry picked from commit 77f1ab29c8, mudler/LocalAI#12771)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Fix the existing NVFP4 variant to use its downloaded BF16 projector.
Assisted-by: Codex:GPT-6 [shell]
(cherry picked from commit 3bfed3b8a0, mudler/LocalAI#12769)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Offer three GGUF builds and a safetensors build for English retrieval.
Pin model files and document the retrieval query prefix.
Assisted-by: Codex:gpt-6
(cherry picked from commit 02413e8190, mudler/LocalAI#12722)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add F16, Q8_0 and Q4_K_M builds with CLS pooling and retrieval usage documentation. Pin the GGUF revision and verify SHA256 values against Hugging Face metadata and download headers.
Assisted-by: Codex:gpt-6
(cherry picked from commit e61c44f317, mudler/LocalAI#12717)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add F16, Q8_0 and Q4_K_M builds for English retrieval. Pin the source
revision and verify each SHA256 against HF metadata and download headers.
Document query prefixes and the 512-token context.
Assisted-by: Codex:gpt-6
(cherry picked from commit b99b669cef, mudler/LocalAI#12713)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add text-only Q4 and Q5 builds for llama.cpp with pinned downloads.
Group the builds as variants and document their supported capabilities.
Assisted-by: Codex:gpt-6
(cherry picked from commit c7cbe18a62, mudler/LocalAI#12707)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4_K_M, Q6_K, and Q8_0 builds for llama.cpp with pinned assets.
Group the builds and document installation, reasoning, and tool setup.
Assisted-by: Codex:gpt-6
(cherry picked from commit f9e7f0bfbe, mudler/LocalAI#12706)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add AtomicChat MiniCPM-V 4.7 35B-A3B Q4 and Q6 builds for llama.cpp.
Include every weight shard and the F16 vision projector.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
(cherry picked from commit eff464d58a, mudler/LocalAI#12690)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Kujira's mixed IQ2 Saluki fine-tune for llama.cpp with plain and MTP
decoding. Pin the weights and document the text-only entries.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
(cherry picked from commit 24f23a66aa, mudler/LocalAI#12688)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4 and IQ4 GGUF builds with a BF16 vision projector and pinned
Hugging Face downloads. Keep the original release available.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
(cherry picked from commit 9c9e18450e, mudler/LocalAI#12670)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Offer Maya S, S24, M, and L builds through the existing GLM group.
Pin all weight shards and the vision projector to an HF revision.
Document the standard llama.cpp configuration and its limits.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
(cherry picked from commit 4aab47ba66, mudler/LocalAI#12666)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Offer the Saluki IQ2 weights with an MTP head for llama.cpp. Pin both
assets and document how to select the variant.
Assisted-by: Codex:GPT-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
(cherry picked from commit c745b96781, mudler/LocalAI#12662)
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): adapt to the multimodal rerank signature
Upstream llama.cpp added query_files and doc_files parameters to
format_prompt_rerank() (ggml-org/llama.cpp#29625). The gRPC rerank
call passed the init options in their place, so grpc-server.cpp no
longer compiled against the new pin.
prepare.sh now probes server-common.h for query_files and writes
LOCALAI_HAS_RERANK_FILES into llama_compat.h. When it is set, the
rerank call passes two empty media lists. The rerank API is text
only, so behaviour is unchanged. Forks that still use the old
signature (turboquant, bonsai) compile as before.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The chat settings sheet, the Talk session sheet and the agent and task
preview sheets set up their focus handling in an effect that depended on
onClose. Every parent passes onClose as an inline arrow, so each re-render
of the page gave it a new identity: the effect cleanup sent focus back to
the opener and the effect ran again and focused the Close button.
Any edit in a sheet re-renders the page, so focus left the control being
used. In the chat sheet, typing in the system prompt moved focus to Close
after the first key, and a following space activated Close and dismissed
the sheet; an open native dropdown in the sheet closed by itself.
Keep the latest onClose in a ref and run the focus effect once per open.
Escape still calls the current onClose; initial focus on Close, the Tab
trap and restoring focus to the opener are unchanged.
The e2e specs type key by key (or switch preview tabs) and assert that
focus stays put; they fail without this change.
Docs-Not-Needed: bug fix, no user-facing behaviour change beyond the fix
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit 773cae7259)
Agent jobs (/api/agent/jobs, local and on distributed agent workers) run
cogito against a LocalAI client that sends no max_tokens of its own, so
every completion gets the client's fixed default of 16384. max_iterations
bounds the number of turns and loop_detection repeated tool calls, but
neither stops one completion that keeps generating, and 16384 is the same
for a small tool-picking model and a reasoning model that thinks at length.
Add agent.max_generation_tokens to the model config. When set, both job
paths apply it to the client with SetMaxTokens; requests that carry their
own cap still win, so cogito's length retry can raise it. Unset keeps
today's behaviour.
The specs run each job path against an MCP fixture and a recording LLM
endpoint and check the max_tokens the completions carry; they fail when
either call site drops the cap.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit 5600a1cabe)
The vLLM and SGLang backends no longer stop at 200 tokens when a request
has no max_tokens. Generation is now bounded by the context window.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
- ApplyReasoningEffort keeps an operator's reasoning.disable for high, xhigh and max
- sglang: _configured_budget() feeds the strict-thinking warning; tested with
effort_budgets alone
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 45a960ca1f)
- warn on an effort_budgets entry without a token count (high=)
- update comments: enable_thinking=false still wins over a listed level
- document that precedence in text-generation.md
- cover xhigh/max and the empty-entry warning in the tests
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 0bd723a652)
The Go side already forwards the request's effort as
Metadata["reasoning_effort"], but the sglang backend only read
enable_thinking: every level got the same model-level thinking_budget,
so low/medium/high produced identical reasoning.
- New model option `effort_budgets:low=…;medium=…;high=…` picks the
budget of the request's level; unlisted levels and requests without an
effort keep `thinking_budget`. Parsing reuses _parse_thinking_budget.
- A level listed in effort_budgets turns reasoning on for that request,
so levels the Go side does not know cannot silently disable it.
- ApplyReasoningEffort now treats `xhigh` and `max` as thinking levels.
Before, they left DisableReasoning unset, so with a model default of
"off" asking for more reasoning produced none.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 879c686ab1)
Mock the request constructor so fake parser tests can run on older
vLLM profiles without ParserManager. Document the streaming version
requirement and fallback behavior.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit ff0130fa89)
With a reasoning parser configured, the streaming path sent every raw
text delta as ChatDelta.content. LocalAI uses the backend's ChatDeltas as
they are, so with thinking enabled the reasoning arrived as
delta.content, and the final chunk then repeated the whole answer and
the reasoning a second time. Non-streaming requests were split correctly.
Use vLLM's composed parser (ParserManager.get_parser, vLLM >= 0.30), the
one vLLM's own OpenAI server streams with: one instance per request,
built with the request's chat_template_kwargs (enable_thinking), fed
each delta via parse_delta. Reasoning, content and tool calls go out in
their own fields as they are produced; the final chunk carries only
token counts and logprobs.
If the parser raises mid-stream, the rest of the output is buffered and
the final chunk sends what the full-text parse has beyond what was
already streamed (suffixes per field and per tool-call index). Where the
full parse disagrees with what went out, nothing is guessed and the
mismatch is logged.
Only active when a reasoning_parser is set; without one, or on a vLLM
without ParserManager, streaming is unchanged.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 5392eb813e)
Both Python backends start from a fixed default (sglang max_new_tokens=200,
vLLM max_tokens=200) and only replace it when the request carries Tokens.
OpenAI clients that leave max_tokens out (several agent frameworks do) were
silently truncated after 200 tokens, and the response still said
finish_reason "stop": the Go side only reports "length" against a budget
the client requested, so it cannot see a backend-side cap.
Use None instead, which allows generation up to the remaining context.
That is what sglang's own OpenAI server does when max_tokens is omitted
(the scheduler turns None into the remaining request capacity; dropping
the key would select SamplingParams' own default of 128), and vLLM treats
max_tokens=None the same way. A requested max_tokens still wins.
Tests check both cases and that the None default passes each library's own
parameter validation.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 5bdab54d97)
Accept response output text when clients replay it as request input.
Keep assistant history in both prompt content fields and preserve user
message boundaries when merging adjacent assistant messages.
Assisted-by: Codex:GPT-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
(cherry picked from commit 1cc5336aaa)
Restoring a snapshot after a failed stop lost the request lifecycle events
that happened while the stop was in progress. A request completing during
the stop found no tracked request, and the restore then resurrected it,
leaving the backend marked busy. A request starting during the stop was
overwritten by the restore, so completing an older request marked the
backend idle while the new one still ran, allowing an unsafe eviction.
An eviction now detaches only the backend identity (model, process, size
estimate) from the registry and marks the address as pending. Request
state stays live, so callbacks keep updating it during the stop. Idle and
busy checks skip pending addresses so they neither re-evict nor wipe that
state. When the stop settles, a backend that still runs is registered
again; otherwise its leftover request state is cleared. A replacement
registered on the address meanwhile is left alone.
Regression tests cover a request completing and a request starting inside
a failing force stop. The idle and busy failure tests now prove re-tracking
through a second stop attempt instead of sampling the registry, which
raced with the pending window.
Assisted-by: Claude:claude-opus-5.5 go vet golangci-lint
Signed-off-by: Hugues CHARBONNIER <hugues.charbonnier@gmail.com>
(cherry picked from commit 9a35647ec0)
An eviction untracks the backend before it stops it. When the stop failed,
the error was only logged: the backend kept running and kept its VRAM, but
the watchdog had already forgotten it. Group and LRU enforcement then saw no
conflict, and the next load started next to it and ran out of memory.
Stopping a model takes its exclusive lifecycle lock, which a model that is
still loading holds for the whole load. The stop then times out ("waiting to
shut down model: context deadline exceeded"), and the watchdog no longer
knows the model is there.
The tracking state is now captured before the backend is untracked, and put
back when the stop fails. This applies to every eviction path that untracks
first: group, LRU, idle timeout and busy timeout. A model that is already
gone (ErrModelNotFound) still counts as stopped, and a replacement
registered on the same address in the meantime is left alone. EvictedCount
only counts backends that really stopped, and NeedMore is set when one did
not, so the loader keeps retrying instead of treating the room as made. The
idle and busy checks retry the stop on their next tick.
The state is only put back while the loader still holds the model at that
address (ModelLoader.HasModelAtAddress). deleteProcess drops a remote-only
model, or the local half of a local+remote placement, before it reports a
failed remote unload; tracking that model again would leave a phantom entry
counted as loaded and as a group conflict. The memory reclaimer applies the
same rule and untracks a model the loader already dropped.
The untrack-before-stop ordering dates from #7535, so this is not a
regression of 4.12.0.
Assisted-by: Claude:claude-sonnet-5.5 go vet
Signed-off-by: Hugues CHARBONNIER <hugues.charbonnier@gmail.com>
(cherry picked from commit 97390f7754)
When --threads / LOCALAI_THREADS is unset, LocalAI uses the host's
physical core count. Inside a container with a CPU limit (docker --cpus,
a Kubernetes limits.cpu) that starts one thread per host core inside a
smaller CFS quota, so the backend is throttled for most of each period.
Since Go 1.25 the default GOMAXPROCS already follows the affinity mask and
the cgroup CPU limit. Use the smaller of the physical core count and
GOMAXPROCS as the default. Hosts without a limit are unchanged.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: manunicholasjacob <manunicholasjacob@gmail.com>
(cherry picked from commit aca800d003)
The menu opens upwards from the sidebar footer and is a child of the
scrollable sidebar. On short windows it was taller than the space above
the trigger, so its top (the first entries) was clipped by the sidebar.
Cap the menu height to the viewport and let it scroll instead.
Signed-off-by: gangna2s-arch <281744444+gangna2s-arch@users.noreply.github.com>
(cherry picked from commit 9a564ef79d)
Describe stop strings, greedy decoding, and seed handling after sampling
parameters are validated during construction.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 05af7b072c)
_build_sampling_params constructed SamplingParams and then assigned the
request values with setattr. protobuf repeated fields (StopPrompts) stay
RepeatedScalarContainer that way, and __post_init__ never sees them:
- vLLM's argument checks assert `isinstance(self.stop, list)` and the
msgpack encoder that ships the params to the engine core cannot
serialise the container, so every request with `stop` failed
(HTTP 500, EngineGenerateError), streaming or not.
- the stop-string hold-back buffer (output_text_buffer_length) was never
set, so the start of a multi-character stop string could be streamed;
seed -1 was not mapped to None.
Collect the values, turn repeated fields into lists and construct
SamplingParams once, as vLLM's OpenAI server does. Values vLLM rejects
(e.g. a penalty out of range, an empty stop string) now fail with vLLM's
validation error instead of being set unchecked.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 48d11fcf74)
The sidebar and the Build overview show the Add-ons and Fine-tuning entries
only when the server reports those switches on. With no add-on owner
running, the server reports the add-ons switch off, and the test server
has the fine-tuning switch off, so the two specs failed on master.
The two tests are about those entries being shown, so they set the switches
on and keep the server's other flags.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The spec "refuses a platform the matrix does not enable" ran package.sh
with windows/amd64 and expected the text "not enabled". That text comes
from the matrix check. The tool now refuses windows/amd64 earlier, in
its format check, with: platform "windows/amd64" is not os/arch of
linux or darwin on amd64 or arm64. The tool still fails and writes no
layout, but the substring is gone, so the add-on packages run failed in
validate package rules.
The format check accepts only linux and darwin on amd64 and arm64, and
Tables lists and enables all four of them. No platform passes the
format check and is missing from the matrix, so the spec cannot reach
the matrix refusal. Assert that the output names the refused platform
and that it gives one of the refusal reasons of the tool. The not this
host check stays as it was.
Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
localai-org-maint-bot