* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): adapt to the multimodal rerank signature
Upstream llama.cpp added query_files and doc_files parameters to
format_prompt_rerank() (ggml-org/llama.cpp#29625). The gRPC rerank
call passed the init options in their place, so grpc-server.cpp no
longer compiled against the new pin.
prepare.sh now probes server-common.h for query_files and writes
LOCALAI_HAS_RERANK_FILES into llama_compat.h. When it is set, the
rerank call passes two empty media lists. The rerank API is text
only, so behaviour is unchanged. Forks that still use the old
signature (turboquant, bonsai) compile as before.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* chore(llama-cpp): bump to a37cc461 and drop the TTS patch
Move the llama.cpp pin to a37cc461, the merge of the Flint
architecture (ggml-org/llama.cpp#30928).
The TTS task patch is gone: upstream now carries
SERVER_TASK_TYPE_TTS, server_task::tts_inp, server_task_result_tts,
server_context_meta::has_cap_tts and the mtmd gen-audio helpers. The
patch is detected as already applied on the new pin. The gRPC server
uses the same symbols. The bonsai and turboquant forks still lack
them and keep using disable-tts-task.sh, so they are unchanged.
The score task patch is still needed: upstream has no
SERVER_TASK_TYPE_SCORE. It is regenerated against the new pin and
applies with no offset or fuzz. Upstream added
SERVER_TASK_TYPE_TRANSCRIBE_RT to server_task::need_logits(), which
broke one hunk, so SCORE is added next to it by hand.
The C++ backend was not built for this change.
Assisted-by: Claude:claude-sonnet-5-5
* feat(gallery): add flint
Add Recursive Flint, a 39.8B mixture-of-experts coding model with
about 2.5B active parameters, from the Unsloth GGUF repository. It
runs on llama.cpp with the Flint architecture support added upstream
in ggml-org/llama.cpp#30928.
Three quantizations are listed, with the default declaring the others
as variants: UD-Q4_K_XL (25 GB), UD-Q2_K_XL (13.6 GB) for small hosts
and UD-Q6_K_XL (36 GB) for larger ones. Sampling defaults follow the
model card (temperature 1, top_p 0.95, top_k 20). The library file
gallery/models/index.yaml is edited and gallery/index.yaml is
regenerated from it.
Assisted-by: Claude:claude-sonnet-5-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): add lazy_mode model option
Expose upstream --lazy-mode (on, auto, off) as a model option, for
example `lazy_mode:on`. It controls on-demand reading of large tensors
such as per-layer embeddings; `on` requires mmap.
The option is also accepted as `lazy-mode` or `lzm`. Invalid values log
a warning and keep the upstream default (auto). The pinned llama.cpp
already provides the lazy_mode field in common_params.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(llama-cpp): bump llama.cpp to c0fd78626405
Update the default llama.cpp pin to the current upstream master. The
changes since the old pin that the backend touches are additive: a new
common_decision_type value, new mtmd audio-stream and download helper
declarations. The two server patches still apply in order, and the
common_params fields and server headers the backend uses are unchanged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
* feat(llama-cpp): add slot prompt-cache save and restore API
Add POST /v1/slots/save and POST /v1/slots/restore. They write one
llama.cpp slot's prompt cache (KV state) to a file and load it back,
so a client can warm a long system prompt in a few hundred
milliseconds instead of re-running prefill. Refs #12685.
The new SlotCache gRPC call posts the same slot save and restore
tasks the llama-server /slots endpoint uses. It is off unless the
model sets the new slot_save_path option, so existing models are not
affected. The /slots monitoring endpoint is not needed here: the
backend has no HTTP server.
The file name comes from the caller, so it is checked twice, in Go
and again in the backend. It must be one path component: no empty
name, "/", "\", "..", NUL, control characters or absolute path. The
resolved path, with symbolic links followed, must be a direct child
of slot_save_path. The routes are admin only because they write
files on the backend host.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): answer 501 when a backend lacks slot cache
A backend that does not implement the SlotCache rpc returns gRPC
UNIMPLEMENTED. The API passed that text through, so the client saw
"unknown method SlotCache". Map the status to a typed error in
core/backend and return 501 with a fixed message instead. Every other
error path is unchanged.
Make the SlotCache comments in backend.proto backend-neutral: slot is
the backend's state slot, filename must stay inside the save directory,
and an empty model_identity means the backend does not check it. Only
comments change, no field or name.
Add tests that the identity sent is the configured model, and for the
error mapping.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): issue slot cache IDs on the server
The slot save and restore API let the client pick the file name.
Restore could load any file in slot_save_path, and the symlink check
ran before llama.cpp opened the file.
The backend now issues the name. A save returns a random 128-bit ID
(32 lowercase hex characters, read from /dev/urandom) and never
overwrites an existing file. Files live in a 0700 directory per model,
named from the SHA-256 of the model identity, with a manifest.json
that records each save. The manifest is written atomically and a
corrupt one stops both save and restore.
A restore takes the ID, finds it in the manifest, refuses an entry of
another model, builds the path from the entry and checks with lstat
that it is a plain file of the recorded size. The directory mode is
the main defence against another local user, since llama.cpp still
opens the path itself.
Each model keeps at most 32 saves and the oldest is deleted first. A
file over 8 GiB is deleted after the save, because llama.cpp does not
report the size before it writes. Responses carry no host path.
The HTTP bodies change: save takes {model, slot}, restore takes
{model, slot, id}, and an unknown ID returns 404. In the proto,
SlotCacheRequest.filename and SlotCacheResponse.filename become id.
SlotCache no longer calls checkModelIdentity, which reads a field this
request does not have.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): copy slot_cache files in prepare.sh
grpc-server.cpp includes slot_cache.h, but prepare.sh copies the helper
headers into llama.cpp/tools/grpc-server/ one by one and skipped it. The
llama-cpp, turboquant and bonsai builds share this step, so they all
failed with "slot_cache.h: No such file or directory".
Copy slot_cache.h and slot_cache_test.cpp, which CMakeLists.txt builds
when LLAMA_GRPC_BUILD_TESTS is on.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(deps): bump LocalAGI to pick up the scheduler start/stop fix
LocalAGI's scheduler could hit a nil pointer in run when Start and Stop
raced. That made the core/services/agentpool tests panic at random and
turned tests-linux red.
Move to mudler/LocalAGI@7aa0c8fea0, the merge of
mudler/LocalAGI#519 ("make Start and Stop safe to call concurrently").
The new module version also adds a few indirect requirements to go.mod
and go.sum.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
Bump LLAMA_VERSION to b76b13379d3b (upstream 0.7.0 plus later
commits). This is the same one-line change as the bot PR #12725.
Add the moe_cache_mib model option. It sets upstream's
moe_cache_size, a GPU cache for MoE experts kept in host memory,
from a size in MiB. It is off by default. Invalid, negative or
non-numeric values are ignored with a log line. Forks that predate
the field ignore the option, using the same prepare.sh probe as the
other fork-skew switches.
Document the option, and cover the MiB to bytes conversion with a
standalone unit test.
Assisted-by: Claude:claude-sonnet-5-5 g++
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): read the rendered chat prompt through str()
llama.cpp now returns the chat prompt as a common_chat_prompt instead of
a std::string. Convert it with a small helper that still accepts the plain
string used by older forks that share this source.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): preserve chat session state
Newer llama.cpp parsers return template state outside the request JSON.
Pass that state to both inference tasks and response readers so grammar,
tool calls, and reasoning remain available in both chat paths.
Detect the parser API during preparation to preserve older fork builds.
Assisted-by: Codex:gpt-6
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Newer ROCm releases split the rocBLAS and hipBLASLt TensileLibrary data
into one folder per GPU architecture (lib/hipblaslt/library/gfx1151/...).
The run.sh of the ROCm-capable backends exports ROCBLAS_TENSILE_LIBPATH and
HIPBLASLT_TENSILE_LIBPATH as the parent folder, and both libraries only look
directly in that folder, so they miss every kernel:
rocblaslt error: Cannot read ".../hipblaslt/library/TensileLibrary_lazy_gfx1151.dat"
hipModuleLoad failed: .../hipblaslt/library/Kernels.so-000-gfx1151.hsaco
and fall back to slower code paths.
rocm_tensile_dir keeps the folder when it has files at the top level (the
older flat layout) or several architecture folders, and descends into the
architecture folder when the bundle carries exactly one.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* fix(schema): preserve SystemOne image inputs
Assisted-by: OpenAI
* test(schema): follow Ginkgo conventions for decision inputs
Assisted-by: OpenAI
* feat(llama-cpp): dispatch native decisions through Score
Upgrade the stock dependency and reconcile Score/TTS patches. Reuse native decision parsing, tasks, formatting and response-reader cleanup; preserve ordinary scoring admission and guard older dependencies.
Assisted-by: OpenAI
* refactor(systemone): share request and model validation
Assisted-by: OpenAI:gpt-5
* fix(systemone): preserve HTTP wire-byte validation limit
Keep structural validation separate from the serialized internal request bound so HTML escaping cannot reject valid HTTP payloads.
Assisted-by: OpenAI:gpt-5
* feat(systemone): bound images and account native decisions
Preserve public wire limits independently from router serialization. Reject unsupported NER images, map native request/capability errors, and stamp explicit usage once. Advertise decisions for stock llama-cpp.
Assisted-by: OpenAI
* fix(systemone): record usage on registered native route
Exercise real registration and billing with a mock native backend. Reject empty native responses, malformed image URLs, trailing JSON, and wire overflow including whitespace.
Assisted-by: OpenAI
* feat(router): add lazy native decision transport
Bind named models through internal ModelSystemOne calls with shared validation and bounded abandoned operations. Remove request and echoed-error contents from decision traces.
Assisted-by: OpenAI:gpt-5
* feat(router): classify overlapping policies with native decisions
Ask independent noul questions, validate probabilities and preserve first-superset routing. Wire the central factory with config-sensitive invalidation and cancellation-safe resolution. Document native framing and bounded operation limits.
Assisted-by: OpenAI:gpt-5
* feat(gallery): add pinned Julia-1 native decision model
Add a separate text-only llama-cpp Q8 entry with pinned Apache-2.0 source provenance and checksum. Installed using the gallery installer and exercised choice, score and noul on CPU.
Assisted-by: OpenAI
* test(router): verify native decisions through central factory
Add an opt-in real-model Ginkgo integration covering the native Go loader and C++ transport, token usage, independent overlapping labels, and candidate selection. Document owned-server execution and the intentionally non-quality threshold.
Assisted-by: Codex:gpt-5
* fix(llama-cpp): align upstream pin and preserve decision signatures
Advance to bed0a856 without losing the automated upstream bump. Detect full-request fill_task support at compile time and forward every question for Nimble framing while retaining the earlier native signature. Preserve reconciled SCORE/TTS patches; add standalone compatibility coverage.
Assisted-by: Codex:gpt-5
* feat(gallery): add native decision family defaults
Pin Laya, Kev-4B, lev, OpenJev and Nimble artifacts. Verify Laya/Kev/lev gallery installs and CPU contracts on both native pins; clearly mark OpenJev/Nimble runtime validation pending and their noncommercial licenses.
Assisted-by: OpenAI
* docs(decisions): clarify integrated Nimble prerequisite
Record the exact combined backend pin while retaining pending OpenJev and Nimble installation/runtime validation status.
Assisted-by: Codex:gpt-5
* fix(gallery): indent native decision model sequences
Match repository yamllint indentation for Laya, Kev, lev and OpenJev list fields. Parsed gallery data is unchanged; reproduce CI gallery lint failure before the whitespace-only fix and pass the same command afterward.
Assisted-by: Codex:gpt-5
* docs(decisions): record OpenJev and Nimble CPU validation
Record gallery installation, checksum/metadata verification and multiquestion native smoke results on bed0a856. Retain noncommercial and text-only limitations without accuracy or deterministic-output claims.
Assisted-by: OpenAI
* fix(ui): expose native Decisions router classifiers
Select classifier models using metadata-driven capability routing, retain tuned thresholds, and validate native decision selections before saving. Cover both native backends and create/save/reopen in the real React editor.
Assisted-by: Codex:gpt-5
* fix(router): exclude aliases from native decision discovery
Check the originally named config before advertising native Decisions eligibility. Retain target capability inheritance for ordinary generation aliases. Exercise the actual capabilities endpoint with native models on both backends, aliases, and disabled models.
Assisted-by: Codex:gpt-5
* feat(systemone): share bounded multimodal input validation
Preserve text wire limits while admitting bounded PNG/JPEG decision input. Share collection and header validation across internal and public callers and keep the native runner response budget independent.
Assisted-by: OpenAI:API-assistant
* fix(systemone): bound admission lifetimes and validate complete images
Retain shared admission leases through actual work completion, including abandoned internal operations. Decode bounded image pixels, cap public native responses before usage stamping, and preserve oversized malformed text status precedence.
Assisted-by: OpenAI:API-assistant
* fix(router): classify images before media fetching
Preserve ordered structured probes for native decisions. Defer OpenAI
media preparation until routing selects the served model, so rejected
decision URLs cannot trigger downloads before shared validation.
Guard direct image collection with context-aware shared admission.
Keep text classifiers and embedding caches from discarding image input.
Retain fail-closed classifier configuration and runtime fallback policy.
Add middleware, typed-content, admission, cancellation and cache tests.
Assisted-by: OpenAI:API-assistant
* fix(router): bound extraction before serialization
Check probe budgets before copying text or marshaling message state.
Count JSON escaping so oversized internal inputs fail before allocation.
Preserve typed Anthropic blocks through selected-model conversion and
fallback. Keep retry coverage in Ginkgo without global test registration.
Assisted-by: OpenAI
* fix(router): bound supported probe serialization
Arbitrary structs can bypass the probe budget through pointer marshalers,
string tags, and promoted fields. Accept concrete chat schema types and
plain JSON values instead of emulating arbitrary struct serialization.
Budget escaped direct prompts before marshaling so raw length cannot hide
serialized expansion. Preserve runtime fallback and reject oversized
input before invoking the decision runner.
Add Ginkgo allocation, boundary, and marshaler invocation regressions.
Six-package tests, three-package race tests, and full-T2 delta lint pass.
Assisted-by: OpenAI:GPT-5 golangci-lint
* feat(decisions): enable bounded OpenJev images
Validate native decision images before permissive media parsing and pixel
allocation. Require both decision image support and a vision projector;
missing or audio-only projectors cannot silently become text decisions.
Pin the OpenJev Q8 projector and document its license and disk footprint.
Add native safety tests, canonical limit parity, gallery and load-option
checks, and a reproducible CPU direct-RPC contrasting-image smoke.
Assisted-by: OpenAI:GPT-5
* fix(decisions): reject incomplete image streams
stb accepts corrupt PNG Adler checksums and truncated JPEG scans.
Use bounded zlib validation and strict libjpeg decoding before parsing.
Keep dimension and aggregate pixel checks ahead of decoder allocations.
Wire decoder dependencies into native builds and runtime packaging.
Add regressions for appended EOI and embedded marker bypasses.
Assisted-by: OpenAI:GPT-5
* fix(ci): gate native decision image validation
Run the decoder security tests outside the stdlib-only native suite.
Fetch vendor headers at the backend pin and provision decoder dependencies.
Gate Go limit parity and production CMake wiring without model downloads.
Assisted-by: OpenAI:GPT-5
* test(decisions): cover multimodal public API paths
Exercise shared image contracts through the registered HTTP routes and
external mock backend. Add opt-in cached gallery installation and real
OpenJev image decisions through SystemOne and both routing APIs.
Assisted-by: Codex:gpt-5
* test(decisions): assert isolation and cache bypass
Observe external RPC calls and compare complete classifier history.
Winner-only and cache-miss checks could hide dropped history or cache use.
Give real inference its own application and model directory so shared
backend mappings and loaded processes cannot affect mixed suite order.
Assisted-by: OpenAI:ChatGPT
* test(decisions): isolate fixture globals
Disable optional global services in the isolated HTTP fixture and register
cleanup before setup assertions. Verify meter provider identity survives
fixture creation and destruction.
Snapshot observed usage before assertions so failures cannot retain the
mutex. Require a successful usage stamp before checking error responses.
Assisted-by: Codex:gpt-5 golangci-lint
* fix(application): honor optional telemetry controls
Skip failover gauge registration when metrics are disabled. Register
against the application meter rather than looking up the global provider.
Allow embedders to retain the bounded routing log without billing stats.
Keep the existing default when stats are disabled. The isolated HTTP
fixture uses this option without losing its native router assertions.
Assisted-by: Codex:gpt-5 golangci-lint
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
When the first result of a streamed request is an error (for example a
prompt that exceeds the context), PredictStream wrote the error message
as a Reply and only then returned the error status. LocalAI treated that
Reply as the first token: it sent the assistant role chunk and the error
text as `content` on an HTTP 200 stream. Because a chunk had already been
written, the pre-stream HTTP error path from #12204 never triggered, so
streaming clients still got a 200 with the error as model output, while
the same request without streaming correctly returns a 400.
Return the error only as the gRPC status. The e2e backend suite gets a
`context_overflow` capability (enabled for llama-cpp) that streams an
over-long prompt and asserts an error status with no content.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
The environment fallback was applied whenever n_parallel was still 1
after option parsing. An explicit `parallel: 1` in the model YAML is
indistinguishable from the default that way, so it was replaced by
LLAMACPP_PARALLEL. The docs say options in the YAML take precedence
over environment variables; a single model could not be forced to one
slot while the global variable was set.
Track whether the options set the slot count and resolve it in a small
helper (parallel_params.h): option first, then LLAMACPP_PARALLEL, then
1. The helper gets a standalone unit test picked up by
`make test-backend-cpp`.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): migrate score batches to the new API
The pinned engine removes common_batch_add and the raw batch view.
Use common_batch entries and llama_process for score suffix decoding.
Read shared-prefix scores from the current common_batch view.
Validation: reproduce both compiler errors on the original patch.
The patched server context and complete grpc-server translation unit
pass g++ -std=c++17 -fsyntax-only with generated protobuf headers.
Assisted-by: Codex:gpt-6
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
grpc-server.cpp forced params.cache_ram_mib = -1 (no limit) since #7009.
Since v4.3 kv_unified and cache_idle_slots are on by default, so every
distinct prompt now leaves its slot KV state in the host-side prompt
cache, and without a limit the backend grows until the host runs out of
memory.
Measured on gfx1151 (Strix Halo, 128 GB), llama-cpp backend, one request
at a time, 100 distinct prompts of ~2000 characters plus a fixed system
prompt, max_tokens 200:
model cache_ram RSS loaded -> after 100
gemma-4-26B-A4B (q8_0 KV) -1 (default) 1.4 GB -> 25.3 GB
Qwen3.6-35B-A3B (q8_0 KV) -1 (default) 1.1 GB -> 19.5 GB
gemma-4-26B-A4B -1, same prompt 100x 1.4 GB -> 1.6 GB
gemma-4-26B-A4B 4096 1.4 GB -> 5.4 GB (flat from
request 20 on, same latency)
Qwen3.6-35B-A3B 4096 1.1 GB -> 5.1 GB (flat)
The memory is not released when idle. In production a document
classification pass pushed the daily chat model to 34 GB RSS overnight.
Drop the override so llama.cpp's own default (8192 MiB) applies; the
cache_ram option still accepts -1 for users who want no limit. Update
both docs tables (the option reference and the prompt-cache table) and
note what -1 does.
Assisted-by: Claude:claude-opus-5-5
Assisted-by: Codex:GPT-6
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): disable PCH for SYCL builds
IntelLLVM cannot bundle the precompiled headers introduced by the latest
llama.cpp update. Disable PCH for both SYCL variants while retaining it
for the other backends.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
grpc::ServerWriter::Write() returns false once the peer is gone, and
PredictStream ignored that result at every call site. The handler kept
pulling decoded tokens and writing them into a dead stream, so the
llama.cpp slot stayed busy until the generation ended on its own terms.
A model configured with max_tokens 0 and a large context ends on its own
terms only at the context limit. On a 35B model at ~41 t/s a 120k context
is about fifty minutes, and a slot held that long is a slot every other
request for that model queues behind. Two abandoned requests were enough
to make a node with free VRAM and a healthy control plane serve nothing:
new requests timed out waiting for a slot, each timeout abandoned another
generation, and the node fell further behind the longer it ran.
Track the peer instead. The first failed write retires it for good, since
a stream never recovers, and the RPC's own cancellation flag folds into
the same predicate so the loop has one condition to test. Returning early
is what frees the slot: ~server_response_reader() posts
SERVER_TASK_TYPE_CANCEL for whatever is still decoding.
TTSStream already checked Write(); this brings PredictStream in line.
Cancellation stays cooperative and is checked between decoded results, so
a batch already in flight may finish before the request stops.
Assisted-by: Claude:claude-opus-5
The generated gRPC source tree omitted the new header and test. Every llama.cpp-derived backend therefore failed when grpc-server.cpp included the missing header.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
llama.cpp reports the same tensor-count error for unsupported model layouts and damaged GGUF files. Add a focused hint so operators can update the backend or verify the model without losing the upstream diagnostic.
Assisted-by: Codex:gpt-5.6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): link librdma from the static ggml-rpc build
ggml-rpc gained an Apple RDMA transport in this llama.cpp range and
declares its librdma dependency with target_link_options(ggml-rpc
PRIVATE "LINKER:-weak_library,..."). Link options are not a usage
requirement of a static library, so the llama-cpp-grpc variant, which
builds with BUILD_SHARED_LIBS=OFF, dropped the flag and left every
ibv_* symbol of transport-apple.cpp undefined when grpc-server linked
on darwin.
prepare.sh now re-declares the same weak link as INTERFACE on the
ggml-rpc target, so the flag reaches whoever links the static library.
The append is guarded on a marker for repeat runs, and on
GGML_RPC_RDMA_APPLE, which the turboquant and bonsai forks lack.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): follow upstream MTMD APIs
The dependency update adds MTMD initialization options to prompt and
bitmap helpers. The gRPC adapter now passes the server options through
each affected path.
The update also replaces the per-layer MoE regex helper. Preparation
probes both APIs because older forks still reuse this adapter.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): adapt to the common JSON API
The llama.cpp bump replaces its nlohmann JSON alias with common_json. Update the gRPC adapter for the new exception, iterator, conversion, and container APIs.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(turboquant): adapt the JSON exception type
The shared gRPC source now follows the upstream common_json API. The
TurboQuant fork still exposes nlohmann JSON and cannot compile the new
exception type.
Translate that exception in the fork-specific source patch so both
llama.cpp variants compile from the shared adapter.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(bonsai): adapt the JSON exception type
The shared gRPC source uses upstream's common_json wrapper. The Bonsai fork still exposes nlohmann JSON and cannot compile that exception type.\n\nTranslate the exception in the fork-specific preparation step and verify that repeated preparation stays idempotent.\n\nAssisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(llama-cpp): let prepare register gRPC
The score patch duplicated the gRPC CMake registration that prepare.sh already owns. Its stale context rejects the current upstream tools file on Darwin before compilation starts.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* chore(llama-cpp): update upstream revision
Assisted-by: Codex:gpt-5.6
* fix(llama-cpp): refresh server patch contexts
The new llama.cpp pin changed the slot reset and prompt batch code. GNU patch accepted stale hunks with fuzz, which left the L4T build with invalid source.
Refresh both server patches against the pinned source so each hunk applies at its intended location.
Assisted-by: Codex:gpt-5
* fix(llama-cpp): adapt metrics result fields
The updated llama.cpp groups cumulative counters under server_metrics. Probe the result layout so the shared adapter also compiles against older forks.
Assisted-by: Codex:gpt-5
* fix(llama-cpp): refresh TTS patch offsets
GNU patch rejects the stale pre-decode hunk after the score patch changes the same file. Anchor the TTS hunks to the pinned llama.cpp source so the full series applies without fuzz.
Assisted-by: Codex:gpt-5.4
* fix(llama-cpp): normalize batch threads
The updated llama.cpp creates its batch threadpool during model initialization, before the context-level fallback can replace the -1 sentinel. Resolve that sentinel from the inference thread count so model loading does not overflow the threadpool allocation.\n\nAssisted-by: Codex:gpt-5.4
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
