mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 18:09:05 -04:00
Code Issues Packages Projects Releases Wiki Activity

chore(llama-cpp): bump llama.cpp and adapt to the load-mode refactor (#11140)

Browse Source 
Bump LLAMA_VERSION to 0d47ea7427463093e69128bf2c2f9cd06b3ee5b3 (73 commits
touching common/, src/ and tools/server/). Two upstream changes break the
backend:

* ggml-org/llama.cpp#20834 folded common_params::use_mmap / use_mlock /
  use_direct_io into a single `load_mode` enum. LocalAI still exposes the three
  as independent settings (`mmap`, `mmlock`, and the `direct_io` option), so
  params_parse folds them once all three have been read, keeping the precedence
  the separate booleans had: direct I/O bypasses the page cache, mlock implies
  mmap, everything off is a plain buffered read. turboquant and bonsai compile
  this same grpc-server.cpp against forks that predate the refactor, so
  prepare.sh probes the checkout for LLAMA_LOAD_MODE_MMAP and generates
  llama_compat.h with LOCALAI_LEGACY_LOAD_MODE set accordingly. Probing beats a
  per-fork build flag here because the fork flavor targets disagree on whether
  they forward CMAKE_ARGS or EXTRA_CMAKE_ARGS, and it heals itself once a fork
  rebases past the refactor.

* The MiniMax M3 patch no longer applies. Upstream merged the model half of
  llama.cpp#24523 (LLM_ARCH_MINIMAX_M3, src/models/minimax-m3.cpp, the gguf-py
  constants and conversion/minimax.py) but not the chat half, so the patch is
  re-cut to carry only the common/chat.cpp template detection and PEG parser,
  rebased onto the new pin and onto the thinking_end_tag -> thinking_end_tags
  rename. Dropping it wholesale (as #11008 did, reverted in #11136) would have
  silently regressed MiniMax M3 tool calling and thinking.

Verified with a CPU docker build of the backend plus LoadModel and Predict
against a real GGUF over gRPC in all four load modes.


Assisted-by: Claude:claude-opus-5 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
mudler's LocalAI [bot]
2026-07-27 09:02:29 +02:00
committed by GitHub
parent d9f3007876
commit 0a8a7fbbb4
5 changed files with 278 additions and 819 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
backend/cpp/llama-cpp/grpc-server.cpp
View File

@@ -52,6 +52,7 @@
#include "common.h"
#include "arg.h"
#include "chat-auto-parser.h"
#include "llama_compat.h" // fork-skew switches, generated by prepare.sh
#include "message_content.h"
#include <getopt.h>
#include <grpcpp/ext/proto_server_reflection_plugin.h>
@@ -613,6 +614,13 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
// starts with '-'. Applied once after the loop via common_params_parse.
std::vector<std::string> extra_argv;
// O_DIRECT intent from the `direct_io` option. Upstream folded
// use_mmap/use_mlock/use_direct_io into a single common_params::load_mode
// enum (ggml-org/llama.cpp#20834), so the three independent LocalAI settings
// can only be reduced to one value once all of them have been read, held
// here until the mmap/mlock fields arrive further down.
bool want_direct_io = false;
auto add_device_options = [&](const std::string & devices) {
const std::regex regex{ R"([,]+)" };
std::sregex_token_iterator it{ devices.begin(), devices.end(), regex, -1 };
@@ -868,9 +876,9 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
// --- O_DIRECT model loading (upstream --direct-io) ---
} else if (!strcmp(optname, "direct_io") || !strcmp(optname, "use_direct_io")) {
if (optval_str == "true" || optval_str == "1" || optval_str == "yes" || optval_str == "on" || optval_str == "enabled") {
params.use_direct_io = true;
want_direct_io = true;
} else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
params.use_direct_io = false;
want_direct_io = false;
}
// --- embedding normalization (upstream --embd-normalize) ---
@@ -1278,8 +1286,28 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
lora_info.ptr = nullptr;
params.lora_adapters.push_back(std::move(lora_info));
}
params.use_mlock = request->mlock();
params.use_mmap = request->mmap();
// LocalAI keeps mmap, mlock and direct-I/O as three independent settings,
// while upstream now carries a single load mode. Fold them with the
// precedence the separate booleans used to give: direct I/O bypasses the
// page cache entirely, mlock implies mmap, and everything off is a plain
// buffered read. Forks that branched before ggml-org/llama.cpp#20834 still
// expose the booleans; prepare.sh probes the checkout and sets
// LOCALAI_LEGACY_LOAD_MODE in the generated llama_compat.h accordingly.
#if LOCALAI_LEGACY_LOAD_MODE
params.use_mlock = request->mlock();
params.use_mmap = request->mmap();
params.use_direct_io = want_direct_io;
#else
if (want_direct_io) {
params.load_mode = LLAMA_LOAD_MODE_DIRECT_IO;
} else if (request->mlock()) {
params.load_mode = LLAMA_LOAD_MODE_MLOCK;
} else if (request->mmap()) {
params.load_mode = LLAMA_LOAD_MODE_MMAP;
} else {
params.load_mode = LLAMA_LOAD_MODE_NONE;
}
#endif
if (request->flashattention() == "on" || request->flashattention() == "enabled") {
params.flash_attn_type = LLAMA_FLASH_ATTN_TYPE_ENABLED;