mirror/LocalAI
1
0 Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-10-28 17:18:49 -04:00
Code Issues Packages Projects Releases Wiki Activity

feat(llama-cpp): add lazy_mode model option (#12743)

Browse Source 
* feat(llama-cpp): add lazy_mode model option

Expose upstream --lazy-mode (on, auto, off) as a model option, for
example `lazy_mode:on`. It controls on-demand reading of large tensors
such as per-layer embeddings; `on` requires mmap.

The option is also accepted as `lazy-mode` or `lzm`. Invalid values log
a warning and keep the upstream default (auto). The pinned llama.cpp
already provides the lazy_mode field in common_params.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(llama-cpp): bump llama.cpp to c0fd78626405

Update the default llama.cpp pin to the current upstream master. The
changes since the old pin that the backend touches are additive: a new
common_decision_type value, new mtmd audio-stream and download helper
declarations. The two server patches still apply in order, and the
common_params fields and server headers the backend uses are unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
mudler-agentClaude Sonnet 5.5Ettore Di Giacinto
 authored and GitHub committed 2026-10-17 22:38:23 +02:00
1 parent e715af1def
commit e7cc993b0f
4 files changed
+46 -1
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

backend/cpp/llama-cpp/grpc-server.cpp
+18
View File
@@ -1212,6 +1212,24 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
#endif
}
}
} else if (!strcmp(optname, "lazy_mode") || !strcmp(optname, "lazy-mode") || !strcmp(optname, "lzm")) {
// On-demand reading of large tensors such as per-layer embeddings
// (upstream -lzm / --lazy-mode). Accepts on, auto or off; "on" needs mmap.
#if LOCALAI_HAS_LAZY_MODE
if (optval != NULL) {
if (optval_str == "on") {
params.lazy_mode = LLAMA_LAZY_MODE_ON;
} else if (optval_str == "auto") {
params.lazy_mode = LLAMA_LAZY_MODE_AUTO;
} else if (optval_str == "off") {
params.lazy_mode = LLAMA_LAZY_MODE_OFF;
} else {
SRV_WRN("lazy_mode: ignoring invalid value '%s' (expected on, auto or off)\n", optval_str.c_str());
}
}
#else
SRV_WRN("%s", "lazy_mode: this llama.cpp build has no lazy mode, ignoring\n");
#endif
} else if (!strcmp(optname, "n_cpu_moe")) {
if (optval != NULL) {
try {