mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-08-03 12:00:43 -04:00
Code Issues Packages Projects Releases Wiki Activity

backend(audio-cpp): correct the status, lifetime and state contracts of LoadedModel

Browse Source 
An environment fault during session creation was reported as UNIMPLEMENTED. A
missing libggml-cpu-*.so surfaced to the client as 'family silero_vad advertises
vad/offline but refused to create the session: Failed to initialize CPU
backend', which tells LocalAI the model cannot do this and must never be
retried, and sends an operator hunting a capability bug instead of a packaging
one. A throw from create_task_session is now a plain runtime_error, so it maps
to INTERNAL. Only a null return, where the family genuinely declined, stays a
CapabilityError.

The model.'s task: option was parsed and then dropped: it lived in a local that
died at the end of LoadModel and had no route to RequestShape::pinned_task.
LoadedModel now keeps it and exposes pinned_task().

The global model becomes a shared_ptr reached through snapshot(). An audio RPC
runs for seconds and cannot hold g_model_mu for its duration, so under a
unique_ptr a Free arriving mid-request would destroy the model underneath it.
Handlers now take a counted reference and whichever finishes last does the
teardown, outside the lock.

session_for documents the streaming state contract rather than resetting the
session itself. Resetting on a cache hit was tried first and is not possible:
silero_vad throws 'session prepare() must be called before Silero VAD reset()',
so it would turn an ordinary second fetch into a hard error. start_stream's base
implementation is already a reset, so a caller that runs prepare then
start_stream per stream gets a clean session; a probe against the bundled
silero_vad confirms an identical replay when it does and a carried-over stream
when it does not.

Also: an unknown backend: name is rejected before the model loads rather than
after; MainGPU is parsed instead of passed through std::atoi, which turned
'gpu1' into device 0 silently; and device carries a device_set flag, because 0
is both the default and a real device index, so MainGPU was overriding an
explicit device:0 that the neighbouring threads: handling promises will win.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
Ettore Di Giacinto
2026-07-26 14:01:34 +00:00
parent a9a59701e6
commit fac26a23df
6 changed files with 151 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
backend/cpp/audio-cpp/loaded_model.cpp
View File

@@ -286,6 +286,22 @@ LoadedModel::LoadedModel(const std::string &resolved_path,
throw ConfigError("audio-cpp: unknown audio.cpp family '" + family + "'");
}
// Session options are built BEFORE the load, because parse_backend_type
// rejects an unknown backend name. Validating after the load would make
// `backend:cudaa` cost a full model load, on a fault a string comparison
// could have caught.
session_options_.backend.type = parse_backend_type(options.backend);
session_options_.backend.device = options.device;
if (options.threads > 0) {
session_options_.backend.threads = options.threads;
}
for (const auto &entry : options.session_options) {
session_options_.options[entry.first] = entry.second;
}
pinned_task_ = options.task;
wait_budget_ceiling_ms_ = options.busy_timeout_ms;
engine::runtime::ModelLoadRequest request;
request.model_path = std::filesystem::path(resolved_path);
request.family_hint = family;
@@ -315,17 +331,6 @@ LoadedModel::LoadedModel(const std::string &resolved_path,
languages_ = engine_caps.languages;
supports_timestamps_ = engine_caps.supports_timestamps;
capabilities_ = to_capabilities(family, engine_caps);
session_options_.backend.type = parse_backend_type(options.backend);
session_options_.backend.device = options.device;
if (options.threads > 0) {
session_options_.backend.threads = options.threads;
}
for (const auto &entry : options.session_options) {
session_options_.options[entry.first] = entry.second;
}
wait_budget_ceiling_ms_ = options.busy_timeout_ms;
}
LoadedModel::Session LoadedModel::session_for(Rpc rpc, const RequestShape &shape) {
@@ -336,7 +341,8 @@ LoadedModel::Session LoadedModel::session_for(Rpc rpc, const RequestShape &shape
const SessionKey key{static_cast<int>(route.task), static_cast<int>(route.mode)};
auto found = sessions_.find(key);
if (found == sessions_.end()) {
const bool cache_hit = found != sessions_.end();
if (!cache_hit) {
engine::runtime::TaskSpec spec;
spec.task = to_engine_task(route.task);
spec.mode = to_engine_mode(route.mode);
@@ -344,13 +350,23 @@ LoadedModel::Session LoadedModel::session_for(Rpc rpc, const RequestShape &shape
try {
created = model_->create_task_session(spec, session_options_);
} catch (const std::exception &err) {
throw CapabilityError(
// NOT a CapabilityError. The family said it supports this pair, and
// a throw from here is overwhelmingly an environment fault: a ggml
// backend .so that package.sh did not ship, an out of memory, a CUDA
// device that is not there. UNIMPLEMENTED would tell LocalAI and
// every client "this model cannot do this, never retry", and send an
// operator hunting a capability bug instead of a packaging one. A
// plain runtime_error maps to INTERNAL, which is what a fixable
// deployment fault should look like.
throw std::runtime_error(
std::string("audio-cpp: family '") + capabilities_.family +
"' advertises " + task_name(route.task) + "/" +
mode_name(route.mode) + " but refused to create the session: " +
err.what());
}
if (created == nullptr) {
// A null return with no throw is the family declining, which is a
// genuine capability answer and stays UNIMPLEMENTED.
throw CapabilityError(std::string("audio-cpp: family '") +
capabilities_.family +
"' returned no session for " +
@@ -373,6 +389,18 @@ LoadedModel::Session LoadedModel::session_for(Rpc rpc, const RequestShape &shape
"' advertises " + task_name(route.task) +
"/streaming but its session is not streaming");
}
// Deliberately NOT reset here, though a cached streaming session does
// carry state across chunks. reset() is not callable at this point:
// silero_vad's implementation throws "session prepare() must be called
// before Silero VAD reset()", so resetting on a cache hit would turn an
// ordinary second fetch into a hard error, which is worse than the leak
// it would prevent.
//
// The state is instead cleared by the sequence every streaming caller
// owes anyway. IStreamingVoiceTaskSession::start_stream's base
// implementation IS a call to reset(), so a caller that runs
// prepare(...) then start_stream(...) at the top of each stream gets a
// clean session for free. See the STATE CONTRACT in loaded_model.h.
} else {
session.offline =
dynamic_cast<engine::runtime::IOfflineVoiceTaskSession *>(raw);