mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 18:09:05 -04:00
e2cad2bf3d
An environment fault during session creation was reported as UNIMPLEMENTED. A missing libggml-cpu-*.so surfaced to the client as 'family silero_vad advertises vad/offline but refused to create the session: Failed to initialize CPU backend', which tells LocalAI the model cannot do this and must never be retried, and sends an operator hunting a capability bug instead of a packaging one. A throw from create_task_session is now a plain runtime_error, so it maps to INTERNAL. Only a null return, where the family genuinely declined, stays a CapabilityError. The model.'s task: option was parsed and then dropped: it lived in a local that died at the end of LoadModel and had no route to RequestShape::pinned_task. LoadedModel now keeps it and exposes pinned_task(). The global model becomes a shared_ptr reached through snapshot(). An audio RPC runs for seconds and cannot hold g_model_mu for its duration, so under a unique_ptr a Free arriving mid-request would destroy the model underneath it. Handlers now take a counted reference and whichever finishes last does the teardown, outside the lock. session_for documents the streaming state contract rather than resetting the session itself. Resetting on a cache hit was tried first and is not possible: silero_vad throws 'session prepare() must be called before Silero VAD reset()', so it would turn an ordinary second fetch into a hard error. start_stream's base implementation is already a reset, so a caller that runs prepare then start_stream per stream gets a clean session; a probe against the bundled silero_vad confirms an identical replay when it does and a carried-over stream when it does not. Also: an unknown backend: name is rejected before the model loads rather than after; MainGPU is parsed instead of passed through std::atoi, which turned 'gpu1' into device 0 silently; and device carries a device_set flag, because 0 is both the default and a real device index, so MainGPU was overriding an explicit device:0 that the neighbouring threads: handling promises will win. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
48 lines
1.7 KiB
C++
48 lines
1.7 KiB
C++
#pragma once
|
|
|
|
// Parses the model YAML's `options:` list (ModelOptions.Options in
|
|
// backend.proto) into a struct. Standard library only: this unit is compiled
|
|
// and tested by backend/cpp/run-unit-tests.sh without an audio.cpp checkout.
|
|
|
|
#include <map>
|
|
#include <string>
|
|
#include <vector>
|
|
|
|
namespace audiocpp_backend {
|
|
|
|
struct ModelOptions {
|
|
// audio.cpp model family. Empty means "derive from the GGUF's embedded
|
|
// audiocpp.model_spec.family key"; a non-GGUF path with an empty family is
|
|
// rejected at load time, not here.
|
|
std::string family;
|
|
// Pins the audio.cpp task, overriding RPC-based routing. Empty means route.
|
|
std::string task;
|
|
// ggml backend: cpu, cuda, vulkan, metal, best.
|
|
std::string backend = "cpu";
|
|
int device = 0;
|
|
// True once a `device:` entry has been seen. 0 is both the default and a
|
|
// legitimate device index, so the value alone cannot tell an explicit
|
|
// `device:0` from an unset option, and a caller merging in its own fallback
|
|
// would silently override the explicit choice.
|
|
bool device_set = false;
|
|
// 0 means "let the runtime decide".
|
|
int threads = 0;
|
|
std::string model_spec_override;
|
|
// 0 disables the run guard's fail-fast, restoring an unbounded wait.
|
|
int busy_timeout_ms = 0;
|
|
// `load.<key>:<value>` entries, prefix stripped.
|
|
std::map<std::string, std::string> load_options;
|
|
// `session.<key>:<value>` entries, prefix stripped.
|
|
std::map<std::string, std::string> session_options;
|
|
};
|
|
|
|
struct ParsedOptions {
|
|
ModelOptions options;
|
|
// Non-empty means the caller must fail the load with INVALID_ARGUMENT.
|
|
std::string error;
|
|
};
|
|
|
|
ParsedOptions parse_model_options(const std::vector<std::string> &entries);
|
|
|
|
} // namespace audiocpp_backend
|