mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 18:09:05 -04:00
2035a4d25e
Three related backend-lifecycle defects, all reachable from the same production incident on a Jetson/Thor worker: a deleted backend's gRPC process survived ~40 minutes with its directory removed from disk, a later model load was routed to that orphan and failed with a certifi path pointing into the deleted directory, and the admin could not stop the model because the frontend reported it as not loaded. 1. backend.delete orphaned the process it claimed to delete ------------------------------------------------------------ s.processes is keyed by `modelID#replicaIndex` (buildProcessKey), so the backend name never appeared in a key and was recorded nowhere on the process. backend.delete resolved its target via isRunning/stopBackend, whose prefix path only matches a bare *modelID* - a delete keyed on a backend name resolved to zero keys, the stop silently no-op'd, and the files were removed out from under a live process. The install fast path then handed that orphan back out: it returns any live process for the (model, replica) slot without checking which backend started it, so a reinstalled variant inherited the deleted backend's port. - Record backendName on backendProcess, threaded installBackend -> startBackend. - Add resolveProcessKeysForBackend, matching the recorded name and resolving alias <-> concrete via ListSystemBackends *before* DeleteBackendFromSystem erases the metadata that carries the alias. Alias resolution failure degrades to name-only matching so a delete never fails on it. - backend.stop goes through resolveStopTargets, which accepts a backend name, a model name, or an exact modelID#replica key. Its payload field is named "backend" but is published with all three meanings: the admin UI sends a backend name, UnloadRemoteModel sends a model name, and the router's abandoned-load reap (#10948) sends an exact replica key. Narrowing it to backend names alone would strand the latter two. backend.delete stays strict - its identifier is unambiguously a backend. - Gate the install fast path on processMatchesBackend so a slot held by a different backend is restarted rather than reused. Processes with no recorded name (pre-upgrade) are accepted, so rollout does not restart every running backend. - stopBackendExact reports a real stop failure - the process still being alive afterwards, which is precisely what finishBackendStop already detects to keep the entry and its port reserved - and backend.delete no longer replies success when it knew about a process and could not kill it. "No process was running" stays a success but is logged, so the orphan case is visible rather than silent. 2. /backend/shutdown reported a running model as missing --------------------------------------------------------- ModelLoader.deleteProcess short-circuits on a miss in this replica's in-memory store. In distributed mode the authoritative record of "is this model loaded" is the shared node registry: a frontend replica that never served the model itself (load balancer picked a peer, or the replica restarted) has no local entry. The remote unload path that pkg/model documents ("when ShutdownModel is called for a model with no local process, UnloadRemoteModel is called") sat behind that short-circuit, unreachable in exactly the case it exists for. #10865 reworked this function but kept the short-circuit at the top, so the gap survived that refactor. - deleteProcess consults the remote unloader on a local-store miss, via a shared unloadRemote helper so this branch and the existing no-local-process branch both prefer #10865's RemoteModelContextUnloader, preserving force propagation across the distributed boundary. - UnloadRemoteModelContext reports ErrRemoteModelNotLoaded when no node has the model; it previously returned nil, making a no-op stop indistinguishable from a real one. The converse case (nodes have it, none could be stopped) already errors since #10865 joined the per-node failures, so that half of the original fix was dropped as redundant. - Only when the model is absent locally AND cluster-wide does the endpoint report not-found, now 404 naming both scopes rather than a bare 500. - modelNotFoundErr becomes the exported ErrModelNotFound so the HTTP layer can map it without string matching; watchdog's identity comparison becomes errors.Is. 3. Coverage for the bounded Free() that #10865 shipped untested ---------------------------------------------------------------- The original branch also bounded the pre-stop Free(), but #10865 landed that fix first (workerBackendFreeTimeout, applied in both stopBackendExact and handleModelUnload). That production change is therefore DROPPED here as superseded - master's version is strictly better, since it also releases the supervisor mutex across the call and keeps the port reserved until termination completes. What #10865 did not ship is a test, and the bound is load-bearing: the router-side reap in #10948 sends backend.stop for an abandoned load, and against a wedged backend an unbounded Free would swallow that stop before it reached the process. Nothing failed if the bound regressed. The spec stands up a real gRPC backend server whose Free handler never returns - what a Python backend looks like when its single worker thread (PYTHON_GRPC_MAX_WORKERS=1 on 37 backends) is occupied by a stuck LoadModel. A stub socket is not sufficient and was tried first: without a completed HTTP/2 handshake, gRPC's own ~20s connect timeout ends the call, so that version passed against the very bug it targets. With the connection READY, only the caller's deadline can end it, so the spec hangs to its 60s limit if the timeout is removed and passes with it. Its fixture process is deliberately never started. go-processmanager v0.1.1 writes Process.pid from readPID() without synchronization, so a live process races its own monitor goroutine under -race - reproducible with a bare Run()+Stop() and unrelated to this spec. Since scripts/model-lifecycle-conformance.sh runs this package with -race and is fail-closed, starting one would turn that gate red on an upstream defect. An unstarted process still proves the point: the stop is reached and the slot released, which is exactly what an unbounded Free prevents. Verified: make lint (new-from-merge-base origin/master) reports 0 issues; scripts/model-lifecycle-conformance.sh passes all three stages including the FizzBee liveness check (1458 states, IsLive: true). Assisted-by: Claude:claude-opus-4-8 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io>