mudler's LocalAI [bot] ec49548c8e fix(modelartifacts): resume interrupted materialization per-file, not from scratch (#11071) ...

materializeLocked built a download task for every file in the resolved
snapshot unconditionally. A completed file is promoted from
.downloads/<hash> into snapshot/<path> and its blob deleted, so on any
re-entry (a controller pod roll, a resubmit, a crash) the new pass built a
task whose .downloads blob no longer existed, re-downloaded the whole file
from Hugging Face, and its AfterDownload even removed the already-complete
snapshot copy first. The only resume that worked was the downloader's
per-file .partial resume for a file caught mid-transfer; completed files
were never skipped.

Production consequence: installing longcat-video-avatar-1.5 (~35 GB after
allow_patterns) on a cluster whose controller rolls hourly (Flux image
automation) never converged across ~14 hours. Each roll restarted from the
first shard; the completed bytes on disk were repeatedly deleted and
re-fetched, and the artifact never promoted. curl of the same files from
inside the pod ran fine, proving the loss was the materializer re-fetching,
not the network.

Before building a task, check whether the file is already materialized and
verified in this staging tree's snapshot/ and, if so, keep it and count it
complete instead of downloading. "Materialized" means a regular file of the
expected size that passes the same verifyDownloadedFile check the download
path uses, so the kept manifest entry is byte-for-byte identical to a fresh
one and integrity is re-checked. The manifest requires a SHA-256 for every
file and non-LFS files carry none to borrow, so a hash is unavoidable for
the manifest anyway; a full re-hash of local disk is still orders of
magnitude cheaper than re-downloading, and the downloader re-verifies any
file it does fetch. Manifest entries are now written at their snapshot index
rather than appended in completion order, so a mix of skipped and downloaded
files keeps the resolved order that committedResult and staging read. The
unconditional root.Remove(destination) now runs only on the fresh-download
path; a kept file survives. Skips are logged at INFO with count and bytes so
an operator can see resume working.

This is the resume-side counterpart to the sibling defects on this path:
read/write error conflation and transient retry (#10985), hash-verify
progress accounting and silent success on an expired deadline (#11026), and
the response-header hang (#11053). The download machinery resumed a single
in-flight file; the materializer above it still threw away every completed
file on restart. It also makes orphan-partial adoption worth its cost:
an adopted tree's completed files were re-downloaded anyway until now.


Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

2026-07-23 10:48:01 +02:00

22 KiB

Raw Blame History

View Raw