mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 09:57:57 -04:00
ec49548c8e
materializeLocked built a download task for every file in the resolved snapshot unconditionally. A completed file is promoted from .downloads/<hash> into snapshot/<path> and its blob deleted, so on any re-entry (a controller pod roll, a resubmit, a crash) the new pass built a task whose .downloads blob no longer existed, re-downloaded the whole file from Hugging Face, and its AfterDownload even removed the already-complete snapshot copy first. The only resume that worked was the downloader's per-file .partial resume for a file caught mid-transfer; completed files were never skipped. Production consequence: installing longcat-video-avatar-1.5 (~35 GB after allow_patterns) on a cluster whose controller rolls hourly (Flux image automation) never converged across ~14 hours. Each roll restarted from the first shard; the completed bytes on disk were repeatedly deleted and re-fetched, and the artifact never promoted. curl of the same files from inside the pod ran fine, proving the loss was the materializer re-fetching, not the network. Before building a task, check whether the file is already materialized and verified in this staging tree's snapshot/ and, if so, keep it and count it complete instead of downloading. "Materialized" means a regular file of the expected size that passes the same verifyDownloadedFile check the download path uses, so the kept manifest entry is byte-for-byte identical to a fresh one and integrity is re-checked. The manifest requires a SHA-256 for every file and non-LFS files carry none to borrow, so a hash is unavoidable for the manifest anyway; a full re-hash of local disk is still orders of magnitude cheaper than re-downloading, and the downloader re-verifies any file it does fetch. Manifest entries are now written at their snapshot index rather than appended in completion order, so a mix of skipped and downloaded files keeps the resolved order that committedResult and staging read. The unconditional root.Remove(destination) now runs only on the fresh-download path; a kept file survives. Skips are logged at INFO with count and bytes so an operator can see resume working. This is the resume-side counterpart to the sibling defects on this path: read/write error conflation and transient retry (#10985), hash-verify progress accounting and silent success on an expired deadline (#11026), and the response-header hang (#11053). The download machinery resumed a single in-flight file; the materializer above it still threw away every completed file on restart. It also makes orphan-partial adoption worth its cost: an adopted tree's completed files were re-downloaded anyway until now. Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>