materializeLocked built a download task for every file in the resolved
snapshot unconditionally. A completed file is promoted from
.downloads/<hash> into snapshot/<path> and its blob deleted, so on any
re-entry (a controller pod roll, a resubmit, a crash) the new pass built a
task whose .downloads blob no longer existed, re-downloaded the whole file
from Hugging Face, and its AfterDownload even removed the already-complete
snapshot copy first. The only resume that worked was the downloader's
per-file .partial resume for a file caught mid-transfer; completed files
were never skipped.
Production consequence: installing longcat-video-avatar-1.5 (~35 GB after
allow_patterns) on a cluster whose controller rolls hourly (Flux image
automation) never converged across ~14 hours. Each roll restarted from the
first shard; the completed bytes on disk were repeatedly deleted and
re-fetched, and the artifact never promoted. curl of the same files from
inside the pod ran fine, proving the loss was the materializer re-fetching,
not the network.
Before building a task, check whether the file is already materialized and
verified in this staging tree's snapshot/ and, if so, keep it and count it
complete instead of downloading. "Materialized" means a regular file of the
expected size that passes the same verifyDownloadedFile check the download
path uses, so the kept manifest entry is byte-for-byte identical to a fresh
one and integrity is re-checked. The manifest requires a SHA-256 for every
file and non-LFS files carry none to borrow, so a hash is unavoidable for
the manifest anyway; a full re-hash of local disk is still orders of
magnitude cheaper than re-downloading, and the downloader re-verifies any
file it does fetch. Manifest entries are now written at their snapshot index
rather than appended in completion order, so a mix of skipped and downloaded
files keeps the resolved order that committedResult and staging read. The
unconditional root.Remove(destination) now runs only on the fresh-download
path; a kept file survives. Skips are logged at INFO with count and bytes so
an operator can see resume working.
This is the resume-side counterpart to the sibling defects on this path:
read/write error conflation and transient retry (#10985), hash-verify
progress accounting and silent success on an expired deadline (#11026), and
the response-header hang (#11053). The download machinery resumed a single
in-flight file; the materializer above it still threw away every completed
file on restart. It also makes orphan-partial adoption worth its cost:
an adopted tree's completed files were re-downloaded anyway until now.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(model-artifacts): load single-file HF snapshots from the file, not the dir
The managed Hugging Face artifact materializer (#10825) always pointed
backends at the snapshot *directory*
(.artifacts/huggingface/<key>/snapshot). For a single-file model
reference such as huggingface://nomic-ai/nomic-embed-text-v1.5-GGUF/nomic-embed-text-v1.5.f16.gguf,
the GGUF lives *inside* that directory, so llama.cpp was handed a
directory and failed with "gguf_init_from_reader: failed to read magic".
This has kept the tests-aio job red on master since the feature merged
(the embeddings e2e tests could not load text-embedding-ada-002).
Record the single file of a one-file snapshot as Resolved.PrimaryFile and
have ModelFileName() resolve to snapshot/<PrimaryFile> when it is set.
Multi-file snapshots (e.g. transformers repos consumed as a directory)
keep pointing at the snapshot directory. PrimaryFile is derived from the
resolved contents and is deliberately excluded from the artifact cache
key. estimateModelSizeBytes now derives the snapshot directory from the
cache key instead of ModelFileName(), so its manifest lookup is unaffected
by the file-vs-directory resolution.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
