mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 18:09:05 -04:00
Code Issues Packages Projects Releases Wiki Activity

fix(model-artifacts): materialize longcat-video on the controller, and support companion repos (#10949)

Browse Source 
* fix(model-artifacts): materialize longcat-video checkpoints on the controller

longcat-video loads a checkpoint directory: its backend.py takes
request.ModelFile when os.path.isdir(request.ModelFile) and otherwise
falls back to snapshot_download. That places it in the same class as
transformers/vllm/diffusers/sglang, but the allow-list added in #10910
did not enumerate it, so PrimaryArtifactSpec returned no managed
artifact for a bare HuggingFace repo id.

The consequence in distributed mode: nothing was acquired on the
controller, ModelFileName fell through to the raw repo id, and staging
skipped the resulting phantom /models/<owner>/<repo> path. The worker
received a blank ModelFile, fell back to request.Model, and downloaded
~83GB from HuggingFace inside the remote LoadModel deadline - so the
load could only ever fail with DeadlineExceeded while an abandoned
backend process kept downloading.

Note this materializes the full repository. The backend restricts its
own snapshot_download with allow_patterns, and the avatar repo ships
both base_model/ and base_model_int8/ where only one is ever loaded;
inferred specs have no way to carry patterns today. Tracked separately.

Assisted-by: Claude:opus-4.8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(distributed): warn when staging skips a non-existent model path

stageModelFiles logs "Staging model files for remote node" up front, then
silently drops any path field that does not exist on the controller. The
skip itself is legitimate and must stay: a backend outside
managedArtifactBackends that takes a bare HuggingFace repo id gets an
optimistically constructed path (ModelFileName falls through to the raw
model reference) that was never materialized, and sources its own weights
on the worker. Erroring would break those configs.

But at debug level the operator is left with a reassuring staging line and
no trace of the skip, so a genuine controller-side acquisition gap is
indistinguishable from a healthy pass-through - it surfaces much later as
a remote LoadModel timeout, on a worker that is quietly downloading tens
of gigabytes. Raise the skip to warn and name the field, path, node and
tracking key. Behavior is unchanged.

Assisted-by: Claude:opus-4.8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(model-artifacts): allow a config to declare companion artifacts

A composed pipeline needs more than one HuggingFace snapshot.
LongCat-Video-Avatar-1.5 loads its own transformer but takes the
tokenizer, text encoder and VAE from the separate LongCat-Video base
repo, so a single-artifact config cannot express it and the backend is
left to fetch the second repo itself at load time.

Widen the artifact model to target: model plus any number of named
target: companion entries. Normalize accepts the new target and
constrains a companion name to [a-z0-9][a-z0-9_-]{0,63} because that
name is the option key the backend later receives; a companion may not
claim primary_file, which only means anything for a load target.
ModelConfig.Validate requires exactly one primary and requires it first,
since Artifacts[0] is what ModelFileName, size estimation and staging all
resolve from.

Both acquisition paths now loop instead of touching index 0 alone:
preloadOne for an already-installed config, bindPrimaryArtifact for a
gallery install. Failure policy differs by provenance. An inferred
primary keeps its warn-and-fall-back, because the legacy download path
still exists for it. Companions are explicit by construction, so they are
all-or-nothing: a config naming one is asserting the backend needs it,
and failing at the acquisition boundary is far more legible than a
missing-weights error surfacing later inside the backend.

The cache key is deliberately unchanged. It hashes source identity only,
never name or target, so every already-installed managed model still hits
its existing snapshot instead of silently re-downloading. Two specs pin
that: one proving a companion and a primary with identical sources agree
on the key, and one pinning the digest of a known primary outright.

Assisted-by: Claude:opus-4.8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(model-artifacts): hand resolved companion snapshots to the backend

A materialized companion is useless until the backend can find it, and
its location is a content-addressed cache key that does not exist until
the artifact resolves. A static gallery override cannot carry that, and
persisting it into the config YAML would rot the moment a re-resolve
produced a new key.

Synthesize it instead at load time: each resolved companion becomes
"<artifact name>:<snapshot path>" in ModelOptions.Options, reusing the
key:value convention backends already parse for options like
attention_backend. The value stays relative to the models directory so a
remote worker can resolve it under its own ModelPath once staging has
rewritten the model root. An option the author set explicitly always
wins, so pinning a companion to a local checkout still beats the managed
snapshot.

longcat-video resolves base_model through ModelPath, the same convention
qwen-tts, voxcpm, outetts and ace-step already use for companion assets.
Its sibling-directory heuristic is deleted: it looked for a LongCat-Video
directory next to the model, which cannot exist under the content
addressed .artifacts/huggingface/<key>/snapshot layout, so it was dead
code the moment the model became managed.

The gallery entry declares both repositories and restricts each with
allow_patterns. The avatar repo ships base_model/ and base_model_int8/
and only ever loads one, so fetching the whole repo would roughly double
the download. The patterns match the entry's own options (use_distill
true, use_int8 default false); enabling use_int8 here also requires
adding base_model_int8/**, which is called out in the entry.

Assisted-by: Claude:opus-4.8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(distributed): stage managed artifact trees from the models root

Staging anchored the worker's models directory on the primary snapshot
whenever a model was managed, so a companion snapshot could not reach the
worker at all.

frontendModelsDir was derived by stripping the Model relative path off
the end of ModelFile. For a managed artifact nothing matches: ModelFile
is .artifacts/huggingface/<key>/snapshot while Model stays a bare
HuggingFace repo id, so the strip was a no-op and the "models directory"
came out as the snapshot itself. Two consequences, both silent. Staging
keys lost the .artifacts/huggingface/<key>/snapshot prefix, so two
snapshots of one model were indistinguishable on the worker. And a
companion, which lives in a sibling snapshot directory outside the
primary, fell outside that directory entirely: StagingKeyMapper.Key
collapsed its files to bare basenames and resolveOptionPath could not
resolve the relative option at all, so it was skipped without a word.

Derive the models root from the artifact tree instead when the path runs
through it, and compute the worker's ModelPath from the file's path
relative to that root rather than from the Model field. The legacy layout
is unaffected: where Model really is the relative path, the new
derivation reduces to the old one, which a regression spec pins.

This deliberately changes an invariant that router_dirstage_test.go
pinned: for a managed primary, ModelFile and ModelPath were both the
snapshot directory, and staging keys were relative to it. Now ModelFile
is the snapshot, ModelPath is the models root above it, and keys keep the
full relative path. That spec is updated rather than accommodated, with
the reasoning recorded inline, because the old invariant is exactly what
made a sibling companion unreachable.

Assisted-by: Claude:opus-4.8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
mudler's LocalAI [bot]
2026-07-19 12:01:36 +02:00
committed by GitHub
parent 09b85ee00e
commit 626ae4d51e
19 changed files with 1021 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
backend/python/longcat-video/backend.py
View File

@@ -109,6 +109,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
self.device_index = 0
self.cp_split_hw = None
self._dist_store_dir = None
self.model_path = ""
def Health(self, request, context):
return backend_pb2.Reply(message=b"OK")
@@ -118,6 +119,11 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
if request.ModelFile and os.path.isdir(request.ModelFile):
model = request.ModelFile
# A managed companion snapshot (see base_model) is passed as a path
# relative to the models directory, which is ModelPath here and a
# different directory on a remote worker once staging has rewritten it.
self.model_path = getattr(request, "ModelPath", "") or ""
model_kind = classify_model(model)
if model_kind is None:
return self._fail(
@@ -376,6 +382,26 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
)
self.cp_split_hw = self.context_parallel_util.get_optimal_split(1)
def _resolve_option_path(self, value):
"""Resolve an option that may name a local directory or a HF repo id.
A managed companion artifact is handed to us relative to the models
directory, so it only becomes a real path once joined with ModelPath;
that indirection is what lets the same config work on a remote worker,
where staging puts the snapshot somewhere else entirely. Anything that
is already an absolute directory, or is not a path at all (a plain repo
id), is passed through untouched for snapshot_download to handle.
"""
if not value:
return value
candidate = normalize_model_source(str(value))
if os.path.isabs(candidate) or not self.model_path:
return value
joined = os.path.join(self.model_path, candidate)
if os.path.isdir(joined):
return joined
return value
def _resolve_checkpoint(self, model, patterns):
source = normalize_model_source(model)
if os.path.isdir(source):
@@ -440,13 +466,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
avatar_patterns.append(f"{model_subfolder}/**")
checkpoint = self._resolve_checkpoint(model, avatar_patterns)
base_model = self.options.get("base_model")
if not base_model and os.path.isdir(normalize_model_source(model)):
sibling = os.path.join(
os.path.dirname(normalize_model_source(model)), "LongCat-Video"
)
if os.path.isdir(sibling):
base_model = sibling
base_model = self._resolve_option_path(self.options.get("base_model"))
base_model = base_model or BASE_MODEL_ID
if classify_model(str(base_model)) != MODEL_KIND_BASE:
raise ValueError("base_model must point to a LongCat-Video checkpoint")