mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 09:57:57 -04:00
feat: materialize Hugging Face model artifacts (#10825)
* feat(config): add model artifact source contract Assisted-by: Codex:GPT-5 [Codex] * feat(downloader): add authenticated raw-byte progress Assisted-by: Codex:GPT-5 [Codex] * feat(huggingface): resolve immutable snapshot manifests Assisted-by: Codex:GPT-5 [Codex] * feat(models): add artifact storage primitives Assisted-by: Codex:GPT-5 [Codex] * feat(models): materialize pinned Hugging Face snapshots Assisted-by: Codex:GPT-5 [Codex] * feat(models): bind managed snapshots at runtime Assisted-by: Codex:GPT-5 [Codex] * feat(gallery): materialize model artifacts during install Assisted-by: Codex:GPT-5 [Codex] * feat(gallery): declare managed Hugging Face artifacts Assisted-by: Codex:GPT-5 [Codex] * feat(models): preload managed model artifacts Assisted-by: Codex:GPT-5 [Codex] * fix(gallery): retain shared artifact caches on delete Assisted-by: Codex:GPT-5 [Codex] * feat(models): report artifact acquisition progress Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): load managed models from ModelFile Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): load staged speech model snapshots Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): use staged snapshots in engine backends Assisted-by: Codex:GPT-5 [Codex] * test(distributed): cover staged artifact snapshots Assisted-by: Codex:GPT-5 [Codex] * docs: explain managed model artifacts Assisted-by: Codex:GPT-5 [Codex] * docs: add product design context Assisted-by: Codex:GPT-5 [Codex] * feat(ui): show model artifact download progress Assisted-by: Codex:GPT-5 [Codex] * Eagerly materialize Hugging Face artifacts Materialize HF-backed model references as managed GGUF artifacts during load, with lazy download retained only as fallback. Assisted-by: Codex:GPT-5 [shell] * Refactor HF downloads through a shared executor Assisted-by: Codex:GPT-5 [shell] * drop Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
LocalAI [bot]
@@ -89,6 +89,62 @@ download_files:
|
||||
sha256: abc123...
|
||||
```
|
||||
|
||||
## Model artifacts
|
||||
|
||||
The `artifacts` section makes installation of a Hugging Face model eager and
|
||||
repeatable. LocalAI resolves the requested revision to an immutable commit,
|
||||
downloads the selected repository files, and commits the complete snapshot
|
||||
before the model installation succeeds.
|
||||
|
||||
```yaml
|
||||
artifacts:
|
||||
- name: model
|
||||
target: model
|
||||
source:
|
||||
type: huggingface
|
||||
repo: Qwen/Qwen3-ASR-1.7B
|
||||
revision: main
|
||||
token_env: HF_TOKEN
|
||||
resolved:
|
||||
endpoint: https://huggingface.co
|
||||
revision: 0123456789abcdef0123456789abcdef01234567
|
||||
cache_key: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
|
||||
|
||||
parameters:
|
||||
model: Qwen/Qwen3-ASR-1.7B
|
||||
```
|
||||
|
||||
Declare `source` when authoring a configuration. LocalAI owns the `resolved`
|
||||
block and writes it after installation; do not choose its values manually.
|
||||
For a public repository, omit `token_env`. For a private or gated repository,
|
||||
set it to `HF_TOKEN` and provide that environment variable to the LocalAI
|
||||
controller.
|
||||
|
||||
| Field | Meaning |
|
||||
|-------|---------|
|
||||
| `name` | Logical artifact name; `model` for the initial primary artifact |
|
||||
| `target` | Binding target; only `model` is supported initially |
|
||||
| `source.type` | `huggingface` |
|
||||
| `source.repo` | `owner/repository` or `hf://owner/repository` |
|
||||
| `source.revision` | Branch, tag, or commit; defaults to `main` and resolves to a commit |
|
||||
| `source.token_env` | Empty or `HF_TOKEN`; the secret value is never persisted |
|
||||
| `source.allow_patterns` | Optional slash-separated glob allow-list |
|
||||
| `source.ignore_patterns` | Optional slash-separated glob deny-list |
|
||||
| `resolved` | Installer-owned immutable endpoint, revision, and cache key |
|
||||
|
||||
Managed installation finishes only after every selected file is committed
|
||||
locally. `parameters.model` remains the logical repository ID. Once
|
||||
`resolved.cache_key` is present, LocalAI derives
|
||||
`.artifacts/huggingface/<cache-key>/snapshot` as the runtime `ModelFile`.
|
||||
Configurations without `artifacts` keep the existing lazy repository-ID
|
||||
behavior.
|
||||
|
||||
The initially migrated backend families are `transformers` and its aliases,
|
||||
`diffusers`, `qwen-asr`, `fish-speech`, `nemo`, `voxcpm`, `qwen-tts`,
|
||||
`liquid-audio`, `vllm`, `vllm-omni`, and `sglang`. Automatic imports add
|
||||
artifact declarations only for this set. Compatible external backends may opt
|
||||
in by declaring the artifact explicitly.
|
||||
|
||||
## Parameters Section
|
||||
|
||||
The `parameters` section contains all OpenAI-compatible request parameters and model-specific options.
|
||||
|
||||
@@ -142,6 +142,23 @@ Set `LOCALAI_DISTRIBUTED_SHARED_MODELS=true` (or `--distributed-shared-models`)
|
||||
|
||||
This flag is a contract you assert: all nodes must mount identical paths. Leave it off (the default) when workers have independent models directories - the frontend stages files to them over HTTP (or S3) as described above.
|
||||
|
||||
### Model artifact staging
|
||||
|
||||
For managed Hugging Face artifacts, the controller resolves the repository and
|
||||
downloads every selected file. Workers receive the committed snapshot through
|
||||
the existing directory stager. They never receive `HF_TOKEN` and do not contact
|
||||
Hugging Face for managed artifacts.
|
||||
|
||||
With `LOCALAI_DISTRIBUTED_SHARED_MODELS` enabled, workers use the shared
|
||||
absolute snapshot path and skip transfer. Otherwise, the controller stages the
|
||||
complete snapshot tree to each worker before loading the backend.
|
||||
|
||||
{{% notice warning %}}
|
||||
Every controller and worker must have enough disk space for its own snapshot
|
||||
copy unless shared-models mode is enabled. Account for temporary partial files
|
||||
during installation as well as the committed snapshot.
|
||||
{{% /notice %}}
|
||||
|
||||
{{% notice warning %}}
|
||||
The worker HTTP file transfer server is authenticated by `LOCALAI_REGISTRATION_TOKEN`. If the token is **empty**, the server **fails open** — anyone who can reach the port gets read/write access to the worker's models/staging/data directories (a remote model-poisoning / exfiltration vector). The worker logs a loud warning at startup in this case. Always set `LOCALAI_REGISTRATION_TOKEN` in distributed mode, and set `LOCALAI_DISTRIBUTED_REQUIRE_AUTH=true` (frontend **and** workers) to make a missing token *or* missing NATS credentials a hard startup error rather than a silent fail-open. Firewall the file-transfer port (gRPC base − 1) so only the frontend can reach it.
|
||||
{{% /notice %}}
|
||||
|
||||
@@ -151,6 +151,25 @@ where:
|
||||
- `bert-embeddings` is the model name in the gallery
|
||||
(read its [config here](https://github.com/mudler/LocalAI/tree/master/gallery/blob/main/bert-embeddings.yaml)).
|
||||
|
||||
### Artifact-backed models
|
||||
|
||||
Gallery models with an `artifacts` declaration are fully materialized during
|
||||
installation. Their operation progresses through these phases:
|
||||
|
||||
```text
|
||||
resolving -> downloading -> verifying -> committing -> persisting
|
||||
```
|
||||
|
||||
The admin Operations Bar and `GET /api/operations` expose `currentBytes` and
|
||||
`totalBytes` as raw transport bytes. Cancelling an active download leaves its
|
||||
partial files in place so a retry can resume. A verification failure never
|
||||
exposes a completed snapshot, while a retry or another installation reuses an
|
||||
already verified content-addressed snapshot.
|
||||
|
||||
Deleting a model configuration does not delete its content-addressed snapshot
|
||||
bytes. This allows another configuration or a later reinstall to reuse the
|
||||
cache; safe cache garbage collection is deferred.
|
||||
|
||||
### How to install a model not part of a gallery
|
||||
|
||||
If you don't want to set any gallery repository, you can still install models by loading a model configuration file.
|
||||
|
||||
Reference in New Issue
Block a user