diff --git a/.github/ci/apexentries/README.md b/.github/ci/apexentries/README.md new file mode 100644 index 000000000..cd4f47537 --- /dev/null +++ b/.github/ci/apexentries/README.md @@ -0,0 +1,194 @@ +# apexentries + +Generates gallery entries for the `mudler/*-APEX-GGUF` HuggingFace repositories. + +Each APEX repo becomes one **family**: one entry per quality rung the repo +publishes and one per quantization rung its unsloth counterpart publishes, all +gathered under the **base model's** entry. LocalAI's variant selector then picks +the build that fits the hardware in front of it. + +## The hub is the base model entry, never a generated `*-apex` parent + +Somebody looking for `qwen3.6-35b-a3b` must find every build of those weights +under that one name: the APEX imatrix rungs, the unsloth quant rungs and any +speculative build. A separate `qwen3.6-35b-a3b-apex` hub competing with the base +entry would split the family in two and leave whichever half the user did not +search for effectively invisible. + +So the generator resolves the hub by stripping the `-APEX`, `-MTP` and `-TQ` +markers and looking the result up in the index, trying both the repo-derived and +the stem-derived candidate the same way `CounterpartCandidates` does. Then: + +- **The hub exists** (14 of the 45 repos, resolving to 10 distinct entries). + Nothing new is emitted for the family root. A `variants:` block is spliced into + the entry that is already there, textually, leaving its description, icon, + tags, overrides and files untouched. The line editing is shared with the + `variantproposals` job via `.github/ci/galleryedit`. +- **The hub is absent** (the other 31). A new hub is emitted, named for the base + model and never for the APEX repo. It carries one of the discovered builds as + its own payload so it is a complete installable entry rather than a bare index, + and that payload is what gives it an `overrides.backend`. Without a declared + backend the verifier would skip it, so a hub carrying feature tags would escape + the tagging check in silence. + +Several APEX repos routinely resolve to one base model, so both paths accumulate +by hub name rather than assuming one family per hub. + +Two references are always filtered out of a hub's list: anything the entry +already declares, and the hub's own name. The self reference is not merely +redundant. An unsloth rung whose weights the gallery already ships under the base +name resolves, through the merge, straight back to the hub, and the verifier +reads a self reference as a variant that declares variants of its own. + +The four hand-written `*-apex` entries (`qwen3.6-35b-a3b-apex`, +`gemma-4-26b-a4b-it-apex`, `qwen3.5-35b-a3b-apex`, +`nemotron-3-nano-omni-30b-a3b-reasoning-apex`) are **ordinary builds**, not hubs. +They are referenced from their hub's variants list like any other rung, and are +never deleted or renamed. + +## Flags + +| Flag | Default | Meaning | +|------|---------|---------| +| `-index ` | `gallery/index.yaml` | Gallery index to dedup against. Read only, unless `-apply` is passed. | +| `-only ` | (all) | Comma-separated full repo names (`mudler/Foo-APEX-GGUF`) to restrict generation to. A name that matches nothing is reported as a warning, since it is a typo rather than an empty result. | +| `-out ` | (none) | Write the entries to add to this file. Nothing is written to the gallery. | +| `-apply` | `false` | Splice the variants into `-index` and append the new entries to it. | +| `-verify ` | (none) | Verify a gallery index and exit. Ignores every other flag. | + +Either `-out` or `-apply` is required, otherwise the run has nothing to do. + +`-apply` splices variant lines into existing entries and **appends** new ones. It +never re-serialises the index: it is roughly 40,000 lines, and a YAML round trip +would reflow the whole file, drop the anchors and merge keys the gallery relies +on, and produce a diff nobody can review. On the three-family sample the splice +is 24 added lines across 3 hunks with zero deletions. + +## Discovery is by filename suffix, never by repo name + +Builds come from the files a repo actually publishes. A filename is never +constructed from a repo name, because the two disagree: +`mudler/gemma-4-26B-A4B-it-APEX-GGUF` ships `gemma-4-26B-A4B-APEX-*.gguf`, and +five other repos likewise drop a suffix (`-it`, `-2603`) or a vendor prefix +(`NVIDIA-`) that the repo name carries. Composing a URL from the repo name would +produce a 404 for every one of them, and the 404 would only surface after the +entry shipped. + +The quality ladder is matched on the trailing tier marker, `-(I-)?(Quality| +Balanced|Compact|Mini|Nano).gguf`. The `I-` prefix marks the imatrix ladder. The +imatrix ladder is emitted when it is non-empty and the plain ladder is used only +as a fallback, because two of the 45 repos publish no imatrix tiers at all and +must still contribute. Eleven repos carry a fifth `I-Nano` rung, so nothing +assumes a fixed number of rungs. + +Every run prints, per repo, the counts that discovery accounted for. If the +number of classified files is short of the number of `.gguf` files the repo +publishes, the shortfall is printed as `UNCLASSIFIED`. That check is a set +difference on counts rather than a second pass over filenames: a second matcher +would duplicate the tier regex and the two copies would drift. The failure it +catches is quiet. A publishing-script typo that breaks every imatrix filename in +a repo does not produce a short ladder; it makes the imatrix ladder empty, and +the fallback then downgrades the whole family to the plain ladder with nothing +said. A downstream HTTP check cannot catch it either, because it validates the +URLs that were emitted, and an undiscovered tier emits none. + +The same reasoning applies to `UNACCOUNTED QUANT`, printed when the unsloth +counterpart demonstrably publishes a wanted quant that produced no build. It is +reported at discovery time because a dropped quant leaves no trace at all in the +finished gallery file. + +## sha256 always comes from the API + +Every file stanza takes its `sha256` from the HuggingFace models API +(`lfs.sha256`). A GGUF the API describes without one is a fatal error for that +family: the repo is reported by name and the run ends non-zero. It is never +substituted from another field, because that is exactly how a Xet hash ends up +masquerading as a content hash. + +## The dflash / mtp tagging rule + +An entry is tagged `dflash` or `mtp` **if and only if** it configures the +matching `spec_type:draft-`. Variant ranking reads tags and nothing +else, so a tag that does not match the configuration either promotes a build +that is no faster or hides one that genuinely is. + +A repo name is not configuration. `mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF` ships +weights that carry MTP heads; an entry that does not enable them is not an MTP +entry and is not tagged as one. + +A generated hub inherits the tags of the build it carries as its payload, rather +than rebuilding them from the base set, so a hub whose payload configures a +`spec_type` stays tagged consistently with the overrides copied alongside it. + +## Reuse reporting: two categories, not one + +Generated entries are deduped against the gallery and against the batch itself. +The run prints the result under two separate headings, because the two cases are +not equivalent: + +- **URI MATCHES** mean the gallery, or an earlier entry in this batch, already + ships exactly these weights. Pointing the hub at the existing entry is correct + and needs no thought. +- **NAME COLLISIONS** mean an entry already owns the name but holds different + weights. Referencing it would point the hub at a build other than the one + generated. Every one of these must be inspected by hand. + +The run then prints `HUBS SPLICED`, listing every reference that will be added to +an entry the gallery already ships along with the line it will be added at, and +`HUBS CREATED` for the families that get a new hub. The splices are the part a +review has to read closely, because they modify entries somebody else wrote. + +Hubs are deliberately kept out of the merge. A new hub carries the family's top +rung as its own payload, so URI dedup would fold the hub into that rung and the +family would lose the very entry point this command exists to create. + +## Workflow: sample first, then the full set + +Never run the full generation straight into the gallery. Generate a small, +deliberately awkward sample, have it reviewed, then run the rest. + +```bash +# 1. Sample three families that between them cover the awkward shapes: +# a standard four-rung repo, one with the extra I-Nano rung AND a file stem +# that differs from its repo name, and one whose unsloth counterpart shards +# its quants across subdirectories. +go run ./.github/ci/apexentries \ + -index gallery/index.yaml \ + -only mudler/Qwen3.6-35B-A3B-APEX-GGUF,mudler/gemma-4-26B-A4B-it-APEX-GGUF,mudler/Step-3.7-Flash-APEX-GGUF \ + -out /tmp/sample.yaml + +# 2. Verify the sample against the gallery it would join, splices included. Apply +# to a COPY, never to the real index, and check that the diff is only the +# intended variant lines. Compare the verifier output to the gallery's own +# baseline: what matters is that the sample adds no new problem, not that the +# total is zero. +cp gallery/index.yaml /tmp/index-copy.yaml +go run ./.github/ci/apexentries -index /tmp/index-copy.yaml -only -apply +diff -u gallery/index.yaml /tmp/index-copy.yaml # expect zero deletions + +go run ./.github/ci/apexentries -verify gallery/index.yaml > /tmp/baseline.log 2>&1 +go run ./.github/ci/apexentries -verify /tmp/index-copy.yaml > /tmp/spliced.log 2>&1 +diff /tmp/baseline.log /tmp/spliced.log + +# 3. Have a human review /tmp/sample.yaml and every reported name collision. + +# 4. Only then, the full set. +go run ./.github/ci/apexentries -index gallery/index.yaml -apply +``` + +## Tests + +```bash +go test ./.github/ci/apexentries/ +``` + +The shared line editor has its own package: + +```bash +go test ./.github/ci/galleryedit/ +``` + +`.github/ci/` is invisible to `go list ./...`, so these specs are not covered by +`make lint` or the repository test run. `.github/workflows/ci-tools-tests.yaml` +names the package explicitly; keep that workflow in step with any package added +under `.github/ci/`. diff --git a/.github/ci/apexentries/discover.go b/.github/ci/apexentries/discover.go new file mode 100644 index 000000000..4320ee5e8 --- /dev/null +++ b/.github/ci/apexentries/discover.go @@ -0,0 +1,70 @@ +package main + +import ( + "regexp" + "strings" +) + +// tierRE matches the tier marker APEX repos put at the end of a weight +// filename. Discovery is by suffix because the stem is not predictable from +// the repo name: six of the 45 repos drop a suffix ("-it", "-2603") or a +// vendor prefix ("NVIDIA-") that the repo name carries. +var tierRE = regexp.MustCompile(`-(I-)?(Quality|Balanced|Compact|Mini|Nano)\.gguf$`) + +// fullPrecisionRE matches the unquantized source weights an APEX repo publishes +// alongside its ladder, flat (-F16.gguf) or sharded across a numbered set +// (-F16-00001-of-00010.gguf). bf16 is accepted because some repos publish that +// instead, and the match is case-insensitive because the casing varies between +// publishing scripts. +// +// These are deliberately not tiers: they are the weights the ladder is quantized +// FROM, and generation is scoped to the ladder itself. +var fullPrecisionRE = regexp.MustCompile(`(?i)-b?f16(-\d{5}-of-\d{5})?\.gguf$`) + +// IsFullPrecision reports whether a weight filename is an unquantized source. +func IsFullPrecision(name string) bool { + return fullPrecisionRE.MatchString(name) +} + +// Tier is one discovered build of an APEX repo. +type Tier struct { + Label string + File GGUFFile +} + +// DiscoverAPEXTiers splits a repo's weight files into the imatrix ladder and +// the plain ladder. mmproj files are never tiers. +func DiscoverAPEXTiers(files []GGUFFile) (imatrix, plain []Tier) { + for _, f := range files { + if strings.HasPrefix(f.Name, "mmproj") { + continue + } + m := tierRE.FindStringSubmatch(f.Name) + if m == nil { + continue + } + if m[1] != "" { + imatrix = append(imatrix, Tier{Label: "I-" + m[2], File: f}) + continue + } + plain = append(plain, Tier{Label: m[2], File: f}) + } + return imatrix, plain +} + +// DiscoverMMProj returns the repo's projector file, if it publishes one. The +// name varies across repos (mmproj.gguf, mmproj-F16.gguf, +// mmproj-step3.7-flash-f16.gguf), so match the prefix rather than a fixed name. +func DiscoverMMProj(files []GGUFFile) (GGUFFile, bool) { + for _, f := range files { + if strings.HasPrefix(f.Name, "mmproj") { + return f, true + } + } + return GGUFFile{}, false +} + +// FileStem returns a tier's filename with its tier suffix removed. +func FileStem(t Tier) string { + return tierRE.ReplaceAllString(t.File.Name, "") +} diff --git a/.github/ci/apexentries/discover_test.go b/.github/ci/apexentries/discover_test.go new file mode 100644 index 000000000..6bca3d2f4 --- /dev/null +++ b/.github/ci/apexentries/discover_test.go @@ -0,0 +1,68 @@ +package main + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("DiscoverAPEXTiers", func() { + It("finds tiers regardless of how the stem relates to the repo name", func() { + // This repo is mudler/gemma-4-26B-A4B-it-APEX-GGUF but its files drop "-it". + files := []GGUFFile{ + {Name: "gemma-4-26B-A4B-APEX-I-Quality.gguf", SHA256: "a"}, + {Name: "gemma-4-26B-A4B-APEX-I-Nano.gguf", SHA256: "b"}, + {Name: "gemma-4-26B-A4B-APEX-Quality.gguf", SHA256: "c"}, + {Name: "mmproj-F16.gguf", SHA256: "d"}, + } + + imatrix, plain := DiscoverAPEXTiers(files) + + Expect(labels(imatrix)).To(ConsistOf("I-Quality", "I-Nano")) + Expect(labels(plain)).To(ConsistOf("Quality")) + }) + + It("excludes mmproj from the tier list", func() { + files := []GGUFFile{{Name: "mmproj.gguf", SHA256: "d"}} + + imatrix, plain := DiscoverAPEXTiers(files) + + Expect(imatrix).To(BeEmpty()) + Expect(plain).To(BeEmpty()) + }) +}) + +var _ = Describe("DiscoverMMProj", func() { + It("finds an mmproj whatever its suffix", func() { + files := []GGUFFile{ + {Name: "Model-APEX-I-Mini.gguf", SHA256: "a"}, + {Name: "mmproj-step3.7-flash-f16.gguf", SHA256: "b"}, + } + + got, ok := DiscoverMMProj(files) + + Expect(ok).To(BeTrue()) + Expect(got.Name).To(Equal("mmproj-step3.7-flash-f16.gguf")) + }) + + It("reports absence when the repo ships none", func() { + _, ok := DiscoverMMProj([]GGUFFile{{Name: "Model-APEX-Quality.gguf", SHA256: "a"}}) + + Expect(ok).To(BeFalse()) + }) +}) + +var _ = Describe("FileStem", func() { + It("strips the tier suffix", func() { + t := Tier{Label: "I-Quality", File: GGUFFile{Name: "gemma-4-26B-A4B-APEX-I-Quality.gguf"}} + + Expect(FileStem(t)).To(Equal("gemma-4-26B-A4B-APEX")) + }) +}) + +func labels(ts []Tier) []string { + out := make([]string, 0, len(ts)) + for _, t := range ts { + out = append(out, t.Label) + } + return out +} diff --git a/.github/ci/apexentries/hf.go b/.github/ci/apexentries/hf.go new file mode 100644 index 000000000..27486064e --- /dev/null +++ b/.github/ci/apexentries/hf.go @@ -0,0 +1,130 @@ +package main + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "strings" + "time" + + "github.com/mudler/LocalAI/pkg/httpclient" +) + +// ErrNoSHA256 marks a GGUF the HuggingFace API describes without an +// lfs.sha256. Emitting an entry without a hash would ship an unverifiable +// download, and guessing one from another field is how a Xet hash ends up +// masquerading as a content hash, so this is fatal rather than skippable. +var ErrNoSHA256 = errors.New("gguf file has no lfs.sha256") + +// GGUFFile is one .gguf sibling of a HuggingFace repo. +type GGUFFile struct { + Name string + Size int64 + SHA256 string +} + +type apiSibling struct { + RFilename string `json:"rfilename"` + Size int64 `json:"size"` + LFS *struct { + SHA256 string `json:"sha256"` + } `json:"lfs"` +} + +type apiModel struct { + Siblings []apiSibling `json:"siblings"` +} + +// ParseRepoFiles returns every .gguf sibling described by a models API body. +func ParseRepoFiles(body []byte) ([]GGUFFile, error) { + var m apiModel + if err := json.Unmarshal(body, &m); err != nil { + return nil, fmt.Errorf("decoding model response: %w", err) + } + + var out []GGUFFile + for _, s := range m.Siblings { + if !strings.HasSuffix(s.RFilename, ".gguf") { + continue + } + if s.LFS == nil || s.LFS.SHA256 == "" { + return nil, fmt.Errorf("%s: %w", s.RFilename, ErrNoSHA256) + } + out = append(out, GGUFFile{Name: s.RFilename, Size: s.Size, SHA256: s.LFS.SHA256}) + } + return out, nil +} + +// FetchOptionalRepoFiles asks the models API for a repo the caller can do +// without, and reports separately whether the repo was merely unreadable. +// +// HuggingFace answers 401 Unauthorized, not 404, for a repo that does not exist +// when the request carries no credentials. Without a token there is therefore no +// way to tell "this repo was never published" from "this repo is private", so an +// optional probe has to treat 401 and 403 exactly like 404: whatever the reason, +// there is nothing here for us to read, so there is no counterpart. +// +// The second return value exists because that collapse is lossy in one +// direction: 401/403 can also mean a real, gated repo whose quants we would +// genuinely want. The caller reports those repos so a silently dropped +// counterpart is visible to a human rather than invisible. +func FetchOptionalRepoFiles(client *http.Client, repo string) ([]GGUFFile, bool, error) { + files, status, err := fetchRepoFiles(client, repo) + if err != nil && (status == http.StatusUnauthorized || status == http.StatusForbidden) { + return nil, true, nil + } + return files, false, err +} + +// FetchRepoFiles asks the models API for one repo. A 404 yields (nil, nil) so +// that probing for an optional counterpart repo is not an error. Every other +// non-200, 401 and 403 included, is an error: for a repo the run REQUIRES there +// is no benign reading of "we cannot see it". +func FetchRepoFiles(client *http.Client, repo string) ([]GGUFFile, error) { + files, _, err := fetchRepoFiles(client, repo) + return files, err +} + +// fetchRepoFiles does the request and returns the HTTP status alongside the +// result, so the optional and required callers can apply different policies to +// the same response without duplicating the request. +func fetchRepoFiles(client *http.Client, repo string) ([]GGUFFile, int, error) { + url := fmt.Sprintf("https://huggingface.co/api/models/%s?blobs=true", repo) + req, err := http.NewRequest(http.MethodGet, url, nil) + if err != nil { + return nil, 0, err + } + req.Header.Set("User-Agent", "localai-apexentries/1.0") + + resp, err := client.Do(req) + if err != nil { + return nil, 0, err + } + defer resp.Body.Close() + + if resp.StatusCode == http.StatusNotFound { + return nil, resp.StatusCode, nil + } + if resp.StatusCode != http.StatusOK { + return nil, resp.StatusCode, fmt.Errorf("%s: unexpected status %d", repo, resp.StatusCode) + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + return nil, resp.StatusCode, err + } + files, err := ParseRepoFiles(body) + return files, resp.StatusCode, err +} + +// newHTTPClient builds the client used against the HuggingFace API. It goes +// through pkg/httpclient rather than a bare &http.Client{} because the std +// client follows redirects and forwards custom credential headers to the +// redirect target on a cross-host hop (GHSA-3mj3-57v2-4636). This caller sends +// only a User-Agent today, but it talks to an external API that could start +// redirecting, and an HF_TOKEN header here later would then leak. +func newHTTPClient() *http.Client { + return httpclient.NewWithTimeout(60 * time.Second) +} diff --git a/.github/ci/apexentries/hf_test.go b/.github/ci/apexentries/hf_test.go new file mode 100644 index 000000000..3dbeb0cc2 --- /dev/null +++ b/.github/ci/apexentries/hf_test.go @@ -0,0 +1,142 @@ +package main + +import ( + "bytes" + "io" + "net/http" + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestApexEntries(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "apexentries") +} + +// stubTransport answers every request with one canned status and body, so the +// status handling of the fetchers can be exercised without reaching the real +// HuggingFace API. +type stubTransport struct { + status int + body string +} + +func (t stubTransport) RoundTrip(req *http.Request) (*http.Response, error) { + return &http.Response{ + StatusCode: t.status, + Body: io.NopCloser(bytes.NewBufferString(t.body)), + Header: make(http.Header), + Request: req, + }, nil +} + +func stubClient(status int, body string) *http.Client { + return &http.Client{Transport: stubTransport{status: status, body: body}} +} + +const oneGGUFBody = `{"siblings":[{"rfilename":"Model-APEX-I-Quality.gguf","size":10,"lfs":{"sha256":"aa","size":10}}]}` + +var _ = Describe("FetchOptionalRepoFiles", func() { + // HuggingFace answers 401 rather than 404 for a repo that does not exist + // when the client carries no credentials, so an optional probe cannot tell + // "absent" from "unauthorized" and must treat both as "no counterpart". + It("treats a 401 as an absent repo and flags it as unavailable", func() { + files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusUnauthorized, ""), "unsloth/Nope-GGUF") + + Expect(err).ToNot(HaveOccurred()) + Expect(files).To(BeEmpty()) + Expect(unavailable).To(BeTrue()) + }) + + It("treats a 403 as an absent repo and flags it as unavailable", func() { + files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusForbidden, ""), "unsloth/Gated-GGUF") + + Expect(err).ToNot(HaveOccurred()) + Expect(files).To(BeEmpty()) + Expect(unavailable).To(BeTrue()) + }) + + // A clean 404 is an unambiguous absence, so it must NOT be reported as + // unavailable: the whole point of the flag is to separate the ambiguous + // case a human may need to look at from the settled one. + It("treats a 404 as an absent repo without flagging it as unavailable", func() { + files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusNotFound, ""), "unsloth/Nope-GGUF") + + Expect(err).ToNot(HaveOccurred()) + Expect(files).To(BeEmpty()) + Expect(unavailable).To(BeFalse()) + }) + + It("parses a 200 body as usual", func() { + files, unavailable, err := FetchOptionalRepoFiles(stubClient(http.StatusOK, oneGGUFBody), "unsloth/Real-GGUF") + + Expect(err).ToNot(HaveOccurred()) + Expect(unavailable).To(BeFalse()) + Expect(files).To(HaveLen(1)) + Expect(files[0].Name).To(Equal("Model-APEX-I-Quality.gguf")) + Expect(files[0].SHA256).To(Equal("aa")) + }) + + // Tolerating 401/403 must not widen into tolerating everything: a 500 is a + // broken API, not evidence about whether the repo exists. + It("still errors on a 500", func() { + _, _, err := FetchOptionalRepoFiles(stubClient(http.StatusInternalServerError, ""), "unsloth/Real-GGUF") + + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("unexpected status 500")) + }) +}) + +var _ = Describe("FetchRepoFiles", func() { + // The APEX repo itself is not optional. A 401 there means the repo the run + // was asked to publish cannot be read, which is a real failure and must not + // be quietly downgraded to "no files". + It("errors on a 401 for a required repo", func() { + _, err := FetchRepoFiles(stubClient(http.StatusUnauthorized, ""), "mudler/Model-APEX-GGUF") + + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("unexpected status 401")) + }) + + It("errors on a 403 for a required repo", func() { + _, err := FetchRepoFiles(stubClient(http.StatusForbidden, ""), "mudler/Model-APEX-GGUF") + + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("unexpected status 403")) + }) + + It("still treats a 404 as an absent repo", func() { + files, err := FetchRepoFiles(stubClient(http.StatusNotFound, ""), "mudler/Model-APEX-GGUF") + + Expect(err).ToNot(HaveOccurred()) + Expect(files).To(BeEmpty()) + }) +}) + +var _ = Describe("ParseRepoFiles", func() { + It("returns gguf siblings with their lfs sha256", func() { + body := []byte(`{"siblings":[ + {"rfilename":"Model-APEX-I-Quality.gguf","size":10,"lfs":{"sha256":"aa","size":10}}, + {"rfilename":"README.md"}, + {"rfilename":"mmproj.gguf","size":5,"lfs":{"sha256":"bb","size":5}} + ]}`) + + files, err := ParseRepoFiles(body) + + Expect(err).ToNot(HaveOccurred()) + Expect(files).To(HaveLen(2)) + Expect(files[0].Name).To(Equal("Model-APEX-I-Quality.gguf")) + Expect(files[0].SHA256).To(Equal("aa")) + Expect(files[1].Name).To(Equal("mmproj.gguf")) + }) + + It("reports a gguf that carries no lfs sha256", func() { + body := []byte(`{"siblings":[{"rfilename":"mmproj.gguf","size":5}]}`) + + _, err := ParseRepoFiles(body) + + Expect(err).To(MatchError(ErrNoSHA256)) + }) +}) diff --git a/.github/ci/apexentries/hub.go b/.github/ci/apexentries/hub.go new file mode 100644 index 000000000..103c30b54 --- /dev/null +++ b/.github/ci/apexentries/hub.go @@ -0,0 +1,143 @@ +package main + +import ( + "fmt" + "os" + "strings" + + "gopkg.in/yaml.v3" + + "github.com/mudler/LocalAI/.github/ci/galleryedit" +) + +// IndexText is the gallery index seen as text: the entries it declares plus the +// exact lines each one occupies, which is what splicing a variants block into an +// entry the gallery already ships requires. +// +// It is a second, narrower read of the same file LoadExisting parses. The two +// answer different questions: LoadExisting answers "do these weights already +// exist anywhere", this one answers "where in the file does this entry live". +type IndexText struct { + Lines []string + Entries []*indexEntry + + byName map[string]*indexEntry +} + +// indexEntry is one entry of the index: its name, the variants it already +// declares, and its coordinates in the file. +type indexEntry struct { + Name string `yaml:"name"` + Variants []VariantRef `yaml:"variants"` + + Pos galleryedit.Entry `yaml:"-"` +} + +// LoadIndexText reads the gallery index for editing. +func LoadIndexText(path string) (*IndexText, error) { + raw, err := os.ReadFile(path) + if err != nil { + return nil, err + } + return ParseIndexText(string(raw)) +} + +// ParseIndexText pairs the decoded entries with the top level list items the +// text actually contains. +// +// If the two views disagree on how many entries there are then every line number +// a splice would compute is suspect, and the failure mode is writing a variants +// block into the wrong model. The parse refuses instead. +func ParseIndexText(text string) (*IndexText, error) { + var entries []*indexEntry + if err := yaml.Unmarshal([]byte(text), &entries); err != nil { + return nil, fmt.Errorf("decoding gallery index: %w", err) + } + + lines, starts := galleryedit.Scan(text) + if len(starts) != len(entries) { + return nil, fmt.Errorf("gallery index has %d decoded entries but %d top level list items; refusing to edit by line number", + len(entries), len(starts)) + } + + ix := &IndexText{Lines: lines, Entries: entries, byName: map[string]*indexEntry{}} + for i, e := range entries { + if e == nil { + return nil, fmt.Errorf("gallery index list item %d is empty; refusing to edit by line number", i) + } + end := len(lines) + if i+1 < len(starts) { + end = starts[i+1] + } + e.Pos = galleryedit.Entry{Name: e.Name, StartLine: starts[i], EndLine: end} + + // First occurrence wins, matching the gallery's own resolution. + key := strings.ToLower(e.Name) + if _, seen := ix.byName[key]; !seen { + ix.byName[key] = e + } + } + return ix, nil +} + +// Find looks an entry up by name, case insensitively. +func (ix *IndexText) Find(name string) *indexEntry { + return ix.byName[strings.ToLower(name)] +} + +// ResolveHub returns the gallery name of a family's hub and whether the gallery +// already ships an entry under it. +// +// The hub is the BASE model entry, never a generated *-apex parent. Somebody +// looking for qwen3.6-35b-a3b has to find every build of those weights under +// that one name: the APEX imatrix rungs, the unsloth quant rungs and any +// speculative build. A separate qwen3.6-35b-a3b-apex hub competing with the base +// entry would split the family in two and leave whichever half the user did not +// search for invisible. +// +// Both candidates are tried for the same reason CounterpartCandidates tries +// both. The repo name and the published file stem disagree for several of these +// repos, and either one may be what the base entry was named after. +func ResolveHub(ix *IndexText, repoBase, stem string) (name string, exists bool) { + candidates := CounterpartCandidates(repoBase, stem) + for _, c := range candidates { + if n := slug(c); ix.Find(n) != nil { + return n, true + } + } + // Nothing matched, so the family needs a hub of its own under the repo + // derived name, which is the more reliable of the two. + return slug(candidates[0]), false +} + +// HubLabel is the human-cased base model name, for prose rather than lookup. +func HubLabel(repoBase, stem string) string { + return CounterpartCandidates(repoBase, stem)[0] +} + +// filterVariants drops the references a hub must not carry: itself, and anything +// it already lists. +// +// The self reference is not merely redundant. A hub that names itself makes the +// verifier resolve the reference back to the hub, see that the hub declares +// variants, and report a variant that declares variants of its own. It arises +// for real rather than in theory: an unsloth rung whose weights the gallery +// already ships under the base model name resolves, through Merge, straight back +// to the hub that is about to reference it. +func filterVariants(hub string, already []VariantRef, want []string) []string { + seen := map[string]bool{strings.ToLower(hub): true} + for _, v := range already { + seen[strings.ToLower(v.Model)] = true + } + + var out []string + for _, w := range want { + key := strings.ToLower(w) + if seen[key] { + continue + } + seen[key] = true + out = append(out, w) + } + return out +} diff --git a/.github/ci/apexentries/main.go b/.github/ci/apexentries/main.go new file mode 100644 index 000000000..d561f108b --- /dev/null +++ b/.github/ci/apexentries/main.go @@ -0,0 +1,795 @@ +// Command apexentries generates gallery entries for the mudler APEX GGUF +// repositories: one entry per imatrix tier and per unsloth quant rung, all +// gathered under the BASE model's entry. Builds off a *-APEX-MTP-GGUF repo turn +// speculative decoding on, because those weights retain the model's MTP heads +// and are only worth their extra size with the heads in use. +// +// The base model entry is the hub. Somebody looking for qwen3.6-35b-a3b must +// find every build of those weights under that one name, so when the gallery +// already ships the base entry this command splices a variants block into it +// rather than emitting a competing *-apex parent beside it. Only a family whose +// base model the gallery does not ship at all gets a new hub entry, and that one +// is still named for the base model. +// +// Builds are discovered by inspecting the filenames a repo actually publishes. +// Repo names do not reliably predict them: mudler/gemma-4-26B-A4B-it-APEX-GGUF +// ships gemma-4-26B-A4B-APEX-*.gguf, and six of the 45 repos drop a suffix or a +// vendor prefix in the same way. +package main + +import ( + "encoding/json" + "flag" + "fmt" + "io" + "net/http" + "os" + "path" + "sort" + "strings" + + "gopkg.in/yaml.v3" + + "github.com/mudler/LocalAI/.github/ci/galleryedit" +) + +const ( + // entryTemplate carries no backend and no parameters of its own, which is + // why RenderChild states everything inline. + entryTemplate = "virtual.yaml" + unslothOwner = "unsloth" + authorListURL = "https://huggingface.co/api/models?author=mudler&limit=300" +) + +// rungRank orders the quality ladder from best to smallest. The HuggingFace API +// returns siblings alphabetically and DiscoverAPEXTiers preserves that order, so +// an unsorted variants list reads I-Balanced, I-Compact, I-Mini, I-Nano, +// I-Quality. Selection ignores authored order, so this is purely so the file a +// human reviews scans in a meaningful sequence. +var rungRank = map[string]int{ + "I-Quality": 0, "I-Balanced": 1, "I-Compact": 2, "I-Mini": 3, "I-Nano": 4, + "Quality": 5, "Balanced": 6, "Compact": 7, "Mini": 8, "Nano": 9, +} + +// baseTags are the tags every generated entry carries. dflash and mtp are never +// among them: RenderChild adds those if and only if the entry configures the +// matching spec_type. +var baseTags = []string{"llm", "gguf", "cpu", "gpu"} + +// childBuild pairs a rendered entry with its position on the quality ladder, so +// the parent's variants list can be sorted without re-parsing entry names. +type childBuild struct { + entry GalleryEntry + rank int +} + +// family is one APEX repo's full generated output. +type family struct { + repo string + repoBase string + stem string + hasMMProj bool + children []childBuild + + // skippedRepos are counterpart candidates HuggingFace would not describe. + // Carried on the family rather than printed and forgotten so the run can + // summarize them next to everything else a reviewer has to eyeball. + skippedRepos []string + census fileCensus + unaccounted int +} + +// fileCensus splits the files discovery emitted nothing for into the ones a +// reviewer must chase and the ones that are deliberately out of scope. +// +// Full-precision sources are the second kind: they are the unquantized weights +// the ladder is derived FROM, not a rung of it. Folding them into the +// unclassified total would leave a permanent benign baseline, and a permanent +// baseline is exactly what hides the one file that ever genuinely matters. +type fileCensus struct { + unclassified int + fullPrecision int +} + +// add accumulates one repo's census into a running total. +func (c *fileCensus) add(o fileCensus) { + c.unclassified += o.unclassified + c.fullPrecision += o.fullPrecision +} + +// sortedChildren returns the family's builds in ladder order, best first. +func (f *family) sortedChildren() []childBuild { + sorted := append([]childBuild{}, f.children...) + sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].rank < sorted[j].rank }) + return sorted +} + +func main() { + verify := flag.String("verify", "", "verify a gallery index and exit") + index := flag.String("index", "gallery/index.yaml", "gallery index to dedup against") + only := flag.String("only", "", "comma-separated repo names to restrict generation to") + out := flag.String("out", "", "write the entries to add to this file") + apply := flag.Bool("apply", false, "append the entries to add to -index") + flag.Parse() + + if *verify != "" { + problems := Verify(*verify) + for _, p := range problems { + fmt.Fprintln(os.Stderr, p) + } + if len(problems) > 0 { + fmt.Fprintf(os.Stderr, "%d problem(s)\n", len(problems)) + os.Exit(1) + } + fmt.Println("index is sound") + return + } + + if err := generate(*index, *only, *out, *apply); err != nil { + fmt.Fprintln(os.Stderr, "error:", err) + os.Exit(1) + } +} + +func generate(indexPath, only, outPath string, apply bool) error { + if outPath == "" && !apply { + return fmt.Errorf("nothing to do: pass -out or -apply") + } + + client := newHTTPClient() + + repos, err := listAPEXRepos(client) + if err != nil { + return err + } + if only != "" { + repos = restrict(repos, only) + } + if len(repos) == 0 { + return fmt.Errorf("no APEX repos selected") + } + fmt.Printf("repos selected: %d\n", len(repos)) + + var families []family + var failed []string + + for _, repo := range repos { + f, err := buildFamily(client, repo) + if err != nil { + // A missing sha256 is fatal for the family rather than skippable: an + // entry without one ships an unverifiable download. Report which repo + // and keep going, so one bad repo does not hide the state of the rest. + fmt.Fprintf(os.Stderr, "FAILED %s: %v\n", repo, err) + failed = append(failed, repo) + continue + } + families = append(families, *f) + } + + existing, err := LoadExisting(indexPath) + if err != nil { + return err + } + ixText, err := LoadIndexText(indexPath) + if err != nil { + return err + } + fmt.Printf("existing index: %d names, %d weight URIs, %d lines\n", + len(existing.ByName), len(existing.ByURI), len(ixText.Lines)) + + // Only the builds go through Merge. A hub is deliberately kept out of it: a + // new hub carries the family's top rung as its own payload, so Merge's URI + // dedup would fold the hub into that rung and the family would lose the very + // entry point this command exists to create. Hub names are checked against + // the index directly, by ResolveHub. + var generated []GalleryEntry + for _, f := range families { + for _, c := range f.children { + generated = append(generated, c.entry) + } + } + + add, reused := Merge(existing, generated) + reportReuse(existing, generated, reused) + + // Variant references are resolved from `reused`, never used to decide what to + // emit: on a within-batch name collision Merge records reused[name] = name + // while the first entry of that name is still in `add`, so treating presence + // in `reused` as "dropped" would silently emit nothing for it. + added := map[string]bool{} + for _, e := range add { + added[e.Name] = true + } + + inserts, newHubs, err := planHubs(families, ixText, reused, added) + if err != nil { + return err + } + reportHubs(ixText, inserts, newHubs) + + skipped, census, fullPrecisionRepos, unaccounted := reportSkipped(families) + + add = append(add, newHubs...) + + fmt.Printf("\nentries generated: %d\nentries to add: %d\nentries reused: %d\nhubs spliced: %d\nhubs created: %d\nrepos skipped: %d\nexcluded (full precision): %d files across %d repos\nunclassified: %d\nunaccounted: %d\n", + len(generated), len(add), len(reused), len(inserts), len(newHubs), len(skipped), + census.fullPrecision, fullPrecisionRepos, census.unclassified, unaccounted) + + lines, err := galleryedit.Apply(ixText.Lines, inserts) + if err != nil { + return err + } + + if err := writeEntries(add, lines, outPath, apply, indexPath); err != nil { + return err + } + + if len(failed) > 0 { + return fmt.Errorf("%d repo(s) failed: %s", len(failed), strings.Join(failed, ", ")) + } + return nil +} + +// resolveVariant maps a generated child name onto whatever entry actually stands +// for it after the merge. `added` is consulted first because a within-batch name +// collision puts a name in BOTH add and reused, and the entry that was emitted +// is the one the parent must reference. +func resolveVariant(name string, reused map[string]string, added map[string]bool) string { + if added[name] { + return name + } + if target, ok := reused[name]; ok { + return target + } + return name +} + +// SpecTypeForRepo reports the speculative decoding mechanism a repo's builds can +// turn on with no extra download. +// +// The *-APEX-MTP-GGUF repos republish the base weights with the model's own MTP +// heads retained, so those builds are only worth their extra size if the heads +// are actually used. Every other APEX repo drops them, and switching MTP on +// there would name a mechanism the weights cannot serve. +// +// The suffix is read off the repo the FILES come from, so nothing downstream has +// to infer a capability from an entry name. +func SpecTypeForRepo(repo string) string { + if strings.HasSuffix(path.Base(repo), "-APEX-MTP-GGUF") { + return "draft-mtp" + } + return "" +} + +// buildFamily discovers everything one APEX repo and its unsloth counterpart +// publish, and renders it. +func buildFamily(client *http.Client, repo string) (*family, error) { + files, err := FetchRepoFiles(client, repo) + if err != nil { + return nil, err + } + if len(files) == 0 { + return nil, fmt.Errorf("no gguf files") + } + + imatrix, plain := DiscoverAPEXTiers(files) + mmproj, hasMMProj := DiscoverMMProj(files) + + census := reportUnclassified(repo, files, imatrix, plain) + + // The imatrix ladder is preferred, but two of the 45 repos publish no + // imatrix tiers at all and must still contribute their plain ladder. + ladder := imatrix + ladderKind := "imatrix" + if len(ladder) == 0 { + ladder = plain + ladderKind = "plain" + } + if len(ladder) == 0 { + return nil, fmt.Errorf("no tiers discovered") + } + + sortTiers(ladder) + + var mm *GGUFFile + if hasMMProj { + mm = &mmproj + } + + repoBase := strings.TrimSuffix(path.Base(repo), "-GGUF") + f := &family{repo: repo, repoBase: repoBase, hasMMProj: hasMMProj, census: census} + + // Only the APEX ladder can carry MTP heads; the unsloth counterpart quantizes + // the plain weights and gets nothing from this. + specType := SpecTypeForRepo(repo) + + for _, t := range ladder { + f.children = append(f.children, childBuild{ + rank: rungRank[t.Label], + entry: RenderChild(ChildInput{ + Name: slug(repoBase) + "-" + slug(t.Label), + Repo: repo, + Template: entryTemplate, + SpecType: specType, + Weights: []GGUFFile{t.File}, + MMProj: mm, + BaseTags: baseTags, + }), + }) + } + + stem := FileStem(ladder[0]) + f.stem = stem + fmt.Printf("%s: %d %s tier(s) [%s], stem %s, mmproj %v\n", + repo, len(ladder), ladderKind, tierLabels(ladder), stem, hasMMProj) + + counterpart, cpFiles, skipped, err := resolveCounterpart(client, repoBase, stem) + f.skippedRepos = skipped + if err != nil { + return nil, err + } + if counterpart != "" { + builds := DiscoverUnslothQuants(cpFiles) + + // Called here rather than inside Verify: a quant dropped at discovery + // leaves no trace at all in the finished gallery file, so the only place + // the shortfall is still visible is the moment of discovery. + unaccounted := UnaccountedQuants(cpFiles, builds) + f.unaccounted = len(unaccounted) + for _, p := range unaccounted { + fmt.Fprintf(os.Stderr, "UNACCOUNTED QUANT %s: %s\n", counterpart, p) + } + + cpMMProj, hasCPMMProj := DiscoverMMProj(cpFiles) + var cpMM *GGUFFile + if hasCPMMProj { + cpMM = &cpMMProj + } + cpBase := strings.TrimSuffix(path.Base(counterpart), "-GGUF") + + for i, b := range builds { + f.children = append(f.children, childBuild{ + rank: 100 + i, + entry: RenderChild(ChildInput{ + Name: slug(cpBase) + "-" + slug(b.Quant), + Repo: counterpart, + Template: entryTemplate, + Weights: b.Files, + MMProj: cpMM, + BaseTags: baseTags, + }), + }) + } + fmt.Printf("%s: counterpart %s, %d quant build(s) %s\n", repo, counterpart, len(builds), quantLabels(builds)) + } else { + fmt.Printf("%s: no unsloth counterpart\n", repo) + } + + return f, nil +} + +// planHubs decides, per family, whether the family's builds are spliced into a +// base model entry the gallery already ships or gathered under a new hub. +// +// Splicing is strongly preferred and is the measured majority-adjacent case. The +// existing entry keeps its description, icon, tags, overrides and files +// untouched; only variant lines are added to it. +func planHubs(families []family, ix *IndexText, reused map[string]string, added map[string]bool) ([]galleryedit.Insert, []GalleryEntry, error) { + // Several APEX repos can resolve to one base model, so both paths accumulate + // by hub name rather than assuming one family per hub. + wantByHub := map[string][]string{} + var spliceOrder []string + + var newHubs []GalleryEntry + hubAt := map[string]int{} + + for i := range families { + f := &families[i] + + hubName, exists := ResolveHub(ix, f.repoBase, f.stem) + want := hubVariants(f, ix, reused, added) + + if exists { + if _, seen := wantByHub[hubName]; !seen { + spliceOrder = append(spliceOrder, hubName) + } + wantByHub[hubName] = append(wantByHub[hubName], want...) + continue + } + + if at, dup := hubAt[hubName]; dup { + for _, v := range filterVariants(hubName, newHubs[at].Variants, want) { + newHubs[at].Variants = append(newHubs[at].Variants, VariantRef{Model: v}) + } + continue + } + + builds := f.sortedChildren() + if len(builds) == 0 { + return nil, nil, fmt.Errorf("%s: no builds to hang a hub on", f.repo) + } + hubAt[hubName] = len(newHubs) + newHubs = append(newHubs, renderHub(hubName, f, builds[0], filterVariants(hubName, nil, want))) + } + + var inserts []galleryedit.Insert + for _, name := range spliceOrder { + e := ix.Find(name) + items := filterVariants(name, e.Variants, wantByHub[name]) + if len(items) == 0 { + continue + } + inserts = append(inserts, galleryedit.Insert{Entry: e.Pos, Variants: items}) + } + return inserts, newHubs, nil +} + +// hubVariants is a family's full build list, in ladder order, named as the hub +// must reference them after the merge. +func hubVariants(f *family, ix *IndexText, reused map[string]string, added map[string]bool) []string { + var out []string + + // A hand-written *-apex entry is an ordinary build of these weights. It is + // never deleted, never renamed and never treated as a hub; it is simply + // referenced like any other rung. + if apex := slug(f.repoBase); ix.Find(apex) != nil { + out = append(out, apex) + } + for _, c := range f.sortedChildren() { + out = append(out, resolveVariant(c.entry.Name, reused, added)) + } + return out +} + +// renderHub builds the hub for a family whose base model the gallery does not +// ship at all. It is named for the BASE model, never for the APEX repo. +// +// It carries one of the discovered builds as its own payload so it is a complete +// installable entry rather than a bare index pointing at other entries. That +// payload is what supplies overrides.backend, which matters beyond installation: +// the verifier can only judge the tagging rule for a backend it can read, so a +// hub carrying feature tags and no backend would escape the check in silence. +// +// The payload's own tags are kept rather than rebuilt from baseTags, so a hub +// whose payload configures a spec_type stays tagged for it and consistent with +// the overrides copied alongside. +func renderHub(name string, f *family, payload childBuild, variants []string) GalleryEntry { + e := payload.entry + e.Name = name + e.Description = fmt.Sprintf( + "%s. Quality ladder and quantization rungs published by %s and its unsloth counterpart; LocalAI picks the build that fits the hardware.", + HubLabel(f.repoBase, f.stem), f.repo) + + e.Tags = append([]string{}, payload.entry.Tags...) + if f.hasMMProj && !hasTag(e.Tags, "vision") { + e.Tags = append(e.Tags, "vision") + } + + e.Variants = nil + for _, v := range variants { + e.Variants = append(e.Variants, VariantRef{Model: v}) + } + return e +} + +func hasTag(tags []string, want string) bool { + for _, t := range tags { + if t == want { + return true + } + } + return false +} + +// resolveCounterpart probes the unsloth candidates in order and returns the +// first that publishes files. +// +// CounterpartCandidates is handed a BARE repo name: its cleaner does not strip +// an owner prefix, so passing "mudler/Foo-APEX-GGUF" would yield "mudler/Foo" +// and compose into the nonsense probe "unsloth/mudler/Foo". +// +// It also returns the candidates HuggingFace refused to describe. Those are +// indistinguishable from absent without credentials, so they are skipped, but +// they are named rather than dropped: one of them could be a real gated repo +// whose quants belong in the gallery. +func resolveCounterpart(client *http.Client, repoBase, stem string) (string, []GGUFFile, []string, error) { + var unavailable []string + for _, cand := range CounterpartCandidates(repoBase, stem) { + repo := unslothOwner + "/" + cand + "-GGUF" + files, unreadable, err := FetchOptionalRepoFiles(client, repo) + if err != nil { + return "", nil, unavailable, fmt.Errorf("probing %s: %w", repo, err) + } + if unreadable { + unavailable = append(unavailable, repo) + continue + } + if len(files) > 0 { + return repo, files, unavailable, nil + } + } + return "", nil, unavailable, nil +} + +// reportUnclassified prints the files discovery turned into nothing. +// +// It is a set difference on COUNTS, not a re-match of filenames: re-matching +// would duplicate the tier regex from discover.go and the two copies would +// drift. The likeliest trigger is a typo or case change from a publishing script +// rather than a genuine sixth tier, and because generation falls back to the +// plain ladder when the imatrix one is empty, a repo whose imatrix files all +// fail to match silently downgrades the whole family instead of erroring. The +// downstream HTTP check cannot catch that: it validates URLs that were emitted, +// and an undiscovered tier emits none. +// It returns the census so the run can total it. +func reportUnclassified(repo string, files []GGUFFile, imatrix, plain []Tier) fileCensus { + mmprojCount, fullPrecision := 0, 0 + for _, f := range files { + // The mmproj test comes first because projectors are themselves often + // published at f16 (mmproj-F16.gguf), and counting such a file in both + // buckets would understate the unclassified remainder. + if strings.HasPrefix(f.Name, "mmproj") { + mmprojCount++ + continue + } + if IsFullPrecision(f.Name) { + fullPrecision++ + } + } + + classified := len(imatrix) + len(plain) + mmprojCount + fullPrecision + if classified >= len(files) { + return fileCensus{fullPrecision: fullPrecision} + } + fmt.Fprintf(os.Stderr, "UNCLASSIFIED %s: %d of %d .gguf files classified, %d unaccounted for\n", + repo, classified, len(files), len(files)-classified) + return fileCensus{unclassified: len(files) - classified, fullPrecision: fullPrecision} +} + +// reportReuse splits Merge's single reused map into the two cases it conflates. +// +// A URI match means the gallery already ships exactly these weights, and +// pointing the parent at the existing entry is correct. A NAME match with a +// different URI means an unrelated entry happens to own the name, and +// referencing it would point the parent at different weights than were +// generated, substituting a build without saying so. Only the first is safe to +// wave through. +func reportReuse(existing *ExistingIndex, generated []GalleryEntry, reused map[string]string) { + byName := map[string]GalleryEntry{} + for _, e := range generated { + if _, seen := byName[e.Name]; !seen { + byName[e.Name] = e + } + } + + var nameCollisions, uriMatches []string + for name, target := range reused { + gen := byName[name] + uri := "" + if len(gen.Files) > 0 { + uri = gen.Files[0].URI + } + + switch { + case hasName(existing, name): + nameCollisions = append(nameCollisions, + fmt.Sprintf(" %s -> gallery entry of the same name (generated uri: %s)", name, orNone(uri))) + case target == name: + nameCollisions = append(nameCollisions, + fmt.Sprintf(" %s -> earlier entry of the same name in this batch (generated uri: %s)", name, orNone(uri))) + default: + uriMatches = append(uriMatches, fmt.Sprintf(" %s -> %s (same weights: %s)", name, target, orNone(uri))) + } + } + sort.Strings(nameCollisions) + sort.Strings(uriMatches) + + fmt.Printf("\nNAME COLLISIONS (%d) - inspect each by hand, the target may hold different weights\n", len(nameCollisions)) + for _, l := range nameCollisions { + fmt.Println(l) + } + fmt.Printf("\nURI MATCHES (%d) - the gallery or this batch already ships these exact weights\n", len(uriMatches)) + for _, l := range uriMatches { + fmt.Println(l) + } +} + +// reportHubs prints exactly what will be written where. The splices are the part +// a human has to read: they modify entries the gallery already ships, so the +// review needs the target, the line, and every added reference spelled out. +func reportHubs(ix *IndexText, inserts []galleryedit.Insert, newHubs []GalleryEntry) { + fmt.Printf("\nHUBS SPLICED (%d) - variants added to the EXISTING base model entry, nothing else touched\n", len(inserts)) + for _, in := range inserts { + e := ix.Find(in.Entry.Name) + fmt.Printf(" %s (line %d, %d variant(s) already declared):\n", in.Entry.Name, in.Entry.StartLine+1, len(e.Variants)) + for _, v := range in.Variants { + fmt.Printf(" + - model: %s\n", galleryedit.QuoteName(v)) + } + } + + fmt.Printf("\nHUBS CREATED (%d) - the gallery ships no base model entry, so one is emitted for it\n", len(newHubs)) + for _, h := range newHubs { + fmt.Printf(" %s:\n", h.Name) + for _, v := range h.Variants { + fmt.Printf(" - model: %s\n", v.Model) + } + } +} + +// reportSkipped names the counterpart repos HuggingFace would not describe, and +// totals the other two silent-shortfall counters alongside them. +// +// A skipped repo is not the same as a clean 404. HuggingFace answers 401 for a +// nonexistent repo to an unauthenticated client, so the overwhelmingly likely +// reading is "there is no such counterpart", which is the normal case for the +// community merges. But a private or gated repo answers 401 too, and that one +// WOULD have quants worth shipping. Printing the list is what keeps that +// possibility auditable instead of silently discarded. +func reportSkipped(families []family) ([]string, fileCensus, int, int) { + var skipped []string + var census fileCensus + fullPrecisionRepos, unaccounted := 0, 0 + for _, f := range families { + skipped = append(skipped, f.skippedRepos...) + census.add(f.census) + if f.census.fullPrecision > 0 { + fullPrecisionRepos++ + } + unaccounted += f.unaccounted + } + sort.Strings(skipped) + + fmt.Printf("\nREPOS SKIPPED AS UNAVAILABLE (%d) - HuggingFace answered 401/403, which is indistinguishable from absent without a token; check none of these is a real gated repo\n", len(skipped)) + for _, r := range skipped { + fmt.Printf(" %s\n", r) + } + return skipped, census, fullPrecisionRepos, unaccounted +} + +func hasName(ix *ExistingIndex, name string) bool { + _, ok := ix.ByName[name] + return ok +} + +func orNone(s string) string { + if s == "" { + return "(no files)" + } + return s +} + +// writeEntries emits the additions. +// +// -apply does two things in one pass: it writes back the spliced lines, which +// differ from the original only by the variant lines galleryedit inserted, and +// then appends the new entries. New entries are APPENDED rather than merged into +// the structure, for the same reason the splice is textual: a YAML round trip +// over 40,000 lines would reflow the whole file into an unreviewable diff. +func writeEntries(add []GalleryEntry, lines []string, outPath string, apply bool, indexPath string) error { + if apply { + if err := os.WriteFile(indexPath, []byte(strings.Join(lines, "\n")), 0o644); err != nil { + return err + } + fmt.Printf("spliced %s\n", indexPath) + } + + if len(add) == 0 { + fmt.Println("nothing to append") + return nil + } + + blob, err := yaml.Marshal(add) + if err != nil { + return err + } + + if outPath != "" { + if err := os.WriteFile(outPath, blob, 0o644); err != nil { + return err + } + fmt.Printf("wrote %d entries to %s\n", len(add), outPath) + } + + if apply { + f, err := os.OpenFile(indexPath, os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + return err + } + defer f.Close() + if _, err := f.Write(blob); err != nil { + return err + } + fmt.Printf("appended %d entries to %s\n", len(add), indexPath) + } + return nil +} + +// listAPEXRepos returns the mudler repos whose name marks them as APEX builds. +func listAPEXRepos(client *http.Client) ([]string, error) { + req, err := http.NewRequest(http.MethodGet, authorListURL, nil) + if err != nil { + return nil, err + } + req.Header.Set("User-Agent", "localai-apexentries/1.0") + + resp, err := client.Do(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, fmt.Errorf("listing models: unexpected status %d", resp.StatusCode) + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + return nil, err + } + + var models []struct { + ID string `json:"id"` + } + if err := json.Unmarshal(body, &models); err != nil { + return nil, fmt.Errorf("decoding model list: %w", err) + } + + var out []string + for _, m := range models { + if strings.Contains(m.ID, "APEX") { + out = append(out, m.ID) + } + } + sort.Strings(out) + return out, nil +} + +func restrict(repos []string, only string) []string { + want := map[string]bool{} + for _, r := range strings.Split(only, ",") { + if r = strings.TrimSpace(r); r != "" { + want[r] = true + } + } + + var out []string + for _, r := range repos { + if want[r] { + out = append(out, r) + delete(want, r) + } + } + // A name in -only that matched nothing is a typo, not an empty result. + for r := range want { + fmt.Fprintf(os.Stderr, "WARNING: -only names %s, which is not an APEX repo of this author\n", r) + } + return out +} + +func sortTiers(tiers []Tier) { + sort.SliceStable(tiers, func(i, j int) bool { return rungRank[tiers[i].Label] < rungRank[tiers[j].Label] }) +} + +func tierLabels(tiers []Tier) string { + var out []string + for _, t := range tiers { + out = append(out, t.Label) + } + return strings.Join(out, ",") +} + +func quantLabels(builds []QuantBuild) string { + var out []string + for _, b := range builds { + l := b.Quant + if b.Sharded { + l += fmt.Sprintf("(%d shards)", len(b.Files)) + } + out = append(out, l) + } + return strings.Join(out, ",") +} + +// slug turns a repo, tier or quant label into a gallery entry name component. +func slug(s string) string { + return strings.ReplaceAll(strings.ToLower(s), "_", "-") +} diff --git a/.github/ci/apexentries/main_test.go b/.github/ci/apexentries/main_test.go new file mode 100644 index 000000000..da4056238 --- /dev/null +++ b/.github/ci/apexentries/main_test.go @@ -0,0 +1,344 @@ +package main + +import ( + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/.github/ci/galleryedit" +) + +func mustIndex(text string) *IndexText { + ix, err := ParseIndexText(text) + ExpectWithOffset(1, err).ToNot(HaveOccurred()) + return ix +} + +// buildOf renders a realistic child so the specs exercise the payload a hub +// actually inherits rather than a bare name. +func buildOf(name, repo, file string, rank int) childBuild { + return childBuild{ + rank: rank, + entry: RenderChild(ChildInput{ + Name: name, + Repo: repo, + Template: entryTemplate, + Weights: []GGUFFile{{Name: file, SHA256: "aa"}}, + BaseTags: baseTags, + }), + } +} + +var _ = Describe("ResolveHub", func() { + It("picks the base model name over the APEX name, even when both are in the gallery", func() { + // The hub is the entry a user searches for. If the *-apex entry were + // chosen the family would be gathered under a name nobody looks up, and + // the base entry would go on advertising only its own build. + ix := mustIndex("- name: qwen3.6-35b-a3b\n url: u\n- name: qwen3.6-35b-a3b-apex\n url: u\n") + + name, exists := ResolveHub(ix, "Qwen3.6-35B-A3B-APEX", "Qwen3.6-35B-A3B-APEX") + + Expect(name).To(Equal("qwen3.6-35b-a3b")) + Expect(exists).To(BeTrue()) + }) + + It("falls back to the stem-derived candidate when the repo-derived one is absent", func() { + // gemma's repo says "-it" and its published files do not, so only one of + // the two candidates can match whatever the base entry was named after. + ix := mustIndex("- name: gemma-4-26b-a4b\n url: u\n") + + name, exists := ResolveHub(ix, "gemma-4-26B-A4B-it-APEX", "gemma-4-26B-A4B-APEX") + + Expect(name).To(Equal("gemma-4-26b-a4b")) + Expect(exists).To(BeTrue()) + }) + + It("reports the base name as absent rather than settling for the APEX entry", func() { + ix := mustIndex("- name: qwen3.5-35b-a3b-apex\n url: u\n") + + name, exists := ResolveHub(ix, "Qwen3.5-35B-A3B-APEX", "Qwen3.5-35B-A3B-APEX") + + Expect(name).To(Equal("qwen3.5-35b-a3b")) + Expect(exists).To(BeFalse()) + }) + + It("strips the MTP and TQ markers as well as APEX", func() { + ix := mustIndex("- name: qwen3.6-35b-a3b\n url: u\n") + + name, exists := ResolveHub(ix, "Qwen3.6-35B-A3B-APEX-MTP", "Qwen3.6-35B-A3B-APEX-MTP") + + Expect(name).To(Equal("qwen3.6-35b-a3b")) + Expect(exists).To(BeTrue()) + }) +}) + +var _ = Describe("planHubs", func() { + noReuse := map[string]string{} + allAdded := func(names ...string) map[string]bool { + out := map[string]bool{} + for _, n := range names { + out[n] = true + } + return out + } + + It("splices into the existing base entry instead of emitting an *-apex parent", func() { + ix := mustIndex("- name: step-3.7-flash\n url: u\n- name: other\n url: u\n") + fams := []family{{ + repo: "mudler/Step-3.7-Flash-APEX-GGUF", + repoBase: "Step-3.7-Flash-APEX", + stem: "Step-3.7-Flash-APEX", + children: []childBuild{buildOf("step-3.7-flash-apex-i-quality", "mudler/Step-3.7-Flash-APEX-GGUF", "a.gguf", 0)}, + }} + + inserts, newHubs, err := planHubs(fams, ix, noReuse, allAdded("step-3.7-flash-apex-i-quality")) + + Expect(err).ToNot(HaveOccurred()) + Expect(newHubs).To(BeEmpty()) + Expect(inserts).To(HaveLen(1)) + Expect(inserts[0].Entry.Name).To(Equal("step-3.7-flash")) + Expect(inserts[0].Variants).To(Equal([]string{"step-3.7-flash-apex-i-quality"})) + }) + + It("merges into an entry that already declares variants, without repeating one", func() { + // The gallery's qwen3.6-35b-a3b already lists its APEX build. Re-adding it + // would put a duplicate key's worth of noise in the diff and a duplicate + // reference in the entry. + ix := mustIndex("- name: qwen3.6-35b-a3b\n variants:\n - model: qwen3.6-35b-a3b-apex\n url: u\n" + + "- name: qwen3.6-35b-a3b-apex\n url: u\n") + fams := []family{{ + repo: "mudler/Qwen3.6-35B-A3B-APEX-GGUF", + repoBase: "Qwen3.6-35B-A3B-APEX", + stem: "Qwen3.6-35B-A3B-APEX", + children: []childBuild{buildOf("qwen3.6-35b-a3b-apex-i-quality", "mudler/Qwen3.6-35B-A3B-APEX-GGUF", "a.gguf", 0)}, + }} + + inserts, newHubs, err := planHubs(fams, ix, noReuse, allAdded("qwen3.6-35b-a3b-apex-i-quality")) + + Expect(err).ToNot(HaveOccurred()) + Expect(newHubs).To(BeEmpty()) + Expect(inserts[0].Variants).To(Equal([]string{"qwen3.6-35b-a3b-apex-i-quality"})) + + out, err := galleryedit.Apply(ix.Lines, inserts) + Expect(err).ToNot(HaveOccurred()) + Expect(strings.Count(strings.Join(out, "\n"), "variants:")).To(Equal(1)) + Expect(out).To(HaveLen(len(ix.Lines) + 1)) + }) + + It("never lets the hub reference itself", func() { + // An unsloth rung whose weights the gallery already ships under the base + // name resolves, through Merge, straight back to the hub. The verifier + // reads a self reference as a variant that declares variants of its own. + ix := mustIndex("- name: step-3.7-flash\n url: u\n") + fams := []family{{ + repo: "mudler/Step-3.7-Flash-APEX-GGUF", + repoBase: "Step-3.7-Flash-APEX", + stem: "Step-3.7-Flash-APEX", + children: []childBuild{buildOf("step-3.7-flash-ud-q4-k-m", "unsloth/Step-3.7-Flash-GGUF", "a.gguf", 100)}, + }} + + inserts, _, err := planHubs(fams, ix, map[string]string{"step-3.7-flash-ud-q4-k-m": "step-3.7-flash"}, map[string]bool{}) + + Expect(err).ToNot(HaveOccurred()) + Expect(inserts).To(BeEmpty()) + }) + + It("emits a hub named for the base model when the gallery has none", func() { + ix := mustIndex("- name: qwen3.5-35b-a3b-apex\n url: u\n") + fams := []family{{ + repo: "mudler/Qwen3.5-35B-A3B-APEX-GGUF", + repoBase: "Qwen3.5-35B-A3B-APEX", + stem: "Qwen3.5-35B-A3B-APEX", + hasMMProj: true, + children: []childBuild{ + buildOf("qwen3.5-35b-a3b-apex-i-quality", "mudler/Qwen3.5-35B-A3B-APEX-GGUF", "a.gguf", 0), + buildOf("qwen3.5-35b-a3b-ud-q6-k", "unsloth/Qwen3.5-35B-A3B-GGUF", "b.gguf", 102), + }, + }} + + inserts, newHubs, err := planHubs(fams, ix, noReuse, + allAdded("qwen3.5-35b-a3b-apex-i-quality", "qwen3.5-35b-a3b-ud-q6-k")) + + Expect(err).ToNot(HaveOccurred()) + Expect(inserts).To(BeEmpty()) + Expect(newHubs).To(HaveLen(1)) + + hub := newHubs[0] + Expect(hub.Name).To(Equal("qwen3.5-35b-a3b")) + Expect(hub.Name).ToNot(HaveSuffix("-apex")) + + // A hand-written *-apex entry is an ordinary build, referenced like any + // other rung and never deleted or renamed. + Expect(hub.Variants).To(Equal([]VariantRef{ + {Model: "qwen3.5-35b-a3b-apex"}, + {Model: "qwen3.5-35b-a3b-apex-i-quality"}, + {Model: "qwen3.5-35b-a3b-ud-q6-k"}, + })) + + // The verifier skips entries with no declared backend, so a hub without + // one would escape the tagging check in silence. + Expect(hub.Overrides).To(HaveKeyWithValue("backend", "llama-cpp")) + Expect(hub.Files).ToNot(BeEmpty()) + Expect(hub.Tags).To(ContainElement("vision")) + }) + + It("gathers two APEX repos that share one base model under a single hub", func() { + ix := mustIndex("- name: unrelated\n url: u\n") + fams := []family{ + { + repo: "mudler/Solo-APEX-GGUF", + repoBase: "Solo-APEX", + stem: "Solo-APEX", + children: []childBuild{buildOf("solo-apex-i-quality", "mudler/Solo-APEX-GGUF", "a.gguf", 0)}, + }, + { + repo: "mudler/Solo-APEX-MTP-GGUF", + repoBase: "Solo-APEX-MTP", + stem: "Solo-APEX-MTP", + children: []childBuild{buildOf("solo-apex-mtp-i-quality", "mudler/Solo-APEX-MTP-GGUF", "b.gguf", 0)}, + }, + } + + _, newHubs, err := planHubs(fams, ix, noReuse, allAdded("solo-apex-i-quality", "solo-apex-mtp-i-quality")) + + Expect(err).ToNot(HaveOccurred()) + Expect(newHubs).To(HaveLen(1)) + Expect(newHubs[0].Name).To(Equal("solo")) + Expect(newHubs[0].Variants).To(Equal([]VariantRef{ + {Model: "solo-apex-i-quality"}, + {Model: "solo-apex-mtp-i-quality"}, + })) + }) +}) + +var _ = Describe("hubVariants", func() { + It("orders builds by quality rung rather than discovery order", func() { + // DiscoverAPEXTiers preserves input order and the HF API returns siblings + // alphabetically, so an unsorted list reads I-Balanced, I-Compact, I-Mini, + // I-Nano, I-Quality. Selection ignores authored order; this is for the + // human reading the file. + f := family{repoBase: "X-APEX", stem: "X-APEX", children: []childBuild{ + {rank: rungRank["I-Nano"], entry: GalleryEntry{Name: "x-i-nano"}}, + {rank: 100, entry: GalleryEntry{Name: "x-ud-q4-k-m"}}, + {rank: rungRank["I-Quality"], entry: GalleryEntry{Name: "x-i-quality"}}, + {rank: rungRank["I-Compact"], entry: GalleryEntry{Name: "x-i-compact"}}, + }} + + got := hubVariants(&f, mustIndex("- name: x\n url: u\n"), map[string]string{}, map[string]bool{}) + + Expect(got).To(Equal([]string{"x-i-quality", "x-i-compact", "x-i-nano", "x-ud-q4-k-m"})) + }) +}) + +var _ = Describe("ParseIndexText", func() { + It("refuses to edit by line number when the two views of the file disagree", func() { + _, err := ParseIndexText("- name: one\n url: u\n-\n") + Expect(err).To(MatchError(ContainSubstring("empty"))) + }) + + It("records the line range of each entry", func() { + ix := mustIndex("- name: first\n url: u\n- name: second\n url: u\n") + + Expect(ix.Find("FIRST").Pos.StartLine).To(Equal(0)) + Expect(ix.Find("first").Pos.EndLine).To(Equal(2)) + Expect(ix.Find("second").Pos.StartLine).To(Equal(2)) + }) +}) + +var _ = Describe("resolveVariant", func() { + It("keeps an entry that was emitted even when it is also in reused", func() { + // A within-batch name collision records reused[name] = name while the + // FIRST entry of that name is still in add. Treating presence in reused as + // "dropped" would emit nothing for it. + added := map[string]bool{"dup": true} + reused := map[string]string{"dup": "dup"} + + Expect(resolveVariant("dup", reused, added)).To(Equal("dup")) + }) + + It("redirects a reused name at the entry that stands in for it", func() { + added := map[string]bool{} + reused := map[string]string{"generated": "already-in-gallery"} + + Expect(resolveVariant("generated", reused, added)).To(Equal("already-in-gallery")) + }) +}) + +var _ = Describe("slug", func() { + It("lowercases and turns quant underscores into hyphens", func() { + Expect(slug("UD-Q4_K_M")).To(Equal("ud-q4-k-m")) + Expect(slug("gemma-4-26B-A4B-it-APEX")).To(Equal("gemma-4-26b-a4b-it-apex")) + Expect(slug("I-Nano")).To(Equal("i-nano")) + }) +}) + +var _ = Describe("sortTiers", func() { + It("puts the imatrix ladder in descending quality order", func() { + tiers := []Tier{ + {Label: "I-Balanced"}, {Label: "I-Compact"}, {Label: "I-Mini"}, + {Label: "I-Nano"}, {Label: "I-Quality"}, + } + sortTiers(tiers) + Expect(tierLabels(tiers)).To(Equal("I-Quality,I-Balanced,I-Compact,I-Mini,I-Nano")) + }) +}) + +var _ = Describe("restrict", func() { + It("keeps only the named repos", func() { + got := restrict([]string{"mudler/A-APEX-GGUF", "mudler/B-APEX-GGUF"}, "mudler/B-APEX-GGUF") + Expect(got).To(Equal([]string{"mudler/B-APEX-GGUF"})) + }) + + It("returns nothing when the filter matches nothing", func() { + Expect(restrict([]string{"mudler/A-APEX-GGUF"}, "mudler/typo")).To(BeEmpty()) + }) +}) + +var _ = Describe("reportUnclassified", func() { + // One real imatrix rung is always present so the specs measure how the + // remaining files are bucketed, not an empty-repo edge case. + tier := Tier{Label: "I-Quality", File: GGUFFile{Name: "Model-APEX-I-Quality.gguf"}} + + censusOf := func(names ...string) fileCensus { + files := []GGUFFile{tier.File} + for _, n := range names { + files = append(files, GGUFFile{Name: n}) + } + return reportUnclassified("mudler/Model-APEX-GGUF", files, []Tier{tier}, nil) + } + + It("counts a flat full-precision source as excluded, not unclassified", func() { + got := censusOf("Carnice-MoE-35B-A3B-F16.gguf") + Expect(got.fullPrecision).To(Equal(1)) + Expect(got.unclassified).To(Equal(0)) + }) + + It("counts every shard of a sharded full-precision source as excluded", func() { + got := censusOf( + "MiniMax-M2.7-APEX-F16-00001-of-00003.gguf", + "MiniMax-M2.7-APEX-F16-00002-of-00003.gguf", + "MiniMax-M2.7-APEX-F16-00003-of-00003.gguf", + ) + Expect(got.fullPrecision).To(Equal(3)) + Expect(got.unclassified).To(Equal(0)) + }) + + It("treats bf16 the same as f16, in either case", func() { + got := censusOf("Model-APEX-BF16.gguf", "Model-APEX-bf16-00001-of-00002.gguf", "Model-APEX-f16.gguf") + Expect(got.fullPrecision).To(Equal(3)) + Expect(got.unclassified).To(Equal(0)) + }) + + It("still reports a genuinely unknown filename as unclassified", func() { + got := censusOf("Model-APEX-Turbo.gguf") + Expect(got.unclassified).To(Equal(1)) + Expect(got.fullPrecision).To(Equal(0)) + }) + + It("separates the two kinds when a repo publishes both", func() { + got := censusOf("Model-APEX-F16.gguf", "Model-APEX-Turbo.gguf") + Expect(got.fullPrecision).To(Equal(1)) + Expect(got.unclassified).To(Equal(1)) + }) +}) diff --git a/.github/ci/apexentries/merge.go b/.github/ci/apexentries/merge.go new file mode 100644 index 000000000..1c8453e65 --- /dev/null +++ b/.github/ci/apexentries/merge.go @@ -0,0 +1,143 @@ +package main + +import ( + "fmt" + "os" + "strings" + + "gopkg.in/yaml.v3" +) + +const ( + hfShorthandPrefix = "huggingface://" + hfResolvePrefix = "https://huggingface.co/" + hfResolveInfix = "/resolve/main/" +) + +// canonicalURI reduces the two interchangeable spellings of a HuggingFace file +// to one key, so a generated resolve/main URI dedups against the shorthand the +// gallery uses for the majority of its entries. +// +// The repo is exactly the first two path segments; everything after is the file +// path, which may itself contain slashes because sharded quants live in a +// subdirectory. Anything that is not recognisably one of the two forms is +// returned unchanged rather than guessed at, so mirrors and other hosts still +// dedup on their literal string. +func canonicalURI(uri string) string { + switch { + case strings.HasPrefix(uri, hfShorthandPrefix): + rest := strings.TrimPrefix(uri, hfShorthandPrefix) + owner, after, ok := strings.Cut(rest, "/") + if !ok { + return uri + } + name, file, ok := strings.Cut(after, "/") + if !ok || owner == "" || name == "" || file == "" { + return uri + } + return hfShorthandPrefix + owner + "/" + name + "/" + file + + case strings.HasPrefix(uri, hfResolvePrefix): + rest := strings.TrimPrefix(uri, hfResolvePrefix) + repo, file, ok := strings.Cut(rest, hfResolveInfix) + if !ok || file == "" { + return uri + } + // A repo is owner/name and nothing more; a longer prefix means this is + // some other huggingface.co URL that must not be rewritten. + owner, name, ok := strings.Cut(repo, "/") + if !ok || owner == "" || name == "" || strings.Contains(name, "/") { + return uri + } + return hfShorthandPrefix + repo + "/" + file + + default: + return uri + } +} + +// ExistingIndex is the lookup built from the current gallery: entry names, and +// which entry claims each weight URI. +type ExistingIndex struct { + ByName map[string]int + ByURI map[string]string +} + +// LoadExisting reads the gallery index for dedup purposes only. It is +// deliberately not used to rewrite the file: the index is 40,000 lines, and a +// YAML round trip would reflow the whole thing into an unreviewable diff. +func LoadExisting(path string) (*ExistingIndex, error) { + raw, err := os.ReadFile(path) + if err != nil { + return nil, err + } + + var entries []struct { + Name string `yaml:"name"` + Files []struct { + URI string `yaml:"uri"` + } `yaml:"files"` + } + if err := yaml.Unmarshal(raw, &entries); err != nil { + return nil, fmt.Errorf("parsing %s: %w", path, err) + } + + ix := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}} + for i, e := range entries { + ix.ByName[e.Name] = i + for _, f := range e.Files { + if f.URI != "" { + ix.ByURI[canonicalURI(f.URI)] = e.Name + } + } + } + return ix, nil +} + +// Merge splits generated entries into those to add and those already covered. +// reused maps a generated name to the existing entry that stands in for it, so +// a parent can reference what is already there instead of duplicating weights. +// Several APEX repos share one base model, so the same counterpart rungs are +// generated more than once in a batch. The batch has to dedup against itself as +// well as against the gallery, tracked locally because the caller may reuse the +// ExistingIndex it passed in. +func Merge(existing *ExistingIndex, generated []GalleryEntry) (add []GalleryEntry, reused map[string]string) { + reused = map[string]string{} + batchNames := map[string]string{} + batchURIs := map[string]string{} + + // Canonicalized into a local copy rather than in place: an ExistingIndex may + // be hand-built or reused by the caller, so Merge must not rewrite it. + existingURIs := make(map[string]string, len(existing.ByURI)) + for uri, owner := range existing.ByURI { + existingURIs[canonicalURI(uri)] = owner + } + + for _, e := range generated { + // Name is checked before URI: a name collision must block the add + // whatever the weights say, since duplicate names corrupt the index. + if _, clash := existing.ByName[e.Name]; clash { + reused[e.Name] = e.Name + continue + } + if claimant, clash := batchNames[e.Name]; clash { + reused[e.Name] = claimant + continue + } + if len(e.Files) > 0 { + uri := canonicalURI(e.Files[0].URI) + if owner, ok := existingURIs[uri]; ok { + reused[e.Name] = owner + continue + } + if claimant, ok := batchURIs[uri]; ok { + reused[e.Name] = claimant + continue + } + batchURIs[uri] = e.Name + } + batchNames[e.Name] = e.Name + add = append(add, e) + } + return add, reused +} diff --git a/.github/ci/apexentries/merge_test.go b/.github/ci/apexentries/merge_test.go new file mode 100644 index 000000000..2c0a7d3f6 --- /dev/null +++ b/.github/ci/apexentries/merge_test.go @@ -0,0 +1,183 @@ +package main + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("Merge", func() { + It("drops a generated entry whose weight URI already exists and reports the existing name", func() { + existing := &ExistingIndex{ + ByName: map[string]int{"qwen3.6-35b-a3b-apex": 0}, + ByURI: map[string]string{ + "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Quality.gguf": "qwen3.6-35b-a3b-apex", + }, + } + gen := []GalleryEntry{{ + Name: "x-apex-i-quality", + Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Quality.gguf"}}, + }} + + add, reused := Merge(existing, gen) + + Expect(add).To(BeEmpty()) + Expect(reused).To(HaveKeyWithValue("x-apex-i-quality", "qwen3.6-35b-a3b-apex")) + }) + + It("keeps a generated entry whose weights are new", func() { + existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}} + gen := []GalleryEntry{{ + Name: "x-apex-i-mini", + Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/X-APEX-I-Mini.gguf"}}, + }} + + add, reused := Merge(existing, gen) + + Expect(add).To(HaveLen(1)) + Expect(reused).To(BeEmpty()) + }) + + It("refuses to add an entry whose name collides with an existing one", func() { + existing := &ExistingIndex{ + ByName: map[string]int{"x-apex-i-mini": 0}, + ByURI: map[string]string{}, + } + gen := []GalleryEntry{{ + Name: "x-apex-i-mini", + Files: []EntryFile{{URI: "https://huggingface.co/mudler/X-APEX-GGUF/resolve/main/other.gguf"}}, + }} + + add, reused := Merge(existing, gen) + + Expect(add).To(BeEmpty()) + Expect(reused).To(HaveKeyWithValue("x-apex-i-mini", "x-apex-i-mini")) + }) + + // The gallery records most of its URIs in huggingface:// shorthand while + // render.go only ever emits the resolve/main form, so without + // canonicalization the majority of the file is invisible to the dedup. + It("matches a generated https URI against the shorthand form recorded in the gallery", func() { + existing := &ExistingIndex{ + ByName: map[string]int{"foo-gguf-q8-0": 0}, + ByURI: map[string]string{ + "huggingface://unsloth/Foo-GGUF/Foo-Q8_0.gguf": "foo-gguf-q8-0", + }, + } + gen := []GalleryEntry{{ + Name: "foo-apex-q8-0", + Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Foo-GGUF/resolve/main/Foo-Q8_0.gguf"}}, + }} + + add, reused := Merge(existing, gen) + + Expect(add).To(BeEmpty()) + Expect(reused).To(HaveKeyWithValue("foo-apex-q8-0", "foo-gguf-q8-0")) + }) + + It("matches a generated shorthand URI against the https form recorded in the gallery", func() { + existing := &ExistingIndex{ + ByName: map[string]int{"foo-gguf-q8-0": 0}, + ByURI: map[string]string{ + "https://huggingface.co/unsloth/Foo-GGUF/resolve/main/Foo-Q8_0.gguf": "foo-gguf-q8-0", + }, + } + gen := []GalleryEntry{{ + Name: "foo-apex-q8-0", + Files: []EntryFile{{URI: "huggingface://unsloth/Foo-GGUF/Foo-Q8_0.gguf"}}, + }} + + add, reused := Merge(existing, gen) + + Expect(add).To(BeEmpty()) + Expect(reused).To(HaveKeyWithValue("foo-apex-q8-0", "foo-gguf-q8-0")) + }) + + // Sharded quants live under a subdirectory, so the file path carries slashes + // of its own and only the first two segments are the repo. + It("matches across both forms when the file path has a subdirectory", func() { + existing := &ExistingIndex{ + ByName: map[string]int{"model-ud-q4-k-m": 0}, + ByURI: map[string]string{ + "huggingface://unsloth/Model-GGUF/UD-Q4_K_M/Model-UD-Q4_K_M-00001-of-00002.gguf": "model-ud-q4-k-m", + }, + } + gen := []GalleryEntry{{ + Name: "model-apex-ud-q4-k-m", + Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Model-GGUF/resolve/main/UD-Q4_K_M/Model-UD-Q4_K_M-00001-of-00002.gguf"}}, + }} + + add, reused := Merge(existing, gen) + + Expect(add).To(BeEmpty()) + Expect(reused).To(HaveKeyWithValue("model-apex-ud-q4-k-m", "model-ud-q4-k-m")) + }) + + // Several APEX repos share one base model, so the same unsloth rungs are + // generated more than once in a single batch. + It("adds only the first of two generated entries sharing a name", func() { + existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}} + gen := []GalleryEntry{ + { + Name: "shared-rung-q8-0", + Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf"}}, + }, + { + Name: "shared-rung-q8-0", + Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Other-GGUF/resolve/main/Other-Q8_0.gguf"}}, + }, + } + + add, reused := Merge(existing, gen) + + Expect(add).To(HaveLen(1)) + Expect(add[0].Files[0].URI).To(Equal("https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf")) + Expect(reused).To(HaveKeyWithValue("shared-rung-q8-0", "shared-rung-q8-0")) + }) + + It("adds only the first of two generated entries sharing a primary URI", func() { + existing := &ExistingIndex{ByName: map[string]int{}, ByURI: map[string]string{}} + gen := []GalleryEntry{ + { + Name: "shared-rung-from-apex", + Files: []EntryFile{{URI: "https://huggingface.co/unsloth/Shared-GGUF/resolve/main/Shared-Q8_0.gguf"}}, + }, + { + Name: "shared-rung-from-apex-mtp", + Files: []EntryFile{{URI: "huggingface://unsloth/Shared-GGUF/Shared-Q8_0.gguf"}}, + }, + } + + add, reused := Merge(existing, gen) + + Expect(add).To(HaveLen(1)) + Expect(add[0].Name).To(Equal("shared-rung-from-apex")) + Expect(reused).To(HaveKeyWithValue("shared-rung-from-apex-mtp", "shared-rung-from-apex")) + }) + + // Anything that is not a HuggingFace URI must survive untouched, so an + // unrecognised scheme still dedups against the very same string. + It("leaves a URI in neither recognised form alone and still dedups it exactly", func() { + existing := &ExistingIndex{ + ByName: map[string]int{"mirrored-model": 0}, + ByURI: map[string]string{ + "https://mirror.example.com/weights/Model-Q8_0.gguf": "mirrored-model", + }, + } + gen := []GalleryEntry{ + { + Name: "mirrored-apex", + Files: []EntryFile{{URI: "https://mirror.example.com/weights/Model-Q8_0.gguf"}}, + }, + { + Name: "elsewhere-apex", + Files: []EntryFile{{URI: "https://mirror.example.com/weights/Other-Q8_0.gguf"}}, + }, + } + + add, reused := Merge(existing, gen) + + Expect(add).To(HaveLen(1)) + Expect(add[0].Name).To(Equal("elsewhere-apex")) + Expect(reused).To(HaveKeyWithValue("mirrored-apex", "mirrored-model")) + }) +}) diff --git a/.github/ci/apexentries/render.go b/.github/ci/apexentries/render.go new file mode 100644 index 000000000..7659c8c67 --- /dev/null +++ b/.github/ci/apexentries/render.go @@ -0,0 +1,175 @@ +package main + +import ( + "fmt" + "path" + "strings" +) + +// EntryFile is one downloadable file of a gallery entry. +type EntryFile struct { + Filename string `yaml:"filename"` + SHA256 string `yaml:"sha256"` + URI string `yaml:"uri"` +} + +// GalleryEntry is the subset of a gallery entry this generator writes. +// +// Named GalleryEntry rather than Entry because the test files dot-import +// Ginkgo, whose table DSL exports an Entry that a package-level Entry would +// collide with. The yaml tags are what the gallery index sees, so the Go +// identifier is free to differ. +type GalleryEntry struct { + Name string `yaml:"name"` + URL string `yaml:"url"` + Description string `yaml:"description,omitempty"` + Tags []string `yaml:"tags,omitempty"` + Overrides map[string]any `yaml:"overrides,omitempty"` + Files []EntryFile `yaml:"files,omitempty"` + Variants []VariantRef `yaml:"variants,omitempty"` +} + +// VariantRef mirrors the gallery's variant reference: a name and nothing else. +type VariantRef struct { + Model string `yaml:"model"` +} + +// ChildInput is everything needed to render one non-parent entry. +type ChildInput struct { + Name string + Repo string + // DraftRepo is the repo publishing the drafter, when it is not the repo + // publishing the weights. Speculative pairings routinely cross repos, so + // the drafter cannot be assumed to sit next to the weights. Empty means + // same-repo, which is how the *-APEX-MTP-GGUF repos ship. + DraftRepo string + Template string + Weights []GGUFFile + MMProj *GGUFFile + SpecType string + DraftFile *GGUFFile + BaseTags []string +} + +// specTuning is the acceptance-window tuning each spec type ships with, copied +// from the hand-written entries that already run these two mechanisms rather +// than invented here. The two differ because the drafters differ: self-drafted +// MTP heads produce a short, high-confidence proposal (15+ hand-written entries +// use 6 with a 0.75 floor), while a separate DFlash drafter is cheap enough to +// run far ahead unconditionally (the five hand-written dflash entries use 15 and +// set no floor). +var specTuning = map[string][]string{ + "draft-mtp": {"spec_n_max:6", "spec_p_min:0.75"}, + "draft-dflash": {"spec_n_max:15"}, +} + +func hfURI(repo, file string) string { + return fmt.Sprintf("https://huggingface.co/%s/resolve/main/%s", repo, file) +} + +// localPath is where a downloaded file lands. +// +// The hand-written entries namespace by the repo's BARE name +// (llama-cpp/models//), which is not unique. LiquidAI/LFM2.5-8B-A1B-GGUF +// and unsloth/LFM2.5-8B-A1B-GGUF share a basename, so both claim +// llama-cpp/models/LFM2.5-8B-A1B-GGUF/, and installing the second after the first +// either overwrites weights whose recorded sha256 belongs to the other file or is +// skipped as already present. Two owners publishing the same model name is the +// normal case for quantizers, not an edge case, so the owner has to be in the path. +// +// The owner becomes its own path segment rather than being folded into the +// directory name: owner/repo is unique on HuggingFace and "/" cannot occur inside +// either half, so this is the only form that is collision-proof by construction. +// It still reads as the hand-written convention with the owner restored, and the +// extra depth is already present in the index for sharded builds. +func localPath(kind, repo, file string) string { + // path.Dir yields "." for a repo named without an owner, which path.Join + // drops, so such a caller keeps the historical two-segment layout. + return path.Join("llama-cpp", kind, path.Dir(repo), path.Base(repo), file) +} + +// RenderChild builds one child entry. +// +// The dflash/mtp tag is added if and only if this entry sets a spec_type, +// because variant ranking reads tags and nothing else, and a tag that does not +// match what the entry configures either promotes a build that is no faster or +// hides one that is. +func RenderChild(in ChildInput) GalleryEntry { + e := GalleryEntry{ + Name: in.Name, + URL: fmt.Sprintf("github:mudler/LocalAI/gallery/%s@master", in.Template), + Tags: append([]string{}, in.BaseTags...), + Overrides: map[string]any{}, + } + + // gallery/virtual.yaml carries no backend, so nothing else would name an + // engine for these entries. Matching the hand-written entries on + // known_usecases too: LocalAI would fall back to the backend defaults, but + // generated entries should not read differently from their neighbours. + e.Overrides["backend"] = "llama-cpp" + e.Overrides["known_usecases"] = []string{"chat"} + + options := []string{"use_jinja:true"} + + for _, w := range in.Weights { + e.Files = append(e.Files, EntryFile{ + Filename: localPath("models", in.Repo, w.Name), + SHA256: w.SHA256, + URI: hfURI(in.Repo, w.Name), + }) + } + e.Overrides["parameters"] = map[string]any{ + "model": localPath("models", in.Repo, in.Weights[0].Name), + } + + if in.MMProj != nil { + // An explicit known_usecases SUPPRESSES the backend-default fallback in + // core/gallery/models_types.go, so a multimodal entry left at chat-only + // never matches FilterGalleryModelsByUsecase(FLAG_VISION) or + // FilterGalleryModelsByMultimodal and vanishes from the UI's vision and + // multimodal filters. 19 of the 45 APEX repos ship an mmproj. + e.Overrides["known_usecases"] = []string{"chat", "vision"} + e.Overrides["mmproj"] = localPath("mmproj", in.Repo, in.MMProj.Name) + e.Files = append(e.Files, EntryFile{ + Filename: localPath("mmproj", in.Repo, in.MMProj.Name), + SHA256: in.MMProj.SHA256, + URI: hfURI(in.Repo, in.MMProj.Name), + }) + } + + // A spec type is configured independently of a drafter FILE. Weights that + // carry their own MTP heads need no second download, and requiring one left + // the *-APEX-MTP-GGUF builds shipping the larger heads-bearing weights with + // the heads switched off: a strictly bigger download at the same speed, + // ranked identically to the plain rung at the same tier. + if in.SpecType != "" { + options = append(options, "spec_type:"+in.SpecType) + options = append(options, specTuning[in.SpecType]...) + // The tag is derived from the spec type this entry sets and from nothing + // else. Variant ranking reads tags only, so a tag taken from a repo or + // entry NAME would promote a build that is no faster whenever the name + // and the configuration disagree. + e.Tags = append(e.Tags, strings.TrimPrefix(in.SpecType, "draft-")) + } + + if in.SpecType != "" && in.DraftFile != nil { + // Fall back to the weights repo so pairings that publish the drafter + // alongside the weights keep working without restating the repo. + draftRepo := in.DraftRepo + if draftRepo == "" { + draftRepo = in.Repo + } + draftPath := localPath("models", draftRepo, in.DraftFile.Name) + + e.Overrides["draft_model"] = draftPath + e.Overrides["flash_attention"] = "on" + e.Files = append(e.Files, EntryFile{ + Filename: draftPath, + SHA256: in.DraftFile.SHA256, + URI: hfURI(draftRepo, in.DraftFile.Name), + }) + } + + e.Overrides["options"] = options + return e +} diff --git a/.github/ci/apexentries/render_test.go b/.github/ci/apexentries/render_test.go new file mode 100644 index 000000000..7a5433ee3 --- /dev/null +++ b/.github/ci/apexentries/render_test.go @@ -0,0 +1,249 @@ +package main + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("RenderChild", func() { + It("tags an entry that configures draft-dflash", func() { + e := RenderChild(ChildInput{ + Name: "qwen3.5-9b-dflash", + Repo: "mudler/Example-APEX-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}}, + SpecType: "draft-dflash", + DraftFile: &GGUFFile{Name: "Example-DFlash.Q8_0.gguf", SHA256: "b"}, + BaseTags: []string{"llm", "gguf"}, + }) + + Expect(e.Tags).To(ContainElement("dflash")) + Expect(e.Tags).ToNot(ContainElement("mtp")) + Expect(e.Overrides["options"]).To(ContainElement("spec_type:draft-dflash")) + Expect(e.Overrides["draft_model"]).ToNot(BeNil()) + }) + + It("does not tag an MTP-named repo that configures no speculation", func() { + // mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF ships MTP-bearing weights. Weights + // that carry the heads are not an entry that enables them, and tagging it + // would win the feature axis without being any faster. + e := RenderChild(ChildInput{ + Name: "qwen3.6-35b-a3b-apex-mtp-i-quality", + Repo: "mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{{Name: "Qwen3.6-35B-A3B-APEX-MTP-I-Quality.gguf", SHA256: "a"}}, + BaseTags: []string{"llm", "gguf"}, + }) + + Expect(e.Tags).ToNot(ContainElement("mtp")) + Expect(e.Tags).ToNot(ContainElement("dflash")) + Expect(e.Overrides).ToNot(HaveKey("draft_model")) + }) + + It("lists every shard of a sharded build and points the model at the first", func() { + e := RenderChild(ChildInput{ + Name: "step-3.7-flash-ud-q4-k-m", + Repo: "unsloth/Step-3.7-Flash-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{ + {Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf", SHA256: "a"}, + {Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00002-of-00002.gguf", SHA256: "b"}, + }, + BaseTags: []string{"llm", "gguf"}, + }) + + Expect(e.Files).To(HaveLen(2)) + params, ok := e.Overrides["parameters"].(map[string]any) + Expect(ok).To(BeTrue()) + Expect(params["model"]).To(HaveSuffix("00001-of-00002.gguf")) + Expect(e.Files[0].URI).To(Equal( + "https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf")) + }) + + It("wires mmproj when the repo publishes one", func() { + e := RenderChild(ChildInput{ + Name: "example-i-mini", + Repo: "mudler/Example-APEX-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{{Name: "Example-APEX-I-Mini.gguf", SHA256: "a"}}, + MMProj: &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"}, + BaseTags: []string{"llm", "gguf"}, + }) + + Expect(e.Overrides["mmproj"]).ToNot(BeNil()) + Expect(e.Files).To(HaveLen(2)) + }) + + It("names the engine and the usecases the hand-written entries name", func() { + // gallery/virtual.yaml supplies no backend, so an entry that omits one + // names no engine at all and cannot load. + e := RenderChild(ChildInput{ + Name: "example-i-mini", + Repo: "mudler/Example-APEX-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{{Name: "Example-APEX-I-Mini.gguf", SHA256: "a"}}, + BaseTags: []string{"llm", "gguf"}, + }) + + Expect(e.Overrides["backend"]).To(Equal("llama-cpp")) + Expect(e.Overrides["known_usecases"]).To(ContainElement("chat")) + }) + + It("draws the drafter from DraftRepo when the pairing spans two repos", func() { + // unsloth/Qwen3-4B-GGUF pairs with a drafter published separately by + // AtomicChat, so a drafter URI built from the weights repo 404s. + e := RenderChild(ChildInput{ + Name: "qwen3-4b-dflash", + Repo: "unsloth/Qwen3-4B-GGUF", + DraftRepo: "AtomicChat/Qwen3-4B-DFlash-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{{Name: "Qwen3-4B-Q4_K_M.gguf", SHA256: "a"}}, + SpecType: "draft-dflash", + DraftFile: &GGUFFile{Name: "Qwen3-4B-DFlash.Q8_0.gguf", SHA256: "b"}, + BaseTags: []string{"llm", "gguf"}, + }) + + Expect(e.Files[0].URI).To(Equal( + "https://huggingface.co/unsloth/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q4_K_M.gguf")) + Expect(e.Files[1].URI).To(Equal( + "https://huggingface.co/AtomicChat/Qwen3-4B-DFlash-GGUF/resolve/main/Qwen3-4B-DFlash.Q8_0.gguf")) + Expect(e.Files[1].Filename).To(Equal( + "llama-cpp/models/AtomicChat/Qwen3-4B-DFlash-GGUF/Qwen3-4B-DFlash.Q8_0.gguf")) + Expect(e.Overrides["draft_model"]).To(Equal( + "llama-cpp/models/AtomicChat/Qwen3-4B-DFlash-GGUF/Qwen3-4B-DFlash.Q8_0.gguf")) + }) + + It("falls back to the weights repo for the drafter when DraftRepo is empty", func() { + // The *-APEX-MTP-GGUF repos ship the drafter alongside the weights. + e := RenderChild(ChildInput{ + Name: "example-apex-dflash", + Repo: "mudler/Example-APEX-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}}, + SpecType: "draft-dflash", + DraftFile: &GGUFFile{Name: "Example-DFlash.Q8_0.gguf", SHA256: "b"}, + BaseTags: []string{"llm", "gguf"}, + }) + + Expect(e.Files[1].URI).To(Equal( + "https://huggingface.co/mudler/Example-APEX-GGUF/resolve/main/Example-DFlash.Q8_0.gguf")) + Expect(e.Files[1].Filename).To(Equal( + "llama-cpp/models/mudler/Example-APEX-GGUF/Example-DFlash.Q8_0.gguf")) + }) +}) + +var _ = Describe("RenderChild known_usecases", func() { + It("declares vision alongside chat when the entry carries an mmproj", func() { + // An explicit known_usecases suppresses the backend-default fallback, so a + // chat-only multimodal entry disappears from the UI's vision filter. + e := RenderChild(ChildInput{ + Name: "example-i-quality", + Repo: "mudler/Example-APEX-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}}, + MMProj: &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"}, + BaseTags: []string{"llm", "gguf"}, + }) + + Expect(e.Overrides["known_usecases"]).To(ConsistOf("chat", "vision")) + }) + + It("leaves a text-only entry at chat", func() { + e := RenderChild(ChildInput{ + Name: "example-i-quality", + Repo: "mudler/Example-APEX-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}}, + BaseTags: []string{"llm", "gguf"}, + }) + + Expect(e.Overrides["known_usecases"]).To(ConsistOf("chat")) + }) +}) + +var _ = Describe("localPath", func() { + It("keeps two repos with the same basename but different owners apart", func() { + // LiquidAI and unsloth both publish LFM2.5-8B-A1B-GGUF. A path built from + // the bare repo name gives both the same local file, so installing the + // second overwrites or skips the first and one of them then serves bytes + // that do not match its recorded sha256. + liquid := RenderChild(ChildInput{ + Name: "lfm2.5-8b-a1b-i-quality", + Repo: "LiquidAI/LFM2.5-8B-A1B-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{{Name: "LFM2.5-8B-A1B-Q8_0.gguf", SHA256: "33ab3b8c"}}, + BaseTags: []string{"llm", "gguf"}, + }) + unsloth := RenderChild(ChildInput{ + Name: "lfm2.5-8b-a1b-q8-0", + Repo: "unsloth/LFM2.5-8B-A1B-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{{Name: "LFM2.5-8B-A1B-Q8_0.gguf", SHA256: "ec11666b"}}, + BaseTags: []string{"llm", "gguf"}, + }) + + Expect(liquid.Files[0].Filename).ToNot(Equal(unsloth.Files[0].Filename)) + Expect(unsloth.Files[0].Filename).To(Equal( + "llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf")) + }) + + It("namespaces the mmproj by owner too", func() { + e := RenderChild(ChildInput{ + Name: "example-i-quality", + Repo: "mudler/Example-APEX-GGUF", + Template: "virtual.yaml", + Weights: []GGUFFile{{Name: "Example-APEX-I-Quality.gguf", SHA256: "a"}}, + MMProj: &GGUFFile{Name: "mmproj-F16.gguf", SHA256: "c"}, + BaseTags: []string{"llm", "gguf"}, + }) + + Expect(e.Overrides["mmproj"]).To(Equal( + "llama-cpp/mmproj/mudler/Example-APEX-GGUF/mmproj-F16.gguf")) + }) +}) + +var _ = Describe("MTP builds", func() { + renderTier := func(repo string) GalleryEntry { + return RenderChild(ChildInput{ + Name: "example-i-quality", + Repo: repo, + Template: "virtual.yaml", + SpecType: SpecTypeForRepo(repo), + Weights: []GGUFFile{{Name: "Example-I-Quality.gguf", SHA256: "a"}}, + BaseTags: []string{"llm", "gguf"}, + }) + } + + It("turns MTP on for a build off an APEX-MTP repo", func() { + // These weights retain the model's own MTP heads, so shipping them with + // speculation off is a strictly larger download at the same speed, + // ranked identically to the plain rung at the same tier. + e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF") + + Expect(e.Overrides["options"]).To(ContainElements( + "spec_type:draft-mtp", "spec_n_max:6", "spec_p_min:0.75")) + Expect(e.Tags).To(ContainElement("mtp")) + }) + + It("needs no drafter file, because the heads travel with the weights", func() { + e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF") + + Expect(e.Overrides).ToNot(HaveKey("draft_model")) + Expect(e.Files).To(HaveLen(1)) + }) + + It("leaves a build off a plain APEX repo alone", func() { + e := renderTier("mudler/Qwen3.6-35B-A3B-APEX-GGUF") + + Expect(e.Tags).ToNot(ContainElement("mtp")) + Expect(e.Overrides["options"]).To(ConsistOf("use_jinja:true")) + }) + + It("leaves an unsloth counterpart rung alone", func() { + // The counterpart quantizes the plain weights; nothing there carries heads. + e := renderTier("unsloth/Qwen3.6-35B-A3B-GGUF") + + Expect(e.Tags).ToNot(ContainElement("mtp")) + Expect(e.Overrides["options"]).To(ConsistOf("use_jinja:true")) + }) +}) diff --git a/.github/ci/apexentries/unsloth.go b/.github/ci/apexentries/unsloth.go new file mode 100644 index 000000000..36530e6ff --- /dev/null +++ b/.github/ci/apexentries/unsloth.go @@ -0,0 +1,71 @@ +package main + +import ( + "regexp" + "sort" + "strings" +) + +// WantedQuants is the fixed unsloth subset this generator emits. It is a +// deliberate subset: unsloth publishes north of 20 quants per repo, and the +// selector needs useful fitness points rather than every rung. +var WantedQuants = []string{"UD-Q4_K_M", "UD-Q5_K_M", "UD-Q6_K", "Q8_0"} + +var shardRE = regexp.MustCompile(`-(\d{5})-of-(\d{5})\.gguf$`) + +// QuantBuild is one unsloth quantization, which may be a single file or an +// ordered set of shards. +type QuantBuild struct { + Quant string + Files []GGUFFile + Sharded bool +} + +// CounterpartCandidates returns the unsloth repo base names worth probing, most +// likely first. Both derivations are needed: the repo name finds +// unsloth/gemma-4-26B-A4B-it-GGUF, while the file stem is what matches for +// repos whose stem is the canonical model name. +func CounterpartCandidates(repoName, fileStem string) []string { + clean := func(s string) string { + s = strings.TrimSuffix(s, "-GGUF") + s = regexp.MustCompile(`-(MTP|TQ)$`).ReplaceAllString(s, "") + s = strings.TrimSuffix(s, "-APEX") + return regexp.MustCompile(`-(MTP|TQ)$`).ReplaceAllString(s, "") + } + + out := []string{clean(repoName)} + if stem := clean(fileStem); stem != out[0] { + out = append(out, stem) + } + return out +} + +// DiscoverUnslothQuants returns the wanted quants a repo publishes, handling +// both the flat single-file layout and the sharded layout where a quant lives +// in its own subdirectory. +func DiscoverUnslothQuants(files []GGUFFile) []QuantBuild { + var out []QuantBuild + + for _, q := range WantedQuants { + var flat []GGUFFile + var shards []GGUFFile + + for _, f := range files { + switch { + case !strings.Contains(f.Name, "/") && strings.HasSuffix(f.Name, "-"+q+".gguf"): + flat = append(flat, f) + case strings.HasPrefix(f.Name, q+"/") && shardRE.MatchString(f.Name): + shards = append(shards, f) + } + } + + switch { + case len(flat) > 0: + out = append(out, QuantBuild{Quant: q, Files: flat}) + case len(shards) > 0: + sort.Slice(shards, func(i, j int) bool { return shards[i].Name < shards[j].Name }) + out = append(out, QuantBuild{Quant: q, Files: shards, Sharded: true}) + } + } + return out +} diff --git a/.github/ci/apexentries/unsloth_test.go b/.github/ci/apexentries/unsloth_test.go new file mode 100644 index 000000000..f7c73c4fc --- /dev/null +++ b/.github/ci/apexentries/unsloth_test.go @@ -0,0 +1,75 @@ +package main + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("CounterpartCandidates", func() { + It("offers both the repo-derived and stem-derived names", func() { + // mudler/gemma-4-26B-A4B-it-APEX-GGUF ships gemma-4-26B-A4B-APEX-*.gguf, + // and only the repo-derived name finds unsloth/gemma-4-26B-A4B-it-GGUF. + got := CounterpartCandidates("gemma-4-26B-A4B-it-APEX-GGUF", "gemma-4-26B-A4B-APEX") + + Expect(got).To(Equal([]string{"gemma-4-26B-A4B-it", "gemma-4-26B-A4B"})) + }) + + It("strips the MTP marker", func() { + got := CounterpartCandidates("Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF", "Qwopus3.6-35B-A3B-v1-APEX-MTP") + + Expect(got[0]).To(Equal("Qwopus3.6-35B-A3B-v1")) + }) + + It("strips the TQ marker", func() { + // This is the branch that folds mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF into + // the qwen3.5-35b-a3b hub. Without it the probe is + // unsloth/Qwen3.5-35B-A3B-TQ-GGUF, which does not exist, so the family + // silently loses every unsloth rung. + got := CounterpartCandidates("Qwen3.5-35B-A3B-APEX-TQ-GGUF", "Qwen3.5-35B-A3B-APEX-TQ") + + Expect(got).To(Equal([]string{"Qwen3.5-35B-A3B"})) + }) + + It("does not repeat a candidate when both derivations agree", func() { + got := CounterpartCandidates("Qwen3.6-35B-A3B-APEX-GGUF", "Qwen3.6-35B-A3B-APEX") + + Expect(got).To(Equal([]string{"Qwen3.6-35B-A3B"})) + }) +}) + +var _ = Describe("DiscoverUnslothQuants", func() { + It("finds flat single-file quants", func() { + files := []GGUFFile{ + {Name: "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", SHA256: "a"}, + {Name: "Qwen3.6-35B-A3B-UD-IQ1_M.gguf", SHA256: "b"}, + } + + got := DiscoverUnslothQuants(files) + + Expect(got).To(HaveLen(1)) + Expect(got[0].Quant).To(Equal("UD-Q4_K_M")) + Expect(got[0].Sharded).To(BeFalse()) + Expect(got[0].Files).To(HaveLen(1)) + }) + + It("collects a sharded quant from its subdirectory in shard order", func() { + files := []GGUFFile{ + {Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00002-of-00002.gguf", SHA256: "b"}, + {Name: "UD-Q4_K_M/Step-3.7-Flash-UD-Q4_K_M-00001-of-00002.gguf", SHA256: "a"}, + } + + got := DiscoverUnslothQuants(files) + + Expect(got).To(HaveLen(1)) + Expect(got[0].Quant).To(Equal("UD-Q4_K_M")) + Expect(got[0].Sharded).To(BeTrue()) + Expect(got[0].Files).To(HaveLen(2)) + Expect(got[0].Files[0].Name).To(HaveSuffix("00001-of-00002.gguf")) + }) + + It("ignores quants outside the wanted subset", func() { + files := []GGUFFile{{Name: "Model-UD-IQ2_XXS.gguf", SHA256: "a"}} + + Expect(DiscoverUnslothQuants(files)).To(BeEmpty()) + }) +}) diff --git a/.github/ci/apexentries/verify.go b/.github/ci/apexentries/verify.go new file mode 100644 index 000000000..62ca82b07 --- /dev/null +++ b/.github/ci/apexentries/verify.go @@ -0,0 +1,312 @@ +package main + +import ( + "fmt" + "os" + "strings" + + "gopkg.in/yaml.v3" +) + +type verifyEntry struct { + Name string `yaml:"name"` + Tags []string `yaml:"tags"` + Variants []VariantRef `yaml:"variants"` + Overrides struct { + // Backend scopes the checks that only hold for one engine. An entry that + // declares none takes its configuration from the referenced url: template, + // which this verifier never reads, so it cannot be judged either way. + Backend string `yaml:"backend"` + Options []string `yaml:"options"` + // MMProj and DraftModel name the files that are not weights. They are + // the only signal for it: a drafter lands in the same models/ prefix as + // the weights, so the path alone cannot tell them apart. + MMProj string `yaml:"mmproj"` + DraftModel string `yaml:"draft_model"` + } `yaml:"overrides"` + Files []struct { + Filename string `yaml:"filename"` + SHA256 string `yaml:"sha256"` + URI string `yaml:"uri"` + } `yaml:"files"` +} + +// Verify checks the invariants the variants schema and the tagging rule +// require. It returns every problem rather than the first, so one run tells the +// author everything that needs fixing. +func Verify(path string) []string { + raw, err := os.ReadFile(path) + if err != nil { + return []string{fmt.Sprintf("reading %s: %v", path, err)} + } + + var entries []verifyEntry + if err := yaml.Unmarshal(raw, &entries); err != nil { + return []string{fmt.Sprintf("parsing %s: %v", path, err)} + } + + var problems []string + + byName := map[string]verifyEntry{} + for _, e := range entries { + if _, seen := byName[e.Name]; seen { + problems = append(problems, fmt.Sprintf("duplicate entry name: %s", e.Name)) + continue + } + byName[e.Name] = e + } + + for _, e := range entries { + for _, v := range e.Variants { + target, ok := byName[v.Model] + if !ok { + problems = append(problems, fmt.Sprintf("%s: variant %q does not exist", e.Name, v.Model)) + continue + } + if len(target.Variants) > 0 { + problems = append(problems, fmt.Sprintf("%s: variant %q declares variants of its own", e.Name, v.Model)) + } + } + + for _, f := range e.Files { + if requiresSHA256(f.Filename) && f.SHA256 == "" { + problems = append(problems, fmt.Sprintf("%s: file %s has no sha256", e.Name, f.Filename)) + } + } + + problems = append(problems, checkWeightCount(e)...) + problems = append(problems, checkFeatureTag(e, "dflash")...) + problems = append(problems, checkFeatureTag(e, "mtp")...) + } + + problems = append(problems, checkPathCollisions(entries)...) + + return problems +} + +// checkPathCollisions catches two different upstream files claiming one local +// path. The install layer keys on the local filename, so whichever entry is +// installed second either overwrites weights the first entry recorded a +// different sha256 for or is skipped as already present. Either way some entry +// afterwards serves bytes that do not match its own checksum, and nothing at +// install time says so. +// +// This is an index-wide invariant rather than a per-entry one: neither entry is +// wrong on its own and the collision exists only in their pairing. The usual +// source is a path scheme built from the repo's BARE name, because two owners +// publishing the same model name is routine for quantizers. +// +// Sharing a path is fine when the uri is the same, which is how several entries +// legitimately reuse one projector. Files with no uri are skipped: there is +// nothing to compare. +func checkPathCollisions(entries []verifyEntry) []string { + type source struct{ uri, entry string } + + first := map[string]source{} + reported := map[string]bool{} + + var problems []string + for _, e := range entries { + for _, f := range e.Files { + if f.Filename == "" || f.URI == "" { + continue + } + prev, seen := first[f.Filename] + if !seen { + first[f.Filename] = source{uri: f.URI, entry: e.Name} + continue + } + if prev.uri == f.URI || reported[f.Filename] { + continue + } + // Reported once per path however many entries pile onto it, so one + // heavily reused filename cannot bury the rest of the report. + reported[f.Filename] = true + problems = append(problems, fmt.Sprintf( + "local path %s is claimed by two different uris: %s (%s) and %s (%s)", + f.Filename, prev.uri, prev.entry, f.URI, e.Name)) + } + } + return problems +} + +// auxiliaryExtensions are the metadata formats an entry ships beside its +// weights, where an unverified download is a nuisance rather than a hole. +// +// The exclusion is stated as a list of metadata formats on purpose. Requiring +// the checksum only on a blessed list of weight formats would silently exempt +// every format nobody has shipped yet, and it already exempted safetensors +// weights, which are downloaded and loaded exactly like GGUF ones. +var auxiliaryExtensions = []string{".json", ".txt", ".md"} + +// requiresSHA256 reports whether an unverified download of this file would be +// a supply-chain hole rather than a cosmetic gap. +func requiresSHA256(filename string) bool { + for _, ext := range auxiliaryExtensions { + if strings.HasSuffix(filename, ext) { + return false + } + } + return true +} + +// checkWeightCount catches an entry carrying two whole models. The flat-match +// branch in DiscoverUnslothQuants appends every match, so a quant label that is +// a suffix of another one (Q8_0 of UD-Q8_0) collects both files into one build +// while the rendered model: points at only the first. The result downloads +// twice the bytes and serves whichever file sorted first, silently. +// +// Shards are exempt because a sharded build is legitimately many files. +// +// The collision is a property of llama-cpp quant discovery, so the check is +// scoped to that backend. Multi-component TTS, ASR and diffusion engines ship an +// encoder, a decoder and a vocoder as one model, and there the second GGUF is +// the design rather than a bug. +func checkWeightCount(e verifyEntry) []string { + if e.Overrides.Backend != "llama-cpp" { + return nil + } + + var weights []string + for _, f := range e.Files { + switch { + case !strings.HasSuffix(f.Filename, ".gguf"): + case shardRE.MatchString(f.Filename): + case f.Filename == e.Overrides.MMProj: + case f.Filename == e.Overrides.DraftModel: + default: + weights = append(weights, f.Filename) + } + } + + if len(weights) > 1 { + return []string{fmt.Sprintf("%s: more than one weight file: %s", e.Name, strings.Join(weights, ", "))} + } + return nil +} + +// checkFeatureTag enforces the rule in both directions. A tag without the +// configuration promotes a build that is no faster; configuration without the +// tag leaves a genuinely faster build ranked as plain. +// +// It only speaks about backends whose declaration it can actually read, because +// a rule applied where the evidence is invisible reports noise rather than bugs. +func checkFeatureTag(e verifyEntry, feature string) []string { + decl, configured, judgeable := featureDeclaration(e, feature) + if !judgeable { + return nil + } + + tagged := false + for _, t := range e.Tags { + if t == feature { + tagged = true + break + } + } + + switch { + case tagged && !configured: + return []string{fmt.Sprintf("%s: tagged %s but sets no %s", e.Name, feature, decl)} + case configured && !tagged: + return []string{fmt.Sprintf("%s: sets %s but is not tagged %s", e.Name, decl, feature)} + } + return nil +} + +// featureDeclaration implements the per-backend table in +// .agents/adding-gallery-models.md. It returns the declaration the backend uses +// to configure the feature, whether the entry carries it, and whether this +// verifier is in a position to answer at all. +func featureDeclaration(e verifyEntry, feature string) (decl string, configured, judgeable bool) { + switch e.Overrides.Backend { + case "llama-cpp": + decl = "spec_type:draft-" + feature + for _, o := range e.Overrides.Options { + if strings.TrimSpace(o) == decl { + return decl, true, true + } + } + return decl, false, true + + case "ds4": + // ds4 carries the MTP heads in the weights and turns them on with + // mtp_path / mtp_draft. It has no dflash counterpart, so dflash is not a + // question that can be asked of a ds4 entry. + if feature != "mtp" { + return "", false, false + } + decl = "mtp_path:" + for _, o := range e.Overrides.Options { + o = strings.TrimSpace(o) + if strings.HasPrefix(o, "mtp_path:") || strings.HasPrefix(o, "mtp_draft:") { + return decl, true, true + } + } + return decl, false, true + + default: + // sglang configures the feature with speculative_algorithm: in the + // referenced gallery/*.yaml, and an entry that declares no backend takes + // its whole configuration from its url: template. Verify reads one index + // file and follows neither, so it must not judge these in either + // direction. + return "", false, false + } +} + +// UnaccountedQuants reports a wanted quant the repo demonstrably publishes but +// that discovery produced no build for. The layout that triggers it today is +// root-level shards, which match neither branch of DiscoverUnslothQuants; no +// counterpart ships that way yet, but a batch generator must not drop a build +// with nothing said about it. +func UnaccountedQuants(files []GGUFFile, builds []QuantBuild) []string { + built := map[string]bool{} + for _, b := range builds { + built[b.Quant] = true + } + + var problems []string + for _, q := range WantedQuants { + if built[q] { + continue + } + for _, f := range files { + if filePublishesQuant(f.Name, q) { + problems = append(problems, fmt.Sprintf("quant %s is published upstream (%s) but produced no build", q, f.Name)) + break + } + } + } + return problems +} + +// filePublishesQuant reports whether an upstream file is a publication of +// quant q. It anchors on the quant label the way DiscoverUnslothQuants does, +// as the trailing token of the base name or as the sharding subdirectory, so +// the diagnostic and the discovery it audits cannot disagree about what a file +// is. +// +// An unanchored match would reproduce the very collision this diagnostic warns +// about: Q8_0 is a substring of UD-Q8_0, so a repo publishing only UD-Q8_0 +// would be reported as publishing an unbuilt Q8_0, which it does not, and +// UD-Q8_0 is not a wanted quant at all. +func filePublishesQuant(name, q string) bool { + if strings.HasPrefix(name, q+"/") { + return true + } + + base := name[strings.LastIndex(name, "/")+1:] + // Shard numbering sits between the quant label and the extension, so it has + // to come off before the label can be read as the trailing token. Root-level + // shards are the layout that matches neither branch of + // DiscoverUnslothQuants, and so the layout this diagnostic mainly catches. + base = shardRE.ReplaceAllString(base, ".gguf") + + if !strings.HasSuffix(base, "-"+q+".gguf") { + return false + } + // UD- is unsloth's dynamic-quant modifier, and UD- is a distinct quant + // label rather than a publication of . + return !strings.HasSuffix(base, "-UD-"+q+".gguf") +} diff --git a/.github/ci/apexentries/verify_test.go b/.github/ci/apexentries/verify_test.go new file mode 100644 index 000000000..b03430cdb --- /dev/null +++ b/.github/ci/apexentries/verify_test.go @@ -0,0 +1,480 @@ +package main + +import ( + "os" + "path/filepath" + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("Verify", func() { + write := func(body string) string { + dir := GinkgoT().TempDir() + p := filepath.Join(dir, "index.yaml") + Expect(os.WriteFile(p, []byte(body), 0o600)).To(Succeed()) + return p + } + + It("passes a sound index", func() { + Expect(Verify(write(` +- name: parent + variants: + - model: child + files: + - filename: a.gguf + sha256: aa + uri: https://example.com/a.gguf +- name: child + files: + - filename: b.gguf + sha256: bb + uri: https://example.com/b.gguf +`))).To(BeEmpty()) + }) + + It("reports a variant pointing at a missing entry", func() { + Expect(Verify(write(` +- name: parent + variants: + - model: ghost + files: + - filename: a.gguf + sha256: aa + uri: https://example.com/a.gguf +`))).To(ContainElement(ContainSubstring("ghost"))) + }) + + It("reports a variant that itself declares variants", func() { + Expect(Verify(write(` +- name: parent + variants: + - model: child + files: + - filename: a.gguf + sha256: aa + uri: https://example.com/a.gguf +- name: child + variants: + - model: grandchild + files: + - filename: b.gguf + sha256: bb + uri: https://example.com/b.gguf +- name: grandchild + files: + - filename: c.gguf + sha256: cc + uri: https://example.com/c.gguf +`))).To(ContainElement(ContainSubstring("declares variants of its own"))) + }) + + It("reports duplicate entry names", func() { + Expect(Verify(write(` +- name: dup + files: + - filename: a.gguf + sha256: aa + uri: https://example.com/a.gguf +- name: dup + files: + - filename: b.gguf + sha256: bb + uri: https://example.com/b.gguf +`))).To(ContainElement(ContainSubstring("duplicate entry name"))) + }) + + It("reports a file with no sha256", func() { + Expect(Verify(write(` +- name: one + files: + - filename: a.gguf + uri: https://example.com/a.gguf +`))).To(ContainElement(ContainSubstring("no sha256"))) + }) + + It("reports an entry tagged dflash without a matching spec_type", func() { + Expect(Verify(write(` +- name: liar + tags: + - dflash + overrides: + backend: llama-cpp + options: + - use_jinja:true + files: + - filename: a.gguf + sha256: aa + uri: https://example.com/a.gguf +`))).To(ContainElement(ContainSubstring("tagged dflash"))) + }) + + It("reports an entry configuring spec_type without the tag", func() { + Expect(Verify(write(` +- name: shy + overrides: + backend: llama-cpp + options: + - spec_type:draft-mtp + files: + - filename: a.gguf + sha256: aa + uri: https://example.com/a.gguf +`))).To(ContainElement(ContainSubstring("not tagged mtp"))) + }) + + // ds4 carries the MTP heads in the weights and names them with mtp_path, so + // the rule holds there in a different vocabulary rather than not at all. + It("reports a ds4 entry configuring mtp_path without the tag", func() { + Expect(Verify(write(` +- name: ds4-shy + overrides: + backend: ds4 + options: + - mtp_path:model-mtp.gguf + - mtp_draft:2 + files: + - filename: a.gguf + sha256: aa + uri: https://example.com/a.gguf +`))).To(ContainElement(ContainSubstring("not tagged mtp"))) + }) + + It("reports a ds4 entry tagged mtp that configures no mtp_path", func() { + Expect(Verify(write(` +- name: ds4-liar + tags: + - mtp + overrides: + backend: ds4 + options: + - context_size:4096 + files: + - filename: a.gguf + sha256: aa + uri: https://example.com/a.gguf +`))).To(ContainElement(ContainSubstring("tagged mtp"))) + }) + + It("accepts a ds4 entry that both configures mtp_path and carries the tag", func() { + Expect(Verify(write(` +- name: ds4-honest + tags: + - mtp + overrides: + backend: ds4 + options: + - mtp_path:model-mtp.gguf + files: + - filename: a.gguf + sha256: aa + uri: https://example.com/a.gguf +`))).To(BeEmpty()) + }) + + // sglang declares speculative_algorithm in the referenced gallery/*.yaml, + // which Verify never reads, so it may not judge such an entry either way. + It("says nothing about an sglang entry tagged mtp", func() { + Expect(Verify(write(` +- name: sglang-mtp + tags: + - mtp + overrides: + backend: sglang + files: [] +`))).To(BeEmpty()) + }) + + It("says nothing about the tag on an entry with no declared backend", func() { + Expect(Verify(write(` +- name: templated + tags: + - mtp + files: + - filename: a.gguf + sha256: aa + uri: https://example.com/a.gguf +`))).To(BeEmpty()) + }) + + // The flat-match branch in unsloth.go appends every match, so a repo + // publishing both a plain and a UD Q8_0 renders one entry holding two full + // models while model: points at only the first. + It("reports an entry holding more than one non-shard weight file", func() { + Expect(Verify(write(` +- name: greedy + overrides: + backend: llama-cpp + options: + - use_jinja:true + parameters: + model: llama-cpp/models/repo/Model-Q8_0.gguf + files: + - filename: llama-cpp/models/repo/Model-Q8_0.gguf + sha256: aa + uri: https://example.com/a.gguf + - filename: llama-cpp/models/repo/Model-UD-Q8_0.gguf + sha256: bb + uri: https://example.com/b.gguf +`))).To(ContainElement(ContainSubstring("more than one weight file"))) + }) + + It("accepts many shards alongside an mmproj and a drafter", func() { + Expect(Verify(write(` +- name: sharded + tags: + - mtp + overrides: + backend: llama-cpp + options: + - spec_type:draft-mtp + mmproj: llama-cpp/mmproj/repo/mm.gguf + draft_model: llama-cpp/models/repo/Model-draft.gguf + files: + - filename: llama-cpp/models/repo/Model-00001-of-00002.gguf + sha256: aa + uri: https://example.com/a.gguf + - filename: llama-cpp/models/repo/Model-00002-of-00002.gguf + sha256: bb + uri: https://example.com/b.gguf + - filename: llama-cpp/mmproj/repo/mm.gguf + sha256: cc + uri: https://example.com/c.gguf + - filename: llama-cpp/models/repo/Model-draft.gguf + sha256: dd + uri: https://example.com/d.gguf +`))).To(BeEmpty()) + }) + + // Multi-component TTS and ASR engines legitimately ship an encoder, a + // tokenizer, a vocoder and so on as one model, so the collision the weight + // count catches does not exist for them. + It("accepts a multi-component non-llama-cpp entry declaring five weights", func() { + Expect(Verify(write(` +- name: multi + overrides: + backend: qwen3-tts-cpp + files: + - filename: talker.gguf + sha256: aa + uri: https://example.com/a.gguf + - filename: tokenizer.gguf + sha256: bb + uri: https://example.com/b.gguf + - filename: vocoder.gguf + sha256: cc + uri: https://example.com/c.gguf + - filename: encoder.gguf + sha256: dd + uri: https://example.com/d.gguf + - filename: vae.gguf + sha256: ee + uri: https://example.com/e.gguf +`))).To(BeEmpty()) + }) + + It("says nothing about the weight count of an entry with no declared backend", func() { + Expect(Verify(write(` +- name: templated-weights + files: + - filename: model-Q4_K_M.gguf + sha256: aa + uri: https://example.com/a.gguf + - filename: model-mmproj-f16.gguf + sha256: bb + uri: https://example.com/b.gguf +`))).To(BeEmpty()) + }) + + It("says nothing about an auxiliary metadata file carrying no sha256", func() { + Expect(Verify(write(` +- name: aux + files: + - filename: a.gguf + sha256: aa + uri: https://example.com/a.gguf + - filename: params.json + sha256: "" + uri: https://example.com/params.json +`))).To(BeEmpty()) + }) + + // safetensors weights are downloaded and loaded exactly like GGUF weights, + // so an unverified one is the same supply-chain hole. + It("reports a safetensors weight carrying no sha256", func() { + Expect(Verify(write(` +- name: vae + files: + - filename: wan_2.1_vae.safetensors + sha256: "" + uri: https://example.com/vae.safetensors +`))).To(ContainElement(ContainSubstring("no sha256"))) + }) + + It("says nothing about a txt or md file carrying no sha256", func() { + Expect(Verify(write(` +- name: docs + files: + - filename: notes.txt + sha256: "" + uri: https://example.com/notes.txt + - filename: README.md + sha256: "" + uri: https://example.com/README.md +`))).To(BeEmpty()) + }) +}) + +var _ = Describe("UnaccountedQuants", func() { + // A quant published only as root-level shards matches neither branch in + // DiscoverUnslothQuants, so without this diagnostic the build would vanish + // from a batch run with nothing said about it. + It("reports a wanted quant upstream publishes but discovery dropped", func() { + files := []GGUFFile{ + {Name: "Model-UD-Q4_K_M-00001-of-00003.gguf", SHA256: "aa"}, + {Name: "Model-UD-Q4_K_M-00002-of-00003.gguf", SHA256: "bb"}, + {Name: "Model-UD-Q4_K_M-00003-of-00003.gguf", SHA256: "cc"}, + } + + Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))). + To(ContainElement(ContainSubstring("UD-Q4_K_M"))) + }) + + It("says nothing when every published wanted quant produced a build", func() { + files := []GGUFFile{ + {Name: "Model-UD-Q4_K_M.gguf", SHA256: "aa"}, + {Name: "UD-Q6_K/Model-UD-Q6_K-00001-of-00002.gguf", SHA256: "bb"}, + {Name: "UD-Q6_K/Model-UD-Q6_K-00002-of-00002.gguf", SHA256: "cc"}, + } + + Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty()) + }) + + It("says nothing about a wanted quant the repo does not publish at all", func() { + files := []GGUFFile{{Name: "Model-UD-Q4_K_M.gguf", SHA256: "aa"}} + + Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty()) + }) + + // UD-Q8_0 is its own quant label and is not a wanted one. Reading it as a + // publication of Q8_0 is the substring collision this diagnostic exists to + // warn about, and subdirectory-sharded UD quants are the normal unsloth + // layout for large repos, so the false positive would fire on every batch. + It("does not read a subdirectory-sharded UD-Q8_0 as a published Q8_0", func() { + files := []GGUFFile{ + {Name: "UD-Q8_0/Model-UD-Q8_0-00001-of-00002.gguf", SHA256: "aa"}, + {Name: "UD-Q8_0/Model-UD-Q8_0-00002-of-00002.gguf", SHA256: "bb"}, + } + + Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))).To(BeEmpty()) + }) + + // A quant in its own subdirectory but not shard-numbered matches neither + // branch of DiscoverUnslothQuants, so it is genuinely published and + // genuinely undiscovered. + It("reports a wanted quant published in its own subdirectory without shard numbering", func() { + files := []GGUFFile{{Name: "Q8_0/Model-Q8_0.gguf", SHA256: "aa"}} + + Expect(UnaccountedQuants(files, DiscoverUnslothQuants(files))). + To(ContainElement(ContainSubstring("quant Q8_0 is published upstream"))) + }) + + // builds is empty on purpose: it isolates the file-to-quant match from + // whatever DiscoverUnslothQuants would have made of the same file. + It("matches the flat single-file layout", func() { + files := []GGUFFile{{Name: "Model-Q8_0.gguf", SHA256: "aa"}} + + Expect(UnaccountedQuants(files, nil)). + To(ConsistOf(ContainSubstring("quant Q8_0 is published upstream"))) + }) +}) + +var _ = Describe("Verify local path collisions", func() { + write := func(body string) string { + dir := GinkgoT().TempDir() + p := filepath.Join(dir, "index.yaml") + Expect(os.WriteFile(p, []byte(body), 0o600)).To(Succeed()) + return p + } + + It("reports one local path claimed by two different uris", func() { + // The shape that shipped: LiquidAI and unsloth both publish + // LFM2.5-8B-A1B-GGUF, so a path built from the bare repo name gives both + // entries the same local file under two different checksums. + Expect(Verify(write(` +- name: lfm2.5-8b-a1b + files: + - filename: llama-cpp/models/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf + sha256: 33ab3b8c + uri: https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-Q8_0.gguf +- name: lfm2.5-8b-a1b-q8-0 + files: + - filename: llama-cpp/models/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf + sha256: ec11666b + uri: https://huggingface.co/unsloth/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-Q8_0.gguf +`))).To(ContainElement(SatisfyAll( + ContainSubstring("claimed by two different uris"), + ContainSubstring("lfm2.5-8b-a1b-q8-0"), + ))) + }) + + It("accepts two entries reusing one file from the same uri", func() { + // Sibling builds of one repo legitimately share a projector. + Expect(Verify(write(` +- name: a + files: + - filename: llama-cpp/mmproj/mudler/Example-GGUF/mmproj-F16.gguf + sha256: cc + uri: https://huggingface.co/mudler/Example-GGUF/resolve/main/mmproj-F16.gguf +- name: b + files: + - filename: llama-cpp/mmproj/mudler/Example-GGUF/mmproj-F16.gguf + sha256: cc + uri: https://huggingface.co/mudler/Example-GGUF/resolve/main/mmproj-F16.gguf +`))).To(BeEmpty()) + }) + + It("reports a collision once however many entries pile onto the path", func() { + problems := Verify(write(` +- name: a + files: + - filename: shared.gguf + sha256: aa + uri: https://example.com/a.gguf +- name: b + files: + - filename: shared.gguf + sha256: bb + uri: https://example.com/b.gguf +- name: c + files: + - filename: shared.gguf + sha256: cc + uri: https://example.com/c.gguf +`)) + + var collisions int + for _, p := range problems { + if strings.Contains(p, "claimed by two different uris") { + collisions++ + } + } + Expect(collisions).To(Equal(1)) + }) + + It("says nothing about files that carry no uri", func() { + // A hand-written entry may record only a checksum. There is no upstream + // to compare, so the check cannot conclude anything either way. + Expect(Verify(write(` +- name: a + files: + - filename: shared.gguf + sha256: aa +- name: b + files: + - filename: shared.gguf + sha256: bb +`))).To(BeEmpty()) + }) +}) diff --git a/.github/ci/galleryedit/edit.go b/.github/ci/galleryedit/edit.go new file mode 100644 index 000000000..62044d025 --- /dev/null +++ b/.github/ci/galleryedit/edit.go @@ -0,0 +1,152 @@ +// Package galleryedit splices variant references into the LocalAI gallery index +// as TEXT. +// +// Re-serialising the index through a YAML marshaller would reflow 40,000 lines, +// drop the anchors and merge keys the gallery relies on, and produce a diff no +// reviewer could read, which makes a pull request worthless even when the +// content inside it is right. Every generator that adds variants to an entry the +// gallery already ships therefore edits lines, and they share this package so +// that two of them cannot drift apart on where a variants block belongs. +package galleryedit + +import ( + "fmt" + "regexp" + "sort" + "strings" +) + +var ( + entryStart = regexp.MustCompile(`^-(?: |$)`) + inlineName = regexp.MustCompile(`^- (?:&\S+ )?name:`) + keyName = regexp.MustCompile(`^ name:`) + keyVariants = regexp.MustCompile(`^ variants:\s*(.*)$`) + variantItem = regexp.MustCompile(`^ - `) + unsafeInName = regexp.MustCompile(`[:#{}\[\],&*?|>'"%@` + "`" + `]|^\s|\s$`) +) + +// Entry is the positional view of one gallery entry: what it is called and +// which lines it occupies. Nothing about what the entry MEANS belongs here, so +// each caller keeps its own semantic decode and only hands over the coordinates. +type Entry struct { + Name string + // StartLine and EndLine bound the entry, zero based and half open. + StartLine int + EndLine int +} + +// Insert is one entry's pending variants addition. The caller owns the contents +// of Variants: this package neither orders nor deduplicates them, because the +// right order and the right dedup rule differ between generators. +type Insert struct { + Entry Entry + Variants []string +} + +// Scan splits index text into lines and reports the line each top level list +// item begins on. +func Scan(text string) (lines []string, starts []int) { + lines = strings.Split(text, "\n") + for i, line := range lines { + if entryStart.MatchString(line) { + starts = append(starts, i) + } + } + return lines, starts +} + +// Apply splices every insert into the index lines and returns the new text. +func Apply(lines []string, inserts []Insert) ([]string, error) { + type edit struct { + at int + remove int + insert []string + } + var edits []edit + + for _, in := range inserts { + if len(in.Variants) == 0 { + continue + } + + items := make([]string, 0, len(in.Variants)) + for _, v := range in.Variants { + items = append(items, " - model: "+QuoteName(v)) + } + + at, remove, err := insertionPoint(lines, in.Entry) + if err != nil { + return nil, err + } + block := items + if remove > 0 || !hasVariantsKey(lines, in.Entry) { + block = append([]string{" variants:"}, items...) + } + edits = append(edits, edit{at: at, remove: remove, insert: block}) + } + + // Applying from the bottom up keeps every line number computed against the + // original text valid while earlier edits are still pending. + sort.Slice(edits, func(i, j int) bool { return edits[i].at > edits[j].at }) + + out := append([]string(nil), lines...) + for _, e := range edits { + tail := append([]string(nil), out[e.at+e.remove:]...) + out = append(out[:e.at], append(append([]string(nil), e.insert...), tail...)...) + } + return out, nil +} + +func hasVariantsKey(lines []string, e Entry) bool { + for i := e.StartLine; i < e.EndLine; i++ { + if keyVariants.MatchString(lines[i]) { + return true + } + } + return false +} + +// insertionPoint reports where new variant items belong, and how many existing +// lines the insertion replaces. +// +// An entry with no variants key gets one right after its name, which is where +// the hand-written families put it. An entry with an empty "variants: []" has +// that line replaced by a block. An entry with a block gets its items appended. +func insertionPoint(lines []string, e Entry) (at int, remove int, err error) { + for i := e.StartLine; i < e.EndLine; i++ { + m := keyVariants.FindStringSubmatch(lines[i]) + if m == nil { + continue + } + if strings.TrimSpace(m[1]) == "[]" { + return i, 1, nil + } + if strings.TrimSpace(m[1]) != "" { + return 0, 0, fmt.Errorf("entry %q writes its variants inline (%q); this job only edits block lists", e.Name, strings.TrimSpace(m[1])) + } + last := i + for j := i + 1; j < e.EndLine && variantItem.MatchString(lines[j]); j++ { + last = j + } + return last + 1, 0, nil + } + + if inlineName.MatchString(lines[e.StartLine]) { + return e.StartLine + 1, 0, nil + } + for i := e.StartLine; i < e.EndLine; i++ { + if keyName.MatchString(lines[i]) { + return i + 1, 0, nil + } + } + return 0, 0, fmt.Errorf("entry %q has no name line to anchor the insertion to", e.Name) +} + +// QuoteName quotes a variant reference when the name would otherwise change +// meaning as bare YAML. Config-suffixed names carry a ":" and always need it. +func QuoteName(name string) string { + if unsafeInName.MatchString(name) { + return `"` + strings.ReplaceAll(name, `"`, `\"`) + `"` + } + return name +} diff --git a/.github/ci/variantproposals/edit.go b/.github/ci/variantproposals/edit.go index 73290f114..47d6e305a 100644 --- a/.github/ci/variantproposals/edit.go +++ b/.github/ci/variantproposals/edit.go @@ -2,119 +2,41 @@ package main import ( "fmt" - "regexp" - "sort" "strings" -) -var ( - inlineName = regexp.MustCompile(`^- (?:&\S+ )?name:`) - keyName = regexp.MustCompile(`^ name:`) - keyVariants = regexp.MustCompile(`^ variants:\s*(.*)$`) - variantItem = regexp.MustCompile(`^ - `) - unsafeInName = regexp.MustCompile(`[:#{}\[\],&*?|>'"%@` + "`" + `]|^\s|\s$`) + "github.com/mudler/LocalAI/.github/ci/galleryedit" ) // ApplyFamilies writes the proposed variant lists into the index text. // -// The edit is textual on purpose. Re-serialising the index through a YAML -// marshaller would reflow 40,000 lines, drop the anchors and merge keys the -// gallery relies on, and produce a diff no reviewer could read, which would -// make the pull request worthless even when the proposals inside it are right. +// The line editing itself lives in galleryedit, shared with the apexentries +// generator. Both jobs add variants to entries the gallery already ships, and a +// second answer to "where does a variants block go" would drift from this one; +// see that package for why the edit is textual rather than a YAML round trip. func ApplyFamilies(ix *Index, families []Family) ([]string, error) { byName, _ := ix.ByName() - type edit struct { - at int - remove int - insert []string - ordinal int - } - var edits []edit - + var inserts []galleryedit.Insert for _, f := range families { entry, ok := byName[strings.ToLower(f.Parent)] if !ok { return nil, fmt.Errorf("parent %q is not in the index", f.Parent) } - items := make([]string, 0, len(f.Proposals)) + + variants := make([]string, 0, len(f.Proposals)) for _, p := range f.Proposals { - items = append(items, " - model: "+quoteName(p.Variant)) + variants = append(variants, p.Variant) } - at, remove, err := insertionPoint(ix, entry) - if err != nil { - return nil, err - } - insert := items - if remove > 0 || !hasVariantsKey(ix, entry) { - insert = append([]string{" variants:"}, items...) - } - edits = append(edits, edit{at: at, remove: remove, insert: insert, ordinal: entry.Index}) + inserts = append(inserts, galleryedit.Insert{ + Entry: galleryedit.Entry{ + Name: entry.Name, + StartLine: entry.StartLine, + EndLine: entry.EndLine, + }, + Variants: variants, + }) } - // Applying from the bottom up keeps every line number computed against the - // original text valid while earlier edits are still pending. - sort.Slice(edits, func(i, j int) bool { return edits[i].at > edits[j].at }) - - lines := append([]string(nil), ix.Lines...) - for _, e := range edits { - tail := append([]string(nil), lines[e.at+e.remove:]...) - lines = append(lines[:e.at], append(append([]string(nil), e.insert...), tail...)...) - } - return lines, nil -} - -func hasVariantsKey(ix *Index, e *GalleryEntry) bool { - for i := e.StartLine; i < e.EndLine; i++ { - if keyVariants.MatchString(ix.Lines[i]) { - return true - } - } - return false -} - -// insertionPoint reports where new variant items belong, and how many existing -// lines the insertion replaces. -// -// An entry with no variants key gets one right after its name, which is where -// the hand-written families put it. An entry with an empty "variants: []" has -// that line replaced by a block. An entry with a block gets its items appended. -func insertionPoint(ix *Index, e *GalleryEntry) (at int, remove int, err error) { - for i := e.StartLine; i < e.EndLine; i++ { - m := keyVariants.FindStringSubmatch(ix.Lines[i]) - if m == nil { - continue - } - if strings.TrimSpace(m[1]) == "[]" { - return i, 1, nil - } - if strings.TrimSpace(m[1]) != "" { - return 0, 0, fmt.Errorf("entry %q writes its variants inline (%q); this job only edits block lists", e.Name, strings.TrimSpace(m[1])) - } - last := i - for j := i + 1; j < e.EndLine && variantItem.MatchString(ix.Lines[j]); j++ { - last = j - } - return last + 1, 0, nil - } - - if inlineName.MatchString(ix.Lines[e.StartLine]) { - return e.StartLine + 1, 0, nil - } - for i := e.StartLine; i < e.EndLine; i++ { - if keyName.MatchString(ix.Lines[i]) { - return i + 1, 0, nil - } - } - return 0, 0, fmt.Errorf("entry %q has no name line to anchor the insertion to", e.Name) -} - -// quoteName quotes a variant reference when the name would otherwise change -// meaning as bare YAML. Config-suffixed names carry a ":" and always need it. -func quoteName(name string) string { - if unsafeInName.MatchString(name) { - return `"` + strings.ReplaceAll(name, `"`, `\"`) + `"` - } - return name + return galleryedit.Apply(ix.Lines, inserts) } diff --git a/.github/ci/variantproposals/index.go b/.github/ci/variantproposals/index.go index 870a1deea..6cce3699d 100644 --- a/.github/ci/variantproposals/index.go +++ b/.github/ci/variantproposals/index.go @@ -8,6 +8,8 @@ import ( "strings" "gopkg.in/yaml.v3" + + "github.com/mudler/LocalAI/.github/ci/galleryedit" ) // File is the subset of a gallery file entry the proposer reads. @@ -57,7 +59,6 @@ type Index struct { } var ( - entryStart = regexp.MustCompile(`^-(?: |$)`) anchorStart = regexp.MustCompile(`^- &(\S+)`) mergeStart = regexp.MustCompile(`^- !!merge <<: \*(\S+)`) ) @@ -82,13 +83,7 @@ func ParseIndex(text string) (*Index, error) { return nil, fmt.Errorf("decoding gallery index: %w", err) } - lines := strings.Split(text, "\n") - var starts []int - for i, line := range lines { - if entryStart.MatchString(line) { - starts = append(starts, i) - } - } + lines, starts := galleryedit.Scan(text) if len(starts) != len(entries) { return nil, fmt.Errorf("gallery index has %d decoded entries but %d top level list items; refusing to edit by line number", len(entries), len(starts)) } diff --git a/.github/workflows/ci-tools-tests.yaml b/.github/workflows/ci-tools-tests.yaml new file mode 100644 index 000000000..604426253 --- /dev/null +++ b/.github/workflows/ci-tools-tests.yaml @@ -0,0 +1,39 @@ +--- +# The packages under .github/ci/ are invisible to `go list ./...`, so neither +# `make lint` nor the repository test run ever touches them. Their specs are +# dead weight until a workflow names each package explicitly. +name: 'CI tool tests' +on: + pull_request: + paths: + - '.github/ci/**' + - '.github/workflows/ci-tools-tests.yaml' + push: + branches: + - master + paths: + - '.github/ci/**' +jobs: + ci-tools: + name: 'Test the .github/ci generators' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: actions/setup-go@v5 + with: + go-version-file: go.mod + cache: false + + # The discovery heuristics are the risky part of these tools. A regression + # produces confident, wrong gallery entries, which is worse than no tool. + - name: 'Test the APEX entry generator' + run: go test ./.github/ci/apexentries/ + + - name: 'Test the variant proposer' + run: go test ./.github/ci/variantproposals/ + + # Shared by both generators above. Its behaviour is exercised through their + # specs; this step exists so a break in the shared package fails under its + # own name rather than as a puzzling failure in whichever caller ran first. + - name: 'Test the shared gallery editor' + run: go test ./.github/ci/galleryedit/ diff --git a/.gitignore b/.gitignore index 689c043d1..5286461b3 100644 --- a/.gitignore +++ b/.gitignore @@ -111,3 +111,10 @@ core/http/react-ui/test-results/ # the realtime-conformance gate; only the .fizz sources are authoritative. formal-verification/*.json formal-verification/out/ + +# `go build ./.github/ci/apexentries` drops a binary of the package name into +# whatever directory it runs in, one `git add -A` away from being committed. +# Both paths are anchored: an unanchored `apexentries` would also match the +# package directory itself and untrack the source. +/apexentries +/.github/ci/apexentries/apexentries diff --git a/gallery/index.yaml b/gallery/index.yaml index d7c777609..88ce6836f 100644 --- a/gallery/index.yaml +++ b/gallery/index.yaml @@ -738,6 +738,15 @@ sha256: 5720d1f671b4996481274fffe01868c3c36e87c135cc8538471cc7bd6087b106 uri: https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B-GGUF/resolve/main/ornith-1.0-9b-Q4_K_M.gguf - name: "ornith-1.0-35b" + variants: + - model: ornith-1.0-35b-apex-i-quality + - model: ornith-1.0-35b-apex-i-balanced + - model: ornith-1.0-35b-apex-i-compact + - model: ornith-1.0-35b-apex-i-mini + - model: ornith-1.0-35b-ud-q4-k-m + - model: ornith-1.0-35b-ud-q5-k-m + - model: ornith-1.0-35b-ud-q6-k + - model: ornith-1.0-35b-q8-0 url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: - https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-GGUF @@ -927,7 +936,7 @@ - https://huggingface.co/Qwen/Qwen3-4B - https://huggingface.co/unsloth/Qwen3-4B-GGUF - https://huggingface.co/z-lab/Qwen3-4B-DFlash - - https://huggingface.co/lym00/Qwen3-4B-DFlash-GGUF-Test + - https://huggingface.co/AtomicChat/Qwen3-4B-DFlash-GGUF description: | Qwen3-4B paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend. This is the canonical DFlash pairing documented upstream (`z-lab/Qwen3-4B-DFlash` + `Qwen/Qwen3-4B`). @@ -948,7 +957,7 @@ overrides: backend: llama-cpp flash_attention: "on" - draft_model: llama-cpp/models/Qwen3-4B-DFlash-GGUF-Test/Qwen3-4B-DFlash-q8_0.gguf + draft_model: llama-cpp/models/Qwen3-4B-DFlash-GGUF/Qwen3-4B-DFlash.Q8_0.gguf function: automatic_tool_parsing_fallback: true grammar: @@ -967,9 +976,9 @@ - filename: llama-cpp/models/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf sha256: f6f851777709861056efcdad3af01da38b31223a3ba26e61a4f8bf3a2195813a uri: https://huggingface.co/unsloth/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q4_K_M.gguf - - filename: llama-cpp/models/Qwen3-4B-DFlash-GGUF-Test/Qwen3-4B-DFlash-q8_0.gguf - sha256: 5ecf02bb269fc42277f43961794387fea11ecc367ea2d99e86b2b71cc249aff6 - uri: https://huggingface.co/lym00/Qwen3-4B-DFlash-GGUF-Test/resolve/main/Qwen3-4B-DFlash-q8_0.gguf + - filename: llama-cpp/models/Qwen3-4B-DFlash-GGUF/Qwen3-4B-DFlash.Q8_0.gguf + sha256: 2913a8a5619b30233c0745ccd58de8fbe144b831b54770eb04d4a371221ca273 + uri: https://huggingface.co/AtomicChat/Qwen3-4B-DFlash-GGUF/resolve/main/Qwen3-4B-DFlash.Q8_0.gguf - name: "qwen3.5-4b-dflash" url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: @@ -1025,7 +1034,7 @@ - https://huggingface.co/Qwen/Qwen3.5-9B - https://huggingface.co/unsloth/Qwen3.5-9B-GGUF - https://huggingface.co/z-lab/Qwen3.5-9B-DFlash - - https://huggingface.co/lym00/Qwen3.5-9B-DFlash-GGUF-Test + - https://huggingface.co/AtomicChat/Qwen3.5-9B-DFlash-GGUF description: | Qwen3.5-9B paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend. @@ -1046,7 +1055,7 @@ overrides: backend: llama-cpp flash_attention: "on" - draft_model: llama-cpp/models/Qwen3.5-9B-DFlash-GGUF-Test/Qwen3.5-9B-DFlash-q8_0.gguf + draft_model: llama-cpp/models/Qwen3.5-9B-DFlash-GGUF/Qwen3.5-9B-DFlash.Q8_0.gguf function: automatic_tool_parsing_fallback: true grammar: @@ -1065,9 +1074,9 @@ - filename: llama-cpp/models/Qwen3.5-9B-GGUF/Qwen3.5-9B-Q4_K_M.gguf sha256: 03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8 uri: https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/main/Qwen3.5-9B-Q4_K_M.gguf - - filename: llama-cpp/models/Qwen3.5-9B-DFlash-GGUF-Test/Qwen3.5-9B-DFlash-q8_0.gguf - sha256: 4f76ecff951dd371eea318be9ca60b4dd0800a677c11e14c092fab3358251dc1 - uri: https://huggingface.co/lym00/Qwen3.5-9B-DFlash-GGUF-Test/resolve/main/Qwen3.5-9B-DFlash-q8_0.gguf + - filename: llama-cpp/models/Qwen3.5-9B-DFlash-GGUF/Qwen3.5-9B-DFlash.Q8_0.gguf + sha256: 27b9d18e605aea9c50ef506e4b63921e0dc624cc91c63c60617a8295f153d436 + uri: https://huggingface.co/AtomicChat/Qwen3.5-9B-DFlash-GGUF/resolve/main/Qwen3.5-9B-DFlash.Q8_0.gguf - name: "qwen3.6-27b-dflash" url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: @@ -2446,6 +2455,14 @@ uri: https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF/resolve/main/mmproj-BF16.gguf sha256: d904b3579a9fbfbd50bc9bf40cb7384909edbd69fa9276db5ddc853e80f0edca - name: "step-3.7-flash" + variants: + - model: step-3.7-flash-apex-i-quality + - model: step-3.7-flash-apex-i-balanced + - model: step-3.7-flash-apex-i-compact + - model: step-3.7-flash-apex-i-mini + - model: step-3.7-flash-ud-q5-k-m + - model: step-3.7-flash-ud-q6-k + - model: step-3.7-flash-q8-0 url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: - https://huggingface.co/unsloth/Step-3.7-Flash-GGUF @@ -2700,6 +2717,15 @@ - jwt - private_key_block - name: "lfm2.5-8b-a1b" + variants: + - model: lfm2.5-8b-a1b-apex-i-quality + - model: lfm2.5-8b-a1b-apex-i-balanced + - model: lfm2.5-8b-a1b-apex-i-compact + - model: lfm2.5-8b-a1b-apex-i-mini + - model: lfm2.5-8b-a1b-ud-q4-k-m + - model: lfm2.5-8b-a1b-ud-q5-k-m + - model: lfm2.5-8b-a1b-ud-q6-k + - model: lfm2.5-8b-a1b-q8-0 url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: - https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-GGUF @@ -2865,6 +2891,16 @@ sha256: fdc443e974cad1f61c45af1cfd5580855855ddce0d6c14cc500a5714c486ac1d uri: https://huggingface.co/DavidAU/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-NEO-CODE-Di-IMatrix-MAX-GGUF/resolve/main/mmproj-F32.gguf - name: "qwopus3.6-35b-a3b-v1" + variants: + - model: qwopus3.6-35b-a3b-v1-apex-i-quality + - model: qwopus3.6-35b-a3b-v1-apex-i-balanced + - model: qwopus3.6-35b-a3b-v1-apex-i-compact + - model: qwopus3.6-35b-a3b-v1-apex-i-mini + - model: qwopus3.6-35b-a3b-v1-apex-mtp-i-quality + - model: qwopus3.6-35b-a3b-v1-apex-mtp-i-balanced + - model: qwopus3.6-35b-a3b-v1-apex-mtp-i-compact + - model: qwopus3.6-35b-a3b-v1-apex-mtp-i-mini + - model: qwopus3.6-35b-a3b-v1-apex-mtp-i-nano url: "github:mudler/LocalAI/gallery/virtual.yaml@master" urls: - https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-v1-GGUF @@ -3429,6 +3465,16 @@ sha256: fdc443e974cad1f61c45af1cfd5580855855ddce0d6c14cc500a5714c486ac1d uri: https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/resolve/main/mmproj-F32.gguf - name: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled + variants: + - model: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-i-quality + - model: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-i-balanced + - model: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-i-compact + - model: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-i-mini + - model: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-mtp-i-quality + - model: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-mtp-i-balanced + - model: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-mtp-i-compact + - model: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-mtp-i-mini + - model: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-mtp-i-nano url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/hesamation/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-GGUF @@ -3628,6 +3674,18 @@ - model: qwen3.6-35b-a3b-dflash - model: qwen3.6-35b-a3b-nvfp4-mtp - model: qwen3.6-35b-a3b-apex + - model: qwen3.6-35b-a3b-apex-i-quality + - model: qwen3.6-35b-a3b-apex-i-balanced + - model: qwen3.6-35b-a3b-apex-i-compact + - model: qwen3.6-35b-a3b-apex-i-mini + - model: qwen3.6-35b-a3b-ud-q5-k-m + - model: qwen3.6-35b-a3b-ud-q6-k + - model: qwen3.6-35b-a3b-q8-0 + - model: qwen3.6-35b-a3b-apex-mtp-i-quality + - model: qwen3.6-35b-a3b-apex-mtp-i-balanced + - model: qwen3.6-35b-a3b-apex-mtp-i-compact + - model: qwen3.6-35b-a3b-apex-mtp-i-mini + - model: qwen3.6-35b-a3b-apex-mtp-i-nano url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF @@ -3728,6 +3786,15 @@ - model: gemma-4-26b-a4b-it-qat - model: gemma-4-26b-a4b-it-qat-q4_0 - model: gemma-4-26b-a4b-it-apex + - model: gemma-4-26b-a4b-it-apex-i-quality + - model: gemma-4-26b-a4b-it-apex-i-balanced + - model: gemma-4-26b-a4b-it-apex-i-compact + - model: gemma-4-26b-a4b-it-apex-i-mini + - model: gemma-4-26b-a4b-it-apex-i-nano + - model: gemma-4-26b-a4b-it-ud-q4-k-m + - model: gemma-4-26b-a4b-it-ud-q5-k-m + - model: gemma-4-26b-a4b-it-ud-q6-k + - model: gemma-4-26b-a4b-it-q8-0 url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/google/gemma-4-26B-A4B-it @@ -4272,6 +4339,14 @@ sha256: a1cd5c1625b44dd0facaec998020e9b36cb78c2225eaee701e73bf2e5b051ce2 uri: https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/main/mmproj-F32.gguf - name: qwen3.5-397b-a17b + variants: + - model: qwen3.5-397b-a17b-apex-quality + - model: qwen3.5-397b-a17b-apex-balanced + - model: qwen3.5-397b-a17b-apex-compact + - model: qwen3.5-397b-a17b-ud-q4-k-m + - model: qwen3.5-397b-a17b-ud-q5-k-m + - model: qwen3.5-397b-a17b-ud-q6-k + - model: qwen3.5-397b-a17b-q8-0 url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF @@ -4373,6 +4448,12 @@ sha256: cb04ce8bd243483434f3e05a51a3821258cac74187e409547742a729452b0756 uri: https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/resolve/main/mmproj-F32.gguf - name: qwen3.5-122b-a10b + variants: + - model: qwen3.5-122b-a10b-apex-i-quality + - model: qwen3.5-122b-a10b-apex-i-balanced + - model: qwen3.5-122b-a10b-apex-i-compact + - model: qwen3.5-122b-a10b-apex-i-mini + - model: qwen3.5-122b-a10b-q8-0 url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF @@ -7165,6 +7246,12 @@ sha256: f25e9612e985adf01869f412f997a7aaace65e1ee0c97d4975070febdcbbb978 uri: https://huggingface.co/mradermacher/mox-small-1-i1-GGUF/resolve/main/mox-small-1.i1-Q4_K_M.gguf - name: glm-4.7-flash + variants: + - model: glm-4.7-flash-apex-i-quality + - model: glm-4.7-flash-apex-i-balanced + - model: glm-4.7-flash-apex-i-compact + - model: glm-4.7-flash-apex-i-mini + - model: glm-4.7-flash-q8-0 url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF @@ -40195,3 +40282,5953 @@ - filename: mimo-tokenizer-q4_k.gguf uri: huggingface://cstr/mimo-tokenizer-GGUF/mimo-tokenizer-q4_k.gguf sha256: 3f3a903b10294ead4ef6a4afec035639fd2113b1d307d42f649a97cc85670e3f +- name: agents-a1-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Agents-A1-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Agents-A1-APEX-GGUF/Agents-A1-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Agents-A1-APEX-GGUF/Agents-A1-APEX-I-Quality.gguf + sha256: c1bf8c3dcf08060f5d525d2026f6905894f245188593e8a53e2597590d30e52e + uri: https://huggingface.co/mudler/Agents-A1-APEX-GGUF/resolve/main/Agents-A1-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Agents-A1-APEX-GGUF/mmproj.gguf + sha256: c8772fe61cfceefd1bc42e3b06f53bb2e751abd15d1450d58ba0c113d3782396 + uri: https://huggingface.co/mudler/Agents-A1-APEX-GGUF/resolve/main/mmproj.gguf +- name: agents-a1-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Agents-A1-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Agents-A1-APEX-GGUF/Agents-A1-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Agents-A1-APEX-GGUF/Agents-A1-APEX-I-Balanced.gguf + sha256: 4ba81259e1b717e80a3957e59ffc91a2f84f5fcf4e44766acbc7ebec19cb265c + uri: https://huggingface.co/mudler/Agents-A1-APEX-GGUF/resolve/main/Agents-A1-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Agents-A1-APEX-GGUF/mmproj.gguf + sha256: c8772fe61cfceefd1bc42e3b06f53bb2e751abd15d1450d58ba0c113d3782396 + uri: https://huggingface.co/mudler/Agents-A1-APEX-GGUF/resolve/main/mmproj.gguf +- name: agents-a1-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Agents-A1-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Agents-A1-APEX-GGUF/Agents-A1-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Agents-A1-APEX-GGUF/Agents-A1-APEX-I-Compact.gguf + sha256: dbdd972647ae4c16a0c018bbf49d4b9f59e08de5f1d8c5f4cabaec275ad9e7cf + uri: https://huggingface.co/mudler/Agents-A1-APEX-GGUF/resolve/main/Agents-A1-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Agents-A1-APEX-GGUF/mmproj.gguf + sha256: c8772fe61cfceefd1bc42e3b06f53bb2e751abd15d1450d58ba0c113d3782396 + uri: https://huggingface.co/mudler/Agents-A1-APEX-GGUF/resolve/main/mmproj.gguf +- name: agents-a1-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Agents-A1-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Agents-A1-APEX-GGUF/Agents-A1-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Agents-A1-APEX-GGUF/Agents-A1-APEX-I-Mini.gguf + sha256: 17ed2881c7ccaf855eeecfb9ff39cf85ade638a0a15e2e4367387ccd8657260f + uri: https://huggingface.co/mudler/Agents-A1-APEX-GGUF/resolve/main/Agents-A1-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Agents-A1-APEX-GGUF/mmproj.gguf + sha256: c8772fe61cfceefd1bc42e3b06f53bb2e751abd15d1450d58ba0c113d3782396 + uri: https://huggingface.co/mudler/Agents-A1-APEX-GGUF/resolve/main/mmproj.gguf +- name: carnice-moe-35b-a3b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/Carnice-MoE-35B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/Carnice-MoE-35B-A3B-APEX-I-Quality.gguf + sha256: 5e4d97368f638edf06fd42014994f8a8e40e73ea42d960f00b602f0aa01a0e78 + uri: https://huggingface.co/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/resolve/main/Carnice-MoE-35B-A3B-APEX-I-Quality.gguf +- name: carnice-moe-35b-a3b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/Carnice-MoE-35B-A3B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/Carnice-MoE-35B-A3B-APEX-I-Balanced.gguf + sha256: 461f796389ceddcf52be0efa99a13bbda6bb2cf75bf63b99516d69eaab28a2b2 + uri: https://huggingface.co/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/resolve/main/Carnice-MoE-35B-A3B-APEX-I-Balanced.gguf +- name: carnice-moe-35b-a3b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/Carnice-MoE-35B-A3B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/Carnice-MoE-35B-A3B-APEX-I-Compact.gguf + sha256: db5b0b1bb19d2606eeba66ca5b6d3c9993b74d037e50f9f1384a441c031b3796 + uri: https://huggingface.co/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/resolve/main/Carnice-MoE-35B-A3B-APEX-I-Compact.gguf +- name: carnice-moe-35b-a3b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/Carnice-MoE-35B-A3B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/Carnice-MoE-35B-A3B-APEX-I-Mini.gguf + sha256: f94e9220e341c282f86ba82bb2cf976d83eaad4c4e155334aa696214894548eb + uri: https://huggingface.co/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/resolve/main/Carnice-MoE-35B-A3B-APEX-I-Mini.gguf +- name: carnice-qwen3.6-moe-35b-a3b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Quality.gguf + sha256: f1fef05db8f1fd8d6b346320af393e64ab91c837ec621c09f5bcf99ece837876 + uri: https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/resolve/main/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Quality.gguf +- name: carnice-qwen3.6-moe-35b-a3b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Balanced.gguf + sha256: 5f130f37310800fd8c0237aca18e68ebdeb1fe456fd87d21e02161302811798a + uri: https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/resolve/main/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Balanced.gguf +- name: carnice-qwen3.6-moe-35b-a3b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Compact.gguf + sha256: 49e4400f8b6c87463545f218d1e448a3a16bf29f87a6c8023ba951b6b93689be + uri: https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/resolve/main/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Compact.gguf +- name: carnice-qwen3.6-moe-35b-a3b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Mini.gguf + sha256: 182ec450fce07a4074e9499cbe3dcb4ab642c7142ec4eba1fc988c7b1a4af162 + uri: https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/resolve/main/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Mini.gguf +- name: carnice-qwen3.6-moe-35b-a3b-apex-mtp-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Quality.gguf + sha256: a605afc0048aceb6c96b86c09e9958b221b20ba33544a5dfaab5869e0b9327da + uri: https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/resolve/main/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Quality.gguf +- name: carnice-qwen3.6-moe-35b-a3b-apex-mtp-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Balanced.gguf + sha256: 9903cde3419584f5bf0e0563337108ba811f258de7c445675def4134904a29d1 + uri: https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/resolve/main/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Balanced.gguf +- name: carnice-qwen3.6-moe-35b-a3b-apex-mtp-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Compact.gguf + sha256: 8c06bb09410cd0e54b52cec4a1d4f020372f402ac43db3d2a8345efbeb36587b + uri: https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/resolve/main/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Compact.gguf +- name: carnice-qwen3.6-moe-35b-a3b-apex-mtp-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Mini.gguf + sha256: e93499610575d1a8dea1836a41823eb94ecf731428191dbbd41090fb687e3aba + uri: https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/resolve/main/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Mini.gguf +- name: carnice-qwen3.6-moe-35b-a3b-apex-mtp-i-nano + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Nano.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Nano.gguf + sha256: dac5480f2dd08e590e4827428412b7dd4fe9c2cceaade263109b86c5b23d5f58 + uri: https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-GGUF/resolve/main/Carnice-Qwen3.6-MoE-35B-A3B-APEX-MTP-I-Nano.gguf +- name: darwin-36b-opus-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Quality.gguf + sha256: aee6e3629b68193680444f5cec3c288348c4b46047af3e5f9ff7d8dd3633df80 + uri: https://huggingface.co/mudler/Darwin-36B-Opus-APEX-GGUF/resolve/main/Darwin-36B-Opus-APEX-I-Quality.gguf +- name: darwin-36b-opus-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Balanced.gguf + sha256: 45436dfc9ed77b0ecc045ab1544f493677496d1a0d6360ed0cf225db7370d02c + uri: https://huggingface.co/mudler/Darwin-36B-Opus-APEX-GGUF/resolve/main/Darwin-36B-Opus-APEX-I-Balanced.gguf +- name: darwin-36b-opus-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Compact.gguf + sha256: 7c5fbc924cc3bcceac40ec42816dd368bf49098707ab0a5f53217add089126c1 + uri: https://huggingface.co/mudler/Darwin-36B-Opus-APEX-GGUF/resolve/main/Darwin-36B-Opus-APEX-I-Compact.gguf +- name: darwin-36b-opus-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Mini.gguf + sha256: efd53dc30617830e0393c22b4c937f1ba134c76519c4f5ed4ba19f61f1c3a87b + uri: https://huggingface.co/mudler/Darwin-36B-Opus-APEX-GGUF/resolve/main/Darwin-36B-Opus-APEX-I-Mini.gguf +- name: darwin-36b-opus-apex-i-nano + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Nano.gguf + files: + - filename: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Nano.gguf + sha256: 0d091db1d46c35e5ca4fa82271b2918809ada4a722611be6ddd92d0646dc921e + uri: https://huggingface.co/mudler/Darwin-36B-Opus-APEX-GGUF/resolve/main/Darwin-36B-Opus-APEX-I-Nano.gguf +- name: glm-4.7-flash-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/GLM-4.7-Flash-APEX-GGUF/GLM-4.7-Flash-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/GLM-4.7-Flash-APEX-GGUF/GLM-4.7-Flash-APEX-I-Quality.gguf + sha256: ebf4d1c46f6ab9218872a2cf2ed14f387ff83cead3da43791cdb6f0737145d75 + uri: https://huggingface.co/mudler/GLM-4.7-Flash-APEX-GGUF/resolve/main/GLM-4.7-Flash-APEX-I-Quality.gguf +- name: glm-4.7-flash-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/GLM-4.7-Flash-APEX-GGUF/GLM-4.7-Flash-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/GLM-4.7-Flash-APEX-GGUF/GLM-4.7-Flash-APEX-I-Balanced.gguf + sha256: 8420668e5cf3c9d8314b6a3f8c1803279916f60882fe9d4ad75d2c2bfa2f08c2 + uri: https://huggingface.co/mudler/GLM-4.7-Flash-APEX-GGUF/resolve/main/GLM-4.7-Flash-APEX-I-Balanced.gguf +- name: glm-4.7-flash-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/GLM-4.7-Flash-APEX-GGUF/GLM-4.7-Flash-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/GLM-4.7-Flash-APEX-GGUF/GLM-4.7-Flash-APEX-I-Compact.gguf + sha256: b9b04d8867de176d39acf73309f2ad90612ea27c86eaf8b15e82038432d42287 + uri: https://huggingface.co/mudler/GLM-4.7-Flash-APEX-GGUF/resolve/main/GLM-4.7-Flash-APEX-I-Compact.gguf +- name: glm-4.7-flash-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/GLM-4.7-Flash-APEX-GGUF/GLM-4.7-Flash-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/GLM-4.7-Flash-APEX-GGUF/GLM-4.7-Flash-APEX-I-Mini.gguf + sha256: 30b9098d2f1395689a307375dbb63a54aad43de425adc07d6d932ba0d5280944 + uri: https://huggingface.co/mudler/GLM-4.7-Flash-APEX-GGUF/resolve/main/GLM-4.7-Flash-APEX-I-Mini.gguf +- name: glm-4.7-flash-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-Q8_0.gguf + files: + - filename: llama-cpp/models/unsloth/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-Q8_0.gguf + sha256: 8c0922b03326932a40315ce92bbee01cc836ad18664268019143c898bdd7dd0b + uri: https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF/resolve/main/GLM-4.7-Flash-Q8_0.gguf +- name: gemopus-4-26b-a4b-it-preview-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/gemopus-4-26B-A4B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/gemopus-4-26B-A4B-APEX-I-Quality.gguf + sha256: 839f2a4df38758ad7db1bdf3e1bd06398429b119cd0fe2cc3dde1f5d133c766c + uri: https://huggingface.co/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/resolve/main/gemopus-4-26B-A4B-APEX-I-Quality.gguf +- name: gemopus-4-26b-a4b-it-preview-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/gemopus-4-26B-A4B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/gemopus-4-26B-A4B-APEX-I-Balanced.gguf + sha256: e00e485855be43ee6133814fe696a085074bc7af4332edec75c25290137453ca + uri: https://huggingface.co/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/resolve/main/gemopus-4-26B-A4B-APEX-I-Balanced.gguf +- name: gemopus-4-26b-a4b-it-preview-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/gemopus-4-26B-A4B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/gemopus-4-26B-A4B-APEX-I-Compact.gguf + sha256: bd2cb748f732c629fa6ebb05f44f54b09043efc7eb4842641da275efa4f3c20b + uri: https://huggingface.co/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/resolve/main/gemopus-4-26B-A4B-APEX-I-Compact.gguf +- name: gemopus-4-26b-a4b-it-preview-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/gemopus-4-26B-A4B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/gemopus-4-26B-A4B-APEX-I-Mini.gguf + sha256: 96b7acfa10749a0b79b29452d367c720c88d4c2af9765982a9ab74577bf1fa8e + uri: https://huggingface.co/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/resolve/main/gemopus-4-26B-A4B-APEX-I-Mini.gguf +- name: holo3-35b-a3b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Holo3-35B-A3B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Holo3-35B-A3B-APEX-GGUF/Holo3-35B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Holo3-35B-A3B-APEX-GGUF/Holo3-35B-A3B-APEX-I-Quality.gguf + sha256: 57ce61073d6f11798c8644bc04fbbd31397e9c2647aa911202ead521bcef8f93 + uri: https://huggingface.co/mudler/Holo3-35B-A3B-APEX-GGUF/resolve/main/Holo3-35B-A3B-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Holo3-35B-A3B-APEX-GGUF/mmproj.gguf + sha256: cd734ca69ed59fd46cf010407f2ddba387db4e5696f21b67c8779f26264b9020 + uri: https://huggingface.co/mudler/Holo3-35B-A3B-APEX-GGUF/resolve/main/mmproj.gguf +- name: holo3-35b-a3b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Holo3-35B-A3B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Holo3-35B-A3B-APEX-GGUF/Holo3-35B-A3B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Holo3-35B-A3B-APEX-GGUF/Holo3-35B-A3B-APEX-I-Balanced.gguf + sha256: 47133973387e56b1869113721aeb457358c5478363b542a252cadbe2c6b24263 + uri: https://huggingface.co/mudler/Holo3-35B-A3B-APEX-GGUF/resolve/main/Holo3-35B-A3B-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Holo3-35B-A3B-APEX-GGUF/mmproj.gguf + sha256: cd734ca69ed59fd46cf010407f2ddba387db4e5696f21b67c8779f26264b9020 + uri: https://huggingface.co/mudler/Holo3-35B-A3B-APEX-GGUF/resolve/main/mmproj.gguf +- name: holo3-35b-a3b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Holo3-35B-A3B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Holo3-35B-A3B-APEX-GGUF/Holo3-35B-A3B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Holo3-35B-A3B-APEX-GGUF/Holo3-35B-A3B-APEX-I-Compact.gguf + sha256: 66fbc642f1d12f0464ab2175bae240efda60dd00dff3aefab08d8861cbaf04ec + uri: https://huggingface.co/mudler/Holo3-35B-A3B-APEX-GGUF/resolve/main/Holo3-35B-A3B-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Holo3-35B-A3B-APEX-GGUF/mmproj.gguf + sha256: cd734ca69ed59fd46cf010407f2ddba387db4e5696f21b67c8779f26264b9020 + uri: https://huggingface.co/mudler/Holo3-35B-A3B-APEX-GGUF/resolve/main/mmproj.gguf +- name: holo3-35b-a3b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Holo3-35B-A3B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Holo3-35B-A3B-APEX-GGUF/Holo3-35B-A3B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Holo3-35B-A3B-APEX-GGUF/Holo3-35B-A3B-APEX-I-Mini.gguf + sha256: 1427ff13f9f6c6278c0715ee8c2faee276eed2fe00634460e1d9bfc8ceb40e5e + uri: https://huggingface.co/mudler/Holo3-35B-A3B-APEX-GGUF/resolve/main/Holo3-35B-A3B-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Holo3-35B-A3B-APEX-GGUF/mmproj.gguf + sha256: cd734ca69ed59fd46cf010407f2ddba387db4e5696f21b67c8779f26264b9020 + uri: https://huggingface.co/mudler/Holo3-35B-A3B-APEX-GGUF/resolve/main/mmproj.gguf +- name: huihui3.5-67b-a3b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Huihui3.5-67B-A3B-APEX-GGUF/Huihui3.5-67B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Huihui3.5-67B-A3B-APEX-GGUF/Huihui3.5-67B-A3B-APEX-I-Quality.gguf + sha256: 48755ebca7650a944ceb520da00ff9e77c97e52ec78b83ac9c428deba7fae6ed + uri: https://huggingface.co/mudler/Huihui3.5-67B-A3B-APEX-GGUF/resolve/main/Huihui3.5-67B-A3B-APEX-I-Quality.gguf +- name: huihui3.5-67b-a3b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Huihui3.5-67B-A3B-APEX-GGUF/Huihui3.5-67B-A3B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Huihui3.5-67B-A3B-APEX-GGUF/Huihui3.5-67B-A3B-APEX-I-Balanced.gguf + sha256: 0dc15029db1e21d69853f7335941df3d784c2103a70933bb726437e8b9750379 + uri: https://huggingface.co/mudler/Huihui3.5-67B-A3B-APEX-GGUF/resolve/main/Huihui3.5-67B-A3B-APEX-I-Balanced.gguf +- name: huihui3.5-67b-a3b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Huihui3.5-67B-A3B-APEX-GGUF/Huihui3.5-67B-A3B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Huihui3.5-67B-A3B-APEX-GGUF/Huihui3.5-67B-A3B-APEX-I-Compact.gguf + sha256: 8cc1b88caca2b2fc68a5f2b5ae55a05ef14c3b18b01115a81db4c0a17eb7466b + uri: https://huggingface.co/mudler/Huihui3.5-67B-A3B-APEX-GGUF/resolve/main/Huihui3.5-67B-A3B-APEX-I-Compact.gguf +- name: huihui3.5-67b-a3b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Huihui3.5-67B-A3B-APEX-GGUF/Huihui3.5-67B-A3B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Huihui3.5-67B-A3B-APEX-GGUF/Huihui3.5-67B-A3B-APEX-I-Mini.gguf + sha256: 2c6f91d6bc6a33469f17a78778d4ae7fed73730301dfafac7dc268ba31bb9c4f + uri: https://huggingface.co/mudler/Huihui3.5-67B-A3B-APEX-GGUF/resolve/main/Huihui3.5-67B-A3B-APEX-I-Mini.gguf +- name: lfm2-24b-a2b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/LFM2-24B-A2B-APEX-GGUF/LFM2-24B-A2B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/LFM2-24B-A2B-APEX-GGUF/LFM2-24B-A2B-APEX-I-Quality.gguf + sha256: 0313aa47547d302ffd88c538d381145888b5b2bd739cf5058c8cdda81268992d + uri: https://huggingface.co/mudler/LFM2-24B-A2B-APEX-GGUF/resolve/main/LFM2-24B-A2B-APEX-I-Quality.gguf +- name: lfm2-24b-a2b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/LFM2-24B-A2B-APEX-GGUF/LFM2-24B-A2B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/LFM2-24B-A2B-APEX-GGUF/LFM2-24B-A2B-APEX-I-Balanced.gguf + sha256: f9599d24f0dd571847c9de06ba8b00206a3b77edb39d27c6cf156283d106a727 + uri: https://huggingface.co/mudler/LFM2-24B-A2B-APEX-GGUF/resolve/main/LFM2-24B-A2B-APEX-I-Balanced.gguf +- name: lfm2-24b-a2b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/LFM2-24B-A2B-APEX-GGUF/LFM2-24B-A2B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/LFM2-24B-A2B-APEX-GGUF/LFM2-24B-A2B-APEX-I-Compact.gguf + sha256: ff25d3481126b5467f38471d1b2adc75277beac4e869bebefbd0b570c11168a9 + uri: https://huggingface.co/mudler/LFM2-24B-A2B-APEX-GGUF/resolve/main/LFM2-24B-A2B-APEX-I-Compact.gguf +- name: lfm2-24b-a2b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/LFM2-24B-A2B-APEX-GGUF/LFM2-24B-A2B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/LFM2-24B-A2B-APEX-GGUF/LFM2-24B-A2B-APEX-I-Mini.gguf + sha256: cb7bb851a0c8b6a7e2620d833c307bdbf9090a28f6aa1bc4cd2e94b8f781404b + uri: https://huggingface.co/mudler/LFM2-24B-A2B-APEX-GGUF/resolve/main/LFM2-24B-A2B-APEX-I-Mini.gguf +- name: lfm2.5-8b-a1b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/LFM2.5-8B-A1B-APEX-GGUF/LFM2.5-8B-A1B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/LFM2.5-8B-A1B-APEX-GGUF/LFM2.5-8B-A1B-APEX-I-Quality.gguf + sha256: be32a7bb170be66d644275f0884f23e98562143a76ac5a08d971e6f0914a9693 + uri: https://huggingface.co/mudler/LFM2.5-8B-A1B-APEX-GGUF/resolve/main/LFM2.5-8B-A1B-APEX-I-Quality.gguf +- name: lfm2.5-8b-a1b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/LFM2.5-8B-A1B-APEX-GGUF/LFM2.5-8B-A1B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/LFM2.5-8B-A1B-APEX-GGUF/LFM2.5-8B-A1B-APEX-I-Balanced.gguf + sha256: c6a7021aeee1af784c755d923e53e544a563bc9f530f483b8cf5480d033aa06d + uri: https://huggingface.co/mudler/LFM2.5-8B-A1B-APEX-GGUF/resolve/main/LFM2.5-8B-A1B-APEX-I-Balanced.gguf +- name: lfm2.5-8b-a1b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/LFM2.5-8B-A1B-APEX-GGUF/LFM2.5-8B-A1B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/LFM2.5-8B-A1B-APEX-GGUF/LFM2.5-8B-A1B-APEX-I-Compact.gguf + sha256: a0bd7d37a474c780306d36cd6925889d8e984a3800c44987da7207a091c6e8c0 + uri: https://huggingface.co/mudler/LFM2.5-8B-A1B-APEX-GGUF/resolve/main/LFM2.5-8B-A1B-APEX-I-Compact.gguf +- name: lfm2.5-8b-a1b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/LFM2.5-8B-A1B-APEX-GGUF/LFM2.5-8B-A1B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/LFM2.5-8B-A1B-APEX-GGUF/LFM2.5-8B-A1B-APEX-I-Mini.gguf + sha256: 2d23a76a36c15afa0c560e37664abc3e6ce065c60e160a39dffa5621d13f298d + uri: https://huggingface.co/mudler/LFM2.5-8B-A1B-APEX-GGUF/resolve/main/LFM2.5-8B-A1B-APEX-I-Mini.gguf +- name: lfm2.5-8b-a1b-ud-q4-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-UD-Q4_K_M.gguf + files: + - filename: llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-UD-Q4_K_M.gguf + sha256: e2c8350d5e6e7c633b2ba4e20805a2c86bc46884cd285e8f3fe36ad917b772c7 + uri: https://huggingface.co/unsloth/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-UD-Q4_K_M.gguf +- name: lfm2.5-8b-a1b-ud-q5-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-UD-Q5_K_M.gguf + files: + - filename: llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-UD-Q5_K_M.gguf + sha256: 90e0b86095c6485fd5a9b7fa9822a02ebb103b6846f3eb3648643cea946fe9cf + uri: https://huggingface.co/unsloth/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-UD-Q5_K_M.gguf +- name: lfm2.5-8b-a1b-ud-q6-k + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-UD-Q6_K.gguf + files: + - filename: llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-UD-Q6_K.gguf + sha256: f5c60637503fcbb479275b1bca9358a315ca5916556f1eeed40013da45a914c3 + uri: https://huggingface.co/unsloth/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-UD-Q6_K.gguf +- name: lfm2.5-8b-a1b-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf + files: + - filename: llama-cpp/models/unsloth/LFM2.5-8B-A1B-GGUF/LFM2.5-8B-A1B-Q8_0.gguf + sha256: ec11666b6129f0b4fe893760b66797f22e1c478a561b40e365f2b6930729b8d2 + uri: https://huggingface.co/unsloth/LFM2.5-8B-A1B-GGUF/resolve/main/LFM2.5-8B-A1B-Q8_0.gguf +- name: minimax-m2.5-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/MiniMax-M2.5-APEX-GGUF/MiniMax-M2.5-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/MiniMax-M2.5-APEX-GGUF/MiniMax-M2.5-APEX-I-Quality.gguf + sha256: 54b4e8c0185aa6b58857dd7d64173e2a8760f6e4a144a45c906211d73a6f7eb8 + uri: https://huggingface.co/mudler/MiniMax-M2.5-APEX-GGUF/resolve/main/MiniMax-M2.5-APEX-I-Quality.gguf +- name: minimax-m2.5-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/MiniMax-M2.5-APEX-GGUF/MiniMax-M2.5-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/MiniMax-M2.5-APEX-GGUF/MiniMax-M2.5-APEX-I-Balanced.gguf + sha256: 04c42f283f860e8c4496ed0ea05cc43b72752be0a9872e6c54eadc1d7e4d6a8b + uri: https://huggingface.co/mudler/MiniMax-M2.5-APEX-GGUF/resolve/main/MiniMax-M2.5-APEX-I-Balanced.gguf +- name: minimax-m2.5-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/MiniMax-M2.5-APEX-GGUF/MiniMax-M2.5-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/MiniMax-M2.5-APEX-GGUF/MiniMax-M2.5-APEX-I-Compact.gguf + sha256: 22bc22cfc26dba0575a0529a27f20e6d9957efab6248a2acb45f1ea10a6dcfaf + uri: https://huggingface.co/mudler/MiniMax-M2.5-APEX-GGUF/resolve/main/MiniMax-M2.5-APEX-I-Compact.gguf +- name: minimax-m2.5-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/MiniMax-M2.5-APEX-GGUF/MiniMax-M2.5-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/MiniMax-M2.5-APEX-GGUF/MiniMax-M2.5-APEX-I-Mini.gguf + sha256: 335a458b0db220433a6db02c6fc4723f91cf01369285257a047c899508ffa341 + uri: https://huggingface.co/mudler/MiniMax-M2.5-APEX-GGUF/resolve/main/MiniMax-M2.5-APEX-I-Mini.gguf +- name: minimax-m2.5-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/MiniMax-M2.5-GGUF/Q8_0/MiniMax-M2.5-Q8_0-00001-of-00006.gguf + files: + - filename: llama-cpp/models/unsloth/MiniMax-M2.5-GGUF/Q8_0/MiniMax-M2.5-Q8_0-00001-of-00006.gguf + sha256: f8d8e8905bc23e108d77e4bd66229060e610314305ccc0dac657ad5c168ff129 + uri: https://huggingface.co/unsloth/MiniMax-M2.5-GGUF/resolve/main/Q8_0/MiniMax-M2.5-Q8_0-00001-of-00006.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.5-GGUF/Q8_0/MiniMax-M2.5-Q8_0-00002-of-00006.gguf + sha256: e47f873a6fba4aa329b3a5cac36b1ba7ebfca1f13d208c4366b65489ebba164f + uri: https://huggingface.co/unsloth/MiniMax-M2.5-GGUF/resolve/main/Q8_0/MiniMax-M2.5-Q8_0-00002-of-00006.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.5-GGUF/Q8_0/MiniMax-M2.5-Q8_0-00003-of-00006.gguf + sha256: 75069c50341a3a706b9b21a622d2f7ff579453f454129557ad50c6421991f197 + uri: https://huggingface.co/unsloth/MiniMax-M2.5-GGUF/resolve/main/Q8_0/MiniMax-M2.5-Q8_0-00003-of-00006.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.5-GGUF/Q8_0/MiniMax-M2.5-Q8_0-00004-of-00006.gguf + sha256: d7f75467a5bb0a5b4a937cdc272e6447ffd20b1e2f93ecd878b33a6a872dc128 + uri: https://huggingface.co/unsloth/MiniMax-M2.5-GGUF/resolve/main/Q8_0/MiniMax-M2.5-Q8_0-00004-of-00006.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.5-GGUF/Q8_0/MiniMax-M2.5-Q8_0-00005-of-00006.gguf + sha256: 9a791588c62dae1da8e8a719efc25553a7170f7766ec100da8cbacb6049a7e4a + uri: https://huggingface.co/unsloth/MiniMax-M2.5-GGUF/resolve/main/Q8_0/MiniMax-M2.5-Q8_0-00005-of-00006.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.5-GGUF/Q8_0/MiniMax-M2.5-Q8_0-00006-of-00006.gguf + sha256: cf16ddb64dd8f9a27678786af45ab55e17016bddd9f684106b85b60c8f1727dc + uri: https://huggingface.co/unsloth/MiniMax-M2.5-GGUF/resolve/main/Q8_0/MiniMax-M2.5-Q8_0-00006-of-00006.gguf +- name: minimax-m2.7-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Quality.gguf + sha256: 1270f5ce024ff1444df43294c86124a7053cd7f4681b4ac7719f2c3a49834405 + uri: https://huggingface.co/mudler/MiniMax-M2.7-APEX-GGUF/resolve/main/MiniMax-M2.7-APEX-I-Quality.gguf +- name: minimax-m2.7-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Balanced.gguf + sha256: bdc9154c2679f710e5ad3af43dd1c33be95c11650c0322ea612e69fd6c6d5a25 + uri: https://huggingface.co/mudler/MiniMax-M2.7-APEX-GGUF/resolve/main/MiniMax-M2.7-APEX-I-Balanced.gguf +- name: minimax-m2.7-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Compact.gguf + sha256: 557ba4fb822d347a8b81f37f5d31cf9d301fd8a668efb58bded63fa7032f910a + uri: https://huggingface.co/mudler/MiniMax-M2.7-APEX-GGUF/resolve/main/MiniMax-M2.7-APEX-I-Compact.gguf +- name: minimax-m2.7-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Mini.gguf + sha256: 9cc9b4a2d66d0d527fc75bec3e2bc84f48b38a061f8215d4b70f3aa796fed93a + uri: https://huggingface.co/mudler/MiniMax-M2.7-APEX-GGUF/resolve/main/MiniMax-M2.7-APEX-I-Mini.gguf +- name: minimax-m2.7-apex-i-nano + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Nano.gguf + files: + - filename: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Nano.gguf + sha256: 42e7b072a54e6a328aff61598314332c84b05b5d675c039312be59fdc06ef01c + uri: https://huggingface.co/mudler/MiniMax-M2.7-APEX-GGUF/resolve/main/MiniMax-M2.7-APEX-I-Nano.gguf +- name: minimax-m2.7-ud-q4-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q4_K_M/MiniMax-M2.7-UD-Q4_K_M-00001-of-00004.gguf + files: + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q4_K_M/MiniMax-M2.7-UD-Q4_K_M-00001-of-00004.gguf + sha256: 92986e39a0c0b5f12c2c9b6a811dad59e3317caaf1b7ad5c7f0d7d12abc4a6e8 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q4_K_M/MiniMax-M2.7-UD-Q4_K_M-00001-of-00004.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q4_K_M/MiniMax-M2.7-UD-Q4_K_M-00002-of-00004.gguf + sha256: 425e6827c5c62ff11d56d99dae144524b0bfb655b51d50bac287e4a581518e46 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q4_K_M/MiniMax-M2.7-UD-Q4_K_M-00002-of-00004.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q4_K_M/MiniMax-M2.7-UD-Q4_K_M-00003-of-00004.gguf + sha256: 27cfd98b49138aae1f30f4681a0a1123f38150b5d054a2f33fd2c8be85ff1fc5 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q4_K_M/MiniMax-M2.7-UD-Q4_K_M-00003-of-00004.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q4_K_M/MiniMax-M2.7-UD-Q4_K_M-00004-of-00004.gguf + sha256: 73882cea9908fb8e84b3dbd5d58c75bdbe187e243aeabc93d2d63fa175322eee + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q4_K_M/MiniMax-M2.7-UD-Q4_K_M-00004-of-00004.gguf +- name: minimax-m2.7-ud-q5-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q5_K_M/MiniMax-M2.7-UD-Q5_K_M-00001-of-00005.gguf + files: + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q5_K_M/MiniMax-M2.7-UD-Q5_K_M-00001-of-00005.gguf + sha256: 1a642e418767ed89a22326bf130259f807b46d4cc94ba72b9c340db7af34222f + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q5_K_M/MiniMax-M2.7-UD-Q5_K_M-00001-of-00005.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q5_K_M/MiniMax-M2.7-UD-Q5_K_M-00002-of-00005.gguf + sha256: 705624d2c1745ee8c5999f4247dfa88047ca145d1e00828cd44e963f1a7249e2 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q5_K_M/MiniMax-M2.7-UD-Q5_K_M-00002-of-00005.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q5_K_M/MiniMax-M2.7-UD-Q5_K_M-00003-of-00005.gguf + sha256: d897449435fa2b54318bd09b1a53f9d4267c6d0786c2a954a8c58c66119fdee6 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q5_K_M/MiniMax-M2.7-UD-Q5_K_M-00003-of-00005.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q5_K_M/MiniMax-M2.7-UD-Q5_K_M-00004-of-00005.gguf + sha256: 494f9073378d88b3f4398397d9e113b0b4248134a83cea4bce42913a6d718677 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q5_K_M/MiniMax-M2.7-UD-Q5_K_M-00004-of-00005.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q5_K_M/MiniMax-M2.7-UD-Q5_K_M-00005-of-00005.gguf + sha256: 7cf686503c65c83a504897d7b0d39a4a38742da63b7d44ab6dcba09dc8793490 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q5_K_M/MiniMax-M2.7-UD-Q5_K_M-00005-of-00005.gguf +- name: minimax-m2.7-ud-q6-k + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q6_K/MiniMax-M2.7-UD-Q6_K-00001-of-00005.gguf + files: + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q6_K/MiniMax-M2.7-UD-Q6_K-00001-of-00005.gguf + sha256: da39783926866dcb87101bebc083ed95d614a21a20741dbd2156720b2294305f + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q6_K/MiniMax-M2.7-UD-Q6_K-00001-of-00005.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q6_K/MiniMax-M2.7-UD-Q6_K-00002-of-00005.gguf + sha256: 58a845e5d484bc1931ed75db9e29c234439c88ea8a17d726e168c2b4e87177dc + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q6_K/MiniMax-M2.7-UD-Q6_K-00002-of-00005.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q6_K/MiniMax-M2.7-UD-Q6_K-00003-of-00005.gguf + sha256: 9be036e595c0bb9e07df2b4c6e71d88f70f468247feb575c54a27fd20bb88b20 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q6_K/MiniMax-M2.7-UD-Q6_K-00003-of-00005.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q6_K/MiniMax-M2.7-UD-Q6_K-00004-of-00005.gguf + sha256: 79740c41bbbddaea62377ad1bb73b086a022ea46f2d004f5832d31211c06a4cb + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q6_K/MiniMax-M2.7-UD-Q6_K-00004-of-00005.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/UD-Q6_K/MiniMax-M2.7-UD-Q6_K-00005-of-00005.gguf + sha256: 7126ca6dc4c937d3ca6c836b54b51d666707ebe78b0e27909b099318866afa7c + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/UD-Q6_K/MiniMax-M2.7-UD-Q6_K-00005-of-00005.gguf +- name: minimax-m2.7-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/Q8_0/MiniMax-M2.7-Q8_0-00001-of-00006.gguf + files: + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/Q8_0/MiniMax-M2.7-Q8_0-00001-of-00006.gguf + sha256: a0991934ab08119f12f00af0ef81cba73e995a07dc5b53f4dd4c6f8dfa86b4e4 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/Q8_0/MiniMax-M2.7-Q8_0-00001-of-00006.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/Q8_0/MiniMax-M2.7-Q8_0-00002-of-00006.gguf + sha256: f5ee2688f9c625917554d9aec273eecd4ef5d270ed391dbbc4a2e02b6617bf4e + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/Q8_0/MiniMax-M2.7-Q8_0-00002-of-00006.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/Q8_0/MiniMax-M2.7-Q8_0-00003-of-00006.gguf + sha256: a3eab2367b732cab02775df7e4a8661f5f52be6ebedd88f3c8243ceeaf121d51 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/Q8_0/MiniMax-M2.7-Q8_0-00003-of-00006.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/Q8_0/MiniMax-M2.7-Q8_0-00004-of-00006.gguf + sha256: c68dc023e46c5ca11afe0d7323c54a8139b46327ee0abc50a1d42431f9a95854 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/Q8_0/MiniMax-M2.7-Q8_0-00004-of-00006.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/Q8_0/MiniMax-M2.7-Q8_0-00005-of-00006.gguf + sha256: 416ef6f47ee60b550d4bb522cb80611a883ffd09010bd1a4707b674cb7810b2a + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/Q8_0/MiniMax-M2.7-Q8_0-00005-of-00006.gguf + - filename: llama-cpp/models/unsloth/MiniMax-M2.7-GGUF/Q8_0/MiniMax-M2.7-Q8_0-00006-of-00006.gguf + sha256: 1a3827ef6b20595bae9e5f19d1f7788e39f1038c8fd2ef63717d6ffb93f6af68 + uri: https://huggingface.co/unsloth/MiniMax-M2.7-GGUF/resolve/main/Q8_0/MiniMax-M2.7-Q8_0-00006-of-00006.gguf +- name: mistral-small-4-119b-2603-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/Mistral-Small-4-119B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/Mistral-Small-4-119B-APEX-I-Quality.gguf + sha256: 2832cb7ef12b934eebd9baec0dfcde3807cd5101237bdfc33f2b265e6283c860 + uri: https://huggingface.co/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/resolve/main/Mistral-Small-4-119B-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/mmproj.gguf + sha256: 2f1f27544c4b4074b28edb8413af92a9111d7166bbc0944fd27db56adc054d0b + uri: https://huggingface.co/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/resolve/main/mmproj.gguf +- name: mistral-small-4-119b-2603-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/Mistral-Small-4-119B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/Mistral-Small-4-119B-APEX-I-Balanced.gguf + sha256: d162ea82330bc1bf18ace7c38d30bdb814f6f0878d22dce8fb84b51d6db21b45 + uri: https://huggingface.co/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/resolve/main/Mistral-Small-4-119B-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/mmproj.gguf + sha256: 2f1f27544c4b4074b28edb8413af92a9111d7166bbc0944fd27db56adc054d0b + uri: https://huggingface.co/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/resolve/main/mmproj.gguf +- name: mistral-small-4-119b-2603-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/Mistral-Small-4-119B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/Mistral-Small-4-119B-APEX-I-Compact.gguf + sha256: e7bc0186e900caf00f58b1b32c2a9665e23aaa9ad27ad48337fec1ad9c4c7d1b + uri: https://huggingface.co/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/resolve/main/Mistral-Small-4-119B-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/mmproj.gguf + sha256: 2f1f27544c4b4074b28edb8413af92a9111d7166bbc0944fd27db56adc054d0b + uri: https://huggingface.co/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/resolve/main/mmproj.gguf +- name: mistral-small-4-119b-2603-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/Mistral-Small-4-119B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/Mistral-Small-4-119B-APEX-I-Mini.gguf + sha256: 7c76c46552b8c3bf054fb3c4cd05d4f3038928ad8e26351f255c12e89714799e + uri: https://huggingface.co/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/resolve/main/Mistral-Small-4-119B-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/mmproj.gguf + sha256: 2f1f27544c4b4074b28edb8413af92a9111d7166bbc0944fd27db56adc054d0b + uri: https://huggingface.co/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/resolve/main/mmproj.gguf +- name: mistral-small-4-119b-2603-ud-q4-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Mistral-Small-4-119B-2603-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q4_K_M/Mistral-Small-4-119B-2603-UD-Q4_K_M-00001-of-00003.gguf + files: + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q4_K_M/Mistral-Small-4-119B-2603-UD-Q4_K_M-00001-of-00003.gguf + sha256: fd3cc46082e4e64eb623eea4611b02583d1de4c20a59411bf1820925d5117dfe + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/UD-Q4_K_M/Mistral-Small-4-119B-2603-UD-Q4_K_M-00001-of-00003.gguf + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q4_K_M/Mistral-Small-4-119B-2603-UD-Q4_K_M-00002-of-00003.gguf + sha256: f357d8dc029824fa4515ae60d7a8b7e108546763614f16c00a1b4ea0cd94145c + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/UD-Q4_K_M/Mistral-Small-4-119B-2603-UD-Q4_K_M-00002-of-00003.gguf + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q4_K_M/Mistral-Small-4-119B-2603-UD-Q4_K_M-00003-of-00003.gguf + sha256: 9dc960d67fb1ef23029d24878b4cf6c63b743ab26cf9137402549591bd770c50 + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/UD-Q4_K_M/Mistral-Small-4-119B-2603-UD-Q4_K_M-00003-of-00003.gguf + - filename: llama-cpp/mmproj/unsloth/Mistral-Small-4-119B-2603-GGUF/mmproj-BF16.gguf + sha256: 2f1f27544c4b4074b28edb8413af92a9111d7166bbc0944fd27db56adc054d0b + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/mmproj-BF16.gguf +- name: mistral-small-4-119b-2603-ud-q5-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Mistral-Small-4-119B-2603-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q5_K_M/Mistral-Small-4-119B-2603-UD-Q5_K_M-00001-of-00003.gguf + files: + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q5_K_M/Mistral-Small-4-119B-2603-UD-Q5_K_M-00001-of-00003.gguf + sha256: 6d3a3d2428d254314a3305628afb4d27141be3597903fa5d61413b69b40e8e30 + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/UD-Q5_K_M/Mistral-Small-4-119B-2603-UD-Q5_K_M-00001-of-00003.gguf + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q5_K_M/Mistral-Small-4-119B-2603-UD-Q5_K_M-00002-of-00003.gguf + sha256: f1340e0d2e7aa979028edffbc344bd3c8e3d4b1f0f8dd267bd4094b676f44142 + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/UD-Q5_K_M/Mistral-Small-4-119B-2603-UD-Q5_K_M-00002-of-00003.gguf + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q5_K_M/Mistral-Small-4-119B-2603-UD-Q5_K_M-00003-of-00003.gguf + sha256: b6181cde0c0dd1e5823ec8abd51e705f98bec50e7be1e01eedb1239e1c9ac434 + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/UD-Q5_K_M/Mistral-Small-4-119B-2603-UD-Q5_K_M-00003-of-00003.gguf + - filename: llama-cpp/mmproj/unsloth/Mistral-Small-4-119B-2603-GGUF/mmproj-BF16.gguf + sha256: 2f1f27544c4b4074b28edb8413af92a9111d7166bbc0944fd27db56adc054d0b + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/mmproj-BF16.gguf +- name: mistral-small-4-119b-2603-ud-q6-k + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Mistral-Small-4-119B-2603-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q6_K/Mistral-Small-4-119B-2603-UD-Q6_K-00001-of-00004.gguf + files: + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q6_K/Mistral-Small-4-119B-2603-UD-Q6_K-00001-of-00004.gguf + sha256: 793d292f8c5235e5aa82780e90bb08702b7152187656842b74299980b2c1ff1f + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/UD-Q6_K/Mistral-Small-4-119B-2603-UD-Q6_K-00001-of-00004.gguf + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q6_K/Mistral-Small-4-119B-2603-UD-Q6_K-00002-of-00004.gguf + sha256: fac090a751b1e422db0f6b6958a0e46844a60af145db0c853988182af445a59e + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/UD-Q6_K/Mistral-Small-4-119B-2603-UD-Q6_K-00002-of-00004.gguf + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q6_K/Mistral-Small-4-119B-2603-UD-Q6_K-00003-of-00004.gguf + sha256: 454312436b12cf122b4ad5ee3ad6857c5118863f83fd2aff5deab350f501dfc3 + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/UD-Q6_K/Mistral-Small-4-119B-2603-UD-Q6_K-00003-of-00004.gguf + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/UD-Q6_K/Mistral-Small-4-119B-2603-UD-Q6_K-00004-of-00004.gguf + sha256: 73a0651f29d23e0de82ac9a9cd8c3ecb3bc838736cb992d10920c97559b9ca95 + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/UD-Q6_K/Mistral-Small-4-119B-2603-UD-Q6_K-00004-of-00004.gguf + - filename: llama-cpp/mmproj/unsloth/Mistral-Small-4-119B-2603-GGUF/mmproj-BF16.gguf + sha256: 2f1f27544c4b4074b28edb8413af92a9111d7166bbc0944fd27db56adc054d0b + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/mmproj-BF16.gguf +- name: mistral-small-4-119b-2603-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Mistral-Small-4-119B-2603-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/Q8_0/Mistral-Small-4-119B-2603-Q8_0-00001-of-00004.gguf + files: + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/Q8_0/Mistral-Small-4-119B-2603-Q8_0-00001-of-00004.gguf + sha256: 204b226d54373fd36d1025565e0f607b002b8a56527751453bad6cac8ca3331e + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/Q8_0/Mistral-Small-4-119B-2603-Q8_0-00001-of-00004.gguf + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/Q8_0/Mistral-Small-4-119B-2603-Q8_0-00002-of-00004.gguf + sha256: 01963f96cfb50f8f3e5fa08eaeb363830ae6e1066095ba46f11894085c7ad5c3 + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/Q8_0/Mistral-Small-4-119B-2603-Q8_0-00002-of-00004.gguf + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/Q8_0/Mistral-Small-4-119B-2603-Q8_0-00003-of-00004.gguf + sha256: 3b99675d4548b400da51a7b3a11404f8e5e87a850cbee69897107f76281b77d5 + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/Q8_0/Mistral-Small-4-119B-2603-Q8_0-00003-of-00004.gguf + - filename: llama-cpp/models/unsloth/Mistral-Small-4-119B-2603-GGUF/Q8_0/Mistral-Small-4-119B-2603-Q8_0-00004-of-00004.gguf + sha256: 421701d8ee946edb30bc355bb5357c2f6fa9c55495a567de6b804af7a6234a85 + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/Q8_0/Mistral-Small-4-119B-2603-Q8_0-00004-of-00004.gguf + - filename: llama-cpp/mmproj/unsloth/Mistral-Small-4-119B-2603-GGUF/mmproj-BF16.gguf + sha256: 2f1f27544c4b4074b28edb8413af92a9111d7166bbc0944fd27db56adc054d0b + uri: https://huggingface.co/unsloth/Mistral-Small-4-119B-2603-GGUF/resolve/main/mmproj-BF16.gguf +- name: nvidia-nemotron-3-super-120b-a12b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/Nemotron-3-Super-120B-A12B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/Nemotron-3-Super-120B-A12B-APEX-I-Quality.gguf + sha256: d6eeef1836655ca3e04ddd851f22feb1dd23edb2599ac50db8326fcdcd21cd14 + uri: https://huggingface.co/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/resolve/main/Nemotron-3-Super-120B-A12B-APEX-I-Quality.gguf +- name: nvidia-nemotron-3-super-120b-a12b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/Nemotron-3-Super-120B-A12B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/Nemotron-3-Super-120B-A12B-APEX-I-Balanced.gguf + sha256: 92b6d4a446c7fb19df85b00560b448c63bf213ba88b2efa870872d04b754b247 + uri: https://huggingface.co/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/resolve/main/Nemotron-3-Super-120B-A12B-APEX-I-Balanced.gguf +- name: nvidia-nemotron-3-super-120b-a12b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/Nemotron-3-Super-120B-A12B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/Nemotron-3-Super-120B-A12B-APEX-I-Compact.gguf + sha256: 4b3da08d42199ccf31ab0f1b6c60f3cd32fe3ecbe81000d87fe5a58aa34119ed + uri: https://huggingface.co/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/resolve/main/Nemotron-3-Super-120B-A12B-APEX-I-Compact.gguf +- name: nvidia-nemotron-3-super-120b-a12b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/Nemotron-3-Super-120B-A12B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/Nemotron-3-Super-120B-A12B-APEX-I-Mini.gguf + sha256: e0f4cf9c14ff81537cae1f6fd5cc7b098726ae3ed5dce913b69ee789cb88d60e + uri: https://huggingface.co/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/resolve/main/Nemotron-3-Super-120B-A12B-APEX-I-Mini.gguf +- name: nvidia-nemotron-3-super-120b-a12b-ud-q4-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_M-00001-of-00003.gguf + files: + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_M-00001-of-00003.gguf + sha256: f22083eb6b15acb52905308ab083e8b0cc38897005cc45e8881abd164580aac2 + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/UD-Q4_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_M-00001-of-00003.gguf + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_M-00002-of-00003.gguf + sha256: 6d65adda88759f1bcea714e4da09dab8ecc42265a57b87f9bf7fe3cb7f53b889 + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/UD-Q4_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_M-00002-of-00003.gguf + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q4_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_M-00003-of-00003.gguf + sha256: 2b126c51396dd31fbb1273fb2b28d0c8502bab5e357a852f67af7a4b6f5ba79c + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/UD-Q4_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_M-00003-of-00003.gguf +- name: nvidia-nemotron-3-super-120b-a12b-ud-q5-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q5_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q5_K_M-00001-of-00004.gguf + files: + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q5_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q5_K_M-00001-of-00004.gguf + sha256: d1d07a4920c4a72e7ca9bb036826cbe330393ea8526604b7e0d3a31bc7ec50d6 + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/UD-Q5_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q5_K_M-00001-of-00004.gguf + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q5_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q5_K_M-00002-of-00004.gguf + sha256: 26f94296c578157ccea3707d94db856f808d9c69b1f1a561fb4bc4d39ebab5b6 + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/UD-Q5_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q5_K_M-00002-of-00004.gguf + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q5_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q5_K_M-00003-of-00004.gguf + sha256: a82c2e832bc4a15dec968823859bfe90a9883a8683a921560e3e73d4f46f2210 + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/UD-Q5_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q5_K_M-00003-of-00004.gguf + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q5_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q5_K_M-00004-of-00004.gguf + sha256: fd984136cc63f5194f959087d6dab31d4b1f2659c43185cc1ea9e709f90ba6d3 + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/UD-Q5_K_M/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q5_K_M-00004-of-00004.gguf +- name: nvidia-nemotron-3-super-120b-a12b-ud-q6-k + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q6_K/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q6_K-00001-of-00004.gguf + files: + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q6_K/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q6_K-00001-of-00004.gguf + sha256: ef6605689bbb052dea4e005edb8b70e25d79168e98e4eb2b1494200975a2a68d + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/UD-Q6_K/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q6_K-00001-of-00004.gguf + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q6_K/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q6_K-00002-of-00004.gguf + sha256: 64d68cfcac0c60719f88f500c14d1ba9f984440e6c07cba39b57bfdb74c18064 + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/UD-Q6_K/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q6_K-00002-of-00004.gguf + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q6_K/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q6_K-00003-of-00004.gguf + sha256: 87f917e7e12a965f57569b4c4f9eaa2f67da543a917373e4ff6620d8229ae8a0 + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/UD-Q6_K/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q6_K-00003-of-00004.gguf + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/UD-Q6_K/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q6_K-00004-of-00004.gguf + sha256: 636e5427391545f6eb2e0df7c5f0efcec1d6aa89c776c6cafe1771f6ef304287 + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/UD-Q6_K/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q6_K-00004-of-00004.gguf +- name: nvidia-nemotron-3-super-120b-a12b-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/Q8_0/NVIDIA-Nemotron-3-Super-120B-A12B-Q8_0-00001-of-00004.gguf + files: + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/Q8_0/NVIDIA-Nemotron-3-Super-120B-A12B-Q8_0-00001-of-00004.gguf + sha256: 7efc3a9c77a65929897e2fed1d1d670d514297508485f80996e77004f5e81332 + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/Q8_0/NVIDIA-Nemotron-3-Super-120B-A12B-Q8_0-00001-of-00004.gguf + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/Q8_0/NVIDIA-Nemotron-3-Super-120B-A12B-Q8_0-00002-of-00004.gguf + sha256: 1dd0a70006703d67ab63a07b49c5ef97cf3081b94549daef6eb792ce225a48b9 + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/Q8_0/NVIDIA-Nemotron-3-Super-120B-A12B-Q8_0-00002-of-00004.gguf + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/Q8_0/NVIDIA-Nemotron-3-Super-120B-A12B-Q8_0-00003-of-00004.gguf + sha256: 31dea2f2172a5ce5456ee5d55f4d4cc2bf18e5ddc6c3e6d512281fe65163deba + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/Q8_0/NVIDIA-Nemotron-3-Super-120B-A12B-Q8_0-00003-of-00004.gguf + - filename: llama-cpp/models/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/Q8_0/NVIDIA-Nemotron-3-Super-120B-A12B-Q8_0-00004-of-00004.gguf + sha256: 4962476f3703a5f9e613b431b866d0344ac361ccae61a0e7e21ed8f8b47b5d3b + uri: https://huggingface.co/unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/resolve/main/Q8_0/NVIDIA-Nemotron-3-Super-120B-A12B-Q8_0-00004-of-00004.gguf +- name: nemotron-3-nano-30b-a3b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/Nemotron-3-Nano-30B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/Nemotron-3-Nano-30B-A3B-APEX-I-Quality.gguf + sha256: 649ccc24aa75282cebf361638f149b056a6a9d022c574aa916d6b01f9296484e + uri: https://huggingface.co/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/resolve/main/Nemotron-3-Nano-30B-A3B-APEX-I-Quality.gguf +- name: nemotron-3-nano-30b-a3b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/Nemotron-3-Nano-30B-A3B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/Nemotron-3-Nano-30B-A3B-APEX-I-Balanced.gguf + sha256: 3b580f27d72a78d0bb9fbfd5e421b6bc89228cb899c07948b4aac38ca595a945 + uri: https://huggingface.co/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/resolve/main/Nemotron-3-Nano-30B-A3B-APEX-I-Balanced.gguf +- name: nemotron-3-nano-30b-a3b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/Nemotron-3-Nano-30B-A3B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/Nemotron-3-Nano-30B-A3B-APEX-I-Compact.gguf + sha256: 1cb1dc86c9bcfd4745fa17d958f4810fe52e781a948da5050183e41fc9f78444 + uri: https://huggingface.co/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/resolve/main/Nemotron-3-Nano-30B-A3B-APEX-I-Compact.gguf +- name: nemotron-3-nano-30b-a3b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/Nemotron-3-Nano-30B-A3B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/Nemotron-3-Nano-30B-A3B-APEX-I-Mini.gguf + sha256: ba320bfaf22ec567c40e70879edc8b450a4a1a45584110984c129a56178fe555 + uri: https://huggingface.co/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/resolve/main/Nemotron-3-Nano-30B-A3B-APEX-I-Mini.gguf +- name: nemotron-3-nano-30b-a3b-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-Q8_0.gguf + files: + - filename: llama-cpp/models/unsloth/Nemotron-3-Nano-30B-A3B-GGUF/Nemotron-3-Nano-30B-A3B-Q8_0.gguf + sha256: 433a9c0d39c6c3434204797b30991c4d147c0f25328690d60862ec39e2a92188 + uri: https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF/resolve/main/Nemotron-3-Nano-30B-A3B-Q8_0.gguf +- name: nemotron-3-nano-omni-30b-a3b-reasoning-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Quality.gguf + sha256: aa1734147522810d52c6c59f269301eed98d34e50438475ab5544d7a6ee1821a + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + sha256: 8e72b13d61ac5972793b94e130e69d93ebbc96bc8e281500c03e762282b1840f + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/mmproj.gguf +- name: nemotron-3-nano-omni-30b-a3b-reasoning-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Balanced.gguf + sha256: 7fda624cb5d43579cf379997f1888399bb3e078c83cf7a421aabfe42a198d297 + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + sha256: 8e72b13d61ac5972793b94e130e69d93ebbc96bc8e281500c03e762282b1840f + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/mmproj.gguf +- name: nemotron-3-nano-omni-30b-a3b-reasoning-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Compact.gguf + sha256: 7f573f83b7c7886e86fbd2b66ad37948271cb755fb6381dae52d923880b28593 + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + sha256: 8e72b13d61ac5972793b94e130e69d93ebbc96bc8e281500c03e762282b1840f + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/mmproj.gguf +- name: nemotron-3-nano-omni-30b-a3b-reasoning-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Mini.gguf + sha256: 362c47fa692be146f54ad87d6797afe3845a41f29dee5deb96f6eb1307c53949 + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + sha256: 8e72b13d61ac5972793b94e130e69d93ebbc96bc8e281500c03e762282b1840f + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/mmproj.gguf +- name: nemotron-3-nano-omni-30b-a3b-reasoning-apex-i-nano + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Nano.gguf + files: + - filename: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Nano.gguf + sha256: 0991fd06fac58525ca55920244cf7752042fa4a95e79113c495a635865e2428f + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Nano.gguf + - filename: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + sha256: 8e72b13d61ac5972793b94e130e69d93ebbc96bc8e281500c03e762282b1840f + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/mmproj.gguf +- name: nex-n2-mini-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Nex-N2-mini-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nex-N2-mini-APEX-GGUF/Nex-N2-mini-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Nex-N2-mini-APEX-GGUF/Nex-N2-mini-APEX-I-Quality.gguf + sha256: 7f8895584cb494f7331cd81e67e898ab68100b94aed9725cf13653b1df9db8b1 + uri: https://huggingface.co/mudler/Nex-N2-mini-APEX-GGUF/resolve/main/Nex-N2-mini-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Nex-N2-mini-APEX-GGUF/mmproj.gguf + sha256: 571ad1e826caa2eb5ff37fdf3ae30b5072edc694d8ebd25897166660ed981469 + uri: https://huggingface.co/mudler/Nex-N2-mini-APEX-GGUF/resolve/main/mmproj.gguf +- name: nex-n2-mini-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Nex-N2-mini-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nex-N2-mini-APEX-GGUF/Nex-N2-mini-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Nex-N2-mini-APEX-GGUF/Nex-N2-mini-APEX-I-Balanced.gguf + sha256: e593a56bb1df6701cd741571fb2c271b022f412e113675e4846d41437b9f0f15 + uri: https://huggingface.co/mudler/Nex-N2-mini-APEX-GGUF/resolve/main/Nex-N2-mini-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Nex-N2-mini-APEX-GGUF/mmproj.gguf + sha256: 571ad1e826caa2eb5ff37fdf3ae30b5072edc694d8ebd25897166660ed981469 + uri: https://huggingface.co/mudler/Nex-N2-mini-APEX-GGUF/resolve/main/mmproj.gguf +- name: nex-n2-mini-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Nex-N2-mini-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nex-N2-mini-APEX-GGUF/Nex-N2-mini-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Nex-N2-mini-APEX-GGUF/Nex-N2-mini-APEX-I-Compact.gguf + sha256: 61ecb5c70070694ee15ec9e7cca79bab44ec91b4110411d47923607287e300ec + uri: https://huggingface.co/mudler/Nex-N2-mini-APEX-GGUF/resolve/main/Nex-N2-mini-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Nex-N2-mini-APEX-GGUF/mmproj.gguf + sha256: 571ad1e826caa2eb5ff37fdf3ae30b5072edc694d8ebd25897166660ed981469 + uri: https://huggingface.co/mudler/Nex-N2-mini-APEX-GGUF/resolve/main/mmproj.gguf +- name: nex-n2-mini-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Nex-N2-mini-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nex-N2-mini-APEX-GGUF/Nex-N2-mini-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Nex-N2-mini-APEX-GGUF/Nex-N2-mini-APEX-I-Mini.gguf + sha256: 6aacb9f16cfc8136f7de784f5e66bd2019753017b2f189122f7e213f04624ef2 + uri: https://huggingface.co/mudler/Nex-N2-mini-APEX-GGUF/resolve/main/Nex-N2-mini-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Nex-N2-mini-APEX-GGUF/mmproj.gguf + sha256: 571ad1e826caa2eb5ff37fdf3ae30b5072edc694d8ebd25897166660ed981469 + uri: https://huggingface.co/mudler/Nex-N2-mini-APEX-GGUF/resolve/main/mmproj.gguf +- name: ornith-1.0-35b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Ornith-1.0-35B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Ornith-1.0-35B-APEX-GGUF/Ornith-1.0-35B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Ornith-1.0-35B-APEX-GGUF/Ornith-1.0-35B-APEX-I-Quality.gguf + sha256: 71291afbbf9158f41d062a391e9418eb9c5e62774e9d3db4d79e29a1ee8a2335 + uri: https://huggingface.co/mudler/Ornith-1.0-35B-APEX-GGUF/resolve/main/Ornith-1.0-35B-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Ornith-1.0-35B-APEX-GGUF/mmproj.gguf + sha256: afa2915185a37d92abb6623d45b2c05bb573ab40a9f1bb4993e265e70edbcc57 + uri: https://huggingface.co/mudler/Ornith-1.0-35B-APEX-GGUF/resolve/main/mmproj.gguf +- name: ornith-1.0-35b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Ornith-1.0-35B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Ornith-1.0-35B-APEX-GGUF/Ornith-1.0-35B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Ornith-1.0-35B-APEX-GGUF/Ornith-1.0-35B-APEX-I-Balanced.gguf + sha256: 22a806250ed1bb459e0b5241b1fbf971db6296dc99d27b922662d6df4eff0e21 + uri: https://huggingface.co/mudler/Ornith-1.0-35B-APEX-GGUF/resolve/main/Ornith-1.0-35B-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Ornith-1.0-35B-APEX-GGUF/mmproj.gguf + sha256: afa2915185a37d92abb6623d45b2c05bb573ab40a9f1bb4993e265e70edbcc57 + uri: https://huggingface.co/mudler/Ornith-1.0-35B-APEX-GGUF/resolve/main/mmproj.gguf +- name: ornith-1.0-35b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Ornith-1.0-35B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Ornith-1.0-35B-APEX-GGUF/Ornith-1.0-35B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Ornith-1.0-35B-APEX-GGUF/Ornith-1.0-35B-APEX-I-Compact.gguf + sha256: c747c8627be16f002597a9ddfaf93310f520d0c02c7d22d2a4419ba140fef7c2 + uri: https://huggingface.co/mudler/Ornith-1.0-35B-APEX-GGUF/resolve/main/Ornith-1.0-35B-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Ornith-1.0-35B-APEX-GGUF/mmproj.gguf + sha256: afa2915185a37d92abb6623d45b2c05bb573ab40a9f1bb4993e265e70edbcc57 + uri: https://huggingface.co/mudler/Ornith-1.0-35B-APEX-GGUF/resolve/main/mmproj.gguf +- name: ornith-1.0-35b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Ornith-1.0-35B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Ornith-1.0-35B-APEX-GGUF/Ornith-1.0-35B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Ornith-1.0-35B-APEX-GGUF/Ornith-1.0-35B-APEX-I-Mini.gguf + sha256: ede66e72de5e5801690d824f0d517dae9792a4b1c80d174c0708ad4a89c108df + uri: https://huggingface.co/mudler/Ornith-1.0-35B-APEX-GGUF/resolve/main/Ornith-1.0-35B-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Ornith-1.0-35B-APEX-GGUF/mmproj.gguf + sha256: afa2915185a37d92abb6623d45b2c05bb573ab40a9f1bb4993e265e70edbcc57 + uri: https://huggingface.co/mudler/Ornith-1.0-35B-APEX-GGUF/resolve/main/mmproj.gguf +- name: ornith-1.0-35b-ud-q4-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Ornith-1.0-35B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Ornith-1.0-35B-GGUF/Ornith-1.0-35B-UD-Q4_K_M.gguf + files: + - filename: llama-cpp/models/unsloth/Ornith-1.0-35B-GGUF/Ornith-1.0-35B-UD-Q4_K_M.gguf + sha256: 4c93716d7c9650285ab7af11e45044aeebd7d3828e513efcd582b0fa988af5ad + uri: https://huggingface.co/unsloth/Ornith-1.0-35B-GGUF/resolve/main/Ornith-1.0-35B-UD-Q4_K_M.gguf + - filename: llama-cpp/mmproj/unsloth/Ornith-1.0-35B-GGUF/mmproj-BF16.gguf + sha256: 8c10da8c1e4b860fa09d1539d40b7ae22b194706046d9d25d390e0d10c680748 + uri: https://huggingface.co/unsloth/Ornith-1.0-35B-GGUF/resolve/main/mmproj-BF16.gguf +- name: ornith-1.0-35b-ud-q5-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Ornith-1.0-35B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Ornith-1.0-35B-GGUF/Ornith-1.0-35B-UD-Q5_K_M.gguf + files: + - filename: llama-cpp/models/unsloth/Ornith-1.0-35B-GGUF/Ornith-1.0-35B-UD-Q5_K_M.gguf + sha256: fd0c2ab1c7b1b0e382e519f62889ddff374a5769873b984fa03966e40fc0d000 + uri: https://huggingface.co/unsloth/Ornith-1.0-35B-GGUF/resolve/main/Ornith-1.0-35B-UD-Q5_K_M.gguf + - filename: llama-cpp/mmproj/unsloth/Ornith-1.0-35B-GGUF/mmproj-BF16.gguf + sha256: 8c10da8c1e4b860fa09d1539d40b7ae22b194706046d9d25d390e0d10c680748 + uri: https://huggingface.co/unsloth/Ornith-1.0-35B-GGUF/resolve/main/mmproj-BF16.gguf +- name: ornith-1.0-35b-ud-q6-k + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Ornith-1.0-35B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Ornith-1.0-35B-GGUF/Ornith-1.0-35B-UD-Q6_K.gguf + files: + - filename: llama-cpp/models/unsloth/Ornith-1.0-35B-GGUF/Ornith-1.0-35B-UD-Q6_K.gguf + sha256: 1ba828db13615279dd9cb0c4b30ed797417696e75ddac7c6c5247c6d6bdfea3f + uri: https://huggingface.co/unsloth/Ornith-1.0-35B-GGUF/resolve/main/Ornith-1.0-35B-UD-Q6_K.gguf + - filename: llama-cpp/mmproj/unsloth/Ornith-1.0-35B-GGUF/mmproj-BF16.gguf + sha256: 8c10da8c1e4b860fa09d1539d40b7ae22b194706046d9d25d390e0d10c680748 + uri: https://huggingface.co/unsloth/Ornith-1.0-35B-GGUF/resolve/main/mmproj-BF16.gguf +- name: ornith-1.0-35b-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Ornith-1.0-35B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Ornith-1.0-35B-GGUF/Ornith-1.0-35B-Q8_0.gguf + files: + - filename: llama-cpp/models/unsloth/Ornith-1.0-35B-GGUF/Ornith-1.0-35B-Q8_0.gguf + sha256: 8f94f093f907065e182924225ef2182a7fc35cae3822a3eab29cc5a8ee02fe4f + uri: https://huggingface.co/unsloth/Ornith-1.0-35B-GGUF/resolve/main/Ornith-1.0-35B-Q8_0.gguf + - filename: llama-cpp/mmproj/unsloth/Ornith-1.0-35B-GGUF/mmproj-BF16.gguf + sha256: 8c10da8c1e4b860fa09d1539d40b7ae22b194706046d9d25d390e0d10c680748 + uri: https://huggingface.co/unsloth/Ornith-1.0-35B-GGUF/resolve/main/mmproj-BF16.gguf +- name: qwen3-coder-30b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3-Coder-30B-APEX-GGUF/Qwen3-Coder-30B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3-Coder-30B-APEX-GGUF/Qwen3-Coder-30B-APEX-I-Quality.gguf + sha256: bf4d49e6f2e526436a29a9481aab7d8dc97987a9e446e2d395a83a0f12bd7134 + uri: https://huggingface.co/mudler/Qwen3-Coder-30B-APEX-GGUF/resolve/main/Qwen3-Coder-30B-APEX-I-Quality.gguf +- name: qwen3-coder-30b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3-Coder-30B-APEX-GGUF/Qwen3-Coder-30B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3-Coder-30B-APEX-GGUF/Qwen3-Coder-30B-APEX-I-Balanced.gguf + sha256: 81530ce8fb2f0bcb354d1bdb9cc874acfa7c68f56de4ded4143946f524738b78 + uri: https://huggingface.co/mudler/Qwen3-Coder-30B-APEX-GGUF/resolve/main/Qwen3-Coder-30B-APEX-I-Balanced.gguf +- name: qwen3-coder-30b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3-Coder-30B-APEX-GGUF/Qwen3-Coder-30B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3-Coder-30B-APEX-GGUF/Qwen3-Coder-30B-APEX-I-Compact.gguf + sha256: e883cea614708d6f0ef26c1d7e6409c56b4a7182ad1e36b5fba3598e1bcf7788 + uri: https://huggingface.co/mudler/Qwen3-Coder-30B-APEX-GGUF/resolve/main/Qwen3-Coder-30B-APEX-I-Compact.gguf +- name: qwen3-coder-next-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3-Coder-Next-APEX-GGUF/Qwen3-Coder-Next-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3-Coder-Next-APEX-GGUF/Qwen3-Coder-Next-APEX-I-Quality.gguf + sha256: a31e629119c0dea4d294cc3ed230e6aac72e552f36ba09218d71d82340b5b755 + uri: https://huggingface.co/mudler/Qwen3-Coder-Next-APEX-GGUF/resolve/main/Qwen3-Coder-Next-APEX-I-Quality.gguf +- name: qwen3-coder-next-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3-Coder-Next-APEX-GGUF/Qwen3-Coder-Next-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3-Coder-Next-APEX-GGUF/Qwen3-Coder-Next-APEX-I-Balanced.gguf + sha256: 2051277beb9d9161daa808cbc6a4de87c58f74b737e5846b865546614da39cd1 + uri: https://huggingface.co/mudler/Qwen3-Coder-Next-APEX-GGUF/resolve/main/Qwen3-Coder-Next-APEX-I-Balanced.gguf +- name: qwen3-coder-next-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3-Coder-Next-APEX-GGUF/Qwen3-Coder-Next-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3-Coder-Next-APEX-GGUF/Qwen3-Coder-Next-APEX-I-Compact.gguf + sha256: 05b67d98ea9754c50c75f74b28ebdbede5b7b62f3265f0843fb245f3d8a93e4a + uri: https://huggingface.co/mudler/Qwen3-Coder-Next-APEX-GGUF/resolve/main/Qwen3-Coder-Next-APEX-I-Compact.gguf +- name: qwen3-coder-next-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3-Coder-Next-APEX-GGUF/Qwen3-Coder-Next-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3-Coder-Next-APEX-GGUF/Qwen3-Coder-Next-APEX-I-Mini.gguf + sha256: c0f3a931366aef306b36d0c3d1803f9801d790bbd1bae96d677eb3c68d2ab15d + uri: https://huggingface.co/mudler/Qwen3-Coder-Next-APEX-GGUF/resolve/main/Qwen3-Coder-Next-APEX-I-Mini.gguf +- name: qwen3-coder-next-ud-q4-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_M.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-UD-Q4_K_M.gguf + sha256: 3a47167ad1bf9e9a79f233f799588d54f85bc65644e72c19251165823e7ae612 + uri: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF/resolve/main/Qwen3-Coder-Next-UD-Q4_K_M.gguf +- name: qwen3-coder-next-ud-q5-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/UD-Q5_K_M/Qwen3-Coder-Next-UD-Q5_K_M-00001-of-00003.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/UD-Q5_K_M/Qwen3-Coder-Next-UD-Q5_K_M-00001-of-00003.gguf + sha256: cc49a0fde9c28ea8ce9bd4bdf053ca96a25c8c1fe4993a33b5db543a8eaf92eb + uri: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF/resolve/main/UD-Q5_K_M/Qwen3-Coder-Next-UD-Q5_K_M-00001-of-00003.gguf + - filename: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/UD-Q5_K_M/Qwen3-Coder-Next-UD-Q5_K_M-00002-of-00003.gguf + sha256: 9e86604dd42bc56b9f36dfbc0c114bf2b98978d3be58d7347d385eff90bf18a8 + uri: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF/resolve/main/UD-Q5_K_M/Qwen3-Coder-Next-UD-Q5_K_M-00002-of-00003.gguf + - filename: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/UD-Q5_K_M/Qwen3-Coder-Next-UD-Q5_K_M-00003-of-00003.gguf + sha256: aa71e21d3eb34d70e725df65ae84341f2c62e2f0cd19c3222c3b78b6629ff5c2 + uri: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF/resolve/main/UD-Q5_K_M/Qwen3-Coder-Next-UD-Q5_K_M-00003-of-00003.gguf +- name: qwen3-coder-next-ud-q6-k + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/UD-Q6_K/Qwen3-Coder-Next-UD-Q6_K-00001-of-00003.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/UD-Q6_K/Qwen3-Coder-Next-UD-Q6_K-00001-of-00003.gguf + sha256: 68f03d9c0cc72c3359ed94e2058230a1ff484e6403b842e5db499c12f9157fd6 + uri: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF/resolve/main/UD-Q6_K/Qwen3-Coder-Next-UD-Q6_K-00001-of-00003.gguf + - filename: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/UD-Q6_K/Qwen3-Coder-Next-UD-Q6_K-00002-of-00003.gguf + sha256: 048f963be0031eb9781a99e69425373c476047dbbeeee42f40923314af64803d + uri: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF/resolve/main/UD-Q6_K/Qwen3-Coder-Next-UD-Q6_K-00002-of-00003.gguf + - filename: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/UD-Q6_K/Qwen3-Coder-Next-UD-Q6_K-00003-of-00003.gguf + sha256: 99b277cae9a934aebecac33351b1142bd1844cb4ebb1ad2791eef66a54d55bd7 + uri: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF/resolve/main/UD-Q6_K/Qwen3-Coder-Next-UD-Q6_K-00003-of-00003.gguf +- name: qwen3-coder-next-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/Q8_0/Qwen3-Coder-Next-Q8_0-00001-of-00003.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/Q8_0/Qwen3-Coder-Next-Q8_0-00001-of-00003.gguf + sha256: 8fceb6409e609f9d4d92658fc926c7fb1dc045d229cf509ef5bbae352867ba41 + uri: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF/resolve/main/Q8_0/Qwen3-Coder-Next-Q8_0-00001-of-00003.gguf + - filename: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/Q8_0/Qwen3-Coder-Next-Q8_0-00002-of-00003.gguf + sha256: 491753e1ed8f20a5b898ae2f47ca58bed2707b56c00a3ec7c0b681b46d953ff8 + uri: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF/resolve/main/Q8_0/Qwen3-Coder-Next-Q8_0-00002-of-00003.gguf + - filename: llama-cpp/models/unsloth/Qwen3-Coder-Next-GGUF/Q8_0/Qwen3-Coder-Next-Q8_0-00003-of-00003.gguf + sha256: 4dced2b07a817f1e0c085fcc332b76c6e32bbab5e0c43311e0224f6249e03366 + uri: https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF/resolve/main/Q8_0/Qwen3-Coder-Next-Q8_0-00003-of-00003.gguf +- name: qwen3.5-122b-a10b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.5-122B-A10B-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-122B-A10B-APEX-GGUF/Qwen3.5-122B-A10B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-122B-A10B-APEX-GGUF/Qwen3.5-122B-A10B-APEX-I-Quality.gguf + sha256: e12fe56bea8435940454c88ee93c334b42f93202f49aabf9833be92cd668e341 + uri: https://huggingface.co/mudler/Qwen3.5-122B-A10B-APEX-GGUF/resolve/main/Qwen3.5-122B-A10B-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.5-122B-A10B-APEX-GGUF/mmproj-F16.gguf + sha256: 691af71bd41d437ce4a1d989f589c4b492237d702b33819ed8f897f9fae6e725 + uri: https://huggingface.co/mudler/Qwen3.5-122B-A10B-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: qwen3.5-122b-a10b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.5-122B-A10B-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-122B-A10B-APEX-GGUF/Qwen3.5-122B-A10B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-122B-A10B-APEX-GGUF/Qwen3.5-122B-A10B-APEX-I-Balanced.gguf + sha256: bbb5f1fa67c213d60d0ada4fe3218fad1dc17191ab619c44de64d6de6213a39c + uri: https://huggingface.co/mudler/Qwen3.5-122B-A10B-APEX-GGUF/resolve/main/Qwen3.5-122B-A10B-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.5-122B-A10B-APEX-GGUF/mmproj-F16.gguf + sha256: 691af71bd41d437ce4a1d989f589c4b492237d702b33819ed8f897f9fae6e725 + uri: https://huggingface.co/mudler/Qwen3.5-122B-A10B-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: qwen3.5-122b-a10b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.5-122B-A10B-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-122B-A10B-APEX-GGUF/Qwen3.5-122B-A10B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-122B-A10B-APEX-GGUF/Qwen3.5-122B-A10B-APEX-I-Compact.gguf + sha256: fee6482b91e2ee1e34562626f827a6b4725c0f2e0576a933be52acf9c0758a9e + uri: https://huggingface.co/mudler/Qwen3.5-122B-A10B-APEX-GGUF/resolve/main/Qwen3.5-122B-A10B-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.5-122B-A10B-APEX-GGUF/mmproj-F16.gguf + sha256: 691af71bd41d437ce4a1d989f589c4b492237d702b33819ed8f897f9fae6e725 + uri: https://huggingface.co/mudler/Qwen3.5-122B-A10B-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: qwen3.5-122b-a10b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.5-122B-A10B-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-122B-A10B-APEX-GGUF/Qwen3.5-122B-A10B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-122B-A10B-APEX-GGUF/Qwen3.5-122B-A10B-APEX-I-Mini.gguf + sha256: 6106f9db580b6e1f44f117bc92f8b507fd7add620518a0cbb5f219f12cc674e7 + uri: https://huggingface.co/mudler/Qwen3.5-122B-A10B-APEX-GGUF/resolve/main/Qwen3.5-122B-A10B-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.5-122B-A10B-APEX-GGUF/mmproj-F16.gguf + sha256: 691af71bd41d437ce4a1d989f589c4b492237d702b33819ed8f897f9fae6e725 + uri: https://huggingface.co/mudler/Qwen3.5-122B-A10B-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: qwen3.5-122b-a10b-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Qwen3.5-122B-A10B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3.5-122B-A10B-GGUF/Q8_0/Qwen3.5-122B-A10B-Q8_0-00001-of-00004.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3.5-122B-A10B-GGUF/Q8_0/Qwen3.5-122B-A10B-Q8_0-00001-of-00004.gguf + sha256: a6cc4bf06d44a96739a200841d2335fc437fa2d790d0975baba9da8af9e228e1 + uri: https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF/resolve/main/Q8_0/Qwen3.5-122B-A10B-Q8_0-00001-of-00004.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-122B-A10B-GGUF/Q8_0/Qwen3.5-122B-A10B-Q8_0-00002-of-00004.gguf + sha256: 81d470090559e5d87459fce8943ccdc9723b7f1b0c643f8289b7d0541ac62a1c + uri: https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF/resolve/main/Q8_0/Qwen3.5-122B-A10B-Q8_0-00002-of-00004.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-122B-A10B-GGUF/Q8_0/Qwen3.5-122B-A10B-Q8_0-00003-of-00004.gguf + sha256: 824422fa8fe3208b6de2c04d96e1240661b07b74f71ee69f0df1dd48360f66b9 + uri: https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF/resolve/main/Q8_0/Qwen3.5-122B-A10B-Q8_0-00003-of-00004.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-122B-A10B-GGUF/Q8_0/Qwen3.5-122B-A10B-Q8_0-00004-of-00004.gguf + sha256: be1ed51c1248c7accc053dca8e7f24d1b831fe1e87747c0104da4006123d68ae + uri: https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF/resolve/main/Q8_0/Qwen3.5-122B-A10B-Q8_0-00004-of-00004.gguf + - filename: llama-cpp/mmproj/unsloth/Qwen3.5-122B-A10B-GGUF/mmproj-BF16.gguf + sha256: c4f22a6b5101ac85930029e0eec321bfaa4c0b16851f569759d3bac3fa31c744 + uri: https://huggingface.co/unsloth/Qwen3.5-122B-A10B-GGUF/resolve/main/mmproj-BF16.gguf +- name: qwen3.5-35b-a3b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.5-35B-A3B-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-GGUF/Qwen3.5-35B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-GGUF/Qwen3.5-35B-A3B-APEX-I-Quality.gguf + sha256: 5e0fa27a9b0841ca8fffc039c6a2e8d933f1d0be12369a14fd4abce090103f20 + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.5-35B-A3B-APEX-GGUF/mmproj-F16.gguf + sha256: a516ab92e8240da4734d68352bdfba84c16e830ee40010b8fac80d69c77272ff + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: qwen3.5-35b-a3b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.5-35B-A3B-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-GGUF/Qwen3.5-35B-A3B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-GGUF/Qwen3.5-35B-A3B-APEX-I-Balanced.gguf + sha256: b3d3b39e079fff18311b24b88217badca56381b76c6798dcf12e2dcccaa4dae7 + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.5-35B-A3B-APEX-GGUF/mmproj-F16.gguf + sha256: a516ab92e8240da4734d68352bdfba84c16e830ee40010b8fac80d69c77272ff + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: qwen3.5-35b-a3b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.5-35B-A3B-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-GGUF/Qwen3.5-35B-A3B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-GGUF/Qwen3.5-35B-A3B-APEX-I-Compact.gguf + sha256: 25fe188e2570366bebbf260c5ebd9fafe714a8458d364fb0b3809cf8b9303866 + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.5-35B-A3B-APEX-GGUF/mmproj-F16.gguf + sha256: a516ab92e8240da4734d68352bdfba84c16e830ee40010b8fac80d69c77272ff + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: qwen3.5-35b-a3b-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Qwen3.5-35B-A3B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-Q8_0.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-Q8_0.gguf + sha256: 3808866c016ab02b4adb26b873f7008a2cdd2c0704a39704050119ab0631db46 + uri: https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/main/Qwen3.5-35B-A3B-Q8_0.gguf + - filename: llama-cpp/mmproj/unsloth/Qwen3.5-35B-A3B-GGUF/mmproj-BF16.gguf + sha256: abe81a7212be307a7723ab47a51a87e5c46d0622273ccb04a6a6feba18b21d63 + uri: https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/main/mmproj-BF16.gguf +- name: qwen3.5-35b-a3b-apex-tq-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF/Qwen3.5-35B-A3B-APEX-TQ-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF/Qwen3.5-35B-A3B-APEX-TQ-Quality.gguf + sha256: 3a9f46c7c3986983d4e465973007259e973a1a9cee9ae7a8d4d3e97f7116a252 + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF/resolve/main/Qwen3.5-35B-A3B-APEX-TQ-Quality.gguf +- name: qwen3.5-35b-a3b-apex-tq-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF/Qwen3.5-35B-A3B-APEX-TQ-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF/Qwen3.5-35B-A3B-APEX-TQ-Balanced.gguf + sha256: c1de8303e61d8d49d6bbf6f0a3db0f7b7a84c3246d2fd8cfe45e5ef3282a4791 + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF/resolve/main/Qwen3.5-35B-A3B-APEX-TQ-Balanced.gguf +- name: qwen3.5-35b-a3b-apex-tq-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF/Qwen3.5-35B-A3B-APEX-TQ-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF/Qwen3.5-35B-A3B-APEX-TQ-Compact.gguf + sha256: 5a8c14979e6f79bbe2a251a63515863e5621a27986ca862ef753100c1e2fb164 + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-TQ-GGUF/resolve/main/Qwen3.5-35B-A3B-APEX-TQ-Compact.gguf +- name: qwen3.5-35b-a3b-claude-distilled-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Quality.gguf + sha256: 99ebd4c27e6957d75396fdccec79cf26f332ffd9503e95751785ed5352e54981 + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Quality.gguf +- name: qwen3.5-35b-a3b-claude-distilled-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Balanced.gguf + sha256: 3c2a866546716193a715e37a7b690421d17b8174c00054311aaa1737bea077fa + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Balanced.gguf +- name: qwen3.5-35b-a3b-claude-distilled-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Compact.gguf + sha256: c278bdacad22d25a57778eed4da7439e1ad86fe72f450d7d6c5335cd095e6e3c + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Compact.gguf +- name: qwen3.5-35b-a3b-claude-distilled-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Mini.gguf + sha256: 09ce279daab7d544cb604350e4d381691fc9ea18a31dd22750bcb6414e6e6e78 + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Mini.gguf +- name: qwen3.5-35b-a3b-uncensored-fernflowerai-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Quality.gguf + sha256: 6b145382be701da5cdc2f6c630a3543c47e54a114c160da5357d6b3ad2abf7ea + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Quality.gguf +- name: qwen3.5-35b-a3b-uncensored-fernflowerai-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Balanced.gguf + sha256: e5dcf17c18e010f48219d787b8bfd5251a9d1f9c8c1115fa73d7744924d65f6f + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Balanced.gguf +- name: qwen3.5-35b-a3b-uncensored-fernflowerai-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Compact.gguf + sha256: 60bdc1727edf663d9a1a2de0b432be86d05d0a8a963e8aa03b994c0f582ab99a + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Compact.gguf +- name: qwen3.5-35b-a3b-uncensored-fernflowerai-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Mini.gguf + sha256: 56b7b012d9202676d2fdad7b802a0598318cb992a95713d2ec10bb05f1b1b227 + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Mini.gguf +- name: qwen3.5-35b-a3b-uncensored-fernflowerai-apex-i-nano + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Nano.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Nano.gguf + sha256: 6fa17618ac657c762d031a41d06527d7425b5c1fcda19638ca8111f2cc7a06ee + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Nano.gguf +- name: qwen3.5-397b-a17b-apex-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.5-397B-A17B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-397B-A17B-APEX-GGUF/Qwen3.5-397B-A17B-APEX-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-397B-A17B-APEX-GGUF/Qwen3.5-397B-A17B-APEX-Quality.gguf + sha256: d41f62529b5738709960b334c3194e64bd66a72315c90f663357d224cf3f5b64 + uri: https://huggingface.co/mudler/Qwen3.5-397B-A17B-APEX-GGUF/resolve/main/Qwen3.5-397B-A17B-APEX-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.5-397B-A17B-APEX-GGUF/mmproj.gguf + sha256: b3624272d7b9b49ffe6c6d0c592980bed6b026ce59cde11708bb230395c2a227 + uri: https://huggingface.co/mudler/Qwen3.5-397B-A17B-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.5-397b-a17b-apex-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.5-397B-A17B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-397B-A17B-APEX-GGUF/Qwen3.5-397B-A17B-APEX-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-397B-A17B-APEX-GGUF/Qwen3.5-397B-A17B-APEX-Balanced.gguf + sha256: 001ca6a45ac1863084b5ce989729da2db26c23ed47cedc1d2f0d611b43d75956 + uri: https://huggingface.co/mudler/Qwen3.5-397B-A17B-APEX-GGUF/resolve/main/Qwen3.5-397B-A17B-APEX-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.5-397B-A17B-APEX-GGUF/mmproj.gguf + sha256: b3624272d7b9b49ffe6c6d0c592980bed6b026ce59cde11708bb230395c2a227 + uri: https://huggingface.co/mudler/Qwen3.5-397B-A17B-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.5-397b-a17b-apex-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.5-397B-A17B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-397B-A17B-APEX-GGUF/Qwen3.5-397B-A17B-APEX-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-397B-A17B-APEX-GGUF/Qwen3.5-397B-A17B-APEX-Compact.gguf + sha256: ce2e94a73f96a06c0d7040913ba9302a74f8ed5626a87cc01760ec7b225d5bd0 + uri: https://huggingface.co/mudler/Qwen3.5-397B-A17B-APEX-GGUF/resolve/main/Qwen3.5-397B-A17B-APEX-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.5-397B-A17B-APEX-GGUF/mmproj.gguf + sha256: b3624272d7b9b49ffe6c6d0c592980bed6b026ce59cde11708bb230395c2a227 + uri: https://huggingface.co/mudler/Qwen3.5-397B-A17B-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.5-397b-a17b-ud-q4-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00001-of-00006.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00001-of-00006.gguf + sha256: 63c290c9be83e1b4dd41833d81bd933afd535d65657579b9f92f5c3f76e0218d + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00001-of-00006.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00002-of-00006.gguf + sha256: dc94995a3605f3130700e96df51ee56cf93bd9340fe891918403450556453ed7 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00002-of-00006.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00003-of-00006.gguf + sha256: 2952dadb60137f413d5f70f6ca3c06007e24198e712c882a094432f58f76c230 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00003-of-00006.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00004-of-00006.gguf + sha256: c7b99959e8fb78c8cfc9b71f3da07a2b4a6d39bf377dfa226f0a7b730c8cf3ba + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00004-of-00006.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00005-of-00006.gguf + sha256: eeea4540f7289ab3baad2b3f2b4b6798e70a1802b9b4b269799a1f04d75b0af0 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00005-of-00006.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00006-of-00006.gguf + sha256: d3bf93bb9fe007910ae9c0fd130d7776d7c6149635c9e7f158312308beb9b754 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q4_K_M/Qwen3.5-397B-A17B-UD-Q4_K_M-00006-of-00006.gguf + - filename: llama-cpp/mmproj/unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-BF16.gguf + sha256: b3624272d7b9b49ffe6c6d0c592980bed6b026ce59cde11708bb230395c2a227 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/mmproj-BF16.gguf +- name: qwen3.5-397b-a17b-ud-q5-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00001-of-00008.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00001-of-00008.gguf + sha256: c82146df8eca3ee9611f9ed1925a19ab7ec32f69c4695ed66e1ed87233cd6a79 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00001-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00002-of-00008.gguf + sha256: ed30d3bf537c22dd834f7eeeebc8d6214404c3d56445616c55fdd66e0a21a988 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00002-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00003-of-00008.gguf + sha256: 612ed2bc62562b096ffec5ba25a51c46a575043c01a33597064b438a38a935a1 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00003-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00004-of-00008.gguf + sha256: 0ca4284becec14270378130904989f09b385de73c196fc225803a0a7b1ac44fc + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00004-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00005-of-00008.gguf + sha256: f46a18407d0b17ed48758af4cd765ea2d5a6931a6384dc5b1417a2ca293c07a6 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00005-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00006-of-00008.gguf + sha256: dd7b94ac3fd1e7a7de01abad02fd3767062d41f3ec7c08ad67c3f1d61c9c7262 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00006-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00007-of-00008.gguf + sha256: 29c6d01ef67357fc085a8b812a5730f2d0a046b295752501ea1bc44cb419ef33 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00007-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00008-of-00008.gguf + sha256: 4bbade915bb05436822b36a088710a90d9da5a7b0f2a5621a978b79a366be35f + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q5_K_M/Qwen3.5-397B-A17B-UD-Q5_K_M-00008-of-00008.gguf + - filename: llama-cpp/mmproj/unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-BF16.gguf + sha256: b3624272d7b9b49ffe6c6d0c592980bed6b026ce59cde11708bb230395c2a227 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/mmproj-BF16.gguf +- name: qwen3.5-397b-a17b-ud-q6-k + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00001-of-00008.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00001-of-00008.gguf + sha256: 0a409159a8b206ef81bdd5736ae6bcc300445ed1adcb7e3faaf9167a8fb79f07 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00001-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00002-of-00008.gguf + sha256: 63aa2ef1048a36faaba2ebb7d320c6bab88eaf9635e20c30964d2b15315cfc09 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00002-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00003-of-00008.gguf + sha256: c874919cd6ba194d3b259e173faad5e41f989168492477d783c3594d3994f7ba + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00003-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00004-of-00008.gguf + sha256: 8a21d496e0b16e85eb74ea679053055a4579a658ffc3e6c00f7875e76b88c4ce + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00004-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00005-of-00008.gguf + sha256: e327fdc4663890a488858d142af6c477f1c060bbbeb4268c8db6e49983cd41d4 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00005-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00006-of-00008.gguf + sha256: ef68b262459a4304d371bd503016de9cf3277eb971e18a073203c4fe10345a45 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00006-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00007-of-00008.gguf + sha256: 4f28a8907d17a21aa670c0a78403d24a3e52a5d5ee5a65987e54a6742b63bbc0 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00007-of-00008.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00008-of-00008.gguf + sha256: 2092241676533c799862ec78d4035af4402907df5822fac13f4109584336a214 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/UD-Q6_K/Qwen3.5-397B-A17B-UD-Q6_K-00008-of-00008.gguf + - filename: llama-cpp/mmproj/unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-BF16.gguf + sha256: b3624272d7b9b49ffe6c6d0c592980bed6b026ce59cde11708bb230395c2a227 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/mmproj-BF16.gguf +- name: qwen3.5-397b-a17b-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/Q8_0/Qwen3.5-397B-A17B-Q8_0-00001-of-00010.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/Q8_0/Qwen3.5-397B-A17B-Q8_0-00001-of-00010.gguf + sha256: ba984516e0fb4000dc2d097bed53add6ecf615930ac4a3da89e351fc23abf17a + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/Q8_0/Qwen3.5-397B-A17B-Q8_0-00001-of-00010.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/Q8_0/Qwen3.5-397B-A17B-Q8_0-00002-of-00010.gguf + sha256: c74a4a3bfe49116cf66cf045b0dfec5c93adaf8a99d84dae44b308f95340bfb6 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/Q8_0/Qwen3.5-397B-A17B-Q8_0-00002-of-00010.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/Q8_0/Qwen3.5-397B-A17B-Q8_0-00003-of-00010.gguf + sha256: dc18c45c75bef79daaca68003ff5919bc85be016554aa5e2d5a1fd3251c6f7ab + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/Q8_0/Qwen3.5-397B-A17B-Q8_0-00003-of-00010.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/Q8_0/Qwen3.5-397B-A17B-Q8_0-00004-of-00010.gguf + sha256: 589ad641d7ca911ad0fc3a378b90e6f07c716d1b1a53d820688cb819441790c0 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/Q8_0/Qwen3.5-397B-A17B-Q8_0-00004-of-00010.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/Q8_0/Qwen3.5-397B-A17B-Q8_0-00005-of-00010.gguf + sha256: c1eb590fc786457447bee156ca32cb1f0cac540c963d121534a899fb0d87b149 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/Q8_0/Qwen3.5-397B-A17B-Q8_0-00005-of-00010.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/Q8_0/Qwen3.5-397B-A17B-Q8_0-00006-of-00010.gguf + sha256: 33708a36679b514f1f2413a1dda5d831d6547ddb84a9735221604f5e9a069357 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/Q8_0/Qwen3.5-397B-A17B-Q8_0-00006-of-00010.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/Q8_0/Qwen3.5-397B-A17B-Q8_0-00007-of-00010.gguf + sha256: ef911df6baa00777ba842c6ed5c45c9c0a3642b8017ebb2c6023ae55fc20c34b + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/Q8_0/Qwen3.5-397B-A17B-Q8_0-00007-of-00010.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/Q8_0/Qwen3.5-397B-A17B-Q8_0-00008-of-00010.gguf + sha256: f840797f800bacddceda674f6f56c2dd7e63152b1afeec935cfaf959f1afcf0a + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/Q8_0/Qwen3.5-397B-A17B-Q8_0-00008-of-00010.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/Q8_0/Qwen3.5-397B-A17B-Q8_0-00009-of-00010.gguf + sha256: ccc0d89d361bd21ac0db4a54fc30cb811803d8389f598370dc1ff121875caa61 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/Q8_0/Qwen3.5-397B-A17B-Q8_0-00009-of-00010.gguf + - filename: llama-cpp/models/unsloth/Qwen3.5-397B-A17B-GGUF/Q8_0/Qwen3.5-397B-A17B-Q8_0-00010-of-00010.gguf + sha256: a0aaf48579fcef07f8e12f97c7378553c0ba77149ac6bcf60cb663ffcfa27ca1 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/Q8_0/Qwen3.5-397B-A17B-Q8_0-00010-of-00010.gguf + - filename: llama-cpp/mmproj/unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-BF16.gguf + sha256: b3624272d7b9b49ffe6c6d0c592980bed6b026ce59cde11708bb230395c2a227 + uri: https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/resolve/main/mmproj-BF16.gguf +- name: qwen3.6-35b-a3b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-GGUF/Qwen3.6-35B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-GGUF/Qwen3.6-35B-A3B-APEX-I-Quality.gguf + sha256: ffb9535b3133942909c3973272d8e65748edafaae5272273baedc7ad78ad2ea6 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-APEX-GGUF/mmproj.gguf + sha256: 356dfaa3111376a4f7165e32e8749713378d1700b37cf52e0c50d9f23322334d + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-GGUF/Qwen3.6-35B-A3B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-GGUF/Qwen3.6-35B-A3B-APEX-I-Balanced.gguf + sha256: cc62ad396e94c783b6fd233d59c2fe5a88e3bdb807e37efd130de350f35d49a2 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-APEX-GGUF/mmproj.gguf + sha256: 356dfaa3111376a4f7165e32e8749713378d1700b37cf52e0c50d9f23322334d + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-GGUF/Qwen3.6-35B-A3B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-GGUF/Qwen3.6-35B-A3B-APEX-I-Compact.gguf + sha256: 50e1122946854f2272b44d466c03d17d410d3f02dcd1c021a1f29f6b384a7126 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-APEX-GGUF/mmproj.gguf + sha256: 356dfaa3111376a4f7165e32e8749713378d1700b37cf52e0c50d9f23322334d + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-GGUF/Qwen3.6-35B-A3B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-GGUF/Qwen3.6-35B-A3B-APEX-I-Mini.gguf + sha256: 5a9cbafc304f7766ff55be089e066b1d86f4beac3bf45bbdf3d27b07949e1d46 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-APEX-GGUF/mmproj.gguf + sha256: 356dfaa3111376a4f7165e32e8749713378d1700b37cf52e0c50d9f23322334d + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-ud-q5-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Qwen3.6-35B-A3B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q5_K_M.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q5_K_M.gguf + sha256: c13ce26253ea334df472bd8fbd2d6da66d8a41195c17f6fcbf44c4d20ece0932 + uri: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q5_K_M.gguf + - filename: llama-cpp/mmproj/unsloth/Qwen3.6-35B-A3B-GGUF/mmproj-BF16.gguf + sha256: 356dfaa3111376a4f7165e32e8749713378d1700b37cf52e0c50d9f23322334d + uri: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/resolve/main/mmproj-BF16.gguf +- name: qwen3.6-35b-a3b-ud-q6-k + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Qwen3.6-35B-A3B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q6_K.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q6_K.gguf + sha256: 4fe53b148b46f9b88830e2a3055c5b15c3a4d1e3ddc9a1384a108d8b9d59f043 + uri: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q6_K.gguf + - filename: llama-cpp/mmproj/unsloth/Qwen3.6-35B-A3B-GGUF/mmproj-BF16.gguf + sha256: 356dfaa3111376a4f7165e32e8749713378d1700b37cf52e0c50d9f23322334d + uri: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/resolve/main/mmproj-BF16.gguf +- name: qwen3.6-35b-a3b-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Qwen3.6-35B-A3B-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-Q8_0.gguf + files: + - filename: llama-cpp/models/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-Q8_0.gguf + sha256: d1a395809f65a43a13ad119eb4e7acdef1ac6d68120f39902c8ab96e72794a59 + uri: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/resolve/main/Qwen3.6-35B-A3B-Q8_0.gguf + - filename: llama-cpp/mmproj/unsloth/Qwen3.6-35B-A3B-GGUF/mmproj-BF16.gguf + sha256: 356dfaa3111376a4f7165e32e8749713378d1700b37cf52e0c50d9f23322334d + uri: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/resolve/main/mmproj-BF16.gguf +- name: qwen3.6-35b-a3b-apex-mtp-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/Qwen3.6-35B-A3B-APEX-MTP-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/Qwen3.6-35B-A3B-APEX-MTP-I-Quality.gguf + sha256: eb0883790326b76ebf90e791b4c3689e9000f337634456f7569060546b650abe + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-APEX-MTP-I-Quality.gguf +- name: qwen3.6-35b-a3b-apex-mtp-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf + sha256: 9bf7d96bb3a9d363e645dd998aee9e9bff8e016a82aec7ff081e0e6cdb53419e + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-APEX-MTP-I-Balanced.gguf +- name: qwen3.6-35b-a3b-apex-mtp-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf + sha256: 7758ab03a600f486e8b50b2a454874a03b52c11a6f813db561749e9beb3ab06e + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf +- name: qwen3.6-35b-a3b-apex-mtp-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/Qwen3.6-35B-A3B-APEX-MTP-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/Qwen3.6-35B-A3B-APEX-MTP-I-Mini.gguf + sha256: 6c2092ff43056701fdf431f01fb62334d60500953dc270dea2a387f1bb53d99b + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-APEX-MTP-I-Mini.gguf +- name: qwen3.6-35b-a3b-apex-mtp-i-nano + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/Qwen3.6-35B-A3B-APEX-MTP-I-Nano.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/Qwen3.6-35B-A3B-APEX-MTP-I-Nano.gguf + sha256: c41cdce38c767a13eccc65a676d28a5506ef285c31614d1e7d822979b9b3559e + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-APEX-MTP-I-Nano.gguf +- name: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Quality.gguf + sha256: 361867cba2a9e3839153de7b660df8b4e6ff7199529266a446e4dd8990b67de0 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Quality.gguf +- name: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Balanced.gguf + sha256: 3857fd436972ffb61c81563f8e22414d308ee8e3d5afb0658b2f477b6597153a + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Balanced.gguf +- name: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Compact.gguf + sha256: 976f3cb1a738a0bf013d9a734453580cc3649d7f1753dec8f71e41583bb1814f + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Compact.gguf +- name: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Mini.gguf + sha256: fcb468276232c32da30d585c1081dee1a5a3bcee8d4d0247df30c0e5a9cf753f + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-I-Mini.gguf +- name: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-mtp-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Quality.gguf + sha256: 11ca498862e628ac8198c127724e0eb999d18688f43b3559c8af1f41101dda83 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Quality.gguf +- name: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-mtp-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Balanced.gguf + sha256: a3a3bd9a195e6d51218e7dbd68e99dcd623d67efc5b25b35be3066fff246e157 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Balanced.gguf +- name: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-mtp-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Compact.gguf + sha256: 2abe5a7acc2a5ef6406c1f83845712e5bb68272dcbd73d8931deaab51d9ca794 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Compact.gguf +- name: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-mtp-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Mini.gguf + sha256: 438d8fee2e1d96fdea7eccb6daf27ac236b7d8bd8dc4891c56f5108fa4a99b60 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Mini.gguf +- name: qwen3.6-35b-a3b-claude-4.6-opus-reasoning-distilled-apex-mtp-i-nano + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Nano.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Nano.gguf + sha256: 2308dc2c04e83f82adbb8e40f9293f8103592c350215d08b79676a5f299ace0b + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled-APEX-MTP-I-Nano.gguf +- name: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Quality.gguf + sha256: 5785096c8c2c0e3fb3d59f5c1a429e7c6345ebd569b317a5356b142459119cf9 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + sha256: 12960e8cc09c5fc0a9f119989a3df717e00c3160c92e7229a964605d62b338fc + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Balanced.gguf + sha256: d027e05e0d2decc01bfcf1d9ce53693a917535a5150d242ef333028cfef2d391 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + sha256: 12960e8cc09c5fc0a9f119989a3df717e00c3160c92e7229a964605d62b338fc + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Compact.gguf + sha256: 6179139dc9d779b0d3d0e7747cf715cc533f97e68fdc079b2e97844819538da6 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + sha256: 12960e8cc09c5fc0a9f119989a3df717e00c3160c92e7229a964605d62b338fc + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Mini.gguf + sha256: ce6f5471856abb7d2a9aed57bd5e9e9a8f2c63166a214118a0b02ddcb4d952c3 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + sha256: 12960e8cc09c5fc0a9f119989a3df717e00c3160c92e7229a964605d62b338fc + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-i-nano + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Nano.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Nano.gguf + sha256: f0c650ca96a452093286181cd5618a7c131a3c0a139471ba6548b3cbedb47450 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Nano.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + sha256: 12960e8cc09c5fc0a9f119989a3df717e00c3160c92e7229a964605d62b338fc + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-mtp-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Quality.gguf + sha256: 44472af62a067e6fbb652d88e26778c31b156caa2be371491120762fe0424d32 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Quality.gguf +- name: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-mtp-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Balanced.gguf + sha256: 47b3b3955eac4da40bd4bafc35bf4ff1d1c377f34ec639a06400a5f405b8178f + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Balanced.gguf +- name: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-mtp-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Compact.gguf + sha256: 78e96b456cc587d416b4dc0abf4e3b205a896d722d0b7a700a5932c10a4ec100 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Compact.gguf +- name: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-mtp-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Mini.gguf + sha256: 42587e47ec309bfd197eaf72e38f71e3e29dedcd2fc69c26aca01bb362be381c + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Mini.gguf +- name: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-mtp-i-nano + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Nano.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Nano.gguf + sha256: 1fb8a998362ebb5f7f3c8ece6d4803a74ba32211c751de2e76b81e3379fbf050 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-MTP-I-Nano.gguf +- name: qwen3.6-35b-a3b-uncensored-heretic-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Quality.gguf + sha256: 667bd985a00bf5a47e5ba582f0bb573578e215c54b504525c13b8d4f573e5b59 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/mmproj.gguf + sha256: 1c625f05cd52e90abc76a5f756226c3a5fe279593379c22f6c6846c970a0cd18 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-uncensored-heretic-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Balanced.gguf + sha256: 7e4d28f5728cb0df835c4bd4cb1afe8157b1ba9faea14b125565fdee0ee7fda5 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/mmproj.gguf + sha256: 1c625f05cd52e90abc76a5f756226c3a5fe279593379c22f6c6846c970a0cd18 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-uncensored-heretic-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Compact.gguf + sha256: aa80e8a52e7fa62fd9101eba8c412095a7abe0db1ef1105ddc3848e65a2cbe51 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/mmproj.gguf + sha256: 1c625f05cd52e90abc76a5f756226c3a5fe279593379c22f6c6846c970a0cd18 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwen3.6-35b-a3b-uncensored-heretic-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Mini.gguf + sha256: 956dd8cde3dc9d5af5467da037e2724677b9434c15ef93f66f52c26a0c271852 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/mmproj.gguf + sha256: 1c625f05cd52e90abc76a5f756226c3a5fe279593379c22f6c6846c970a0cd18 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwopus-moe-35b-a3b-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/Qwopus-MoE-35B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/Qwopus-MoE-35B-A3B-APEX-I-Quality.gguf + sha256: 36a7848c43b3c7db701aba61b71729e2a172da7b2c5338f5121d2c31f2ab86ee + uri: https://huggingface.co/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/resolve/main/Qwopus-MoE-35B-A3B-APEX-I-Quality.gguf +- name: qwopus-moe-35b-a3b-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/Qwopus-MoE-35B-A3B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/Qwopus-MoE-35B-A3B-APEX-I-Balanced.gguf + sha256: 18e6979a777f313cec57329cb9c5a736cf1fc9120f8c8d3284d1b50f57fff0ce + uri: https://huggingface.co/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/resolve/main/Qwopus-MoE-35B-A3B-APEX-I-Balanced.gguf +- name: qwopus-moe-35b-a3b-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/Qwopus-MoE-35B-A3B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/Qwopus-MoE-35B-A3B-APEX-I-Compact.gguf + sha256: 1c4b0b62ae97e183532e307cf945f9658d89f958dd15e449957983f5d2eed4f0 + uri: https://huggingface.co/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/resolve/main/Qwopus-MoE-35B-A3B-APEX-I-Compact.gguf +- name: qwopus-moe-35b-a3b-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/Qwopus-MoE-35B-A3B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/Qwopus-MoE-35B-A3B-APEX-I-Mini.gguf + sha256: 166892da3c52a5a67c560393fcd69daf3c68961d955bfdd29c7ffec4690eac99 + uri: https://huggingface.co/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/resolve/main/Qwopus-MoE-35B-A3B-APEX-I-Mini.gguf +- name: qwopus3.6-35b-a3b-coder-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-I-Quality.gguf + sha256: b86b2398d38337b0485217d0be0a90ac32adec1d25ff9d2c4dfa84e1a826ebce + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/resolve/main/Qwopus3.6-35B-A3B-Coder-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/mmproj.gguf + sha256: 551d7a6187874b37f69a0c926e2aaad65df5c426a2e243b68f11bfe7e85b3d12 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwopus3.6-35b-a3b-coder-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-I-Balanced.gguf + sha256: 059faf65390b7341da46b60ebe2bc052601c87ee3b3395a8f28f321d87417c34 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/resolve/main/Qwopus3.6-35B-A3B-Coder-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/mmproj.gguf + sha256: 551d7a6187874b37f69a0c926e2aaad65df5c426a2e243b68f11bfe7e85b3d12 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwopus3.6-35b-a3b-coder-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-I-Compact.gguf + sha256: 16835025e2cfde01085527d804c0a4d6392ff1c08236bd086b7a9a99d43d1435 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/resolve/main/Qwopus3.6-35B-A3B-Coder-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/mmproj.gguf + sha256: 551d7a6187874b37f69a0c926e2aaad65df5c426a2e243b68f11bfe7e85b3d12 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwopus3.6-35b-a3b-coder-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-I-Mini.gguf + sha256: edf46cefe889fdb71e9af47f7908bc79315d656d78dbb5501da9db3fe70d3f48 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/resolve/main/Qwopus3.6-35B-A3B-Coder-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/mmproj.gguf + sha256: 551d7a6187874b37f69a0c926e2aaad65df5c426a2e243b68f11bfe7e85b3d12 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/resolve/main/mmproj.gguf +- name: qwopus3.6-35b-a3b-coder-apex-mtp-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/mmproj.gguf + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Quality.gguf + sha256: 83016f27e7eced4811f36c7dc73a07b34b8447fa7ccc5d31b0681993b28dc395 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/resolve/main/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/mmproj.gguf + sha256: 551d7a6187874b37f69a0c926e2aaad65df5c426a2e243b68f11bfe7e85b3d12 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/resolve/main/mmproj.gguf +- name: qwopus3.6-35b-a3b-coder-apex-mtp-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/mmproj.gguf + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Balanced.gguf + sha256: 63ac79da814cdeb131f3ad739787ae124ee0c58580fc9b110c3b757236820d04 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/resolve/main/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/mmproj.gguf + sha256: 551d7a6187874b37f69a0c926e2aaad65df5c426a2e243b68f11bfe7e85b3d12 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/resolve/main/mmproj.gguf +- name: qwopus3.6-35b-a3b-coder-apex-mtp-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/mmproj.gguf + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Compact.gguf + sha256: 22cc915b2e8f0f449e33db4a56cdbd45a464c9207e2940200337a16084f95815 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/resolve/main/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/mmproj.gguf + sha256: 551d7a6187874b37f69a0c926e2aaad65df5c426a2e243b68f11bfe7e85b3d12 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/resolve/main/mmproj.gguf +- name: qwopus3.6-35b-a3b-coder-apex-mtp-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/mmproj.gguf + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Mini.gguf + sha256: 9f4a27e173fa27e7b827e298d5058066c2e8ad94fd4d219574dacf15e1ae2845 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/resolve/main/Qwopus3.6-35B-A3B-Coder-APEX-MTP-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/mmproj.gguf + sha256: 551d7a6187874b37f69a0c926e2aaad65df5c426a2e243b68f11bfe7e85b3d12 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-MTP-GGUF/resolve/main/mmproj.gguf +- name: qwopus3.6-35b-a3b-v1-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/Qwopus3.6-35B-A3B-v1-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/Qwopus3.6-35B-A3B-v1-APEX-I-Quality.gguf + sha256: 2abdc3df406413b4c492d8189216d7914d6525652d2fe9f8fb6d56b156d7da70 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/resolve/main/Qwopus3.6-35B-A3B-v1-APEX-I-Quality.gguf +- name: qwopus3.6-35b-a3b-v1-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/Qwopus3.6-35B-A3B-v1-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/Qwopus3.6-35B-A3B-v1-APEX-I-Balanced.gguf + sha256: 91ccaee664d01f73c0e090b486084f02671e327eecf78133e729d3fa10c008f5 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/resolve/main/Qwopus3.6-35B-A3B-v1-APEX-I-Balanced.gguf +- name: qwopus3.6-35b-a3b-v1-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/Qwopus3.6-35B-A3B-v1-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/Qwopus3.6-35B-A3B-v1-APEX-I-Compact.gguf + sha256: 38dcb1fa39acb3284f3f9be780a9aa6ea0699b72d400a35671db34251d831320 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/resolve/main/Qwopus3.6-35B-A3B-v1-APEX-I-Compact.gguf +- name: qwopus3.6-35b-a3b-v1-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/Qwopus3.6-35B-A3B-v1-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/Qwopus3.6-35B-A3B-v1-APEX-I-Mini.gguf + sha256: 426127010859519ff190d0e6ea734e8212f7982c7a7cc23fe3a9f9f324b2cfd7 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-v1-APEX-GGUF/resolve/main/Qwopus3.6-35B-A3B-v1-APEX-I-Mini.gguf +- name: qwopus3.6-35b-a3b-v1-apex-mtp-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Quality.gguf + sha256: 1a777fcda83a60228fe4978ab5a1ce373d92bf38a5a58d98d3d0ffbb4f7f00a8 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/resolve/main/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Quality.gguf +- name: qwopus3.6-35b-a3b-v1-apex-mtp-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Balanced.gguf + sha256: 016e26ee859045e6e914ce2a844133c6707fc1314dd261ad19c1b8d32a668abf + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/resolve/main/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Balanced.gguf +- name: qwopus3.6-35b-a3b-v1-apex-mtp-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Compact.gguf + sha256: 595c68e7b1ed2b7fd070ca07c98c9fc830c0b7ec6e225578ff9aeb0355f3a9a6 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/resolve/main/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Compact.gguf +- name: qwopus3.6-35b-a3b-v1-apex-mtp-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Mini.gguf + sha256: b82e1ac06fb28783e638e2d9b63a9d8118cc0462bed3f542e1b57f465a09863c + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/resolve/main/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Mini.gguf +- name: qwopus3.6-35b-a3b-v1-apex-mtp-i-nano + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + - mtp + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + - spec_type:draft-mtp + - spec_n_max:6 + - spec_p_min:0.75 + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Nano.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Nano.gguf + sha256: ee6e4d5bc0cb3240f1d159d4eab86058196b0ac580aa19c2ddda3e766ee76c48 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-v1-APEX-MTP-GGUF/resolve/main/Qwopus3.6-35B-A3B-v1-APEX-MTP-I-Nano.gguf +- name: step-3.5-flash-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Step-3.5-Flash-APEX-GGUF/Step-3.5-Flash-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Step-3.5-Flash-APEX-GGUF/Step-3.5-Flash-APEX-I-Quality.gguf + sha256: e4fdf83cd3ad931ee0d0448104460c4685a0ee419002e6a7cdd217ff352a03c4 + uri: https://huggingface.co/mudler/Step-3.5-Flash-APEX-GGUF/resolve/main/Step-3.5-Flash-APEX-I-Quality.gguf +- name: step-3.5-flash-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Step-3.5-Flash-APEX-GGUF/Step-3.5-Flash-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Step-3.5-Flash-APEX-GGUF/Step-3.5-Flash-APEX-I-Balanced.gguf + sha256: bf57255e8825fafccf0d5a7d387a2e99df10bee1d053ed99cdb155793f41828f + uri: https://huggingface.co/mudler/Step-3.5-Flash-APEX-GGUF/resolve/main/Step-3.5-Flash-APEX-I-Balanced.gguf +- name: step-3.5-flash-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Step-3.5-Flash-APEX-GGUF/Step-3.5-Flash-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Step-3.5-Flash-APEX-GGUF/Step-3.5-Flash-APEX-I-Compact.gguf + sha256: 893161c2ec99cd8fe9d9491609fda311e0f7fc9a6f3cb98f525b5503e48852b8 + uri: https://huggingface.co/mudler/Step-3.5-Flash-APEX-GGUF/resolve/main/Step-3.5-Flash-APEX-I-Compact.gguf +- name: step-3.5-flash-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Step-3.5-Flash-APEX-GGUF/Step-3.5-Flash-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Step-3.5-Flash-APEX-GGUF/Step-3.5-Flash-APEX-I-Mini.gguf + sha256: 360f2f9d220d8eef5fa6a7b4f2537f5812e4dd604b3a45421dae62929f9cfb79 + uri: https://huggingface.co/mudler/Step-3.5-Flash-APEX-GGUF/resolve/main/Step-3.5-Flash-APEX-I-Mini.gguf +- name: step-3.7-flash-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Step-3.7-Flash-APEX-GGUF/mmproj-step3.7-flash-f16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Step-3.7-Flash-APEX-GGUF/Step-3.7-Flash-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Step-3.7-Flash-APEX-GGUF/Step-3.7-Flash-APEX-I-Quality.gguf + sha256: fe8db9753d4a0bfc0adb890510f3a7389d3fa79cf07af1c10bb1a37140da6e73 + uri: https://huggingface.co/mudler/Step-3.7-Flash-APEX-GGUF/resolve/main/Step-3.7-Flash-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Step-3.7-Flash-APEX-GGUF/mmproj-step3.7-flash-f16.gguf + sha256: 5f25d11f92235c69682ca820af5f4cb125ae1142c8c33c018d0b3c9000a2ec1c + uri: https://huggingface.co/mudler/Step-3.7-Flash-APEX-GGUF/resolve/main/mmproj-step3.7-flash-f16.gguf +- name: step-3.7-flash-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Step-3.7-Flash-APEX-GGUF/mmproj-step3.7-flash-f16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Step-3.7-Flash-APEX-GGUF/Step-3.7-Flash-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/Step-3.7-Flash-APEX-GGUF/Step-3.7-Flash-APEX-I-Balanced.gguf + sha256: 58a81b534d648440c7f6570fbd4d8ab88908d7603116baaea9de0f4a1d553201 + uri: https://huggingface.co/mudler/Step-3.7-Flash-APEX-GGUF/resolve/main/Step-3.7-Flash-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/Step-3.7-Flash-APEX-GGUF/mmproj-step3.7-flash-f16.gguf + sha256: 5f25d11f92235c69682ca820af5f4cb125ae1142c8c33c018d0b3c9000a2ec1c + uri: https://huggingface.co/mudler/Step-3.7-Flash-APEX-GGUF/resolve/main/mmproj-step3.7-flash-f16.gguf +- name: step-3.7-flash-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Step-3.7-Flash-APEX-GGUF/mmproj-step3.7-flash-f16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Step-3.7-Flash-APEX-GGUF/Step-3.7-Flash-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/Step-3.7-Flash-APEX-GGUF/Step-3.7-Flash-APEX-I-Compact.gguf + sha256: 16acaffa8d6c276db2d11afeac3a6cfdb6d25e1e353d792ef49e09c627c3a681 + uri: https://huggingface.co/mudler/Step-3.7-Flash-APEX-GGUF/resolve/main/Step-3.7-Flash-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/Step-3.7-Flash-APEX-GGUF/mmproj-step3.7-flash-f16.gguf + sha256: 5f25d11f92235c69682ca820af5f4cb125ae1142c8c33c018d0b3c9000a2ec1c + uri: https://huggingface.co/mudler/Step-3.7-Flash-APEX-GGUF/resolve/main/mmproj-step3.7-flash-f16.gguf +- name: step-3.7-flash-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Step-3.7-Flash-APEX-GGUF/mmproj-step3.7-flash-f16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Step-3.7-Flash-APEX-GGUF/Step-3.7-Flash-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/Step-3.7-Flash-APEX-GGUF/Step-3.7-Flash-APEX-I-Mini.gguf + sha256: 946033b1f38d3661561978857aee0ea1fce4e575d2120d728f0e9914bb703d5e + uri: https://huggingface.co/mudler/Step-3.7-Flash-APEX-GGUF/resolve/main/Step-3.7-Flash-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/Step-3.7-Flash-APEX-GGUF/mmproj-step3.7-flash-f16.gguf + sha256: 5f25d11f92235c69682ca820af5f4cb125ae1142c8c33c018d0b3c9000a2ec1c + uri: https://huggingface.co/mudler/Step-3.7-Flash-APEX-GGUF/resolve/main/mmproj-step3.7-flash-f16.gguf +- name: step-3.7-flash-ud-q5-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Step-3.7-Flash-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/UD-Q5_K_M/Step-3.7-Flash-UD-Q5_K_M-00001-of-00004.gguf + files: + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/UD-Q5_K_M/Step-3.7-Flash-UD-Q5_K_M-00001-of-00004.gguf + sha256: 87a5ae1d4ec56045181d6cbbb7991d909e300cde4717d6e6504bbdd985a3f19d + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q5_K_M/Step-3.7-Flash-UD-Q5_K_M-00001-of-00004.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/UD-Q5_K_M/Step-3.7-Flash-UD-Q5_K_M-00002-of-00004.gguf + sha256: 85830c607c29e5d57978c5d3440726eccb44d3e04b7dab61804951648dca6165 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q5_K_M/Step-3.7-Flash-UD-Q5_K_M-00002-of-00004.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/UD-Q5_K_M/Step-3.7-Flash-UD-Q5_K_M-00003-of-00004.gguf + sha256: 059509555257fb0e7aab79578c6d76faaa052acc5a9d0ec254851f848ce09ba8 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q5_K_M/Step-3.7-Flash-UD-Q5_K_M-00003-of-00004.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/UD-Q5_K_M/Step-3.7-Flash-UD-Q5_K_M-00004-of-00004.gguf + sha256: cd8327dde4818ee116d6199a9835f0af7cfca5482fde4877af4f3fe1f34e79cf + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q5_K_M/Step-3.7-Flash-UD-Q5_K_M-00004-of-00004.gguf + - filename: llama-cpp/mmproj/unsloth/Step-3.7-Flash-GGUF/mmproj-BF16.gguf + sha256: 1425406962c8d89eb09caf9116a86fa02fa6710a8ab529d852976bb5b3120250 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/mmproj-BF16.gguf +- name: step-3.7-flash-ud-q6-k + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Step-3.7-Flash-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/UD-Q6_K/Step-3.7-Flash-UD-Q6_K-00001-of-00005.gguf + files: + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/UD-Q6_K/Step-3.7-Flash-UD-Q6_K-00001-of-00005.gguf + sha256: 8d64916e4c34dfffdbd30f788d90c0782eeb20979330eb642e0c2928b9f4250d + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q6_K/Step-3.7-Flash-UD-Q6_K-00001-of-00005.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/UD-Q6_K/Step-3.7-Flash-UD-Q6_K-00002-of-00005.gguf + sha256: 1a403920bcff976dd9577e88613799150dfe4f8e7ff95b78feaf91cc403d870f + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q6_K/Step-3.7-Flash-UD-Q6_K-00002-of-00005.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/UD-Q6_K/Step-3.7-Flash-UD-Q6_K-00003-of-00005.gguf + sha256: 4431aacbf3a204c9dd4fc86d4ed74a54fdaaeefebb442936ae2f91b29741e7dd + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q6_K/Step-3.7-Flash-UD-Q6_K-00003-of-00005.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/UD-Q6_K/Step-3.7-Flash-UD-Q6_K-00004-of-00005.gguf + sha256: 5b4a26c65680156b6e12f42e2bd1967573d7bea028d1f0d4efe44361daf48aa5 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q6_K/Step-3.7-Flash-UD-Q6_K-00004-of-00005.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/UD-Q6_K/Step-3.7-Flash-UD-Q6_K-00005-of-00005.gguf + sha256: 5c5127950a197a4faa88ca1b65cf1abe929cdba6b2c81d28208bca7cc2f024f1 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/UD-Q6_K/Step-3.7-Flash-UD-Q6_K-00005-of-00005.gguf + - filename: llama-cpp/mmproj/unsloth/Step-3.7-Flash-GGUF/mmproj-BF16.gguf + sha256: 1425406962c8d89eb09caf9116a86fa02fa6710a8ab529d852976bb5b3120250 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/mmproj-BF16.gguf +- name: step-3.7-flash-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/Step-3.7-Flash-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/Q8_0/Step-3.7-Flash-Q8_0-00001-of-00006.gguf + files: + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/Q8_0/Step-3.7-Flash-Q8_0-00001-of-00006.gguf + sha256: 66e8a389e5dd943108f40fafca8970fd7edb76a96609550252839d0a67db9eb6 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/Q8_0/Step-3.7-Flash-Q8_0-00001-of-00006.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/Q8_0/Step-3.7-Flash-Q8_0-00002-of-00006.gguf + sha256: e273dd4e1485898ec30c9dfead623db323d18820fa61fe0483e0fa54191bf2e4 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/Q8_0/Step-3.7-Flash-Q8_0-00002-of-00006.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/Q8_0/Step-3.7-Flash-Q8_0-00003-of-00006.gguf + sha256: d0b131501f4e5c1d8b24c0e88d4ec9b7f46e497ccdf9320625df7afc888b6ada + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/Q8_0/Step-3.7-Flash-Q8_0-00003-of-00006.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/Q8_0/Step-3.7-Flash-Q8_0-00004-of-00006.gguf + sha256: e369492d48d4da4b1f132b26fb4db0b43e3a2e3a7f58f0d7990891fb99eed052 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/Q8_0/Step-3.7-Flash-Q8_0-00004-of-00006.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/Q8_0/Step-3.7-Flash-Q8_0-00005-of-00006.gguf + sha256: d74c1f9da8085204dc5f6899453d16777cc3ffe017565510316a8bf4f3a7a271 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/Q8_0/Step-3.7-Flash-Q8_0-00005-of-00006.gguf + - filename: llama-cpp/models/unsloth/Step-3.7-Flash-GGUF/Q8_0/Step-3.7-Flash-Q8_0-00006-of-00006.gguf + sha256: a65aea5c133cffce3afd9e76d41ab1de5c6862856aee566309f42a2d2134b7a9 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/Q8_0/Step-3.7-Flash-Q8_0-00006-of-00006.gguf + - filename: llama-cpp/mmproj/unsloth/Step-3.7-Flash-GGUF/mmproj-BF16.gguf + sha256: 1425406962c8d89eb09caf9116a86fa02fa6710a8ab529d852976bb5b3120250 + uri: https://huggingface.co/unsloth/Step-3.7-Flash-GGUF/resolve/main/mmproj-BF16.gguf +- name: gemma-4-26b-a4b-it-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-APEX-GGUF/gemma-4-26B-A4B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-APEX-GGUF/gemma-4-26B-A4B-APEX-I-Quality.gguf + sha256: 472828ccd00bcf52d6ca72e97d49526fd254371a90ae6a77505409e6e2bf3304 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-APEX-GGUF/resolve/main/gemma-4-26B-A4B-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-APEX-GGUF/mmproj-F16.gguf + sha256: cfc8dc4e41ab1d0c4846ed63ba4a62186846b04eb25fb38e1f2555ce2d00cb26 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: gemma-4-26b-a4b-it-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-APEX-GGUF/gemma-4-26B-A4B-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-APEX-GGUF/gemma-4-26B-A4B-APEX-I-Balanced.gguf + sha256: 0640553a14cd6010b285288e0d91d7c34811579a3e3ef0f7369e5c645e8e41b6 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-APEX-GGUF/resolve/main/gemma-4-26B-A4B-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-APEX-GGUF/mmproj-F16.gguf + sha256: cfc8dc4e41ab1d0c4846ed63ba4a62186846b04eb25fb38e1f2555ce2d00cb26 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: gemma-4-26b-a4b-it-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-APEX-GGUF/gemma-4-26B-A4B-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-APEX-GGUF/gemma-4-26B-A4B-APEX-I-Compact.gguf + sha256: 6af4e64b2060af7afa01aa44863e6192a215ee43379bd75bab4688e1f95d4212 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-APEX-GGUF/resolve/main/gemma-4-26B-A4B-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-APEX-GGUF/mmproj-F16.gguf + sha256: cfc8dc4e41ab1d0c4846ed63ba4a62186846b04eb25fb38e1f2555ce2d00cb26 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: gemma-4-26b-a4b-it-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-APEX-GGUF/gemma-4-26B-A4B-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-APEX-GGUF/gemma-4-26B-A4B-APEX-I-Mini.gguf + sha256: 1b0b1d3986e5213f291f04edb5910f33c975352ab1ab06b717ef3753da8e6b7a + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-APEX-GGUF/resolve/main/gemma-4-26B-A4B-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-APEX-GGUF/mmproj-F16.gguf + sha256: cfc8dc4e41ab1d0c4846ed63ba4a62186846b04eb25fb38e1f2555ce2d00cb26 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: gemma-4-26b-a4b-it-apex-i-nano + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-APEX-GGUF/gemma-4-26B-A4B-APEX-I-Nano.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-APEX-GGUF/gemma-4-26B-A4B-APEX-I-Nano.gguf + sha256: 2629f050b2ed313b40b7290d3fda64dbd4ab948ccba55db4a5023ac7ed87d351 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-APEX-GGUF/resolve/main/gemma-4-26B-A4B-APEX-I-Nano.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-APEX-GGUF/mmproj-F16.gguf + sha256: cfc8dc4e41ab1d0c4846ed63ba4a62186846b04eb25fb38e1f2555ce2d00cb26 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-APEX-GGUF/resolve/main/mmproj-F16.gguf +- name: gemma-4-26b-a4b-it-ud-q4-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf + files: + - filename: llama-cpp/models/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf + sha256: f2c28b3dc4776931ac6f879e11f203dec637ea0f14267a86ec8f6165f63f293f + uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/resolve/main/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf + - filename: llama-cpp/mmproj/unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf + sha256: 41926ed5f1403cf5add23b0684992805ea6f97253096132e769e65646b8cef9d + uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/resolve/main/mmproj-BF16.gguf +- name: gemma-4-26b-a4b-it-ud-q5-k-m + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q5_K_M.gguf + files: + - filename: llama-cpp/models/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q5_K_M.gguf + sha256: 769d386a69d43782321c1bad04d41d29a2e84b2c06e6a277cd99fd6265ec0e80 + uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/resolve/main/gemma-4-26B-A4B-it-UD-Q5_K_M.gguf + - filename: llama-cpp/mmproj/unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf + sha256: 41926ed5f1403cf5add23b0684992805ea6f97253096132e769e65646b8cef9d + uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/resolve/main/mmproj-BF16.gguf +- name: gemma-4-26b-a4b-it-ud-q6-k + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q6_K.gguf + files: + - filename: llama-cpp/models/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-UD-Q6_K.gguf + sha256: 76bf3eee49cca5b076b34a4b575ecd505dabc470512b46f1207b4f9da08385f0 + uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/resolve/main/gemma-4-26B-A4B-it-UD-Q6_K.gguf + - filename: llama-cpp/mmproj/unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf + sha256: 41926ed5f1403cf5add23b0684992805ea6f97253096132e769e65646b8cef9d + uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/resolve/main/mmproj-BF16.gguf +- name: gemma-4-26b-a4b-it-q8-0 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-Q8_0.gguf + files: + - filename: llama-cpp/models/unsloth/gemma-4-26B-A4B-it-GGUF/gemma-4-26B-A4B-it-Q8_0.gguf + sha256: 5f7cbd0f4564e84342fc34321a09acb54b1a3da9215124e5bf444baa6dda152c + uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/resolve/main/gemma-4-26B-A4B-it-Q8_0.gguf + - filename: llama-cpp/mmproj/unsloth/gemma-4-26B-A4B-it-GGUF/mmproj-BF16.gguf + sha256: 41926ed5f1403cf5add23b0684992805ea6f97253096132e769e65646b8cef9d + uri: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/resolve/main/mmproj-BF16.gguf +- name: gemma-4-26b-a4b-it-claude-opus-distill-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/gemma-4-26B-A4B-Claude-Distill-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/gemma-4-26B-A4B-Claude-Distill-APEX-I-Quality.gguf + sha256: 4b5a331ce4edaf1b286f269eb6db8d878ebd0c56fa9e0635be505c87867b673e + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/resolve/main/gemma-4-26B-A4B-Claude-Distill-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/mmproj.gguf + sha256: fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/resolve/main/mmproj.gguf +- name: gemma-4-26b-a4b-it-claude-opus-distill-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/gemma-4-26B-A4B-Claude-Distill-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/gemma-4-26B-A4B-Claude-Distill-APEX-I-Balanced.gguf + sha256: bf9377ea45ee44e920565be56326b2c356f00173ed8597260c32453f1fb9b935 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/resolve/main/gemma-4-26B-A4B-Claude-Distill-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/mmproj.gguf + sha256: fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/resolve/main/mmproj.gguf +- name: gemma-4-26b-a4b-it-claude-opus-distill-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/gemma-4-26B-A4B-Claude-Distill-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/gemma-4-26B-A4B-Claude-Distill-APEX-I-Compact.gguf + sha256: 05ae030d1a227ec8c3b438baa2e4a5916364ab7689399017b6954f55818108a0 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/resolve/main/gemma-4-26B-A4B-Claude-Distill-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/mmproj.gguf + sha256: fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/resolve/main/mmproj.gguf +- name: gemma-4-26b-a4b-it-claude-opus-distill-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/gemma-4-26B-A4B-Claude-Distill-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/gemma-4-26B-A4B-Claude-Distill-APEX-I-Mini.gguf + sha256: 15263c2222c6f202a114c1eab40cffb4de56c33932e5db292ccbd937b3179218 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/resolve/main/gemma-4-26B-A4B-Claude-Distill-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/mmproj.gguf + sha256: fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/resolve/main/mmproj.gguf +- name: gemma-4-26b-a4b-it-heretic-apex-i-quality + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/gemma-4-26B-A4B-heretic-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/gemma-4-26B-A4B-heretic-APEX-I-Quality.gguf + sha256: 2def9529490558630809bdb1c9f3b82c6b8567409c850bbf4a3eeb5c35986de6 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/resolve/main/gemma-4-26B-A4B-heretic-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/mmproj.gguf + sha256: fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/resolve/main/mmproj.gguf +- name: gemma-4-26b-a4b-it-heretic-apex-i-balanced + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/gemma-4-26B-A4B-heretic-APEX-I-Balanced.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/gemma-4-26B-A4B-heretic-APEX-I-Balanced.gguf + sha256: 964d8ce13421263198c06eff327e09c2533213605b298e8404f084a7103cc40b + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/resolve/main/gemma-4-26B-A4B-heretic-APEX-I-Balanced.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/mmproj.gguf + sha256: fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/resolve/main/mmproj.gguf +- name: gemma-4-26b-a4b-it-heretic-apex-i-compact + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/gemma-4-26B-A4B-heretic-APEX-I-Compact.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/gemma-4-26B-A4B-heretic-APEX-I-Compact.gguf + sha256: ff2e7a1f7949e80ec011dd60b32a0659272fbe7571090754e87f6ca042293c67 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/resolve/main/gemma-4-26B-A4B-heretic-APEX-I-Compact.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/mmproj.gguf + sha256: fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/resolve/main/mmproj.gguf +- name: gemma-4-26b-a4b-it-heretic-apex-i-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/gemma-4-26B-A4B-heretic-APEX-I-Mini.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/gemma-4-26B-A4B-heretic-APEX-I-Mini.gguf + sha256: 1dee90dc11557a249fd5799886c69a583fbcfd440ac6391dae569f94fd499f1c + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/resolve/main/gemma-4-26B-A4B-heretic-APEX-I-Mini.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/mmproj.gguf + sha256: fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/resolve/main/mmproj.gguf +- name: agents-a1 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Agents-A1. Quality ladder and quantization rungs published by mudler/Agents-A1-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + - vision + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Agents-A1-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Agents-A1-APEX-GGUF/Agents-A1-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Agents-A1-APEX-GGUF/Agents-A1-APEX-I-Quality.gguf + sha256: c1bf8c3dcf08060f5d525d2026f6905894f245188593e8a53e2597590d30e52e + uri: https://huggingface.co/mudler/Agents-A1-APEX-GGUF/resolve/main/Agents-A1-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Agents-A1-APEX-GGUF/mmproj.gguf + sha256: c8772fe61cfceefd1bc42e3b06f53bb2e751abd15d1450d58ba0c113d3782396 + uri: https://huggingface.co/mudler/Agents-A1-APEX-GGUF/resolve/main/mmproj.gguf + variants: + - model: agents-a1-apex-i-quality + - model: agents-a1-apex-i-balanced + - model: agents-a1-apex-i-compact + - model: agents-a1-apex-i-mini +- name: carnice-moe-35b-a3b + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Carnice-MoE-35B-A3B. Quality ladder and quantization rungs published by mudler/Carnice-MoE-35B-A3B-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/Carnice-MoE-35B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/Carnice-MoE-35B-A3B-APEX-I-Quality.gguf + sha256: 5e4d97368f638edf06fd42014994f8a8e40e73ea42d960f00b602f0aa01a0e78 + uri: https://huggingface.co/mudler/Carnice-MoE-35B-A3B-APEX-GGUF/resolve/main/Carnice-MoE-35B-A3B-APEX-I-Quality.gguf + variants: + - model: carnice-moe-35b-a3b-apex-i-quality + - model: carnice-moe-35b-a3b-apex-i-balanced + - model: carnice-moe-35b-a3b-apex-i-compact + - model: carnice-moe-35b-a3b-apex-i-mini +- name: carnice-qwen3.6-moe-35b-a3b + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Carnice-Qwen3.6-MoE-35B-A3B. Quality ladder and quantization rungs published by mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Quality.gguf + sha256: f1fef05db8f1fd8d6b346320af393e64ab91c837ec621c09f5bcf99ece837876 + uri: https://huggingface.co/mudler/Carnice-Qwen3.6-MoE-35B-A3B-APEX-GGUF/resolve/main/Carnice-Qwen3.6-MoE-35B-A3B-APEX-I-Quality.gguf + variants: + - model: carnice-qwen3.6-moe-35b-a3b-apex-i-quality + - model: carnice-qwen3.6-moe-35b-a3b-apex-i-balanced + - model: carnice-qwen3.6-moe-35b-a3b-apex-i-compact + - model: carnice-qwen3.6-moe-35b-a3b-apex-i-mini + - model: carnice-qwen3.6-moe-35b-a3b-apex-mtp-i-quality + - model: carnice-qwen3.6-moe-35b-a3b-apex-mtp-i-balanced + - model: carnice-qwen3.6-moe-35b-a3b-apex-mtp-i-compact + - model: carnice-qwen3.6-moe-35b-a3b-apex-mtp-i-mini + - model: carnice-qwen3.6-moe-35b-a3b-apex-mtp-i-nano +- name: darwin-36b-opus + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Darwin-36B-Opus. Quality ladder and quantization rungs published by mudler/Darwin-36B-Opus-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Darwin-36B-Opus-APEX-GGUF/Darwin-36B-Opus-APEX-I-Quality.gguf + sha256: aee6e3629b68193680444f5cec3c288348c4b46047af3e5f9ff7d8dd3633df80 + uri: https://huggingface.co/mudler/Darwin-36B-Opus-APEX-GGUF/resolve/main/Darwin-36B-Opus-APEX-I-Quality.gguf + variants: + - model: darwin-36b-opus-apex-i-quality + - model: darwin-36b-opus-apex-i-balanced + - model: darwin-36b-opus-apex-i-compact + - model: darwin-36b-opus-apex-i-mini + - model: darwin-36b-opus-apex-i-nano +- name: gemopus-4-26b-a4b-it-preview + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Gemopus-4-26B-A4B-it-Preview. Quality ladder and quantization rungs published by mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/gemopus-4-26B-A4B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/gemopus-4-26B-A4B-APEX-I-Quality.gguf + sha256: 839f2a4df38758ad7db1bdf3e1bd06398429b119cd0fe2cc3dde1f5d133c766c + uri: https://huggingface.co/mudler/Gemopus-4-26B-A4B-it-Preview-APEX-GGUF/resolve/main/gemopus-4-26B-A4B-APEX-I-Quality.gguf + variants: + - model: gemopus-4-26b-a4b-it-preview-apex-i-quality + - model: gemopus-4-26b-a4b-it-preview-apex-i-balanced + - model: gemopus-4-26b-a4b-it-preview-apex-i-compact + - model: gemopus-4-26b-a4b-it-preview-apex-i-mini +- name: holo3-35b-a3b + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Holo3-35B-A3B. Quality ladder and quantization rungs published by mudler/Holo3-35B-A3B-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + - vision + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Holo3-35B-A3B-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Holo3-35B-A3B-APEX-GGUF/Holo3-35B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Holo3-35B-A3B-APEX-GGUF/Holo3-35B-A3B-APEX-I-Quality.gguf + sha256: 57ce61073d6f11798c8644bc04fbbd31397e9c2647aa911202ead521bcef8f93 + uri: https://huggingface.co/mudler/Holo3-35B-A3B-APEX-GGUF/resolve/main/Holo3-35B-A3B-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Holo3-35B-A3B-APEX-GGUF/mmproj.gguf + sha256: cd734ca69ed59fd46cf010407f2ddba387db4e5696f21b67c8779f26264b9020 + uri: https://huggingface.co/mudler/Holo3-35B-A3B-APEX-GGUF/resolve/main/mmproj.gguf + variants: + - model: holo3-35b-a3b-apex-i-quality + - model: holo3-35b-a3b-apex-i-balanced + - model: holo3-35b-a3b-apex-i-compact + - model: holo3-35b-a3b-apex-i-mini +- name: huihui3.5-67b-a3b + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Huihui3.5-67B-A3B. Quality ladder and quantization rungs published by mudler/Huihui3.5-67B-A3B-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Huihui3.5-67B-A3B-APEX-GGUF/Huihui3.5-67B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Huihui3.5-67B-A3B-APEX-GGUF/Huihui3.5-67B-A3B-APEX-I-Quality.gguf + sha256: 48755ebca7650a944ceb520da00ff9e77c97e52ec78b83ac9c428deba7fae6ed + uri: https://huggingface.co/mudler/Huihui3.5-67B-A3B-APEX-GGUF/resolve/main/Huihui3.5-67B-A3B-APEX-I-Quality.gguf + variants: + - model: huihui3.5-67b-a3b-apex-i-quality + - model: huihui3.5-67b-a3b-apex-i-balanced + - model: huihui3.5-67b-a3b-apex-i-compact + - model: huihui3.5-67b-a3b-apex-i-mini +- name: lfm2-24b-a2b + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: LFM2-24B-A2B. Quality ladder and quantization rungs published by mudler/LFM2-24B-A2B-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/LFM2-24B-A2B-APEX-GGUF/LFM2-24B-A2B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/LFM2-24B-A2B-APEX-GGUF/LFM2-24B-A2B-APEX-I-Quality.gguf + sha256: 0313aa47547d302ffd88c538d381145888b5b2bd739cf5058c8cdda81268992d + uri: https://huggingface.co/mudler/LFM2-24B-A2B-APEX-GGUF/resolve/main/LFM2-24B-A2B-APEX-I-Quality.gguf + variants: + - model: lfm2-24b-a2b-apex-i-quality + - model: lfm2-24b-a2b-apex-i-balanced + - model: lfm2-24b-a2b-apex-i-compact + - model: lfm2-24b-a2b-apex-i-mini +- name: minimax-m2.5 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: MiniMax-M2.5. Quality ladder and quantization rungs published by mudler/MiniMax-M2.5-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/MiniMax-M2.5-APEX-GGUF/MiniMax-M2.5-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/MiniMax-M2.5-APEX-GGUF/MiniMax-M2.5-APEX-I-Quality.gguf + sha256: 54b4e8c0185aa6b58857dd7d64173e2a8760f6e4a144a45c906211d73a6f7eb8 + uri: https://huggingface.co/mudler/MiniMax-M2.5-APEX-GGUF/resolve/main/MiniMax-M2.5-APEX-I-Quality.gguf + variants: + - model: minimax-m2.5-apex-i-quality + - model: minimax-m2.5-apex-i-balanced + - model: minimax-m2.5-apex-i-compact + - model: minimax-m2.5-apex-i-mini + - model: minimax-m2.5-q8-0 +- name: minimax-m2.7 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: MiniMax-M2.7. Quality ladder and quantization rungs published by mudler/MiniMax-M2.7-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/MiniMax-M2.7-APEX-GGUF/MiniMax-M2.7-APEX-I-Quality.gguf + sha256: 1270f5ce024ff1444df43294c86124a7053cd7f4681b4ac7719f2c3a49834405 + uri: https://huggingface.co/mudler/MiniMax-M2.7-APEX-GGUF/resolve/main/MiniMax-M2.7-APEX-I-Quality.gguf + variants: + - model: minimax-m2.7-apex-i-quality + - model: minimax-m2.7-apex-i-balanced + - model: minimax-m2.7-apex-i-compact + - model: minimax-m2.7-apex-i-mini + - model: minimax-m2.7-apex-i-nano + - model: minimax-m2.7-ud-q4-k-m + - model: minimax-m2.7-ud-q5-k-m + - model: minimax-m2.7-ud-q6-k + - model: minimax-m2.7-q8-0 +- name: mistral-small-4-119b-2603 + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Mistral-Small-4-119B-2603. Quality ladder and quantization rungs published by mudler/Mistral-Small-4-119B-2603-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + - vision + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/Mistral-Small-4-119B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/Mistral-Small-4-119B-APEX-I-Quality.gguf + sha256: 2832cb7ef12b934eebd9baec0dfcde3807cd5101237bdfc33f2b265e6283c860 + uri: https://huggingface.co/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/resolve/main/Mistral-Small-4-119B-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/mmproj.gguf + sha256: 2f1f27544c4b4074b28edb8413af92a9111d7166bbc0944fd27db56adc054d0b + uri: https://huggingface.co/mudler/Mistral-Small-4-119B-2603-APEX-GGUF/resolve/main/mmproj.gguf + variants: + - model: mistral-small-4-119b-2603-apex-i-quality + - model: mistral-small-4-119b-2603-apex-i-balanced + - model: mistral-small-4-119b-2603-apex-i-compact + - model: mistral-small-4-119b-2603-apex-i-mini + - model: mistral-small-4-119b-2603-ud-q4-k-m + - model: mistral-small-4-119b-2603-ud-q5-k-m + - model: mistral-small-4-119b-2603-ud-q6-k + - model: mistral-small-4-119b-2603-q8-0 +- name: nvidia-nemotron-3-super-120b-a12b + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: NVIDIA-Nemotron-3-Super-120B-A12B. Quality ladder and quantization rungs published by mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/Nemotron-3-Super-120B-A12B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/Nemotron-3-Super-120B-A12B-APEX-I-Quality.gguf + sha256: d6eeef1836655ca3e04ddd851f22feb1dd23edb2599ac50db8326fcdcd21cd14 + uri: https://huggingface.co/mudler/NVIDIA-Nemotron-3-Super-120B-A12B-APEX-GGUF/resolve/main/Nemotron-3-Super-120B-A12B-APEX-I-Quality.gguf + variants: + - model: nvidia-nemotron-3-super-120b-a12b-apex-i-quality + - model: nvidia-nemotron-3-super-120b-a12b-apex-i-balanced + - model: nvidia-nemotron-3-super-120b-a12b-apex-i-compact + - model: nvidia-nemotron-3-super-120b-a12b-apex-i-mini + - model: nvidia-nemotron-3-super-120b-a12b-ud-q4-k-m + - model: nvidia-nemotron-3-super-120b-a12b-ud-q5-k-m + - model: nvidia-nemotron-3-super-120b-a12b-ud-q6-k + - model: nvidia-nemotron-3-super-120b-a12b-q8-0 +- name: nemotron-3-nano-30b-a3b + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Nemotron-3-Nano-30B-A3B. Quality ladder and quantization rungs published by mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/Nemotron-3-Nano-30B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/Nemotron-3-Nano-30B-A3B-APEX-I-Quality.gguf + sha256: 649ccc24aa75282cebf361638f149b056a6a9d022c574aa916d6b01f9296484e + uri: https://huggingface.co/mudler/Nemotron-3-Nano-30B-A3B-APEX-GGUF/resolve/main/Nemotron-3-Nano-30B-A3B-APEX-I-Quality.gguf + variants: + - model: nemotron-3-nano-30b-a3b-apex-i-quality + - model: nemotron-3-nano-30b-a3b-apex-i-balanced + - model: nemotron-3-nano-30b-a3b-apex-i-compact + - model: nemotron-3-nano-30b-a3b-apex-i-mini + - model: nemotron-3-nano-30b-a3b-q8-0 +- name: nemotron-3-nano-omni-30b-a3b-reasoning + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Nemotron-3-Nano-Omni-30B-A3B-Reasoning. Quality ladder and quantization rungs published by mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + - vision + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Quality.gguf + sha256: aa1734147522810d52c6c59f269301eed98d34e50438475ab5544d7a6ee1821a + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/mmproj.gguf + sha256: 8e72b13d61ac5972793b94e130e69d93ebbc96bc8e281500c03e762282b1840f + uri: https://huggingface.co/mudler/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-APEX-GGUF/resolve/main/mmproj.gguf + variants: + - model: nemotron-3-nano-omni-30b-a3b-reasoning-apex + - model: nemotron-3-nano-omni-30b-a3b-reasoning-apex-i-quality + - model: nemotron-3-nano-omni-30b-a3b-reasoning-apex-i-balanced + - model: nemotron-3-nano-omni-30b-a3b-reasoning-apex-i-compact + - model: nemotron-3-nano-omni-30b-a3b-reasoning-apex-i-mini + - model: nemotron-3-nano-omni-30b-a3b-reasoning-apex-i-nano +- name: nex-n2-mini + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Nex-N2-mini. Quality ladder and quantization rungs published by mudler/Nex-N2-mini-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + - vision + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Nex-N2-mini-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Nex-N2-mini-APEX-GGUF/Nex-N2-mini-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Nex-N2-mini-APEX-GGUF/Nex-N2-mini-APEX-I-Quality.gguf + sha256: 7f8895584cb494f7331cd81e67e898ab68100b94aed9725cf13653b1df9db8b1 + uri: https://huggingface.co/mudler/Nex-N2-mini-APEX-GGUF/resolve/main/Nex-N2-mini-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Nex-N2-mini-APEX-GGUF/mmproj.gguf + sha256: 571ad1e826caa2eb5ff37fdf3ae30b5072edc694d8ebd25897166660ed981469 + uri: https://huggingface.co/mudler/Nex-N2-mini-APEX-GGUF/resolve/main/mmproj.gguf + variants: + - model: nex-n2-mini-apex-i-quality + - model: nex-n2-mini-apex-i-balanced + - model: nex-n2-mini-apex-i-compact + - model: nex-n2-mini-apex-i-mini +- name: qwen3-coder-30b + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Qwen3-Coder-30B. Quality ladder and quantization rungs published by mudler/Qwen3-Coder-30B-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3-Coder-30B-APEX-GGUF/Qwen3-Coder-30B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3-Coder-30B-APEX-GGUF/Qwen3-Coder-30B-APEX-I-Quality.gguf + sha256: bf4d49e6f2e526436a29a9481aab7d8dc97987a9e446e2d395a83a0f12bd7134 + uri: https://huggingface.co/mudler/Qwen3-Coder-30B-APEX-GGUF/resolve/main/Qwen3-Coder-30B-APEX-I-Quality.gguf + variants: + - model: qwen3-coder-30b-apex-i-quality + - model: qwen3-coder-30b-apex-i-balanced + - model: qwen3-coder-30b-apex-i-compact +- name: qwen3-coder-next + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Qwen3-Coder-Next. Quality ladder and quantization rungs published by mudler/Qwen3-Coder-Next-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3-Coder-Next-APEX-GGUF/Qwen3-Coder-Next-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3-Coder-Next-APEX-GGUF/Qwen3-Coder-Next-APEX-I-Quality.gguf + sha256: a31e629119c0dea4d294cc3ed230e6aac72e552f36ba09218d71d82340b5b755 + uri: https://huggingface.co/mudler/Qwen3-Coder-Next-APEX-GGUF/resolve/main/Qwen3-Coder-Next-APEX-I-Quality.gguf + variants: + - model: qwen3-coder-next-apex-i-quality + - model: qwen3-coder-next-apex-i-balanced + - model: qwen3-coder-next-apex-i-compact + - model: qwen3-coder-next-apex-i-mini + - model: qwen3-coder-next-ud-q4-k-m + - model: qwen3-coder-next-ud-q5-k-m + - model: qwen3-coder-next-ud-q6-k + - model: qwen3-coder-next-q8-0 +- name: qwen3.5-35b-a3b + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Qwen3.5-35B-A3B. Quality ladder and quantization rungs published by mudler/Qwen3.5-35B-A3B-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + - vision + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.5-35B-A3B-APEX-GGUF/mmproj-F16.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-GGUF/Qwen3.5-35B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-APEX-GGUF/Qwen3.5-35B-A3B-APEX-I-Quality.gguf + sha256: 5e0fa27a9b0841ca8fffc039c6a2e8d933f1d0be12369a14fd4abce090103f20 + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.5-35B-A3B-APEX-GGUF/mmproj-F16.gguf + sha256: a516ab92e8240da4734d68352bdfba84c16e830ee40010b8fac80d69c77272ff + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-GGUF/resolve/main/mmproj-F16.gguf + variants: + - model: qwen3.5-35b-a3b-apex + - model: qwen3.5-35b-a3b-apex-i-quality + - model: qwen3.5-35b-a3b-apex-i-balanced + - model: qwen3.5-35b-a3b-apex-i-compact + - model: qwen3.5-35b-a3b-q8-0 + - model: qwen3.5-35b-a3b-apex-tq-quality + - model: qwen3.5-35b-a3b-apex-tq-balanced + - model: qwen3.5-35b-a3b-apex-tq-compact +- name: qwen3.5-35b-a3b-claude-distilled + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Qwen3.5-35B-A3B-Claude-Distilled. Quality ladder and quantization rungs published by mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Quality.gguf + sha256: 99ebd4c27e6957d75396fdccec79cf26f332ffd9503e95751785ed5352e54981 + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-Claude-Distilled-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-Claude-Distilled-APEX-I-Quality.gguf + variants: + - model: qwen3.5-35b-a3b-claude-distilled-apex-i-quality + - model: qwen3.5-35b-a3b-claude-distilled-apex-i-balanced + - model: qwen3.5-35b-a3b-claude-distilled-apex-i-compact + - model: qwen3.5-35b-a3b-claude-distilled-apex-i-mini +- name: qwen3.5-35b-a3b-uncensored-fernflowerai + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Qwen3.5-35B-A3B-Uncensored-FernflowerAI. Quality ladder and quantization rungs published by mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Quality.gguf + sha256: 6b145382be701da5cdc2f6c630a3543c47e54a114c160da5357d6b3ad2abf7ea + uri: https://huggingface.co/mudler/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-GGUF/resolve/main/Qwen3.5-35B-A3B-Uncensored-FernflowerAI-APEX-I-Quality.gguf + variants: + - model: qwen3.5-35b-a3b-uncensored-fernflowerai-apex-i-quality + - model: qwen3.5-35b-a3b-uncensored-fernflowerai-apex-i-balanced + - model: qwen3.5-35b-a3b-uncensored-fernflowerai-apex-i-compact + - model: qwen3.5-35b-a3b-uncensored-fernflowerai-apex-i-mini + - model: qwen3.5-35b-a3b-uncensored-fernflowerai-apex-i-nano +- name: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled. Quality ladder and quantization rungs published by mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + - vision + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Quality.gguf + sha256: 5785096c8c2c0e3fb3d59f5c1a429e7c6345ebd569b317a5356b142459119cf9 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/mmproj.gguf + sha256: 12960e8cc09c5fc0a9f119989a3df717e00c3160c92e7229a964605d62b338fc + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-APEX-GGUF/resolve/main/mmproj.gguf + variants: + - model: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-i-quality + - model: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-i-balanced + - model: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-i-compact + - model: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-i-mini + - model: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-i-nano + - model: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-mtp-i-quality + - model: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-mtp-i-balanced + - model: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-mtp-i-compact + - model: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-mtp-i-mini + - model: qwen3.6-35b-a3b-claude-4.7-opus-reasoning-distilled-apex-mtp-i-nano +- name: qwen3.6-35b-a3b-uncensored-heretic + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Qwen3.6-35B-A3B-uncensored-heretic. Quality ladder and quantization rungs published by mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + - vision + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Quality.gguf + sha256: 667bd985a00bf5a47e5ba582f0bb573578e215c54b504525c13b8d4f573e5b59 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/resolve/main/Qwen3.6-35B-A3B-uncensored-heretic-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/mmproj.gguf + sha256: 1c625f05cd52e90abc76a5f756226c3a5fe279593379c22f6c6846c970a0cd18 + uri: https://huggingface.co/mudler/Qwen3.6-35B-A3B-uncensored-heretic-APEX-GGUF/resolve/main/mmproj.gguf + variants: + - model: qwen3.6-35b-a3b-uncensored-heretic-apex-i-quality + - model: qwen3.6-35b-a3b-uncensored-heretic-apex-i-balanced + - model: qwen3.6-35b-a3b-uncensored-heretic-apex-i-compact + - model: qwen3.6-35b-a3b-uncensored-heretic-apex-i-mini +- name: qwopus-moe-35b-a3b + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Qwopus-MoE-35B-A3B. Quality ladder and quantization rungs published by mudler/Qwopus-MoE-35B-A3B-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/Qwopus-MoE-35B-A3B-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/Qwopus-MoE-35B-A3B-APEX-I-Quality.gguf + sha256: 36a7848c43b3c7db701aba61b71729e2a172da7b2c5338f5121d2c31f2ab86ee + uri: https://huggingface.co/mudler/Qwopus-MoE-35B-A3B-APEX-GGUF/resolve/main/Qwopus-MoE-35B-A3B-APEX-I-Quality.gguf + variants: + - model: qwopus-moe-35b-a3b-apex-i-quality + - model: qwopus-moe-35b-a3b-apex-i-balanced + - model: qwopus-moe-35b-a3b-apex-i-compact + - model: qwopus-moe-35b-a3b-apex-i-mini +- name: qwopus3.6-35b-a3b-coder + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Qwopus3.6-35B-A3B-Coder. Quality ladder and quantization rungs published by mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + - vision + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/Qwopus3.6-35B-A3B-Coder-APEX-I-Quality.gguf + sha256: b86b2398d38337b0485217d0be0a90ac32adec1d25ff9d2c4dfa84e1a826ebce + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/resolve/main/Qwopus3.6-35B-A3B-Coder-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/mmproj.gguf + sha256: 551d7a6187874b37f69a0c926e2aaad65df5c426a2e243b68f11bfe7e85b3d12 + uri: https://huggingface.co/mudler/Qwopus3.6-35B-A3B-Coder-APEX-GGUF/resolve/main/mmproj.gguf + variants: + - model: qwopus3.6-35b-a3b-coder-apex-i-quality + - model: qwopus3.6-35b-a3b-coder-apex-i-balanced + - model: qwopus3.6-35b-a3b-coder-apex-i-compact + - model: qwopus3.6-35b-a3b-coder-apex-i-mini + - model: qwopus3.6-35b-a3b-coder-apex-mtp-i-quality + - model: qwopus3.6-35b-a3b-coder-apex-mtp-i-balanced + - model: qwopus3.6-35b-a3b-coder-apex-mtp-i-compact + - model: qwopus3.6-35b-a3b-coder-apex-mtp-i-mini +- name: step-3.5-flash + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: Step-3.5-Flash. Quality ladder and quantization rungs published by mudler/Step-3.5-Flash-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + known_usecases: + - chat + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/Step-3.5-Flash-APEX-GGUF/Step-3.5-Flash-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/Step-3.5-Flash-APEX-GGUF/Step-3.5-Flash-APEX-I-Quality.gguf + sha256: e4fdf83cd3ad931ee0d0448104460c4685a0ee419002e6a7cdd217ff352a03c4 + uri: https://huggingface.co/mudler/Step-3.5-Flash-APEX-GGUF/resolve/main/Step-3.5-Flash-APEX-I-Quality.gguf + variants: + - model: step-3.5-flash-apex-i-quality + - model: step-3.5-flash-apex-i-balanced + - model: step-3.5-flash-apex-i-compact + - model: step-3.5-flash-apex-i-mini +- name: gemma-4-26b-a4b-it-claude-opus-distill + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: gemma-4-26B-A4B-it-Claude-Opus-Distill. Quality ladder and quantization rungs published by mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + - vision + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/gemma-4-26B-A4B-Claude-Distill-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/gemma-4-26B-A4B-Claude-Distill-APEX-I-Quality.gguf + sha256: 4b5a331ce4edaf1b286f269eb6db8d878ebd0c56fa9e0635be505c87867b673e + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/resolve/main/gemma-4-26B-A4B-Claude-Distill-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/mmproj.gguf + sha256: fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-Claude-Opus-Distill-APEX-GGUF/resolve/main/mmproj.gguf + variants: + - model: gemma-4-26b-a4b-it-claude-opus-distill-apex-i-quality + - model: gemma-4-26b-a4b-it-claude-opus-distill-apex-i-balanced + - model: gemma-4-26b-a4b-it-claude-opus-distill-apex-i-compact + - model: gemma-4-26b-a4b-it-claude-opus-distill-apex-i-mini +- name: gemma-4-26b-a4b-it-heretic + url: github:mudler/LocalAI/gallery/virtual.yaml@master + description: gemma-4-26B-A4B-it-heretic. Quality ladder and quantization rungs published by mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF and its unsloth counterpart; LocalAI picks the build that fits the hardware. + tags: + - llm + - gguf + - cpu + - gpu + - vision + overrides: + backend: llama-cpp + known_usecases: + - chat + - vision + mmproj: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/mmproj.gguf + options: + - use_jinja:true + parameters: + model: llama-cpp/models/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/gemma-4-26B-A4B-heretic-APEX-I-Quality.gguf + files: + - filename: llama-cpp/models/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/gemma-4-26B-A4B-heretic-APEX-I-Quality.gguf + sha256: 2def9529490558630809bdb1c9f3b82c6b8567409c850bbf4a3eeb5c35986de6 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/resolve/main/gemma-4-26B-A4B-heretic-APEX-I-Quality.gguf + - filename: llama-cpp/mmproj/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/mmproj.gguf + sha256: fc2ebf4c44528daa2cea7b39891712847ca5e4f87dcf578054a06c46bfe6da27 + uri: https://huggingface.co/mudler/gemma-4-26B-A4B-it-heretic-APEX-GGUF/resolve/main/mmproj.gguf + variants: + - model: gemma-4-26b-a4b-it-heretic-apex-i-quality + - model: gemma-4-26b-a4b-it-heretic-apex-i-balanced + - model: gemma-4-26b-a4b-it-heretic-apex-i-compact + - model: gemma-4-26b-a4b-it-heretic-apex-i-mini