diff --git a/docs/content/features/model-gallery.md b/docs/content/features/model-gallery.md index c430a6a71..bc33ddbc8 100644 --- a/docs/content/features/model-gallery.md +++ b/docs/content/features/model-gallery.md @@ -331,6 +331,24 @@ When browsing the gallery or importing a model by URI, LocalAI can show **estima - **Hardware fit indicator**: When your system reports GPU or RAM capacity, each row shows whether the estimated memory at the chosen context length fits, using a 95% headroom rule. A model that is too big for the GPU but would run from system RAM is marked as spilling to the CPU, with how much; a model too big for both is marked as over by the shortfall. - Estimates are best-effort and may be missing if the server does not support HEAD/Range or the request times out. +## EmbeddingGemma 2 + +The `embeddinggemma-2` family serves text embeddings through `llama-cpp`. +It offers Q8_0 and BF16 weights, plus smaller Unsloth UD-Q4_K_XL, +UD-Q5_K_XL, and UD-Q6_K_XL variants. These entries use the 270M-parameter +text backbone with an 8192-token context and 768-dimensional output. + +To select a smaller build explicitly: + +```sh +local-ai models install embeddinggemma-2 --variant embeddinggemma-2-ud-q4-k-xl +``` + +Send text to `/v1/embeddings`. For retrieval, prefix queries with +`task: search result | query: ` and documents with `title: {title} | text: `. +See the [model card](https://huggingface.co/google/embeddinggemma-2) +for other task prefixes. These gallery entries support text input only. + ## Useful Links and resources - [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) - here you can find a list of the most performing models on the Open LLM benchmark. Keep in mind models compatible with LocalAI must be quantized in the `gguf` format. diff --git a/gallery/index.yaml b/gallery/index.yaml index 93a07b968..69f1fd5b4 100644 --- a/gallery/index.yaml +++ b/gallery/index.yaml @@ -38545,6 +38545,9 @@ - name: embeddinggemma-2 variants: - model: embeddinggemma-2-bf16 + - model: embeddinggemma-2-ud-q4-k-xl + - model: embeddinggemma-2-ud-q5-k-xl + - model: embeddinggemma-2-ud-q6-k-xl url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: - https://huggingface.co/google/embeddinggemma-2 @@ -38602,6 +38605,93 @@ - filename: llama-cpp/models/embeddinggemma-2/embeddinggemma-2-BF16.gguf sha256: 68bae29d62fb8c7d23e98d21fd4662753ddd636e6b62b8a70dfe059a9844f216 uri: https://huggingface.co/ggml-org/embeddinggemma-2-GGUF/resolve/bfcd298762cc34d0357ece5ebdd31791a3a374d8/embeddinggemma-2-BF16.gguf +- name: embeddinggemma-2-ud-q4-k-xl + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://huggingface.co/google/embeddinggemma-2 + - https://huggingface.co/unsloth/embeddinggemma-2-GGUF + description: | + EmbeddingGemma 2 is Google's multilingual embedding model. + This Unsloth UD-Q4_K_XL GGUF serves the 270M-parameter text backbone through llama.cpp + and produces 768-dimensional embeddings for retrieval and semantic similarity. + This entry supports text input with an 8192-token context. + Use the model's task prefixes for queries and documents. + license: apache-2.0 + tags: + - embeddings + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + embeddings: true + context_size: 8192 + known_usecases: + - embeddings + parameters: + model: llama-cpp/models/embeddinggemma-2/embeddinggemma-2-UD-Q4_K_XL.gguf + files: + - filename: llama-cpp/models/embeddinggemma-2/embeddinggemma-2-UD-Q4_K_XL.gguf + sha256: ea905fd08e8061db77a0031cbb5096d459d7cfd2bdd0ad2ba66e483f16719493 + uri: https://huggingface.co/unsloth/embeddinggemma-2-GGUF/resolve/031f0d4b35536f69ab3509d4893c923264fcf253/embeddinggemma-2-UD-Q4_K_XL.gguf +- name: embeddinggemma-2-ud-q5-k-xl + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://huggingface.co/google/embeddinggemma-2 + - https://huggingface.co/unsloth/embeddinggemma-2-GGUF + description: | + EmbeddingGemma 2 is Google's multilingual embedding model. + This Unsloth UD-Q5_K_XL GGUF serves the 270M-parameter text backbone through llama.cpp + and produces 768-dimensional embeddings for retrieval and semantic similarity. + This entry supports text input with an 8192-token context. + Use the model's task prefixes for queries and documents. + license: apache-2.0 + tags: + - embeddings + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + embeddings: true + context_size: 8192 + known_usecases: + - embeddings + parameters: + model: llama-cpp/models/embeddinggemma-2/embeddinggemma-2-UD-Q5_K_XL.gguf + files: + - filename: llama-cpp/models/embeddinggemma-2/embeddinggemma-2-UD-Q5_K_XL.gguf + sha256: a9d7a3b7eb421113f05e964e10882676f10fff83811b7876d5bb983db7c7ac41 + uri: https://huggingface.co/unsloth/embeddinggemma-2-GGUF/resolve/031f0d4b35536f69ab3509d4893c923264fcf253/embeddinggemma-2-UD-Q5_K_XL.gguf +- name: embeddinggemma-2-ud-q6-k-xl + url: github:mudler/LocalAI/gallery/virtual.yaml@master + urls: + - https://huggingface.co/google/embeddinggemma-2 + - https://huggingface.co/unsloth/embeddinggemma-2-GGUF + description: | + EmbeddingGemma 2 is Google's multilingual embedding model. + This Unsloth UD-Q6_K_XL GGUF serves the 270M-parameter text backbone through llama.cpp + and produces 768-dimensional embeddings for retrieval and semantic similarity. + This entry supports text input with an 8192-token context. + Use the model's task prefixes for queries and documents. + license: apache-2.0 + tags: + - embeddings + - gguf + - cpu + - gpu + overrides: + backend: llama-cpp + embeddings: true + context_size: 8192 + known_usecases: + - embeddings + parameters: + model: llama-cpp/models/embeddinggemma-2/embeddinggemma-2-UD-Q6_K_XL.gguf + files: + - filename: llama-cpp/models/embeddinggemma-2/embeddinggemma-2-UD-Q6_K_XL.gguf + sha256: dc84f042bc3ffbe6122b3b38b2fb299d66f5f37b276c9d458db85d4a4f1e6ba3 + uri: https://huggingface.co/unsloth/embeddinggemma-2-GGUF/resolve/031f0d4b35536f69ab3509d4893c923264fcf253/embeddinggemma-2-UD-Q6_K_XL.gguf - name: embeddinggemma-300m url: github:mudler/LocalAI/gallery/virtual.yaml@master urls: