wip

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

.agents/adding-backends.md

@@ -1,143 +0,0 @@
-# Adding a New Backend
-
-When adding a new backend to LocalAI, you need to update several files to ensure the backend is properly built, tested, and registered. Here's a step-by-step guide based on the pattern used for adding backends like `moonshine`:
-
-## 1. Create Backend Directory Structure
-
-Create the backend directory under the appropriate location:
-- **Python backends**: `backend/python/<backend-name>/`
-- **Go backends**: `backend/go/<backend-name>/`
-- **C++ backends**: `backend/cpp/<backend-name>/`
-
-For Python backends, you'll typically need:
-- `backend.py` - Main gRPC server implementation
-- `Makefile` - Build configuration
-- `install.sh` - Installation script for dependencies
-- `protogen.sh` - Protocol buffer generation script
-- `requirements.txt` - Python dependencies
-- `run.sh` - Runtime script
-- `test.py` / `test.sh` - Test files
-
-## 2. Add Build Configurations to `.github/workflows/backend.yml`
-
-Add build matrix entries for each platform/GPU type you want to support. Look at similar backends (e.g., `chatterbox`, `faster-whisper`) for reference.
-
-**Placement in file:**
-- CPU builds: Add after other CPU builds (e.g., after `cpu-chatterbox`)
-- CUDA 12 builds: Add after other CUDA 12 builds (e.g., after `gpu-nvidia-cuda-12-chatterbox`)
-- CUDA 13 builds: Add after other CUDA 13 builds (e.g., after `gpu-nvidia-cuda-13-chatterbox`)
-
-**Additional build types you may need:**
-- ROCm/HIP: Use `build-type: 'hipblas'` with `base-image: "rocm/dev-ubuntu-24.04:6.4.4"`
-- Intel/SYCL: Use `build-type: 'intel'` or `build-type: 'sycl_f16'`/`sycl_f32` with `base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"`
-- L4T (ARM): Use `build-type: 'l4t'` with `platforms: 'linux/arm64'` and `runs-on: 'ubuntu-24.04-arm'`
-
-## 3. Add Backend Metadata to `backend/index.yaml`
-
-**Step 3a: Add Meta Definition**
-
-Add a YAML anchor definition in the `## metas` section (around line 2-300). Look for similar backends to use as a template such as `diffusers` or `chatterbox`
-
-**Step 3b: Add Image Entries**
-
-Add image entries at the end of the file, following the pattern of similar backends such as `diffusers` or `chatterbox`. Include both `latest` (production) and `master` (development) tags.
-
-## 4. Update the Makefile
-
-The Makefile needs to be updated in several places to support building and testing the new backend:
-
-**Step 4a: Add to `.NOTPARALLEL`**
-
-Add `backends/<backend-name>` to the `.NOTPARALLEL` line (around line 2) to prevent parallel execution conflicts:
-
-```makefile
-.NOTPARALLEL: ... backends/<backend-name>
-```
-
-**Step 4b: Add to `prepare-test-extra`**
-
-Add the backend to the `prepare-test-extra` target (around line 312) to prepare it for testing:
-
-```makefile
-prepare-test-extra: protogen-python
-	...
-	$(MAKE) -C backend/python/<backend-name>
-```
-
-**Step 4c: Add to `test-extra`**
-
-Add the backend to the `test-extra` target (around line 319) to run its tests:
-
-```makefile
-test-extra: prepare-test-extra
-	...
-	$(MAKE) -C backend/python/<backend-name> test
-```
-
-**Step 4d: Add Backend Definition**
-
-Add a backend definition variable in the backend definitions section (around line 428-457). The format depends on the backend type:
-
-**For Python backends with root context** (like `faster-whisper`, `coqui`):
-```makefile
-BACKEND_<BACKEND_NAME> = <backend-name>|python|.|false|true
-```
-
-**For Python backends with `./backend` context** (like `chatterbox`, `moonshine`):
-```makefile
-BACKEND_<BACKEND_NAME> = <backend-name>|python|./backend|false|true
-```
-
-**For Go backends**:
-```makefile
-BACKEND_<BACKEND_NAME> = <backend-name>|golang|.|false|true
-```
-
-**Step 4e: Generate Docker Build Target**
-
-Add an eval call to generate the docker-build target (around line 480-501):
-
-```makefile
-$(eval $(call generate-docker-build-target,$(BACKEND_<BACKEND_NAME>)))
-```
-
-**Step 4f: Add to `docker-build-backends`**
-
-Add `docker-build-<backend-name>` to the `docker-build-backends` target (around line 507):
-
-```makefile
-docker-build-backends: ... docker-build-<backend-name>
-```
-
-**Determining the Context:**
-
-- If the backend is in `backend/python/<backend-name>/` and uses `./backend` as context in the workflow file, use `./backend` context
-- If the backend is in `backend/python/<backend-name>/` but uses `.` as context in the workflow file, use `.` context
-- Check similar backends to determine the correct context
-
-## 5. Verification Checklist
-
-After adding a new backend, verify:
-
-- [ ] Backend directory structure is complete with all necessary files
-- [ ] Build configurations added to `.github/workflows/backend.yml` for all desired platforms
-- [ ] Meta definition added to `backend/index.yaml` in the `## metas` section
-- [ ] Image entries added to `backend/index.yaml` for all build variants (latest + development)
-- [ ] Tag suffixes match between workflow file and index.yaml
-- [ ] Makefile updated with all 6 required changes (`.NOTPARALLEL`, `prepare-test-extra`, `test-extra`, backend definition, docker-build target eval, `docker-build-backends`)
-- [ ] No YAML syntax errors (check with linter)
-- [ ] No Makefile syntax errors (check with linter)
-- [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern)
-
-## 6. Example: Adding a Python Backend
-
-For reference, when `moonshine` was added:
-- **Files created**: `backend/python/moonshine/{backend.py, Makefile, install.sh, protogen.sh, requirements.txt, run.sh, test.py, test.sh}`
-- **Workflow entries**: 3 build configurations (CPU, CUDA 12, CUDA 13)
-- **Index entries**: 1 meta definition + 6 image entries (cpu, cuda12, cuda13 x latest/development)
-- **Makefile updates**:
-  - Added to `.NOTPARALLEL` line
-  - Added to `prepare-test-extra` and `test-extra` targets
-  - Added `BACKEND_MOONSHINE = moonshine|python|./backend|false|true`
-  - Added eval for docker-build target generation
-  - Added `docker-build-moonshine` to `docker-build-backends`

.agents/building-and-testing.md

@@ -1,16 +0,0 @@
-# Build and Testing
-
-Building and testing the project depends on the components involved and the platform where development is taking place. Due to the amount of context required it's usually best not to try building or testing the project unless the user requests it. If you must build the project then inspect the Makefile in the project root and the Makefiles of any backends that are effected by changes you are making. In addition the workflows in .github/workflows can be used as a reference when it is unclear how to build or test a component. The primary Makefile contains targets for building inside or outside Docker, if the user has not previously specified a preference then ask which they would like to use.
-
-## Building a specified backend
-
-Let's say the user wants to build a particular backend for a given platform. For example let's say they want to build coqui for ROCM/hipblas
-
-- The Makefile has targets like `docker-build-coqui` created with `generate-docker-build-target` at the time of writing. Recently added backends may require a new target.
-- At a minimum we need to set the BUILD_TYPE, BASE_IMAGE build-args
-  - Use .github/workflows/backend.yml as a reference it lists the needed args in the `include` job strategy matrix
-  - l4t and cublas also requires the CUDA major and minor version
-- You can pretty print a command like `DOCKER_MAKEFLAGS=-j$(nproc --ignore=1) BUILD_TYPE=hipblas BASE_IMAGE=rocm/dev-ubuntu-24.04:6.4.4 make docker-build-coqui`
-- Unless the user specifies that they want you to run the command, then just print it because not all agent frontends handle long running jobs well and the output may overflow your context
-- The user may say they want to build AMD or ROCM instead of hipblas, or Intel instead of SYCL or NVIDIA insted of l4t or cublas. Ask for confirmation if there is ambiguity.
-- Sometimes the user may need extra parameters to be added to `docker build` (e.g. `--platform` for cross-platform builds or `--progress` to view the full logs), in which case you can generate the `docker build` command directly.

.agents/coding-style.md

@@ -1,51 +0,0 @@
-# Coding Style
-
-The project has the following .editorconfig:
-
-```
-root = true
-
-[*]
-indent_style = space
-indent_size = 2
-end_of_line = lf
-charset = utf-8
-trim_trailing_whitespace = true
-insert_final_newline = true
-
-[*.go]
-indent_style = tab
-
-[Makefile]
-indent_style = tab
-
-[*.proto]
-indent_size = 2
-
-[*.py]
-indent_size = 4
-
-[*.js]
-indent_size = 2
-
-[*.yaml]
-indent_size = 2
-
-[*.md]
-trim_trailing_whitespace = false
-```
-
-- Use comments sparingly to explain why code does something, not what it does. Comments are there to add context that would be difficult to deduce from reading the code.
-- Prefer modern Go e.g. use `any` not `interface{}`
-
-## Logging
-
-Use `github.com/mudler/xlog` for logging which has the same API as slog.
-
-## Documentation
-
-The project documentation is located in `docs/content`. When adding new features or changing existing functionality, it is crucial to update the documentation to reflect these changes. This helps users understand how to use the new capabilities and ensures the documentation stays relevant.
-
-- **Feature Documentation**: If you add a new feature (like a new backend or API endpoint), create a new markdown file in `docs/content/features/` explaining what it is, how to configure it, and how to use it.
-- **Configuration**: If you modify configuration options, update the relevant sections in `docs/content/`.
-- **Examples**: providing concrete examples (like YAML configuration blocks) is highly encouraged to help users get started quickly.

.agents/llama-cpp-backend.md

@@ -1,77 +0,0 @@
-# llama.cpp Backend
-
-The llama.cpp backend (`backend/cpp/llama-cpp/grpc-server.cpp`) is a gRPC adaptation of the upstream HTTP server (`llama.cpp/tools/server/server.cpp`). It uses the same underlying server infrastructure from `llama.cpp/tools/server/server-context.cpp`.
-
-## Building and Testing
-
-- Test llama.cpp backend compilation: `make backends/llama-cpp`
-- The backend is built as part of the main build process
-- Check `backend/cpp/llama-cpp/Makefile` for build configuration
-
-## Architecture
-
-- **grpc-server.cpp**: gRPC server implementation, adapts HTTP server patterns to gRPC
-- Uses shared server infrastructure: `server-context.cpp`, `server-task.cpp`, `server-queue.cpp`, `server-common.cpp`
-- The gRPC server mirrors the HTTP server's functionality but uses gRPC instead of HTTP
-
-## Common Issues When Updating llama.cpp
-
-When fixing compilation errors after upstream changes:
-1. Check how `server.cpp` (HTTP server) handles the same change
-2. Look for new public APIs or getter methods
-3. Store copies of needed data instead of accessing private members
-4. Update function calls to match new signatures
-5. Test with `make backends/llama-cpp`
-
-## Key Differences from HTTP Server
-
-- gRPC uses `BackendServiceImpl` class with gRPC service methods
-- HTTP server uses `server_routes` with HTTP handlers
-- Both use the same `server_context` and task queue infrastructure
-- gRPC methods: `LoadModel`, `Predict`, `PredictStream`, `Embedding`, `Rerank`, `TokenizeString`, `GetMetrics`, `Health`
-
-## Tool Call Parsing Maintenance
-
-When working on JSON/XML tool call parsing functionality, always check llama.cpp for reference implementation and updates:
-
-### Checking for XML Parsing Changes
-
-1. **Review XML Format Definitions**: Check `llama.cpp/common/chat-parser-xml-toolcall.h` for `xml_tool_call_format` struct changes
-2. **Review Parsing Logic**: Check `llama.cpp/common/chat-parser-xml-toolcall.cpp` for parsing algorithm updates
-3. **Review Format Presets**: Check `llama.cpp/common/chat-parser.cpp` for new XML format presets (search for `xml_tool_call_format form`)
-4. **Review Model Lists**: Check `llama.cpp/common/chat.h` for `COMMON_CHAT_FORMAT_*` enum values that use XML parsing:
-   - `COMMON_CHAT_FORMAT_GLM_4_5`
-   - `COMMON_CHAT_FORMAT_MINIMAX_M2`
-   - `COMMON_CHAT_FORMAT_KIMI_K2`
-   - `COMMON_CHAT_FORMAT_QWEN3_CODER_XML`
-   - `COMMON_CHAT_FORMAT_APRIEL_1_5`
-   - `COMMON_CHAT_FORMAT_XIAOMI_MIMO`
-   - Any new formats added
-
-### Model Configuration Options
-
-Always check `llama.cpp` for new model configuration options that should be supported in LocalAI:
-
-1. **Check Server Context**: Review `llama.cpp/tools/server/server-context.cpp` for new parameters
-2. **Check Chat Params**: Review `llama.cpp/common/chat.h` for `common_chat_params` struct changes
-3. **Check Server Options**: Review `llama.cpp/tools/server/server.cpp` for command-line argument changes
-4. **Examples of options to check**:
-   - `ctx_shift` - Context shifting support
-   - `parallel_tool_calls` - Parallel tool calling
-   - `reasoning_format` - Reasoning format options
-   - Any new flags or parameters
-
-### Implementation Guidelines
-
-1. **Feature Parity**: Always aim for feature parity with llama.cpp's implementation
-2. **Test Coverage**: Add tests for new features matching llama.cpp's behavior
-3. **Documentation**: Update relevant documentation when adding new formats or options
-4. **Backward Compatibility**: Ensure changes don't break existing functionality
-
-### Files to Monitor
-
-- `llama.cpp/common/chat-parser-xml-toolcall.h` - Format definitions
-- `llama.cpp/common/chat-parser-xml-toolcall.cpp` - Parsing logic
-- `llama.cpp/common/chat-parser.cpp` - Format presets and model-specific handlers
-- `llama.cpp/common/chat.h` - Format enums and parameter structures
-- `llama.cpp/tools/server/server-context.cpp` - Server configuration options

.agents/testing-mcp-apps.md

@@ -1,120 +0,0 @@
-# Testing MCP Apps (Interactive Tool UIs)
-
-MCP Apps is an extension to MCP where tools declare interactive HTML UIs via `_meta.ui.resourceUri`. When the LLM calls such a tool, the UI renders the app in a sandboxed iframe inline in the chat. The app communicates bidirectionally with the host via `postMessage` (JSON-RPC) and can call server tools, send messages, and update model context.
-
-Spec: https://modelcontextprotocol.io/extensions/apps/overview
-
-## Quick Start: Run a Test MCP App Server
-
-The `@modelcontextprotocol/server-basic-react` npm package is a ready-to-use test server that exposes a `get-time` tool with an interactive React clock UI. It requires Node >= 20, so run it in Docker:
-
-```bash
-docker run -d --name mcp-app-test -p 3001:3001 node:22-slim \
-  sh -c 'npx -y @modelcontextprotocol/server-basic-react'
-```
-
-Wait ~10 seconds for it to start, then verify:
-
-```bash
-# Check it's running
-docker logs mcp-app-test
-# Expected: "MCP server listening on http://localhost:3001/mcp"
-
-# Verify MCP protocol works
-curl -s -X POST http://localhost:3001/mcp \
-  -H 'Content-Type: application/json' \
-  -H 'Accept: application/json, text/event-stream' \
-  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"test","version":"1.0.0"}}}'
-
-# List tools — should show get-time with _meta.ui.resourceUri
-curl -s -X POST http://localhost:3001/mcp \
-  -H 'Content-Type: application/json' \
-  -H 'Accept: application/json, text/event-stream' \
-  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'
-```
-
-The `tools/list` response should contain:
-```json
-{
-  "name": "get-time",
-  "_meta": {
-    "ui": { "resourceUri": "ui://get-time/mcp-app.html" }
-  }
-}
-```
-
-## Testing in LocalAI's UI
-
-1. Make sure LocalAI is running (e.g. `http://localhost:8080`)
-2. Build the React UI: `cd core/http/react-ui && npm install && npm run build`
-3. Open the Chat page in your browser
-4. Click **"Client MCP"** in the chat header
-5. Add a new client MCP server:
-   - **URL**: `http://localhost:3001/mcp`
-   - **Use CORS proxy**: enabled (default) — required because the browser can't hit `localhost:3001` directly due to CORS; LocalAI's proxy at `/api/cors-proxy` handles it
-6. The server should connect and discover the `get-time` tool
-7. Select a model and send: **"What time is it?"**
-8. The LLM should call the `get-time` tool
-9. The tool result should render the interactive React clock app in an iframe as a standalone chat message (not inside the collapsed activity group)
-
-## What to Verify
-
-- [ ] Tool appears in the connected tools list (not filtered — `get-time` is callable by the LLM)
-- [ ] The iframe renders as a standalone chat message with a puzzle-piece icon
-- [ ] The app loads and is interactive (clock UI, buttons work)
-- [ ] No "Reconnect to MCP server" overlay (connection is live)
-- [ ] Console logs show bidirectional communication:
-  - `tools/call` messages from app to host (app calling server tools)
-  - `ui/message` notifications (app sending messages)
-- [ ] After the app renders, the LLM continues and produces a text response with the time
-- [ ] Non-UI tools continue to work normally (text-only results)
-- [ ] Page reload shows the HTML statically with a reconnect overlay until you reconnect
-
-## Console Log Patterns
-
-Healthy bidirectional communication looks like:
-
-```
-Parsed message { jsonrpc: "2.0", id: N, result: {...} }     // Bridge init
-get-time result: { content: [...] }                          // Tool result received
-Calling get-time tool...                                     // App calls tool
-Sending message { method: "tools/call", ... }                // App -> host -> server
-Parsed message { jsonrpc: "2.0", id: N, result: {...} }     // Server response
-Sending message text to Host: ...                            // App sends message
-Sending message { method: "ui/message", ... }                // Message notification
-Message accepted                                             // Host acknowledged
-```
-
-Benign warnings to ignore:
-- `Source map error: ... about:srcdoc` — browser devtools can't find source maps for srcdoc iframes
-- `Ignoring message from unknown source` — duplicate postMessage from iframe navigation
-- `notifications/cancelled` — app cleaning up previous requests
-
-## Architecture Notes
-
-- **No server-side changes needed** — the MCP App protocol runs entirely in the browser
-- `PostMessageTransport` wraps `window.postMessage` between host and `srcdoc` iframe
-- `AppBridge` (from `@modelcontextprotocol/ext-apps`) auto-forwards `tools/call`, `resources/read`, `resources/list` from the app to the MCP server via the host's `Client`
-- The iframe uses `sandbox="allow-scripts allow-forms"` (no `allow-same-origin`) — opaque origin, no access to host cookies/DOM/localStorage
-- App-only tools (`_meta.ui.visibility: "app-only"`) are filtered from the LLM's tool list but remain callable by the app iframe
-
-## Key Files
-
-- `core/http/react-ui/src/components/MCPAppFrame.jsx` — iframe + AppBridge component
-- `core/http/react-ui/src/hooks/useMCPClient.js` — MCP client hook with app UI helpers (`hasAppUI`, `getAppResource`, `getClientForTool`, `getToolDefinition`)
-- `core/http/react-ui/src/hooks/useChat.js` — agentic loop, attaches `appUI` to tool_result messages
-- `core/http/react-ui/src/pages/Chat.jsx` — renders MCPAppFrame as standalone chat messages
-
-## Other Test Servers
-
-The `@modelcontextprotocol/ext-apps` repo has many example servers:
-- `@modelcontextprotocol/server-basic-react` — simple clock (React)
-- More examples at https://github.com/modelcontextprotocol/ext-apps/tree/main/examples
-
-All examples support both stdio and HTTP transport. Run without `--stdio` for HTTP mode on port 3001.
-
-## Cleanup
-
-```bash
-docker rm -f mcp-app-test
-```

.air.toml

@@ -1,8 +0,0 @@
-# .air.toml
-[build]
-cmd = "make build"
-bin = "./local-ai"
-args_bin = [ "--debug" ]
-include_ext = ["go", "html", "yaml", "toml", "json", "txt", "md"]
-exclude_dir = ["pkg/grpc/proto"]
-delay = 1000

.devcontainer/docker-compose-devcontainer.yml

@@ -10,8 +10,7 @@ services:
       - 8080:8080
     volumes:
       - localai_workspace:/workspace
-      - models:/host-models
+      - ../models:/host-models
-      - backends:/host-backends
       - ./customization:/devcontainer-customization
     command: /bin/sh -c "while sleep 1000; do :; done"
     cap_add:
@@ -40,9 +39,6 @@ services:
       - GF_SECURITY_ADMIN_PASSWORD=grafana
     volumes:
       - ./grafana:/etc/grafana/provisioning/datasources
-
 volumes:
   prom_data:
-  localai_workspace:
+  localai_workspace:
-  models:
-  backends:

.env

@@ -26,15 +26,21 @@
 ## Disables COMPEL (Diffusers)
 # COMPEL=0
 
-## Disables SD_EMBED (Diffusers)
-# SD_EMBED=0
-
 ## Enable/Disable single backend (useful if only one GPU is available)
 # LOCALAI_SINGLE_ACTIVE_BACKEND=true
 
 # Forces shutdown of the backends if busy (only if LOCALAI_SINGLE_ACTIVE_BACKEND is set)
 # LOCALAI_FORCE_BACKEND_SHUTDOWN=true
 
+## Specify a build type. Available: cublas, openblas, clblas.
+## cuBLAS: This is a GPU-accelerated version of the complete standard BLAS (Basic Linear Algebra Subprograms) library. It's provided by Nvidia and is part of their CUDA toolkit.
+## OpenBLAS: This is an open-source implementation of the BLAS library that aims to provide highly optimized code for various platforms. It includes support for multi-threading and can be compiled to use hardware-specific features for additional performance. OpenBLAS can run on many kinds of hardware, including CPUs from Intel, AMD, and ARM.
+## clBLAS:   This is an open-source implementation of the BLAS library that uses OpenCL, a framework for writing programs that execute across heterogeneous platforms consisting of CPUs, GPUs, and other processors. clBLAS is designed to take advantage of the parallel computing power of GPUs but can also run on any hardware that supports OpenCL. This includes hardware from different vendors like Nvidia, AMD, and Intel.
+# BUILD_TYPE=openblas
+
+## Uncomment and set to true to enable rebuilding from source
+# REBUILD=true
+
 ## Path where to store generated images
 # LOCALAI_IMAGE_PATH=/tmp/generated/images
 

.github/gallery-agent/agent.go

@@ -2,19 +2,14 @@ package main
 
 import (
 	"context"
-	"encoding/json"
 	"fmt"
-	"io"
-	"net/http"
 	"os"
-	"regexp"
 	"slices"
 	"strings"
 
-	"github.com/ghodss/yaml"
+	"github.com/go-skynet/LocalAI/.github/gallery-agent/hfapi"
-	hfapi "github.com/mudler/LocalAI/pkg/huggingface-api"
 	"github.com/mudler/cogito"
-	"github.com/mudler/cogito/clients"
+
 	"github.com/mudler/cogito/structures"
 	"github.com/sashabaranov/go-openai/jsonschema"
 )
@@ -25,7 +20,7 @@ var (
 	openAIBaseURL    = os.Getenv("OPENAI_BASE_URL")
 	galleryIndexPath = os.Getenv("GALLERY_INDEX_PATH")
 	//defaultclient
-	llm = clients.NewOpenAILLM(openAIModel, openAIKey, openAIBaseURL)
+	llm = cogito.NewOpenAILLM(openAIModel, openAIKey, openAIBaseURL)
 )
 
 // cleanTextContent removes trailing spaces, tabs, and normalizes line endings
@@ -50,12 +45,7 @@ func cleanTextContent(text string) string {
 	}
 	// Remove trailing empty lines from the result
 	result := strings.Join(cleanedLines, "\n")
-	return stripThinkingTags(strings.TrimRight(result, "\n"))
+	return strings.TrimRight(result, "\n")
-}
-
-type galleryModel struct {
-	Name string   `yaml:"name"`
-	Urls []string `yaml:"urls"`
 }
 
 // isModelExisting checks if a specific model ID exists in the gallery using text search
@@ -66,20 +56,9 @@ func isModelExisting(modelID string) (bool, error) {
 		return false, fmt.Errorf("failed to read %s: %w", indexPath, err)
 	}
 
-	var galleryModels []galleryModel
+	contentStr := string(content)
-
+	// Simple text search - if the model ID appears anywhere in the file, it exists
-	err = yaml.Unmarshal(content, &galleryModels)
+	return strings.Contains(contentStr, modelID), nil
-	if err != nil {
-		return false, fmt.Errorf("failed to unmarshal %s: %w", indexPath, err)
-	}
-
-	for _, galleryModel := range galleryModels {
-		if slices.Contains(galleryModel.Urls, modelID) {
-			return true, nil
-		}
-	}
-
-	return false, nil
 }
 
 // filterExistingModels removes models that already exist in the gallery
@@ -113,16 +92,6 @@ func getGalleryIndexPath() string {
 	return "gallery/index.yaml"
 }
 
-func stripThinkingTags(content string) string {
-	// Remove content between <thinking> and </thinking> (including multi-line)
-	content = regexp.MustCompile(`(?s)<thinking>.*?</thinking>`).ReplaceAllString(content, "")
-	// Remove content between <think> and </think> (including multi-line)
-	content = regexp.MustCompile(`(?s)<think>.*?</think>`).ReplaceAllString(content, "")
-	// Clean up any extra whitespace
-	content = strings.TrimSpace(content)
-	return content
-}
-
 func getRealReadme(ctx context.Context, repository string) (string, error) {
 	// Create a conversation fragment
 	fragment := cogito.NewEmptyFragment().
@@ -141,21 +110,16 @@ func getRealReadme(ctx context.Context, repository string) (string, error) {
 	result = result.AddMessage("user", "Describe the model in a clear and concise way that can be shared in a model gallery.")
 
 	// Get a response
-	_, err = llm.Ask(ctx, result)
+	newFragment, err := llm.Ask(ctx, result)
 	if err != nil {
 		return "", err
 	}
 
-	content := result.LastMessage().Content
+	content := newFragment.LastMessage().Content
 	return cleanTextContent(content), nil
 }
 
 func selectMostInterestingModels(ctx context.Context, searchResult *SearchResult) ([]ProcessedModel, error) {
-
-	if len(searchResult.Models) == 1 {
-		return searchResult.Models, nil
-	}
-
 	// Create a conversation fragment
 	fragment := cogito.NewEmptyFragment().
 		AddMessage("user",
@@ -254,192 +218,71 @@ Return your analysis and selection reasoning.`)
 	return filteredModels, nil
 }
 
-// ModelMetadata represents extracted metadata from a model
+// ModelFamily represents a YAML anchor/family
-type ModelMetadata struct {
+type ModelFamily struct {
-	Tags    []string `json:"tags"`
+	Anchor string `json:"anchor"`
-	License string   `json:"license"`
+	Name   string `json:"name"`
 }
 
-// extractModelMetadata extracts tags and license from model README and documentation
+// selectModelFamily selects the appropriate model family/anchor for a given model
-func extractModelMetadata(ctx context.Context, model ProcessedModel) ([]string, string, error) {
+func selectModelFamily(ctx context.Context, model ProcessedModel, availableFamilies []ModelFamily) (string, error) {
 	// Create a conversation fragment
 	fragment := cogito.NewEmptyFragment().
 		AddMessage("user",
-			`Your task is to extract metadata from an AI model's README and documentation. You will be provided with:
+			`Your task is to select the most appropriate model family/anchor for a given AI model. You will be provided with:
-1. Model information (ID, author, description)
+1. Information about the model (name, description, etc.)
-2. README content
+2. A list of available model families/anchors
 
-You need to extract:
+You need to select the family that best matches the model's architecture, capabilities, or characteristics. Consider:
-1. **Tags**: An array of relevant tags that describe the model. Use common tags from the gallery such as:
+- Model architecture (e.g., Llama, Qwen, Mistral, etc.)
-   - llm, gguf, gpu, cpu, multimodal, image-to-text, text-to-text, text-to-speech, tts
+- Model capabilities (e.g., vision, coding, chat, etc.)
-   - thinking, reasoning, chat, instruction-tuned, code, vision
+- Model size/type (e.g., small, medium, large)
-   - Model family names (e.g., llama, qwen, mistral, gemma) if applicable
+- Model purpose (e.g., general purpose, specialized, etc.)
-   - Any other relevant descriptive tags
-   Select 3-8 most relevant tags.
 
-2. **License**: The license identifier (e.g., "apache-2.0", "mit", "llama2", "gpl-3.0", "bsd", "cc-by-4.0").
+Return the anchor name that best fits the model.`)
-   If no license is found, return an empty string.
-
-Return the extracted metadata in a structured format.`)
 
 	// Add model information
 	modelInfo := "Model Information:\n"
 	modelInfo += fmt.Sprintf("  ID: %s\n", model.ModelID)
 	modelInfo += fmt.Sprintf("  Author: %s\n", model.Author)
 	modelInfo += fmt.Sprintf("  Downloads: %d\n", model.Downloads)
-	if model.ReadmeContent != "" {
+	modelInfo += fmt.Sprintf("  Description: %s\n", model.ReadmeContentPreview)
-		modelInfo += fmt.Sprintf("  README Content:\n%s\n", model.ReadmeContent)
-	} else if model.ReadmeContentPreview != "" {
-		modelInfo += fmt.Sprintf("  README Preview: %s\n", model.ReadmeContentPreview)
-	}
 
 	fragment = fragment.AddMessage("user", modelInfo)
-	fragment = fragment.AddMessage("user", "Extract the tags and license from the model information. Return the metadata as a JSON object with 'tags' (array of strings) and 'license' (string).")
+
+	// Add available families
+	familiesInfo := "Available Model Families:\n"
+	for _, family := range availableFamilies {
+		familiesInfo += fmt.Sprintf("  - %s (%s)\n", family.Anchor, family.Name)
+	}
+
+	fragment = fragment.AddMessage("user", familiesInfo)
+	fragment = fragment.AddMessage("user", "Select the most appropriate family anchor for this model. Return just the anchor name.")
 
 	// Get a response
 	newFragment, err := llm.Ask(ctx, fragment)
 	if err != nil {
-		return nil, "", err
+		return "", err
 	}
 
-	// Extract structured metadata
+	// Extract the selected family
-	metadata := ModelMetadata{}
+	selectedFamily := strings.TrimSpace(newFragment.LastMessage().Content)
 
-	s := structures.Structure{
+	// Validate that the selected family exists in our list
-		Schema: jsonschema.Definition{
+	for _, family := range availableFamilies {
-			Type:                 jsonschema.Object,
+		if family.Anchor == selectedFamily {
-			AdditionalProperties: false,
+			return selectedFamily, nil
-			Properties: map[string]jsonschema.Definition{
-				"tags": {
-					Type:        jsonschema.Array,
-					Items:       &jsonschema.Definition{Type: jsonschema.String},
-					Description: "Array of relevant tags describing the model",
-				},
-				"license": {
-					Type:        jsonschema.String,
-					Description: "License identifier (e.g., apache-2.0, mit, llama2). Empty string if not found.",
-				},
-			},
-			Required: []string{"tags", "license"},
-		},
-		Object: &metadata,
-	}
-
-	err = newFragment.ExtractStructure(ctx, llm, s)
-	if err != nil {
-		return nil, "", err
-	}
-
-	return metadata.Tags, metadata.License, nil
-}
-
-// extractIconFromReadme scans the README content for image URLs and returns the first suitable icon URL found
-func extractIconFromReadme(readmeContent string) string {
-	if readmeContent == "" {
-		return ""
-	}
-
-	// Regular expressions to match image URLs in various formats (case-insensitive)
-	// Match markdown image syntax: ![alt](url) - case insensitive extensions
-	markdownImageRegex := regexp.MustCompile(`(?i)!\[[^\]]*\]\(([^)]+\.(png|jpg|jpeg|svg|webp|gif))\)`)
-	// Match HTML img tags: <img src="url">
-	htmlImageRegex := regexp.MustCompile(`(?i)<img[^>]+src=["']([^"']+\.(png|jpg|jpeg|svg|webp|gif))["']`)
-	// Match plain URLs ending with image extensions
-	plainImageRegex := regexp.MustCompile(`(?i)https?://[^\s<>"']+\.(png|jpg|jpeg|svg|webp|gif)`)
-
-	// Try markdown format first
-	matches := markdownImageRegex.FindStringSubmatch(readmeContent)
-	if len(matches) > 1 && matches[1] != "" {
-		url := strings.TrimSpace(matches[1])
-		// Prefer HuggingFace CDN URLs or absolute URLs
-		if strings.HasPrefix(strings.ToLower(url), "http") {
-			return url
 		}
 	}
 
-	// Try HTML img tags
+	// If no exact match, try to find a close match
-	matches = htmlImageRegex.FindStringSubmatch(readmeContent)
+	for _, family := range availableFamilies {
-	if len(matches) > 1 && matches[1] != "" {
+		if strings.Contains(strings.ToLower(family.Anchor), strings.ToLower(selectedFamily)) ||
-		url := strings.TrimSpace(matches[1])
+			strings.Contains(strings.ToLower(selectedFamily), strings.ToLower(family.Anchor)) {
-		if strings.HasPrefix(strings.ToLower(url), "http") {
+			return family.Anchor, nil
-			return url
 		}
 	}
 
-	// Try plain URLs
+	// Default fallback
-	matches = plainImageRegex.FindStringSubmatch(readmeContent)
+	return "llama3", nil
-	if len(matches) > 0 {
-		url := strings.TrimSpace(matches[0])
-		if strings.HasPrefix(strings.ToLower(url), "http") {
-			return url
-		}
-	}
-
-	return ""
-}
-
-// getHuggingFaceAvatarURL attempts to get the HuggingFace avatar URL for a user
-func getHuggingFaceAvatarURL(author string) string {
-	if author == "" {
-		return ""
-	}
-
-	// Try to fetch user info from HuggingFace API
-	// HuggingFace API endpoint: https://huggingface.co/api/users/{username}
-	baseURL := "https://huggingface.co"
-	userURL := fmt.Sprintf("%s/api/users/%s", baseURL, author)
-
-	req, err := http.NewRequest("GET", userURL, nil)
-	if err != nil {
-		return ""
-	}
-
-	client := &http.Client{}
-	resp, err := client.Do(req)
-	if err != nil {
-		return ""
-	}
-	defer resp.Body.Close()
-
-	if resp.StatusCode != http.StatusOK {
-		return ""
-	}
-
-	// Parse the response to get avatar URL
-	var userInfo map[string]interface{}
-	body, err := io.ReadAll(resp.Body)
-	if err != nil {
-		return ""
-	}
-
-	if err := json.Unmarshal(body, &userInfo); err != nil {
-		return ""
-	}
-
-	// Try to extract avatar URL from response
-	if avatar, ok := userInfo["avatarUrl"].(string); ok && avatar != "" {
-		return avatar
-	}
-	if avatar, ok := userInfo["avatar"].(string); ok && avatar != "" {
-		return avatar
-	}
-
-	return ""
-}
-
-// extractModelIcon extracts icon URL from README or falls back to HuggingFace avatar
-func extractModelIcon(model ProcessedModel) string {
-	// First, try to extract icon from README
-	if icon := extractIconFromReadme(model.ReadmeContent); icon != "" {
-		return icon
-	}
-
-	// Fallback: Try to get HuggingFace user avatar
-	if model.Author != "" {
-		if avatar := getHuggingFaceAvatarURL(model.Author); avatar != "" {
-			return avatar
-		}
-	}
-
-	return ""
 }

.github/gallery-agent/gallery.go

@@ -2,61 +2,13 @@ package main
 
 import (
 	"context"
-	"encoding/json"
 	"fmt"
 	"os"
 	"strings"
-
-	"github.com/ghodss/yaml"
-	"github.com/mudler/LocalAI/core/gallery/importers"
 )
 
-func formatTextContent(text string) string {
-	return formatTextContentWithIndent(text, 4, 6)
-}
-
-// formatTextContentWithIndent formats text content with specified base and list item indentation
-func formatTextContentWithIndent(text string, baseIndent int, listItemIndent int) string {
-	var formattedLines []string
-	lines := strings.Split(text, "\n")
-	for _, line := range lines {
-		trimmed := strings.TrimRight(line, " \t\r")
-		if trimmed == "" {
-			// Keep empty lines as empty (no indentation)
-			formattedLines = append(formattedLines, "")
-		} else {
-			// Preserve relative indentation from yaml.Marshal output
-			// Count existing leading spaces to preserve relative structure
-			leadingSpaces := len(trimmed) - len(strings.TrimLeft(trimmed, " \t"))
-			trimmedStripped := strings.TrimLeft(trimmed, " \t")
-
-			var totalIndent int
-			if strings.HasPrefix(trimmedStripped, "-") {
-				// List items: use listItemIndent (ignore existing leading spaces)
-				totalIndent = listItemIndent
-			} else {
-				// Regular lines: use baseIndent + preserve relative indentation
-				// This handles both top-level keys (leadingSpaces=0) and nested properties (leadingSpaces>0)
-				totalIndent = baseIndent + leadingSpaces
-			}
-
-			indentStr := strings.Repeat(" ", totalIndent)
-			formattedLines = append(formattedLines, indentStr+trimmedStripped)
-		}
-	}
-	formattedText := strings.Join(formattedLines, "\n")
-	// Remove any trailing spaces from the formatted description
-	formattedText = strings.TrimRight(formattedText, " \t")
-	return formattedText
-}
-
 // generateYAMLEntry generates a YAML entry for a model using the specified anchor
-func generateYAMLEntry(model ProcessedModel, quantization string) string {
+func generateYAMLEntry(model ProcessedModel, familyAnchor string) string {
-	modelConfig, err := importers.DiscoverModelConfig("https://huggingface.co/"+model.ModelID, json.RawMessage(`{ "quantization": "`+quantization+`"}`))
-	if err != nil {
-		panic(err)
-	}
-
 	// Extract model name from ModelID
 	parts := strings.Split(model.ModelID, "/")
 	modelName := model.ModelID
@@ -70,6 +22,18 @@ func generateYAMLEntry(model ProcessedModel, quantization string) string {
 	modelName = strings.ReplaceAll(modelName, "-q3_k_m", "")
 	modelName = strings.ReplaceAll(modelName, "-q2_k", "")
 
+	fileName := ""
+	checksum := ""
+	if model.PreferredModelFile != nil {
+		fileParts := strings.Split(model.PreferredModelFile.Path, "/")
+		if len(fileParts) > 0 {
+			fileName = fileParts[len(fileParts)-1]
+		}
+		checksum = model.PreferredModelFile.SHA256
+	} else {
+		fileName = model.ModelID
+	}
+
 	description := model.ReadmeContent
 	if description == "" {
 		description = fmt.Sprintf("AI model: %s", modelName)
@@ -77,88 +41,142 @@ func generateYAMLEntry(model ProcessedModel, quantization string) string {
 
 	// Clean up description to prevent YAML linting issues
 	description = cleanTextContent(description)
-	formattedDescription := formatTextContent(description)
 
-	configFile := formatTextContent(modelConfig.ConfigFile)
+	// Format description for YAML (indent each line and ensure no trailing spaces)
-
+	lines := strings.Split(description, "\n")
-	filesYAML, _ := yaml.Marshal(modelConfig.Files)
+	var formattedLines []string
-
+	for _, line := range lines {
-	// Files section: list items need 4 spaces (not 6), since files: is at 2 spaces
+		if strings.TrimSpace(line) == "" {
-	files := formatTextContentWithIndent(string(filesYAML), 4, 4)
+			// Keep empty lines as empty (no indentation)
-
+			formattedLines = append(formattedLines, "")
-	// Build metadata sections
+		} else {
-	var metadataSections []string
+			// Add indentation to non-empty lines
-
+			formattedLines = append(formattedLines, "    "+line)
-	// Add license if present
+		}
-	if model.License != "" {
-		metadataSections = append(metadataSections, fmt.Sprintf(`  license: "%s"`, model.License))
 	}
-
+	formattedDescription := strings.Join(formattedLines, "\n")
-	// Add tags if present
+	// Remove any trailing spaces from the formatted description
-	if len(model.Tags) > 0 {
+	formattedDescription = strings.TrimRight(formattedDescription, " \t")
-		tagsYAML, _ := yaml.Marshal(model.Tags)
-		tagsFormatted := formatTextContentWithIndent(string(tagsYAML), 4, 4)
-		tagsFormatted = strings.TrimRight(tagsFormatted, "\n")
-		metadataSections = append(metadataSections, fmt.Sprintf("  tags:\n%s", tagsFormatted))
-	}
-
-	// Add icon if present
-	if model.Icon != "" {
-		metadataSections = append(metadataSections, fmt.Sprintf(`  icon: %s`, model.Icon))
-	}
-
-	// Build the metadata block
-	metadataBlock := ""
-	if len(metadataSections) > 0 {
-		metadataBlock = strings.Join(metadataSections, "\n") + "\n"
-	}
-
 	yamlTemplate := ""
-	yamlTemplate = `- name: "%s"
+	if checksum != "" {
-  url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
+		yamlTemplate = `- !!merge <<: *%s
+  name: "%s"
   urls:
     - https://huggingface.co/%s
   description: |
-%s%s
-  overrides:
 %s
+  overrides:
+    parameters:
+      model: %s
   files:
-%s`
+    - filename: %s
-	// Trim trailing newlines from formatted sections to prevent extra blank lines
+      sha256: %s
-	formattedDescription = strings.TrimRight(formattedDescription, "\n")
+      uri: huggingface://%s/%s`
-	configFile = strings.TrimRight(configFile, "\n")
+		return fmt.Sprintf(yamlTemplate,
-	files = strings.TrimRight(files, "\n")
+			familyAnchor,
-	// Add newline before metadata block if present
+			modelName,
-	if metadataBlock != "" {
+			model.ModelID,
-		metadataBlock = "\n" + strings.TrimRight(metadataBlock, "\n")
+			formattedDescription,
+			fileName,
+			fileName,
+			checksum,
+			model.ModelID,
+			fileName,
+		)
+	} else {
+		yamlTemplate = `- !!merge <<: *%s
+  name: "%s"
+  urls:
+    - https://huggingface.co/%s
+  description: |
+%s
+  overrides:
+    parameters:
+      model: %s`
+		return fmt.Sprintf(yamlTemplate,
+			familyAnchor,
+			modelName,
+			model.ModelID,
+			formattedDescription,
+			fileName,
+		)
 	}
-	return fmt.Sprintf(yamlTemplate,
+}
-		modelName,
+
-		model.ModelID,
+// extractModelFamilies extracts all YAML anchors from the gallery index.yaml file
-		formattedDescription,
+func extractModelFamilies() ([]ModelFamily, error) {
-		metadataBlock,
+	// Read the index.yaml file
-		configFile,
+	indexPath := getGalleryIndexPath()
-		files,
+	content, err := os.ReadFile(indexPath)
-	)
+	if err != nil {
+		return nil, fmt.Errorf("failed to read %s: %w", indexPath, err)
+	}
+
+	lines := strings.Split(string(content), "\n")
+	var families []ModelFamily
+
+	for _, line := range lines {
+		line = strings.TrimSpace(line)
+		// Look for YAML anchors (lines starting with "- &")
+		if strings.HasPrefix(line, "- &") {
+			// Extract the anchor name (everything after "- &")
+			anchor := strings.TrimPrefix(line, "- &")
+			// Remove any trailing colon or other characters
+			anchor = strings.Split(anchor, ":")[0]
+			anchor = strings.Split(anchor, " ")[0]
+
+			if anchor != "" {
+				families = append(families, ModelFamily{
+					Anchor: anchor,
+					Name:   anchor, // Use anchor as name for now
+				})
+			}
+		}
+	}
+
+	return families, nil
 }
 
 // generateYAMLForModels generates YAML entries for selected models and appends to index.yaml
-func generateYAMLForModels(ctx context.Context, models []ProcessedModel, quantization string) error {
+func generateYAMLForModels(ctx context.Context, models []ProcessedModel) error {
+	// Extract available model families
+	families, err := extractModelFamilies()
+	if err != nil {
+		return fmt.Errorf("failed to extract model families: %w", err)
+	}
+
+	fmt.Printf("Found %d model families: %v\n", len(families),
+		func() []string {
+			var names []string
+			for _, f := range families {
+				names = append(names, f.Anchor)
+			}
+			return names
+		}())
 
 	// Generate YAML entries for each model
 	var yamlEntries []string
 	for _, model := range models {
-		fmt.Printf("Generating YAML entry for model: %s\n", model.ModelID)
+		fmt.Printf("Selecting family for model: %s\n", model.ModelID)
+
+		// Select appropriate family for this model
+		familyAnchor, err := selectModelFamily(ctx, model, families)
+		if err != nil {
+			fmt.Printf("Error selecting family for %s: %v, using default\n", model.ModelID, err)
+			familyAnchor = "llama3" // Default fallback
+		}
+
+		fmt.Printf("Selected family '%s' for model %s\n", familyAnchor, model.ModelID)
 
 		// Generate YAML entry
-		yamlEntry := generateYAMLEntry(model, quantization)
+		yamlEntry := generateYAMLEntry(model, familyAnchor)
 		yamlEntries = append(yamlEntries, yamlEntry)
 	}
 
-	// Prepend to index.yaml (write at the top)
+	// Append to index.yaml
 	if len(yamlEntries) > 0 {
 		indexPath := getGalleryIndexPath()
-		fmt.Printf("Prepending YAML entries to %s...\n", indexPath)
+		fmt.Printf("Appending YAML entries to %s...\n", indexPath)
 
 		// Read current content
 		content, err := os.ReadFile(indexPath)
@@ -166,26 +184,11 @@ func generateYAMLForModels(ctx context.Context, models []ProcessedModel, quantiz
 			return fmt.Errorf("failed to read %s: %w", indexPath, err)
 		}
 
-		existingContent := string(content)
+		// Append new entries
+		// Remove trailing whitespace from existing content and join entries without extra newlines
+		existingContent := strings.TrimRight(string(content), " \t\n\r")
 		yamlBlock := strings.Join(yamlEntries, "\n")
-
+		newContent := existingContent + "\n" + yamlBlock + "\n"
-		// Check if file starts with "---"
-		var newContent string
-		if strings.HasPrefix(existingContent, "---\n") {
-			// File starts with "---", prepend new entries after it
-			restOfContent := strings.TrimPrefix(existingContent, "---\n")
-			// Ensure proper spacing: "---\n" + new entries + "\n" + rest of content
-			newContent = "---\n" + yamlBlock + "\n" + restOfContent
-		} else if strings.HasPrefix(existingContent, "---") {
-			// File starts with "---" but no newline after
-			restOfContent := strings.TrimPrefix(existingContent, "---")
-			newContent = "---\n" + yamlBlock + "\n" + strings.TrimPrefix(restOfContent, "\n")
-		} else {
-			// No "---" at start, prepend new entries at the very beginning
-			// Trim leading whitespace from existing content
-			existingContent = strings.TrimLeft(existingContent, " \t\n\r")
-			newContent = yamlBlock + "\n" + existingContent
-		}
 
 		// Write back to file
 		err = os.WriteFile(indexPath, []byte(newContent), 0644)
@@ -193,7 +196,7 @@ func generateYAMLForModels(ctx context.Context, models []ProcessedModel, quantiz
 			return fmt.Errorf("failed to write %s: %w", indexPath, err)
 		}
 
-		fmt.Printf("Successfully prepended %d models to %s\n", len(yamlEntries), indexPath)
+		fmt.Printf("Successfully added %d models to %s\n", len(yamlEntries), indexPath)
 	}
 
 	return nil

.github/gallery-agent/go.mod

@@ -0,0 +1,39 @@
+module github.com/go-skynet/LocalAI/.github/gallery-agent
+
+go 1.24.1
+
+require (
+	github.com/mudler/cogito v0.3.0
+	github.com/onsi/ginkgo/v2 v2.25.3
+	github.com/onsi/gomega v1.38.2
+	github.com/sashabaranov/go-openai v1.41.2
+	github.com/tmc/langchaingo v0.1.13
+	gopkg.in/yaml.v3 v3.0.1
+)
+
+require (
+	dario.cat/mergo v1.0.1 // indirect
+	github.com/Masterminds/goutils v1.1.1 // indirect
+	github.com/Masterminds/semver/v3 v3.4.0 // indirect
+	github.com/Masterminds/sprig/v3 v3.3.0 // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
+	github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
+	github.com/google/go-cmp v0.7.0 // indirect
+	github.com/google/jsonschema-go v0.3.0 // indirect
+	github.com/google/pprof v0.0.0-20250403155104-27863c87afa6 // indirect
+	github.com/google/uuid v1.6.0 // indirect
+	github.com/huandu/xstrings v1.5.0 // indirect
+	github.com/mitchellh/copystructure v1.2.0 // indirect
+	github.com/mitchellh/reflectwalk v1.0.2 // indirect
+	github.com/modelcontextprotocol/go-sdk v1.0.0 // indirect
+	github.com/shopspring/decimal v1.4.0 // indirect
+	github.com/spf13/cast v1.7.0 // indirect
+	github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
+	go.uber.org/automaxprocs v1.6.0 // indirect
+	go.yaml.in/yaml/v3 v3.0.4 // indirect
+	golang.org/x/crypto v0.41.0 // indirect
+	golang.org/x/net v0.43.0 // indirect
+	golang.org/x/sys v0.35.0 // indirect
+	golang.org/x/text v0.28.0 // indirect
+	golang.org/x/tools v0.36.0 // indirect
+)

.github/gallery-agent/go.sum

@@ -0,0 +1,168 @@
+dario.cat/mergo v1.0.1 h1:Ra4+bf83h2ztPIQYNP99R6m+Y7KfnARDfID+a+vLl4s=
+dario.cat/mergo v1.0.1/go.mod h1:uNxQE+84aUszobStD9th8a29P2fMDhsBdgRYvZOxGmk=
+github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 h1:L/gRVlceqvL25UVaW/CKtUDjefjrs0SPonmDGUVOYP0=
+github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
+github.com/Masterminds/goutils v1.1.1 h1:5nUrii3FMTL5diU80unEVvNevw1nH4+ZV4DSLVJLSYI=
+github.com/Masterminds/goutils v1.1.1/go.mod h1:8cTjp+g8YejhMuvIA5y2vz3BpJxksy863GQaJW2MFNU=
+github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
+github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
+github.com/Masterminds/sprig/v3 v3.3.0 h1:mQh0Yrg1XPo6vjYXgtf5OtijNAKJRNcTdOOGZe3tPhs=
+github.com/Masterminds/sprig/v3 v3.3.0/go.mod h1:Zy1iXRYNqNLUolqCpL4uhk6SHUMAOSCzdgBfDb35Lz0=
+github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
+github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
+github.com/cenkalti/backoff v2.2.1+incompatible h1:tNowT99t7UNflLxfYYSlKYsBpXdEet03Pg2g16Swow4=
+github.com/cenkalti/backoff/v4 v4.2.1 h1:y4OZtCnogmCPw98Zjyt5a6+QwPLGkiQsYW5oUqylYbM=
+github.com/cenkalti/backoff/v4 v4.2.1/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
+github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
+github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M=
+github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE=
+github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmCQvQ4GpJHEqRLVk=
+github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
+github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
+github.com/containerd/platforms v0.2.1 h1:zvwtM3rz2YHPQsF2CHYM8+KtB5dvhISiXh5ZpSBQv6A=
+github.com/containerd/platforms v0.2.1/go.mod h1:XHCb+2/hzowdiut9rkudds9bE5yJ7npe7dG/wG+uFPw=
+github.com/cpuguy83/dockercfg v0.3.2 h1:DlJTyZGBDlXqUZ2Dk2Q3xHs/FtnooJJVaad2S9GKorA=
+github.com/cpuguy83/dockercfg v0.3.2/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHfjj5/jFyUJc=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
+github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
+github.com/docker/docker v28.2.2+incompatible h1:CjwRSksz8Yo4+RmQ339Dp/D2tGO5JxwYeqtMOEe0LDw=
+github.com/docker/docker v28.2.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
+github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c=
+github.com/docker/go-connections v0.5.0/go.mod h1:ov60Kzw0kKElRwhNs9UlUHAE/F9Fe6GLaXnqyDdmEXc=
+github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
+github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
+github.com/ebitengine/purego v0.8.4 h1:CF7LEKg5FFOsASUj0+QwaXf8Ht6TlFxg09+S9wz0omw=
+github.com/ebitengine/purego v0.8.4/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
+github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
+github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
+github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
+github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
+github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
+github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
+github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
+github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
+github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
+github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
+github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
+github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
+github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
+github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
+github.com/google/jsonschema-go v0.3.0 h1:6AH2TxVNtk3IlvkkhjrtbUc4S8AvO0Xii0DxIygDg+Q=
+github.com/google/jsonschema-go v0.3.0/go.mod h1:r5quNTdLOYEz95Ru18zA0ydNbBuYoo9tgaYcxEYhJVE=
+github.com/google/pprof v0.0.0-20250403155104-27863c87afa6 h1:BHT72Gu3keYf3ZEu2J0b1vyeLSOYI8bm5wbJM/8yDe8=
+github.com/google/pprof v0.0.0-20250403155104-27863c87afa6/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/huandu/xstrings v1.5.0 h1:2ag3IFq9ZDANvthTwTiqSSZLjDc+BedvHPAp5tJy2TI=
+github.com/huandu/xstrings v1.5.0/go.mod h1:y5/lhBue+AyNmUVz9RLU9xbLR0o4KIIExikq4ovT0aE=
+github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
+github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
+github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
+github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
+github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
+github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4=
+github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
+github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE=
+github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
+github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw=
+github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s=
+github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ=
+github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
+github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
+github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
+github.com/moby/go-archive v0.1.0 h1:Kk/5rdW/g+H8NHdJW2gsXyZ7UnzvJNOy6VKJqueWdcQ=
+github.com/moby/go-archive v0.1.0/go.mod h1:G9B+YoujNohJmrIYFBpSd54GTUB4lt9S+xVQvsJyFuo=
+github.com/moby/patternmatcher v0.6.0 h1:GmP9lR19aU5GqSSFko+5pRqHi+Ohk1O69aFiKkVGiPk=
+github.com/moby/patternmatcher v0.6.0/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc=
+github.com/moby/sys/sequential v0.6.0 h1:qrx7XFUd/5DxtqcoH1h438hF5TmOvzC/lspjy7zgvCU=
+github.com/moby/sys/sequential v0.6.0/go.mod h1:uyv8EUTrca5PnDsdMGXhZe6CCe8U/UiTWd+lL+7b/Ko=
+github.com/moby/sys/user v0.4.0 h1:jhcMKit7SA80hivmFJcbB1vqmw//wU61Zdui2eQXuMs=
+github.com/moby/sys/user v0.4.0/go.mod h1:bG+tYYYJgaMtRKgEmuueC0hJEAZWwtIbZTB+85uoHjs=
+github.com/moby/sys/userns v0.1.0 h1:tVLXkFOxVu9A64/yh59slHVv9ahO9UIev4JZusOLG/g=
+github.com/moby/sys/userns v0.1.0/go.mod h1:IHUYgu/kao6N8YZlp9Cf444ySSvCmDlmzUcYfDHOl28=
+github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0=
+github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y=
+github.com/modelcontextprotocol/go-sdk v1.0.0 h1:Z4MSjLi38bTgLrd/LjSmofqRqyBiVKRyQSJgw8q8V74=
+github.com/modelcontextprotocol/go-sdk v1.0.0/go.mod h1:nYtYQroQ2KQiM0/SbyEPUWQ6xs4B95gJjEalc9AQyOs=
+github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A=
+github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
+github.com/mudler/cogito v0.3.0 h1:NbVAO3bLkK5oGSY0xq87jlz8C9OIsLW55s+8Hfzeu9s=
+github.com/mudler/cogito v0.3.0/go.mod h1:abMwl+CUjCp87IufA2quZdZt0bbLaHHN79o17HbUKxU=
+github.com/onsi/ginkgo/v2 v2.25.3 h1:Ty8+Yi/ayDAGtk4XxmmfUy4GabvM+MegeB4cDLRi6nw=
+github.com/onsi/ginkgo/v2 v2.25.3/go.mod h1:43uiyQC4Ed2tkOzLsEYm7hnrb7UJTWHYNsuy3bG/snE=
+github.com/onsi/gomega v1.38.2 h1:eZCjf2xjZAqe+LeWvKb5weQ+NcPwX84kqJ0cZNxok2A=
+github.com/onsi/gomega v1.38.2/go.mod h1:W2MJcYxRGV63b418Ai34Ud0hEdTVXq9NW9+Sx6uXf3k=
+github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
+github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
+github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
+github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
+github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c h1:ncq/mPwQF4JjgDlrVEn3C11VoGHZN7m8qihwgMEtzYw=
+github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
+github.com/prashantv/gostub v1.1.0 h1:BTyx3RfQjRHnUWaGF9oQos79AlQ5k8WNktv7VGvVH4g=
+github.com/prashantv/gostub v1.1.0/go.mod h1:A5zLQHz7ieHGG7is6LLXLz7I8+3LZzsrV0P1IAHhP5U=
+github.com/rogpeppe/go-internal v1.11.0 h1:cWPaGQEPrBb5/AsnsZesgZZ9yb1OQ+GOISoDNXVBh4M=
+github.com/rogpeppe/go-internal v1.11.0/go.mod h1:ddIwULY96R17DhadqLgMfk9H9tvdUzkipdSkR5nkCZA=
+github.com/sashabaranov/go-openai v1.41.2 h1:vfPRBZNMpnqu8ELsclWcAvF19lDNgh1t6TVfFFOPiSM=
+github.com/sashabaranov/go-openai v1.41.2/go.mod h1:lj5b/K+zjTSFxVLijLSTDZuP7adOgerWeFyZLUhAKRg=
+github.com/shirou/gopsutil/v4 v4.25.5 h1:rtd9piuSMGeU8g1RMXjZs9y9luK5BwtnG7dZaQUJAsc=
+github.com/shirou/gopsutil/v4 v4.25.5/go.mod h1:PfybzyydfZcN+JMMjkF6Zb8Mq1A/VcogFFg7hj50W9c=
+github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
+github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
+github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
+github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
+github.com/spf13/cast v1.7.0 h1:ntdiHjuueXFgm5nzDRdOS4yfT43P5Fnud6DH50rz/7w=
+github.com/spf13/cast v1.7.0/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
+github.com/testcontainers/testcontainers-go v0.38.0 h1:d7uEapLcv2P8AvH8ahLqDMMxda2W9gQN1nRbHS28HBw=
+github.com/testcontainers/testcontainers-go v0.38.0/go.mod h1:C52c9MoHpWO+C4aqmgSU+hxlR5jlEayWtgYrb8Pzz1w=
+github.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFAEVmqU=
+github.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI=
+github.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk=
+github.com/tklauser/numcpus v0.6.1/go.mod h1:1XfjsgE2zo8GVw7POkMbHENHzVg3GzmoZ9fESEdAacY=
+github.com/tmc/langchaingo v0.1.13 h1:rcpMWBIi2y3B90XxfE4Ao8dhCQPVDMaNPnN5cGB1CaA=
+github.com/tmc/langchaingo v0.1.13/go.mod h1:vpQ5NOIhpzxDfTZK9B6tf2GM/MoaHewPWM5KXXGh7hg=
+github.com/yosida95/uritemplate/v3 v3.0.2 h1:Ed3Oyj9yrmi9087+NczuL5BwkIc4wvTb5zIM+UJPGz4=
+github.com/yosida95/uritemplate/v3 v3.0.2/go.mod h1:ILOh0sOhIJR3+L/8afwt/kE++YT040gmv5BQTMR2HP4=
+github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
+github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
+go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
+go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
+go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.51.0 h1:Xs2Ncz0gNihqu9iosIZ5SkBbWo5T8JhhLJFMQL1qmLI=
+go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.51.0/go.mod h1:vy+2G/6NvVMpwGX/NyLqcC41fxepnuKHk16E6IZUcJc=
+go.opentelemetry.io/otel v1.38.0 h1:RkfdswUDRimDg0m2Az18RKOsnI8UDzppJAtj01/Ymk8=
+go.opentelemetry.io/otel v1.38.0/go.mod h1:zcmtmQ1+YmQM9wrNsTGV/q/uyusom3P8RxwExxkZhjM=
+go.opentelemetry.io/otel/metric v1.38.0 h1:Kl6lzIYGAh5M159u9NgiRkmoMKjvbsKtYRwgfrA6WpA=
+go.opentelemetry.io/otel/metric v1.38.0/go.mod h1:kB5n/QoRM8YwmUahxvI3bO34eVtQf2i4utNVLr9gEmI=
+go.opentelemetry.io/otel/trace v1.38.0 h1:Fxk5bKrDZJUH+AMyyIXGcFAPah0oRcT+LuNtJrmcNLE=
+go.opentelemetry.io/otel/trace v1.38.0/go.mod h1:j1P9ivuFsTceSWe1oY+EeW3sc+Pp42sO++GHkg4wwhs=
+go.uber.org/automaxprocs v1.6.0 h1:O3y2/QNTOdbF+e/dpXNNW7Rx2hZ4sTIPyybbxyNqTUs=
+go.uber.org/automaxprocs v1.6.0/go.mod h1:ifeIMSnPZuznNm6jmdzmU3/bfk01Fe2fotchwEFJ8r8=
+go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
+go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
+golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4=
+golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc=
+golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE=
+golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg=
+golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
+golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
+golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
+golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
+golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg=
+golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s=
+google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
+google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

.github/gallery-agent/hfapi/client.go

@@ -51,7 +51,6 @@ type ModelFile struct {
 	Size     int64
 	SHA256   string
 	IsReadme bool
-	URL      string
 }
 
 // ModelDetails represents detailed information about a model
@@ -148,15 +147,10 @@ func (c *Client) SetBaseURL(url string) {
 	c.baseURL = url
 }
 
-// listFilesInPath lists all files in a specific path of a HuggingFace repository (recursive helper)
+// ListFiles lists all files in a HuggingFace repository
-func (c *Client) listFilesInPath(repoID, path string) ([]FileInfo, error) {
+func (c *Client) ListFiles(repoID string) ([]FileInfo, error) {
 	baseURL := strings.TrimSuffix(c.baseURL, "/api/models")
-	var url string
+	url := fmt.Sprintf("%s/api/models/%s/tree/main", baseURL, repoID)
-	if path == "" {
-		url = fmt.Sprintf("%s/api/models/%s/tree/main", baseURL, repoID)
-	} else {
-		url = fmt.Sprintf("%s/api/models/%s/tree/main/%s", baseURL, repoID, path)
-	}
 
 	req, err := http.NewRequest("GET", url, nil)
 	if err != nil {
@@ -178,52 +172,19 @@ func (c *Client) listFilesInPath(repoID, path string) ([]FileInfo, error) {
 		return nil, fmt.Errorf("failed to read response body: %w", err)
 	}
 
-	var items []FileInfo
+	var files []FileInfo
-	if err := json.Unmarshal(body, &items); err != nil {
+	if err := json.Unmarshal(body, &files); err != nil {
 		return nil, fmt.Errorf("failed to parse JSON response: %w", err)
 	}
 
-	var allFiles []FileInfo
+	return files, nil
-	for _, item := range items {
-		switch item.Type {
-		// If it's a directory/folder, recursively list its contents
-		case "directory", "folder":
-			// Build the subfolder path
-			subPath := item.Path
-			if path != "" {
-				subPath = fmt.Sprintf("%s/%s", path, item.Path)
-			}
-
-			// Recursively get files from subfolder
-			// The recursive call will already prepend the subPath to each file's path
-			subFiles, err := c.listFilesInPath(repoID, subPath)
-			if err != nil {
-				return nil, fmt.Errorf("failed to list files in subfolder %s: %w", subPath, err)
-			}
-
-			allFiles = append(allFiles, subFiles...)
-		case "file":
-			// It's a file, prepend the current path to make it relative to root
-			//	if path != "" {
-			//		item.Path = fmt.Sprintf("%s/%s", path, item.Path)
-			//	}
-			allFiles = append(allFiles, item)
-		}
-	}
-
-	return allFiles, nil
-}
-
-// ListFiles lists all files in a HuggingFace repository, including files in subfolders
-func (c *Client) ListFiles(repoID string) ([]FileInfo, error) {
-	return c.listFilesInPath(repoID, "")
 }
 
 // GetFileSHA gets the SHA256 checksum for a specific file by searching through the file list
 func (c *Client) GetFileSHA(repoID, fileName string) (string, error) {
 	files, err := c.ListFiles(repoID)
 	if err != nil {
-		return "", fmt.Errorf("failed to list files while getting SHA: %w", err)
+		return "", fmt.Errorf("failed to list files: %w", err)
 	}
 
 	for _, file := range files {
@@ -254,7 +215,6 @@ func (c *Client) GetModelDetails(repoID string) (*ModelDetails, error) {
 	}
 
 	// Process each file
-	baseURL := strings.TrimSuffix(c.baseURL, "/api/models")
 	for _, file := range files {
 		fileName := filepath.Base(file.Path)
 		isReadme := strings.Contains(strings.ToLower(fileName), "readme")
@@ -267,16 +227,11 @@ func (c *Client) GetModelDetails(repoID string) (*ModelDetails, error) {
 			sha256 = file.Oid
 		}
 
-		// Construct the full URL for the file
-		// Use /resolve/main/ for downloading files (handles LFS properly)
-		fileURL := fmt.Sprintf("%s/%s/resolve/main/%s", baseURL, repoID, file.Path)
-
 		modelFile := ModelFile{
 			Path:     file.Path,
 			Size:     file.Size,
 			SHA256:   sha256,
 			IsReadme: isReadme,
-			URL:      fileURL,
 		}
 
 		details.Files = append(details.Files, modelFile)

.github/gallery-agent/hfapi/client_test.go

@@ -1,7 +1,6 @@
 package hfapi_test
 
 import (
-	"fmt"
 	"net/http"
 	"net/http/httptest"
 	"strings"
@@ -9,7 +8,7 @@ import (
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
 
-	hfapi "github.com/mudler/LocalAI/pkg/huggingface-api"
+	"github.com/go-skynet/LocalAI/.github/gallery-agent/hfapi"
 )
 
 var _ = Describe("HuggingFace API Client", func() {
@@ -271,15 +270,6 @@ var _ = Describe("HuggingFace API Client", func() {
 		})
 	})
 
-	Context("when getting file SHA on remote model", func() {
-		It("should get file SHA successfully", func() {
-			sha, err := client.GetFileSHA(
-				"mudler/LocalAI-functioncall-qwen2.5-7b-v0.5-Q4_K_M-GGUF", "localai-functioncall-qwen2.5-7b-v0.5-q4_k_m.gguf")
-			Expect(err).ToNot(HaveOccurred())
-			Expect(sha).To(Equal("4e7b7fe1d54b881f1ef90799219dc6cc285d29db24f559c8998d1addb35713d4"))
-		})
-	})
-
 	Context("when listing files", func() {
 		BeforeEach(func() {
 			mockFilesResponse := `[
@@ -337,158 +327,25 @@ var _ = Describe("HuggingFace API Client", func() {
 		})
 	})
 
-	Context("when listing files with subfolders", func() {
-		BeforeEach(func() {
-			// Mock response for root directory with files and a subfolder
-			mockRootResponse := `[
-				{
-					"type": "file",
-					"path": "README.md",
-					"size": 5000,
-					"oid": "readme123"
-				},
-				{
-					"type": "directory",
-					"path": "subfolder",
-					"size": 0,
-					"oid": "dir123"
-				},
-				{
-					"type": "file",
-					"path": "config.json",
-					"size": 1000,
-					"oid": "config123"
-				}
-			]`
-
-			// Mock response for subfolder directory
-			mockSubfolderResponse := `[
-				{
-					"type": "file",
-					"path": "subfolder/file.bin",
-					"size": 2000000,
-					"oid": "filebin123",
-					"lfs": {
-						"oid": "filebin456",
-						"size": 2000000,
-						"pointerSize": 135
-					}
-				},
-				{
-					"type": "directory",
-					"path": "nested",
-					"size": 0,
-					"oid": "nesteddir123"
-				}
-			]`
-
-			// Mock response for nested subfolder
-			mockNestedResponse := `[
-				{
-					"type": "file",
-					"path": "subfolder/nested/nested_file.gguf",
-					"size": 5000000,
-					"oid": "nested123",
-					"lfs": {
-						"oid": "nested456",
-						"size": 5000000,
-						"pointerSize": 135
-					}
-				}
-			]`
-
-			server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-				urlPath := r.URL.Path
-				w.Header().Set("Content-Type", "application/json")
-				w.WriteHeader(http.StatusOK)
-
-				if strings.Contains(urlPath, "/tree/main/subfolder/nested") {
-					w.Write([]byte(mockNestedResponse))
-				} else if strings.Contains(urlPath, "/tree/main/subfolder") {
-					w.Write([]byte(mockSubfolderResponse))
-				} else if strings.Contains(urlPath, "/tree/main") {
-					w.Write([]byte(mockRootResponse))
-				} else {
-					w.WriteHeader(http.StatusNotFound)
-				}
-			}))
-
-			client.SetBaseURL(server.URL)
-		})
-
-		It("should recursively list all files including those in subfolders", func() {
-			files, err := client.ListFiles("test/model")
-
-			Expect(err).ToNot(HaveOccurred())
-			Expect(files).To(HaveLen(4))
-
-			// Verify root level files
-			readmeFile := findFileByPath(files, "README.md")
-			Expect(readmeFile).ToNot(BeNil())
-			Expect(readmeFile.Size).To(Equal(int64(5000)))
-			Expect(readmeFile.Oid).To(Equal("readme123"))
-
-			configFile := findFileByPath(files, "config.json")
-			Expect(configFile).ToNot(BeNil())
-			Expect(configFile.Size).To(Equal(int64(1000)))
-			Expect(configFile.Oid).To(Equal("config123"))
-
-			// Verify subfolder file with relative path
-			subfolderFile := findFileByPath(files, "subfolder/file.bin")
-			Expect(subfolderFile).ToNot(BeNil())
-			Expect(subfolderFile.Size).To(Equal(int64(2000000)))
-			Expect(subfolderFile.LFS).ToNot(BeNil())
-			Expect(subfolderFile.LFS.Oid).To(Equal("filebin456"))
-
-			// Verify nested subfolder file
-			nestedFile := findFileByPath(files, "subfolder/nested/nested_file.gguf")
-			Expect(nestedFile).ToNot(BeNil())
-			Expect(nestedFile.Size).To(Equal(int64(5000000)))
-			Expect(nestedFile.LFS).ToNot(BeNil())
-			Expect(nestedFile.LFS.Oid).To(Equal("nested456"))
-		})
-
-		It("should handle files with correct relative paths", func() {
-			files, err := client.ListFiles("test/model")
-
-			Expect(err).ToNot(HaveOccurred())
-
-			// Check that all paths are relative and correct
-			paths := make([]string, len(files))
-			for i, file := range files {
-				paths[i] = file.Path
-			}
-
-			Expect(paths).To(ContainElements(
-				"README.md",
-				"config.json",
-				"subfolder/file.bin",
-				"subfolder/nested/nested_file.gguf",
-			))
-		})
-	})
-
 	Context("when getting file SHA", func() {
 		BeforeEach(func() {
-			mockFilesResponse := `[
+			mockFileInfoResponse := `{
-				{
+				"path": "model-Q4_K_M.gguf",
-					"type": "file",
+				"size": 1000000,
-					"path": "model-Q4_K_M.gguf",
+				"oid": "abc123",
+				"lfs": {
+					"oid": "sha256:def456",
 					"size": 1000000,
-					"oid": "abc123",
+					"pointer": "version https://git-lfs.github.com/spec/v1",
-					"lfs": {
+					"sha256": "def456789"
-						"oid": "def456789",
-						"size": 1000000,
-						"pointerSize": 135
-					}
 				}
-			]`
+			}`
 
 			server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-				if strings.Contains(r.URL.Path, "/tree/main") {
+				if strings.Contains(r.URL.Path, "/paths-info") {
 					w.Header().Set("Content-Type", "application/json")
 					w.WriteHeader(http.StatusOK)
-					w.Write([]byte(mockFilesResponse))
+					w.Write([]byte(mockFileInfoResponse))
 				} else {
 					w.WriteHeader(http.StatusNotFound)
 				}
@@ -506,29 +363,18 @@ var _ = Describe("HuggingFace API Client", func() {
 
 		It("should handle missing SHA gracefully", func() {
 			server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-				if strings.Contains(r.URL.Path, "/tree/main") {
+				w.Header().Set("Content-Type", "application/json")
-					w.Header().Set("Content-Type", "application/json")
+				w.WriteHeader(http.StatusOK)
-					w.WriteHeader(http.StatusOK)
+				w.Write([]byte(`{"path": "file.txt", "size": 100}`))
-					w.Write([]byte(`[
-						{
-							"type": "file",
-							"path": "file.txt",
-							"size": 100,
-							"oid": "file123"
-						}
-					]`))
-				} else {
-					w.WriteHeader(http.StatusNotFound)
-				}
 			}))
 
 			client.SetBaseURL(server.URL)
 
 			sha, err := client.GetFileSHA("test/model", "file.txt")
 
-			Expect(err).ToNot(HaveOccurred())
+			Expect(err).To(HaveOccurred())
-			// When there's no LFS, it should return the OID
+			Expect(err.Error()).To(ContainSubstring("no SHA256 found"))
-			Expect(sha).To(Equal("file123"))
+			Expect(sha).To(Equal(""))
 		})
 	})
 
@@ -536,7 +382,6 @@ var _ = Describe("HuggingFace API Client", func() {
 		BeforeEach(func() {
 			mockFilesResponse := `[
 				{
-					"type": "file",
 					"path": "model-Q4_K_M.gguf",
 					"size": 1000000,
 					"oid": "abc123",
@@ -548,7 +393,6 @@ var _ = Describe("HuggingFace API Client", func() {
 					}
 				},
 				{
-					"type": "file",
 					"path": "README.md",
 					"size": 5000,
 					"oid": "readme123"
@@ -595,13 +439,6 @@ var _ = Describe("HuggingFace API Client", func() {
 			Expect(details.ReadmeFile).ToNot(BeNil())
 			Expect(details.ReadmeFile.Path).To(Equal("README.md"))
 			Expect(details.ReadmeFile.IsReadme).To(BeTrue())
-
-			// Verify URLs are set for all files
-			baseURL := strings.TrimSuffix(server.URL, "/api/models")
-			for _, file := range details.Files {
-				expectedURL := fmt.Sprintf("%s/test/model/resolve/main/%s", baseURL, file.Path)
-				Expect(file.URL).To(Equal(expectedURL))
-			}
 		})
 	})
 
@@ -671,84 +508,4 @@ var _ = Describe("HuggingFace API Client", func() {
 			Expect(preferred).To(BeNil())
 		})
 	})
-
-	Context("integration test with real HuggingFace API", func() {
-		It("should recursively list all files including subfolders from real repository", func() {
-			// This test makes actual API calls to HuggingFace
-			// Skip if running in CI or if network is not available
-			realClient := hfapi.NewClient()
-			repoID := "bartowski/Qwen_Qwen3-Next-80B-A3B-Instruct-GGUF"
-
-			files, err := realClient.ListFiles(repoID)
-
-			Expect(err).ToNot(HaveOccurred())
-			Expect(files).ToNot(BeEmpty(), "should return at least some files")
-
-			// Verify that we get files from subfolders
-			// Based on the repository structure, there should be files in subfolders like:
-			// - Qwen_Qwen3-Next-80B-A3B-Instruct-Q4_1/...
-			// - Qwen_Qwen3-Next-80B-A3B-Instruct-Q5_K_L/...
-			// etc.
-			hasSubfolderFiles := false
-			rootLevelFiles := 0
-			subfolderFiles := 0
-
-			for _, file := range files {
-				if strings.Contains(file.Path, "/") {
-					hasSubfolderFiles = true
-					subfolderFiles++
-					// Verify the path format is correct (subfolder/file.gguf)
-					Expect(file.Path).ToNot(HavePrefix("/"), "paths should be relative, not absolute")
-					Expect(file.Path).ToNot(HaveSuffix("/"), "file paths should not end with /")
-				} else {
-					rootLevelFiles++
-				}
-			}
-
-			Expect(hasSubfolderFiles).To(BeTrue(), "should find files in subfolders")
-			Expect(rootLevelFiles).To(BeNumerically(">", 0), "should find files at root level")
-			Expect(subfolderFiles).To(BeNumerically(">", 0), "should find files in subfolders")
-			// Verify specific expected files exist
-			// Root level files
-			readmeFile := findFileByPath(files, "README.md")
-			Expect(readmeFile).ToNot(BeNil(), "README.md should exist at root level")
-
-			// Verify we can find files in subfolders
-			// Look for any file in a subfolder (the exact structure may vary, can be nested)
-			foundSubfolderFile := false
-			for _, file := range files {
-				if strings.Contains(file.Path, "/") && strings.HasSuffix(file.Path, ".gguf") {
-					foundSubfolderFile = true
-					// Verify the path structure: can be nested like subfolder/subfolder/file.gguf
-					parts := strings.Split(file.Path, "/")
-					Expect(len(parts)).To(BeNumerically(">=", 2), "subfolder files should have at least subfolder/file.gguf format")
-					// The last part should be the filename
-					Expect(parts[len(parts)-1]).To(HaveSuffix(".gguf"), "file in subfolder should be a .gguf file")
-					Expect(parts[len(parts)-1]).ToNot(BeEmpty(), "filename should not be empty")
-					break
-				}
-			}
-			Expect(foundSubfolderFile).To(BeTrue(), "should find at least one .gguf file in a subfolder")
-
-			// Verify file properties are populated
-			for _, file := range files {
-				Expect(file.Path).ToNot(BeEmpty(), "file path should not be empty")
-				Expect(file.Type).To(Equal("file"), "all returned items should be files, not directories")
-				// Size might be 0 for some files, but OID should be present
-				if file.LFS == nil {
-					Expect(file.Oid).ToNot(BeEmpty(), "file should have an OID if no LFS")
-				}
-			}
-		})
-	})
 })
-
-// findFileByPath is a helper function to find a file by its path in a slice of FileInfo
-func findFileByPath(files []hfapi.FileInfo, path string) *hfapi.FileInfo {
-	for i := range files {
-		if files[i].Path == path {
-			return &files[i]
-		}
-	}
-	return nil
-}

.github/gallery-agent/hfapi/hfapi_suite_test.go

@@ -11,5 +11,3 @@ func TestHfapi(t *testing.T) {
 	RegisterFailHandler(Fail)
 	RunSpecs(t, "HuggingFace API Suite")
 }
-
-

.github/gallery-agent/main.go

@@ -9,7 +9,7 @@ import (
 	"strings"
 	"time"
 
-	hfapi "github.com/mudler/LocalAI/pkg/huggingface-api"
+	"github.com/go-skynet/LocalAI/.github/gallery-agent/hfapi"
 )
 
 // ProcessedModelFile represents a processed model file with additional metadata
@@ -34,9 +34,6 @@ type ProcessedModel struct {
 	ReadmeContentPreview    string               `json:"readme_content_preview,omitempty"`
 	QuantizationPreferences []string             `json:"quantization_preferences"`
 	ProcessingError         string               `json:"processing_error,omitempty"`
-	Tags                    []string             `json:"tags,omitempty"`
-	License                 string               `json:"license,omitempty"`
-	Icon                    string               `json:"icon,omitempty"`
 }
 
 // SearchResult represents the complete result of searching and processing models
@@ -119,24 +116,14 @@ func main() {
 	}
 
 	fmt.Println(result.FormattedOutput)
-	var models []ProcessedModel
 
-	if len(result.Models) > 1 {
+	// Use AI agent to select the most interesting models
-		fmt.Println("More than one model found (", len(result.Models), "), using AI agent to select the most interesting models")
+	fmt.Println("Using AI agent to select the most interesting models...")
-		for _, model := range result.Models {
+	models, err := selectMostInterestingModels(context.Background(), result)
-			fmt.Println("Model: ", model.ModelID)
+	if err != nil {
-		}
+		fmt.Fprintf(os.Stderr, "Error in model selection: %v\n", err)
-		// Use AI agent to select the most interesting models
+		// Continue with original result if selection fails
-		fmt.Println("Using AI agent to select the most interesting models...")
-		models, err = selectMostInterestingModels(context.Background(), result)
-		if err != nil {
-			fmt.Fprintf(os.Stderr, "Error in model selection: %v\n", err)
-			// Continue with original result if selection fails
-			models = result.Models
-		}
-	} else if len(result.Models) == 1 {
 		models = result.Models
-		fmt.Println("Only one model found, using it directly")
 	}
 
 	fmt.Print(models)
@@ -167,7 +154,7 @@ func main() {
 			addedModelURLs = append(addedModelURLs, modelURL)
 		}
 		fmt.Println("Generating YAML entries for selected models...")
-		err = generateYAMLForModels(context.Background(), models, quantization)
+		err = generateYAMLForModels(context.Background(), models)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Error generating YAML entries: %v\n", err)
 			os.Exit(1)
@@ -325,28 +312,9 @@ func searchAndProcessModels(searchTerm string, limit int, quantization string) (
 				outputBuilder.WriteString(fmt.Sprintf("   README Content Preview: %s\n",
 					processedModel.ReadmeContentPreview))
 			} else {
-				fmt.Printf("   Warning: Failed to get real readme: %v\n", err)
+				continue
 			}
 			fmt.Println("Real readme got", readmeContent)
-
-			// Extract metadata (tags, license) from README using LLM
-			fmt.Println("Extracting metadata for", model.ModelID, "waiting...")
-			tags, license, err := extractModelMetadata(context.Background(), processedModel)
-			if err == nil {
-				processedModel.Tags = tags
-				processedModel.License = license
-				outputBuilder.WriteString(fmt.Sprintf("   Tags: %v\n", tags))
-				outputBuilder.WriteString(fmt.Sprintf("   License: %s\n", license))
-			} else {
-				fmt.Printf("   Warning: Failed to extract metadata: %v\n", err)
-			}
-
-			// Extract icon from README or use HuggingFace avatar
-			icon := extractModelIcon(processedModel)
-			if icon != "" {
-				processedModel.Icon = icon
-				outputBuilder.WriteString(fmt.Sprintf("   Icon: %s\n", icon))
-			}
 			// Get README content
 			// readmeContent, err := client.GetReadmeContent(model.ModelID, details.ReadmeFile.Path)
 			// if err == nil {

.github/gallery-agent/testing.go

@@ -25,7 +25,7 @@ func runSyntheticMode() error {
 
 	// Generate YAML entries and append to gallery/index.yaml
 	fmt.Println("Generating YAML entries for synthetic models...")
-	err := generateYAMLForModels(context.Background(), models, "Q4_K_M")
+	err := generateYAMLForModels(context.Background(), models)
 	if err != nil {
 		return fmt.Errorf("error generating YAML entries: %w", err)
 	}
@@ -138,25 +138,6 @@ func (g *SyntheticDataGenerator) GenerateProcessedModel() ProcessedModel {
 
 	readmeContent := g.generateReadmeContent(modelName, author)
 
-	// Generate sample metadata
-	licenses := []string{"apache-2.0", "mit", "llama2", "gpl-3.0", "bsd", ""}
-	license := licenses[g.rand.Intn(len(licenses))]
-
-	sampleTags := []string{"llm", "gguf", "gpu", "cpu", "text-to-text", "chat", "instruction-tuned"}
-	numTags := g.rand.Intn(4) + 3 // 3-6 tags
-	tags := make([]string, numTags)
-	for i := 0; i < numTags; i++ {
-		tags[i] = sampleTags[g.rand.Intn(len(sampleTags))]
-	}
-	// Remove duplicates
-	tags = g.removeDuplicates(tags)
-
-	// Optionally include icon (50% chance)
-	icon := ""
-	if g.rand.Intn(2) == 0 {
-		icon = fmt.Sprintf("https://cdn-avatars.huggingface.co/v1/production/uploads/%s.png", g.randomString(24))
-	}
-
 	return ProcessedModel{
 		ModelID:                 modelID,
 		Author:                  author,
@@ -169,9 +150,6 @@ func (g *SyntheticDataGenerator) GenerateProcessedModel() ProcessedModel {
 		ReadmeContentPreview:    truncateString(readmeContent, 200),
 		QuantizationPreferences: []string{"Q4_K_M", "Q4_K_S", "Q3_K_M", "Q2_K"},
 		ProcessingError:         "",
-		Tags:                    tags,
-		License:                 license,
-		Icon:                    icon,
 	}
 }
 
@@ -201,18 +179,6 @@ func (g *SyntheticDataGenerator) randomDate() string {
 	return pastDate.Format("2006-01-02T15:04:05.000Z")
 }
 
-func (g *SyntheticDataGenerator) removeDuplicates(slice []string) []string {
-	keys := make(map[string]bool)
-	result := []string{}
-	for _, item := range slice {
-		if !keys[item] {
-			keys[item] = true
-			result = append(result, item)
-		}
-	}
-	return result
-}
-
 func (g *SyntheticDataGenerator) generateReadmeContent(modelName, author string) string {
 	templates := []string{
 		fmt.Sprintf("# %s Model\n\nThis is a %s model developed by %s. It's designed for various natural language processing tasks including text generation, question answering, and conversation.\n\n## Features\n\n- High-quality text generation\n- Efficient inference\n- Multiple quantization options\n- Easy to use with LocalAI\n\n## Usage\n\nUse this model with LocalAI for various AI tasks.", strings.Title(modelName), modelName, author),

.github/gallery-agent/tools.go

@@ -3,9 +3,9 @@ package main
 import (
 	"fmt"
 
-	hfapi "github.com/mudler/LocalAI/pkg/huggingface-api"
+	"github.com/go-skynet/LocalAI/.github/gallery-agent/hfapi"
-	openai "github.com/sashabaranov/go-openai"
+	"github.com/sashabaranov/go-openai"
-	jsonschema "github.com/sashabaranov/go-openai/jsonschema"
+	"github.com/tmc/langchaingo/jsonschema"
 )
 
 // Get repository README from HF
@@ -13,16 +13,16 @@ type HFReadmeTool struct {
 	client *hfapi.Client
 }
 
-func (s *HFReadmeTool) Execute(args map[string]any) (string, any, error) {
+func (s *HFReadmeTool) Run(args map[string]any) (string, error) {
 	q, ok := args["repository"].(string)
 	if !ok {
-		return "", nil, fmt.Errorf("no query")
+		return "", fmt.Errorf("no query")
 	}
 	readme, err := s.client.GetReadmeContent(q, "README.md")
 	if err != nil {
-		return "", nil, err
+		return "", err
 	}
-	return readme, nil, nil
+	return readme, nil
 }
 
 func (s *HFReadmeTool) Tool() openai.Tool {

.github/workflows/backend_build.yml

@@ -1,5 +1,5 @@
 ---
-name: 'build backend container images (reusable)'
+name: 'build python backend container images (reusable)'
 
 on:
   workflow_call:
@@ -53,11 +53,6 @@ on:
         description: 'Skip drivers'
         default: 'false'
         type: string
-      ubuntu-version:
-        description: 'Ubuntu version'
-        required: false
-        default: '2204'
-        type: string
     secrets:
       dockerUsername:
         required: false
@@ -102,7 +97,7 @@ jobs:
           && sudo apt-get install -y git
 
       - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
 
       - name: Release space from worker
         if: inputs.runs-on == 'ubuntu-latest'
@@ -149,7 +144,7 @@ jobs:
       - name: Docker meta
         id: meta
         if: github.event_name != 'pull_request'
-        uses: docker/metadata-action@v6
+        uses: docker/metadata-action@v5
         with:
           images: |
             quay.io/go-skynet/local-ai-backends
@@ -165,7 +160,7 @@ jobs:
       - name: Docker meta for PR
         id: meta_pull_request
         if: github.event_name == 'pull_request'
-        uses: docker/metadata-action@v6
+        uses: docker/metadata-action@v5
         with:
           images: |
             quay.io/go-skynet/ci-tests
@@ -188,21 +183,21 @@ jobs:
 
       - name: Login to DockerHub
         if: github.event_name != 'pull_request'
-        uses: docker/login-action@v4
+        uses: docker/login-action@v3
         with:
           username: ${{ secrets.dockerUsername }}
           password: ${{ secrets.dockerPassword }}
 
       - name: Login to Quay.io
         if: ${{ env.quay_username != '' }}
-        uses: docker/login-action@v4
+        uses: docker/login-action@v3
         with:
           registry: quay.io
           username: ${{ secrets.quayUsername }}
           password: ${{ secrets.quayPassword }}
 
       - name: Build and push
-        uses: docker/build-push-action@v7
+        uses: docker/build-push-action@v6
         if: github.event_name != 'pull_request'
         with:
           builder: ${{ steps.buildx.outputs.name }}
@@ -213,7 +208,6 @@ jobs:
             CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
             BASE_IMAGE=${{ inputs.base-image }}
             BACKEND=${{ inputs.backend }}
-            UBUNTU_VERSION=${{ inputs.ubuntu-version }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
           cache-from: type=gha
@@ -223,7 +217,7 @@ jobs:
           labels: ${{ steps.meta.outputs.labels }}
 
       - name: Build and push (PR)
-        uses: docker/build-push-action@v7
+        uses: docker/build-push-action@v6
         if: github.event_name == 'pull_request'
         with:
           builder: ${{ steps.buildx.outputs.name }}
@@ -234,7 +228,6 @@ jobs:
             CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
             BASE_IMAGE=${{ inputs.base-image }}
             BACKEND=${{ inputs.backend }}
-            UBUNTU_VERSION=${{ inputs.ubuntu-version }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
           cache-from: type=gha

.github/workflows/backend_build_darwin.yml

@@ -50,7 +50,7 @@ jobs:
         go-version: ['${{ inputs.go-version }}']
     steps:
       - name: Clone
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           submodules: true
 
@@ -74,7 +74,7 @@ jobs:
           BACKEND=${{ inputs.backend }} BUILD_TYPE=${{ inputs.build-type }} USE_PIP=${{ inputs.use-pip }} make build-darwin-${{ inputs.lang }}-backend
 
       - name: Upload ${{ inputs.backend }}.tar
-        uses: actions/upload-artifact@v7
+        uses: actions/upload-artifact@v5
         with:
           name: ${{ inputs.backend }}-tar
           path: backend-images/${{ inputs.backend }}.tar
@@ -85,7 +85,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Download ${{ inputs.backend }}.tar
-        uses: actions/download-artifact@v8
+        uses: actions/download-artifact@v6
         with:
           name: ${{ inputs.backend }}-tar
           path: .
@@ -105,7 +105,7 @@ jobs:
 
       - name: Docker meta
         id: meta
-        uses: docker/metadata-action@v6
+        uses: docker/metadata-action@v5
         with:
           images: |
             localai/localai-backends
@@ -119,7 +119,7 @@ jobs:
 
       - name: Docker meta
         id: quaymeta
-        uses: docker/metadata-action@v6
+        uses: docker/metadata-action@v5
         with:
           images: |
             quay.io/go-skynet/local-ai-backends

.github/workflows/backend_pr.yml

@@ -17,7 +17,7 @@ jobs:
       has-backends-darwin: ${{ steps.set-matrix.outputs.has-backends-darwin }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
 
       - name: Setup Bun
         uses: oven-sh/setup-bun@v2
@@ -52,7 +52,6 @@ jobs:
       dockerfile: ${{ matrix.dockerfile }}
       skip-drivers: ${{ matrix.skip-drivers }}
       context: ${{ matrix.context }}
-      ubuntu-version: ${{ matrix.ubuntu-version }}
     secrets:
       quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
       quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
@@ -70,7 +69,7 @@ jobs:
       tag-suffix: ${{ matrix.tag-suffix }}
       lang: ${{ matrix.lang || 'python' }}
       use-pip: ${{ matrix.backend == 'diffusers' }}
-      runs-on: "macos-latest"
+      runs-on: "macOS-14"
     secrets:
       quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
       quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}

.github/workflows/build-test.yaml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           fetch-depth: 0
       - name: Set up Go
@@ -25,7 +25,7 @@ jobs:
     runs-on: macos-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           fetch-depth: 0
       - name: Set up Go
@@ -37,7 +37,7 @@ jobs:
           make build-launcher-darwin
           ls -liah dist
       - name: Upload macOS launcher artifacts
-        uses: actions/upload-artifact@v7
+        uses: actions/upload-artifact@v5
         with:
           name: launcher-macos
           path: dist/
@@ -47,7 +47,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           fetch-depth: 0
       - name: Set up Go
@@ -60,7 +60,7 @@ jobs:
           sudo apt-get install golang gcc libgl1-mesa-dev xorg-dev libxkbcommon-dev
           make build-launcher-linux
       - name: Upload Linux launcher artifacts
-        uses: actions/upload-artifact@v7
+        uses: actions/upload-artifact@v5
         with:
           name: launcher-linux
           path: local-ai-launcher-linux.tar.xz

.github/workflows/bump_deps.yaml

@@ -1,11 +1,10 @@
-name: Bump Backend dependencies
+name: Bump dependencies
 on:
   schedule:
     - cron: 0 20 * * *
   workflow_dispatch:
 jobs:
-  bump-backends:
+  bump:
-    if: github.repository == 'mudler/LocalAI'
     strategy:
       fail-fast: false
       matrix:
@@ -18,6 +17,10 @@ jobs:
             variable: "WHISPER_CPP_VERSION"
             branch: "master"
             file: "backend/go/whisper/Makefile"
+          - repository: "PABannier/bark.cpp"
+            variable: "BARKCPP_VERSION"
+            branch: "main"
+            file: "Makefile"
           - repository: "leejet/stable-diffusion.cpp"
             variable: "STABLEDIFFUSION_GGML_VERSION"
             branch: "master"
@@ -26,17 +29,9 @@ jobs:
             variable: "PIPER_VERSION"
             branch: "master"
             file: "backend/go/piper/Makefile"
-          - repository: "antirez/voxtral.c"
-            variable: "VOXTRAL_VERSION"
-            branch: "main"
-            file: "backend/go/voxtral/Makefile"
-          - repository: "ace-step/acestep.cpp"
-            variable: "ACESTEP_CPP_VERSION"
-            branch: "master"
-            file: "backend/go/acestep-cpp/Makefile"
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v5
       - name: Bump dependencies 🔧
         id: bump
         run: |
@@ -54,7 +49,7 @@ jobs:
           rm -rfv ${{ matrix.variable }}_message.txt
           rm -rfv ${{ matrix.variable }}_commit.txt
       - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v8
+        uses: peter-evans/create-pull-request@v7
         with:
           token: ${{ secrets.UPDATE_BOT_TOKEN }}
           push-to-fork: ci-forks/LocalAI

.github/workflows/bump_docs.yaml

@@ -1,11 +1,10 @@
-name: Bump Documentation
+name: Bump dependencies
 on:
   schedule:
     - cron: 0 20 * * *
   workflow_dispatch:
 jobs:
-  bump-docs:
+  bump:
-    if: github.repository == 'mudler/LocalAI'
     strategy:
       fail-fast: false
       matrix:
@@ -13,12 +12,12 @@ jobs:
           - repository: "mudler/LocalAI"
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v5
       - name: Bump dependencies 🔧
         run: |
           bash .github/bump_docs.sh ${{ matrix.repository }}
       - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v8
+        uses: peter-evans/create-pull-request@v7
         with:
           token: ${{ secrets.UPDATE_BOT_TOKEN }}
           push-to-fork: ci-forks/LocalAI

.github/workflows/checksum_checker.yaml

@@ -5,7 +5,6 @@ on:
   workflow_dispatch:
 jobs:
   checksum_check:
-    if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
       - name: Force Install GIT latest
@@ -16,7 +15,7 @@ jobs:
           && sudo add-apt-repository -y ppa:git-core/ppa \
           && sudo apt-get update \
           && sudo apt-get install -y git
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v5
       - name: Install dependencies
         run: |
           sudo apt-get update
@@ -36,7 +35,7 @@ jobs:
           sudo chmod 777 /hf_cache
           bash .github/checksum_checker.sh gallery/index.yaml
       - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v8
+        uses: peter-evans/create-pull-request@v7
         with:
           token: ${{ secrets.UPDATE_BOT_TOKEN }}
           push-to-fork: ci-forks/LocalAI

.github/workflows/dependabot_auto.yml

@@ -9,18 +9,18 @@ permissions:
 
 jobs:
   dependabot:
-    if: github.repository == 'mudler/LocalAI' && github.actor == 'dependabot[bot]'
     runs-on: ubuntu-latest
+    if: ${{ github.actor == 'dependabot[bot]' }}
     steps:
       - name: Dependabot metadata
         id: metadata
-        uses: dependabot/fetch-metadata@v2.5.0
+        uses: dependabot/fetch-metadata@v2.4.0
         with:
           github-token: "${{ secrets.GITHUB_TOKEN }}"
           skip-commit-verification: true
 
       - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
 
       - name: Approve a PR if not already approved
         run: |

.github/workflows/deploy-explorer.yaml

@@ -12,11 +12,10 @@ concurrency:
 
 jobs:
   build-linux:
-    if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           submodules: true
       - uses: actions/setup-go@v5
@@ -34,7 +33,7 @@ jobs:
         run: |
           CGO_ENABLED=0 make build
       - name: rm
-        uses: appleboy/ssh-action@v1.2.5
+        uses: appleboy/ssh-action@v1.2.2
         with:
             host: ${{ secrets.EXPLORER_SSH_HOST }}
             username: ${{ secrets.EXPLORER_SSH_USERNAME }}
@@ -54,7 +53,7 @@ jobs:
             rm: true
             target: ./local-ai
       - name: restarting
-        uses: appleboy/ssh-action@v1.2.5
+        uses: appleboy/ssh-action@v1.2.2
         with:
             host: ${{ secrets.EXPLORER_SSH_HOST }}
             username: ${{ secrets.EXPLORER_SSH_USERNAME }}

.github/workflows/gallery-agent.yaml

@@ -2,7 +2,7 @@ name: Gallery Agent
 on:
 
   schedule:
-    - cron: '0 */3 * * *'  # Run every 4 hours
+    - cron: '0 */1 * * *'  # Run every 4 hours
   workflow_dispatch:
     inputs:
       search_term:
@@ -27,11 +27,10 @@ on:
         type: string
 jobs:
   gallery-agent:
-    if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -39,33 +38,27 @@ jobs:
         uses: actions/setup-go@v5
         with:
           go-version: '1.21'
-      - name: Proto Dependencies
+
+      - name: Build gallery agent
         run: |
-          # Install protoc
+          cd .github/gallery-agent
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
+          go mod download
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+          go build -o gallery-agent .
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - uses: mudler/localai-github-action@v1.1
-        with:
-          model: 'https://huggingface.co/unsloth/Qwen3.5-2B-GGUF'
 
       - name: Run gallery agent
         env:
-          #OPENAI_MODEL: ${{ secrets.OPENAI_MODEL }}
+          OPENAI_MODEL: ${{ secrets.OPENAI_MODEL }}
-          OPENAI_MODE: Qwen3.5-2B-GGUF
-          OPENAI_BASE_URL: "http://localhost:8080"
           OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
-          #OPENAI_BASE_URL: ${{ secrets.OPENAI_BASE_URL }}
+          OPENAI_BASE_URL: ${{ secrets.OPENAI_BASE_URL }}
           SEARCH_TERM: ${{ github.event.inputs.search_term || 'GGUF' }}
           LIMIT: ${{ github.event.inputs.limit || '15' }}
           QUANTIZATION: ${{ github.event.inputs.quantization || 'Q4_K_M' }}
           MAX_MODELS: ${{ github.event.inputs.max_models || '1' }}
         run: |
           export GALLERY_INDEX_PATH=$PWD/gallery/index.yaml
-          go run ./.github/gallery-agent
+          cd .github/gallery-agent
+          ./gallery-agent
+          rm -rf gallery-agent
 
       - name: Check for changes
         id: check_changes
@@ -83,28 +76,28 @@ jobs:
         id: read_summary
         if: steps.check_changes.outputs.changes == 'true'
         run: |
-          if [ -f "./gallery-agent-summary.json" ]; then
+          if [ -f ".github/gallery-agent/gallery-agent-summary.json" ]; then
             echo "summary_exists=true" >> $GITHUB_OUTPUT
             # Extract summary data using jq
-            echo "search_term=$(jq -r '.search_term' ./gallery-agent-summary.json)" >> $GITHUB_OUTPUT
+            echo "search_term=$(jq -r '.search_term' .github/gallery-agent/gallery-agent-summary.json)" >> $GITHUB_OUTPUT
-            echo "total_found=$(jq -r '.total_found' ./gallery-agent-summary.json)" >> $GITHUB_OUTPUT
+            echo "total_found=$(jq -r '.total_found' .github/gallery-agent/gallery-agent-summary.json)" >> $GITHUB_OUTPUT
-            echo "models_added=$(jq -r '.models_added' ./gallery-agent-summary.json)" >> $GITHUB_OUTPUT
+            echo "models_added=$(jq -r '.models_added' .github/gallery-agent/gallery-agent-summary.json)" >> $GITHUB_OUTPUT
-            echo "quantization=$(jq -r '.quantization' ./gallery-agent-summary.json)" >> $GITHUB_OUTPUT
+            echo "quantization=$(jq -r '.quantization' .github/gallery-agent/gallery-agent-summary.json)" >> $GITHUB_OUTPUT
-            echo "processing_time=$(jq -r '.processing_time' ./gallery-agent-summary.json)" >> $GITHUB_OUTPUT
+            echo "processing_time=$(jq -r '.processing_time' .github/gallery-agent/gallery-agent-summary.json)" >> $GITHUB_OUTPUT
             
             # Create a formatted list of added models with URLs
-            added_models=$(jq -r 'range(0; .added_model_ids | length) as $i | "- [\(.added_model_ids[$i])](\(.added_model_urls[$i]))"' ./gallery-agent-summary.json | tr '\n' '\n')
+            added_models=$(jq -r 'range(0; .added_model_ids | length) as $i | "- [\(.added_model_ids[$i])](\(.added_model_urls[$i]))"' .github/gallery-agent/gallery-agent-summary.json | tr '\n' '\n')
             echo "added_models<<EOF" >> $GITHUB_OUTPUT
             echo "$added_models" >> $GITHUB_OUTPUT
             echo "EOF" >> $GITHUB_OUTPUT
-            rm -f ./gallery-agent-summary.json
+            rm -f .github/gallery-agent/gallery-agent-summary.json
           else
             echo "summary_exists=false" >> $GITHUB_OUTPUT
           fi
 
       - name: Create Pull Request
         if: steps.check_changes.outputs.changes == 'true'
-        uses: peter-evans/create-pull-request@v8
+        uses: peter-evans/create-pull-request@v7
         with:
           token: ${{ secrets.UPDATE_BOT_TOKEN }}
           push-to-fork: ci-forks/LocalAI

.github/workflows/generate_grpc_cache.yaml

@@ -13,11 +13,10 @@ concurrency:
 
 jobs:
   generate_caches:
-    if: github.repository == 'mudler/LocalAI'
     strategy:
       matrix:
         include:
-          - grpc-base-image: ubuntu:24.04
+          - grpc-base-image: ubuntu:22.04
             runs-on: 'ubuntu-latest'
             platforms: 'linux/amd64,linux/arm64'
     runs-on: ${{matrix.runs-on}}
@@ -74,10 +73,10 @@ jobs:
         uses: docker/setup-buildx-action@master
 
       - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
 
       - name: Cache GRPC
-        uses: docker/build-push-action@v7
+        uses: docker/build-push-action@v6
         with:
           builder: ${{ steps.buildx.outputs.name }}
           # The build-args MUST be an EXACT match between the image cache and other workflow steps that want to use that cache.

.github/workflows/generate_intel_image.yaml

@@ -12,12 +12,11 @@ concurrency:
 
 jobs:
   generate_caches:
-    if: github.repository == 'mudler/LocalAI'
     strategy:
       matrix:
         include:
-          - base-image: intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04
+          - base-image: intel/oneapi-basekit:2025.2.0-0-devel-ubuntu22.04
-            runs-on: 'arc-runner-set'
+            runs-on: 'ubuntu-latest'
             platforms: 'linux/amd64'
     runs-on: ${{matrix.runs-on}}
     steps:
@@ -27,14 +26,14 @@ jobs:
           platforms: all
       - name: Login to DockerHub
         if: github.event_name != 'pull_request'
-        uses: docker/login-action@v4
+        uses: docker/login-action@v3
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_PASSWORD }}
 
       - name: Login to quay
         if: github.event_name != 'pull_request'
-        uses: docker/login-action@v4
+        uses: docker/login-action@v3
         with:
           registry: quay.io
           username: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
@@ -44,17 +43,17 @@ jobs:
         uses: docker/setup-buildx-action@master
 
       - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
 
       - name: Cache Intel images
-        uses: docker/build-push-action@v7
+        uses: docker/build-push-action@v6
         with:
           builder: ${{ steps.buildx.outputs.name }}
           build-args: |
             BASE_IMAGE=${{ matrix.base-image }}
           context: .
           file: ./Dockerfile
-          tags: quay.io/go-skynet/intel-oneapi-base:24.04
+          tags: quay.io/go-skynet/intel-oneapi-base:latest
           push: true
           target: intel
           platforms: ${{ matrix.platforms }}

.github/workflows/image-pr.yml

@@ -1,95 +1,68 @@
 ---
-  name: 'build container images tests'
+name: 'build container images tests'
-  
+
-  on:
+on:
-    pull_request:
+  pull_request:
-  
+
-  concurrency:
+concurrency:
-    group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
-    cancel-in-progress: true
+  cancel-in-progress: true
-  
+
-  jobs:
+jobs:
-    image-build:
+  image-build:
-      uses: ./.github/workflows/image_build.yml
+    uses: ./.github/workflows/image_build.yml
-      with:
+    with:
-        tag-latest: ${{ matrix.tag-latest }}
+      tag-latest: ${{ matrix.tag-latest }}
-        tag-suffix: ${{ matrix.tag-suffix }}
+      tag-suffix: ${{ matrix.tag-suffix }}
-        build-type: ${{ matrix.build-type }}
+      build-type: ${{ matrix.build-type }}
-        cuda-major-version: ${{ matrix.cuda-major-version }}
+      cuda-major-version: ${{ matrix.cuda-major-version }}
-        cuda-minor-version: ${{ matrix.cuda-minor-version }}
+      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-        platforms: ${{ matrix.platforms }}
+      platforms: ${{ matrix.platforms }}
-        runs-on: ${{ matrix.runs-on }}
+      runs-on: ${{ matrix.runs-on }}
-        base-image: ${{ matrix.base-image }}
+      base-image: ${{ matrix.base-image }}
-        grpc-base-image: ${{ matrix.grpc-base-image }}
+      grpc-base-image: ${{ matrix.grpc-base-image }}
-        makeflags: ${{ matrix.makeflags }}
+      makeflags: ${{ matrix.makeflags }}
-        ubuntu-version: ${{ matrix.ubuntu-version }}
+    secrets:
-      secrets:
+      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
-      strategy:
+      # Pushing with all jobs in parallel
-        # Pushing with all jobs in parallel
+      # eats the bandwidth of all the nodes
-        # eats the bandwidth of all the nodes
+      max-parallel: ${{ github.event_name != 'pull_request' && 4 || 8 }}
-        max-parallel: ${{ github.event_name != 'pull_request' && 4 || 8 }}
+      fail-fast: false
-        fail-fast: false
+      matrix:
-        matrix:
+        include:
-          include:
+          - build-type: 'cublas'
-            - build-type: 'cublas'
+            cuda-major-version: "12"
-              cuda-major-version: "12"
+            cuda-minor-version: "0"
-              cuda-minor-version: "8"
+            platforms: 'linux/amd64'
-              platforms: 'linux/amd64'
+            tag-latest: 'false'
-              tag-latest: 'false'
+            tag-suffix: '-gpu-nvidia-cuda-12'
-              tag-suffix: '-gpu-nvidia-cuda-12'
+            runs-on: 'ubuntu-latest'
-              runs-on: 'ubuntu-latest'
+            base-image: "ubuntu:22.04"
-              base-image: "ubuntu:24.04"
+            makeflags: "--jobs=3 --output-sync=target"
-              makeflags: "--jobs=3 --output-sync=target"
+          - build-type: 'hipblas'
-              ubuntu-version: '2404'
+            platforms: 'linux/amd64'
-            - build-type: 'cublas'
+            tag-latest: 'false'
-              cuda-major-version: "13"
+            tag-suffix: '-hipblas'
-              cuda-minor-version: "0"
+            base-image: "rocm/dev-ubuntu-22.04:6.4.3"
-              platforms: 'linux/amd64'
+            grpc-base-image: "ubuntu:22.04"
-              tag-latest: 'false'
+            runs-on: 'ubuntu-latest'
-              tag-suffix: '-gpu-nvidia-cuda-13'
+            makeflags: "--jobs=3 --output-sync=target"
-              runs-on: 'ubuntu-latest'
+          - build-type: 'sycl'
-              base-image: "ubuntu:22.04"
+            platforms: 'linux/amd64'
-              makeflags: "--jobs=3 --output-sync=target"
+            tag-latest: 'false'
-              ubuntu-version: '2404'
+            base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
-            - build-type: 'hipblas'
+            grpc-base-image: "ubuntu:22.04"
-              platforms: 'linux/amd64'
+            tag-suffix: 'sycl'
-              tag-latest: 'false'
+            runs-on: 'ubuntu-latest'
-              tag-suffix: '-hipblas'
+            makeflags: "--jobs=3 --output-sync=target"
-              base-image: "rocm/dev-ubuntu-24.04:6.4.4"
+          - build-type: 'vulkan'
-              grpc-base-image: "ubuntu:24.04"
+            platforms: 'linux/amd64'
-              runs-on: 'ubuntu-latest'
+            tag-latest: 'false'
-              makeflags: "--jobs=3 --output-sync=target"
+            tag-suffix: '-vulkan-core'
-              ubuntu-version: '2404'
+            runs-on: 'ubuntu-latest'
-            - build-type: 'sycl'
+            base-image: "ubuntu:22.04"
-              platforms: 'linux/amd64'
+            makeflags: "--jobs=4 --output-sync=target"
-              tag-latest: 'false'
-              base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-              grpc-base-image: "ubuntu:24.04"
-              tag-suffix: 'sycl'
-              runs-on: 'ubuntu-latest'
-              makeflags: "--jobs=3 --output-sync=target"
-              ubuntu-version: '2404'
-            - build-type: 'vulkan'
-              platforms: 'linux/amd64,linux/arm64'
-              tag-latest: 'false'
-              tag-suffix: '-vulkan-core'
-              runs-on: 'ubuntu-latest'
-              base-image: "ubuntu:24.04"
-              makeflags: "--jobs=4 --output-sync=target"
-              ubuntu-version: '2404'
-            - build-type: 'cublas'
-              cuda-major-version: "13"
-              cuda-minor-version: "0"
-              platforms: 'linux/arm64'
-              tag-latest: 'false'
-              tag-suffix: '-nvidia-l4t-arm64-cuda-13'
-              base-image: "ubuntu:24.04"
-              runs-on: 'ubuntu-24.04-arm'
-              makeflags: "--jobs=4 --output-sync=target"
-              skip-drivers: 'false'
-              ubuntu-version: '2404'
-  

.github/workflows/image.yml

@@ -1,181 +1,154 @@
 ---
-  name: 'build container images'
+name: 'build container images'
-  
+
-  on:
+on:
-    push:
+  push:
-      branches:
+    branches:
-        - master
+      - master
-      tags:
+    tags:
-        - '*'
+      - '*'
-  
+
-  concurrency:
+concurrency:
-    group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
+  group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
-    cancel-in-progress: true
+  cancel-in-progress: true
-  
+
-  jobs:
+jobs:
-    hipblas-jobs:
+  hipblas-jobs:
-      if: github.repository == 'mudler/LocalAI'
+    uses: ./.github/workflows/image_build.yml
-      uses: ./.github/workflows/image_build.yml
+    with:
-      with:
+      tag-latest: ${{ matrix.tag-latest }}
-        tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
-        tag-suffix: ${{ matrix.tag-suffix }}
+      build-type: ${{ matrix.build-type }}
-        build-type: ${{ matrix.build-type }}
+      cuda-major-version: ${{ matrix.cuda-major-version }}
-        cuda-major-version: ${{ matrix.cuda-major-version }}
+      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-        cuda-minor-version: ${{ matrix.cuda-minor-version }}
+      platforms: ${{ matrix.platforms }}
-        platforms: ${{ matrix.platforms }}
+      runs-on: ${{ matrix.runs-on }}
-        runs-on: ${{ matrix.runs-on }}
+      base-image: ${{ matrix.base-image }}
-        base-image: ${{ matrix.base-image }}
+      grpc-base-image: ${{ matrix.grpc-base-image }}
-        grpc-base-image: ${{ matrix.grpc-base-image }}
+      aio: ${{ matrix.aio }}
-        makeflags: ${{ matrix.makeflags }}
+      makeflags: ${{ matrix.makeflags }}
-        ubuntu-version: ${{ matrix.ubuntu-version }}
+    secrets:
-        ubuntu-codename: ${{ matrix.ubuntu-codename }}
+      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-      secrets:
+      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+    strategy:
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+      matrix:
-      strategy:
+        include:
-        matrix:
+          - build-type: 'hipblas'
-          include:
+            platforms: 'linux/amd64'
-            - build-type: 'hipblas'
+            tag-latest: 'auto'
-              platforms: 'linux/amd64'
+            tag-suffix: '-gpu-hipblas'
-              tag-latest: 'auto'
+            base-image: "rocm/dev-ubuntu-22.04:6.4.3"
-              tag-suffix: '-gpu-hipblas'
+            grpc-base-image: "ubuntu:22.04"
-              base-image: "rocm/dev-ubuntu-24.04:6.4.4"
+            runs-on: 'ubuntu-latest'
-              grpc-base-image: "ubuntu:24.04"
+            makeflags: "--jobs=3 --output-sync=target"
-              runs-on: 'ubuntu-latest'
+            aio: "-aio-gpu-hipblas"
-              makeflags: "--jobs=3 --output-sync=target"
+
-              ubuntu-version: '2404'
+  core-image-build:
-              ubuntu-codename: 'noble'
+    uses: ./.github/workflows/image_build.yml
-  
+    with:
-    core-image-build:
+      tag-latest: ${{ matrix.tag-latest }}
-      if: github.repository == 'mudler/LocalAI'
+      tag-suffix: ${{ matrix.tag-suffix }}
-      uses: ./.github/workflows/image_build.yml
+      build-type: ${{ matrix.build-type }}
-      with:
+      cuda-major-version: ${{ matrix.cuda-major-version }}
-        tag-latest: ${{ matrix.tag-latest }}
+      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-        tag-suffix: ${{ matrix.tag-suffix }}
+      platforms: ${{ matrix.platforms }}
-        build-type: ${{ matrix.build-type }}
+      runs-on: ${{ matrix.runs-on }}
-        cuda-major-version: ${{ matrix.cuda-major-version }}
+      aio: ${{ matrix.aio }}
-        cuda-minor-version: ${{ matrix.cuda-minor-version }}
+      base-image: ${{ matrix.base-image }}
-        platforms: ${{ matrix.platforms }}
+      grpc-base-image: ${{ matrix.grpc-base-image }}
-        runs-on: ${{ matrix.runs-on }}
+      makeflags: ${{ matrix.makeflags }}
-        base-image: ${{ matrix.base-image }}
+      skip-drivers: ${{ matrix.skip-drivers }}
-        grpc-base-image: ${{ matrix.grpc-base-image }}
+    secrets:
-        makeflags: ${{ matrix.makeflags }}
+      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        skip-drivers: ${{ matrix.skip-drivers }}
+      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        ubuntu-version: ${{ matrix.ubuntu-version }}
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        ubuntu-codename: ${{ matrix.ubuntu-codename }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-      secrets:
+    strategy:
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+      #max-parallel: ${{ github.event_name != 'pull_request' && 2 || 4 }}
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+      matrix:
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+        include:
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+          - build-type: ''
-      strategy:
+            platforms: 'linux/amd64,linux/arm64'
-        #max-parallel: ${{ github.event_name != 'pull_request' && 2 || 4 }}
+            tag-latest: 'auto'
-        matrix:
+            tag-suffix: ''
-          include:
+            base-image: "ubuntu:22.04"
-            - build-type: ''
+            runs-on: 'ubuntu-latest'
-              platforms: 'linux/amd64,linux/arm64'
+            aio: "-aio-cpu"
-              tag-latest: 'auto'
+            makeflags: "--jobs=4 --output-sync=target"
-              tag-suffix: ''
+            skip-drivers: 'false'
-              base-image: "ubuntu:24.04"
+          - build-type: 'cublas'
-              runs-on: 'ubuntu-latest'
+            cuda-major-version: "11"
-              makeflags: "--jobs=4 --output-sync=target"
+            cuda-minor-version: "7"
-              skip-drivers: 'false'
+            platforms: 'linux/amd64'
-              ubuntu-version: '2404'
+            tag-latest: 'auto'
-              ubuntu-codename: 'noble'
+            tag-suffix: '-gpu-nvidia-cuda-11'
-            - build-type: 'cublas'
+            runs-on: 'ubuntu-latest'
-              cuda-major-version: "12"
+            base-image: "ubuntu:22.04"
-              cuda-minor-version: "8"
+            makeflags: "--jobs=4 --output-sync=target"
-              platforms: 'linux/amd64'
+            skip-drivers: 'false'
-              tag-latest: 'auto'
+            aio: "-aio-gpu-nvidia-cuda-11"
-              tag-suffix: '-gpu-nvidia-cuda-12'
+          - build-type: 'cublas'
-              runs-on: 'ubuntu-latest'
+            cuda-major-version: "12"
-              base-image: "ubuntu:24.04"
+            cuda-minor-version: "0"
-              skip-drivers: 'false'
+            platforms: 'linux/amd64'
-              makeflags: "--jobs=4 --output-sync=target"
+            tag-latest: 'auto'
-              ubuntu-version: '2404'
+            tag-suffix: '-gpu-nvidia-cuda-12'
-              ubuntu-codename: 'noble'
+            runs-on: 'ubuntu-latest'
-            - build-type: 'cublas'
+            base-image: "ubuntu:22.04"
-              cuda-major-version: "13"
+            skip-drivers: 'false'
-              cuda-minor-version: "0"
+            makeflags: "--jobs=4 --output-sync=target"
-              platforms: 'linux/amd64'
+            aio: "-aio-gpu-nvidia-cuda-12"
-              tag-latest: 'auto'
+          - build-type: 'vulkan'
-              tag-suffix: '-gpu-nvidia-cuda-13'
+            platforms: 'linux/amd64'
-              runs-on: 'ubuntu-latest'
+            tag-latest: 'auto'
-              base-image: "ubuntu:22.04"
+            tag-suffix: '-gpu-vulkan'
-              skip-drivers: 'false'
+            runs-on: 'ubuntu-latest'
-              makeflags: "--jobs=4 --output-sync=target"
+            base-image: "ubuntu:22.04"
-              ubuntu-version: '2404'
+            skip-drivers: 'false'
-              ubuntu-codename: 'noble'
+            makeflags: "--jobs=4 --output-sync=target"
-            - build-type: 'vulkan'
+            aio: "-aio-gpu-vulkan"
-              platforms: 'linux/amd64,linux/arm64'
+          - build-type: 'intel'
-              tag-latest: 'auto'
+            platforms: 'linux/amd64'
-              tag-suffix: '-gpu-vulkan'
+            tag-latest: 'auto'
-              runs-on: 'ubuntu-latest'
+            base-image: "quay.io/go-skynet/intel-oneapi-base:latest"
-              base-image: "ubuntu:24.04"
+            grpc-base-image: "ubuntu:22.04"
-              skip-drivers: 'false'
+            tag-suffix: '-gpu-intel'
-              makeflags: "--jobs=4 --output-sync=target"
+            runs-on: 'ubuntu-latest'
-              ubuntu-version: '2404'
+            makeflags: "--jobs=3 --output-sync=target"
-              ubuntu-codename: 'noble'
+            aio: "-aio-gpu-intel"
-            - build-type: 'intel'
+
-              platforms: 'linux/amd64'
+  gh-runner:
-              tag-latest: 'auto'
+    uses: ./.github/workflows/image_build.yml
-              base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    with:
-              grpc-base-image: "ubuntu:24.04"
+      tag-latest: ${{ matrix.tag-latest }}
-              tag-suffix: '-gpu-intel'
+      tag-suffix: ${{ matrix.tag-suffix }}
-              runs-on: 'ubuntu-latest'
+      build-type: ${{ matrix.build-type }}
-              makeflags: "--jobs=3 --output-sync=target"
+      cuda-major-version: ${{ matrix.cuda-major-version }}
-              ubuntu-version: '2404'
+      cuda-minor-version: ${{ matrix.cuda-minor-version }}
-              ubuntu-codename: 'noble'
+      platforms: ${{ matrix.platforms }}
-  
+      runs-on: ${{ matrix.runs-on }}
-    gh-runner:
+      aio: ${{ matrix.aio }}
-      if: github.repository == 'mudler/LocalAI'
+      base-image: ${{ matrix.base-image }}
-      uses: ./.github/workflows/image_build.yml
+      grpc-base-image: ${{ matrix.grpc-base-image }}
-      with:
+      makeflags: ${{ matrix.makeflags }}
-        tag-latest: ${{ matrix.tag-latest }}
+      skip-drivers: ${{ matrix.skip-drivers }}
-        tag-suffix: ${{ matrix.tag-suffix }}
+    secrets:
-        build-type: ${{ matrix.build-type }}
+      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
-        cuda-major-version: ${{ matrix.cuda-major-version }}
+      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
-        cuda-minor-version: ${{ matrix.cuda-minor-version }}
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
-        platforms: ${{ matrix.platforms }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
-        runs-on: ${{ matrix.runs-on }}
+    strategy:
-        base-image: ${{ matrix.base-image }}
+      matrix:
-        grpc-base-image: ${{ matrix.grpc-base-image }}
+        include:
-        makeflags: ${{ matrix.makeflags }}
+          - build-type: 'cublas'
-        skip-drivers: ${{ matrix.skip-drivers }}
+            cuda-major-version: "12"
-        ubuntu-version: ${{ matrix.ubuntu-version }}
+            cuda-minor-version: "0"
-        ubuntu-codename: ${{ matrix.ubuntu-codename }}
+            platforms: 'linux/arm64'
-      secrets:
+            tag-latest: 'auto'
-        dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+            tag-suffix: '-nvidia-l4t-arm64'
-        dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+            base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
-        quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+            runs-on: 'ubuntu-24.04-arm'
-        quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+            makeflags: "--jobs=4 --output-sync=target"
-      strategy:
+            skip-drivers: 'true'
-        matrix:
-          include:
-            - build-type: 'cublas'
-              cuda-major-version: "12"
-              cuda-minor-version: "0"
-              platforms: 'linux/arm64'
-              tag-latest: 'auto'
-              tag-suffix: '-nvidia-l4t-arm64'
-              base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
-              runs-on: 'ubuntu-24.04-arm'
-              makeflags: "--jobs=4 --output-sync=target"
-              skip-drivers: 'true'
-              ubuntu-version: "2204"
-              ubuntu-codename: 'jammy'
-            - build-type: 'cublas'
-              cuda-major-version: "13"
-              cuda-minor-version: "0"
-              platforms: 'linux/arm64'
-              tag-latest: 'auto'
-              tag-suffix: '-nvidia-l4t-arm64-cuda-13'
-              base-image: "ubuntu:24.04"
-              runs-on: 'ubuntu-24.04-arm'
-              makeflags: "--jobs=4 --output-sync=target"
-              skip-drivers: 'false'
-              ubuntu-version: '2404'
-              ubuntu-codename: 'noble'
-  

.github/workflows/image_build.yml

@@ -23,7 +23,7 @@ on:
         type: string
       cuda-minor-version:
         description: 'CUDA minor version'
-        default: "9"
+        default: "4"
         type: string
       platforms:
         description: 'Platforms'
@@ -51,15 +51,10 @@ on:
         required: false
         default: '--jobs=4 --output-sync=target'
         type: string
-      ubuntu-version:
+      aio:
-        description: 'Ubuntu version'
+        description: 'AIO Image Name'
         required: false
-        default: '2204'
+        default: ''
-        type: string
-      ubuntu-codename:
-        description: 'Ubuntu codename'
-        required: false
-        default: 'noble'
         type: string
     secrets:
       dockerUsername:
@@ -99,7 +94,7 @@ jobs:
           && sudo apt-get update \
           && sudo apt-get install -y git
       - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
 
       - name: Release space from worker
         if: inputs.runs-on == 'ubuntu-latest'
@@ -146,7 +141,7 @@ jobs:
       - name: Docker meta
         id: meta
         if: github.event_name != 'pull_request'
-        uses: docker/metadata-action@v6
+        uses: docker/metadata-action@v5
         with:
           images: |
             quay.io/go-skynet/local-ai
@@ -161,7 +156,7 @@ jobs:
       - name: Docker meta for PR
         id: meta_pull_request
         if: github.event_name == 'pull_request'
-        uses: docker/metadata-action@v6
+        uses: docker/metadata-action@v5
         with:
           images: |
             quay.io/go-skynet/ci-tests
@@ -172,6 +167,34 @@ jobs:
           flavor: |
             latest=${{ inputs.tag-latest }}
             suffix=${{ inputs.tag-suffix }}
+      - name: Docker meta AIO (quay.io)
+        if: inputs.aio != ''
+        id: meta_aio
+        uses: docker/metadata-action@v5
+        with:
+          images: |
+            quay.io/go-skynet/local-ai
+          tags: |
+            type=ref,event=branch
+            type=semver,pattern={{raw}}
+          flavor: |
+            latest=${{ inputs.tag-latest }}
+            suffix=${{ inputs.aio }},onlatest=true
+
+      - name: Docker meta AIO (dockerhub)
+        if: inputs.aio != ''
+        id: meta_aio_dockerhub
+        uses: docker/metadata-action@v5
+        with:
+          images: |
+            localai/localai
+          tags: |
+            type=ref,event=branch
+            type=semver,pattern={{raw}}
+          flavor: |
+            latest=${{ inputs.tag-latest }}
+            suffix=${{ inputs.aio }},onlatest=true
+
       - name: Set up QEMU
         uses: docker/setup-qemu-action@master
         with:
@@ -183,21 +206,21 @@ jobs:
 
       - name: Login to DockerHub
         if: github.event_name != 'pull_request'
-        uses: docker/login-action@v4
+        uses: docker/login-action@v3
         with:
           username: ${{ secrets.dockerUsername }}
           password: ${{ secrets.dockerPassword }}
 
       - name: Login to DockerHub
         if: github.event_name != 'pull_request'
-        uses: docker/login-action@v4
+        uses: docker/login-action@v3
         with:
           registry: quay.io
           username: ${{ secrets.quayUsername }}
           password: ${{ secrets.quayPassword }}
 
       - name: Build and push
-        uses: docker/build-push-action@v7
+        uses: docker/build-push-action@v6
         if: github.event_name != 'pull_request'
         with:
           builder: ${{ steps.buildx.outputs.name }}
@@ -215,8 +238,6 @@ jobs:
             GRPC_VERSION=v1.65.0
             MAKEFLAGS=${{ inputs.makeflags }}
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
-            UBUNTU_VERSION=${{ inputs.ubuntu-version }}
-            UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
           context: .
           file: ./Dockerfile
           cache-from: type=gha
@@ -226,7 +247,7 @@ jobs:
           labels: ${{ steps.meta.outputs.labels }}
 ### Start testing image
       - name: Build and push
-        uses: docker/build-push-action@v7
+        uses: docker/build-push-action@v6
         if: github.event_name == 'pull_request'
         with:
           builder: ${{ steps.buildx.outputs.name }}
@@ -244,8 +265,6 @@ jobs:
             GRPC_VERSION=v1.65.0
             MAKEFLAGS=${{ inputs.makeflags }}
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
-            UBUNTU_VERSION=${{ inputs.ubuntu-version }}
-            UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
           context: .
           file: ./Dockerfile
           cache-from: type=gha
@@ -254,6 +273,41 @@ jobs:
           tags: ${{ steps.meta_pull_request.outputs.tags }}
           labels: ${{ steps.meta_pull_request.outputs.labels }}
 ## End testing image
+      - name: Build and push AIO image
+        if: inputs.aio != ''
+        uses: docker/build-push-action@v6
+        with:
+          builder: ${{ steps.buildx.outputs.name }}
+          build-args: |
+            BASE_IMAGE=quay.io/go-skynet/local-ai:${{ steps.meta.outputs.version }}
+            MAKEFLAGS=${{ inputs.makeflags }}
+          context: .
+          file: ./Dockerfile.aio
+          platforms: ${{ inputs.platforms }}
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.meta_aio.outputs.tags }}
+          labels: ${{ steps.meta_aio.outputs.labels }}
+
+      - name: Build and push AIO image (dockerhub)
+        if: inputs.aio != ''
+        uses: docker/build-push-action@v6
+        with:
+          builder: ${{ steps.buildx.outputs.name }}
+          build-args: |
+            BASE_IMAGE=localai/localai:${{ steps.meta.outputs.version }}
+            MAKEFLAGS=${{ inputs.makeflags }}
+          context: .
+          file: ./Dockerfile.aio
+          platforms: ${{ inputs.platforms }}
+          push: ${{ github.event_name != 'pull_request' }}
+          tags: ${{ steps.meta_aio_dockerhub.outputs.tags }}
+          labels: ${{ steps.meta_aio_dockerhub.outputs.labels }}
+
       - name: job summary
         run: |
           echo "Built image: ${{ steps.meta.outputs.labels }}" >> $GITHUB_STEP_SUMMARY
+
+      - name: job summary(AIO)
+        if: inputs.aio != ''
+        run: |
+          echo "Built image: ${{ steps.meta_aio.outputs.labels }}" >> $GITHUB_STEP_SUMMARY

.github/workflows/localaibot_automerge.yml

@@ -10,11 +10,11 @@ permissions:
   actions: write # to dispatch publish workflow
 jobs:
   dependabot:
-    if: github.repository == 'mudler/LocalAI' && github.actor == 'localai-bot' && contains(github.event.pull_request.title, 'chore:')
     runs-on: ubuntu-latest
+    if: ${{ github.actor == 'localai-bot' && !contains(github.event.pull_request.title, 'chore(model gallery):') }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
 
       - name: Approve a PR if not already approved
         run: |

.github/workflows/notify-models.yaml

@@ -10,12 +10,12 @@ permissions:
 
 jobs:
   notify-discord:
-    if: github.repository == 'mudler/LocalAI' && (github.event.pull_request.merged == true) && (contains(github.event.pull_request.labels.*.name, 'area/ai-model'))
+    if: ${{ (github.event.pull_request.merged == true) && (contains(github.event.pull_request.labels.*.name, 'area/ai-model')) }}
     env:
         MODEL_NAME: gemma-3-12b-it-qat
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v5
       with:
         fetch-depth: 0 # needed to checkout all branches for this Action to work
         ref: ${{ github.event.pull_request.head.sha }} # Checkout the PR head to get the actual changes
@@ -90,12 +90,12 @@ jobs:
         connect-timeout-seconds: 180
         limit-access-to-actor: true
   notify-twitter:
-    if: github.repository == 'mudler/LocalAI' && (github.event.pull_request.merged == true) && (contains(github.event.pull_request.labels.*.name, 'area/ai-model'))
+    if: ${{ (github.event.pull_request.merged == true) && (contains(github.event.pull_request.labels.*.name, 'area/ai-model')) }}
     env:
         MODEL_NAME: gemma-3-12b-it-qat
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v5
       with:
         fetch-depth: 0 # needed to checkout all branches for this Action to work
         ref: ${{ github.event.pull_request.head.sha }} # Checkout the PR head to get the actual changes

.github/workflows/notify-releases.yaml

@@ -6,7 +6,6 @@ on:
 
 jobs:
   notify-discord:
-    if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     env:
         RELEASE_BODY: ${{ github.event.release.body }}

.github/workflows/release.yaml

@@ -10,7 +10,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           fetch-depth: 0
       - name: Set up Go
@@ -18,7 +18,7 @@ jobs:
         with:
           go-version: 1.23
       - name: Run GoReleaser
-        uses: goreleaser/goreleaser-action@v7
+        uses: goreleaser/goreleaser-action@v6
         with:
           version: v2.11.0
           args: release --clean
@@ -28,7 +28,7 @@ jobs:
     runs-on: macos-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           fetch-depth: 0
       - name: Set up Go
@@ -46,7 +46,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           fetch-depth: 0
       - name: Set up Go

.github/workflows/secscan.yaml

@@ -14,7 +14,7 @@ jobs:
       GO111MODULE: on
     steps:
       - name: Checkout Source
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         if: ${{ github.actor != 'dependabot[bot]' }}
       - name: Run Gosec Security Scanner
         if: ${{ github.actor != 'dependabot[bot]' }}

.github/workflows/stalebot.yml

@@ -8,10 +8,9 @@ on:
 
 jobs:
   stale:
-    if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/stale@b5d41d4e1d5dceea10e7104786b73624c18a190f # v9
+      - uses: actions/stale@5f858e3efba33a5ca4407a664cc011ad407f2008 # v9
         with:
           stale-issue-message: 'This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.'
           stale-pr-message: 'This PR is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.'

.github/workflows/test-extra.yml

@@ -19,7 +19,7 @@ jobs:
   #   runs-on: ubuntu-latest
   #   steps:
   #     - name: Clone
-  #       uses: actions/checkout@v6
+  #       uses: actions/checkout@v5
   #       with:
   #         submodules: true
   #     - name: Dependencies
@@ -40,7 +40,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           submodules: true
       - name: Dependencies
@@ -61,7 +61,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           submodules: true
       - name: Dependencies
@@ -83,7 +83,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           submodules: true
       - name: Dependencies
@@ -104,7 +104,7 @@ jobs:
   #  runs-on: ubuntu-latest
   #  steps:
   #    - name: Clone
-  #      uses: actions/checkout@v6
+  #      uses: actions/checkout@v5
   #      with:
   #        submodules: true
   #    - name: Dependencies
@@ -124,7 +124,7 @@ jobs:
   #   runs-on: ubuntu-latest
   #   steps:
   #     - name: Clone
-  #       uses: actions/checkout@v6
+  #       uses: actions/checkout@v5
   #       with:
   #         submodules: true
   #     - name: Dependencies
@@ -186,7 +186,7 @@ jobs:
   #           sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
   #           df -h
   #     - name: Clone
-  #       uses: actions/checkout@v6
+  #       uses: actions/checkout@v5
   #       with:
   #         submodules: true
   #     - name: Dependencies
@@ -211,7 +211,7 @@ jobs:
   #   runs-on: ubuntu-latest
   #   steps:
   #     - name: Clone
-  #       uses: actions/checkout@v6
+  #       uses: actions/checkout@v5
   #       with:
   #         submodules: true
   #     - name: Dependencies
@@ -232,13 +232,13 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Clone
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           submodules: true
       - name: Dependencies
         run: |
           sudo apt-get update
-          sudo apt-get install -y build-essential ffmpeg
+          sudo apt-get install build-essential ffmpeg
           sudo apt-get install -y ca-certificates cmake curl patch espeak espeak-ng python3-pip
           # Install UV
           curl -LsSf https://astral.sh/uv/install.sh | sh
@@ -247,200 +247,3 @@ jobs:
         run: |
           make --jobs=5 --output-sync=target -C backend/python/coqui
           make --jobs=5 --output-sync=target -C backend/python/coqui test
-  tests-moonshine:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential ffmpeg
-          sudo apt-get install -y ca-certificates cmake curl patch python3-pip
-          # Install UV
-          curl -LsSf https://astral.sh/uv/install.sh | sh
-          pip install --user --no-cache-dir grpcio-tools==1.64.1
-      - name: Test moonshine
-        run: |
-          make --jobs=5 --output-sync=target -C backend/python/moonshine
-          make --jobs=5 --output-sync=target -C backend/python/moonshine test
-  tests-pocket-tts:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential ffmpeg
-          sudo apt-get install -y ca-certificates cmake curl patch python3-pip
-          # Install UV
-          curl -LsSf https://astral.sh/uv/install.sh | sh
-          pip install --user --no-cache-dir grpcio-tools==1.64.1
-      - name: Test pocket-tts
-        run: |
-          make --jobs=5 --output-sync=target -C backend/python/pocket-tts
-          make --jobs=5 --output-sync=target -C backend/python/pocket-tts test
-  tests-qwen-tts:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential ffmpeg
-          sudo apt-get install -y ca-certificates cmake curl patch python3-pip
-          # Install UV
-          curl -LsSf https://astral.sh/uv/install.sh | sh
-          pip install --user --no-cache-dir grpcio-tools==1.64.1
-      - name: Test qwen-tts
-        run: |
-          make --jobs=5 --output-sync=target -C backend/python/qwen-tts
-          make --jobs=5 --output-sync=target -C backend/python/qwen-tts test
-  # TODO: s2-pro model is too large to load on CPU-only CI runners — re-enable
-  # when we have GPU runners or a smaller test model.
-  # tests-fish-speech:
-  #   runs-on: ubuntu-latest
-  #   timeout-minutes: 45
-  #   steps:
-  #     - name: Clone
-  #       uses: actions/checkout@v6
-  #       with:
-  #         submodules: true
-  #     - name: Dependencies
-  #       run: |
-  #         sudo apt-get update
-  #         sudo apt-get install -y build-essential ffmpeg portaudio19-dev
-  #         sudo apt-get install -y ca-certificates cmake curl patch python3-pip
-  #         # Install UV
-  #         curl -LsSf https://astral.sh/uv/install.sh | sh
-  #         pip install --user --no-cache-dir grpcio-tools==1.64.1
-  #     - name: Test fish-speech
-  #       run: |
-  #         make --jobs=5 --output-sync=target -C backend/python/fish-speech
-  #         make --jobs=5 --output-sync=target -C backend/python/fish-speech test
-  tests-qwen-asr:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential ffmpeg sox
-          sudo apt-get install -y ca-certificates cmake curl patch python3-pip
-          # Install UV
-          curl -LsSf https://astral.sh/uv/install.sh | sh
-          pip install --user --no-cache-dir grpcio-tools==1.64.1
-      - name: Test qwen-asr
-        run: |
-          make --jobs=5 --output-sync=target -C backend/python/qwen-asr
-          make --jobs=5 --output-sync=target -C backend/python/qwen-asr test
-  tests-nemo:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential ffmpeg sox
-          sudo apt-get install -y ca-certificates cmake curl patch python3-pip
-          # Install UV
-          curl -LsSf https://astral.sh/uv/install.sh | sh
-          pip install --user --no-cache-dir grpcio-tools==1.64.1
-      - name: Test nemo
-        run: |
-          make --jobs=5 --output-sync=target -C backend/python/nemo
-          make --jobs=5 --output-sync=target -C backend/python/nemo test
-  tests-voxcpm:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install build-essential ffmpeg
-          sudo apt-get install -y ca-certificates cmake curl patch python3-pip
-          # Install UV
-          curl -LsSf https://astral.sh/uv/install.sh | sh
-          pip install --user --no-cache-dir grpcio-tools==1.64.1
-      - name: Test voxcpm
-        run: |
-          make --jobs=5 --output-sync=target -C backend/python/voxcpm
-          make --jobs=5 --output-sync=target -C backend/python/voxcpm test
-  tests-acestep-cpp:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential cmake curl libopenblas-dev ffmpeg
-      - name: Setup Go
-        uses: actions/setup-go@v5
-      - name: Display Go version
-        run: go version
-      - name: Proto Dependencies
-        run: |
-          # Install protoc
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Build acestep-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/acestep-cpp
-      - name: Test acestep-cpp
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/acestep-cpp test
-  tests-voxtral:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential cmake curl libopenblas-dev ffmpeg
-      - name: Setup Go
-        uses: actions/setup-go@v5
-      # You can test your matrix by printing the current Go version
-      - name: Display Go version
-        run: go version
-      - name: Proto Dependencies
-        run: |
-          # Install protoc
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Build voxtral
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/voxtral
-      - name: Test voxtral
-        run: |
-          make --jobs=5 --output-sync=target -C backend/go/voxtral test

.github/workflows/test.yml

@@ -70,7 +70,7 @@ jobs:
           sudo rm -rfv build || true
           df -h
       - name: Clone
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           submodules: true
       - name: Setup Go ${{ matrix.go-version }}
@@ -93,21 +93,35 @@ jobs:
       - name: Dependencies
         run: |
           sudo apt-get update
-          sudo apt-get install curl ffmpeg libopus-dev
+          sudo apt-get install build-essential ccache upx-ucl curl ffmpeg
-      - name: Setup Node.js
+          sudo apt-get install -y libgmock-dev clang
-        uses: actions/setup-node@v6
+          # Install UV
-        with:
+          curl -LsSf https://astral.sh/uv/install.sh | sh
-          node-version: '22'
+          sudo apt-get install -y ca-certificates cmake patch python3-pip unzip
-      - name: Build React UI
+          sudo apt-get install -y libopencv-dev
-        run: make react-ui
+
-      - name: Build backends
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-        run: |
+          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          make backends/transformers
+          rm protoc.zip
-          mkdir external && mv backends/transformers external/transformers
+
-          make backends/llama-cpp backends/local-store backends/silero-vad backends/piper backends/whisper backends/stablediffusion-ggml
+          curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
+          sudo dpkg -i cuda-keyring_1.1-1_all.deb
+          sudo apt-get update
+          sudo apt-get install -y cuda-nvcc-${CUDA_VERSION} libcublas-dev-${CUDA_VERSION}
+          export CUDACXX=/usr/local/cuda/bin/nvcc
+
+
+          # The python3-grpc-tools package in 22.04 is too old
+          pip install --user grpcio-tools==1.71.0 grpcio==1.71.0
+
+          make -C backend/python/transformers
+
+          make backends/huggingface backends/llama-cpp backends/local-store backends/silero-vad backends/piper backends/whisper backends/stablediffusion-ggml
+        env:
+          CUDA_VERSION: 12-4
       - name: Test
         run: |
-          TRANSFORMER_BACKEND=$PWD/external/transformers/run.sh PATH="$PATH:/root/go/bin" GO_TAGS="tts" make --jobs 5 --output-sync=target test
+          PATH="$PATH:/root/go/bin" GO_TAGS="tts" make --jobs 5 --output-sync=target test
       - name: Setup tmate session if tests fail
         if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23
@@ -116,7 +130,7 @@ jobs:
           connect-timeout-seconds: 180
           limit-access-to-actor: true
 
-  tests-e2e-container:
+  tests-aio-container:
     runs-on: ubuntu-latest
     steps:
       - name: Release space from worker
@@ -152,7 +166,7 @@ jobs:
           sudo rm -rfv build || true
           df -h
       - name: Clone
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           submodules: true
       - name: Dependencies
@@ -166,7 +180,7 @@ jobs:
           PATH="$PATH:$HOME/go/bin" make protogen-go
       - name: Test
         run: |
-            PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
+            PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-aio e2e-aio
       - name: Setup tmate session if tests fail
         if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23
@@ -176,13 +190,13 @@ jobs:
           limit-access-to-actor: true
 
   tests-apple:
-    runs-on: macos-latest
+    runs-on: macOS-14
     strategy:
       matrix:
         go-version: ['1.25.x']
     steps:
       - name: Clone
-        uses: actions/checkout@v6
+        uses: actions/checkout@v5
         with:
           submodules: true
       - name: Setup Go ${{ matrix.go-version }}
@@ -195,14 +209,8 @@ jobs:
         run: go version
       - name: Dependencies
         run: |
-          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm opus
+          brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm
-          pip install --user --no-cache-dir grpcio-tools grpcio
+          pip install --user --no-cache-dir grpcio-tools==1.71.0 grpcio==1.71.0
-      - name: Setup Node.js
-        uses: actions/setup-node@v6
-        with:
-          node-version: '22'
-      - name: Build React UI
-        run: make react-ui
       - name: Build llama-cpp-darwin
         run: |
           make protogen-go

.github/workflows/tests-e2e.yml

@@ -1,62 +0,0 @@
----
-name: 'E2E Backend Tests'
-
-on:
-  pull_request:
-  push:
-    branches:
-      - master
-    tags:
-      - '*'
-
-concurrency:
-  group: ci-tests-e2e-backend-${{ github.head_ref || github.ref }}-${{ github.repository }}
-  cancel-in-progress: true
-
-jobs:
-  tests-e2e-backend:
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        go-version: ['1.25.x']
-    steps:
-      - name: Clone
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-      - name: Setup Go ${{ matrix.go-version }}
-        uses: actions/setup-go@v5
-        with:
-          go-version: ${{ matrix.go-version }}
-          cache: false
-      - name: Display Go version
-        run: go version
-      - name: Proto Dependencies
-        run: |
-          # Install protoc
-          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
-          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
-          rm protoc.zip
-          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
-          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
-          PATH="$PATH:$HOME/go/bin" make protogen-go
-      - name: Dependencies
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential libopus-dev
-      - name: Setup Node.js
-        uses: actions/setup-node@v6
-        with:
-          node-version: '22'
-      - name: Build React UI
-        run: make react-ui
-      - name: Test Backend E2E
-        run: |
-          PATH="$PATH:$HOME/go/bin" make build-mock-backend test-e2e
-      - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
-        uses: mxschmitt/action-tmate@v3.23
-        with:
-          detached: true
-          connect-timeout-seconds: 180
-          limit-access-to-actor: true

.github/workflows/update_swagger.yaml

@@ -5,12 +5,11 @@ on:
   workflow_dispatch:
 jobs:
   swagger:
-    if: github.repository == 'mudler/LocalAI'
     strategy:
       fail-fast: false
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@v5
       - uses: actions/setup-go@v5
         with:
           go-version: 'stable'
@@ -26,7 +25,7 @@ jobs:
         run: |
           make protogen-go swagger
       - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v8
+        uses: peter-evans/create-pull-request@v7
         with:
           token: ${{ secrets.UPDATE_BOT_TOKEN }}
           push-to-fork: ci-forks/LocalAI

.gitignore

@@ -25,7 +25,6 @@ go-bert
 # LocalAI build binary
 LocalAI
 /local-ai
-/local-ai-launcher
 # prevent above rules from omitting the helm chart
 !charts/*
 # prevent above rules from omitting the api/localai folder
@@ -36,8 +35,6 @@ LocalAI
 models/*
 test-models/
 test-dir/
-tests/e2e-aio/backends
-mock-backend
 
 release/
 
@@ -65,10 +62,3 @@ docs/static/gallery.html
 
 # per-developer customization files for the development container
 .devcontainer/customization/*
-
-# React UI build artifacts (keep placeholder dist/index.html)
-core/http/react-ui/node_modules/
-core/http/react-ui/dist
-
-# Extracted backend binaries for container-based testing
-local-backends/

.gitmodules

@@ -1,3 +1,6 @@
 [submodule "docs/themes/hugo-theme-relearn"]
 	path = docs/themes/hugo-theme-relearn
 	url = https://github.com/McShelby/hugo-theme-relearn.git
+[submodule "docs/themes/lotusdocs"]
+	path = docs/themes/lotusdocs
+	url = https://github.com/colinwilson/lotusdocs

.goreleaser.yaml

@@ -2,7 +2,6 @@ version: 2
 before:
   hooks:
     - make protogen-go
-    - make react-ui
     - go mod tidy
 dist: release
 source:
@@ -23,9 +22,6 @@ builds:
     goarch:
       - amd64
       - arm64
-    ignore:
-      - goos: darwin
-        goarch: amd64
 archives:
   - formats: [ 'binary' ] # this removes the tar of the archives, leaving the binaries alone
     name_template: local-ai-{{ .Tag }}-{{ .Os }}-{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}

AGENTS.md

@@ -1,22 +0,0 @@
-# LocalAI Agent Instructions
-
-This file is an index to detailed topic guides in the `.agents/` directory. Read the relevant file(s) for the task at hand — you don't need to load all of them.
-
-## Topics
-
-| File | When to read |
-|------|-------------|
-| [.agents/building-and-testing.md](.agents/building-and-testing.md) | Building the project, running tests, Docker builds for specific platforms |
-| [.agents/adding-backends.md](.agents/adding-backends.md) | Adding a new backend (Python, Go, or C++) — full step-by-step checklist |
-| [.agents/coding-style.md](.agents/coding-style.md) | Code style, editorconfig, logging, documentation conventions |
-| [.agents/llama-cpp-backend.md](.agents/llama-cpp-backend.md) | Working on the llama.cpp backend — architecture, updating, tool call parsing |
-| [.agents/testing-mcp-apps.md](.agents/testing-mcp-apps.md) | Testing MCP Apps (interactive tool UIs) in the React UI |
-
-## Quick Reference
-
-- **Logging**: Use `github.com/mudler/xlog` (same API as slog)
-- **Go style**: Prefer `any` over `interface{}`
-- **Comments**: Explain *why*, not *what*
-- **Docs**: Update `docs/content/` when adding features or changing config
-- **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
-- **UI**: The active UI is the React app in `core/http/react-ui/`. The older Alpine.js/HTML UI in `core/http/static/` is pending deprecation — all new UI work goes in the React UI

CLAUDE.md

@@ -1 +0,0 @@
-AGENTS.md

CONTRIBUTING.md

@@ -7,10 +7,8 @@ Thank you for your interest in contributing to LocalAI! We appreciate your time
 - [Getting Started](#getting-started)
   - [Prerequisites](#prerequisites)
   - [Setting up the Development Environment](#setting-up-the-development-environment)
-  - [Environment Variables](#environment-variables)
 - [Contributing](#contributing)
   - [Submitting an Issue](#submitting-an-issue)
-  - [Development Workflow](#development-workflow)
   - [Creating a Pull Request (PR)](#creating-a-pull-request-pr)
 - [Coding Guidelines](#coding-guidelines)
 - [Testing](#testing)
@@ -21,122 +19,17 @@ Thank you for your interest in contributing to LocalAI! We appreciate your time
 
 ### Prerequisites
 
-- **Go 1.21+** (the project currently uses Go 1.26 in `go.mod`, but 1.21 is the minimum supported version)
+- Golang [1.21]
-  - [Download Go](https://go.dev/dl/) or install via your package manager
+- Git
-  - macOS: `brew install go`
+- macOS/Linux
-  - Ubuntu/Debian: follow the [official instructions](https://go.dev/doc/install) (the `apt` version is often outdated)
-  - Verify: `go version`
-- **Git**
-- **GNU Make**
-- **GCC / C/C++ toolchain** (required for CGo and native backends)
-- **Protocol Buffers compiler** (`protoc`) — needed for gRPC code generation
 
-#### System dependencies by platform
+### Setting up the Development Environment and running localAI in the local environment
 
-<details>
+1. Clone the repository: `git clone https://github.com/go-skynet/LocalAI.git`
-<summary><strong>Ubuntu / Debian</strong></summary>
+2. Navigate to the project directory: `cd LocalAI`
-
+3. Install the required dependencies ( see https://localai.io/basics/build/#build-localai-locally )
-```bash
+4. Build LocalAI: `make build`
-sudo apt-get update
+5. Run LocalAI: `./local-ai`
-sudo apt-get install -y build-essential gcc g++ cmake git wget \
-  protobuf-compiler libprotobuf-dev pkg-config \
-  libopencv-dev libgrpc-dev
-```
-
-</details>
-
-<details>
-<summary><strong>CentOS / RHEL / Fedora</strong></summary>
-
-```bash
-sudo dnf groupinstall -y "Development Tools"
-sudo dnf install -y cmake git wget protobuf-compiler protobuf-devel \
-  opencv-devel grpc-devel
-```
-
-</details>
-
-<details>
-<summary><strong>macOS</strong></summary>
-
-```bash
-xcode-select --install
-brew install cmake git protobuf grpc opencv wget
-```
-
-</details>
-
-<details>
-<summary><strong>Windows</strong></summary>
-
-Use [WSL 2](https://learn.microsoft.com/en-us/windows/wsl/install) with an Ubuntu distribution, then follow the Ubuntu instructions above.
-
-</details>
-
-### Setting up the Development Environment
-
-1. **Clone the repository:**
-
-   ```bash
-   git clone https://github.com/mudler/LocalAI.git
-   cd LocalAI
-   ```
-
-2. **Build LocalAI:**
-
-   ```bash
-   make build
-   ```
-
-   This runs protobuf generation, installs Go tools, builds the React UI, and compiles the `local-ai` binary. Key build variables you can set:
-
-   | Variable | Description | Example |
-   |---|---|---|
-   | `BUILD_TYPE` | GPU/accelerator type (`cublas`, `hipblas`, `intel`, ``) | `BUILD_TYPE=cublas make build` |
-   | `GO_TAGS` | Additional Go build tags | `GO_TAGS=debug make build` |
-   | `CUDA_MAJOR_VERSION` | CUDA major version (default: `13`) | `CUDA_MAJOR_VERSION=12` |
-
-3. **Run LocalAI:**
-
-   ```bash
-   ./local-ai
-   ```
-
-4. **Development mode with live reload:**
-
-   ```bash
-   make build-dev
-   ```
-
-   This installs [`air`](https://github.com/air-verse/air) automatically and watches for file changes, rebuilding and restarting the server on each save.
-
-5. **Containerized build** (no local toolchain needed):
-
-   ```bash
-   make docker
-   ```
-
-   For GPU-specific Docker builds, see the `docker-build-*` targets in the Makefile and refer to [CLAUDE.md](CLAUDE.md) for detailed backend build instructions.
-
-### Environment Variables
-
-LocalAI is configured primarily through environment variables (or equivalent CLI flags). The most useful ones for development are:
-
-| Variable | Description | Default |
-|---|---|---|
-| `LOCALAI_DEBUG` | Enable debug mode | `false` |
-| `LOCALAI_LOG_LEVEL` | Log verbosity (`error`, `warn`, `info`, `debug`, `trace`) | — |
-| `LOCALAI_LOG_FORMAT` | Log format (`default`, `text`, `json`) | `default` |
-| `LOCALAI_MODELS_PATH` | Path to model files | `./models` |
-| `LOCALAI_BACKENDS_PATH` | Path to backend binaries | `./backends` |
-| `LOCALAI_CONFIG_DIR` | Directory for dynamic config files (API keys, external backends) | `./configuration` |
-| `LOCALAI_THREADS` | Number of threads for inference | — |
-| `LOCALAI_ADDRESS` | Bind address for the API server | `:8080` |
-| `LOCALAI_API_KEY` | API key(s) for authentication | — |
-| `LOCALAI_CORS` | Enable CORS | `false` |
-| `LOCALAI_DISABLE_WEBUI` | Disable the web UI | `false` |
-
-See `core/cli/run.go` for the full list of supported environment variables.
 
 ## Contributing
 
@@ -146,143 +39,44 @@ We welcome contributions from everyone! To get started, follow these steps:
 
 If you find a bug, have a feature request, or encounter any issues, please check the [issue tracker](https://github.com/go-skynet/LocalAI/issues) to see if a similar issue has already been reported. If not, feel free to [create a new issue](https://github.com/go-skynet/LocalAI/issues/new) and provide as much detail as possible.
 
-### Development Workflow
+### Creating a Pull Request (PR)
-
-#### Branch naming conventions
-
-Use a descriptive branch name that indicates the type and scope of the change:
-
-- `feature/<short-description>` — new functionality
-- `fix/<short-description>` — bug fixes
-- `docs/<short-description>` — documentation changes
-- `refactor/<short-description>` — code refactoring
-
-#### Commit messages
-
-- Use a short, imperative subject line (e.g., "feat: add whisper backend support", not "Added whisper backend support")
-- Keep the subject under 72 characters
-- Use the body to explain **why** the change was made when the subject alone is not sufficient
-- Use [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/)
-
-#### Creating a Pull Request (PR)
-
-Before jumping into a PR for a massive feature or big change, it is preferred to discuss it first via an issue.
 
 1. Fork the repository.
-2. Create a new branch: `git checkout -b feature/my-change`
+2. Create a new branch with a descriptive name: `git checkout -b [branch name]`
-3. Make your changes, keeping commits focused and atomic.
+3. Make your changes and commit them.
-4. Run tests locally before pushing (see [Testing](#testing) below).
+4. Push the changes to your fork: `git push origin [branch name]`
-5. Push to your fork: `git push origin feature/my-change`
+5. Create a new pull request from your branch to the main project's `main` or `master` branch.
-6. Open a pull request against the `master` branch.
+6. Provide a clear description of your changes in the pull request.
-7. Fill in the PR description with:
+7. Make any requested changes during the review process.
-   - What the change does and why
+8. Once your PR is approved, it will be merged into the main project.
-   - How it was tested
-   - Any breaking changes or migration steps
-8. Respond to review feedback promptly. Push follow-up commits rather than force-pushing amended commits so reviewers can see incremental changes.
-9. Once approved, a maintainer will merge your PR.
 
 ## Coding Guidelines
 
-This project uses an [`.editorconfig`](.editorconfig) file to define formatting standards (indentation, line endings, charset, etc.). Please configure your editor to respect it.
+- No specific coding guidelines at the moment. Please make sure the code can be tested. The most popular lint tools like [`golangci-lint`](https://golangci-lint.run) can help you here.
-
-For AI-assisted development, see [`CLAUDE.md`](CLAUDE.md) for agent-specific guidelines including build instructions and backend architecture details.
-
-### General Principles
-
-- Write code that can be tested. All new features and bug fixes should include test coverage.
-- Use comments sparingly to explain **why** code does something, not **what** it does. Comments should add context that would be difficult to deduce from reading the code alone.
-- Keep changes focused. Avoid unrelated refactors, formatting changes, or feature additions in the same PR.
-
-### Go Code
-
-- Prefer modern Go idioms — for example, use `any` instead of `interface{}`.
-- Use [`golangci-lint`](https://golangci-lint.run) to catch common issues before submitting a PR.
-- Use [`github.com/mudler/xlog`](https://github.com/mudler/xlog) for logging (same API as `slog`). Do not use `fmt.Println` or the standard `log` package for operational logging.
-- Use tab indentation for Go files (as defined in `.editorconfig`).
-
-### Python Code
-
-- Use 4-space indentation (as defined in `.editorconfig`).
-- Include a `requirements.txt` for any new dependencies.
-
-### Code Review
-
-- All contributions go through code review via pull requests.
-- Reviewers will check for correctness, test coverage, adherence to these guidelines, and clarity of intent.
-- Be responsive to review feedback and keep discussions constructive.
 
 ## Testing
 
-All new features and bug fixes should include test coverage. The project uses [Ginkgo](https://onsi.github.io/ginkgo/) as its test framework.
+`make test` cannot handle all the model now. Please be sure to add a test case for the new features or the part was changed.
 
-### Running unit tests
+### Running AIO tests
 
-```bash
+All-In-One images has a set of tests that automatically verifies that most of the endpoints works correctly, a flow can be :
-make test
-```
-
-This downloads test model fixtures, runs protobuf generation, and executes the full test suite including llama-gguf, TTS, and stable-diffusion tests. Note: some tests require model files to be downloaded, so the first run may take longer.
-
-To run tests for a specific package:
-
-```bash
-go test ./core/config/...
-go test ./pkg/model/...
-```
-
-To run a specific test by name using Ginkgo's `--focus` flag:
-
-```bash
-go run github.com/onsi/ginkgo/v2/ginkgo --focus="should load a model" -v -r ./core/
-```
-
-### Running end-to-end tests
-
-The e2e tests run LocalAI in a Docker container and exercise the API:
-
-```bash
-make test-e2e
-```
-
-### Running E2E container tests
-
-These tests build a standard LocalAI Docker image and run it with pre-configured model configs to verify that most endpoints work correctly:
 
 ```bash
 # Build the LocalAI docker image
-make docker-build-e2e
+make DOCKER_IMAGE=local-ai docker
 
-# Run the e2e tests (uses model configs from tests/e2e-aio/models/)
+# Build the corresponding AIO image
-make e2e-aio
+BASE_IMAGE=local-ai DOCKER_AIO_IMAGE=local-ai-aio:test make docker-aio
-```
 
-### Testing backends
+# Run the AIO e2e tests
-
+LOCALAI_IMAGE_TAG=test LOCALAI_IMAGE=local-ai-aio make run-e2e-aio
-To prepare and test extra (Python) backends:
-
-```bash
-make prepare-test-extra   # build Python backends for testing
-make test-extra           # run backend-specific tests
 ```
 
 ## Documentation
 
-We welcome contributions to the documentation. Please open a new PR or create a new issue. The documentation is available under `docs/` https://github.com/mudler/LocalAI/tree/master/docs
+We are welcome the contribution of the documents, please open new PR or create a new issue. The documentation is available under `docs/` https://github.com/mudler/LocalAI/tree/master/docs
-
+ 
-### Gallery YAML Schema
-
-LocalAI provides a JSON Schema for gallery model YAML files at:
-
-`core/schema/gallery-model.schema.json`
-
-This schema mirrors the internal gallery model configuration and can be used by editors (such as VS Code) to enable autocomplete, validation, and inline documentation when creating or modifying gallery files.
-
-To use it with the YAML language server, add the following comment at the top of a gallery YAML file:
-
-```yaml
-# yaml-language-server: $schema=../core/schema/gallery-model.schema.json
-```
-
 ## Community and Communication
 
 - You can reach out via the Github issue tracker.

Dockerfile

@@ -1,7 +1,6 @@
-ARG BASE_IMAGE=ubuntu:24.04
+ARG BASE_IMAGE=ubuntu:22.04
 ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
 ARG INTEL_BASE_IMAGE=${BASE_IMAGE}
-ARG UBUNTU_CODENAME=noble
 
 FROM ${BASE_IMAGE} AS requirements
 
@@ -10,7 +9,7 @@ ENV DEBIAN_FRONTEND=noninteractive
 RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         ca-certificates curl wget espeak-ng libgomp1 \
-        ffmpeg libopenblas0 libopenblas-dev libopus0 sox && \
+        ffmpeg libopenblas-base libopenblas-dev && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
@@ -24,7 +23,6 @@ ARG SKIP_DRIVERS=false
 ARG TARGETARCH
 ARG TARGETVARIANT
 ENV BUILD_TYPE=${BUILD_TYPE}
-ARG UBUNTU_VERSION=2404
 
 RUN mkdir -p /run/localai
 RUN echo "default" > /run/localai/capability
@@ -35,45 +33,11 @@ RUN <<EOT bash
         apt-get update && \
         apt-get install -y  --no-install-recommends \
             software-properties-common pciutils wget gpg-agent && \
-        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
+        wget -qO - https://packages.lunarg.com/lunarg-signing-key-pub.asc | apt-key add - && \
-            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
+        wget -qO /etc/apt/sources.list.d/lunarg-vulkan-jammy.list https://packages.lunarg.com/vulkan/lunarg-vulkan-jammy.list && \
-            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
+        apt-get update && \
-            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
+        apt-get install -y \
-            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
+            vulkan-sdk && \
-            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils mesa-vulkan-drivers
-        if [ "amd64" = "$TARGETARCH" ]; then
-            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
-            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            mkdir -p /opt/vulkan-sdk && \
-            mv 1.4.335.0 /opt/vulkan-sdk/ && \
-            cd /opt/vulkan-sdk/1.4.335.0 && \
-            ./vulkansdk --no-deps --maxjobs \
-                vulkan-loader \
-                vulkan-validationlayers \
-                vulkan-extensionlayer \
-                vulkan-tools \
-                shaderc && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
-            rm -rf /opt/vulkan-sdk
-        fi
-        if [ "arm64" = "$TARGETARCH" ]; then
-            mkdir vulkan && cd vulkan && \
-            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
-            tar -xvf vulkan-sdk.tar.xz && \
-            rm vulkan-sdk.tar.xz && \
-            cd 1.4.335.0 && \
-            cp -rfv aarch64/bin/* /usr/bin/ && \
-            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
-            cp -rfv aarch64/include/* /usr/include/ && \
-            cp -rfv aarch64/share/* /usr/share/ && \
-            cd ../.. && \
-            rm -rf vulkan
-        fi
-        ldconfig && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \
         echo "vulkan" > /run/localai/capability
@@ -82,19 +46,15 @@ EOT
 
 # CuBLAS requirements
 RUN <<EOT bash
-    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
+    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y  --no-install-recommends \
             software-properties-common pciutils
         if [ "amd64" = "$TARGETARCH" ]; then
-            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
         fi
         if [ "arm64" = "$TARGETARCH" ]; then
-            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/arm64/cuda-keyring_1.1-1_all.deb
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
-            else
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
-            fi
         fi
         dpkg -i cuda-keyring_1.1-1_all.deb && \
         rm -f cuda-keyring_1.1-1_all.deb && \
@@ -105,34 +65,26 @@ RUN <<EOT bash
             libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} && \
-        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
-            apt-get install -y --no-install-recommends \
-            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
-        fi
         apt-get clean && \
         rm -rf /var/lib/apt/lists/* && \
-        echo "nvidia-cuda-${CUDA_MAJOR_VERSION}" > /run/localai/capability
+        echo "nvidia" > /run/localai/capability
     fi
 EOT
 
 RUN <<EOT bash
     if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
-        echo "nvidia-l4t-cuda-${CUDA_MAJOR_VERSION}" > /run/localai/capability
+        echo "nvidia-l4t" > /run/localai/capability
     fi
 EOT
 
 # https://github.com/NVIDIA/Isaac-GR00T/issues/343
 RUN <<EOT bash
     if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
-        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu2204-0.6.0_0.6.0-1_arm64.deb && \
-        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        dpkg -i cudss-local-tegra-repo-ubuntu2204-0.6.0_0.6.0-1_arm64.deb && \
-        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
+        cp /var/cudss-local-tegra-repo-ubuntu2204-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
+        apt-get update && apt-get -y install cudss
-        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get install -y nvpl
     fi
 EOT
 
@@ -176,12 +128,13 @@ ENV PATH=/opt/rocm/bin:${PATH}
 # The requirements-core target is common to all images.  It should not be placed in requirements-core unless every single build will use it.
 FROM requirements-drivers AS build-requirements
 
-ARG GO_VERSION=1.25.4
+ARG GO_VERSION=1.22.6
-ARG CMAKE_VERSION=3.31.10
+ARG CMAKE_VERSION=3.26.4
 ARG CMAKE_FROM_SOURCE=false
 ARG TARGETARCH
 ARG TARGETVARIANT
 
+
 RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
@@ -190,7 +143,6 @@ RUN apt-get update && \
         curl libssl-dev \
         git \
         git-lfs \
-        libopus-dev pkg-config \
         unzip upx-ucl python3 python-is-python3 && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
@@ -219,6 +171,14 @@ RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2 && \
 COPY --chmod=644 custom-ca-certs/* /usr/local/share/ca-certificates/
 RUN update-ca-certificates
 
+
+# OpenBLAS requirements and stable diffusion
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        libopenblas-dev && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
 RUN test -n "$TARGETARCH" \
     || (echo 'warn: missing $TARGETARCH, either set this `ARG` manually, or run using `docker buildkit`')
 
@@ -239,10 +199,9 @@ WORKDIR /build
 # https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/APT-Repository-not-working-signatures-invalid/m-p/1599436/highlight/true#M36143
 # This is a temporary workaround until Intel fixes their repository
 FROM ${INTEL_BASE_IMAGE} AS intel
-ARG UBUNTU_CODENAME=noble
 RUN wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | \
 gpg --yes --dearmor --output /usr/share/keyrings/intel-graphics.gpg
-RUN echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu ${UBUNTU_CODENAME}/lts/2350 unified" > /etc/apt/sources.list.d/intel-graphics.list
+RUN echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu jammy/lts/2350 unified" > /etc/apt/sources.list.d/intel-graphics.list
 RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         intel-oneapi-runtime-libs && \
@@ -292,17 +251,6 @@ EOT
 ###################################
 ###################################
 
-# Build React UI
-FROM node:25-slim AS react-ui-builder
-WORKDIR /app
-COPY core/http/react-ui/package*.json ./
-RUN npm install
-COPY core/http/react-ui/ ./
-RUN npm run build
-
-###################################
-###################################
-
 # Compile backends first in a separate stage
 FROM builder-base AS builder-backends
 ARG TARGETARCH
@@ -332,9 +280,6 @@ WORKDIR /build
 
 COPY . .
 
-# Copy pre-built React UI
-COPY --from=react-ui-builder /app/dist ./core/http/react-ui/dist
-
 ## Build the binary
 ## If we're on arm64 AND using cublas/hipblas, skip some of the llama-compat backends to save space
 ## Otherwise just run the normal build
@@ -379,17 +324,14 @@ COPY ./entrypoint.sh .
 
 # Copy the binary
 COPY --from=builder /build/local-ai ./
-# Copy the opus shim if it was built
-RUN --mount=from=builder,src=/build/,dst=/mnt/build \
-    if [ -f /mnt/build/libopusshim.so ]; then cp /mnt/build/libopusshim.so ./; fi
 
 # Make sure the models directory exists
-RUN mkdir -p /models /backends /data
+RUN mkdir -p /models /backends
 
 # Define the health check command
 HEALTHCHECK --interval=1m --timeout=10m --retries=10 \
   CMD curl -f ${HEALTHCHECK_ENDPOINT} || exit 1
 
-VOLUME /models /backends /configuration /data
+VOLUME /models /backends
 EXPOSE 8080
 ENTRYPOINT [ "/entrypoint.sh" ]

Dockerfile.aio

@@ -0,0 +1,8 @@
+ARG BASE_IMAGE=ubuntu:22.04
+
+FROM ${BASE_IMAGE} 
+
+RUN apt-get update && apt-get install -y pciutils && apt-get clean
+
+COPY aio/ /aio
+ENTRYPOINT [ "/aio/entrypoint.sh" ]

Makefile

@@ -1,20 +1,12 @@
-# Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus
-
 GOCMD=go
 GOTEST=$(GOCMD) test
 GOVET=$(GOCMD) vet
 BINARY_NAME=local-ai
 LAUNCHER_BINARY_NAME=local-ai-launcher
 
-UBUNTU_VERSION?=2404
-UBUNTU_CODENAME?=noble
-
 GORELEASER?=
 
 export BUILD_TYPE?=
-export CUDA_MAJOR_VERSION?=13
-export CUDA_MINOR_VERSION?=0
 
 GO_TAGS?=
 BUILD_ID?=
@@ -91,23 +83,8 @@ install-go-tools:
 	go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
 	go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
 
-## React UI:
-react-ui:
-ifneq ($(wildcard core/http/react-ui/dist),)
-	@echo "react-ui dist already exists, skipping build"
-else
-	cd core/http/react-ui && npm install && npm run build
-endif
-
-react-ui-docker:
-	docker run --entrypoint /bin/bash -v $(CURDIR):/app:z oven/bun:1 \
-	  -c "cd /app/core/http/react-ui && bun install && bun run build"
-
-core/http/react-ui/dist: react-ui
-
 ## Build:
-
+build: protogen-go install-go-tools ## Build the project
-build: protogen-go install-go-tools core/http/react-ui/dist ## Build the project
 	$(info ${GREEN}I local-ai build info:${RESET})
 	$(info ${GREEN}I BUILD_TYPE: ${YELLOW}$(BUILD_TYPE)${RESET})
 	$(info ${GREEN}I GO_TAGS: ${YELLOW}$(GO_TAGS)${RESET})
@@ -126,10 +103,6 @@ build-launcher: ## Build the launcher application
 
 build-all: build build-launcher ## Build both server and launcher
 
-build-dev: ## Run LocalAI in dev mode with live reload
-	@command -v air >/dev/null 2>&1 || go install github.com/air-verse/air@latest
-	air -c .air.toml
-
 dev-dist:
 	$(GORELEASER) build --snapshot --clean
 
@@ -164,7 +137,6 @@ test: test-models/testmodel.ggml protogen-go
 	@echo 'Running tests'
 	export GO_TAGS="debug"
 	$(MAKE) prepare-test
-	OPUS_SHIM_LIBRARY=$(abspath ./pkg/opus/shim/libopusshim.so) \
 	HUGGINGFACE_GRPC=$(abspath ./)/backend/python/transformers/run.sh TEST_DIR=$(abspath ./)/test-dir/ FIXTURES=$(abspath ./)/tests/fixtures CONFIG_FILE=$(abspath ./)/test-models/config.yaml MODELS_PATH=$(abspath ./)/test-models BACKENDS_PATH=$(abspath ./)/backends \
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="!llama-gguf"  --flake-attempts $(TEST_FLAKES) --fail-fast -v -r $(TEST_PATHS)
 	$(MAKE) test-llama-gguf
@@ -172,27 +144,18 @@ test: test-models/testmodel.ggml protogen-go
 	$(MAKE) test-stablediffusion
 
 ########################################################
-## E2E AIO tests (uses standard image with pre-configured models)
+## AIO tests
 ########################################################
 
-docker-build-e2e:
+docker-build-aio:
-	docker build \
+	docker build --build-arg MAKEFLAGS="--jobs=5 --output-sync=target" -t local-ai:tests -f Dockerfile .
-		--build-arg MAKEFLAGS="--jobs=5 --output-sync=target" \
+	BASE_IMAGE=local-ai:tests DOCKER_AIO_IMAGE=local-ai-aio:test $(MAKE) docker-aio
-		--build-arg BASE_IMAGE=$(BASE_IMAGE) \
-		--build-arg IMAGE_TYPE=$(IMAGE_TYPE) \
-		--build-arg BUILD_TYPE=$(BUILD_TYPE) \
-		--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
-		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
-		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
-		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg GO_TAGS="$(GO_TAGS)" \
-		-t local-ai:tests -f Dockerfile .
 
 e2e-aio:
 	LOCALAI_BACKEND_DIR=$(abspath ./backends) \
-	LOCALAI_MODELS_DIR=$(abspath ./tests/e2e-aio/models) \
+	LOCALAI_MODELS_DIR=$(abspath ./models) \
-	LOCALAI_IMAGE_TAG=tests \
+	LOCALAI_IMAGE_TAG=test \
-	LOCALAI_IMAGE=local-ai \
+	LOCALAI_IMAGE=local-ai-aio \
 	$(MAKE) run-e2e-aio
 
 run-e2e-aio: protogen-go
@@ -204,29 +167,20 @@ run-e2e-aio: protogen-go
 ########################################################
 
 prepare-e2e:
-	docker build \
+	mkdir -p $(TEST_DIR)
-		--build-arg IMAGE_TYPE=core \
+	cp -rfv $(abspath ./tests/e2e-fixtures)/gpu.yaml $(TEST_DIR)/gpu.yaml
-		--build-arg BUILD_TYPE=$(BUILD_TYPE) \
+	test -e $(TEST_DIR)/ggllm-test-model.bin || wget -q https://huggingface.co/TheBloke/CodeLlama-7B-Instruct-GGUF/resolve/main/codellama-7b-instruct.Q2_K.gguf -O $(TEST_DIR)/ggllm-test-model.bin
-		--build-arg BASE_IMAGE=$(BASE_IMAGE) \
+	docker build --build-arg IMAGE_TYPE=core --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg CUDA_MAJOR_VERSION=12 --build-arg CUDA_MINOR_VERSION=0 -t localai-tests .
-		--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
-		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
-		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
-		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		--build-arg GO_TAGS="$(GO_TAGS)" \
-		--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
-		-t localai-tests .
 
 run-e2e-image:
-	docker run -p 5390:8080 -e MODELS_PATH=/models -e THREADS=1 -e DEBUG=true -d --rm -v $(TEST_DIR):/models --name e2e-tests-$(RANDOM) localai-tests
+	ls -liah $(abspath ./tests/e2e-fixtures)
+	docker run -p 5390:8080 -e MODELS_PATH=/models -e THREADS=1 -e DEBUG=true -d --rm -v $(TEST_DIR):/models --gpus all --name e2e-tests-$(RANDOM) localai-tests
 
-test-e2e: build-mock-backend prepare-e2e run-e2e-image
+test-e2e:
 	@echo 'Running e2e tests'
 	BUILD_TYPE=$(BUILD_TYPE) \
-	LOCALAI_API=http://$(E2E_BRIDGE_IP):5390 \
+	LOCALAI_API=http://$(E2E_BRIDGE_IP):5390/v1 \
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
-	$(MAKE) clean-mock-backend
-	$(MAKE) teardown-e2e
-	docker rmi localai-tests
 
 teardown-e2e:
 	rm -rf $(TEST_DIR) || true
@@ -251,88 +205,6 @@ test-stablediffusion: prepare-test
 test-stores:
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="stores" --flake-attempts $(TEST_FLAKES) -v -r tests/integration
 
-test-opus:
-	@echo 'Running opus backend tests'
-	$(MAKE) -C backend/go/opus libopusshim.so
-	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./backend/go/opus/...
-
-test-opus-docker:
-	@echo 'Running opus backend tests in Docker'
-	docker build --target builder \
-	  --build-arg BUILD_TYPE=$(or $(BUILD_TYPE),) \
-	  --build-arg BASE_IMAGE=$(or $(BASE_IMAGE),ubuntu:24.04) \
-	  --build-arg BACKEND=opus \
-	  -t localai-opus-test -f backend/Dockerfile.golang .
-	docker run --rm localai-opus-test \
-	  bash -c 'cd /LocalAI && go run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./backend/go/opus/...'
-
-test-realtime: build-mock-backend
-	@echo 'Running realtime e2e tests (mock backend)'
-	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="Realtime && !real-models" --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
-
-# Real-model realtime tests. Set REALTIME_TEST_MODEL to use your own pipeline,
-# or leave unset to auto-build one from the component env vars below.
-REALTIME_VAD?=silero-vad-ggml
-REALTIME_STT?=whisper-1
-REALTIME_LLM?=qwen3-0.6b
-REALTIME_TTS?=tts-1
-REALTIME_BACKENDS_PATH?=$(abspath ./)/backends
-
-test-realtime-models: build-mock-backend
-	@echo 'Running realtime e2e tests (real models)'
-	REALTIME_TEST_MODEL=$${REALTIME_TEST_MODEL:-realtime-test-pipeline} \
-	REALTIME_VAD=$(REALTIME_VAD) \
-	REALTIME_STT=$(REALTIME_STT) \
-	REALTIME_LLM=$(REALTIME_LLM) \
-	REALTIME_TTS=$(REALTIME_TTS) \
-	REALTIME_BACKENDS_PATH=$(REALTIME_BACKENDS_PATH) \
-	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="Realtime" --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
-
-# --- Container-based real-model testing ---
-
-REALTIME_BACKEND_NAMES ?= silero-vad whisper llama-cpp kokoro
-REALTIME_MODELS_DIR ?= $(abspath ./models)
-REALTIME_BACKENDS_DIR ?= $(abspath ./local-backends)
-REALTIME_DOCKER_FLAGS ?= --gpus all
-
-local-backends:
-	mkdir -p local-backends
-
-extract-backend-%: docker-build-% local-backends
-	@echo "Extracting backend $*..."
-	@CID=$$(docker create local-ai-backend:$*) && \
-	  rm -rf local-backends/$* && mkdir -p local-backends/$* && \
-	  docker cp $$CID:/ - | tar -xf - -C local-backends/$* && \
-	  docker rm $$CID > /dev/null
-
-extract-realtime-backends: $(addprefix extract-backend-,$(REALTIME_BACKEND_NAMES))
-
-test-realtime-models-docker: build-mock-backend
-	docker build --target build-requirements \
-	  --build-arg BUILD_TYPE=$(or $(BUILD_TYPE),cublas) \
-	  --build-arg CUDA_MAJOR_VERSION=$(or $(CUDA_MAJOR_VERSION),13) \
-	  --build-arg CUDA_MINOR_VERSION=$(or $(CUDA_MINOR_VERSION),0) \
-	  -t localai-test-runner .
-	docker run --rm \
-	  $(REALTIME_DOCKER_FLAGS) \
-	  -v $(abspath ./):/build \
-	  -v $(REALTIME_MODELS_DIR):/models:ro \
-	  -v $(REALTIME_BACKENDS_DIR):/backends \
-	  -v localai-go-cache:/root/go/pkg/mod \
-	  -v localai-go-build-cache:/root/.cache/go-build \
-	  -e REALTIME_TEST_MODEL=$${REALTIME_TEST_MODEL:-realtime-test-pipeline} \
-	  -e REALTIME_VAD=$(REALTIME_VAD) \
-	  -e REALTIME_STT=$(REALTIME_STT) \
-	  -e REALTIME_LLM=$(REALTIME_LLM) \
-	  -e REALTIME_TTS=$(REALTIME_TTS) \
-	  -e REALTIME_BACKENDS_PATH=/backends \
-	  -e REALTIME_MODELS_PATH=/models \
-	  -w /build \
-	  localai-test-runner \
-	  bash -c 'git config --global --add safe.directory /build && \
-	    make protogen-go && make build-mock-backend && \
-	    go run github.com/onsi/ginkgo/v2/ginkgo --label-filter="Realtime" --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e'
-
 test-container:
 	docker build --target requirements -t local-ai-test-container .
 	docker run -ti --rm --entrypoint /bin/bash -ti -v $(abspath ./):/build local-ai-test-container
@@ -389,7 +261,7 @@ protoc:
 	  echo "Unsupported OS: $$OS_NAME"; exit 1; \
 	fi; \
 	URL=https://github.com/protocolbuffers/protobuf/releases/download/v31.1/$$FILE; \
-	curl -L $$URL -o protoc.zip && \
+	curl -L -s $$URL -o protoc.zip && \
 	unzip -j -d $(CURDIR) protoc.zip bin/protoc && rm protoc.zip
 
 .PHONY: protogen-go
@@ -405,45 +277,20 @@ protogen-go-clean:
 
 prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/transformers
-	$(MAKE) -C backend/python/outetts
 	$(MAKE) -C backend/python/diffusers
 	$(MAKE) -C backend/python/chatterbox
 	$(MAKE) -C backend/python/vllm
-	$(MAKE) -C backend/python/vllm-omni
-	$(MAKE) -C backend/python/vibevoice
-	$(MAKE) -C backend/python/moonshine
-	$(MAKE) -C backend/python/pocket-tts
-	$(MAKE) -C backend/python/qwen-tts
-	$(MAKE) -C backend/python/fish-speech
-	$(MAKE) -C backend/python/faster-qwen3-tts
-	$(MAKE) -C backend/python/qwen-asr
-	$(MAKE) -C backend/python/nemo
-	$(MAKE) -C backend/python/voxcpm
-	$(MAKE) -C backend/python/whisperx
-	$(MAKE) -C backend/python/ace-step
 
 test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/transformers test
-	$(MAKE) -C backend/python/outetts test
 	$(MAKE) -C backend/python/diffusers test
 	$(MAKE) -C backend/python/chatterbox test
 	$(MAKE) -C backend/python/vllm test
-	$(MAKE) -C backend/python/vllm-omni test
-	$(MAKE) -C backend/python/vibevoice test
-	$(MAKE) -C backend/python/moonshine test
-	$(MAKE) -C backend/python/pocket-tts test
-	$(MAKE) -C backend/python/qwen-tts test
-	$(MAKE) -C backend/python/fish-speech test
-	$(MAKE) -C backend/python/faster-qwen3-tts test
-	$(MAKE) -C backend/python/qwen-asr test
-	$(MAKE) -C backend/python/nemo test
-	$(MAKE) -C backend/python/voxcpm test
-	$(MAKE) -C backend/python/whisperx test
-	$(MAKE) -C backend/python/ace-step test
 
 DOCKER_IMAGE?=local-ai
+DOCKER_AIO_IMAGE?=local-ai-aio
 IMAGE_TYPE?=core
-BASE_IMAGE?=ubuntu:24.04
+BASE_IMAGE?=ubuntu:22.04
 
 docker:
 	docker build \
@@ -452,52 +299,86 @@ docker:
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
 		--build-arg BUILD_TYPE=$(BUILD_TYPE) \
-		--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
-		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
-		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
-		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
 		-t $(DOCKER_IMAGE) .
 
-docker-cuda12:
+docker-cuda11:
 	docker build \
-		--build-arg CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
+		--build-arg CUDA_MAJOR_VERSION=11 \
-		--build-arg CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
+		--build-arg CUDA_MINOR_VERSION=8 \
 		--build-arg BASE_IMAGE=$(BASE_IMAGE) \
 		--build-arg IMAGE_TYPE=$(IMAGE_TYPE) \
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
 		--build-arg BUILD_TYPE=$(BUILD_TYPE) \
-		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
+		-t $(DOCKER_IMAGE)-cuda-11 .
-		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
+
-		-t $(DOCKER_IMAGE)-cuda-12 .
+docker-aio:
+	@echo "Building AIO image with base $(BASE_IMAGE) as $(DOCKER_AIO_IMAGE)"
+	docker build \
+		--build-arg BASE_IMAGE=$(BASE_IMAGE) \
+		--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
+		-t $(DOCKER_AIO_IMAGE) -f Dockerfile.aio .
+
+docker-aio-all:
+	$(MAKE) docker-aio DOCKER_AIO_SIZE=cpu
+	$(MAKE) docker-aio DOCKER_AIO_SIZE=cpu
 
 docker-image-intel:
 	docker build \
-		--build-arg BASE_IMAGE=intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04 \
+		--build-arg BASE_IMAGE=quay.io/go-skynet/intel-oneapi-base:latest \
 		--build-arg IMAGE_TYPE=$(IMAGE_TYPE) \
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
-		--build-arg BUILD_TYPE=intel \
+		--build-arg BUILD_TYPE=intel -t $(DOCKER_IMAGE) .
-		--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
-		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
-		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
-		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		-t $(DOCKER_IMAGE) .
 
 ########################################################
 ## Backends
 ########################################################
 
-# Pattern rule for standard backends (docker-based)
-# This matches all backends that use docker-build-* and docker-save-*
-backends/%: docker-build-% docker-save-% build
-	./local-ai backends install "ocifile://$(abspath ./backend-images/$*.tar)"
 
-# Darwin-specific backends (keep as explicit targets since they have special build logic)
+backends/diffusers: docker-build-diffusers docker-save-diffusers build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/diffusers.tar)"
+
+backends/llama-cpp: docker-build-llama-cpp docker-save-llama-cpp build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/llama-cpp.tar)"
+
+backends/piper: docker-build-piper docker-save-piper build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/piper.tar)"
+
+backends/stablediffusion-ggml: docker-build-stablediffusion-ggml docker-save-stablediffusion-ggml build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/stablediffusion-ggml.tar)"
+
+backends/whisper: docker-build-whisper docker-save-whisper build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/whisper.tar)"
+
+backends/silero-vad: docker-build-silero-vad docker-save-silero-vad build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/silero-vad.tar)"
+
+backends/local-store: docker-build-local-store docker-save-local-store build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/local-store.tar)"
+
+backends/huggingface: docker-build-huggingface docker-save-huggingface build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/huggingface.tar)"
+
+backends/rfdetr: docker-build-rfdetr docker-save-rfdetr build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/rfdetr.tar)"
+
+backends/kitten-tts: docker-build-kitten-tts docker-save-kitten-tts build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/kitten-tts.tar)"
+
+backends/kokoro: docker-build-kokoro docker-save-kokoro build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/kokoro.tar)"
+
+backends/chatterbox: docker-build-chatterbox docker-save-chatterbox build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/chatterbox.tar)"
+
 backends/llama-cpp-darwin: build
 	bash ./scripts/build/llama-cpp-darwin.sh
 	./local-ai backends install "ocifile://$(abspath ./backend-images/llama-cpp.tar)"
 
+backends/neutts: docker-build-neutts docker-save-neutts build
+	./local-ai backends install "ocifile://$(abspath ./backend-images/neutts.tar)"
+
 build-darwin-python-backend: build
 	bash ./scripts/build/python-darwin.sh
 
@@ -520,10 +401,6 @@ backends/mlx-audio:
 	BACKEND=mlx-audio $(MAKE) build-darwin-python-backend
 	./local-ai backends install "ocifile://$(abspath ./backend-images/mlx-audio.tar)"
 
-backends/mlx-distributed:
-	BACKEND=mlx-distributed $(MAKE) build-darwin-python-backend
-	./local-ai backends install "ocifile://$(abspath ./backend-images/mlx-distributed.tar)"
-
 backends/stablediffusion-ggml-darwin:
 	BACKEND=stablediffusion-ggml BUILD_TYPE=metal $(MAKE) build-darwin-go-backend
 	./local-ai backends install "ocifile://$(abspath ./backend-images/stablediffusion-ggml.tar)"
@@ -531,120 +408,112 @@ backends/stablediffusion-ggml-darwin:
 backend-images:
 	mkdir -p backend-images
 
-# Backend metadata: BACKEND_NAME | DOCKERFILE_TYPE | BUILD_CONTEXT | PROGRESS_FLAG | NEEDS_BACKEND_ARG
+docker-build-llama-cpp:
-# llama-cpp is special - uses llama-cpp Dockerfile and doesn't need BACKEND arg
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:llama-cpp -f backend/Dockerfile.llama-cpp .
-BACKEND_LLAMA_CPP = llama-cpp|llama-cpp|.|false|false
 
-# Golang backends
+docker-build-bark-cpp:
-BACKEND_PIPER = piper|golang|.|false|true
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:bark-cpp -f backend/Dockerfile.golang --build-arg BACKEND=bark-cpp .
-BACKEND_LOCAL_STORE = local-store|golang|.|false|true
-BACKEND_HUGGINGFACE = huggingface|golang|.|false|true
-BACKEND_SILERO_VAD = silero-vad|golang|.|false|true
-BACKEND_STABLEDIFFUSION_GGML = stablediffusion-ggml|golang|.|--progress=plain|true
-BACKEND_WHISPER = whisper|golang|.|false|true
-BACKEND_VOXTRAL = voxtral|golang|.|false|true
-BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
-BACKEND_OPUS = opus|golang|.|false|true
 
-# Python backends with root context
+docker-build-piper:
-BACKEND_RERANKERS = rerankers|python|.|false|true
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:piper -f backend/Dockerfile.golang --build-arg BACKEND=piper .
-BACKEND_TRANSFORMERS = transformers|python|.|false|true
-BACKEND_OUTETTS = outetts|python|.|false|true
-BACKEND_FASTER_WHISPER = faster-whisper|python|.|false|true
-BACKEND_COQUI = coqui|python|.|false|true
-BACKEND_RFDETR = rfdetr|python|.|false|true
-BACKEND_KITTEN_TTS = kitten-tts|python|.|false|true
-BACKEND_NEUTTS = neutts|python|.|false|true
-BACKEND_KOKORO = kokoro|python|.|false|true
-BACKEND_VLLM = vllm|python|.|false|true
-BACKEND_VLLM_OMNI = vllm-omni|python|.|false|true
-BACKEND_DIFFUSERS = diffusers|python|.|--progress=plain|true
-BACKEND_CHATTERBOX = chatterbox|python|.|false|true
-BACKEND_VIBEVOICE = vibevoice|python|.|--progress=plain|true
-BACKEND_MOONSHINE = moonshine|python|.|false|true
-BACKEND_POCKET_TTS = pocket-tts|python|.|false|true
-BACKEND_QWEN_TTS = qwen-tts|python|.|false|true
-BACKEND_FISH_SPEECH = fish-speech|python|.|false|true
-BACKEND_FASTER_QWEN3_TTS = faster-qwen3-tts|python|.|false|true
-BACKEND_QWEN_ASR = qwen-asr|python|.|false|true
-BACKEND_NEMO = nemo|python|.|false|true
-BACKEND_VOXCPM = voxcpm|python|.|false|true
-BACKEND_WHISPERX = whisperx|python|.|false|true
-BACKEND_ACE_STEP = ace-step|python|.|false|true
-BACKEND_MLX_DISTRIBUTED = mlx-distributed|python|./|false|true
 
-# Helper function to build docker image for a backend
+docker-build-local-store:
-# Usage: $(call docker-build-backend,BACKEND_NAME,DOCKERFILE_TYPE,BUILD_CONTEXT,PROGRESS_FLAG,NEEDS_BACKEND_ARG)
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:local-store -f backend/Dockerfile.golang --build-arg BACKEND=local-store .
-define docker-build-backend
-	docker build $(if $(filter-out false,$(4)),$(4)) \
-		--build-arg BUILD_TYPE=$(BUILD_TYPE) \
-		--build-arg BASE_IMAGE=$(BASE_IMAGE) \
-		--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
-		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
-		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
-		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
-		$(if $(filter true,$(5)),--build-arg BACKEND=$(1)) \
-		-t local-ai-backend:$(1) -f backend/Dockerfile.$(2) $(3)
-endef
 
-# Generate docker-build targets from backend definitions
+docker-build-huggingface:
-define generate-docker-build-target
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:huggingface -f backend/Dockerfile.golang --build-arg BACKEND=huggingface .
-docker-build-$(word 1,$(subst |, ,$(1))):
-	$$(call docker-build-backend,$(word 1,$(subst |, ,$(1))),$(word 2,$(subst |, ,$(1))),$(word 3,$(subst |, ,$(1))),$(word 4,$(subst |, ,$(1))),$(word 5,$(subst |, ,$(1))))
-endef
 
-# Generate all docker-build targets
+docker-build-rfdetr:
-$(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP)))
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:rfdetr -f backend/Dockerfile.python --build-arg BACKEND=rfdetr ./backend
-$(eval $(call generate-docker-build-target,$(BACKEND_PIPER)))
-$(eval $(call generate-docker-build-target,$(BACKEND_LOCAL_STORE)))
-$(eval $(call generate-docker-build-target,$(BACKEND_HUGGINGFACE)))
-$(eval $(call generate-docker-build-target,$(BACKEND_SILERO_VAD)))
-$(eval $(call generate-docker-build-target,$(BACKEND_STABLEDIFFUSION_GGML)))
-$(eval $(call generate-docker-build-target,$(BACKEND_WHISPER)))
-$(eval $(call generate-docker-build-target,$(BACKEND_VOXTRAL)))
-$(eval $(call generate-docker-build-target,$(BACKEND_OPUS)))
-$(eval $(call generate-docker-build-target,$(BACKEND_RERANKERS)))
-$(eval $(call generate-docker-build-target,$(BACKEND_TRANSFORMERS)))
-$(eval $(call generate-docker-build-target,$(BACKEND_OUTETTS)))
-$(eval $(call generate-docker-build-target,$(BACKEND_FASTER_WHISPER)))
-$(eval $(call generate-docker-build-target,$(BACKEND_COQUI)))
-$(eval $(call generate-docker-build-target,$(BACKEND_RFDETR)))
-$(eval $(call generate-docker-build-target,$(BACKEND_KITTEN_TTS)))
-$(eval $(call generate-docker-build-target,$(BACKEND_NEUTTS)))
-$(eval $(call generate-docker-build-target,$(BACKEND_KOKORO)))
-$(eval $(call generate-docker-build-target,$(BACKEND_VLLM)))
-$(eval $(call generate-docker-build-target,$(BACKEND_VLLM_OMNI)))
-$(eval $(call generate-docker-build-target,$(BACKEND_DIFFUSERS)))
-$(eval $(call generate-docker-build-target,$(BACKEND_CHATTERBOX)))
-$(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE)))
-$(eval $(call generate-docker-build-target,$(BACKEND_MOONSHINE)))
-$(eval $(call generate-docker-build-target,$(BACKEND_POCKET_TTS)))
-$(eval $(call generate-docker-build-target,$(BACKEND_QWEN_TTS)))
-$(eval $(call generate-docker-build-target,$(BACKEND_FISH_SPEECH)))
-$(eval $(call generate-docker-build-target,$(BACKEND_FASTER_QWEN3_TTS)))
-$(eval $(call generate-docker-build-target,$(BACKEND_QWEN_ASR)))
-$(eval $(call generate-docker-build-target,$(BACKEND_NEMO)))
-$(eval $(call generate-docker-build-target,$(BACKEND_VOXCPM)))
-$(eval $(call generate-docker-build-target,$(BACKEND_WHISPERX)))
-$(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
-$(eval $(call generate-docker-build-target,$(BACKEND_MLX_DISTRIBUTED)))
 
-# Pattern rule for docker-save targets
+docker-build-kitten-tts:
-docker-save-%: backend-images
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:kitten-tts -f backend/Dockerfile.python --build-arg BACKEND=kitten-tts ./backend
-	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed
+docker-save-kitten-tts: backend-images
+	docker save local-ai-backend:kitten-tts -o backend-images/kitten-tts.tar
 
-########################################################
+docker-save-chatterbox: backend-images
-### Mock Backend for E2E Tests
+	docker save local-ai-backend:chatterbox -o backend-images/chatterbox.tar
-########################################################
 
-build-mock-backend: protogen-go
+docker-build-neutts:
-	$(GOCMD) build -o tests/e2e/mock-backend/mock-backend ./tests/e2e/mock-backend
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:neutts -f backend/Dockerfile.python --build-arg BACKEND=neutts ./backend
 
-clean-mock-backend:
+docker-save-neutts: backend-images
-	rm -f tests/e2e/mock-backend/mock-backend
+	docker save local-ai-backend:neutts -o backend-images/neutts.tar
+
+docker-build-kokoro:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:kokoro -f backend/Dockerfile.python --build-arg BACKEND=kokoro ./backend
+
+docker-save-kokoro: backend-images
+	docker save local-ai-backend:kokoro -o backend-images/kokoro.tar
+
+docker-save-rfdetr: backend-images
+	docker save local-ai-backend:rfdetr -o backend-images/rfdetr.tar
+
+docker-save-huggingface: backend-images
+	docker save local-ai-backend:huggingface -o backend-images/huggingface.tar
+
+docker-save-local-store: backend-images
+	docker save local-ai-backend:local-store -o backend-images/local-store.tar
+
+docker-build-silero-vad:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:silero-vad -f backend/Dockerfile.golang --build-arg BACKEND=silero-vad .
+
+docker-save-silero-vad: backend-images
+	docker save local-ai-backend:silero-vad -o backend-images/silero-vad.tar
+
+docker-save-piper: backend-images
+	docker save local-ai-backend:piper -o backend-images/piper.tar
+
+docker-save-llama-cpp: backend-images
+	docker save local-ai-backend:llama-cpp -o backend-images/llama-cpp.tar
+
+docker-save-bark-cpp: backend-images
+	docker save local-ai-backend:bark-cpp -o backend-images/bark-cpp.tar
+
+docker-build-stablediffusion-ggml:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:stablediffusion-ggml -f backend/Dockerfile.golang --build-arg BACKEND=stablediffusion-ggml .
+
+docker-save-stablediffusion-ggml: backend-images
+	docker save local-ai-backend:stablediffusion-ggml -o backend-images/stablediffusion-ggml.tar
+
+docker-build-rerankers:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:rerankers -f backend/Dockerfile.python --build-arg BACKEND=rerankers .
+
+docker-build-vllm:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:vllm -f backend/Dockerfile.python --build-arg BACKEND=vllm .
+
+docker-build-transformers:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:transformers -f backend/Dockerfile.python --build-arg BACKEND=transformers .
+
+docker-build-diffusers:
+	docker build --progress=plain --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:diffusers -f backend/Dockerfile.python --build-arg BACKEND=diffusers ./backend
+
+docker-save-diffusers: backend-images
+	docker save local-ai-backend:diffusers -o backend-images/diffusers.tar
+
+docker-build-whisper:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:whisper -f backend/Dockerfile.golang --build-arg BACKEND=whisper  .
+
+docker-save-whisper: backend-images
+	docker save local-ai-backend:whisper -o backend-images/whisper.tar
+
+docker-build-faster-whisper:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:faster-whisper -f backend/Dockerfile.python --build-arg BACKEND=faster-whisper .
+
+docker-build-coqui:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:coqui -f backend/Dockerfile.python --build-arg BACKEND=coqui .
+
+docker-build-bark:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:bark -f backend/Dockerfile.python --build-arg BACKEND=bark .
+
+docker-build-chatterbox:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:chatterbox -f backend/Dockerfile.python --build-arg BACKEND=chatterbox ./backend
+
+docker-build-exllama2:
+	docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:exllama2 -f backend/Dockerfile.python --build-arg BACKEND=exllama2 .
+
+docker-build-backends: docker-build-llama-cpp docker-build-rerankers docker-build-vllm docker-build-transformers docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-bark docker-build-chatterbox docker-build-exllama2
 
 ########################################################
 ### END Backends
@@ -654,7 +523,6 @@ clean-mock-backend:
 swagger:
 	swag init -g core/http/app.go --output swagger
 
-# DEPRECATED: gen-assets is for the legacy Alpine.js UI. Remove when legacy UI is removed.
 .PHONY: gen-assets
 gen-assets:
 	$(GOCMD) run core/dependencies_manager/manager.go webui_static.yaml core/http/static/assets

README.md

@@ -19,12 +19,6 @@
 </a>
 </p>
 
-<p align="center">
-<a href="LICENSE" target="blank">
-<img src="https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge" alt="LocalAI License"/>
-</a>
-</p>
-
 <p align="center">
 <a href="https://hub.docker.com/r/localai/localai" target="blank">
 <img src="https://img.shields.io/badge/dockerhub-images-important.svg?logo=Docker" alt="LocalAI Docker hub"/>
@@ -39,7 +33,7 @@
 <img src="https://img.shields.io/badge/X-%23000000.svg?style=for-the-badge&logo=X&logoColor=white&label=LocalAI_API" alt="Follow LocalAI_API"/>
 </a>
 <a href="https://discord.gg/uJAeKSAGDy" target="blank">
-<img src="https://img.shields.io/badge/dynamic/json?color=blue&label=Discord&style=for-the-badge&query=approximate_member_count&url=https%3A%2F%2Fdiscordapp.com%2Fapi%2Finvites%2FuJAeKSAGDy%3Fwith_counts%3Dtrue&logo=discord" alt="Join LocalAI Discord Community"/>
+<img src="https://dcbadge.vercel.app/api/server/uJAeKSAGDy?style=flat-square&theme=default-inverted" alt="Join LocalAI Discord Community"/>
 </a>
 </p>
 
@@ -49,92 +43,72 @@
 
 > :bulb: Get help - [❓FAQ](https://localai.io/faq/) [💭Discussions](https://github.com/go-skynet/LocalAI/discussions) [:speech_balloon: Discord](https://discord.gg/uJAeKSAGDy) [:book: Documentation website](https://localai.io/)
 >
-> [💻 Quickstart](https://localai.io/basics/getting_started/) [🖼️ Models](https://models.localai.io/) [🚀 Roadmap](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap) [🛫 Examples](https://github.com/mudler/LocalAI-examples) Try on 
+> [💻 Quickstart](https://localai.io/basics/getting_started/) [🖼️ Models](https://models.localai.io/) [🚀 Roadmap](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap) [🌍 Explorer](https://explorer.localai.io) [🛫 Examples](https://github.com/mudler/LocalAI-examples) Try on 
 [![Telegram](https://img.shields.io/badge/Telegram-2CA5E0?style=for-the-badge&logo=telegram&logoColor=white)](https://t.me/localaiofficial_bot)
 
 [![tests](https://github.com/go-skynet/LocalAI/actions/workflows/test.yml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/test.yml)[![Build and Release](https://github.com/go-skynet/LocalAI/actions/workflows/release.yaml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/release.yaml)[![build container images](https://github.com/go-skynet/LocalAI/actions/workflows/image.yml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/image.yml)[![Bump dependencies](https://github.com/go-skynet/LocalAI/actions/workflows/bump_deps.yaml/badge.svg)](https://github.com/go-skynet/LocalAI/actions/workflows/bump_deps.yaml)[![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/localai)](https://artifacthub.io/packages/search?repo=localai)
 
-<p align="center">
-<a href="https://github.com/mudler/LocalAI-examples" target="blank">
-<img src="https://img.shields.io/badge/📦_Examples_Repository-Browse_Ready--to--Run_Examples-blue?style=for-the-badge" alt="LocalAI Examples Repository"/>
-</a>
-</p>
-
 **LocalAI** is the free, Open Source OpenAI alternative. LocalAI act as a drop-in replacement REST API that's compatible with OpenAI (Elevenlabs, Anthropic... ) API specifications for local AI inferencing. It allows you to run LLMs, generate images, audio (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families. Does not require GPU. It is created and maintained by [Ettore Di Giacinto](https://github.com/mudler).
 
-<details>
-<summary><strong>Table of Contents</strong></summary>
 
-- [Local Stack Family](#local-stack-family)
+## 📚🆕 Local Stack Family
-- [Screenshots / Video](#screenshots--video)
-- [Quickstart](#-quickstart)
-  - [macOS Download](#macos-download)
-  - [Containers (Docker, podman, ...)](#containers-docker-podman-)
-- [Latest project news](#-latest-project-news)
-- [Features](#-features)
-- [Supported Backends & Acceleration](#-supported-backends--acceleration)
-  - [Text Generation & Language Models](#text-generation--language-models)
-  - [Audio & Speech Processing](#audio--speech-processing)
-  - [Image & Video Generation](#image--video-generation)
-  - [Specialized AI Tasks](#specialized-ai-tasks)
-  - [Hardware Acceleration Matrix](#hardware-acceleration-matrix)
-  - [Community and integrations](#-community-and-integrations)
-  - [Resources](#-resources)
-- [Media, Blogs, Social](#book--media-blogs-social)
-- [Autonomous Development Team](#-autonomous-development-team)
-- [Citation](#citation)
-- [Sponsors](#️-sponsors)
-  - [Individual sponsors](#individual-sponsors)
-- [Star history](#-star-history)
-- [License](#-license)
-- [Acknowledgements](#-acknowledgements)
-- [Contributors](#-contributors)
 
-</details>
+🆕 LocalAI is now part of a comprehensive suite of AI tools designed to work together:
 
-## Local Stack Family
+<table>
+  <tr>
+    <td width="50%" valign="top">
+      <a href="https://github.com/mudler/LocalAGI">
+        <img src="https://raw.githubusercontent.com/mudler/LocalAGI/refs/heads/main/webui/react-ui/public/logo_2.png" width="300" alt="LocalAGI Logo">
+      </a>
+    </td>
+    <td width="50%" valign="top">
+      <h3><a href="https://github.com/mudler/LocalAGI">LocalAGI</a></h3>
+      <p>A powerful Local AI agent management platform that serves as a drop-in replacement for OpenAI's Responses API, enhanced with advanced agentic capabilities.</p>
+    </td>
+  </tr>
+  <tr>
+    <td width="50%" valign="top">
+      <a href="https://github.com/mudler/LocalRecall">
+        <img src="https://raw.githubusercontent.com/mudler/LocalRecall/refs/heads/main/static/localrecall_horizontal.png" width="300" alt="LocalRecall Logo">
+      </a>
+    </td>
+    <td width="50%" valign="top">
+      <h3><a href="https://github.com/mudler/LocalRecall">LocalRecall</a></h3>
+      <p>A REST-ful API and knowledge base management system that provides persistent memory and storage capabilities for AI agents.</p>
+    </td>
+  </tr>
+</table>
 
-Liking LocalAI? LocalAI is part of an integrated suite of AI infrastructure tools, you might also like:
+## Screenshots
 
-- **[LocalAGI](https://github.com/mudler/LocalAGI)** - AI agent orchestration platform with OpenAI Responses API compatibility and advanced agentic capabilities
-- **[LocalRecall](https://github.com/mudler/LocalRecall)** - MCP/REST API knowledge base system providing persistent memory and storage for AI agents
-- 🆕 **[Cogito](https://github.com/mudler/cogito)** - Go library for building intelligent, co-operative agentic software and LLM-powered workflows, focusing on improving results for small, open source language models that scales to any LLM. Powers LocalAGI and LocalAI MCP/Agentic capabilities
-- 🆕 **[Wiz](https://github.com/mudler/wiz)** - Terminal-based AI agent accessible via Ctrl+Space keybinding. Portable, local-LLM friendly shell assistant with TUI/CLI modes, tool execution with approval, MCP protocol support, and multi-shell compatibility (zsh, bash, fish)
-- 🆕 **[SkillServer](https://github.com/mudler/skillserver)** - Simple, centralized skills database for AI agents via MCP. Manages skills as Markdown files with MCP server integration, web UI for editing, Git synchronization, and full-text search capabilities
-
-
-## Screenshots / Video
-
-### Youtube video
-
-<h1 align="center">
-  <br>
-  <a href="https://www.youtube.com/watch?v=PDqYhB9nNHA" target="_blank"> <img width="300" src="https://img.youtube.com/vi/PDqYhB9nNHA/0.jpg"> </a><br>
-<br>
-</h1>
-
-
-### Screenshots
 
 | Talk Interface | Generate Audio |
 | --- | --- |
-| ![LocalAI - Talk](./docs/assets/images/screenshots/screenshot_talk.png) | ![LocalAI - Generate Audio](./docs/assets/images/screenshots/screenshot_tts.png) |
+| ![Screenshot 2025-03-31 at 12-01-36 LocalAI - Talk](./docs/assets/images/screenshots/screenshot_tts.png) | ![Screenshot 2025-03-31 at 12-01-29 LocalAI - Generate audio with voice-en-us-ryan-low](./docs/assets/images/screenshots/screenshot_tts.png) |
 
 | Models Overview | Generate Images |
 | --- | --- |
-| ![LocalAI - Models](./docs/assets/images/screenshots/screenshot_gallery.png) | ![LocalAI - Generate Images](./docs/assets/images/screenshots/screenshot_image.png) |
+| ![Screenshot 2025-03-31 at 12-01-20 LocalAI - Models](./docs/assets/images/screenshots/screenshot_gallery.png) | ![Screenshot 2025-03-31 at 12-31-41 LocalAI - Generate images with flux 1-dev](./docs/assets/images/screenshots/screenshot_image.png) |
 
 | Chat Interface | Home |
 | --- | --- |
-| ![LocalAI - Chat](./docs/assets/images/screenshots/screenshot_chat.png) | ![LocalAI - Home](./docs/assets/images/screenshots/screenshot_home.png) |
+| ![Screenshot 2025-03-31 at 11-57-44 LocalAI - Chat with localai-functioncall-qwen2 5-7b-v0 5](./docs/assets/images/screenshots/screenshot_chat.png) | ![Screenshot 2025-03-31 at 11-57-23 LocalAI API - c2a39e3 (c2a39e3639227cfd94ffffe9f5691239acc275a8)](./docs/assets/images/screenshots/screenshot_home.png) |
 
 | Login | Swarm |
 | --- | --- |
-| ![LocalAI - Login](./docs/assets/images/screenshots/screenshot_login.png) | ![LocalAI - P2P Dashboard](./docs/assets/images/screenshots/screenshot_p2p.png) |
+|![Screenshot 2025-03-31 at 12-09-59 ](./docs/assets/images/screenshots/screenshot_login.png) | ![Screenshot 2025-03-31 at 12-10-39 LocalAI - P2P dashboard](./docs/assets/images/screenshots/screenshot_p2p.png) |
 
 ## 💻 Quickstart
 
+Run the installer script:
 
+```bash
+# Basic installation
+curl https://localai.io/install.sh | sh
+```
+
+For more installation options, see [Installer Options](https://localai.io/docs/advanced/installer/).
 
 ### macOS Download:
 
@@ -142,9 +116,7 @@ Liking LocalAI? LocalAI is part of an integrated suite of AI infrastructure tool
   <img src="https://img.shields.io/badge/Download-macOS-blue?style=for-the-badge&logo=apple&logoColor=white" alt="Download LocalAI for macOS"/>
 </a>
 
-> Note: the DMGs are not signed by Apple as quarantined. See https://github.com/mudler/LocalAI/issues/6268 for a workaround, fix is tracked here: https://github.com/mudler/LocalAI/issues/6244
+Or run with docker:
-
-### Containers (Docker, podman, ...)
 
 > **💡 Docker Run vs Docker Start**
 > 
@@ -153,47 +125,64 @@ Liking LocalAI? LocalAI is part of an integrated suite of AI infrastructure tool
 > 
 > If you've already run LocalAI before and want to start it again, use: `docker start -i local-ai`
 
-#### CPU only image:
+### CPU only image:
 
 ```bash
 docker run -ti --name local-ai -p 8080:8080 localai/localai:latest
 ```
 
-#### NVIDIA GPU Images:
+### NVIDIA GPU Images:
 
 ```bash
-# CUDA 13.0
-docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13
-
 # CUDA 12.0
 docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12
 
-# NVIDIA Jetson (L4T) ARM64
+# CUDA 11.7
-# CUDA 12 (for Nvidia AGX Orin and similar platforms)
+docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-11
-docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64
 
-# CUDA 13 (for Nvidia DGX Spark)
+# NVIDIA Jetson (L4T) ARM64
-docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64-cuda-13
+docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-nvidia-l4t-arm64
 ```
 
-#### AMD GPU Images (ROCm):
+### AMD GPU Images (ROCm):
 
 ```bash
 docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/localai:latest-gpu-hipblas
 ```
 
-#### Intel GPU Images (oneAPI):
+### Intel GPU Images (oneAPI):
 
 ```bash
 docker run -ti --name local-ai -p 8080:8080 --device=/dev/dri/card1 --device=/dev/dri/renderD128 localai/localai:latest-gpu-intel
 ```
 
-#### Vulkan GPU Images:
+### Vulkan GPU Images:
 
 ```bash
 docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-vulkan
 ```
 
+### AIO Images (pre-downloaded models):
+
+```bash
+# CPU version
+docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-aio-cpu
+
+# NVIDIA CUDA 12 version
+docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-aio-gpu-nvidia-cuda-12
+
+# NVIDIA CUDA 11 version
+docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-aio-gpu-nvidia-cuda-11
+
+# Intel GPU version
+docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-aio-gpu-intel
+
+# AMD GPU version
+docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/localai:latest-aio-gpu-hipblas
+```
+
+For more information about the AIO images and pre-downloaded models, see [Container Documentation](https://localai.io/basics/container/).
+
 To load models:
 
 ```bash
@@ -211,14 +200,10 @@ local-ai run oci://localai/phi-2:latest
 
 > ⚡ **Automatic Backend Detection**: When you install models from the gallery or YAML files, LocalAI automatically detects your system's GPU capabilities (NVIDIA, AMD, Intel) and downloads the appropriate backend. For advanced configuration options, see [GPU Acceleration](https://localai.io/features/gpu-acceleration/#automatic-backend-detection).
 
-For more information, see [💻 Getting started](https://localai.io/basics/getting_started/index.html), if you are interested in our roadmap items and future enhancements, you can see the [Issues labeled as Roadmap here](https://github.com/mudler/LocalAI/issues?q=is%3Aissue+is%3Aopen+label%3Aroadmap)
+For more information, see [💻 Getting started](https://localai.io/basics/getting_started/index.html)
 
 ## 📰 Latest project news
-- March 2026: [Agent management](https://github.com/mudler/LocalAI/pull/8820), [New React UI](https://github.com/mudler/LocalAI/pull/8772), [WebRTC](https://github.com/mudler/LocalAI/pull/8790),[MLX-distributed via P2P and RDMA](https://github.com/mudler/LocalAI/pull/8801), [MCP Apps, MCP Client-side](https://github.com/mudler/LocalAI/pull/8947)
+
-- February 2026: [Realtime API for audio-to-audio with tool calling](https://github.com/mudler/LocalAI/pull/6245), [ACE-Step 1.5 support](https://github.com/mudler/LocalAI/pull/8396)
-- January 2026: **LocalAI 3.10.0** - Major release with Anthropic API support, Open Responses API for stateful agents, video & image generation suite (LTX-2), unified GPU backends, tool streaming & XML parsing, system-aware backend gallery, crash fixes for AVX-only CPUs and AMD VRAM reporting, request tracing, and new backends: **Moonshine** (ultra-fast transcription), **Pocket-TTS** (lightweight TTS). Vulkan arm64 builds now available. [Release notes](https://github.com/mudler/LocalAI/releases/tag/v3.10.0).
-- December 2025: [Dynamic Memory Resource reclaimer](https://github.com/mudler/LocalAI/pull/7583), [Automatic fitting of models to multiple GPUS(llama.cpp)](https://github.com/mudler/LocalAI/pull/7584), [Added Vibevoice backend](https://github.com/mudler/LocalAI/pull/7494)
-- November 2025: Major improvements to the UX. Among these: [Import models via URL](https://github.com/mudler/LocalAI/pull/7245) and [Multiple chats and history](https://github.com/mudler/LocalAI/pull/7325)
 - October 2025: 🔌 [Model Context Protocol (MCP)](https://localai.io/docs/features/mcp/) support added for agentic capabilities with external tools
 - September 2025: New Launcher application for MacOS and Linux, extended support to many backends for Mac and Nvidia L4T devices. Models: Added MLX-Audio, WAN 2.2. WebUI improvements and Python-based backends now ships portable python environments.
 - August 2025: MLX, MLX-VLM, Diffusers and llama.cpp are now supported on Mac M1/M2/M3+ chips ( with `development` suffix in the gallery ): https://github.com/mudler/LocalAI/pull/6049 https://github.com/mudler/LocalAI/pull/6119 https://github.com/mudler/LocalAI/pull/6121 https://github.com/mudler/LocalAI/pull/6060
@@ -229,7 +214,7 @@ For more information, see [💻 Getting started](https://localai.io/basics/getti
 - May 2025: Important: image name changes [See release](https://github.com/mudler/LocalAI/releases/tag/v2.29.0)
 - Apr 2025: Rebrand, WebUI enhancements
 - Apr 2025: [LocalAGI](https://github.com/mudler/LocalAGI) and [LocalRecall](https://github.com/mudler/LocalRecall) join the LocalAI family stack.
-- Apr 2025: WebUI overhaul
+- Apr 2025: WebUI overhaul, AIO images updates
 - Feb 2025: Backend cleanup, Breaking changes, new backends (kokoro, OutelTTS, faster-whisper), Nvidia L4T images
 - Jan 2025: LocalAI model release: https://huggingface.co/mudler/LocalAI-functioncall-phi-4-v0.3, SANA support in diffusers: https://github.com/mudler/LocalAI/pull/4603
 - Dec 2024: stablediffusion.cpp backend (ggml) added ( https://github.com/mudler/LocalAI/pull/4289 )
@@ -249,10 +234,9 @@ Roadmap items: [List of issues](https://github.com/mudler/LocalAI/issues?q=is%3A
 - 🧩 [Backend Gallery](https://localai.io/backends/): Install/remove backends on the fly, powered by OCI images — fully customizable and API-driven.
 - 📖 [Text generation with GPTs](https://localai.io/features/text-generation/) (`llama.cpp`, `transformers`, `vllm` ... [:book: and more](https://localai.io/model-compatibility/index.html#model-compatibility-table))
 - 🗣 [Text to Audio](https://localai.io/features/text-to-audio/)
-- 🔈 [Audio to Text](https://localai.io/features/audio-to-text/)
+- 🔈 [Audio to Text](https://localai.io/features/audio-to-text/) (Audio transcription with `whisper.cpp`)
 - 🎨 [Image generation](https://localai.io/features/image-generation)
 - 🔥 [OpenAI-alike tools API](https://localai.io/features/openai-functions/) 
-- ⚡ [Realtime API](https://localai.io/features/openai-realtime/) (Speech-to-speech) 
 - 🧠 [Embeddings generation for vector databases](https://localai.io/features/embeddings/)
 - ✍️ [Constrained grammars](https://localai.io/features/constrained_grammars/)
 - 🖼️ [Download Models directly from Huggingface ](https://localai.io/models/)
@@ -261,7 +245,6 @@ Roadmap items: [List of issues](https://github.com/mudler/LocalAI/issues?q=is%3A
 - 📈 [Reranker API](https://localai.io/features/reranker/)
 - 🆕🖧 [P2P Inferencing](https://localai.io/features/distribute/)
 - 🆕🔌 [Model Context Protocol (MCP)](https://localai.io/docs/features/mcp/) - Agentic capabilities with external tools and [LocalAGI's Agentic capabilities](https://github.com/mudler/LocalAGI)
-- 🆕🤖 [Built-in Agents](https://localai.io/features/agents/) - Autonomous AI agents with tool use, knowledge base (RAG), skills, SSE streaming, import/export, and [Agent Hub](https://agenthub.localai.io) — powered by [LocalAGI](https://github.com/mudler/LocalAGI)
 - 🔊 Voice activity detection (Silero-VAD support)
 - 🌍 Integrated WebUI!
 
@@ -272,48 +255,39 @@ LocalAI supports a comprehensive range of AI backends with multiple acceleration
 ### Text Generation & Language Models
 | Backend | Description | Acceleration Support |
 |---------|-------------|---------------------|
-| **llama.cpp** | LLM inference in C/C++ | CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, CPU |
+| **llama.cpp** | LLM inference in C/C++ | CUDA 11/12, ROCm, Intel SYCL, Vulkan, Metal, CPU |
-| **vLLM** | Fast LLM inference with PagedAttention | CUDA 12/13, ROCm, Intel |
+| **vLLM** | Fast LLM inference with PagedAttention | CUDA 12, ROCm, Intel |
-| **transformers** | HuggingFace transformers framework | CUDA 12/13, ROCm, Intel, CPU |
+| **transformers** | HuggingFace transformers framework | CUDA 11/12, ROCm, Intel, CPU |
+| **exllama2** | GPTQ inference library | CUDA 12 |
 | **MLX** | Apple Silicon LLM inference | Metal (M1/M2/M3+) |
 | **MLX-VLM** | Apple Silicon Vision-Language Models | Metal (M1/M2/M3+) |
-| **vLLM Omni** | Multimodal vLLM with vision and audio | CUDA 12/13, ROCm, Intel |
 
 ### Audio & Speech Processing
 | Backend | Description | Acceleration Support |
 |---------|-------------|---------------------|
-| **whisper.cpp** | OpenAI Whisper in C/C++ | CUDA 12/13, ROCm, Intel SYCL, Vulkan, CPU |
+| **whisper.cpp** | OpenAI Whisper in C/C++ | CUDA 12, ROCm, Intel SYCL, Vulkan, CPU |
-| **faster-whisper** | Fast Whisper with CTranslate2 | CUDA 12/13, ROCm, Intel, CPU |
+| **faster-whisper** | Fast Whisper with CTranslate2 | CUDA 12, ROCm, Intel, CPU |
-| **moonshine** | Ultra-fast transcription engine for low-end devices | CUDA 12/13, Metal, CPU |
+| **bark** | Text-to-audio generation | CUDA 12, ROCm, Intel |
-| **coqui** | Advanced TTS with 1100+ languages | CUDA 12/13, ROCm, Intel, CPU |
+| **bark-cpp** | C++ implementation of Bark | CUDA, Metal, CPU |
-| **kokoro** | Lightweight TTS model | CUDA 12/13, ROCm, Intel, CPU |
+| **coqui** | Advanced TTS with 1100+ languages | CUDA 12, ROCm, Intel, CPU |
-| **chatterbox** | Production-grade TTS | CUDA 12/13, CPU |
+| **kokoro** | Lightweight TTS model | CUDA 12, ROCm, Intel, CPU |
+| **chatterbox** | Production-grade TTS | CUDA 11/12, CPU |
 | **piper** | Fast neural TTS system | CPU |
 | **kitten-tts** | Kitten TTS models | CPU |
 | **silero-vad** | Voice Activity Detection | CPU |
-| **neutts** | Text-to-speech with voice cloning | CUDA 12/13, ROCm, CPU |
+| **neutts** | Text-to-speech with voice cloning | CUDA 12, ROCm, CPU |
-| **vibevoice** | Real-time TTS with voice cloning | CUDA 12/13, ROCm, Intel, CPU |
-| **pocket-tts** | Lightweight CPU-based TTS | CUDA 12/13, ROCm, Intel, CPU |
-| **qwen-tts** | High-quality TTS with custom voice, voice design, and voice cloning | CUDA 12/13, ROCm, Intel, CPU |
-| **nemo** | NVIDIA NeMo framework for speech models | CUDA 12/13, ROCm, Intel, CPU |
-| **outetts** | OuteTTS with voice cloning | CUDA 12/13, CPU |
-| **faster-qwen3-tts** | Faster Qwen3 TTS | CUDA 12/13, ROCm, Intel, CPU |
-| **qwen-asr** | Qwen ASR speech recognition | CUDA 12/13, ROCm, Intel, CPU |
-| **voxcpm** | VoxCPM speech understanding | CUDA 12/13, Metal, CPU |
-| **whisperx** | Enhanced Whisper transcription | CUDA 12/13, ROCm, Intel, CPU |
-| **ace-step** | Music generation from text descriptions, lyrics, or audio samples | CUDA 12/13, ROCm, Intel, Metal, CPU |
 
 ### Image & Video Generation
 | Backend | Description | Acceleration Support |
 |---------|-------------|---------------------|
-| **stablediffusion.cpp** | Stable Diffusion in C/C++ | CUDA 12/13, Intel SYCL, Vulkan, CPU |
+| **stablediffusion.cpp** | Stable Diffusion in C/C++ | CUDA 12, Intel SYCL, Vulkan, CPU |
-| **diffusers** | HuggingFace diffusion models | CUDA 12/13, ROCm, Intel, Metal, CPU |
+| **diffusers** | HuggingFace diffusion models | CUDA 11/12, ROCm, Intel, Metal, CPU |
 
 ### Specialized AI Tasks
 | Backend | Description | Acceleration Support |
 |---------|-------------|---------------------|
-| **rfdetr** | Real-time object detection | CUDA 12/13, Intel, CPU |
+| **rfdetr** | Real-time object detection | CUDA 12, Intel, CPU |
-| **rerankers** | Document reranking API | CUDA 12/13, ROCm, Intel, CPU |
+| **rerankers** | Document reranking API | CUDA 11/12, ROCm, Intel, CPU |
 | **local-store** | Vector database | CPU |
 | **huggingface** | HuggingFace API integration | API-based |
 
@@ -321,14 +295,13 @@ LocalAI supports a comprehensive range of AI backends with multiple acceleration
 
 | Acceleration Type | Supported Backends | Hardware Support |
 |-------------------|-------------------|------------------|
+| **NVIDIA CUDA 11** | llama.cpp, whisper, stablediffusion, diffusers, rerankers, bark, chatterbox | Nvidia hardware |
 | **NVIDIA CUDA 12** | All CUDA-compatible backends | Nvidia hardware |
-| **NVIDIA CUDA 13** | All CUDA-compatible backends | Nvidia hardware |
+| **AMD ROCm** | llama.cpp, whisper, vllm, transformers, diffusers, rerankers, coqui, kokoro, bark, neutts | AMD Graphics |
-| **AMD ROCm** | llama.cpp, whisper, vllm, transformers, diffusers, rerankers, coqui, kokoro, neutts, vibevoice, pocket-tts, qwen-tts, ace-step | AMD Graphics |
+| **Intel oneAPI** | llama.cpp, whisper, stablediffusion, vllm, transformers, diffusers, rfdetr, rerankers, exllama2, coqui, kokoro, bark | Intel Arc, Intel iGPUs |
-| **Intel oneAPI** | llama.cpp, whisper, stablediffusion, vllm, transformers, diffusers, rfdetr, rerankers, coqui, kokoro, vibevoice, pocket-tts, qwen-tts, ace-step | Intel Arc, Intel iGPUs |
+| **Apple Metal** | llama.cpp, whisper, diffusers, MLX, MLX-VLM, bark-cpp | Apple M1/M2/M3+ |
-| **Apple Metal** | llama.cpp, whisper, diffusers, MLX, MLX-VLM, moonshine, ace-step | Apple M1/M2/M3+ |
 | **Vulkan** | llama.cpp, whisper, stablediffusion | Cross-platform GPUs |
-| **NVIDIA Jetson (CUDA 12)** | llama.cpp, whisper, stablediffusion, diffusers, rfdetr, ace-step | ARM64 embedded AI (AGX Orin, etc.) |
+| **NVIDIA Jetson** | llama.cpp, whisper, stablediffusion, diffusers, rfdetr | ARM64 embedded AI |
-| **NVIDIA Jetson (CUDA 13)** | llama.cpp, whisper, stablediffusion, diffusers, rfdetr | ARM64 embedded AI (DGX Spark) |
 | **CPU Optimized** | All backends | AVX/AVX2/AVX512, quantization support |
 
 ### 🔗 Community and integrations
@@ -347,10 +320,6 @@ Agentic Libraries:
 MCPs:
 - https://github.com/mudler/MCPs
 
-OS Assistant:
-
-- https://github.com/mudler/Keygeist - Keygeist is an AI-powered keyboard operator that listens for key combinations and responds with AI-generated text typed directly into your Linux box.
-
 Model galleries
 - https://github.com/go-skynet/model-gallery
 
@@ -363,7 +332,7 @@ Other:
 - Langchain: https://python.langchain.com/docs/integrations/providers/localai/
 - Terminal utility https://github.com/djcopley/ShellOracle
 - Local Smart assistant https://github.com/mudler/LocalAGI
-- Home Assistant https://github.com/drndos/hass-openai-custom-conversation / https://github.com/valentinfrlch/ha-llmvision / https://github.com/loryanstrant/HA-LocalAI-Monitor
+- Home Assistant https://github.com/sammcj/homeassistant-localai / https://github.com/drndos/hass-openai-custom-conversation / https://github.com/valentinfrlch/ha-gpt4vision
 - Discord bot https://github.com/mudler/LocalAGI/tree/main/examples/discord
 - Slack bot https://github.com/mudler/LocalAGI/tree/main/examples/slack
 - Shell-Pilot(Interact with LLM using LocalAI models via pure shell scripts on your Linux or MacOS system) https://github.com/reid41/shell-pilot
@@ -385,8 +354,6 @@ Other:
 
 ## :book: 🎥 [Media, Blogs, Social](https://localai.io/basics/news/#media-blogs-social)
 
-- 🆕 [LocalAI Autonomous Dev Team Blog Post](https://mudler.pm/posts/2026/02/28/a-call-to-open-source-maintainers-stop-babysitting-ai-how-i-built-a-100-local-autonomous-dev-team-to-maintain-localai-and-why-you-should-too/)
-
 - [Run Visual studio code with LocalAI (SUSE)](https://www.suse.com/c/running-ai-locally/)
 - 🆕 [Run LocalAI on Jetson Nano Devkit](https://mudler.pm/posts/local-ai-jetson-nano-devkit/)
 - [Run LocalAI on AWS EKS with Pulumi](https://www.pulumi.com/blog/low-code-llm-apps-with-local-ai-flowise-and-pulumi/)
@@ -396,15 +363,6 @@ Other:
 - [Question Answering on Documents locally with LangChain, LocalAI, Chroma, and GPT4All](https://mudler.pm/posts/localai-question-answering/)
 - [Tutorial to use k8sgpt with LocalAI](https://medium.com/@tyler_97636/k8sgpt-localai-unlock-kubernetes-superpowers-for-free-584790de9b65)
 
-
-## 🤖 Autonomous Development Team
-
-LocalAI is now helped being maintained (for small tasks!) by a full team of autonomous AI agents led by an AI Scrum Master! This experiment demonstrates how open source projects can leverage AI agents for sustainable, long-term maintenance.
-
-- **📊 Live Reports**: [Automatically generated reports](http://reports.localai.io)
-- **📋 Project Board**: [Agent task tracking](https://github.com/users/mudler/projects/6)
-- **📝 Blog Post**: [Learn about the autonomous dev team experiment](https://mudler.pm/posts/2026/02/28/a-call-to-open-source-maintainers-stop-babysitting-ai-how-i-built-a-100-local-autonomous-dev-team-to-maintain-localai-and-why-you-should-too/)
-
 ## Citation
 
 If you utilize this repository, data in a downstream project, please consider citing it with:
@@ -436,10 +394,6 @@ A huge thank you to our generous sponsors who support this project covering CI e
   </a>
 </p>
 
-### Individual sponsors
-
-A special thanks to individual sponsors that contributed to the project, a full list is in [Github](https://github.com/sponsors/mudler) and [buymeacoffee](https://buymeacoffee.com/mudler), a special shout out goes to [drikster80](https://github.com/drikster80) for being generous. Thank you everyone!
-
 ## 🌟 Star history
 
 [![LocalAI Star history Chart](https://api.star-history.com/svg?repos=go-skynet/LocalAI&type=Date)](https://star-history.com/#go-skynet/LocalAI&Date)
@@ -461,7 +415,6 @@ LocalAI couldn't have been built without the help of great software already avai
 - https://github.com/EdVince/Stable-Diffusion-NCNN
 - https://github.com/ggerganov/whisper.cpp
 - https://github.com/rhasspy/piper
-- [exo](https://github.com/exo-explore/exo) for the MLX distributed auto-parallel sharding implementation
 
 ## 🤗 Contributors
 

SECURITY.md

@@ -8,24 +8,10 @@ At LocalAI, we take the security of our software seriously. We understand the im
 
 We provide support and updates for certain versions of our software. The following table outlines which versions are currently supported with security updates:
 
-| Version Series | Support Level | Details |
+| Version | Supported          |
-| -------------- | ------------- | ------- |
+| ------- | ------------------ |
-| 3.x | :white_check_mark: Actively supported | Full security updates and bug fixes for the latest minor versions. |
+| > 2.0   | :white_check_mark: |
-| 2.x | :warning: Security fixes only | Critical security patches only, until **December 31, 2025**. |
+| < 2.0   | :x:                |
-| 1.x | :x: End-of-life (EOL) | No longer supported as of **January 1, 2024**. No security fixes will be provided. |
-
-### What each support level means
-
-- **Actively supported (3.x):** Receives all security updates, bug fixes, and new features. Users should stay on the latest 3.x minor release for the best protection.
-- **Security fixes only (2.x):** Receives only critical security patches (e.g., remote code execution, authentication bypass, data exposure). No bug fixes or new features. Support ends December 31, 2025.
-- **End-of-life (1.x):** No updates of any kind. Users on 1.x are strongly encouraged to upgrade immediately, as known vulnerabilities will not be patched.
-
-### Migrating from older versions
-
-If you are running an unsupported or soon-to-be-unsupported version, we recommend upgrading as soon as possible:
-
-- **From 1.x to 3.x:** Version 1.x reached end-of-life on January 1, 2024. Review the [release notes](https://github.com/mudler/LocalAI/releases) for breaking changes across major versions, and upgrade directly to the latest 3.x release.
-- **From 2.x to 3.x:** While 2.x still receives critical security patches until December 31, 2025, we recommend planning your migration to 3.x to benefit from ongoing improvements and full support.
 
 Please ensure that you are using a supported version to receive the latest security updates.
 

aio/cpu/README.md

@@ -0,0 +1,5 @@
+## AIO CPU size
+
+Use this image with CPU-only.
+
+Please keep using only C++ backends so the base image is as small as possible (without CUDA, cuDNN, python, etc).

aio/cpu/embeddings.yaml

@@ -0,0 +1,13 @@
+embeddings: true
+name: text-embedding-ada-002
+backend: llama-cpp
+parameters:
+  model: huggingface://bartowski/granite-embedding-107m-multilingual-GGUF/granite-embedding-107m-multilingual-f16.gguf
+
+usage: |
+    You can test this model with curl like this:
+
+    curl http://localhost:8080/embeddings -X POST -H "Content-Type: application/json" -d '{
+      "input": "Your text string goes here",
+      "model": "text-embedding-ada-002"
+    }'

aio/cpu/image-gen.yaml

@@ -12,3 +12,12 @@ download_files:
 - filename: "stable-diffusion-v1-5-pruned-emaonly-Q4_0.gguf"
   sha256: "b8944e9fe0b69b36ae1b5bb0185b3a7b8ef14347fe0fa9af6c64c4829022261f"
   uri: "huggingface://second-state/stable-diffusion-v1-5-GGUF/stable-diffusion-v1-5-pruned-emaonly-Q4_0.gguf"
+
+usage: |
+        curl http://localhost:8080/v1/images/generations \
+          -H "Content-Type: application/json" \
+          -d '{
+            "prompt": "<positive prompt>|<negative prompt>",
+            "step": 25,
+            "size": "512x512"
+          }'

aio/cpu/rerank.yaml

@@ -0,0 +1,33 @@
+name: jina-reranker-v1-base-en
+reranking: true
+f16: true
+parameters:
+  model: jina-reranker-v1-tiny-en.f16.gguf
+backend: llama-cpp
+download_files:
+  - filename: jina-reranker-v1-tiny-en.f16.gguf
+    sha256: 5f696cf0d0f3d347c4a279eee8270e5918554cdac0ed1f632f2619e4e8341407
+    uri: huggingface://mradermacher/jina-reranker-v1-tiny-en-GGUF/jina-reranker-v1-tiny-en.f16.gguf 
+
+usage: |
+    You can test this model with curl like this:
+
+    curl http://localhost:8080/v1/rerank \
+      -H "Content-Type: application/json" \
+      -d '{
+      "model": "jina-reranker-v1-base-en",
+      "query": "Organic skincare products for sensitive skin",
+      "documents": [
+        "Eco-friendly kitchenware for modern homes",
+        "Biodegradable cleaning supplies for eco-conscious consumers",
+        "Organic cotton baby clothes for sensitive skin",
+        "Natural organic skincare range for sensitive skin",
+        "Tech gadgets for smart homes: 2024 edition",
+        "Sustainable gardening tools and compost solutions",
+        "Sensitive skin-friendly facial cleansers and toners",
+        "Organic food wraps and storage solutions",
+        "All-natural pet food for dogs with allergies",
+        "Yoga mats made from recycled materials"
+      ],
+      "top_n": 3
+    }'

aio/cpu/speech-to-text.yaml

@@ -0,0 +1,18 @@
+name: whisper-1
+backend: whisper
+parameters:
+  model: ggml-whisper-base.bin
+
+usage: |
+    ## example audio file
+    wget --quiet --show-progress -O gb1.ogg https://upload.wikimedia.org/wikipedia/commons/1/1f/George_W_Bush_Columbia_FINAL.ogg
+
+    ## Send the example audio file to the transcriptions endpoint
+    curl http://localhost:8080/v1/audio/transcriptions \
+         -H "Content-Type: multipart/form-data" \
+         -F file="@$PWD/gb1.ogg" -F model="whisper-1"
+
+download_files:
+- filename: "ggml-whisper-base.bin"
+  sha256: "60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe"
+  uri: "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.bin"

aio/cpu/text-to-speech.yaml

@@ -0,0 +1,15 @@
+name: tts-1
+download_files:
+  - filename: voice-en-us-amy-low.tar.gz
+    uri: https://github.com/rhasspy/piper/releases/download/v0.0.2/voice-en-us-amy-low.tar.gz
+backend: piper
+parameters:
+  model: en-us-amy-low.onnx
+
+usage: |
+    To test if this model works as expected, you can use the following curl command:
+
+    curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
+      "model":"voice-en-us-amy-low",
+      "input": "Hi, this is a test."
+    }'

aio/cpu/text-to-text.yaml

@@ -55,4 +55,4 @@ template:
 download_files:
 - filename: Hermes-3-Llama-3.2-3B-Q4_K_M.gguf
   sha256: 2e220a14ba4328fee38cf36c2c068261560f999fadb5725ce5c6d977cb5126b5
-  uri: huggingface://bartowski/Hermes-3-Llama-3.2-3B-GGUF/Hermes-3-Llama-3.2-3B-Q4_K_M.gguf
+  uri: huggingface://bartowski/Hermes-3-Llama-3.2-3B-GGUF/Hermes-3-Llama-3.2-3B-Q4_K_M.gguf

aio/cpu/vad.yaml

@@ -1,8 +1,8 @@
-backend: silero-vad
+backend: silero-vad
-name: silero-vad
+name: silero-vad
-parameters:
+parameters:
-  model: silero-vad.onnx
+  model: silero-vad.onnx
-download_files:
+download_files:
-- filename: silero-vad.onnx
+- filename: silero-vad.onnx
-  uri: https://huggingface.co/onnx-community/silero-vad/resolve/main/onnx/model.onnx
+  uri: https://huggingface.co/onnx-community/silero-vad/resolve/main/onnx/model.onnx
-  sha256: a4a068cd6cf1ea8355b84327595838ca748ec29a25bc91fc82e6c299ccdc5808
+  sha256: a4a068cd6cf1ea8355b84327595838ca748ec29a25bc91fc82e6c299ccdc5808

aio/cpu/vision.yaml

@@ -47,4 +47,4 @@ download_files:
   uri: huggingface://openbmb/MiniCPM-V-4_5-gguf/ggml-model-Q4_K_M.gguf
 - filename: minicpm-v-4_5-mmproj-f16.gguf
   uri: huggingface://openbmb/MiniCPM-V-4_5-gguf/mmproj-model-f16.gguf
-  sha256: 7a7225a32e8d453aaa3d22d8c579b5bf833c253f784cdb05c99c9a76fd616df8
+  sha256: 7a7225a32e8d453aaa3d22d8c579b5bf833c253f784cdb05c99c9a76fd616df8

aio/entrypoint.sh

@@ -0,0 +1,138 @@
+#!/bin/bash
+
+echo "===> LocalAI All-in-One (AIO) container starting..."
+
+GPU_ACCELERATION=false
+GPU_VENDOR=""
+
+function check_intel() {
+    if lspci | grep -E 'VGA|3D' | grep -iq intel; then
+        echo "Intel GPU detected"
+        if [ -d /opt/intel ]; then
+            GPU_ACCELERATION=true
+            GPU_VENDOR=intel
+        else
+            echo "Intel GPU detected, but Intel GPU drivers are not installed. GPU acceleration will not be available."
+        fi
+    fi
+}
+
+function check_nvidia_wsl() {
+    if lspci | grep -E 'VGA|3D' | grep -iq "Microsoft Corporation Device 008e"; then
+        # We make the assumption this WSL2 cars is NVIDIA, then check for nvidia-smi
+        # Make sure the container was run with `--gpus all` as the only required parameter
+        echo "NVIDIA GPU detected via WSL2"
+        # nvidia-smi should be installed in the container
+        if nvidia-smi; then
+            GPU_ACCELERATION=true
+            GPU_VENDOR=nvidia
+        else
+            echo "NVIDIA GPU detected via WSL2, but nvidia-smi is not installed. GPU acceleration will not be available."
+        fi
+    fi
+}
+
+function check_amd() {
+    if lspci | grep -E 'VGA|3D' | grep -iq amd; then
+        echo "AMD GPU detected"
+        # Check if ROCm is installed
+        if [ -d /opt/rocm ]; then
+            GPU_ACCELERATION=true
+            GPU_VENDOR=amd
+        else
+            echo "AMD GPU detected, but ROCm is not installed. GPU acceleration will not be available."
+        fi
+    fi
+}
+
+function check_nvidia() {
+    if lspci | grep -E 'VGA|3D' | grep -iq nvidia; then
+        echo "NVIDIA GPU detected"
+        # nvidia-smi should be installed in the container
+        if nvidia-smi; then
+            GPU_ACCELERATION=true
+            GPU_VENDOR=nvidia
+        else
+            echo "NVIDIA GPU detected, but nvidia-smi is not installed. GPU acceleration will not be available."
+        fi
+    fi
+}
+
+function check_metal() {
+    if system_profiler SPDisplaysDataType | grep -iq 'Metal'; then
+        echo "Apple Metal supported GPU detected"
+        GPU_ACCELERATION=true
+        GPU_VENDOR=apple
+    fi
+}
+
+function detect_gpu() {
+    case "$(uname -s)" in
+        Linux)
+            check_nvidia
+            check_amd
+            check_intel
+            check_nvidia_wsl
+            ;;
+        Darwin)
+            check_metal
+            ;;
+    esac
+}
+
+function detect_gpu_size() {
+    # Attempting to find GPU memory size for NVIDIA GPUs
+    if [ "$GPU_ACCELERATION" = true ] && [ "$GPU_VENDOR" = "nvidia" ]; then
+        echo "NVIDIA GPU detected. Attempting to find memory size..."
+        # Using head -n 1 to get the total memory of the 1st NVIDIA GPU detected.
+        # If handling multiple GPUs is required in the future, this is the place to do it
+        nvidia_sm=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | head -n 1)
+        if [ ! -z "$nvidia_sm" ]; then
+            echo "Total GPU Memory: $nvidia_sm MiB"
+            # if bigger than 8GB, use 16GB
+            #if [ "$nvidia_sm" -gt 8192 ]; then
+            #    GPU_SIZE=gpu-16g
+            #else
+            GPU_SIZE=gpu-8g
+            #fi
+        else
+            echo "Unable to determine NVIDIA GPU memory size. Falling back to CPU."
+            GPU_SIZE=gpu-8g
+        fi
+    elif [ "$GPU_ACCELERATION" = true ] && [ "$GPU_VENDOR" = "intel" ]; then
+        GPU_SIZE=intel
+    # Default to a generic GPU size until we implement GPU size detection for non NVIDIA GPUs
+    elif [ "$GPU_ACCELERATION" = true ]; then
+        echo "Non-NVIDIA GPU detected. Specific GPU memory size detection is not implemented."
+        GPU_SIZE=gpu-8g
+
+    # default to cpu if GPU_SIZE is not set
+    else
+        echo "GPU acceleration is not enabled or supported. Defaulting to CPU."
+        GPU_SIZE=cpu
+    fi
+}
+
+function check_vars() {
+    if [ -z "$MODELS" ]; then
+        echo "MODELS environment variable is not set. Please set it to a comma-separated list of model YAML files to load."
+        exit 1
+    fi
+
+    if [ -z "$PROFILE" ]; then
+        echo "PROFILE environment variable is not set. Please set it to one of the following: cpu, gpu-8g, gpu-16g, apple"
+        exit 1
+    fi
+}
+
+detect_gpu
+detect_gpu_size
+
+PROFILE="${PROFILE:-$GPU_SIZE}" # default to cpu
+export MODELS="${MODELS:-/aio/${PROFILE}/embeddings.yaml,/aio/${PROFILE}/rerank.yaml,/aio/${PROFILE}/text-to-speech.yaml,/aio/${PROFILE}/image-gen.yaml,/aio/${PROFILE}/text-to-text.yaml,/aio/${PROFILE}/speech-to-text.yaml,/aio/${PROFILE}/vad.yaml,/aio/${PROFILE}/vision.yaml}"
+
+check_vars
+
+echo "===> Starting LocalAI[$PROFILE] with the following models: $MODELS"
+
+exec /entrypoint.sh "$@"

aio/gpu-8g/embeddings.yaml

@@ -0,0 +1,13 @@
+embeddings: true
+name: text-embedding-ada-002
+backend: llama-cpp
+parameters:
+  model: huggingface://bartowski/granite-embedding-107m-multilingual-GGUF/granite-embedding-107m-multilingual-f16.gguf
+
+usage: |
+    You can test this model with curl like this:
+
+    curl http://localhost:8080/embeddings -X POST -H "Content-Type: application/json" -d '{
+      "input": "Your text string goes here",
+      "model": "text-embedding-ada-002"
+    }'

aio/gpu-8g/image-gen.yaml

@@ -0,0 +1,25 @@
+name: stablediffusion
+parameters:
+  model: DreamShaper_8_pruned.safetensors
+backend: diffusers
+step: 25
+f16: true
+
+diffusers:
+  pipeline_type: StableDiffusionPipeline
+  cuda: true
+  enable_parameters: "negative_prompt,num_inference_steps"
+  scheduler_type: "k_dpmpp_2m"
+
+download_files:
+- filename: DreamShaper_8_pruned.safetensors
+  uri: huggingface://Lykon/DreamShaper/DreamShaper_8_pruned.safetensors
+
+usage: |
+        curl http://localhost:8080/v1/images/generations \
+          -H "Content-Type: application/json" \
+          -d '{
+            "prompt": "<positive prompt>|<negative prompt>",
+            "step": 25,
+            "size": "512x512"
+          }'

aio/gpu-8g/rerank.yaml

@@ -0,0 +1,33 @@
+name: jina-reranker-v1-base-en
+reranking: true
+f16: true
+parameters:
+  model: jina-reranker-v1-tiny-en.f16.gguf
+backend: llama-cpp
+download_files:
+  - filename: jina-reranker-v1-tiny-en.f16.gguf
+    sha256: 5f696cf0d0f3d347c4a279eee8270e5918554cdac0ed1f632f2619e4e8341407
+    uri: huggingface://mradermacher/jina-reranker-v1-tiny-en-GGUF/jina-reranker-v1-tiny-en.f16.gguf 
+
+usage: |
+    You can test this model with curl like this:
+
+    curl http://localhost:8080/v1/rerank \
+      -H "Content-Type: application/json" \
+      -d '{
+      "model": "jina-reranker-v1-base-en",
+      "query": "Organic skincare products for sensitive skin",
+      "documents": [
+        "Eco-friendly kitchenware for modern homes",
+        "Biodegradable cleaning supplies for eco-conscious consumers",
+        "Organic cotton baby clothes for sensitive skin",
+        "Natural organic skincare range for sensitive skin",
+        "Tech gadgets for smart homes: 2024 edition",
+        "Sustainable gardening tools and compost solutions",
+        "Sensitive skin-friendly facial cleansers and toners",
+        "Organic food wraps and storage solutions",
+        "All-natural pet food for dogs with allergies",
+        "Yoga mats made from recycled materials"
+      ],
+      "top_n": 3
+    }'

aio/gpu-8g/speech-to-text.yaml

@@ -0,0 +1,18 @@
+name: whisper-1
+backend: whisper
+parameters:
+  model: ggml-whisper-base.bin
+
+usage: |
+    ## example audio file
+    wget --quiet --show-progress -O gb1.ogg https://upload.wikimedia.org/wikipedia/commons/1/1f/George_W_Bush_Columbia_FINAL.ogg
+
+    ## Send the example audio file to the transcriptions endpoint
+    curl http://localhost:8080/v1/audio/transcriptions \
+         -H "Content-Type: multipart/form-data" \
+         -F file="@$PWD/gb1.ogg" -F model="whisper-1"
+
+download_files:
+- filename: "ggml-whisper-base.bin"
+  sha256: "60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe"
+  uri: "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.bin"

aio/gpu-8g/text-to-speech.yaml

@@ -0,0 +1,15 @@
+name: tts-1
+download_files:
+  - filename: voice-en-us-amy-low.tar.gz
+    uri: https://github.com/rhasspy/piper/releases/download/v0.0.2/voice-en-us-amy-low.tar.gz
+backend: piper
+parameters:
+  model: en-us-amy-low.onnx
+
+usage: |
+    To test if this model works as expected, you can use the following curl command:
+
+    curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
+      "model":"tts-1",
+      "input": "Hi, this is a test."
+    }'

aio/gpu-8g/text-to-text.yaml

@@ -0,0 +1,54 @@
+context_size: 4096
+f16: true
+backend: llama-cpp
+function:
+  capture_llm_results:
+  - (?s)<Thought>(.*?)</Thought>
+  grammar:
+    properties_order: name,arguments
+  json_regex_match:
+  - (?s)<Output>(.*?)</Output>
+  replace_llm_results:
+  - key: (?s)<Thought>(.*?)</Thought>
+    value: ""
+mmap: true
+name: gpt-4
+parameters:
+  model: localai-functioncall-qwen2.5-7b-v0.5-q4_k_m.gguf
+stopwords:
+- <|im_end|>
+- <dummy32000>
+- </s>
+template:
+  chat: |
+    {{.Input -}}
+    <|im_start|>assistant
+  chat_message: |
+    <|im_start|>{{ .RoleName }}
+    {{ if .FunctionCall -}}
+    Function call:
+    {{ else if eq .RoleName "tool" -}}
+    Function response:
+    {{ end -}}
+    {{ if .Content -}}
+    {{.Content }}
+    {{ end -}}
+    {{ if .FunctionCall -}}
+    {{toJson .FunctionCall}}
+    {{ end -}}<|im_end|>
+  completion: |
+    {{.Input}}
+  function: |
+    <|im_start|>system
+    You are an AI assistant that executes function calls, and these are the tools at your disposal:
+    {{range .Functions}}
+    {'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
+    {{end}}
+    <|im_end|>
+    {{.Input -}}
+    <|im_start|>assistant
+
+download_files:
+- filename: localai-functioncall-qwen2.5-7b-v0.5-q4_k_m.gguf
+  sha256: 4e7b7fe1d54b881f1ef90799219dc6cc285d29db24f559c8998d1addb35713d4
+  uri: huggingface://mudler/LocalAI-functioncall-qwen2.5-7b-v0.5-Q4_K_M-GGUF/localai-functioncall-qwen2.5-7b-v0.5-q4_k_m.gguf

aio/gpu-8g/vad.yaml

@@ -0,0 +1,8 @@
+backend: silero-vad
+name: silero-vad
+parameters:
+  model: silero-vad.onnx
+download_files:
+- filename: silero-vad.onnx
+  uri: https://huggingface.co/onnx-community/silero-vad/resolve/main/onnx/model.onnx
+  sha256: a4a068cd6cf1ea8355b84327595838ca748ec29a25bc91fc82e6c299ccdc5808

aio/gpu-8g/vision.yaml

@@ -0,0 +1,50 @@
+context_size: 4096
+backend: llama-cpp
+f16: true
+mmap: true
+mmproj: minicpm-v-4_5-mmproj-f16.gguf
+name: gpt-4o
+parameters:
+  model: minicpm-v-4_5-Q4_K_M.gguf
+stopwords:
+- <|im_end|>
+- <dummy32000>
+- </s>
+- <|endoftext|>
+template:
+  chat: |
+    {{.Input -}}
+    <|im_start|>assistant
+  chat_message: |
+    <|im_start|>{{ .RoleName }}
+    {{ if .FunctionCall -}}
+    Function call:
+    {{ else if eq .RoleName "tool" -}}
+    Function response:
+    {{ end -}}
+    {{ if .Content -}}
+    {{.Content }}
+    {{ end -}}
+    {{ if .FunctionCall -}}
+    {{toJson .FunctionCall}}
+    {{ end -}}<|im_end|>
+  completion: |
+    {{.Input}}
+  function: |
+    <|im_start|>system
+    You are a function calling AI model. You are provided with functions to execute. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
+    {{range .Functions}}
+    {'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
+    {{end}}
+    For each function call return a json object with function name and arguments
+    <|im_end|>
+    {{.Input -}}
+    <|im_start|>assistant
+
+download_files:
+- filename: minicpm-v-4_5-Q4_K_M.gguf
+  sha256: c1c3c33100b15b4caf7319acce4e23c0eb0ce1cbd12f70e8d24f05aa67b7512f
+  uri: huggingface://openbmb/MiniCPM-V-4_5-gguf/ggml-model-Q4_K_M.gguf
+- filename: minicpm-v-4_5-mmproj-f16.gguf
+  uri: huggingface://openbmb/MiniCPM-V-4_5-gguf/mmproj-model-f16.gguf
+  sha256: 7a7225a32e8d453aaa3d22d8c579b5bf833c253f784cdb05c99c9a76fd616df8

aio/intel/embeddings.yaml

@@ -0,0 +1,13 @@
+embeddings: true
+name: text-embedding-ada-002
+backend: llama-cpp
+parameters:
+  model: huggingface://bartowski/granite-embedding-107m-multilingual-GGUF/granite-embedding-107m-multilingual-f16.gguf
+
+usage: |
+    You can test this model with curl like this:
+
+    curl http://localhost:8080/embeddings -X POST -H "Content-Type: application/json" -d '{
+      "input": "Your text string goes here",
+      "model": "text-embedding-ada-002"
+    }'

aio/intel/image-gen.yaml

@@ -0,0 +1,20 @@
+name: stablediffusion
+parameters:
+  model: Lykon/dreamshaper-8
+backend: diffusers
+step: 25
+f16: true
+diffusers:
+  pipeline_type: StableDiffusionPipeline
+  cuda: true
+  enable_parameters: "negative_prompt,num_inference_steps"
+  scheduler_type: "k_dpmpp_2m"
+
+usage: |
+        curl http://localhost:8080/v1/images/generations \
+          -H "Content-Type: application/json" \
+          -d '{
+            "prompt": "<positive prompt>|<negative prompt>",
+            "step": 25,
+            "size": "512x512"
+          }'

aio/intel/rerank.yaml

@@ -0,0 +1,33 @@
+name: jina-reranker-v1-base-en
+reranking: true
+f16: true
+parameters:
+  model: jina-reranker-v1-tiny-en.f16.gguf
+backend: llama-cpp
+download_files:
+  - filename: jina-reranker-v1-tiny-en.f16.gguf
+    sha256: 5f696cf0d0f3d347c4a279eee8270e5918554cdac0ed1f632f2619e4e8341407
+    uri: huggingface://mradermacher/jina-reranker-v1-tiny-en-GGUF/jina-reranker-v1-tiny-en.f16.gguf 
+
+usage: |
+    You can test this model with curl like this:
+
+    curl http://localhost:8080/v1/rerank \
+      -H "Content-Type: application/json" \
+      -d '{
+      "model": "jina-reranker-v1-base-en",
+      "query": "Organic skincare products for sensitive skin",
+      "documents": [
+        "Eco-friendly kitchenware for modern homes",
+        "Biodegradable cleaning supplies for eco-conscious consumers",
+        "Organic cotton baby clothes for sensitive skin",
+        "Natural organic skincare range for sensitive skin",
+        "Tech gadgets for smart homes: 2024 edition",
+        "Sustainable gardening tools and compost solutions",
+        "Sensitive skin-friendly facial cleansers and toners",
+        "Organic food wraps and storage solutions",
+        "All-natural pet food for dogs with allergies",
+        "Yoga mats made from recycled materials"
+      ],
+      "top_n": 3
+    }'

aio/intel/speech-to-text.yaml

@@ -0,0 +1,18 @@
+name: whisper-1
+backend: whisper
+parameters:
+  model: ggml-whisper-base.bin
+
+usage: |
+    ## example audio file
+    wget --quiet --show-progress -O gb1.ogg https://upload.wikimedia.org/wikipedia/commons/1/1f/George_W_Bush_Columbia_FINAL.ogg
+
+    ## Send the example audio file to the transcriptions endpoint
+    curl http://localhost:8080/v1/audio/transcriptions \
+         -H "Content-Type: multipart/form-data" \
+         -F file="@$PWD/gb1.ogg" -F model="whisper-1"
+
+download_files:
+- filename: "ggml-whisper-base.bin"
+  sha256: "60ed5bc3dd14eea856493d334349b405782ddcaf0028d4b5df4088345fba2efe"
+  uri: "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.bin"

aio/intel/text-to-speech.yaml

@@ -0,0 +1,15 @@
+name: tts-1
+download_files:
+  - filename: voice-en-us-amy-low.tar.gz
+    uri: https://github.com/rhasspy/piper/releases/download/v0.0.2/voice-en-us-amy-low.tar.gz
+backend: piper
+parameters:
+  model: en-us-amy-low.onnx
+
+usage: |
+    To test if this model works as expected, you can use the following curl command:
+
+    curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
+      "model":"tts-1",
+      "input": "Hi, this is a test."
+    }'

aio/intel/text-to-text.yaml

@@ -0,0 +1,54 @@
+context_size: 4096
+f16: true
+backend: llama-cpp
+function:
+  capture_llm_results:
+  - (?s)<Thought>(.*?)</Thought>
+  grammar:
+    properties_order: name,arguments
+  json_regex_match:
+  - (?s)<Output>(.*?)</Output>
+  replace_llm_results:
+  - key: (?s)<Thought>(.*?)</Thought>
+    value: ""
+mmap: true
+name: gpt-4
+parameters:
+  model: localai-functioncall-qwen2.5-7b-v0.5-q4_k_m.gguf
+stopwords:
+- <|im_end|>
+- <dummy32000>
+- </s>
+template:
+  chat: |
+    {{.Input -}}
+    <|im_start|>assistant
+  chat_message: |
+    <|im_start|>{{ .RoleName }}
+    {{ if .FunctionCall -}}
+    Function call:
+    {{ else if eq .RoleName "tool" -}}
+    Function response:
+    {{ end -}}
+    {{ if .Content -}}
+    {{.Content }}
+    {{ end -}}
+    {{ if .FunctionCall -}}
+    {{toJson .FunctionCall}}
+    {{ end -}}<|im_end|>
+  completion: |
+    {{.Input}}
+  function: |
+    <|im_start|>system
+    You are an AI assistant that executes function calls, and these are the tools at your disposal:
+    {{range .Functions}}
+    {'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
+    {{end}}
+    <|im_end|>
+    {{.Input -}}
+    <|im_start|>assistant
+
+download_files:
+- filename: localai-functioncall-phi-4-v0.3-q4_k_m.gguf
+  sha256: 23fee048ded2a6e2e1a7b6bbefa6cbf83068f194caa9552aecbaa00fec8a16d5
+  uri: huggingface://mudler/LocalAI-functioncall-phi-4-v0.3-Q4_K_M-GGUF/localai-functioncall-phi-4-v0.3-q4_k_m.gguf

aio/intel/vad.yaml

@@ -0,0 +1,8 @@
+backend: silero-vad
+name: silero-vad
+parameters:
+  model: silero-vad.onnx
+download_files:
+- filename: silero-vad.onnx
+  uri: https://huggingface.co/onnx-community/silero-vad/resolve/main/onnx/model.onnx
+  sha256: a4a068cd6cf1ea8355b84327595838ca748ec29a25bc91fc82e6c299ccdc5808

aio/intel/vision.yaml

@@ -0,0 +1,51 @@
+context_size: 4096
+backend: llama-cpp
+f16: true
+mmap: true
+mmproj: minicpm-v-4_5-mmproj-f16.gguf
+name: gpt-4o
+parameters:
+  model: minicpm-v-4_5-Q4_K_M.gguf
+stopwords:
+- <|im_end|>
+- <dummy32000>
+- </s>
+- <|endoftext|>
+template:
+  chat: |
+    {{.Input -}}
+    <|im_start|>assistant
+  chat_message: |
+    <|im_start|>{{ .RoleName }}
+    {{ if .FunctionCall -}}
+    Function call:
+    {{ else if eq .RoleName "tool" -}}
+    Function response:
+    {{ end -}}
+    {{ if .Content -}}
+    {{.Content }}
+    {{ end -}}
+    {{ if .FunctionCall -}}
+    {{toJson .FunctionCall}}
+    {{ end -}}<|im_end|>
+  completion: |
+    {{.Input}}
+  function: |
+    <|im_start|>system
+    You are a function calling AI model. You are provided with functions to execute. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
+    {{range .Functions}}
+    {'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
+    {{end}}
+    For each function call return a json object with function name and arguments
+    <|im_end|>
+    {{.Input -}}
+    <|im_start|>assistant
+
+
+download_files:
+- filename: minicpm-v-4_5-Q4_K_M.gguf
+  sha256: c1c3c33100b15b4caf7319acce4e23c0eb0ce1cbd12f70e8d24f05aa67b7512f
+  uri: huggingface://openbmb/MiniCPM-V-4_5-gguf/ggml-model-Q4_K_M.gguf
+- filename: minicpm-v-4_5-mmproj-f16.gguf
+  uri: huggingface://openbmb/MiniCPM-V-4_5-gguf/mmproj-model-f16.gguf
+  sha256: 7a7225a32e8d453aaa3d22d8c579b5bf833c253f784cdb05c99c9a76fd616df8

backend/Dockerfile.golang

@@ -1,4 +1,4 @@
-ARG BASE_IMAGE=ubuntu:24.04
+ARG BASE_IMAGE=ubuntu:22.04
 
 FROM ${BASE_IMAGE} AS builder
 ARG BACKEND=rerankers
@@ -12,15 +12,14 @@ ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
 ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
-ARG GO_VERSION=1.25.4
+ARG GO_VERSION=1.22.6
-ARG UBUNTU_VERSION=2404
 
 RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         git ccache \
         ca-certificates \
-        make cmake wget libopenblas-dev \
+        make cmake \
         curl unzip \
         libssl-dev && \
     apt-get clean && \
@@ -33,52 +32,17 @@ ENV PATH=/usr/local/cuda/bin:${PATH}
 # HipBLAS requirements
 ENV PATH=/opt/rocm/bin:${PATH}
 
-
 # Vulkan requirements
 RUN <<EOT bash
     if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y  --no-install-recommends \
             software-properties-common pciutils wget gpg-agent && \
-        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
+        wget -qO - https://packages.lunarg.com/lunarg-signing-key-pub.asc | apt-key add - && \
-            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
+        wget -qO /etc/apt/sources.list.d/lunarg-vulkan-jammy.list https://packages.lunarg.com/vulkan/lunarg-vulkan-jammy.list && \
-            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
+        apt-get update && \
-            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
+        apt-get install -y \
-            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
+            vulkan-sdk && \
-            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
-        if [ "amd64" = "$TARGETARCH" ]; then
-            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
-            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            mkdir -p /opt/vulkan-sdk && \
-            mv 1.4.335.0 /opt/vulkan-sdk/ && \
-            cd /opt/vulkan-sdk/1.4.335.0 && \
-            ./vulkansdk --no-deps --maxjobs \
-                vulkan-loader \
-                vulkan-validationlayers \
-                vulkan-extensionlayer \
-                vulkan-tools \
-                shaderc && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
-            rm -rf /opt/vulkan-sdk
-        fi
-        if [ "arm64" = "$TARGETARCH" ]; then
-            mkdir vulkan && cd vulkan && \
-            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
-            tar -xvf vulkan-sdk.tar.xz && \
-            rm vulkan-sdk.tar.xz && \
-            cd 1.4.335.0 && \
-            cp -rfv aarch64/bin/* /usr/bin/ && \
-            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
-            cp -rfv aarch64/include/* /usr/include/ && \
-            cp -rfv aarch64/share/* /usr/share/ && \
-            cd ../.. && \
-            rm -rf vulkan
-        fi
-        ldconfig && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/*
     fi
@@ -86,19 +50,15 @@ EOT
 
 # CuBLAS requirements
 RUN <<EOT bash
-    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
+    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y  --no-install-recommends \
             software-properties-common pciutils
         if [ "amd64" = "$TARGETARCH" ]; then
-            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
         fi
         if [ "arm64" = "$TARGETARCH" ]; then
-            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/arm64/cuda-keyring_1.1-1_all.deb
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
-            else
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
-            fi
         fi
         dpkg -i cuda-keyring_1.1-1_all.deb && \
         rm -f cuda-keyring_1.1-1_all.deb && \
@@ -109,31 +69,12 @@ RUN <<EOT bash
             libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} && \
-        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
-            apt-get install -y --no-install-recommends \
-            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
-        fi
         apt-get clean && \
         rm -rf /var/lib/apt/lists/*
     fi
 EOT
 
-
-# https://github.com/NVIDIA/Isaac-GR00T/issues/343
-RUN <<EOT bash
-    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
-        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
-        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get install -y nvpl
-    fi
-EOT
-
 # If we are building with clblas support, we need the libraries for the builds
 RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
         apt-get update && \
@@ -180,15 +121,8 @@ RUN <<EOT bash
     fi
 EOT
 
-RUN if [ "${BACKEND}" = "opus" ]; then \
-    apt-get update && apt-get install -y --no-install-recommends libopus-dev pkg-config && \
-    apt-get clean && rm -rf /var/lib/apt/lists/*; \
-fi
-
 COPY . /LocalAI
 
-RUN git config --global --add safe.directory /LocalAI
-
 RUN cd /LocalAI && make protogen-go && make -C /LocalAI/backend/go/${BACKEND} build
 
 FROM scratch

backend/Dockerfile.llama-cpp

@@ -1,4 +1,4 @@
-ARG BASE_IMAGE=ubuntu:24.04
+ARG BASE_IMAGE=ubuntu:22.04
 ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
 
 
@@ -10,8 +10,7 @@ FROM ${GRPC_BASE_IMAGE} AS grpc
 ARG GRPC_MAKEFLAGS="-j4 -Otarget"
 ARG GRPC_VERSION=v1.65.0
 ARG CMAKE_FROM_SOURCE=false
-# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
+ARG CMAKE_VERSION=3.26.4
-ARG CMAKE_VERSION=3.31.10
 
 ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
 
@@ -21,13 +20,13 @@ RUN apt-get update && \
     apt-get install -y --no-install-recommends \
         ca-certificates \
         build-essential curl libssl-dev \
-        git wget && \
+        git && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
 # Install CMake (the version in 22.04 is too old)
 RUN <<EOT bash
-    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+    if [ "${CMAKE_FROM_SOURCE}}" = "true" ]; then
         curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
     else
         apt-get update && \
@@ -51,13 +50,6 @@ RUN git clone --recurse-submodules --jobs 4 -b ${GRPC_VERSION} --depth 1 --shall
     rm -rf /build
 
 FROM ${BASE_IMAGE} AS builder
-ARG CMAKE_FROM_SOURCE=false
-ARG CMAKE_VERSION=3.31.10
-# We can target specific CUDA ARCHITECTURES like --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
-ARG CUDA_DOCKER_ARCH
-ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
-ARG CMAKE_ARGS
-ENV CMAKE_ARGS=${CMAKE_ARGS}
 ARG BACKEND=rerankers
 ARG BUILD_TYPE
 ENV BUILD_TYPE=${BUILD_TYPE}
@@ -69,8 +61,7 @@ ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
 ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
-ARG GO_VERSION=1.25.4
+ARG GO_VERSION=1.22.6
-ARG UBUNTU_VERSION=2404
 
 RUN apt-get update && \
     apt-get install -y --no-install-recommends \
@@ -78,9 +69,8 @@ RUN apt-get update && \
         ccache git \
         ca-certificates \
         make \
-        pkg-config libcurl4-openssl-dev \
         curl unzip \
-        libssl-dev wget && \
+        libssl-dev && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
@@ -90,52 +80,17 @@ ENV PATH=/usr/local/cuda/bin:${PATH}
 # HipBLAS requirements
 ENV PATH=/opt/rocm/bin:${PATH}
 
-
 # Vulkan requirements
 RUN <<EOT bash
     if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y  --no-install-recommends \
             software-properties-common pciutils wget gpg-agent && \
-        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
+        wget -qO - https://packages.lunarg.com/lunarg-signing-key-pub.asc | apt-key add - && \
-            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
+        wget -qO /etc/apt/sources.list.d/lunarg-vulkan-jammy.list https://packages.lunarg.com/vulkan/lunarg-vulkan-jammy.list && \
-            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
+        apt-get update && \
-            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
+        apt-get install -y \
-            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
+            vulkan-sdk && \
-            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
-        if [ "amd64" = "$TARGETARCH" ]; then
-            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
-            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            mkdir -p /opt/vulkan-sdk && \
-            mv 1.4.335.0 /opt/vulkan-sdk/ && \
-            cd /opt/vulkan-sdk/1.4.335.0 && \
-            ./vulkansdk --no-deps --maxjobs \
-                vulkan-loader \
-                vulkan-validationlayers \
-                vulkan-extensionlayer \
-                vulkan-tools \
-                shaderc && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
-            rm -rf /opt/vulkan-sdk
-        fi
-        if [ "arm64" = "$TARGETARCH" ]; then
-            mkdir vulkan && cd vulkan && \
-            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
-            tar -xvf vulkan-sdk.tar.xz && \
-            rm vulkan-sdk.tar.xz && \
-            cd 1.4.335.0 && \
-            cp -rfv aarch64/bin/* /usr/bin/ && \
-            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
-            cp -rfv aarch64/include/* /usr/include/ && \
-            cp -rfv aarch64/share/* /usr/share/ && \
-            cd ../.. && \
-            rm -rf vulkan
-        fi
-        ldconfig && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/*
     fi
@@ -143,19 +98,15 @@ EOT
 
 # CuBLAS requirements
 RUN <<EOT bash
-    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
+    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y  --no-install-recommends \
             software-properties-common pciutils
         if [ "amd64" = "$TARGETARCH" ]; then
-            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
         fi
         if [ "arm64" = "$TARGETARCH" ]; then
-            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/arm64/cuda-keyring_1.1-1_all.deb
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
-            else
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
-            fi
         fi
         dpkg -i cuda-keyring_1.1-1_all.deb && \
         rm -f cuda-keyring_1.1-1_all.deb && \
@@ -166,31 +117,12 @@ RUN <<EOT bash
             libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} && \
-        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
-            apt-get install -y --no-install-recommends \
-            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
-        fi
         apt-get clean && \
         rm -rf /var/lib/apt/lists/*
     fi
 EOT
 
-
-# https://github.com/NVIDIA/Isaac-GR00T/issues/343
-RUN <<EOT bash
-    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
-        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
-        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get install -y nvpl
-    fi
-EOT
-
 # If we are building with clblas support, we need the libraries for the builds
 RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
         apt-get update && \
@@ -232,7 +164,7 @@ EOT
 
 # Install CMake (the version in 22.04 is too old)
 RUN <<EOT bash
-    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+    if [ "${CMAKE_FROM_SOURCE}}" = "true" ]; then
         curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
     else
         apt-get update && \
@@ -248,30 +180,19 @@ COPY --from=grpc /opt/grpc /usr/local
 
 COPY . /LocalAI
 
-RUN <<'EOT' bash
+## Otherwise just run the normal build
-set -euxo pipefail
+RUN <<EOT bash
-
+if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then \
-if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
+        cd /LocalAI/backend/cpp/llama-cpp && make llama-cpp-fallback && \
-  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
+        make llama-cpp-grpc && make llama-cpp-rpc-server; \
-  export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
+    else \
-  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
+        cd /LocalAI/backend/cpp/llama-cpp && make llama-cpp-avx && \
-  rm -rf /LocalAI/backend/cpp/llama-cpp-*-build
+        make llama-cpp-avx2 && \
-fi
+        make llama-cpp-avx512 && \
-
+        make llama-cpp-fallback && \
-if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
+        make llama-cpp-grpc && \
-  cd /LocalAI/backend/cpp/llama-cpp
+        make llama-cpp-rpc-server; \
-  make llama-cpp-fallback
+    fi  
-  make llama-cpp-grpc
-  make llama-cpp-rpc-server
-else
-  cd /LocalAI/backend/cpp/llama-cpp
-  make llama-cpp-avx
-  make llama-cpp-avx2
-  make llama-cpp-avx512
-  make llama-cpp-fallback
-  make llama-cpp-grpc
-  make llama-cpp-rpc-server
-fi
 EOT
 
 

backend/Dockerfile.python

@@ -1,4 +1,4 @@
-ARG BASE_IMAGE=ubuntu:24.04
+ARG BASE_IMAGE=ubuntu:22.04
 
 FROM ${BASE_IMAGE} AS builder
 ARG BACKEND=rerankers
@@ -12,7 +12,6 @@ ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
 ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
-ARG UBUNTU_VERSION=2404
 
 RUN apt-get update && \
     apt-get install -y --no-install-recommends \
@@ -22,7 +21,7 @@ RUN apt-get update && \
         espeak-ng \
         curl \
         libssl-dev \
-        git wget \
+        git \
         git-lfs \
         unzip clang \
         upx-ucl \
@@ -31,15 +30,8 @@ RUN apt-get update && \
         python3-dev llvm \
         python3-venv make cmake && \
     apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
+    rm -rf /var/lib/apt/lists/* && \
-
+    pip install --upgrade pip
-RUN <<EOT bash
-    if [ "${UBUNTU_VERSION}" = "2404" ]; then
-        pip install --break-system-packages --user --upgrade pip
-    else
-        pip install --upgrade pip
-    fi
-EOT
 
 
 # Cuda
@@ -54,45 +46,11 @@ RUN <<EOT bash
         apt-get update && \
         apt-get install -y  --no-install-recommends \
             software-properties-common pciutils wget gpg-agent && \
-        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
+        wget -qO - https://packages.lunarg.com/lunarg-signing-key-pub.asc | apt-key add - && \
-            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
+        wget -qO /etc/apt/sources.list.d/lunarg-vulkan-jammy.list https://packages.lunarg.com/vulkan/lunarg-vulkan-jammy.list && \
-            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
+        apt-get update && \
-            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
+        apt-get install -y \
-            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
+            vulkan-sdk && \
-            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
-        if [ "amd64" = "$TARGETARCH" ]; then
-            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
-            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
-            mkdir -p /opt/vulkan-sdk && \
-            mv 1.4.335.0 /opt/vulkan-sdk/ && \
-            cd /opt/vulkan-sdk/1.4.335.0 && \
-            ./vulkansdk --no-deps --maxjobs \
-                vulkan-loader \
-                vulkan-validationlayers \
-                vulkan-extensionlayer \
-                vulkan-tools \
-                shaderc && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
-            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
-            rm -rf /opt/vulkan-sdk
-        fi
-        if [ "arm64" = "$TARGETARCH" ]; then
-            mkdir vulkan && cd vulkan && \
-            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
-            tar -xvf vulkan-sdk.tar.xz && \
-            rm vulkan-sdk.tar.xz && \
-            cd 1.4.335.0 && \
-            cp -rfv aarch64/bin/* /usr/bin/ && \
-            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
-            cp -rfv aarch64/include/* /usr/include/ && \
-            cp -rfv aarch64/share/* /usr/share/ && \
-            cd ../.. && \
-            rm -rf vulkan
-        fi
-        ldconfig && \
         apt-get clean && \
         rm -rf /var/lib/apt/lists/*
     fi
@@ -100,19 +58,15 @@ EOT
 
 # CuBLAS requirements
 RUN <<EOT bash
-    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
+    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         apt-get update && \
         apt-get install -y  --no-install-recommends \
             software-properties-common pciutils
         if [ "amd64" = "$TARGETARCH" ]; then
-            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
         fi
         if [ "arm64" = "$TARGETARCH" ]; then
-            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/arm64/cuda-keyring_1.1-1_all.deb
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
-            else
-                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
-            fi
         fi
         dpkg -i cuda-keyring_1.1-1_all.deb && \
         rm -f cuda-keyring_1.1-1_all.deb && \
@@ -123,31 +77,12 @@ RUN <<EOT bash
             libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
             libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
-            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} && \
-        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
-            apt-get install -y --no-install-recommends \
-            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
-        fi
         apt-get clean && \
         rm -rf /var/lib/apt/lists/*
     fi
 EOT
 
-
-# https://github.com/NVIDIA/Isaac-GR00T/issues/343
-RUN <<EOT bash
-    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
-        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
-        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
-        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
-        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
-        apt-get update && apt-get install -y nvpl
-    fi
-EOT
-
 # If we are building with clblas support, we need the libraries for the builds
 RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
         apt-get update && \
@@ -168,45 +103,21 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
         # to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
         ldconfig \
     ; fi
-
-RUN if [ "${BUILD_TYPE}" = "hipblas" ]; then \
-    ln -s /opt/rocm-**/lib/llvm/lib/libomp.so /usr/lib/libomp.so \
-    ; fi
-
 # Install uv as a system package
 RUN curl -LsSf https://astral.sh/uv/install.sh | UV_INSTALL_DIR=/usr/bin sh
 ENV PATH="/root/.cargo/bin:${PATH}"
-# Increase timeout for uv installs behind slow networks
+
-ENV UV_HTTP_TIMEOUT=180
 RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
 
 # Install grpcio-tools (the version in 22.04 is too old)
-RUN <<EOT bash
+RUN pip install --user grpcio-tools==1.71.0 grpcio==1.71.0
-    if [ "${UBUNTU_VERSION}" = "2404" ]; then
-        pip install --break-system-packages --user grpcio-tools==1.71.0 grpcio==1.71.0
-    else
-        pip install grpcio-tools==1.71.0 grpcio==1.71.0
-    fi
-EOT
 
-
+COPY python/${BACKEND} /${BACKEND}
-COPY backend/python/${BACKEND} /${BACKEND}
+COPY backend.proto /${BACKEND}/backend.proto
-COPY backend/backend.proto /${BACKEND}/backend.proto
+COPY python/common/ /${BACKEND}/common
-COPY backend/python/common/ /${BACKEND}/common
-COPY scripts/build/package-gpu-libs.sh /package-gpu-libs.sh
 
 RUN cd /${BACKEND} && PORTABLE_PYTHON=true make
 
-# Package GPU libraries into the backend's lib directory
-RUN mkdir -p /${BACKEND}/lib && \
-    TARGET_LIB_DIR="/${BACKEND}/lib" BUILD_TYPE="${BUILD_TYPE}" CUDA_MAJOR_VERSION="${CUDA_MAJOR_VERSION}" \
-    bash /package-gpu-libs.sh "/${BACKEND}/lib"
-
-# Run backend-specific packaging if a package.sh exists
-RUN if [ -f "/${BACKEND}/package.sh" ]; then \
-        cd /${BACKEND} && bash package.sh; \
-    fi
-
 FROM scratch
 ARG BACKEND=rerankers
 COPY --from=builder /${BACKEND}/ /

backend/README.md

@@ -46,14 +46,16 @@ The backend system provides language-specific Dockerfiles that handle the build
 - **vllm**: High-performance LLM inference
 - **mlx**: Apple Silicon optimization
 - **diffusers**: Stable Diffusion models
-- **Audio**: coqui, faster-whisper, kitten-tts
+- **Audio**: bark, coqui, faster-whisper, kitten-tts
 - **Vision**: mlx-vlm, rfdetr
 - **Specialized**: rerankers, chatterbox, kokoro
 
 #### Go Backends (`go/`)
 - **whisper**: OpenAI Whisper speech recognition in Go with GGML cpp backend (whisper.cpp)
 - **stablediffusion-ggml**: Stable Diffusion in Go with GGML Cpp backend
+- **huggingface**: Hugging Face model integration
 - **piper**: Text-to-speech synthesis Golang with C bindings using rhaspy/piper
+- **bark-cpp**: Bark TTS models Golang with Cpp bindings
 - **local-store**: Vector storage backend
 
 #### C++ Backends (`cpp/`)
@@ -63,7 +65,7 @@ The backend system provides language-specific Dockerfiles that handle the build
 ## Hardware Acceleration Support
 
 ### CUDA (NVIDIA)
-- **Versions**: CUDA 12.x, 13.x
+- **Versions**: CUDA 11.x, 12.x
 - **Features**: cuBLAS, cuDNN, TensorRT optimization
 - **Targets**: x86_64, ARM64 (Jetson)
 
@@ -130,7 +132,8 @@ For ARM64/Mac builds, docker can't be used, and the makefile in the respective b
 ### Build Types
 
 - **`cpu`**: CPU-only optimization
-- **`cublas12`**, **`cublas13`**: CUDA 12.x, 13.x with cuBLAS
+- **`cublas11`**: CUDA 11.x with cuBLAS
+- **`cublas12`**: CUDA 12.x with cuBLAS
 - **`hipblas`**: ROCm with rocBLAS
 - **`intel`**: Intel oneAPI optimization
 - **`vulkan`**: Vulkan-based acceleration
@@ -207,4 +210,4 @@ When contributing to the backend system:
 2. **Add Tests**: Include comprehensive test coverage
 3. **Document**: Provide clear usage examples
 4. **Optimize**: Consider performance and resource usage
-5. **Validate**: Test across different hardware targets
+5. **Validate**: Test across different hardware targets

backend/backend.proto

@@ -9,7 +9,6 @@ package backend;
 
 service Backend {
   rpc Health(HealthMessage) returns (Reply) {}
-  rpc Free(HealthMessage) returns (Result) {}
   rpc Predict(PredictOptions) returns (Reply) {}
   rpc LoadModel(ModelOptions) returns (Result) {}
   rpc PredictStream(PredictOptions) returns (stream Reply) {}
@@ -18,7 +17,6 @@ service Backend {
   rpc GenerateVideo(GenerateVideoRequest) returns (Result) {}
   rpc AudioTranscription(TranscriptRequest) returns (TranscriptResult) {}
   rpc TTS(TTSRequest) returns (Result) {}
-  rpc TTSStream(TTSRequest) returns (stream Reply) {}
   rpc SoundGeneration(SoundGenerationRequest) returns (Result) {}
   rpc TokenizeString(PredictOptions) returns (TokenizationResponse) {}
   rpc Status(HealthMessage) returns (StatusResponse) {}
@@ -34,11 +32,6 @@ service Backend {
   rpc GetMetrics(MetricsRequest) returns (MetricsResponse);
 
   rpc VAD(VADRequest) returns (VADResponse) {}
-
-  rpc AudioEncode(AudioEncodeRequest) returns (AudioEncodeResult) {}
-  rpc AudioDecode(AudioDecodeRequest) returns (AudioDecodeResult) {}
-
-  rpc ModelMetadata(ModelOptions) returns (ModelMetadataResponse) {}
 }
 
 // Define the empty request
@@ -161,27 +154,6 @@ message PredictOptions {
   repeated string Videos = 45;
   repeated string Audios = 46;
   string CorrelationId = 47;
-  string Tools = 48;  // JSON array of available tools/functions for tool calling
-  string ToolChoice = 49;  // JSON string or object specifying tool choice behavior
-  int32 Logprobs = 50;  // Number of top logprobs to return (maps to OpenAI logprobs parameter)
-  int32 TopLogprobs = 51;  // Number of top logprobs to return per token (maps to OpenAI top_logprobs parameter)
-  map<string, string> Metadata = 52;  // Generic per-request metadata (e.g., enable_thinking)
-}
-
-// ToolCallDelta represents an incremental tool call update from the C++ parser.
-// Used for both streaming (partial diffs) and non-streaming (final tool calls).
-message ToolCallDelta {
-  int32 index = 1;           // tool call index (0-based)
-  string id = 2;             // tool call ID (e.g., "call_abc123")
-  string name = 3;           // function name (set on first appearance)
-  string arguments = 4;      // arguments chunk (incremental in streaming, full in non-streaming)
-}
-
-// ChatDelta represents incremental content/reasoning/tool_call updates parsed by the C++ backend.
-message ChatDelta {
-  string content = 1;                       // content text delta
-  string reasoning_content = 2;             // reasoning/thinking text delta
-  repeated ToolCallDelta tool_calls = 3;    // tool call deltas
 }
 
 // The response message containing the result
@@ -192,8 +164,6 @@ message Reply {
   double timing_prompt_processing = 4;
   double timing_token_generation = 5;
   bytes audio = 6;
-  bytes logprobs = 7;  // JSON-encoded logprobs data matching OpenAI format
-  repeated ChatDelta chat_deltas = 8;       // Parsed chat deltas from C++ autoparser (streaming + non-streaming)
 }
 
 message GrammarTrigger {
@@ -307,7 +277,6 @@ message TranscriptRequest {
   uint32 threads = 4;
   bool translate = 5;
   bool diarize = 6;
-  string prompt = 7;
 }
 
 message TranscriptResult {
@@ -321,12 +290,12 @@ message TranscriptSegment {
   int64 end = 3;
   string text = 4;
   repeated int32 tokens = 5;
-  string speaker = 6;
 }
 
 message GenerateImageRequest {
   int32 height = 1;
   int32 width = 2;
+  int32 mode = 3;
   int32 step = 4;
   int32 seed = 5;
   string positive_prompt = 6;
@@ -387,14 +356,6 @@ message SoundGenerationRequest {
   optional bool sample = 6;
   optional string src = 7;
   optional int32 src_divisor = 8;
-  optional bool think = 9;
-  optional string caption = 10;
-  optional string lyrics = 11;
-  optional int32 bpm = 12;
-  optional string keyscale = 13;
-  optional string language = 14;
-  optional string timesignature = 15;
-  optional bool instrumental = 17;
 }
 
 message TokenizationResponse {
@@ -421,11 +382,6 @@ message StatusResponse {
 message Message {
   string role = 1;
   string content = 2;
-  // Optional fields for OpenAI-compatible message format
-  string name = 3;                    // Tool name (for tool messages)
-  string tool_call_id = 4;            // Tool call ID (for tool messages)
-  string reasoning_content = 5;       // Reasoning content (for thinking models)
-  string tool_calls = 6;              // Tool calls as JSON string (for assistant messages with tool calls)
 }
 
 message DetectOptions {
@@ -444,87 +400,3 @@ message Detection {
 message DetectResponse {
   repeated Detection Detections = 1;
 }
-
-message ToolFormatMarkers {
-  string format_type = 1;           // "json_native", "tag_with_json", "tag_with_tagged"
-
-  // Tool section markers
-  string section_start = 2;         // e.g., "<tool_call>", "[TOOL_CALLS]"
-  string section_end = 3;           // e.g., "</tool_call>"
-  string per_call_start = 4;        // e.g., "<|tool_call_begin|>"
-  string per_call_end = 5;          // e.g., "<|tool_call_end|>"
-
-  // Function name markers (TAG_WITH_JSON / TAG_WITH_TAGGED)
-  string func_name_prefix = 6;     // e.g., "<function="
-  string func_name_suffix = 7;     // e.g., ">"
-  string func_close = 8;           // e.g., "</function>"
-
-  // Argument markers (TAG_WITH_TAGGED)
-  string arg_name_prefix = 9;      // e.g., "<param="
-  string arg_name_suffix = 10;     // e.g., ">"
-  string arg_value_prefix = 11;
-  string arg_value_suffix = 12;    // e.g., "</param>"
-  string arg_separator = 13;       // e.g., "\n"
-
-  // JSON format fields (JSON_NATIVE)
-  string name_field = 14;          // e.g., "name"
-  string args_field = 15;          // e.g., "arguments"
-  string id_field = 16;            // e.g., "id"
-  bool fun_name_is_key = 17;
-  bool tools_array_wrapped = 18;
-  bool uses_python_dicts = 19;
-
-  // Reasoning markers
-  string reasoning_start = 20;     // e.g., "<think>"
-  string reasoning_end = 21;       // e.g., "</think>"
-
-  // Content markers
-  string content_start = 22;
-  string content_end = 23;
-
-  // Args wrapper markers
-  string args_start = 24;          // e.g., "<args>"
-  string args_end = 25;            // e.g., "</args>"
-
-  // JSON parameter ordering
-  string function_field = 26;      // e.g., "function" (wrapper key in JSON)
-  repeated string parameter_order = 27;
-
-  // Generated ID field (alternative field name for generated IDs)
-  string gen_id_field = 28;        // e.g., "call_id"
-
-  // Call ID markers (position and delimiters for tool call IDs)
-  string call_id_position = 29;    // "none", "pre_func_name", "between_func_and_args", "post_args"
-  string call_id_prefix = 30;      // e.g., "[CALL_ID]"
-  string call_id_suffix = 31;      // e.g., ""
-}
-
-message AudioEncodeRequest {
-  bytes pcm_data = 1;
-  int32 sample_rate = 2;
-  int32 channels = 3;
-  map<string, string> options = 4;
-}
-
-message AudioEncodeResult {
-  repeated bytes frames = 1;
-  int32 sample_rate = 2;
-  int32 samples_per_frame = 3;
-}
-
-message AudioDecodeRequest {
-  repeated bytes frames = 1;
-  map<string, string> options = 2;
-}
-
-message AudioDecodeResult {
-  bytes pcm_data = 1;
-  int32 sample_rate = 2;
-  int32 samples_per_frame = 3;
-}
-
-message ModelMetadataResponse {
-  bool supports_thinking = 1;
-  string rendered_template = 2;  // The rendered chat template with enable_thinking=true (empty if not applicable)
-  ToolFormatMarkers tool_format = 3;  // Auto-detected tool format markers from differential template analysis
-}

backend/cpp/llama-cpp/CMakeLists.txt

@@ -57,7 +57,7 @@ add_library(hw_grpc_proto
   ${hw_proto_srcs}
   ${hw_proto_hdrs} )
 
-add_executable(${TARGET} grpc-server.cpp json.hpp httplib.h)
+add_executable(${TARGET} grpc-server.cpp utils.hpp json.hpp httplib.h)
 
 target_include_directories(${TARGET} PRIVATE ../llava)
 target_include_directories(${TARGET} PRIVATE ${CMAKE_SOURCE_DIR})
@@ -70,4 +70,4 @@ target_link_libraries(${TARGET} PRIVATE common llama mtmd ${CMAKE_THREAD_LIBS_IN
 target_compile_features(${TARGET} PRIVATE cxx_std_11)
 if(TARGET BUILD_INFO)
   add_dependencies(${TARGET} BUILD_INFO)
-endif()
+endif()

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=e30f1fdf74ea9238ff562901aa974c75aab6619b
+LLAMA_VERSION?=5a4ff43e7dd049e35942bc3d12361dab2f155544
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=
@@ -7,8 +7,7 @@ BUILD_TYPE?=
 NATIVE?=false
 ONEAPI_VARS?=/opt/intel/oneapi/setvars.sh
 TARGET?=--target grpc-server
-JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
+JOBS?=$(shell nproc)
-ARCH?=$(shell uname -m)
 
 # Disable Shared libs as we are linking on static gRPC and we can't mix shared and static
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF -DLLAMA_CURL=OFF
@@ -107,21 +106,21 @@ llama-cpp-avx: llama.cpp
 	cp -rf $(CURRENT_MAKEFILE_DIR)/../llama-cpp $(CURRENT_MAKEFILE_DIR)/../llama-cpp-avx-build
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-avx-build purge
 	$(info ${GREEN}I llama-cpp build info:avx${RESET})
-	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) VARIANT="llama-cpp-avx-build" build-llama-cpp-grpc-server
+	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off" $(MAKE) VARIANT="llama-cpp-avx-build" build-llama-cpp-grpc-server
 	cp -rfv $(CURRENT_MAKEFILE_DIR)/../llama-cpp-avx-build/grpc-server llama-cpp-avx
 
 llama-cpp-fallback: llama.cpp
 	cp -rf $(CURRENT_MAKEFILE_DIR)/../llama-cpp $(CURRENT_MAKEFILE_DIR)/../llama-cpp-fallback-build
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-fallback-build purge
 	$(info ${GREEN}I llama-cpp build info:fallback${RESET})
-	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) VARIANT="llama-cpp-fallback-build" build-llama-cpp-grpc-server
+	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off" $(MAKE) VARIANT="llama-cpp-fallback-build" build-llama-cpp-grpc-server
 	cp -rfv $(CURRENT_MAKEFILE_DIR)/../llama-cpp-fallback-build/grpc-server llama-cpp-fallback
 
 llama-cpp-grpc: llama.cpp
 	cp -rf $(CURRENT_MAKEFILE_DIR)/../llama-cpp $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build purge
 	$(info ${GREEN}I llama-cpp build info:grpc${RESET})
-	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" TARGET="--target grpc-server --target rpc-server" $(MAKE) VARIANT="llama-cpp-grpc-build" build-llama-cpp-grpc-server
+	CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off" TARGET="--target grpc-server --target rpc-server" $(MAKE) VARIANT="llama-cpp-grpc-build" build-llama-cpp-grpc-server
 	cp -rfv $(CURRENT_MAKEFILE_DIR)/../llama-cpp-grpc-build/grpc-server llama-cpp-grpc
 
 llama-cpp-rpc-server: llama-cpp-grpc

backend/cpp/llama-cpp/package.sh

@@ -6,7 +6,6 @@
 set -e
 
 CURDIR=$(dirname "$(realpath $0)")
-REPO_ROOT="${CURDIR}/../../.."
 
 # Create lib directory
 mkdir -p $CURDIR/package/lib
@@ -38,15 +37,6 @@ else
     exit 1
 fi
 
-# Package GPU libraries based on BUILD_TYPE
-# The GPU library packaging script will detect BUILD_TYPE and copy appropriate GPU libraries
-GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
-if [ -f "$GPU_LIB_SCRIPT" ]; then
-    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
-    package_gpu_libs
-fi
-
 echo "Packaging completed successfully" 
 ls -liah $CURDIR/package/
 ls -liah $CURDIR/package/lib/

backend/cpp/llama-cpp/patches/01-llava.patch

@@ -0,0 +1,13 @@
+diff --git a/tools/mtmd/clip.cpp b/tools/mtmd/clip.cpp
+index 3cd0d2fa..6c5e811a 100644
+--- a/tools/mtmd/clip.cpp
++++ b/tools/mtmd/clip.cpp
+@@ -2608,7 +2608,7 @@ bool clip_image_batch_encode(clip_ctx * ctx, const int n_threads, const clip_ima
+                 struct ggml_tensor * patches = ggml_graph_get_tensor(gf, "patches");
+                 int* patches_data = (int*)malloc(ggml_nbytes(patches));
+                 for (int i = 0; i < num_patches; i++) {
+-                    patches_data[i] = i + 1;
++                    patches_data[i] = i;
+                 }
+                 ggml_backend_tensor_set(patches, patches_data, 0, ggml_nbytes(patches));
+                 free(patches_data);

backend/cpp/llama-cpp/prepare.sh

@@ -1,24 +1,18 @@
 #!/bin/bash
 
 ## Patches
-
 ## Apply patches from the `patches` directory
-if [ -d "patches" ]; then
+for patch in $(ls patches); do
-    for patch in $(ls patches); do
+    echo "Applying patch $patch"
-        echo "Applying patch $patch"
+    patch -d llama.cpp/ -p1 < patches/$patch
-        patch -d llama.cpp/ -p1 < patches/$patch
+done 
-    done 
-fi
 
 set -e
 
-for file in $(ls llama.cpp/tools/server/); do
-    cp -rfv llama.cpp/tools/server/$file llama.cpp/tools/grpc-server/
-done
-
 cp -r CMakeLists.txt llama.cpp/tools/grpc-server/
 cp -r grpc-server.cpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/tools/grpc-server/
+cp -rfv llama.cpp/tools/server/utils.hpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/cpp-httplib/httplib.h llama.cpp/tools/grpc-server/
 
 set +e
@@ -29,3 +23,30 @@ else
 fi
 set -e
 
+# Now to keep maximum compatibility with the original server.cpp, we need to remove the index.html.gz.hpp and loading.html.hpp includes
+# and remove the main function
+# TODO: upstream this to the original server.cpp by extracting the upstream main function to a separate file
+awk '
+/int[ \t]+main[ \t]*\(/ {          # If the line starts the main function
+    in_main=1;                     # Set a flag
+    open_braces=0;                 # Track number of open braces
+}
+in_main {
+    open_braces += gsub(/\{/, "{"); # Count opening braces
+    open_braces -= gsub(/\}/, "}"); # Count closing braces
+    if (open_braces == 0) {         # If all braces are closed
+        in_main=0;                  # End skipping
+    }
+    next;                           # Skip lines inside main
+}
+!in_main                           # Print lines not inside main
+' "llama.cpp/tools/server/server.cpp" > llama.cpp/tools/grpc-server/server.cpp
+
+# remove index.html.gz.hpp and loading.html.hpp includes
+if [[ "$OSTYPE" == "darwin"* ]]; then
+    # macOS
+    sed -i '' '/#include "index\.html\.gz\.hpp"/d; /#include "loading\.html\.hpp"/d' llama.cpp/tools/grpc-server/server.cpp
+else
+    # Linux and others
+    sed -i '/#include "index\.html\.gz\.hpp"/d; /#include "loading\.html\.hpp"/d' llama.cpp/tools/grpc-server/server.cpp
+fi

backend/go/acestep-cpp/CMakeLists.txt

@@ -1,54 +0,0 @@
-cmake_minimum_required(VERSION 3.14)
-project(goacestepcpp LANGUAGES C CXX)
-set(CMAKE_POSITION_INDEPENDENT_CODE ON)
-set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
-
-set(ACESTEP_DIR ${CMAKE_CURRENT_SOURCE_DIR}/sources/acestep.cpp)
-
-# Override upstream's CMAKE_CUDA_ARCHITECTURES before add_subdirectory.
-# Upstream sets 120a/121a for CUDA >= 12.8, but those archs require a newer
-# toolkit than 12.8.x ships. Pre-defining this variable makes the upstream
-# "if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)" guard skip its broken defaults.
-if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
-    set(CMAKE_CUDA_ARCHITECTURES "75-virtual;80-virtual;86-real;89-real")
-endif()
-
-# EXCLUDE_FROM_ALL: only build targets we explicitly depend on (acestep-core, ggml),
-# skip upstream standalone executables (ace-understand, dit-vae, etc.)
-add_subdirectory(${ACESTEP_DIR} acestep EXCLUDE_FROM_ALL)
-
-add_library(goacestepcpp MODULE cpp/goacestepcpp.cpp)
-target_link_libraries(goacestepcpp PRIVATE acestep-core ggml ggml-base ggml-cpu)
-
-# Include dirs matching link_ggml_backends macro, but with absolute paths
-target_include_directories(goacestepcpp PRIVATE ${ACESTEP_DIR}/src ${ACESTEP_DIR})
-target_include_directories(goacestepcpp SYSTEM PRIVATE ${ACESTEP_DIR}/ggml/include)
-
-# Link GPU backends if available (mirrors link_ggml_backends macro)
-foreach(backend blas cuda metal vulkan)
-    if(TARGET ggml-${backend})
-        target_link_libraries(goacestepcpp PRIVATE ggml-${backend})
-        string(TOUPPER ${backend} BACKEND_UPPER)
-        target_compile_definitions(goacestepcpp PRIVATE ACESTEP_HAVE_${BACKEND_UPPER})
-        if(backend STREQUAL "cuda")
-            find_package(CUDAToolkit QUIET)
-            if(CUDAToolkit_FOUND)
-                target_link_libraries(goacestepcpp PRIVATE CUDA::cudart)
-            endif()
-        endif()
-    endif()
-endforeach()
-
-if(MSVC)
-    target_compile_options(goacestepcpp PRIVATE /W4 /wd4100 /wd4505)
-else()
-    target_compile_options(goacestepcpp PRIVATE -Wall -Wextra -Wshadow -Wconversion
-                          -Wno-unused-parameter -Wno-unused-function -Wno-sign-conversion)
-endif()
-
-if(CMAKE_CXX_COMPILER_ID MATCHES "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9.0)
-    target_link_libraries(goacestepcpp PRIVATE stdc++fs)
-endif()
-
-set_property(TARGET goacestepcpp PROPERTY CXX_STANDARD 17)
-set_target_properties(goacestepcpp PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})

backend/go/acestep-cpp/Makefile

@@ -1,127 +0,0 @@
-CMAKE_ARGS?=
-BUILD_TYPE?=
-NATIVE?=false
-
-GOCMD?=go
-GO_TAGS?=
-JOBS?=$(shell nproc --ignore=1)
-
-# acestep.cpp version
-ACESTEP_REPO?=https://github.com/ace-step/acestep.cpp
-ACESTEP_CPP_VERSION?=5aa065445541094cba934299cd498bbb9fa5c434
-SO_TARGET?=libgoacestepcpp.so
-
-CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
-
-ifeq ($(NATIVE),false)
-	CMAKE_ARGS+=-DGGML_NATIVE=OFF
-endif
-
-ifeq ($(BUILD_TYPE),cublas)
-	CMAKE_ARGS+=-DGGML_CUDA=ON
-else ifeq ($(BUILD_TYPE),openblas)
-	CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
-else ifeq ($(BUILD_TYPE),clblas)
-	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
-else ifeq ($(BUILD_TYPE),hipblas)
-	CMAKE_ARGS+=-DGGML_HIPBLAS=ON
-else ifeq ($(BUILD_TYPE),vulkan)
-	CMAKE_ARGS+=-DGGML_VULKAN=ON
-else ifeq ($(OS),Darwin)
-	ifneq ($(BUILD_TYPE),metal)
-		CMAKE_ARGS+=-DGGML_METAL=OFF
-	else
-		CMAKE_ARGS+=-DGGML_METAL=ON
-		CMAKE_ARGS+=-DGGML_METAL_EMBED_LIBRARY=ON
-	endif
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f16)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx \
-		-DGGML_SYCL_F16=ON
-endif
-
-ifeq ($(BUILD_TYPE),sycl_f32)
-	CMAKE_ARGS+=-DGGML_SYCL=ON \
-		-DCMAKE_C_COMPILER=icx \
-		-DCMAKE_CXX_COMPILER=icpx
-endif
-
-sources/acestep.cpp:
-	mkdir -p sources/acestep.cpp
-	cd sources/acestep.cpp && \
-	git init && \
-	git remote add origin $(ACESTEP_REPO) && \
-	git fetch origin && \
-	git checkout $(ACESTEP_CPP_VERSION) && \
-	git submodule update --init --recursive --depth 1 --single-branch
-
-# Detect OS
-UNAME_S := $(shell uname -s)
-
-# Only build CPU variants on Linux
-ifeq ($(UNAME_S),Linux)
-	VARIANT_TARGETS = libgoacestepcpp-avx.so libgoacestepcpp-avx2.so libgoacestepcpp-avx512.so libgoacestepcpp-fallback.so
-else
-	# On non-Linux (e.g., Darwin), build only fallback variant
-	VARIANT_TARGETS = libgoacestepcpp-fallback.so
-endif
-
-acestep-cpp: main.go goacestepcpp.go $(VARIANT_TARGETS)
-	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o acestep-cpp ./
-
-package: acestep-cpp
-	bash package.sh
-
-build: package
-
-clean: purge
-	rm -rf libgoacestepcpp*.so package sources/acestep.cpp acestep-cpp
-
-purge:
-	rm -rf build*
-
-# Variants must build sequentially: each uses its own build-<name> directory,
-# but parallel builds can still race on shared resources (jobserver, disk I/O).
-.NOTPARALLEL:
-
-# Build all variants (Linux only)
-ifeq ($(UNAME_S),Linux)
-libgoacestepcpp-avx.so: sources/acestep.cpp
-	$(info ${GREEN}I acestep-cpp build info:avx${RESET})
-	SO_TARGET=libgoacestepcpp-avx.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgoacestepcpp-custom
-	rm -rf build-libgoacestepcpp-avx.so
-
-libgoacestepcpp-avx2.so: sources/acestep.cpp
-	$(info ${GREEN}I acestep-cpp build info:avx2${RESET})
-	SO_TARGET=libgoacestepcpp-avx2.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgoacestepcpp-custom
-	rm -rf build-libgoacestepcpp-avx2.so
-
-libgoacestepcpp-avx512.so: sources/acestep.cpp
-	$(info ${GREEN}I acestep-cpp build info:avx512${RESET})
-	SO_TARGET=libgoacestepcpp-avx512.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on -DGGML_BMI2=on" $(MAKE) libgoacestepcpp-custom
-	rm -rf build-libgoacestepcpp-avx512.so
-endif
-
-# Build fallback variant (all platforms)
-libgoacestepcpp-fallback.so: sources/acestep.cpp
-	$(info ${GREEN}I acestep-cpp build info:fallback${RESET})
-	SO_TARGET=libgoacestepcpp-fallback.so CMAKE_ARGS="$(CMAKE_ARGS) -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off" $(MAKE) libgoacestepcpp-custom
-	rm -rf build-libgoacestepcpp-fallback.so
-
-libgoacestepcpp-custom: CMakeLists.txt cpp/goacestepcpp.cpp cpp/goacestepcpp.h
-	mkdir -p build-$(SO_TARGET) && \
-	cd build-$(SO_TARGET) && \
-	cmake .. $(CMAKE_ARGS) && \
-	cmake --build . --config Release -j$(JOBS) --target goacestepcpp && \
-	cd .. && \
-	mv build-$(SO_TARGET)/libgoacestepcpp.so ./$(SO_TARGET)
-
-test: acestep-cpp
-	@echo "Running acestep-cpp tests..."
-	bash test.sh
-	@echo "acestep-cpp tests completed."
-
-all: acestep-cpp package

backend/go/acestep-cpp/acestepcpp_test.go

@@ -1,202 +0,0 @@
-package main
-
-import (
-	"context"
-	"os"
-	"os/exec"
-	"path/filepath"
-	"testing"
-	"time"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	"google.golang.org/grpc"
-	"google.golang.org/grpc/credentials/insecure"
-)
-
-const (
-	testAddr    = "localhost:50051"
-	startupWait = 5 * time.Second
-)
-
-func skipIfNoModel(t *testing.T) string {
-	t.Helper()
-	modelDir := os.Getenv("ACESTEP_MODEL_DIR")
-	if modelDir == "" {
-		t.Skip("ACESTEP_MODEL_DIR not set, skipping test (set to directory with GGUF models)")
-	}
-	if _, err := os.Stat(filepath.Join(modelDir, "acestep-5Hz-lm-0.6B-Q8_0.gguf")); os.IsNotExist(err) {
-		t.Skipf("LM model file not found in %s, skipping", modelDir)
-	}
-	if _, err := os.Stat(filepath.Join(modelDir, "Qwen3-Embedding-0.6B-Q8_0.gguf")); os.IsNotExist(err) {
-		t.Skipf("Text encoder model file not found in %s, skipping", modelDir)
-	}
-	if _, err := os.Stat(filepath.Join(modelDir, "acestep-v15-turbo-Q8_0.gguf")); os.IsNotExist(err) {
-		t.Skipf("DiT model file not found in %s, skipping", modelDir)
-	}
-	if _, err := os.Stat(filepath.Join(modelDir, "vae-BF16.gguf")); os.IsNotExist(err) {
-		t.Skipf("VAE model file not found in %s, skipping", modelDir)
-	}
-	return modelDir
-}
-
-func startServer(t *testing.T) *exec.Cmd {
-	t.Helper()
-	binary := os.Getenv("ACESTEP_BINARY")
-	if binary == "" {
-		binary = "./acestep-cpp"
-	}
-	if _, err := os.Stat(binary); os.IsNotExist(err) {
-		t.Skipf("Backend binary not found at %s, skipping", binary)
-	}
-	cmd := exec.Command(binary, "--addr", testAddr)
-	cmd.Stdout = os.Stderr
-	cmd.Stderr = os.Stderr
-	if err := cmd.Start(); err != nil {
-		t.Fatalf("Failed to start server: %v", err)
-	}
-	time.Sleep(startupWait)
-	return cmd
-}
-
-func stopServer(cmd *exec.Cmd) {
-	if cmd != nil && cmd.Process != nil {
-		cmd.Process.Kill()
-		cmd.Wait()
-	}
-}
-
-func dialGRPC(t *testing.T) *grpc.ClientConn {
-	t.Helper()
-	conn, err := grpc.Dial(testAddr,
-		grpc.WithTransportCredentials(insecure.NewCredentials()),
-		grpc.WithDefaultCallOptions(
-			grpc.MaxCallRecvMsgSize(50*1024*1024),
-			grpc.MaxCallSendMsgSize(50*1024*1024),
-		),
-	)
-	if err != nil {
-		t.Fatalf("Failed to dial gRPC: %v", err)
-	}
-	return conn
-}
-
-func TestServerHealth(t *testing.T) {
-	cmd := startServer(t)
-	defer stopServer(cmd)
-
-	conn := dialGRPC(t)
-	defer conn.Close()
-
-	client := pb.NewBackendClient(conn)
-	resp, err := client.Health(context.Background(), &pb.HealthMessage{})
-	if err != nil {
-		t.Fatalf("Health check failed: %v", err)
-	}
-	if string(resp.Message) != "OK" {
-		t.Fatalf("Expected OK, got %s", string(resp.Message))
-	}
-}
-
-func TestLoadModel(t *testing.T) {
-	modelDir := skipIfNoModel(t)
-	cmd := startServer(t)
-	defer stopServer(cmd)
-
-	conn := dialGRPC(t)
-	defer conn.Close()
-
-	client := pb.NewBackendClient(conn)
-	
-	// Get base directory from main model file for relative paths
-	mainModelPath := filepath.Join(modelDir, "acestep-5Hz-lm-0.6B-Q8_0.gguf")
-	
-	resp, err := client.LoadModel(context.Background(), &pb.ModelOptions{
-		ModelFile: mainModelPath,
-		Options: []string{
-			"text_encoder_model:Qwen3-Embedding-0.6B-Q8_0.gguf",
-			"dit_model:acestep-v15-turbo-Q8_0.gguf",
-			"vae_model:vae-BF16.gguf",
-		},
-	})
-	if err != nil {
-		t.Fatalf("LoadModel failed: %v", err)
-	}
-	if !resp.Success {
-		t.Fatalf("LoadModel returned failure: %s", resp.Message)
-	}
-}
-
-func TestSoundGeneration(t *testing.T) {
-	modelDir := skipIfNoModel(t)
-
-	tmpDir, err := os.MkdirTemp("", "acestep-test")
-	if err != nil {
-		t.Fatal(err)
-	}
-	defer os.RemoveAll(tmpDir)
-
-	outputFile := filepath.Join(tmpDir, "output.wav")
-
-	cmd := startServer(t)
-	defer stopServer(cmd)
-
-	conn := dialGRPC(t)
-	defer conn.Close()
-
-	client := pb.NewBackendClient(conn)
-
-	// Get base directory from main model file for relative paths
-	mainModelPath := filepath.Join(modelDir, "acestep-5Hz-lm-0.6B-Q8_0.gguf")
-
-	// Load models
-	loadResp, err := client.LoadModel(context.Background(), &pb.ModelOptions{
-		ModelFile: mainModelPath,
-		Options: []string{
-			"text_encoder_model:Qwen3-Embedding-0.6B-Q8_0.gguf",
-			"dit_model:acestep-v15-turbo-Q8_0.gguf",
-			"vae_model:vae-BF16.gguf",
-		},
-	})
-	if err != nil {
-		t.Fatalf("LoadModel failed: %v", err)
-	}
-	if !loadResp.Success {
-		t.Fatalf("LoadModel returned failure: %s", loadResp.Message)
-	}
-
-	// Generate music
-	duration := float32(10.0)
-	temperature := float32(0.85)
-	bpm := int32(120)
-	caption := "A cheerful electronic dance track"
-	timesig := "4/4"
-
-	_, err = client.SoundGeneration(context.Background(), &pb.SoundGenerationRequest{
-		Text:          caption,
-		Caption:       &caption,
-		Dst:           outputFile,
-		Duration:      &duration,
-		Temperature:   &temperature,
-		Bpm:           &bpm,
-		Timesignature: &timesig,
-	})
-	if err != nil {
-		t.Fatalf("SoundGeneration failed: %v", err)
-	}
-
-	// Verify output file exists and has content
-	info, err := os.Stat(outputFile)
-	if os.IsNotExist(err) {
-		t.Fatal("Output audio file was not created")
-	}
-	if err != nil {
-		t.Fatalf("Failed to stat output file: %v", err)
-	}
-
-	t.Logf("Output file size: %d bytes", info.Size())
-
-	// WAV header is 44 bytes minimum; any real audio should be much larger
-	if info.Size() < 1000 {
-		t.Errorf("Output file too small (%d bytes), expected real audio data", info.Size())
-	}
-}

backend/go/acestep-cpp/cpp/goacestepcpp.cpp

@@ -1,306 +0,0 @@
-#include "goacestepcpp.h"
-#include "ggml-backend.h"
-
-#include "audio-io.h"
-#include "bpe.h"
-#include "cond-enc.h"
-#include "dit-sampler.h"
-#include "dit.h"
-#include "gguf-weights.h"
-#include "philox.h"
-#include "qwen3-enc.h"
-#include "qwen3-lm.h"
-#include "request.h"
-#include "vae.h"
-
-#include <cmath>
-#include <cstdio>
-#include <cstdlib>
-#include <cstring>
-#include <random>
-#include <string>
-#include <vector>
-
-// Global model contexts (loaded once, reused across requests)
-static DiTGGML       g_dit       = {};
-static DiTGGMLConfig g_dit_cfg;
-static VAEGGML       g_vae       = {};
-static bool          g_dit_loaded = false;
-static bool          g_vae_loaded = false;
-static bool          g_is_turbo   = false;
-
-// Silence latent [15000, 64] — read once from DiT GGUF
-static std::vector<float> g_silence_full;
-
-// Paths for per-request loading (text encoder, tokenizer)
-static std::string g_text_enc_path;
-static std::string g_dit_path;
-static std::string g_lm_path;
-
-static void ggml_log_cb(enum ggml_log_level level, const char * log, void * data) {
-    const char * level_str;
-    if (!log)
-        return;
-    switch (level) {
-    case GGML_LOG_LEVEL_DEBUG:
-        level_str = "DEBUG";
-        break;
-    case GGML_LOG_LEVEL_INFO:
-        level_str = "INFO";
-        break;
-    case GGML_LOG_LEVEL_WARN:
-        level_str = "WARN";
-        break;
-    case GGML_LOG_LEVEL_ERROR:
-        level_str = "ERROR";
-        break;
-    default:
-        level_str = "?????";
-        break;
-    }
-    fprintf(stderr, "[%-5s] ", level_str);
-    fputs(log, stderr);
-    fflush(stderr);
-}
-
-int load_model(const char * lm_model_path, const char * text_encoder_path,
-               const char * dit_model_path, const char * vae_model_path) {
-    ggml_log_set(ggml_log_cb, nullptr);
-    ggml_backend_load_all();
-
-    g_lm_path       = lm_model_path;
-    g_text_enc_path = text_encoder_path;
-    g_dit_path      = dit_model_path;
-
-    // Load DiT model
-    fprintf(stderr, "[acestep-cpp] Loading DiT from %s\n", dit_model_path);
-    dit_ggml_init_backend(&g_dit);
-    if (!dit_ggml_load(&g_dit, dit_model_path, g_dit_cfg, nullptr, 0.0f)) {
-        fprintf(stderr, "[acestep-cpp] FATAL: failed to load DiT from %s\n", dit_model_path);
-        return 1;
-    }
-    g_dit_loaded = true;
-
-    // Read DiT GGUF metadata + silence_latent
-    {
-        GGUFModel gf = {};
-        if (gf_load(&gf, dit_model_path)) {
-            g_is_turbo           = gf_get_bool(gf, "acestep.is_turbo");
-            const void * sl_data = gf_get_data(gf, "silence_latent");
-            if (sl_data) {
-                g_silence_full.resize(15000 * 64);
-                memcpy(g_silence_full.data(), sl_data, 15000 * 64 * sizeof(float));
-                fprintf(stderr, "[acestep-cpp] silence_latent: [15000, 64] loaded\n");
-            } else {
-                fprintf(stderr, "[acestep-cpp] FATAL: silence_latent not found in %s\n", dit_model_path);
-                gf_close(&gf);
-                return 2;
-            }
-            gf_close(&gf);
-        } else {
-            fprintf(stderr, "[acestep-cpp] FATAL: cannot read GGUF metadata from %s\n", dit_model_path);
-            return 2;
-        }
-    }
-
-    // Load VAE model
-    fprintf(stderr, "[acestep-cpp] Loading VAE from %s\n", vae_model_path);
-    vae_ggml_load(&g_vae, vae_model_path);
-    g_vae_loaded = true;
-
-    fprintf(stderr, "[acestep-cpp] All models loaded successfully (turbo=%d)\n", g_is_turbo);
-    return 0;
-}
-
-int generate_music(const char * caption, const char * lyrics, int bpm,
-                   const char * keyscale, const char * timesignature,
-                   float duration, float temperature, bool instrumental,
-                   int seed, const char * dst, int threads) {
-    if (!g_dit_loaded || !g_vae_loaded) {
-        fprintf(stderr, "[acestep-cpp] ERROR: models not loaded\n");
-        return 1;
-    }
-
-    const int FRAMES_PER_SECOND = 25;
-
-    // Defaults
-    if (duration <= 0)
-        duration = 30.0f;
-    std::string cap_str    = caption ? caption : "";
-    std::string lyrics_str = (instrumental || !lyrics) ? "" : lyrics;
-    std::string ks_str     = keyscale ? keyscale : "N/A";
-    std::string ts_str     = timesignature ? timesignature : "4/4";
-    std::string lang_str   = "unknown";
-    char        bpm_str[16];
-    if (bpm > 0) {
-        snprintf(bpm_str, sizeof(bpm_str), "%d", bpm);
-    } else {
-        snprintf(bpm_str, sizeof(bpm_str), "N/A");
-    }
-
-    int   num_steps      = 8;
-    float guidance_scale = g_is_turbo ? 1.0f : 7.0f;
-    float shift          = 1.0f;
-
-    if (seed < 0) {
-        std::random_device rd;
-        seed = (int)(rd() & 0x7FFFFFFF);
-    }
-
-    // Compute T (latent frames at 25Hz)
-    int T = (int)(duration * FRAMES_PER_SECOND);
-    T     = ((T + g_dit_cfg.patch_size - 1) / g_dit_cfg.patch_size) * g_dit_cfg.patch_size;
-    int S = T / g_dit_cfg.patch_size;
-
-    if (T > 15000) {
-        fprintf(stderr, "[acestep-cpp] ERROR: T=%d exceeds max 15000\n", T);
-        return 2;
-    }
-
-    int Oc     = g_dit_cfg.out_channels;      // 64
-    int ctx_ch = g_dit_cfg.in_channels - Oc;  // 128
-
-    fprintf(stderr, "[acestep-cpp] T=%d, S=%d, duration=%.1fs, seed=%d\n", T, S, duration, seed);
-
-    // 1. Load BPE tokenizer from text encoder GGUF
-    BPETokenizer tok;
-    if (!load_bpe_from_gguf(&tok, g_text_enc_path.c_str())) {
-        fprintf(stderr, "[acestep-cpp] FATAL: failed to load BPE tokenizer\n");
-        return 3;
-    }
-
-    // 2. Build formatted prompts (matches dit-vae.cpp text2music template)
-    std::string instruction = "Fill the audio semantic mask based on the given conditions:";
-
-    char metas[512];
-    snprintf(metas, sizeof(metas),
-             "- bpm: %s\n- timesignature: %s\n- keyscale: %s\n- duration: %d seconds\n",
-             bpm_str, ts_str.c_str(), ks_str.c_str(), (int)duration);
-
-    std::string text_str  = std::string("# Instruction\n") + instruction + "\n\n" +
-                            "# Caption\n" + cap_str + "\n\n" +
-                            "# Metas\n" + metas + "<|endoftext|>\n";
-    std::string lyric_str = std::string("# Languages\n") + lang_str + "\n\n# Lyric\n" +
-                            lyrics_str + "<|endoftext|>";
-
-    // 3. Tokenize
-    auto text_ids  = bpe_encode(&tok, text_str.c_str(), true);
-    auto lyric_ids = bpe_encode(&tok, lyric_str.c_str(), true);
-    int  S_text    = (int)text_ids.size();
-    int  S_lyric   = (int)lyric_ids.size();
-
-    fprintf(stderr, "[acestep-cpp] caption: %d tokens, lyrics: %d tokens\n", S_text, S_lyric);
-
-    // 4. Text encoder forward
-    Qwen3GGML text_enc = {};
-    qwen3_init_backend(&text_enc);
-    if (!qwen3_load_text_encoder(&text_enc, g_text_enc_path.c_str())) {
-        fprintf(stderr, "[acestep-cpp] FATAL: failed to load text encoder\n");
-        return 4;
-    }
-
-    int                H_text = text_enc.cfg.hidden_size;  // 1024
-    std::vector<float> text_hidden(H_text * S_text);
-
-    qwen3_forward(&text_enc, text_ids.data(), S_text, text_hidden.data());
-    fprintf(stderr, "[acestep-cpp] TextEncoder forward done\n");
-
-    // 5. Lyric embedding
-    std::vector<float> lyric_embed(H_text * S_lyric);
-    qwen3_embed_lookup(&text_enc, lyric_ids.data(), S_lyric, lyric_embed.data());
-
-    // 6. Condition encoder
-    CondGGML cond = {};
-    cond_ggml_init_backend(&cond);
-    if (!cond_ggml_load(&cond, g_dit_path.c_str())) {
-        fprintf(stderr, "[acestep-cpp] FATAL: failed to load condition encoder\n");
-        qwen3_free(&text_enc);
-        return 5;
-    }
-
-    const int          S_ref = 750;
-    std::vector<float> silence_feats(S_ref * 64);
-    memcpy(silence_feats.data(), g_silence_full.data(), S_ref * 64 * sizeof(float));
-
-    int                enc_S = 0;
-    std::vector<float> enc_hidden;
-    cond_ggml_forward(&cond, text_hidden.data(), S_text, lyric_embed.data(), S_lyric,
-                      silence_feats.data(), S_ref, enc_hidden, &enc_S);
-    fprintf(stderr, "[acestep-cpp] ConditionEncoder done, enc_S=%d\n", enc_S);
-
-    qwen3_free(&text_enc);
-    cond_ggml_free(&cond);
-
-    // 7. Build context [T, ctx_ch] = silence[64] + mask[64]
-    std::vector<float> context(T * ctx_ch);
-    for (int t = 0; t < T; t++) {
-        const float * src = g_silence_full.data() + t * Oc;
-        for (int c = 0; c < Oc; c++) {
-            context[t * ctx_ch + c] = src[c];
-        }
-        for (int c = 0; c < Oc; c++) {
-            context[t * ctx_ch + Oc + c] = 1.0f;
-        }
-    }
-
-    // 8. Build schedule
-    std::vector<float> schedule(num_steps);
-    for (int i = 0; i < num_steps; i++) {
-        float t     = 1.0f - (float)i / (float)num_steps;
-        schedule[i] = shift * t / (1.0f + (shift - 1.0f) * t);
-    }
-
-    // 9. Generate noise (Philox)
-    std::vector<float> noise(Oc * T);
-    philox_randn((long long)seed, noise.data(), Oc * T, true);
-
-    // 10. DiT generate
-    std::vector<float> output(Oc * T);
-    fprintf(stderr, "[acestep-cpp] DiT generate: T=%d, steps=%d, guidance=%.1f\n", T, num_steps, guidance_scale);
-
-    dit_ggml_generate(&g_dit, noise.data(), context.data(), enc_hidden.data(), enc_S,
-                      T, 1, num_steps, schedule.data(), output.data(), guidance_scale,
-                      nullptr, nullptr, -1);
-    fprintf(stderr, "[acestep-cpp] DiT generation done\n");
-
-    // 11. VAE decode
-    int                T_audio_max = T * 1920;
-    std::vector<float> audio(2 * T_audio_max);
-
-    int T_audio = vae_ggml_decode_tiled(&g_vae, output.data(), T, audio.data(), T_audio_max, 256, 64);
-    if (T_audio < 0) {
-        fprintf(stderr, "[acestep-cpp] ERROR: VAE decode failed\n");
-        return 6;
-    }
-    fprintf(stderr, "[acestep-cpp] VAE decode done: %d samples (%.2fs @ 48kHz)\n", T_audio,
-            (float)T_audio / 48000.0f);
-
-    // 12. Peak normalization to -1.0 dB
-    {
-        float peak      = 0.0f;
-        int   n_samples = 2 * T_audio;
-        for (int i = 0; i < n_samples; i++) {
-            float a = audio[i] < 0 ? -audio[i] : audio[i];
-            if (a > peak) {
-                peak = a;
-            }
-        }
-        if (peak > 1e-6f) {
-            const float target_amp = powf(10.0f, -1.0f / 20.0f);
-            float       gain       = target_amp / peak;
-            for (int i = 0; i < n_samples; i++) {
-                audio[i] *= gain;
-            }
-        }
-    }
-
-    // 13. Write WAV output
-    if (!audio_write_wav(dst, audio.data(), T_audio, 48000)) {
-        fprintf(stderr, "[acestep-cpp] ERROR: failed to write %s\n", dst);
-        return 7;
-    }
-
-    fprintf(stderr, "[acestep-cpp] Wrote %s: %d samples (%.2fs @ 48kHz stereo)\n",
-            dst, T_audio, (float)T_audio / 48000.0f);
-    return 0;
-}

backend/go/acestep-cpp/cpp/goacestepcpp.h

@@ -1,11 +0,0 @@
-#include <cstddef>
-#include <cstdint>
-
-extern "C" {
-int load_model(const char *lm_model_path, const char *text_encoder_path,
-               const char *dit_model_path, const char *vae_model_path);
-int generate_music(const char *caption, const char *lyrics, int bpm,
-                   const char *keyscale, const char *timesignature,
-                   float duration, float temperature, bool instrumental,
-                   int seed, const char *dst, int threads);
-}

backend/go/acestep-cpp/goacestepcpp.go

@@ -1,109 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"os"
-	"path/filepath"
-	"strings"
-
-	"github.com/mudler/LocalAI/pkg/grpc/base"
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-)
-
-var (
-	CppLoadModel    func(lmModelPath, textEncoderPath, ditModelPath, vaeModelPath string) int
-	CppGenerateMusic func(caption, lyrics string, bpm int, keyscale, timesignature string, duration, temperature float32, instrumental bool, seed int, dst string, threads int) int
-)
-
-type AceStepCpp struct {
-	base.SingleThread
-}
-
-func (a *AceStepCpp) Load(opts *pb.ModelOptions) error {
-	// ModelFile is the LM model path
-	lmModel := opts.ModelFile
-
-	// Get the base directory from ModelFile for resolving relative paths
-	baseDir := filepath.Dir(lmModel)
-
-	var textEncoderModel, ditModel, vaeModel string
-
-	for _, oo := range opts.Options {
-		parts := strings.SplitN(oo, ":", 2)
-		if len(parts) != 2 {
-			fmt.Fprintf(os.Stderr, "Unrecognized option: %v\n", oo)
-			continue
-		}
-		switch parts[0] {
-		case "text_encoder_model":
-			textEncoderModel = parts[1]
-		case "dit_model":
-			ditModel = parts[1]
-		case "vae_model":
-			vaeModel = parts[1]
-		default:
-			fmt.Fprintf(os.Stderr, "Unrecognized option: %v\n", oo)
-		}
-	}
-
-	if textEncoderModel == "" {
-		return fmt.Errorf("text_encoder_model option is required")
-	}
-	if ditModel == "" {
-		return fmt.Errorf("dit_model option is required")
-	}
-	if vaeModel == "" {
-		return fmt.Errorf("vae_model option is required")
-	}
-
-	// Resolve relative paths to the base directory
-	// If the path doesn't start with "/" it's relative
-	if !filepath.IsAbs(textEncoderModel) {
-		textEncoderModel = filepath.Join(baseDir, textEncoderModel)
-	}
-	if !filepath.IsAbs(ditModel) {
-		ditModel = filepath.Join(baseDir, ditModel)
-	}
-	if !filepath.IsAbs(vaeModel) {
-		vaeModel = filepath.Join(baseDir, vaeModel)
-	}
-
-	// Also resolve the lmModel if it's relative
-	if !filepath.IsAbs(lmModel) {
-		lmModel = filepath.Join(baseDir, lmModel)
-	}
-
-	fmt.Fprintf(os.Stderr, "[acestep-cpp] Resolved paths:\n")
-	fmt.Fprintf(os.Stderr, "  LM Model: %s\n", lmModel)
-	fmt.Fprintf(os.Stderr, "  Text Encoder: %s\n", textEncoderModel)
-	fmt.Fprintf(os.Stderr, "  DiT Model: %s\n", ditModel)
-	fmt.Fprintf(os.Stderr, "  VAE Model: %s\n", vaeModel)
-
-	if ret := CppLoadModel(lmModel, textEncoderModel, ditModel, vaeModel); ret != 0 {
-		return fmt.Errorf("failed to load acestep models (error code: %d)", ret)
-	}
-
-	return nil
-}
-
-func (a *AceStepCpp) SoundGeneration(req *pb.SoundGenerationRequest) error {
-	caption := req.GetCaption()
-	if caption == "" {
-		caption = req.GetText()
-	}
-	lyrics := req.GetLyrics()
-	bpm := int(req.GetBpm())
-	keyscale := req.GetKeyscale()
-	timesignature := req.GetTimesignature()
-	duration := req.GetDuration()
-	temperature := req.GetTemperature()
-	instrumental := req.GetInstrumental()
-	seed := 42
-	threads := 4
-
-	if ret := CppGenerateMusic(caption, lyrics, bpm, keyscale, timesignature, duration, temperature, instrumental, seed, req.GetDst(), threads); ret != 0 {
-		return fmt.Errorf("failed to generate music (error code: %d)", ret)
-	}
-
-	return nil
-}