mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
mudler-agentandEttore Di Giacinto
c60a2e47d6
* feat(grpc): add a Forecast RPC for time-series models Add Forecast to the backend service with columnar request and response messages: flat row-major arrays, NaN for missing values, and one message per series so a caller can send series of different shapes in one call. The response keeps the request order. Wire the RPC through the server, client, embedded client, connection evicting client, in-flight tracking client and the base backend, which answers unimplemented. core/backend gains a Forecast helper that stamps the model identity and records a trace, like the other single-purpose RPCs. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(config): add the forecast usecase FLAG_FORECAST marks a model as a time-series forecaster. The timestral-cpp backend declares it as its default usecase, so a config with that backend needs no extra lines. The backend is also listed as a non-text backend, so a forecast model is never offered as a chat model. It shows up in the gallery filter and in the capability list. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(api): add POST /v1/forecast Take one or more numeric series and return quantile trajectories for the next prediction_length steps. Targets are [variates][steps] or a flat array for one variate; JSON null marks a missing value. Past and known covariates and categorical codes are checked for length before the backend sees them. Quantiles default to 0.1, 0.5 and 0.9 in the backend. The route is gated by a new default-on forecast API feature. The endpoint is listed in the swagger spec, the /api/instructions index, the UI model filter and the capability symbols. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(backends): add the timestral-cpp forecast backend Wrap the timestral.cpp C ABI with purego, the way the other ggml backends do: libtimestral is built from a pinned commit with ggml embedded, and the Go binary loads it at start. Forecast validates the counts against the array lengths, groups the series by shape, makes one library call per group and puts the results back in the order the caller sent them. Quantiles default to 0.1, 0.5 and 0.9, and a level the model does not have is refused with the list of valid ones. A mutex guards the handle because the library is not reentrant. Request size is bounded by two model options. The unit specs use a fake library. The integration specs and a gRPC spec run against the 5M F32 model that test.sh downloads and checks against a fixed SHA-256. Assisted-by: Claude:claude-sonnet-5-5 go-vet * build(backends): register timestral-cpp in the build and the library Add the CPU image to the backend matrix for amd64 and arm64, plus an Apple silicon entry that is not built or tested yet. The image has no GPU keys because the engine runs on the CPU only. Wire the make targets, the path filter and the dependency bump bot. The library entries were generated with backendcatalog. The images are not published yet, so the release entries are unpinned until a maintainer runs --repin. The gallery reader test now counts the names in the library instead of a fixed number, so adding a backend does not break it. Assisted-by: Claude:claude-sonnet-5-5 go-vet * feat(gallery): add the Timestral forecast models Add Timestral 5M, 28M, 123M and 209M as F32 entries, each with its F16 build as a variant. Q8_0 and the K-quants are left out until the quality question is settled. Each entry runs on the timestral-cpp backend, on the CPU only. The weights use Mistral's Modified MIT licence, which bars use by companies above a monthly revenue threshold. The first paragraph after the summary says so, the licence file is installed next to the weights, and the revenue-restricted-license tag makes the entries filterable. The memory each build needs has not been measured, so no size is set. Assisted-by: Claude:claude-sonnet-5-5 * docs(forecast): document /v1/forecast and the Timestral models Add a feature page with the request and response shapes, the error cases, the model options and the licence note. List the engine in the README, the compatibility table and the backend categories. Assisted-by: Claude:claude-sonnet-5-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
16 KiB
16 KiB
+++ disableToc = false title = "Model compatibility table" weight = 24 url = "/model-compatibility/" +++
Besides llama based models, LocalAI is compatible also with other architectures. The table below lists all the backends, compatible models families and the associated repository.
{{% notice note %}}
LocalAI will attempt to automatically load models which are not explicitly configured for a specific backend. You can specify the backend to use by configuring a model with a YAML file. See [the advanced section]({{%relref "advanced" %}}) for more details.
All backends listed here can be installed on demand from the [Backend Gallery]({{%relref "features/backends" %}}). The exact set of acceleration variants published for each backend is defined in the backend library, gallery/backends/index.yaml. backend/index.yaml is generated from it.
{{% /notice %}}
Text Generation & Language Models
| Backend | Description | Capability | Embeddings | Streaming | Acceleration |
|---|---|---|---|---|---|
| llama.cpp | LLM inference in C/C++. Supports LLaMA, Mamba, RWKV, Falcon, Starcoder, GPT-2, and many others | GPT, Functions | yes | yes | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| ik_llama.cpp | Hard fork of llama.cpp optimized for CPU/hybrid CPU+GPU with IQK quants, custom quant mixes, and MLA for DeepSeek | GPT | yes | yes | CPU (AVX2+) |
| turboquant | llama.cpp fork adding the TurboQuant KV-cache quantization scheme | GPT | yes | yes | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Jetson L4T |
| ds4 | DeepSeek V4 Flash single-model inference engine, optimized for Metal and CUDA | GPT | no | yes | CPU, CUDA 12/13, Metal, Jetson L4T |
| vllm.cpp | From-scratch C++20 port of vLLM by the LocalAI team: paged KV cache, continuous batching, prefix caching, safetensors + GGUF, engine-enforced structured output, no Python at inference | GPT, Functions | no | yes | CPU, CUDA 13 (Ampere through Blackwell), Vulkan, Metal, Jetson L4T (Orin, Thor, GB10) |
| vLLM | Fast LLM serving with PagedAttention; GPTQ/AWQ/FP8 quantization | GPT, Functions, Multimodal | no | yes | CUDA 12/13, ROCm, Intel SYCL, Jetson L4T |
| vLLM Omni | Unified multimodal generation (text, image, video, audio) on top of vLLM | Multimodal GPT, Functions | no | yes | CUDA 12/13, ROCm, Jetson L4T |
| SGLang | Fast serving framework for LLMs and vision-language models with speculative decoding | GPT, Functions, Multimodal | no | yes | CUDA 12/13, ROCm, Intel SYCL, Jetson L4T |
| transformers | HuggingFace Transformers framework | GPT, Embeddings, Multimodal | yes | yes* | CUDA 12/13, ROCm, Intel SYCL, Metal |
| MLX | Apple Silicon LLM inference | GPT, Functions | no | yes | CPU, CUDA 12/13, Metal, Jetson L4T |
| MLX-VLM | Vision-Language Models on Apple Silicon | Multimodal GPT, Functions | no | yes | CPU, CUDA 12/13, Metal, Jetson L4T |
| MLX Distributed | Distributed LLM inference across multiple Apple Silicon Macs | GPT | no | no | CPU, CUDA 12/13, Metal, Jetson L4T |
| tinygrad | Minimalist deep-learning framework with zero runtime dependencies | GPT, Embeddings, Multimodal | yes | yes | CPU |
| RKLLM | Rockchip NPU runtime connected through its OpenAI-compatible server and LocalAI's [cloud-proxy]({{% relref "features/rkllm" %}}) configuration | GPT | no | yes | Rockchip NPU (RK3588, RK3576, RV1126B, RK3562) |
Speech-to-Text
| Backend | Description | Acceleration |
|---|---|---|
| whisper.cpp | OpenAI Whisper in C/C++ | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| faster-whisper | Fast Whisper with CTranslate2 | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal, Jetson L4T |
| WhisperX | Word-level timestamps and speaker diarization | CPU, CUDA 12/13, Metal, Jetson L4T |
| moonshine | Ultra-fast transcription for low-end devices (ONNX) | CPU, CUDA 12/13, Metal |
| parakeet.cpp | C++/GGML port of NVIDIA NeMo Parakeet (tdt/ctc/rnnt/hybrid), with cache-aware streaming | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| CrispASR | Unified speech engine (whisper.cpp fork) supporting Parakeet, Canary, and many ASR architectures, plus TTS | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| voxtral | Voxtral Realtime 4B speech-to-text in pure C | CPU, Metal |
| Qwen3-ASR | Qwen3 automatic speech recognition | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal, Jetson L4T |
| NeMo | NVIDIA NeMo ASR toolkit | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal |
| sherpa-onnx | Sherpa-ONNX ASR (Whisper, Paraformer, SenseVoice) and TTS | CPU, CUDA 12, Metal |
| audio.cpp | Multi-family GGML audio engine: transcription, forced alignment and speaker diarization (nemotron_asr, qwen3_asr, voxtral_realtime, sortformer_diar and more). See audio.cpp backend |
CPU, CUDA 12/13, Metal |
| NeMo-Speech.cpp | NVIDIA C++/GGML runtime for Nemotron Speech: offline, streaming and live transcription, Sortformer diarization, MagpieTTS synthesis and Riva-Translate translation from one backend. See NeMo-Speech.cpp backend | CPU, CUDA 12/13, Vulkan, Metal, Jetson L4T |
Text-to-Speech
| Backend | Description | Acceleration |
|---|---|---|
| piper | Fast neural TTS | CPU, Metal |
| Coqui TTS | TTS with 1100+ languages and voice cloning | CUDA 12, ROCm, Intel SYCL, Metal |
| Kokoro | Lightweight TTS (82M params) | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal, Jetson L4T |
| Kokoros | Pure Rust Kokoro TTS via ONNX | CPU |
| Chatterbox | Production-grade TTS with emotion control | CPU, CUDA 12/13, Metal, Jetson L4T |
| VibeVoice | Real-time TTS with voice cloning | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal, Jetson L4T |
| vibevoice.cpp | Native C++/GGML port of VibeVoice for TTS (voice cloning) and long-form ASR with diarization | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| moss-tts.cpp | Native C++/GGML port of OpenMOSS MOSS-TTS-Local v1.5: 48 kHz stereo TTS with reference-audio voice cloning | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| magpie-tts.cpp | Native C++/GGML port of NVIDIA Magpie TTS Multilingual 357M: 22.05 kHz mono TTS, 5 baked voices, 9+ languages | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| Qwen3-TTS | TTS with custom voice, voice design, and voice cloning | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal, Jetson L4T |
| qwentts.cpp | Native C++/GGML Qwen3-TTS with streaming, named speakers, and voice design | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| OmniVoice | Native C++/GGML TTS with voice cloning, voice design, and streaming | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| fish-speech | High-quality TTS with voice cloning | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal, Jetson L4T |
| Pocket TTS | Lightweight CPU-efficient TTS with voice cloning | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal, Jetson L4T |
| OuteTTS | TTS with custom speaker voices | CPU, CUDA 12 |
| faster-qwen3-tts | Real-time Qwen3-TTS with CUDA graph capture | CPU, CUDA 12/13, Jetson L4T |
| NeuTTS Air | Instant voice cloning, on-device TTS | CPU, CUDA 12, ROCm |
| VoxCPM | Expressive end-to-end TTS | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal |
| Kitten TTS | Kitten TTS model | CPU, Metal |
| Supertonic | Lightning-fast on-device multilingual TTS via ONNX | CPU |
| MLX-Audio | Audio models on Apple Silicon | CPU, CUDA 12/13, Metal, Jetson L4T |
| liquid-audio | LFM2 end-to-end speech-to-speech, ASR, and TTS | CPU, CUDA 12/13, ROCm, Intel SYCL, Jetson L4T |
| audio.cpp | Multi-family GGML audio engine: TTS, voice cloning and voice design (supertonic, vibevoice, qwen3_tts, chatterbox and more). See audio.cpp backend |
CPU, CUDA 12/13, Metal |
| NeMo-Speech.cpp | NVIDIA MagpieTTS over NanoCodec, served by the same backend as Nemotron ASR. See NeMo-Speech.cpp backend | CPU, CUDA 12/13, Vulkan, Metal, Jetson L4T |
Music & Sound Generation
| Backend | Description | Acceleration |
|---|---|---|
| ACE-Step | Music generation from text descriptions, lyrics, or audio | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal |
| acestep.cpp | ACE-Step 1.5 C++ backend using GGML | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| audio.cpp | Multi-family GGML audio engine: music and sound generation (ace_step, stable_audio, heartmula). See audio.cpp backend |
CPU, CUDA 12/13, Metal |
Image & Video Generation
| Backend | Description | Acceleration |
|---|---|---|
| stable-diffusion.cpp | Stable Diffusion, Flux, PhotoMaker, Ideogram, Qwen-Image in C/C++ | CPU, CUDA 12/13, Intel SYCL, Vulkan, Metal, Jetson L4T |
| diffusers | HuggingFace diffusion models (image and video generation) | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal, Jetson L4T |
| mlx-video | LTX-2 and converted Wan2.1/Wan2.2 video generation | Metal (Apple Silicon) |
| vLLM Omni | Multimodal generation including text-to-image and text-to-video | CUDA 12/13, ROCm, Jetson L4T |
Vision, Detection & Recognition
| Backend | Description | Acceleration |
|---|---|---|
| RF-DETR | Real-time transformer-based object detection (Python) | CPU, CUDA 12/13, Intel SYCL, Metal, Jetson L4T |
| rf-detr.cpp | Native RF-DETR object detection and instance segmentation in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T |
| locate-anything.cpp | Open-vocabulary object detection and visual grounding (LocateAnything-3B) in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T |
| depth-anything.cpp | Depth Anything 3 monocular metric depth + camera pose in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T |
| sam3.cpp | Segment Anything (SAM 3/2/EdgeTAM) with text/point/box prompts in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T |
| timestral.cpp | Mistral Timestral time-series forecasting through /v1/forecast, in C/C++ using GGML |
CPU (Linux amd64 and arm64) |
| face-detect.cpp | Native face detection, recognition, embedding, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace) in C/C++ using GGML | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| voice-detect.cpp | Native speaker (voice) recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2) in C/C++ using GGML | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| insightface | Face verification, embedding, and anti-spoofing liveness (ONNX Runtime) | CPU, CUDA 12 |
| speaker-recognition | Speaker (voice) recognition via SpeechBrain ECAPA-TDNN | CPU, CUDA 12, Metal |
Audio Processing
| Backend | Description | Acceleration |
|---|---|---|
| Silero VAD | Voice Activity Detection | CPU, Metal |
| LocalVQE | Joint acoustic echo cancellation, noise suppression, and dereverberation in C/C++ using GGML | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Jetson L4T |
| Opus | Audio codec for WebRTC / Realtime API | CPU, Metal |
| audio.cpp | Multi-family GGML audio engine: voice activity detection (bundled silero_vad and marblenet_vad, no download), source separation into named stems, and voice conversion. See audio.cpp backend |
CPU, CUDA 12/13, Metal |
Utilities & Other
| Backend | Description | Acceleration |
|---|---|---|
| rerankers | Document reranking for RAG | CUDA 12, ROCm, Intel SYCL, Metal |
| privacy-filter.cpp | Standalone GGML engine for the openai-privacy-filter PII/NER token-classification model family (powers LocalAI's PII redaction tier) | CPU, CUDA 13, Vulkan |
| local-store | Local-first vector database for embeddings | CPU, Metal |
| valkey-store | Durable vector store for embeddings backed by Valkey Search (FLAT or HNSW) | CPU, Metal |
| qdrant-store | Durable vector store for embeddings backed by Qdrant (HNSW; self-hosted, clustered or Qdrant Cloud) | CPU, Metal |
| TRL | Fine-tuning (SFT, DPO, GRPO, RLOO, KTO, ORPO) | CPU, CUDA 12/13 |
| llama.cpp quantization | HuggingFace → GGUF model conversion and quantization | CPU, Metal |
Acceleration Support Summary
GPU Acceleration
- NVIDIA CUDA: CUDA 12.0, CUDA 13.0 support across most backends
- AMD ROCm: HIP-based acceleration for AMD GPUs
- Intel oneAPI: SYCL-based acceleration for Intel GPUs (F16/F32 precision)
- Vulkan: Cross-platform GPU acceleration
- Metal: Apple Silicon GPU acceleration (M1/M2/M3+)
Specialized Hardware
- NVIDIA Jetson (L4T CUDA 12): ARM64 support for embedded AI (AGX Orin, Jetson Nano, Jetson Xavier NX, Jetson AGX Xavier)
- NVIDIA Jetson (L4T CUDA 13): ARM64 support for embedded AI (DGX Spark)
- Apple Silicon: Native Metal acceleration for Mac M1/M2/M3+
- Darwin x86: Intel Mac support
CPU Optimization
- AVX/AVX2/AVX512: Advanced vector extensions for x86
- Quantization: 4-bit, 5-bit, 8-bit integer quantization support
- Mixed Precision: F16/F32 mixed precision support
Note: any backend name listed above can be used in the backend field of the model configuration file (See [the advanced section]({{%relref "advanced" %}})).
- * Only for CUDA and OpenVINO CPU/XPU acceleration.