mirror/LocalAI
1
0 Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-10-30 10:09:13 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
1ac2ecb2df6d100618f1cb44b468f9e45d40fc5c
LocalAI/Makefile
T
mudler-agentandEttore Di Giacinto 2cadf319a0 feat(realtime): LFM3-Audio speech-to-speech on llama.cpp, remove liquid-audio (#12958) 
* chore(backends): remove the liquid-audio Python backend

The backend targets liquid-audio, which loads LFM2 checkpoints only.
Remove its directory, the gallery file and the LFM2.5-Audio model
entries, the library and generated backend gallery entries, the
backend matrix and CI job, the Makefile targets and the golden test
fixtures that listed it.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactor(config): drop liquid-audio from capabilities, importers and UI

With the liquid-audio backend gone, remove its capability entry, its
importer, its artifact-gate entry and the realtime_audio heuristic
that only matched that backend. realtime_audio is now set only
through known_usecases. Update the Talk and fine-tuning pages, the
docs and the swagger text that named the backend.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(llama-cpp): bump llama.cpp to e486a4b5

Move the llama.cpp pin to e486a4b559617dc88bf79978996f6c670b6bbc9d,
the master head of 2026-10-28. It contains the LFM3-Audio support
(ggml-org/llama.cpp#31257) and the realtime server split
(ggml-org/llama.cpp#31269).

The score task patch is regenerated against the new pin. It applies
with git apply and with patch -p1, with no offset and no fuzz. Its
hunks in server-task.h and server-context.cpp were rebased by hand
next to SERVER_TASK_TYPE_REALTIME and the control dispatch.

The C++ backend was built on CPU for this change: the llama, mtmd and
server-context libraries and the grpc-server binary link.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): compile server-realtime.cpp into grpc-server

server-context.cpp includes server-realtime.h, and its realtime slot
type is defined in server-realtime.cpp. Upstream compiles that file
into the server library. The gRPC server is one translation unit, so
it must include the file too, or the link fails.

The include is guarded by __has_include. Forks that predate the
realtime split do not ship the file, so they keep building as before.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(llama-cpp): probe realtime and TTS voice support in prepare.sh

prepare.sh writes llama_compat.h from probes on the checkout. Add two:
LOCALAI_HAS_REALTIME (SERVER_TASK_TYPE_REALTIME in server-task.h) and
LOCALAI_HAS_TTS_VOICE (set_voice in mtmd-helper.h). The probes let the
gRPC server choose the realtime and voice code paths per checkout, so
forks without them keep compiling.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): advertise speech recognition and speech-to-speech

llama-cpp now registers the AudioTranscription and AudioToAudioStream
RPCs, with the transcript and realtime_audio usecases. The default
usecase stays chat, so a plain GGUF is still a chat model. The
backend implements the RPCs in a follow-up change; until then the
calls return UNIMPLEMENTED.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(config): add pipeline.any_to_any for speech-to-speech models

A realtime_audio model that serves the whole speech loop itself sets
pipeline.any_to_any. The flag is off by default, so chained pipelines
and existing realtime_audio configs are unchanged. The realtime_audio
heuristic stays explicit: a llama-cpp model is never a Talk-page model
unless known_usecases says so.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(config): read the audio encoder and speech-to-speech projector flags

Add HasAudioEncoderProjector for clip.has_audio_encoder and
HasSpeechToSpeechProjector, which requires the LFM3 audio encoder
family together with the gen-audio decoder. Qwen3-TTS sets both encoder
flags too, so the flags alone cannot tell it apart from LFM3.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(importer): import LFM3-Audio projectors as speech-to-speech

An mmproj that carries the LFM3 audio encoder and the gen-audio decoder
now imports with known_usecases realtime_audio, chat, transcript and
tts, pipeline.any_to_any: true, the parallel:2 option and the tokenizer
template. Other gen-audio projectors keep the TTS-only result, Qwen3-TTS
included. The probe runs against a local HTTP server in the tests.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): named-voice TTS and the LFM3 ASR prompt

Models without a speaker reference (LFM3-Audio) take a named voice
through their "tts" chat template instead of a reference file. The
request is validated with needs_reference=false, the prompt is rendered
with common_chat_get_tts_prompt(), and the voice is set on the gen-audio
input. Qwen3-TTS Base keeps the reference-file rule. The branch is
guarded by LOCALAI_HAS_TTS_VOICE, so forks without named voices keep the
old behaviour.

Transcription takes its prompt from common_chat_get_asr_prompt(). LFM3
gets a system turn that names the language and an empty user turn.
Qwen3-ASR's preset is unchanged: fixed user text, no system turn, and a
caller prompt is still appended after a newline. The preset call is
guarded by a new LOCALAI_HAS_ASR_PRESET probe.

The pure parts sit in tts_request_options.h and transcription_prompt.h,
with standalone tests run by backend/cpp/run-unit-tests.sh.

Add the test-extra-backend-llama-cpp-lfm3 target. It runs the tts,
ttsstream and transcription specs against LFM3-Audio GGUF weights. It is
a manual run and is not wired into CI.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): mock AudioToAudioStream and pending realtime S2S specs

The mock backend serves AudioToAudioStream deterministically: session
created, commit with RMS, a tool call on the first turn with tools, then
transcript pieces and paced 24 kHz tone frames, with response.cancel
ending a reply as cancelled.

The realtime S2S specs cover the mock and real-weights variants. They
skip until the /v1/realtime S2S path lands, so the mock suite stays
green in CI.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): speech-to-speech AudioToAudioStream on the realtime slot

Implement AudioToAudioStream on upstream's realtime slot, so a model
that answers after the audio (LFM3-Audio) can hold a conversation over
the existing duplex RPC. The proto schema does not change: tools travel
in the config, tool calls leave as function_call_arguments.done and tool
results arrive as a function_call_output item.

realtime_session.h keeps what needs no model: the turn text derived
from the chat template (with a tool turn), the conversation state, a
filter that keeps <tool_call> blocks out of the transcript and drops
audio produced inside them, and the translation of slot results into
events. grpc-server.cpp wires it to the slot with control tasks and
parses calls with the parser the chat template selects.

The unit test uses a fake renderer. A ctest-only check renders the real
LFM3 template and compares the slot text with the full render.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(gallery): add the LFM3-Audio-2B entries

Two llama-cpp entries for LiquidAI/LFM3-Audio-2B-GGUF, the Q4_K_M
default (lfm3-audio-2b) and the Q8_0 build (lfm3-audio-2b-q8), both
with the Q8_0 audio projector. Each declares pipeline.any_to_any and the
realtime_audio, chat, transcript and tts usecases the importer writes
for an LFM3 projector, with parallel:2 and the tokenizer template. The
Q4 entry lists the Q8 one as a variant.

File sizes and sha256 come from the Hugging Face API for the pinned
repo revision. Speaker reference files and the VAD model are not part
of the entries: VAD stays optional.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: describe LFM3-Audio speech-to-speech and its gallery entries

Adds a speech-to-speech section to the Realtime API page (any_to_any
configuration, VAD versus manual commit, tools, transcripts, barge-in,
limits), a pipeline.any_to_any row, a llama.cpp speech-to-speech row in
the compatibility table, and LFM3-Audio mentions on the TTS and ASR
pages. The feature is marked experimental: the behaviour follows the
backend session contract and has not been run with real weights.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(realtime): run any-to-any models on AudioToAudioStream

A realtime_audio model with pipeline.any_to_any now opens one duplex
stream to the backend and keeps it for the session, replacing the
"any-to-any models not implemented" branch. The chained pipeline is
unchanged.

The endpoint forwards input audio (with a pre-roll from speech start
when a VAD model is set, otherwise every chunk and a client commit),
sends response.create, and turns the backend's events into the usual
realtime events: audio and transcript deltas, tool calls through the
existing function_call items, and a single response.done. A function
call output is forwarded before the next response. Barge-in sends
response.cancel ahead of the new speech and stops forwarding audio at
once. The user's speech is transcribed on the side and reported on the
user item; a failure there does not touch the response.

Server VAD is on only when pipeline.vad is set. Classifier mode,
compaction, streaming.llm, voice gating, sound detection and
semantic_vad are rejected for these models.

The mock backend now matches the llama.cpp contract (session.updated,
the cancel error when idle) and its VAD reports the loud stretches of
the audio, so the S2S e2e specs run a VAD turn. The specs are enabled.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): add the real-weights realtime S2S variant and the gRPC s2s spec

The realtime S2S specs now check reply audio on real models: duration,
loudness, clipping, and keywords found in the model transcript or in a
separate ASR of the output audio. run-realtime-lfm3.sh stages the llama-cpp
and sherpa-onnx backends, the LFM3-Audio files and the sherpa VAD, ASR
and TTS models, then runs the real variant. It is manual and never part of
CI: `make test-realtime` still filters out real-models.

The backend suite gains an s2s capability (spoken turn, tool call, tool
result, spoken answer over AudioToAudioStream) and optional TTS duration
and RMS bounds plus a transcript keyword check, all enabled for
test-extra-backend-llama-cpp-lfm3.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): expect the idle-cancel error from the mock S2S

The mock now answers a response.cancel with no reply running with
response_cancel_not_active, as the llama.cpp backend does. The cancel
spec still treated that cancel as a no-op and read the error as extra
data after the stream ended.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(talk): show any-to-any models as one speech-to-speech model

/api/pipeline-models lists a pipeline.any_to_any model with any_to_any
set and its optional VAD. A realtime_audio model that is not any-to-any
is listed only when it has the four pipeline slots, since the session
takes the chained path for it.

The Talk page shows a speech-to-speech model as one model: no four-slot
view, the VAD slot only when a VAD model is set, and the female and male
voices offered. Without a VAD model the server does not detect turns, so
the page offers Done speaking, which sends input_audio_buffer.commit and
then response.create. Text-only replies stream through
response.output_text.delta and .done.

self_contained stays in the response for clients built before any_to_any.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): refuse a llama-cpp realtime model without pipeline.any_to_any

llama.cpp has no VAD, transcription or TTS RPCs, so filling the empty
pipeline slots with the model name failed later with UNIMPLEMENTED.
Fail at session start and name the missing setting instead.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): hold manual-commit S2S audio until the commit

In manual-commit mode the endpoint streamed every appended chunk to the
backend, so input_audio_buffer.clear only emptied the Go buffer while the
model kept the audio, and idle audio filled its context between turns.
Keep the audio in the input buffer and send it, behind pending text and
tool-result items, when the client commits or asks for a response.

Frames and controls now go through one ordered write queue, so no gRPC
send happens under the audio buffer lock. A tool result created while a
VAD turn is open is refused with a clear error. When the stream ends
outside a response the session reports it and closes, and a bare EOF
from Send reports the status that Recv holds.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): drop the S2S readiness gate and stale comments

The gate was always true. Some comments referred to work items that
readers of the repository cannot see.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(realtime): add the liquid-audio migration note and the clear semantics

Say that the Python liquid-audio backend is gone, which gallery entries
replace it and that LFM2.5-Audio has no replacement. Describe what
input_audio_buffer.clear drops for speech-to-speech sessions now that
manual-commit audio is held back until the commit.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(talk): do not promise turn detection when no VAD model is set

The idle note said the server notices when you stop talking. A
speech-to-speech model without a VAD model needs Done speaking, so the
note now says that.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): make the mock S2S backend keep the llama-cpp session limits

The mock accepted what the real backend refuses and hid two endpoint bugs.
It now treats output_modalities from the config as a fixed session mode and
picks the default per response from the tools set at that time. A fixed text
plus audio mode with tools reads the tool call out loud. After the first
commit or message, session.update of instructions, voice or tools is an
error that does not belong to a response, committing an empty buffer is an
error, and response.create while a reply runs is refused. Errors of a
refused response.create carry response_scoped in their meta.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): leave the S2S output mode to the backend unless the client sets one

The stream config sent text plus audio when the session had no tools yet.
The llama-cpp backend keeps a mode from the config as a fixed session mode,
and tools added later by session.update never cleared it. The model then
read tool calls out loud. Send output_modalities only when the client chose
them. The backend picks the default of each response from the tools it has
when the response starts.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): mark the realtime errors that answer a response.create

The error event now carries response_scoped: true when it ends the request
for a response (a refused response.create, a failed commit made by it, or
a stream failure). Other errors, such as a refused session.update or a
commit of an empty buffer, are not about a response and one that runs
keeps generating. The client needs the difference to know whether a
response.done is still coming.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): end an S2S response only on the errors that belong to it

Every backend error ended the response in flight, although the backend
also reports errors about other things (a refused session.update, a commit
of an empty buffer) and goes on generating. The rest of the reply was lost
and the next response.create met either a refusal or the leftover events of
the old reply. End the response only on an error marked response_scoped,
forward the others to the client and keep waiting for response.done. Drop
the events buffered by an earlier response when a new one starts, and skip
the commit of a turn that forwarded no audio.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): run the S2S input transcription detached from the commit slot

The commit waited for the side transcription of the user audio while it
held the commit slot, and response.create waits for that slot. The ASR
latency was added before every reply, and on a backend with few slots a
busy ASR task could stall the reply. The model already has the audio, so
run the transcription detached under the session context and release the
slot at once. The transcript is still reported on the user item.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(realtime): do not pin the place of the input transcript in the S2S order

The transcript of the user audio is reported on the side and may land
before or after the first response events. Check the order of the response
events without it.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): stop an S2S reader leak and a cancel that waits on the write queue

The reader waited on a channel it closes itself when the events buffer was
full, so it outlived the session after a turn ended on an error or timed
out. It now also stops when the session closes.

Stopping a response sent the cancel through the write queue and waited for
it before the grace timer started. With backpressure on the stream the wait
never ended and nothing read the events meanwhile. The timer now starts
first and the cancel is only queued.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(realtime): let a chained pipeline run without a VAD model

A pipeline that names no pipeline.vad failed to build, because the
VAD slot was always loaded, and a new session still started with
server_vad. The slot is now optional for chained pipelines as it
already is for any-to-any models: the session starts with turn
detection off and the client commits each turn. Turning server_vad on
for such a session is refused, and a switch to a pipeline without a
VAD drops turn detection.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(gallery): add LFM3-Audio VAD and chained-pipeline entries

Add lfm3-audio-2b-vad and -q8-vad (any-to-any with silero-vad-ggml as
the turn detector), and lfm3-audio-2b-pipeline, -pipeline-vad and the
-q8 forms (the model chained through its own transcription, chat and
speech slots, with and without the VAD). The gallery has no
dependencies between models, so the VAD entries state that
silero-vad-ggml must be installed first.

Drop the variants list from lfm3-audio-2b: it let the installer pick
the larger Q8 build on any host with enough memory, so a plain install
of the Q4 entry did not get Q4. All entries are now independent. Tests
cover the entry shapes, the single-build install and their acceptance
by the realtime endpoint.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): run the realtime specs for any-to-any and chained shapes

The realtime specs covered one any-to-any model with a VAD. They now run
for any-to-any and for a chained pipeline, each with and without a VAD,
on the mock backend in CI. VAD-less shapes also check that turn
detection starts off and that server_vad is refused. The manual
real-weights script writes one config per shape that mirrors the gallery
entries and runs the four shapes in turn (LFM3_MATRIX picks a subset).

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(realtime): document the LFM3-Audio VAD and chained-pipeline entries

List the eight LFM3-Audio gallery entries, explain that silero-vad-ggml
must be installed before the -vad entries, describe the chained pipeline
on one model and the behaviour of a pipeline without a VAD, and
describe the shape matrix of the e2e specs.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(realtime): wait for the detached input transcription in the S2S specs

The input transcription runs in its own goroutine, so the transcription
completed and failed events can arrive after the response ends. Poll for
them instead of reading the event list right after the response.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): release the S2S stream before running server-side tool calls

A response held the stream slot until it returned, and it ran the tool
calls from inside that scope. A server-side tool starts a follow-up
response on the same stream, which then waited for the slot its parent
still held, so the client never got response.done. Run the tool calls
once the slot is free; the event order is unchanged.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): end an S2S session whose stream dies, and stop queuing after close

A stream that died during a response left the session open: later
requests failed one by one and VAD frames kept being forwarded. The
response now ends the session through the transport, and enqueue drops
requests once the session is closed so the write queue cannot grow.

The writer is now the only caller of Send and of CloseSend. close()
used to call CloseSend while the writer could be inside Send, which
gRPC does not allow.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): close the open S2S turn when the VAD stops

Switching turn detection to manual in the middle of a VAD turn left the
turn forwarding, and the next client commit sent the whole buffer a
second time. Stopping the VAD now commits the forwarded audio on the
backend and drops it from the input buffer.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): keep a tool result pending while an S2S user turn is open

The check that refuses a tool result during a user turn ran when the
item arrived, but the item was forwarded later, and the result was
marked as sent before the backend took it. If the user spoke in
between, the backend refused it and it was never retried. The result is
now claimed and queued under the audio lock only when no turn is open,
and otherwise stays pending for the next commit or response.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(talk): empty the input buffer for models without a VAD

Without voice detection the microphone fills the server's input buffer
for the whole session, so Done speaking committed idle silence and the
echo of the previous reply. Clear the buffer when the session starts and
when a reply ends.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(importer): keep an existing parallel option on LFM3-Audio imports

The speech-to-speech import added parallel:2 even when the options
already carried another parallel value. Use the same check as the
hardware defaults, which also covers n_parallel.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(talk): list chained pipelines that have no VAD

/api/pipeline-models required a VAD for a chained pipeline, so the
VAD-less LFM3-Audio pipeline entries never showed up on the Talk page.
Any pipeline with transcription, LLM and speech slots now qualifies.
The Talk page treats every model without a VAD as manual-turn, and the
session sheet says so for chained pipelines too.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(llama-cpp): bump llama.cpp to 0ac1a7fd

Move the llama.cpp pin to 0ac1a7fd35c127a18c0cbc6c57d12e256a6a385b,
the master head of 2026-10-29. It contains tool calls in the realtime
server (ggml-org/llama.cpp#31287).

The score task patch is regenerated so that it applies with no offset
and no fuzz.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(functions): keep a null tool argument null

A tagged parameter whose text was "null" came out as an empty string:
unmarshalling JSON null into a string succeeds and leaves it empty, so
the value was taken for a string. Check the first byte of the JSON
value instead. A parameter with the text "null" now becomes JSON null,
as in llama.cpp.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(functions): add an lfm3 XML format preset

The LFM3 chat template writes a call as <tool_call>, <function=NAME>,
one <parameter=KEY> block per argument, then the closing tags. The
qwen3-coder format has the same shape and auto-detection already found
it, but xml_format_preset: lfm3 was not a known name.

Add the preset, with tests for one call, a call without arguments, two
calls, string and non-string values (numbers, booleans, null, arrays,
objects), a call between text, and unfinished input.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(realtime): parse tool calls with the chat template, as upstream

Since ggml-org/llama.cpp#31287 the realtime server handles tools with
the parser and grammar of the chat template. Do the same in the
speech-to-speech stream instead of a hard-coded tag filter and a parse
of the whole text at the end.

- Each response gets a common_chat_session with the tools of the
  session and parallel calls. Its grammar and preserved tokens go into
  the sampling of the response, so the slot keeps the call tokens in
  the text and constrains the call.
- The text of the response is fed to the session piece by piece
  (task_result_state::update_chat_msg). Content deltas become the
  transcript; tool-call deltas are collected into one
  function_call_arguments.done per call at the end. Audio after the
  start of a call is dropped.
- A tool_choice of auto, none or required is accepted in the params of
  the config, in session.update and in response.create, with the
  response overriding the session.
- The tool turn is derived from a conversation in which the assistant
  called a tool, like upstream, not from a plain reply.
- response.done carries raw_text, everything the model wrote with the
  call markup, for a caller that parses it another way. If the parser
  throws, the rest of the response is content and the raw text is the
  way out.

ToolCallFilter is no longer used and goes in the next commit.

The unit tests cover the splitter contract (content, call deltas, the
final piece, whole characters only). realtime_chat_check feeds
LFM3-format text through the session in pieces of 1, 7 and all bytes
and checks the call, the transcript and the grammar.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactor(realtime): drop the tool-call text filter

The parser of the chat template now splits content from tool calls, so
ToolCallFilter, which looked for the literal <tool_call> tags, has no
caller. Remove it with its tests. Every case it covered (a call
between text, split tags, several calls, an unfinished call, text after
a call) is handled by the parser, and realtime_chat_check runs the
LFM3 format through it in pieces of different sizes.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(realtime): fall back to the raw text for S2S tool calls

For an any-to-any model, the calls the backend parsed still win. When
there are none and the request had tools, parse the raw text that
response.done now carries the way the chained pipeline parses the text
of its LLM (CleanupLLMResult, ParseFunctionCall, ParseTextContent), so
a call format the backend parser does not know can still be served
with function.xml_format_preset or the other function settings of the
model. The content around the calls replaces the streamed transcript
only if the config captures it, as in the chained path.

Also pass tool_choice to the backend: the session mode with the stream
config and with session.update, and the mode of a response.create
override with that event. A named function cannot be forced, so it is
not sent.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): carry raw_text in the mock S2S response.done

Match the llama-cpp session: with tools set, response.done has raw_text,
the call markup of the LFM3 template for a tool turn and the transcript
for any other reply.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(functions): parse tool blocks that have text between them

The iterative XML parser stopped at the first scope block that was
preceded by text, so "<tool_call>A</tool_call> then <tool_call>B" gave
one call. Skip text between blocks once a call was found; text before
the first block still ends the parse as before. Blocks that follow each
other, or only whitespace, already worked.

Tests cover three calls back to back and with text between them, with
the lfm3 preset and with auto-detection. They also pin the current
reading of argument values: the parser has no tool schema, so a value
such as 42 or true is read as JSON even when the schema types it as a
string.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(gallery): read LFM3-Audio tool calls with the lfm3 preset

Set function.xml_format_preset: lfm3 on the eight LFM3-Audio-2B entries
and in the config the importer writes for a speech-to-speech projector,
so the server parses the call format of the LFM3 chat template when the
backend returns no call (and in the chained pipeline entries, which use
the regular chat path). gallery/index.yaml is regenerated from the
library.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(realtime): describe how S2S tool calls are read

Replace the description of the hard-coded tag filter with the parser
and grammar of the chat template, tool_choice, and the raw-text
fallback in the server. Document function.xml_format_preset, and set it
in the real-weights e2e model configs like the gallery entries do.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(functions): keep earlier calls when a later block does not parse

After a valid call, text that holds the opening tag again (a prose
mention, a markdown example, a malformed second block) made the parser
treat it as a new block, fail, and return no calls at all. Remember the
state after each complete block and fall back to it when a later one
fails: the calls found so far are kept, parsing stops there and the
remaining text stays in the content.

Text skipped between two valid blocks now goes to the content instead of
being dropped, so StripToolCallMarkup keeps it.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): do not drop audio after a tool call started

In mixed mode the speech for text written before a call can still be
pending when the call starts, so dropping every audio frame after the
first call delta cut the speech while its transcript was sent. The
llama.cpp server never drops audio when it reads a result; do the same.
If a model speaks the call markup aloud, that is model behaviour to
observe on real weights, not something to filter here.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): flush held-back text when a response is cancelled

The parser got is_partial=true on a cancelled response, so text it held
back (a possible start of a tag) never reached the transcript. The llama.cpp
server parses a cancelled response with is_partial=false; do the same. A
cancelled response still makes no tool call. The path where the parser
throws already sends the text as content and needs no change.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(realtime): stop the fake S2S stream from sending after Close

The barge-in spec failed now and then with a data race and "send on
closed channel": its script pushes audio from a goroutine, and the
session closes the stream while that goroutine still runs. A push now
does nothing once the stream is closed, and Close waits for a push in
flight. The production close order is right: the writer closes the
stream only after it stops sending.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): align the S2S tool-call fallback with the backend

Four gaps in how the Go side reads the tool calls of an any-to-any reply:

- With tool_choice none (the response value wins over the session value)
  the backend builds no tool parser, yet the raw-text fallback still
  turned a call the model wrote into a function_call item. Skip it.
- Without a capture regex the fallback never cleaned the transcript, so the
  stored item and transcript.done kept the raw call markup. When the call
  is XML, use the text left after the markup is cut out; a reply that is
  only a call now has an empty transcript.
- A named function as tool_choice mapped to nothing, so the backend kept
  the previous mode. Refuse it with an error in session.update and
  response.create.
- A response-level tools list changed what the fallback looked for, but
  the backend only has the tools of the session. Use those in both.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(realtime): say how S2S tool_choice and response tools behave

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(e2e): use the real llama-cpp backend in the real realtime run

The suite always registered the mock backend under the name llama-cpp, so
a run with REALTIME_S2S_MODEL set never reached the llama-cpp backend
found in REALTIME_BACKENDS_PATH. Register the mock only when that run is
off.

The real run converts the input audio with ffmpeg. Check for it in the
suite and in run-realtime-lfm3.sh before any download or backend start,
with a message that says what to install, and list it as a prerequisite
in the script header and the Makefile target.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(realtime): cover the tools path of a client session.update to the S2S backend

A tool call that never came back from an any-to-any model raised the
question whether the tools of the session reach the backend at all. Add a
spec that sends the session.update of the real-weights e2e (flat
Responses API tool, no session type) and checks that the backend gets
the tool in the chat-completions form.

Log the number of tools sent when the stream opens and when a
session.update carries them, so a run log shows it without reading the
wire. The "No tools received" line of the llama-cpp backend does not
show it: the server prints it for the input transcription request, which
has no tools by design.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): interrupt the chained pipeline reply while it is in flight

With unary stages, the default of the lfm3 pipeline gallery entries, the
chained pipeline synthesizes the whole reply and sends it as one audio
delta right before response.done. The barge-in spec waited for that
delta and then rendered the interruption with the TTS, so the first
response was already done when the speech started and ended completed
instead of cancelled. Real runs on a GPU showed it with both
quantizations.

For a pipeline, start the interruption at response.created, which comes
before the LLM and TTS stages. Render the interruption audio before the
first utterance is sent, so nothing sits between the reply and the
interruption. The any-to-any shape still waits for the first audio
delta, which that backend streams while it generates.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): accept a number word and its homophone in the spoken-answer check

The check that the reply audio says what the transcript says needs a
shared word of 4+ letters. A short answer failed it: the model wrote
"Four" and the ASR heard "for". Compare numbers in one spoken form, so
digits, number words and their homophones match each other. A homophone
only counts when the other text has the number itself, so "for" on both
sides still shares nothing and the check does not pass on a stop word.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(llama-cpp): ask LFM3-Audio for function calls when a realtime session has tools

The default system prompt of a realtime session was "Respond with
interleaved text and audio." whatever the tools. Under that prompt
LFM3-Audio answers in words ("I'll check the weather ...") and never
writes the call. The model card, and the llama.cpp realtime example
client, use "Respond with function calls, and interleaved text and
audio." when the session has tools, and the model then calls the tool.

Pick the default by the tools of the session, and treat the plain default
that the LocalAI server sends back as no choice: it follows the tools too,
also when they arrive with a session.update before the first audio. A
prompt the client wrote is kept as it is.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): transcribe the input of an any-to-any turn between responses

The input transcript of a speech-to-speech turn ran at the same time as
the reply, on the same model. llama.cpp serves both from one context: the
realtime slot wants embeddings on, the chat slot of the transcript wants
them off, and each flip makes it reserve its compute buffers again. In a
GB10 run that was two reserves of about 175 ms per step, the reply fell to
2.6 tokens per second and the transcript to 380 ms per token, until the
transcript was done.

When pipeline.transcription is the model of the stream, run the transcript
after the response, or before one that is not started yet, and keep
responses out while it runs. A separate transcription model still runs in
parallel.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): send the input transcript of an any-to-any turn before its reply

The previous change moved the same-model transcript after the response.
A client that reads until response.done then never saw
conversation.item.input_audio_transcription.completed: the transcript
was made (the log shows it) but sent after the end of the window.

Run it in line with the commit instead. A VAD turn transcribes and then
responds. A client response.create already waits for the commit slot, so
it starts once the transcript is out. The transcript event now comes
before any output of the response of its turn, for a manual commit and for
a VAD turn, and never overlaps the response on the llama.cpp context. A
transcription model of its own still runs in parallel.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(realtime): transcribe the input of an any-to-any model only when asked

The stream answers from the audio, so the transcript of the input is for
the client only. On the model of the stream it is costly: llama.cpp
serves the transcript request and the realtime stream from one context
and rebuilds its compute buffers on every step while both are active.
On a GB10 that was 26 to 29 reserves of about 175 ms per turn, against 0
to 1 on the llama.cpp server, and the reply ran at a few tokens per
second.

An any-to-any model no longer fills pipeline.transcription with its own
name. Without it the session does not transcribe the input; with another
model it does, in parallel with the reply; with the model itself named it
does, before the reply. The reply transcript always comes from the stream.
The gallery entries set none, so they get the fast path.

The mock configs and run-realtime-lfm3.sh name a transcription model, so
the real-weights spec still covers the input transcript.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): end a response as cancelled when a barge-in lands during the LLM call

The chained pipeline cancels the response context on a barge-in. When that
happens while the model is generating, the backend call fails with the
context error, and the response ended as failed: an error event and no
response.done. A client that waits for the terminal of the interrupted
response then waits forever. On a GB10 the q4 model answered before the
barge-in arrived, so only the slower q8 model showed it: the spec read
events for 30 s and timed out.

When the context is cancelled, a failed Predict call or a failed result is
a cancel, as the TTS step and the streamed LLM path already treat it. The
response ends with response.done{cancelled} and no error.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): wait for the input transcript that comes from a model of its own

An any-to-any model now gets its input transcript from a separate ASR
that runs next to the reply. The reply is often over first, so a spec that
stops reading at response.done saw no transcript in the VAD shape. Keep
reading, up to 30 s, until the transcript arrives. The contract is that it
comes, not when.

Add a guard spec for what the input transcription is handed: the whole
turn, in a WAV that says the rate it was recorded at (the session rate for
a client commit, 16 kHz for a VAD turn).

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): check the errors of Close in the realtime specs

golangci-lint (errcheck) flagged the unchecked Close of the websocket
connections, the response body and the fake stream. Use the
`defer func() { _ = x.Close() }()` form the other e2e specs use.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): write the mock S2S specs with Ginkgo and drop the env read

forbidigo only allows Ginkgo/Gomega specs and no os.Getenv outside the
sanctioned places. Convert the mock S2S tests to Describe/It with a suite
bootstrap (the package had none), with the same 11 cases. The mock no
longer reads MOCK_S2S_FRAMES: only these specs set it, so the reply length
is a package variable they change per spec and restore afterwards.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-10-29 22:06:41 +01:00

1779 lines
91 KiB
Makefile
Raw Blame History

# Disable parallel execution for backend builds
.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/nemo-speech-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/qdrant-store backends/cloud-proxy backends/localai-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/mlx-video backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/supertonic backends/depth-anything-cpp backends/timestral-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin
.NOTPARALLEL: backends/whisper-medusa
.NOTPARALLEL: backends/funasr
GOCMD=go
GOTEST=$(GOCMD) test
GOVET=$(GOCMD) vet
BINARY_NAME=local-ai
LAUNCHER_BINARY_NAME=local-ai-launcher
UBUNTU_VERSION?=2404
UBUNTU_CODENAME?=noble
# Optional Ubuntu apt mirror overrides forwarded to docker builds.
# Empty = use upstream archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com.
# Set e.g. APT_MIRROR=http://azure.archive.ubuntu.com to route apt traffic
# during outages of the default Ubuntu pool.
APT_MIRROR?=
APT_PORTS_MIRROR?=
GORELEASER?=
export BUILD_TYPE?=
export CUDA_MAJOR_VERSION?=13
export CUDA_MINOR_VERSION?=0
GO_TAGS?=
BUILD_ID?=
NATIVE?=false
TEST_DIR=/tmp/test
TEST_FLAKES?=5
RANDOM := $(shell bash -c 'echo $$RANDOM')
VERSION?=$(shell git describe --always --tags || echo "dev" )
# fyne package only accepts numeric x[.y[.z]] app versions, so reduce git
# describe output (v4.9.0, v4.9.0-14-gabc1234, or a bare sha on untagged
# checkouts) to its numeric core; anything non-numeric falls back to 0.0.0.
# Without this the packaged launcher reports itself as version 0.0.0 (#11673).
LAUNCHER_APP_VERSION?=$(shell v=$$(echo "$(VERSION)" | sed -E 's/^v//; s/[+-].*$$//'); echo "$$v" | grep -qE '^[0-9]+(\.[0-9]+){0,2}$$' && echo "$$v" || echo "0.0.0")
# go tool nm ./local-ai | grep Commit
LD_FLAGS?=-s -w
override LD_FLAGS += -X "github.com/mudler/LocalAI/internal.Version=$(VERSION)"
override LD_FLAGS += -X "github.com/mudler/LocalAI/internal.Commit=$(shell git rev-parse HEAD)"
# The model source the binary reads by default: "library" (the model library) or
# "gallery" (the gallery named localai). A build from master reads the library.
# A build of a commit that carries a tag reads the gallery until the release
# workflow cuts over by setting LOCALAI_MODEL_SOURCE=library (see release.yaml).
LOCALAI_MODEL_SOURCE?=$(shell git describe --exact-match --tags >/dev/null 2>&1 && echo gallery || echo library)
override LD_FLAGS += -X "github.com/mudler/LocalAI/core/library.ModelSource=$(LOCALAI_MODEL_SOURCE)"
# The backend source follows the same rule: "library" (the backend library,
# gallery/backends/index.yaml) or "gallery" (the gallery named localai-backends,
# backend/index.yaml, which is generated from the library). A build from master
# reads the library; a build of a commit that carries a tag reads the gallery
# until the release workflow cuts over with LOCALAI_BACKEND_SOURCE=library.
LOCALAI_BACKEND_SOURCE?=$(shell git describe --exact-match --tags >/dev/null 2>&1 && echo gallery || echo library)
override LD_FLAGS += -X "github.com/mudler/LocalAI/core/library.BackendSource=$(LOCALAI_BACKEND_SOURCE)"
OPTIONAL_TARGETS?=
export OS := $(shell uname -s)
ARCH := $(shell uname -m)
GREEN := $(shell tput -Txterm setaf 2)
YELLOW := $(shell tput -Txterm setaf 3)
WHITE := $(shell tput -Txterm setaf 7)
CYAN := $(shell tput -Txterm setaf 6)
RESET := $(shell tput -Txterm sgr0)
# Default Docker bridge IP
E2E_BRIDGE_IP?=172.17.0.1
ifndef UNAME_S
UNAME_S := $(shell uname -s)
endif
ifeq ($(OS),Darwin)
ifeq ($(OSX_SIGNING_IDENTITY),)
OSX_SIGNING_IDENTITY := $(shell security find-identity -v -p codesigning | grep '"' | head -n 1 | sed -E 's/.*"(.*)"/\1/')
endif
endif
# check if goreleaser exists
ifeq (, $(shell which goreleaser))
GORELEASER=curl -sfL https://goreleaser.com/static/run | bash -s --
else
GORELEASER=$(shell which goreleaser)
endif
TEST_PATHS?=./api/... ./pkg/... ./core/... ./backend/go/cloud-proxy/... ./backend/go/localai-proxy/... ./backend/go/local-store/... ./backend/go/valkey-store/... ./backend/go/qdrant-store/...
## Coverage output and the committed baseline that CI compares against.
## The gate is strict: total coverage must never decrease (no tolerance).
## covermode=atomic makes line coverage deterministic regardless of test
## ordering or flake retries, so there is no run-to-run jitter to absorb.
COVERAGE_DIR?=$(abspath ./coverage)
COVERAGE_PROFILE?=$(COVERAGE_DIR)/coverage.out
COVERAGE_BASELINE?=coverage-baseline.txt
## Coverage is collected one recursive root at a time and merged (see
## scripts/run-coverage.sh): passing several recursive roots to a single
## ginkgo invocation only keeps one root's coverprofile. Mirrors TEST_PATHS
## minus ./api (which doesn't exist).
COVERAGE_ROOTS?=./pkg ./core
## Build tags for the coverage build. `auth` is required to compile the real
## auth implementation and its ~150 `//go:build auth` tests (otherwise they're
## invisible and the gate scores auth against a stub). `debug` matches `test`.
COVERAGE_TAGS?=debug auth
## Coverage is attributed to these packages via --coverpkg, so the in-process
## integration suites (COVERAGE_E2E_ROOTS) credit the core/http handlers they
## drive over HTTP — not just their own test package.
COVERAGE_COVERPKG?=github.com/mudler/LocalAI/core/...,github.com/mudler/LocalAI/pkg/...
## In-process integration suites folded into coverage. Run non-recursively
## (excludes tests/e2e/distributed, which needs containers) with the mock
## backend built by prepare-test. real-models specs need a downloaded model,
## so they're filtered out. NOTE: tests/integration is intentionally NOT here —
## it needs the local-store backend built (`make backends/local-store`), which
## the coverage CI job doesn't do.
COVERAGE_E2E_ROOTS?=./tests/e2e
COVERAGE_E2E_LABELS?=!real-models
## Drop generated protobuf from the denominator (it has no tests by design).
COVERAGE_EXCLUDE_RE?=grpc/proto/.*[.]pb[.]go
.PHONY: all test test-coverage test-coverage-baseline test-coverage-check test-backend-cpp test-build-scripts test-ui test-ui-stale-chunk test-ui-coverage-baseline test-ui-coverage-check build vendor lint lint-all
all: help
## GENERIC
rebuild: ## Rebuilds the project
$(GOCMD) clean -cache
$(MAKE) build
clean: ## Remove build related file
$(GOCMD) clean -cache
rm -f prepare
rm -rf $(BINARY_NAME)
rm -rf release/
$(MAKE) protogen-clean
rmdir pkg/grpc/proto || true
clean-tests:
rm -rf test-models
rm -rf test-dir
rm -f tests/e2e/mock-backend/mock-backend
## Install Go tools
install-go-tools:
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
## React UI:
react-ui:
ifneq ($(wildcard core/http/react-ui/dist),)
@echo "react-ui dist already exists, skipping build"
else
cd core/http/react-ui && npm install && npm run build
endif
react-ui-docker:
docker run --entrypoint /bin/bash -v $(CURDIR):/app:z oven/bun:1 \
-c "cd /app/core/http/react-ui && bun install && bun run build"
core/http/react-ui/dist: react-ui
## Build:
build: protogen-go generate install-go-tools core/http/react-ui/dist ## Build the project
$(info ${GREEN}I local-ai build info:${RESET})
$(info ${GREEN}I BUILD_TYPE: ${YELLOW}$(BUILD_TYPE)${RESET})
$(info ${GREEN}I GO_TAGS: ${YELLOW}$(GO_TAGS)${RESET})
$(info ${GREEN}I LD_FLAGS: ${YELLOW}$(LD_FLAGS)${RESET})
$(info ${GREEN}I UPX: ${YELLOW}$(UPX)${RESET})
rm -rf $(BINARY_NAME) || true
CGO_LDFLAGS="$(CGO_LDFLAGS)" $(GOCMD) build -ldflags "$(LD_FLAGS)" -tags "$(GO_TAGS)" -o $(BINARY_NAME) ./cmd/local-ai
build-launcher: ## Build the launcher application
$(info ${GREEN}I local-ai launcher build info:${RESET})
$(info ${GREEN}I BUILD_TYPE: ${YELLOW}$(BUILD_TYPE)${RESET})
$(info ${GREEN}I GO_TAGS: ${YELLOW}$(GO_TAGS)${RESET})
$(info ${GREEN}I LD_FLAGS: ${YELLOW}$(LD_FLAGS)${RESET})
rm -rf $(LAUNCHER_BINARY_NAME) || true
CGO_LDFLAGS="$(CGO_LDFLAGS)" $(GOCMD) build -ldflags "$(LD_FLAGS)" -tags "$(GO_TAGS)" -o $(LAUNCHER_BINARY_NAME) ./cmd/launcher
build-all: build build-launcher ## Build both server and launcher
build-dev: ## Run LocalAI in dev mode with live reload
@command -v air >/dev/null 2>&1 || go install github.com/air-verse/air@latest
air -c .air.toml
dev-dist:
$(GORELEASER) build --snapshot --clean
## PR-time variant of dev-dist: builds only the host platform instead of all
## three release targets (linux/amd64, linux/arm64, darwin/arm64). The point of
## running goreleaser on a PR is to catch a broken config or a broken
## before-hook (protogen-go, react-ui, go mod tidy), and --single-target still
## exercises every one of those. Nothing consumes a PR's cross-compiled
## binaries. master pushes and tags still run the full dev-dist/dist.
dev-dist-single:
$(GORELEASER) build --snapshot --clean --single-target
dist:
$(GORELEASER) build --clean
osx-signed: build
codesign --deep --force --sign "$(OSX_SIGNING_IDENTITY)" --entitlements "./Entitlements.plist" "./$(BINARY_NAME)"
## Run
run: ## run local-ai
CGO_LDFLAGS="$(CGO_LDFLAGS)" $(GOCMD) run ./cmd/local-ai
prepare-test: protogen-go build-mock-backend
########################################################
## Tests
########################################################
## Test targets
## After the test-suite reorg (see plans/test-reorg) the default `make test`
## no longer downloads multi-GB GGUF/whisper fixtures or builds llama-cpp /
## transformers / piper / whisper / stablediffusion-ggml. core/http/app_test.go
## now drives the mock-backend binary built by build-mock-backend; real-backend
## inference moved into tests/e2e-backends/ (per-backend, path-filtered) and
## tests/e2e-aio/ (nightly).
test: prepare-test
@echo 'Running tests'
export GO_TAGS="debug"
OPUS_SHIM_LIBRARY=$(abspath ./pkg/opus/shim/libopusshim.so) \
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) --fail-fast -v -r $(TEST_PATHS)
## Compiles and runs the standalone C++ unit tests for the backends (pure
## helpers that depend only on the stdlib + nlohmann/json, no full backend
## build). Discovers every *_test.cpp under backend/cpp/ - see
## backend/cpp/run-unit-tests.sh. Set NLOHMANN_INCLUDE to skip the header fetch.
test-backend-cpp:
bash backend/cpp/run-unit-tests.sh
## Runs the shell-level regression tests for the image packaging scripts
## (scripts/build/*_test.sh). These guard invariants that only ever break
## inside a container build - a missing transitive dep, a partial cuDNN
## family - and that no Go test can observe. Needs only bash + gcc + ldd.
test-build-scripts:
@set -e; for t in scripts/build/*_test.sh; do echo "== $$t"; bash "$$t"; done
## Runs the unit tests for the CI helper scripts under scripts/lib/. Currently
## the backend matrix path filter, whose failure mode is invisible in CI: it
## emits an empty matrix, every job goes green, and the change ships to no
## image at all (see PR #10946). Plain `node --test`, no dependencies.
test-ci-scripts:
@set -e; for t in scripts/lib/*_test.mjs; do echo "== $$t"; node --test "$$t"; done
## Runs the unit tests for the shared python backend helpers. These modules are
## pure stdlib on purpose so they run without any backend venv; the list is
## explicit because their siblings (model_identity_test) import grpc and the
## generated protobufs, which only exist inside a built backend.
PYTHON_HELPER_TESTS?=python_utils_test vllm_utils_test model_utils_test mlx_utils_test parent_watch_test temp_utils_test
test-python-helpers:
cd backend/python/common && python3 -m unittest $(PYTHON_HELPER_TESTS)
## Runs the core suite ($(TEST_PATHS)) with statement-coverage instrumentation
## and writes a merged profile to $(COVERAGE_PROFILE). Deliberately omits
## --fail-fast so a single failure doesn't truncate the coverage number, and
## uses covermode=atomic so the result is deterministic. Prints the total.
test-coverage: prepare-test
@echo 'Running tests with coverage'
GINKGO_TAGS="$(COVERAGE_TAGS)" \
COVERAGE_COVERPKG="$(COVERAGE_COVERPKG)" \
COVERAGE_E2E_ROOTS="$(COVERAGE_E2E_ROOTS)" \
COVERAGE_E2E_LABELS="$(COVERAGE_E2E_LABELS)" \
COVERAGE_EXCLUDE_RE='$(COVERAGE_EXCLUDE_RE)' \
OPUS_SHIM_LIBRARY=$(abspath ./pkg/opus/shim/libopusshim.so) \
scripts/run-coverage.sh $(COVERAGE_DIR) $(COVERAGE_PROFILE) $(TEST_FLAKES) $(COVERAGE_ROOTS)
@$(GOCMD) tool cover -html=$(COVERAGE_PROFILE) -o $(COVERAGE_DIR)/coverage.html
@$(GOCMD) tool cover -func=$(COVERAGE_PROFILE) | tail -n1
## Writes the current total coverage to $(COVERAGE_BASELINE). Run this (and
## commit the result) whenever a change legitimately raises coverage so the
## ratchet moves up. Never lower it by hand.
test-coverage-baseline: test-coverage
@$(GOCMD) tool cover -func=$(COVERAGE_PROFILE) | awk '/^total:/{gsub(/%/,"",$$NF); print $$NF}' > $(COVERAGE_BASELINE)
@echo "Saved coverage baseline: $$(cat $(COVERAGE_BASELINE))%"
## CI gate: fails if total coverage dropped more than COVERAGE_TOLERANCE
## (default 0.5pp) below the committed baseline. A small tolerance absorbs the
## run-to-run jitter from the in-process tests/e2e suite folded in via
## --coverpkg (timing-dependent which handler lines execute).
test-coverage-check: test-coverage
@scripts/coverage-check.sh $(COVERAGE_PROFILE) $(COVERAGE_BASELINE)
########################################################
## Lint
########################################################
## Runs golangci-lint with config from .golangci.yml. Includes the standard
## linter set plus forbidigo, which enforces the Ginkgo/Gomega-only test
## convention documented in .agents/coding-style.md.
##
## LINT_EXCLUDE_DIRS_RE matches directories whose Go packages can't typecheck
## without C/C++ headers we don't install in the lint runner (cgo wrappers
## around llama.cpp, piper/spdlog, silero-vad/onnxruntime, and Fyne/OpenGL for
## the launcher). Their compile-time correctness is enforced by their own
## build pipelines. Keep this as a deny list — `go list ./...` discovers
## everything else automatically, so new packages are scanned by default.
LINT_EXCLUDE_DIRS_RE=/(backend/go/(piper|silero-vad|llm)|cmd/launcher)(/|$$)
## Set LINT_NEW_FROM to a git ref to override .golangci.yml's
## new-from-merge-base (origin/master). Useful from a fork clone where
## origin/master is stale relative to the canonical repo.
LINT_NEW_FROM?=
lint:
@command -v golangci-lint >/dev/null 2>&1 || { \
echo 'golangci-lint not installed. Install: go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest'; \
exit 1; \
}
golangci-lint run $(if $(LINT_NEW_FROM),--new-from-merge-base=$(LINT_NEW_FROM),) $$(go list -e -f '{{.Dir}}' ./... | grep -vE '$(LINT_EXCLUDE_DIRS_RE)')
## Like `lint` but reports every issue, including the pre-existing baseline
## that `lint` ignores via .golangci.yml's new-from-merge-base. Use this to
## see what's available to clean up.
lint-all:
@command -v golangci-lint >/dev/null 2>&1 || { \
echo 'golangci-lint not installed. Install: go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest'; \
exit 1; \
}
golangci-lint run --new=false --new-from-merge-base= --new-from-rev= $$(go list -e -f '{{.Dir}}' ./... | grep -vE '$(LINT_EXCLUDE_DIRS_RE)')
########################################################
## E2E AIO tests (uses standard image with pre-configured models)
########################################################
docker-build-e2e:
docker build \
--build-arg MAKEFLAGS="--jobs=5 --output-sync=target" \
--build-arg BASE_IMAGE=$(BASE_IMAGE) \
--build-arg IMAGE_TYPE=$(IMAGE_TYPE) \
--build-arg BUILD_TYPE=$(BUILD_TYPE) \
--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
--build-arg APT_MIRROR=$(APT_MIRROR) \
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
--build-arg GO_TAGS="$(GO_TAGS)" \
-t local-ai:tests -f Dockerfile .
e2e-aio:
LOCALAI_BACKEND_DIR=$(abspath ./backends) \
LOCALAI_MODELS_DIR=$(abspath ./tests/e2e-aio/models) \
LOCALAI_IMAGE_TAG=tests \
LOCALAI_IMAGE=local-ai \
$(MAKE) run-e2e-aio
run-e2e-aio: protogen-go
@echo 'Running e2e AIO tests'
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e-aio
# Distributed architecture e2e (PostgreSQL + NATS via testcontainers).
# Includes NatsJWT specs (JWT-enabled NATS). Requires Docker.
# VLLMMultinode is excluded here; use test-e2e-vllm-multinode for that.
test-e2e-distributed: protogen-go
@echo 'Running distributed e2e tests (label Distributed, incl. NatsJWT)'
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter='Distributed && !VLLMMultinode' --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e/distributed
# vLLM multi-node DP smoke (CPU). Builds local-ai:tests and the
# cpu-vllm backend from the current working tree, then drives a
# head + headless follower via testcontainers-go and asserts a chat
# completion. BuildKit caches both images, so re-runs only rebuild
# what changed. The test lives under tests/e2e/distributed and is
# selected by the VLLMMultinode label so it doesn't run alongside
# test-e2e-distributed.
test-e2e-vllm-multinode: docker-build-e2e extract-backend-vllm protogen-go
@echo 'Running e2e vLLM multi-node DP test'
LOCALAI_IMAGE=local-ai \
LOCALAI_IMAGE_TAG=tests \
LOCALAI_VLLM_BACKEND_DIR=$(abspath ./local-backends/vllm) \
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter='VLLMMultinode' -v -r ./tests/e2e/distributed
########################################################
## E2E tests
########################################################
prepare-e2e:
docker build \
--build-arg IMAGE_TYPE=core \
--build-arg BUILD_TYPE=$(BUILD_TYPE) \
--build-arg BASE_IMAGE=$(BASE_IMAGE) \
--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
--build-arg APT_MIRROR=$(APT_MIRROR) \
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
--build-arg GO_TAGS="$(GO_TAGS)" \
--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
-t localai-tests .
run-e2e-image:
docker run -p 5390:8080 -e MODELS_PATH=/models -e THREADS=1 -e DEBUG=true -d --rm -v $(TEST_DIR):/models --name e2e-tests-$(RANDOM) localai-tests
test-e2e: build-mock-backend build-cloud-proxy-backend build-localai-proxy-backend prepare-e2e run-e2e-image
@echo 'Running e2e tests'
BUILD_TYPE=$(BUILD_TYPE) \
LOCALAI_API=http://$(E2E_BRIDGE_IP):5390 \
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
$(MAKE) clean-mock-backend
$(MAKE) clean-cloud-proxy-backend
$(MAKE) clean-localai-proxy-backend
$(MAKE) teardown-e2e
docker rmi localai-tests
# `docker stop` returns as soon as the container exits, but Docker reaps a
# `--rm` container asynchronously after that. The `docker rmi localai-tests` in
# test-e2e then loses the race against the reaper and fails on a still
# referenced image, turning a green suite red. Removing the container ourselves
# is synchronous, so the image reference is gone before we return. It also
# covers the case where nothing is running, which `docker stop` could not
# because it rejects an empty argument list.
teardown-e2e:
rm -rf $(TEST_DIR) || true
@CONTAINERS=$$(docker ps -aq --filter ancestor=localai-tests 2>/dev/null); \
if [ -n "$$CONTAINERS" ]; then docker rm -f $$CONTAINERS || true; fi
########################################################
## Integration and unit tests
########################################################
## Storage / vector-store integration. Requires the local-store backend to
## be available — we build it on demand and pass its location via
## BACKENDS_PATH (the model loader looks there for the gRPC binary).
test-stores: backends/local-store
BACKENDS_PATH=$(abspath ./)/backends \
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r tests/integration
## Valkey-backed vector-store integration. Requires a running Valkey Search
## server (valkey/valkey-bundle:9.1.0) reachable at $$VALKEY_ADDR — the suite
## skips itself when VALKEY_ADDR is unset. Builds the backend on demand and
## points the model loader at it via BACKENDS_PATH. Label-filtered to the
## valkey specs so it does not also run the in-memory local-store suite.
test-valkey-store: backends/valkey-store
BACKENDS_PATH=$(abspath ./)/backends \
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) --label-filter='valkey' -v -r tests/integration
## Needs a Qdrant gRPC endpoint in $$QDRANT_ADDR, e.g. localhost:6334.
test-qdrant-store: backends/qdrant-store
BACKENDS_PATH=$(abspath ./)/backends \
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) --label-filter='qdrant' -v -r tests/integration
test-opus:
@echo 'Running opus backend tests'
$(MAKE) -C backend/go/opus libopusshim.so
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./backend/go/opus/...
test-opus-docker:
@echo 'Running opus backend tests in Docker'
docker build --target builder \
--build-arg BUILD_TYPE=$(or $(BUILD_TYPE),) \
--build-arg BASE_IMAGE=$(or $(BASE_IMAGE),ubuntu:24.04) \
--build-arg APT_MIRROR=$(APT_MIRROR) \
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
--build-arg BACKEND=opus \
-t localai-opus-test -f backend/Dockerfile.golang .
docker run --rm localai-opus-test \
bash -c 'cd /LocalAI && go run github.com/onsi/ginkgo/v2/ginkgo --flake-attempts $(TEST_FLAKES) -v -r ./backend/go/opus/...'
test-realtime: build-mock-backend
@echo 'Running realtime e2e tests (mock backend)'
$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="Realtime && !real-models" --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
# Verify the realtime state-machine implementations conform to their formal
# designs (Go transition/rapid tests under -race + FizzBee model check of the
# authoritative specs). See docs/design/realtime-state-machines.md (Part 6) and
# docs/design/specs/README.md.
test-realtime-conformance:
GOCMD=$(GOCMD) ./scripts/realtime-conformance.sh
# Verify the shared model-loader shutdown behavior independently of any API
# modality (focused loader/gRPC/distributed/worker tests under -race + FizzBee).
test-model-lifecycle-conformance:
GOCMD=$(GOCMD) ./scripts/model-lifecycle-conformance.sh
# Install the pinned, checksum-verified FizzBee model checker (into .tools/,
# gitignored) used by the conformance targets. Idempotent; no-op if present.
install-fizzbee:
./scripts/install-fizzbee.sh
# Container-based real-model realtime testing. Build env vars / pipeline
# definition kept here so test-realtime-models-docker can drive a fully wired
# pipeline (VAD + STT + LLM + TTS) from inside a containerised runner.
REALTIME_VAD?=silero-vad-ggml
REALTIME_STT?=whisper-1
REALTIME_LLM?=qwen3-0.6b
REALTIME_TTS?=tts-1
# --- Container-based real-model testing ---
REALTIME_BACKEND_NAMES ?= silero-vad whisper llama-cpp kokoro
REALTIME_MODELS_DIR ?= $(abspath ./models)
REALTIME_BACKENDS_DIR ?= $(abspath ./local-backends)
REALTIME_DOCKER_FLAGS ?= --gpus all
local-backends:
mkdir -p local-backends
extract-backend-%: docker-build-% local-backends
@echo "Extracting backend $*..."
@CID=$$(docker create --entrypoint=/run.sh local-ai-backend:$*) && \
rm -rf local-backends/$* && mkdir -p local-backends/$* && \
docker cp $$CID:/ - | tar -xf - -C local-backends/$* && \
docker rm $$CID > /dev/null
extract-realtime-backends: $(addprefix extract-backend-,$(REALTIME_BACKEND_NAMES))
test-realtime-models-docker: build-mock-backend
docker build --target build-requirements \
--build-arg BUILD_TYPE=$(or $(BUILD_TYPE),cublas) \
--build-arg CUDA_MAJOR_VERSION=$(or $(CUDA_MAJOR_VERSION),13) \
--build-arg CUDA_MINOR_VERSION=$(or $(CUDA_MINOR_VERSION),0) \
--build-arg APT_MIRROR=$(APT_MIRROR) \
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
-t localai-test-runner .
docker run --rm \
$(REALTIME_DOCKER_FLAGS) \
-v $(abspath ./):/build \
-v $(REALTIME_MODELS_DIR):/models:ro \
-v $(REALTIME_BACKENDS_DIR):/backends \
-v localai-go-cache:/root/go/pkg/mod \
-v localai-go-build-cache:/root/.cache/go-build \
-e REALTIME_TEST_MODEL=$${REALTIME_TEST_MODEL:-realtime-test-pipeline} \
-e REALTIME_VAD=$(REALTIME_VAD) \
-e REALTIME_STT=$(REALTIME_STT) \
-e REALTIME_LLM=$(REALTIME_LLM) \
-e REALTIME_TTS=$(REALTIME_TTS) \
-e REALTIME_BACKENDS_PATH=/backends \
-e REALTIME_MODELS_PATH=/models \
-w /build \
localai-test-runner \
bash -c 'git config --global --add safe.directory /build && \
make protogen-go && make build-mock-backend && \
go run github.com/onsi/ginkgo/v2/ginkgo --label-filter="Realtime" --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e'
test-container:
docker build --target requirements -t local-ai-test-container .
docker run -ti --rm --entrypoint /bin/bash -ti -v $(abspath ./):/build local-ai-test-container
########################################################
## Help
########################################################
## Help:
help: ## Show this help.
@echo ''
@echo 'Usage:'
@echo ' ${YELLOW}make${RESET} ${GREEN}<target>${RESET}'
@echo ''
@echo 'Targets:'
@awk 'BEGIN {FS = ":.*?## "} { \
if (/^[a-zA-Z_-]+:.*?##.*$$/) {printf " ${YELLOW}%-20s${GREEN}%s${RESET}\n", $$1, $$2} \
else if (/^## .*$$/) {printf " ${CYAN}%s${RESET}\n", substr($$1,4)} \
}' $(MAKEFILE_LIST)
########################################################
## Backends
########################################################
.PHONY: protogen
protogen: protogen-go
protoc:
@OS_NAME=$$(uname -s | tr '[:upper:]' '[:lower:]'); \
ARCH_NAME=$$(uname -m); \
if [ "$$OS_NAME" = "darwin" ]; then \
if [ "$$ARCH_NAME" = "arm64" ]; then \
FILE=protoc-31.1-osx-aarch_64.zip; \
elif [ "$$ARCH_NAME" = "x86_64" ]; then \
FILE=protoc-31.1-osx-x86_64.zip; \
else \
echo "Unsupported macOS architecture: $$ARCH_NAME"; exit 1; \
fi; \
elif [ "$$OS_NAME" = "linux" ]; then \
if [ "$$ARCH_NAME" = "x86_64" ]; then \
FILE=protoc-31.1-linux-x86_64.zip; \
elif [ "$$ARCH_NAME" = "aarch64" ] || [ "$$ARCH_NAME" = "arm64" ]; then \
FILE=protoc-31.1-linux-aarch_64.zip; \
elif [ "$$ARCH_NAME" = "ppc64le" ]; then \
FILE=protoc-31.1-linux-ppcle_64.zip; \
elif [ "$$ARCH_NAME" = "s390x" ]; then \
FILE=protoc-31.1-linux-s390_64.zip; \
elif [ "$$ARCH_NAME" = "i386" ] || [ "$$ARCH_NAME" = "x86" ]; then \
FILE=protoc-31.1-linux-x86_32.zip; \
else \
echo "Unsupported Linux architecture: $$ARCH_NAME"; exit 1; \
fi; \
else \
echo "Unsupported OS: $$OS_NAME"; exit 1; \
fi; \
URL=https://github.com/protocolbuffers/protobuf/releases/download/v31.1/$$FILE; \
curl -fsSL --retry 5 --retry-all-errors --retry-delay 5 $$URL -o protoc.zip && \
unzip -j -d $(CURDIR) protoc.zip bin/protoc && rm protoc.zip
.PHONY: protogen-go
protogen-go: protoc install-go-tools
mkdir -p pkg/grpc/proto
# install-go-tools writes protoc-gen-go and protoc-gen-go-grpc into
# $(shell go env GOPATH)/bin, which isn't on every dev's PATH. protoc
# resolves its code-gen plugins via PATH, so without this prefix the
# generate step fails with "protoc-gen-go: program not found". Prepend
# GOPATH/bin so the freshly-installed plugins win without requiring a
# shell-profile change.
PATH="$$(go env GOPATH)/bin:$$PATH" ./protoc --experimental_allow_proto3_optional -Ibackend/ --go_out=pkg/grpc/proto/ --go_opt=paths=source_relative --go-grpc_out=pkg/grpc/proto/ --go-grpc_opt=paths=source_relative \
backend/backend.proto
PATH="$$(go env GOPATH)/bin:$$PATH" ./protoc -Ipkg/motion/proto --go_out=pkg/motion/proto --go_opt=paths=source_relative pkg/motion/proto/motion.proto
core/config/inference_defaults.json: ## Fetch inference defaults from unsloth (only if missing)
$(GOCMD) generate ./core/config/...
.PHONY: generate
generate: core/config/inference_defaults.json ## Ensure inference defaults exist
.PHONY: generate-force
generate-force: ## Re-fetch inference defaults from unsloth (always)
$(GOCMD) generate ./core/config/...
.PHONY: protogen-go-clean
protogen-go-clean:
$(RM) pkg/grpc/proto/backend.pb.go pkg/grpc/proto/backend_grpc.pb.go pkg/motion/proto/motion.pb.go
$(RM) bin/*
prepare-test-extra: protogen-python
$(MAKE) -C backend/python/transformers
$(MAKE) -C backend/python/outetts
$(MAKE) -C backend/python/diffusers
$(MAKE) -C backend/python/chatterbox
$(MAKE) -C backend/python/vllm
$(MAKE) -C backend/python/vllm-omni
$(MAKE) -C backend/python/longcat-video
$(MAKE) -C backend/python/mlx-video
$(MAKE) -C backend/python/sglang
$(MAKE) -C backend/python/vibevoice
$(MAKE) -C backend/python/moonshine
$(MAKE) -C backend/python/pocket-tts
$(MAKE) -C backend/python/qwen-tts
$(MAKE) -C backend/python/fish-speech
$(MAKE) -C backend/python/faster-qwen3-tts
$(MAKE) -C backend/python/qwen-asr
$(MAKE) -C backend/python/funasr
$(MAKE) -C backend/python/nemo
$(MAKE) -C backend/python/voxcpm
$(MAKE) -C backend/python/faster-whisper
$(MAKE) -C backend/python/whisper-medusa
$(MAKE) -C backend/python/whisperx
$(MAKE) -C backend/python/ace-step
$(MAKE) -C backend/python/trl
$(MAKE) -C backend/python/tinygrad
$(MAKE) -C backend/python/insightface
$(MAKE) -C backend/python/speaker-recognition
$(MAKE) -C backend/rust/kokoros kokoros-grpc
$(MAKE) -C backend/go/rfdetr-cpp
$(MAKE) -C backend/go/locate-anything-cpp
$(MAKE) -C backend/go/timestral-cpp
$(MAKE) -C backend/go/trellis2cpp
$(MAKE) -C backend/go/kimodocpp
$(MAKE) -C backend/go/gemxcpp
$(MAKE) -C backend/go/valkey-store
$(MAKE) -C backend/go/qdrant-store
test-extra: prepare-test-extra
$(MAKE) -C backend/python/transformers test
$(MAKE) -C backend/python/outetts test
$(MAKE) -C backend/python/diffusers test
$(MAKE) -C backend/python/chatterbox test
$(MAKE) -C backend/python/vllm test
$(MAKE) -C backend/python/vllm-omni test
$(MAKE) -C backend/python/longcat-video test
$(MAKE) -C backend/python/mlx-video test
$(MAKE) -C backend/python/vibevoice test
$(MAKE) -C backend/python/moonshine test
$(MAKE) -C backend/python/pocket-tts test
$(MAKE) -C backend/python/qwen-tts test
$(MAKE) -C backend/python/fish-speech test
$(MAKE) -C backend/python/faster-qwen3-tts test
$(MAKE) -C backend/python/qwen-asr test
$(MAKE) -C backend/python/funasr test
$(MAKE) -C backend/python/nemo test
$(MAKE) -C backend/python/voxcpm test
$(MAKE) -C backend/python/faster-whisper test
$(MAKE) -C backend/python/whisper-medusa test
$(MAKE) -C backend/python/whisperx test
$(MAKE) -C backend/python/ace-step test
$(MAKE) -C backend/python/trl test
$(MAKE) -C backend/python/tinygrad test
$(MAKE) -C backend/python/insightface test
$(MAKE) -C backend/python/speaker-recognition test
$(MAKE) -C backend/rust/kokoros test
$(MAKE) -C backend/go/rfdetr-cpp test
$(MAKE) -C backend/go/locate-anything-cpp test
$(MAKE) -C backend/go/depth-anything-cpp test
$(MAKE) -C backend/go/timestral-cpp test
$(MAKE) -C backend/go/supertonic test
$(MAKE) -C backend/go/vllm-cpp test
$(MAKE) -C backend/go/nemo-speech-cpp test
$(MAKE) -C backend/go/trellis2cpp test
$(MAKE) -C backend/go/kimodocpp test
$(MAKE) -C backend/go/gemxcpp test
$(MAKE) -C backend/go/valkey-store test
$(MAKE) -C backend/go/qdrant-store test
##
## End-to-end gRPC tests that exercise a built backend container image.
##
## The test suite in tests/e2e-backends is backend-agnostic. You drive it via env
## vars (see tests/e2e-backends/backend_test.go for the full list) and the
## capability-driven harness picks which gRPC RPCs to exercise:
##
## BACKEND_IMAGE Required. Docker image to test, e.g. local-ai-backend:llama-cpp.
## BACKEND_TEST_MODEL_URL URL of a model file to download and load.
## BACKEND_TEST_MODEL_FILE Path to an already-downloaded model (skips download).
## BACKEND_TEST_MODEL_NAME HuggingFace repo id (e.g. Qwen/Qwen2.5-0.5B-Instruct).
## Use this instead of MODEL_URL for backends that
## resolve HF model ids natively (vllm, vllm-omni).
## BACKEND_TEST_CAPS Comma-separated capabilities, default "health,load,predict,stream".
## Adds "tools" to exercise ChatDelta tool call extraction.
## BACKEND_TEST_PROMPT Override the prompt used in predict/stream specs.
## BACKEND_TEST_OPTIONS Comma-separated Options[] entries forwarded to LoadModel,
## e.g. "tool_parser:hermes,reasoning_parser:qwen3".
## BACKEND_TEST_EMBEDDING_LAYOUT Expected EmbeddingResult layout: "final" or "per_token".
##
## Direct usage (image already built, no docker-build-* dependency):
##
## make test-extra-backend BACKEND_IMAGE=local-ai-backend:llama-cpp \
## BACKEND_TEST_MODEL_URL=https://.../model.gguf
##
## Convenience wrappers below build a specific backend image first, then run the
## suite against it.
##
BACKEND_TEST_MODEL_URL?=https://huggingface.co/Qwen/Qwen3-0.6B-GGUF/resolve/main/Qwen3-0.6B-Q8_0.gguf
## Suite timeout for `go test`. Wrappers whose model download alone can eat
## most of the default (multi-GB models on a slow HF CDN day) override this.
BACKEND_TEST_TIMEOUT?=30m
## Generic target — runs the suite against whatever BACKEND_IMAGE points at.
## Depends on protogen-go so pkg/grpc/proto is generated before `go test`.
test-extra-backend: protogen-go
@test -n "$$BACKEND_IMAGE" || { echo "BACKEND_IMAGE must be set" >&2; exit 1; }
BACKEND_IMAGE="$$BACKEND_IMAGE" \
BACKEND_TEST_MODEL_URL="$${BACKEND_TEST_MODEL_URL:-$(BACKEND_TEST_MODEL_URL)}" \
BACKEND_TEST_MODEL_FILE="$$BACKEND_TEST_MODEL_FILE" \
BACKEND_TEST_MODEL_NAME="$$BACKEND_TEST_MODEL_NAME" \
BACKEND_TEST_MMPROJ_URL="$$BACKEND_TEST_MMPROJ_URL" \
BACKEND_TEST_MMPROJ_FILE="$$BACKEND_TEST_MMPROJ_FILE" \
BACKEND_TEST_AUDIO_URL="$$BACKEND_TEST_AUDIO_URL" \
BACKEND_TEST_AUDIO_FILE="$$BACKEND_TEST_AUDIO_FILE" \
BACKEND_TEST_CAPS="$$BACKEND_TEST_CAPS" \
BACKEND_TEST_PROMPT="$$BACKEND_TEST_PROMPT" \
BACKEND_TEST_OPTIONS="$$BACKEND_TEST_OPTIONS" \
BACKEND_TEST_EMBEDDING_LAYOUT="$$BACKEND_TEST_EMBEDDING_LAYOUT" \
BACKEND_TEST_TOOL_PROMPT="$$BACKEND_TEST_TOOL_PROMPT" \
BACKEND_TEST_TOOL_NAME="$$BACKEND_TEST_TOOL_NAME" \
BACKEND_TEST_CACHE_TYPE_K="$$BACKEND_TEST_CACHE_TYPE_K" \
BACKEND_TEST_CACHE_TYPE_V="$$BACKEND_TEST_CACHE_TYPE_V" \
BACKEND_TEST_FACE_IMAGE_1_URL="$$BACKEND_TEST_FACE_IMAGE_1_URL" \
BACKEND_TEST_FACE_IMAGE_1_FILE="$$BACKEND_TEST_FACE_IMAGE_1_FILE" \
BACKEND_TEST_FACE_IMAGE_2_URL="$$BACKEND_TEST_FACE_IMAGE_2_URL" \
BACKEND_TEST_FACE_IMAGE_2_FILE="$$BACKEND_TEST_FACE_IMAGE_2_FILE" \
BACKEND_TEST_FACE_IMAGE_3_URL="$$BACKEND_TEST_FACE_IMAGE_3_URL" \
BACKEND_TEST_FACE_IMAGE_3_FILE="$$BACKEND_TEST_FACE_IMAGE_3_FILE" \
BACKEND_TEST_VERIFY_DISTANCE_CEILING="$$BACKEND_TEST_VERIFY_DISTANCE_CEILING" \
go test -v -timeout $(BACKEND_TEST_TIMEOUT) ./tests/e2e-backends/...
## Convenience wrappers: build the image, then exercise it.
test-extra-backend-llama-cpp: docker-build-llama-cpp
BACKEND_IMAGE=local-ai-backend:llama-cpp \
BACKEND_TEST_CAPS=health,load,predict,stream,logprobs,logit_bias,context_overflow \
$(MAKE) test-extra-backend
## Raw llama.cpp embeddings are required by Go-side pooling. This exercises the
## real C++ backend and verifies that it marks the flattened matrix per-token.
test-extra-backend-llama-cpp-embeddings: docker-build-llama-cpp
BACKEND_IMAGE=local-ai-backend:llama-cpp \
BACKEND_TEST_CAPS=health,load,embeddings \
BACKEND_TEST_OPTIONS=pooling:none \
BACKEND_TEST_EMBEDDING_LAYOUT=per_token \
$(MAKE) test-extra-backend
test-extra-backend-ik-llama-cpp: docker-build-ik-llama-cpp
BACKEND_IMAGE=local-ai-backend:ik-llama-cpp $(MAKE) test-extra-backend
## turboquant: exercises the llama.cpp-fork backend with the fork's
## *TurboQuant-specific* KV-cache types (turbo3 for both K and V). turbo3
## is what makes this backend distinct from stock llama-cpp — picking q8_0
## here would only test the standard llama.cpp code path that the upstream
## llama-cpp backend already covers. The fork auto-enables flash_attention
## when turbo3/turbo4 are active, so we don't need to set it explicitly.
test-extra-backend-turboquant: docker-build-turboquant
BACKEND_IMAGE=local-ai-backend:turboquant \
BACKEND_TEST_CACHE_TYPE_K=q8_0 \
BACKEND_TEST_CACHE_TYPE_V=turbo3 \
$(MAKE) test-extra-backend
## bonsai: exercises the llama.cpp-fork backend with a real Q1_0 (1-bit) model —
## the PrismML Bonsai-8B GGUF, whose weight quant is *only* decodable by the fork's
## Q1_0 kernels. Loading it is what makes this backend distinct from stock llama-cpp;
## a standard-quant model would only test the upstream code path the llama-cpp backend
## already covers.
test-extra-backend-bonsai: docker-build-bonsai
BACKEND_IMAGE=local-ai-backend:bonsai \
BACKEND_TEST_MODEL_URL=https://huggingface.co/prism-ml/Bonsai-8B-gguf/resolve/main/Bonsai-8B-Q1_0.gguf \
$(MAKE) test-extra-backend
## Audio transcription wrapper for the llama-cpp backend.
## Drives the new AudioTranscription / AudioTranscriptionStream RPCs against
## ggml-org/Qwen3-ASR-0.6B-GGUF (a small ASR model that requires its mmproj
## audio encoder companion). The audio fixture is a short public-domain
## "jfk.wav" clip ggml-org bundles with whisper.cpp's CI assets.
test-extra-backend-llama-cpp-transcription: docker-build-llama-cpp
BACKEND_IMAGE=local-ai-backend:llama-cpp \
BACKEND_TEST_MODEL_URL=https://huggingface.co/ggml-org/Qwen3-ASR-0.6B-GGUF/resolve/main/Qwen3-ASR-0.6B-Q8_0.gguf \
BACKEND_TEST_MMPROJ_URL=https://huggingface.co/ggml-org/Qwen3-ASR-0.6B-GGUF/resolve/main/mmproj-Qwen3-ASR-0.6B-Q8_0.gguf \
BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
BACKEND_TEST_CAPS=health,load,transcription \
BACKEND_TEST_CTX_SIZE=2048 \
$(MAKE) test-extra-backend
## LFM3-Audio (LiquidAI) through the llama-cpp backend: a named-voice TTS
## (no speaker reference), the LFM3 ASR preset, and the tts/ttsstream/transcription
## s2s specs. Real weights, so it is a manual run and not part of CI. CPU is enough.
test-extra-backend-llama-cpp-lfm3: docker-build-llama-cpp
BACKEND_IMAGE=local-ai-backend:llama-cpp \
BACKEND_TEST_MODEL_URL=https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/main/LFM3-Audio-2B-Q4_K_M.gguf \
BACKEND_TEST_MMPROJ_URL=https://huggingface.co/LiquidAI/LFM3-Audio-2B-GGUF/resolve/main/mmproj-LFM3-Audio-2B-Q8_0.gguf \
BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
BACKEND_TEST_CAPS=health,load,predict,tts,ttsstream,transcription,s2s \
BACKEND_TEST_CTX_SIZE=4096 \
BACKEND_TEST_TTS_TEXT="Hello from LocalAI." \
BACKEND_TEST_TTS_MIN_SECONDS=0.3 \
BACKEND_TEST_TTS_MAX_SECONDS=15 \
BACKEND_TEST_TTS_MIN_RMS=100 \
BACKEND_TEST_TRANSCRIPT_CONTAINS=country \
BACKEND_TEST_OPTIONS=parallel:2 \
$(MAKE) test-extra-backend
## Real-weights realtime e2e: LFM3-Audio on llama-cpp behind /v1/realtime, as
## an any-to-any model and as a chained pipeline, each with and without a VAD
## (LFM3_MATRIX picks the shapes). The client speaks generated audio (sherpa
## VITS) and a separate sherpa ASR checks the reply audio. MANUAL: not run in CI
## (the CI variant is `make test-realtime`, which uses the mock backend and the
## `!real-models` label filter). A GPU is recommended. Build the llama-cpp image
## for your hardware first. The host needs ffmpeg on PATH (input audio
## conversion). LFM3_VARIANT=q8 selects lfm3-audio-2b-q8.
test-realtime-s2s-real: build-mock-backend docker-build-llama-cpp docker-build-sherpa-onnx protogen-go
bash tests/e2e/run-realtime-lfm3.sh
## Same, without rebuilding the backend images (they must already exist).
test-realtime-s2s-real-nobuild: build-mock-backend protogen-go
bash tests/e2e/run-realtime-lfm3.sh
## privacy-filter: the PII/NER token-classification backend. Exercises the
## TokenClassify RPC and asserts byte-correct, UTF-8-aligned span offsets
## against the openai-privacy-filter multilingual GGUF (CPU-runnable, ~50M
## active params). This is the live-backend coverage for the PII NER tier.
test-extra-backend-privacy-filter: docker-build-privacy-filter
BACKEND_IMAGE=local-ai-backend:privacy-filter \
BACKEND_TEST_MODEL_URL=https://huggingface.co/LocalAI-io/privacy-filter-multilingual-GGUF/resolve/main/privacy-filter-multilingual-f16.gguf \
BACKEND_TEST_CAPS=health,load,token_classify \
$(MAKE) test-extra-backend
## vllm is resolved from a HuggingFace model id (no file download) and
## exercises Predict + streaming + tool-call extraction via the hermes parser.
## Requires a host CPU with the SIMD instructions the prebuilt vllm CPU
## wheel was compiled against (AVX-512 VNNI/BF16); older CPUs will SIGILL
## on import — on CI this means using the bigger-runner label.
test-extra-backend-vllm: docker-build-vllm
BACKEND_IMAGE=local-ai-backend:vllm \
BACKEND_TEST_MODEL_NAME=Qwen/Qwen2.5-0.5B-Instruct \
BACKEND_TEST_CAPS=health,load,predict,stream,tools \
BACKEND_TEST_OPTIONS=tool_parser:hermes \
$(MAKE) test-extra-backend
## vllm multi-node data-parallel smoke test. Runs LocalAI head + a
## `local-ai p2p-worker vllm` follower in docker compose against
## Qwen2.5-0.5B with data_parallel_size=2. Requires 2 NVIDIA GPUs and
## nvidia-container-runtime on the host — vLLM v1's DP coordinator is
## not viable on CPU so this cannot run in CI without GPU.
test-extra-backend-vllm-multinode:
./tests/e2e/vllm-multinode/smoke.sh
## tinygrad mirrors the vllm target (same model, same caps, same parser) so
## the two backends are directly comparable. The LLM path covers Predict,
## streaming and native tool-call extraction. Companion targets below cover
## embeddings, Stable Diffusion and Whisper — run them individually or via
## the `test-extra-backend-tinygrad-all` aggregate.
test-extra-backend-tinygrad: docker-build-tinygrad
BACKEND_IMAGE=local-ai-backend:tinygrad \
BACKEND_TEST_MODEL_NAME=Qwen/Qwen3-0.6B \
BACKEND_TEST_CAPS=health,load,predict,stream,tools \
BACKEND_TEST_OPTIONS=tool_parser:hermes \
$(MAKE) test-extra-backend
## tinygrad — embeddings via LLM last-hidden-state pooling. Reuses the same
## Qwen3-0.6B as the chat target so we don't need a separate BERT vendor;
## the Embedding RPC mean-pools and L2-normalizes the last-layer hidden
## state.
test-extra-backend-tinygrad-embeddings: docker-build-tinygrad
BACKEND_IMAGE=local-ai-backend:tinygrad \
BACKEND_TEST_MODEL_NAME=Qwen/Qwen3-0.6B \
BACKEND_TEST_CAPS=health,load,embeddings \
BACKEND_TEST_EMBEDDING_LAYOUT=final \
$(MAKE) test-extra-backend
## tinygrad — Stable Diffusion 1.5. The original CompVis/runwayml repos have
## been gated, so we use the community-maintained mirror at
## stable-diffusion-v1-5/stable-diffusion-v1-5 with the EMA-only pruned
## checkpoint (~4.3GB). Step count is kept low (4) so a CPU-only run finishes
## in a few minutes; bump BACKEND_TEST_IMAGE_STEPS for higher quality.
test-extra-backend-tinygrad-sd: docker-build-tinygrad
BACKEND_IMAGE=local-ai-backend:tinygrad \
BACKEND_TEST_MODEL_NAME=stable-diffusion-v1-5/stable-diffusion-v1-5 \
BACKEND_TEST_CAPS=health,load,image \
$(MAKE) test-extra-backend
## tinygrad — Whisper. Loads OpenAI's tiny.en checkpoint (smallest at ~75MB)
## from the original azure CDN through tinygrad's `fetch` helper, and
## transcribes the canonical jfk.wav fixture from whisper.cpp's CI samples.
## Exercises both AudioTranscription and AudioTranscriptionStream.
test-extra-backend-tinygrad-whisper: docker-build-tinygrad
BACKEND_IMAGE=local-ai-backend:tinygrad \
BACKEND_TEST_MODEL_NAME=openai/whisper-tiny.en \
BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
BACKEND_TEST_CAPS=health,load,transcription \
$(MAKE) test-extra-backend
test-extra-backend-tinygrad-all: \
test-extra-backend-tinygrad \
test-extra-backend-tinygrad-embeddings \
test-extra-backend-tinygrad-sd \
test-extra-backend-tinygrad-whisper
## insightface — face recognition.
##
## Face fixtures default to the sample images shipped in the
## deepinsight/insightface repository (MIT-licensed). For offline/local
## runs override with BACKEND_TEST_FACE_IMAGE_{1,2,3}_FILE pointing at
## local paths.
FACE_IMAGE_1_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/t1.jpg
FACE_IMAGE_2_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/t1.jpg
FACE_IMAGE_3_URL ?= https://github.com/deepinsight/insightface/raw/master/python-package/insightface/data/images/mask_white.jpg
## Known spoof fixture used by the face_antispoof e2e cap. This is
## upstream's own `image_F2.jpg` (Silent-Face repo, via yakhyo mirror)
## — verified to classify as is_real=false with score < 0.05 on the
## MiniFASNetV2 + MiniFASNetV1SE ensemble.
FACE_SPOOF_IMAGE_URL ?= https://github.com/yakhyo/face-anti-spoofing/raw/main/assets/image_F2.jpg
## Host-side cache for the OpenCV Zoo face ONNX files used by the
## opencv e2e target. The backend image no longer bakes model weights —
## gallery installs bring them via `files:` — but the e2e suite drives
## LoadModel over gRPC directly without going through the gallery. We
## pre-download the ONNX files to a stable host path and pass absolute
## paths in BACKEND_TEST_OPTIONS; `make` skips the downloads when the
## SHA-256 already matches.
INSIGHTFACE_OPENCV_DIR := /tmp/localai-insightface-opencv-cache
INSIGHTFACE_OPENCV_YUNET_URL := https://github.com/opencv/opencv_zoo/raw/main/models/face_detection_yunet/face_detection_yunet_2023mar.onnx
INSIGHTFACE_OPENCV_SFACE_URL := https://github.com/opencv/opencv_zoo/raw/main/models/face_recognition_sface/face_recognition_sface_2021dec.onnx
INSIGHTFACE_OPENCV_YUNET_SHA := 8f2383e4dd3cfbb4553ea8718107fc0423210dc964f9f4280604804ed2552fa4
INSIGHTFACE_OPENCV_SFACE_SHA := 0ba9fbfa01b5270c96627c4ef784da859931e02f04419c829e83484087c34e79
## buffalo_sc (insightface) — pack zip + SHA-256 mirrors the gallery
## entry so the e2e target matches exactly what `local-ai models install
## insightface-buffalo-sc` would have fetched. Smallest insightface pack
## (~16MB) — keeps CI fast while still covering the insightface engine
## code path end-to-end.
INSIGHTFACE_BUFFALO_SC_DIR := /tmp/localai-insightface-buffalo-sc-cache
INSIGHTFACE_BUFFALO_SC_URL := https://github.com/deepinsight/insightface/releases/download/v0.7/buffalo_sc.zip
INSIGHTFACE_BUFFALO_SC_SHA := 57d31b56b6ffa911c8a73cfc1707c73cab76efe7f13b675a05223bf42de47c72
## Silent-Face antispoofing (MiniFASNetV2 + MiniFASNetV1SE) — shared
## between the buffalo_sc and opencv e2e targets. Both ONNX files are
## ~1.7MB, Apache 2.0. URLs + SHAs mirror the gallery entries.
INSIGHTFACE_ANTISPOOF_DIR := /tmp/localai-insightface-antispoof-cache
INSIGHTFACE_ANTISPOOF_V2_URL := https://github.com/yakhyo/face-anti-spoofing/releases/download/weights/MiniFASNetV2.onnx
INSIGHTFACE_ANTISPOOF_V2_SHA := b32929adc2d9c34b9486f8c4c7bc97c1b69bc0ea9befefc380e4faae4e463907
INSIGHTFACE_ANTISPOOF_V1SE_URL := https://github.com/yakhyo/face-anti-spoofing/releases/download/weights/MiniFASNetV1SE.onnx
INSIGHTFACE_ANTISPOOF_V1SE_SHA := ebab7f90c7833fbccd46d3a555410e78d969db5438e169b6524be444862b3676
.PHONY: insightface-opencv-models
insightface-opencv-models:
@mkdir -p $(INSIGHTFACE_OPENCV_DIR)
@if [ "$$(sha256sum $(INSIGHTFACE_OPENCV_DIR)/yunet.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_OPENCV_YUNET_SHA)" ]; then \
echo "Fetching YuNet..."; \
curl -fsSL -o $(INSIGHTFACE_OPENCV_DIR)/yunet.onnx $(INSIGHTFACE_OPENCV_YUNET_URL); \
echo "$(INSIGHTFACE_OPENCV_YUNET_SHA) $(INSIGHTFACE_OPENCV_DIR)/yunet.onnx" | sha256sum -c; \
fi
@if [ "$$(sha256sum $(INSIGHTFACE_OPENCV_DIR)/sface.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_OPENCV_SFACE_SHA)" ]; then \
echo "Fetching SFace..."; \
curl -fsSL -o $(INSIGHTFACE_OPENCV_DIR)/sface.onnx $(INSIGHTFACE_OPENCV_SFACE_URL); \
echo "$(INSIGHTFACE_OPENCV_SFACE_SHA) $(INSIGHTFACE_OPENCV_DIR)/sface.onnx" | sha256sum -c; \
fi
.PHONY: insightface-antispoof-models
insightface-antispoof-models:
@mkdir -p $(INSIGHTFACE_ANTISPOOF_DIR)
@if [ "$$(sha256sum $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_ANTISPOOF_V2_SHA)" ]; then \
echo "Fetching MiniFASNetV2..."; \
curl -fsSL -o $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx $(INSIGHTFACE_ANTISPOOF_V2_URL); \
echo "$(INSIGHTFACE_ANTISPOOF_V2_SHA) $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx" | sha256sum -c; \
fi
@if [ "$$(sha256sum $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_ANTISPOOF_V1SE_SHA)" ]; then \
echo "Fetching MiniFASNetV1SE..."; \
curl -fsSL -o $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx $(INSIGHTFACE_ANTISPOOF_V1SE_URL); \
echo "$(INSIGHTFACE_ANTISPOOF_V1SE_SHA) $(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx" | sha256sum -c; \
fi
.PHONY: insightface-buffalo-sc-models
insightface-buffalo-sc-models:
@mkdir -p $(INSIGHTFACE_BUFFALO_SC_DIR)
@if [ "$$(sha256sum $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip 2>/dev/null | awk '{print $$1}')" != "$(INSIGHTFACE_BUFFALO_SC_SHA)" ]; then \
echo "Fetching buffalo_sc..."; \
curl -fsSL -o $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip $(INSIGHTFACE_BUFFALO_SC_URL); \
echo "$(INSIGHTFACE_BUFFALO_SC_SHA) $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip" | sha256sum -c; \
rm -f $(INSIGHTFACE_BUFFALO_SC_DIR)/*.onnx; \
fi
@if [ ! -f "$(INSIGHTFACE_BUFFALO_SC_DIR)/det_500m.onnx" ]; then \
echo "Extracting buffalo_sc..."; \
unzip -o -q $(INSIGHTFACE_BUFFALO_SC_DIR)/buffalo_sc.zip -d $(INSIGHTFACE_BUFFALO_SC_DIR); \
fi
## buffalo_sc — smallest insightface pack (SCRFD-500MF detector + MBF
## recognizer, ~16MB). Exercises the insightface engine code path
## (model_zoo-backed inference) without the ~326MB buffalo_l download.
## No age/gender/landmark heads — face_analyze is dropped from caps.
## The pack is pre-fetched on the host and passed as `root:<dir>` since
## the e2e suite drives LoadModel directly without going through
## LocalAI's gallery flow (which is what would normally populate
## ModelPath and in turn the engine's `_model_dir` option).
test-extra-backend-insightface-buffalo-sc: docker-build-insightface insightface-buffalo-sc-models insightface-antispoof-models
BACKEND_IMAGE=local-ai-backend:insightface \
BACKEND_TEST_MODEL_NAME=insightface-buffalo-sc \
BACKEND_TEST_OPTIONS=engine:insightface,model_pack:buffalo_sc,root:$(INSIGHTFACE_BUFFALO_SC_DIR),antispoof_v2_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx,antispoof_v1se_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx \
BACKEND_TEST_CAPS=health,load,face_detect,face_embed,face_verify,face_antispoof \
BACKEND_TEST_FACE_IMAGE_1_URL=$(FACE_IMAGE_1_URL) \
BACKEND_TEST_FACE_IMAGE_2_URL=$(FACE_IMAGE_2_URL) \
BACKEND_TEST_FACE_IMAGE_3_URL=$(FACE_IMAGE_3_URL) \
BACKEND_TEST_FACE_SPOOF_IMAGE_URL=$(FACE_SPOOF_IMAGE_URL) \
BACKEND_TEST_VERIFY_DISTANCE_CEILING=0.55 \
$(MAKE) test-extra-backend
## OpenCV Zoo YuNet + SFace — Apache 2.0, commercial-safe. face_analyze
## cap is dropped (SFace has no demographic head). The ONNX files are
## pre-fetched on the host via the insightface-opencv-models target and
## passed as absolute paths, since the e2e suite drives LoadModel
## directly without going through LocalAI's gallery flow.
test-extra-backend-insightface-opencv: docker-build-insightface insightface-opencv-models insightface-antispoof-models
BACKEND_IMAGE=local-ai-backend:insightface \
BACKEND_TEST_MODEL_NAME=insightface-opencv \
BACKEND_TEST_OPTIONS=engine:onnx_direct,detector_onnx:$(INSIGHTFACE_OPENCV_DIR)/yunet.onnx,recognizer_onnx:$(INSIGHTFACE_OPENCV_DIR)/sface.onnx,antispoof_v2_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV2.onnx,antispoof_v1se_onnx:$(INSIGHTFACE_ANTISPOOF_DIR)/MiniFASNetV1SE.onnx \
BACKEND_TEST_CAPS=health,load,face_detect,face_embed,face_verify,face_antispoof \
BACKEND_TEST_FACE_IMAGE_1_URL=$(FACE_IMAGE_1_URL) \
BACKEND_TEST_FACE_IMAGE_2_URL=$(FACE_IMAGE_2_URL) \
BACKEND_TEST_FACE_IMAGE_3_URL=$(FACE_IMAGE_3_URL) \
BACKEND_TEST_FACE_SPOOF_IMAGE_URL=$(FACE_SPOOF_IMAGE_URL) \
BACKEND_TEST_VERIFY_DISTANCE_CEILING=0.55 \
$(MAKE) test-extra-backend
## Aggregate — runs both face-recognition model configurations so CI
## catches regressions across engines together.
test-extra-backend-insightface-all: \
test-extra-backend-insightface-buffalo-sc \
test-extra-backend-insightface-opencv
## speaker-recognition — voice (speaker) biometrics.
##
## Audio fixtures default to the speechbrain test samples served
## straight from their GitHub repo — public, no auth needed, and they
## ship as 16kHz mono WAV/FLAC which is exactly what the engine wants.
## example{1,2,5} are three different speakers; the suite treats
## example1 as the "same-image twin" probe (verify(clip, clip) must
## return distance≈0) and the other two as cross-speaker ceilings.
## Override with BACKEND_TEST_VOICE_AUDIO_{1,2,3}_FILE for offline runs.
VOICE_AUDIO_1_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example1.wav
VOICE_AUDIO_2_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example2.flac
VOICE_AUDIO_3_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example5.wav
## ECAPA-TDNN via SpeechBrain — default CI configuration. Auto-downloads
## the checkpoint from HuggingFace on first LoadModel (bundled in the
## backend image pip install). 192-d embeddings, cosine-distance based.
## The e2e suite drives LoadModel directly so we don't rely on LocalAI's
## gallery flow here.
test-extra-backend-speaker-recognition-ecapa: docker-build-speaker-recognition
BACKEND_IMAGE=local-ai-backend:speaker-recognition \
BACKEND_TEST_MODEL_NAME=speechbrain/spkrec-ecapa-voxceleb \
BACKEND_TEST_OPTIONS=engine:speechbrain,source:speechbrain/spkrec-ecapa-voxceleb \
BACKEND_TEST_CAPS=health,load,voice_embed,voice_verify \
BACKEND_TEST_VOICE_AUDIO_1_URL=$(VOICE_AUDIO_1_URL) \
BACKEND_TEST_VOICE_AUDIO_2_URL=$(VOICE_AUDIO_2_URL) \
BACKEND_TEST_VOICE_AUDIO_3_URL=$(VOICE_AUDIO_3_URL) \
BACKEND_TEST_VOICE_VERIFY_DISTANCE_CEILING=0.4 \
$(MAKE) test-extra-backend
## Aggregate — today there's only one voice config; the target exists
## so the CI workflow matches the insightface-all naming convention and
## can grow to include WeSpeaker / 3D-Speaker later.
test-extra-backend-speaker-recognition-all: \
test-extra-backend-speaker-recognition-ecapa
## Realtime e2e with sherpa-onnx driving VAD + STT + TTS against a mocked
## LLM. Extracts the sherpa-onnx Docker image rootfs, downloads the three
## gallery-referenced model bundles (silero-vad, omnilingual-asr, vits-ljs),
## writes the corresponding model config YAMLs, and runs the realtime
## websocket spec in tests/e2e with REALTIME_* env vars wiring the sherpa
## slots into the pipeline. The LLM slot stays on the in-repo mock-backend
## registered unconditionally by tests/e2e/e2e_suite_test.go. See
## tests/e2e/run-realtime-sherpa.sh for the full orchestration.
test-extra-e2e-realtime-sherpa: build-mock-backend docker-build-sherpa-onnx protogen-go react-ui
bash tests/e2e/run-realtime-sherpa.sh
## Streaming ASR via the sherpa-onnx online recognizer. Uses the streaming
## zipformer English model (encoder/decoder/joiner int8 + tokens) from the
## sherpa-onnx gallery entry. Drives both AudioTranscription and
## AudioTranscriptionStream via the e2e-backends gRPC harness; streaming
## emits real partial deltas during decode. Each file is renamed on download
## to the shape sherpa-onnx's online loader expects (encoder.int8.onnx etc.).
test-extra-backend-sherpa-onnx-transcription: docker-build-sherpa-onnx
BACKEND_IMAGE=local-ai-backend:sherpa-onnx \
BACKEND_TEST_MODEL_URL='https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/encoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx#encoder.int8.onnx' \
BACKEND_TEST_EXTRA_FILES='https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/decoder-epoch-99-avg-1-chunk-16-left-128.int8.onnx#decoder.int8.onnx|https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/joiner-epoch-99-avg-1-chunk-16-left-128.int8.onnx#joiner.int8.onnx|https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-en-2023-06-26/resolve/main/tokens.txt' \
BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
BACKEND_TEST_CAPS=health,load,transcription \
BACKEND_TEST_OPTIONS=subtype=online \
$(MAKE) test-extra-backend
## VITS TTS via the sherpa-onnx backend. Pulls the individual files from
## HuggingFace (the vits-ljs release tarball lives on the k2-fsa github
## but is also mirrored as discrete files on HF). Exercises both
## TTS (write-to-file) and TTSStream (PCM chunks + WAV header) via the
## e2e-backends gRPC harness.
test-extra-backend-sherpa-onnx-tts: docker-build-sherpa-onnx
BACKEND_IMAGE=local-ai-backend:sherpa-onnx \
BACKEND_TEST_MODEL_URL='https://huggingface.co/csukuangfj/vits-ljs/resolve/main/vits-ljs.onnx#vits-ljs.onnx' \
BACKEND_TEST_EXTRA_FILES='https://huggingface.co/csukuangfj/vits-ljs/resolve/main/tokens.txt|https://huggingface.co/csukuangfj/vits-ljs/resolve/main/lexicon.txt' \
BACKEND_TEST_CAPS=health,load,tts \
$(MAKE) test-extra-backend
## VibeVoice TTS via the vibevoice-cpp backend. ModelFile is the
## realtime gguf; the supplementary tokenizer + voice prompt land
## alongside it under the harness's models dir and are wired through
## via the standard Options[] convention (tokenizer=, voice=).
test-extra-backend-vibevoice-cpp-tts: docker-build-vibevoice-cpp
BACKEND_IMAGE=local-ai-backend:vibevoice-cpp \
BACKEND_TEST_MODEL_URL='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/vibevoice-realtime-0.5B-q8_0.gguf#vibevoice-realtime-0.5B-q8_0.gguf' \
BACKEND_TEST_EXTRA_FILES='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/tokenizer.gguf#tokenizer.gguf|https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/voice-en-Carter_man.gguf#voice-en-Carter_man.gguf' \
BACKEND_TEST_OPTIONS=tokenizer:tokenizer.gguf,voice:voice-en-Carter_man.gguf \
BACKEND_TEST_CAPS=health,load,tts \
$(MAKE) test-extra-backend
## VibeVoice ASR (long-form, with diarization). type=asr tells the
## backend's Load() to slot ModelFile into the asr_model role; the
## tokenizer is supplied via Options[]. Uses the Q4_K quant (~10 GB)
## rather than Q8_0 (~14 GB) so the bundle fits inside ubuntu-latest's
## post-image disk budget.
test-extra-backend-vibevoice-cpp-transcription: docker-build-vibevoice-cpp
BACKEND_IMAGE=local-ai-backend:vibevoice-cpp \
BACKEND_TEST_TIMEOUT=120m \
BACKEND_TEST_MODEL_URL='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/vibevoice-asr-q4_k.gguf#vibevoice-asr-q4_k.gguf' \
BACKEND_TEST_EXTRA_FILES='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/tokenizer.gguf#tokenizer.gguf' \
BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
BACKEND_TEST_OPTIONS=type:asr,tokenizer:tokenizer.gguf \
BACKEND_TEST_CAPS=health,load,transcription \
$(MAKE) test-extra-backend
## Audio transcription wrapper for the whisper.cpp backend.
## Drives the AudioTranscription / AudioTranscriptionStream RPCs against
## ggml-base.en (~145 MB) using the JFK 11s clip. The streaming spec
## asserts len(deltas) >= 1 and concat(deltas) == final.Text - whisper-
## specific multi-segment assertions live in backend/go/whisper/gowhisper_test.go.
test-extra-backend-whisper-transcription: docker-build-whisper
BACKEND_IMAGE=local-ai-backend:whisper \
BACKEND_TEST_MODEL_URL=https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin \
BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
BACKEND_TEST_CAPS=health,load,transcription \
$(MAKE) test-extra-backend
## Audio transcription wrapper for the parakeet-cpp (parakeet.cpp ggml port)
## backend. Mirrors test-extra-backend-whisper-transcription: drives the
## AudioTranscription / AudioTranscriptionStream RPCs against a published
## Parakeet GGUF using the JFK 11s clip from whisper.cpp's CI samples. Not
## part of the default test suite - run explicitly once the pinned model URL
## is reachable.
test-extra-backend-parakeet-cpp-transcription: docker-build-parakeet-cpp
BACKEND_IMAGE=local-ai-backend:parakeet-cpp \
BACKEND_TEST_MODEL_URL=https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/realtime_eou_120m-v1-f16.gguf \
BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
BACKEND_TEST_CAPS=health,load,transcription \
$(MAKE) test-extra-backend
## Audio transcription wrapper for the moss-transcribe-cpp (moss-transcribe.cpp
## ggml port) backend. Mirrors test-extra-backend-parakeet-cpp-transcription:
## drives the AudioTranscription RPC against a published MOSS GGUF using the JFK
## 11s clip from whisper.cpp's CI samples. Not part of the default test suite -
## run explicitly once the pinned model URL is reachable.
test-extra-backend-moss-transcribe-cpp-transcription: docker-build-moss-transcribe-cpp
BACKEND_IMAGE=local-ai-backend:moss-transcribe-cpp \
BACKEND_TEST_MODEL_URL=https://huggingface.co/mudler/moss-transcribe.cpp-gguf/resolve/main/moss-transcribe-q5_k.gguf \
BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
BACKEND_TEST_CAPS=health,load,transcription \
$(MAKE) test-extra-backend
## LocalVQE audio transform (joint AEC + noise suppression + dereverb).
## Exercises the audio_transform capability end-to-end: batch transform
## of a real WAV fixture and bidi streaming of synthetic silent frames.
test-extra-backend-localvqe-transform: docker-build-localvqe
BACKEND_IMAGE=local-ai-backend:localvqe \
BACKEND_TEST_MODEL_URL='https://huggingface.co/LocalAI-io/LocalVQE/resolve/main/localvqe-v1-1.3M-f32.gguf#localvqe-v1-1.3M-f32.gguf' \
BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
BACKEND_TEST_CAPS=health,load,audio_transform \
$(MAKE) test-extra-backend
## sglang mirrors the vllm setup: HuggingFace model id, same tiny Qwen,
## tool-call extraction via sglang's native qwen parser. CPU builds use
## sglang's upstream pyproject_cpu.toml recipe (see backend/python/sglang/install.sh).
test-extra-backend-sglang: docker-build-sglang
BACKEND_IMAGE=local-ai-backend:sglang \
BACKEND_TEST_MODEL_NAME=Qwen/Qwen2.5-0.5B-Instruct \
BACKEND_TEST_CAPS=health,load,predict,stream,tools \
BACKEND_TEST_OPTIONS=tool_parser:qwen \
$(MAKE) test-extra-backend
## mlx is Apple-Silicon-first — the MLX backend auto-detects the right tool
## parser from the chat template, so no tool_parser: option is needed (it
## would be ignored at runtime). Run this on macOS / arm64 with Metal; the
## Linux/CPU mlx variant is untested in CI.
test-extra-backend-mlx: docker-build-mlx
BACKEND_IMAGE=local-ai-backend:mlx \
BACKEND_TEST_MODEL_NAME=mlx-community/Qwen2.5-0.5B-Instruct-4bit \
BACKEND_TEST_CAPS=health,load,predict,stream,tools \
$(MAKE) test-extra-backend
test-extra-backend-mlx-vlm: docker-build-mlx-vlm
BACKEND_IMAGE=local-ai-backend:mlx-vlm \
BACKEND_TEST_MODEL_NAME=mlx-community/Qwen2.5-0.5B-Instruct-4bit \
BACKEND_TEST_CAPS=health,load,predict,stream,tools \
$(MAKE) test-extra-backend
DOCKER_IMAGE?=local-ai
IMAGE_TYPE?=core
BASE_IMAGE?=ubuntu:24.04
docker:
docker build \
--build-arg BASE_IMAGE=$(BASE_IMAGE) \
--build-arg IMAGE_TYPE=$(IMAGE_TYPE) \
--build-arg GO_TAGS="$(GO_TAGS)" \
--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
--build-arg BUILD_TYPE=$(BUILD_TYPE) \
--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
--build-arg APT_MIRROR=$(APT_MIRROR) \
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
-t $(DOCKER_IMAGE) .
docker-cuda12:
docker build \
--build-arg CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
--build-arg CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
--build-arg BASE_IMAGE=$(BASE_IMAGE) \
--build-arg IMAGE_TYPE=$(IMAGE_TYPE) \
--build-arg GO_TAGS="$(GO_TAGS)" \
--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
--build-arg BUILD_TYPE=$(BUILD_TYPE) \
--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
--build-arg APT_MIRROR=$(APT_MIRROR) \
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
-t $(DOCKER_IMAGE)-cuda-12 .
docker-image-intel:
docker build \
--build-arg BASE_IMAGE=intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04 \
--build-arg IMAGE_TYPE=$(IMAGE_TYPE) \
--build-arg GO_TAGS="$(GO_TAGS)" \
--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
--build-arg BUILD_TYPE=intel \
--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
--build-arg APT_MIRROR=$(APT_MIRROR) \
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
-t $(DOCKER_IMAGE) .
########################################################
## Backends
########################################################
# Pattern rule for standard backends (docker-based)
# This matches all backends that use docker-build-* and docker-save-*
backends/%: docker-build-% docker-save-% build
./local-ai backends install "ocifile://$(abspath ./backend-images/$*.tar)"
# Darwin-specific backends (keep as explicit targets since they have special build logic)
backends/llama-cpp-darwin: build
bash ./scripts/build/llama-cpp-darwin.sh
./local-ai backends install "ocifile://$(abspath ./backend-images/llama-cpp.tar)"
backends/ds4-darwin: build
bash ./scripts/build/ds4-darwin.sh
./local-ai backends install "ocifile://$(abspath ./backend-images/ds4.tar)"
backends/privacy-filter-darwin: build
bash ./scripts/build/privacy-filter-darwin.sh
./local-ai backends install "ocifile://$(abspath ./backend-images/privacy-filter.tar)"
backends/audio-cpp-darwin: build
bash ./scripts/build/audio-cpp-darwin.sh
./local-ai backends install "ocifile://$(abspath ./backend-images/audio-cpp.tar)"
build-darwin-python-backend: build
bash ./scripts/build/python-darwin.sh
build-darwin-go-backend: build
bash ./scripts/build/golang-darwin.sh
backends/mlx:
BACKEND=mlx $(MAKE) build-darwin-python-backend
./local-ai backends install "ocifile://$(abspath ./backend-images/mlx.tar)"
backends/mlx-video:
BACKEND=mlx-video $(MAKE) build-darwin-python-backend
./local-ai backends install "ocifile://$(abspath ./backend-images/mlx-video.tar)"
backends/diffuser-darwin:
BACKEND=diffusers $(MAKE) build-darwin-python-backend
./local-ai backends install "ocifile://$(abspath ./backend-images/diffusers.tar)"
backends/mlx-vlm:
BACKEND=mlx-vlm $(MAKE) build-darwin-python-backend
./local-ai backends install "ocifile://$(abspath ./backend-images/mlx-vlm.tar)"
backends/mlx-audio:
BACKEND=mlx-audio $(MAKE) build-darwin-python-backend
./local-ai backends install "ocifile://$(abspath ./backend-images/mlx-audio.tar)"
backends/mlx-distributed:
BACKEND=mlx-distributed $(MAKE) build-darwin-python-backend
./local-ai backends install "ocifile://$(abspath ./backend-images/mlx-distributed.tar)"
backends/stablediffusion-ggml-darwin:
BACKEND=stablediffusion-ggml BUILD_TYPE=metal $(MAKE) build-darwin-go-backend
./local-ai backends install "ocifile://$(abspath ./backend-images/stablediffusion-ggml.tar)"
backends/trellis2cpp-darwin:
BACKEND=trellis2cpp BUILD_TYPE=metal $(MAKE) build-darwin-go-backend
./local-ai backends install "ocifile://$(abspath ./backend-images/trellis2cpp.tar)"
backend-images:
mkdir -p backend-images
# Backend metadata: BACKEND_NAME | DOCKERFILE_TYPE | BUILD_CONTEXT | PROGRESS_FLAG | NEEDS_BACKEND_ARG
# llama-cpp is special - uses llama-cpp Dockerfile and doesn't need BACKEND arg
BACKEND_LLAMA_CPP = llama-cpp|llama-cpp|.|false|false
# ik-llama-cpp is a fork of llama.cpp with superior CPU performance
BACKEND_IK_LLAMA_CPP = ik-llama-cpp|ik-llama-cpp|.|false|false
# turboquant is a llama.cpp fork with TurboQuant KV-cache quantization.
# Reuses backend/cpp/llama-cpp grpc-server sources via a thin wrapper Makefile.
BACKEND_TURBOQUANT = turboquant|turboquant|.|false|false
# bonsai is a llama.cpp fork (PrismML) adding the Q1_0 (1-bit) and Q2_0 (ternary)
# weight-quant kernels the Bonsai / Ternary-Bonsai models ship in. Reuses
# backend/cpp/llama-cpp grpc-server sources via a thin wrapper Makefile.
BACKEND_BONSAI = bonsai|bonsai|.|false|false
# ds4 is antirez/ds4, a DeepSeek V4 Flash-specific inference engine.
# Single-model; hardware-only validation lives at tests/e2e-backends/
# (BACKEND_BINARY mode); see docs/superpowers/plans/2026-05-11-ds4-backend.md.
BACKEND_DS4 = ds4|ds4|.|false|false
# privacy-filter wraps the standalone privacy-filter.cpp GGML engine (the
# openai-privacy-filter PII/NER token classifier) — the TokenClassify RPC for
# the PII redactor tier, on stock ggml with no llama.cpp carry-patches.
BACKEND_PRIVACY_FILTER = privacy-filter|privacy-filter|.|false|false
# audio-cpp wraps 0xShug0/audio.cpp, a multi-family ggml audio inference engine
# (TTS, ASR, VAD, diarization, source separation, music generation). Builds
# against apt gRPC/protobuf rather than a prebuilt base-grpc image; the reason
# is on the audio-cpp block in .github/backend-matrix.yml.
BACKEND_AUDIO_CPP = audio-cpp|audio-cpp|.|false|false
# Golang backends
BACKEND_PIPER = piper|golang|.|false|true
BACKEND_LOCAL_STORE = local-store|golang|.|false|true
BACKEND_VALKEY_STORE = valkey-store|golang|.|false|true
BACKEND_QDRANT_STORE = qdrant-store|golang|.|false|true
BACKEND_CLOUD_PROXY = cloud-proxy|golang|.|false|true
BACKEND_LOCALAI_PROXY = localai-proxy|golang|.|false|true
BACKEND_HUGGINGFACE = huggingface|golang|.|false|true
BACKEND_SILERO_VAD = silero-vad|golang|.|false|true
BACKEND_STABLEDIFFUSION_GGML = stablediffusion-ggml|golang|.|--progress=plain|true
BACKEND_TRELLIS2CPP = trellis2cpp|golang|.|--progress=plain|true
BACKEND_GEMXCPP = gemxcpp|golang|.|--progress=plain|true
BACKEND_KIMODOCPP = kimodocpp|golang|.|--progress=plain|true
BACKEND_WHISPER = whisper|golang|.|false|true
BACKEND_CRISPASR = crispasr|golang|.|false|true
BACKEND_PARAKEET_CPP = parakeet-cpp|golang|.|false|true
BACKEND_MOSS_TRANSCRIBE_CPP = moss-transcribe-cpp|golang|.|false|true
BACKEND_NEMO_SPEECH_CPP = nemo-speech-cpp|golang|.|false|true
BACKEND_DEPTH_ANYTHING_CPP = depth-anything-cpp|golang|.|false|true
BACKEND_TIMESTRAL_CPP = timestral-cpp|golang|.|false|true
BACKEND_VOXTRAL = voxtral|golang|.|false|true
BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
BACKEND_QWEN3_TTS_CPP = qwen3-tts-cpp|golang|.|false|true
BACKEND_MOSS_TTS_CPP = moss-tts-cpp|golang|.|false|true
BACKEND_MAGPIE_TTS_CPP = magpie-tts-cpp|golang|.|false|true
BACKEND_VLLM_CPP = vllm-cpp|golang|.|false|true
BACKEND_OMNIVOICE_CPP = omnivoice-cpp|golang|.|false|true
BACKEND_VIBEVOICE_CPP = vibevoice-cpp|golang|.|false|true
BACKEND_LOCALVQE = localvqe|golang|.|false|true
BACKEND_OPUS = opus|golang|.|false|true
BACKEND_SHERPA_ONNX = sherpa-onnx|golang|.|false|true
BACKEND_SUPERTONIC = supertonic|golang|.|false|true
# Python backends with root context
BACKEND_RERANKERS = rerankers|python|.|false|true
BACKEND_TRANSFORMERS = transformers|python|.|false|true
BACKEND_OUTETTS = outetts|python|.|false|true
BACKEND_FASTER_WHISPER = faster-whisper|python|.|false|true
BACKEND_WHISPER_MEDUSA = whisper-medusa|python|.|false|true
BACKEND_COQUI = coqui|python|.|false|true
BACKEND_RFDETR = rfdetr|python|.|false|true
BACKEND_INSIGHTFACE = insightface|python|.|false|true
BACKEND_SPEAKER_RECOGNITION = speaker-recognition|python|.|false|true
BACKEND_KITTEN_TTS = kitten-tts|python|.|false|true
BACKEND_NEUTTS = neutts|python|.|false|true
BACKEND_KOKORO = kokoro|python|.|false|true
BACKEND_VLLM = vllm|python|.|false|true
BACKEND_VLLM_OMNI = vllm-omni|python|.|false|true
BACKEND_LONGCAT_VIDEO = longcat-video|python|.|--progress=plain|true
BACKEND_SGLANG = sglang|python|.|false|true
BACKEND_DIFFUSERS = diffusers|python|.|--progress=plain|true
BACKEND_CHATTERBOX = chatterbox|python|.|false|true
BACKEND_VIBEVOICE = vibevoice|python|.|--progress=plain|true
BACKEND_MOONSHINE = moonshine|python|.|false|true
BACKEND_POCKET_TTS = pocket-tts|python|.|false|true
BACKEND_QWEN_TTS = qwen-tts|python|.|false|true
BACKEND_FISH_SPEECH = fish-speech|python|.|false|true
BACKEND_FASTER_QWEN3_TTS = faster-qwen3-tts|python|.|false|true
BACKEND_QWEN_ASR = qwen-asr|python|.|false|true
BACKEND_FUNASR = funasr|python|.|false|true
BACKEND_NEMO = nemo|python|.|false|true
BACKEND_VOXCPM = voxcpm|python|.|false|true
BACKEND_WHISPERX = whisperx|python|.|false|true
BACKEND_ACE_STEP = ace-step|python|.|false|true
BACKEND_MLX = mlx|python|.|false|true
BACKEND_MLX_VLM = mlx-vlm|python|.|false|true
BACKEND_MLX_DISTRIBUTED = mlx-distributed|python|./|false|true
BACKEND_TRL = trl|python|.|false|true
BACKEND_LLAMA_CPP_QUANTIZATION = llama-cpp-quantization|python|.|false|true
BACKEND_TINYGRAD = tinygrad|python|.|false|true
# Rust backends
BACKEND_KOKOROS = kokoros|rust|.|false|true
# C++ backends (Go wrapper with purego)
BACKEND_SAM3_CPP = sam3-cpp|golang|.|false|true
BACKEND_RFDETR_CPP = rfdetr-cpp|golang|.|false|true
# Helper function to build docker image for a backend
# Usage: $(call docker-build-backend,BACKEND_NAME,DOCKERFILE_TYPE,BUILD_CONTEXT,PROGRESS_FLAG,NEEDS_BACKEND_ARG)
define docker-build-backend
docker build $(if $(filter-out false,$(4)),$(4)) \
--build-arg BUILD_TYPE=$(BUILD_TYPE) \
--build-arg BASE_IMAGE=$(BASE_IMAGE) \
--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
--build-arg APT_MIRROR=$(APT_MIRROR) \
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
$(if $(FROM_SOURCE),--build-arg FROM_SOURCE=$(FROM_SOURCE)) \
$(if $(AMDGPU_TARGETS),--build-arg AMDGPU_TARGETS=$(AMDGPU_TARGETS)) \
$(if $(filter true,$(5)),--build-arg BACKEND=$(1)) \
-t local-ai-backend:$(1) -f backend/Dockerfile.$(2) $(3)
endef
# Generate docker-build targets from backend definitions
define generate-docker-build-target
docker-build-$(word 1,$(subst |, ,$(1))):
$$(call docker-build-backend,$(word 1,$(subst |, ,$(1))),$(word 2,$(subst |, ,$(1))),$(word 3,$(subst |, ,$(1))),$(word 4,$(subst |, ,$(1))),$(word 5,$(subst |, ,$(1))))
endef
# Generate all docker-build targets
$(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_IK_LLAMA_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_TURBOQUANT)))
$(eval $(call generate-docker-build-target,$(BACKEND_BONSAI)))
$(eval $(call generate-docker-build-target,$(BACKEND_DS4)))
$(eval $(call generate-docker-build-target,$(BACKEND_PRIVACY_FILTER)))
$(eval $(call generate-docker-build-target,$(BACKEND_AUDIO_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_PIPER)))
$(eval $(call generate-docker-build-target,$(BACKEND_LOCAL_STORE)))
$(eval $(call generate-docker-build-target,$(BACKEND_VALKEY_STORE)))
$(eval $(call generate-docker-build-target,$(BACKEND_QDRANT_STORE)))
$(eval $(call generate-docker-build-target,$(BACKEND_CLOUD_PROXY)))
$(eval $(call generate-docker-build-target,$(BACKEND_LOCALAI_PROXY)))
$(eval $(call generate-docker-build-target,$(BACKEND_HUGGINGFACE)))
$(eval $(call generate-docker-build-target,$(BACKEND_SILERO_VAD)))
$(eval $(call generate-docker-build-target,$(BACKEND_STABLEDIFFUSION_GGML)))
$(eval $(call generate-docker-build-target,$(BACKEND_TRELLIS2CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_KIMODOCPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_GEMXCPP)))
.NOTPARALLEL: backends/kimodocpp backends/kimodocpp-darwin
docker-build-backends: docker-build-kimodocpp
backends/kimodocpp-darwin:
BACKEND=kimodocpp BUILD_TYPE=cpu $(MAKE) build-darwin-go-backend
./local-ai backends install "ocifile://$(abspath ./backend-images/kimodocpp.tar)"
.NOTPARALLEL: backends/gemxcpp backends/gemxcpp-darwin
docker-build-backends: docker-build-gemxcpp
backends/gemxcpp-darwin:
BACKEND=gemxcpp BUILD_TYPE=cpu $(MAKE) build-darwin-go-backend
./local-ai backends install "ocifile://$(abspath ./backend-images/gemxcpp.tar)"
$(eval $(call generate-docker-build-target,$(BACKEND_WHISPER)))
$(eval $(call generate-docker-build-target,$(BACKEND_CRISPASR)))
$(eval $(call generate-docker-build-target,$(BACKEND_PARAKEET_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TRANSCRIBE_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_NEMO_SPEECH_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_DEPTH_ANYTHING_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_TIMESTRAL_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_VOXTRAL)))
$(eval $(call generate-docker-build-target,$(BACKEND_OPUS)))
$(eval $(call generate-docker-build-target,$(BACKEND_RERANKERS)))
$(eval $(call generate-docker-build-target,$(BACKEND_TRANSFORMERS)))
$(eval $(call generate-docker-build-target,$(BACKEND_OUTETTS)))
$(eval $(call generate-docker-build-target,$(BACKEND_FASTER_WHISPER)))
$(eval $(call generate-docker-build-target,$(BACKEND_WHISPER_MEDUSA)))
$(eval $(call generate-docker-build-target,$(BACKEND_COQUI)))
$(eval $(call generate-docker-build-target,$(BACKEND_RFDETR)))
$(eval $(call generate-docker-build-target,$(BACKEND_INSIGHTFACE)))
$(eval $(call generate-docker-build-target,$(BACKEND_SPEAKER_RECOGNITION)))
$(eval $(call generate-docker-build-target,$(BACKEND_KITTEN_TTS)))
$(eval $(call generate-docker-build-target,$(BACKEND_NEUTTS)))
$(eval $(call generate-docker-build-target,$(BACKEND_KOKORO)))
$(eval $(call generate-docker-build-target,$(BACKEND_VLLM)))
$(eval $(call generate-docker-build-target,$(BACKEND_VLLM_OMNI)))
$(eval $(call generate-docker-build-target,$(BACKEND_LONGCAT_VIDEO)))
$(eval $(call generate-docker-build-target,$(BACKEND_SGLANG)))
$(eval $(call generate-docker-build-target,$(BACKEND_DIFFUSERS)))
$(eval $(call generate-docker-build-target,$(BACKEND_CHATTERBOX)))
$(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE)))
$(eval $(call generate-docker-build-target,$(BACKEND_MOONSHINE)))
$(eval $(call generate-docker-build-target,$(BACKEND_POCKET_TTS)))
$(eval $(call generate-docker-build-target,$(BACKEND_QWEN_TTS)))
$(eval $(call generate-docker-build-target,$(BACKEND_FISH_SPEECH)))
$(eval $(call generate-docker-build-target,$(BACKEND_FASTER_QWEN3_TTS)))
$(eval $(call generate-docker-build-target,$(BACKEND_QWEN_ASR)))
$(eval $(call generate-docker-build-target,$(BACKEND_FUNASR)))
$(eval $(call generate-docker-build-target,$(BACKEND_NEMO)))
$(eval $(call generate-docker-build-target,$(BACKEND_VOXCPM)))
$(eval $(call generate-docker-build-target,$(BACKEND_WHISPERX)))
$(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
$(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_QWEN3_TTS_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TTS_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_MAGPIE_TTS_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_VLLM_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_OMNIVOICE_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_VIBEVOICE_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_LOCALVQE)))
$(eval $(call generate-docker-build-target,$(BACKEND_MLX)))
$(eval $(call generate-docker-build-target,$(BACKEND_MLX_VLM)))
$(eval $(call generate-docker-build-target,$(BACKEND_MLX_DISTRIBUTED)))
$(eval $(call generate-docker-build-target,$(BACKEND_TRL)))
$(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP_QUANTIZATION)))
$(eval $(call generate-docker-build-target,$(BACKEND_TINYGRAD)))
$(eval $(call generate-docker-build-target,$(BACKEND_KOKOROS)))
$(eval $(call generate-docker-build-target,$(BACKEND_SAM3_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_RFDETR_CPP)))
$(eval $(call generate-docker-build-target,$(BACKEND_SHERPA_ONNX)))
$(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC)))
# Pattern rule for docker-save targets
docker-save-%: backend-images
docker save local-ai-backend:$* -o backend-images/$*.tar
docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-localai-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-nemo-speech-cpp docker-build-privacy-filter docker-build-trellis2cpp docker-build-valkey-store docker-build-qdrant-store docker-build-audio-cpp docker-build-timestral-cpp
docker-build-backends: docker-build-whisper-medusa
docker-build-backends: docker-build-funasr
########################################################
### Mock Backend for E2E Tests
########################################################
build-mock-backend: protogen-go
$(GOCMD) build -o tests/e2e/mock-backend/mock-backend ./tests/e2e/mock-backend
clean-mock-backend:
rm -f tests/e2e/mock-backend/mock-backend
build-cloud-proxy-backend: protogen-go
$(GOCMD) build -o tests/e2e/mock-backend/cloud-proxy ./backend/go/cloud-proxy
clean-cloud-proxy-backend:
rm -f tests/e2e/mock-backend/cloud-proxy
build-localai-proxy-backend: protogen-go
$(GOCMD) build -o tests/e2e/mock-backend/localai-proxy ./backend/go/localai-proxy
clean-localai-proxy-backend:
rm -f tests/e2e/mock-backend/localai-proxy
########################################################
### UI E2E Test Server
########################################################
build-ui-test-server: build-mock-backend react-ui protogen-go
$(GOCMD) build -o tests/e2e-ui/ui-test-server ./tests/e2e-ui
test-ui-e2e: build-ui-test-server
cd core/http/react-ui && npm install && npx playwright install --with-deps chromium && npx playwright test
## Optional Playwright worker count for the UI e2e targets below. Pass
## UI_TEST_WORKERS=N (e.g. `make test-ui-coverage UI_TEST_WORKERS=20`) to
## override Playwright's default (cores/2). Empty by default so Playwright
## picks its own worker count.
UI_TEST_WORKERS ?=
PLAYWRIGHT_WORKERS_FLAG = $(if $(UI_TEST_WORKERS),--workers=$(UI_TEST_WORKERS),)
## Fast Playwright e2e run for local React UI validation.
## Force-rebuilds the (non-instrumented) dist so the suite tests the working
## tree — not a stale dist the `react-ui` skip-guard would leave — re-embeds
## it into ui-test-server, and runs the specs. Uses the nix-provided browser
## when PLAYWRIGHT_CHROMIUM_PATH is set (flake dev shell), else falls back to
## downloading it as `test-ui-e2e` does.
test-ui: build-mock-backend protogen-go
cd core/http/react-ui && bun install && bun run build
$(GOCMD) build -o tests/e2e-ui/ui-test-server ./tests/e2e-ui
cd core/http/react-ui && sh $(CURDIR)/scripts/ensure-playwright-browser.sh && bunx playwright test $(PLAYWRIGHT_WORKERS_FLAG)
## The stale-chunk specs need the production code-split bundle. The V8 coverage
## bundle below inlines dynamic imports to keep every page in its denominator.
test-ui-stale-chunk: build-mock-backend protogen-go
cd core/http/react-ui && bun install && bun run build
$(GOCMD) build -o tests/e2e-ui/ui-test-server ./tests/e2e-ui
cd core/http/react-ui && sh $(CURDIR)/scripts/ensure-playwright-browser.sh && bunx playwright test --grep @production-chunks --workers=1
## React UI code coverage from the Playwright e2e suite. Builds a
## NON-instrumented bundle with source maps (COVERAGE_V8=true), re-embeds it
## into the ui-test-server (the dist is //go:embed'ed at compile time), runs the
## Playwright specs which collect native Chromium V8 coverage (PW_V8_COVERAGE=1)
## — far cheaper than istanbul's build-time counters (~40% faster end-to-end) —
## convert it to istanbul via v8-to-istanbul in the coverage fixture, and write
## an nyc report to core/http/react-ui/coverage/. Removes the dist afterwards so
## normal builds aren't served source-mapped assets. (The legacy istanbul path
## still exists: `bun run build:coverage` + unset PW_V8_COVERAGE.)
test-ui-coverage: build-mock-backend protogen-go
trap 'rm -rf "$(CURDIR)/core/http/react-ui/dist"' EXIT; \
( cd core/http/react-ui && bun install && bun run build:coverage-v8 ) && \
$(GOCMD) build -o tests/e2e-ui/ui-test-server ./tests/e2e-ui && \
( cd core/http/react-ui && rm -rf .nyc_output coverage && \
sh $(CURDIR)/scripts/ensure-playwright-browser.sh && \
PW_V8_COVERAGE=1 bunx playwright test --grep-invert @production-chunks $(PLAYWRIGHT_WORKERS_FLAG) && bun run coverage:report )
## UI coverage baseline (committed) and the strict gate that compares against
## it — the React mirror of test-coverage-baseline / test-coverage-check.
test-ui-coverage-baseline: test-ui-coverage
@node -e 'const fs=require("fs");process.stdout.write(String(JSON.parse(fs.readFileSync("core/http/react-ui/coverage/coverage-summary.json")).total.lines.pct))' > core/http/react-ui/coverage-baseline.txt
@echo "Saved UI coverage baseline: $$(cat core/http/react-ui/coverage-baseline.txt)% lines"
test-ui-coverage-check: test-ui-coverage
sh $(CURDIR)/scripts/ui-coverage-check.sh core/http/react-ui/coverage/coverage-summary.json core/http/react-ui/coverage-baseline.txt
test-ui-e2e-docker:
docker build -t localai-ui-e2e -f tests/e2e-ui/Dockerfile .
docker run --rm localai-ui-e2e
clean-ui-test-server:
rm -f tests/e2e-ui/ui-test-server
########################################################
### END Backends
########################################################
.PHONY: swagger
swagger:
swag init -g core/http/app.go --output swagger
# DEPRECATED: gen-assets is for the legacy Alpine.js UI. Remove when legacy UI is removed.
.PHONY: gen-assets
gen-assets:
$(GOCMD) run core/dependencies_manager/manager.go webui_static.yaml core/http/static/assets
## Documentation and website
# The published site is two Hugo sites: website/ owns the root, docs/ is nested
# under /docs/. Serve them separately while editing; use `make site` to get the
# merged tree (including the legacy URL redirects) that GitHub Pages deploys.
SITE_BASE_URL?=http://localhost:8000
docs/layouts/_default:
mkdir -p docs/layouts/_default
docs/static/gallery.html: docs/layouts/_default
$(GOCMD) run ./.github/ci/modelslist.go ./gallery/index.yaml > docs/static/gallery.html
docs/public: docs/layouts/_default docs/static/gallery.html
cd docs && hugo --minify
docs-clean:
rm -rf docs/public
rm -rf website/public
rm -rf docs/static/gallery.html
.PHONY: docs
docs: docs/static/gallery.html
cd docs && hugo serve
.PHONY: website
website:
cd website && hugo serve
.PHONY: site
site: docs/static/gallery.html
rm -rf website/public docs/public
cd website && hugo --minify --baseURL "$(SITE_BASE_URL)/"
cd docs && hugo --minify --baseURL "$(SITE_BASE_URL)/docs/"
mkdir -p website/public/docs
cp -R docs/public/. website/public/docs/
./.github/ci/gen-redirects.sh website/public "$(SITE_BASE_URL)/"
.PHONY: site-serve
site-serve: site
cd website/public && python3 -m http.server 8000
########################################################
## Platform-specific builds
########################################################
## fyne cross-platform build
# Build LocalAI.app from the launcher via fyne (metadata read from cmd/launcher/FyneApp.toml).
# Signing happens via contrib/macos/sign-and-notarize.sh, which is a no-op when the signing
# secrets are unset, so unsigned local/fork builds keep working.
build-launcher-darwin:
rm -rf dist/LocalAI.app cmd/launcher/LocalAI.app
mkdir -p dist
cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME) --app-version $(LAUNCHER_APP_VERSION)
mv cmd/launcher/LocalAI.app dist/LocalAI.app
bash contrib/macos/sign-and-notarize.sh sign dist/LocalAI.app
# Notarize + staple the .app itself, then wrap it into a drag-to-Applications
# DMG via hdiutil and sign the DMG. The app is stapled BEFORE packaging so the
# bundle carries its own ticket and verifies offline (a dmg-only staple leaves
# the app relying on an online Gatekeeper check, which fails offline / once the
# app is copied out of the dmg). No-op without notary secrets.
dmg-launcher-darwin: build-launcher-darwin
bash contrib/macos/sign-and-notarize.sh notarize-app dist/LocalAI.app
rm -rf dist/dmg dist/LocalAI.dmg
mkdir -p dist/dmg
cp -R dist/LocalAI.app dist/dmg/LocalAI.app
ln -s /Applications dist/dmg/Applications
hdiutil create -volname "LocalAI" -srcfolder dist/dmg -ov -format UDZO dist/LocalAI.dmg
bash contrib/macos/sign-and-notarize.sh sign dist/LocalAI.dmg
# Submit the DMG to Apple notarization and staple the ticket (no-op without notary secrets).
notarize-launcher-darwin: dmg-launcher-darwin
bash contrib/macos/sign-and-notarize.sh notarize dist/LocalAI.dmg
# Single entrypoint for CI: build -> sign app -> notarize+staple app -> dmg -> sign dmg -> notarize+staple dmg.
release-launcher-darwin: notarize-launcher-darwin
@echo "dist/LocalAI.dmg is ready"
build-launcher-linux:
cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux --app-version $(LAUNCHER_APP_VERSION) && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz
# Thin native launcher only; retains the first-run LocalAI runtime downloader.
.PHONY: build-launcher-appimage test-launcher-appimage
build-launcher-appimage:
APPIMAGE_ARCH=$(or $(APPIMAGE_ARCH),amd64) bash contrib/appimage/build.sh
test-launcher-appimage:
python3 -m unittest discover -s contrib/appimage -p 'test_*.py' -v
Reference in new issue View Git Blame Copy Permalink