mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-18 10:53:22 -04:00
localai-org-maint-botandlocalai-org-maint-bot
35819d9e0d
* feat(cli): benchmark configured text models Compare model aliases through a running LocalAI server. Report full request latency and server-reported throughput with raw JSON samples. Keep warmups separate and fail before writing results on request errors. Assisted-by: Codex:GPT-6 * fix(cli): satisfy benchmark error checks Explicitly discard errors from buffered report writes, HTTP response cleanup, and test server writes to pass errcheck without changing behavior. Assisted-by: Codex:gpt-6 golangci-lint --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
85 lines
4.0 KiB
Markdown
85 lines
4.0 KiB
Markdown
+++
|
|
title = "Benchmark text models"
|
|
weight = 35
|
|
+++
|
|
|
|
Use `local-ai benchmark` to compare text inference through a running LocalAI
|
|
server. Install and configure the models first. The command sends sequential,
|
|
non-streaming requests to `/v1/chat/completions` and reports full request latency
|
|
and end-to-end completion tokens per second.
|
|
|
|
Pass one or more configured model names. To compare backends, configure separate
|
|
model aliases with the desired backend, then pass those aliases. Backend names
|
|
alone are not model names. The command does not install, discover, or unload
|
|
models; the server's normal loading and eviction settings still apply.
|
|
|
|
For example, with models named `text-llama-cpp` and `text-vllm` configured:
|
|
|
|
```sh
|
|
local-ai benchmark text-llama-cpp text-vllm --runs 5 --warmup 1 \
|
|
--prompt 'Explain how a rainbow forms.' --max-tokens 128
|
|
```
|
|
|
|
To save settings, per-model summaries, and every measured sample:
|
|
|
|
```sh
|
|
local-ai benchmark text-llama-cpp text-vllm --json > benchmark.json
|
|
```
|
|
|
|
For an authenticated server, set `LOCALAI_API_KEY` or `API_KEY` in the environment.
|
|
The API key is excluded from the JSON report. Use `--endpoint` for a remote server
|
|
or reverse proxy:
|
|
|
|
```sh
|
|
local-ai benchmark text-llama-cpp --endpoint https://localai.example.org/proxy/v1
|
|
```
|
|
|
|
The endpoint accepts a server root, an optional `/v1` suffix, and trailing
|
|
slashes. A reverse proxy path prefix is preserved. Redirects are refused.
|
|
Credentials in the URL, query strings, and fragments are rejected.
|
|
|
|
## Arguments and flags
|
|
|
|
| Argument or flag | Default | Description |
|
|
|---|---|---|
|
|
| `MODEL ...` | Required | One or more configured text model names. |
|
|
| `--endpoint` | `http://127.0.0.1:8080` | Server URL, optionally ending in `/v1`. |
|
|
| `--api-key` | Unset | API key; also reads `LOCALAI_API_KEY`, then `API_KEY`. |
|
|
| `--prompt` | `Explain why the sky is blue.` | Nonblank user message repeated for every request. |
|
|
| `--max-tokens` | `128` | Positive maximum number of completion tokens per request. |
|
|
| `--runs` | `3` | Positive number of measured requests per model. |
|
|
| `--warmup` | `1` | Unmeasured requests before each model; zero disables warmups. |
|
|
| `--timeout` | `5m` | Positive timeout per request, including reading its response. |
|
|
| `--json` | `false` | Write JSON instead of a table. |
|
|
| `-h`, `--help` | | Show command help. |
|
|
|
|
Every request sets temperature to `0` and streaming to `false`. Interrupting the
|
|
command cancels the active request. A failed request stops the benchmark with a
|
|
model and run error; results are written only after every model succeeds.
|
|
|
|
## Reading the results
|
|
|
|
Each model has minimum, mean, and maximum latency across measured requests.
|
|
Latency runs from sending the request through parsing the complete response.
|
|
It includes transport, queueing, prompt processing, generation, and response
|
|
parsing. This command does not measure time to first token.
|
|
|
|
End-to-end completion tokens per second is the sum of server-reported completion
|
|
tokens divided by the sum of full request durations. It is not decode-only speed
|
|
or a substitute for `llama-bench` kernel measurements. If any measured response
|
|
omits completion token usage, throughput is `null` in JSON and `N/A` in the table.
|
|
Reported zero tokens remain zero. Missing prompt or completion counts remain
|
|
`null` in each JSON sample; the command never estimates tokens from text length.
|
|
|
|
Warmups run separately for each model and do not appear in measurements. They can
|
|
absorb model loading time, but repeated prompts can also benefit from prompt
|
|
caching. With `--warmup 0`, measured requests can include model loading. Other
|
|
clients and server queueing can affect results; compare under similar load.
|
|
|
|
Keep hardware, quantization, context size, backend settings, and prompt consistent
|
|
when comparing engines. Different model tokenizers can report different token
|
|
counts for the same text, and models can stop before `--max-tokens`. Temperature
|
|
zero does not guarantee identical output across models or engines. The JSON
|
|
settings describe the benchmark requests, not the server's full model
|
|
configuration; record that configuration alongside the report.
|