mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
feat(agents): per-model output cap for agent job completions
Agent jobs (/api/agent/jobs, local and on distributed agent workers) run
cogito against a LocalAI client that sends no max_tokens of its own, so
every completion gets the client's fixed default of 16384. max_iterations
bounds the number of turns and loop_detection repeated tool calls, but
neither stops one completion that keeps generating, and 16384 is the same
for a small tool-picking model and a reasoning model that thinks at length.
Add agent.max_generation_tokens to the model config. When set, both job
paths apply it to the client with SetMaxTokens; requests that carry their
own cap still win, so cogito's length retry can raise it. Unset keeps
today's behaviour.
The specs run each job path against an MCP fixture and a recording LLM
endpoint and check the max_tokens the completions carry; they fail when
either call site drops the cap.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit 5600a1cabe)
This commit is contained in:
1 parent
3302c76964
commit
6d7e8171be
9 files changed
+328
No files matched your search
@@ -1015,6 +1015,7 @@ Agent/autonomous agent configuration:
|
||||
| `agent.enable_planning` | bool | Enable planning capabilities |
|
||||
| `agent.enable_mcp_prompts` | bool | Enable MCP prompts |
|
||||
| `agent.enable_plan_re_evaluator` | bool | Enable plan re-evaluation |
|
||||
| `agent.max_generation_tokens` | int | Output-token cap for each completion in the agent loop (agent jobs, `/api/agent/jobs`). Bounds a single completion that keeps generating, which `max_iterations` and `loop_detection` do not. A request that already carries a cap keeps it, so a length retry can still raise it. `0` (default) keeps the client default of 16384 |
|
||||
|
||||
## Reasoning Configuration
|
||||
|
||||
|
||||
@@ -191,6 +191,7 @@ different tools share one loaded model.
|
||||
#### Agent Configuration (`agent`)
|
||||
|
||||
- **`max_iterations`**: Maximum number of MCP tool execution loop iterations (default: 10). Each iteration allows the model to call tools and receive results before generating the next response.
|
||||
- **`max_generation_tokens`**: Output-token cap for each completion of an MCP agent job (default: the client default, 16384). `max_iterations` bounds the number of turns, not the length of one; this bounds a single completion that keeps generating. Raise it for reasoning models that think at length, lower it for models that should only pick a tool and answer briefly. It applies to agent jobs (`/api/agent/jobs`, also when run on a distributed agent worker); an MCP tool loop on `/v1/chat/completions` uses the request's own `max_tokens`.
|
||||
|
||||
## Agent-scoped MCP servers
|
||||
|
||||
|
||||
Reference in new issue
Block a user