mirror of
https://github.com/mudler/LocalAI.git
synced 2026-08-03 03:53:15 -04:00
feat(chat): local-ai chat is now a terminal agent (#11291)
* chore(deps): bump cogito to v0.11 ahead of the nib harness nib is the agent harness that becomes 'local-ai chat'. It requires cogito v0.11, so pull that bump forward on its own: minimal version selection would apply it to LocalAI anyway, and both repos use cogito and cogito/clients. Landing it separately keeps the harness change reviewable. nib itself is not pinned yet. Nothing in LocalAI imports it, and 'go mod tidy' runs as a goreleaser before-hook in CI, so an unimported require line does not survive. It lands with its first importer. No LocalAI call site needed a change. Both cogito.WithMaxAttempts callers guard the argument above zero, so v0.11's new clamp is unreachable, and LocalAI's Multimedia values implement only URL(), so v0.11's new TypedMultimedia routing treats them as images exactly as v0.10 did. Binary size (cmd/local-ai): 200,301,381 -> 200,336,045 bytes (+34,664). A throwaway probe that links nib measured 201,042,243 bytes (+740,862 over the pre-change baseline). Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): resolve and seed the agent state directory Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): write the agent config atomically and tighten its modes Replacing config.yaml in place truncated it first, so an interrupted write would have destroyed the api_key nib keeps in the same file. Stage through a sibling temp file and rename over the target instead, and match nib's 0700 directory mode. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): probe the endpoint and classify failures Probe lists what a LocalAI endpoint advertises and separates the two failures that need different advice: nothing listening, and rejected credentials. go-openai reports a rejected key as one of two concrete types depending on the error body, and both occur against a real LocalAI. The normal error handler sends an OpenAI error envelope, which arrives as *openai.APIError; the opaque-errors handler replies with a bare status and no body, which arrives as *openai.RequestError. Classifying on only one of them misses half the cases, so the status is read from either. A cancelled probe is not reported as an unreachable server, because it learned nothing about the endpoint, and neither is a reply that could not be parsed, because something did answer. Both would otherwise send the user off to start a server that may already be running. The model list is returned verbatim and in server order. LocalAI lists whatever it finds in the models directory, including stray archives and dotfiles, and deciding which advertised ids are real belongs to whoever presents them. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): resolve the model from flag, config, or the server Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(chat): pin that model resolution sorts a copy of the caller's slice The sort spec asserted only on what the chooser was offered, so replacing the defensive copy with an in-place sort of req.Available still passed all 37 specs. Assert the input slice's order after the call, so the guarantee cannot be dropped silently by a later refactor. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): offer to start a server when none is reachable Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): bound the server wait and pin readiness and stop semantics Set cmd.WaitDelay so a backend subprocess holding the child's stderr pipe cannot block cmd.Wait forever, which would leave exited unclosed, burn the whole shutdown grace on a clean exit, and leak the waiter goroutine. Two test gaps closed alongside it: the readiness spec now counts polls, so treating 503 as ready is observable, and Stop's single-interrupt contract is pinned by giving StartedServer interrupt/kill hooks that a spec can count. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(chat): drive Stop through one process interface, hide exec plumbing Two independent interrupt/kill func fields plus a nil check admitted wirings no test could distinguish: the pair swapped, so a SIGKILL would strand the backends SIGINT exists to let local-ai run clean up, or kill left nil, so a wedged server never escalates. One two-method interface that *os.Process already satisfies leaves nothing to swap and nothing to nil. Also translate exec.ErrWaitDelay, whose text names an os/exec struct field, into what the user can act on. os/exec only substitutes that sentinel when the process exited without an error of its own, so no exit status is swallowed. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): replace the REPL with the built-in agent local-ai chat is now the nib agent harness compiled into the binary: tool use behind an approval gate, sub-agents, MCP, plugins, and skills, all auto-configured against the local server. The REPL goes with it. Its model listing and its 401 classifier were duplicates of the ones Probe now owns, and the classifier was the version that misreads a bare 401 with no OpenAI error envelope, so keeping either would leave the package with two divergent answers to the same question. github.com/mudler/nib lands in go.mod in this commit rather than earlier: go mod tidy runs as a goreleaser before-hook on every PR, so a require line with no importer is stripped before it reaches CI. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(chat): split the pre-agent phase out of Run and pin it Everything before the handoff is testable and nothing after it is: once app.Run owns the terminal there is no seam left. prepare draws that line, takes interactivity as a parameter so the prompts can be driven over a pipe, and hands Run the state dir, the model, and any server it started. The questions move onto one prompter that owns its buffered reader. A fresh bufio.Reader per question reads ahead and discards what it buffered, so the model choice typed behind an answer to "start a server?" was lost and the next question saw EOF. choose answers with a list index and refuses an empty offer, so a value that was never on the list cannot reach ResolveModel, which persists it and starts every later run against it. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): bound each server check with a deadline Nothing bounded the model listing, so pointing chat at an address that accepts the connection and then never replies left the user with no output and no offer to start a server. The budget is context.WithTimeout rather than a cancel plus a timer. Probe deliberately refuses to call an endpoint unreachable on a context.Canceled, since a caller who gave up learned nothing about the server, and only honours a deadline. A cancel-based budget therefore expires as the one error that suppresses ErrUnreachable, exactly for the hung servers the offer exists to rescue. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): tell the user when their model choice cannot be saved The choice is meant to be asked for once. When saving it fails the user is silently asked again on the next run, and the only trace was an xlog.Warn: the agent runs at log level error, and a --log-level=error run swallows it entirely. ModelRequest gains Notify for exactly this class of problem, one that is worth telling the user about but not worth failing over, and the chat wiring points it at the same writer the question was asked on. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): stop a session's server when the process is signalled A server started for the session is stopped by a deferred call, and a signal skips deferred calls: a SIGTERM between the spawn and the exit left 'local-ai run' reparented to init with nothing left that knew to shut it down. Ctrl+C was already safe, but only incidentally, because the child shares this process' foreground process group. A signal handler rather than Pdeathsig on the child. Pdeathsig is Linux-only and, in Go, is delivered when the OS thread that forked exits rather than when the process does, so it can fire on a healthy parent. Setpgid would break the Ctrl+C that works today by taking the child out of the foreground group. SIGHUP joins SIGINT and SIGTERM: a terminal program whose terminal is gone has nobody left to talk to. The same context is what cancels the agent, which nib leaves to its embedder. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): only skip the server checks for work that stays local Two argument shapes were classified wrongly. Every 'mcp ...' invocation counted as management, so 'local-ai chat mcp --stdio', which serves the agent over MCP and needs a model like any other session, was handed an empty one. And --init, whose shell snippet a user pastes into an rc file long before any server exists, went the other way: it demanded a running server to print a static string. The mcp split is asked of nib's own IsMCPManageSubcommand rather than restated here, so a verb added upstream cannot drift out of this list. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): exit with the agent's status instead of reporting it twice nib writes what went wrong to stderr and returns nothing but an exit code, so returning that error unchanged had main log "Error running the application error=exit status 1" underneath the message the user had just read. The refusal to render the full-screen interface into a pipe is the one they meet in practice: it names --cli, and burying that hides the fix. ExitCodeError says "already reported, exit with this status". main honours it and prints nothing more, so a piped or redirected chat still fails a script the way it should. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(chat): route interactive chatter through one writer helper The prompts and notices all write to a terminal, where a failed write is not worth failing the session over and the read that follows the question reports the real problem. say says that once instead of five discarded error returns. The command's one-line help comes along: chat is no longer "an interactive chat session". Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(chat): record why the agent gets this process' streams Injecting them is what makes nib refuse to draw its full-screen interface into a pipe and name --cli, instead of rendering onto a terminal the caller may not own. The tradeoff is worth stating where the wiring is. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): stop the session's server on cancellation, not on the way out The deferred Stop is reached only if the agent returns, and cancelling the context does not make it: nib hands the TUI to bubbletea without the context, so what actually unwinds a running session today is bubbletea's own SIGINT and SIGTERM handler. SIGHUP has no such backstop, and registering for it removed the default disposition that used to end the process outright, so kill -HUP left a live TUI with a cancelled context and the started server still running. runSession watches the context alongside the agent and stops the server the moment it is cancelled, so the guarantee no longer depends on what the agent does with cancellation. Stop is idempotent, so the deferred call stays correct and free. The doc comment on shutdownContext described the mechanism it was supposed to work by rather than the one that does. Corrected, bubbletea's handler included. ResolveModel now checks the chooser's answer against what it offered. The shipped chooser answers by list index and cannot be wrong, but ModelChooser is exported, the answer is persisted, and every later run starts against it, so the invariant belongs at the consumer. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(chat): bump nib to v0.5.1 v0.5.1 carries four fixes that matter to 'local-ai chat': - --init now names the embedder's command, so the emitted widget invokes 'local-ai chat' rather than a bare 'nib' the user does not have. - A piped CLI session that succeeds exits 0 instead of failing with EOF. - EOF at a tool-approval prompt denies the call rather than approving it, and the session exits 3 (app.ExitCodeApprovalNoInput) so a script can tell "answered" from "refused to act" without reading stdout. Read-only tools are unaffected and still run. ExitStatus already unwraps app.ExitError, so the code propagates with no change here. - RunTUI passes the context to bubbletea and gives up bubbletea's own signal handler, which makes shutdownContext the single owner of the signal and stops a SIGHUP leaving a wedged TUI behind. Verified against a live server on 127.0.0.1:8080: the three --init shells, a piped prompt exiting 0, a denied 'touch' that left no file and exited 3, a read-only 'ls' that still ran and exited 0, and a SIGHUP that unwound a TUI running under a pty. Two comment blocks in run.go described the old TUI behavior and are now wrong, so they are corrected in the same change. No behavior change: both shutdownContext and runSession are untouched, and stopping the server on cancellation is still worth keeping independent of how promptly nib unwinds. One known gap, not addressed here. The widget --init now emits runs 'output=$(local-ai chat --height 50%)', and runAgent injects Stdout unconditionally, so under $(...) nib refuses the TUI for a non-terminal stream. This is the cost the runAgent comment already anticipated, now that the snippets no longer hardcode standalone nib. Ctrl+Space should not be documented until that is decided. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): let nib own stdout, so the Ctrl+Space widget works The widget 'local-ai chat --init' emits runs 'output=$(local-ai chat --height 50%)', which puts a pipe on stdout by construction. runAgent injected os.Stdout unconditionally, and nib refuses every mode but --cli when a stream it was handed is not a terminal, so Ctrl+Space printed "Re-run with --cli to use the injected streams" and inserted nothing. Verified against a pty before and after. nib reads a nil stream as "not injected" and falls back to the process stream, which is how an embedder asks for nib's own behavior. That is what stdout needs: the interface renders on /dev/tty but writes the chosen command to stdout even when stdout is a pipe, and that write is the whole of the shell-capture idiom. Stdin is deliberately left injected. A piped or redirected stdin really is ignored by the interface, so the refusal is the honest answer there, and it is the one users meet: 'echo q | local-ai chat' still says to re-run with --cli, once, exit 1. Nilling stdin the way stdout is nilled would delete that silently. Stderr is not gated by nib at all and is unchanged. One case does change and cannot be kept: 'local-ai chat > out.txt' from a terminal no longer refuses, because it is indistinguishable from the widget. It renders on /dev/tty and writes the capture line to the file, which is what standalone nib does. The app.Options literal moves into agentOptions so the decision is reachable from a spec rather than being a detail of a function that takes the terminal. Both sides of the asymmetry are pinned: reinstating 'Stdout: opts.Out' fails "hands nib nothing for the process stdout", and nilling stdin fails "hands the process stdin over". Also rewrites the last comments describing the pre-v0.5.1 behavior. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(chat): say what the stream refusal actually keys on Two comments still called it the refusal to render the interface "into a pipe". That was true when both stdin and stdout were injected, but a pipe on stdout no longer refuses, so the wording now points at precisely the case that was un-refused to make Ctrl+Space work. Only a stdin that cannot be read triggers it, and both comments now say so and name the command a user meets it with, 'echo q | local-ai chat'. The agentOptions doc also said a "file a caller chose" stays injected and refused, which reads as though 'local-ai chat > out.txt' still refuses. It does not: a shell redirect arrives as os.Stdout and is nil-ed like the widget's pipe, because the two differ only in being a regular file rather than a FIFO and nib's gate does not look at that. What stays injected is a writer an in-process caller chose for itself. Says that now, in the doc and in the spec comment that had the same ambiguity. Comments only. No behavior change. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: local-ai chat is now the built-in terminal agent `local-ai chat` was a plain chat prompt and is now an agent that runs shell commands behind an approval gate, so the pages that described a REPL were wrong rather than merely thin. Adds a Terminal agent feature page at /features/terminal-agent covering the approval gate, piped runs and their exit codes, Ctrl+Space, model resolution, state directory, and the pass-through management commands (including the `--yes` caveat that leaves a plugin installed but disabled in a script). The three-way "looking for something else" notice becomes four-way and moves into an agentic-routing shortcode. Four hand-kept copies of the same paragraph is what produced the drift the new page would otherwise have added to; the shortcode takes `current=` so each page still marks itself, and errors the build on a name that is not one of the four. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * website: the agent is in the binary, not a second install The nib section sold a separate tool you also install, with a GitHub link as the only way in, which is now the wrong order: the agent ships compiled into local-ai, and the standalone binary is the second reason to care rather than the first. Leads with `local-ai chat`, keeps nib as the SSH-anywhere story, and adds a docs CTA pointing at the new Terminal agent page. id="nib" is left alone because localai.io/#nib is linked from outside. The two credits on the demo clip named nib as the thing that drove the machine; they now credit the agent in LocalAI, which is the same agent. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * website: fix the exit keys, the plugin warning, and the redirect gap Three claims on the chat-agent pages that the code does not back. try-it-out told readers to press Ctrl+D. nib has no Ctrl+D handler: the full-screen interface quits on Esc or Ctrl+C, and Ctrl+D is only an exit in --cli, where it arrives as ordinary tty EOF. That sentence had replaced the removed /exit and /quit text, so the page was left with no working way to leave a session. Document both modes, since they differ. The plugin warning said nothing tells you the install stopped short. It does: the command prints that the plugin was left disabled. What it does not do is say so in its exit code, which is 0 either way. That is the part a script cannot work around, and it is the reason to pass --yes. Overstating it in the paragraph that gives the advice only makes the advice easier to dismiss. Redirecting stdout no longer refuses; the interface goes to /dev/tty and only the yanked command reaches the file. It is what lets the Ctrl+Space widget capture a command at all, since a redirect and out=$(...) are the same thing to the stream gate. It was documented nowhere. A non-terminal stdin is still refused, and the new text says which of the two it is. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): make the CLI flags outrank the agent config file local-ai chat routed --endpoint, --model, --api-key, --trace-dir and --yolo through nib's app.Options.Defaults. Defaults are seeds: they sit beneath the config file, so the file silently undoes them. That made the flags accepted and inert, and not in an edge case, since EnsureStateDir writes base_url on the first run and the interactive picker writes model, so from the second run on the file carried a value for both. Observed against a live server: with base_url: http://127.0.0.1:9999/v1 in the config and --endpoint http://127.0.0.1:8080 on the command line, the probe hit 8080 and every agent turn posted to 9999. With model: gemma-4-e2b-it-qat-q4_0 in the config, --model lfm2.5-8b-a1b was ignored on the wire. nib v0.6.0 adds app.Options.Overrides, applied above the config file and above the bare environment block. Move the whole block there: all five values are decisions this invocation already made on the user's behalf, and a flag the config file can undo is not a flag. Nothing is left in Defaults, because LocalAI's one genuine seed, the initial base_url, is written into the config file by EnsureStateDir rather than handed to nib. Two limits come with the channel and are documented on agentOptions rather than worked around. An override can only raise a field, since nib cannot tell "set to the zero value" from "not set", so --yolo can turn approval off but nothing on the command line turns it back on over an approval_mode: auto in the file. And nib's own NIB_TRACE_DIR and NIB_YOLO are resolved after the config load and still outrank these, deliberately, upstream. The existing spec pinned that the right values reach app.Options, which they always did, which is exactly why it could not see nib discarding them. The new specs resolve the config the way app.Run resolves it, against a real config file that disagrees with every flag, and one asserts Defaults stays empty. docs/content/features/terminal-agent.md already documented --model as winning over the saved model; that was false before this change and is true now, so no docs edit was needed. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): document intentional config file read Assisted-by: Codex:gpt-5 [gosec] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
This commit is contained in:
mudler's LocalAI [bot]
committed by
GitHub
GitHub
parent
359bd4850d
commit
d0119bf62c
@@ -161,7 +161,7 @@ local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
|
||||
local-ai run oci://localai/phi-2:latest
|
||||
```
|
||||
|
||||
To test a running LocalAI server from the terminal, open an interactive chat session from another shell. Inside the prompt, `/models` lists installed models and `/model <name>` switches between them.
|
||||
To work with a running LocalAI server from the terminal, start the built-in agent from another shell. It answers questions, reads your files and runs commands on your machine, asking you to approve anything that changes state. Inside a session, `/models` lists installed models and `/model <name>` switches between them. See the [Terminal agent](https://localai.io/docs/features/terminal-agent/) docs.
|
||||
|
||||
```bash
|
||||
# Terminal 1
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
@@ -107,6 +108,13 @@ For documentation and support:
|
||||
// Run the thing!
|
||||
err = ctx.Run(&cli.CLI.Context)
|
||||
if err != nil {
|
||||
// A command that has already told the user what went wrong returns
|
||||
// only a status. Logging it as well would print a bare "exit status 1"
|
||||
// underneath the explanation they just read.
|
||||
var reported cli.ExitCodeError
|
||||
if errors.As(err, &reported) {
|
||||
os.Exit(reported.Code)
|
||||
}
|
||||
xlog.Fatal("Error running the application", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Options struct {
|
||||
Model string
|
||||
BaseURL string
|
||||
APIKey string
|
||||
In io.Reader
|
||||
Out io.Writer
|
||||
}
|
||||
|
||||
func Run(ctx context.Context, opts Options) error {
|
||||
if opts.In == nil {
|
||||
opts.In = strings.NewReader("")
|
||||
}
|
||||
if opts.Out == nil {
|
||||
opts.Out = io.Discard
|
||||
}
|
||||
|
||||
session, err := newChatSession(ctx, newLocalAIChatClient(opts.BaseURL, opts.APIKey), opts.Model)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return runTerminalChat(ctx, session, opts.In, opts.Out)
|
||||
}
|
||||
@@ -1,172 +0,0 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("Run chat", func() {
|
||||
It("streams a single chat response", func() {
|
||||
var capturedModel string
|
||||
var capturedAuth string
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/v1/models" {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
writeResponse(w, `{"object":"list","data":[{"id":"test-model","object":"model"}]}`)
|
||||
return
|
||||
}
|
||||
|
||||
Expect(r.URL.Path).To(Equal("/v1/chat/completions"))
|
||||
capturedAuth = r.Header.Get("Authorization")
|
||||
|
||||
var body struct {
|
||||
Model string `json:"model"`
|
||||
Messages []struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
} `json:"messages"`
|
||||
}
|
||||
Expect(json.NewDecoder(r.Body).Decode(&body)).To(Succeed())
|
||||
capturedModel = body.Model
|
||||
Expect(body.Messages).To(HaveLen(1))
|
||||
Expect(body.Messages[0].Role).To(Equal("user"))
|
||||
Expect(body.Messages[0].Content).To(Equal("hello"))
|
||||
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
writeResponse(w, "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"hi\"}}]}\n\n")
|
||||
writeResponse(w, "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\"}}]}\n\n")
|
||||
writeResponse(w, "data: [DONE]\n\n")
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Run(GinkgoT().Context(), Options{
|
||||
Model: "test-model",
|
||||
BaseURL: server.URL + "/v1",
|
||||
APIKey: "secret",
|
||||
In: strings.NewReader("hello\n/exit\n"),
|
||||
Out: &out,
|
||||
})
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(capturedModel).To(Equal("test-model"))
|
||||
Expect(capturedAuth).To(Equal("Bearer secret"))
|
||||
Expect(out.String()).To(ContainSubstring("assistant: hi!"))
|
||||
Expect(out.String()).To(ContainSubstring("bye"))
|
||||
})
|
||||
|
||||
It("auto-selects the only available model", func() {
|
||||
server := chatTestServer([]string{"solo"}, nil)
|
||||
defer server.Close()
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Run(GinkgoT().Context(), Options{
|
||||
BaseURL: server.URL + "/v1",
|
||||
In: strings.NewReader("/exit\n"),
|
||||
Out: &out,
|
||||
})
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(out.String()).To(ContainSubstring("LocalAI chat (solo)"))
|
||||
})
|
||||
|
||||
It("returns an actionable error when no models are installed", func() {
|
||||
server := chatTestServer(nil, nil)
|
||||
defer server.Close()
|
||||
|
||||
err := Run(GinkgoT().Context(), Options{
|
||||
BaseURL: server.URL + "/v1",
|
||||
In: strings.NewReader(""),
|
||||
})
|
||||
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("no chat models are installed"))
|
||||
Expect(err.Error()).To(ContainSubstring("local-ai models install <model>"))
|
||||
})
|
||||
|
||||
It("returns an actionable error when multiple models are available without a selection", func() {
|
||||
server := chatTestServer([]string{"alpha", "beta"}, nil)
|
||||
defer server.Close()
|
||||
|
||||
err := Run(GinkgoT().Context(), Options{
|
||||
BaseURL: server.URL + "/v1",
|
||||
In: strings.NewReader(""),
|
||||
})
|
||||
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("multiple models are available"))
|
||||
Expect(err.Error()).To(ContainSubstring("--model"))
|
||||
Expect(err.Error()).To(ContainSubstring("alpha"))
|
||||
Expect(err.Error()).To(ContainSubstring("beta"))
|
||||
})
|
||||
|
||||
It("lists and switches models inside the chat", func() {
|
||||
requestedModels := []string{}
|
||||
server := chatTestServer([]string{"alpha", "beta"}, func(model string) {
|
||||
requestedModels = append(requestedModels, model)
|
||||
})
|
||||
defer server.Close()
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Run(GinkgoT().Context(), Options{
|
||||
Model: "alpha",
|
||||
BaseURL: server.URL + "/v1",
|
||||
In: strings.NewReader("/models\n/model beta\nhello\n/exit\n"),
|
||||
Out: &out,
|
||||
})
|
||||
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(out.String()).To(ContainSubstring("* alpha"))
|
||||
Expect(out.String()).To(ContainSubstring(" beta"))
|
||||
Expect(out.String()).To(ContainSubstring("switched to beta; conversation cleared"))
|
||||
Expect(requestedModels).To(Equal([]string{"beta"}))
|
||||
})
|
||||
})
|
||||
|
||||
func chatTestServer(models []string, onChat func(model string)) *httptest.Server {
|
||||
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/v1/models":
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
writeResponse(w, `{"object":"list","data":[`)
|
||||
for i, model := range models {
|
||||
if i > 0 {
|
||||
writeResponse(w, ",")
|
||||
}
|
||||
writeResponsef(w, `{"id":%q,"object":"model"}`, model)
|
||||
}
|
||||
writeResponse(w, `]}`)
|
||||
case "/v1/chat/completions":
|
||||
var body struct {
|
||||
Model string `json:"model"`
|
||||
}
|
||||
Expect(json.NewDecoder(r.Body).Decode(&body)).To(Succeed())
|
||||
if onChat != nil {
|
||||
onChat(body.Model)
|
||||
}
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
writeResponse(w, "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"ok\"}}]}\n\n")
|
||||
writeResponse(w, "data: [DONE]\n\n")
|
||||
default:
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
}))
|
||||
}
|
||||
|
||||
func writeResponse(w io.Writer, text string) {
|
||||
_, err := fmt.Fprint(w, text)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
}
|
||||
|
||||
func writeResponsef(w io.Writer, format string, args ...any) {
|
||||
_, err := fmt.Fprintf(w, format, args...)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
}
|
||||
@@ -1,114 +0,0 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
openai "github.com/sashabaranov/go-openai"
|
||||
)
|
||||
|
||||
type chatClient interface {
|
||||
ListModels(ctx context.Context) ([]string, error)
|
||||
StreamChat(ctx context.Context, model string, messages []chatMessage, out io.Writer) (string, error)
|
||||
}
|
||||
|
||||
type localAIChatClient struct {
|
||||
client *openai.Client
|
||||
}
|
||||
|
||||
func newLocalAIChatClient(baseURL string, apiKey string) *localAIChatClient {
|
||||
cfg := openai.DefaultConfig(apiKey)
|
||||
cfg.BaseURL = baseURL
|
||||
return &localAIChatClient{client: openai.NewClientWithConfig(cfg)}
|
||||
}
|
||||
|
||||
func (c *localAIChatClient) ListModels(ctx context.Context) ([]string, error) {
|
||||
resp, err := c.client.ListModels(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
models := make([]string, 0, len(resp.Models))
|
||||
for _, model := range resp.Models {
|
||||
if model.ID != "" {
|
||||
models = append(models, model.ID)
|
||||
}
|
||||
}
|
||||
sort.Strings(models)
|
||||
return models, nil
|
||||
}
|
||||
|
||||
func (c *localAIChatClient) StreamChat(ctx context.Context, model string, messages []chatMessage, out io.Writer) (string, error) {
|
||||
stream, err := c.client.CreateChatCompletionStream(ctx, openai.ChatCompletionRequest{
|
||||
Model: model,
|
||||
Messages: openAIChatMessages(messages),
|
||||
})
|
||||
if err != nil {
|
||||
return "", friendlyChatError(err, model)
|
||||
}
|
||||
defer func() {
|
||||
_ = stream.Close()
|
||||
}()
|
||||
|
||||
var answer strings.Builder
|
||||
for {
|
||||
resp, err := stream.Recv()
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return answer.String(), friendlyChatError(err, model)
|
||||
}
|
||||
if len(resp.Choices) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
token := resp.Choices[0].Delta.Content
|
||||
if token == "" {
|
||||
continue
|
||||
}
|
||||
answer.WriteString(token)
|
||||
if _, err := fmt.Fprint(out, token); err != nil {
|
||||
return answer.String(), err
|
||||
}
|
||||
}
|
||||
|
||||
return answer.String(), nil
|
||||
}
|
||||
|
||||
func openAIChatMessages(messages []chatMessage) []openai.ChatCompletionMessage {
|
||||
converted := make([]openai.ChatCompletionMessage, len(messages))
|
||||
for i, message := range messages {
|
||||
converted[i] = openai.ChatCompletionMessage{
|
||||
Role: message.Role,
|
||||
Content: message.Content,
|
||||
}
|
||||
}
|
||||
return converted
|
||||
}
|
||||
|
||||
func friendlyChatError(err error, model string) error {
|
||||
var apiErr *openai.APIError
|
||||
if errors.As(err, &apiErr) {
|
||||
switch apiErr.HTTPStatusCode {
|
||||
case 404:
|
||||
return fmt.Errorf("model %q is not available. Run `local-ai models list`, install a model with `local-ai models install <model>`, or switch with `/model <name>`", model)
|
||||
case 403:
|
||||
return fmt.Errorf("model %q is disabled. Enable it from LocalAI settings or choose another model with `/model <name>`", model)
|
||||
}
|
||||
if apiErr.Message != "" {
|
||||
return errors.New(apiErr.Message)
|
||||
}
|
||||
}
|
||||
|
||||
msg := err.Error()
|
||||
if strings.Contains(msg, "model") && strings.Contains(msg, "not found") {
|
||||
return fmt.Errorf("model %q is not available. Run `local-ai models list`, install a model with `local-ai models install <model>`, or switch with `/model <name>`", model)
|
||||
}
|
||||
|
||||
return err
|
||||
}
|
||||
@@ -1,17 +0,0 @@
|
||||
package chat
|
||||
|
||||
import "strings"
|
||||
|
||||
func formatChatModelList(models []string, current string) string {
|
||||
var b strings.Builder
|
||||
for _, model := range models {
|
||||
prefix := " "
|
||||
if model == current {
|
||||
prefix = "* "
|
||||
}
|
||||
b.WriteString(prefix)
|
||||
b.WriteString(model)
|
||||
b.WriteByte('\n')
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
153
core/cli/chat/paths.go
Normal file
153
core/cli/chat/paths.go
Normal file
@@ -0,0 +1,153 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// stateDirMode matches the mode nib uses for the same directory. The directory
|
||||
// holds an API key, so it stays owner-only.
|
||||
const stateDirMode = 0o700
|
||||
|
||||
// configFileMode keeps the config owner-only: nib stores the user's API key in
|
||||
// it alongside the keys written here.
|
||||
const configFileMode = 0o600
|
||||
|
||||
// StateDir resolves where the chat agent keeps its config, plugins, and
|
||||
// skills. This is user-scoped rather than server-scoped: chat is a client that
|
||||
// may target a remote LocalAI, so it does not belong under LOCALAI_CONFIG_DIR.
|
||||
func StateDir(override string) (string, error) {
|
||||
if override != "" {
|
||||
return override, nil
|
||||
}
|
||||
if xdg := os.Getenv("XDG_CONFIG_HOME"); xdg != "" {
|
||||
return filepath.Join(xdg, "localai", "chat"), nil
|
||||
}
|
||||
home, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("resolving home directory for the agent state dir: %w", err)
|
||||
}
|
||||
return filepath.Join(home, ".config", "localai", "chat"), nil
|
||||
}
|
||||
|
||||
// ConfigPath is the agent's config file inside dir.
|
||||
func ConfigPath(dir string) string { return filepath.Join(dir, "config.yaml") }
|
||||
|
||||
// EnsureStateDir creates dir and, on first run only, seeds a config file
|
||||
// pointing at baseURL. It deliberately does not seed a model: a baked-in model
|
||||
// name goes stale as soon as the user installs a different one.
|
||||
//
|
||||
// The config file is machine-managed from here on: nib rewrites it whenever it
|
||||
// self-configures, so hand-written comments in it do not survive.
|
||||
func EnsureStateDir(dir, baseURL string) error {
|
||||
if err := os.MkdirAll(dir, stateDirMode); err != nil {
|
||||
return fmt.Errorf("creating agent state dir %s: %w", dir, err)
|
||||
}
|
||||
path := ConfigPath(dir)
|
||||
if _, err := os.Stat(path); err == nil {
|
||||
return nil // already configured; never overwrite the user's file
|
||||
} else if !os.IsNotExist(err) {
|
||||
return fmt.Errorf("checking agent config %s: %w", path, err)
|
||||
}
|
||||
|
||||
seed := map[string]string{"base_url": baseURL}
|
||||
data, err := yaml.Marshal(seed)
|
||||
if err != nil {
|
||||
return fmt.Errorf("encoding seed agent config: %w", err)
|
||||
}
|
||||
if err := writeConfigFile(path, data); err != nil {
|
||||
return fmt.Errorf("writing seed agent config: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PersistModel records the chosen model in the agent config, preserving every
|
||||
// other key the user may have set, including the api_key nib writes there.
|
||||
//
|
||||
// The file is machine-managed: this overlays the model onto the parsed keys and
|
||||
// re-marshals, which drops comments. That is deliberate rather than an
|
||||
// oversight, because nib's own save path does the same thing and would erase
|
||||
// them on its next write regardless.
|
||||
func PersistModel(dir, model string) error {
|
||||
// PersistModel is callable before EnsureStateDir, so it cannot assume the
|
||||
// directory exists.
|
||||
if err := os.MkdirAll(dir, stateDirMode); err != nil {
|
||||
return fmt.Errorf("creating agent state dir %s: %w", dir, err)
|
||||
}
|
||||
path := ConfigPath(dir)
|
||||
|
||||
values := map[string]any{}
|
||||
// #nosec G304 -- path is the fixed config.yaml name under the user-selected
|
||||
// chat state directory; selecting that directory is the documented override.
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("reading agent config %s: %w", path, err)
|
||||
}
|
||||
if err == nil {
|
||||
if err := yaml.Unmarshal(data, &values); err != nil {
|
||||
return fmt.Errorf("parsing agent config %s: %w", path, err)
|
||||
}
|
||||
}
|
||||
values["model"] = model
|
||||
|
||||
out, err := yaml.Marshal(values)
|
||||
if err != nil {
|
||||
return fmt.Errorf("encoding agent config: %w", err)
|
||||
}
|
||||
if err := writeConfigFile(path, out); err != nil {
|
||||
return fmt.Errorf("writing agent config: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeConfigFile replaces path with data atomically: it writes a temporary
|
||||
// file next to the target and renames it over the target. Writing the target in
|
||||
// place would truncate it first, so an interrupted or out-of-disk write would
|
||||
// leave a half-written config and destroy the api_key nib keeps in the same
|
||||
// file. The temporary file must share the directory because rename is only
|
||||
// atomic within one filesystem.
|
||||
func writeConfigFile(path string, data []byte) error {
|
||||
dir := filepath.Dir(path)
|
||||
|
||||
// A randomized name rather than a fixed config.yaml.tmp, so two concurrent
|
||||
// writers cannot corrupt each other's temporary file.
|
||||
tmp, err := os.CreateTemp(dir, "config.yaml.*.tmp")
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating temp file in %s: %w", dir, err)
|
||||
}
|
||||
tmpPath := tmp.Name()
|
||||
renamed := false
|
||||
defer func() {
|
||||
if !renamed {
|
||||
// Leave no litter behind on any failure path.
|
||||
_ = os.Remove(tmpPath)
|
||||
}
|
||||
}()
|
||||
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("writing %s: %w", tmpPath, err)
|
||||
}
|
||||
// Flush before the rename: renaming a file whose contents are still only in
|
||||
// the page cache can still lose them across a crash.
|
||||
if err := tmp.Sync(); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("syncing %s: %w", tmpPath, err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("closing %s: %w", tmpPath, err)
|
||||
}
|
||||
// CreateTemp already asks for 0600, but the umask can only ever clear bits,
|
||||
// so set the mode explicitly rather than inheriting whatever survived.
|
||||
if err := os.Chmod(tmpPath, configFileMode); err != nil {
|
||||
return fmt.Errorf("setting mode on %s: %w", tmpPath, err)
|
||||
}
|
||||
if err := os.Rename(tmpPath, path); err != nil {
|
||||
return fmt.Errorf("replacing %s: %w", path, err)
|
||||
}
|
||||
renamed = true
|
||||
return nil
|
||||
}
|
||||
186
core/cli/chat/paths_test.go
Normal file
186
core/cli/chat/paths_test.go
Normal file
@@ -0,0 +1,186 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// richConfig stands in for a config nib has already taken ownership of: a
|
||||
// comment, a secret, and a nested block. A flat scalar alone would not catch a
|
||||
// writer that mangles structure or drops a key it does not know about.
|
||||
const richConfig = `# hand written note
|
||||
base_url: http://x.invalid/v1
|
||||
api_key: secret-token
|
||||
mcp_servers:
|
||||
files:
|
||||
command: mcp-files
|
||||
args:
|
||||
- --root
|
||||
- /tmp
|
||||
`
|
||||
|
||||
var _ = Describe("Agent state directory", func() {
|
||||
Describe("StateDir", func() {
|
||||
It("prefers an explicit override", func() {
|
||||
Expect(StateDir("/custom/dir")).To(Equal("/custom/dir"))
|
||||
})
|
||||
|
||||
It("uses XDG_CONFIG_HOME when set", func() {
|
||||
tmp := GinkgoT().TempDir()
|
||||
GinkgoT().Setenv("XDG_CONFIG_HOME", tmp)
|
||||
Expect(StateDir("")).To(Equal(filepath.Join(tmp, "localai", "chat")))
|
||||
})
|
||||
|
||||
It("falls back to ~/.config/localai/chat", func() {
|
||||
tmp := GinkgoT().TempDir()
|
||||
GinkgoT().Setenv("XDG_CONFIG_HOME", "")
|
||||
GinkgoT().Setenv("HOME", tmp)
|
||||
Expect(StateDir("")).To(Equal(filepath.Join(tmp, ".config", "localai", "chat")))
|
||||
})
|
||||
|
||||
It("fails when neither XDG_CONFIG_HOME nor a home directory is resolvable", func() {
|
||||
GinkgoT().Setenv("XDG_CONFIG_HOME", "")
|
||||
GinkgoT().Setenv("HOME", "")
|
||||
|
||||
dir, err := StateDir("")
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("agent state dir"))
|
||||
// No silent fallback to a relative path: writing an API key into the
|
||||
// working directory would be worse than refusing.
|
||||
Expect(dir).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
|
||||
Describe("EnsureStateDir", func() {
|
||||
It("creates the directory and seeds base_url on first run", func() {
|
||||
dir := filepath.Join(GinkgoT().TempDir(), "chat")
|
||||
Expect(EnsureStateDir(dir, "http://127.0.0.1:8080/v1")).To(Succeed())
|
||||
|
||||
data, err := os.ReadFile(ConfigPath(dir))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(string(data)).To(ContainSubstring("base_url: http://127.0.0.1:8080/v1"))
|
||||
// A model must NOT be seeded: it goes stale as soon as the user
|
||||
// installs a different one.
|
||||
Expect(string(data)).ToNot(ContainSubstring("model:"))
|
||||
})
|
||||
|
||||
It("keeps the seeded config and its directory owner-only", func() {
|
||||
dir := filepath.Join(GinkgoT().TempDir(), "chat")
|
||||
Expect(EnsureStateDir(dir, "http://127.0.0.1:8080/v1")).To(Succeed())
|
||||
|
||||
// nib writes the user's api_key into this same file, so the modes are
|
||||
// load-bearing, not cosmetic.
|
||||
config, err := os.Stat(ConfigPath(dir))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(config.Mode().Perm()).To(Equal(os.FileMode(0o600)))
|
||||
|
||||
state, err := os.Stat(dir)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(state.Mode().Perm()).To(Equal(os.FileMode(0o700)))
|
||||
})
|
||||
|
||||
It("leaves an existing config byte-for-byte untouched", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
Expect(os.WriteFile(ConfigPath(dir), []byte(richConfig), 0o600)).To(Succeed())
|
||||
|
||||
Expect(EnsureStateDir(dir, "http://127.0.0.1:8080/v1")).To(Succeed())
|
||||
|
||||
data, err := os.ReadFile(ConfigPath(dir))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
// Byte-exact against a fixture carrying a comment and a nested block:
|
||||
// an implementation that "preserves" by re-marshaling through a map
|
||||
// fails here rather than passing on a flat scalar.
|
||||
Expect(string(data)).To(Equal(richConfig))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("PersistModel", func() {
|
||||
It("adds a model to an existing config, preserving other keys", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
Expect(os.WriteFile(ConfigPath(dir), []byte("base_url: http://x.invalid/v1\n"), 0o600)).To(Succeed())
|
||||
|
||||
Expect(PersistModel(dir, "chosen-model")).To(Succeed())
|
||||
|
||||
data, err := os.ReadFile(ConfigPath(dir))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(string(data)).To(ContainSubstring("base_url: http://x.invalid/v1"))
|
||||
Expect(string(data)).To(ContainSubstring("model: chosen-model"))
|
||||
})
|
||||
|
||||
It("replaces an existing model rather than duplicating the key", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
Expect(os.WriteFile(ConfigPath(dir), []byte("model: old\nbase_url: http://x.invalid/v1\n"), 0o600)).To(Succeed())
|
||||
|
||||
Expect(PersistModel(dir, "new")).To(Succeed())
|
||||
|
||||
data, err := os.ReadFile(ConfigPath(dir))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(string(data)).To(ContainSubstring("model: new"))
|
||||
Expect(string(data)).ToNot(ContainSubstring("model: old"))
|
||||
})
|
||||
|
||||
It("preserves secrets and nested blocks it does not understand", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
Expect(os.WriteFile(ConfigPath(dir), []byte(richConfig), 0o600)).To(Succeed())
|
||||
|
||||
Expect(PersistModel(dir, "chosen-model")).To(Succeed())
|
||||
|
||||
data, err := os.ReadFile(ConfigPath(dir))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
var got map[string]any
|
||||
Expect(yaml.Unmarshal(data, &got)).To(Succeed())
|
||||
Expect(got).To(HaveKeyWithValue("model", "chosen-model"))
|
||||
Expect(got).To(HaveKeyWithValue("base_url", "http://x.invalid/v1"))
|
||||
// Losing this key logs the user out of their own server.
|
||||
Expect(got).To(HaveKeyWithValue("api_key", "secret-token"))
|
||||
Expect(got).To(HaveKeyWithValue("mcp_servers",
|
||||
HaveKeyWithValue("files", And(
|
||||
HaveKeyWithValue("command", "mcp-files"),
|
||||
HaveKeyWithValue("args", ConsistOf("--root", "/tmp")),
|
||||
)),
|
||||
))
|
||||
|
||||
// Documented, accepted behavior rather than an aspiration: the overlay
|
||||
// re-marshals, so comments do not survive. nib's own save path erases
|
||||
// them too, so preserving them here would buy nothing.
|
||||
Expect(string(data)).ToNot(ContainSubstring("# hand written note"))
|
||||
})
|
||||
|
||||
It("keeps the rewritten config owner-only and leaves no temp file behind", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
Expect(os.WriteFile(ConfigPath(dir), []byte(richConfig), 0o600)).To(Succeed())
|
||||
|
||||
Expect(PersistModel(dir, "chosen-model")).To(Succeed())
|
||||
|
||||
info, err := os.Stat(ConfigPath(dir))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(info.Mode().Perm()).To(Equal(os.FileMode(0o600)))
|
||||
|
||||
// The atomic write stages through a sibling temp file; it must not
|
||||
// survive a successful write.
|
||||
entries, err := os.ReadDir(dir)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
names := []string{}
|
||||
for _, entry := range entries {
|
||||
names = append(names, entry.Name())
|
||||
}
|
||||
Expect(names).To(ConsistOf("config.yaml"))
|
||||
})
|
||||
|
||||
It("creates the state directory when it does not exist yet", func() {
|
||||
// Task 4 may persist a picked model before anything else has run.
|
||||
dir := filepath.Join(GinkgoT().TempDir(), "chat")
|
||||
|
||||
Expect(PersistModel(dir, "chosen-model")).To(Succeed())
|
||||
|
||||
data, err := os.ReadFile(ConfigPath(dir))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(string(data)).To(ContainSubstring("model: chosen-model"))
|
||||
})
|
||||
})
|
||||
})
|
||||
86
core/cli/chat/probe.go
Normal file
86
core/cli/chat/probe.go
Normal file
@@ -0,0 +1,86 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
openai "github.com/sashabaranov/go-openai"
|
||||
)
|
||||
|
||||
var (
|
||||
// ErrUnreachable means nothing answered at the endpoint. Callers use this
|
||||
// to decide whether offering to start a server makes sense.
|
||||
ErrUnreachable = errors.New("no LocalAI server reachable")
|
||||
// ErrUnauthorized means the server answered but rejected the credentials.
|
||||
ErrUnauthorized = errors.New("LocalAI server rejected the API key")
|
||||
)
|
||||
|
||||
// Probe lists the models the endpoint advertises. It classifies the two
|
||||
// failures that need different advice: nothing listening, and bad credentials.
|
||||
//
|
||||
// The returned list is what the server advertises, verbatim and in server
|
||||
// order. LocalAI happily lists non-model entries it finds in the models
|
||||
// directory (stray archives, dotfiles), and guessing which advertised IDs are
|
||||
// real belongs to whoever presents them, not here.
|
||||
func Probe(ctx context.Context, baseURL, apiKey string) ([]string, error) {
|
||||
cfg := openai.DefaultConfig(apiKey)
|
||||
cfg.BaseURL = baseURL
|
||||
|
||||
resp, err := openai.NewClientWithConfig(cfg).ListModels(ctx)
|
||||
if err != nil {
|
||||
if status, answered := responseStatus(err); answered {
|
||||
if status == http.StatusUnauthorized || status == http.StatusForbidden {
|
||||
return nil, fmt.Errorf("%w: %w", ErrUnauthorized, err)
|
||||
}
|
||||
// The server answered, so it is up; surface its error as-is.
|
||||
return nil, fmt.Errorf("listing models at %s: %w", baseURL, err)
|
||||
}
|
||||
// A caller who cancelled the probe learned nothing about the endpoint,
|
||||
// so claiming it is unreachable would send them to fix a server that
|
||||
// may be fine. A deadline is left alone: an endpoint that cannot answer
|
||||
// within the probe's budget is unreachable for our purposes.
|
||||
var urlErr *url.Error
|
||||
if errors.As(err, &urlErr) && !errors.Is(err, context.Canceled) {
|
||||
// Only a failure to complete the round trip means nothing is
|
||||
// listening. A reply we could not parse is a different problem,
|
||||
// so it falls through to the generic error below.
|
||||
return nil, fmt.Errorf("%w at %s: %w", ErrUnreachable, baseURL, err)
|
||||
}
|
||||
return nil, fmt.Errorf("listing models at %s: %w", baseURL, err)
|
||||
}
|
||||
|
||||
models := make([]string, 0, len(resp.Models))
|
||||
for _, m := range resp.Models {
|
||||
if m.ID != "" {
|
||||
models = append(models, m.ID)
|
||||
}
|
||||
}
|
||||
return models, nil
|
||||
}
|
||||
|
||||
// responseStatus reports the HTTP status a failed call came back with, and
|
||||
// whether there was one at all.
|
||||
//
|
||||
// go-openai splits this across two types depending on the error body, and both
|
||||
// occur against a real LocalAI: it returns *openai.APIError when the body
|
||||
// parses as an OpenAI error envelope, which is what LocalAI's normal error
|
||||
// handler sends, and *openai.RequestError when it does not, which is what
|
||||
// LocalAI sends when started with opaque errors, since that handler replies
|
||||
// with a bare status and no body.
|
||||
func responseStatus(err error) (int, bool) {
|
||||
// *RequestError is checked first because it is the outer type when
|
||||
// go-openai nests one error inside the other; the inner value in that case
|
||||
// carries no status.
|
||||
var reqErr *openai.RequestError
|
||||
if errors.As(err, &reqErr) {
|
||||
return reqErr.HTTPStatusCode, true
|
||||
}
|
||||
var apiErr *openai.APIError
|
||||
if errors.As(err, &apiErr) {
|
||||
return apiErr.HTTPStatusCode, true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
169
core/cli/chat/probe_test.go
Normal file
169
core/cli/chat/probe_test.go
Normal file
@@ -0,0 +1,169 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"time"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("Probe", func() {
|
||||
It("returns the advertised models", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
Expect(json.NewEncoder(w).Encode(map[string]any{
|
||||
"object": "list",
|
||||
"data": []map[string]string{
|
||||
{"id": "model-a", "object": "model"},
|
||||
{"id": "model-b", "object": "model"},
|
||||
},
|
||||
})).To(Succeed())
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
models, err := Probe(context.Background(), srv.URL+"/v1", "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(models).To(Equal([]string{"model-a", "model-b"}))
|
||||
})
|
||||
|
||||
It("reports an unreachable server distinguishably", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
|
||||
url := srv.URL
|
||||
srv.Close() // nothing is listening now
|
||||
|
||||
_, err := Probe(context.Background(), url+"/v1", "")
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(errors.Is(err, ErrUnreachable)).To(BeTrue(), "want ErrUnreachable, got %v", err)
|
||||
})
|
||||
|
||||
It("reports an auth failure distinguishably", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
_, err := Probe(context.Background(), srv.URL+"/v1", "bad-key")
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(errors.Is(err, ErrUnauthorized)).To(BeTrue(), "want ErrUnauthorized, got %v", err)
|
||||
})
|
||||
|
||||
// LocalAI's normal error handler replies with an OpenAI error envelope, and
|
||||
// its opaque-errors handler replies with a bare status and no body. Those
|
||||
// reach the client as two different go-openai types, so both have to be
|
||||
// classified the same way.
|
||||
It("reports an auth failure carrying an error envelope distinguishably", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
Expect(json.NewEncoder(w).Encode(map[string]any{
|
||||
"error": map[string]any{"message": "invalid api key", "code": http.StatusUnauthorized},
|
||||
})).To(Succeed())
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
_, err := Probe(context.Background(), srv.URL+"/v1", "bad-key")
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(errors.Is(err, ErrUnauthorized)).To(BeTrue(), "want ErrUnauthorized, got %v", err)
|
||||
})
|
||||
|
||||
It("does not call a server that answered with an error unreachable", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusInternalServerError)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
_, err := Probe(context.Background(), srv.URL+"/v1", "")
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(errors.Is(err, ErrUnreachable)).To(BeFalse(), "a server that replied is not unreachable, got %v", err)
|
||||
Expect(errors.Is(err, ErrUnauthorized)).To(BeFalse(), "500 is not an auth failure, got %v", err)
|
||||
})
|
||||
|
||||
// Pointing chat at some other service that happens to be listening is a
|
||||
// different problem from nothing listening, and needs different advice.
|
||||
It("does not call a reply it could not parse unreachable", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, err := w.Write([]byte("<html><body>not LocalAI</body></html>"))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
_, err := Probe(context.Background(), srv.URL+"/v1", "")
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(errors.Is(err, ErrUnreachable)).To(BeFalse(), "something answered, got %v", err)
|
||||
})
|
||||
|
||||
It("returns every advertised id, including ones that are not models", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
Expect(json.NewEncoder(w).Encode(map[string]any{
|
||||
"object": "list",
|
||||
"data": []map[string]string{
|
||||
{"id": "zeta", "object": "model"},
|
||||
{"id": ".gitignore", "object": "model"},
|
||||
{"id": "alpha", "object": "model"},
|
||||
{"id": "voice.tar.bz2", "object": "model"},
|
||||
},
|
||||
})).To(Succeed())
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
// Verbatim and in server order: deciding which of these are real, and
|
||||
// what order to show them in, belongs to the caller.
|
||||
models, err := Probe(context.Background(), srv.URL+"/v1", "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(models).To(Equal([]string{"zeta", ".gitignore", "alpha", "voice.tar.bz2"}))
|
||||
})
|
||||
|
||||
It("stops early when the context is already cancelled", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
Expect(json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": []any{}})).To(Succeed())
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
_, err := Probe(ctx, srv.URL+"/v1", "")
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(errors.Is(err, context.Canceled)).To(BeTrue(), "want the cancellation preserved, got %v", err)
|
||||
// A cancelled probe learned nothing about the endpoint, so it must not
|
||||
// send the caller off to start a server that may already be running.
|
||||
Expect(errors.Is(err, ErrUnreachable)).To(BeFalse(), "cancelling is not a verdict on the server, got %v", err)
|
||||
})
|
||||
|
||||
It("reports a server that never answers as unreachable", func() {
|
||||
release := make(chan struct{})
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
<-release
|
||||
}))
|
||||
defer srv.Close()
|
||||
defer close(release)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
_, err := Probe(ctx, srv.URL+"/v1", "")
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(errors.Is(err, ErrUnreachable)).To(BeTrue(), "want ErrUnreachable, got %v", err)
|
||||
Expect(errors.Is(err, context.DeadlineExceeded)).To(BeTrue(), "want the deadline preserved, got %v", err)
|
||||
})
|
||||
|
||||
It("returns an empty list when the server has no models", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
Expect(json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": []any{}})).To(Succeed())
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
models, err := Probe(context.Background(), srv.URL+"/v1", "")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(models).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
95
core/cli/chat/resolve.go
Normal file
95
core/cli/chat/resolve.go
Normal file
@@ -0,0 +1,95 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/mudler/xlog"
|
||||
)
|
||||
|
||||
// ModelChooser asks the user to pick one of models. It is nil when the session
|
||||
// is not interactive.
|
||||
type ModelChooser func(models []string) (string, error)
|
||||
|
||||
// ModelRequest is everything model resolution needs.
|
||||
type ModelRequest struct {
|
||||
Flag string // --model
|
||||
Configured string // model recorded in the agent config
|
||||
Available []string // models the server advertises
|
||||
StateDir string // where an interactive choice is persisted
|
||||
Choose ModelChooser // nil means non-interactive
|
||||
// Notify reports a problem that is worth telling the user about but not
|
||||
// worth failing over. Nil discards it. It exists because the one such
|
||||
// problem here, a choice that could not be saved, changes what the user
|
||||
// should expect next: they will be asked again. A log line does not reach
|
||||
// them, since the agent runs at log level error by default.
|
||||
Notify func(message string)
|
||||
}
|
||||
|
||||
// ResolveModel picks the model for this invocation. A flag or a configured
|
||||
// value wins outright and is not persisted; only an interactive choice is
|
||||
// written back, so the prompt appears at most once.
|
||||
//
|
||||
// Available is used exactly as the server gave it. LocalAI advertises stray
|
||||
// files it finds in the models directory alongside real models, but real model
|
||||
// IDs contain dots too (lfm2.5-8b-a1b), so any client-side "looks like a
|
||||
// filename" heuristic would eventually hide a model the user has. Deciding
|
||||
// which advertised IDs are real belongs to the endpoint, not to a guess here.
|
||||
func ResolveModel(req ModelRequest) (string, error) {
|
||||
if req.Flag != "" {
|
||||
return req.Flag, nil
|
||||
}
|
||||
if req.Configured != "" {
|
||||
return req.Configured, nil
|
||||
}
|
||||
|
||||
// The server's /v1/models ordering is not stable between calls, so sort
|
||||
// before showing or listing: the same number must mean the same model on
|
||||
// the next run. Sort a copy; the caller's slice is not ours to reorder.
|
||||
available := append([]string(nil), req.Available...)
|
||||
sort.Strings(available)
|
||||
|
||||
switch len(available) {
|
||||
case 0:
|
||||
return "", errors.New("the LocalAI server has no models installed. Install one with 'local-ai models install <name>', then run 'local-ai chat' again")
|
||||
case 1:
|
||||
return available[0], nil
|
||||
}
|
||||
|
||||
if req.Choose == nil {
|
||||
return "", fmt.Errorf(
|
||||
"several models are available; pick one with --model. Available: %s",
|
||||
strings.Join(available, ", "),
|
||||
)
|
||||
}
|
||||
|
||||
chosen, err := req.Choose(available)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
// Choose is an interface, so its answer is checked rather than trusted.
|
||||
// What comes back is persisted and every later run starts against it, so a
|
||||
// chooser that returns an empty string or a name of its own would record a
|
||||
// model the server never offered and there would be nothing left to catch
|
||||
// it.
|
||||
if !slices.Contains(available, chosen) {
|
||||
return "", fmt.Errorf(
|
||||
"the model chooser answered %q, which is not one of the available models: %s",
|
||||
chosen, strings.Join(available, ", "),
|
||||
)
|
||||
}
|
||||
if req.StateDir != "" {
|
||||
if err := PersistModel(req.StateDir, chosen); err != nil {
|
||||
// A failure to remember the choice must not block the session: the
|
||||
// user picked a model, so honour it and say what will happen.
|
||||
xlog.Warn("could not save the model choice", "error", err, "model", chosen)
|
||||
if req.Notify != nil {
|
||||
req.Notify(fmt.Sprintf("Your choice of %s could not be saved, so this question comes back next time: %v", chosen, err))
|
||||
}
|
||||
}
|
||||
}
|
||||
return chosen, nil
|
||||
}
|
||||
156
core/cli/chat/resolve_test.go
Normal file
156
core/cli/chat/resolve_test.go
Normal file
@@ -0,0 +1,156 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("ResolveModel", func() {
|
||||
It("prefers the flag over everything", func() {
|
||||
got, err := ResolveModel(ModelRequest{
|
||||
Flag: "from-flag",
|
||||
Configured: "from-config",
|
||||
Available: []string{"a", "b"},
|
||||
})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(got).To(Equal("from-flag"))
|
||||
})
|
||||
|
||||
It("uses the configured model when no flag is given", func() {
|
||||
got, err := ResolveModel(ModelRequest{
|
||||
Configured: "from-config",
|
||||
Available: []string{"a", "b"},
|
||||
})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(got).To(Equal("from-config"))
|
||||
})
|
||||
|
||||
It("auto-selects when the server offers exactly one model", func() {
|
||||
got, err := ResolveModel(ModelRequest{Available: []string{"only-one"}})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(got).To(Equal("only-one"))
|
||||
})
|
||||
|
||||
It("errors and lists the options when several models exist and there is no chooser", func() {
|
||||
_, err := ResolveModel(ModelRequest{Available: []string{"a", "b"}})
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("a"))
|
||||
Expect(err.Error()).To(ContainSubstring("b"))
|
||||
Expect(err.Error()).To(ContainSubstring("--model"))
|
||||
})
|
||||
|
||||
It("sorts before offering, so the same number means the same model next run", func() {
|
||||
var offered []string
|
||||
available := []string{"zeta", "alpha", "mid"}
|
||||
_, err := ResolveModel(ModelRequest{
|
||||
Available: available,
|
||||
StateDir: GinkgoT().TempDir(),
|
||||
Choose: func(models []string) (string, error) {
|
||||
offered = models
|
||||
return models[0], nil
|
||||
},
|
||||
})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
// The server's /v1/models ordering is unstable between calls.
|
||||
Expect(offered).To(Equal([]string{"alpha", "mid", "zeta"}))
|
||||
// Sorting must happen on a copy: the caller still owns this slice, and
|
||||
// reordering it under them would move whatever they index into it.
|
||||
Expect(available).To(Equal([]string{"zeta", "alpha", "mid"}))
|
||||
})
|
||||
|
||||
It("lists models in sorted order in the several-models error", func() {
|
||||
_, err := ResolveModel(ModelRequest{Available: []string{"zeta", "alpha"}})
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("alpha, zeta"))
|
||||
})
|
||||
|
||||
It("asks the chooser when several models exist, and persists the answer", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
got, err := ResolveModel(ModelRequest{
|
||||
Available: []string{"a", "b"},
|
||||
StateDir: dir,
|
||||
Choose: func(models []string) (string, error) { return models[1], nil },
|
||||
})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(got).To(Equal("b"))
|
||||
|
||||
data, err := os.ReadFile(ConfigPath(dir))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(string(data)).To(ContainSubstring("model: b"))
|
||||
})
|
||||
|
||||
// The answer is persisted and every later run starts against it, and
|
||||
// ModelChooser is exported, so the invariant has to hold for choosers this
|
||||
// package did not write.
|
||||
DescribeTable("refuses an answer the chooser was not offered",
|
||||
func(answer string) {
|
||||
dir := GinkgoT().TempDir()
|
||||
got, err := ResolveModel(ModelRequest{
|
||||
Available: []string{"alpha", "zeta"},
|
||||
StateDir: dir,
|
||||
Choose: func([]string) (string, error) { return answer, nil },
|
||||
})
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(got).To(BeEmpty())
|
||||
Expect(err.Error()).To(ContainSubstring("alpha, zeta"))
|
||||
|
||||
_, statErr := os.Stat(ConfigPath(dir))
|
||||
Expect(os.IsNotExist(statErr)).To(BeTrue(), "nothing may be recorded for an answer that was refused")
|
||||
},
|
||||
Entry("nothing at all", ""),
|
||||
Entry("a model the server never offered", "gamma"),
|
||||
Entry("an offered model with stray whitespace", " alpha"),
|
||||
Entry("an offered model in the wrong case", "Alpha"),
|
||||
)
|
||||
|
||||
It("notifies, and still honours the choice, when it cannot be persisted", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
// A directory where the config file belongs: the write fails for any
|
||||
// user, including root.
|
||||
Expect(os.MkdirAll(ConfigPath(dir), 0o700)).To(Succeed())
|
||||
|
||||
var notices []string
|
||||
got, err := ResolveModel(ModelRequest{
|
||||
Available: []string{"a", "b"},
|
||||
StateDir: dir,
|
||||
Choose: func(models []string) (string, error) { return models[0], nil },
|
||||
Notify: func(message string) { notices = append(notices, message) },
|
||||
})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(got).To(Equal("a"))
|
||||
Expect(notices).To(HaveLen(1))
|
||||
Expect(notices[0]).To(ContainSubstring("a"))
|
||||
Expect(notices[0]).To(ContainSubstring("could not be saved"))
|
||||
})
|
||||
|
||||
It("says nothing when the choice was saved", func() {
|
||||
var notices []string
|
||||
_, err := ResolveModel(ModelRequest{
|
||||
Available: []string{"a", "b"},
|
||||
StateDir: GinkgoT().TempDir(),
|
||||
Choose: func(models []string) (string, error) { return models[0], nil },
|
||||
Notify: func(message string) { notices = append(notices, message) },
|
||||
})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(notices).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("propagates a chooser cancellation", func() {
|
||||
cancelled := errors.New("cancelled")
|
||||
_, err := ResolveModel(ModelRequest{
|
||||
Available: []string{"a", "b"},
|
||||
StateDir: GinkgoT().TempDir(),
|
||||
Choose: func([]string) (string, error) { return "", cancelled },
|
||||
})
|
||||
Expect(errors.Is(err, cancelled)).To(BeTrue())
|
||||
})
|
||||
|
||||
It("errors with an install hint when the server has no models", func() {
|
||||
_, err := ResolveModel(ModelRequest{Available: nil})
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("local-ai models install"))
|
||||
})
|
||||
})
|
||||
475
core/cli/chat/run.go
Normal file
475
core/cli/chat/run.go
Normal file
@@ -0,0 +1,475 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/nib/app"
|
||||
nibcmd "github.com/mudler/nib/cmd"
|
||||
nibconfig "github.com/mudler/nib/config"
|
||||
nibtypes "github.com/mudler/nib/types"
|
||||
"golang.org/x/term"
|
||||
)
|
||||
|
||||
// Options is everything the chat command passes down from its flags.
|
||||
type Options struct {
|
||||
Args []string // forwarded to the agent verbatim
|
||||
Endpoint string // the server root, e.g. http://127.0.0.1:8080
|
||||
BaseURL string // the API base, e.g. http://127.0.0.1:8080/v1
|
||||
APIKey string
|
||||
Model string
|
||||
StateDir string
|
||||
TraceDir string
|
||||
Yolo bool
|
||||
// ProbeTimeout bounds each check of the server. Zero means
|
||||
// defaultProbeTimeout.
|
||||
ProbeTimeout time.Duration
|
||||
|
||||
In io.Reader
|
||||
Out io.Writer
|
||||
ErrOut io.Writer
|
||||
}
|
||||
|
||||
// ExitStatus reports the status the process should exit with for an agent run
|
||||
// that failed, and whether err is such a failure.
|
||||
//
|
||||
// nib writes what went wrong to the error stream itself and hands back nothing
|
||||
// but a code, so an error that satisfies this has already been explained to the
|
||||
// user and must not be reported a second time. The refusal to open a
|
||||
// full-screen session on a stdin that cannot be read arrives this way, and it
|
||||
// is the one a user is most likely to meet: 'echo q | local-ai chat' names
|
||||
// --cli, and burying that under a second message would hide the fix.
|
||||
func ExitStatus(err error) (int, bool) {
|
||||
var exit app.ExitError
|
||||
if errors.As(err, &exit) {
|
||||
return exit.Code, true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// shutdownSignals end the session. SIGHUP is one of them because this is a
|
||||
// terminal program: once the terminal is gone there is nobody left to talk to,
|
||||
// and a server started for the session has to go with it.
|
||||
var shutdownSignals = []os.Signal{os.Interrupt, syscall.SIGTERM, syscall.SIGHUP}
|
||||
|
||||
// shutdownContext derives a context that is cancelled when the process is
|
||||
// asked to stop.
|
||||
//
|
||||
// Without it a signal kills this process where it stands, skipping every
|
||||
// deferred call, and a 'local-ai run' started for the session is reparented to
|
||||
// init with nothing left that knows to shut it down. An interactive Ctrl+C is
|
||||
// safe on its own, because the child shares this process' foreground process
|
||||
// group and the terminal signals all of it, but a SIGTERM from a supervisor or
|
||||
// a script reaches only this process.
|
||||
//
|
||||
// Since nib v0.5.1 cancelling this context does end the session: RunTUI passes
|
||||
// it to bubbletea, which unwinds the program and reports the context's own
|
||||
// error. The server is still stopped on cancellation rather than on the way
|
||||
// out (see runSession), because registering here removes SIGHUP's default
|
||||
// terminate disposition, and a guarantee about a server this process owns is
|
||||
// not worth resting on how promptly a third party unwinds its interface.
|
||||
//
|
||||
// A handler rather than SysProcAttr.Pdeathsig on the child: Pdeathsig is
|
||||
// Linux-only, and in Go it is delivered when the OS thread that forked exits
|
||||
// rather than when the process does, so it can fire on a perfectly healthy
|
||||
// parent. Setpgid is not an alternative either, since taking the child out of
|
||||
// the foreground process group is what would break the Ctrl+C that works
|
||||
// today. SIGKILL stays uncovered, as it must: nothing in the process can
|
||||
// observe it.
|
||||
func shutdownContext(parent context.Context) (context.Context, context.CancelFunc) {
|
||||
return signal.NotifyContext(parent, shutdownSignals...)
|
||||
}
|
||||
|
||||
// Run starts the agent: resolve where state lives, make sure a server is
|
||||
// reachable, pick a model, then hand off to nib.
|
||||
func Run(ctx context.Context, opts Options) error {
|
||||
ctx, stop := shutdownContext(ctx)
|
||||
defer stop()
|
||||
|
||||
p, err := prepare(ctx, opts, isTerminal(opts.In))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// A server this process started belongs to this session, and Stop is
|
||||
// nil-safe and idempotent, so one defer covers both cases and costs nothing
|
||||
// when runSession has already stopped it.
|
||||
defer p.server.Stop()
|
||||
|
||||
return runSession(ctx, p.server, func(ctx context.Context) error {
|
||||
return runAgent(ctx, p.dir, p.model, opts)
|
||||
})
|
||||
}
|
||||
|
||||
// runSession hands the terminal to agent, and stops a server started for this
|
||||
// session as soon as the context is cancelled rather than when agent returns.
|
||||
//
|
||||
// The difference matters because the deferred Stop in Run is only reached once
|
||||
// agent returns, and how long that takes is nib's business rather than ours.
|
||||
// nib v0.5.1 does unwind the TUI on a cancelled context, so it does return; a
|
||||
// SIGHUP no longer leaves the interface on screen with the server behind it,
|
||||
// which it did before, when bubbletea's own SIGINT and SIGTERM handler was the
|
||||
// only thing that ever quit the program and registering for SIGHUP had removed
|
||||
// the default disposition that used to end the process. Watching the context
|
||||
// keeps the guarantee independent of what the agent does with it.
|
||||
func runSession(ctx context.Context, server *StartedServer, agent func(context.Context) error) error {
|
||||
returned := make(chan struct{})
|
||||
defer close(returned)
|
||||
|
||||
go func() {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
server.Stop()
|
||||
case <-returned:
|
||||
}
|
||||
}()
|
||||
|
||||
return agent(ctx)
|
||||
}
|
||||
|
||||
// preparation is what the agent needs once the environment is ready: where its
|
||||
// state lives, which model to talk to, and the server this process started on
|
||||
// the user's behalf, if any.
|
||||
type preparation struct {
|
||||
dir string
|
||||
model string
|
||||
server *StartedServer
|
||||
}
|
||||
|
||||
// prepare does everything that has to happen before the agent takes over the
|
||||
// terminal. It is split out of Run because all of it is testable and none of
|
||||
// what follows is: once app.Run has the terminal there is no seam left.
|
||||
//
|
||||
// interactive says whether there is a user to prompt. It is a parameter rather
|
||||
// than a second read of opts.In so the prompts can be driven over a pipe.
|
||||
func prepare(ctx context.Context, opts Options, interactive bool) (_ *preparation, err error) {
|
||||
dir, dirErr := StateDir(opts.StateDir)
|
||||
if dirErr != nil {
|
||||
return nil, dirErr
|
||||
}
|
||||
if err := EnsureStateDir(dir, opts.BaseURL); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if isLocalOnlyArgs(opts.Args) {
|
||||
return &preparation{dir: dir}, nil
|
||||
}
|
||||
|
||||
// One prompter for every question this run asks; see its doc comment for
|
||||
// why the reader cannot be rebuilt per question.
|
||||
var prompts *prompter
|
||||
if interactive {
|
||||
prompts = newPrompter(opts.In, opts.ErrOut)
|
||||
}
|
||||
|
||||
var started *StartedServer
|
||||
defer func() {
|
||||
// Nothing after the spawn may leave a server behind: the caller only
|
||||
// learns about it through a successful return.
|
||||
if err != nil {
|
||||
started.Stop()
|
||||
}
|
||||
}()
|
||||
|
||||
models, err := probeModels(ctx, opts)
|
||||
if err != nil {
|
||||
if errors.Is(err, ErrUnauthorized) {
|
||||
return nil, fmt.Errorf("the LocalAI server at %s rejected the API key. Pass --api-key or set LOCALAI_API_KEY", opts.Endpoint)
|
||||
}
|
||||
if !errors.Is(err, ErrUnreachable) {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var confirm Confirmer
|
||||
if interactive {
|
||||
confirm = prompts.yesNo
|
||||
}
|
||||
var startErr error
|
||||
started, startErr = OfferToStart(ctx, StartOptions{
|
||||
Endpoint: opts.Endpoint,
|
||||
Confirm: confirm,
|
||||
Stderr: opts.ErrOut,
|
||||
})
|
||||
if startErr != nil {
|
||||
err = startErr
|
||||
if errors.Is(startErr, ErrDeclined) {
|
||||
err = fmt.Errorf("no LocalAI server at %s. Start one with 'local-ai run', or point elsewhere with --endpoint", opts.Endpoint)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
say(opts.ErrOut, "Started a temporary LocalAI server; it stops when you exit. Use 'local-ai run' for a persistent one.\n")
|
||||
|
||||
if models, err = probeModels(ctx, opts); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
var chooser ModelChooser
|
||||
if interactive {
|
||||
chooser = prompts.choose
|
||||
}
|
||||
model, err := ResolveModel(ModelRequest{
|
||||
Flag: opts.Model,
|
||||
Configured: configuredModel(dir),
|
||||
Available: models,
|
||||
StateDir: dir,
|
||||
Choose: chooser,
|
||||
Notify: func(message string) { say(opts.ErrOut, "%s\n", message) },
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &preparation{dir: dir, model: model, server: started}, nil
|
||||
}
|
||||
|
||||
func runAgent(ctx context.Context, dir, model string, opts Options) error {
|
||||
return app.Run(ctx, agentOptions(dir, model, opts))
|
||||
}
|
||||
|
||||
// agentOptions builds the request handed to nib. It is split out of runAgent
|
||||
// because app.Run takes the terminal and cannot be called from a test, while
|
||||
// what is asked of it is exactly the part worth pinning.
|
||||
//
|
||||
// The stream fields are the interesting ones, and they are not symmetric.
|
||||
//
|
||||
// nib reads a non-nil stream as "the embedder wants this used", and refuses
|
||||
// every mode but --cli when such a stream is not a terminal, because the
|
||||
// full-screen interface renders on /dev/tty and would otherwise ignore it in
|
||||
// silence. Nil means "not injected": nib falls back to the process stream and
|
||||
// behaves as standalone nib does.
|
||||
//
|
||||
// Stdin is passed through as it comes. A piped or redirected stdin really is
|
||||
// ignored by the interface, so the refusal is the honest answer there, and it
|
||||
// is the one users meet: 'echo q | local-ai chat' says to re-run with --cli
|
||||
// rather than opening a full-screen session that will never read the question.
|
||||
//
|
||||
// Stdout is different, and the process stream is deliberately sent as nil. The
|
||||
// interface does write to stdout even when it is a pipe: that is the whole of
|
||||
// nib's shell-capture idiom, out=$(local-ai chat --height 50%), which is what
|
||||
// the Ctrl+Space widget emitted by --init is built on. Injecting os.Stdout
|
||||
// there would refuse the widget for a stream nib was going to use anyway.
|
||||
//
|
||||
// The test is identity with os.Stdout rather than whether it happens to be a
|
||||
// terminal, which means a shell redirect goes the same way as the widget:
|
||||
// 'local-ai chat > out.txt' no longer refuses either, and renders on /dev/tty
|
||||
// with the capture line landing in the file. That is not a second decision, it
|
||||
// is the same one. Both are the process stdout as the shell handed it over,
|
||||
// differing only in being a pipe rather than a regular file, which nib's gate
|
||||
// does not look at and should not. Refusing one would refuse the other.
|
||||
//
|
||||
// What stays injected, and so stays subject to the refusal, is a writer some
|
||||
// in-process caller chose for itself rather than inherited: a bytes.Buffer, or
|
||||
// an *os.File it opened. The specs rely on that.
|
||||
//
|
||||
// Stderr is never gated by nib, so it is passed through unchanged.
|
||||
//
|
||||
// The config values go through Overrides rather than Defaults, and that is not
|
||||
// a detail. Defaults are seeds: they sit BENEATH the config file, so the file
|
||||
// silently undoes them. Everything here is a decision this invocation already
|
||||
// made on the user's behalf, and a flag that the file can undo is not a flag.
|
||||
// It was not a rare case either, since EnsureStateDir writes base_url on the
|
||||
// first run and an interactive choice writes model, so from the second run on
|
||||
// the file carried a value for both and --endpoint and --model did nothing.
|
||||
//
|
||||
// The one asymmetry to plan around is that nib cannot tell "set to the zero
|
||||
// value" from "not set", so an override only ever raises a field. --yolo can
|
||||
// turn approval off, but nothing on the command line can turn it back on over
|
||||
// an approval_mode: auto in the file; that needs a config edit. Same shape for
|
||||
// the strings, which is what makes an unset --api-key or --trace-dir leave the
|
||||
// file's value standing, as it should.
|
||||
//
|
||||
// nib's own --trace-dir and --yolo, and their NIB_TRACE_DIR and NIB_YOLO twins,
|
||||
// are resolved after the config load and so still outrank these. That is
|
||||
// deliberate upstream: they are instructions to nib rather than ambient
|
||||
// environment.
|
||||
func agentOptions(dir, model string, opts Options) app.Options {
|
||||
// Model is the model this run resolved, which already prefers --model and
|
||||
// falls back to the file's own model, so the override restates the file's
|
||||
// value rather than fighting it whenever no flag was given.
|
||||
//
|
||||
// BaseURL is the endpoint this run probed, offered to start a server for,
|
||||
// and seeded the config with. Handing nib a different one is precisely the
|
||||
// split that made --endpoint a no-op, so the agent talks to the server
|
||||
// LocalAI checked. Pointing somewhere else for good is LOCALAI_CHAT_ENDPOINT
|
||||
// or --endpoint, not a hand-edited base_url the probe never reads.
|
||||
//
|
||||
// APIKey and TraceDir are the flags as given, empty when they were not, and
|
||||
// an empty override leaves the file alone. TraceDir is runtime-only in nib
|
||||
// (yaml:"-"), so no file value exists for it to beat today; it belongs here
|
||||
// with the other flags rather than one rung down for a reason that could
|
||||
// quietly stop being true.
|
||||
overrides := nibtypes.Config{
|
||||
Model: model,
|
||||
APIKey: opts.APIKey,
|
||||
BaseURL: opts.BaseURL,
|
||||
TraceDir: opts.TraceDir,
|
||||
}
|
||||
if opts.Yolo {
|
||||
overrides.ApprovalMode = "auto"
|
||||
}
|
||||
|
||||
return app.Options{
|
||||
Args: opts.Args,
|
||||
ProgramName: "local-ai chat",
|
||||
BaseDir: dir,
|
||||
Overrides: overrides,
|
||||
SkipSetup: true,
|
||||
SkipBareEnv: true,
|
||||
Stdin: opts.In,
|
||||
Stdout: ownStdout(opts.Out),
|
||||
Stderr: opts.ErrOut,
|
||||
}
|
||||
}
|
||||
|
||||
// ownStdout reports the writer as nib's own rather than as an injected one when
|
||||
// it is the process stdout, by answering nil for it. See agentOptions for why
|
||||
// that distinction is the difference between a working Ctrl+Space widget and a
|
||||
// refused one.
|
||||
func ownStdout(w io.Writer) io.Writer {
|
||||
if f, ok := w.(*os.File); ok && f == os.Stdout {
|
||||
return nil
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// defaultProbeTimeout bounds a check of the server. Listing models is cheap,
|
||||
// so this is long enough that a loaded server is never given up on and short
|
||||
// enough that a hung one does not leave the user staring at nothing.
|
||||
const defaultProbeTimeout = 30 * time.Second
|
||||
|
||||
// probeModels lists what the endpoint offers, under a budget.
|
||||
func probeModels(ctx context.Context, opts Options) ([]string, error) {
|
||||
timeout := opts.ProbeTimeout
|
||||
if timeout <= 0 {
|
||||
timeout = defaultProbeTimeout
|
||||
}
|
||||
// A real deadline rather than a cancel plus a timer. Probe reads
|
||||
// context.Canceled as "the caller gave up", which is a statement about the
|
||||
// caller and not about the endpoint, and only a deadline as "nothing
|
||||
// answered in time". Expiring the budget as a cancellation would stop
|
||||
// ErrUnreachable firing for precisely the hung servers that the offer to
|
||||
// start one exists for.
|
||||
probeCtx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
return Probe(probeCtx, opts.BaseURL, opts.APIKey)
|
||||
}
|
||||
|
||||
// isLocalOnlyArgs reports whether the forwarded arguments do their work
|
||||
// without ever reaching a model, in which case demanding a running server (and
|
||||
// offering to start one) would be an obstacle rather than a service.
|
||||
//
|
||||
// Two groups qualify. The management subcommands edit nib's own state: plugin,
|
||||
// skill, and the mcp verbs that add or remove configured servers, which is
|
||||
// asked of nib rather than restated, because bare 'mcp' and its transport
|
||||
// flags do serve the agent and do need a model. The other group is the flags
|
||||
// that only print something, above all --init: its shell snippet goes into an
|
||||
// rc file, typically long before any server exists.
|
||||
func isLocalOnlyArgs(args []string) bool {
|
||||
if len(args) == 0 {
|
||||
return false
|
||||
}
|
||||
// A scan rather than a look at args[0]: the mode flags this command
|
||||
// translates are prepended, so --init is not necessarily first. Positional
|
||||
// text cannot be mistaken for a flag here, since nib ignores what is left
|
||||
// after flag parsing.
|
||||
for _, a := range args {
|
||||
switch {
|
||||
case a == "--init", a == "-init", strings.HasPrefix(a, "--init="), strings.HasPrefix(a, "-init="):
|
||||
return true
|
||||
case a == "--version", a == "-version":
|
||||
return true
|
||||
}
|
||||
}
|
||||
switch args[0] {
|
||||
case "plugin", "skill":
|
||||
return true
|
||||
case "mcp":
|
||||
return len(args) >= 2 && nibcmd.IsMCPManageSubcommand(args[1])
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// configuredModel reads the model already recorded in the agent config, if any.
|
||||
func configuredModel(dir string) string {
|
||||
cfg := nibconfig.LoadWith(nibconfig.LoadOptions{BaseDir: dir, SkipBareEnv: true})
|
||||
return cfg.Model
|
||||
}
|
||||
|
||||
func isTerminal(in io.Reader) bool {
|
||||
f, ok := in.(*os.File)
|
||||
return ok && term.IsTerminal(int(f.Fd()))
|
||||
}
|
||||
|
||||
// say writes a line of interactive chatter: a question, or a notice about
|
||||
// something that did not stop the session. A write that fails is not worth
|
||||
// failing over, and when the terminal really is gone the read that follows the
|
||||
// question says so.
|
||||
func say(w io.Writer, format string, args ...any) {
|
||||
_, _ = fmt.Fprintf(w, format, args...)
|
||||
}
|
||||
|
||||
// prompter asks this run's questions on the user's terminal.
|
||||
//
|
||||
// It owns the buffered reader rather than wrapping opts.In per question,
|
||||
// because bufio reads ahead: a throwaway reader for the "start a server?"
|
||||
// question swallows the model choice that was typed behind it, and the next
|
||||
// question then sees EOF. A real run asks both, one after the other.
|
||||
type prompter struct {
|
||||
in *bufio.Reader
|
||||
out io.Writer
|
||||
}
|
||||
|
||||
func newPrompter(in io.Reader, out io.Writer) *prompter {
|
||||
return &prompter{in: bufio.NewReader(in), out: out}
|
||||
}
|
||||
|
||||
// yesNo satisfies Confirmer. Anything that is not an explicit yes is a no, so
|
||||
// a closed stream declines rather than proceeding on the user's behalf.
|
||||
func (p *prompter) yesNo(question string) (bool, error) {
|
||||
say(p.out, "%s [y/N]: ", question)
|
||||
line, err := p.in.ReadString('\n')
|
||||
if err != nil && !errors.Is(err, io.EOF) {
|
||||
return false, fmt.Errorf("reading the answer: %w", err)
|
||||
}
|
||||
switch strings.ToLower(strings.TrimSpace(line)) {
|
||||
case "y", "yes":
|
||||
return true, nil
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// choose satisfies ModelChooser. It answers with a list index rather than with
|
||||
// what the user typed, so the result can only ever be one of the models it was
|
||||
// offered: a model name is not something to accept unvalidated here, since
|
||||
// ResolveModel persists whatever comes back and every later run then starts
|
||||
// against it.
|
||||
func (p *prompter) choose(models []string) (string, error) {
|
||||
if len(models) == 0 {
|
||||
return "", errors.New("there is nothing to choose from")
|
||||
}
|
||||
say(p.out, "Several models are available:\n")
|
||||
for i, m := range models {
|
||||
say(p.out, " %d) %s\n", i+1, m)
|
||||
}
|
||||
say(p.out, "Pick one [1-%d]: ", len(models))
|
||||
|
||||
line, err := p.in.ReadString('\n')
|
||||
if err != nil && !errors.Is(err, io.EOF) {
|
||||
return "", fmt.Errorf("reading the choice: %w", err)
|
||||
}
|
||||
answer := strings.TrimSpace(line)
|
||||
n, err := strconv.Atoi(answer)
|
||||
if err != nil || n < 1 || n > len(models) {
|
||||
return "", fmt.Errorf("not a valid choice: %q. Pick a number between 1 and %d, or pass --model", answer, len(models))
|
||||
}
|
||||
return models[n-1], nil
|
||||
}
|
||||
629
core/cli/chat/run_test.go
Normal file
629
core/cli/chat/run_test.go
Normal file
@@ -0,0 +1,629 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/nib/app"
|
||||
nibconfig "github.com/mudler/nib/config"
|
||||
nibtypes "github.com/mudler/nib/types"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// modelServer answers /v1/models with the given ids, as LocalAI does.
|
||||
func modelServer(ids ...string) *httptest.Server {
|
||||
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
data := make([]map[string]string, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
data = append(data, map[string]string{"id": id, "object": "model"})
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
Expect(json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": data})).To(Succeed())
|
||||
}))
|
||||
}
|
||||
|
||||
var _ = Describe("prepare", func() {
|
||||
var (
|
||||
dir string
|
||||
errOut *bytes.Buffer
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
dir = GinkgoT().TempDir()
|
||||
errOut = &bytes.Buffer{}
|
||||
})
|
||||
|
||||
// optionsFor points a run at srv, with no input to read: the default is a
|
||||
// session nobody can be asked anything in.
|
||||
optionsFor := func(srv *httptest.Server) Options {
|
||||
endpoint := "http://127.0.0.1:0"
|
||||
base := endpoint + "/v1"
|
||||
if srv != nil {
|
||||
endpoint, base = srv.URL, srv.URL+"/v1"
|
||||
}
|
||||
return Options{
|
||||
Endpoint: endpoint,
|
||||
BaseURL: base,
|
||||
StateDir: dir,
|
||||
In: strings.NewReader(""),
|
||||
Out: &bytes.Buffer{},
|
||||
ErrOut: errOut,
|
||||
}
|
||||
}
|
||||
|
||||
It("uses the only model the server offers", func() {
|
||||
srv := modelServer("the-only-model")
|
||||
defer srv.Close()
|
||||
|
||||
p, err := prepare(context.Background(), optionsFor(srv), false)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(p.model).To(Equal("the-only-model"))
|
||||
Expect(p.dir).To(Equal(dir))
|
||||
Expect(p.server).To(BeNil(), "nothing was started, so nothing is owned")
|
||||
})
|
||||
|
||||
It("seeds the agent config with the endpoint on first run", func() {
|
||||
srv := modelServer("m")
|
||||
defer srv.Close()
|
||||
|
||||
_, err := prepare(context.Background(), optionsFor(srv), false)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
|
||||
data, err := os.ReadFile(ConfigPath(dir))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(string(data)).To(ContainSubstring(srv.URL + "/v1"))
|
||||
})
|
||||
|
||||
It("lets --model win over what the server offers", func() {
|
||||
srv := modelServer("a", "b")
|
||||
defer srv.Close()
|
||||
|
||||
opts := optionsFor(srv)
|
||||
opts.Model = "not-listed-yet"
|
||||
p, err := prepare(context.Background(), opts, false)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(p.model).To(Equal("not-listed-yet"))
|
||||
})
|
||||
|
||||
It("advises about the API key when the server rejects it", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
_, err := prepare(context.Background(), optionsFor(srv), false)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("--api-key"))
|
||||
Expect(err.Error()).To(ContainSubstring(srv.URL))
|
||||
})
|
||||
|
||||
// Not interactive means nobody can answer the offer, so the advice has to
|
||||
// stand on its own.
|
||||
It("advises how to start a server when none is reachable", func() {
|
||||
srv := modelServer()
|
||||
url := srv.URL
|
||||
srv.Close() // nothing is listening now
|
||||
|
||||
opts := optionsFor(nil)
|
||||
opts.Endpoint, opts.BaseURL = url, url+"/v1"
|
||||
_, err := prepare(context.Background(), opts, false)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("local-ai run"))
|
||||
Expect(err.Error()).To(ContainSubstring(url))
|
||||
})
|
||||
|
||||
// A server that accepts the connection and then never replies is the case
|
||||
// the offer to start one exists for, so the budget has to expire as a
|
||||
// deadline: Probe reads a cancellation as "the caller gave up" and refuses
|
||||
// to call the endpoint unreachable on the strength of it.
|
||||
It("treats a server that never answers as one that is not there", func(ctx SpecContext) {
|
||||
release := make(chan struct{})
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
select {
|
||||
case <-release:
|
||||
case <-r.Context().Done():
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
defer close(release)
|
||||
|
||||
opts := optionsFor(srv)
|
||||
opts.ProbeTimeout = 100 * time.Millisecond
|
||||
_, err := prepare(context.Background(), opts, false)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("local-ai run"), "want the offer-a-server advice, got %v", err)
|
||||
}, SpecTimeout(30*time.Second))
|
||||
|
||||
It("asks which model to use and remembers the answer", func() {
|
||||
srv := modelServer("zeta", "alpha")
|
||||
defer srv.Close()
|
||||
|
||||
opts := optionsFor(srv)
|
||||
opts.In = strings.NewReader("2\n")
|
||||
p, err := prepare(context.Background(), opts, true)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
// The list is sorted before it is shown, so 2 is zeta, not the second
|
||||
// thing the server happened to name.
|
||||
Expect(p.model).To(Equal("zeta"))
|
||||
Expect(errOut.String()).To(ContainSubstring("1) alpha"))
|
||||
Expect(errOut.String()).To(ContainSubstring("2) zeta"))
|
||||
|
||||
data, err := os.ReadFile(ConfigPath(dir))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(string(data)).To(ContainSubstring("zeta"))
|
||||
})
|
||||
|
||||
// The choice is prompted for once and remembered. When remembering it fails
|
||||
// the user is about to be asked again on every future run, so they have to
|
||||
// be told here: a log line is invisible at the default log level.
|
||||
It("says so on the prompt when the choice cannot be remembered", func() {
|
||||
srv := modelServer("zeta", "alpha")
|
||||
defer srv.Close()
|
||||
|
||||
// A directory where the config file belongs: writable state dir,
|
||||
// unwritable config, on any platform and as any user.
|
||||
Expect(os.MkdirAll(ConfigPath(dir), 0o700)).To(Succeed())
|
||||
|
||||
opts := optionsFor(srv)
|
||||
opts.In = strings.NewReader("1\n")
|
||||
p, err := prepare(context.Background(), opts, true)
|
||||
|
||||
// Failing to remember the choice must not cost the user their session.
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(p.model).To(Equal("alpha"))
|
||||
Expect(errOut.String()).To(ContainSubstring("could not be saved"), "the user has to learn they will be asked again")
|
||||
})
|
||||
|
||||
It("does not ask again once a model is recorded", func() {
|
||||
srv := modelServer("zeta", "alpha")
|
||||
defer srv.Close()
|
||||
|
||||
Expect(PersistModel(dir, "alpha")).To(Succeed())
|
||||
|
||||
opts := optionsFor(srv)
|
||||
opts.In = strings.NewReader("") // an answer would have nothing to read
|
||||
p, err := prepare(context.Background(), opts, true)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(p.model).To(Equal("alpha"))
|
||||
Expect(errOut.String()).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("says what to install when the server has no models", func() {
|
||||
srv := modelServer()
|
||||
defer srv.Close()
|
||||
|
||||
_, err := prepare(context.Background(), optionsFor(srv), false)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("models install"))
|
||||
})
|
||||
|
||||
Describe("arguments that only touch local state", func() {
|
||||
unreachable := func(args ...string) Options {
|
||||
opts := optionsFor(nil) // port 0: nothing can ever answer here
|
||||
opts.Args = args
|
||||
return opts
|
||||
}
|
||||
|
||||
DescribeTable("skips the server entirely",
|
||||
func(args ...string) {
|
||||
p, err := prepare(context.Background(), unreachable(args...), false)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(p.model).To(BeEmpty())
|
||||
Expect(p.server).To(BeNil())
|
||||
},
|
||||
Entry("plugin", "plugin", "list"),
|
||||
Entry("skill", "skill", "list"),
|
||||
Entry("mcp add", "mcp", "add", "srv"),
|
||||
Entry("mcp list", "mcp", "list"),
|
||||
// The shell snippet is what a user puts in their rc file, long
|
||||
// before any server exists.
|
||||
Entry("the shell integration script", "--init", "zsh"),
|
||||
Entry("the version", "--version"),
|
||||
)
|
||||
|
||||
// Bare 'mcp' and its transport flags serve the agent over MCP, so they
|
||||
// need a model like any other session. Only the verbs that edit the
|
||||
// configured servers are local.
|
||||
DescribeTable("still needs a server",
|
||||
func(args ...string) {
|
||||
_, err := prepare(context.Background(), unreachable(args...), false)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("local-ai run"))
|
||||
},
|
||||
Entry("mcp over stdio", "mcp", "--stdio"),
|
||||
Entry("bare mcp", "mcp"),
|
||||
)
|
||||
})
|
||||
|
||||
// A reader per question would read ahead into a buffer it then discards, so
|
||||
// the second question would see EOF whenever both answers were typed ahead.
|
||||
// That is the shape of a real run: the offer to start a server is followed
|
||||
// by the model prompt.
|
||||
It("keeps reading answers from the same stream across questions", func() {
|
||||
out := &bytes.Buffer{}
|
||||
p := newPrompter(strings.NewReader("y\n2\n"), out)
|
||||
|
||||
yes, err := p.yesNo("Start one now?")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(yes).To(BeTrue())
|
||||
|
||||
chosen, err := p.choose([]string{"alpha", "zeta"})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(chosen).To(Equal("zeta"))
|
||||
})
|
||||
|
||||
// Whatever the chooser returns is persisted and used for every later run,
|
||||
// so an answer that is not one of the offered models must never come back
|
||||
// as one.
|
||||
Describe("the model prompt", func() {
|
||||
offered := []string{"alpha", "zeta"}
|
||||
|
||||
DescribeTable("refuses an answer that is not one of the numbers shown",
|
||||
func(answer string) {
|
||||
chosen, err := newPrompter(strings.NewReader(answer), &bytes.Buffer{}).choose(offered)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(chosen).To(BeEmpty())
|
||||
},
|
||||
Entry("nothing at all", ""),
|
||||
Entry("a blank line", "\n"),
|
||||
Entry("only spaces", " \n"),
|
||||
Entry("zero", "0\n"),
|
||||
Entry("past the end", "3\n"),
|
||||
Entry("negative", "-1\n"),
|
||||
Entry("a model name", "zeta\n"),
|
||||
Entry("a number with a suffix", "1x\n"),
|
||||
)
|
||||
|
||||
It("says how to answer when the answer was not a number", func() {
|
||||
_, err := newPrompter(strings.NewReader("banana\n"), &bytes.Buffer{}).choose(offered)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("between 1 and 2"))
|
||||
Expect(err.Error()).To(ContainSubstring("--model"))
|
||||
})
|
||||
|
||||
It("returns the model shown against the number", func() {
|
||||
chosen, err := newPrompter(strings.NewReader("1\n"), &bytes.Buffer{}).choose(offered)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(chosen).To(Equal("alpha"))
|
||||
})
|
||||
|
||||
It("refuses to ask when there is nothing to offer", func() {
|
||||
chosen, err := newPrompter(strings.NewReader("1\n"), &bytes.Buffer{}).choose(nil)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(chosen).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
|
||||
// A server started for this session is stopped by a deferred call, which a
|
||||
// signal skips: the process dies where it stands and leaves 'local-ai run'
|
||||
// reparented to init.
|
||||
Describe("shutdown signals", func() {
|
||||
It("ends the session when the terminal goes away", func() {
|
||||
ctx, stop := shutdownContext(context.Background())
|
||||
defer stop()
|
||||
|
||||
self, err := os.FindProcess(os.Getpid())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(self.Signal(syscall.SIGHUP)).To(Succeed())
|
||||
|
||||
Eventually(ctx.Done()).WithTimeout(5 * time.Second).Should(BeClosed())
|
||||
Expect(ctx.Err()).To(MatchError(context.Canceled))
|
||||
})
|
||||
|
||||
// SIGINT and SIGTERM cannot be delivered here to prove the same thing:
|
||||
// Ginkgo registers for both to abort the suite, and a signal goes to
|
||||
// every registered listener.
|
||||
It("also listens for an interrupt and a terminate", func() {
|
||||
Expect(shutdownSignals).To(ContainElements(os.Signal(os.Interrupt), os.Signal(syscall.SIGTERM)))
|
||||
})
|
||||
})
|
||||
|
||||
// Cancelling the context does unwind nib's TUI since v0.5.1, but how long
|
||||
// that takes is nib's business, and the deferred Stop in Run is only reached
|
||||
// once the agent returns. A server this process started is ours to end, so
|
||||
// the guarantee is made here instead, where it does not depend on the agent
|
||||
// at all. Before v0.5.1 there was no guarantee to be had on the SIGHUP path:
|
||||
// bubbletea's own SIGINT and SIGTERM handler was the only thing that ever
|
||||
// quit the program, and registering for SIGHUP took away the default
|
||||
// disposition that used to end the process.
|
||||
Describe("runSession", func() {
|
||||
It("stops the session's server on cancellation, without waiting for the agent", func() {
|
||||
server, proc := stoppableServer()
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
err := runSession(ctx, server, func(ctx context.Context) error {
|
||||
cancel()
|
||||
Eventually(func() int32 { return proc.interrupts.Load() }).
|
||||
WithTimeout(5 * time.Second).
|
||||
Should(BeNumerically(">", 0), "the server has to be stopped while the agent is still running")
|
||||
return nil
|
||||
})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(proc.lastSignal.Load()).To(Equal(os.Interrupt))
|
||||
})
|
||||
|
||||
It("leaves the server alone for as long as the session lasts", func() {
|
||||
server, proc := stoppableServer()
|
||||
|
||||
Expect(runSession(context.Background(), server, func(context.Context) error {
|
||||
return nil
|
||||
})).To(Succeed())
|
||||
Expect(proc.interrupts.Load()).To(BeZero())
|
||||
Expect(proc.kills.Load()).To(BeZero())
|
||||
})
|
||||
|
||||
It("returns what the agent returned", func() {
|
||||
failed := errors.New("the agent gave up")
|
||||
server, _ := stoppableServer()
|
||||
|
||||
Expect(runSession(context.Background(), server, func(context.Context) error {
|
||||
return failed
|
||||
})).To(MatchError(failed))
|
||||
})
|
||||
|
||||
// Most sessions run against a server the user already had, and there is
|
||||
// nothing to stop then.
|
||||
It("copes with a session that started no server", func() {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
Expect(runSession(ctx, nil, func(context.Context) error {
|
||||
return nil
|
||||
})).To(Succeed())
|
||||
})
|
||||
})
|
||||
|
||||
// Which streams reach nib decides two user-visible behaviours at once, and
|
||||
// they pull in opposite directions, so both are pinned here rather than left
|
||||
// to whoever next edits the literal.
|
||||
//
|
||||
// nib refuses every mode but --cli when a stream it was handed is not a
|
||||
// terminal. That refusal is wanted for stdin, where it is what tells someone
|
||||
// piping a question to re-run with --cli. It is not wanted for the process
|
||||
// stdout, where it would refuse the Ctrl+Space widget that --init emits:
|
||||
// out=$(local-ai chat --height 50%) puts a pipe on stdout by construction,
|
||||
// and writing the chosen command into that pipe is the entire point.
|
||||
Describe("agentOptions", func() {
|
||||
// optionsWithStreams is a request that differs from the next only in
|
||||
// what it was told to read and write.
|
||||
optionsWithStreams := func(in io.Reader, out, errOut io.Writer) Options {
|
||||
return Options{
|
||||
BaseURL: "http://127.0.0.1:8080/v1",
|
||||
In: in,
|
||||
Out: out,
|
||||
ErrOut: errOut,
|
||||
}
|
||||
}
|
||||
|
||||
Describe("stdout", func() {
|
||||
// The regression this exists to catch: reinstating
|
||||
// 'Stdout: opts.Out' breaks Ctrl+Space and nothing else notices.
|
||||
It("hands nib nothing for the process stdout, so the capture widget is not refused", func() {
|
||||
o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr))
|
||||
Expect(o.Stdout).To(BeNil(), "injecting os.Stdout is what refuses out=$(local-ai chat)")
|
||||
})
|
||||
|
||||
It("keeps a stdout the caller chose, which the refusal still guards", func() {
|
||||
out := &bytes.Buffer{}
|
||||
o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, out, os.Stderr))
|
||||
Expect(o.Stdout).To(BeIdenticalTo(out))
|
||||
})
|
||||
|
||||
// Being an *os.File is not what makes a stream nib's own; being the
|
||||
// process stdout is. This is a file an in-process caller opened for
|
||||
// itself, not one a shell redirect handed over as stdout, which
|
||||
// still arrives as os.Stdout and is still nil-ed. It was never going
|
||||
// to receive the interface, so it stays injected and stays refused.
|
||||
It("keeps a file that is not the process stdout", func() {
|
||||
f, err := os.CreateTemp(GinkgoT().TempDir(), "captured")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
DeferCleanup(f.Close)
|
||||
|
||||
o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, f, os.Stderr))
|
||||
Expect(o.Stdout).To(BeIdenticalTo(f))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("stdin", func() {
|
||||
// The opposite regression: nilling stdin the way stdout is nilled
|
||||
// would silently drop the refusal that names --cli.
|
||||
It("hands the process stdin over, so a piped session is still refused", func() {
|
||||
o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr))
|
||||
Expect(o.Stdin).To(BeIdenticalTo(os.Stdin))
|
||||
})
|
||||
|
||||
It("hands over a stdin the caller chose", func() {
|
||||
in := strings.NewReader("a question")
|
||||
o := agentOptions(dir, "a-model", optionsWithStreams(in, os.Stdout, os.Stderr))
|
||||
Expect(o.Stdin).To(BeIdenticalTo(in))
|
||||
})
|
||||
})
|
||||
|
||||
// nib gates stdin and stdout and nothing else, so there is no reason to
|
||||
// hide the error stream from it.
|
||||
It("hands the error stream over whatever it is", func() {
|
||||
errOut := &bytes.Buffer{}
|
||||
o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, errOut))
|
||||
Expect(o.Stderr).To(BeIdenticalTo(errOut))
|
||||
|
||||
o = agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr))
|
||||
Expect(o.Stderr).To(BeIdenticalTo(os.Stderr))
|
||||
})
|
||||
|
||||
It("names the command a user would type, not the binary nib ships as", func() {
|
||||
o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr))
|
||||
Expect(o.ProgramName).To(Equal("local-ai chat"),
|
||||
"the --init widget invokes this name, so a user has to be able to run it")
|
||||
})
|
||||
|
||||
It("carries the resolved session through to nib", func() {
|
||||
opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
|
||||
opts.Args = []string{"--cli"}
|
||||
opts.APIKey = "a-key"
|
||||
opts.TraceDir = "/traces"
|
||||
|
||||
o := agentOptions(dir, "the-model", opts)
|
||||
Expect(o.Args).To(Equal([]string{"--cli"}))
|
||||
Expect(o.BaseDir).To(Equal(dir))
|
||||
Expect(o.Overrides.Model).To(Equal("the-model"))
|
||||
Expect(o.Overrides.APIKey).To(Equal("a-key"))
|
||||
Expect(o.Overrides.BaseURL).To(Equal("http://127.0.0.1:8080/v1"))
|
||||
Expect(o.Overrides.TraceDir).To(Equal("/traces"))
|
||||
// The model and the server are settled before nib starts, and the
|
||||
// bare MODEL and API_KEY variables belong to some other tool.
|
||||
Expect(o.SkipSetup).To(BeTrue())
|
||||
Expect(o.SkipBareEnv).To(BeTrue())
|
||||
})
|
||||
|
||||
// Defaults sit beneath the config file. Anything routed through them is
|
||||
// accepted from the command line and then thrown away the moment the
|
||||
// file carries the same key, which is the normal state rather than an
|
||||
// edge case. Nothing this command resolves belongs there, so the channel
|
||||
// stays empty and this says so: it is what fails if the block is moved
|
||||
// back a rung.
|
||||
It("seeds nothing, because a seed is not a flag", func() {
|
||||
opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
|
||||
opts.APIKey = "a-key"
|
||||
opts.TraceDir = "/traces"
|
||||
opts.Yolo = true
|
||||
|
||||
Expect(agentOptions(dir, "the-model", opts).Defaults).To(Equal(nibtypes.Config{}),
|
||||
"Defaults lose to the config file, so a value placed there is a flag that does nothing")
|
||||
})
|
||||
|
||||
It("asks for automatic approval only when --yolo was given", func() {
|
||||
opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
|
||||
Expect(agentOptions(dir, "a-model", opts).Overrides.ApprovalMode).To(BeEmpty())
|
||||
|
||||
opts.Yolo = true
|
||||
Expect(agentOptions(dir, "a-model", opts).Overrides.ApprovalMode).To(Equal("auto"))
|
||||
})
|
||||
|
||||
// The specs above pin what is handed over. These pin what nib does with
|
||||
// it, which is the part that was wrong: every value below reached
|
||||
// app.Options intact and was then discarded by the config load, so a
|
||||
// spec that stops at the struct cannot see the bug. Resolving the config
|
||||
// the way app.Run resolves it can.
|
||||
Describe("the config nib actually resolves", func() {
|
||||
// writeConfig puts a config file where nib will read it, with values
|
||||
// that disagree with every flag under test.
|
||||
writeConfig := func(body string) {
|
||||
Expect(os.WriteFile(ConfigPath(dir), []byte(body), 0o600)).To(Succeed())
|
||||
}
|
||||
|
||||
// resolve loads the config exactly as app.Run does, so the precedence
|
||||
// under test is nib's own rather than a restatement of it here.
|
||||
resolve := func(o app.Options) nibtypes.Config {
|
||||
return nibconfig.LoadWith(nibconfig.LoadOptions{
|
||||
BaseDir: o.BaseDir,
|
||||
Defaults: o.Defaults,
|
||||
Overrides: o.Overrides,
|
||||
SkipBareEnv: o.SkipBareEnv,
|
||||
})
|
||||
}
|
||||
|
||||
It("sends the requests to the endpoint the flag named, not the one on disk", func() {
|
||||
writeConfig("base_url: http://127.0.0.1:9999/v1\n")
|
||||
|
||||
opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
|
||||
opts.BaseURL = "http://127.0.0.1:8080/v1"
|
||||
|
||||
cfg := resolve(agentOptions(dir, "a-model", opts))
|
||||
Expect(cfg.BaseURL).To(Equal("http://127.0.0.1:8080/v1"),
|
||||
"--endpoint probed 8080; every turn has to go there too")
|
||||
})
|
||||
|
||||
It("uses the model the flag named, not the one the picker recorded", func() {
|
||||
writeConfig("model: recorded-model\n")
|
||||
|
||||
cfg := resolve(agentOptions(dir, "flag-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)))
|
||||
Expect(cfg.Model).To(Equal("flag-model"))
|
||||
})
|
||||
|
||||
It("uses the key the flag named, not the one nib saved", func() {
|
||||
writeConfig("api_key: saved-key\n")
|
||||
|
||||
opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
|
||||
opts.APIKey = "flag-key"
|
||||
|
||||
cfg := resolve(agentOptions(dir, "a-model", opts))
|
||||
Expect(cfg.APIKey).To(Equal("flag-key"))
|
||||
})
|
||||
|
||||
It("turns approval off for --yolo even when the file demands it", func() {
|
||||
writeConfig("approval_mode: prompt\n")
|
||||
|
||||
opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
|
||||
opts.Yolo = true
|
||||
|
||||
cfg := resolve(agentOptions(dir, "a-model", opts))
|
||||
Expect(cfg.ApprovalMode).To(Equal("auto"))
|
||||
})
|
||||
|
||||
// The other half of the same rule, and the reason an unset flag is
|
||||
// not a demand for the empty string: an override only ever raises a
|
||||
// field, so what the user configured survives a run that said
|
||||
// nothing about it.
|
||||
It("leaves what the file configured alone when no flag was given", func() {
|
||||
writeConfig("api_key: saved-key\napproval_mode: prompt\n")
|
||||
|
||||
cfg := resolve(agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)))
|
||||
Expect(cfg.APIKey).To(Equal("saved-key"))
|
||||
Expect(cfg.ApprovalMode).To(Equal("prompt"))
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
// nib reports its own failures on the error stream and returns nothing but
|
||||
// a status, so anything that reaches here as one has already been explained
|
||||
// once. The refusal to open a full-screen session on a stdin that cannot be
|
||||
// read is the one users meet: 'echo q | local-ai chat' names --cli, and a
|
||||
// second message on top would bury the fix.
|
||||
Describe("ExitStatus", func() {
|
||||
It("recognises a status the agent already explained", func() {
|
||||
code, reported := ExitStatus(app.ExitError{Code: 2})
|
||||
Expect(reported).To(BeTrue())
|
||||
Expect(code).To(Equal(2))
|
||||
})
|
||||
|
||||
It("finds one that has been wrapped", func() {
|
||||
code, reported := ExitStatus(fmt.Errorf("running the agent: %w", app.ExitError{Code: 1}))
|
||||
Expect(reported).To(BeTrue())
|
||||
Expect(code).To(Equal(1))
|
||||
})
|
||||
|
||||
It("leaves an ordinary failure to be reported", func() {
|
||||
_, reported := ExitStatus(errors.New("no LocalAI server at http://127.0.0.1:8080"))
|
||||
Expect(reported).To(BeFalse())
|
||||
})
|
||||
|
||||
It("says nothing about a run that succeeded", func() {
|
||||
_, reported := ExitStatus(nil)
|
||||
Expect(reported).To(BeFalse())
|
||||
})
|
||||
})
|
||||
|
||||
It("reports a state dir it cannot create", func() {
|
||||
blocked := filepath.Join(dir, "a-file")
|
||||
Expect(os.WriteFile(blocked, []byte("not a dir"), 0o600)).To(Succeed())
|
||||
|
||||
opts := optionsFor(nil)
|
||||
opts.StateDir = filepath.Join(blocked, "chat")
|
||||
_, err := prepare(context.Background(), opts, false)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(ContainSubstring("agent state dir"))
|
||||
})
|
||||
})
|
||||
276
core/cli/chat/server.go
Normal file
276
core/cli/chat/server.go
Normal file
@@ -0,0 +1,276 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/pkg/httpclient"
|
||||
)
|
||||
|
||||
// ErrDeclined means no server was started, either because the session is not
|
||||
// interactive or because the user said no.
|
||||
var ErrDeclined = errors.New("no server started")
|
||||
|
||||
// errServerExited means the process we spawned died before it ever reported
|
||||
// ready, so there is no point in polling out the rest of the budget.
|
||||
var errServerExited = errors.New("the LocalAI server exited before it became ready")
|
||||
|
||||
const (
|
||||
// defaultReadyTimeout bounds the wait for a freshly spawned server. A cold
|
||||
// start probes hardware and may pull a backend, so the budget is generous.
|
||||
defaultReadyTimeout = 2 * time.Minute
|
||||
// readyPollInterval is how long to wait between readiness polls.
|
||||
readyPollInterval = 500 * time.Millisecond
|
||||
// readyProbeTimeout bounds a single readiness request, so one connection
|
||||
// that hangs cannot swallow the whole budget.
|
||||
readyProbeTimeout = 5 * time.Second
|
||||
// shutdownGrace is how long a server we started gets to unload models and
|
||||
// stop its backends after SIGINT before it is killed outright.
|
||||
shutdownGrace = 10 * time.Second
|
||||
// childOutputDrainDelay bounds how long cmd.Wait keeps copying the child's
|
||||
// output after the child itself has exited.
|
||||
//
|
||||
// This is not a theoretical guard for LocalAI. 'local-ai run' spawns backend
|
||||
// subprocesses, and they inherit the write end of the pipe exec created for
|
||||
// the child's stderr. A backend that outlives its parent holds that pipe
|
||||
// open, so an unbounded cmd.Wait would block on the copy goroutine long
|
||||
// after the server itself is gone: exited would never close, Stop would burn
|
||||
// its whole grace period even on a clean shutdown, and the waiter goroutine
|
||||
// would leak.
|
||||
//
|
||||
// The value is long enough that a legitimate final burst of logs is never
|
||||
// truncated even on a loaded machine, where the copy itself takes
|
||||
// microseconds. It must stay strictly below shutdownGrace: at or above it,
|
||||
// every wedged-pipe shutdown would exhaust the grace period and then SIGKILL
|
||||
// a process that had already exited cleanly.
|
||||
childOutputDrainDelay = 5 * time.Second
|
||||
)
|
||||
|
||||
// Confirmer asks a yes/no question. Nil means the session is not interactive.
|
||||
type Confirmer func(question string) (bool, error)
|
||||
|
||||
// StartOptions configures OfferToStart.
|
||||
type StartOptions struct {
|
||||
// Endpoint is the address the user expected a server on, used in the
|
||||
// question and polled for readiness. This is the endpoint root, not the
|
||||
// /v1 API base URL: readiness is served at the root.
|
||||
Endpoint string
|
||||
// Confirm asks whether to start a server. Nil means never start.
|
||||
Confirm Confirmer
|
||||
// Stderr receives the child's output.
|
||||
Stderr io.Writer
|
||||
// Executable overrides the binary to run. Empty means os.Executable().
|
||||
Executable string
|
||||
// ReadyTimeout bounds the wait for readiness. Zero means defaultReadyTimeout.
|
||||
ReadyTimeout time.Duration
|
||||
}
|
||||
|
||||
// StartedServer is a server this process started and is responsible for.
|
||||
type StartedServer struct {
|
||||
// exited is closed once the child has been reaped. One background waiter
|
||||
// owns cmd.Wait: it may only be called once, and it is what closes the
|
||||
// pipes exec created for Stdout/Stderr and joins the goroutines copying
|
||||
// them, so calling os.Process.Wait directly instead would leak both.
|
||||
exited chan struct{}
|
||||
// waitErr is the child's exit status. It is written before exited is
|
||||
// closed and must only be read after that channel is observed closed.
|
||||
waitErr error
|
||||
|
||||
// proc is the child. It is an interface rather than *os.Process so that
|
||||
// Stop's contract, in particular that the child is asked to stop exactly
|
||||
// once however often Stop is called, can be pinned without a live process
|
||||
// to signal. Nil means nothing was ever started.
|
||||
proc processControl
|
||||
|
||||
stopOnce sync.Once
|
||||
}
|
||||
|
||||
// processControl is the part of *os.Process that Stop needs.
|
||||
//
|
||||
// One interface rather than a pair of independent function fields: two fields
|
||||
// can be wired to each other's operation, or one left nil, and no test can tell,
|
||||
// because a fake satisfies any combination. There is nothing to swap or forget
|
||||
// here, since the sole implementation is the real process and the method names
|
||||
// carry the meaning.
|
||||
type processControl interface {
|
||||
Signal(os.Signal) error
|
||||
Kill() error
|
||||
}
|
||||
|
||||
// *os.Process satisfies processControl unmodified, so production needs no
|
||||
// adapter and no nil branch: the wiring is a single assignment.
|
||||
var _ processControl = (*os.Process)(nil)
|
||||
|
||||
// newServerCommand builds the child process. Split out from OfferToStart so the
|
||||
// process' configuration can be asserted on without spawning anything.
|
||||
func newServerCommand(bin string, stderr io.Writer) *exec.Cmd {
|
||||
cmd := exec.Command(bin, "run")
|
||||
// Stdin is left nil, so the child gets /dev/null: it is a background
|
||||
// server, and sharing the terminal would have it stealing keystrokes from
|
||||
// the agent.
|
||||
cmd.Stdout = stderr // the child's logs are diagnostics, not chat output
|
||||
cmd.Stderr = stderr
|
||||
// Bound the wait for the child's output pipes; see childOutputDrainDelay.
|
||||
cmd.WaitDelay = childOutputDrainDelay
|
||||
return cmd
|
||||
}
|
||||
|
||||
// OfferToStart asks whether to start a LocalAI server and, if allowed, spawns
|
||||
// one and waits for it to report ready.
|
||||
//
|
||||
// A child process rather than an in-process boot: RunCMD.Run installs its own
|
||||
// signal handling and blocks until shutdown, so re-entering it from a chat
|
||||
// session would entangle two lifecycles in one process.
|
||||
func OfferToStart(ctx context.Context, opts StartOptions) (*StartedServer, error) {
|
||||
if opts.Confirm == nil {
|
||||
// Not interactive. Spawning a server nobody asked for is the one thing
|
||||
// this function must never do: in CI, in a pipeline, or under a
|
||||
// supervisor there is no one to see it or shut it down.
|
||||
return nil, ErrDeclined
|
||||
}
|
||||
ok, err := opts.Confirm(fmt.Sprintf("No LocalAI server at %s. Start one now?", opts.Endpoint))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("asking whether to start a server: %w", err)
|
||||
}
|
||||
if !ok {
|
||||
return nil, ErrDeclined
|
||||
}
|
||||
|
||||
bin := opts.Executable
|
||||
if bin == "" {
|
||||
if bin, err = os.Executable(); err != nil {
|
||||
return nil, fmt.Errorf("locating the local-ai binary: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
cmd := newServerCommand(bin, opts.Stderr)
|
||||
if err := cmd.Start(); err != nil {
|
||||
return nil, fmt.Errorf("starting a LocalAI server with %s: %w", bin, err)
|
||||
}
|
||||
|
||||
s := &StartedServer{exited: make(chan struct{}), proc: cmd.Process}
|
||||
go func() {
|
||||
s.waitErr = cmd.Wait()
|
||||
close(s.exited)
|
||||
}()
|
||||
|
||||
timeout := opts.ReadyTimeout
|
||||
if timeout <= 0 {
|
||||
timeout = defaultReadyTimeout
|
||||
}
|
||||
if err := waitReady(ctx, opts.Endpoint, timeout, s.exited); err != nil {
|
||||
if errors.Is(err, errServerExited) {
|
||||
// Safe to read: errServerExited is only returned once exited has
|
||||
// been observed closed, which happens after waitErr is written.
|
||||
err = describeExit(err, s.waitErr)
|
||||
}
|
||||
s.Stop()
|
||||
return nil, fmt.Errorf("%w. Run 'local-ai run' in another terminal to see why it did not come up", err)
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// describeExit adds what is known about how the child died to exitErr, without
|
||||
// putting os/exec's plumbing in front of the user.
|
||||
//
|
||||
// waitErr is exec.ErrWaitDelay when the child exited cleanly but something it
|
||||
// spawned still held its output pipe open past childOutputDrainDelay. The
|
||||
// sentinel's own text names the WaitDelay field, which is meaningless to a
|
||||
// user, so it is translated. Nothing is swallowed: os/exec only substitutes
|
||||
// ErrWaitDelay when the process itself exited without an error of its own (see
|
||||
// Cmd.Wait, "Report an error from the copying goroutines only if the program
|
||||
// otherwise exited normally"), so it can never stand in for an *ExitError.
|
||||
func describeExit(exitErr, waitErr error) error {
|
||||
switch {
|
||||
case waitErr == nil:
|
||||
return exitErr
|
||||
case errors.Is(waitErr, exec.ErrWaitDelay):
|
||||
return fmt.Errorf("%w, and left a subprocess of its own still running", exitErr)
|
||||
default:
|
||||
return fmt.Errorf("%w: %w", exitErr, waitErr)
|
||||
}
|
||||
}
|
||||
|
||||
// Stop terminates the server this process started, giving it a chance to shut
|
||||
// down cleanly first. It is safe to call on a nil or never-started server, and
|
||||
// safe to call more than once.
|
||||
func (s *StartedServer) Stop() {
|
||||
if s == nil || s.proc == nil {
|
||||
return
|
||||
}
|
||||
s.stopOnce.Do(func() {
|
||||
// SIGINT rather than SIGKILL: local-ai run installs its own handler and
|
||||
// needs it to unload models and stop backend subprocesses. Killing it
|
||||
// outright would strand those children.
|
||||
_ = s.proc.Signal(os.Interrupt)
|
||||
|
||||
select {
|
||||
case <-s.exited:
|
||||
case <-time.After(shutdownGrace):
|
||||
// It ignored the interrupt or wedged on the way down. The user is
|
||||
// waiting on their shell prompt, so stop being polite.
|
||||
_ = s.proc.Kill()
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// waitReady polls the endpoint's /readyz until the server reports ready, the
|
||||
// budget expires, the caller gives up, or exited signals that the process we
|
||||
// are waiting on is gone. A nil exited channel means there is no process to
|
||||
// watch.
|
||||
//
|
||||
// Readiness lives on the endpoint ROOT, not under the /v1 API base URL, and it
|
||||
// answers 503 for as long as startup is still in progress.
|
||||
func waitReady(ctx context.Context, endpoint string, timeout time.Duration, exited <-chan struct{}) error {
|
||||
url := strings.TrimSuffix(endpoint, "/") + "/readyz"
|
||||
|
||||
// A real deadline rather than context.WithCancel plus a timer: the latter
|
||||
// expires as context.Canceled, which every classifier here reads as "the
|
||||
// caller gave up" rather than "the endpoint never answered".
|
||||
waitCtx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
client := httpclient.NewWithTimeout(readyProbeTimeout)
|
||||
ticker := time.NewTicker(readyPollInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-exited:
|
||||
return errServerExited
|
||||
case <-waitCtx.Done():
|
||||
// Distinguish our budget from the caller's: only ours is advice
|
||||
// about the server.
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
return fmt.Errorf("the LocalAI server did not become ready within %s", timeout)
|
||||
case <-ticker.C:
|
||||
}
|
||||
|
||||
req, err := http.NewRequestWithContext(waitCtx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("building the readiness request for %s: %w", url, err)
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
continue // nothing listening yet
|
||||
}
|
||||
// Drain before closing so the next poll can reuse the connection
|
||||
// instead of opening a socket every 500ms for two minutes.
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
_ = resp.Body.Close()
|
||||
if resp.StatusCode == http.StatusOK {
|
||||
return nil
|
||||
}
|
||||
// Anything else means startup is still in progress; keep polling.
|
||||
}
|
||||
}
|
||||
375
core/cli/chat/server_test.go
Normal file
375
core/cli/chat/server_test.go
Normal file
@@ -0,0 +1,375 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// unusedPort is a loopback address nothing listens on, used wherever a spec
|
||||
// needs a readiness poll to keep failing. Port 1 is privileged, so no test
|
||||
// process could have bound it.
|
||||
const unusedPort = "http://127.0.0.1:1"
|
||||
|
||||
var _ = Describe("OfferToStart", func() {
|
||||
It("never spawns anything when there is no confirmer", func() {
|
||||
started, err := OfferToStart(context.Background(), StartOptions{
|
||||
Endpoint: "http://127.0.0.1:59999",
|
||||
Confirm: nil,
|
||||
Stderr: io.Discard,
|
||||
Executable: "/nonexistent/binary-that-must-not-run",
|
||||
})
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(errors.Is(err, ErrDeclined)).To(BeTrue(), "want ErrDeclined, got %v", err)
|
||||
Expect(started).To(BeNil())
|
||||
})
|
||||
|
||||
It("does not spawn when the user declines", func() {
|
||||
asked := false
|
||||
started, err := OfferToStart(context.Background(), StartOptions{
|
||||
Endpoint: "http://127.0.0.1:59999",
|
||||
Confirm: func(string) (bool, error) {
|
||||
asked = true
|
||||
return false, nil
|
||||
},
|
||||
Stderr: io.Discard,
|
||||
Executable: "/nonexistent/binary-that-must-not-run",
|
||||
})
|
||||
Expect(asked).To(BeTrue(), "the user should have been asked")
|
||||
Expect(errors.Is(err, ErrDeclined)).To(BeTrue())
|
||||
Expect(started).To(BeNil())
|
||||
})
|
||||
|
||||
It("names the endpoint in the question", func() {
|
||||
var question string
|
||||
_, _ = OfferToStart(context.Background(), StartOptions{
|
||||
Endpoint: "http://example.invalid:9090",
|
||||
Confirm: func(q string) (bool, error) {
|
||||
question = q
|
||||
return false, nil
|
||||
},
|
||||
Stderr: io.Discard,
|
||||
Executable: "/nonexistent/binary-that-must-not-run",
|
||||
})
|
||||
Expect(question).To(ContainSubstring("http://example.invalid:9090"))
|
||||
})
|
||||
|
||||
It("propagates a confirmer error", func() {
|
||||
boom := errors.New("boom")
|
||||
_, err := OfferToStart(context.Background(), StartOptions{
|
||||
Endpoint: "http://127.0.0.1:59999",
|
||||
Confirm: func(string) (bool, error) { return false, boom },
|
||||
Stderr: io.Discard,
|
||||
Executable: "/nonexistent/binary-that-must-not-run",
|
||||
})
|
||||
Expect(errors.Is(err, boom)).To(BeTrue())
|
||||
})
|
||||
|
||||
It("reports which binary it failed to launch", func() {
|
||||
started, err := OfferToStart(context.Background(), StartOptions{
|
||||
Endpoint: "http://127.0.0.1:59999",
|
||||
Confirm: func(string) (bool, error) { return true, nil },
|
||||
Stderr: io.Discard,
|
||||
Executable: "/nonexistent/binary-that-must-not-run",
|
||||
})
|
||||
Expect(started).To(BeNil())
|
||||
Expect(err).To(MatchError(ContainSubstring("starting a LocalAI server")))
|
||||
Expect(err).To(MatchError(ContainSubstring("/nonexistent/binary-that-must-not-run")))
|
||||
})
|
||||
|
||||
It("stops waiting as soon as the process it started exits", func() {
|
||||
// A harmless no-op binary rather than a real server: this exercises the
|
||||
// early-exit path without starting LocalAI, binding a port, or running
|
||||
// 'local-ai run'. Without early-exit detection the call would sit here
|
||||
// polling until ReadyTimeout.
|
||||
bin, lookErr := exec.LookPath("true")
|
||||
if lookErr != nil {
|
||||
Skip("no 'true' binary on PATH to stand in for a server that dies at once")
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
started, err := OfferToStart(context.Background(), StartOptions{
|
||||
Endpoint: unusedPort,
|
||||
Confirm: func(string) (bool, error) { return true, nil },
|
||||
Stderr: io.Discard,
|
||||
Executable: bin,
|
||||
ReadyTimeout: 30 * time.Second,
|
||||
})
|
||||
Expect(started).To(BeNil())
|
||||
Expect(err).To(MatchError(ContainSubstring("exited before it became ready")))
|
||||
Expect(time.Since(start)).To(BeNumerically("<", 10*time.Second),
|
||||
"the wait should end with the process, not with the readiness budget")
|
||||
})
|
||||
|
||||
It("gives up on a child whose grandchildren still hold its output pipe", func() {
|
||||
// The real LocalAI shape: 'local-ai run' exits but a backend
|
||||
// subprocess it spawned inherited the stderr pipe and keeps it open.
|
||||
// Without cmd.WaitDelay, cmd.Wait blocks on the copy goroutine, exited
|
||||
// never closes, and the readiness wait runs out the full budget instead
|
||||
// of reporting that the server died.
|
||||
sh, lookErr := exec.LookPath("sh")
|
||||
if lookErr != nil {
|
||||
Skip("no 'sh' binary on PATH to stand in for a server with a lingering child")
|
||||
}
|
||||
|
||||
dir := GinkgoT().TempDir()
|
||||
pidFile := filepath.Join(dir, "grandchild.pid")
|
||||
script := filepath.Join(dir, "server-with-lingering-child")
|
||||
// #nosec G306 -- this has to be executable to stand in for a binary.
|
||||
Expect(os.WriteFile(script,
|
||||
[]byte("#!"+sh+"\nsleep 30 &\necho $! > "+pidFile+"\nexit 0\n"),
|
||||
0o700)).To(Succeed())
|
||||
|
||||
// Reap the grandchild whatever happens: it outlives its own parent by
|
||||
// design, so nothing else will clean it up.
|
||||
DeferCleanup(func() {
|
||||
raw, err := os.ReadFile(pidFile)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
pid, err := strconv.Atoi(strings.TrimSpace(string(raw)))
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
proc, err := os.FindProcess(pid)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
_ = proc.Kill()
|
||||
_, _ = proc.Wait()
|
||||
})
|
||||
|
||||
start := time.Now()
|
||||
started, err := OfferToStart(context.Background(), StartOptions{
|
||||
Endpoint: unusedPort,
|
||||
Confirm: func(string) (bool, error) { return true, nil },
|
||||
Stderr: io.Discard,
|
||||
Executable: script,
|
||||
ReadyTimeout: 25 * time.Second,
|
||||
})
|
||||
elapsed := time.Since(start)
|
||||
|
||||
Expect(started).To(BeNil())
|
||||
Expect(err).To(MatchError(ContainSubstring("exited before it became ready")),
|
||||
"an unbounded cmd.Wait would report a readiness timeout instead")
|
||||
Expect(elapsed).To(BeNumerically("<", 20*time.Second),
|
||||
"the wait must be bounded by the output drain, not by the readiness budget")
|
||||
|
||||
// This is the case where cmd.Wait returns exec.ErrWaitDelay, whose own
|
||||
// text names a struct field of os/exec. Users get told what happened
|
||||
// instead.
|
||||
Expect(err).NotTo(MatchError(ContainSubstring("WaitDelay")),
|
||||
"os/exec plumbing must not reach the user")
|
||||
Expect(err).NotTo(MatchError(ContainSubstring("exec:")))
|
||||
Expect(err).To(MatchError(ContainSubstring("left a subprocess of its own still running")))
|
||||
})
|
||||
|
||||
It("reports the exit status of a server that failed outright", func() {
|
||||
// The counterpart to the case above: translating ErrWaitDelay must not
|
||||
// cost a real exit status, which is the one diagnostic worth having.
|
||||
bin, lookErr := exec.LookPath("false")
|
||||
if lookErr != nil {
|
||||
Skip("no 'false' binary on PATH to stand in for a server that fails")
|
||||
}
|
||||
|
||||
_, err := OfferToStart(context.Background(), StartOptions{
|
||||
Endpoint: unusedPort,
|
||||
Confirm: func(string) (bool, error) { return true, nil },
|
||||
Stderr: io.Discard,
|
||||
Executable: bin,
|
||||
ReadyTimeout: 30 * time.Second,
|
||||
})
|
||||
Expect(err).To(MatchError(ContainSubstring("exited before it became ready")))
|
||||
Expect(err).To(MatchError(ContainSubstring("exit status 1")))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("StartedServer.Stop", func() {
|
||||
It("is a no-op on a server that was never started", func() {
|
||||
var nilServer *StartedServer
|
||||
Expect(nilServer.Stop).NotTo(Panic())
|
||||
Expect((&StartedServer{}).Stop).NotTo(Panic())
|
||||
})
|
||||
|
||||
It("interrupts the child exactly once however often it is called", func() {
|
||||
s, proc := stoppableServer()
|
||||
|
||||
s.Stop()
|
||||
s.Stop()
|
||||
s.Stop()
|
||||
|
||||
Expect(proc.interrupts.Load()).To(Equal(int32(1)),
|
||||
"a second Stop must not signal the child again")
|
||||
Expect(proc.kills.Load()).To(BeZero(), "a child that already exited must not be killed")
|
||||
})
|
||||
|
||||
It("interrupts the child exactly once when called concurrently", func() {
|
||||
// The realistic double-Stop: a deferred Stop on the way out racing the
|
||||
// signal handler that also owns shutting the server down.
|
||||
const callers = 8
|
||||
|
||||
s, proc := stoppableServer()
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(callers)
|
||||
for range callers {
|
||||
go func() {
|
||||
defer GinkgoRecover()
|
||||
defer wg.Done()
|
||||
s.Stop()
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
Expect(proc.interrupts.Load()).To(Equal(int32(1)))
|
||||
Expect(proc.kills.Load()).To(BeZero())
|
||||
})
|
||||
|
||||
It("asks the child to interrupt rather than killing it outright", func() {
|
||||
// The escalation order is the whole point of the grace period: SIGKILL
|
||||
// first would strand the backend subprocesses local-ai run owns.
|
||||
s, proc := stoppableServer()
|
||||
|
||||
s.Stop()
|
||||
|
||||
Expect(proc.lastSignal.Load()).To(Equal(os.Interrupt))
|
||||
Expect(proc.kills.Load()).To(BeZero())
|
||||
})
|
||||
})
|
||||
|
||||
// countingProcess stands in for the *os.Process that Stop drives, recording
|
||||
// what it was asked to do.
|
||||
type countingProcess struct {
|
||||
interrupts atomic.Int32
|
||||
kills atomic.Int32
|
||||
lastSignal atomic.Value
|
||||
}
|
||||
|
||||
func (p *countingProcess) Signal(sig os.Signal) error {
|
||||
p.interrupts.Add(1)
|
||||
p.lastSignal.Store(sig)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *countingProcess) Kill() error {
|
||||
p.kills.Add(1)
|
||||
return nil
|
||||
}
|
||||
|
||||
// stoppableServer builds a StartedServer whose child has already exited, driven
|
||||
// by a countingProcess rather than a real one. Nothing is spawned.
|
||||
func stoppableServer() (*StartedServer, *countingProcess) {
|
||||
proc := &countingProcess{}
|
||||
exited := make(chan struct{})
|
||||
close(exited)
|
||||
return &StartedServer{exited: exited, proc: proc}, proc
|
||||
}
|
||||
|
||||
var _ = Describe("newServerCommand", func() {
|
||||
It("bounds how long it will wait for the child's output pipes", func() {
|
||||
cmd := newServerCommand("/nonexistent/binary-that-must-not-run", io.Discard)
|
||||
|
||||
// An unbounded wait is the failure mode: backend subprocesses inherit
|
||||
// the child's stderr pipe and can hold it open long after the server
|
||||
// itself is gone.
|
||||
Expect(cmd.WaitDelay).To(BeNumerically(">", 0), "cmd.Wait must not be unbounded")
|
||||
Expect(cmd.WaitDelay).To(BeNumerically("<", shutdownGrace),
|
||||
"a drain longer than the shutdown grace would kill a cleanly exited server")
|
||||
})
|
||||
|
||||
It("runs the server subcommand without giving it the terminal", func() {
|
||||
cmd := newServerCommand("/nonexistent/binary-that-must-not-run", io.Discard)
|
||||
|
||||
Expect(cmd.Args).To(Equal([]string{"/nonexistent/binary-that-must-not-run", "run"}))
|
||||
Expect(cmd.Stdin).To(BeNil(), "the child must not compete with the agent for stdin")
|
||||
Expect(cmd.Stdout).NotTo(BeNil())
|
||||
Expect(cmd.Stderr).NotTo(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("waitReady", func() {
|
||||
It("polls /readyz on the endpoint root and returns only once it answers 200", func() {
|
||||
// readyOnPoll is deliberately above 1. A handler that answers 200 to the
|
||||
// first poll cannot tell a correct implementation apart from one that
|
||||
// treats 503 as ready, because both return after a single request; the
|
||||
// poll count is what makes 503-as-ready observable.
|
||||
const readyOnPoll = 3
|
||||
|
||||
var polls atomic.Int32
|
||||
var paths atomic.Value
|
||||
paths.Store("")
|
||||
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
paths.Store(r.URL.Path)
|
||||
if polls.Add(1) < readyOnPoll {
|
||||
// What LocalAI answers while startup is still in progress.
|
||||
w.WriteHeader(http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
Expect(waitReady(context.Background(), srv.URL, 20*time.Second, nil)).To(Succeed())
|
||||
Expect(paths.Load()).To(Equal("/readyz"), "readiness lives on the endpoint root, not under /v1")
|
||||
Expect(polls.Load()).To(BeNumerically(">=", readyOnPoll),
|
||||
"503 means startup is still in progress and must never be accepted as ready")
|
||||
})
|
||||
|
||||
It("tolerates a trailing slash on the endpoint", func() {
|
||||
var path atomic.Value
|
||||
path.Store("")
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
path.Store(r.URL.Path)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
Expect(waitReady(context.Background(), srv.URL+"/", 20*time.Second, nil)).To(Succeed())
|
||||
Expect(path.Load()).To(Equal("/readyz"))
|
||||
})
|
||||
|
||||
It("reports a timeout, not a cancellation, when the budget runs out", func() {
|
||||
err := waitReady(context.Background(), unusedPort, 1200*time.Millisecond, nil)
|
||||
Expect(err).To(HaveOccurred())
|
||||
// A budget built from context.WithCancel plus a timer would surface as
|
||||
// context.Canceled, which downstream code reads as "the caller gave up"
|
||||
// and would stop classifying a hung server as unreachable.
|
||||
Expect(errors.Is(err, context.Canceled)).To(BeFalse(), "got %v", err)
|
||||
Expect(err).To(MatchError(ContainSubstring("did not become ready")))
|
||||
})
|
||||
|
||||
It("returns the caller's cancellation when the caller gives up", func() {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
go func() {
|
||||
defer GinkgoRecover()
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
cancel()
|
||||
}()
|
||||
defer cancel()
|
||||
|
||||
err := waitReady(ctx, unusedPort, time.Minute, nil)
|
||||
Expect(errors.Is(err, context.Canceled)).To(BeTrue(), "got %v", err)
|
||||
})
|
||||
|
||||
It("gives up when the process it is waiting on has exited", func() {
|
||||
exited := make(chan struct{})
|
||||
close(exited)
|
||||
|
||||
err := waitReady(context.Background(), unusedPort, time.Minute, exited)
|
||||
Expect(err).To(MatchError(ContainSubstring("exited before it became ready")))
|
||||
})
|
||||
})
|
||||
@@ -1,112 +0,0 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"slices"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const (
|
||||
chatRoleUser = "user"
|
||||
chatRoleAssistant = "assistant"
|
||||
)
|
||||
|
||||
type chatMessage struct {
|
||||
Role string
|
||||
Content string
|
||||
}
|
||||
|
||||
type chatSession struct {
|
||||
client chatClient
|
||||
model string
|
||||
models []string
|
||||
messages []chatMessage
|
||||
}
|
||||
|
||||
func newChatSession(ctx context.Context, client chatClient, requestedModel string) (*chatSession, error) {
|
||||
models, err := client.ListModels(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list models: %w", err)
|
||||
}
|
||||
|
||||
model, err := resolveChatModel(requestedModel, models)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &chatSession{
|
||||
client: client,
|
||||
model: model,
|
||||
models: models,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (s *chatSession) CurrentModel() string {
|
||||
return s.model
|
||||
}
|
||||
|
||||
func (s *chatSession) Models() []string {
|
||||
models := make([]string, len(s.models))
|
||||
copy(models, s.models)
|
||||
return models
|
||||
}
|
||||
|
||||
func (s *chatSession) Clear() {
|
||||
s.messages = nil
|
||||
}
|
||||
|
||||
func (s *chatSession) SwitchModel(model string) error {
|
||||
if !slices.Contains(s.models, model) {
|
||||
return fmt.Errorf("model %q is not available. Use /models to see installed models", model)
|
||||
}
|
||||
s.model = model
|
||||
s.Clear()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *chatSession) Send(ctx context.Context, prompt string, out io.Writer) error {
|
||||
s.messages = append(s.messages, chatMessage{
|
||||
Role: chatRoleUser,
|
||||
Content: prompt,
|
||||
})
|
||||
|
||||
answer, err := s.client.StreamChat(ctx, s.model, s.messages, out)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
s.messages = append(s.messages, chatMessage{
|
||||
Role: chatRoleAssistant,
|
||||
Content: answer,
|
||||
})
|
||||
return nil
|
||||
}
|
||||
|
||||
func resolveChatModel(requested string, models []string) (string, error) {
|
||||
switch {
|
||||
case requested == "" && len(models) == 0:
|
||||
return "", errors.New(`no chat models are installed.
|
||||
|
||||
Install a model first, for example:
|
||||
local-ai models list
|
||||
local-ai models install <model>
|
||||
local-ai run
|
||||
|
||||
Then start a chat session:
|
||||
local-ai chat --model <model>`)
|
||||
case requested == "" && len(models) == 1:
|
||||
return models[0], nil
|
||||
case requested == "" && len(models) > 1:
|
||||
var b strings.Builder
|
||||
b.WriteString("multiple models are available; choose one with --model:\n")
|
||||
b.WriteString(formatChatModelList(models, ""))
|
||||
return "", errors.New(b.String())
|
||||
case !slices.Contains(models, requested):
|
||||
return "", fmt.Errorf("model %q is not available. Use `local-ai models list` and `local-ai models install <model>`, or pass an installed model with --model", requested)
|
||||
default:
|
||||
return requested, nil
|
||||
}
|
||||
}
|
||||
@@ -1,56 +0,0 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("Chat session", func() {
|
||||
It("keeps model switching and message history out of the terminal adapter", func() {
|
||||
client := &fakeChatClient{
|
||||
models: []string{"alpha", "beta"},
|
||||
answer: "pong",
|
||||
}
|
||||
|
||||
session, err := newChatSession(context.Background(), client, "alpha")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(session.CurrentModel()).To(Equal("alpha"))
|
||||
|
||||
Expect(session.SwitchModel("beta")).To(Succeed())
|
||||
Expect(session.CurrentModel()).To(Equal("beta"))
|
||||
Expect(session.Send(context.Background(), "ping", io.Discard)).To(Succeed())
|
||||
|
||||
Expect(client.requests).To(HaveLen(1))
|
||||
Expect(client.requests[0].model).To(Equal("beta"))
|
||||
Expect(client.requests[0].messages).To(HaveLen(1))
|
||||
Expect(client.requests[0].messages[0].Content).To(Equal("ping"))
|
||||
})
|
||||
})
|
||||
|
||||
type fakeChatClient struct {
|
||||
models []string
|
||||
answer string
|
||||
requests []fakeChatRequest
|
||||
}
|
||||
|
||||
type fakeChatRequest struct {
|
||||
model string
|
||||
messages []chatMessage
|
||||
}
|
||||
|
||||
func (c *fakeChatClient) ListModels(context.Context) ([]string, error) {
|
||||
return c.models, nil
|
||||
}
|
||||
|
||||
func (c *fakeChatClient) StreamChat(_ context.Context, model string, messages []chatMessage, out io.Writer) (string, error) {
|
||||
copied := make([]chatMessage, len(messages))
|
||||
copy(copied, messages)
|
||||
c.requests = append(c.requests, fakeChatRequest{model: model, messages: copied})
|
||||
if _, err := io.WriteString(out, c.answer); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return c.answer, nil
|
||||
}
|
||||
@@ -1,93 +0,0 @@
|
||||
package chat
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func runTerminalChat(ctx context.Context, session *chatSession, in io.Reader, out io.Writer) error {
|
||||
scanner := bufio.NewScanner(in)
|
||||
scanner.Buffer(make([]byte, 0, 64*1024), 4*1024*1024)
|
||||
|
||||
if err := writeChat(out, "LocalAI chat (%s)\n", session.CurrentModel()); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := writeChat(out, "Type /exit to quit, /clear to reset the conversation, /models to list models.\n"); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for {
|
||||
if err := writeChat(out, "\n> "); err != nil {
|
||||
return err
|
||||
}
|
||||
if !scanner.Scan() {
|
||||
break
|
||||
}
|
||||
|
||||
prompt := strings.TrimSpace(scanner.Text())
|
||||
switch prompt {
|
||||
case "":
|
||||
continue
|
||||
case "/bye", "/exit", "/quit":
|
||||
return writeChat(out, "bye\n")
|
||||
case "/clear":
|
||||
session.Clear()
|
||||
if err := writeChat(out, "conversation cleared\n"); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
case "/models":
|
||||
if err := printChatModels(out, session.Models(), session.CurrentModel()); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
if nextModel, ok := strings.CutPrefix(prompt, "/model "); ok {
|
||||
nextModel = strings.TrimSpace(nextModel)
|
||||
if nextModel == "" {
|
||||
if err := writeChat(out, "usage: /model <name>\n"); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err := session.SwitchModel(nextModel); err != nil {
|
||||
if writeErr := writeChat(out, "%s\n", err); writeErr != nil {
|
||||
return writeErr
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err := writeChat(out, "switched to %s; conversation cleared\n", session.CurrentModel()); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
if err := writeChat(out, "assistant: "); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := session.Send(ctx, prompt, out); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := writeChat(out, "\n"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return scanner.Err()
|
||||
}
|
||||
|
||||
func printChatModels(out io.Writer, models []string, current string) error {
|
||||
if len(models) == 0 {
|
||||
return writeChat(out, "no models installed\n")
|
||||
}
|
||||
return writeChat(out, "%s", formatChatModelList(models, current))
|
||||
}
|
||||
|
||||
func writeChat(out io.Writer, format string, args ...any) error {
|
||||
_, err := fmt.Fprintf(out, format, args...)
|
||||
return err
|
||||
}
|
||||
@@ -8,18 +8,72 @@ import (
|
||||
cliContext "github.com/mudler/LocalAI/core/cli/context"
|
||||
)
|
||||
|
||||
// ChatCMD runs the built-in terminal agent. Everything after the first
|
||||
// positional argument is forwarded to the agent verbatim, so its own
|
||||
// subcommands (plugin, skill, mcp) and their flags work unchanged. LocalAI's
|
||||
// own flags must therefore come first.
|
||||
type ChatCMD struct {
|
||||
Model string `short:"m" help:"Model name to use. Defaults to the only model returned by the server when exactly one is available"`
|
||||
Endpoint string `env:"LOCALAI_CHAT_ENDPOINT" default:"http://127.0.0.1:8080" help:"LocalAI server endpoint. The /v1 path is added automatically when omitted"`
|
||||
APIKey string `env:"LOCALAI_API_KEY,API_KEY" help:"API key to use when the LocalAI server requires authentication"`
|
||||
Model string `short:"m" help:"Model to use. Defaults to the only model the server offers, or asks when there are several"`
|
||||
Endpoint string `env:"LOCALAI_CHAT_ENDPOINT" default:"http://127.0.0.1:8080" help:"LocalAI server endpoint. The /v1 path is added automatically when omitted"`
|
||||
APIKey string `env:"LOCALAI_API_KEY,API_KEY" help:"API key to use when the LocalAI server requires authentication"`
|
||||
ConfigDir string `env:"LOCALAI_CHAT_CONFIG_DIR" help:"Directory holding the agent's config, plugins, and skills. Defaults to ~/.config/localai/chat" type:"path"`
|
||||
TraceDir string `env:"LOCALAI_CHAT_TRACE_DIR" help:"Write a session LLM trace (NDJSON) to this directory" type:"path"`
|
||||
|
||||
CLI bool `help:"Run in plain CLI mode instead of the full-screen interface"`
|
||||
TUI bool `help:"Force the full-screen interface"`
|
||||
Height string `help:"Run as an inline drop-down of this height, e.g. '40%'"`
|
||||
Tmux bool `help:"Run in a tmux split"`
|
||||
NoTmux bool `name:"no-tmux" help:"Never use a tmux split, even inside tmux"`
|
||||
Init string `help:"Print the shell integration script for Ctrl+Space (zsh, bash, or fish)"`
|
||||
Yolo bool `env:"LOCALAI_CHAT_YOLO" help:"Auto-approve every tool call without prompting"`
|
||||
|
||||
Args []string `arg:"" optional:"" passthrough:"" help:"Arguments forwarded to the agent, e.g. 'plugin install <url>', 'skill list', 'mcp add'"`
|
||||
}
|
||||
|
||||
func (c *ChatCMD) Run(ctx *cliContext.Context) error {
|
||||
return chatcli.Run(context.Background(), chatcli.Options{
|
||||
Model: c.Model,
|
||||
BaseURL: chatAPIBaseURL(c.Endpoint),
|
||||
APIKey: c.APIKey,
|
||||
In: os.Stdin,
|
||||
Out: os.Stdout,
|
||||
err := chatcli.Run(context.Background(), chatcli.Options{
|
||||
Args: c.agentArgs(),
|
||||
Endpoint: c.Endpoint,
|
||||
BaseURL: chatAPIBaseURL(c.Endpoint),
|
||||
APIKey: c.APIKey,
|
||||
Model: c.Model,
|
||||
StateDir: c.ConfigDir,
|
||||
TraceDir: c.TraceDir,
|
||||
Yolo: c.Yolo,
|
||||
In: os.Stdin,
|
||||
Out: os.Stdout,
|
||||
ErrOut: os.Stderr,
|
||||
})
|
||||
// The agent explains its own failures on stderr and hands back a code, so
|
||||
// carry the code out and leave the explanation to stand alone.
|
||||
if code, reported := chatcli.ExitStatus(err); reported {
|
||||
return ExitCodeError{Code: code}
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// agentArgs rebuilds the argument vector the agent expects: LocalAI's mode
|
||||
// flags are declared here for discoverability and shell completion, so they
|
||||
// have to be translated back into the agent's own flag names.
|
||||
func (c *ChatCMD) agentArgs() []string {
|
||||
var args []string
|
||||
if c.CLI {
|
||||
args = append(args, "--cli")
|
||||
}
|
||||
if c.TUI {
|
||||
args = append(args, "--tui")
|
||||
}
|
||||
if c.Height != "" {
|
||||
args = append(args, "--height", c.Height)
|
||||
}
|
||||
if c.Tmux {
|
||||
args = append(args, "--tmux")
|
||||
}
|
||||
if c.NoTmux {
|
||||
args = append(args, "--no-tmux")
|
||||
}
|
||||
if c.Init != "" {
|
||||
args = append(args, "--init", c.Init)
|
||||
}
|
||||
return append(args, c.Args...)
|
||||
}
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"github.com/alecthomas/kong"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
@@ -24,4 +28,70 @@ var _ = Describe("Chat command wiring", func() {
|
||||
Expect(chatAPIBaseURL("http://127.0.0.1:8080/localai")).To(Equal("http://127.0.0.1:8080/localai/v1"))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("argument parsing", func() {
|
||||
parse := func(args ...string) *ChatCMD {
|
||||
var cli struct {
|
||||
Chat ChatCMD `cmd:""`
|
||||
}
|
||||
parser, err := kong.New(&cli)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
_, err = parser.Parse(append([]string{"chat"}, args...))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
return &cli.Chat
|
||||
}
|
||||
|
||||
It("leaves Args empty for a bare invocation", func() {
|
||||
Expect(parse().Args).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("binds flags that precede the forwarded arguments", func() {
|
||||
c := parse("--endpoint", "http://host:9090", "--model", "m", "plugin", "list")
|
||||
Expect(c.Endpoint).To(Equal("http://host:9090"))
|
||||
Expect(c.Model).To(Equal("m"))
|
||||
Expect(c.Args).To(Equal([]string{"plugin", "list"}))
|
||||
})
|
||||
|
||||
It("forwards flags that follow the first positional to the agent", func() {
|
||||
c := parse("plugin", "install", "https://example.invalid/p", "--yes")
|
||||
Expect(c.Args).To(Equal([]string{"plugin", "install", "https://example.invalid/p", "--yes"}))
|
||||
})
|
||||
|
||||
It("parses its own mode flags", func() {
|
||||
c := parse("--cli")
|
||||
Expect(c.CLI).To(BeTrue())
|
||||
Expect(c.Args).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
|
||||
// The agent prints its own diagnosis and hands back a status. main exits
|
||||
// with that status and prints nothing more, so the user reads one message
|
||||
// rather than an "exit status 1" stacked under it.
|
||||
Describe("ExitCodeError", func() {
|
||||
It("carries the status out", func() {
|
||||
Expect(ExitCodeError{Code: 2}.Code).To(Equal(2))
|
||||
})
|
||||
|
||||
It("is recognisable after wrapping", func() {
|
||||
var got ExitCodeError
|
||||
Expect(errors.As(fmt.Errorf("chat: %w", ExitCodeError{Code: 2}), &got)).To(BeTrue())
|
||||
Expect(got.Code).To(Equal(2))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("agentArgs", func() {
|
||||
It("translates mode flags into the agent's own flags", func() {
|
||||
c := &ChatCMD{CLI: true}
|
||||
Expect(c.agentArgs()).To(Equal([]string{"--cli"}))
|
||||
})
|
||||
|
||||
It("puts forwarded arguments after the translated flags", func() {
|
||||
c := &ChatCMD{Height: "40%", Args: []string{"plugin", "list"}}
|
||||
Expect(c.agentArgs()).To(Equal([]string{"--height", "40%", "plugin", "list"}))
|
||||
})
|
||||
|
||||
It("returns nothing for a bare invocation", func() {
|
||||
Expect((&ChatCMD{}).agentArgs()).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -9,7 +9,7 @@ var CLI struct {
|
||||
cliContext.Context `embed:""`
|
||||
|
||||
Run RunCMD `cmd:"" help:"Run LocalAI, this the default command if no other command is specified. Run 'local-ai run --help' for more information" default:"withargs"`
|
||||
Chat ChatCMD `cmd:"" help:"Open an interactive chat session against a running LocalAI server"`
|
||||
Chat ChatCMD `cmd:"" help:"Run the built-in terminal agent against a LocalAI server"`
|
||||
Federated FederatedCLI `cmd:"" help:"Run LocalAI in federated mode"`
|
||||
Models ModelsCMD `cmd:"" help:"Manage LocalAI models and definitions"`
|
||||
Backends BackendsCMD `cmd:"" help:"Manage LocalAI backends and definitions"`
|
||||
|
||||
15
core/cli/exit.go
Normal file
15
core/cli/exit.go
Normal file
@@ -0,0 +1,15 @@
|
||||
package cli
|
||||
|
||||
import "fmt"
|
||||
|
||||
// ExitCodeError is a failure a command has already reported to the user. It
|
||||
// carries nothing but the status the process should exit with, and main prints
|
||||
// nothing more for it.
|
||||
//
|
||||
// It exists for commands that hand their terminal to something that does its
|
||||
// own error reporting. Returning that subordinate's error instead would put a
|
||||
// bare "exit status 1" underneath the explanation the user has just read, and
|
||||
// returning nil would tell a script the run succeeded.
|
||||
type ExitCodeError struct{ Code int }
|
||||
|
||||
func (e ExitCodeError) Error() string { return fmt.Sprintf("exit status %d", e.Code) }
|
||||
@@ -24,6 +24,7 @@ LocalAI provides a comprehensive set of features for running AI models locally.
|
||||
- **[Agent Actions]({{% relref "features/agent-actions" %}})** - The catalog of actions an agent can run.
|
||||
- **[Model Context Protocol (MCP)]({{% relref "features/mcp" %}})** - Give a model external tools over MCP.
|
||||
- **[LocalAI Assistant]({{% relref "features/localai-assistant" %}})** - Chat to administer your LocalAI instance.
|
||||
- **[Terminal agent]({{% relref "features/terminal-agent" %}})** - `local-ai chat`, an agent in your shell that runs commands behind an approval gate.
|
||||
|
||||
## Audio
|
||||
|
||||
|
||||
@@ -11,13 +11,7 @@ LocalAI includes a built-in agent platform powered by [LocalAGI](https://github.
|
||||
|
||||
LocalAGI is embedded in LocalAI. There is nothing separate to install or run.
|
||||
|
||||
{{% notice info %}}
|
||||
**Looking for something else?** LocalAI has three related agentic features that are easy to confuse:
|
||||
|
||||
- **Agents** (this page): autonomous agents you build that reason, use tools, and act on their own. Start here if you want to create an agent.
|
||||
- **LocalAI Assistant** ({{% relref "features/localai-assistant" %}}): an admin chat modality for administering LocalAI itself (install models, manage backends) by chatting.
|
||||
- **MCP** ({{% relref "features/mcp" %}}): a way to give a model external tools through the Model Context Protocol.
|
||||
{{% /notice %}}
|
||||
{{< agentic-routing current="agents" >}}
|
||||
|
||||
{{% notice tip %}}
|
||||
New to agents? The [Build your first agent]({{% relref "getting-started/first-agent" %}}) walkthrough takes you from an empty Agents page to an agent that answers a message and uses one tool.
|
||||
|
||||
@@ -5,13 +5,7 @@ weight = 23
|
||||
url = '/features/localai-assistant'
|
||||
+++
|
||||
|
||||
{{% notice info %}}
|
||||
**Looking for something else?** LocalAI has three related agentic features that are easy to confuse:
|
||||
|
||||
- **LocalAI Assistant** (this page): an admin chat modality for administering LocalAI itself (install models, manage backends) by chatting.
|
||||
- **Agents** ({{% relref "features/agents" %}}): autonomous agents you build that reason, use tools, and act on their own.
|
||||
- **MCP** ({{% relref "features/mcp" %}}): a way to give a model external tools through the Model Context Protocol.
|
||||
{{% /notice %}}
|
||||
{{< agentic-routing current="localai-assistant" >}}
|
||||
|
||||
LocalAI Assistant is an admin-only chat modality. When enabled on a chat session, the conversation is wired to an in-process MCP server that exposes LocalAI's own admin/management surface as tools. You can install models, manage backends, edit model configs and check system status by chatting, with no REST calls or YAML edits.
|
||||
|
||||
|
||||
@@ -12,13 +12,7 @@ categories = ["Features"]
|
||||
|
||||
LocalAI now supports the **Model Context Protocol (MCP)**, enabling powerful agentic capabilities by connecting AI models to external tools and services. This feature allows your LocalAI models to interact with various MCP servers, providing access to real-time data, APIs, and specialized tools.
|
||||
|
||||
{{% notice info %}}
|
||||
**Looking for something else?** LocalAI has three related agentic features that are easy to confuse:
|
||||
|
||||
- **MCP** (this page): a way to give a model external tools through the Model Context Protocol.
|
||||
- **Agents** ({{% relref "features/agents" %}}): autonomous agents you build that reason, use tools, and act on their own.
|
||||
- **LocalAI Assistant** ({{% relref "features/localai-assistant" %}}): an admin chat modality for administering LocalAI itself (install models, manage backends) by chatting.
|
||||
{{% /notice %}}
|
||||
{{< agentic-routing current="mcp" >}}
|
||||
|
||||
## What is MCP?
|
||||
|
||||
|
||||
152
docs/content/features/terminal-agent.md
Normal file
152
docs/content/features/terminal-agent.md
Normal file
@@ -0,0 +1,152 @@
|
||||
+++
|
||||
disableToc = false
|
||||
title = "Terminal agent"
|
||||
weight = 24
|
||||
url = '/features/terminal-agent'
|
||||
+++
|
||||
|
||||
{{< agentic-routing current="terminal-agent" >}}
|
||||
|
||||
`local-ai chat` opens a terminal agent that runs against your LocalAI server. It is not a
|
||||
separate tool: the agent is compiled into the `local-ai` binary, so if you have LocalAI you
|
||||
already have it.
|
||||
|
||||
It works in the shell you already have open. It reads your files, runs commands on your
|
||||
machine, delegates to sub-agents, and loads MCP servers, plugins and skills. Read-only work
|
||||
happens on its own; anything that can change something asks you first.
|
||||
|
||||
The agent is [nib](https://github.com/mudler/nib) embedded in LocalAI, which is why its
|
||||
plugin, skill and MCP ecosystem works here unchanged.
|
||||
|
||||
{{% notice warning %}}
|
||||
**Breaking change.** `local-ai chat` used to be a plain chat prompt. It is now an agent that
|
||||
executes shell commands on the machine it runs on, behind an approval gate you answer. The
|
||||
old `/clear` command is gone, and `/compact` is the nearest equivalent.
|
||||
{{% /notice %}}
|
||||
|
||||
## Start a session
|
||||
|
||||
```bash
|
||||
# Terminal 1
|
||||
local-ai run
|
||||
|
||||
# Terminal 2
|
||||
local-ai chat
|
||||
```
|
||||
|
||||
That opens the full-screen interface. Pass `--cli` for a plain, pipe-friendly session
|
||||
instead.
|
||||
|
||||
If nothing is listening, an interactive session offers to start a server for you and stops it
|
||||
again when you exit. Anywhere else, for instance in a script, it fails and points you at
|
||||
`local-ai run`.
|
||||
|
||||
### How the model is chosen
|
||||
|
||||
The agent picks the model in this order, and stops at the first one that answers:
|
||||
|
||||
1. `--model`, if you passed it.
|
||||
2. The model saved in the agent's config.
|
||||
3. The only model the server offers, when there is exactly one.
|
||||
4. An interactive picker, when there are several and you are at a terminal. Your choice is
|
||||
saved, so the question is asked once.
|
||||
5. Otherwise it fails, listing the models the server does offer.
|
||||
|
||||
## Summon it with Ctrl+Space
|
||||
|
||||
`local-ai chat --init` prints a shell widget that binds `Ctrl+Space` to the agent, so you can
|
||||
call it from a half-typed command line without losing what you were doing:
|
||||
|
||||
```bash
|
||||
# zsh
|
||||
echo 'eval "$(local-ai chat --init zsh)"' >> ~/.zshrc
|
||||
# bash
|
||||
echo 'eval "$(local-ai chat --init bash)"' >> ~/.bashrc
|
||||
# fish
|
||||
echo 'local-ai chat --init fish | source' >> ~/.config/fish/config.fish
|
||||
```
|
||||
|
||||
## Inside a session
|
||||
|
||||
- `/models` lists the models the server offers, marking the one you are using.
|
||||
- `/model <name>` switches model and keeps the conversation you are in.
|
||||
- `/compact` summarises the history so far to free up context.
|
||||
|
||||
## Tool approval
|
||||
|
||||
Read-only tools run without asking. That covers reads and searches, and read-only shell
|
||||
commands such as `ls` and `cat`, so the agent can look around and answer you without
|
||||
interrupting. Everything else prompts for approval, and you can approve, deny, or trust the
|
||||
tool for the rest of the session.
|
||||
|
||||
nib's README keeps the
|
||||
[exact list of what counts as read-only](https://github.com/mudler/nib#what-a-piped-run-may-and-may-not-do),
|
||||
along with the [approval modes](https://github.com/mudler/nib#tool-approval) you can set in
|
||||
the agent's own config.
|
||||
|
||||
`--yolo`, or `LOCALAI_CHAT_YOLO=1`, approves every tool call without asking. It is the right
|
||||
setting for a sandbox and the wrong one for your laptop.
|
||||
|
||||
## Piping and scripting
|
||||
|
||||
`--cli` makes the agent answer a piped question and exit:
|
||||
|
||||
```bash
|
||||
echo "what is 2+2" | local-ai chat --cli
|
||||
```
|
||||
|
||||
That exits `0`. Read-only tools still run, so a piped question can inspect a repository or a
|
||||
log file to answer you.
|
||||
|
||||
A tool call that is not read-only is a different matter: there is nobody at the keyboard to
|
||||
approve it, so the call is denied and the session exits `3`. The code is deliberately not `1`,
|
||||
so a script can tell "I refused to act" apart from "I crashed".
|
||||
|
||||
Pipe a question in without `--cli` and the agent refuses, with a message naming `--cli`. It
|
||||
will not render a full-screen interface into a pipe that cannot show it.
|
||||
|
||||
Redirecting the output is not the same thing and is not refused: `local-ai chat > out.txt`
|
||||
from a terminal draws the interface on `/dev/tty` and writes only the command you pick with
|
||||
`Ctrl+Y` to the file. That is the same capture the `Ctrl+Space` widget is built on, so both
|
||||
work. It is the stdin that has to be a terminal.
|
||||
|
||||
## Where the agent keeps its state
|
||||
|
||||
Config, plugins and skills live in `~/.config/localai/chat/`, or under `$XDG_CONFIG_HOME`
|
||||
when you have set it. Override it with `--config-dir`, or with
|
||||
`LOCALAI_CHAT_CONFIG_DIR`. The directory is user-scoped rather than server-scoped, because
|
||||
the agent is a client and may be pointed at a remote LocalAI.
|
||||
|
||||
## Plugins, skills and MCP servers
|
||||
|
||||
The agent's own management commands are reached by passing them through:
|
||||
|
||||
```bash
|
||||
local-ai chat plugin install https://github.com/user/plugin
|
||||
local-ai chat skill list
|
||||
local-ai chat mcp add my-server -- npx -y @modelcontextprotocol/server-filesystem /tmp
|
||||
```
|
||||
|
||||
Everything after the first positional argument is forwarded verbatim, so LocalAI's own flags
|
||||
have to come first:
|
||||
|
||||
```bash
|
||||
local-ai chat --config-dir /srv/agent plugin list
|
||||
```
|
||||
|
||||
Claude Code plugins install as they are, so a plugin written for that format needs no
|
||||
conversion.
|
||||
|
||||
{{% notice warning %}}
|
||||
**Pass `--yes` in scripts.** The management commands do not read the streams LocalAI hands
|
||||
them, so `plugin install` without `--yes` in a non-interactive context installs the plugin
|
||||
and leaves it **disabled**. It prints that it did, on the line
|
||||
`Plugin "name" installed but left disabled`, but it exits `0` either way, so a script that
|
||||
only checks the exit code cannot tell the two outcomes apart. Always pass `--yes` when you
|
||||
are not at a terminal.
|
||||
{{% /notice %}}
|
||||
|
||||
## Flags
|
||||
|
||||
Every flag, with its environment variable, is in
|
||||
[Chat flags]({{% relref "reference/cli-reference" %}}#chat-flags).
|
||||
@@ -30,7 +30,11 @@ local-ai run
|
||||
local-ai chat --model qwen3-4b
|
||||
```
|
||||
|
||||
`local-ai chat` connects to a running LocalAI server, opens an interactive chat prompt, and exits when you type `/exit`, `/quit`, or `/bye`. Use `/models` to list installed models, `/model <name>` to switch models, and `/clear` to reset the current conversation. If the server exposes exactly one model, LocalAI uses that model automatically:
|
||||
`local-ai chat` connects to a running LocalAI server and opens the built-in terminal agent. It is more than a chat prompt: it can read your files and run commands on your machine, so the first time it wants to do something that changes state it stops and asks you to approve the call. Reads and searches run without asking. Press `Esc` or `Ctrl+C` to end the session. `Esc` is a key of the full-screen interface: in the plain `--cli` mode, leave with `Ctrl+C`, `Ctrl+D`, or by typing `exit`.
|
||||
|
||||
Use `/models` to list installed models, `/model <name>` to switch models while keeping the conversation, and `/compact` to summarize the history so far when the context fills up. The full picture is on the [Terminal agent]({{% relref "features/terminal-agent" %}}) page.
|
||||
|
||||
If the server exposes exactly one model, LocalAI uses that model automatically:
|
||||
|
||||
```bash
|
||||
# Terminal 1
|
||||
@@ -40,7 +44,7 @@ local-ai run qwen3-4b
|
||||
local-ai chat
|
||||
```
|
||||
|
||||
When more than one model is configured, pass `--model` with the installed model name to avoid ambiguity. Use `--endpoint` to connect to a non-default server, for example `local-ai chat --endpoint http://127.0.0.1:8081 --model qwen3-4b`.
|
||||
When more than one model is configured, the agent asks you to pick one and remembers your answer, or you can pass `--model` with the installed model name. Use `--endpoint` to connect to a non-default server, for example `local-ai chat --endpoint http://127.0.0.1:8081 --model qwen3-4b`.
|
||||
|
||||
You can also test out the API endpoints using `curl`. A few examples are listed below.
|
||||
|
||||
|
||||
@@ -123,18 +123,65 @@ See [Authentication & Authorization]({{%relref "features/authentication" %}}) fo
|
||||
|
||||
## Chat Flags
|
||||
|
||||
Use `local-ai chat` to open an interactive terminal chat session against a running LocalAI server.
|
||||
Use `local-ai chat` to run the built-in terminal agent against a LocalAI server.
|
||||
The agent can run shell commands, delegate to sub-agents, and use MCP tools. Read-only
|
||||
calls run on their own; everything else goes through an approval prompt you answer.
|
||||
See [Terminal agent]({{% relref "features/terminal-agent" %}}) for the full feature page.
|
||||
|
||||
| Parameter | Default | Description | Environment Variable |
|
||||
|-----------|---------|-------------|----------------------|
|
||||
| `--endpoint` | `http://127.0.0.1:8080` | LocalAI server endpoint. The `/v1` path is added automatically when omitted. | `$LOCALAI_CHAT_ENDPOINT` |
|
||||
| `--model` | | Model name to use. If omitted, LocalAI uses the only model returned by the server when exactly one is available. | |
|
||||
| `--model` | | Model to use. Defaults to the only model the server offers, or asks when there are several. | |
|
||||
| `--api-key` | | API key to use when the LocalAI server requires authentication. | `$LOCALAI_API_KEY`, `$API_KEY` |
|
||||
| `--config-dir` | `~/.config/localai/chat` | Directory holding the agent's config, plugins, and skills. | `$LOCALAI_CHAT_CONFIG_DIR` |
|
||||
| `--trace-dir` | | Write a session LLM trace (NDJSON) to this directory. | `$LOCALAI_CHAT_TRACE_DIR` |
|
||||
| `--cli` | `false` | Plain CLI mode instead of the full-screen interface. Pipe stdin for one-shot use. | |
|
||||
| `--tui` | `false` | Force the full-screen interface. | |
|
||||
| `--height` | | Run as an inline drop-down of this height, e.g. `40%`. | |
|
||||
| `--tmux` / `--no-tmux` | | Control the tmux split. | |
|
||||
| `--init` | | Print the shell integration script for `Ctrl+Space` (zsh, bash, or fish). | |
|
||||
| `--yolo` | `false` | Auto-approve every tool call. Use with care. | `$LOCALAI_CHAT_YOLO` |
|
||||
|
||||
- Inside the chat prompt:
|
||||
- Use `/models` to list installed models.
|
||||
- Use `/model <name>` to switch to a different model and clear the conversation.
|
||||
- Use `/clear` to reset the current conversation.
|
||||
The agent's own subcommands are reached by passing them through. LocalAI's flags
|
||||
must come first, because everything after the first positional argument is
|
||||
forwarded verbatim:
|
||||
|
||||
```bash
|
||||
local-ai chat plugin install https://github.com/user/plugin
|
||||
local-ai chat skill list
|
||||
local-ai chat mcp add my-server -- npx -y @modelcontextprotocol/server-filesystem /tmp
|
||||
local-ai chat --config-dir /srv/agent plugin list # flags first
|
||||
```
|
||||
|
||||
{{% notice warning %}}
|
||||
In a script, pass `--yes` to those subcommands. Without it, a non-interactive
|
||||
`plugin install` installs the plugin, leaves it disabled, and still exits `0`.
|
||||
{{% /notice %}}
|
||||
|
||||
Summon the agent from any shell prompt with `Ctrl+Space`:
|
||||
|
||||
```bash
|
||||
echo 'eval "$(local-ai chat --init zsh)"' >> ~/.zshrc
|
||||
```
|
||||
|
||||
Inside a session:
|
||||
- `/models` lists the models the server offers, marking the current one.
|
||||
- `/model <name>` switches model, keeping the conversation.
|
||||
- `/compact` summarizes the conversation so far to free up context.
|
||||
- `/skill <name>`, `/agent <name> <task>`, `/attach <file>`, `/goal <text>`.
|
||||
|
||||
If no server is reachable, the agent offers to start one for the session
|
||||
(interactive terminals only) and otherwise points you at `local-ai run`.
|
||||
|
||||
Piped use needs `--cli`, and exits `0` when it answers:
|
||||
|
||||
```bash
|
||||
echo "what is 2+2" | local-ai chat --cli
|
||||
```
|
||||
|
||||
Read-only tools still run in a piped session. A tool call that is not read-only has
|
||||
nobody to approve it, so it is denied and the session exits `3`, a code chosen to be
|
||||
distinct from the `1` a failure reports.
|
||||
|
||||
## P2P Flags
|
||||
|
||||
@@ -153,7 +200,7 @@ LocalAI supports several subcommands beyond `run`:
|
||||
|
||||
- `local-ai models` - Manage LocalAI models and definitions
|
||||
- `local-ai backends` - Manage LocalAI backends and definitions
|
||||
- `local-ai chat` - Open an interactive chat session against a running LocalAI server
|
||||
- `local-ai chat` - Run the built-in terminal agent against a LocalAI server
|
||||
- `local-ai tts` - Convert text to speech
|
||||
- `local-ai sound-generation` - Generate audio files from text or audio
|
||||
- `local-ai transcript` - Convert audio to text
|
||||
|
||||
77
docs/layouts/shortcodes/agentic-routing.html
Normal file
77
docs/layouts/shortcodes/agentic-routing.html
Normal file
@@ -0,0 +1,77 @@
|
||||
{{- /*
|
||||
The four agentic surfaces are easy to mistake for each other, so every one of
|
||||
their pages opens with the same routing block. It lives here rather than being
|
||||
pasted into each page, because four hand-kept copies drifted apart the last
|
||||
time and a reader who lands on the wrong one reads a stale description of the
|
||||
page they actually wanted.
|
||||
|
||||
Call it with angle brackets. The list is built here as markdown and rendered
|
||||
before it reaches the notice partial, which is what a percent-form `notice`
|
||||
does with its own inner content; handing the partial raw markdown instead
|
||||
leaves the asterisks on the page, because it drops its content into an HTML
|
||||
block that goldmark then declines to look inside.
|
||||
|
||||
`current` names the page it is rendered on. That entry loses its link, is
|
||||
marked as the current page, and moves to the top, which is what the copies
|
||||
used to do by hand. Omitting it renders the block as a plain four-way index,
|
||||
which is what a page outside the four wants.
|
||||
*/ -}}
|
||||
{{- $current := .Get "current" | default (.Get 0) -}}
|
||||
{{- $entries := slice
|
||||
(dict
|
||||
"id" "terminal-agent"
|
||||
"title" "Terminal agent"
|
||||
"ref" "features/terminal-agent"
|
||||
"text" "an agent in your own shell. `local-ai chat` reads your files and runs commands on your machine, behind an approval gate you control.")
|
||||
(dict
|
||||
"id" "agents"
|
||||
"title" "Agents"
|
||||
"ref" "features/agents"
|
||||
"text" "autonomous agents you build and run inside LocalAI, that reason, use tools, and act on their own.")
|
||||
(dict
|
||||
"id" "localai-assistant"
|
||||
"title" "LocalAI Assistant"
|
||||
"ref" "features/localai-assistant"
|
||||
"text" "an admin chat modality for administering LocalAI itself: install models, manage backends and edit configs by chatting.")
|
||||
(dict
|
||||
"id" "mcp"
|
||||
"title" "MCP"
|
||||
"ref" "features/mcp"
|
||||
"text" "the Model Context Protocol itself, the way a model is handed external tools. It sits underneath the other three rather than beside them.")
|
||||
-}}
|
||||
|
||||
{{- /* A typo in `current` would silently render the block with nothing marked
|
||||
as the current page, which is the one failure a reader cannot spot. */ -}}
|
||||
{{- if $current -}}
|
||||
{{- $known := false -}}
|
||||
{{- range $entries -}}
|
||||
{{- if eq .id $current -}}{{- $known = true -}}{{- end -}}
|
||||
{{- end -}}
|
||||
{{- if not $known -}}
|
||||
{{- errorf "agentic-routing: current=%q is not one of the four surfaces (terminal-agent, agents, localai-assistant, mcp)" $current -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- $ordered := slice -}}
|
||||
{{- range $entries -}}
|
||||
{{- if eq .id $current -}}{{- $ordered = $ordered | append . -}}{{- end -}}
|
||||
{{- end -}}
|
||||
{{- range $entries -}}
|
||||
{{- if ne .id $current -}}{{- $ordered = $ordered | append . -}}{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- $lines := slice -}}
|
||||
{{- range $ordered -}}
|
||||
{{- if eq .id $current -}}
|
||||
{{- $lines = $lines | append (printf "- **%s** (this page): %s" .title .text) -}}
|
||||
{{- else -}}
|
||||
{{- $lines = $lines | append (printf "- **[%s](%s)**: %s" .title (relref $.Page .ref) .text) -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- $intro := "**Looking for something else?** LocalAI has four agentic surfaces that are easy to confuse:" -}}
|
||||
{{- partial "shortcodes/notice.html" (dict
|
||||
"page" .Page
|
||||
"style" "info"
|
||||
"content" (printf "%s\n\n%s\n" $intro (delimit $lines "\n") | markdownify)
|
||||
) -}}
|
||||
19
go.mod
19
go.mod
@@ -36,10 +36,11 @@ require (
|
||||
github.com/mholt/archiver/v3 v3.5.1
|
||||
github.com/microcosm-cc/bluemonday v1.0.27
|
||||
github.com/modelcontextprotocol/go-sdk v1.5.0
|
||||
github.com/mudler/cogito v0.10.1-0.20260609212329-bf4010d31047
|
||||
github.com/mudler/cogito v0.11.1-0.20260721122412-6eece18a6bb6
|
||||
github.com/mudler/edgevpn v0.34.0
|
||||
github.com/mudler/go-processmanager v0.1.2-0.20260720195933-3d64f5c974fc
|
||||
github.com/mudler/memory v0.0.0-20260406210934-424c1ecf2cf8
|
||||
github.com/mudler/nib v0.6.0
|
||||
github.com/mudler/xlog v0.0.6
|
||||
github.com/nats-io/jwt/v2 v2.7.4
|
||||
github.com/nats-io/nats.go v1.52.0
|
||||
@@ -85,6 +86,7 @@ require (
|
||||
filippo.io/bigmod v0.1.1-0.20260103110540-f8a47775ebe5 // indirect
|
||||
filippo.io/keygen v0.0.0-20260114151900-8e2790ea4c5b // indirect
|
||||
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
|
||||
github.com/atotto/clipboard v0.1.4 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.9 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect
|
||||
@@ -103,12 +105,19 @@ require (
|
||||
github.com/blang/semver v3.5.1+incompatible // indirect
|
||||
github.com/buger/jsonparser v1.1.2 // indirect
|
||||
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
|
||||
github.com/charmbracelet/bubbles v0.21.0 // indirect
|
||||
github.com/charmbracelet/bubbletea v1.3.10 // indirect
|
||||
github.com/chromedp/cdproto v0.0.0-20260321001828-e3e3800016bc // indirect
|
||||
github.com/chromedp/chromedp v0.15.1 // indirect
|
||||
github.com/chromedp/sysutil v1.1.0 // indirect
|
||||
github.com/cyberphone/json-canonicalization v0.0.0-20241213102144-19d51d7fe467 // indirect
|
||||
github.com/digitorus/pkcs7 v0.0.0-20230818184609-3a137a874352 // indirect
|
||||
github.com/digitorus/timestamp v0.0.0-20231217203849-220c5c2851b7 // indirect
|
||||
github.com/dunglas/httpsfv v1.1.0 // indirect
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
|
||||
github.com/filecoin-project/go-clock v0.1.0 // indirect
|
||||
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
|
||||
github.com/go-json-experiment/json v0.0.0-20260214004413-d219187c3433 // indirect
|
||||
github.com/go-openapi/analysis v0.24.1 // indirect
|
||||
github.com/go-openapi/errors v0.22.4 // indirect
|
||||
github.com/go-openapi/loads v0.23.2 // indirect
|
||||
@@ -127,6 +136,9 @@ require (
|
||||
github.com/go-openapi/swag/yamlutils v0.25.4 // indirect
|
||||
github.com/go-openapi/validate v0.25.1 // indirect
|
||||
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
|
||||
github.com/gobwas/httphead v0.1.0 // indirect
|
||||
github.com/gobwas/pool v0.2.1 // indirect
|
||||
github.com/gobwas/ws v1.4.0 // indirect
|
||||
github.com/google/certificate-transparency-go v1.3.2 // indirect
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect
|
||||
github.com/in-toto/attestation v1.1.2 // indirect
|
||||
@@ -136,9 +148,12 @@ require (
|
||||
github.com/jinzhu/now v1.1.5 // indirect
|
||||
github.com/jolestar/go-commons-pool/v2 v2.1.2 // indirect
|
||||
github.com/klippa-app/go-pdfium v1.19.2 // indirect
|
||||
github.com/mattn/go-localereader v0.0.1 // indirect
|
||||
github.com/mattn/go-sqlite3 v1.14.28 // indirect
|
||||
github.com/moby/moby/api v1.54.2 // indirect
|
||||
github.com/moby/moby/client v0.4.1 // indirect
|
||||
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
|
||||
github.com/muesli/cancelreader v0.2.2 // indirect
|
||||
github.com/nats-io/nuid v1.0.1 // indirect
|
||||
github.com/oklog/ulid v1.3.1 // indirect
|
||||
github.com/secure-systems-lab/go-securesystemslib v0.9.1 // indirect
|
||||
@@ -344,7 +359,7 @@ require (
|
||||
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
|
||||
github.com/KyleBanks/depth v1.2.1 // indirect
|
||||
github.com/Masterminds/goutils v1.1.1 // indirect
|
||||
github.com/Masterminds/semver/v3 v3.4.0 // indirect
|
||||
github.com/Masterminds/semver/v3 v3.5.0 // indirect
|
||||
github.com/Microsoft/go-winio v0.6.2 // indirect
|
||||
github.com/Microsoft/hcsshim v0.11.7 // indirect
|
||||
github.com/alecthomas/chroma/v2 v2.20.0 // indirect
|
||||
|
||||
49
go.sum
49
go.sum
@@ -91,10 +91,12 @@ github.com/JohannesKaufmann/html-to-markdown/v2 v2.4.0 h1:C0/TerKdQX9Y9pbYi1EsLr
|
||||
github.com/JohannesKaufmann/html-to-markdown/v2 v2.4.0/go.mod h1:OLaKh+giepO8j7teevrNwiy/fwf8LXgoc9g7rwaE1jk=
|
||||
github.com/KyleBanks/depth v1.2.1 h1:5h8fQADFrWtarTdtDudMmGsC7GPbOAu6RVB3ffsVFHc=
|
||||
github.com/KyleBanks/depth v1.2.1/go.mod h1:jzSb9d0L43HxTQfT+oSA1EEp2q+ne2uh6XgeJcm8brE=
|
||||
github.com/MakeNowJust/heredoc v1.0.0 h1:cXCdzVdstXyiTqTvfqk9SDHpKNjxuom+DOlyEeQ4pzQ=
|
||||
github.com/MakeNowJust/heredoc v1.0.0/go.mod h1:mG5amYoWBHf8vpLOuehzbGGw0EHxpZZ6lCpQ4fNJ8LE=
|
||||
github.com/Masterminds/goutils v1.1.1 h1:5nUrii3FMTL5diU80unEVvNevw1nH4+ZV4DSLVJLSYI=
|
||||
github.com/Masterminds/goutils v1.1.1/go.mod h1:8cTjp+g8YejhMuvIA5y2vz3BpJxksy863GQaJW2MFNU=
|
||||
github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
|
||||
github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
|
||||
github.com/Masterminds/semver/v3 v3.5.0 h1:kQceYJfbupGfZOKZQg0kou0DgAKhzDg2NZPAwZ/2OOE=
|
||||
github.com/Masterminds/semver/v3 v3.5.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
|
||||
github.com/Masterminds/sprig/v3 v3.3.0 h1:mQh0Yrg1XPo6vjYXgtf5OtijNAKJRNcTdOOGZe3tPhs=
|
||||
github.com/Masterminds/sprig/v3 v3.3.0/go.mod h1:Zy1iXRYNqNLUolqCpL4uhk6SHUMAOSCzdgBfDb35Lz0=
|
||||
github.com/Microsoft/go-winio v0.5.2/go.mod h1:WpS1mjBmmwHBEWmogvA2mj8546UReBk4v8QkMxJ6pZY=
|
||||
@@ -142,6 +144,8 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPd
|
||||
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
|
||||
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3dyBCFEj5IhUbnKptjxatkF07cF2ak3yi77so=
|
||||
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
|
||||
github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4=
|
||||
github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
|
||||
github.com/aws/aws-sdk-go v1.55.7 h1:UJrkFq7es5CShfBwlWAC8DA077vp8PyVbQd3lqLiztE=
|
||||
github.com/aws/aws-sdk-go v1.55.7/go.mod h1:eRwEWoyTWFMVYVQzKMNHWP5/RV4xIUGMQfXQHfHkpNU=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.7 h1:DWpAJt66FmnnaRIOT/8ASTucrvuDPZASqhhLey6tLY8=
|
||||
@@ -253,6 +257,10 @@ github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F9
|
||||
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/charmbracelet/bubbles v0.21.0 h1:9TdC97SdRVg/1aaXNVWfFH3nnLAwOXr8Fn6u6mfQdFs=
|
||||
github.com/charmbracelet/bubbles v0.21.0/go.mod h1:HF+v6QUR4HkEpz62dx7ym2xc71/KBHg+zKwJtMw+qtg=
|
||||
github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
|
||||
github.com/charmbracelet/bubbletea v1.3.10/go.mod h1:ORQfo0fk8U+po9VaNvnV95UPWA1BitP1E0N6xJPlHr4=
|
||||
github.com/charmbracelet/colorprofile v0.2.3-0.20250311203215-f60798e515dc h1:4pZI35227imm7yK2bGPcfpFEmuY1gc2YSTShr4iJBfs=
|
||||
github.com/charmbracelet/colorprofile v0.2.3-0.20250311203215-f60798e515dc/go.mod h1:X4/0JoqgTIPSFcRA/P6INZzIuyqdFY5rm8tb41s9okk=
|
||||
github.com/charmbracelet/glamour v1.0.0 h1:AWMLOVFHTsysl4WV8T8QgkQ0s/ZNZo7CiE4WKhk8l08=
|
||||
@@ -263,14 +271,20 @@ github.com/charmbracelet/x/ansi v0.10.2 h1:ith2ArZS0CJG30cIUfID1LXN7ZFXRCww6RUvA
|
||||
github.com/charmbracelet/x/ansi v0.10.2/go.mod h1:HbLdJjQH4UH4AqA2HpRWuWNluRE6zxJH/yteYEYCFa8=
|
||||
github.com/charmbracelet/x/cellbuf v0.0.13 h1:/KBBKHuVRbq1lYx5BzEHBAFBP8VcQzJejZ/IA3iR28k=
|
||||
github.com/charmbracelet/x/cellbuf v0.0.13/go.mod h1:xe0nKWGd3eJgtqZRaN9RjMtK7xUYchjzPr7q6kcvCCs=
|
||||
github.com/charmbracelet/x/exp/golden v0.0.0-20240806155701-69247e0abc2a h1:G99klV19u0QnhiizODirwVksQB91TJKV/UaTnACcG30=
|
||||
github.com/charmbracelet/x/exp/golden v0.0.0-20240806155701-69247e0abc2a/go.mod h1:wDlXFlCrmJ8J+swcL/MnGUuYnqgQdW9rhSD61oNMb6U=
|
||||
github.com/charmbracelet/x/exp/golden v0.0.0-20241011142426-46044092ad91 h1:payRxjMjKgx2PaCWLZ4p3ro9y97+TVLZNaRZgJwSVDQ=
|
||||
github.com/charmbracelet/x/exp/golden v0.0.0-20241011142426-46044092ad91/go.mod h1:wDlXFlCrmJ8J+swcL/MnGUuYnqgQdW9rhSD61oNMb6U=
|
||||
github.com/charmbracelet/x/exp/slice v0.0.0-20250327172914-2fdc97757edf h1:rLG0Yb6MQSDKdB52aGX55JT1oi0P0Kuaj7wi1bLUpnI=
|
||||
github.com/charmbracelet/x/exp/slice v0.0.0-20250327172914-2fdc97757edf/go.mod h1:B3UgsnsBZS/eX42BlaNiJkD1pPOUa+oF1IYC6Yd2CEU=
|
||||
github.com/charmbracelet/x/term v0.2.1 h1:AQeHeLZ1OqSXhrAWpYUtZyX1T3zVxfpZuEQMIQaGIAQ=
|
||||
github.com/charmbracelet/x/term v0.2.1/go.mod h1:oQ4enTYFV7QN4m0i9mzHrViD7TQKvNEEkHUMCmsxdUg=
|
||||
github.com/chengxilo/virtualterm v1.0.4 h1:Z6IpERbRVlfB8WkOmtbHiDbBANU7cimRIof7mk9/PwM=
|
||||
github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0iwWCOK1q10rlY=
|
||||
github.com/chromedp/cdproto v0.0.0-20260321001828-e3e3800016bc h1:wkN/LMi5vc60pBRWx6qpbk/aEvq3/ZVNpnMvsw8PVVU=
|
||||
github.com/chromedp/cdproto v0.0.0-20260321001828-e3e3800016bc/go.mod h1:cbyjALe67vDvlvdiG9369P8w5U2w6IshwtyD2f2Tvag=
|
||||
github.com/chromedp/chromedp v0.15.1 h1:EJWiPm7BNqDqjYy6U0lTSL5wNH+iNt9GjC3a4gfjNyQ=
|
||||
github.com/chromedp/chromedp v0.15.1/go.mod h1:CdTHtUqD/dqaFw/cvFWtTydoEQS44wLBuwbMR9EkOY4=
|
||||
github.com/chromedp/sysutil v1.1.0 h1:PUFNv5EcprjqXZD9nJb9b/c9ibAbxiYo4exNWZyipwM=
|
||||
github.com/chromedp/sysutil v1.1.0/go.mod h1:WiThHUdltqCNKGc4gaU50XgYjwjYIhKWoHGPTUfWTJ8=
|
||||
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
|
||||
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
|
||||
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
|
||||
@@ -378,6 +392,8 @@ github.com/envoyproxy/go-control-plane v0.9.7/go.mod h1:cwu0lG7PUMfa9snN8LXBig5y
|
||||
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
|
||||
github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
|
||||
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6baUTXGLOoWe4PQhGxaX0KpnayAqC48p4=
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
|
||||
github.com/eritikass/githubmarkdownconvertergo v0.1.10 h1:mL93ADvYMOeT15DcGtK9AaFFc+RcWcy6kQBC6yS/5f4=
|
||||
github.com/eritikass/githubmarkdownconvertergo v0.1.10/go.mod h1:BdpHs6imOtzE5KorbUtKa6bZ0ZBh1yFcrTTAL8FwDKY=
|
||||
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
|
||||
@@ -441,6 +457,8 @@ github.com/go-gl/glfw/v3.3/glfw v0.0.0-20240506104042-037f3cc74f2a h1:vxnBhFDDT+
|
||||
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20240506104042-037f3cc74f2a/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
|
||||
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
|
||||
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
|
||||
github.com/go-json-experiment/json v0.0.0-20260214004413-d219187c3433 h1:vymEbVwYFP/L05h5TKQxvkXoKxNvTpjxYKdF1Nlwuao=
|
||||
github.com/go-json-experiment/json v0.0.0-20260214004413-d219187c3433/go.mod h1:tphK2c80bpPhMOI4v6bIc2xWywPfbqi1Z06+RcrMkDg=
|
||||
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
|
||||
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
@@ -518,6 +536,12 @@ github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlnd
|
||||
github.com/go-yaml/yaml v2.1.0+incompatible/go.mod h1:w2MrLa16VYP0jy6N7M5kHaCkaLENm+P+Tv+MfurjSw0=
|
||||
github.com/gobwas/glob v0.2.3 h1:A4xDbljILXROh+kObIiy5kIaPYD8e96x1tgBhUI5J+Y=
|
||||
github.com/gobwas/glob v0.2.3/go.mod h1:d3Ez4x06l9bZtSvzIay5+Yzi0fmZzPgnTbPcKjJAkT8=
|
||||
github.com/gobwas/httphead v0.1.0 h1:exrUm0f4YX0L7EBwZHuCF4GDp8aJfVeBrlLQrs6NqWU=
|
||||
github.com/gobwas/httphead v0.1.0/go.mod h1:O/RXo79gxV8G+RqlR/otEwx4Q36zl9rqC5u12GKvMCM=
|
||||
github.com/gobwas/pool v0.2.1 h1:xfeeEhW7pwmX8nuLVlqbzVc7udMDrwetjEv+TZIz1og=
|
||||
github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw=
|
||||
github.com/gobwas/ws v1.4.0 h1:CTaoG1tojrh4ucGPcoJFiAQUAsEWekEWvLy7GsVNqGs=
|
||||
github.com/gobwas/ws v1.4.0/go.mod h1:G3gNqMNtPppf5XUz7O4shetPpcZ1VJ7zt18dlUeakrc=
|
||||
github.com/goccy/go-yaml v1.18.0 h1:8W7wMFS12Pcas7KU+VVkaiCng+kG8QiFeFwzFb+rwuw=
|
||||
github.com/goccy/go-yaml v1.18.0/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
|
||||
github.com/gocolly/colly v1.2.0 h1:qRz9YAn8FIH0qzgNUw+HT9UN7wm1oF9OBAilwEWpyrI=
|
||||
@@ -823,6 +847,8 @@ github.com/labstack/echo/v4 v4.15.1 h1:S9keusg26gZpjMmPqB5hOEvNKnmd1lNmcHrbbH2ln
|
||||
github.com/labstack/echo/v4 v4.15.1/go.mod h1:xmw1clThob0BSVRX1CRQkGQ/vjwcpOMjQZSZa9fKA/c=
|
||||
github.com/labstack/gommon v0.4.2 h1:F8qTUNXgG1+6WQmqoUWnz8WiEU60mXVVw0P4ht1WRA0=
|
||||
github.com/labstack/gommon v0.4.2/go.mod h1:QlUFxVM+SNXhDL/Z7YhocGIBYOiwB0mXm1+1bAPHPyU=
|
||||
github.com/ledongthuc/pdf v0.0.0-20250511090121-5959a4027728 h1:QwWKgMY28TAXaDl+ExRDqGQltzXqN/xypdKP86niVn8=
|
||||
github.com/ledongthuc/pdf v0.0.0-20250511090121-5959a4027728/go.mod h1:1fEHWurg7pvf5SG6XNE5Q8UZmOwex51Mkx3SLhrW5B4=
|
||||
github.com/letsencrypt/boulder v0.20251110.0 h1:J8MnKICeilO91dyQ2n5eBbab24neHzUpYMUIOdOtbjc=
|
||||
github.com/letsencrypt/boulder v0.20251110.0/go.mod h1:ogKCJQwll82m7OVHWyTuf8eeFCjuzdRQlgnZcCl0V+8=
|
||||
github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
|
||||
@@ -885,6 +911,8 @@ github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/
|
||||
github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||
github.com/mattn/go-isatty v0.0.22 h1:j8l17JJ9i6VGPUFUYoTUKPSgKe/83EYU2zBC7YNKMw4=
|
||||
github.com/mattn/go-isatty v0.0.22/go.mod h1:ZXfXG4SQHsB/w3ZeOYbR0PrPwLy+n6xiMrJlRFqopa4=
|
||||
github.com/mattn/go-localereader v0.0.1 h1:ygSAOl7ZXTx4RdPYinUpg6W99U8jWvWi9Ye2JC/oIi4=
|
||||
github.com/mattn/go-localereader v0.0.1/go.mod h1:8fBrzywKY7BI3czFoHkuzRoWE9C+EiG4R1k4Cjx5p88=
|
||||
github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
|
||||
github.com/mattn/go-runewidth v0.0.12/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk=
|
||||
github.com/mattn/go-runewidth v0.0.17 h1:78v8ZlW0bP43XfmAfPsdXcoNCelfMHsDmd/pkENfrjQ=
|
||||
@@ -968,8 +996,8 @@ github.com/mschoch/smat v0.2.0 h1:8imxQsjDm8yFEAVBe7azKmKSgzSkZXDuKkSq9374khM=
|
||||
github.com/mschoch/smat v0.2.0/go.mod h1:kc9mz7DoBKqDyiRL7VZN8KvXQMWeTaVnttLRXOlotKw=
|
||||
github.com/mudler/LocalAGI v0.0.0-20260606071251-14aed1ae4336 h1:iKBkSnpisOvMVxFoYsAObvAuOqXBakRPMD0PWxWG5EE=
|
||||
github.com/mudler/LocalAGI v0.0.0-20260606071251-14aed1ae4336/go.mod h1:U+g6u8mF2wQxhkdBl3dr8G4db1cv3n7KTKmraoJ7D0c=
|
||||
github.com/mudler/cogito v0.10.1-0.20260609212329-bf4010d31047 h1:wJ8WbDah1YcpBNRDmovQro8JiR228YFk7TUqPCS4m04=
|
||||
github.com/mudler/cogito v0.10.1-0.20260609212329-bf4010d31047/go.mod h1:6sfja3lcu2nWRzEc0wwqGNu/eCG3EWgij+8s7xyUeQ4=
|
||||
github.com/mudler/cogito v0.11.1-0.20260721122412-6eece18a6bb6 h1:eYTR8od5HdaHlh9AKCkxkRoHs2/wmx24BF5qrUh2TRY=
|
||||
github.com/mudler/cogito v0.11.1-0.20260721122412-6eece18a6bb6/go.mod h1:6sfja3lcu2nWRzEc0wwqGNu/eCG3EWgij+8s7xyUeQ4=
|
||||
github.com/mudler/edgevpn v0.34.0 h1:qDrD/rCPFY/FdURbXudIZWihVKY4VOX3nMn3CcbeQEU=
|
||||
github.com/mudler/edgevpn v0.34.0/go.mod h1:yki7uMi5LR9gSMrw8PdPieuxsrk8BLV2Ui7VBEmbbIA=
|
||||
github.com/mudler/go-piper v0.0.0-20241023091659-2494246fd9fc h1:RxwneJl1VgvikiX28EkpdAyL4yQVnJMrbquKospjHyA=
|
||||
@@ -980,12 +1008,18 @@ github.com/mudler/localrecall v0.6.3 h1:uXOrP9JmetzxgVKzSrawviyBHZfAcvPBBIrvVUdZ
|
||||
github.com/mudler/localrecall v0.6.3/go.mod h1:28k5n19raUrkuwXkacdNsBlj8yuSnGhpT16tu+2+4dU=
|
||||
github.com/mudler/memory v0.0.0-20260406210934-424c1ecf2cf8 h1:Ry8RiWy8fZ6Ff4E7dPmjRsBrnHOnPeOOj2LhCgyjQu0=
|
||||
github.com/mudler/memory v0.0.0-20260406210934-424c1ecf2cf8/go.mod h1:EA8Ashhd56o32qN7ouPKFSRUs/Z+LrRCF4v6R2Oarm8=
|
||||
github.com/mudler/nib v0.6.0 h1:6l2bQJkgHT5+o6S0wmmk/ipTCbFzmiSp1pM/tGQv4Eo=
|
||||
github.com/mudler/nib v0.6.0/go.mod h1:d+Ymgi7PxDLnGxpYAkugv+mwRDtR1CMcgyykKNipwWA=
|
||||
github.com/mudler/skillserver v0.0.7-0.20260520220837-a7317cbf9145 h1:z59tA3IDYPt71nzH1jpxeaA1LuDw8aZfpTQFNU43Zb8=
|
||||
github.com/mudler/skillserver v0.0.7-0.20260520220837-a7317cbf9145/go.mod h1:z3yFhcL9bSykmmh6xgGu0hyoItd4CnxgtWMEWw8uFJU=
|
||||
github.com/mudler/water v0.0.0-20250808092830-dd90dcf09025 h1:WFLP5FHInarYGXi6B/Ze204x7Xy6q/I4nCZnWEyPHK0=
|
||||
github.com/mudler/water v0.0.0-20250808092830-dd90dcf09025/go.mod h1:QuIFdRstyGJt+MTTkWY+mtD7U6xwjOR6SwKUjmLZtR4=
|
||||
github.com/mudler/xlog v0.0.6 h1:3nBV4THK8kY0Y8FDXXvWAnuAJoOyO7EAXteJeAoHUC0=
|
||||
github.com/mudler/xlog v0.0.6/go.mod h1:3pO/Dsp3ViWl1QLyNtK7VQDJqMlntVu+rm5lP5PkX1g=
|
||||
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 h1:ZK8zHtRHOkbHy6Mmr5D264iyp3TiX5OmNcI5cIARiQI=
|
||||
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6/go.mod h1:CJlz5H+gyd6CUWT45Oy4q24RdLyn7Md9Vj2/ldJBSIo=
|
||||
github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELUXHmA=
|
||||
github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
|
||||
github.com/muesli/reflow v0.3.0 h1:IFsN6K9NfGtjeggFP+68I4chLZV2yIKsXJFNZ+eWh6s=
|
||||
github.com/muesli/reflow v0.3.0/go.mod h1:pbwTDkVPibjO2kyvBQRBxTWEEGDGq0FlB1BIKtnHY/8=
|
||||
github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
|
||||
@@ -1054,6 +1088,8 @@ github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJw
|
||||
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
|
||||
github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+1B0VhjKrZUs=
|
||||
github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc=
|
||||
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde h1:x0TT0RDC7UhAVbbWWBzr41ElhJx5tXPWkIHA2HWPRuw=
|
||||
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde/go.mod h1:nZgzbfBr3hhjoZnS66nKrHmduYNpc34ny7RK4z5/HM0=
|
||||
github.com/otiai10/copy v1.14.1 h1:5/7E6qsUMBaH5AnQ0sSLzzTg1oTECmcCmT6lvF45Na8=
|
||||
github.com/otiai10/copy v1.14.1/go.mod h1:oQwrEDDOci3IM8dJF0d8+jnbfPDllW6vUjNc3DoZm9I=
|
||||
github.com/otiai10/mint v1.6.3 h1:87qsV/aw1F5as1eH1zS/yqHY85ANKVMgkDrf9rcxbQs=
|
||||
@@ -1666,6 +1702,7 @@ golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBc
|
||||
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
|
||||
@@ -146,9 +146,9 @@
|
||||
</div>
|
||||
</div>
|
||||
<div>
|
||||
<p class="lede rv">Every part of this clip came out of LocalAI, and nib drove the machine that made it. The agent opened the app, ran the demo and captured the screen, while a local model wrote the script, a cloned voice read it, and the video endpoint generated and lip-synced the presenter. No human touched the keyboard, and nothing left the building.</p>
|
||||
<p class="lede rv">Every part of this clip came out of LocalAI, and the agent in the binary drove the machine that made it. It opened the app, ran the demo and captured the screen, while a local model wrote the script, a cloned voice read it, and the video endpoint generated and lip-synced the presenter. No human touched the keyboard, and nothing left the building.</p>
|
||||
<div class="recipe rv">
|
||||
<div><b>Direction</b><span>nib, our agent harness, drove the machine end to end</span></div>
|
||||
<div><b>Direction</b><span>The agent in LocalAI drove the machine end to end</span></div>
|
||||
<div><b>Script</b><span>Written by a local language model</span></div>
|
||||
<div><b>Voice</b><span>Cloned from a few seconds of reference audio</span></div>
|
||||
<div><b>Presenter</b><span>Generated and lip-synced through the video endpoint</span></div>
|
||||
@@ -343,18 +343,22 @@
|
||||
<section class="deepbg" id="nib">
|
||||
<div class="shell">
|
||||
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
|
||||
<p class="kicker rv">nib</p>
|
||||
<p class="kicker rv">the agent</p>
|
||||
<div class="duo">
|
||||
<div>
|
||||
<h2 class="rv mt1" style="max-width:16ch">An agent you can drop on any box you SSH into.</h2>
|
||||
<p class="lede rv mt2">One Go binary, about 20 MB, no runtime and no daemon. Press <span class="mono" style="color:var(--cyan-hi)">Ctrl+Space</span> anywhere and it opens. Point it at any OpenAI-compatible endpoint, including a model running on your own laptop, and it works. Tool calls go through an approval gate you control, and Claude Code plugins load as they are.</p>
|
||||
<div class="chips rv"><span>Go</span><span>zero dependencies</span><span>MCP</span><span>plugins</span><span>skills</span><span>sub-agents</span></div>
|
||||
<p class="rv mt2"><a class="btn btn--o" href="https://github.com/mudler/nib">nib on GitHub ↗</a></p>
|
||||
<h2 class="rv mt1" style="max-width:18ch">The agent is already in the binary.</h2>
|
||||
<p class="lede rv mt2">Run <span class="mono" style="color:var(--cyan-hi)">local-ai chat</span> and you are talking to an agent that already knows where your models are. It runs shell commands behind an approval gate you control, delegates to sub-agents, and loads MCP servers, plugins and skills. Claude Code plugins load as they are.</p>
|
||||
<p class="lede rv mt2">It is also nib, a single ~20 MB Go binary with no runtime and no daemon, so you can drop the same agent on any box you SSH into and press <span class="mono" style="color:var(--cyan-hi)">Ctrl+Space</span>.</p>
|
||||
<div class="chips rv"><span>local-ai chat</span><span>MCP</span><span>plugins</span><span>skills</span><span>sub-agents</span><span>zero dependencies</span></div>
|
||||
<div class="acts rv">
|
||||
<a class="btn" href="/docs/features/terminal-agent/">Read the docs <span>→</span></a>
|
||||
<a class="btn btn--o" href="https://github.com/mudler/nib">nib on GitHub ↗</a>
|
||||
</div>
|
||||
</div>
|
||||
<div class="duo__m rv">
|
||||
<figure class="screen" style="margin:0">
|
||||
<figcaption class="screen__bar"><i></i> nib <b>~20 MB, one binary</b></figcaption>
|
||||
<video src="/media/nib.mp4" muted loop playsinline preload="none" data-lazy aria-label="The nib terminal agent running a task"></video>
|
||||
<figcaption class="screen__bar"><i></i> local-ai chat <b>the agent, in the binary</b></figcaption>
|
||||
<video src="/media/nib.mp4" muted loop playsinline preload="none" data-lazy aria-label="The terminal agent built into LocalAI running a task"></video>
|
||||
</figure>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
Reference in New Issue
Block a user