mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
feat(addons): installable add-ons foundation and the Tables add-on (#12702)
* feat(addons): add the add-on runtime, navigation and first-party add-ons Add the add-on runtime to LocalAI: manifests, the host operation broker, admin gating by account role, the add-on UI bundle builder and the SDK for add-on pages. Move the navigation to Chat, Models, Add-ons and Settings, with pinned add-ons in the sidebar and an Add-ons drawer. Ship the first-party add-ons (voice, face, fine-tuning, quantization) as packages for linux/amd64, linux/arm64 and darwin/arm64. Intel macOS (darwin/amd64) is enabled in the platform files. Squashed from the review branch. The tree is unchanged; the pre-squash history is kept on a local backup branch. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gosec): tighten file modes and annotate false positives Write job state, configs and imported models with mode 0600, clean the paths that are read from disk, and mark the route tables, the header name and the retry jitter as reviewed false positives. Assisted-by: Claude Code:claude-sonnet-5-5 gosec Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): keep API keys off recovery mutations and same-site cookies POST /api/addons/dispatch forwarded the recovery and update mutations to the owner for any administrator key, which skipped the browser-session rule of their own routes. Refuse those actions there. Also apply the same-origin rule to the legacy token cookie, not only the session cookie, because the global CSRF check lets same-site requests through. Do not fail the start of LocalAI when an add-on library cannot be refreshed; log it and keep the installed add-ons up. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): refuse a reinstall that would migrate retained data unbacked An upgrade takes a verified backup before the new package migrates the data. A reinstall over retained data has no running generation to stop, so the new package would migrate it without one. Refuse a reinstall whose data format differs from the retained one. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): release the manager lock while a release is fetched Submit held the state lock across the catalog lookup and the package download, so the apps list and status reads stalled for the whole install. Serialize Submit and Reconcile with their own lock and drop the state lock around the fetch only. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): kill an add-on only after three missed health probes One slow 100 ms probe killed a busy add-on and dropped the work it was serving. Require three misses in a row. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): move settings under the operate hub The Operate hub already has Settings as a tab, so the sidebar no longer shows a separate Settings entry. The /app/settings route stays, and the hub tab still reaches it. The navigation test now expects Chat, Models and Add-ons as the primary items. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): accept same-host browser requests that carry no fetch metadata Browsers do not send Sec-Fetch-* headers over plain HTTP on an address that is not localhost, so every add-on request from a LAN address was refused as cross-site. Keep refusing a Sec-Fetch-Site value other than same-origin. When the header is absent, accept the request only if its Origin, or failing that its Referer, names the same host and port as the request. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): show the server's reason when an add-on request is refused A 403 or 400 answer showed a generic message and hid the reason in the response body, so a refusal by the browser guard could not be told from a missing permission. Show error.message from the body as the server sent it, and keep the generic text when the body has none. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): run the add-on owner in the normal binary The owner was only enabled when an embedding host set the add-on config, so a normal run answered every add-on request as unavailable. Add --addons-enabled (default true) and --addons-ownership-mode (default named-user, or local-owner; anything else stops startup). The node is its own owner: state lives under the data path, and the carrier uses loopback keys made at each start and kept in memory. --addons-enabled=false turns the owner off completely. Named-user mode needs the account database. With authentication off the owner stays off and a warning says how to use local-owner. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): keep the loopback owner address stable across restarts The single-node loopback transport took a new ephemeral port at each start. The owner stores its address in its identity file and refuses to start when the address differs, so every start after the first failed with "invalid addon owner configuration". Choose the port once, keep it in the add-on state root (loopback-port, mode 0600) and reuse it. If the port is taken at a later start, stop with an error that names the port and --addons-enabled=false, because another port would break the identity check. The keys and certificate stay in memory. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): let add-on pages keep the host modules they were written against A page written for the web UI imports React, the router, i18next and a few shared components. The add-on builder refuses a bundle that holds a host file, so moving such a page into an add-on meant rewriting it. Add a compat mode to the shared builder. It answers those imports from a directory of stand-ins that read the SDK, bundles the dynamic import of the page code into the one output file, and collects the style sheets the page imports. A relative import that is not in the add-on's own source resolves to the file at the same path under the compat directory. Widen the SDK for these pages, without a version change: more host components (spinner, modal, lightbox, model picker) and hooks (models, resources, voice profiles, operations, MCP client), a style on icons, the icon names the pages draw, sdk.navigation.openApp to open a page of the app (only addresses under /app, and the way back as the only state) and sdk.portal. The shell passes the query string to the page. The stand-ins cover the router (the tab in the path, the model in the query), the translator over the host locale files, the core API calls through sdk.core.fetch, and byte copies of the pure helpers, with tests that keep the copies equal to the originals. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): ship Studio and Talk as add-ons instead of built-in pages Studio and Talk were compiled into the web bundle and always listed in the Add-ons drawer. They are add-ons now, like Voice, Face, fine-tuning and quantization: a person installs and removes them, the drawer lists them only while they are installed, and they are packaged the same way. Move the pages, their components, hooks and helpers into addons/studio and addons/talk. Each has a manifest, a UI bundle, a release platform matrix and a catalog template, and is built by the compat mode of the shared builder. The core API stays where it is: the pages call it with the signed-in person's session. The history keeps its browser storage keys. The unit tests of the moved helpers move with them. In the core, drop the two pages, the Create group of the sidebar (Chat joins Home and Models in the old layout), the built-in catalog entries (the catalog is empty) and the talk locale namespace. The old addresses (/app/studio, /app/talk and the generator links) redirect to the add-on pages, with the model as ?model=. The Home /studio command is offered only while the Studio add-on is installed and opens it. The release workflow and its tests now list seven add-ons. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): run the Studio and Talk specs against the add-on bundles The pages are add-ons now, so a spec that draws one installs the real built bundle first: the apps list names it and its module and style sheet are served from the build output, the way the add-on owner serves them. studio-addon.js holds that helper, and the Studio and Talk fixtures call it. Update the specs for the new addresses (/app/addons/studio/studio, the model as ?model=), the session permission the add-on route needs, the legacy redirects, the Home /studio command that appears only while the add-on is installed, and the sidebar without a Create group. Add specs for the add-ons themselves: every tab draws with no script error, the shell titles the page once, the old addresses say so when the add-on is not installed, a button of the page reaches a core page with the way back, and the drawer lists both only after they are installed. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): centre the icon of a built-in tile in its icon box The drawer drew the icon of a built-in tile (Studio's palette was one) in a bare 40px box. Only the placeholder box was laid out as a centred grid with the 20px size, so the real icon sat in the top-left corner at the size of the page text. Give the built-in tile the same classes as the placeholder box, and check in a browser that the icon is centred. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): describe Studio and Talk as add-ons Replace the built-in list (now empty) with the Studio and Talk add-on sections, the new targets of the old links, the Home command, and the seven packaged add-ons. Describe the SDK additions the pages use: components, hooks, icons, openApp and portal. Say on the Studio page that it is installed from the drawer. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): click the shell title in the Studio slash-key spec The add-on shell titles the page, and the Studio overview no longer draws a second title, but the generator tabs keep theirs, so a bare h1 locator matched two elements. Click the first, which is the shell's. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): read the loopback port file through os.Root The persisted port file was read with a path built from the add-on root, which gosec reports as a file inclusion through a variable. Open it through os.Root so the name cannot leave the root, even by a symbolic link, and require a regular file with no group or other access. A looser file now fails closed instead of being trusted. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): expect Settings under the Operate hub, not the sidebar Settings moved out of the sidebar into the Operate hub. The navigation spec still expected a sidebar link. Assert that the sidebar has no Settings entry, and that the Operate hub opens Settings as a tab. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): drop Settings from the 404 places Settings is a tab of the Operate hub and not a destination of its own, so the 404 page does not list it. Assert that it is absent. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): drive the media history and TTS specs through the Studio add-on The image, video, sound and speech generators are pages of the Studio add-on now, and the old addresses redirect to it. Install the add-on with the Studio fixtures, open its pages and use its compose card, so the specs run against the page that exists. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): count the first-run steps, not every list item The recommended models are a list inside the second step, so counting all list items read 3 only while the models were still loading. Count the step elements, which does not depend on the load order. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): fail the node refresh by state, not by call count The spec served the list to the first request and failed the rest. The first request is not always the one from the page, so the page could get the failure first and show no list. Serve the list until the page shows it, then fail the refreshes. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): default to local-owner when auth is off A server without authentication has no accounts, so named-user left the add-on owner off. Resolve the unset mode in one place: local-owner when authentication is off, named-user when it is on. A flag or environment value still wins, and an unknown value is still refused. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): register an official add-on library by default Add gallery/addons/index.yaml, a YAML index of the four first-party add-ons (fine-tuning, quantization, voice, face) in the library format. The entries come from addonpkg catalog over locally built linux/amd64 packages, so the digests pin those images. On the first start, when the owner has no sources.json, register the raw GitHub URL of that index on master as the "official" library and write the file. The file is then the record: a library the admin removes stays removed, and an existing file is never changed. --addons-official-library=false skips the default. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): document the ownership default and the official library Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): list Studio and Talk in the first-party index Add the Studio and Talk entries to gallery/addons/index.yaml next to the four existing ones. They come from addonpkg catalog over locally built linux/amd64 packages, so the digests pin those images. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): expect Studio and Talk in the official index The official index now lists six add-ons. Update the test that decodes it so it accepts Studio and Talk. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): add a real icon to each first-party add-on Each icon is a 256 by 256 PNG: a rounded square in one colour of the UI kit palette with a white outline glyph from the kit sprite. Fine-tuning uses sliders, quantization layers, voice waveform, face user, studio palette, talk phone and tables database. They were rendered with headless Chrome from the sprite paths. Assisted-by: Claude Code:claude-sonnet-5-5 chrome Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): ship a committed icon.png instead of the generated square The shared UI builder and the Tables builder copy ui/icon.png from the add-on source tree to the package unchanged when the file exists. They write the generated placeholder only when it does not. Add a fixture with an icon and a test for both cases. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): list the tables add-on in the first-party index Add the Tables entry to gallery/addons/index.yaml next to the six existing ones. It comes from addonpkg catalog over a locally built linux/amd64 package, so the digest pins that image. The official index test now expects seven add-ons. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): make the Voice add-on one page with tabs The Voice add-on had four pages: recognition, library, create and speech, and the drawer listed two of them. It now has one page, "voice", and one drawer entry. Recognition, Library, Text to speech and From a recording are tabs of that page, drawn with the same tab strip the recognition page already used. The tab is the first path segment (voice/library), as in the Studio add-on, and the form that adds a voice is a view of the Library tab (voice/library/new). The old page names stay declared in the manifest. Each one redirects to its tab and keeps the query string, so saved links still open. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): send the old voice routes to the Voice tabs /app/voice, /app/voice-library and /app/voice-library/new now open the matching tab of the Voice add-on page while the add-on is installed and running, and the core pages otherwise. The moved-page helper compares the first segment of the target with the declared pages, so a target can name a tab. The docs and the navigation spec list the new addresses. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): cover the Voice tabs and the old page redirects The add-on unit tests render the tabs instead of the four pages. The Playwright spec checks the tab strip, the address of each tab, the missing Library tab for a member, and that the old page addresses of the add-on redirect to their tab with the query string kept. The migration table test follows the new targets. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(studio): keep the model menu inside a narrow screen The Home model picker sizes its menu from the viewport, and assumes the picker sits at the page gutter. In the Studio composer card the field is further in, so on a 390px screen the menu of the Sound expansion picker ended 7px past the right edge. Size the menu from the field it opens under instead. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): build the add-on bundles once for the whole run Each worker built the Studio and Talk bundles into the same dist directory, and the build empties that directory first. A worker that started, or restarted after a failure, removed the files another worker was serving, so a Studio spec saw "the style sheet of the add-on could not be loaded". Build once in a global setup before the workers start. A run without the setup still builds per worker. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): register the worker when the add-node spec says so The mocked roster added the worker on the third read of /api/nodes. The shell and the page both read it, and which one asks first depends on load, so the page could take the new worker as part of the roster it opened with and never show it as found. The worker now joins after the page shows it is listening. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): hold the gallery answer until the skeleton is seen The spec delayed the answer by a fixed 800 ms and then looked for the skeleton. The assertion polls at growing intervals, so on a slow machine it could step over that window. The answer now waits until the skeleton has been seen, and the spec then checks that the skeleton goes away. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): give the PDF tag the time the other PDF specs have pdf.js and its worker load on first use. The other specs in this file wait up to ten seconds for the result of that load. The first one waited five, and missed the tag on a busy machine. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): build the Face and Fine-tuning bundles in the global setup The navigation spec serves the built Fine-tuning bundle, but nothing built it, so two specs failed on a fresh checkout when the dist folder was missing. The spec also built the Face bundle in a hook that every worker runs, and a build rewrites the folder the other workers serve. The global setup now builds Studio, Talk, Face and Fine-tuning once, before the workers start. The spec builds the two bundles itself only when it runs without the global setup. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): fix the Home answers in the stale pin spec The spec asserts that the page shows no alert. The Home page raises its own alert when the model list fails or a staging operation has failed, and both answers come from the shared test server. A slow reply or a failed install left there counted as an alert about the pin. The spec now fixes both answers, so the alert count tells only about the pin. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): read the server address in the add-node spec from the page The spec expected the register address to carry port 8089, so it failed on any other port. It now expects the origin the page was opened from. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): give the navigation routes spec its own catalog No page is built in today, so the catalog is empty and the three specs that expect the studio and talk entries failed on the list and on the pin seed. They passed only while Studio and Talk were built in. The spec now sets a catalog of two ungated entries for each example and restores the old one afterwards. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(cli): report a failed write of the MLX hostfile The tunnel callback ignored the error of os.WriteFile and set the hostfile variable even when the file was not written. It now logs the error and leaves the variable alone. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): discard the close error of the held listener errcheck flags the deferred Close of the listener that keeps the port busy. The test has no use for that error, so the defer drops it explicitly. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the add-on tool tests to Ginkgo The fine-tuning, quantization and add-on host library tests used t.Errorf, t.Fatal and t.Run, which the forbidigo rule rejects. They now use Describe, It and Gomega, with a suite bootstrap in each package. The cases and the assertions are the same. The tables in the host client and the tool schema tests keep one entry per case. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the host operation tests to Ginkgo The hostops tests used the testing package calls that forbidigo rejects. They now run as Ginkgo specs under one suite bootstrap. Every check and every per-operation case is kept. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the pin store tests to Ginkgo The pin service, file store, sweep and shared store contract tests used the testing package calls that forbidigo rejects. The store contract is now a function that registers its specs for each store, so the file store, the memory-only store and the database store run the same cases. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the remaining add-on tests to Ginkgo The built-in catalog, tool arguments, loopback carrier, host operation contract, packaging, application, config, CLI and endpoint tests used the testing package calls that forbidigo rejects. They now run as Ginkgo specs. The endpoint tests also drop the unchecked Close errors that errcheck reports when the auth tag is on. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): wait for the targeted Reinstall button before the install click The "generic deletion blocks targeted install at confirmation" spec dispatches a popstate to set the target node, then clicks Reinstall with a script click. The click can run before the targeted page has rendered or while the button is still disabled, so no install request is sent and the poll on the install count times out. It failed 1 run in 10 with two workers and on every retry in CI. The spec now waits for the button to be enabled first. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(addons): refresh the voice package digest in the first-party index The Voice add-on became one page with tabs and got a new icon, so the digest in gallery/addons/index.yaml no longer matched the build. Render the entry again with addonpkg catalog over a locally built linux/amd64 package; the main page is now voice. The other entries are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): wait for the target node's Reinstall button after the popstate The "generic deletion blocks targeted install at confirmation" spec dispatches a popstate to set the target node, then waits for a button named Reinstall and clicks it. Until React renders the targeted page, the page still draws the generic Reinstall button. It is enabled and has the same name, so the wait passed on it, the click started a global install, and no node install request was sent. The poll on the install count timed out with the confirm dialog still open. With the CPU throttled to 8x, the spec failed on 20 of 20 runs, and the button it clicked had the title "Reinstall". The spec now waits for the button titled "Reinstall on <node>". With the throttle it passes 30 of 30 runs. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): give the Studio hand-off spec a budget for the cold bundle load The "a result made through a hand-off records which result it came from" spec opens the Studio add-on straight from an address, twice. Each is a cold load: the page shows "Loading" until the add-on module has run, and the next assertion had the default five seconds. The load takes about two seconds on an idle run and seven to ten under CPU load, so the spec failed on a loaded machine with the hand-off note not yet drawn. With the CPU throttled to 6x and eight busy loops running, it failed on 21 of 30 runs. The first assertion after each load now waits up to 20 seconds, and the spec then passed 30 of 30 under the same load. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): carry the identity page styles in the Face add-on The Face page is drawn with the shared identity views, but its style sheet held only the clip and error rules. The page, card, model list, known-faces table, enrol sheet and distance scale classes had no rules. They looked right only when the host had already loaded the style sheet of its own Faces page, which a visit through the add-on drawer does not do. Opened that way, the cards had no gap, the model list showed bullets, and the install buttons ran into the names. The Face sheet now holds the same identity rules as the core sheet, so the page looks the same whichever way it is opened. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): scope the narrow-screen identity rules of the Voice add-on The rules for narrow screens (single-column pairs, smaller tool padding, compact rank rows) were copied into the Voice style sheet without their media query, so they applied at every width. They now sit in a max-width 720px query, and the reduced-motion rule in its own query, as in the core sheet. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): drop the repeated title from the Voice and Face pages The add-on shell already titles the page, so the recognition pages showed the name twice: "Voice" above "Voices", "Face" above "Faces". Keep the supporting line and the model picker, as the Studio page does, and remove the unused title strings. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): replace the Pinned sidebar section with an Apps entry The sidebar no longer lists pinned add-ons. It shows one Apps entry that opens the add-ons drawer, with a Show all link below it that opens the same drawer. Pins are used from the drawer, where a pinned tile keeps its pin and move buttons. The server pins API is unchanged. Remove the PinnedNav component and its locale title, rename the items.addons label to Apps, and update the e2e specs and the add-ons documentation to match. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): assert the single shell title on the Voice and Face pages The add-on shell now shows the one title, "Voice" or "Face", and the page no longer repeats it. The tests still looked for the h1 "Voices" and "Faces". Assert that the add-on page has exactly one h1 and that it is the shell title. The core pages keep their own "Voices" and "Faces" titles. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): drop the repeated title from the Voice Library and Text to speech pages The add-on shell already titles the page "Voice", so these two tabs showed a second heading below it ("Speech voices", "Speak text"). Keep the supporting line, as the Studio and recognition pages do, and remove the unused title strings. The e2e spec now checks that each tab has exactly one h1, the shell title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): list pinned apps under Apps in the sidebar A pinned app now shows as an indented child of the Apps entry, with its icon and title, in pin order. Show all stays the last child. The pins come from GET /api/addons/pins through the navigation context, and the apps from the add-on registry, which the drawer feeds on every list read, so the links follow the drawer. A pin without an installed app is not drawn, nothing extra shows when no app is pinned, and the children are hidden when the sidebar is collapsed to the rail. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): say that pinned apps are listed under Apps in the sidebar The sidebar lists the pinned apps between Apps and Show all, in pin order. The pins section and the navigation section said that pins do not appear in the sidebar. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): remove the Show all link under Apps in the sidebar The link repeated the Apps entry, which opens the same drawer, and looked redundant when no app was pinned. Apps is now the only add-on entry; pinned apps still list under it in pin order. The link, its locale key and the style comment are gone, and the e2e spec and the add-ons docs now say that Apps is the only entry. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show a first-run card on Apps when nothing is installed An administrator who opens Apps with no add-on installed now sees a short card with three steps: choose a library, browse and install, start the add-on. Its button opens the Browse tab. The card is hidden once an add-on is installed and for anyone who cannot install. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): hide the pinned apps in the icon rail When the window is narrower than 1024 px the sidebar shrinks to the icon rail by CSS alone, without the collapsed class. The pinned app links were still drawn there, with the indent meant for the full sidebar, so their icons sat to the right of the rail and were cut off. Hide the pinned app links in both rail states. The Apps entry keeps its centred icon. The e2e spec checks that the rail shows no pinned app and that every icon is centred at 1280 px collapsed and at 1000 px. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): stop the backend redirect test racing the installed list The test reads the redirected URL right after the path changes. The Backends page then loads the installed list and drops a selected backend that is not installed, so the backend parameter was sometimes already gone when the test read it. The mock server has no llama-cpp, and about 15 of 40 repeated runs failed on the backend assertion. The redirect itself was correct, including show_all. Serve the selected backend in the installed list, so the page keeps the selection and the assertions stay as strict as before. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): accept a same-host Referer or Origin for human confirmation Browsers send no Sec-Fetch-* headers over plain HTTP on an address that is not localhost, so creating a resource from a LAN address failed with "Creating a resource requires your own browser session". When Sec-Fetch-Site is present it must still be same-origin with an empty destination, as before. When it is absent, Origin or, failing that, Referer must name the request host and port. A destination or mode that is present must still be the expected one. This matches the rule in BrowserProtection. The credential, token cookie, API key and tool-context checks are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): keep hub entries off the Apps route in the sidebar The Build hub still owned /app/addons through its legacy Add-ons tab, so the sidebar lit both Apps and Build on the Apps page. A tab whose gate is closed now owns no route when the sidebar asks, so only the Apps entry is active there. The hubs keep their own routes, and the legacy navigation still gives /app/addons to the Build hub. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): label the pin control on app tiles and in the detail view The pin was a small icon-only button and hard to find. Each app tile now shows a button reading Pin, or Pinned with a filled pin, and the detail view has the same control in its header. Both toggle the same stored pin. The Apps grid lists pinned apps first, in pin order, then the others. The sidebar list and the server pins API are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): check the Voice From a recording tab has one title The Library and Text to speech tabs already assert that the add-on shell title is the only h1. Do the same for the From a recording tab, so a heading added there later fails the navigation spec. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): stop the add-on shell drawing a title over add-on pages The app decides what its page shows. The shell drew the add-on name as a header above every add-on page, so an add-on could not choose its own title. The shell now draws no header on a live, loading or frozen add-on page; each add-on titles its own pages. The unavailable and missing-page states keep the name, because no add-on page is drawn there. The runtime and shell specs now check that the shell adds no page title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): let each add-on draw its own page title The shell no longer titles add-on pages, so each add-on draws one h1 per page. Voice titles its tabs "Voices" (Recognition and From a recording), "Speech voices" (Library) and "Speak text" (Text to speech); "Add a speech voice" already had its own. Face draws "Faces", Studio "Studio" and Talk "Talk". Tables promotes the database list and database name from h2 to h1. Fine-tuning and Quantization already drew a title. The add-on unit tests and the navigation and Studio specs now assert exactly one h1 per page, with the page's own title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): add a kind to library entries and read unknown kinds tolerantly A library entry now has an optional kind: addon (the default when it is absent), backend or model. Every index written before kinds existed stays valid and its entries are add-ons. A reader keeps the entries of the kinds it knows and skips the others without reading their fields, so a newer kind never makes an older host refuse the whole index. The refresh result and the library state report how many entries were skipped and of which kinds, and the refresh logs a warning. A kind that is not a plain kind name still makes the index malformed. Search lists add-ons unless the caller asks for a kind, and Resolve returns add-ons only, so an add-on install cannot be pointed at a backend image. A library policy may name the signer with an anchored identityRegex instead of an exact identity, so builds from master and from tags both verify. An add-on install refuses a policy that carries one. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backends): install a backend from a library entry A library can now list backends. An entry of kind backend names the image of the backend gallery by digest, and POST /backends/library/install installs it on the local node. The gallery stays the source of truth and its install path is the one used: the library resolves the entry, takes the signature policy from the library, pins the image by digest and calls the same install as the gallery. Installing a name that came from a gallery, another library or by hand is refused and changes nothing. The signature policy of a backend library names the signer with an anchored identityRegex, so images signed by the merge workflow on master and on tags both verify. A pattern that is not anchored is refused. An install is bounded by a size limit per kind: add-ons keep 512 MiB and a backend image defaults to 8 GiB, set with --addons-backend-max-bytes. The limit reads the manifest after the signature check and before any layer is fetched. The metadata of an installed backend records the library it came from. scripts/backends/cmd/backendcatalog renders entries from backend/index.yaml and pins them against the registry. The pilot is the CPU build of local-store, in gallery/backends/index.yaml. The library is registered on the first start only with --addons-backend-library, which is off by default. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show the kind of library entries and install backends from them The Browse tab lists the entries of every kind this host reads and marks each one as an add-on or a backend. A backend installs through the backend route and not through the add-on lifecycle: the confirmation says that the backend gallery is not changed and that a backend runs native code, and a refusal of the host, such as a name installed from another origin, is shown with its own reason. The details give the kind, the alias and the signer pattern. The library editor can name the signer with a pattern. Without this a save would drop the pattern of a backend library. The Libraries tab reports the entries a refresh skipped because their kind is not read. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): describe backends in the library Document the entry kinds, how a reader skips a kind it does not know, the backend entry format, the signer identity pattern, the install route, the size limit of each kind, the pilot library and the command that renders it. The backend page links to it. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(mcp): install a backend from a library entry The backend library route installs a backend from an add-on library entry, and the admin REST surface and the MCP catalog must stay in step. Add the install_library_backend tool with an in-process and an HTTP client, the route map entry, the safety-rule listing and a skill recipe. Listing library entries stays on the add-on routes, which the assistant does not expose, so no list tool is added. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): carry the add-on library keys in every locale The add-on library and backend library strings existed only in English, so the other locales had no value for them. Add the missing keys of the addons namespace to de, es, id, it, ko, pt-BR and zh-CN. The values are the English text until a translation is contributed. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(mcp): install a backend from a library entry" This reverts commit7493cff727. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "docs(addons): describe backends in the library" This reverts commit57816ed7e6. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(ui): show the kind of library entries and install backends from them" This reverts commit49b2b151b7. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(backends): install a backend from a library entry" This reverts commit1d8d3e3421. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(addons): add a kind to library entries and read unknown kinds tolerantly" This reverts commit484813f856. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): drop the backend library keys from the locales Revert the part of39f77ae29that added strings for backend libraries: the kind labels, the backend install texts, the signer identity pattern and the skipped entries notice. The add-on library keys stay. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add an add-ons control to the chat header Add-on tools were on or off only through the /addons command, with a toast and no state a person could see. The chat header now has an Add-ons button in the style of the MCP dropdown. It lists each running add-on that ships tools with a switch, and a badge shows how many are on. The chat keeps the ids that are on (addonToolIds) next to the older boolean, and both are saved with the chat. A chat that has only the boolean means every add-on. The request sends addon_tools alone when every add-on is on, and adds addon_tool_ids when only some are. /addons turns all on or all off and uses the same state. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): let a chat request name the add-ons whose tools it uses addon_tools=true still offers the tools of every add-on the caller may use. A new addon_tool_ids metadata key, a comma-separated list of add-on ids, narrows that to some add-ons. The list counts only with the opt-in, and a list that names nothing offers no tool rather than all of them. The executor drops the tools of add-ons that are not listed and refuses a call to one as an unknown tool, so a model cannot reach a tool the person turned off. Both keys are consumed so they do not reach the backend. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): cover the per-add-on tool list and the chat header control Go tests cover the list form (the list filters the tools offered and the calls allowed) and the unchanged boolean form. Unit tests cover the chat add-on rules. Playwright specs cover the header control: listing only add-ons that ship tools, toggling each one, the count, /addons in step with the switches, and the request metadata. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): check each add-on page has one title when not enabled The shell names an add-on only where the add-on page cannot mount (not enabled, failed, no UI, bundle not loaded, no such page). Once the page mounts it draws its own title, including when it says it is not enabled on this server, and the shell draws none. The shell code already works this way; this change states the rule beside the unavailable state and pins it in the specs. The shell spec now expects exactly one title in the unavailable states. The navigation spec checks the Face page without a face model (the add-on title only) and the Face page of a stopped add-on (the shell title only). Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): upgrade add-ons on their own when the setting is on A new auto_upgrade_addons setting (flag LOCALAI_AUTO_UPGRADE_ADDONS, default off) lets the owner upgrade an installed add-on when its library lists a newer version. It follows the backend setting: a startup flag, a runtime setting, and the same schedule as the backend update check (a first pass shortly after start, then every six hours). Each pass fetches the library of every installed add-on, and for each available update builds the same pinned upgrade command an administrator sends and runs it through the same lifecycle path. The channel rules, the signer identity, the digest checks, the SDK compatibility check and the backup before the upgrade all apply unchanged. The only difference is the gate: the owner acts on its own setting, so no administrator is asked. A refused or failed upgrade changes nothing and is logged; a candidate that failed is not retried until the library lists another digest. With the setting off, nothing runs and upgrades stay manual. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add the Upgrade add-ons automatically toggle to Settings The toggle sits beside Upgrade backends automatically and is off by default. It saves the auto_upgrade_addons runtime setting. The English locale gets the label key, and the Settings test fixture lists the new field. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): cover the automatic upgrade setting and document it The setting defaults to off in the config, the runtime settings and the flag, and is read back from the saved settings file. With it on, a real owner upgrades an installed add-on through the manager; with it off, nothing changes, and a change of the setting at run time takes effect on the next pass. The manager specs show the upgrade goes through the same pinned path as the manual one, and that a corrupted package, a failed signature policy, an incompatible release, a missing update checker and a caller who is not an administrator each leave the installed version in place. The new specs are added to the lifecycle case inventory. The add-ons page describes the setting, its schedule and its default. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(addons): name the library entry identifier id The library entry of an add-on used addonId while the shared envelope and the model entries use id. Release now has the field ID, written as id, and every writer emits it: the index encoder, the package workflow templates, the catalog generator and gallery/addons/index.yaml. The digests in the index did not change. DecodeIndex still reads addonId as a legacy alias for one release, so an existing library and a library served by an older tool keep working. An entry that holds both names is refused. The Apps UI reads id and falls back to addonId. Assisted-by: Claude Code:claude-sonnet-5-5 go node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): use the id field in the auto-upgrade test release The auto-upgrade test arrived after the identifier rename and still named the release field AddonID, so the application tests did not build. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): annotate the auto-upgrade actor label for gosec The actor label looks like a credential to gosec G101. It is a fixed name for upgrades the owner starts itself. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(schema): persist the per-chat add-on tool fields The chat header persists addonTools and addonToolIds, but Conversation had no fields for them, so the server dropped them on save. Add both and update the generated swagger files by hand because swag is not installed here. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): count the add-ons auto-upgrade setting in the settings groups The setting sits in the Backends and galleries group, once. The page now lists 54 settings plus the three branding images, so the spec expects 57. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): pause the fake clock ahead of now in the clear-all spec The fake clock runs from install, so pausing at the current time failed with 'Cannot fast-forward to the past' on a slow runner. The save itself is synchronous and was not lost. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): hold the Studio capabilities reply longer in the skeleton spec 800 ms was shorter than a slow runner needs to render the page, so the skeleton was gone before the check. The product shows it. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(i18n): add the French add-ons locale Master added the fr locale after this branch forked. The add-ons unit test requires every locale to carry the add-ons namespace, so fr needs its own addons.json. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
mudler-agentandEttore Di Giacinto authored and GitHub committed
1 parent
099d6c3dd0
commit
e45dc5e9f7
964 files changed
+159560
-515
No files matched your search
@@ -0,0 +1,244 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
// This executable is a real protocol fixture, not a model backend.
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/subtle"
|
||||
"encoding/json"
|
||||
addonruntime "github.com/mudler/LocalAI/core/services/addons/runtime"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if len(os.Args) != 2 || os.Args[1] != "--localai-addon-v1" {
|
||||
os.Exit(2)
|
||||
}
|
||||
parent := os.NewFile(4, "startup")
|
||||
d := json.NewDecoder(parent)
|
||||
var startup addonruntime.Startup
|
||||
if d.Decode(&startup) != nil {
|
||||
os.Exit(3)
|
||||
}
|
||||
var config struct {
|
||||
Mode string `json:"mode"`
|
||||
// Effects names a file that makes the add-on's data durable: one line
|
||||
// per committed operation, and "<Effects>.applies" with one line per
|
||||
// apply request that reached the process, duplicates included. Empty
|
||||
// keeps the receipts in memory only.
|
||||
Effects string `json:"effects"`
|
||||
}
|
||||
b, _ := os.ReadFile("fixture.json")
|
||||
_ = json.Unmarshal(b, &config)
|
||||
// The "orphan" mode ignores the closing of the startup channel, as a
|
||||
// misbehaving add-on would: only the owner's ledger can then stop it.
|
||||
if config.Mode != "orphan" {
|
||||
go func() { _, _ = io.Copy(io.Discard, io.MultiReader(d.Buffered(), parent)); os.Exit(0) }()
|
||||
}
|
||||
_ = os.WriteFile("fixture.pid", []byte(strconv.Itoa(os.Getpid())), 0600)
|
||||
// The data directory the owner passed is recorded as received, so a test
|
||||
// can assert what the startup object carried. Writing a probe file there
|
||||
// shows the add-on can use it.
|
||||
_ = os.WriteFile("fixture.datadir", []byte(startup.DataDir), 0600)
|
||||
if startup.DataDir != "" {
|
||||
_ = os.WriteFile(filepath.Join(startup.DataDir, "fixture.probe"), []byte("ok"), 0600)
|
||||
}
|
||||
// The host operation endpoint and credential, as received, so a test can
|
||||
// assert that an add-on the host does not name gets neither.
|
||||
_ = os.WriteFile("fixture.hostops", []byte(startup.HostEndpoint+"\n"+startup.HostCredential), 0600)
|
||||
if config.Mode == "exit-start" {
|
||||
os.Exit(4)
|
||||
}
|
||||
if config.Mode == "env-check" && (os.Getenv("LOCALAI_RUNTIME_TEST_SECRET") != "" || os.Getenv("HTTP_PROXY") != "" || len(os.Environ()) != 2) { //nolint:forbidigo // the fixture checks the exact environment the host gives a child
|
||||
os.Exit(5)
|
||||
}
|
||||
listener, e := net.FileListener(os.NewFile(3, "listener"))
|
||||
if e != nil {
|
||||
os.Exit(6)
|
||||
}
|
||||
var mu sync.Mutex
|
||||
receipts := map[string]contract.Receipt{}
|
||||
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if subtle.ConstantTimeCompare([]byte(r.Header.Get("Authorization")), []byte("Bearer "+startup.Credential)) != 1 || r.Header.Get("X-LocalAI-Incarnation") != startup.Registration.Incarnation {
|
||||
http.Error(w, "denied", 401)
|
||||
return
|
||||
}
|
||||
if r.Method != "POST" || r.Header.Get("Content-Type") != "application/json" {
|
||||
http.Error(w, "invalid request", 400)
|
||||
return
|
||||
}
|
||||
r.Body = http.MaxBytesReader(w, r.Body, startup.Limits.EnvelopeBytes)
|
||||
defer func() { _ = r.Body.Close() }()
|
||||
decoder := json.NewDecoder(r.Body)
|
||||
decoder.DisallowUnknownFields()
|
||||
read := func(v any) bool {
|
||||
if decoder.Decode(v) != nil || decoder.Decode(new(any)) != io.EOF {
|
||||
http.Error(w, "invalid request", 400)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
valid := func(v contract.Envelope, scope string) bool {
|
||||
if v.Validate(time.Now(), startup.Limits) != nil || v.Authority.Scope != scope || v.Request.AddonID != startup.Registration.AddonID || v.Request.Generation != startup.Registration.Generation {
|
||||
http.Error(w, "invalid authority", 403)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch r.URL.Path {
|
||||
case "/health":
|
||||
if config.Mode == "never-ready" {
|
||||
http.Error(w, "starting", 503)
|
||||
return
|
||||
}
|
||||
reg := startup.Registration
|
||||
switch config.Mode {
|
||||
case "forged":
|
||||
reg.Capabilities.Tools = []string{"forged"}
|
||||
case "nonloopback":
|
||||
reg.Endpoint = "http://192.0.2.1:80"
|
||||
case "stale":
|
||||
reg.Incarnation = "stale"
|
||||
case "wrong-version":
|
||||
reg.Protocol = 2
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(reg)
|
||||
case "/read", "/preview":
|
||||
var v contract.Envelope
|
||||
if !read(&v) || !valid(v, r.URL.Path[1:]) {
|
||||
return
|
||||
}
|
||||
if v.Request.Action == "delay" {
|
||||
select {
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
case <-time.After(time.Second):
|
||||
}
|
||||
}
|
||||
if r.URL.Path == "/preview" {
|
||||
_ = json.NewEncoder(w).Encode(contract.Preview{OperationID: v.Request.OperationID, Revision: contract.Revision{Schema: 1, Data: 1}, ExpiresAt: time.Now().Add(time.Minute), Summary: json.RawMessage(`{"rows":1}`)})
|
||||
return
|
||||
}
|
||||
payload, _ := json.Marshal(map[string]any{"actor": v.Authority.Actor.UserID, "arguments": v.Request.Arguments})
|
||||
result := contract.Result{Payload: payload}
|
||||
// The tool "export" answers with a file result, and the owner
|
||||
// turns it into a private artifact.
|
||||
var tool struct {
|
||||
Tool string `json:"tool"`
|
||||
}
|
||||
if json.Unmarshal(v.Request.Arguments, &tool) == nil && tool.Tool == "export" {
|
||||
result.Outputs = []contract.OutputDescription{{Handle: "report", Name: "report.txt", MediaType: "text/plain", Size: int64(len("fixture\n"))}}
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(result)
|
||||
case "/apply":
|
||||
var v addonruntime.ApplyRequest
|
||||
if !read(&v) || !valid(v.Envelope, "apply") {
|
||||
return
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
id := v.Envelope.Request.OperationID
|
||||
noteLine(config.Effects+".applies", id)
|
||||
receipt, ok := lookup(config.Effects, receipts, id)
|
||||
fresh := !ok
|
||||
if fresh {
|
||||
receipt = contract.Receipt{OperationID: id, State: contract.ReceiptCommitted, Revision: contract.Revision{Schema: v.Revision.Schema, Data: v.Revision.Data + 1}}
|
||||
receipts[id] = receipt
|
||||
noteLine(config.Effects, id)
|
||||
}
|
||||
// "lose-reply": the change is committed and the connection is
|
||||
// dropped before the reply, as a crash or a network cut would.
|
||||
if fresh && config.Mode == "lose-reply" {
|
||||
if conn, _, e := w.(http.Hijacker).Hijack(); e == nil {
|
||||
_ = conn.Close()
|
||||
return
|
||||
}
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(receipt)
|
||||
case "/receipt":
|
||||
var v addonruntime.ReceiptRequest
|
||||
if !read(&v) {
|
||||
return
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
receipt, ok := lookup(config.Effects, receipts, v.OperationID)
|
||||
if !ok {
|
||||
receipt = contract.Receipt{OperationID: v.OperationID, State: contract.ReceiptUnknown}
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(receipt)
|
||||
case "/output":
|
||||
var v addonruntime.OutputRequest
|
||||
if !read(&v) || !valid(v.Envelope, "output") {
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/octet-stream")
|
||||
w.WriteHeader(200)
|
||||
if v.Handle == "slow" {
|
||||
w.(http.Flusher).Flush()
|
||||
<-r.Context().Done()
|
||||
return
|
||||
}
|
||||
if v.Handle == "oversized" {
|
||||
for n := int64(0); n <= startup.Limits.ArtifactBytes; n++ {
|
||||
if _, e := w.Write([]byte{'x'}); e != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
_, _ = io.WriteString(w, "fixture\n")
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
})
|
||||
if config.Mode == "crash" {
|
||||
go func() { time.Sleep(100 * time.Millisecond); os.Exit(7) }()
|
||||
}
|
||||
server := &http.Server{Handler: handler, ReadHeaderTimeout: time.Second, ReadTimeout: 2 * time.Second, IdleTimeout: time.Second, MaxHeaderBytes: 16 << 10}
|
||||
_ = server.Serve(listener)
|
||||
}
|
||||
|
||||
// noteLine appends one line to a durable log. An empty path keeps nothing.
|
||||
func noteLine(path, line string) {
|
||||
if path == "" || path == ".applies" {
|
||||
return
|
||||
}
|
||||
f, e := os.OpenFile(path, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0600) // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
if e != nil {
|
||||
return
|
||||
}
|
||||
_, _ = f.WriteString(line + "\n")
|
||||
_ = f.Sync()
|
||||
_ = f.Close()
|
||||
}
|
||||
|
||||
// lookup finds the receipt of an operation, in memory and then in the durable
|
||||
// log, so that a restarted process still knows what it committed.
|
||||
func lookup(path string, memory map[string]contract.Receipt, id string) (contract.Receipt, bool) {
|
||||
if r, ok := memory[id]; ok {
|
||||
return r, true
|
||||
}
|
||||
if path == "" {
|
||||
return contract.Receipt{}, false
|
||||
}
|
||||
b, e := os.ReadFile(path) // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
if e != nil {
|
||||
return contract.Receipt{}, false
|
||||
}
|
||||
for _, line := range strings.Split(string(b), "\n") {
|
||||
if line == id {
|
||||
return contract.Receipt{OperationID: id, State: contract.ReceiptCommitted, Revision: contract.Revision{Schema: 1, Data: 1}}, true
|
||||
}
|
||||
}
|
||||
return contract.Receipt{}, false
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"schemaVersion":1,"id":"bare","version":"1.0.0","executable":"bin/fixture","os":"linux","arch":"amd64","host":{"min":1,"max":2},"api":{"min":1,"max":1},"protocol":{"min":1,"max":1},"dataFormat":{"min":1,"max":2},"launchMode":"always","capabilities":{"service":true,"tools":["ping"],"ui":{"entry":"ui.js","sdk":{"min":1,"max":2},"pages":["home"],"mainPage":"home","title":"Bare import","description":"Imports React by bare specifier, which the host never resolves."}}}
|
||||
@@ -0,0 +1,7 @@
|
||||
// What a bundler-less build that did not follow the SDK contract looks like:
|
||||
// a bare import of React. The host has no import map, so the module fails to
|
||||
// load and no second React is ever resolved for it.
|
||||
import React from 'react'
|
||||
export function createAddon() {
|
||||
return { Component: () => React.createElement('p', null, 'unreachable'), dispose() {} }
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"schemaVersion":1,"id":"cssb","version":"1.0.0","executable":"bin/fixture","os":"linux","arch":"amd64","host":{"min":1,"max":2},"api":{"min":1,"max":1},"protocol":{"min":1,"max":1},"dataFormat":{"min":1,"max":2},"launchMode":"always","capabilities":{"service":true,"tools":["ping"],"ui":{"entry":"ui.js","css":["style.css"],"sdk":{"min":1,"max":2},"pages":["home"],"navigation":["home"],"mainPage":"home","title":"CSS B","description":"A second add-on with the same class names."}}}
|
||||
@@ -0,0 +1,2 @@
|
||||
.title { color: rgb(0, 0, 255); }
|
||||
body { margin: 40px; }
|
||||
@@ -0,0 +1,8 @@
|
||||
// A second add-on whose style sheet uses the same class names as the first.
|
||||
export function createAddon(sdk) {
|
||||
const h = sdk.React.createElement
|
||||
return {
|
||||
Component: () => h('section', null, h('h2', { className: 'title', 'data-testid': 'cssb-title' }, 'Second add-on')),
|
||||
dispose() {},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"schemaVersion":1,"id":"slots","version":"1.0.0","executable":"bin/fixture","os":"linux","arch":"amd64","host":{"min":1,"max":2},"api":{"min":1,"max":1},"protocol":{"min":1,"max":1},"dataFormat":{"min":1,"max":2},"launchMode":"always","capabilities":{"service":true,"tools":["ping","summarize"],"ui":{"entry":"ui.js","css":["ui/style.css"],"sdk":{"min":1,"max":2},"pages":["home"],"mainPage":"home","title":{"en":"Slots fixture","it":"Fixture degli slot"},"description":"Contributes to every host slot, with a failing contribution and a style collision.","chatActions":[{"id":"summarize","label":{"en":"Summarize chat","it":"Riassumi la chat"},"icon":"sparkles","placement":"composer","target":{"tool":"summarize"}},{"id":"approve","label":"Review a request","icon":"check-circle","placement":"composer","target":{"component":"ApprovePanel"}},{"id":"inspect","label":"Inspect message","icon":"eye","placement":"message","target":{"component":"InspectPanel"}},{"id":"explode","label":"Break on purpose","icon":"warning","placement":"message","target":{"component":"ExplodePanel"}}],"slots":[{"id":"composer-chip","slot":"chat.composer.actions","component":"ComposerChip"},{"id":"broken-chip","slot":"chat.composer.actions","component":"BrokenChip"},{"id":"message-chip","slot":"chat.message.actions","component":"MessageChip"},{"id":"nav-chip","slot":"app.nav","component":"NavChip"},{"id":"settings-card","slot":"settings.sections","component":"SettingsCard"},{"id":"broken-section","slot":"settings.sections","component":"BrokenSection"}],"renderers":[{"kind":"approval-card","tool":"ping","component":"PingCard"},{"kind":"tool-result","tool":"summarize","component":"SummaryResult"}]}}}
|
||||
@@ -0,0 +1,67 @@
|
||||
// Slots fixture, version 1. Hand-written in the compiled form an add-on build
|
||||
// produces: no bare `react` import. It contributes to every host slot, with a
|
||||
// contribution that throws and a style sheet that collides with host classes.
|
||||
const state = (window.__slotsFixture = window.__slotsFixture || { events: [] })
|
||||
const log = (name) => { state.events.push(name) }
|
||||
|
||||
export function createAddon(sdk) {
|
||||
const h = sdk.React.createElement
|
||||
const { Button } = sdk.ui
|
||||
const generation = sdk.generation
|
||||
log('create:' + generation)
|
||||
sdk.signal.addEventListener('abort', () => log('abort:' + generation))
|
||||
|
||||
const props = (testid) => (p) => h('span', { className: 'fx-chip', 'data-testid': testid, 'data-props': JSON.stringify(p) }, testid)
|
||||
|
||||
function NavChip() {
|
||||
const href = sdk.navigation.href('home')
|
||||
return h('a', {
|
||||
className: 'fx-chip', href, 'data-testid': 'fx-nav',
|
||||
onClick: (e) => { e.preventDefault(); sdk.navigation.navigate('home') },
|
||||
}, 'Slots home')
|
||||
}
|
||||
function SettingsCard() {
|
||||
return h('div', { className: 'fx-card', 'data-testid': 'fx-settings' },
|
||||
h('h2', null, 'Fixture setting'),
|
||||
h('p', null, 'Settings contributed by the slots fixture.'),
|
||||
h(Button, null, 'Fixture button'))
|
||||
}
|
||||
function BrokenChip() { throw new Error('fixture composer contribution failure') }
|
||||
function BrokenSection() { throw new Error('fixture settings contribution failure') }
|
||||
function ExplodePanel() { throw new Error('fixture dialog failure') }
|
||||
function InspectPanel(p) {
|
||||
return h('div', { 'data-testid': 'fx-inspect', 'data-props': JSON.stringify({ conversationId: p.conversationId, messageId: p.messageId, role: p.role }) },
|
||||
h('p', null, 'Inspecting message ' + p.messageId),
|
||||
h(Button, { onClick: p.close }, 'Done'))
|
||||
}
|
||||
function ApprovePanel(p) {
|
||||
return h('div', { 'data-testid': 'fx-approve' },
|
||||
h('p', null, 'This add-on asks the host to show a request.'),
|
||||
h(Button, { onClick: () => { p.close(); sdk.approvals.present('op-fixture-1') } }, 'Show the request'))
|
||||
}
|
||||
function PingCard(p) {
|
||||
return h('div', { className: 'fx-card', 'data-testid': 'fx-card' },
|
||||
h('p', null, 'Card for ' + p.tool + ': ' + p.effects.count + ' changes'),
|
||||
h(Button, { onClick: () => log('card-clicked') }, 'Card button'))
|
||||
}
|
||||
function SummaryResult(p) {
|
||||
const ctx = p.result && p.result.arguments && p.result.arguments.context
|
||||
return h('div', { className: 'fx-card', 'data-testid': 'fx-result-card' },
|
||||
h('p', null, 'Summary of ' + (ctx && ctx.conversationId)),
|
||||
h('p', null, 'tool ' + p.tool))
|
||||
}
|
||||
function Home() { return h('section', null, h('h2', null, 'Slots home')) }
|
||||
|
||||
return {
|
||||
Component: Home,
|
||||
components: {
|
||||
ComposerChip: props('fx-composer-chip'),
|
||||
MessageChip: props('fx-message-chip'),
|
||||
NavChip, SettingsCard, BrokenChip, BrokenSection, ExplodePanel, InspectPanel, ApprovePanel, PingCard, SummaryResult,
|
||||
// Not in the manifest: the host must never draw these.
|
||||
Rogue: () => h('span', { 'data-testid': 'fx-rogue' }, 'ROGUE'),
|
||||
},
|
||||
registerSlot() { log('register-slot-called') },
|
||||
dispose() { log('dispose:' + generation) },
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
/* Selectors that collide with host classes on purpose. The host scopes this
|
||||
sheet to the add-on root, so none of them may reach a host node. */
|
||||
.cx-acts { display: none !important; }
|
||||
.home-attach-btn { background: red !important; width: 1px !important; }
|
||||
.dk-btn { background: red !important; color: yellow !important; }
|
||||
.page-title, h1, h2 { color: red !important; font-size: 48px !important; }
|
||||
.hub-subnav__link { display: none !important; }
|
||||
body { background: red !important; }
|
||||
.fx-chip { color: rgb(1, 2, 3); font-weight: 700; }
|
||||
.fx-card { border: 2px solid rgb(4, 5, 6); padding: 4px; }
|
||||
@@ -0,0 +1 @@
|
||||
{"schemaVersion":1,"id":"uifixture","version":"2.0.0","executable":"bin/fixture","os":"linux","arch":"amd64","host":{"min":1,"max":2},"api":{"min":1,"max":1},"protocol":{"min":1,"max":1},"dataFormat":{"min":1,"max":2},"launchMode":"always","capabilities":{"service":true,"tools":["ping"],"ui":{"entry":"ui.js","css":["ui/style.css"],"sdk":{"min":1,"max":2},"pages":["home","boom","dialog"],"navigation":["home","dialog"],"chatActions":["summarize"],"mainPage":"home","title":{"en":"UI fixture","it":"Fixture interfaccia"},"description":"Exercises the add-on UI runtime."}}}
|
||||
@@ -0,0 +1,11 @@
|
||||
// UI runtime fixture, version 2: the package an update installs.
|
||||
const events = (window.__uiFixture = window.__uiFixture || { events: [] }).events
|
||||
export function createAddon(sdk) {
|
||||
const h = sdk.React.createElement
|
||||
events.push('create-v2:' + sdk.generation)
|
||||
return {
|
||||
Component: () => h('section', null, h('h2', { 'data-testid': 'fx-title' }, 'Fixture version 2'), h('p', { 'data-testid': 'fx-generation' }, 'generation ' + sdk.generation + ' version 2')),
|
||||
chatActions: [],
|
||||
dispose() { events.push('dispose-v2:' + sdk.generation) },
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
@import url("https://evil.example/theme.css");
|
||||
/* Rules that would restyle the host if they were not scoped. */
|
||||
body { background: rgb(255, 0, 255); }
|
||||
h1 { color: rgb(255, 0, 0); font-size: 11px; }
|
||||
.page-title { color: rgb(255, 0, 0); }
|
||||
.dk-btn { outline: 5px solid rgb(255, 0, 0); }
|
||||
:root { --fx-token: 1; }
|
||||
.title { color: rgb(1, 2, 3); }
|
||||
.fx-home { display: block; background-image: url(https://evil.example/pixel.gif); }
|
||||
@@ -0,0 +1 @@
|
||||
{"schemaVersion":1,"id":"uifixture","version":"1.0.0","executable":"bin/fixture","os":"linux","arch":"amd64","host":{"min":1,"max":2},"api":{"min":1,"max":1},"protocol":{"min":1,"max":1},"dataFormat":{"min":1,"max":2},"launchMode":"always","capabilities":{"service":true,"tools":["ping"],"ui":{"entry":"ui.js","css":["ui/style.css"],"sdk":{"min":1,"max":2},"pages":["home","boom","dialog"],"navigation":["home","dialog"],"chatActions":["summarize"],"mainPage":"home","title":{"en":"UI fixture","it":"Fixture interfaccia"},"description":"Exercises the add-on UI runtime."}}}
|
||||
@@ -0,0 +1,95 @@
|
||||
// UI runtime fixture, version 1. Hand-written in the compiled form an add-on
|
||||
// build produces: classic JSX turned into factory-scoped React.createElement
|
||||
// calls over the React the host SDK hands in. There is no bare `react` import
|
||||
// and no bundler at test time.
|
||||
const events = (window.__uiFixture = window.__uiFixture || { events: [] }).events
|
||||
const log = (name) => { events.push(name) }
|
||||
|
||||
export function createAddon(sdk) {
|
||||
const React = sdk.React
|
||||
const h = React.createElement
|
||||
const { Button, Stack, Text, Badge } = sdk.ui
|
||||
const generation = sdk.generation
|
||||
log('create:' + generation)
|
||||
sdk.signal.addEventListener('abort', () => log('abort:' + generation))
|
||||
sdk.onDispose(() => log('sdk-dispose:' + generation))
|
||||
sdk.locale.define('en', { greeting: 'Hello {{name}}' })
|
||||
sdk.locale.define('it', { greeting: 'Ciao {{name}}' })
|
||||
|
||||
function Home(props) {
|
||||
const [count, setCount] = React.useState(0)
|
||||
const [draft, setDraft] = React.useState('')
|
||||
const [result, setResult] = React.useState('idle')
|
||||
React.useEffect(() => { log('home-mounted:' + generation); return () => log('home-unmounted:' + generation) }, [])
|
||||
const onInput = (event) => {
|
||||
setDraft(event.target.value)
|
||||
sdk.markDirty(event.target.value !== '')
|
||||
}
|
||||
const read = async () => {
|
||||
setResult('pending')
|
||||
try {
|
||||
const out = await sdk.api.read({ arguments: { q: draft } })
|
||||
setResult('ok:' + JSON.stringify(out))
|
||||
} catch (error) {
|
||||
setResult('error:' + (error && (error.kind || error.name)) + ':' + (error && error.status))
|
||||
}
|
||||
}
|
||||
return h('section', { className: 'fx-home' },
|
||||
h('h2', { className: 'title', 'data-testid': 'fx-title' }, sdk.locale.t('greeting', 'Hello {{name}}', { name: 'fixture' })),
|
||||
h('p', { 'data-testid': 'fx-where' }, 'page=' + props.page + ' path=' + props.path),
|
||||
h('p', { 'data-testid': 'fx-generation' }, 'generation ' + generation + ' version 1'),
|
||||
h(Stack, null,
|
||||
h(Badge, { tone: 'healthy' }, 'hooks ' + count),
|
||||
h(Button, { onClick: () => setCount((n) => n + 1) }, 'Increment'),
|
||||
h('label', null, 'Draft ', h('input', { 'data-testid': 'fx-draft', value: draft, onChange: onInput })),
|
||||
h(Button, { onClick: read }, 'Read'),
|
||||
h(Button, { onClick: () => { sdk.markDirty(false); setDraft('') } }, 'Save draft'),
|
||||
h(Button, { onClick: () => sdk.navigation.navigate('dialog/opened?from=home') }, 'Go to dialog'),
|
||||
h(Button, { onClick: () => { try { sdk.navigation.navigate('/app/settings') } catch (e) { setResult('refused:' + e.code) } } }, 'Leave'),
|
||||
h('p', { 'data-testid': 'fx-result' }, result),
|
||||
h(sdk.Icon, { name: 'table', title: 'table' })));
|
||||
}
|
||||
|
||||
function Boom() {
|
||||
throw new Error('fixture render failure')
|
||||
}
|
||||
|
||||
function Dialog(props) {
|
||||
return h('section', null,
|
||||
h('h2', { className: 'title' }, 'Dialog page'),
|
||||
h('p', { 'data-testid': 'fx-where' }, 'page=' + props.page + ' path=' + props.path),
|
||||
h(Button, {
|
||||
onClick: () => sdk.layers.openDialog({
|
||||
title: 'Fixture dialog',
|
||||
render: ({ close }) => h('div', null, h('p', null, 'Opened by the add-on.'), h(Button, { onClick: close }, 'Done')),
|
||||
}),
|
||||
}, 'Open dialog'),
|
||||
h(Button, {
|
||||
onClick: () => sdk.layers.openDialog({
|
||||
title: 'Broken dialog',
|
||||
render: () => { throw new Error('fixture layer failure') },
|
||||
}),
|
||||
}, 'Open broken dialog'),
|
||||
h(Button, {
|
||||
onClick: () => sdk.layers.openPanel({
|
||||
title: 'Broken panel',
|
||||
render: () => { throw new Error('fixture panel failure') },
|
||||
}),
|
||||
}, 'Open broken panel'))
|
||||
}
|
||||
|
||||
function Root(props) {
|
||||
if (props.page === 'boom') return h(Boom)
|
||||
if (props.page === 'dialog') return h(Dialog, props)
|
||||
return h(Home, props)
|
||||
}
|
||||
|
||||
return {
|
||||
Component: Root,
|
||||
chatActions: [{
|
||||
id: 'summarize', label: 'Summarize conversation', icon: 'sparkles',
|
||||
open: (context) => { log('chat-action:' + JSON.stringify(context)) },
|
||||
}],
|
||||
dispose() { log('dispose:' + generation) },
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
@import url("https://evil.example/theme.css");
|
||||
/* Rules that would restyle the host if they were not scoped. */
|
||||
body { background: rgb(255, 0, 255); }
|
||||
h1 { color: rgb(255, 0, 0); font-size: 11px; }
|
||||
.page-title { color: rgb(255, 0, 0); }
|
||||
.dk-btn { outline: 5px solid rgb(255, 0, 0); }
|
||||
:root { --fx-token: 1; }
|
||||
.title { color: rgb(1, 2, 3); }
|
||||
.fx-home { display: block; background-image: url(https://evil.example/pixel.gif); }
|
||||
@@ -0,0 +1,55 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"id": "sample",
|
||||
"version": "1.2.3",
|
||||
"executable": "bin/sample",
|
||||
"os": "linux",
|
||||
"arch": "amd64",
|
||||
"host": {
|
||||
"min": 1,
|
||||
"max": 2
|
||||
},
|
||||
"api": {
|
||||
"min": 1,
|
||||
"max": 1
|
||||
},
|
||||
"protocol": {
|
||||
"min": 1,
|
||||
"max": 1
|
||||
},
|
||||
"dataFormat": {
|
||||
"min": 1,
|
||||
"max": 2
|
||||
},
|
||||
"launchMode": "always",
|
||||
"capabilities": {
|
||||
"service": true,
|
||||
"tools": [
|
||||
"query"
|
||||
],
|
||||
"ui": {
|
||||
"entry": "ui/main.js",
|
||||
"css": [
|
||||
"ui/main.css"
|
||||
],
|
||||
"sdk": {
|
||||
"min": 1,
|
||||
"max": 1
|
||||
},
|
||||
"pages": [
|
||||
"home",
|
||||
"settings"
|
||||
],
|
||||
"navigation": [
|
||||
"home"
|
||||
],
|
||||
"mainPage": "home",
|
||||
"icon": "ui/icon.png",
|
||||
"title": {
|
||||
"en": "Sample",
|
||||
"it": "Esempio"
|
||||
},
|
||||
"description": "A sample addon."
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"id": "sample",
|
||||
"version": "1.2.3",
|
||||
"executable": "bin/sample",
|
||||
"os": "linux",
|
||||
"arch": "amd64",
|
||||
"host": {
|
||||
"min": 1,
|
||||
"max": 2
|
||||
},
|
||||
"api": {
|
||||
"min": 1,
|
||||
"max": 1
|
||||
},
|
||||
"protocol": {
|
||||
"min": 1,
|
||||
"max": 1
|
||||
},
|
||||
"dataFormat": {
|
||||
"min": 1,
|
||||
"max": 2
|
||||
},
|
||||
"capabilities": {
|
||||
"ui": {
|
||||
"entry": "ui/main.js",
|
||||
"sdk": {
|
||||
"min": 1,
|
||||
"max": 1
|
||||
},
|
||||
"chatActions": [
|
||||
"open"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"request":{"addonId":"sample","resourceId":"resource-1","operationId":"operation-1","generation":1,"action":"query","arguments":{"limit":1}},"authority":{"actor":{"userId":"user-1","credentialId":"key-1","mode":"named-user"},"scope":"read","expiresAt":"2030-01-01T00:00:00Z"}}
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"id": "sample",
|
||||
"version": "1.2.3",
|
||||
"executable": "bin/sample",
|
||||
"os": "linux",
|
||||
"arch": "amd64",
|
||||
"host": {
|
||||
"min": 1,
|
||||
"max": 2
|
||||
},
|
||||
"api": {
|
||||
"min": 1,
|
||||
"max": 1
|
||||
},
|
||||
"protocol": {
|
||||
"min": 1,
|
||||
"max": 1
|
||||
},
|
||||
"dataFormat": {
|
||||
"min": 1,
|
||||
"max": 2
|
||||
},
|
||||
"capabilities": {
|
||||
"service": true,
|
||||
"tools": [
|
||||
"query"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
[
|
||||
{
|
||||
"host": 3,
|
||||
"api": 1,
|
||||
"protocol": 1,
|
||||
"sdk": 1,
|
||||
"dataFormat": 1,
|
||||
"os": "linux",
|
||||
"arch": "amd64"
|
||||
},
|
||||
{
|
||||
"host": 1,
|
||||
"api": 3,
|
||||
"protocol": 1,
|
||||
"sdk": 1,
|
||||
"dataFormat": 1,
|
||||
"os": "linux",
|
||||
"arch": "amd64"
|
||||
},
|
||||
{
|
||||
"host": 1,
|
||||
"api": 1,
|
||||
"protocol": 3,
|
||||
"sdk": 1,
|
||||
"dataFormat": 1,
|
||||
"os": "linux",
|
||||
"arch": "amd64"
|
||||
},
|
||||
{
|
||||
"host": 1,
|
||||
"api": 1,
|
||||
"protocol": 1,
|
||||
"sdk": 3,
|
||||
"dataFormat": 1,
|
||||
"os": "linux",
|
||||
"arch": "amd64"
|
||||
},
|
||||
{
|
||||
"host": 1,
|
||||
"api": 1,
|
||||
"protocol": 1,
|
||||
"sdk": 1,
|
||||
"dataFormat": 3,
|
||||
"os": "linux",
|
||||
"arch": "amd64"
|
||||
},
|
||||
{
|
||||
"host": 1,
|
||||
"api": 1,
|
||||
"protocol": 1,
|
||||
"sdk": 1,
|
||||
"dataFormat": 1,
|
||||
"os": "unsupported",
|
||||
"arch": "amd64"
|
||||
},
|
||||
{
|
||||
"host": 1,
|
||||
"api": 1,
|
||||
"protocol": 1,
|
||||
"sdk": 1,
|
||||
"dataFormat": 1,
|
||||
"os": "linux",
|
||||
"arch": "unsupported"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"id": "sample",
|
||||
"version": "1.2.3",
|
||||
"executable": "bin/sample",
|
||||
"os": "linux",
|
||||
"arch": "amd64",
|
||||
"host": {
|
||||
"min": 1,
|
||||
"max": 2
|
||||
},
|
||||
"api": {
|
||||
"min": 1,
|
||||
"max": 1
|
||||
},
|
||||
"protocol": {
|
||||
"min": 1,
|
||||
"max": 1
|
||||
},
|
||||
"dataFormat": {
|
||||
"min": 1,
|
||||
"max": 2
|
||||
},
|
||||
"capabilities": {
|
||||
"ui": {
|
||||
"entry": "ui/main.js",
|
||||
"css": [
|
||||
"ui/main.css"
|
||||
],
|
||||
"sdk": {
|
||||
"min": 1,
|
||||
"max": 1
|
||||
},
|
||||
"pages": [
|
||||
"home"
|
||||
],
|
||||
"navigation": [
|
||||
"home"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"schemaVersion":1,"id":"sample","version":"1.2.3","executable":"bin/sample","os":"linux","arch":"amd64","host":{"min":1,"max":2},"api":{"min":1,"max":1},"protocol":{"min":1,"max":1},"dataFormat":{"min":1,"max":2},"capabilities":{"service":true}}
|
||||
@@ -0,0 +1 @@
|
||||
{"schemaVersion":1,"id":"sample","version":"1.2.3","executable":"bin/sample","os":"linux","arch":"amd64","host":{"min":1,"max":2},"api":{"min":1,"max":1},"protocol":{"min":1,"max":1},"dataFormat":{"min":1,"max":2},"capabilities":{"service":true}}
|
||||
@@ -0,0 +1,158 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
// Package signing builds hermetic cryptographic fixtures, never production trust.
|
||||
package signing
|
||||
|
||||
import (
|
||||
"crypto"
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/asn1"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"math/big"
|
||||
"time"
|
||||
|
||||
"encoding/pem"
|
||||
"github.com/cyberphone/json-canonicalization/go/src/webpki.org/jsoncanonicalizer"
|
||||
ct "github.com/google/certificate-transparency-go"
|
||||
"github.com/google/certificate-transparency-go/tls"
|
||||
ctx509util "github.com/google/certificate-transparency-go/x509util"
|
||||
"github.com/sigstore/sigstore-go/pkg/root"
|
||||
)
|
||||
|
||||
type Authority struct {
|
||||
root.BaseTrustedMaterial
|
||||
parent *x509.Certificate
|
||||
key, logKey, rekorKey *ecdsa.PrivateKey
|
||||
logID, rekorID [32]byte
|
||||
}
|
||||
|
||||
func New() (*Authority, error) {
|
||||
key, e := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
tmpl := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "fixture root"}, NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour), IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign}
|
||||
raw, e := x509.CreateCertificate(rand.Reader, tmpl, tmpl, key.Public(), key)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
cert, e := x509.ParseCertificate(raw)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
rekorKey, e := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
rekorDER, e := x509.MarshalPKIXPublicKey(rekorKey.Public())
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
logKey, e := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
der, e := x509.MarshalPKIXPublicKey(logKey.Public())
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
return &Authority{parent: cert, key: key, logKey: logKey, rekorKey: rekorKey, logID: sha256.Sum256(der), rekorID: sha256.Sum256(rekorDER)}, nil
|
||||
}
|
||||
func (a *Authority) FulcioCertificateAuthorities() []root.CertificateAuthority {
|
||||
return []root.CertificateAuthority{&root.FulcioCertificateAuthority{Root: a.parent}}
|
||||
}
|
||||
func (a *Authority) CTLogs() map[string]*root.TransparencyLog {
|
||||
return map[string]*root.TransparencyLog{hex.EncodeToString(a.logID[:]): {PublicKey: a.logKey.Public()}}
|
||||
}
|
||||
|
||||
// Bundle signs a manifest digest and embeds a real SCT and Rekor signed entry
|
||||
// timestamp (the inclusion promise supported by Sigstore bundle v0.1).
|
||||
func (a *Authority) Bundle(digest, identity, issuer string) ([]byte, error) {
|
||||
key, e := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
now := time.Now().Add(-time.Second)
|
||||
tmpl := &x509.Certificate{SerialNumber: big.NewInt(2), EmailAddresses: []string{identity}, NotBefore: now.Add(-time.Minute), NotAfter: now.Add(time.Hour), KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageCodeSigning}, ExtraExtensions: []pkix.Extension{{Id: asn1.ObjectIdentifier{1, 3, 6, 1, 4, 1, 57264, 1, 1}, Value: []byte(issuer)}}}
|
||||
der, e := x509.CreateCertificate(rand.Reader, tmpl, a.parent, key.Public(), a.key)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
pre, e := x509.ParseCertificate(der)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
sct := ct.SignedCertificateTimestamp{SCTVersion: ct.V1, LogID: ct.LogID{KeyID: a.logID}, Timestamp: uint64(now.UnixMilli())}
|
||||
entry := ct.LogEntry{Leaf: ct.MerkleTreeLeaf{Version: ct.V1, LeafType: ct.TimestampedEntryLeafType, TimestampedEntry: &ct.TimestampedEntry{Timestamp: sct.Timestamp, EntryType: ct.PrecertLogEntryType, PrecertEntry: &ct.PreCert{IssuerKeyHash: sha256.Sum256(a.parent.RawSubjectPublicKeyInfo), TBSCertificate: pre.RawTBSCertificate}}}}
|
||||
input, e := ct.SerializeSCTSignatureInput(sct, entry)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
hash := sha256.Sum256(input)
|
||||
sig, e := a.logKey.Sign(rand.Reader, hash[:], crypto.SHA256)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
sct.Signature = ct.DigitallySigned{Algorithm: tls.SignatureAndHashAlgorithm{Hash: tls.SHA256, Signature: tls.ECDSA}, Signature: sig}
|
||||
list, e := ctx509util.MarshalSCTsIntoSCTList([]*ct.SignedCertificateTimestamp{&sct})
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
raw, e := tls.Marshal(*list)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
raw, e = asn1.Marshal(raw)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
tmpl.ExtraExtensions = append(tmpl.ExtraExtensions, pkix.Extension{Id: asn1.ObjectIdentifier{1, 3, 6, 1, 4, 1, 11129, 2, 4, 2}, Value: raw})
|
||||
der, e = x509.CreateCertificate(rand.Reader, tmpl, a.parent, key.Public(), a.key)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
cert, e := x509.ParseCertificate(der)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
digestBytes, e := hex.DecodeString(digest)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
sig, e = key.Sign(rand.Reader, digestBytes, crypto.SHA256)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: cert.Raw})
|
||||
body, e := json.Marshal(map[string]any{"apiVersion": "0.0.1", "kind": "hashedrekord", "spec": map[string]any{"data": map[string]any{"hash": map[string]string{"algorithm": "sha256", "value": digest}}, "signature": map[string]any{"content": sig, "publicKey": map[string]any{"content": certPEM}}}})
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
body, e = jsoncanonicalizer.Transform(body)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
payload, e := json.Marshal(map[string]any{"body": body, "integratedTime": now.Unix(), "logIndex": 0, "logID": hex.EncodeToString(a.rekorID[:])})
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
payload, e = jsoncanonicalizer.Transform(payload)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
hash = sha256.Sum256(payload)
|
||||
set, e := a.rekorKey.Sign(rand.Reader, hash[:], crypto.SHA256)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
return json.Marshal(map[string]any{"mediaType": "application/vnd.dev.sigstore.bundle+json;version=0.1", "messageSignature": map[string]any{"messageDigest": map[string]any{"algorithm": "SHA2_256", "digest": digestBytes}, "signature": sig}, "verificationMaterial": map[string]any{"x509CertificateChain": map[string]any{"certificates": []any{map[string]any{"rawBytes": base64.StdEncoding.EncodeToString(der)}}}, "tlogEntries": []any{map[string]any{"logIndex": "0", "logId": map[string]any{"keyId": a.rekorID[:]}, "kindVersion": map[string]string{"kind": "hashedrekord", "version": "0.0.1"}, "integratedTime": now.Unix(), "inclusionPromise": map[string]any{"signedEntryTimestamp": set}, "canonicalizedBody": body}}}})
|
||||
}
|
||||
func (a *Authority) RekorLogs() map[string]*root.TransparencyLog {
|
||||
return map[string]*root.TransparencyLog{hex.EncodeToString(a.rekorID[:]): {PublicKey: a.rekorKey.Public(), ValidityPeriodStart: time.Now().Add(-time.Hour)}}
|
||||
}
|
||||
@@ -0,0 +1,187 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/addons/catalog"
|
||||
)
|
||||
|
||||
// Package describes one release of a fixture add-on.
|
||||
type Package struct {
|
||||
ID string
|
||||
Version string
|
||||
// Capabilities is the manifest "capabilities" object.
|
||||
Capabilities map[string]any
|
||||
// Fixture is written as fixture.json into the package (modes of the
|
||||
// protocol fixture). Durable effect logs are enabled with Effects.
|
||||
Fixture map[string]any
|
||||
// Effects makes the fixture keep its commits in <effects dir>/<id>.log.
|
||||
Effects string
|
||||
// Files are extra package files, by package path.
|
||||
Files map[string]string
|
||||
// Executable is the package path of the executable. The default is
|
||||
// bin/fixture, which holds the protocol fixture.
|
||||
Executable string
|
||||
// Binary replaces the library's executable for this package, for a real
|
||||
// add-on built by the test.
|
||||
Binary []byte
|
||||
}
|
||||
|
||||
// Library serves releases as a catalog source on loopback.
|
||||
type Library struct {
|
||||
mu sync.Mutex
|
||||
listener net.Listener
|
||||
server *http.Server
|
||||
exe []byte
|
||||
releases []catalog.Release
|
||||
archives map[string][]byte
|
||||
}
|
||||
|
||||
// NewLibrary starts a catalog server. exe is the fixture executable that goes
|
||||
// into every package built by Add. An empty path starts a library that only
|
||||
// serves finished archives (AddArchive).
|
||||
func NewLibrary(exePath string) (*Library, error) {
|
||||
var exe []byte
|
||||
if exePath != "" {
|
||||
b, err := os.ReadFile(exePath) // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
exe = b
|
||||
}
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
lib := &Library{listener: l, exe: exe, archives: map[string][]byte{}}
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/library.yaml", func(w http.ResponseWriter, r *http.Request) {
|
||||
lib.mu.Lock()
|
||||
defer lib.mu.Unlock()
|
||||
_ = catalog.WriteIndex(w, lib.releases)
|
||||
})
|
||||
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
|
||||
lib.mu.Lock()
|
||||
data, ok := lib.archives[strings.TrimPrefix(r.URL.Path, "/")]
|
||||
lib.mu.Unlock()
|
||||
if !ok {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write(data)
|
||||
})
|
||||
lib.server = &http.Server{Handler: mux, ReadHeaderTimeout: 5 * time.Second}
|
||||
go func() { _ = lib.server.Serve(l) }()
|
||||
return lib, nil
|
||||
}
|
||||
|
||||
// URL is the library document address.
|
||||
func (l *Library) URL() string { return "http://" + l.listener.Addr().String() + "/library.yaml" }
|
||||
|
||||
// Close stops the server.
|
||||
func (l *Library) Close() { _ = l.server.Close() }
|
||||
|
||||
// Add builds a package archive and lists it. effectsDir receives the durable
|
||||
// effect log of packages that ask for one.
|
||||
func (l *Library) Add(p Package, effectsDir string) error {
|
||||
executable, binary := p.Executable, p.Binary
|
||||
if executable == "" {
|
||||
executable = "bin/fixture"
|
||||
}
|
||||
if binary == nil {
|
||||
binary = l.exe
|
||||
}
|
||||
manifest := map[string]any{
|
||||
"schemaVersion": 1, "id": p.ID, "version": p.Version, "executable": executable,
|
||||
"os": runtime.GOOS, "arch": runtime.GOARCH,
|
||||
"host": map[string]int{"min": 1, "max": 2}, "api": map[string]int{"min": 1, "max": 1},
|
||||
"protocol": map[string]int{"min": 1, "max": 1}, "dataFormat": map[string]int{"min": 1, "max": 2},
|
||||
"launchMode": "always", "capabilities": p.Capabilities,
|
||||
}
|
||||
raw, err := json.Marshal(manifest)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
tw := tar.NewWriter(&buf)
|
||||
add := func(name string, data []byte, mode int64) error {
|
||||
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: mode, Size: int64(len(data))}); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := tw.Write(data)
|
||||
return err
|
||||
}
|
||||
if err := add("manifest.json", raw, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := add(executable, binary, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
fixture := map[string]any{}
|
||||
for k, v := range p.Fixture {
|
||||
fixture[k] = v
|
||||
}
|
||||
if p.Effects != "" {
|
||||
fixture["effects"] = filepath.Join(effectsDir, p.ID+".log")
|
||||
}
|
||||
if len(fixture) > 0 {
|
||||
b, _ := json.Marshal(fixture)
|
||||
if err := add("fixture.json", b, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
names := make([]string, 0, len(p.Files))
|
||||
for n := range p.Files {
|
||||
names = append(names, n)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, n := range names {
|
||||
if err := add(n, []byte(p.Files[n]), 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := tw.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
name := fmt.Sprintf("%s-%s.tar", p.ID, p.Version)
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.archives[name] = buf.Bytes()
|
||||
l.releases = append(l.releases, catalog.Release{
|
||||
ID: p.ID, Name: p.ID, Version: p.Version, OS: runtime.GOOS, Arch: runtime.GOARCH,
|
||||
Digest: fmt.Sprintf("sha256:%x", sha256.Sum256(buf.Bytes())),
|
||||
Artifact: "http://" + l.listener.Addr().String() + "/" + name,
|
||||
})
|
||||
return nil
|
||||
}
|
||||
|
||||
// AddArchive lists a finished package archive (a plain tar with manifest.json
|
||||
// at its root) as one release. The catalog digest is the digest of the archive
|
||||
// bytes, which the installer verifies before it unpacks anything.
|
||||
func (l *Library) AddArchive(id, version string, archive []byte) (string, error) {
|
||||
name := fmt.Sprintf("%s-%s.tar", id, version)
|
||||
digest := fmt.Sprintf("sha256:%x", sha256.Sum256(archive))
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
l.archives[name] = archive
|
||||
l.releases = append(l.releases, catalog.Release{
|
||||
ID: id, Name: id, Version: version, OS: runtime.GOOS, Arch: runtime.GOARCH,
|
||||
Digest: digest,
|
||||
Artifact: "http://" + l.listener.Addr().String() + "/" + name,
|
||||
})
|
||||
return digest, nil
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Cluster is a running fixture: the catalog, the configuration and the three
|
||||
// server processes.
|
||||
type Cluster struct {
|
||||
Config *Config
|
||||
Library *Library
|
||||
Nodes map[string]*Node
|
||||
Creds *Credentials
|
||||
// Effects is where the fixture add-ons keep their durable commit logs.
|
||||
Effects string
|
||||
|
||||
mu sync.Mutex
|
||||
down bool
|
||||
}
|
||||
|
||||
// Paths are the executables a cluster needs.
|
||||
type Paths struct {
|
||||
Server, Mock, Fixture string
|
||||
}
|
||||
|
||||
// New generates a configuration under dir and starts the catalog. No process
|
||||
// is started: call Up.
|
||||
func New(dir string, p Paths) (*Cluster, error) {
|
||||
lib, err := NewLibrary(p.Fixture)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cfg, err := Generate(dir, lib.URL())
|
||||
if err != nil {
|
||||
lib.Close()
|
||||
return nil, err
|
||||
}
|
||||
c := &Cluster{Config: cfg, Library: lib, Nodes: map[string]*Node{}, Effects: filepath.Join(cfg.Dir, "effects")}
|
||||
if err := os.MkdirAll(c.Effects, 0o700); err != nil {
|
||||
lib.Close()
|
||||
return nil, err
|
||||
}
|
||||
for _, name := range Instances {
|
||||
c.Nodes[name] = &Node{Name: name, Config: cfg, ServerPath: p.Server, MockPath: p.Mock}
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// Up starts the owner, waits for it, and then starts the frontends.
|
||||
func (c *Cluster) Up() error {
|
||||
for _, name := range Instances {
|
||||
c.mu.Lock()
|
||||
if c.down {
|
||||
c.mu.Unlock()
|
||||
return fmt.Errorf("the cluster is down")
|
||||
}
|
||||
err := c.Nodes[name].Start()
|
||||
c.mu.Unlock()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := c.Nodes[name].Ready(120 * time.Second); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
creds, err := LoadCredentials(c.Config.Credentials)
|
||||
if err != nil {
|
||||
return fmt.Errorf("the owner wrote no credentials: %w", err)
|
||||
}
|
||||
c.Creds = creds
|
||||
return nil
|
||||
}
|
||||
|
||||
// Down stops every process and the catalog. It is idempotent, safe to call
|
||||
// from a signal handler, and no node starts after it.
|
||||
func (c *Cluster) Down() {
|
||||
c.mu.Lock()
|
||||
first := !c.down
|
||||
c.down = true
|
||||
c.mu.Unlock()
|
||||
for _, name := range []string{FrontB, FrontA, Owner} {
|
||||
c.Nodes[name].Stop()
|
||||
}
|
||||
if first {
|
||||
c.Library.Close()
|
||||
}
|
||||
}
|
||||
|
||||
// Token returns a client of one node authenticated with the API key of a user.
|
||||
func (c *Cluster) Token(node, user string) Client {
|
||||
return Client{Base: c.Nodes[node].URL(), Bearer: c.Creds.Users[user].Token}
|
||||
}
|
||||
|
||||
// Session returns a client of one node with the browser session of a user.
|
||||
func (c *Cluster) Session(node, user string) Client {
|
||||
return Client{Base: c.Nodes[node].URL(), Session: c.Creds.Users[user].Session}
|
||||
}
|
||||
|
||||
// Control calls the owner's test control route.
|
||||
func (c *Cluster) Control(path string, body any) (*Response, error) {
|
||||
return Client{Base: "http://" + c.Config.ControlAddr}.Do("POST", "/"+path, body, map[string]string{"X-Foundation-Control": c.Config.Control})
|
||||
}
|
||||
@@ -0,0 +1,292 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
// Package cluster builds the foundation acceptance fixture: one add-on owner
|
||||
// and two non-owner frontends, each a separate UI test server process with its
|
||||
// own listener, sharing one account database and speaking the real mTLS
|
||||
// carrier. The fixture add-on is the protocol fixture binary; nothing here
|
||||
// imports a Tables package.
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"crypto/ed25519"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"fmt"
|
||||
"math/big"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/addons/gateway"
|
||||
)
|
||||
|
||||
// Owner and the two frontends are the fixed instance names.
|
||||
const (
|
||||
Owner = "owner"
|
||||
FrontA = "front-a"
|
||||
FrontB = "front-b"
|
||||
)
|
||||
|
||||
// Instances lists every instance name in start order.
|
||||
var Instances = []string{Owner, FrontA, FrontB}
|
||||
|
||||
// Secret is the API key HMAC secret every instance shares.
|
||||
const Secret = "foundation-fixture-secret"
|
||||
|
||||
// Instance is one process of the fixture.
|
||||
type Instance struct {
|
||||
Name string `json:"name"`
|
||||
HTTPPort int `json:"httpPort"`
|
||||
CarrierAddr string `json:"carrierAddr"`
|
||||
// Dir holds the instance's models, data and logs. Root is the add-on root,
|
||||
// which only the owner opens.
|
||||
Dir string `json:"dir"`
|
||||
Root string `json:"root"`
|
||||
Cert string `json:"cert"`
|
||||
Key string `json:"key"`
|
||||
Sign string `json:"sign"`
|
||||
}
|
||||
|
||||
// Config is the cluster.json every process reads.
|
||||
type Config struct {
|
||||
Dir string `json:"dir"`
|
||||
AuthDB string `json:"authDb"`
|
||||
Control string `json:"control"`
|
||||
ControlAddr string `json:"controlAddr"`
|
||||
CatalogURL string `json:"catalogUrl"`
|
||||
Credentials string `json:"credentials"`
|
||||
CA string `json:"ca"`
|
||||
Pins map[string]string `json:"pins"`
|
||||
Verify map[string]string `json:"verify"`
|
||||
Nodes map[string]Instance `json:"nodes"`
|
||||
// ArtifactExpiryMillis is the lifetime of a private file result in
|
||||
// milliseconds; zero is the product default. A test that needs expiry
|
||||
// sets it and saves the configuration before the cluster starts.
|
||||
ArtifactExpiryMillis int64 `json:"artifactExpiryMillis,omitempty"`
|
||||
// Alias names a derived configuration (see Derive); it only changes the
|
||||
// file the configuration is saved to.
|
||||
Alias string `json:"alias,omitempty"`
|
||||
}
|
||||
|
||||
// Derive copies the configuration under another file name with the ports of
|
||||
// the owner replaced, so a second owner can be started on the same storage.
|
||||
func (c *Config) Derive(alias string) (*Config, error) {
|
||||
d := *c
|
||||
d.Alias = alias
|
||||
d.Nodes = map[string]Instance{}
|
||||
for k, v := range c.Nodes {
|
||||
d.Nodes[k] = v
|
||||
}
|
||||
owner := d.Nodes[Owner]
|
||||
var err error
|
||||
if owner.HTTPPort, err = freePort(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// The carrier address stays: it is what the frontends dial, so the second
|
||||
// owner would need the same listener and must fail before it gets it.
|
||||
d.Nodes[Owner] = owner
|
||||
if d.ControlAddr, err = freeAddr(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
releasePorts()
|
||||
return &d, d.Save()
|
||||
}
|
||||
|
||||
func freeAddr() (string, error) {
|
||||
p, err := freePort()
|
||||
return fmt.Sprintf("127.0.0.1:%d", p), err
|
||||
}
|
||||
|
||||
// Generate writes a fresh cluster configuration under dir. Ports are chosen
|
||||
// by the operating system and released again; the processes bind them later.
|
||||
func Generate(dir, catalogURL string) (*Config, error) {
|
||||
dir, err := filepath.Abs(dir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := os.MkdirAll(dir, 0o700); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pub, key, err := ed25519.GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ca := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "foundation CA"}, NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(48 * time.Hour), IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign}
|
||||
der, err := x509.CreateCertificate(rand.Reader, ca, ca, pub, key)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if ca, err = x509.ParseCertificate(der); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
control := make([]byte, 16)
|
||||
_, _ = rand.Read(control)
|
||||
controlPort, err := freePort()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c := &Config{
|
||||
ControlAddr: fmt.Sprintf("127.0.0.1:%d", controlPort),
|
||||
Dir: dir, AuthDB: filepath.Join(dir, "auth.db"), Control: hex.EncodeToString(control), CatalogURL: catalogURL,
|
||||
Credentials: filepath.Join(dir, "credentials.json"),
|
||||
CA: string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})),
|
||||
Pins: map[string]string{}, Verify: map[string]string{}, Nodes: map[string]Instance{},
|
||||
}
|
||||
for i, name := range Instances {
|
||||
leafPub, leafKey, err := ed25519.GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
leaf := &x509.Certificate{SerialNumber: big.NewInt(int64(i + 2)), Subject: pkix.Name{CommonName: name}, NotBefore: ca.NotBefore, NotAfter: ca.NotAfter, IPAddresses: []net.IP{net.ParseIP("127.0.0.1")}, KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth}}
|
||||
leafDER, err := x509.CreateCertificate(rand.Reader, leaf, ca, leafPub, key)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
keyDER, err := x509.MarshalPKCS8PrivateKey(leafKey)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
signPub, signKey, err := ed25519.GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pin := sha256.Sum256(leafDER)
|
||||
c.Pins[name] = hex.EncodeToString(pin[:])
|
||||
c.Verify[name] = hex.EncodeToString(signPub)
|
||||
httpPort, err := freePort()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
carrier := ""
|
||||
if name == Owner {
|
||||
p, err := freePort()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
carrier = fmt.Sprintf("127.0.0.1:%d", p)
|
||||
}
|
||||
node := Instance{
|
||||
Name: name, HTTPPort: httpPort, CarrierAddr: carrier, Dir: filepath.Join(dir, name), Root: filepath.Join(dir, name, "addons"),
|
||||
Cert: string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: leafDER})),
|
||||
Key: string(pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: keyDER})),
|
||||
Sign: hex.EncodeToString(signKey),
|
||||
}
|
||||
if err := os.MkdirAll(node.Dir, 0o700); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.Nodes[name] = node
|
||||
}
|
||||
releasePorts()
|
||||
return c, c.Save()
|
||||
}
|
||||
|
||||
// held keeps every chosen port bound until the configuration is complete, so
|
||||
// that two choices can never return the same number.
|
||||
var (
|
||||
heldMu sync.Mutex
|
||||
held []net.Listener
|
||||
)
|
||||
|
||||
func freePort() (int, error) {
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
heldMu.Lock()
|
||||
held = append(held, l)
|
||||
heldMu.Unlock()
|
||||
return l.Addr().(*net.TCPAddr).Port, nil
|
||||
}
|
||||
|
||||
// releasePorts frees the chosen ports for the processes that will bind them.
|
||||
func releasePorts() {
|
||||
heldMu.Lock()
|
||||
defer heldMu.Unlock()
|
||||
for _, l := range held {
|
||||
_ = l.Close()
|
||||
}
|
||||
held = nil
|
||||
}
|
||||
|
||||
// Path is where cluster.json lives.
|
||||
func (c *Config) Path() string {
|
||||
if c.Alias != "" {
|
||||
return filepath.Join(c.Dir, "cluster-"+c.Alias+".json")
|
||||
}
|
||||
return filepath.Join(c.Dir, "cluster.json")
|
||||
}
|
||||
|
||||
// Save writes cluster.json.
|
||||
func (c *Config) Save() error {
|
||||
b, err := json.MarshalIndent(c, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(c.Path(), b, 0o600)
|
||||
}
|
||||
|
||||
// Load reads a cluster.json.
|
||||
func Load(path string) (*Config, error) {
|
||||
b, err := os.ReadFile(path) // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var c Config
|
||||
if err := json.Unmarshal(b, &c); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &c, nil
|
||||
}
|
||||
|
||||
// OwnerAddress is the address every instance dials for the owner.
|
||||
func (c *Config) OwnerAddress() string { return "https://" + c.Nodes[Owner].CarrierAddr }
|
||||
|
||||
// Carrier builds the carrier identity of one instance.
|
||||
func (c *Config) Carrier(name string) (*gateway.CarrierConfig, error) {
|
||||
node, ok := c.Nodes[name]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("unknown instance %q", name)
|
||||
}
|
||||
pair, err := tls.X509KeyPair([]byte(node.Cert), []byte(node.Key))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
roots := x509.NewCertPool()
|
||||
if !roots.AppendCertsFromPEM([]byte(c.CA)) {
|
||||
return nil, fmt.Errorf("invalid CA")
|
||||
}
|
||||
signRaw, err := hex.DecodeString(node.Sign)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pins := map[string][32]byte{}
|
||||
for id, h := range c.Pins {
|
||||
raw, err := hex.DecodeString(h)
|
||||
if err != nil || len(raw) != 32 {
|
||||
return nil, fmt.Errorf("invalid pin")
|
||||
}
|
||||
var pin [32]byte
|
||||
copy(pin[:], raw)
|
||||
pins[id] = pin
|
||||
}
|
||||
keys := map[string]ed25519.PublicKey{}
|
||||
for id, h := range c.Verify {
|
||||
raw, err := hex.DecodeString(h)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
keys[id] = ed25519.PublicKey(raw)
|
||||
}
|
||||
return &gateway.CarrierConfig{
|
||||
InstanceID: name, OwnerID: Owner, OwnerAddress: c.OwnerAddress(),
|
||||
Certificate: pair, Roots: roots, Pins: pins, SigningKey: ed25519.PrivateKey(signRaw), VerificationKeys: keys,
|
||||
}, nil
|
||||
}
|
||||
@@ -0,0 +1,354 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/pkg/httpclient"
|
||||
)
|
||||
|
||||
// Node is one UI test server process of the cluster.
|
||||
type Node struct {
|
||||
Name string
|
||||
Config *Config
|
||||
ServerPath string
|
||||
MockPath string
|
||||
|
||||
mu sync.Mutex
|
||||
cmd *exec.Cmd
|
||||
done chan struct{}
|
||||
// groups are the process groups of every process this node ever started.
|
||||
// A server leads a group of its own and its add-on and mock processes
|
||||
// stay in it, even after the server is killed on purpose, so a sweep of
|
||||
// the groups reaches an orphan that the scenario left behind.
|
||||
groups []int
|
||||
}
|
||||
|
||||
// URL is the HTTP address of the node.
|
||||
func (n *Node) URL() string {
|
||||
return fmt.Sprintf("http://127.0.0.1:%d", n.Config.Nodes[n.Name].HTTPPort)
|
||||
}
|
||||
|
||||
// Start launches the process. It does not wait for readiness.
|
||||
func (n *Node) Start() error {
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
if n.cmd != nil && !n.exitedLocked() {
|
||||
return fmt.Errorf("%s is already running", n.Name)
|
||||
}
|
||||
logFile, err := os.OpenFile(filepath.Join(n.Config.Dir, n.Name+".log"), os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
node := n.Config.Nodes[n.Name]
|
||||
cmd := exec.Command(n.ServerPath, // #nosec G204 -- test harness starts the binaries it built
|
||||
"--mock-backend="+n.MockPath,
|
||||
"--port="+strconv.Itoa(node.HTTPPort),
|
||||
"--foundation="+n.Config.Path(),
|
||||
"--foundation-role="+n.Name)
|
||||
cmd.Stdout, cmd.Stderr = logFile, logFile
|
||||
cmd.Env = append(cmd.Environ(), "TMPDIR="+node.Dir)
|
||||
// Own process group, so teardown reaches every descendant. The server
|
||||
// dies with the harness (Linux), so a killed test run leaves no server.
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true, Pdeathsig: syscall.SIGKILL}
|
||||
if err := cmd.Start(); err != nil {
|
||||
_ = logFile.Close()
|
||||
return err
|
||||
}
|
||||
done := make(chan struct{})
|
||||
n.cmd, n.done = cmd, done
|
||||
n.groups = append(n.groups, cmd.Process.Pid)
|
||||
go func() { _ = cmd.Wait(); _ = logFile.Close(); close(done) }()
|
||||
return nil
|
||||
}
|
||||
|
||||
// PID is the process id, or zero when it was never started.
|
||||
func (n *Node) PID() int {
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
if n.cmd == nil || n.cmd.Process == nil {
|
||||
return 0
|
||||
}
|
||||
return n.cmd.Process.Pid
|
||||
}
|
||||
|
||||
// Exited reports whether the process has ended.
|
||||
func (n *Node) Exited() bool {
|
||||
n.mu.Lock()
|
||||
defer n.mu.Unlock()
|
||||
return n.exitedLocked()
|
||||
}
|
||||
|
||||
func (n *Node) exitedLocked() bool {
|
||||
if n.done == nil {
|
||||
return true
|
||||
}
|
||||
select {
|
||||
case <-n.done:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// Wait blocks until the process ended or the timeout passed.
|
||||
func (n *Node) Wait(timeout time.Duration) bool {
|
||||
n.mu.Lock()
|
||||
done := n.done
|
||||
n.mu.Unlock()
|
||||
if done == nil {
|
||||
return true
|
||||
}
|
||||
select {
|
||||
case <-done:
|
||||
return true
|
||||
case <-time.After(timeout):
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// Ready waits until the node answers its readiness probe.
|
||||
func (n *Node) Ready(timeout time.Duration) error {
|
||||
deadline := time.Now().Add(timeout)
|
||||
for time.Now().Before(deadline) {
|
||||
if n.Exited() {
|
||||
return fmt.Errorf("%s exited before it was ready: %s", n.Name, n.Tail(15))
|
||||
}
|
||||
resp, err := httpclient.NewWithTimeout(5 * time.Second).Get(n.URL() + "/readyz")
|
||||
if err == nil {
|
||||
_ = resp.Body.Close()
|
||||
if resp.StatusCode == 200 {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
return fmt.Errorf("%s was not ready in %s: %s", n.Name, timeout, n.Tail(15))
|
||||
}
|
||||
|
||||
// Kill ends the server process alone, without any chance to clean up. Its
|
||||
// descendants are left running on purpose: a scenario uses this to orphan an
|
||||
// add-on. Stop sweeps them.
|
||||
func (n *Node) Kill() {
|
||||
n.mu.Lock()
|
||||
cmd := n.cmd
|
||||
n.mu.Unlock()
|
||||
if cmd != nil && cmd.Process != nil && !n.Exited() {
|
||||
_ = cmd.Process.Signal(syscall.SIGKILL)
|
||||
n.Wait(15 * time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
// Stop asks the server to end, waits a bounded time, and then kills every
|
||||
// process of every group the node started. It is idempotent and also reaps
|
||||
// what an earlier Kill left behind.
|
||||
func (n *Node) Stop() {
|
||||
n.mu.Lock()
|
||||
cmd := n.cmd
|
||||
n.mu.Unlock()
|
||||
if cmd != nil && cmd.Process != nil && !n.Exited() {
|
||||
_ = cmd.Process.Signal(syscall.SIGTERM)
|
||||
if !n.Wait(20 * time.Second) {
|
||||
n.Kill()
|
||||
}
|
||||
}
|
||||
n.sweep(5 * time.Second)
|
||||
}
|
||||
|
||||
// sweep stops what is left in the process groups of the node: SIGTERM, a
|
||||
// bounded wait, then SIGKILL.
|
||||
func (n *Node) sweep(grace time.Duration) {
|
||||
n.mu.Lock()
|
||||
groups := append([]int(nil), n.groups...)
|
||||
n.mu.Unlock()
|
||||
live := func() []int {
|
||||
var out []int
|
||||
for _, g := range groups {
|
||||
if syscall.Kill(-g, 0) == nil {
|
||||
out = append(out, g)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
for _, g := range live() {
|
||||
_ = syscall.Kill(-g, syscall.SIGTERM)
|
||||
}
|
||||
deadline := time.Now().Add(grace)
|
||||
for len(live()) > 0 && time.Now().Before(deadline) {
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
for _, g := range live() {
|
||||
_ = syscall.Kill(-g, syscall.SIGKILL)
|
||||
}
|
||||
for len(live()) > 0 && time.Now().Before(deadline.Add(5*time.Second)) {
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// Tail returns the last lines of the node log.
|
||||
func (n *Node) Tail(lines int) string {
|
||||
b, err := os.ReadFile(filepath.Join(n.Config.Dir, n.Name+".log"))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
parts := strings.Split(strings.TrimSpace(string(b)), "\n")
|
||||
if len(parts) > lines {
|
||||
parts = parts[len(parts)-lines:]
|
||||
}
|
||||
return strings.Join(parts, "\n")
|
||||
}
|
||||
|
||||
// ChildrenOf lists the live direct children of a process, as pid and the
|
||||
// command line, by reading /proc.
|
||||
func ChildrenOf(pid int) map[int]string {
|
||||
out := map[int]string{}
|
||||
entries, _ := os.ReadDir("/proc")
|
||||
for _, e := range entries {
|
||||
child, err := strconv.Atoi(e.Name())
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
stat, err := os.ReadFile("/proc/" + e.Name() + "/stat")
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
s := string(stat)
|
||||
end := strings.LastIndex(s, ")")
|
||||
if end < 0 {
|
||||
continue
|
||||
}
|
||||
f := strings.Fields(s[end+1:])
|
||||
// f[0] is the state and f[1] the parent.
|
||||
if len(f) < 2 || f[0] == "Z" {
|
||||
continue
|
||||
}
|
||||
if ppid, _ := strconv.Atoi(f[1]); ppid == pid {
|
||||
cmdline, _ := os.ReadFile("/proc/" + e.Name() + "/cmdline")
|
||||
out[child] = strings.ReplaceAll(strings.TrimRight(string(cmdline), "\x00"), "\x00", " ")
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Alive reports whether a process id is a live, non-zombie process.
|
||||
func Alive(pid int) bool {
|
||||
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
s := string(b)
|
||||
end := strings.LastIndex(s, ")")
|
||||
if end < 0 {
|
||||
return false
|
||||
}
|
||||
f := strings.Fields(s[end+1:])
|
||||
return len(f) > 0 && f[0] != "Z"
|
||||
}
|
||||
|
||||
// Credentials are the accounts the owner seeded.
|
||||
type Credentials struct {
|
||||
Users map[string]User `json:"users"`
|
||||
Packages []string `json:"packages"`
|
||||
}
|
||||
|
||||
// User is one seeded account with its API key and browser session.
|
||||
type User struct {
|
||||
ID string `json:"id"`
|
||||
Role string `json:"role"`
|
||||
Token string `json:"token"`
|
||||
Session string `json:"session"`
|
||||
KeyID string `json:"keyId"`
|
||||
}
|
||||
|
||||
// LoadCredentials reads the file the owner wrote.
|
||||
func LoadCredentials(path string) (*Credentials, error) {
|
||||
b, err := os.ReadFile(path) // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var c Credentials
|
||||
return &c, json.Unmarshal(b, &c)
|
||||
}
|
||||
|
||||
// Response is one HTTP answer.
|
||||
type Response struct {
|
||||
Status int
|
||||
Header http.Header
|
||||
Body []byte
|
||||
}
|
||||
|
||||
// JSON decodes the body.
|
||||
func (r *Response) JSON(v any) error { return json.Unmarshal(r.Body, v) }
|
||||
|
||||
// String is the body as text.
|
||||
func (r *Response) String() string { return string(r.Body) }
|
||||
|
||||
// Client talks to one node as one account.
|
||||
type Client struct {
|
||||
Base string
|
||||
Bearer string
|
||||
Session string
|
||||
// Headers are sent on every request, after the credential headers.
|
||||
Headers map[string]string
|
||||
}
|
||||
|
||||
// Do sends one request. A session client sends the browser fetch metadata the
|
||||
// add-on routes require; a bearer client sends none.
|
||||
func (c Client) Do(method, path string, body any, extra map[string]string) (*Response, error) {
|
||||
var reader io.Reader
|
||||
if body != nil {
|
||||
if raw, ok := body.(string); ok {
|
||||
reader = strings.NewReader(raw)
|
||||
} else {
|
||||
b, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
reader = bytes.NewReader(b)
|
||||
}
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(ctx, method, c.Base+path, reader)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
switch {
|
||||
case c.Bearer != "":
|
||||
req.Header.Set("Authorization", "Bearer "+c.Bearer)
|
||||
case c.Session != "":
|
||||
req.Header.Set("Cookie", "session="+c.Session)
|
||||
req.Header.Set("Sec-Fetch-Site", "same-origin")
|
||||
req.Header.Set("Sec-Fetch-Mode", "cors")
|
||||
req.Header.Set("Sec-Fetch-Dest", "empty")
|
||||
}
|
||||
for k, v := range c.Headers {
|
||||
req.Header.Set(k, v)
|
||||
}
|
||||
for k, v := range extra {
|
||||
req.Header.Set(k, v)
|
||||
}
|
||||
resp, err := httpclient.New().Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
data, err := io.ReadAll(io.LimitReader(resp.Body, 16<<20))
|
||||
return &Response{Status: resp.StatusCode, Header: resp.Header, Body: data}, err
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package cluster
|
||||
|
||||
// The fixture packages. All run the protocol fixture; they differ only in what
|
||||
// the manifest declares, so the three shapes the host must support are covered:
|
||||
// a UI-bearing package, a tool package without a UI and a service only package.
|
||||
|
||||
// UIModule is a minimal add-on interface in the compiled form of the SDK
|
||||
// contract: it draws a title and a button that reads through the host.
|
||||
const UIModule = `export function createAddon(sdk) {
|
||||
const h = sdk.React.createElement
|
||||
return {
|
||||
Component: () => h('section', null,
|
||||
h('h2', { 'data-testid': 'fnd-title' }, 'Foundation fixture'),
|
||||
h('button', { 'data-testid': 'fnd-read', onClick: () => sdk.api.read({ arguments: { tool: 'echo' } }).then(
|
||||
(out) => { window.__fnd = { ok: out } }, (e) => { window.__fnd = { error: e && e.status } }) }, 'Read')),
|
||||
dispose() {},
|
||||
}
|
||||
}
|
||||
`
|
||||
|
||||
// UIPackage is the UI-bearing add-on with tools.
|
||||
func UIPackage(version string, fixture map[string]any) Package {
|
||||
return Package{
|
||||
ID: "found-ui", Version: version, Effects: "yes", Fixture: fixture,
|
||||
Capabilities: map[string]any{
|
||||
"service": true, "tools": []string{"echo", "export"},
|
||||
"ui": map[string]any{"entry": "ui.js", "css": []string{"ui.css"}, "sdk": map[string]int{"min": 1, "max": 2}, "pages": []string{"home"}, "mainPage": "home", "title": "Foundation", "description": "Foundation fixture."},
|
||||
},
|
||||
Files: map[string]string{"ui.js": UIModule, "ui.css": "body{}\n", "private.txt": "not declared"},
|
||||
}
|
||||
}
|
||||
|
||||
// ToolPackage is a tool add-on without a UI.
|
||||
func ToolPackage(version string, fixture map[string]any) Package {
|
||||
return Package{ID: "found-tool", Version: version, Effects: "yes", Fixture: fixture, Capabilities: map[string]any{"service": true, "tools": []string{"echo", "export"}}}
|
||||
}
|
||||
|
||||
// ServicePackage declares only a service: no tool, no page.
|
||||
func ServicePackage(version string, fixture map[string]any) Package {
|
||||
return Package{ID: "found-svc", Version: version, Effects: "yes", Fixture: fixture, Capabilities: map[string]any{"service": true}}
|
||||
}
|
||||
@@ -0,0 +1,182 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/subtle"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/mudler/LocalAI/core/application"
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/http/auth"
|
||||
"github.com/mudler/LocalAI/core/services/addons/catalog"
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
)
|
||||
|
||||
// AppOptions are the application options of one role. The account database is
|
||||
// the same file for every instance, so a session or key is valid on all.
|
||||
func (c *Config) AppOptions(role string) ([]config.AppOption, error) {
|
||||
node, ok := c.Nodes[role]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("unknown role %q", role)
|
||||
}
|
||||
transport, err := c.Carrier(role)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
addon := config.AddonConfig{
|
||||
Enabled: true, InstanceID: role, OwnerID: Owner, OwnerAddress: c.OwnerAddress(), StorageID: "disk",
|
||||
Root: node.Root, Transport: transport,
|
||||
Host: contract.HostCompatibility{Host: 1, API: 1, Protocol: 1, SDK: 1, DataFormat: 1, OS: runtime.GOOS, Arch: runtime.GOARCH},
|
||||
}
|
||||
if role == Owner {
|
||||
addon.ListenAddress = node.CarrierAddr
|
||||
addon.ArtifactExpiry = time.Duration(c.ArtifactExpiryMillis) * time.Millisecond
|
||||
}
|
||||
return []config.AppOption{
|
||||
config.WithAuthEnabled(true),
|
||||
config.WithAuthDatabaseURL(c.AuthDB),
|
||||
config.WithAuthAPIKeyHMACSecret(Secret),
|
||||
config.WithAddons(addon),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Seed creates the accounts and the catalog source once, on the first start of
|
||||
// the owner, and writes the credentials. A restart keeps them.
|
||||
func (c *Config) Seed(ctx context.Context, app *application.Application) error {
|
||||
if _, err := os.Stat(c.Credentials); err == nil {
|
||||
return nil
|
||||
}
|
||||
db := app.AuthDB()
|
||||
if db == nil {
|
||||
return fmt.Errorf("authentication is not enabled")
|
||||
}
|
||||
creds := Credentials{Users: map[string]User{}}
|
||||
var admin contract.Actor
|
||||
for _, u := range []struct {
|
||||
name, role string
|
||||
feature bool
|
||||
}{{"admin", auth.RoleAdmin, true}, {"member", auth.RoleUser, true}, {"friend", auth.RoleUser, true}, {"outsider", auth.RoleUser, false},
|
||||
// Two more ordinary accounts for suites that need a viewer and an
|
||||
// editor next to the owner of a resource.
|
||||
{"viewer", auth.RoleUser, true}, {"editor", auth.RoleUser, true}} {
|
||||
user := auth.User{ID: "fnd-" + u.name, Email: u.name + "@foundation.test", Name: u.name, Provider: auth.ProviderLocal, Role: u.role, Status: auth.StatusActive}
|
||||
if err := db.Create(&user).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if u.feature && u.role != auth.RoleAdmin {
|
||||
if err := db.Create(&auth.UserPermission{ID: uuid.NewString(), UserID: user.ID, Permissions: auth.PermissionMap{auth.FeatureAddons: true}}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
token, key, err := auth.CreateAPIKey(db, user.ID, "foundation", u.role, Secret, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
session, err := auth.CreateSession(db, user.ID, Secret)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
creds.Users[u.name] = User{ID: user.ID, Role: u.role, Token: token, Session: session, KeyID: key.ID}
|
||||
if u.name == "admin" {
|
||||
admin = contract.Actor{UserID: user.ID, CredentialID: "key:" + key.ID, Mode: auth.AddonModeNamedUser}
|
||||
}
|
||||
}
|
||||
sources := app.AddonSources()
|
||||
if sources == nil {
|
||||
return fmt.Errorf("the add-on owner did not start")
|
||||
}
|
||||
if err := sources.Save(ctx, admin, catalog.Source{ID: "fixture", Name: "Fixture", URL: c.CatalogURL, Enabled: true, AllowLoopbackHTTP: true}); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := sources.Refresh(ctx, "fixture"); err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := json.MarshalIndent(creds, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := c.Credentials + ".tmp"
|
||||
if err := os.WriteFile(tmp, b, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, c.Credentials)
|
||||
}
|
||||
|
||||
// ControlServer serves the owner-only test control routes on loopback: the
|
||||
// fixture has no product route that creates a resource, so a test registers
|
||||
// one through the owner's own policy here. It never grants more than the
|
||||
// policy itself would.
|
||||
func (c *Config) ControlServer(app *application.Application) (*http.Server, error) {
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/resource", func(w http.ResponseWriter, r *http.Request) {
|
||||
if subtle.ConstantTimeCompare([]byte(r.Header.Get("X-Foundation-Control")), []byte(c.Control)) != 1 || r.Method != http.MethodPost {
|
||||
http.Error(w, "denied", http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
AddonID string `json:"addonId"`
|
||||
Owner string `json:"owner"`
|
||||
Grants map[string]string `json:"grants"`
|
||||
}
|
||||
if json.NewDecoder(r.Body).Decode(&req) != nil {
|
||||
http.Error(w, "invalid", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
creds, err := LoadCredentials(c.Credentials)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
actorOf := func(name string) (contract.Actor, bool) {
|
||||
u, ok := creds.Users[name]
|
||||
return contract.Actor{UserID: u.ID, CredentialID: "key:" + u.KeyID, Mode: auth.AddonModeNamedUser}, ok
|
||||
}
|
||||
owner, ok := actorOf(req.Owner)
|
||||
p := app.AddonPolicy()
|
||||
if !ok || p == nil {
|
||||
http.Error(w, "no owner policy", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
key, err := p.Register(ctx, owner, req.AddonID)
|
||||
if err == nil {
|
||||
err = p.Commit(ctx, owner, key)
|
||||
}
|
||||
for name, role := range req.Grants {
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
u, ok := creds.Users[name]
|
||||
if !ok {
|
||||
err = fmt.Errorf("unknown user %q", name)
|
||||
break
|
||||
}
|
||||
err = p.SetGrant(ctx, owner, key, u.ID, policy.Role(role))
|
||||
}
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"resourceId": key.ResourceID})
|
||||
})
|
||||
l, err := net.Listen("tcp", c.ControlAddr)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
s := &http.Server{Handler: mux, ReadHeaderTimeout: 5 * time.Second}
|
||||
go func() { _ = s.Serve(l) }()
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// StateDir is where one instance keeps its models, data and generated files.
|
||||
func (c *Config) StateDir(role string) string { return filepath.Join(c.Nodes[role].Dir, "state") }
|
||||
@@ -0,0 +1,98 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package cluster
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// fakeServer is a server stand-in: it starts two helpers that ignore SIGTERM,
|
||||
// as an add-on in orphan mode does, records their ids and then waits.
|
||||
const fakeServer = `#!/bin/sh
|
||||
dir=$(dirname "$0")
|
||||
for n in 1 2; do
|
||||
( trap '' TERM; exec sleep 300 ) &
|
||||
echo $! >> "$dir/helpers"
|
||||
done
|
||||
echo ready > "$dir/ready"
|
||||
while :; do sleep 1; done
|
||||
`
|
||||
|
||||
func helperPIDs(t *testing.T, dir string) []int {
|
||||
t.Helper()
|
||||
var pids []int
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if _, err := os.Stat(filepath.Join(dir, "ready")); err == nil {
|
||||
break
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, "helpers"))
|
||||
NewWithT(t).Expect(err).NotTo(HaveOccurred())
|
||||
for _, f := range strings.Fields(string(b)) {
|
||||
n, _ := strconv.Atoi(f)
|
||||
pids = append(pids, n)
|
||||
}
|
||||
NewWithT(t).Expect(pids).To(HaveLen(2), "want two helpers")
|
||||
return pids
|
||||
}
|
||||
|
||||
func newFakeNode(t *testing.T) (*Node, string) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
script := filepath.Join(dir, "server.sh")
|
||||
NewWithT(t).Expect(os.WriteFile(script, []byte(fakeServer), 0o700)).To(Succeed())
|
||||
cfg := &Config{Dir: dir, Nodes: map[string]Instance{Owner: {}}}
|
||||
return &Node{Name: Owner, Config: cfg, ServerPath: script}, dir
|
||||
}
|
||||
|
||||
func assertGone(t *testing.T, pids ...int) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
alive := false
|
||||
for _, p := range pids {
|
||||
alive = alive || Alive(p)
|
||||
}
|
||||
if !alive {
|
||||
return
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
NewWithT(t).Expect(false).To(BeTrue(), "processes survived the teardown: %v", pids)
|
||||
}
|
||||
|
||||
// A failing scenario ends in Stop: the helpers that ignore SIGTERM must go.
|
||||
func TestStopReapsDescendantsThatIgnoreSIGTERM(t *testing.T) {
|
||||
n, dir := newFakeNode(t)
|
||||
NewWithT(t).Expect(n.Start()).To(Succeed())
|
||||
helpers := helperPIDs(t, dir)
|
||||
n.Stop()
|
||||
assertGone(t, append(helpers, n.PID())...)
|
||||
n.Stop() // idempotent
|
||||
}
|
||||
|
||||
// A scenario that kills the server on purpose orphans its helpers; Stop must
|
||||
// still reach them, across a restart of the same node.
|
||||
func TestStopReapsOrphansLeftByKill(t *testing.T) {
|
||||
n, dir := newFakeNode(t)
|
||||
NewWithT(t).Expect(n.Start()).To(Succeed())
|
||||
first := helperPIDs(t, dir)
|
||||
n.Kill()
|
||||
for _, p := range first {
|
||||
NewWithT(t).Expect(Alive(p)).To(BeTrue(), "Kill must leave the descendants running, %d is gone", p)
|
||||
}
|
||||
_ = os.Remove(filepath.Join(dir, "ready"))
|
||||
_ = os.Remove(filepath.Join(dir, "helpers"))
|
||||
NewWithT(t).Expect(n.Start()).To(Succeed())
|
||||
second := helperPIDs(t, dir)
|
||||
n.Stop()
|
||||
assertGone(t, append(first, second...)...)
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
// Command foundation-cluster runs the foundation fixture for the browser
|
||||
// checks: one owner and two frontends, the catalog on loopback, and the UI
|
||||
// package already installed and enabled through a frontend. It writes a
|
||||
// ready file with the addresses and accounts and runs until it is stopped.
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
)
|
||||
|
||||
func main() {
|
||||
dir := flag.String("dir", "", "directory of the cluster state (created)")
|
||||
server := flag.String("server", "", "UI test server executable")
|
||||
mock := flag.String("mock-backend", "", "mock backend executable")
|
||||
fixture := flag.String("fixture", "", "protocol fixture executable")
|
||||
ready := flag.String("ready", "", "file written when the cluster is serving")
|
||||
flag.Parse()
|
||||
if err := run(*dir, cluster.Paths{Server: *server, Mock: *mock, Fixture: *fixture}, *ready); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "foundation-cluster:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func run(dir string, p cluster.Paths, ready string) error {
|
||||
if dir == "" || ready == "" || p.Server == "" || p.Mock == "" || p.Fixture == "" {
|
||||
return fmt.Errorf("-dir, -server, -mock-backend, -fixture and -ready are required")
|
||||
}
|
||||
c, err := cluster.New(dir, p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer c.Down()
|
||||
// A signal during start-up or serving tears everything down first.
|
||||
stop := make(chan os.Signal, 1)
|
||||
signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
|
||||
interrupted := make(chan struct{})
|
||||
go func() {
|
||||
<-stop
|
||||
c.Down()
|
||||
close(interrupted)
|
||||
os.Exit(1)
|
||||
}()
|
||||
if err := c.Library.Add(cluster.UIPackage("1.0.0", nil), c.Effects); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := c.Up(); err != nil {
|
||||
return err
|
||||
}
|
||||
// Installed and enabled through a frontend, as an administrator would.
|
||||
admin := c.Token(cluster.FrontA, "admin")
|
||||
for i, action := range []string{"install", "enable"} {
|
||||
args := map[string]any{}
|
||||
if action == "install" {
|
||||
args = map[string]any{"sourceId": "fixture", "version": "1.0.0"}
|
||||
}
|
||||
r, err := admin.Do("POST", "/api/addons/dispatch", map[string]any{"operationId": fmt.Sprintf("browser-%s-%d", action, i), "addonId": "found-ui", "generation": 1, "action": action, "arguments": args}, nil)
|
||||
if err != nil || r.Status != 200 {
|
||||
return fmt.Errorf("%s: %v %v", action, err, r)
|
||||
}
|
||||
}
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
for {
|
||||
r, err := admin.Do("GET", "/api/addons/apps", nil, nil)
|
||||
if err == nil && r.Status == 200 && strings.Contains(string(r.Body), `"state":"running"`) {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return fmt.Errorf("the add-on did not start")
|
||||
}
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
}
|
||||
out := map[string]any{"urls": map[string]string{}, "users": c.Creds.Users, "dir": c.Config.Dir}
|
||||
for name, n := range c.Nodes {
|
||||
out["urls"].(map[string]string)[name] = n.URL()
|
||||
}
|
||||
b, _ := json.MarshalIndent(out, "", " ")
|
||||
tmp := filepath.Clean(ready) + ".tmp"
|
||||
if err := os.WriteFile(tmp, b, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Rename(tmp, ready); err != nil {
|
||||
return err
|
||||
}
|
||||
<-interrupted
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package foundation_test
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestFoundation(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Add-ons foundation acceptance")
|
||||
}
|
||||
|
||||
// paths are the executables every cluster of this suite runs.
|
||||
var paths cluster.Paths
|
||||
|
||||
// repoRoot is the root of this repository.
|
||||
var repoRoot string
|
||||
|
||||
var _ = BeforeSuite(func() {
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
repoRoot = filepath.Clean(filepath.Join(filepath.Dir(file), "..", "..", ".."))
|
||||
scratch, err := os.MkdirTemp("", "foundation-bin-*")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() { _ = os.RemoveAll(scratch) })
|
||||
build := func(env, out, pkg string) string {
|
||||
if p := os.Getenv(env); p != "" {
|
||||
return p
|
||||
}
|
||||
target := filepath.Join(scratch, out)
|
||||
cmd := exec.Command("go", "build", "-p", "2", "-tags", "auth", "-o", target, pkg)
|
||||
cmd.Dir = repoRoot
|
||||
b, err := cmd.CombinedOutput()
|
||||
Expect(err).NotTo(HaveOccurred(), string(b))
|
||||
return target
|
||||
}
|
||||
paths.Fixture = build("FOUNDATION_FIXTURE", "fixture-addon", "./tests/addons/fixture/cmd")
|
||||
paths.Server = build("FOUNDATION_SERVER", "ui-test-server", "./tests/e2e-ui")
|
||||
paths.Mock = os.Getenv("FOUNDATION_MOCK")
|
||||
if paths.Mock == "" {
|
||||
paths.Mock = filepath.Join(repoRoot, "tests/e2e/mock-backend/mock-backend")
|
||||
}
|
||||
Expect(paths.Mock).To(BeAnExistingFile(), "build tests/e2e/mock-backend first (scripts/addons/prepare.sh --integration)")
|
||||
})
|
||||
|
||||
// start brings a cluster up in a fresh directory and takes it down at the end
|
||||
// of the spec container. Package directories are read-only, so the cleanup
|
||||
// restores permissions first.
|
||||
func start() *cluster.Cluster {
|
||||
dir, err := os.MkdirTemp("", "foundation-cluster-*")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
c, err := cluster.New(dir, paths)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() {
|
||||
c.Down()
|
||||
_ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, e error) error {
|
||||
if e == nil && d.IsDir() {
|
||||
_ = os.Chmod(p, 0o700)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
_ = os.RemoveAll(dir)
|
||||
})
|
||||
return c
|
||||
}
|
||||
|
||||
// ---- small API helpers over the HTTP client ----
|
||||
|
||||
type apiError struct {
|
||||
Error struct {
|
||||
Type string `json:"type"`
|
||||
Message string `json:"message"`
|
||||
OperationID string `json:"operationId"`
|
||||
} `json:"error"`
|
||||
}
|
||||
|
||||
var counter int
|
||||
|
||||
func nextID(prefix string) string {
|
||||
counter++
|
||||
return fmt.Sprintf("%s-%d-%d", prefix, time.Now().UnixNano()%1_000_000, counter)
|
||||
}
|
||||
|
||||
func dispatch(c cluster.Client, request map[string]any) *cluster.Response {
|
||||
if _, ok := request["operationId"]; !ok {
|
||||
request["operationId"] = nextID("op")
|
||||
}
|
||||
if _, ok := request["arguments"]; !ok {
|
||||
request["arguments"] = map[string]any{}
|
||||
}
|
||||
r, err := c.Do("POST", "/api/addons/dispatch", request, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}
|
||||
|
||||
type app struct {
|
||||
ID string `json:"id"`
|
||||
Generation uint64 `json:"generation"`
|
||||
Enabled bool `json:"enabled"`
|
||||
State string `json:"state"`
|
||||
Tools []string `json:"tools"`
|
||||
HasUI bool `json:"hasUI"`
|
||||
UI *struct {
|
||||
Entry string `json:"entry"`
|
||||
CSS []string `json:"css"`
|
||||
Assets *struct {
|
||||
Digest string `json:"digest"`
|
||||
Base string `json:"base"`
|
||||
} `json:"assets"`
|
||||
} `json:"ui"`
|
||||
Version string `json:"version"`
|
||||
}
|
||||
|
||||
func apps(c cluster.Client) (int, []app) {
|
||||
r, err := c.Do("GET", "/api/addons/apps", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
var out struct {
|
||||
Apps []app `json:"apps"`
|
||||
}
|
||||
if r.Status == 200 {
|
||||
Expect(r.JSON(&out)).To(Succeed(), r.String())
|
||||
}
|
||||
return r.Status, out.Apps
|
||||
}
|
||||
|
||||
func appOf(c cluster.Client, id string) *app {
|
||||
status, list := apps(c)
|
||||
Expect(status).To(Equal(200))
|
||||
for i := range list {
|
||||
if list[i].ID == id {
|
||||
return &list[i]
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// settle polls a condition with a deadline; the condition is a fact of the
|
||||
// system under test, never a stand-in for synchronisation.
|
||||
func settle(what string, timeout time.Duration, f func() bool) {
|
||||
GinkgoHelper()
|
||||
deadline := time.Now().Add(timeout)
|
||||
for time.Now().Before(deadline) {
|
||||
if f() {
|
||||
return
|
||||
}
|
||||
time.Sleep(150 * time.Millisecond)
|
||||
}
|
||||
Fail("timed out waiting for " + what)
|
||||
}
|
||||
|
||||
func sha(b []byte) string { h := sha256.Sum256(b); return hex.EncodeToString(h[:]) }
|
||||
|
||||
// hashTree digests a file or directory so a spec can prove it did not change.
|
||||
func hashTree(path string) string {
|
||||
h := sha256.New()
|
||||
info, err := os.Stat(path)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if !info.IsDir() {
|
||||
b, err := os.ReadFile(path)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return sha(b)
|
||||
}
|
||||
Expect(filepath.WalkDir(path, func(p string, d fs.DirEntry, e error) error {
|
||||
if e != nil || d.IsDir() {
|
||||
return e
|
||||
}
|
||||
rel, _ := filepath.Rel(path, p)
|
||||
b, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = fmt.Fprintf(h, "%s\x00%s\x00", rel, sha(b))
|
||||
return nil
|
||||
})).To(Succeed())
|
||||
return hex.EncodeToString(h.Sum(nil))
|
||||
}
|
||||
@@ -0,0 +1,506 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package foundation_test
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// lifecycle submits one administrator lifecycle action.
|
||||
func lifecycle(cl cluster.Client, id, action string, generation uint64, version string) *cluster.Response {
|
||||
args := map[string]any{}
|
||||
if version != "" {
|
||||
args = map[string]any{"sourceId": "fixture", "version": version}
|
||||
}
|
||||
return dispatch(cl, map[string]any{"addonId": id, "action": action, "generation": generation, "arguments": args})
|
||||
}
|
||||
|
||||
// resource registers one resource on the owner and returns its identifier.
|
||||
func resource(c *cluster.Cluster, addon, owner string, grants map[string]string) string {
|
||||
GinkgoHelper()
|
||||
r, err := c.Control("resource", map[string]any{"addonId": addon, "owner": owner, "grants": grants})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
var out struct {
|
||||
ResourceID string `json:"resourceId"`
|
||||
}
|
||||
Expect(r.JSON(&out)).To(Succeed())
|
||||
return out.ResourceID
|
||||
}
|
||||
|
||||
type result struct {
|
||||
Payload struct {
|
||||
Actor string `json:"actor"`
|
||||
Arguments map[string]any `json:"arguments"`
|
||||
} `json:"payload"`
|
||||
Outputs []struct {
|
||||
Handle string `json:"handle"`
|
||||
Name string `json:"name"`
|
||||
MediaType string `json:"mediaType"`
|
||||
Size int64 `json:"size"`
|
||||
} `json:"outputs"`
|
||||
}
|
||||
|
||||
// read is a typed read of a resource through one client.
|
||||
func read(cl cluster.Client, addon, rid string, generation uint64, args map[string]any) *cluster.Response {
|
||||
return dispatch(cl, map[string]any{"addonId": addon, "resourceId": rid, "generation": generation, "action": "read", "arguments": args})
|
||||
}
|
||||
|
||||
func actorOf(r *cluster.Response) string {
|
||||
GinkgoHelper()
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
var out result
|
||||
Expect(r.JSON(&out)).To(Succeed())
|
||||
return out.Payload.Actor
|
||||
}
|
||||
|
||||
func errorType(r *cluster.Response) string {
|
||||
var e apiError
|
||||
_ = r.JSON(&e)
|
||||
return e.Error.Type
|
||||
}
|
||||
|
||||
// lines counts the non-empty lines of a durable fixture log.
|
||||
func lines(path string) int {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
n := 0
|
||||
for _, l := range strings.Split(string(b), "\n") {
|
||||
if strings.TrimSpace(l) != "" {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// fixtureChildren counts the live fixture processes that are children of pid.
|
||||
func fixtureChildren(pid int) int {
|
||||
n := 0
|
||||
for _, cmdline := range cluster.ChildrenOf(pid) {
|
||||
if strings.Contains(cmdline, "bin/fixture") {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func dataDir(c *cluster.Cluster, id string) string {
|
||||
return filepath.Join(c.Config.Nodes[cluster.Owner].Root, "data", id)
|
||||
}
|
||||
|
||||
// retainData writes a marker the add-on's retained data must keep.
|
||||
func retainData(c *cluster.Cluster, id string) string {
|
||||
GinkgoHelper()
|
||||
dir := dataDir(c, id)
|
||||
Expect(os.MkdirAll(dir, 0o700)).To(Succeed())
|
||||
marker := filepath.Join(dir, "rows.db")
|
||||
Expect(os.WriteFile(marker, []byte("retained rows"), 0o600)).To(Succeed())
|
||||
return marker
|
||||
}
|
||||
|
||||
func expectRetained(marker string) {
|
||||
GinkgoHelper()
|
||||
b, err := os.ReadFile(marker)
|
||||
Expect(err).NotTo(HaveOccurred(), "retained data must survive")
|
||||
Expect(string(b)).To(Equal("retained rows"))
|
||||
}
|
||||
|
||||
var forged = map[string]string{
|
||||
"X-Forwarded-Actor": "fnd-admin", "X-LocalAI-Actor": "fnd-admin", "X-LocalAI-User": "fnd-admin",
|
||||
"X-Actor-Id": "fnd-admin", "X-Forwarded-User": "fnd-admin", "X-Remote-User": "fnd-admin",
|
||||
"X-LocalAI-Credential": "key:forged", "X-LocalAI-Authority": `{"actor":{"userId":"fnd-admin"},"scope":"read"}`,
|
||||
}
|
||||
|
||||
var _ = Describe("foundation fixture through the owner and two frontends", Ordered, func() {
|
||||
var (
|
||||
c *cluster.Cluster
|
||||
admin cluster.Client
|
||||
hostBin string
|
||||
hostUI string
|
||||
binSum string
|
||||
uiSum string
|
||||
toolRes string
|
||||
uiRes string
|
||||
svcRes string
|
||||
exportID string
|
||||
)
|
||||
front := []string{cluster.FrontA, cluster.FrontB}
|
||||
|
||||
BeforeAll(func() {
|
||||
hostBin = paths.Server
|
||||
hostUI = filepath.Join(repoRoot, "core/http/react-ui/dist")
|
||||
if _, err := os.Stat(hostUI); err != nil {
|
||||
hostUI = ""
|
||||
}
|
||||
binSum = hashTree(hostBin)
|
||||
if hostUI != "" {
|
||||
uiSum = hashTree(hostUI)
|
||||
}
|
||||
c = start()
|
||||
Expect(c.Library.Add(cluster.UIPackage("1.0.0", nil), c.Effects)).To(Succeed())
|
||||
Expect(c.Library.Add(cluster.UIPackage("2.0.0", nil), c.Effects)).To(Succeed())
|
||||
Expect(c.Library.Add(cluster.ToolPackage("1.0.0", nil), c.Effects)).To(Succeed())
|
||||
Expect(c.Library.Add(cluster.ServicePackage("1.0.0", map[string]any{"mode": "lose-reply"}), c.Effects)).To(Succeed())
|
||||
Expect(c.Up()).To(Succeed())
|
||||
admin = c.Token(cluster.FrontA, "admin")
|
||||
})
|
||||
|
||||
It("installs the UI, tool and service-only packages from the catalog through a frontend without starting them and leaves the host binary and UI build unchanged", func() {
|
||||
for _, id := range []string{"found-ui", "found-tool", "found-svc"} {
|
||||
r := lifecycle(admin, id, "install", 0, "1.0.0")
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
}
|
||||
status, list := apps(admin)
|
||||
Expect(status).To(Equal(200))
|
||||
Expect(list).To(HaveLen(3))
|
||||
for _, a := range list {
|
||||
Expect(a.Enabled).To(BeFalse(), a.ID)
|
||||
Expect(a.State).To(Equal("stopped"), a.ID)
|
||||
}
|
||||
Expect(fixtureChildren(c.Nodes[cluster.Owner].PID())).To(BeZero(), "installing must not start any process")
|
||||
Expect(hashTree(hostBin)).To(Equal(binSum), "the host binary must not change")
|
||||
if hostUI != "" {
|
||||
Expect(hashTree(hostUI)).To(Equal(uiSum), "the shipped UI build must not change")
|
||||
}
|
||||
})
|
||||
|
||||
It("activates each package through the other frontend and runs one process per add-on under the owner only", func() {
|
||||
viaB := c.Token(cluster.FrontB, "admin")
|
||||
for _, id := range []string{"found-ui", "found-tool", "found-svc"} {
|
||||
r := lifecycle(viaB, id, "enable", 1, "")
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
}
|
||||
settle("three running add-ons", 30*time.Second, func() bool {
|
||||
_, list := apps(admin)
|
||||
n := 0
|
||||
for _, a := range list {
|
||||
if a.State == "running" {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n == 3
|
||||
})
|
||||
Expect(fixtureChildren(c.Nodes[cluster.Owner].PID())).To(Equal(3))
|
||||
for _, f := range front {
|
||||
Expect(fixtureChildren(c.Nodes[f].PID())).To(BeZero(), f+" must never start an add-on")
|
||||
Expect(filepath.Join(c.Config.Nodes[f].Root)).NotTo(BeADirectory(), f+" must not open add-on storage")
|
||||
}
|
||||
toolRes = resource(c, "found-tool", "member", map[string]string{"friend": "viewer"})
|
||||
uiRes = resource(c, "found-ui", "member", map[string]string{"friend": "viewer"})
|
||||
svcRes = resource(c, "found-svc", "member", nil)
|
||||
})
|
||||
|
||||
It("lists the same apps from the owner and both frontends and shows only what each declared", func() {
|
||||
var seen [][]string
|
||||
for _, n := range []string{cluster.Owner, cluster.FrontA, cluster.FrontB} {
|
||||
status, list := apps(c.Token(n, "member"))
|
||||
Expect(status).To(Equal(200), n)
|
||||
byID := map[string]app{}
|
||||
ids := []string{}
|
||||
for _, a := range list {
|
||||
byID[a.ID] = a
|
||||
ids = append(ids, a.ID+":"+a.State)
|
||||
}
|
||||
seen = append(seen, ids)
|
||||
Expect(byID["found-ui"].HasUI).To(BeTrue())
|
||||
Expect(byID["found-tool"].HasUI).To(BeFalse(), "the tool package declares no UI")
|
||||
Expect(byID["found-tool"].UI).To(BeNil())
|
||||
Expect(byID["found-tool"].Tools).To(ConsistOf("echo", "export"))
|
||||
Expect(byID["found-svc"].HasUI).To(BeFalse())
|
||||
Expect(byID["found-svc"].Tools).To(BeEmpty(), "the service-only package declares no tool")
|
||||
}
|
||||
Expect(seen[1]).To(ConsistOf(seen[0]))
|
||||
Expect(seen[2]).To(ConsistOf(seen[0]))
|
||||
})
|
||||
|
||||
It("serves reads of the UI, tool and service-only packages alternately through the owner and both frontends", func() {
|
||||
nodes := []string{cluster.FrontA, cluster.FrontB, cluster.Owner, cluster.FrontB, cluster.FrontA, cluster.Owner}
|
||||
for i, n := range nodes {
|
||||
Expect(actorOf(read(c.Token(n, "member"), "found-ui", uiRes, 1, map[string]any{"tool": "echo", "i": i}))).To(Equal("fnd-member"), n)
|
||||
Expect(actorOf(read(c.Token(n, "friend"), "found-tool", toolRes, 1, map[string]any{"tool": "echo"}))).To(Equal("fnd-friend"), n)
|
||||
Expect(actorOf(read(c.Token(n, "member"), "found-svc", svcRes, 1, map[string]any{"q": "plain"}))).To(Equal("fnd-member"), n)
|
||||
}
|
||||
})
|
||||
|
||||
It("denies an account without the feature, a stranger to the resource and an anonymous caller the same way on every node", func() {
|
||||
for _, n := range []string{cluster.Owner, cluster.FrontA, cluster.FrontB} {
|
||||
r := read(c.Token(n, "outsider"), "found-tool", toolRes, 1, map[string]any{"tool": "echo"})
|
||||
Expect(r.Status).To(Equal(403), n+" outsider: "+r.String())
|
||||
// The friend has a grant on the tool resource only.
|
||||
r = read(c.Token(n, "friend"), "found-svc", svcRes, 1, map[string]any{})
|
||||
Expect(r.Status).To(Equal(403), n+" friend on a foreign resource: "+r.String())
|
||||
anon, err := cluster.Client{Base: c.Nodes[n].URL()}.Do("POST", "/api/addons/dispatch", `{"addonId":"found-tool","action":"read"}`, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(anon.Status).To(BeElementOf(401, 403), n)
|
||||
}
|
||||
})
|
||||
|
||||
It("takes the actor from the authenticated credential only and ignores forged actor and authority headers", func() {
|
||||
for _, n := range []string{cluster.FrontA, cluster.FrontB, cluster.Owner} {
|
||||
cl := c.Token(n, "member")
|
||||
cl.Headers = forged
|
||||
Expect(actorOf(read(cl, "found-tool", toolRes, 1, map[string]any{"tool": "echo"}))).To(Equal("fnd-member"), n)
|
||||
// A forged administrator cannot reach lifecycle either.
|
||||
r := lifecycle(cl, "found-tool", "disable", 1, "")
|
||||
Expect(r.Status).To(Equal(403), n)
|
||||
}
|
||||
// A body that claims another actor is not part of the request type.
|
||||
r := dispatch(c.Token(cluster.FrontA, "member"), map[string]any{"addonId": "found-tool", "resourceId": toolRes, "generation": 1, "action": "read", "actor": "fnd-admin", "arguments": map[string]any{"tool": "echo"}})
|
||||
Expect(r.Status).To(Equal(400), r.String())
|
||||
})
|
||||
|
||||
It("denies a tool the add-on did not declare and a generation that is not current on every node", func() {
|
||||
for _, n := range []string{cluster.FrontA, cluster.FrontB, cluster.Owner} {
|
||||
m := c.Token(n, "member")
|
||||
r := read(m, "found-tool", toolRes, 1, map[string]any{"tool": "drop-everything"})
|
||||
Expect(r.Status).To(Equal(403), n+": "+r.String())
|
||||
r = read(m, "found-tool", toolRes, 7, map[string]any{"tool": "echo"})
|
||||
Expect(r.Status).To(Equal(403), n+": "+r.String())
|
||||
// Without a tool name only the generation of the running add-on
|
||||
// can refuse the call.
|
||||
r = read(m, "found-tool", toolRes, 7, map[string]any{"q": "plain"})
|
||||
Expect(r.Status).To(Equal(403), n+" plain read of a stale generation: "+r.String())
|
||||
}
|
||||
})
|
||||
|
||||
It("relays JSON and a private file result and downloads the file through a different frontend than the one that produced it", func() {
|
||||
m := c.Token(cluster.FrontA, "member")
|
||||
r := read(m, "found-tool", toolRes, 1, map[string]any{"tool": "export"})
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
var out result
|
||||
Expect(r.JSON(&out)).To(Succeed())
|
||||
Expect(out.Outputs).To(HaveLen(1))
|
||||
exportID = out.Outputs[0].Handle
|
||||
Expect(exportID).NotTo(Equal("report"), "the host never hands out the add-on's own handle")
|
||||
Expect(out.Outputs[0].Name).To(Equal("report.txt"))
|
||||
|
||||
viaB := c.Token(cluster.FrontB, "member")
|
||||
// The file result is transferred in the background: the read returns
|
||||
// the artifact as transferring, and a download answers 403 until it
|
||||
// is complete. Wait on the documented status lookup, bounded, so a
|
||||
// loaded machine cannot turn the transfer time into a failure.
|
||||
Eventually(func(g Gomega) string {
|
||||
st, err := viaB.Do("GET", "/api/addons/artifacts/"+exportID, nil, nil)
|
||||
g.Expect(err).NotTo(HaveOccurred())
|
||||
g.Expect(st.Status).To(Equal(200), st.String())
|
||||
var a struct {
|
||||
State string `json:"state"`
|
||||
}
|
||||
g.Expect(st.JSON(&a)).To(Succeed())
|
||||
return a.State
|
||||
}, 30*time.Second, 100*time.Millisecond).Should(Equal("complete"))
|
||||
dl, err := viaB.Do("GET", "/api/addons/artifacts/"+exportID+"/download", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(dl.Status).To(Equal(200))
|
||||
Expect(dl.String()).To(Equal("fixture\n"))
|
||||
Expect(dl.Header.Get("Content-Disposition")).To(ContainSubstring("attachment"))
|
||||
// Another account, even one with a grant on the resource, is denied
|
||||
// the bytes of someone else's result.
|
||||
other, err := c.Token(cluster.FrontB, "friend").Do("GET", "/api/addons/artifacts/"+exportID+"/download", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(other.Status).To(Equal(403))
|
||||
|
||||
j, err := viaB.Do("POST", "/api/addons/relay", map[string]any{"kind": "json", "request": map[string]any{
|
||||
"addonId": "found-tool", "resourceId": toolRes, "generation": 1, "operationId": nextID("json"), "action": "read", "arguments": map[string]any{"tool": "echo", "shape": "json"}}}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(j.Status).To(Equal(200), j.String())
|
||||
Expect(j.String()).To(ContainSubstring("fnd-member"))
|
||||
})
|
||||
|
||||
It("relays lifecycle status and progress of an operation to the administrator who made it and to nobody else", func() {
|
||||
r := lifecycle(admin, "found-tool", "enable", 1, "")
|
||||
// The add-on is already enabled: an identical enable is an idempotent
|
||||
// no-op or a conflict, but never a server error.
|
||||
Expect(r.Status).To(BeNumerically("<", 500), r.String())
|
||||
op := nextID("progress")
|
||||
r = dispatch(admin, map[string]any{"operationId": op, "addonId": "found-svc", "action": "enable", "generation": 1})
|
||||
Expect(r.Status).To(BeNumerically("<", 500), r.String())
|
||||
for _, kind := range []string{"status", "progress"} {
|
||||
rel, err := c.Token(cluster.FrontB, "admin").Do("POST", "/api/addons/relay", map[string]any{"kind": kind, "request": map[string]any{
|
||||
"addonId": "found-svc", "generation": 1, "operationId": op, "action": "operation-status", "arguments": map[string]any{}}}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(rel.Status).To(Equal(200), kind+": "+rel.String())
|
||||
Expect(rel.String()).To(ContainSubstring("completed"), kind)
|
||||
// Another account never sees the operation.
|
||||
no, err := c.Token(cluster.FrontB, "member").Do("POST", "/api/addons/relay", map[string]any{"kind": kind, "request": map[string]any{
|
||||
"addonId": "found-svc", "generation": 1, "operationId": op, "action": "operation-status", "arguments": map[string]any{}}}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(no.Status).To(Equal(403), kind)
|
||||
}
|
||||
})
|
||||
|
||||
It("serves the immutable UI assets by package digest through a non-owner frontend and refuses everything else", func() {
|
||||
a := appOf(c.Token(cluster.FrontA, "member"), "found-ui")
|
||||
Expect(a).NotTo(BeNil())
|
||||
Expect(a.UI).NotTo(BeNil())
|
||||
for _, n := range front {
|
||||
r, err := c.Token(n, "member").Do("GET", a.UI.Entry, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), n)
|
||||
Expect(r.String()).To(ContainSubstring("createAddon"))
|
||||
Expect(r.Header.Get("Content-Type")).To(ContainSubstring("javascript"))
|
||||
Expect(r.Header.Get("Cache-Control")).To(ContainSubstring("immutable"))
|
||||
// The same asset without a credential is refused.
|
||||
anon, err := cluster.Client{Base: c.Nodes[n].URL()}.Do("GET", a.UI.Entry, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(anon.Status).To(BeElementOf(401, 403))
|
||||
out, err := c.Token(n, "outsider").Do("GET", a.UI.Entry, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(out.Status).To(Equal(403))
|
||||
}
|
||||
base := strings.TrimSuffix(a.UI.Entry, "ui.js")
|
||||
for _, bad := range []string{base + "private.txt", base + "../manifest.json", base + "bin/fixture", strings.Replace(base, a.UI.Entry[len("/api/addons/found-ui/assets/"):len("/api/addons/found-ui/assets/")+64], strings.Repeat("0", 64), 1) + "ui.js"} {
|
||||
r, err := c.Token(cluster.FrontB, "member").Do("GET", bad, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(BeElementOf(400, 403, 404), bad)
|
||||
Expect(r.String()).NotTo(ContainSubstring("not declared"))
|
||||
}
|
||||
// The tool package has no UI: there is nothing to serve.
|
||||
r, err := c.Token(cluster.FrontB, "member").Do("GET", "/api/addons/found-tool/assets/"+strings.Repeat("a", 64)+"/ui.js", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(BeElementOf(403, 404))
|
||||
})
|
||||
|
||||
It("applies one confirmed change exactly once through a different frontend than the one that proposed it", func() {
|
||||
pid := nextID("write")
|
||||
r, err := c.Token(cluster.FrontA, "member").Do("POST", "/api/addons/proposals", map[string]any{
|
||||
"addonId": "found-tool", "resourceId": toolRes, "generation": 1, "operationId": pid, "action": "write", "arguments": map[string]any{"note": "one"}}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
var p struct {
|
||||
ID, Digest, State string
|
||||
}
|
||||
Expect(r.JSON(&p)).To(Succeed())
|
||||
Expect(p.State).To(Equal("pending"))
|
||||
// A bearer key can never confirm.
|
||||
bad, err := c.Token(cluster.FrontB, "member").Do("POST", "/api/addons/proposals/"+pid+"/confirm", map[string]any{"digest": p.Digest}, map[string]string{"X-LocalAI-Confirm": pid})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(bad.Status).To(Equal(403))
|
||||
// Another account cannot confirm it either.
|
||||
other, err := c.Session(cluster.FrontB, "friend").Do("POST", "/api/addons/proposals/"+pid+"/confirm", map[string]any{"digest": p.Digest}, map[string]string{"X-LocalAI-Confirm": pid})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(other.Status).To(Equal(403))
|
||||
Expect(lines(filepath.Join(c.Effects, "found-tool.log"))).To(BeZero())
|
||||
|
||||
confirm := func(node string) *cluster.Response {
|
||||
resp, err := c.Session(node, "member").Do("POST", "/api/addons/proposals/"+pid+"/confirm", map[string]any{"digest": p.Digest}, map[string]string{"X-LocalAI-Confirm": pid})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return resp
|
||||
}
|
||||
first := confirm(cluster.FrontB)
|
||||
Expect(first.Status).To(Equal(200), first.String())
|
||||
Expect(first.String()).To(ContainSubstring(`"state":"applied"`))
|
||||
again := confirm(cluster.FrontA)
|
||||
Expect(again.Status).To(Equal(200), again.String())
|
||||
Expect(again.String()).To(ContainSubstring(`"state":"applied"`))
|
||||
Expect(lines(filepath.Join(c.Effects, "found-tool.log"))).To(Equal(1), "one committed effect")
|
||||
Expect(lines(filepath.Join(c.Effects, "found-tool.log.applies"))).To(Equal(1), "one apply request reached the add-on")
|
||||
})
|
||||
|
||||
It("reports a lost reply as outcome unknown with the original operation ID and settles it by receipt lookup without a second apply", func() {
|
||||
pid := nextID("lost")
|
||||
r, err := c.Token(cluster.FrontA, "member").Do("POST", "/api/addons/proposals", map[string]any{
|
||||
"addonId": "found-svc", "resourceId": svcRes, "generation": 1, "operationId": pid, "action": "write", "arguments": map[string]any{"note": "lost"}}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
var p struct{ Digest string }
|
||||
Expect(r.JSON(&p)).To(Succeed())
|
||||
confirm := func(node string) *cluster.Response {
|
||||
resp, err := c.Session(node, "member").Do("POST", "/api/addons/proposals/"+pid+"/confirm", map[string]any{"digest": p.Digest}, map[string]string{"X-LocalAI-Confirm": pid})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return resp
|
||||
}
|
||||
// The owner and each frontend report the same original operation.
|
||||
first := confirm(cluster.FrontB)
|
||||
Expect(first.Status).To(Equal(503), first.String())
|
||||
var e apiError
|
||||
Expect(first.JSON(&e)).To(Succeed())
|
||||
Expect(e.Error.Type).To(Equal("outcome_unknown"))
|
||||
Expect(e.Error.OperationID).To(Equal(pid))
|
||||
Expect(lines(filepath.Join(c.Effects, "found-svc.log"))).To(Equal(1), "the change did commit")
|
||||
// Repeating the confirmation from every node never submits it again.
|
||||
for _, n := range []string{cluster.FrontA, cluster.Owner, cluster.FrontB} {
|
||||
rep := confirm(n)
|
||||
if rep.Status == 503 {
|
||||
Expect(errorType(rep)).To(Equal("outcome_unknown"), n)
|
||||
var again apiError
|
||||
Expect(rep.JSON(&again)).To(Succeed())
|
||||
Expect(again.Error.OperationID).To(Equal(pid), n)
|
||||
}
|
||||
}
|
||||
Expect(lines(filepath.Join(c.Effects, "found-svc.log.applies"))).To(Equal(1), "no second apply may reach the add-on")
|
||||
// A status lookup is read-only and settles by receipt.
|
||||
rec, err := c.Session(cluster.FrontA, "member").Do("POST", "/api/addons/proposals/"+pid+"/reconcile", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(rec.Status).To(Equal(200), rec.String())
|
||||
Expect(rec.String()).To(ContainSubstring(`"state":"applied"`))
|
||||
Expect(lines(filepath.Join(c.Effects, "found-svc.log"))).To(Equal(1))
|
||||
Expect(lines(filepath.Join(c.Effects, "found-svc.log.applies"))).To(Equal(1))
|
||||
})
|
||||
|
||||
It("upgrades to a new generation, denies the previous generation and its assets, and keeps the retained data", func() {
|
||||
marker := retainData(c, "found-ui")
|
||||
before := appOf(admin, "found-ui")
|
||||
Expect(before.Generation).To(Equal(uint64(1)))
|
||||
r := lifecycle(admin, "found-ui", "upgrade", 0, "2.0.0")
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
settle("the upgraded add-on running", 30*time.Second, func() bool {
|
||||
a := appOf(admin, "found-ui")
|
||||
return a != nil && a.Generation == 2 && a.State == "running"
|
||||
})
|
||||
expectRetained(marker)
|
||||
for _, n := range []string{cluster.FrontA, cluster.FrontB, cluster.Owner} {
|
||||
stale := read(c.Token(n, "member"), "found-ui", uiRes, 1, map[string]any{"q": "plain"})
|
||||
Expect(stale.Status).To(Equal(403), n+" stale generation: "+stale.String())
|
||||
Expect(actorOf(read(c.Token(n, "member"), "found-ui", uiRes, 2, map[string]any{"tool": "echo"}))).To(Equal("fnd-member"), n)
|
||||
}
|
||||
old, err := c.Token(cluster.FrontB, "member").Do("GET", before.UI.Entry, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(old.Status).To(BeElementOf(403, 404), "the previous package digest is no longer served")
|
||||
now := appOf(c.Token(cluster.FrontB, "member"), "found-ui")
|
||||
Expect(now.UI.Entry).NotTo(Equal(before.UI.Entry))
|
||||
cur, err := c.Token(cluster.FrontB, "member").Do("GET", now.UI.Entry, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(cur.Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("disables, removes and reinstalls an add-on while its data and grants remain", func() {
|
||||
marker := retainData(c, "found-tool")
|
||||
Expect(lifecycle(admin, "found-tool", "disable", 1, "").Status).To(Equal(200))
|
||||
for _, n := range []string{cluster.FrontA, cluster.FrontB, cluster.Owner} {
|
||||
r := read(c.Token(n, "member"), "found-tool", toolRes, 1, map[string]any{"tool": "echo"})
|
||||
Expect(r.Status).To(BeElementOf(403, 503), n+" disabled: "+r.String())
|
||||
}
|
||||
Expect(fixtureChildren(c.Nodes[cluster.Owner].PID())).To(Equal(2))
|
||||
Expect(lifecycle(admin, "found-tool", "uninstall", 0, "").Status).To(Equal(200))
|
||||
expectRetained(marker)
|
||||
Expect(lifecycle(admin, "found-tool", "install", 0, "1.0.0").Status).To(Equal(200))
|
||||
cur := appOf(admin, "found-tool")
|
||||
Expect(cur).NotTo(BeNil())
|
||||
Expect(lifecycle(admin, "found-tool", "enable", cur.Generation, "").Status).To(Equal(200))
|
||||
settle("the reinstalled add-on running", 30*time.Second, func() bool {
|
||||
a := appOf(admin, "found-tool")
|
||||
return a != nil && a.State == "running"
|
||||
})
|
||||
expectRetained(marker)
|
||||
g := appOf(admin, "found-tool").Generation
|
||||
// The grants of the resource outlived the removal: the friend reads it.
|
||||
Expect(actorOf(read(c.Token(cluster.FrontB, "friend"), "found-tool", toolRes, g, map[string]any{"tool": "echo"}))).To(Equal("fnd-friend"))
|
||||
})
|
||||
|
||||
It("never changes the host binary or the shipped UI build over the whole lifecycle", func() {
|
||||
Expect(hashTree(hostBin)).To(Equal(binSum))
|
||||
if hostUI != "" {
|
||||
Expect(hashTree(hostUI)).To(Equal(uiSum))
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,129 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package foundation_test
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// A new generation or an enabled add-on whose process cannot start, and an
|
||||
// owner that dies while one is starting, must leave the last good code, the
|
||||
// retained data and a usable lifecycle.
|
||||
var _ = Describe("failed starts and interrupted lifecycle", func() {
|
||||
It("keeps the last good generation running and its data when an upgrade fails to start", func() {
|
||||
c, res := serving([]cluster.Package{
|
||||
cluster.ToolPackage("1.0.0", nil),
|
||||
cluster.ToolPackage("2.0.0", map[string]any{"mode": "exit-start"}),
|
||||
cluster.ToolPackage("3.0.0", nil),
|
||||
}, map[string]string{"found-tool": "1.0.0"})
|
||||
admin := c.Token(cluster.FrontB, "admin")
|
||||
marker := retainData(c, "found-tool")
|
||||
r := lifecycle(admin, "found-tool", "upgrade", 0, "2.0.0")
|
||||
Expect(r.Status).NotTo(Equal(200), "an upgrade whose process cannot start must not report success: "+r.String())
|
||||
a := appOf(admin, "found-tool")
|
||||
Expect(a).NotTo(BeNil())
|
||||
Expect(a.State).To(Equal("running"))
|
||||
Expect(a.Generation).To(Equal(uint64(1)))
|
||||
Expect(fixtureChildren(c.Nodes[cluster.Owner].PID())).To(Equal(1))
|
||||
Expect(actorOf(read(c.Token(cluster.FrontA, "member"), "found-tool", res["found-tool"], 1, map[string]any{"tool": "echo"}))).To(Equal("fnd-member"))
|
||||
expectRetained(marker)
|
||||
// The failure left nothing behind that blocks a later good upgrade.
|
||||
r = lifecycle(admin, "found-tool", "upgrade", 0, "3.0.0")
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
settle("the third release running", 30*time.Second, func() bool {
|
||||
cur := appOf(admin, "found-tool")
|
||||
return cur != nil && cur.State == "running" && cur.Generation > 1
|
||||
})
|
||||
expectRetained(marker)
|
||||
})
|
||||
|
||||
It("keeps the last good code and data when the owner is killed in the middle of an upgrade and recovers on restart", func() {
|
||||
c, res := serving([]cluster.Package{
|
||||
cluster.ToolPackage("1.0.0", nil),
|
||||
cluster.ToolPackage("2.0.0", map[string]any{"mode": "never-ready"}),
|
||||
cluster.ToolPackage("3.0.0", nil),
|
||||
}, map[string]string{"found-tool": "1.0.0"})
|
||||
admin := c.Token(cluster.FrontA, "admin")
|
||||
marker := retainData(c, "found-tool")
|
||||
owner := c.Nodes[cluster.Owner]
|
||||
ownerPID := owner.PID()
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer GinkgoRecover()
|
||||
defer close(done)
|
||||
// The connection dies with the owner; the result is not the point.
|
||||
_, _ = admin.Do("POST", "/api/addons/dispatch", map[string]any{"operationId": nextID("mid"), "addonId": "found-tool", "action": "upgrade", "generation": 0, "arguments": map[string]any{"sourceId": "fixture", "version": "2.0.0"}}, nil)
|
||||
}()
|
||||
var lastGood int
|
||||
for pid := range cluster.ChildrenOf(ownerPID) {
|
||||
lastGood = pid
|
||||
}
|
||||
// The last good process is replaced by the candidate, which never
|
||||
// becomes ready: the owner is killed while that start is in flight.
|
||||
settle("the candidate generation being started", 30*time.Second, func() bool {
|
||||
for pid, cmdline := range cluster.ChildrenOf(ownerPID) {
|
||||
if pid != lastGood && strings.Contains(cmdline, "bin/fixture") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
})
|
||||
owner.Kill()
|
||||
Eventually(done, 30*time.Second).Should(BeClosed())
|
||||
settle("every add-on process ending", 20*time.Second, func() bool { return fixtureChildren(ownerPID) == 0 })
|
||||
|
||||
restartOwner(c)
|
||||
settle("the last good generation running", 60*time.Second, func() bool {
|
||||
a := appOf(c.Token(cluster.FrontB, "admin"), "found-tool")
|
||||
return a != nil && a.State == "running"
|
||||
})
|
||||
a := appOf(c.Token(cluster.FrontB, "admin"), "found-tool")
|
||||
Expect(a.Generation).To(Equal(uint64(1)), "the interrupted upgrade must not have replaced the last good code")
|
||||
Expect(a.Version).To(Equal("1.0.0"))
|
||||
Expect(actorOf(read(c.Token(cluster.FrontB, "member"), "found-tool", res["found-tool"], 1, map[string]any{"tool": "echo"}))).To(Equal("fnd-member"))
|
||||
expectRetained(marker)
|
||||
Expect(fixtureChildren(c.Nodes[cluster.Owner].PID())).To(Equal(1))
|
||||
// A later good upgrade is not blocked by the interrupted one.
|
||||
r := lifecycle(c.Token(cluster.FrontB, "admin"), "found-tool", "upgrade", 0, "3.0.0")
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
expectRetained(marker)
|
||||
})
|
||||
|
||||
It("keeps retained data and a consistent lifecycle when the owner is killed in the middle of an enable", func() {
|
||||
c := start()
|
||||
Expect(c.Library.Add(cluster.ServicePackage("1.0.0", map[string]any{"mode": "never-ready"}), c.Effects)).To(Succeed())
|
||||
Expect(c.Up()).To(Succeed())
|
||||
admin := c.Token(cluster.FrontA, "admin")
|
||||
Expect(lifecycle(admin, "found-svc", "install", 0, "1.0.0").Status).To(Equal(200))
|
||||
marker := retainData(c, "found-svc")
|
||||
owner := c.Nodes[cluster.Owner]
|
||||
ownerPID := owner.PID()
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer GinkgoRecover()
|
||||
defer close(done)
|
||||
_, _ = admin.Do("POST", "/api/addons/dispatch", map[string]any{"operationId": nextID("enable"), "addonId": "found-svc", "action": "enable", "generation": 1, "arguments": map[string]any{}}, nil)
|
||||
}()
|
||||
settle("the process being started", 30*time.Second, func() bool { return fixtureChildren(ownerPID) == 1 })
|
||||
owner.Kill()
|
||||
Eventually(done, 30*time.Second).Should(BeClosed())
|
||||
settle("the process ending", 20*time.Second, func() bool { return fixtureChildren(ownerPID) == 0 })
|
||||
|
||||
restartOwner(c)
|
||||
a := appOf(c.Token(cluster.FrontB, "admin"), "found-svc")
|
||||
Expect(a).NotTo(BeNil(), "the installed add-on is not lost")
|
||||
Expect(a.State).NotTo(Equal("running"), "a process that never became ready is never reported as running")
|
||||
expectRetained(marker)
|
||||
// The lifecycle is not wedged: the add-on can be disabled and removed.
|
||||
for _, action := range []string{"disable", "uninstall"} {
|
||||
r := lifecycle(c.Token(cluster.FrontB, "admin"), "found-svc", action, a.Generation, "")
|
||||
Expect(r.Status).To(Equal(200), action+": "+r.String())
|
||||
}
|
||||
expectRetained(marker)
|
||||
Expect(fixtureChildren(c.Nodes[cluster.Owner].PID())).To(BeZero())
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,186 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package foundation_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// serving brings up a cluster with the given packages published, the named
|
||||
// add-ons installed and enabled, and one resource of each registered.
|
||||
func serving(pkgs []cluster.Package, install map[string]string) (*cluster.Cluster, map[string]string) {
|
||||
GinkgoHelper()
|
||||
c := start()
|
||||
for _, p := range pkgs {
|
||||
Expect(c.Library.Add(p, c.Effects)).To(Succeed())
|
||||
}
|
||||
Expect(c.Up()).To(Succeed())
|
||||
admin := c.Token(cluster.FrontA, "admin")
|
||||
resources := map[string]string{}
|
||||
for id, version := range install {
|
||||
Expect(lifecycle(admin, id, "install", 0, version).Status).To(Equal(200))
|
||||
Expect(lifecycle(admin, id, "enable", 1, "").Status).To(Equal(200))
|
||||
resources[id] = resource(c, id, "member", map[string]string{"friend": "viewer"})
|
||||
}
|
||||
settle("every add-on running", 30*time.Second, func() bool {
|
||||
_, list := apps(admin)
|
||||
n := 0
|
||||
for _, a := range list {
|
||||
if a.State == "running" {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n == len(install)
|
||||
})
|
||||
return c, resources
|
||||
}
|
||||
|
||||
func restartOwner(c *cluster.Cluster) {
|
||||
GinkgoHelper()
|
||||
Expect(c.Nodes[cluster.Owner].Start()).To(Succeed())
|
||||
Expect(c.Nodes[cluster.Owner].Ready(120 * time.Second)).To(Succeed())
|
||||
}
|
||||
|
||||
var _ = Describe("owner failure and recovery", func() {
|
||||
It("fails closed on every frontend while the owner is down, executes nothing locally and recovers the last good package and data on restart", func() {
|
||||
c, res := serving([]cluster.Package{cluster.ToolPackage("1.0.0", nil), cluster.UIPackage("1.0.0", nil)}, map[string]string{"found-tool": "1.0.0", "found-ui": "1.0.0"})
|
||||
marker := retainData(c, "found-tool")
|
||||
pkgDir := filepath.Join(c.Config.Nodes[cluster.Owner].Root, "packages")
|
||||
packageSum := hashTree(pkgDir)
|
||||
before := lines(filepath.Join(c.Effects, "found-tool.log"))
|
||||
ui := appOf(c.Token(cluster.FrontA, "member"), "found-ui")
|
||||
|
||||
owner := c.Nodes[cluster.Owner]
|
||||
ownerPID := owner.PID()
|
||||
owner.Kill()
|
||||
settle("the add-on processes ending with the owner", 20*time.Second, func() bool { return fixtureChildren(ownerPID) == 0 })
|
||||
|
||||
var bodies []string
|
||||
for _, n := range []string{cluster.FrontA, cluster.FrontB} {
|
||||
cl := c.Token(n, "member")
|
||||
r := read(cl, "found-tool", res["found-tool"], 1, map[string]any{"tool": "echo"})
|
||||
Expect(r.Status).To(Equal(503), n+": "+r.String())
|
||||
Expect(errorType(r)).To(Equal("addon_error"))
|
||||
bodies = append(bodies, r.String())
|
||||
status, _ := apps(cl)
|
||||
Expect(status).To(Equal(503), n+" apps")
|
||||
asset, err := cl.Do("GET", ui.UI.Entry, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(asset.Status).To(Equal(503), n+" asset")
|
||||
life := lifecycle(c.Token(n, "admin"), "found-tool", "disable", 1, "")
|
||||
Expect(life.Status).To(Equal(503), n+" lifecycle: "+life.String())
|
||||
prop, err := cl.Do("POST", "/api/addons/proposals", map[string]any{"addonId": "found-tool", "resourceId": res["found-tool"], "generation": 1, "operationId": nextID("down"), "action": "write", "arguments": map[string]any{}}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(prop.Status).To(Equal(503), n+" propose: "+prop.String())
|
||||
Expect(fixtureChildren(c.Nodes[n].PID())).To(BeZero(), n+" must execute nothing locally")
|
||||
Expect(c.Config.Nodes[n].Root).NotTo(BeADirectory(), n+" must not create add-on storage")
|
||||
}
|
||||
Expect(bodies[1]).To(Equal(bodies[0]), "the same typed error from every frontend")
|
||||
Expect(fixtureChildren(ownerPID)).To(BeZero())
|
||||
|
||||
restartOwner(c)
|
||||
settle("the add-ons running again", 30*time.Second, func() bool {
|
||||
a := appOf(c.Token(cluster.FrontB, "admin"), "found-tool")
|
||||
b := appOf(c.Token(cluster.FrontB, "admin"), "found-ui")
|
||||
return a != nil && b != nil && a.State == "running" && b.State == "running"
|
||||
})
|
||||
a := appOf(c.Token(cluster.FrontA, "admin"), "found-tool")
|
||||
Expect(a.Generation).To(Equal(uint64(1)), "the last good generation is the one that runs")
|
||||
for _, n := range []string{cluster.FrontA, cluster.FrontB} {
|
||||
Expect(actorOf(read(c.Token(n, "friend"), "found-tool", res["found-tool"], 1, map[string]any{"tool": "echo"}))).To(Equal("fnd-friend"))
|
||||
}
|
||||
Expect(fixtureChildren(c.Nodes[cluster.Owner].PID())).To(Equal(2), "exactly one process per add-on")
|
||||
Expect(hashTree(pkgDir)).To(Equal(packageSum), "package code is unchanged")
|
||||
expectRetained(marker)
|
||||
Expect(lines(filepath.Join(c.Effects, "found-tool.log"))).To(Equal(before))
|
||||
})
|
||||
|
||||
It("refuses a second owner on the same storage and keeps one process per add-on", func() {
|
||||
c, res := serving([]cluster.Package{cluster.ToolPackage("1.0.0", nil)}, map[string]string{"found-tool": "1.0.0"})
|
||||
owner := c.Nodes[cluster.Owner]
|
||||
Expect(fixtureChildren(owner.PID())).To(Equal(1))
|
||||
|
||||
dup, err := c.Config.Derive("dup")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
second := &cluster.Node{Name: cluster.Owner, Config: dup, ServerPath: paths.Server, MockPath: paths.Mock}
|
||||
Expect(second.Start()).To(Succeed())
|
||||
DeferCleanup(second.Stop)
|
||||
Expect(second.Wait(60*time.Second)).To(BeTrue(), "a second owner on the same storage must not keep running: "+second.Tail(8))
|
||||
Expect(fixtureChildren(second.PID())).To(BeZero())
|
||||
// The first owner keeps its single process and keeps serving.
|
||||
Expect(fixtureChildren(owner.PID())).To(Equal(1))
|
||||
Expect(actorOf(read(c.Token(cluster.FrontB, "member"), "found-tool", res["found-tool"], 1, map[string]any{"tool": "echo"}))).To(Equal("fnd-member"))
|
||||
})
|
||||
|
||||
It("stops an add-on process orphaned by an owner crash before it starts the next one", func() {
|
||||
c, res := serving([]cluster.Package{cluster.ToolPackage("1.0.0", map[string]any{"mode": "orphan"})}, map[string]string{"found-tool": "1.0.0"})
|
||||
owner := c.Nodes[cluster.Owner]
|
||||
var orphan int
|
||||
for pid := range cluster.ChildrenOf(owner.PID()) {
|
||||
orphan = pid
|
||||
}
|
||||
Expect(orphan).NotTo(BeZero())
|
||||
owner.Kill()
|
||||
// An add-on that ignores the closed startup channel outlives its owner.
|
||||
Consistently(func() bool { return cluster.Alive(orphan) }, 1500*time.Millisecond, 100*time.Millisecond).Should(BeTrue(), "the orphan mode must really orphan the process")
|
||||
|
||||
restartOwner(c)
|
||||
settle("the orphan being stopped", 20*time.Second, func() bool { return !cluster.Alive(orphan) })
|
||||
settle("the add-on serving again", 30*time.Second, func() bool {
|
||||
a := appOf(c.Token(cluster.FrontA, "admin"), "found-tool")
|
||||
return a != nil && a.State == "running"
|
||||
})
|
||||
children := cluster.ChildrenOf(c.Nodes[cluster.Owner].PID())
|
||||
Expect(children).To(HaveLen(1))
|
||||
for pid := range children {
|
||||
Expect(pid).NotTo(Equal(orphan))
|
||||
}
|
||||
Expect(actorOf(read(c.Token(cluster.FrontA, "member"), "found-tool", res["found-tool"], 1, map[string]any{"tool": "echo"}))).To(Equal("fnd-member"))
|
||||
})
|
||||
|
||||
It("leaves no process behind when a cluster with an orphaned add-on is taken down", func() {
|
||||
c, _ := serving([]cluster.Package{cluster.ToolPackage("1.0.0", map[string]any{"mode": "orphan"})}, map[string]string{"found-tool": "1.0.0"})
|
||||
owner := c.Nodes[cluster.Owner]
|
||||
first := owner.PID()
|
||||
var orphan int
|
||||
for pid := range cluster.ChildrenOf(first) {
|
||||
orphan = pid
|
||||
}
|
||||
Expect(orphan).NotTo(BeZero())
|
||||
owner.Kill()
|
||||
restartOwner(c)
|
||||
var current int
|
||||
for pid := range cluster.ChildrenOf(owner.PID()) {
|
||||
current = pid
|
||||
}
|
||||
Expect(current).NotTo(BeZero())
|
||||
// The teardown a failing spec reaches: every process must be gone.
|
||||
c.Down()
|
||||
for _, pid := range []int{first, owner.PID(), orphan, current, c.Nodes[cluster.FrontA].PID(), c.Nodes[cluster.FrontB].PID()} {
|
||||
Expect(cluster.Alive(pid)).To(BeFalse(), "process %d survived Down", pid)
|
||||
}
|
||||
c.Down()
|
||||
})
|
||||
|
||||
It("keeps the files of the package of an installed add-on read only", func() {
|
||||
c, _ := serving([]cluster.Package{cluster.ToolPackage("1.0.0", nil)}, map[string]string{"found-tool": "1.0.0"})
|
||||
root := filepath.Join(c.Config.Nodes[cluster.Owner].Root, "packages")
|
||||
var files int
|
||||
Expect(filepath.WalkDir(root, func(p string, d os.DirEntry, e error) error {
|
||||
if e != nil || d.IsDir() {
|
||||
return e
|
||||
}
|
||||
info, err := d.Info()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
files++
|
||||
Expect(info.Mode().Perm()&0o222).To(BeZero(), p+" must not be writable")
|
||||
return nil
|
||||
})).To(Succeed())
|
||||
Expect(files).To(BeNumerically(">", 0))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,279 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package foundation_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
goruntime "runtime"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/addons"
|
||||
"github.com/mudler/LocalAI/core/services/addons/catalog"
|
||||
"github.com/mudler/LocalAI/core/services/addons/packages"
|
||||
addonruntime "github.com/mudler/LocalAI/core/services/addons/runtime"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// owner is the lifecycle service of an owner, built the way the application
|
||||
// builds it (manager, installer, supervisor) over the real fixture process.
|
||||
type owner struct {
|
||||
m *addons.Manager
|
||||
s *addonruntime.Supervisor
|
||||
actor contract.Actor
|
||||
root string
|
||||
}
|
||||
|
||||
// openOwner opens or reopens the owner on root.
|
||||
func openOwner(root string, lib *cluster.Library, ready time.Duration) (*owner, error) {
|
||||
ctx := context.Background()
|
||||
actor := contract.Actor{UserID: "admin", CredentialID: "key:admin", Mode: "named-user"}
|
||||
admin := func(_ context.Context, a contract.Actor) error {
|
||||
if a != actor {
|
||||
return errors.New("denied")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
sources, err := catalog.NewSources(filepath.Join(root, "sources.json"), admin, catalog.CredentialResolver{})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := sources.Save(ctx, actor, catalog.Source{ID: "fixture", Name: "Fixture", URL: lib.URL(), Enabled: true, AllowLoopbackHTTP: true}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, err := sources.Refresh(ctx, "fixture"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cat := catalog.Catalog{Sources: sources, TargetOS: goruntime.GOOS, TargetArch: goruntime.GOARCH}
|
||||
host := contract.HostCompatibility{Host: 1, API: 1, Protocol: 1, SDK: 1, DataFormat: 1, OS: goruntime.GOOS, Arch: goruntime.GOARCH}
|
||||
installer := &packages.Installer{Root: filepath.Join(root, "packages"), Catalog: cat, Host: host}
|
||||
m, err := addons.NewManager(ctx, addons.OwnerConfig{InstanceID: "owner", OwnerID: "owner", OwnerAddress: "https://owner.example", StorageID: "disk", Root: root}, cat, installer, admin)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := m.Reconcile(ctx); err != nil {
|
||||
_ = m.Close()
|
||||
return nil, err
|
||||
}
|
||||
limits := contract.DefaultLimits()
|
||||
s, err := addonruntime.NewSupervisor(addonruntime.Config{StateRoot: filepath.Join(root, "runtime"), Limits: limits, ReadyTimeout: ready, StableHealth: limits.BackoffMax})
|
||||
if err != nil {
|
||||
_ = m.Close()
|
||||
return nil, err
|
||||
}
|
||||
if err := m.AttachRuntime(ctx, s); err != nil {
|
||||
_ = s.Close(context.Background())
|
||||
_ = m.Close()
|
||||
return nil, err
|
||||
}
|
||||
return &owner{m: m, s: s, actor: actor, root: root}, nil
|
||||
}
|
||||
|
||||
func (o *owner) submit(id, action string, generation uint64, version string) (addons.OperationStatus, error) {
|
||||
return o.m.Submit(context.Background(), o.actor, addons.LifecycleCommand{OperationID: nextID("svc"), AddonID: id, Action: action, SourceID: "fixture", Version: version, Generation: generation})
|
||||
}
|
||||
|
||||
func (o *owner) ready(id string, generation uint64) bool {
|
||||
regs, err := o.s.Registry.Snapshot(context.Background())
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
for _, r := range regs {
|
||||
if r.AddonID == id && r.Generation == generation {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func makeOwner(lib *cluster.Library, ready time.Duration) (*owner, string) {
|
||||
GinkgoHelper()
|
||||
root, err := os.MkdirTemp("", "foundation-owner-*")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() {
|
||||
_ = filepath.Walk(root, func(p string, i os.FileInfo, e error) error {
|
||||
if e == nil && i.IsDir() {
|
||||
_ = os.Chmod(p, 0o700)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
_ = os.RemoveAll(root)
|
||||
})
|
||||
o, err := openOwner(root, lib, ready)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() { _ = o.s.Close(context.Background()); _ = o.m.Close() })
|
||||
return o, root
|
||||
}
|
||||
|
||||
var _ = Describe("lifecycle service over the real fixture process", func() {
|
||||
var lib *cluster.Library
|
||||
var effects string
|
||||
|
||||
BeforeEach(func() {
|
||||
var err error
|
||||
lib, err = cluster.NewLibrary(paths.Fixture)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(lib.Close)
|
||||
effects = GinkgoT().TempDir()
|
||||
})
|
||||
|
||||
It("keeps the last good generation running and the retained data when the new generation cannot start", func() {
|
||||
Expect(lib.Add(cluster.ToolPackage("1.0.0", nil), effects)).To(Succeed())
|
||||
Expect(lib.Add(cluster.ToolPackage("2.0.0", map[string]any{"mode": "exit-start"}), effects)).To(Succeed())
|
||||
Expect(lib.Add(cluster.ToolPackage("3.0.0", nil), effects)).To(Succeed())
|
||||
o, _ := makeOwner(lib, 5*time.Second)
|
||||
ctx := context.Background()
|
||||
_, err := o.submit("found-tool", "install", 0, "1.0.0")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
_, err = o.submit("found-tool", "enable", 1, "")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(o.ready("found-tool", 1)).To(BeTrue())
|
||||
before, err := o.m.Status(ctx, "found-tool")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(os.MkdirAll(before.DataDirectory, 0o700)).To(Succeed())
|
||||
marker := filepath.Join(before.DataDirectory, "rows.db")
|
||||
Expect(os.WriteFile(marker, []byte("retained rows"), 0o600)).To(Succeed())
|
||||
firstPackage := hashTree(before.Generations[0].Receipt.Directory)
|
||||
|
||||
status, err := o.submit("found-tool", "upgrade", 0, "2.0.0")
|
||||
Expect(err).To(HaveOccurred(), "an upgrade that cannot start is a failed operation: %+v", status)
|
||||
|
||||
after, err := o.m.Status(ctx, "found-tool")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(after.Generation).To(Equal(uint64(1)), "the last good generation stays the installed one")
|
||||
Expect(after.ObservedHealth).To(Equal("ready"), "and it stays running")
|
||||
Expect(o.ready("found-tool", 1)).To(BeTrue())
|
||||
Expect(hashTree(before.Generations[0].Receipt.Directory)).To(Equal(firstPackage), "its code is untouched")
|
||||
b, err := os.ReadFile(marker)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(string(b)).To(Equal("retained rows"))
|
||||
// The failed attempt leaves nothing that blocks a good upgrade.
|
||||
_, err = o.submit("found-tool", "upgrade", 0, "3.0.0")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(o.ready("found-tool", 2)).To(BeTrue(), "the good release is the next generation")
|
||||
})
|
||||
|
||||
It("restores the last good generation and resumes the same operation when the request is canceled during an upgrade start", func() {
|
||||
Expect(lib.Add(cluster.ToolPackage("1.0.0", nil), effects)).To(Succeed())
|
||||
Expect(lib.Add(cluster.ToolPackage("2.0.0", map[string]any{"mode": "never-ready"}), effects)).To(Succeed())
|
||||
Expect(lib.Add(cluster.ToolPackage("3.0.0", nil), effects)).To(Succeed())
|
||||
o, root := makeOwner(lib, 5*time.Second)
|
||||
bg := context.Background()
|
||||
_, err := o.submit("found-tool", "install", 0, "1.0.0")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
_, err = o.submit("found-tool", "enable", 1, "")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
first, err := o.m.Runtime(bg, "found-tool")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
command := addons.LifecycleCommand{OperationID: nextID("cancel"), AddonID: "found-tool", Action: "upgrade", SourceID: "fixture", Version: "2.0.0"}
|
||||
timeout, cancel := context.WithTimeout(bg, 800*time.Millisecond)
|
||||
defer cancel()
|
||||
started := time.Now()
|
||||
status, err := o.m.Submit(timeout, o.actor, command)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(status.Phase).NotTo(Equal("failed"), "a canceled request is not a terminal failure of the operation")
|
||||
Expect(time.Since(started)).To(BeNumerically("<", 10*time.Second), "the restore is bounded")
|
||||
// The last good generation runs again, as a new process, and the
|
||||
// canceled candidate and the first process do not overlap with it.
|
||||
current, err := o.m.Runtime(bg, "found-tool")
|
||||
Expect(err).NotTo(HaveOccurred(), "the last good generation is back without another command")
|
||||
Expect(current.Generation).To(Equal(uint64(1)))
|
||||
Expect(current.PID).NotTo(Equal(first.PID))
|
||||
Expect(syscall.Kill(first.PID, 0)).NotTo(Succeed(), "the stopped process is gone")
|
||||
Expect(o.ready("found-tool", 1)).To(BeTrue())
|
||||
regs, err := o.s.Registry.Snapshot(bg)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(regs).To(HaveLen(1))
|
||||
|
||||
// The same operation ID is retried and settles on the same outcome
|
||||
// as the first attempt would have without the cancellation: the
|
||||
// candidate never becomes ready, so it is a terminal failure with
|
||||
// the last good generation kept.
|
||||
_, err = o.m.Submit(bg, o.actor, command)
|
||||
Expect(err).To(MatchError(addons.ErrOperationFailed))
|
||||
_, err = o.m.Runtime(bg, "found-tool")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
again, err := o.m.Submit(bg, o.actor, command)
|
||||
Expect(err).To(MatchError(addons.ErrOperationFailed))
|
||||
Expect(again.Phase).To(Equal("failed"))
|
||||
Expect(o.ready("found-tool", 1)).To(BeTrue())
|
||||
// The failure leaves nothing behind that blocks a good upgrade.
|
||||
_, err = o.submit("found-tool", "upgrade", 0, "3.0.0")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(o.ready("found-tool", 2)).To(BeTrue())
|
||||
_ = root
|
||||
})
|
||||
|
||||
It("converges on the last good generation after a crash that follows the rollback start and precedes any journal write", func() {
|
||||
Expect(lib.Add(cluster.ToolPackage("1.0.0", nil), effects)).To(Succeed())
|
||||
Expect(lib.Add(cluster.ToolPackage("2.0.0", map[string]any{"mode": "never-ready"}), effects)).To(Succeed())
|
||||
o, root := makeOwner(lib, 5*time.Second)
|
||||
bg := context.Background()
|
||||
_, err := o.submit("found-tool", "install", 0, "1.0.0")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
_, err = o.submit("found-tool", "enable", 1, "")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
command := addons.LifecycleCommand{OperationID: nextID("crash"), AddonID: "found-tool", Action: "upgrade", SourceID: "fixture", Version: "2.0.0"}
|
||||
timeout, cancel := context.WithTimeout(bg, 800*time.Millisecond)
|
||||
defer cancel()
|
||||
_, err = o.m.Submit(timeout, o.actor, command)
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(o.ready("found-tool", 1)).To(BeTrue())
|
||||
// The owner dies here: the journal still says generation 1 and the
|
||||
// operation never reached prepared.
|
||||
Expect(o.s.Close(bg)).To(Succeed())
|
||||
Expect(o.m.Close()).To(Succeed())
|
||||
again, err := openOwner(root, lib, 5*time.Second)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() { _ = again.s.Close(bg); _ = again.m.Close() })
|
||||
status, err := again.m.Status(bg, "found-tool")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(status.Generation).To(Equal(uint64(1)))
|
||||
Expect(again.ready("found-tool", 1)).To(BeTrue())
|
||||
regs, err := again.s.Registry.Snapshot(bg)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(regs).To(HaveLen(1))
|
||||
})
|
||||
|
||||
It("starts the owner when an enabled add-on cannot become ready and fails only that add-on", func() {
|
||||
Expect(lib.Add(cluster.ToolPackage("1.0.0", nil), effects)).To(Succeed())
|
||||
Expect(lib.Add(cluster.ServicePackage("1.0.0", map[string]any{"mode": "exit-start"}), effects)).To(Succeed())
|
||||
o, root := makeOwner(lib, 5*time.Second)
|
||||
for _, id := range []string{"found-tool", "found-svc"} {
|
||||
_, err := o.submit(id, "install", 0, "1.0.0")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
}
|
||||
_, err := o.submit("found-tool", "enable", 1, "")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
// The broken add-on is enabled, and its own command says it did not start.
|
||||
_, err = o.submit("found-svc", "enable", 1, "")
|
||||
Expect(err).To(HaveOccurred())
|
||||
status, err := o.m.Status(context.Background(), "found-svc")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
marker := filepath.Join(status.DataDirectory, "rows.db")
|
||||
Expect(os.MkdirAll(status.DataDirectory, 0o700)).To(Succeed())
|
||||
Expect(os.WriteFile(marker, []byte("retained rows"), 0o600)).To(Succeed())
|
||||
Expect(o.s.Close(context.Background())).To(Succeed())
|
||||
Expect(o.m.Close()).To(Succeed())
|
||||
|
||||
// A restart must still bring the owner up, with the healthy add-on
|
||||
// running and the broken one stopped.
|
||||
again, err := openOwner(root, lib, 5*time.Second)
|
||||
Expect(err).NotTo(HaveOccurred(), "one add-on that cannot start must not stop the owner")
|
||||
DeferCleanup(func() { _ = again.s.Close(context.Background()); _ = again.m.Close() })
|
||||
Expect(again.ready("found-tool", 1)).To(BeTrue())
|
||||
Expect(again.ready("found-svc", 1)).To(BeFalse())
|
||||
// A command on the healthy add-on is not failed by the broken one.
|
||||
_, err = again.submit("found-tool", "disable", 1, "")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
_, err = again.submit("found-svc", "disable", 1, "")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
b, err := os.ReadFile(marker)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(string(b)).To(Equal("retained rows"))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,102 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package foundation_test
|
||||
|
||||
import (
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// Two separate server processes, each with its own listener and carrier, share
|
||||
// one account store and forward every add-on request to the one owner.
|
||||
var _ = Describe("two non-owner frontends with real sessions", Ordered, func() {
|
||||
var (
|
||||
c *cluster.Cluster
|
||||
res string
|
||||
)
|
||||
nodes := []string{cluster.Owner, cluster.FrontA, cluster.FrontB}
|
||||
|
||||
BeforeAll(func() {
|
||||
c, _ = serving([]cluster.Package{cluster.ToolPackage("1.0.0", nil)}, map[string]string{"found-tool": "1.0.0"})
|
||||
res = resource(c, "found-tool", "member", nil)
|
||||
})
|
||||
|
||||
It("gives an ordinary account the apps list and no management on any node", func() {
|
||||
for _, n := range nodes {
|
||||
member := c.Session(n, "member")
|
||||
status, list := apps(member)
|
||||
Expect(status).To(Equal(200), n)
|
||||
Expect(list).To(HaveLen(1))
|
||||
raw, err := member.Do("GET", "/api/addons/apps", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(raw.String()).NotTo(ContainSubstring(`"admin"`), n+": the apps list carries no controls for a member")
|
||||
for _, action := range []string{"install", "upgrade", "enable", "disable", "uninstall", "catalog-sources", "addons-installed"} {
|
||||
r := dispatch(member, map[string]any{"addonId": "found-tool", "action": action, "generation": 1, "arguments": map[string]any{}})
|
||||
Expect(r.Status).To(Equal(403), n+" "+action+": "+r.String())
|
||||
}
|
||||
for _, verb := range []string{"start", "stop"} {
|
||||
r, err := member.Do("POST", "/api/addons/apps/found-tool/"+verb, map[string]any{"operationId": nextID(verb), "generation": 1}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(403), n+" "+verb)
|
||||
}
|
||||
}
|
||||
// The administrator sees the controls on every node.
|
||||
for _, n := range nodes {
|
||||
raw, err := c.Session(n, "admin").Do("GET", "/api/addons/apps", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(raw.String()).To(ContainSubstring(`"admin"`), n)
|
||||
}
|
||||
})
|
||||
|
||||
It("returns the same typed denial from the owner and from each frontend", func() {
|
||||
var bodies []string
|
||||
for _, n := range nodes {
|
||||
r := read(c.Token(n, "friend"), "found-tool", res, 1, map[string]any{"tool": "echo"})
|
||||
Expect(r.Status).To(Equal(403))
|
||||
bodies = append(bodies, r.String())
|
||||
}
|
||||
Expect(bodies[1]).To(Equal(bodies[0]))
|
||||
Expect(bodies[2]).To(Equal(bodies[0]))
|
||||
var unknown []string
|
||||
for _, n := range nodes {
|
||||
r := read(c.Token(n, "member"), "found-tool", "r-not-a-resource", 1, map[string]any{"tool": "echo"})
|
||||
Expect(r.Status).To(Equal(403))
|
||||
unknown = append(unknown, r.String())
|
||||
}
|
||||
Expect(unknown[1]).To(Equal(unknown[0]))
|
||||
Expect(unknown[2]).To(Equal(unknown[0]))
|
||||
Expect(unknown[0]).To(Equal(bodies[0]), "a missing resource and a forbidden one look the same")
|
||||
})
|
||||
|
||||
It("refuses a session on every node once it was ended through one frontend", func() {
|
||||
// A dedicated session: the shared fixture sessions stay valid.
|
||||
victim := c.Session(cluster.FrontA, "friend")
|
||||
for _, n := range nodes {
|
||||
status, _ := apps(c.Session(n, "friend"))
|
||||
Expect(status).To(Equal(200), n)
|
||||
}
|
||||
out, err := victim.Do("POST", "/api/auth/logout", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(out.Status).To(Equal(200), out.String())
|
||||
for _, n := range nodes {
|
||||
status, _ := apps(c.Session(n, "friend"))
|
||||
Expect(status).To(Equal(401), n+": the ended session is refused everywhere")
|
||||
asset, err := c.Session(n, "friend").Do("GET", "/api/addons/found-tool/assets/"+"0000000000000000000000000000000000000000000000000000000000000000"+"/ui.js", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(asset.Status).To(Equal(401), n)
|
||||
}
|
||||
})
|
||||
|
||||
It("refuses an API key on every node once it was revoked through one frontend", func() {
|
||||
member := c.Creds.Users["member"]
|
||||
bearer := c.Token(cluster.FrontB, "member")
|
||||
Expect(actorOf(read(bearer, "found-tool", res, 1, map[string]any{"tool": "echo"}))).To(Equal("fnd-member"))
|
||||
r, err := c.Session(cluster.FrontA, "member").Do("DELETE", "/api/auth/api-keys/"+member.KeyID, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
for _, n := range nodes {
|
||||
got := read(c.Token(n, "member"), "found-tool", res, 1, map[string]any{"tool": "echo"})
|
||||
Expect(got.Status).To(Equal(401), n+": "+got.String())
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,14 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package hostslots_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestHostSlots(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Host UI slots contract")
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package hostslots_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/addons/apps"
|
||||
"github.com/mudler/LocalAI/core/services/addons/packages"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func host() contract.HostCompatibility {
|
||||
return contract.HostCompatibility{Host: 1, API: 1, Protocol: 1, SDK: 1, DataFormat: 1, OS: "linux", Arch: "amd64"}
|
||||
}
|
||||
|
||||
// manifest builds a manifest with the given ui object. Tools are ping and list.
|
||||
func manifest(ui string) []byte {
|
||||
return []byte(`{"schemaVersion":1,"id":"notes","version":"1.0.0","executable":"bin/x","os":"linux","arch":"amd64",` +
|
||||
`"host":{"min":1,"max":2},"api":{"min":1,"max":1},"protocol":{"min":1,"max":1},"dataFormat":{"min":1,"max":2},` +
|
||||
`"capabilities":{"tools":["ping","list"],"ui":` + ui + `}}`)
|
||||
}
|
||||
|
||||
const base = `"entry":"ui.js","sdk":{"min":1,"max":1},"icon":"ui/icon.png","pages":["home"]`
|
||||
|
||||
func validate(ui string) error {
|
||||
m, e := contract.ParseManifest(manifest(ui))
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
return contract.ValidateManifest(m, host())
|
||||
}
|
||||
|
||||
var _ = Describe("Host UI contributions in the manifest", func() {
|
||||
It("accepts tool and component chat actions, slots and own-tool renderers", func() {
|
||||
ui := `{` + base + `,"chatActions":[
|
||||
{"id":"summarize","label":"Summarize","icon":"sparkles","placement":"composer","target":{"tool":"ping"}},
|
||||
{"id":"inspect","label":{"en":"Inspect","it":"Ispeziona"},"icon":"ui/icon.png","placement":"message","target":{"component":"InspectPanel"}}],
|
||||
"slots":[{"id":"nav","slot":"app.nav","component":"NavBit"},{"id":"set","slot":"settings.sections","component":"SettingsBit"},
|
||||
{"id":"cx","slot":"chat.composer.actions","component":"ComposerBit"},{"id":"mx","slot":"chat.message.actions","component":"MessageBit"}],
|
||||
"renderers":[{"kind":"approval-card","tool":"ping","component":"PingCard"},{"kind":"tool-result","tool":"list","component":"ListView"}]}`
|
||||
Expect(validate(ui)).To(Succeed())
|
||||
m, _ := contract.ParseManifest(manifest(ui))
|
||||
Expect(m.Capabilities.UI.ChatActions).To(HaveLen(2))
|
||||
Expect(m.Capabilities.UI.ChatActions[1].Label.ByLanguage).To(HaveKeyWithValue("it", "Ispeziona"))
|
||||
// The canonical form is stable.
|
||||
out, e := json.Marshal(m)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
again, e := contract.ParseManifest(out)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(again).To(Equal(m))
|
||||
})
|
||||
|
||||
It("keeps bare chat action identifiers and every earlier fixture valid", func() {
|
||||
Expect(validate(`{` + base + `,"chatActions":["open","summarize"]}`)).To(Succeed())
|
||||
for _, name := range []string{"service", "headless", "page", "chat", "app"} {
|
||||
b, e := os.ReadFile("../fixtures/contracts/" + name + ".json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := contract.ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(contract.ValidateManifest(m, host())).To(Succeed(), name)
|
||||
}
|
||||
for _, dir := range []string{"bare", "cssb", "uifixture", "uifixture-2"} {
|
||||
b, e := os.ReadFile("../fixture/ui/" + dir + "/manifest.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := contract.ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(contract.ValidateManifest(m, host())).To(Succeed(), dir)
|
||||
}
|
||||
})
|
||||
|
||||
It("rejects every undeclared, unsafe or inconsistent contribution at install", func() {
|
||||
action := func(body string) string {
|
||||
return `{` + base + `,"chatActions":[` + body + `]}`
|
||||
}
|
||||
bad := map[string]string{
|
||||
"unknown slot": `{` + base + `,"slots":[{"id":"a","slot":"chat.footer","component":"Bit"}]}`,
|
||||
"unknown placement": action(`{"id":"a","label":"A","icon":"star","placement":"sidebar","target":{"tool":"ping"}}`),
|
||||
"missing placement": action(`{"id":"a","label":"A","icon":"star","target":{"tool":"ping"}}`),
|
||||
"remote icon": action(`{"id":"a","label":"A","icon":"https://evil.example/i.png","placement":"composer","target":{"tool":"ping"}}`),
|
||||
"absolute icon": action(`{"id":"a","label":"A","icon":"/etc/icon.png","placement":"composer","target":{"tool":"ping"}}`),
|
||||
"unknown icon name": action(`{"id":"a","label":"A","icon":"skull","placement":"composer","target":{"tool":"ping"}}`),
|
||||
"other packaged icon": action(`{"id":"a","label":"A","icon":"ui/other.png","placement":"composer","target":{"tool":"ping"}}`),
|
||||
"undeclared tool": action(`{"id":"a","label":"A","icon":"star","placement":"composer","target":{"tool":"delete-everything"}}`),
|
||||
"both targets": action(`{"id":"a","label":"A","icon":"star","placement":"composer","target":{"tool":"ping","component":"Panel"}}`),
|
||||
"no target": action(`{"id":"a","label":"A","icon":"star","placement":"composer","target":{}}`),
|
||||
"duplicate ids": action(`{"id":"a","label":"A","icon":"star","placement":"composer","target":{"tool":"ping"}},{"id":"a","label":"B","icon":"star","placement":"message","target":{"tool":"ping"}}`),
|
||||
"markup label": action(`{"id":"a","label":"<b>A</b>","icon":"star","placement":"composer","target":{"tool":"ping"}}`),
|
||||
"long label": action(`{"id":"a","label":"` + strings.Repeat("x", 61) + `","icon":"star","placement":"composer","target":{"tool":"ping"}}`),
|
||||
"unsafe component": action(`{"id":"a","label":"A","icon":"star","placement":"message","target":{"component":"__proto__"}}`),
|
||||
"dotted component": action(`{"id":"a","label":"A","icon":"star","placement":"message","target":{"component":"A.b"}}`),
|
||||
"unknown field": action(`{"id":"a","label":"A","icon":"star","placement":"composer","target":{"tool":"ping"},"onClick":"x"}`),
|
||||
"slot id clashes": `{` + base + `,"chatActions":["a"],"slots":[{"id":"a","slot":"app.nav","component":"Bit"}]}`,
|
||||
"duplicate slot id": `{` + base + `,"slots":[{"id":"a","slot":"app.nav","component":"Bit"},{"id":"a","slot":"settings.sections","component":"Bit"}]}`,
|
||||
"lowercase component": `{` + base + `,"slots":[{"id":"a","slot":"app.nav","component":"constructor"}]}`,
|
||||
"renderer foreign": `{` + base + `,"renderers":[{"kind":"approval-card","tool":"other-addon-tool","component":"Card"}]}`,
|
||||
"renderer kind": `{` + base + `,"renderers":[{"kind":"banner","tool":"ping","component":"Card"}]}`,
|
||||
"renderer duplicate": `{` + base + `,"renderers":[{"kind":"tool-result","tool":"ping","component":"A"},{"kind":"tool-result","tool":"ping","component":"B"}]}`,
|
||||
"too many actions": `{` + base + `,"chatActions":[` + manyActions(17) + `]}`,
|
||||
"too many slots": `{` + base + `,"slots":[` + manySlots(17) + `]}`,
|
||||
"slot without id": `{` + base + `,"slots":[{"slot":"app.nav","component":"Bit"}]}`,
|
||||
"slot unknown field": `{` + base + `,"slots":[{"id":"a","slot":"app.nav","component":"Bit","props":{}}]}`,
|
||||
"renderer unknown key": `{` + base + `,"renderers":[{"kind":"tool-result","tool":"ping","component":"A","style":"x"}]}`,
|
||||
}
|
||||
for name, ui := range bad {
|
||||
Expect(validate(ui)).NotTo(Succeed(), name)
|
||||
}
|
||||
// A valid control for each family, so the table above cannot pass by
|
||||
// failing for an unrelated reason.
|
||||
Expect(validate(action(`{"id":"a","label":"A","icon":"star","placement":"composer","target":{"tool":"ping"}}`))).To(Succeed())
|
||||
Expect(validate(`{` + base + `,"slots":[{"id":"a","slot":"app.nav","component":"Bit"}]}`)).To(Succeed())
|
||||
Expect(validate(`{` + base + `,"renderers":[{"kind":"tool-result","tool":"ping","component":"A"}]}`)).To(Succeed())
|
||||
})
|
||||
|
||||
It("keeps the closed slot list and the icon allowlist equal to the host", func() {
|
||||
Expect(contract.HostSlots).To(Equal([]string{"chat.composer.actions", "chat.message.actions", "app.nav", "settings.sections"}))
|
||||
src, e := os.ReadFile("../../../core/http/react-ui/src/addons/sdk.js")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
block := regexp.MustCompile(`(?s)ICON_NAMES = Object\.freeze\(\[(.*?)\]\)`).FindSubmatch(src)
|
||||
Expect(block).NotTo(BeNil())
|
||||
var names []string
|
||||
for _, m := range regexp.MustCompile(`'([a-z-]+)'`).FindAllSubmatch(block[1], -1) {
|
||||
names = append(names, string(m[1]))
|
||||
}
|
||||
Expect(slices.Equal(names, contract.IconNames)).To(BeTrue(), "sdk.js ICON_NAMES and contract.IconNames differ")
|
||||
slots, e := os.ReadFile("../../../core/http/react-ui/src/addons/slots.js")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
for _, id := range contract.HostSlots {
|
||||
Expect(string(slots)).To(ContainSubstring("'" + id + "'"))
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
func manyActions(n int) string {
|
||||
var out []string
|
||||
for i := 0; i < n; i++ {
|
||||
out = append(out, `{"id":"a`+string(rune('a'+i))+`","label":"A","icon":"star","placement":"composer","target":{"tool":"ping"}}`)
|
||||
}
|
||||
return strings.Join(out, ",")
|
||||
}
|
||||
|
||||
func manySlots(n int) string {
|
||||
var out []string
|
||||
for i := 0; i < n; i++ {
|
||||
out = append(out, `{"id":"s`+string(rune('a'+i))+`","slot":"app.nav","component":"Bit"}`)
|
||||
}
|
||||
return strings.Join(out, ",")
|
||||
}
|
||||
|
||||
var _ = Describe("The apps list carries the declared contributions", func() {
|
||||
snapshot := func(enabled bool, ui string) apps.Snapshot {
|
||||
m, e := contract.ParseManifest(manifest(ui))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return apps.Snapshot{AddonID: "notes", Installed: true, Enabled: enabled, Generation: 2, State: "running",
|
||||
Receipt: packages.PackageReceipt{Digest: "sha256:" + strings.Repeat("ab", 32), Manifest: m,
|
||||
Icon: &packages.PackageIcon{Path: "ui/icon.png"}}}
|
||||
}
|
||||
ui := `{` + base + `,"chatActions":["legacy",
|
||||
{"id":"summarize","label":{"en":"Summarize","it":"Riassumi"},"icon":"sparkles","placement":"composer","target":{"tool":"ping"}},
|
||||
{"id":"inspect","label":"Inspect","icon":"ui/icon.png","placement":"message","target":{"component":"InspectPanel"}}],
|
||||
"slots":[{"id":"set","slot":"settings.sections","component":"SettingsBit"}],
|
||||
"renderers":[{"kind":"approval-card","tool":"ping","component":"PingCard"}]}`
|
||||
|
||||
It("lists actions, slots and renderers for an enabled addon and keeps the identifiers", func() {
|
||||
a := apps.Build([]apps.Snapshot{snapshot(true, ui)}, false)[0]
|
||||
Expect(a.UI).NotTo(BeNil())
|
||||
Expect(a.UI.ChatActions).To(Equal([]string{"legacy", "summarize", "inspect"}))
|
||||
Expect(a.UI.Actions).To(Equal([]apps.ChatActionView{
|
||||
{ID: "summarize", Label: "Summarize", Labels: map[string]string{"en": "Summarize", "it": "Riassumi"}, Icon: "sparkles", Placement: "composer", Tool: "ping"},
|
||||
{ID: "inspect", Label: "Inspect", PackagedIcon: true, Placement: "message", Component: "InspectPanel"},
|
||||
}))
|
||||
Expect(a.UI.Slots).To(Equal([]apps.SlotView{{ID: "set", Slot: "settings.sections", Component: "SettingsBit"}}))
|
||||
Expect(a.UI.Renderers).To(Equal([]contract.Renderer{{Kind: "approval-card", Tool: "ping", Component: "PingCard"}}))
|
||||
b, e := json.Marshal(a)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(string(b)).To(ContainSubstring(`"slot":"settings.sections"`))
|
||||
})
|
||||
|
||||
It("lists no contribution for a disabled addon, whoever asks", func() {
|
||||
for _, admin := range []bool{false, true} {
|
||||
for _, a := range apps.Build([]apps.Snapshot{snapshot(false, ui)}, admin) {
|
||||
Expect(a.UI).To(BeNil())
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,300 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
// Package packaging_test installs the package that scripts/addons/package.sh
|
||||
// builds, through the real installer and a real local OCI distribution, and
|
||||
// starts the packaged Tables executable under the real supervisor. The
|
||||
// registry is an in-process distribution on a loopback port; nothing is
|
||||
// pushed anywhere else and nothing is signed with a real identity.
|
||||
package packaging_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
goruntime "runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/go-containerregistry/pkg/name"
|
||||
"github.com/google/go-containerregistry/pkg/registry"
|
||||
v1 "github.com/google/go-containerregistry/pkg/v1"
|
||||
"github.com/google/go-containerregistry/pkg/v1/empty"
|
||||
"github.com/google/go-containerregistry/pkg/v1/mutate"
|
||||
"github.com/google/go-containerregistry/pkg/v1/remote"
|
||||
"github.com/google/go-containerregistry/pkg/v1/static"
|
||||
"github.com/google/go-containerregistry/pkg/v1/types"
|
||||
"github.com/mudler/LocalAI/core/services/addons/catalog"
|
||||
"github.com/mudler/LocalAI/core/services/addons/packages"
|
||||
addonruntime "github.com/mudler/LocalAI/core/services/addons/runtime"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
"github.com/mudler/LocalAI/pkg/addons/packaging"
|
||||
"github.com/mudler/LocalAI/pkg/credentials"
|
||||
"github.com/mudler/LocalAI/tests/addons/fixtures/signing"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
func TestPackagingContract(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Add-on package installer contract")
|
||||
}
|
||||
|
||||
const (
|
||||
repoRoot = "../../.."
|
||||
revision = "0123456789abcdef0123456789abcdef01234567"
|
||||
bundle = "application/vnd.dev.sigstore.bundle.v0.3+json"
|
||||
issuer = "https://token.actions.githubusercontent.com"
|
||||
)
|
||||
|
||||
var (
|
||||
binary string
|
||||
hostPlat = goruntime.GOOS + "/" + goruntime.GOARCH
|
||||
)
|
||||
|
||||
var _ = BeforeSuite(func() {
|
||||
binary = filepath.Join(GinkgoT().TempDir(), "tables")
|
||||
cmd := exec.Command("go", "build", "-trimpath", "-o", binary, filepath.Join(repoRoot, "addons/tables/cmd/tables"))
|
||||
cmd.Env = append(os.Environ(), "CGO_ENABLED=1")
|
||||
out, err := cmd.CombinedOutput()
|
||||
Expect(err).NotTo(HaveOccurred(), string(out))
|
||||
_, err = os.Stat(filepath.Join(repoRoot, "addons/tables/ui/dist/ui/tables.js"))
|
||||
Expect(err).NotTo(HaveOccurred(), "build the UI first: node addons/tables/ui/build.mjs")
|
||||
})
|
||||
|
||||
// matrix writes a platform matrix that enables the platform of this host.
|
||||
func matrix() string {
|
||||
GinkgoHelper()
|
||||
p := filepath.Join(GinkgoT().TempDir(), "platforms.yaml")
|
||||
body := "schema: 1\naddon: tables\nexecutable: tables\nmanifest: addons/tables/manifest.json\nui: addons/tables/ui/dist\nplatforms:\n" +
|
||||
" - platform: " + hostPlat + "\n enabled: true\n runner: native\n native_check: executable-start\n"
|
||||
Expect(os.WriteFile(p, []byte(body), 0o644)).To(Succeed())
|
||||
return p
|
||||
}
|
||||
|
||||
// packageSh runs the real entrypoint script.
|
||||
func packageSh(args ...string) (string, error) {
|
||||
cmd := exec.Command("sh", append([]string{filepath.Join(repoRoot, "scripts/addons/package.sh")}, args...)...)
|
||||
out, err := cmd.CombinedOutput()
|
||||
return string(out), err
|
||||
}
|
||||
|
||||
type published struct {
|
||||
repository string
|
||||
digest string // platform image digest the catalog pins
|
||||
index string
|
||||
entry map[string]any
|
||||
registry string
|
||||
authority *signing.Authority
|
||||
}
|
||||
|
||||
// publishPackage builds, merges, publishes (to the local distribution) and
|
||||
// renders the catalog entry of the package for the host platform.
|
||||
func publishPackage(version string, signAs string) published {
|
||||
GinkgoHelper()
|
||||
cfg := matrix()
|
||||
work := GinkgoT().TempDir()
|
||||
layout := filepath.Join(work, "leg")
|
||||
out, err := packageSh("--config", cfg, "--layout", layout, "--version", version, "--platform", hostPlat, "--executable", binary, "--revision", revision)
|
||||
Expect(err).NotTo(HaveOccurred(), out)
|
||||
merged := filepath.Join(work, "merged")
|
||||
Expect(packaging.Run([]string{"merge", "--config", cfg, "--out", merged, "--version", version, "--revision", revision, layout}, io.Discard)).To(Succeed())
|
||||
|
||||
handler := registry.New(registry.Logger(log.New(io.Discard, "", 0)))
|
||||
server := httptest.NewServer(handler)
|
||||
DeferCleanup(server.Close)
|
||||
host := strings.TrimPrefix(server.URL, "http://")
|
||||
repository := host + "/addons/tables"
|
||||
var pub strings.Builder
|
||||
Expect(packaging.Run([]string{"publish", "--config", cfg, "--layout", merged, "--repository", repository, "--tag", version}, &pub)).To(Succeed())
|
||||
|
||||
catalogFile := filepath.Join(work, "catalog.yaml")
|
||||
Expect(packaging.Run([]string{"catalog", "--config", cfg, "--template", filepath.Join(repoRoot, "gallery/addons/tables.yaml.tmpl"),
|
||||
"--merged", merged, "--repository", repository, "--tag", version, "--channel", "latest",
|
||||
"--owner-repo", "example/localai", "--signer-ref", "refs/tags/v" + version, "--out", catalogFile}, io.Discard)).To(Succeed())
|
||||
b, err := os.ReadFile(catalogFile)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
// The host's own reader is the judge of the generated index.
|
||||
hostEntries, err := catalog.DecodeIndex(b)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(hostEntries).To(HaveLen(1))
|
||||
var entries []map[string]any
|
||||
Expect(yaml.Unmarshal(b, &entries)).To(Succeed())
|
||||
Expect(entries).To(HaveLen(1))
|
||||
|
||||
p := published{repository: repository, entry: entries[0], registry: host, authority: nil}
|
||||
p.digest = entries[0]["digest"].(string)
|
||||
for _, l := range strings.Split(pub.String(), "\n") {
|
||||
if v, ok := strings.CutPrefix(l, "digest="); ok {
|
||||
p.index = v
|
||||
}
|
||||
}
|
||||
// Sign the platform image the catalog pins, as the workflow's signing
|
||||
// step does (the identity is what the workflow certificate carries).
|
||||
authority, err := signing.New()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
p.authority = authority
|
||||
identity := entries[0]["policy"].(map[string]any)["identity"].(string)
|
||||
if signAs != "" {
|
||||
identity = signAs
|
||||
}
|
||||
ref, err := name.ParseReference(repository + "@" + p.digest)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
blob, err := authority.Bundle(strings.TrimPrefix(p.digest, "sha256:"), identity, issuer)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
img, err := mutate.AppendLayers(empty.Image, static.NewLayer(blob, types.MediaType(bundle)))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
h, err := img.Digest()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
size, err := img.Size()
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
index := mutate.AppendManifests(empty.Index, mutate.IndexAddendum{Add: img, Descriptor: v1.Descriptor{Digest: h, Size: size, MediaType: types.DockerManifestSchema2, ArtifactType: bundle}})
|
||||
Expect(remote.WriteIndex(ref.Context().Tag(strings.Replace(p.digest, ":", "-", 1)), index)).To(Succeed())
|
||||
Expect(remote.Write(ref.Context().Tag("signature"), img)).To(Succeed())
|
||||
return p
|
||||
}
|
||||
|
||||
// installer is an owner: a catalog source that serves the rendered entry
|
||||
// with the signer policy the owner configures, and a private package root.
|
||||
func installer(p published, policy catalog.VerificationPolicy) (*packages.Installer, catalog.Release) {
|
||||
GinkgoHelper()
|
||||
entry := p.entry
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Header.Get("Authorization") != "Bearer catalog-only" {
|
||||
w.WriteHeader(403)
|
||||
return
|
||||
}
|
||||
doc, err := yaml.Marshal([]map[string]any{entry})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
_, _ = w.Write(doc)
|
||||
}))
|
||||
DeferCleanup(server.Close)
|
||||
store, err := credentials.Parse([]byte("- match: "+server.URL+"\n allow_insecure: true\n bearer: catalog-only"), nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
sources, err := catalog.NewSources(filepath.Join(GinkgoT().TempDir(), "sources"), func(context.Context, contract.Actor) error { return nil },
|
||||
catalog.CredentialResolver{References: map[string]catalog.CredentialBinding{"catalog": {Origin: server.URL, Store: store}}})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
src := catalog.Source{ID: "first-party", Name: "First party", Enabled: true, AllowLoopbackHTTP: true, URL: server.URL, AuthReference: "catalog", Policy: policy}
|
||||
Expect(sources.Save(context.Background(), contract.Actor{}, src)).To(Succeed())
|
||||
_, err = sources.Refresh(context.Background(), "first-party")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
c := catalog.Catalog{Sources: sources, TargetOS: goruntime.GOOS, TargetArch: goruntime.GOARCH}
|
||||
release, err := c.Resolve(context.Background(), "first-party", "tables", "latest")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
root := GinkgoT().TempDir()
|
||||
DeferCleanup(func() {
|
||||
_ = filepath.Walk(root, func(p string, info os.FileInfo, e error) error {
|
||||
if e == nil && info.IsDir() {
|
||||
_ = os.Chmod(p, 0o700)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
})
|
||||
return &packages.Installer{
|
||||
Root: root, Catalog: c,
|
||||
Host: contract.HostCompatibility{Host: 1, API: 1, Protocol: 1, SDK: 1, DataFormat: 1, OS: goruntime.GOOS, Arch: goruntime.GOARCH},
|
||||
}, release
|
||||
}
|
||||
|
||||
var _ = Describe("first-party package", func() {
|
||||
It("installs the package built by package.sh and starts its executable", func() {
|
||||
p := publishPackage("1.2.3", "")
|
||||
identity := p.entry["policy"].(map[string]any)["identity"].(string)
|
||||
Expect(identity).To(Equal("https://github.com/example/localai/.github/workflows/addons-package.yml@refs/tags/v1.2.3"))
|
||||
i, release := installer(p, catalog.VerificationPolicy{RequireSignature: true, Issuer: issuer, Identity: identity})
|
||||
i.SignatureTrust = p.authority
|
||||
Expect(release.Digest).To(Equal(p.digest))
|
||||
staged, err := i.Stage(context.Background(), release, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
receipt, err := i.Commit(context.Background(), staged)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(receipt.Verification).To(Equal("sha256+sigstore"))
|
||||
Expect(receipt.Digest).To(Equal(p.digest))
|
||||
Expect(receipt.Manifest.ID).To(Equal("tables"))
|
||||
Expect(receipt.Manifest.Version).To(Equal("1.2.3"))
|
||||
Expect(receipt.Manifest.OS).To(Equal(goruntime.GOOS))
|
||||
Expect(receipt.Manifest.Arch).To(Equal(goruntime.GOARCH))
|
||||
Expect(receipt.Icon).NotTo(BeNil())
|
||||
info, err := os.Stat(filepath.Join(receipt.Directory, "tables"))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(info.Mode().Perm() & 0o100).NotTo(BeZero())
|
||||
Expect(filepath.Join(receipt.Directory, "ui", "tables.js")).To(BeAnExistingFile())
|
||||
Expect(filepath.Join(receipt.Directory, "ui", "tables.css")).To(BeAnExistingFile())
|
||||
|
||||
limits := contract.DefaultLimits()
|
||||
sup, err := addonruntime.NewSupervisor(addonruntime.Config{StateRoot: filepath.Join(GinkgoT().TempDir(), "runtime"), DataRoot: filepath.Join(GinkgoT().TempDir(), "data"), Limits: limits, ReadyTimeout: 20 * time.Second, StableHealth: limits.BackoffMax})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() { Expect(sup.Close(context.Background())).To(Succeed()) })
|
||||
inst, err := sup.Start(context.Background(), receipt, 1)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(inst.Generation).To(Equal(uint64(1)))
|
||||
})
|
||||
|
||||
It("is refused when the signature carries another workflow identity", func() {
|
||||
p := publishPackage("1.2.4", "https://github.com/example/localai/.github/workflows/backend_merge.yml@refs/heads/master")
|
||||
identity := p.entry["policy"].(map[string]any)["identity"].(string)
|
||||
i, release := installer(p, catalog.VerificationPolicy{RequireSignature: true, Issuer: issuer, Identity: identity})
|
||||
i.SignatureTrust = p.authority
|
||||
_, err := i.Stage(context.Background(), release, nil)
|
||||
Expect(err).To(MatchError(packages.ErrPackage))
|
||||
entries, err := os.ReadDir(i.Root)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(entries).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("is refused when the owner policy names the backend workflow", func() {
|
||||
p := publishPackage("1.2.5", "")
|
||||
i, release := installer(p, catalog.VerificationPolicy{RequireSignature: true, Issuer: issuer, Identity: "https://github.com/example/localai/.github/workflows/backend_merge.yml@refs/heads/master"})
|
||||
i.SignatureTrust = p.authority
|
||||
_, err := i.Stage(context.Background(), release, nil)
|
||||
Expect(err).To(MatchError(packages.ErrPackage))
|
||||
})
|
||||
|
||||
It("is refused when the pinned digest is not the published image", func() {
|
||||
p := publishPackage("1.2.6", "")
|
||||
identity := p.entry["policy"].(map[string]any)["identity"].(string)
|
||||
i, release := installer(p, catalog.VerificationPolicy{RequireSignature: true, Issuer: issuer, Identity: identity})
|
||||
i.SignatureTrust = p.authority
|
||||
release.Digest = "sha256:" + strings.Repeat("0", 64)
|
||||
_, err := i.Stage(context.Background(), release, nil)
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
|
||||
It("gives identical digests for two builds of the same inputs", func() {
|
||||
cfg := matrix()
|
||||
var digests []string
|
||||
for n := 0; n < 2; n++ {
|
||||
layout := filepath.Join(GinkgoT().TempDir(), "leg")
|
||||
out, err := packageSh("--config", cfg, "--layout", layout, "--version", "2.0.0", "--platform", hostPlat, "--executable", binary, "--revision", revision)
|
||||
Expect(err).NotTo(HaveOccurred(), out)
|
||||
for _, l := range strings.Split(out, "\n") {
|
||||
if v, ok := strings.CutPrefix(l, "digest="); ok {
|
||||
digests = append(digests, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
Expect(digests).To(HaveLen(2))
|
||||
Expect(digests[0]).To(Equal(digests[1]))
|
||||
})
|
||||
|
||||
It("refuses a platform the matrix does not enable and a missing executable of another platform", func() {
|
||||
layout := filepath.Join(GinkgoT().TempDir(), "leg")
|
||||
out, err := packageSh("--layout", layout, "--version", "1.0.0", "--platform", hostPlat, "--executable", binary)
|
||||
Expect(err).To(HaveOccurred(), out)
|
||||
Expect(out).To(ContainSubstring("not enabled"))
|
||||
Expect(layout).NotTo(BeADirectory())
|
||||
other := "linux/arm64"
|
||||
if hostPlat == other {
|
||||
other = "linux/amd64"
|
||||
}
|
||||
out, err = packageSh("--config", matrix(), "--layout", filepath.Join(GinkgoT().TempDir(), "x"), "--version", "1.0.0", "--platform", other)
|
||||
Expect(err).To(HaveOccurred(), out)
|
||||
Expect(out).To(ContainSubstring("not this host"))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,475 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package recovery_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/ed25519"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"math/big"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
goruntime "runtime"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/labstack/echo/v4"
|
||||
"github.com/mudler/LocalAI/core/application"
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
httpapi "github.com/mudler/LocalAI/core/http"
|
||||
"github.com/mudler/LocalAI/core/http/auth"
|
||||
"github.com/mudler/LocalAI/core/http/routes"
|
||||
addonsvc "github.com/mudler/LocalAI/core/services/addons"
|
||||
"github.com/mudler/LocalAI/core/services/addons/catalog"
|
||||
"github.com/mudler/LocalAI/core/services/addons/gateway"
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
"github.com/mudler/LocalAI/pkg/system"
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestRecovery(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Add-on data recovery")
|
||||
}
|
||||
|
||||
var binary string
|
||||
|
||||
var _ = BeforeSuite(func() {
|
||||
binary = filepath.Join(GinkgoT().TempDir(), "tables")
|
||||
cmd := exec.Command("go", "build", "-o", binary, "../../../addons/tables/cmd/tables")
|
||||
cmd.Env = append(os.Environ(), "CGO_ENABLED=1")
|
||||
out, e := cmd.CombinedOutput()
|
||||
Expect(e).NotTo(HaveOccurred(), string(out))
|
||||
})
|
||||
|
||||
const appSecret = "tables-admin-secret"
|
||||
|
||||
// transports builds the transport identities of one owner and one forwarding
|
||||
// frontend: a CA, two pinned leaves and two signing keys.
|
||||
func transports(port int) (owner, front *gateway.CarrierConfig) {
|
||||
GinkgoHelper()
|
||||
pub, key, e := ed25519.GenerateKey(rand.Reader)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
ca := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "test CA"}, NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour), IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign}
|
||||
der, e := x509.CreateCertificate(rand.Reader, ca, ca, pub, key)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
ca, e = x509.ParseCertificate(der)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
pool := x509.NewCertPool()
|
||||
pool.AddCert(ca)
|
||||
pins := map[string][32]byte{}
|
||||
verify := map[string]ed25519.PublicKey{}
|
||||
pairs := map[string]tls.Certificate{}
|
||||
signing := map[string]ed25519.PrivateKey{}
|
||||
for i, id := range []string{"owner", "front-a"} {
|
||||
lp, lk, e := ed25519.GenerateKey(rand.Reader)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
leaf := &x509.Certificate{SerialNumber: big.NewInt(int64(i + 2)), Subject: pkix.Name{CommonName: id}, NotBefore: ca.NotBefore, NotAfter: ca.NotAfter, IPAddresses: []net.IP{net.ParseIP("127.0.0.1")}, KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth}}
|
||||
d, e := x509.CreateCertificate(rand.Reader, leaf, ca, lp, key)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
kd, e := x509.MarshalPKCS8PrivateKey(lk)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
pair, e := tls.X509KeyPair(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: d}), pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: kd}))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
sp, sk, e := ed25519.GenerateKey(rand.Reader)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
pins[id], verify[id], pairs[id], signing[id] = sha256.Sum256(d), sp, pair, sk
|
||||
}
|
||||
address := fmt.Sprintf("https://127.0.0.1:%d", port)
|
||||
mk := func(id string) *gateway.CarrierConfig {
|
||||
return &gateway.CarrierConfig{InstanceID: id, OwnerID: "owner", OwnerAddress: address, Certificate: pairs[id], Roots: pool, Pins: pins, SigningKey: signing[id], VerificationKeys: verify}
|
||||
}
|
||||
return mk("owner"), mk("front-a")
|
||||
}
|
||||
|
||||
func freePort() int {
|
||||
GinkgoHelper()
|
||||
l, e := net.Listen("tcp", "127.0.0.1:0")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
defer func() { _ = l.Close() }()
|
||||
return l.Addr().(*net.TCPAddr).Port
|
||||
}
|
||||
|
||||
// person is an account of the fixture.
|
||||
type person struct {
|
||||
id, role string
|
||||
feature bool
|
||||
status string
|
||||
}
|
||||
|
||||
var people = []person{
|
||||
{"alice", auth.RoleAdmin, false, auth.StatusActive},
|
||||
{"bob", auth.RoleUser, true, auth.StatusActive},
|
||||
{"carol", auth.RoleUser, true, auth.StatusActive},
|
||||
{"dave", auth.RoleUser, true, auth.StatusActive},
|
||||
{"erin", auth.RoleUser, false, auth.StatusActive},
|
||||
{"frank", auth.RoleUser, true, auth.StatusPending},
|
||||
}
|
||||
|
||||
// host is one owner in process: the real application bootstrap over real
|
||||
// account storage, the real supervisor and the real Tables executable.
|
||||
type host struct {
|
||||
dir string
|
||||
libraryURL string
|
||||
owner config.AddonConfig
|
||||
front *gateway.CarrierConfig
|
||||
app *application.Application
|
||||
web *echo.Echo
|
||||
cancel context.CancelFunc
|
||||
local bool
|
||||
tokens map[string]string
|
||||
sessions map[string]string
|
||||
keys map[string]string
|
||||
}
|
||||
|
||||
func newHost(local bool) *host {
|
||||
GinkgoHelper()
|
||||
dir := GinkgoT().TempDir()
|
||||
h := &host{dir: dir, local: local, tokens: map[string]string{}, sessions: map[string]string{}, keys: map[string]string{}}
|
||||
DeferCleanup(func() {
|
||||
_ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, e error) error {
|
||||
if e == nil && d.IsDir() {
|
||||
_ = os.Chmod(p, 0o700)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
})
|
||||
port := freePort()
|
||||
var owner *gateway.CarrierConfig
|
||||
owner, h.front = transports(port)
|
||||
h.owner = config.AddonConfig{
|
||||
Enabled: true, InstanceID: "owner", OwnerID: "owner", OwnerAddress: owner.OwnerAddress, StorageID: "local",
|
||||
Root: filepath.Join(dir, "addons"), ListenAddress: fmt.Sprintf("127.0.0.1:%d", port), Transport: owner,
|
||||
Host: contract.HostCompatibility{Host: 1, API: 1, Protocol: 1, SDK: 1, DataFormat: 1, OS: goruntime.GOOS, Arch: goruntime.GOARCH},
|
||||
}
|
||||
if local {
|
||||
h.owner.OwnershipMode = auth.AddonModeLocalOwner
|
||||
}
|
||||
return h
|
||||
}
|
||||
|
||||
func (h *host) mode() string {
|
||||
if h.local {
|
||||
return auth.AddonModeLocalOwner
|
||||
}
|
||||
return auth.AddonModeNamedUser
|
||||
}
|
||||
|
||||
func (h *host) start() {
|
||||
GinkgoHelper()
|
||||
models, backends := filepath.Join(h.dir, "models"), filepath.Join(h.dir, "backends")
|
||||
Expect(os.MkdirAll(models, 0o750)).To(Succeed())
|
||||
Expect(os.MkdirAll(backends, 0o750)).To(Succeed())
|
||||
state, e := system.GetSystemState(system.WithBackendPath(backends), system.WithModelPath(models))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
h.cancel = cancel
|
||||
opts := []config.AppOption{config.WithContext(ctx), config.WithSystemState(state), config.WithDataPath(h.dir), config.WithAddons(h.owner)}
|
||||
if !h.local {
|
||||
opts = append(opts, config.WithAuthEnabled(true), config.WithAuthDatabaseURL(filepath.Join(h.dir, "auth.db")), config.WithAuthAPIKeyHMACSecret(appSecret))
|
||||
}
|
||||
app, e := application.New(opts...)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.app = app
|
||||
h.web, e = httpapi.API(app)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
}
|
||||
|
||||
// stop ends the application as a restart does and waits until the owner
|
||||
// released its durable state.
|
||||
func (h *host) stop() {
|
||||
GinkgoHelper()
|
||||
if h.cancel == nil {
|
||||
return
|
||||
}
|
||||
Expect(h.app.Shutdown()).To(Succeed())
|
||||
h.cancel()
|
||||
h.cancel = nil
|
||||
mode := policy.ModeNamedUser
|
||||
if h.local {
|
||||
mode = policy.ModeLocalOwner
|
||||
}
|
||||
Eventually(func() error {
|
||||
s, e := policy.OpenStore(filepath.Join(h.owner.Root, "policy"), mode)
|
||||
if e == nil {
|
||||
_ = s.Close()
|
||||
}
|
||||
return e
|
||||
}, 10*time.Second, 50*time.Millisecond).Should(Succeed())
|
||||
}
|
||||
|
||||
func (h *host) actor(user string) contract.Actor {
|
||||
if h.local {
|
||||
return contract.Actor{UserID: auth.AddonLocalOwnerID, CredentialID: auth.AddonLocalOwnerID, Mode: auth.AddonModeLocalOwner}
|
||||
}
|
||||
return contract.Actor{UserID: user, CredentialID: "key:" + h.keys[user], Mode: auth.AddonModeNamedUser}
|
||||
}
|
||||
|
||||
// seed creates the accounts, installs the real Tables package from a local
|
||||
// catalog and waits until it runs.
|
||||
func (h *host) seed(lib *cluster.Library) {
|
||||
GinkgoHelper()
|
||||
ctx := context.Background()
|
||||
if !h.local {
|
||||
db := h.app.AuthDB()
|
||||
Expect(db).NotTo(BeNil())
|
||||
for _, u := range people {
|
||||
Expect(db.Create(&auth.User{ID: u.id, Email: u.id + "@tables.test", Name: "Name of " + u.id, Provider: auth.ProviderLocal, Role: u.role, Status: u.status}).Error).To(Succeed())
|
||||
if u.feature {
|
||||
Expect(db.Create(&auth.UserPermission{ID: uuid.NewString(), UserID: u.id, Permissions: auth.PermissionMap{auth.FeatureAddons: true}}).Error).To(Succeed())
|
||||
}
|
||||
token, key, e := auth.CreateAPIKey(db, u.id, "agent", u.role, appSecret, nil)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
session, e := auth.CreateSession(db, u.id, appSecret)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.tokens[u.id], h.sessions[u.id], h.keys[u.id] = token, session, key.ID
|
||||
}
|
||||
}
|
||||
admin := h.actor("alice")
|
||||
Expect(h.app.AddonSources().Save(ctx, admin, catalog.Source{ID: "local", Name: "Local", URL: lib.URL(), Enabled: true, AllowLoopbackHTTP: true})).To(Succeed())
|
||||
_, e := h.app.AddonSources().Refresh(ctx, "local")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m := h.app.AddonManager()
|
||||
_, e = m.Submit(ctx, admin, addonsvc.LifecycleCommand{OperationID: "install-tables", AddonID: "tables", Action: "install", SourceID: "local", Version: "0.1.0"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = m.Submit(ctx, admin, addonsvc.LifecycleCommand{OperationID: "enable-tables", AddonID: "tables", Action: "enable", Generation: 1})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.running()
|
||||
}
|
||||
|
||||
func (h *host) running() {
|
||||
GinkgoHelper()
|
||||
Eventually(func() string {
|
||||
list, e := h.app.AddonManager().Apps(context.Background(), h.actor("alice"))
|
||||
if e != nil {
|
||||
return e.Error()
|
||||
}
|
||||
for _, a := range list {
|
||||
if a.ID == "tables" {
|
||||
return a.State
|
||||
}
|
||||
}
|
||||
return "absent"
|
||||
}, 30*time.Second, 100*time.Millisecond).Should(Equal("running"))
|
||||
}
|
||||
|
||||
func (h *host) library() *cluster.Library {
|
||||
GinkgoHelper()
|
||||
lib, e := cluster.NewLibrary(binary)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
DeferCleanup(lib.Close)
|
||||
h.libraryURL = lib.URL()
|
||||
raw, e := os.ReadFile("../../../addons/tables/manifest.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var m struct {
|
||||
Capabilities map[string]any `json:"capabilities"`
|
||||
}
|
||||
Expect(json.Unmarshal(raw, &m)).To(Succeed())
|
||||
// These specs run the tools, not the page: the package carries no UI
|
||||
// files, so the declaration of them is left out.
|
||||
delete(m.Capabilities, "ui")
|
||||
exe, e := os.ReadFile(binary)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
for _, v := range []string{"0.1.0", "0.1.1"} {
|
||||
Expect(lib.Add(cluster.Package{ID: "tables", Version: v, Capabilities: m.Capabilities, Executable: "bin/tables", Binary: exe}, GinkgoT().TempDir())).To(Succeed())
|
||||
}
|
||||
// A release whose process fails at start: its upgrade must be undone.
|
||||
Expect(lib.Add(cluster.Package{ID: "tables", Version: "0.2.0", Capabilities: m.Capabilities, Executable: "bin/tables", Binary: []byte("#!/bin/sh\nexit 9\n")}, GinkgoT().TempDir())).To(Succeed())
|
||||
return lib
|
||||
}
|
||||
|
||||
// ready is the usual start of a spec: a running host with the accounts and
|
||||
// the package.
|
||||
func ready(local bool) *host {
|
||||
GinkgoHelper()
|
||||
h := newHost(local)
|
||||
lib := h.library()
|
||||
h.start()
|
||||
DeferCleanup(h.stop)
|
||||
h.seed(lib)
|
||||
return h
|
||||
}
|
||||
|
||||
// reply is one HTTP answer.
|
||||
type reply struct {
|
||||
code int
|
||||
body map[string]any
|
||||
raw string
|
||||
}
|
||||
|
||||
func (r reply) errType() string {
|
||||
e, _ := r.body["error"].(map[string]any)
|
||||
s, _ := e["type"].(string)
|
||||
return s
|
||||
}
|
||||
|
||||
func serve(e *echo.Echo, method, path, body string, headers ...func(*http.Request)) reply {
|
||||
GinkgoHelper()
|
||||
r := httptest.NewRequest(method, path, bytes.NewBufferString(body))
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
for _, h := range headers {
|
||||
h(r)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
e.ServeHTTP(w, r)
|
||||
out := reply{code: w.Code, body: map[string]any{}, raw: w.Body.String()}
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &out.body)
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *host) send(method, path, body string, headers ...func(*http.Request)) reply {
|
||||
GinkgoHelper()
|
||||
return serve(h.web, method, path, body, headers...)
|
||||
}
|
||||
|
||||
func (h *host) bearer(user string) func(*http.Request) {
|
||||
return func(r *http.Request) { r.Header.Set("Authorization", "Bearer "+h.tokens[user]) }
|
||||
}
|
||||
|
||||
// tab is what the host's own pages send: the user's browser session, a
|
||||
// same-origin fetch and the identifier the action concerns in the header. In
|
||||
// local-owner mode there is no session.
|
||||
func (h *host) tab(user, id string) func(*http.Request) {
|
||||
return func(r *http.Request) {
|
||||
if !h.local {
|
||||
r.AddCookie(&http.Cookie{Name: "session", Value: h.sessions[user]})
|
||||
}
|
||||
r.Header.Set("Sec-Fetch-Site", "same-origin")
|
||||
r.Header.Set("Sec-Fetch-Mode", "cors")
|
||||
r.Header.Set("Sec-Fetch-Dest", "empty")
|
||||
r.Header.Set("X-LocalAI-Confirm", id)
|
||||
}
|
||||
}
|
||||
|
||||
// session is a plain same-origin session request with no confirmation header.
|
||||
func (h *host) session(user string) func(*http.Request) {
|
||||
return func(r *http.Request) {
|
||||
r.AddCookie(&http.Cookie{Name: "session", Value: h.sessions[user]})
|
||||
r.Header.Set("Sec-Fetch-Site", "same-origin")
|
||||
r.Header.Set("Sec-Fetch-Mode", "cors")
|
||||
r.Header.Set("Sec-Fetch-Dest", "empty")
|
||||
}
|
||||
}
|
||||
|
||||
// frontend is a second LocalAI instance that holds no addon state and
|
||||
// forwards every resource call to the owner over the real carrier.
|
||||
func (h *host) frontend() *echo.Echo {
|
||||
GinkgoHelper()
|
||||
carrier, e := gateway.NewCarrier(*h.front)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var db = h.app.AuthDB()
|
||||
g := &gateway.Gateway{Forward: carrier, Revalidate: auth.AddonRevalidate(db, h.mode())}
|
||||
web := echo.New()
|
||||
web.Use(auth.Middleware(db, h.app.ApplicationConfig()))
|
||||
routes.RegisterAddonRoutesFor(web, g, db, h.mode())
|
||||
return web
|
||||
}
|
||||
|
||||
// Typed calls of the management routes, as the person's own browser tab.
|
||||
|
||||
func (h *host) create(user, op, name string) reply {
|
||||
GinkgoHelper()
|
||||
body := fmt.Sprintf(`{"addonId":"tables","operationId":%q,"generation":1,"arguments":{"name":%q}}`, op, name)
|
||||
return h.send("POST", "/api/addons/resources", body, h.tab(user, op))
|
||||
}
|
||||
|
||||
func (h *host) list(user string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("GET", "/api/addons/resources?addonId=tables", "", h.bearer(user))
|
||||
}
|
||||
|
||||
// resources returns the listed resources by ID.
|
||||
func resourcesOf(r reply) map[string]map[string]any {
|
||||
GinkgoHelper()
|
||||
out := map[string]map[string]any{}
|
||||
list, _ := r.body["resources"].([]any)
|
||||
for _, x := range list {
|
||||
m := x.(map[string]any)
|
||||
out[m["id"].(string)] = m
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *host) grant(user, res, target, role string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("PUT", "/api/addons/resources/"+res+"/grants/"+target, fmt.Sprintf(`{"role":%q}`, role), h.tab(user, res))
|
||||
}
|
||||
|
||||
func (h *host) revoke(user, res, target string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("DELETE", "/api/addons/resources/"+res+"/grants/"+target, "", h.tab(user, res))
|
||||
}
|
||||
|
||||
func (h *host) grants(user, res string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("GET", "/api/addons/resources/"+res+"/grants", "", h.tab(user, res))
|
||||
}
|
||||
|
||||
func (h *host) rename(user, res, op, name string, rev map[string]any) reply {
|
||||
GinkgoHelper()
|
||||
b, _ := json.Marshal(map[string]any{"operationId": op, "generation": 1, "revision": rev, "name": name})
|
||||
return h.send("POST", "/api/addons/resources/"+res+"/rename", string(b), h.tab(user, res))
|
||||
}
|
||||
|
||||
// revision reads the revision the owner role is shown for a resource.
|
||||
func (h *host) revision(user, res string) map[string]any {
|
||||
GinkgoHelper()
|
||||
got := resourcesOf(h.list(user))[res]
|
||||
Expect(got).NotTo(BeNil())
|
||||
rev, _ := got["revision"].(map[string]any)
|
||||
Expect(rev).NotTo(BeNil())
|
||||
return rev
|
||||
}
|
||||
|
||||
// name reads the display name of a resource through the owner's gateway, as
|
||||
// the person who may read it.
|
||||
func (h *host) name(user, res string) (string, error) {
|
||||
out, e := h.app.AddonGateway().Dispatch(context.Background(), h.actor(user), contract.Request{
|
||||
AddonID: "tables", ResourceID: res, OperationID: "inspect-" + uuid.NewString()[:8], Generation: h.currentGeneration(), Action: "read",
|
||||
Arguments: json.RawMessage(`{"tool":"databases-inspect"}`),
|
||||
})
|
||||
if e != nil {
|
||||
return "", e
|
||||
}
|
||||
var v struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
Expect(json.Unmarshal(out.Payload, &v)).To(Succeed())
|
||||
return v.Name, nil
|
||||
}
|
||||
|
||||
// database creates a database for the user and returns its ID.
|
||||
func (h *host) database(user, op, name string) string {
|
||||
GinkgoHelper()
|
||||
got := h.create(user, op, name)
|
||||
Expect(got.code).To(Equal(200), got.raw)
|
||||
id, _ := got.body["id"].(string)
|
||||
Expect(id).NotTo(BeEmpty())
|
||||
return id
|
||||
}
|
||||
|
||||
func (h *host) dbFile(res string) string {
|
||||
return filepath.Join(h.owner.Root, "data", "tables", "tables", "databases", res+".db")
|
||||
}
|
||||
|
||||
func (h *host) currentGeneration() uint64 {
|
||||
st, e := h.app.AddonManager().Status(context.Background(), "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return st.Generation
|
||||
}
|
||||
@@ -0,0 +1,463 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package recovery_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/labstack/echo/v4"
|
||||
"github.com/mudler/LocalAI/core/http/auth"
|
||||
addonsvc "github.com/mudler/LocalAI/core/services/addons"
|
||||
"github.com/mudler/LocalAI/core/services/addons/catalog"
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
"github.com/mudler/LocalAI/core/services/addons/tools"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// admin sends one recovery mutation as the administrator's own browser tab.
|
||||
func (h *host) admin(path, op string, body map[string]any) reply {
|
||||
GinkgoHelper()
|
||||
body["operationId"] = op
|
||||
b, _ := json.Marshal(body)
|
||||
return h.send("POST", path, string(b), h.tab("alice", op))
|
||||
}
|
||||
|
||||
func (h *host) status(user string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("GET", "/api/addons/tables/recovery", "", h.session(user))
|
||||
}
|
||||
|
||||
func (h *host) stopTables() {
|
||||
GinkgoHelper()
|
||||
st, e := h.app.AddonManager().Status(context.Background(), "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = h.app.AddonManager().Submit(context.Background(), h.actor("alice"), addonsvc.LifecycleCommand{OperationID: fmt.Sprintf("disable-%d", time.Now().UnixNano()), AddonID: "tables", Action: "disable", Generation: st.Generation})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Eventually(func() string {
|
||||
list, _ := h.app.AddonManager().Apps(context.Background(), h.actor("alice"))
|
||||
for _, a := range list {
|
||||
if a.ID == "tables" {
|
||||
return a.State
|
||||
}
|
||||
}
|
||||
return "absent"
|
||||
}, 20*time.Second, 100*time.Millisecond).Should(Equal("stopped"))
|
||||
}
|
||||
|
||||
func (h *host) startTables() {
|
||||
GinkgoHelper()
|
||||
st, e := h.app.AddonManager().Status(context.Background(), "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = h.app.AddonManager().Submit(context.Background(), h.actor("alice"), addonsvc.LifecycleCommand{OperationID: fmt.Sprintf("enable-%d", time.Now().UnixNano()), AddonID: "tables", Action: "enable", Generation: st.Generation})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.running()
|
||||
}
|
||||
|
||||
func (h *host) generation() uint64 {
|
||||
GinkgoHelper()
|
||||
st, e := h.app.AddonManager().Status(context.Background(), "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return st.Generation
|
||||
}
|
||||
|
||||
func backups(r reply) []map[string]any {
|
||||
GinkgoHelper()
|
||||
var out []map[string]any
|
||||
list, _ := r.body["backups"].([]any)
|
||||
for _, x := range list {
|
||||
out = append(out, x.(map[string]any))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
var _ = Describe("Add-on data recovery over the real owner", func() {
|
||||
It("makes a backup only for an administrator's own browser session and lists it", func() {
|
||||
h := ready(false)
|
||||
id := h.database("bob", "db-1", "Sales")
|
||||
Expect(id).NotTo(BeEmpty())
|
||||
// A person with the feature is not an administrator.
|
||||
Expect(h.send("POST", "/api/addons/tables/recovery/backup", `{"operationId":"b-1"}`, h.tab("bob", "b-1")).code).To(Equal(403))
|
||||
Expect(h.status("bob").code).To(Equal(403))
|
||||
// An administrator's API key is not a confirmation surface.
|
||||
Expect(h.send("POST", "/api/addons/tables/recovery/backup", `{"operationId":"b-1"}`, h.bearer("alice"), func(r *http.Request) { r.Header.Set("X-LocalAI-Confirm", "b-1") }).code).To(Equal(403))
|
||||
// The header must name the operation, and an operation is required.
|
||||
Expect(h.send("POST", "/api/addons/tables/recovery/backup", `{"operationId":"b-1"}`, h.tab("alice", "other")).code).To(Equal(403))
|
||||
Expect(h.send("POST", "/api/addons/tables/recovery/backup", `{}`, h.tab("alice", "")).code).To(Equal(400))
|
||||
Expect(h.send("POST", "/api/addons/Tables/recovery/backup", `{"operationId":"b-1"}`, h.tab("alice", "b-1")).code).To(Equal(403))
|
||||
got := h.admin("/api/addons/tables/recovery/backup", "b-1", map[string]any{})
|
||||
Expect(got.code).To(Equal(200), got.raw)
|
||||
backupID, _ := got.body["id"].(string)
|
||||
Expect(backupID).To(HavePrefix("b"))
|
||||
Expect(got.body["verified"]).To(BeTrue())
|
||||
// The same operation returns the same backup and makes no second one.
|
||||
again := h.admin("/api/addons/tables/recovery/backup", "b-1", map[string]any{})
|
||||
Expect(again.code).To(Equal(200))
|
||||
Expect(again.body["id"]).To(Equal(backupID))
|
||||
list := backups(h.status("alice"))
|
||||
Expect(list).To(HaveLen(1))
|
||||
Expect(list[0]["verification"]).To(Equal("verified"))
|
||||
// The add-on runs again and the database still answers.
|
||||
h.running()
|
||||
name, e := h.name("bob", id)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Sales"))
|
||||
// The backup holds the database file and is private.
|
||||
dir := filepath.Join(h.owner.Root, "backups", "tables", backupID)
|
||||
Expect(filepath.Join(dir, "data", "tables", "databases", id+".db")).To(BeAnExistingFile())
|
||||
info, e := os.Stat(dir)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(info.Mode().Perm()).To(Equal(os.FileMode(0o700)))
|
||||
})
|
||||
|
||||
It("restores with warnings and does not give back a share that was revoked after the backup", func() {
|
||||
h := ready(false)
|
||||
id := h.database("bob", "db-1", "Sales")
|
||||
Expect(h.grant("bob", id, "carol", "editor").code).To(Equal(200))
|
||||
Expect(h.grant("bob", id, "dave", "viewer").code).To(Equal(200))
|
||||
got := h.admin("/api/addons/tables/recovery/backup", "b-1", map[string]any{})
|
||||
Expect(got.code).To(Equal(200), got.raw)
|
||||
backupID := got.body["id"].(string)
|
||||
// After the backup: carol loses access and the database is renamed.
|
||||
Expect(h.revoke("bob", id, "carol").code).To(Equal(200))
|
||||
rc := h.rename("bob", id, "rename-1", "Renamed", h.revision("bob", id))
|
||||
Expect(rc.code).To(Equal(200), rc.raw)
|
||||
// A restore needs the add-on stopped, the typed ID and a verified backup.
|
||||
running := h.admin("/api/addons/tables/recovery/restore", "r-1", map[string]any{"backupId": backupID, "confirm": "tables"})
|
||||
Expect(running.code).To(Equal(409))
|
||||
Expect(running.errType()).To(Equal("addon_running"))
|
||||
h.stopTables()
|
||||
wrong := h.admin("/api/addons/tables/recovery/restore", "r-2", map[string]any{"backupId": backupID, "confirm": "Tables"})
|
||||
Expect(wrong.code).To(Equal(400))
|
||||
Expect(wrong.errType()).To(Equal("confirmation_mismatch"))
|
||||
missing := h.admin("/api/addons/tables/recovery/restore", "r-3", map[string]any{"backupId": "b-nope", "confirm": "tables"})
|
||||
Expect(missing.code).To(Equal(404))
|
||||
Expect(missing.errType()).To(Equal("backup_not_found"))
|
||||
done := h.admin("/api/addons/tables/recovery/restore", "r-4", map[string]any{"backupId": backupID, "confirm": "tables"})
|
||||
Expect(done.code).To(Equal(200), done.raw)
|
||||
warnings, _ := done.body["warnings"].([]any)
|
||||
Expect(warnings).NotTo(BeEmpty())
|
||||
Expect(fmt.Sprint(warnings)).To(ContainSubstring("Changes made after that time are not in it"))
|
||||
prev, _ := done.body["previousCopy"].(string)
|
||||
Expect(prev).To(HavePrefix("previous-"))
|
||||
Expect(filepath.Join(h.owner.Root, "recovery", "tables", prev)).To(BeADirectory())
|
||||
// The same operation does not restore again.
|
||||
repeat := h.admin("/api/addons/tables/recovery/restore", "r-4", map[string]any{"backupId": backupID, "confirm": "tables"})
|
||||
Expect(repeat.code).To(Equal(200))
|
||||
Expect(repeat.body["at"]).To(Equal(done.body["at"]))
|
||||
h.startTables()
|
||||
// The data is as it was at the backup: the old name is back. Access
|
||||
// is as it is now: carol stays out, dave is a viewer.
|
||||
name, e := h.name("bob", id)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Sales"))
|
||||
_, e = h.name("carol", id)
|
||||
Expect(e).To(HaveOccurred())
|
||||
name, e = h.name("dave", id)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Sales"))
|
||||
Expect(h.grants("bob", id).code).To(Equal(200))
|
||||
g := h.grants("bob", id)
|
||||
Expect(g.raw).NotTo(ContainSubstring("carol"))
|
||||
Expect(g.raw).To(ContainSubstring("dave"))
|
||||
pol := done.body["policy"].(map[string]any)
|
||||
Expect(pol["grantsDropped"]).To(BeNumerically("==", 1))
|
||||
Expect(pol["revoked"]).To(BeNumerically("==", 1))
|
||||
st := h.status("alice")
|
||||
Expect(st.body["state"]).To(Equal("ready"))
|
||||
Expect(st.body["lastRestore"]).NotTo(BeNil())
|
||||
})
|
||||
|
||||
It("restores only a backup of the add-on in the route and touches nothing otherwise", func() {
|
||||
h := ready(false)
|
||||
id := h.database("bob", "db-1", "Sales")
|
||||
got := h.admin("/api/addons/tables/recovery/backup", "b-1", map[string]any{})
|
||||
Expect(got.code).To(Equal(200), got.raw)
|
||||
backupID := got.body["id"].(string)
|
||||
rc := h.rename("bob", id, "rename-1", "Renamed", h.revision("bob", id))
|
||||
Expect(rc.code).To(Equal(200), rc.raw)
|
||||
h.stopTables()
|
||||
// The typed ID and the route name another add-on than the backup's.
|
||||
for _, path := range []string{"verify", "delete-backup"} {
|
||||
r := h.admin("/api/addons/notes/recovery/"+path, "x-"+path, map[string]any{"backupId": backupID})
|
||||
Expect(r.code).To(Equal(404), path+": "+r.raw)
|
||||
}
|
||||
bad := h.admin("/api/addons/notes/recovery/restore", "r-x", map[string]any{"backupId": backupID, "confirm": "notes"})
|
||||
Expect(bad.code).To(Equal(400), bad.raw)
|
||||
Expect(bad.errType()).To(Equal("invalid_request"))
|
||||
bad = h.admin("/api/addons/notes/recovery/restore", "r-y", map[string]any{"backupId": backupID, "confirm": "tables"})
|
||||
Expect(bad.code).To(Equal(400), bad.raw)
|
||||
Expect(filepath.Join(h.owner.Root, "backups", "tables", backupID)).To(BeADirectory())
|
||||
entries, _ := os.ReadDir(filepath.Join(h.owner.Root, "recovery", "tables"))
|
||||
for _, en := range entries {
|
||||
Expect(en.Name()).NotTo(HavePrefix("previous-"))
|
||||
}
|
||||
h.startTables()
|
||||
// Nothing was swapped in: the rename made after the backup stands.
|
||||
name, e := h.name("bob", id)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Renamed"))
|
||||
})
|
||||
|
||||
It("gives a forwarding frontend the same typed answers as the owner", func() {
|
||||
h := ready(false)
|
||||
h.database("bob", "db-1", "Sales")
|
||||
front := h.frontend()
|
||||
backup := func(web *echo.Echo, op string) reply {
|
||||
GinkgoHelper()
|
||||
b, _ := json.Marshal(map[string]any{"operationId": op})
|
||||
return serve(web, "POST", "/api/addons/tables/recovery/backup", string(b), h.tab("alice", op))
|
||||
}
|
||||
ownerGot := backup(h.web, "p-1")
|
||||
frontGot := backup(front, "p-2")
|
||||
Expect(ownerGot.code).To(Equal(200), ownerGot.raw)
|
||||
Expect(frontGot.code).To(Equal(200), frontGot.raw)
|
||||
// A restore of a running add-on is refused the same way from both.
|
||||
ask := func(web *echo.Echo, op string) reply {
|
||||
b, _ := json.Marshal(map[string]any{"operationId": op, "backupId": ownerGot.body["id"], "confirm": "tables"})
|
||||
return serve(web, "POST", "/api/addons/tables/recovery/restore", string(b), h.tab("alice", op))
|
||||
}
|
||||
a, b := ask(h.web, "q-1"), ask(front, "q-2")
|
||||
Expect(a.code).To(Equal(409))
|
||||
Expect(b.code).To(Equal(a.code))
|
||||
Expect(b.errType()).To(Equal(a.errType()))
|
||||
// A person who is not an administrator is denied the same way.
|
||||
bob := serve(front, "GET", "/api/addons/tables/recovery", "", h.session("bob"))
|
||||
Expect(bob.code).To(Equal(403))
|
||||
list := serve(front, "GET", "/api/addons/tables/recovery", "", h.session("alice"))
|
||||
Expect(list.code).To(Equal(200), list.raw)
|
||||
Expect(backups(list)).To(HaveLen(2))
|
||||
})
|
||||
|
||||
It("purges only with the typed ID, a stopped add-on and an explicit scope, and touches nothing else", func() {
|
||||
h := ready(false)
|
||||
id := h.database("bob", "db-1", "Sales")
|
||||
backupID := h.admin("/api/addons/tables/recovery/backup", "b-1", map[string]any{}).body["id"].(string)
|
||||
// Running: refused.
|
||||
got := h.admin("/api/addons/tables/purge", "x-1", map[string]any{"scope": map[string]any{"data": true}, "confirm": "tables"})
|
||||
Expect(got.code).To(Equal(409))
|
||||
Expect(got.errType()).To(Equal("addon_running"))
|
||||
h.stopTables()
|
||||
Expect(h.admin("/api/addons/tables/purge", "x-2", map[string]any{"scope": map[string]any{"data": true}, "confirm": "nope"}).errType()).To(Equal("confirmation_mismatch"))
|
||||
Expect(h.admin("/api/addons/tables/purge", "x-3", map[string]any{"scope": map[string]any{}, "confirm": "tables"}).errType()).To(Equal("invalid_request"))
|
||||
// A person with the feature cannot, and neither can a database
|
||||
// deletion or an uninstall do it: the data is still there.
|
||||
Expect(h.send("POST", "/api/addons/tables/purge", `{"operationId":"x-4","scope":{"data":true},"confirm":"tables"}`, h.tab("bob", "x-4")).code).To(Equal(403))
|
||||
Expect(h.dbFile(id)).To(BeAnExistingFile())
|
||||
done := h.admin("/api/addons/tables/purge", "x-5", map[string]any{"scope": map[string]any{"data": true}, "confirm": "tables"})
|
||||
Expect(done.code).To(Equal(200), done.raw)
|
||||
Expect(h.dbFile(id)).NotTo(BeAnExistingFile())
|
||||
// The backups were not in the scope.
|
||||
Expect(backups(h.status("alice"))).To(HaveLen(1))
|
||||
Expect(filepath.Join(h.owner.Root, "backups", "tables", backupID)).To(BeADirectory())
|
||||
// The owner no longer lists the database for its person.
|
||||
Expect(resourcesOf(h.list("bob"))).To(BeEmpty())
|
||||
bk := h.admin("/api/addons/tables/purge", "x-6", map[string]any{"scope": map[string]any{"backups": true}, "confirm": "tables"})
|
||||
Expect(bk.code).To(Equal(200), bk.raw)
|
||||
Expect(backups(h.status("alice"))).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("is never reachable by a model: no tool, no declared name and no route for an API key", func() {
|
||||
h := ready(false)
|
||||
h.database("bob", "db-1", "Sales")
|
||||
adapter := tools.NewToolAdapter(h.app.AddonGateway())
|
||||
defs, e := adapter.Discover(context.Background(), h.actor("alice"))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
for _, d := range defs {
|
||||
Expect(d.Name).NotTo(ContainSubstring("recovery"))
|
||||
Expect(d.Name).NotTo(ContainSubstring("purge"))
|
||||
Expect(d.Name).NotTo(ContainSubstring("backup"))
|
||||
Expect(d.Name).NotTo(ContainSubstring("restore"))
|
||||
}
|
||||
for _, name := range []string{"addon__tables__recovery-backup", "addon__tables__purge", "addon__tables__recovery-restore", "addon__tables__recovery-status"} {
|
||||
_, e := adapter.Execute(context.Background(), h.actor("alice"), name, json.RawMessage(`{}`))
|
||||
Expect(e).To(HaveOccurred(), name)
|
||||
}
|
||||
// The typed dispatch of a key names the action as an administrator
|
||||
// action, but a key never reaches the human-surface mutations.
|
||||
for _, path := range []string{"/api/addons/tables/recovery/backup", "/api/addons/tables/recovery/restore", "/api/addons/tables/purge", "/api/addons/tables/recovery/prune"} {
|
||||
got := h.send("POST", path, `{"operationId":"k-1"}`, h.bearer("alice"))
|
||||
Expect(got.code).To(Equal(403), path)
|
||||
}
|
||||
// Nothing was made by those calls.
|
||||
Expect(backups(h.status("alice"))).To(BeEmpty())
|
||||
var manifest struct {
|
||||
Capabilities struct {
|
||||
Tools []string `json:"tools"`
|
||||
AdminTools []string `json:"adminTools"`
|
||||
} `json:"capabilities"`
|
||||
}
|
||||
raw, e := os.ReadFile("../../../addons/tables/manifest.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(json.Unmarshal(raw, &manifest)).To(Succeed())
|
||||
for _, t := range append(manifest.Capabilities.Tools, manifest.Capabilities.AdminTools...) {
|
||||
Expect(t).NotTo(MatchRegexp(`recovery|purge|backup|restore`))
|
||||
}
|
||||
})
|
||||
|
||||
It("upgrades an add-on with data through a backup and a migration and keeps its databases", func() {
|
||||
h := ready(false)
|
||||
id := h.database("bob", "db-1", "Sales")
|
||||
Expect(h.grant("bob", id, "carol", "viewer").code).To(Equal(200))
|
||||
before := h.generation()
|
||||
_, e := h.app.AddonManager().Submit(context.Background(), h.actor("alice"), addonsvc.LifecycleCommand{OperationID: "upgrade-1", AddonID: "tables", Action: "upgrade", SourceID: "local", Version: "0.1.1"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(h.generation()).To(Equal(before + 1))
|
||||
h.running()
|
||||
// The data is intact and the shares are as they were.
|
||||
name, e := h.name("carol", id)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Sales"))
|
||||
// The add-on migrated its own data at start and said so.
|
||||
raw, e := os.ReadFile(filepath.Join(h.owner.Root, "data", "tables", "migration.json"))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(string(raw)).To(ContainSubstring(`"complete"`))
|
||||
// The backup taken before is the recovery copy and is kept.
|
||||
list := backups(h.status("alice"))
|
||||
Expect(list).To(HaveLen(1))
|
||||
Expect(list[0]["reason"]).To(Equal("upgrade"))
|
||||
Expect(list[0]["pinned"]).To(BeTrue())
|
||||
Expect(list[0]["version"]).To(Equal("0.1.0"))
|
||||
st := h.status("alice")
|
||||
Expect(st.body["state"]).To(Equal("ready"))
|
||||
// The sole recovery copy cannot be deleted.
|
||||
del := h.admin("/api/addons/tables/recovery/delete-backup", "d-1", map[string]any{"backupId": list[0]["id"]})
|
||||
Expect(del.code).To(Equal(409))
|
||||
Expect(del.errType()).To(Equal("sole_recovery_copy"))
|
||||
})
|
||||
|
||||
It("undoes an upgrade whose package does not start and runs the old generation on its own data", func() {
|
||||
h := ready(false)
|
||||
id := h.database("bob", "db-1", "Sales")
|
||||
before := h.generation()
|
||||
_, e := h.app.AddonManager().Submit(context.Background(), h.actor("alice"), addonsvc.LifecycleCommand{OperationID: "upgrade-bad", AddonID: "tables", Action: "upgrade", SourceID: "local", Version: "0.2.0"})
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(h.generation()).To(Equal(before))
|
||||
h.running()
|
||||
name, e := h.name("bob", id)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Sales"))
|
||||
st := h.status("alice")
|
||||
Expect(st.body["state"]).To(Equal("ready"))
|
||||
// The backup that made it safe is still there.
|
||||
Expect(backups(st)).To(HaveLen(1))
|
||||
// A new database can be made: nothing is blocked.
|
||||
Expect(h.create("bob", "db-2", "More").code).To(Equal(200))
|
||||
})
|
||||
|
||||
It("hands retained data only to the same trusted source on a reinstall", func() {
|
||||
h := ready(false)
|
||||
id := h.database("bob", "db-1", "Sales")
|
||||
ctx := context.Background()
|
||||
st, e := h.app.AddonManager().Status(ctx, "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = h.app.AddonManager().Submit(ctx, h.actor("alice"), addonsvc.LifecycleCommand{OperationID: "uninstall-1", AddonID: "tables", Action: "uninstall"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_ = st
|
||||
// The same library under another source ID is another trusted
|
||||
// identity: the data stays where it is.
|
||||
lib := h.libraryURL
|
||||
Expect(h.app.AddonSources().Save(ctx, h.actor("alice"), catalog.Source{ID: "mirror", Name: "Mirror", URL: lib, Enabled: true, AllowLoopbackHTTP: true})).To(Succeed())
|
||||
_, e = h.app.AddonSources().Refresh(ctx, "mirror")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = h.app.AddonManager().Submit(ctx, h.actor("alice"), addonsvc.LifecycleCommand{OperationID: "install-other", AddonID: "tables", Action: "install", SourceID: "mirror", Version: "0.1.0"})
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(h.dbFile(id)).To(BeAnExistingFile())
|
||||
// The original source is accepted and the data is served again.
|
||||
_, e = h.app.AddonManager().Submit(ctx, h.actor("alice"), addonsvc.LifecycleCommand{OperationID: "install-same", AddonID: "tables", Action: "install", SourceID: "local", Version: "0.1.0"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
sts, e := h.app.AddonManager().Status(ctx, "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = h.app.AddonManager().Submit(ctx, h.actor("alice"), addonsvc.LifecycleCommand{OperationID: "enable-again", AddonID: "tables", Action: "enable", Generation: sts.Generation})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.running()
|
||||
name, e := h.name("bob", id)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Sales"))
|
||||
})
|
||||
|
||||
It("documents every route and every emitted field in the Swagger spec, the instructions and the feature registry", func() {
|
||||
spec, err := os.ReadFile("../../../swagger/swagger.json")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
var doc struct {
|
||||
Paths map[string]map[string]json.RawMessage `json:"paths"`
|
||||
Definitions map[string]struct {
|
||||
Properties map[string]any `json:"properties"`
|
||||
} `json:"definitions"`
|
||||
}
|
||||
Expect(json.Unmarshal(spec, &doc)).To(Succeed())
|
||||
for path, method := range map[string]string{
|
||||
"/api/addons/{id}/recovery": "get",
|
||||
"/api/addons/{id}/recovery/backup": "post",
|
||||
"/api/addons/{id}/recovery/verify": "post",
|
||||
"/api/addons/{id}/recovery/restore": "post",
|
||||
"/api/addons/{id}/recovery/delete-backup": "post",
|
||||
"/api/addons/{id}/recovery/prune": "post",
|
||||
"/api/addons/{id}/purge": "post",
|
||||
} {
|
||||
Expect(doc.Paths).To(HaveKey(path))
|
||||
text := string(doc.Paths[path][method])
|
||||
Expect(text).To(ContainSubstring("MCP"), path)
|
||||
Expect(text).To(ContainSubstring(`"Addons"`), path)
|
||||
}
|
||||
h := ready(false)
|
||||
h.database("bob", "db-1", "Sales")
|
||||
backup := h.admin("/api/addons/tables/recovery/backup", "b-1", map[string]any{})
|
||||
Expect(backup.code).To(Equal(200))
|
||||
// Emitted bodies and documented definitions list the same fields.
|
||||
check := func(name string, got map[string]any) {
|
||||
GinkgoHelper()
|
||||
def, ok := doc.Definitions[name]
|
||||
Expect(ok).To(BeTrue(), name)
|
||||
for field := range got {
|
||||
Expect(def.Properties).To(HaveKey(field), name+"."+field)
|
||||
}
|
||||
}
|
||||
check("recovery.BackupReceipt", backup.body)
|
||||
st := h.status("alice")
|
||||
check("recovery.Status", st.body)
|
||||
for _, b := range backups(st) {
|
||||
check("recovery.BackupInfo", b)
|
||||
}
|
||||
h.stopTables()
|
||||
rec := h.admin("/api/addons/tables/recovery/restore", "r-1", map[string]any{"backupId": backup.body["id"], "confirm": "tables"})
|
||||
Expect(rec.code).To(Equal(200), rec.raw)
|
||||
check("recovery.RestoreReceipt", rec.body)
|
||||
check("recovery.RestoreReport", rec.body["policy"].(map[string]any))
|
||||
purge := h.admin("/api/addons/tables/purge", "p-1", map[string]any{"scope": map[string]any{"recoveryCopies": true}, "confirm": "tables"})
|
||||
Expect(purge.code).To(Equal(200), purge.raw)
|
||||
check("recovery.PurgeReceipt", purge.body)
|
||||
check("recovery.PurgeScope", purge.body["scope"].(map[string]any))
|
||||
registered := map[string]bool{}
|
||||
for _, entry := range auth.RouteFeatureRegistry {
|
||||
if entry.Feature == auth.FeatureAddons {
|
||||
registered[entry.Method+" "+entry.Pattern] = true
|
||||
}
|
||||
}
|
||||
for _, key := range []string{"GET /api/addons/:id/recovery", "POST /api/addons/:id/recovery/backup", "POST /api/addons/:id/recovery/verify", "POST /api/addons/:id/recovery/restore", "POST /api/addons/:id/recovery/delete-backup", "POST /api/addons/:id/recovery/prune", "POST /api/addons/:id/purge"} {
|
||||
Expect(registered).To(HaveKey(key))
|
||||
}
|
||||
ins := h.send("GET", "/api/instructions/addons", "", h.bearer("alice"))
|
||||
Expect(ins.code).To(Equal(200))
|
||||
for _, want := range []string{"/api/addons/{id}/recovery", "needs-recovery", "purge", "VACUUM"} {
|
||||
if want == "VACUUM" {
|
||||
continue
|
||||
}
|
||||
Expect(ins.raw).To(ContainSubstring(want), want)
|
||||
}
|
||||
docs, err := os.ReadFile("../../../docs/content/features/addons.md")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
for _, want := range []string{"needs-recovery", "sole recovery copy", "retained data", "ownership mapping", "migration.json"} {
|
||||
Expect(strings.ToLower(string(docs))).To(ContainSubstring(strings.ToLower(want)), want)
|
||||
}
|
||||
_ = policy.ModeNamedUser
|
||||
_ = contract.Actor{}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,393 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tables_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/ed25519"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"math/big"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
goruntime "runtime"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/labstack/echo/v4"
|
||||
"github.com/mudler/LocalAI/core/application"
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
httpapi "github.com/mudler/LocalAI/core/http"
|
||||
"github.com/mudler/LocalAI/core/http/auth"
|
||||
addonsvc "github.com/mudler/LocalAI/core/services/addons"
|
||||
"github.com/mudler/LocalAI/core/services/addons/catalog"
|
||||
"github.com/mudler/LocalAI/core/services/addons/gateway"
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
"github.com/mudler/LocalAI/pkg/system"
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
const appSecret = "tables-application-secret"
|
||||
|
||||
// ownerCarrier builds the smallest real transport identity of an owner: one
|
||||
// pinned certificate, one signing key and a loopback listen address.
|
||||
func ownerCarrier() *gateway.CarrierConfig {
|
||||
GinkgoHelper()
|
||||
pub, key, e := ed25519.GenerateKey(rand.Reader)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
ca := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "test CA"}, NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour), IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign}
|
||||
der, e := x509.CreateCertificate(rand.Reader, ca, ca, pub, key)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
ca, e = x509.ParseCertificate(der)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
pool := x509.NewCertPool()
|
||||
pool.AddCert(ca)
|
||||
lp, lk, e := ed25519.GenerateKey(rand.Reader)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
leaf := &x509.Certificate{SerialNumber: big.NewInt(2), Subject: pkix.Name{CommonName: "owner"}, NotBefore: ca.NotBefore, NotAfter: ca.NotAfter, IPAddresses: []net.IP{net.ParseIP("127.0.0.1")}, KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth}}
|
||||
d, e := x509.CreateCertificate(rand.Reader, leaf, ca, lp, key)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
kd, e := x509.MarshalPKCS8PrivateKey(lk)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
pair, e := tls.X509KeyPair(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: d}), pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: kd}))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
sp, sk, e := ed25519.GenerateKey(rand.Reader)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return &gateway.CarrierConfig{InstanceID: "owner", OwnerID: "owner", OwnerAddress: "https://owner.example", Certificate: pair, Roots: pool,
|
||||
Pins: map[string][32]byte{"owner": sha256.Sum256(d)}, SigningKey: sk, VerificationKeys: map[string]ed25519.PublicKey{"owner": sp}}
|
||||
}
|
||||
|
||||
// host is one owner process-in-process: the real application bootstrap over
|
||||
// real account storage, the real supervisor and the real Tables executable.
|
||||
type host struct {
|
||||
dir string
|
||||
owner config.AddonConfig
|
||||
app *application.Application
|
||||
web *echo.Echo
|
||||
cancel context.CancelFunc
|
||||
// Credentials of the accounts, created once and kept across restarts.
|
||||
tokens map[string]string
|
||||
sessions map[string]string
|
||||
keys map[string]string
|
||||
}
|
||||
|
||||
func newHost() *host {
|
||||
GinkgoHelper()
|
||||
dir := GinkgoT().TempDir()
|
||||
h := &host{dir: dir, tokens: map[string]string{}, sessions: map[string]string{}, keys: map[string]string{}}
|
||||
root := filepath.Join(dir, "addons")
|
||||
// Package directories are read-only; restore permissions before removal.
|
||||
DeferCleanup(func() {
|
||||
_ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, e error) error {
|
||||
if e == nil && d.IsDir() {
|
||||
_ = os.Chmod(p, 0o700)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
})
|
||||
h.owner = config.AddonConfig{
|
||||
Enabled: true, InstanceID: "owner", OwnerID: "owner", OwnerAddress: "https://owner.example", StorageID: "local",
|
||||
Root: root, ListenAddress: "127.0.0.1:0", Transport: ownerCarrier(),
|
||||
Host: contract.HostCompatibility{Host: 1, API: 1, Protocol: 1, SDK: 1, DataFormat: 1, OS: goruntime.GOOS, Arch: goruntime.GOARCH},
|
||||
}
|
||||
return h
|
||||
}
|
||||
|
||||
// start runs the application bootstrap, which reconciles reservations and
|
||||
// recovers proposals before it serves.
|
||||
func (h *host) start() {
|
||||
GinkgoHelper()
|
||||
models, backends := filepath.Join(h.dir, "models"), filepath.Join(h.dir, "backends")
|
||||
Expect(os.MkdirAll(models, 0o750)).To(Succeed())
|
||||
Expect(os.MkdirAll(backends, 0o750)).To(Succeed())
|
||||
state, e := system.GetSystemState(system.WithBackendPath(backends), system.WithModelPath(models))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
h.cancel = cancel
|
||||
app, e := application.New(
|
||||
config.WithContext(ctx), config.WithSystemState(state), config.WithDataPath(h.dir),
|
||||
config.WithAuthEnabled(true), config.WithAuthDatabaseURL(filepath.Join(h.dir, "auth.db")), config.WithAuthAPIKeyHMACSecret(appSecret),
|
||||
config.WithAddons(h.owner),
|
||||
)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.app = app
|
||||
h.web, e = httpapi.API(app)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
}
|
||||
|
||||
// stop ends the application as a restart does and waits until the owner
|
||||
// released its durable state.
|
||||
func (h *host) stop() {
|
||||
GinkgoHelper()
|
||||
if h.cancel == nil {
|
||||
return
|
||||
}
|
||||
Expect(h.app.Shutdown()).To(Succeed())
|
||||
h.cancel()
|
||||
h.cancel = nil
|
||||
Eventually(func() error {
|
||||
s, e := policy.OpenStore(filepath.Join(h.owner.Root, "policy"), policy.ModeNamedUser)
|
||||
if e == nil {
|
||||
_ = s.Close()
|
||||
}
|
||||
return e
|
||||
}, 10*time.Second, 50*time.Millisecond).Should(Succeed())
|
||||
}
|
||||
|
||||
func (h *host) actor(user string) contract.Actor {
|
||||
return contract.Actor{UserID: user, CredentialID: "key:" + h.keys[user], Mode: auth.AddonModeNamedUser}
|
||||
}
|
||||
|
||||
// seed creates the accounts, installs the real Tables package from a local
|
||||
// catalog and waits until it runs.
|
||||
func (h *host) seed(lib *cluster.Library) {
|
||||
GinkgoHelper()
|
||||
ctx := context.Background()
|
||||
db := h.app.AuthDB()
|
||||
Expect(db).NotTo(BeNil())
|
||||
for _, u := range []struct {
|
||||
id, role string
|
||||
feature bool
|
||||
}{{"alice", auth.RoleAdmin, false}, {"bob", auth.RoleUser, true}} {
|
||||
Expect(db.Create(&auth.User{ID: u.id, Email: u.id + "@tables.test", Name: u.id, Provider: auth.ProviderLocal, Role: u.role, Status: auth.StatusActive}).Error).To(Succeed())
|
||||
if u.feature {
|
||||
Expect(db.Create(&auth.UserPermission{ID: uuid.NewString(), UserID: u.id, Permissions: auth.PermissionMap{auth.FeatureAddons: true}}).Error).To(Succeed())
|
||||
}
|
||||
token, key, e := auth.CreateAPIKey(db, u.id, "agent", u.role, appSecret, nil)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
session, e := auth.CreateSession(db, u.id, appSecret)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.tokens[u.id], h.sessions[u.id], h.keys[u.id] = token, session, key.ID
|
||||
}
|
||||
admin := h.actor("alice")
|
||||
Expect(h.app.AddonSources().Save(ctx, admin, catalog.Source{ID: "local", Name: "Local", URL: lib.URL(), Enabled: true, AllowLoopbackHTTP: true})).To(Succeed())
|
||||
_, e := h.app.AddonSources().Refresh(ctx, "local")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m := h.app.AddonManager()
|
||||
_, e = m.Submit(ctx, admin, addonsvc.LifecycleCommand{OperationID: "install-tables", AddonID: "tables", Action: "install", SourceID: "local", Version: "0.1.0"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = m.Submit(ctx, admin, addonsvc.LifecycleCommand{OperationID: "enable-tables", AddonID: "tables", Action: "enable", Generation: 1})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.running()
|
||||
}
|
||||
|
||||
// running waits until the Tables process is observed running.
|
||||
func (h *host) running() {
|
||||
GinkgoHelper()
|
||||
Eventually(func() string {
|
||||
list, e := h.app.AddonManager().Apps(context.Background(), h.actor("alice"))
|
||||
if e != nil {
|
||||
return e.Error()
|
||||
}
|
||||
for _, a := range list {
|
||||
if a.ID == "tables" {
|
||||
return a.State
|
||||
}
|
||||
}
|
||||
return "absent"
|
||||
}, 30*time.Second, 100*time.Millisecond).Should(Equal("running"))
|
||||
}
|
||||
|
||||
type webReply struct {
|
||||
code int
|
||||
body map[string]any
|
||||
}
|
||||
|
||||
func (h *host) send(method, path, body string, headers func(*http.Request)) webReply {
|
||||
GinkgoHelper()
|
||||
r := httptest.NewRequest(method, path, bytes.NewBufferString(body))
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
if headers != nil {
|
||||
headers(r)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
h.web.ServeHTTP(w, r)
|
||||
out := webReply{code: w.Code, body: map[string]any{}}
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &out.body)
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *host) bearer(user string) func(*http.Request) {
|
||||
return func(r *http.Request) { r.Header.Set("Authorization", "Bearer "+h.tokens[user]) }
|
||||
}
|
||||
|
||||
// humanTab is what the host confirmation surface sends: the user's own browser
|
||||
// session, a same-origin fetch and the proposal named in the header.
|
||||
func (h *host) humanTab(user, id string) func(*http.Request) {
|
||||
return func(r *http.Request) {
|
||||
r.AddCookie(&http.Cookie{Name: "session", Value: h.sessions[user]})
|
||||
r.Header.Set("Sec-Fetch-Site", "same-origin")
|
||||
r.Header.Set("Sec-Fetch-Mode", "cors")
|
||||
r.Header.Set("Sec-Fetch-Dest", "empty")
|
||||
r.Header.Set("X-LocalAI-Confirm", id)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *host) chatCreate(user, op, name string) webReply {
|
||||
GinkgoHelper()
|
||||
body := fmt.Sprintf(`{"addonId":"tables","operationId":%q,"generation":1,"action":"create","arguments":{"tool":"databases-create","name":%q}}`, op, name)
|
||||
return h.send("POST", "/api/addons/proposals", body, h.bearer(user))
|
||||
}
|
||||
|
||||
func (h *host) confirm(user, id, digest string) webReply {
|
||||
GinkgoHelper()
|
||||
return h.send("POST", "/api/addons/proposals/"+id+"/confirm", fmt.Sprintf(`{"digest":%q}`, digest), h.humanTab(user, id))
|
||||
}
|
||||
|
||||
// databaseName reads the name of a database through the owner's gateway, as
|
||||
// the person who may read it.
|
||||
func (h *host) databaseName(user, resource string) (string, error) {
|
||||
res, e := h.app.AddonGateway().Dispatch(context.Background(), h.actor(user), contract.Request{
|
||||
AddonID: "tables", ResourceID: resource, OperationID: "inspect-" + uuid.NewString()[:8], Generation: 1, Action: "read",
|
||||
Arguments: json.RawMessage(`{"tool":"databases-inspect"}`),
|
||||
})
|
||||
if e != nil {
|
||||
return "", e
|
||||
}
|
||||
var out struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
Expect(json.Unmarshal(res.Payload, &out)).To(Succeed())
|
||||
return out.Name, nil
|
||||
}
|
||||
|
||||
func (h *host) library() *cluster.Library {
|
||||
GinkgoHelper()
|
||||
lib, e := cluster.NewLibrary(binary)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
DeferCleanup(lib.Close)
|
||||
raw, e := os.ReadFile("../../../addons/tables/manifest.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var m struct {
|
||||
Capabilities map[string]any `json:"capabilities"`
|
||||
}
|
||||
Expect(json.Unmarshal(raw, &m)).To(Succeed())
|
||||
// These specs run the tools, not the page: the package carries no UI
|
||||
// files, so the declaration of them is left out. The UI suite installs
|
||||
// the whole package.
|
||||
delete(m.Capabilities, "ui")
|
||||
exe, e := os.ReadFile(binary)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(lib.Add(cluster.Package{ID: "tables", Version: "0.1.0", Capabilities: m.Capabilities, Executable: "bin/tables", Binary: exe}, GinkgoT().TempDir())).To(Succeed())
|
||||
return lib
|
||||
}
|
||||
|
||||
func accessibleTo(h *host, user string) []string {
|
||||
GinkgoHelper()
|
||||
keys, e := h.app.AddonPolicy().Accessible(context.Background(), h.actor(user), "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var ids []string
|
||||
for _, k := range keys {
|
||||
ids = append(ids, k.ResourceID)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
var _ = Describe("Tables in the application bootstrap", Label("application"), func() {
|
||||
It("creates a database from a chat proposal that only the proposer's session confirms", func() {
|
||||
h := newHost()
|
||||
lib := h.library()
|
||||
h.start()
|
||||
DeferCleanup(h.stop)
|
||||
h.seed(lib)
|
||||
|
||||
p := h.chatCreate("alice", "chat-create-1", "Sales")
|
||||
Expect(p.code).To(Equal(200), "%v", p.body)
|
||||
Expect(p.body["state"]).To(Equal("pending"))
|
||||
Expect(p.body["action"]).To(Equal("create"))
|
||||
id, digest, resource := p.body["id"].(string), p.body["digest"].(string), p.body["resourceId"].(string)
|
||||
Expect(resource).NotTo(BeEmpty())
|
||||
// The reservation is reachable by nobody, not even by its proposer.
|
||||
Expect(accessibleTo(h, "alice")).To(BeEmpty())
|
||||
_, e := h.databaseName("alice", resource)
|
||||
Expect(e).To(HaveOccurred())
|
||||
// An API key and another user's session cannot approve it.
|
||||
Expect(h.send("POST", "/api/addons/proposals/"+id+"/confirm", fmt.Sprintf(`{"digest":%q}`, digest), h.bearer("alice")).code).To(Equal(403))
|
||||
Expect(h.confirm("bob", id, digest).code).To(Equal(403))
|
||||
Expect(accessibleTo(h, "alice")).To(BeEmpty())
|
||||
|
||||
done := h.confirm("alice", id, digest)
|
||||
Expect(done.code).To(Equal(200), "%v", done.body)
|
||||
Expect(done.body["state"]).To(Equal("applied"))
|
||||
Expect(accessibleTo(h, "alice")).To(Equal([]string{resource}))
|
||||
Expect(accessibleTo(h, "bob")).To(BeEmpty())
|
||||
name, e := h.databaseName("alice", resource)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Sales"))
|
||||
_, e = h.databaseName("bob", resource)
|
||||
Expect(e).To(HaveOccurred())
|
||||
// A withdrawn proposal leaves nothing behind.
|
||||
q := h.chatCreate("alice", "chat-create-2", "Scratch")
|
||||
Expect(q.code).To(Equal(200))
|
||||
Expect(h.send("POST", "/api/addons/proposals/chat-create-2/cancel", "", h.bearer("alice")).code).To(Equal(200))
|
||||
pending, e := h.app.AddonPolicy().Pending()
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(pending).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("settles reservations at bootstrap by asking the real addon and keeps those of pending proposals", func() {
|
||||
h := newHost()
|
||||
lib := h.library()
|
||||
h.start()
|
||||
DeferCleanup(h.stop)
|
||||
h.seed(lib)
|
||||
ctx := context.Background()
|
||||
alice := h.actor("alice")
|
||||
p := h.app.AddonPolicy()
|
||||
m := h.app.AddonManager()
|
||||
|
||||
// A crash after the addon created the database and before the commit.
|
||||
made, e := p.Register(ctx, alice, "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
create := policy.RuntimeCreate(m, m, contract.DefaultLimits(), alice, json.RawMessage(`{"tool":"databases-create","name":"Recovered"}`))
|
||||
Expect(create(ctx, made)).To(Succeed())
|
||||
// A crash after the reservation and before any database.
|
||||
lost, e := p.Register(ctx, alice, "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
// A chat proposal that nobody confirmed yet.
|
||||
live := h.chatCreate("alice", "chat-create-live", "Later")
|
||||
Expect(live.code).To(Equal(200), "%v", live.body)
|
||||
held := live.body["resourceId"].(string)
|
||||
before, e := p.Pending()
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(before).To(ConsistOf(made.ResourceID, lost.ResourceID, held))
|
||||
|
||||
h.stop()
|
||||
h.start()
|
||||
h.running()
|
||||
|
||||
p = h.app.AddonPolicy()
|
||||
// The real addon said the database exists: committed for its owner.
|
||||
Expect(accessibleTo(h, "alice")).To(Equal([]string{made.ResourceID}))
|
||||
name, e := h.databaseName("alice", made.ResourceID)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Recovered"))
|
||||
// It said nothing exists: dropped. The live proposal keeps its own.
|
||||
pending, e := p.Pending()
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(pending).To(Equal([]string{held}))
|
||||
// The proposal is still confirmable after the restart.
|
||||
got := h.send("GET", "/api/addons/proposals/chat-create-live", "", h.bearer("alice"))
|
||||
Expect(got.code).To(Equal(200), "%v", got.body)
|
||||
Expect(got.body["state"]).To(Equal("pending"))
|
||||
done := h.confirm("alice", "chat-create-live", got.body["digest"].(string))
|
||||
Expect(done.code).To(Equal(200), "%v", done.body)
|
||||
Expect(accessibleTo(h, "alice")).To(ConsistOf(made.ResourceID, held))
|
||||
name, e = h.databaseName("alice", held)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Later"))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,270 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tables_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("Tables databases under the real owner", func() {
|
||||
ctx := context.Background()
|
||||
alice, bob := user("alice"), user("bob")
|
||||
|
||||
It("starts the real executable through the authenticated handshake and refuses unauthenticated callers", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
regs, e := r.sup.Registry.Snapshot(ctx)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(regs).To(HaveLen(1))
|
||||
Expect(regs[0].AddonID).To(Equal("tables"))
|
||||
Expect(regs[0].Capabilities.Tools).To(ConsistOf("resource-exists", "databases-create", "databases-rename", "databases-inspect", "tables-create", "tables-rename", "tables-drop", "columns-add", "columns-rename", "columns-drop", "rows-query", "rows-aggregate", "rows-export", "rows-insert", "rows-update", "rows-delete"))
|
||||
// The package declares the native pages: the entry, the style sheet, the
|
||||
// one page and the chat action that selects a database.
|
||||
ui := regs[0].Capabilities.UI
|
||||
Expect(ui).NotTo(BeNil())
|
||||
Expect(ui.Entry).To(Equal("ui/tables.js"))
|
||||
Expect(ui.CSS).To(Equal([]string{"ui/tables.css"}))
|
||||
Expect(ui.Pages).To(Equal([]string{"databases"}))
|
||||
Expect(ui.ChatActions).To(HaveLen(1))
|
||||
Expect(ui.ChatActions[0].ID).To(Equal("select-database"))
|
||||
// The data directory the owner created is private and holds the addon's data.
|
||||
fi, e := os.Stat(filepath.Join(r.dataRoot, "tables"))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(fi.Mode().Perm()).To(Equal(os.FileMode(0o700)))
|
||||
// Anyone who reaches the loopback port without the credential is denied.
|
||||
for _, h := range []map[string]string{
|
||||
{},
|
||||
{"X-LocalAI-Incarnation": regs[0].Incarnation},
|
||||
{"X-LocalAI-Incarnation": "stale", "Authorization": "Bearer guess"},
|
||||
} {
|
||||
for _, path := range []string{"/health", "/read", "/apply", "/receipt"} {
|
||||
req, _ := http.NewRequest("POST", regs[0].Endpoint+path, bytes.NewReader([]byte(`{}`)))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
for k, v := range h {
|
||||
req.Header.Set(k, v)
|
||||
}
|
||||
resp, e := http.DefaultClient.Do(req)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
_ = resp.Body.Close()
|
||||
Expect(resp.StatusCode).To(Equal(http.StatusUnauthorized), path)
|
||||
}
|
||||
}
|
||||
_ = inst
|
||||
})
|
||||
|
||||
It("creates a database under the host-assigned ID and lists it only for people who may read it", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
key, e := r.create(inst, alice, "Sales")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(key.AddonID).To(Equal("tables"))
|
||||
fi, e := os.Stat(r.dbFile(key.ResourceID))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(fi.Mode().Perm()).To(Equal(os.FileMode(0o600)))
|
||||
Expect(r.policy.Authorize(ctx, alice, key, policy.ActionSchema)).To(Succeed())
|
||||
Expect(r.policy.Authorize(ctx, bob, key, policy.ActionRead)).To(MatchError(policy.ErrForbidden))
|
||||
mine, e := r.policy.Accessible(ctx, alice, "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(mine).To(Equal([]policy.ResourceKey{key}))
|
||||
theirs, e := r.policy.Accessible(ctx, bob, "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(theirs).To(BeEmpty())
|
||||
res, e := r.read(inst, alice, key.ResourceID, "databases-inspect")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(schemaName(res)).To(Equal("Sales"))
|
||||
// Sharing is the host's decision and widens the listing.
|
||||
Expect(r.policy.SetGrant(ctx, alice, key, "bob", policy.RoleViewer)).To(Succeed())
|
||||
theirs, _ = r.policy.Accessible(ctx, bob, "tables")
|
||||
Expect(theirs).To(Equal([]policy.ResourceKey{key}))
|
||||
})
|
||||
|
||||
It("drops the reservation when the addon refuses the name, and leaves no database", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
_, e := r.create(inst, alice, " ")
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(e).To(MatchError(policy.ErrNotCreated))
|
||||
got, _ := r.policy.Accessible(ctx, alice, "tables")
|
||||
Expect(got).To(BeEmpty())
|
||||
Expect(r.reservations()).To(BeEmpty())
|
||||
Expect(filesIn(filepath.Join(r.dataRoot, "tables", "tables", "databases"))).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("drops a reservation at reconcile when the crash came before the database was created", func() {
|
||||
r := newRig()
|
||||
r.start(1)
|
||||
key, e := r.policy.Register(ctx, alice, "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
// Pending is invisible to its own owner.
|
||||
Expect(r.policy.Authorize(ctx, alice, key, policy.ActionRead)).To(MatchError(policy.ErrForbidden))
|
||||
r.closePolicy()
|
||||
r.openPolicy()
|
||||
Expect(r.policy.Reconcile(ctx, r.backing())).To(Succeed())
|
||||
Expect(r.policy.Authorize(ctx, alice, key, policy.ActionRead)).To(MatchError(policy.ErrForbidden))
|
||||
Expect(r.reservations()).To(BeEmpty())
|
||||
got, _ := r.policy.Accessible(ctx, alice, "tables")
|
||||
Expect(got).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("commits a reservation at reconcile when the crash came after the database was created", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
actor := alice
|
||||
key, e := r.policy.Register(ctx, actor, "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
// The database is created; the owner dies before it commits.
|
||||
Expect(policy.RuntimeCreate(r, r, contract.DefaultLimits(), actor, args(map[string]any{"tool": "databases-create", "name": "Recovered"}))(ctx, key)).To(Succeed())
|
||||
Expect(r.policy.Authorize(ctx, actor, key, policy.ActionRead)).To(MatchError(policy.ErrForbidden))
|
||||
r.closePolicy()
|
||||
r.openPolicy()
|
||||
Expect(r.policy.Reconcile(ctx, r.backing())).To(Succeed())
|
||||
Expect(r.policy.Authorize(ctx, actor, key, policy.ActionSchema)).To(Succeed())
|
||||
Expect(r.policy.Authorize(ctx, bob, key, policy.ActionRead)).To(MatchError(policy.ErrForbidden))
|
||||
res, e := r.read(inst, actor, key.ResourceID, "databases-inspect")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(schemaName(res)).To(Equal("Recovered"))
|
||||
})
|
||||
|
||||
It("never exposes a database that has no host record", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
// The addon holds a database for an ID the host never recorded, as after
|
||||
// a lost policy store: nobody can resolve or list it.
|
||||
const orphan = "rffffffffffffffffffffffffffffffff"
|
||||
a := args(map[string]any{"tool": "databases-create", "name": "Orphan"})
|
||||
p, e := inst.Client.Preview(ctx, r.envelope(inst, alice, "preview", "create-"+orphan, orphan, a))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = inst.Client.Apply(ctx, r.envelope(inst, alice, "apply", "create-"+orphan, orphan, a), p.Revision)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(r.dbFile(orphan)).To(BeAnExistingFile())
|
||||
for _, who := range []contract.Actor{alice, bob, user("admin")} {
|
||||
_, e = r.policy.Resolve(ctx, who, contract.Request{AddonID: "tables", ResourceID: orphan})
|
||||
Expect(e).To(MatchError(policy.ErrForbidden))
|
||||
got, _ := r.policy.Accessible(ctx, who, "tables")
|
||||
Expect(got).To(BeEmpty())
|
||||
}
|
||||
Expect(r.policy.Reconcile(ctx, r.backing())).To(Succeed())
|
||||
_, e = r.policy.Resolve(ctx, alice, contract.Request{AddonID: "tables", ResourceID: orphan})
|
||||
Expect(e).To(MatchError(policy.ErrForbidden))
|
||||
})
|
||||
|
||||
It("renames one database by operation, replays safely, and leaves every other database alone", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
a, e := r.create(inst, alice, "Sales")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
b, e := r.create(inst, bob, "Costs")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
before, _ := os.ReadFile(r.dbFile(b.ResourceID))
|
||||
|
||||
rec, e := r.rename(inst, alice, a.ResourceID, "Revenue", "rename-1")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rec.State).To(Equal(contract.ReceiptCommitted))
|
||||
again, e := r.rename(inst, alice, a.ResourceID, "Revenue", "rename-1")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(again.State).To(Equal(contract.ReceiptCommitted))
|
||||
Expect(again.OperationID).To(Equal("rename-1"))
|
||||
got, e := inst.Client.Receipt(ctx, "rename-1")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(got.State).To(Equal(contract.ReceiptCommitted))
|
||||
|
||||
res, _ := r.read(inst, alice, a.ResourceID, "databases-inspect")
|
||||
Expect(schemaName(res)).To(Equal("Revenue"))
|
||||
res, _ = r.read(inst, bob, b.ResourceID, "databases-inspect")
|
||||
Expect(schemaName(res)).To(Equal("Costs"))
|
||||
after, _ := os.ReadFile(r.dbFile(b.ResourceID))
|
||||
Expect(after).To(Equal(before))
|
||||
// The rename is not portable: the same operation on another database is rejected.
|
||||
other, e := r.rename(inst, alice, b.ResourceID, "Revenue", "rename-1")
|
||||
if e == nil {
|
||||
Expect(other.State).To(Equal(contract.ReceiptRejected))
|
||||
}
|
||||
res, _ = r.read(inst, bob, b.ResourceID, "databases-inspect")
|
||||
Expect(schemaName(res)).To(Equal("Costs"))
|
||||
// Host policy: an editor may not change the schema, so no rename is sent.
|
||||
Expect(r.policy.SetGrant(ctx, bob, b, "alice", policy.RoleEditor)).To(Succeed())
|
||||
Expect(r.policy.Authorize(ctx, alice, b, policy.ActionSchema)).To(MatchError(policy.ErrForbidden))
|
||||
Expect(r.policy.Authorize(ctx, alice, b, policy.ActionWrite)).To(Succeed())
|
||||
})
|
||||
|
||||
It("rejects an undeclared tool, a forged addon binding and an unknown resource", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
key, e := r.create(inst, alice, "Sales")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = r.read(inst, alice, key.ResourceID, "databases-drop")
|
||||
Expect(e).To(HaveOccurred())
|
||||
forged := r.envelope(inst, alice, "read", "forged-1", key.ResourceID, args(map[string]any{"tool": "databases-inspect"}))
|
||||
forged.Request.AddonID = "other"
|
||||
_, e = inst.Client.Read(ctx, forged)
|
||||
Expect(e).To(HaveOccurred())
|
||||
forged = r.envelope(inst, alice, "read", "forged-2", key.ResourceID, args(map[string]any{"tool": "databases-inspect"}))
|
||||
forged.Request.Generation = 99
|
||||
_, e = inst.Client.Read(ctx, forged)
|
||||
Expect(e).To(HaveOccurred())
|
||||
for _, bad := range []string{"../etc/passwd", "r00000000000000000000000000000000"} {
|
||||
_, e = r.read(inst, alice, bad, "databases-inspect")
|
||||
Expect(e).To(HaveOccurred(), bad)
|
||||
}
|
||||
Expect(filesIn(filepath.Join(r.dataRoot, "tables", "tables", "databases"))).To(Equal([]string{key.ResourceID + ".db"}))
|
||||
})
|
||||
|
||||
It("keeps databases and operation receipts across an addon restart", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
key, e := r.create(inst, alice, "Sales")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = r.rename(inst, alice, key.ResourceID, "Renamed", "rename-keep")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(r.sup.Stop(ctx, inst)).To(Succeed())
|
||||
inst = r.start(2)
|
||||
res, e := r.read(inst, alice, key.ResourceID, "databases-inspect")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(schemaName(res)).To(Equal("Renamed"))
|
||||
rec, e := inst.Client.Receipt(ctx, "rename-keep")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rec.State).To(Equal(contract.ReceiptCommitted))
|
||||
replay, e := r.rename(inst, alice, key.ResourceID, "Renamed", "rename-keep")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(replay.State).To(Equal(contract.ReceiptCommitted))
|
||||
Expect(filesIn(filepath.Join(r.dataRoot, "tables", "tables", "databases"))).To(HaveLen(1))
|
||||
})
|
||||
|
||||
It("fails closed when the addon cannot answer, and drops only for an addon that is not installed", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
key, e := r.policy.Register(ctx, alice, "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(policy.RuntimeCreate(r, r, contract.DefaultLimits(), alice, args(map[string]any{"tool": "databases-create", "name": "Kept"}))(ctx, key)).To(Succeed())
|
||||
Expect(r.sup.Stop(ctx, inst)).To(Succeed())
|
||||
// Installed but down: the record is kept pending, not dropped, and
|
||||
// the pass does not fail.
|
||||
rep, e := r.policy.ReconcileReport(ctx, r.backing())
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rep).To(Equal(policy.ReconcileReport{Kept: 1}))
|
||||
Expect(r.reservations()).To(ConsistOf(key.ResourceID))
|
||||
// Gone for good: nothing can back it, so the record is dropped.
|
||||
r.installed = false
|
||||
Expect(r.policy.Reconcile(ctx, r.backing())).To(Succeed())
|
||||
Expect(r.reservations()).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
|
||||
func filesIn(dir string) []string {
|
||||
es, _ := os.ReadDir(dir)
|
||||
out := []string{}
|
||||
for _, e := range es {
|
||||
out = append(out, e.Name())
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,284 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tables_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
addonruntime "github.com/mudler/LocalAI/core/services/addons/runtime"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// rowEnv is an envelope under the host's read or write action.
|
||||
func (r *rig) rowEnv(inst addonruntime.Instance, actor contract.Actor, scope, action, op, resource string, a map[string]any) contract.Envelope {
|
||||
e := r.envelope(inst, actor, scope, op, resource, args(a))
|
||||
e.Request.Action = action
|
||||
return e
|
||||
}
|
||||
|
||||
func (r *rig) rowsRead(inst addonruntime.Instance, actor contract.Actor, resource, op string, a map[string]any) (contract.Result, error) {
|
||||
return inst.Client.Read(context.Background(), r.rowEnv(inst, actor, "read", "read", op, resource, a))
|
||||
}
|
||||
|
||||
// rowsWrite previews and applies one row mutation as the host's broker does.
|
||||
func (r *rig) rowsWrite(inst addonruntime.Instance, actor contract.Actor, resource, op string, a map[string]any) (contract.Preview, contract.Receipt, error) {
|
||||
ctx := context.Background()
|
||||
p, e := inst.Client.Preview(ctx, r.rowEnv(inst, actor, "preview", "write", op, resource, a))
|
||||
if e != nil {
|
||||
return p, contract.Receipt{}, e
|
||||
}
|
||||
rc, e := inst.Client.Apply(ctx, r.rowEnv(inst, actor, "apply", "write", op, resource, a), p.Revision)
|
||||
return p, rc, e
|
||||
}
|
||||
|
||||
type page struct {
|
||||
Columns []string `json:"columns"`
|
||||
Rows []struct {
|
||||
ID int64 `json:"id"`
|
||||
Values []any `json:"values"`
|
||||
} `json:"rows"`
|
||||
HasMore bool `json:"hasMore"`
|
||||
Cursor string `json:"cursor"`
|
||||
Revision contract.Revision `json:"revision"`
|
||||
}
|
||||
|
||||
func asPage(res contract.Result) page {
|
||||
GinkgoHelper()
|
||||
var p page
|
||||
Expect(json.Unmarshal(res.Payload, &p)).To(Succeed())
|
||||
return p
|
||||
}
|
||||
|
||||
var _ = Describe("Tables rows under the real owner", func() {
|
||||
ctx := context.Background()
|
||||
alice, bob := user("alice"), user("bob")
|
||||
|
||||
setup := func() (*rig, addonruntime.Instance, string, policy.ResourceKey) {
|
||||
GinkgoHelper()
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
key, e := r.create(inst, alice, "Sales")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, _, e = r.schemaCall(inst, alice, key.ResourceID, "s-t", map[string]any{"tool": "tables-create", "table": "Orders"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
for i, c := range [][2]string{{"Item", "text"}, {"Qty", "integer"}} {
|
||||
_, rc, e := r.schemaCall(inst, alice, key.ResourceID, fmt.Sprint("s-c", i), map[string]any{"tool": "columns-add", "table": "Orders", "column": c[0], "type": c[1]})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rc.State).To(Equal(contract.ReceiptCommitted))
|
||||
}
|
||||
return r, inst, key.ResourceID, key
|
||||
}
|
||||
|
||||
It("reads and mutates rows with the exact preview, refuses wrong actions and keeps receipts across a restart", func() {
|
||||
r, inst, id, key := setup()
|
||||
p, rc, e := r.rowsWrite(inst, alice, id, "ins-1", map[string]any{"tool": "rows-insert", "table": "Orders",
|
||||
"rows": []map[string]any{{"Item": "pen", "Qty": 3}, {"Item": "ink", "Qty": 9}, {"Item": "pad"}}})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rc.State).To(Equal(contract.ReceiptCommitted))
|
||||
Expect(rc.Revision.Data).To(Equal(p.Revision.Data + 1))
|
||||
Expect(rc.Revision.Schema).To(Equal(p.Revision.Schema))
|
||||
var shape struct {
|
||||
Count int `json:"count"`
|
||||
Sample []map[string]any `json:"sample"`
|
||||
}
|
||||
Expect(json.Unmarshal(p.Summary, &shape)).To(Succeed())
|
||||
Expect(shape.Count).To(Equal(3))
|
||||
|
||||
res, e := r.rowsRead(inst, alice, id, "q-1", map[string]any{"tool": "rows-query", "table": "orders", "columns": []string{"Item"},
|
||||
"filter": map[string]any{"column": "Qty", "op": "gt", "value": 2}, "sort": []map[string]any{{"column": "Qty", "desc": true}}})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
got := asPage(res)
|
||||
Expect(got.Columns).To(Equal([]string{"Item"}))
|
||||
Expect(got.Rows).To(HaveLen(2))
|
||||
Expect(got.Rows[0].Values).To(Equal([]any{"ink"}))
|
||||
agg, e := r.rowsRead(inst, alice, id, "q-2", map[string]any{"tool": "rows-aggregate", "table": "Orders",
|
||||
"aggregates": []map[string]any{{"fn": "sum", "column": "Qty"}, {"fn": "count"}}})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(string(agg.Payload)).To(ContainSubstring(`"values":[12,3]`))
|
||||
|
||||
// A filter update shows its exact set; a delete by id removes one row.
|
||||
pv, rc, e := r.rowsWrite(inst, alice, id, "upd-1", map[string]any{"tool": "rows-update", "table": "Orders",
|
||||
"filter": map[string]any{"column": "Qty", "op": "is-null"}, "set": map[string]any{"Qty": 1}})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rc.State).To(Equal(contract.ReceiptCommitted))
|
||||
Expect(json.Unmarshal(pv.Summary, &shape)).To(Succeed())
|
||||
Expect(shape.Count).To(Equal(1))
|
||||
_, rc, e = r.rowsWrite(inst, alice, id, "del-1", map[string]any{"tool": "rows-delete", "table": "Orders", "ids": []int64{1}})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rc.State).To(Equal(contract.ReceiptCommitted))
|
||||
|
||||
// Roles: a viewer reads, an editor writes, neither changes the schema.
|
||||
Expect(r.policy.SetGrant(ctx, alice, key, "bob", policy.RoleViewer)).To(Succeed())
|
||||
Expect(r.policy.Authorize(ctx, bob, key, policy.ActionRead)).To(Succeed())
|
||||
Expect(r.policy.Authorize(ctx, bob, key, policy.ActionWrite)).To(MatchError(policy.ErrForbidden))
|
||||
Expect(r.policy.SetGrant(ctx, alice, key, "bob", policy.RoleEditor)).To(Succeed())
|
||||
Expect(r.policy.Authorize(ctx, bob, key, policy.ActionWrite)).To(Succeed())
|
||||
Expect(r.policy.Authorize(ctx, bob, key, policy.ActionSchema)).To(MatchError(policy.ErrForbidden))
|
||||
|
||||
// The addon refuses wrong actions on its own, whatever the host did.
|
||||
del := map[string]any{"tool": "rows-delete", "table": "Orders", "ids": []int64{2}}
|
||||
before, _ := r.rowsRead(inst, alice, id, "q-3", map[string]any{"tool": "rows-query", "table": "Orders"})
|
||||
for _, act := range []string{"schema", "read", "share", "delete", "unknown", ""} {
|
||||
env := r.rowEnv(inst, bob, "preview", act, "wrong-"+act, id, del)
|
||||
_, e = inst.Client.Preview(ctx, env)
|
||||
Expect(e).To(HaveOccurred(), act)
|
||||
env = r.rowEnv(inst, bob, "apply", act, "wrong-"+act, id, del)
|
||||
_, e = inst.Client.Apply(ctx, env, p.Revision)
|
||||
Expect(e).To(HaveOccurred(), act)
|
||||
}
|
||||
_, e = inst.Client.Apply(ctx, r.rowEnv(inst, contract.Actor{}, "apply", "write", "wrong-noactor", id, del), p.Revision)
|
||||
Expect(e).To(HaveOccurred())
|
||||
// A row read under the write or schema action, and a mutation tool as a read.
|
||||
for _, act := range []string{"write", "schema", "share"} {
|
||||
_, e = inst.Client.Read(ctx, r.rowEnv(inst, bob, "read", act, "rd-"+act, id, map[string]any{"tool": "rows-query", "table": "Orders"}))
|
||||
Expect(e).To(HaveOccurred(), act)
|
||||
}
|
||||
_, e = r.rowsRead(inst, bob, id, "rd-mut", del)
|
||||
Expect(e).To(HaveOccurred())
|
||||
// A schema tool under the write action is refused too.
|
||||
env := r.rowEnv(inst, bob, "preview", "write", "sch-as-write", id, map[string]any{"tool": "tables-drop", "table": "Orders"})
|
||||
_, e = inst.Client.Preview(ctx, env)
|
||||
Expect(e).To(HaveOccurred())
|
||||
// Unknown argument fields are refused, so a typo cannot widen a mutation.
|
||||
_, e = inst.Client.Preview(ctx, r.rowEnv(inst, bob, "preview", "write", "typo", id, map[string]any{"tool": "rows-delete", "table": "Orders", "filters": map[string]any{"column": "Qty", "op": "eq", "value": 1}}))
|
||||
Expect(e).To(HaveOccurred())
|
||||
after, _ := r.rowsRead(inst, alice, id, "q-4", map[string]any{"tool": "rows-query", "table": "Orders"})
|
||||
b, a := asPage(before), asPage(after)
|
||||
Expect(a.Rows).To(Equal(b.Rows))
|
||||
rec, e := inst.Client.Receipt(ctx, "wrong-write")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rec.State).To(Equal(contract.ReceiptUnknown))
|
||||
|
||||
// Stale: two previews of one revision, the second apply is rejected as stale.
|
||||
first := map[string]any{"tool": "rows-update", "table": "Orders", "filter": map[string]any{"column": "Qty", "op": "gte", "value": 0}, "set": map[string]any{"Item": "A"}}
|
||||
second := map[string]any{"tool": "rows-update", "table": "Orders", "filter": map[string]any{"column": "Qty", "op": "gte", "value": 0}, "set": map[string]any{"Item": "B"}}
|
||||
p1, e := inst.Client.Preview(ctx, r.rowEnv(inst, bob, "preview", "write", "race-1", id, first))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
p2, e := inst.Client.Preview(ctx, r.rowEnv(inst, bob, "preview", "write", "race-2", id, second))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
rc1, e := inst.Client.Apply(ctx, r.rowEnv(inst, bob, "apply", "write", "race-1", id, first), p1.Revision)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rc1.State).To(Equal(contract.ReceiptCommitted))
|
||||
rc2, e := inst.Client.Apply(ctx, r.rowEnv(inst, bob, "apply", "write", "race-2", id, second), p2.Revision)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rc2.State).To(Equal(contract.ReceiptRejected))
|
||||
Expect(string(rc2.Result.Payload)).To(ContainSubstring(`"error":"stale"`))
|
||||
// A rejected operation has no receipt to replay and its ID is free.
|
||||
rec, e = inst.Client.Receipt(ctx, "race-2")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rec.State).To(Equal(contract.ReceiptUnknown))
|
||||
|
||||
// A restart keeps the data and the receipt, and a replay is the same receipt.
|
||||
Expect(r.sup.Stop(ctx, inst)).To(Succeed())
|
||||
inst = r.start(2)
|
||||
got2, e := inst.Client.Receipt(ctx, "race-1")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(got2.State).To(Equal(contract.ReceiptCommitted))
|
||||
Expect(got2.Revision).To(Equal(rc1.Revision))
|
||||
res, e = r.rowsRead(inst, alice, id, "q-5", map[string]any{"tool": "rows-query", "table": "Orders", "columns": []string{"Item"}})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
for _, row := range asPage(res).Rows {
|
||||
Expect(row.Values).To(Equal([]any{"A"}))
|
||||
}
|
||||
})
|
||||
|
||||
It("declares the row tools and refuses a row tool the manifest does not name", func() {
|
||||
r, inst, id, _ := setup()
|
||||
regs, e := r.sup.Registry.Snapshot(ctx)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(regs[0].Capabilities.Tools).To(ContainElements("rows-query", "rows-aggregate", "rows-insert", "rows-update", "rows-delete"))
|
||||
_, _, e = r.rowsWrite(inst, alice, id, "tr-1", map[string]any{"tool": "rows-truncate", "table": "Orders"})
|
||||
Expect(e).To(HaveOccurred())
|
||||
_, e = r.rowsRead(inst, alice, id, "tr-2", map[string]any{"tool": "rows-sql", "table": "Orders"})
|
||||
Expect(e).To(HaveOccurred())
|
||||
_, _, e = r.rowsWrite(inst, alice, id, "tr-3", map[string]any{"tool": "rows-delete", "table": "Orders"})
|
||||
Expect(e).To(HaveOccurred())
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("Tables row proposals in the application bootstrap", Label("application"), func() {
|
||||
It("applies row changes only on the proposer's confirmation and rejects a proposal whose database moved", func() {
|
||||
h := newHost()
|
||||
lib := h.library()
|
||||
h.start()
|
||||
DeferCleanup(h.stop)
|
||||
h.seed(lib)
|
||||
ctx := context.Background()
|
||||
|
||||
made := h.chatCreate("alice", "rows-create", "Ledger")
|
||||
Expect(made.code).To(Equal(200), "%v", made.body)
|
||||
Expect(h.confirm("alice", "rows-create", made.body["digest"].(string)).code).To(Equal(200))
|
||||
res := made.body["resourceId"].(string)
|
||||
propose := func(user, op, action string, a map[string]any) webReply {
|
||||
body, _ := json.Marshal(map[string]any{"addonId": "tables", "resourceId": res, "operationId": op, "generation": 1, "action": action, "arguments": a})
|
||||
return h.send("POST", "/api/addons/proposals", string(body), h.bearer(user))
|
||||
}
|
||||
run := func(user, op, action string, a map[string]any) webReply {
|
||||
GinkgoHelper()
|
||||
p := propose(user, op, action, a)
|
||||
Expect(p.code).To(Equal(200), "%v", p.body)
|
||||
done := h.confirm(user, op, p.body["digest"].(string))
|
||||
Expect(done.code).To(Equal(200), "%v", done.body)
|
||||
Expect(done.body["state"]).To(Equal("applied"), "%v", done.body)
|
||||
return done
|
||||
}
|
||||
run("alice", "ap-t", "schema", map[string]any{"tool": "tables-create", "table": "Entries"})
|
||||
run("alice", "ap-c", "schema", map[string]any{"tool": "columns-add", "table": "Entries", "column": "Memo", "type": "text"})
|
||||
run("alice", "ap-n", "schema", map[string]any{"tool": "columns-add", "table": "Entries", "column": "Amount", "type": "integer"})
|
||||
|
||||
// An editor is needed for rows; a stranger and a viewer cannot propose.
|
||||
Expect(propose("bob", "bob-denied", "write", map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []map[string]any{{"Memo": "x"}}}).code).To(Equal(403))
|
||||
key := policy.ResourceKey{AddonID: "tables", ResourceID: res}
|
||||
alice := h.actor("alice")
|
||||
Expect(h.app.AddonPolicy().SetGrant(ctx, alice, key, "bob", policy.RoleViewer)).To(Succeed())
|
||||
Expect(propose("bob", "bob-viewer", "write", map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []map[string]any{{"Memo": "x"}}}).code).To(Equal(403))
|
||||
read := func(user string, a map[string]any) (page, error) {
|
||||
out, e := h.app.AddonGateway().Dispatch(ctx, h.actor(user), contract.Request{AddonID: "tables", ResourceID: res,
|
||||
OperationID: "rd-" + uuid.NewString()[:8], Generation: 1, Action: "read", Arguments: args(a)})
|
||||
if e != nil {
|
||||
return page{}, e
|
||||
}
|
||||
return asPage(out), nil
|
||||
}
|
||||
_, e := read("bob", map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(h.app.AddonPolicy().SetGrant(ctx, alice, key, "bob", policy.RoleEditor)).To(Succeed())
|
||||
|
||||
// The proposal is a preview until the proposer's own session confirms.
|
||||
p := propose("bob", "ins-1", "write", map[string]any{"tool": "rows-insert", "table": "Entries",
|
||||
"rows": []map[string]any{{"Memo": "rent", "Amount": 100}, {"Memo": "food", "Amount": 40}}})
|
||||
Expect(p.code).To(Equal(200), "%v", p.body)
|
||||
Expect(p.body["state"]).To(Equal("pending"))
|
||||
pg, e := read("alice", map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(pg.Rows).To(BeEmpty())
|
||||
Expect(h.send("POST", "/api/addons/proposals/ins-1/confirm", fmt.Sprintf(`{"digest":%q}`, p.body["digest"]), h.bearer("bob")).code).To(Equal(403))
|
||||
Expect(h.confirm("alice", "ins-1", p.body["digest"].(string)).code).To(Equal(403))
|
||||
Expect(h.confirm("bob", "ins-1", p.body["digest"].(string)).code).To(Equal(200))
|
||||
pg, e = read("alice", map[string]any{"tool": "rows-query", "table": "Entries", "sort": []map[string]any{{"column": "Amount", "desc": true}}})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(pg.Rows).To(HaveLen(2))
|
||||
Expect(pg.Rows[0].Values).To(Equal([]any{"rent", float64(100)}))
|
||||
|
||||
// Two proposals on one revision: the second no longer applies, and the
|
||||
// filter is never re-evaluated against the rows the first one made.
|
||||
del := propose("alice", "del-all", "write", map[string]any{"tool": "rows-delete", "table": "Entries",
|
||||
"filter": map[string]any{"column": "Amount", "op": "gt", "value": 0}})
|
||||
Expect(del.code).To(Equal(200), "%v", del.body)
|
||||
ins := propose("bob", "ins-2", "write", map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []map[string]any{{"Memo": "late", "Amount": 5}}})
|
||||
Expect(ins.code).To(Equal(200), "%v", ins.body)
|
||||
Expect(h.confirm("bob", "ins-2", ins.body["digest"].(string)).code).To(Equal(200))
|
||||
stale := h.confirm("alice", "del-all", del.body["digest"].(string))
|
||||
Expect(stale.body["state"]).NotTo(Equal("applied"), "%v", stale.body)
|
||||
pg, e = read("alice", map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(pg.Rows).To(HaveLen(3))
|
||||
_ = time.Second
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,182 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tables_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
|
||||
_ "github.com/mattn/go-sqlite3"
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
addonruntime "github.com/mudler/LocalAI/core/services/addons/runtime"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// schemaCall sends one schema preview and apply through the real runtime
|
||||
// transport, as the owner's broker does after it admitted an owner.
|
||||
func (r *rig) schemaCall(inst addonruntime.Instance, actor contract.Actor, resource, op string, a map[string]any) (contract.Preview, contract.Receipt, error) {
|
||||
body := args(a)
|
||||
p, e := inst.Client.Preview(context.Background(), r.envelope(inst, actor, "preview", op, resource, body))
|
||||
if e != nil {
|
||||
return p, contract.Receipt{}, e
|
||||
}
|
||||
rc, e := inst.Client.Apply(context.Background(), r.envelope(inst, actor, "apply", op, resource, body), p.Revision)
|
||||
return p, rc, e
|
||||
}
|
||||
|
||||
var _ = Describe("Tables schema under the real owner", func() {
|
||||
ctx := context.Background()
|
||||
alice, bob := user("alice"), user("bob")
|
||||
|
||||
It("changes the schema as the owner, shows data loss in the preview and refuses an editor at the host and the addon", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
key, e := r.create(inst, alice, "Sales")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
id := key.ResourceID
|
||||
|
||||
p, rc, e := r.schemaCall(inst, alice, id, "tbl-create-1", map[string]any{"tool": "tables-create", "table": "Orders"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rc.State).To(Equal(contract.ReceiptCommitted))
|
||||
Expect(rc.Revision.Schema).To(Equal(p.Revision.Schema + 1))
|
||||
var shape struct {
|
||||
Count int `json:"count"`
|
||||
Sample []map[string]any `json:"sample"`
|
||||
}
|
||||
Expect(json.Unmarshal(p.Summary, &shape)).To(Succeed())
|
||||
Expect(shape.Count).To(Equal(1))
|
||||
Expect(shape.Sample[0]["action"]).To(Equal("create table"))
|
||||
|
||||
_, rc, e = r.schemaCall(inst, alice, id, "col-add-1", map[string]any{"tool": "columns-add", "table": "orders", "column": "Total", "type": "real", "required": true, "default": 0})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rc.State).To(Equal(contract.ReceiptCommitted))
|
||||
res, e := r.read(inst, alice, id, "databases-inspect")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var s struct {
|
||||
Tables []string `json:"tables"`
|
||||
Detail []struct {
|
||||
Name string `json:"name"`
|
||||
Columns []struct {
|
||||
Name, Type string
|
||||
Required bool
|
||||
} `json:"columns"`
|
||||
} `json:"detail"`
|
||||
}
|
||||
Expect(json.Unmarshal(res.Payload, &s)).To(Succeed())
|
||||
Expect(s.Tables).To(Equal([]string{"Orders"}))
|
||||
Expect(s.Detail[0].Columns).To(HaveLen(1))
|
||||
Expect(s.Detail[0].Columns[0].Name).To(Equal("Total"))
|
||||
Expect(s.Detail[0].Columns[0].Type).To(Equal("real"))
|
||||
Expect(s.Detail[0].Columns[0].Required).To(BeTrue())
|
||||
|
||||
// Rows exist: dropping the column reports the data it would destroy.
|
||||
raw, e := sql.Open("sqlite3", r.dbFile(id))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
defer func() { _ = raw.Close() }()
|
||||
var tableID, colID string
|
||||
Expect(raw.QueryRow("SELECT table_id FROM tables_catalog_tables").Scan(&tableID)).To(Succeed())
|
||||
Expect(raw.QueryRow("SELECT column_id FROM tables_catalog_columns").Scan(&colID)).To(Succeed())
|
||||
_, e = raw.Exec("INSERT INTO " + tableID + " DEFAULT VALUES")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
for i := 0; i < 3; i++ {
|
||||
_, e = raw.Exec("INSERT INTO "+tableID+"("+colID+") VALUES(?)", float64(i))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
}
|
||||
dropArgs := map[string]any{"tool": "columns-drop", "table": "Orders", "column": "Total"}
|
||||
p, e = inst.Client.Preview(ctx, r.envelope(inst, alice, "preview", "col-drop-1", id, args(dropArgs)))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var loss struct {
|
||||
DataLoss map[string]float64 `json:"dataLoss"`
|
||||
}
|
||||
Expect(json.Unmarshal(p.Summary, &loss)).To(Succeed())
|
||||
// The row made by DEFAULT VALUES holds the default 0, so four rows hold a value.
|
||||
Expect(loss.DataLoss["rows"]).To(BeEquivalentTo(4))
|
||||
|
||||
// An editor: the host refuses the action before any call is sent.
|
||||
Expect(r.policy.SetGrant(ctx, alice, key, "bob", policy.RoleEditor)).To(Succeed())
|
||||
Expect(r.policy.Authorize(ctx, bob, key, policy.ActionSchema)).To(MatchError(policy.ErrForbidden))
|
||||
Expect(r.policy.Authorize(ctx, bob, key, policy.ActionWrite)).To(Succeed())
|
||||
// Direct runtime transport: a schema tool under a row action, or
|
||||
// without an actor, is refused by the addon itself and changes nothing.
|
||||
before, _ := r.read(inst, alice, id, "databases-inspect")
|
||||
for _, act := range []string{"write", "read", "share", "delete", "unknown", ""} {
|
||||
env := r.envelope(inst, bob, "preview", "direct-"+act, id, args(dropArgs))
|
||||
env.Request.Action = act
|
||||
_, e = inst.Client.Preview(ctx, env)
|
||||
Expect(e).To(HaveOccurred(), act)
|
||||
env = r.envelope(inst, bob, "apply", "direct-"+act, id, args(dropArgs))
|
||||
env.Request.Action = act
|
||||
_, e = inst.Client.Apply(ctx, env, p.Revision)
|
||||
if e == nil {
|
||||
Fail("apply under action " + act + " must fail")
|
||||
}
|
||||
}
|
||||
noActor := r.envelope(inst, contract.Actor{}, "apply", "direct-noactor", id, args(dropArgs))
|
||||
_, e = inst.Client.Apply(ctx, noActor, p.Revision)
|
||||
Expect(e).To(HaveOccurred())
|
||||
after, _ := r.read(inst, alice, id, "databases-inspect")
|
||||
Expect(after.Payload).To(MatchJSON(before.Payload))
|
||||
rec, e := inst.Client.Receipt(ctx, "direct-write")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rec.State).To(Equal(contract.ReceiptUnknown))
|
||||
})
|
||||
|
||||
It("rejects a stale schema revision, replays an operation and keeps receipts across a restart", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
key, e := r.create(inst, alice, "Sales")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
id := key.ResourceID
|
||||
a := args(map[string]any{"tool": "tables-create", "table": "A"})
|
||||
stale, e := inst.Client.Preview(ctx, r.envelope(inst, alice, "preview", "sch-a", id, a))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, rc, e := r.schemaCall(inst, alice, id, "sch-b", map[string]any{"tool": "tables-create", "table": "B"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rc.State).To(Equal(contract.ReceiptCommitted))
|
||||
// The preview of "A" was made before "B" changed the schema.
|
||||
rj, e := inst.Client.Apply(ctx, r.envelope(inst, alice, "apply", "sch-a", id, a), stale.Revision)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rj.State).To(Equal(contract.ReceiptRejected))
|
||||
var body struct {
|
||||
Error string `json:"error"`
|
||||
Revision contract.Revision `json:"revision"`
|
||||
}
|
||||
Expect(json.Unmarshal(rj.Result.Payload, &body)).To(Succeed())
|
||||
Expect(body.Error).To(Equal("stale"))
|
||||
Expect(body.Revision).To(Equal(rc.Revision))
|
||||
res, _ := r.read(inst, alice, id, "databases-inspect")
|
||||
Expect(string(res.Payload)).NotTo(ContainSubstring(`"A"`))
|
||||
// The operation replays to one receipt and the same ID with another
|
||||
// request is a conflict at the transport.
|
||||
_, again, e := r.schemaCall(inst, alice, id, "sch-b", map[string]any{"tool": "tables-create", "table": "B"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(again).To(Equal(rc))
|
||||
_, _, e = r.schemaCall(inst, alice, id, "sch-b", map[string]any{"tool": "tables-create", "table": "C"})
|
||||
Expect(e).To(HaveOccurred())
|
||||
// A restart keeps the receipt.
|
||||
Expect(r.sup.Stop(ctx, inst)).To(Succeed())
|
||||
inst = r.start(2)
|
||||
got, e := inst.Client.Receipt(ctx, "sch-b")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(got.State).To(Equal(contract.ReceiptCommitted))
|
||||
Expect(got.Revision).To(Equal(rc.Revision))
|
||||
Expect(errors.Is(e, context.Canceled)).To(BeFalse())
|
||||
})
|
||||
|
||||
It("declares the schema tools and refuses any schema tool the manifest does not name", func() {
|
||||
r := newRig()
|
||||
inst := r.start(1)
|
||||
key, e := r.create(inst, alice, "Sales")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
regs, e := r.sup.Registry.Snapshot(ctx)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(regs[0].Capabilities.Tools).To(ContainElements("tables-create", "tables-rename", "tables-drop", "columns-add", "columns-rename", "columns-drop"))
|
||||
_, _, e = r.schemaCall(inst, alice, key.ResourceID, "sch-x", map[string]any{"tool": "columns-retype", "table": "A", "column": "b", "type": "integer"})
|
||||
Expect(e).To(HaveOccurred())
|
||||
_, _, e = r.schemaCall(inst, alice, key.ResourceID, "sch-y", map[string]any{"tool": "tables-create", "table": "badname"})
|
||||
Expect(e).To(HaveOccurred())
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,159 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tables_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
goruntime "runtime"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/addons/packages"
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
addonruntime "github.com/mudler/LocalAI/core/services/addons/runtime"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestTablesDatabases(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Tables databases addon")
|
||||
}
|
||||
|
||||
var binary string
|
||||
|
||||
var _ = BeforeSuite(func() {
|
||||
binary = filepath.Join(GinkgoT().TempDir(), "tables")
|
||||
cmd := exec.Command("go", "build", "-o", binary, "../../../addons/tables/cmd/tables")
|
||||
cmd.Env = append(os.Environ(), "CGO_ENABLED=1")
|
||||
out, e := cmd.CombinedOutput()
|
||||
Expect(e).NotTo(HaveOccurred(), string(out))
|
||||
})
|
||||
|
||||
// rig is one owner: a supervisor over the real Tables executable, a policy
|
||||
// store, and the host's bridge to the current runtime.
|
||||
type rig struct {
|
||||
root string
|
||||
dataRoot string
|
||||
sup *addonruntime.Supervisor
|
||||
pkg packages.PackageReceipt
|
||||
gen uint64
|
||||
policy *policy.Policy
|
||||
store *policy.Store
|
||||
policyDir string
|
||||
installed bool
|
||||
}
|
||||
|
||||
func (r *rig) Runtime(ctx context.Context, id string) (addonruntime.Instance, error) {
|
||||
return r.sup.Current(ctx, id, r.gen)
|
||||
}
|
||||
func (r *rig) DeclaredTools(_ context.Context, id string, _ uint64) ([]string, error) {
|
||||
return r.pkg.Manifest.Capabilities.Tools, nil
|
||||
}
|
||||
func (r *rig) Installed(context.Context, string) (bool, error) { return r.installed, nil }
|
||||
func (r *rig) backing() func(context.Context, policy.ResourceKey) (bool, error) {
|
||||
return policy.BackingExists(policy.BackingOptions{Runtime: r, Tools: r, Installed: r.Installed})
|
||||
}
|
||||
|
||||
func user(id string) contract.Actor {
|
||||
return contract.Actor{UserID: id, CredentialID: "session:" + id, Mode: "named-user"}
|
||||
}
|
||||
|
||||
func newRig() *rig {
|
||||
GinkgoHelper()
|
||||
root := GinkgoT().TempDir()
|
||||
r := &rig{root: root, dataRoot: filepath.Join(root, "data"), installed: true, gen: 1}
|
||||
data, e := os.ReadFile("../../../addons/tables/manifest.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := contract.ParseManifest(data)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m.OS, m.Arch = goruntime.GOOS, goruntime.GOARCH
|
||||
dir := filepath.Join(root, "package")
|
||||
Expect(os.MkdirAll(dir, 0o700)).To(Succeed())
|
||||
exe, e := os.ReadFile(binary)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(os.WriteFile(filepath.Join(dir, m.Executable), exe, 0o500)).To(Succeed())
|
||||
r.pkg = packages.PackageReceipt{Manifest: m, Directory: dir, Digest: "sha256:tables"}
|
||||
b, e := json.Marshal(r.pkg)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(os.WriteFile(filepath.Join(dir, "receipt.json"), b, 0o400)).To(Succeed())
|
||||
limits := contract.DefaultLimits()
|
||||
r.sup, e = addonruntime.NewSupervisor(addonruntime.Config{StateRoot: filepath.Join(root, "runtime"), DataRoot: r.dataRoot, Limits: limits, ReadyTimeout: 10 * time.Second, StableHealth: limits.BackoffMax})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() { Expect(r.sup.Close(context.Background())).To(Succeed()) })
|
||||
r.openPolicy()
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *rig) openPolicy() {
|
||||
GinkgoHelper()
|
||||
r.policyDir = filepath.Join(r.root, "policy")
|
||||
st, e := policy.OpenStore(r.policyDir, policy.ModeNamedUser)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
r.store = st
|
||||
r.policy = policy.New(st, policy.Options{
|
||||
Revalidate: func(context.Context, contract.Actor) error { return nil },
|
||||
CreateAuthority: func(context.Context, contract.Actor, string) error { return nil },
|
||||
})
|
||||
}
|
||||
|
||||
func (r *rig) closePolicy() { Expect(r.store.Close()).To(Succeed()) }
|
||||
|
||||
func (r *rig) start(gen uint64) addonruntime.Instance {
|
||||
GinkgoHelper()
|
||||
r.gen = gen
|
||||
i, e := r.sup.Start(context.Background(), r.pkg, gen)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return i
|
||||
}
|
||||
|
||||
func (r *rig) dbFile(id string) string {
|
||||
return filepath.Join(r.dataRoot, "tables", "tables", "databases", id+".db")
|
||||
}
|
||||
|
||||
func args(v map[string]any) json.RawMessage { b, _ := json.Marshal(v); return b }
|
||||
|
||||
func (r *rig) envelope(inst addonruntime.Instance, actor contract.Actor, scope, op, resource string, a json.RawMessage) contract.Envelope {
|
||||
return contract.Envelope{
|
||||
Request: contract.Request{AddonID: "tables", ResourceID: resource, OperationID: op, Generation: inst.Generation, Action: "schema", Arguments: a},
|
||||
Authority: contract.Authority{Actor: actor, Scope: scope, ExpiresAt: time.Now().Add(time.Minute)},
|
||||
}
|
||||
}
|
||||
|
||||
func (r *rig) create(inst addonruntime.Instance, actor contract.Actor, name string) (policy.ResourceKey, error) {
|
||||
return r.policy.Create(context.Background(), actor, "tables", policy.RuntimeCreate(r, r, contract.DefaultLimits(), actor, args(map[string]any{"tool": "databases-create", "name": name})))
|
||||
}
|
||||
|
||||
func (r *rig) read(inst addonruntime.Instance, actor contract.Actor, resource, tool string) (contract.Result, error) {
|
||||
return inst.Client.Read(context.Background(), r.envelope(inst, actor, "read", "read-"+resource, resource, args(map[string]any{"tool": tool})))
|
||||
}
|
||||
|
||||
func (r *rig) rename(inst addonruntime.Instance, actor contract.Actor, resource, name, op string) (contract.Receipt, error) {
|
||||
a := args(map[string]any{"tool": "databases-rename", "name": name})
|
||||
p, e := inst.Client.Preview(context.Background(), r.envelope(inst, actor, "preview", op, resource, a))
|
||||
if e != nil {
|
||||
return contract.Receipt{}, e
|
||||
}
|
||||
return inst.Client.Apply(context.Background(), r.envelope(inst, actor, "apply", op, resource, a), p.Revision)
|
||||
}
|
||||
|
||||
func schemaName(res contract.Result) string {
|
||||
GinkgoHelper()
|
||||
var out struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
Expect(json.Unmarshal(res.Payload, &out)).To(Succeed())
|
||||
return out.Name
|
||||
}
|
||||
|
||||
// reservations lists the IDs still pending in the policy store, read from its file.
|
||||
func (r *rig) reservations() []string {
|
||||
GinkgoHelper()
|
||||
got, e := r.policy.Pending()
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return got
|
||||
}
|
||||
@@ -0,0 +1,240 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tables_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
|
||||
"github.com/mudler/LocalAI/core/http/auth"
|
||||
"github.com/mudler/LocalAI/core/services/addons/gateway"
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
"github.com/mudler/LocalAI/core/services/addons/tools"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// callTool runs one model tool call as a user's API key, the credential of an
|
||||
// unattended agent. The adapter is the real one over the real owner gateway.
|
||||
func (h *host) callTool(user, name, body string) (tools.ToolResult, error) {
|
||||
GinkgoHelper()
|
||||
adapter := tools.NewToolAdapter(h.app.AddonGateway())
|
||||
return adapter.Execute(context.Background(), h.actor(user), name, json.RawMessage(body))
|
||||
}
|
||||
|
||||
func (h *host) pending(user string) []map[string]any {
|
||||
GinkgoHelper()
|
||||
got := h.send("GET", "/api/addons/proposals?state=pending", "", h.bearer(user))
|
||||
Expect(got.code).To(Equal(200), "%v", got.body)
|
||||
var out []map[string]any
|
||||
list, _ := got.body["proposals"].([]any)
|
||||
for _, x := range list {
|
||||
out = append(out, x.(map[string]any))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
var _ = Describe("Tables tools under the real owner", Label("application"), func() {
|
||||
It("offers the manifest tools to a permitted user and nothing to a user without the feature or after revocation", func() {
|
||||
h := newHost()
|
||||
lib := h.library()
|
||||
h.start()
|
||||
DeferCleanup(h.stop)
|
||||
h.seed(lib)
|
||||
adapter := tools.NewToolAdapter(h.app.AddonGateway())
|
||||
ctx := context.Background()
|
||||
|
||||
defs, e := adapter.Discover(ctx, h.actor("alice"))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(defs).To(HaveLen(15))
|
||||
for _, d := range defs {
|
||||
Expect(d.Name).To(HavePrefix("addon__tables__"))
|
||||
}
|
||||
defs, e = adapter.Discover(ctx, h.actor("bob"))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(defs).To(HaveLen(15))
|
||||
|
||||
// Another user's identity cannot be claimed by changing a field of
|
||||
// the actor: the owner checks the stored credential.
|
||||
forged := h.actor("bob")
|
||||
forged.CredentialID = "key:" + h.keys["alice"]
|
||||
_, e = adapter.Discover(ctx, forged)
|
||||
Expect(e).To(HaveOccurred())
|
||||
|
||||
// Revocation of the feature withdraws every tool at once.
|
||||
Expect(h.app.AuthDB().Where("user_id = ?", "bob").Delete(&auth.UserPermission{}).Error).To(Succeed())
|
||||
_, e = adapter.Discover(ctx, h.actor("bob"))
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
_, e = h.callTool("bob", "addon__tables__databases-create", `{"name":"Nope"}`)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
})
|
||||
|
||||
It("stores every change as a pending proposal and applies it only on the proposer's own session, exactly once", func() {
|
||||
h := newHost()
|
||||
lib := h.library()
|
||||
h.start()
|
||||
DeferCleanup(h.stop)
|
||||
h.seed(lib)
|
||||
|
||||
created, e := h.callTool("alice", "addon__tables__databases-create", `{"name":"Ledger"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(created.Kind).To(Equal(tools.ResultProposal))
|
||||
ref := created.Proposal
|
||||
Expect(ref.State).To(Equal("pending"))
|
||||
Expect(ref.Count).To(Equal(uint64(1)))
|
||||
Expect(ref.ResourceID).NotTo(BeEmpty())
|
||||
// Nothing exists yet, for anyone, including a read by the same agent.
|
||||
Expect(accessibleTo(h, "alice")).To(BeEmpty())
|
||||
_, e = h.callTool("alice", "addon__tables__databases-inspect", fmt.Sprintf(`{"resourceId":%q}`, ref.ResourceID))
|
||||
Expect(e).To(HaveOccurred())
|
||||
|
||||
// The proposal waits in the proposer's own list, not in anyone else's.
|
||||
listed := h.pending("alice")
|
||||
Expect(listed).To(HaveLen(1))
|
||||
Expect(listed[0]["id"]).To(Equal(ref.ID))
|
||||
Expect(listed[0]["digest"]).To(Equal(ref.Digest))
|
||||
Expect(h.pending("bob")).To(BeEmpty())
|
||||
// The card's authoritative record is the same one the tool referenced.
|
||||
card := h.send("GET", "/api/addons/proposals/"+ref.ID, "", h.bearer("alice"))
|
||||
Expect(card.code).To(Equal(200))
|
||||
Expect(card.body["resourceId"]).To(Equal(ref.ResourceID))
|
||||
Expect(card.body["effects"].(map[string]any)["count"]).To(BeEquivalentTo(1))
|
||||
Expect(h.send("GET", "/api/addons/proposals/"+ref.ID, "", h.bearer("bob")).code).To(Equal(403))
|
||||
|
||||
// A tool call, a key and another user's session cannot approve.
|
||||
Expect(h.send("POST", "/api/addons/proposals/"+ref.ID+"/confirm", fmt.Sprintf(`{"digest":%q}`, ref.Digest), h.bearer("alice")).code).To(Equal(403))
|
||||
toolCall := func(r *http.Request) { h.humanTab("alice", ref.ID)(r); r.Header.Set("X-LocalAI-Tool-Call", "1") }
|
||||
Expect(h.send("POST", "/api/addons/proposals/"+ref.ID+"/confirm", fmt.Sprintf(`{"digest":%q}`, ref.Digest), toolCall).code).To(Equal(403))
|
||||
Expect(h.confirm("bob", ref.ID, ref.Digest).code).To(Equal(403))
|
||||
Expect(accessibleTo(h, "alice")).To(BeEmpty())
|
||||
|
||||
done := h.confirm("alice", ref.ID, ref.Digest)
|
||||
Expect(done.code).To(Equal(200), "%v", done.body)
|
||||
Expect(done.body["state"]).To(Equal("applied"))
|
||||
Expect(h.pending("alice")).To(BeEmpty())
|
||||
res := ref.ResourceID
|
||||
|
||||
step := func(name, body string) *tools.ProposalRef {
|
||||
GinkgoHelper()
|
||||
r, e := h.callTool("alice", name, body)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(r.Kind).To(Equal(tools.ResultProposal))
|
||||
ok := h.confirm("alice", r.Proposal.ID, r.Proposal.Digest)
|
||||
Expect(ok.code).To(Equal(200), "%v", ok.body)
|
||||
Expect(ok.body["state"]).To(Equal("applied"), "%v", ok.body)
|
||||
return r.Proposal
|
||||
}
|
||||
step("addon__tables__tables-create", fmt.Sprintf(`{"resourceId":%q,"table":"Entries"}`, res))
|
||||
step("addon__tables__columns-add", fmt.Sprintf(`{"resourceId":%q,"table":"Entries","column":"Memo","type":"text"}`, res))
|
||||
|
||||
rows, e := h.callTool("alice", "addon__tables__rows-insert", fmt.Sprintf(`{"resourceId":%q,"table":"Entries","rows":[{"Memo":"rent"},{"Memo":"food"}]}`, res))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rows.Proposal.Count).To(Equal(uint64(2)))
|
||||
count := func() int {
|
||||
GinkgoHelper()
|
||||
r, e := h.callTool("alice", "addon__tables__rows-query", fmt.Sprintf(`{"resourceId":%q,"table":"Entries"}`, res))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(r.Kind).To(Equal(tools.ResultData))
|
||||
var p page
|
||||
Expect(json.Unmarshal(r.Result, &p)).To(Succeed())
|
||||
return len(p.Rows)
|
||||
}
|
||||
Expect(count()).To(Equal(0), "a proposal changes nothing")
|
||||
Expect(h.confirm("alice", rows.Proposal.ID, rows.Proposal.Digest).code).To(Equal(200))
|
||||
Expect(count()).To(Equal(2))
|
||||
// A repeated confirmation returns the stored receipt and runs nothing.
|
||||
again := h.confirm("alice", rows.Proposal.ID, rows.Proposal.Digest)
|
||||
Expect(again.body["state"]).To(Equal("applied"))
|
||||
Expect(count()).To(Equal(2))
|
||||
})
|
||||
|
||||
It("keeps the roles of the host for tool calls and rejects forged fields before the owner sees them", func() {
|
||||
h := newHost()
|
||||
lib := h.library()
|
||||
h.start()
|
||||
DeferCleanup(h.stop)
|
||||
h.seed(lib)
|
||||
ctx := context.Background()
|
||||
|
||||
created, e := h.callTool("alice", "addon__tables__databases-create", `{"name":"Team"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(h.confirm("alice", created.Proposal.ID, created.Proposal.Digest).code).To(Equal(200))
|
||||
res := created.Proposal.ResourceID
|
||||
mk := func(table string) {
|
||||
r, e := h.callTool("alice", "addon__tables__tables-create", fmt.Sprintf(`{"resourceId":%q,"table":%q}`, res, table))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(h.confirm("alice", r.Proposal.ID, r.Proposal.Digest).code).To(Equal(200))
|
||||
}
|
||||
mk("Entries")
|
||||
|
||||
// A stranger sees nothing: the same denial for every tool.
|
||||
for _, call := range []struct{ tool, body string }{
|
||||
{"addon__tables__databases-inspect", fmt.Sprintf(`{"resourceId":%q}`, res)},
|
||||
{"addon__tables__rows-query", fmt.Sprintf(`{"resourceId":%q,"table":"Entries"}`, res)},
|
||||
{"addon__tables__rows-insert", fmt.Sprintf(`{"resourceId":%q,"table":"Entries","rows":[{}]}`, res)},
|
||||
{"addon__tables__tables-drop", fmt.Sprintf(`{"resourceId":%q,"table":"Entries"}`, res)},
|
||||
} {
|
||||
_, e := h.callTool("bob", call.tool, call.body)
|
||||
Expect(e).To(HaveOccurred(), call.tool)
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403), call.tool)
|
||||
}
|
||||
|
||||
key := policy.ResourceKey{AddonID: "tables", ResourceID: res}
|
||||
Expect(h.app.AddonPolicy().SetGrant(ctx, h.actor("alice"), key, "bob", policy.RoleViewer)).To(Succeed())
|
||||
_, e = h.callTool("bob", "addon__tables__rows-query", fmt.Sprintf(`{"resourceId":%q,"table":"Entries"}`, res))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = h.callTool("bob", "addon__tables__rows-insert", fmt.Sprintf(`{"resourceId":%q,"table":"Entries","rows":[{}]}`, res))
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
|
||||
Expect(h.app.AddonPolicy().SetGrant(ctx, h.actor("alice"), key, "bob", policy.RoleEditor)).To(Succeed())
|
||||
r, e := h.callTool("bob", "addon__tables__rows-insert", fmt.Sprintf(`{"resourceId":%q,"table":"Entries","rows":[{}]}`, res))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(r.Kind).To(Equal(tools.ResultProposal))
|
||||
// An editor never changes the schema, by tool or by any other name.
|
||||
_, e = h.callTool("bob", "addon__tables__tables-drop", fmt.Sprintf(`{"resourceId":%q,"table":"Entries"}`, res))
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
before := len(h.pending("bob"))
|
||||
|
||||
// Forged identity, role and approval fields and approval aliases never
|
||||
// create a proposal and never reach the addon.
|
||||
for _, field := range []string{`"actor":"alice"`, `"userId":"alice"`, `"role":"owner"`, `"approved":true`, `"proposalId":"x"`, `"tool":"tables-drop"`} {
|
||||
_, e := h.callTool("bob", "addon__tables__rows-insert", fmt.Sprintf(`{"resourceId":%q,"table":"Entries","rows":[{}],%s}`, res, field))
|
||||
Expect(e).To(MatchError(tools.ErrForbiddenArgument), field)
|
||||
}
|
||||
for _, name := range []string{"confirm", "approve", "addon__tables__confirm", "addon__tables__share", "addon__tables__databases-delete", "addon__tables__databases-purge"} {
|
||||
_, e := h.callTool("bob", name, fmt.Sprintf(`{"resourceId":%q}`, res))
|
||||
Expect(e).To(MatchError(tools.ErrUnknownTool), name)
|
||||
}
|
||||
Expect(h.pending("bob")).To(HaveLen(before))
|
||||
})
|
||||
|
||||
It("lists at most the bound of pending proposals and refuses more than the user's quota", func() {
|
||||
h := newHost()
|
||||
lib := h.library()
|
||||
h.start()
|
||||
DeferCleanup(h.stop)
|
||||
h.seed(lib)
|
||||
created, e := h.callTool("alice", "addon__tables__databases-create", `{"name":"Many"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(h.confirm("alice", created.Proposal.ID, created.Proposal.Digest).code).To(Equal(200))
|
||||
res := created.Proposal.ResourceID
|
||||
for _, step := range []string{
|
||||
fmt.Sprintf(`{"resourceId":%q,"table":"T"}`, res),
|
||||
} {
|
||||
r, e := h.callTool("alice", "addon__tables__tables-create", step)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(h.confirm("alice", r.Proposal.ID, r.Proposal.Digest).code).To(Equal(200))
|
||||
}
|
||||
for i := 0; i < policy.MaxProposalsPerUser; i++ {
|
||||
_, e := h.callTool("alice", "addon__tables__rows-insert", fmt.Sprintf(`{"resourceId":%q,"table":"T","rows":[{}]}`, res))
|
||||
Expect(e).NotTo(HaveOccurred(), "proposal %d", i)
|
||||
}
|
||||
Expect(len(h.pending("alice"))).To(Equal(gateway.MaxPendingListed))
|
||||
_, e = h.callTool("alice", "addon__tables__rows-insert", fmt.Sprintf(`{"resourceId":%q,"table":"T","rows":[{}]}`, res))
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1 @@
|
||||
.cache/
|
||||
@@ -0,0 +1,437 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesacceptance_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// Role separation and credential changes, decided by the real servers.
|
||||
var _ = Describe("Tables authorization across the owner and two frontends", Ordered, func() {
|
||||
var (
|
||||
d *deployment
|
||||
res string
|
||||
)
|
||||
|
||||
BeforeAll(func() {
|
||||
c := start("0.1.0")
|
||||
d = deploy(c, "0.1.0")
|
||||
res = d.seedLedger(cluster.FrontA, "member", "Ledger", map[string]any{"Item": "pen", "Qty": 3})
|
||||
Expect(d.grant(cluster.FrontB, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
Expect(d.grant(cluster.Owner, "member", res, "viewer", "viewer").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("separates owner, editor, viewer, administrator and stranger across the browser routes, the REST routes and the chat tool consumer", func() {
|
||||
row := map[string]any{"Item": "ink", "Qty": 1}
|
||||
expect := map[string]struct {
|
||||
read, rows, schema, share int
|
||||
proposals int
|
||||
}{
|
||||
"member": {200, 200, 200, 200, 1},
|
||||
"editor": {200, 200, 403, 403, 1},
|
||||
"viewer": {200, 403, 403, 403, 0},
|
||||
"friend": {403, 403, 403, 403, 0},
|
||||
"admin": {403, 403, 403, 403, 0},
|
||||
"outsider": {403, 403, 403, 403, 0},
|
||||
}
|
||||
order := []string{"member", "editor", "viewer", "friend", "admin", "outsider"}
|
||||
// The one denial every refusal must equal: a database that does not
|
||||
// exist, read by an account that has the feature.
|
||||
probe := d.read(d.c.Token(cluster.FrontB, "member"), "r-not-a-database", map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(probe.Status).To(Equal(403), probe.String())
|
||||
denial := probe.String()
|
||||
for i, user := range order {
|
||||
want := expect[user]
|
||||
// Every role goes through a different node in turn: the answer
|
||||
// never depends on which process the person reached.
|
||||
node := nodes[i%len(nodes)]
|
||||
other := nodes[(i+1)%len(nodes)]
|
||||
|
||||
// REST read with the API key, on the node and on another one.
|
||||
for _, n := range []string{node, other} {
|
||||
r := d.read(d.c.Token(n, user), res, map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(r.Status).To(Equal(want.read), fmt.Sprintf("%s reads on %s: %s", user, n, r.String()))
|
||||
if want.read == 403 && user != "outsider" {
|
||||
// A stranger and an administrator read the same denial, on
|
||||
// every node; it does not tell a missing database from a
|
||||
// private one.
|
||||
Expect(r.String()).To(Equal(denial), user+" on "+n)
|
||||
}
|
||||
}
|
||||
missing := d.read(d.c.Token(node, user), "r-not-a-database", map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(missing.Status).To(Equal(403), user)
|
||||
if user != "outsider" {
|
||||
Expect(missing.String()).To(Equal(denial), "a missing database looks like a private one: "+user)
|
||||
}
|
||||
|
||||
// The person's own tab: a row change and a schema change.
|
||||
rows := d.apply(node, user, res, opID("acc-r"), "rows-insert", d.revisionOf(res), map[string]any{"table": "Entries", "rows": []any{row}})
|
||||
Expect(rows.Status).To(Equal(want.rows), fmt.Sprintf("%s inserts on %s: %s", user, node, rows.String()))
|
||||
column := "Col" + strings.ToUpper(user[:1]) + user[1:]
|
||||
schema := d.apply(other, user, res, opID("acc-s"), "columns-add", d.revisionOf(res), map[string]any{"table": "Entries", "column": column, "type": "text"})
|
||||
Expect(schema.Status).To(Equal(want.schema), fmt.Sprintf("%s changes the schema on %s: %s", user, other, schema.String()))
|
||||
share := d.grant(node, user, res, "friend", "viewer")
|
||||
Expect(share.Status).To(Equal(want.share), fmt.Sprintf("%s shares on %s: %s", user, node, share.String()))
|
||||
if want.share == 200 {
|
||||
Expect(d.revoke(other, user, res, "friend").Status).To(Equal(200))
|
||||
}
|
||||
|
||||
// The typed proposal route: a write is a proposal for the owner
|
||||
// role and the editor, and never for a viewer or a stranger.
|
||||
prop := d.propose(other, user, res, "write", opID("acc-p"), map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{row}})
|
||||
if want.rows == 200 {
|
||||
Expect(prop.Status).To(Equal(200), user+": "+prop.String())
|
||||
} else {
|
||||
Expect(prop.Status).To(Equal(403), user+": "+prop.String())
|
||||
}
|
||||
|
||||
// The chat consumer: the scripted model calls the tool as this key.
|
||||
before := 0
|
||||
if want.read == 200 || want.rows == 200 {
|
||||
before = len(d.pending(node, user))
|
||||
}
|
||||
d.chatTool(other, user, "rows-insert", map[string]any{"resourceId": res, "table": "Entries", "rows": []any{row}})
|
||||
if user != "outsider" {
|
||||
after := len(d.pending(node, user))
|
||||
Expect(after-before).To(Equal(want.proposals), fmt.Sprintf("%s chat proposals (%d -> %d)", user, before, after))
|
||||
}
|
||||
}
|
||||
|
||||
// The same data through each node tells the same story: the owner role
|
||||
// and the editor wrote, the others did not.
|
||||
for _, n := range nodes {
|
||||
count := d.rowCount(n, "viewer", res)
|
||||
Expect(count).To(Equal(1 + 2))
|
||||
}
|
||||
})
|
||||
|
||||
It("serves the browser module to every account with the feature and to nobody else, from the owner and each frontend", func() {
|
||||
for _, user := range []string{"member", "editor", "viewer", "friend", "admin"} {
|
||||
for _, n := range nodes {
|
||||
a := d.app(d.c.Session(n, user))
|
||||
Expect(a).NotTo(BeNil(), user+" on "+n)
|
||||
ui, _ := a["ui"].(map[string]any)
|
||||
Expect(ui).NotTo(BeNil())
|
||||
entry, _ := ui["entry"].(string)
|
||||
Expect(entry).To(MatchRegexp(`^/api/addons/tables/assets/[0-9a-f]{64}/`))
|
||||
asset, err := d.c.Session(n, user).Do("GET", entry, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(asset.Status).To(Equal(200), user+" "+n)
|
||||
Expect(asset.Body).NotTo(BeEmpty())
|
||||
// The page of the SPA is the same shell for everyone; what the
|
||||
// account may do is the server's answer above.
|
||||
page, err := d.c.Session(n, user).Do("GET", "/app/addons/tables/databases", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(page.Status).To(Equal(200), user+" "+n)
|
||||
}
|
||||
}
|
||||
for _, n := range nodes {
|
||||
r, err := d.c.Session(n, "outsider").Do("GET", "/api/addons/apps", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(403), n)
|
||||
anon, err := cluster.Client{Base: d.c.Nodes[n].URL()}.Do("GET", "/api/addons/apps", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(anon.Status).To(Equal(401), n)
|
||||
}
|
||||
})
|
||||
|
||||
It("fails every blocked consumer explicitly and never serves add-on tools to it", func() {
|
||||
member := func(n string) cluster.Client { return d.c.Token(n, "member") }
|
||||
for _, n := range nodes {
|
||||
for _, path := range []string{"/v1/messages", "/v1/responses", "/v1/mcp/chat/completions"} {
|
||||
r, err := member(n).Do("POST", path, map[string]any{
|
||||
"model": "mock-model", "metadata": map[string]string{"addon_tools": "true"},
|
||||
"max_tokens": 5, "input": "hi", "messages": []map[string]any{{"role": "user", "content": "hi"}},
|
||||
}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(http.StatusBadRequest), n+" "+path+": "+r.String())
|
||||
Expect(errType(r)).To(Equal("addon_tools_unsupported"), n+" "+path)
|
||||
}
|
||||
// The chat route itself refuses an account without the feature
|
||||
// and an anonymous caller, with typed denials.
|
||||
out := d.chatTool(n, "outsider", "rows-query", map[string]any{"resourceId": res, "table": "Entries"})
|
||||
Expect(out.Status).To(Equal(403), n+": "+out.String())
|
||||
anon, err := cluster.Client{Base: d.c.Nodes[n].URL()}.Do("POST", "/v1/chat/completions", map[string]any{
|
||||
"model": "mock-model", "metadata": map[string]string{"addon_tools": "true"},
|
||||
"messages": []map[string]any{{"role": "user", "content": "hi"}},
|
||||
}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(anon.Status).To(Equal(401), n)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
// revisionOf reads the revision as the owner of a database. It is used where
|
||||
// the caller is whoever may read; the revision is the same for everyone.
|
||||
func (d *deployment) revisionOf(res string) map[string]any {
|
||||
GinkgoHelper()
|
||||
return d.revision(cluster.Owner, "member", res)
|
||||
}
|
||||
|
||||
// A person's rights change while the person's tabs and agents are working.
|
||||
var _ = Describe("Tables authorization when rights change in the middle of a session", Ordered, func() {
|
||||
var (
|
||||
d *deployment
|
||||
res string
|
||||
row = map[string]any{"Item": "x", "Qty": 1}
|
||||
)
|
||||
|
||||
BeforeAll(func() {
|
||||
d = deploy(start("0.1.0"), "0.1.0")
|
||||
res = d.seedLedger(cluster.Owner, "member", "Shared", map[string]any{"Item": "pen", "Qty": 3})
|
||||
Expect(d.grant(cluster.FrontA, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
// everywhere runs one check on the owner and on both frontends.
|
||||
everywhere := func(check func(node string)) {
|
||||
for _, n := range nodes {
|
||||
check(n)
|
||||
}
|
||||
}
|
||||
editorCanWrite := func(node string) int {
|
||||
return d.apply(node, "editor", res, opID("mid"), "rows-insert", d.revisionOf(res), map[string]any{"table": "Entries", "rows": []any{row}}).Status
|
||||
}
|
||||
|
||||
It("denies the next call of a paused key on every node and accepts it again when resumed", func() {
|
||||
everywhere(func(n string) {
|
||||
Expect(d.read(d.c.Token(n, "editor"), res, map[string]any{"tool": "rows-query", "table": "Entries"}).Status).To(Equal(200), n)
|
||||
})
|
||||
key := d.c.Creds.Users["editor"].KeyID
|
||||
r := d.tab(cluster.FrontB, "editor", "PATCH", "/api/auth/api-keys/"+key, map[string]any{"disabled": true}, "")
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
everywhere(func(n string) {
|
||||
got := d.read(d.c.Token(n, "editor"), res, map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(got.Status).To(BeElementOf(401, 403), n+": "+got.String())
|
||||
chat := d.chatTool(n, "editor", "rows-insert", map[string]any{"resourceId": res, "table": "Entries", "rows": []any{row}})
|
||||
Expect(chat.Status).To(BeElementOf(401, 403), n)
|
||||
})
|
||||
// The person's own session is a different credential and still works.
|
||||
Expect(editorCanWrite(cluster.FrontA)).To(Equal(200))
|
||||
r = d.tab(cluster.FrontA, "editor", "PATCH", "/api/auth/api-keys/"+key, map[string]any{"disabled": false}, "")
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
everywhere(func(n string) {
|
||||
Expect(d.read(d.c.Token(n, "editor"), res, map[string]any{"tool": "rows-query", "table": "Entries"}).Status).To(Equal(200), n)
|
||||
})
|
||||
})
|
||||
|
||||
It("denies the browser, the key and the chat everywhere when the add-ons feature is withdrawn", func() {
|
||||
admin := d.c.Token(cluster.FrontA, "admin")
|
||||
id := d.c.Creds.Users["editor"].ID
|
||||
r, err := admin.Do("PUT", "/api/auth/admin/users/"+id+"/permissions", map[string]bool{"addons": false}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
everywhere(func(n string) {
|
||||
Expect(editorCanWrite(n)).To(Equal(403), n)
|
||||
Expect(d.read(d.c.Token(n, "editor"), res, map[string]any{"tool": "rows-query", "table": "Entries"}).Status).To(Equal(403), n)
|
||||
Expect(d.chatTool(n, "editor", "rows-insert", map[string]any{"resourceId": res, "table": "Entries", "rows": []any{row}}).Status).To(Equal(403), n)
|
||||
apps, err := d.c.Session(n, "editor").Do("GET", "/api/addons/apps", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(apps.Status).To(Equal(403), n)
|
||||
})
|
||||
r, err = admin.Do("PUT", "/api/auth/admin/users/"+id+"/permissions", map[string]bool{"addons": true}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
Expect(editorCanWrite(cluster.FrontB)).To(Equal(200))
|
||||
})
|
||||
|
||||
It("lowers an editor to a viewer for the next call everywhere and drops the proposals the editor still held", func() {
|
||||
held := d.propose(cluster.FrontB, "editor", res, "write", opID("held"), map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{row}})
|
||||
Expect(held.Status).To(Equal(200), held.String())
|
||||
heldID, _ := body(held)["id"].(string)
|
||||
heldDigest, _ := body(held)["digest"].(string)
|
||||
Expect(heldID).NotTo(BeEmpty(), held.String())
|
||||
Expect(len(d.pending(cluster.Owner, "editor"))).To(BeNumerically(">=", 1))
|
||||
|
||||
Expect(d.grant(cluster.FrontB, "member", res, "editor", "viewer").Status).To(Equal(200))
|
||||
everywhere(func(n string) {
|
||||
Expect(editorCanWrite(n)).To(Equal(403), n)
|
||||
Expect(d.read(d.c.Token(n, "editor"), res, map[string]any{"tool": "rows-query", "table": "Entries"}).Status).To(Equal(200), n)
|
||||
})
|
||||
// The proposal the editor held does not apply any more.
|
||||
Expect(d.confirm(cluster.FrontA, "editor", heldID, heldDigest).Status).NotTo(Equal(200))
|
||||
everywhere(func(n string) {
|
||||
for _, p := range d.pending(n, "editor") {
|
||||
Expect(p["id"]).NotTo(Equal(heldID), n)
|
||||
}
|
||||
})
|
||||
Expect(d.revoke(cluster.Owner, "member", res, "editor").Status).To(Equal(200))
|
||||
everywhere(func(n string) {
|
||||
Expect(d.read(d.c.Token(n, "editor"), res, map[string]any{"tool": "rows-query", "table": "Entries"}).Status).To(Equal(403), n)
|
||||
})
|
||||
Expect(d.grant(cluster.FrontA, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("refuses the session and the key of a disabled account on every node", func() {
|
||||
admin := d.c.Token(cluster.FrontB, "admin")
|
||||
id := d.c.Creds.Users["editor"].ID
|
||||
r, err := admin.Do("PUT", "/api/auth/admin/users/"+id+"/status", map[string]string{"status": "disabled"}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
everywhere(func(n string) {
|
||||
Expect(editorCanWrite(n)).To(BeElementOf(401, 403), n)
|
||||
got := d.read(d.c.Token(n, "editor"), res, map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(got.Status).To(BeElementOf(401, 403), n)
|
||||
})
|
||||
r, err = admin.Do("PUT", "/api/auth/admin/users/"+id+"/status", map[string]string{"status": "active"}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
Expect(editorCanWrite(cluster.FrontA)).To(Equal(200))
|
||||
})
|
||||
})
|
||||
|
||||
// Authority is the host's. Whatever a browser, a script or a model puts in a
|
||||
// request about who it is or whether it was approved is only data.
|
||||
var _ = Describe("Tables authority and approval cannot be forged", Ordered, func() {
|
||||
var (
|
||||
d *deployment
|
||||
res string
|
||||
row = map[string]any{"Item": "x", "Qty": 1}
|
||||
)
|
||||
|
||||
BeforeAll(func() {
|
||||
d = deploy(start("0.1.0"), "0.1.0")
|
||||
res = d.seedLedger(cluster.FrontA, "member", "Private", map[string]any{"Item": "pen", "Qty": 3})
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
Expect(d.grant(cluster.Owner, "member", res, "viewer", "viewer").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("ignores actor, forwarding and authority headers and fields on every node", func() {
|
||||
forged := map[string]string{
|
||||
"X-Forwarded-User": d.c.Creds.Users["member"].ID, "X-Forwarded-Actor": d.c.Creds.Users["member"].ID,
|
||||
"X-Addon-Actor": d.c.Creds.Users["member"].ID, "X-LocalAI-Actor": d.c.Creds.Users["member"].ID,
|
||||
"X-Remote-User": d.c.Creds.Users["member"].ID, "X-User-Id": d.c.Creds.Users["member"].ID,
|
||||
"X-Addon-Peer": "owner",
|
||||
}
|
||||
for _, n := range nodes {
|
||||
// A stranger with forged identity headers still reads nothing.
|
||||
stranger := d.c.Token(n, "friend")
|
||||
stranger.Headers = forged
|
||||
r := d.read(stranger, res, map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(r.Status).To(Equal(403), n+": "+r.String())
|
||||
Expect(r.String()).NotTo(ContainSubstring("pen"))
|
||||
// A viewer with them still cannot write.
|
||||
viewer := d.c.Session(n, "viewer")
|
||||
viewer.Headers = forged
|
||||
w, err := viewer.Do("POST", "/api/addons/resources/"+res+"/apply", map[string]any{
|
||||
"operationId": opID("forge"), "generation": d.gen, "revision": d.revisionOf(res), "tool": "rows-insert",
|
||||
"arguments": map[string]any{"table": "Entries", "rows": []any{row}},
|
||||
}, map[string]string{"X-LocalAI-Confirm": res})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(w.Status).To(Equal(403), n+": "+w.String())
|
||||
}
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(1))
|
||||
})
|
||||
|
||||
It("derives the action of a proposal from the tool kind, so an editor cannot confirm a schema change as a write", func() {
|
||||
rename := map[string]any{"tool": "databases-rename", "name": "Stolen"}
|
||||
for _, n := range nodes {
|
||||
// Model rules R4 and S4: the editor claims the data action for the
|
||||
// rename tool, and the schema action; both are refused.
|
||||
for _, action := range []string{"write", "schema"} {
|
||||
r := d.propose(n, "editor", res, action, opID("kind-e"), rename)
|
||||
Expect(r.Status).To(Equal(403), n+" editor "+action+": "+r.String())
|
||||
}
|
||||
// The owner role may not claim the data action for it either.
|
||||
r := d.propose(n, "member", res, "write", opID("kind-m"), rename)
|
||||
Expect(r.Status).To(Equal(403), n+" owner write: "+r.String())
|
||||
}
|
||||
Expect(d.pending(cluster.Owner, "editor")).To(BeEmpty())
|
||||
// The owner role proposes the rename under the schema action.
|
||||
r := d.propose(cluster.FrontA, "member", res, "schema", opID("kind-ok"), rename)
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
})
|
||||
|
||||
It("denies authority, role and approval fields in the arguments of every consumer and creates nothing", func() {
|
||||
before := len(d.pending(cluster.Owner, "editor"))
|
||||
for _, field := range []map[string]any{
|
||||
{"actor": d.c.Creds.Users["member"].ID}, {"userId": d.c.Creds.Users["member"].ID}, {"role": "owner"},
|
||||
{"approved": true}, {"confirmed": true}, {"proposalId": "x"}, {"authority": map[string]any{"scope": "apply"}},
|
||||
} {
|
||||
args := map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{row}}
|
||||
for k, v := range field {
|
||||
args[k] = v
|
||||
}
|
||||
for _, n := range nodes {
|
||||
r := d.propose(n, "editor", res, "write", opID("fa"), args)
|
||||
// A field the tool does not declare is refused as invalid, with
|
||||
// the same typed 400 from every node, and nothing is stored.
|
||||
Expect(r.Status).To(Equal(400), fmt.Sprintf("%s %v: %s", n, field, r.String()))
|
||||
Expect(errType(r)).To(Equal("invalid_request"), fmt.Sprintf("%s %v: %s", n, field, r.String()))
|
||||
chatArgs := map[string]any{"resourceId": res, "table": "Entries", "rows": []any{row}}
|
||||
for k, v := range field {
|
||||
chatArgs[k] = v
|
||||
}
|
||||
// The chat consumer's own argument binding rejects the field
|
||||
// before the add-on is asked: the turn is served, nothing ran
|
||||
// (the pending list below proves it).
|
||||
chat := d.chatTool(n, "editor", "rows-insert", chatArgs)
|
||||
Expect(chat.Status).To(Equal(200), chat.String())
|
||||
Expect(chat.String()).NotTo(ContainSubstring("pen"), fmt.Sprintf("%s %v", n, field))
|
||||
}
|
||||
}
|
||||
Expect(len(d.pending(cluster.FrontB, "editor"))).To(Equal(before))
|
||||
// The tool of the arguments must be the declared one, exactly.
|
||||
for _, tool := range []string{"Rows-insert", "rows-insert ", "tables-drop-all", ""} {
|
||||
r := d.read(d.c.Token(cluster.FrontA, "editor"), res, map[string]any{"tool": tool, "table": "Entries"})
|
||||
Expect(r.Status).To(Equal(403), tool+": "+r.String())
|
||||
Expect(r.String()).NotTo(ContainSubstring("pen"))
|
||||
}
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(1))
|
||||
})
|
||||
|
||||
It("applies a proposal only on the confirmation of the proposer's own tab, with its digest, exactly once", func() {
|
||||
op := opID("approve")
|
||||
p := d.propose(cluster.FrontA, "editor", res, "write", op, map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{row}})
|
||||
Expect(p.Status).To(Equal(200), p.String())
|
||||
id, _ := body(p)["id"].(string)
|
||||
digest, _ := body(p)["digest"].(string)
|
||||
Expect(id).To(Equal(op))
|
||||
bad := map[string]*cluster.Response{}
|
||||
// A key never approves, whoever it belongs to.
|
||||
// The header that names the proposal is sent as well: the key is what
|
||||
// fails, not a missing header.
|
||||
named := map[string]string{"X-LocalAI-Confirm": id, "Sec-Fetch-Site": "same-origin", "Sec-Fetch-Mode": "cors", "Sec-Fetch-Dest": "empty"}
|
||||
keyed, _ := d.c.Token(cluster.FrontB, "editor").Do("POST", "/api/addons/proposals/"+id+"/confirm", map[string]any{"digest": digest}, named)
|
||||
bad["the proposer's key"] = keyed
|
||||
keyedOwner, _ := d.c.Token(cluster.Owner, "member").Do("POST", "/api/addons/proposals/"+id+"/confirm", map[string]any{"digest": digest}, named)
|
||||
bad["the owner's key"] = keyedOwner
|
||||
keyedAdmin, _ := d.c.Token(cluster.FrontA, "admin").Do("POST", "/api/addons/proposals/"+id+"/confirm", map[string]any{"digest": digest}, named)
|
||||
bad["an administrator's key"] = keyedAdmin
|
||||
// The resource owner's own tab is not the proposer's.
|
||||
bad["the resource owner's tab"] = d.confirm(cluster.FrontB, "member", id, digest)
|
||||
// A viewer's tab, an administrator's tab.
|
||||
bad["a viewer's tab"] = d.confirm(cluster.Owner, "viewer", id, digest)
|
||||
bad["an administrator's tab"] = d.confirm(cluster.FrontA, "admin", id, digest)
|
||||
// The proposer's tab without the confirmation header, with another ID
|
||||
// and with a tool-call marker.
|
||||
bad["no confirmation header"] = d.tab(cluster.FrontB, "editor", "POST", "/api/addons/proposals/"+id+"/confirm", map[string]any{"digest": digest}, "")
|
||||
bad["another proposal's header"] = d.tab(cluster.FrontB, "editor", "POST", "/api/addons/proposals/"+id+"/confirm", map[string]any{"digest": digest}, "other-proposal")
|
||||
marked, _ := d.c.Session(cluster.FrontB, "editor").Do("POST", "/api/addons/proposals/"+id+"/confirm", map[string]any{"digest": digest},
|
||||
map[string]string{"X-LocalAI-Confirm": id, "X-LocalAI-Tool-Call": "1"})
|
||||
bad["a tool-call marker"] = marked
|
||||
for what, r := range bad {
|
||||
Expect(r.Status).To(Equal(403), what+": "+r.String())
|
||||
}
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(1), "nothing was applied by a forged approval")
|
||||
// A wrong digest is not the proposal the person saw.
|
||||
wrong := d.confirm(cluster.FrontB, "editor", id, "sha256:"+strings.Repeat("0", 64))
|
||||
Expect(wrong.Status).NotTo(Equal(200), wrong.String())
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(1))
|
||||
// The proposer's own tab, through another node than the one that
|
||||
// stored the proposal, applies it once; repeats are the same receipt.
|
||||
ok := d.confirm(cluster.FrontB, "editor", id, digest)
|
||||
Expect(ok.Status).To(Equal(200), ok.String())
|
||||
Expect(body(ok)["state"]).To(Equal("applied"))
|
||||
for _, n := range nodes {
|
||||
again := d.confirm(n, "editor", id, digest)
|
||||
Expect(again.Status).To(Equal(200), n+": "+again.String())
|
||||
Expect(body(again)["state"]).To(Equal("applied"))
|
||||
}
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(2))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,157 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
// Command tables-cluster runs the acceptance fixture for the browser checks:
|
||||
// one owner and two forwarding frontends over the real carrier, a local
|
||||
// catalog that serves the packaged Tables, and Tables installed and enabled
|
||||
// through a frontend as an administrator would. It writes a ready file with
|
||||
// the addresses, the accounts and a loopback control address, and runs until
|
||||
// it is stopped. The control address lets a browser spec stop and start the
|
||||
// owner, so it can look at the page when the owner is gone.
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/subtle"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
"github.com/mudler/LocalAI/tests/addons/tablesacceptance/tablespkg"
|
||||
)
|
||||
|
||||
func main() {
|
||||
dir := flag.String("dir", "", "directory of the cluster state (created)")
|
||||
server := flag.String("server", "", "UI test server executable")
|
||||
mock := flag.String("mock-backend", "", "mock backend executable")
|
||||
tables := flag.String("tables", "", "Tables executable")
|
||||
root := flag.String("root", "", "repository root")
|
||||
ready := flag.String("ready", "", "file written when the cluster is serving")
|
||||
flag.Parse()
|
||||
if err := run(*dir, *root, cluster.Paths{Server: *server, Mock: *mock}, *tables, *ready); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "tables-cluster:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func run(dir, root string, p cluster.Paths, tables, ready string) error {
|
||||
if dir == "" || root == "" || ready == "" || p.Server == "" || p.Mock == "" || tables == "" {
|
||||
return fmt.Errorf("-dir, -root, -server, -mock-backend, -tables and -ready are required")
|
||||
}
|
||||
scratch := filepath.Join(dir, "packages")
|
||||
if err := os.MkdirAll(scratch, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
releases, err := tablespkg.Build(root, scratch, tables, nil, "0.1.0")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
c, err := cluster.New(filepath.Join(dir, "cluster"), p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer c.Down()
|
||||
stop := make(chan os.Signal, 1)
|
||||
signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
|
||||
interrupted := make(chan struct{})
|
||||
go func() {
|
||||
<-stop
|
||||
c.Down()
|
||||
close(interrupted)
|
||||
os.Exit(1)
|
||||
}()
|
||||
if _, err := c.Library.AddArchive("tables", "0.1.0", releases["0.1.0"].Archive); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := c.Up(); err != nil {
|
||||
return err
|
||||
}
|
||||
admin := c.Token(cluster.FrontA, "admin")
|
||||
for i, action := range []string{"install", "enable"} {
|
||||
args := map[string]any{}
|
||||
if action == "install" {
|
||||
args = map[string]any{"sourceId": "fixture", "version": "0.1.0"}
|
||||
}
|
||||
r, err := admin.Do("POST", "/api/addons/dispatch", map[string]any{"operationId": fmt.Sprintf("browser-%s-%d", action, i), "addonId": "tables", "generation": 1, "action": action, "arguments": args}, nil)
|
||||
if err != nil || r.Status != 200 {
|
||||
return fmt.Errorf("%s: %v %v", action, err, r)
|
||||
}
|
||||
}
|
||||
deadline := time.Now().Add(60 * time.Second)
|
||||
for {
|
||||
r, err := admin.Do("GET", "/api/addons/apps", nil, nil)
|
||||
if err == nil && r.Status == 200 && strings.Contains(string(r.Body), `"state":"running"`) {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
return fmt.Errorf("Tables did not start")
|
||||
}
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
}
|
||||
|
||||
// Owner control: stop (as a crash) and start again.
|
||||
token := make([]byte, 16)
|
||||
_, _ = rand.Read(token)
|
||||
secret := hex.EncodeToString(token)
|
||||
mux := http.NewServeMux()
|
||||
guard := func(w http.ResponseWriter, r *http.Request) bool {
|
||||
if r.Method != http.MethodPost || subtle.ConstantTimeCompare([]byte(r.Header.Get("X-Control")), []byte(secret)) != 1 {
|
||||
http.Error(w, "denied", http.StatusForbidden)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
mux.HandleFunc("/owner/stop", func(w http.ResponseWriter, r *http.Request) {
|
||||
if guard(w, r) {
|
||||
c.Nodes[cluster.Owner].Kill()
|
||||
_, _ = w.Write([]byte("stopped"))
|
||||
}
|
||||
})
|
||||
mux.HandleFunc("/owner/start", func(w http.ResponseWriter, r *http.Request) {
|
||||
if !guard(w, r) {
|
||||
return
|
||||
}
|
||||
n := c.Nodes[cluster.Owner]
|
||||
if err := n.Start(); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if err := n.Ready(120 * time.Second); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte("started"))
|
||||
})
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
srv := &http.Server{Handler: mux, ReadHeaderTimeout: 5 * time.Second}
|
||||
go func() { _ = srv.Serve(l) }()
|
||||
defer func() { _ = srv.Close() }()
|
||||
|
||||
out := map[string]any{"urls": map[string]string{}, "users": c.Creds.Users, "dir": c.Config.Dir,
|
||||
"control": map[string]string{"url": "http://" + l.Addr().String(), "token": secret}}
|
||||
for name, n := range c.Nodes {
|
||||
out["urls"].(map[string]string)[name] = n.URL()
|
||||
}
|
||||
b, _ := json.MarshalIndent(out, "", " ")
|
||||
tmp := filepath.Clean(ready) + ".tmp"
|
||||
if err := os.WriteFile(tmp, b, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Rename(tmp, ready); err != nil {
|
||||
return err
|
||||
}
|
||||
<-interrupted
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,265 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesacceptance_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"sync"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("Tables staleness, replay and races across the owner and two frontends", Ordered, func() {
|
||||
var (
|
||||
d *deployment
|
||||
res string
|
||||
)
|
||||
|
||||
BeforeAll(func() {
|
||||
d = deploy(start("0.1.0"), "0.1.0")
|
||||
res = d.seedLedger(cluster.FrontA, "member", "Shared", map[string]any{"Item": "pen", "Qty": 3})
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("refuses a change prepared on an old schema or data revision from two tabs of two people and applies nothing", func() {
|
||||
// Both people open the database at the same revision.
|
||||
ownerView := d.revision(cluster.FrontA, "member", res)
|
||||
editorView := d.revision(cluster.FrontB, "editor", res)
|
||||
Expect(editorView).To(Equal(ownerView))
|
||||
|
||||
// The editor changes the data first, through another node.
|
||||
first := d.apply(cluster.Owner, "editor", res, opID("st-1"), "rows-insert", editorView, map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "ink", "Qty": 1}}})
|
||||
Expect(first.Status).To(Equal(200), first.String())
|
||||
ids, _ := body(first)["result"].(map[string]any)["ids"].([]any)
|
||||
Expect(ids).To(HaveLen(1))
|
||||
|
||||
// The owner's stale tab: a data change and a schema change both fail
|
||||
// as stale and change nothing.
|
||||
for _, c := range []struct {
|
||||
tool string
|
||||
args map[string]any
|
||||
}{
|
||||
{"rows-insert", map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "late", "Qty": 9}}}},
|
||||
{"rows-update", map[string]any{"table": "Entries", "ids": ids, "set": map[string]any{"Qty": 7}}},
|
||||
{"columns-add", map[string]any{"table": "Entries", "column": "Note", "type": "text"}},
|
||||
} {
|
||||
op := opID("st-old")
|
||||
receiptsBefore := len(d.receipts(res))
|
||||
r := d.apply(cluster.FrontB, "member", res, op, c.tool, ownerView, c.args)
|
||||
Expect(len(d.receipts(res))).To(Equal(receiptsBefore), "a stale change leaves no receipt")
|
||||
Expect(r.Status).To(Equal(http.StatusConflict), c.tool+": "+r.String())
|
||||
Expect(errType(r)).To(Equal("stale"), c.tool)
|
||||
_ = op
|
||||
}
|
||||
Expect(d.rowCount(cluster.FrontA, "member", res)).To(Equal(2))
|
||||
|
||||
// The owner changes the schema; the editor's old view is stale for a
|
||||
// row change that uses the new column too, and never half-applies.
|
||||
Expect(d.applyOK(cluster.FrontA, "member", res, "columns-add", map[string]any{"table": "Entries", "column": "Note", "type": "text"})).NotTo(BeNil())
|
||||
op := opID("st-schema")
|
||||
r := d.apply(cluster.FrontB, "editor", res, op, "rows-insert", editorView, map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "x", "Note": "n"}}})
|
||||
Expect(r.Status).To(Equal(http.StatusConflict), r.String())
|
||||
Expect(errType(r)).To(Equal("stale"))
|
||||
Expect(d.rowCount(cluster.Owner, "editor", res)).To(Equal(2))
|
||||
|
||||
// A proposal prepared before the revision moved is rejected when it is
|
||||
// confirmed, and the data stays as it is.
|
||||
held := d.propose(cluster.FrontA, "editor", res, "write", opID("st-prop"), map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{map[string]any{"Item": "held", "Qty": 2}}})
|
||||
Expect(held.Status).To(Equal(200), held.String())
|
||||
heldID, _ := body(held)["id"].(string)
|
||||
heldDigest, _ := body(held)["digest"].(string)
|
||||
d.applyOK(cluster.Owner, "member", res, "columns-add", map[string]any{"table": "Entries", "column": "Tag", "type": "text"})
|
||||
receiptsAtHold := len(d.receipts(res))
|
||||
got := d.confirm(cluster.FrontB, "editor", heldID, heldDigest)
|
||||
Expect(body(got)["state"]).To(Equal("rejected"), got.String())
|
||||
Expect(len(d.receipts(res))).To(Equal(receiptsAtHold), "the rejected proposal left no receipt")
|
||||
Expect(d.rowCount(cluster.FrontA, "member", res)).To(Equal(2))
|
||||
|
||||
// With the revision it read, the same person applies.
|
||||
ok := d.apply(cluster.FrontB, "editor", res, opID("st-fresh"), "rows-insert", d.revision(cluster.FrontB, "editor", res),
|
||||
map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "fresh", "Qty": 4}}})
|
||||
Expect(ok.Status).To(Equal(200), ok.String())
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(3))
|
||||
})
|
||||
|
||||
It("replays an operation ID as one receipt from every node, conflicts on a different request and never applies twice or under a new ID", func() {
|
||||
before := d.rowCount(cluster.Owner, "member", res)
|
||||
receiptsBefore := len(d.receipts(res))
|
||||
rev := d.revision(cluster.FrontA, "member", res)
|
||||
args := map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "once", "Qty": 1}}}
|
||||
op := opID("replay")
|
||||
first := d.apply(cluster.FrontA, "editor", res, op, "rows-insert", rev, args)
|
||||
Expect(first.Status).To(Equal(200), first.String())
|
||||
hash := body(first)["requestHash"]
|
||||
Expect(hash).NotTo(BeEmpty())
|
||||
// No sticky session: the replays alternate over every node and over
|
||||
// the credential kinds, and each is the same receipt.
|
||||
for _, n := range append(append([]string{}, nodes...), nodes...) {
|
||||
again := d.apply(n, "editor", res, op, "rows-insert", rev, args)
|
||||
Expect(again.Status).To(Equal(200), n+": "+again.String())
|
||||
Expect(body(again)["revision"]).To(Equal(body(first)["revision"]))
|
||||
Expect(body(again)["requestHash"]).To(Equal(hash))
|
||||
Expect(body(again)["result"]).To(Equal(body(first)["result"]))
|
||||
}
|
||||
// Another request under the same ID is a conflict on every node.
|
||||
for _, n := range nodes {
|
||||
other := d.apply(n, "editor", res, op, "rows-insert", rev, map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "other", "Qty": 2}}})
|
||||
Expect(other.Status).To(Equal(http.StatusConflict), n+": "+other.String())
|
||||
Expect(errType(other)).To(Equal("operation_conflict"))
|
||||
}
|
||||
Expect(d.rowCount(cluster.FrontB, "member", res)).To(Equal(before + 1))
|
||||
Expect(len(d.receipts(res))).To(Equal(receiptsBefore+1), "the add-on committed the operation once")
|
||||
|
||||
// A burst from the three nodes at once, one fresh ID each: every
|
||||
// acknowledged ID is applied exactly once, and the add-on holds no
|
||||
// receipt that no one asked for (no retry under a new ID).
|
||||
var wg sync.WaitGroup
|
||||
results := make([]*cluster.Response, 12)
|
||||
ops := make([]string, 12)
|
||||
gate := make(chan struct{})
|
||||
for i := range results {
|
||||
wg.Add(1)
|
||||
ops[i] = opID("burst")
|
||||
go func(i int) {
|
||||
defer GinkgoRecover()
|
||||
defer wg.Done()
|
||||
<-gate
|
||||
// Revisions race, so a stale answer is correct; the caller
|
||||
// retries with the revision it reads, under the same ID.
|
||||
for attempt := 0; attempt < 40; attempt++ {
|
||||
n := nodes[(i+attempt)%len(nodes)]
|
||||
rev := d.revision(n, "member", res)
|
||||
r := d.apply(n, "member", res, ops[i], "rows-insert", rev, map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": fmt.Sprintf("b%d", i), "Qty": i}}})
|
||||
// A stale answer retries with the revision just read. A
|
||||
// busy owner (the add-on admits four calls at once) answers
|
||||
// outcome_unknown: the documented recovery is the same call
|
||||
// under the same ID, which the add-on applies once.
|
||||
stale := r.Status == http.StatusConflict && errType(r) == "stale"
|
||||
unknown := r.Status == http.StatusServiceUnavailable && errType(r) == "outcome_unknown"
|
||||
if !stale && !unknown {
|
||||
results[i] = r
|
||||
return
|
||||
}
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
close(gate)
|
||||
wg.Wait()
|
||||
applied := 0
|
||||
for i, r := range results {
|
||||
Expect(r).NotTo(BeNil(), ops[i])
|
||||
Expect(r.Status).To(Equal(200), ops[i]+": "+r.String())
|
||||
applied++
|
||||
}
|
||||
Expect(len(d.receipts(res))).To(Equal(receiptsBefore+1+applied), "one receipt per acknowledged ID and none more")
|
||||
Expect(d.rowCount(cluster.FrontA, "member", res)).To(Equal(before + 1 + applied))
|
||||
})
|
||||
|
||||
It("settles a revocation that races a confirmation: one outcome, applied at most once, and no work after the revocation", func() {
|
||||
unresolved := ""
|
||||
blockedSeen := 0
|
||||
for round := 0; round < 6; round++ {
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
if unresolved != "" {
|
||||
// A confirmation that the revocation cut short is unresolved
|
||||
// until its receipt is read, and it blocks every other write
|
||||
// on the database. The block is the host's, not the add-on's.
|
||||
blocked := d.propose(cluster.FrontB, "editor", res, "write", opID("blocked"), map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{map[string]any{"Item": "blocked"}}})
|
||||
Expect(blocked.Status).To(Equal(403), blocked.String())
|
||||
blockedSeen++
|
||||
rec := d.tab(nodes[round%3], "editor", "POST", "/api/addons/proposals/"+unresolved+"/reconcile", map[string]any{}, unresolved)
|
||||
Expect(rec.Status).To(Equal(200), rec.String())
|
||||
if body(rec)["state"] == "outcome-unknown" {
|
||||
// The add-on has no receipt, which does not prove it never
|
||||
// ran: the block stays until the owner role resolves it.
|
||||
stillBlocked := d.propose(cluster.FrontA, "editor", res, "write", opID("still"), map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{map[string]any{"Item": "blocked"}}})
|
||||
Expect(stillBlocked.Status).To(Equal(403), stillBlocked.String())
|
||||
listed := d.tab(nodes[(round+1)%3], "member", "GET", "/api/addons/resources/"+res+"/proposals?state=unresolved", nil, res)
|
||||
Expect(listed.Status).To(Equal(200), listed.String())
|
||||
entries, _ := body(listed)["proposals"].([]any)
|
||||
Expect(entries).NotTo(BeEmpty(), listed.String())
|
||||
// Abandoned earlier rounds stay listed; this one is the open one.
|
||||
var entry map[string]any
|
||||
for _, e := range entries {
|
||||
if m := e.(map[string]any); m["id"] == unresolved {
|
||||
entry = m
|
||||
}
|
||||
}
|
||||
Expect(entry).NotTo(BeNil(), listed.String())
|
||||
Expect(entry["state"]).To(Equal("outcome-unknown"))
|
||||
// Neither the proposer nor an administrator can resolve it.
|
||||
for _, user := range []string{"editor", "admin"} {
|
||||
denied := d.tab(cluster.FrontB, user, "POST", "/api/addons/proposals/"+unresolved+"/resolve", map[string]any{"digest": entry["digest"], "reason": "not mine"}, unresolved)
|
||||
Expect(denied.Status).To(Equal(403), user+": "+denied.String())
|
||||
}
|
||||
done := d.tab(nodes[(round+2)%3], "member", "POST", "/api/addons/proposals/"+unresolved+"/resolve", map[string]any{"digest": entry["digest"], "reason": "checked the rows"}, unresolved)
|
||||
Expect(done.Status).To(Equal(200), done.String())
|
||||
Expect(body(done)["state"]).To(Equal("abandoned-unknown"), done.String())
|
||||
}
|
||||
unresolved = ""
|
||||
}
|
||||
before := d.rowCount(cluster.Owner, "member", res)
|
||||
receiptsBefore := len(d.receipts(res))
|
||||
op := opID("race")
|
||||
p := d.propose(cluster.FrontA, "editor", res, "write", op, map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{map[string]any{"Item": "race", "Qty": round}}})
|
||||
Expect(p.Status).To(Equal(200), p.String())
|
||||
digest, _ := body(p)["digest"].(string)
|
||||
|
||||
var confirmed, revoked *cluster.Response
|
||||
var wg sync.WaitGroup
|
||||
gate := make(chan struct{})
|
||||
wg.Add(2)
|
||||
go func() {
|
||||
defer GinkgoRecover()
|
||||
defer wg.Done()
|
||||
<-gate
|
||||
confirmed = d.confirm(nodes[round%3], "editor", op, digest)
|
||||
}()
|
||||
go func() {
|
||||
defer GinkgoRecover()
|
||||
defer wg.Done()
|
||||
<-gate
|
||||
revoked = d.revoke(nodes[(round+1)%3], "member", res, "editor")
|
||||
}()
|
||||
close(gate)
|
||||
wg.Wait()
|
||||
Expect(revoked.Status).To(Equal(200), revoked.String())
|
||||
|
||||
// The revocation is acknowledged: the editor is out everywhere.
|
||||
for _, n := range nodes {
|
||||
Expect(d.read(d.c.Token(n, "editor"), res, map[string]any{"tool": "rows-query", "table": "Entries"}).Status).To(Equal(403), n)
|
||||
}
|
||||
after := d.rowCount(cluster.FrontB, "member", res)
|
||||
switch {
|
||||
case confirmed.Status == 200:
|
||||
Expect(body(confirmed)["state"]).To(Equal("applied"))
|
||||
Expect(after).To(Equal(before+1), fmt.Sprintf("round %d: applied once", round))
|
||||
case confirmed.Status == 503 && errType(confirmed) == "outcome_unknown":
|
||||
unresolved = op
|
||||
Expect(after).To(BeElementOf(before, before+1), fmt.Sprintf("round %d: unknown, at most once", round))
|
||||
Expect(len(d.receipts(res))).To(Equal(receiptsBefore + after - before))
|
||||
default:
|
||||
Expect(confirmed.Status).To(BeElementOf(403, 409, 503), confirmed.String())
|
||||
Expect(after).To(Equal(before), fmt.Sprintf("round %d: denied, nothing applied (%s)", round, confirmed.String()))
|
||||
Expect(len(d.receipts(res))).To(Equal(receiptsBefore))
|
||||
}
|
||||
// The same confirmation again, after the revocation, does nothing.
|
||||
// It may return the stored record of what already happened, and it
|
||||
// can only say "applied" when the row is there.
|
||||
late := d.confirm(nodes[(round+2)%3], "editor", op, digest)
|
||||
if late.Status == 200 && body(late)["state"] == "applied" {
|
||||
Expect(after).To(Equal(before+1), "an applied record needs its row: "+late.String())
|
||||
}
|
||||
Expect(late.Status == 200 && body(late)["state"] == "applied" && after == before).To(BeFalse(), late.String())
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(after))
|
||||
if late.Status == 200 && (body(late)["state"] == "applied" || body(late)["state"] == "rejected") {
|
||||
// The repeated call read the receipt: nothing is left to settle.
|
||||
unresolved = ""
|
||||
}
|
||||
}
|
||||
// Leave the grant as the other specs expect it.
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,159 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesacceptance_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"runtime"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// TestEvidenceManifest checks the evidence manifest the acceptance run wrote.
|
||||
// scripts/addons/tables_acceptance.sh runs it after the suite with
|
||||
// TABLES_ACCEPTANCE_CHECK_EVIDENCE=1; a plain `go test` of the package skips
|
||||
// it, because the manifest does not exist before the suite ran.
|
||||
//
|
||||
// The manifest is evidence, so the check is strict: the platform is the native
|
||||
// one, the commands are listed, both package digests are well formed and
|
||||
// distinct, the case inventory is exactly the inventory of suites.json, every
|
||||
// case passed, and a case that did not has a retained trace that exists.
|
||||
func TestEvidenceManifest(t *testing.T) {
|
||||
if os.Getenv("TABLES_ACCEPTANCE_CHECK_EVIDENCE") != "1" {
|
||||
// Ginkgo's Skip works only inside a spec; this is a plain Go test.
|
||||
t.Skip("runs after the suite, from scripts/addons/tables_acceptance.sh") //nolint:forbidigo
|
||||
}
|
||||
g := NewWithT(t)
|
||||
// Every broken rule is reported, not only the first one.
|
||||
var problems []string
|
||||
problem := func(format string, args ...any) { problems = append(problems, fmt.Sprintf(format, args...)) }
|
||||
defer func() { g.Expect(problems).To(BeEmpty()) }()
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
root := filepath.Clean(filepath.Join(filepath.Dir(file), "..", "..", ".."))
|
||||
path, err := evidenceLocation(os.Getenv("TABLES_ACCEPTANCE_EVIDENCE"), root)
|
||||
g.Expect(err).NotTo(HaveOccurred())
|
||||
raw, err := os.ReadFile(path)
|
||||
g.Expect(err).NotTo(HaveOccurred(), "the evidence manifest is missing")
|
||||
var ev evidence
|
||||
dec := json.NewDecoder(strings.NewReader(string(raw)))
|
||||
dec.DisallowUnknownFields()
|
||||
g.Expect(dec.Decode(&ev)).To(Succeed(), "the evidence manifest does not parse")
|
||||
g.Expect(ev.Schema).To(Equal(1), "unexpected schema")
|
||||
g.Expect(ev.Suite).To(Equal("tables-acceptance"))
|
||||
g.Expect(ev.Platform.OS).To(Equal(runtime.GOOS), "the evidence is not of this native platform: %+v", ev.Platform)
|
||||
g.Expect(ev.Platform.Arch).To(Equal(runtime.GOARCH), "the evidence is not of this native platform: %+v", ev.Platform)
|
||||
g.Expect(ev.Platform.Native).To(BeTrue(), "the evidence is not of this native platform: %+v", ev.Platform)
|
||||
g.Expect(ev.Publication).To(HavePrefix("none"), "the manifest must say nothing was published")
|
||||
joined := make([]string, 0, len(ev.Commands))
|
||||
for _, c := range ev.Commands {
|
||||
joined = append(joined, strings.Join(c, " "))
|
||||
}
|
||||
for _, want := range []string{"scripts/addons/package.sh", "go test"} {
|
||||
found := false
|
||||
for _, c := range joined {
|
||||
if strings.Contains(c, want) {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
problem("no recorded command contains %q: %v", want, joined)
|
||||
}
|
||||
}
|
||||
digest := regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
|
||||
seen := map[string]string{}
|
||||
for _, v := range []string{"0.1.0", "0.2.0"} {
|
||||
d, ok := ev.Artifact.Versions[v]
|
||||
g.Expect(ok).To(BeTrue(), "no digest of version %s", v)
|
||||
for what, value := range map[string]string{"archive": d.Archive, "image": d.Image} {
|
||||
if !digest.MatchString(value) {
|
||||
problem("%s digest of %s is malformed: %q", what, v, value)
|
||||
}
|
||||
if other, dup := seen[value]; dup {
|
||||
problem("%s digest of %s equals %s", what, v, other)
|
||||
}
|
||||
seen[value] = v + " " + what
|
||||
}
|
||||
}
|
||||
if ev.Artifact.Addon != "tables" {
|
||||
problem("the artifact is %q", ev.Artifact.Addon)
|
||||
}
|
||||
|
||||
inventory, err := os.ReadFile(filepath.Join(root, "scripts/addons/suites.json"))
|
||||
g.Expect(err).NotTo(HaveOccurred())
|
||||
var suites map[string]struct {
|
||||
Cases []string `json:"cases"`
|
||||
}
|
||||
g.Expect(json.Unmarshal(inventory, &suites)).To(Succeed())
|
||||
var want, wantBrowser []string
|
||||
for _, c := range suites["tables-acceptance"].Cases {
|
||||
if name, ok := strings.CutPrefix(c, "go: "); ok {
|
||||
want = append(want, name)
|
||||
}
|
||||
if strings.HasPrefix(c, "tables-acceptance: ") {
|
||||
wantBrowser = append(wantBrowser, c)
|
||||
}
|
||||
}
|
||||
var got []string
|
||||
for _, c := range ev.Cases {
|
||||
got = append(got, c.Name)
|
||||
if c.State != "passed" {
|
||||
problem("case %q is %s", c.Name, c.State)
|
||||
if c.Trace == "" {
|
||||
problem("case %q failed and no trace was retained", c.Name)
|
||||
} else if _, err := os.Stat(c.Trace); err != nil {
|
||||
problem("the trace of %q is missing: %v", c.Name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
var gotBrowser []string
|
||||
for _, c := range ev.Browser {
|
||||
gotBrowser = append(gotBrowser, c.Name)
|
||||
if c.State != "passed" {
|
||||
problem("browser case %q is %s", c.Name, c.State)
|
||||
if c.Trace == "" {
|
||||
problem("browser case %q failed and no trace was retained", c.Name)
|
||||
} else if _, err := os.Stat(c.Trace); err != nil {
|
||||
problem("the trace of %q is missing: %v", c.Name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Strings(wantBrowser)
|
||||
sort.Strings(gotBrowser)
|
||||
if strings.Join(wantBrowser, "\n") != strings.Join(gotBrowser, "\n") {
|
||||
problem("the browser cases of the manifest differ from suites.json:\nmanifest: %q\ninventory: %q", gotBrowser, wantBrowser)
|
||||
}
|
||||
if len(wantBrowser) == 0 {
|
||||
problem("the inventory has no browser case")
|
||||
}
|
||||
sort.Strings(want)
|
||||
sort.Strings(got)
|
||||
if strings.Join(want, "\n") != strings.Join(got, "\n") {
|
||||
problem("the case inventory of the manifest differs from suites.json:\nmanifest: %q\ninventory: %q", got, want)
|
||||
}
|
||||
if len(want) == 0 {
|
||||
problem("the inventory has no Go case")
|
||||
}
|
||||
}
|
||||
|
||||
// A relative evidence path resolves against the package directory the test
|
||||
// binary runs in, which once left a stale manifest inside the tree. The suite
|
||||
// refuses it, whatever the writer or the checker.
|
||||
func TestEvidencePathMustBeAbsolute(t *testing.T) {
|
||||
g := NewWithT(t)
|
||||
_, err := evidenceLocation(".cache/addons/tables-acceptance-evidence.json", "/repo")
|
||||
g.Expect(err).To(HaveOccurred(), "a relative evidence path must be refused")
|
||||
_, err = evidenceLocation("../evidence.json", "/repo")
|
||||
g.Expect(err).To(HaveOccurred(), "a parent-relative evidence path must be refused")
|
||||
p, err := evidenceLocation("", "/repo")
|
||||
g.Expect(err).NotTo(HaveOccurred())
|
||||
g.Expect(p).To(Equal("/repo/.cache/addons/tables-acceptance-evidence.json"), "the default is under the repository cache")
|
||||
p, err = evidenceLocation("/work/evidence.json", "/repo")
|
||||
g.Expect(err).NotTo(HaveOccurred())
|
||||
g.Expect(p).To(Equal("/work/evidence.json"), "an absolute path is used as given")
|
||||
}
|
||||
@@ -0,0 +1,211 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesacceptance_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
"github.com/mudler/LocalAI/tests/addons/tablesacceptance/tablespkg"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// snapshot lists a directory tree with the size and time of every file, so a
|
||||
// spec can prove that something did not touch it.
|
||||
func snapshot(root string) []string {
|
||||
GinkgoHelper()
|
||||
var out []string
|
||||
err := filepath.WalkDir(root, func(p string, e fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
info, err := e.Info()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
out = append(out, fmt.Sprintf("%s %d %d", p, info.Size(), info.ModTime().UnixNano()))
|
||||
return nil
|
||||
})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// Faults that the real executable cannot show on its own: an artifact that
|
||||
// expires, a transfer that is withdrawn and a change that committed and lost
|
||||
// its answer. They run against a test-only build of the same executable
|
||||
// (addons/tables/cmd/tables-faults) that a spec arms by creating a file in the
|
||||
// "fault" directory of the add-on's data, and a short artifact lifetime that
|
||||
// only the fixture's launcher sets. The product executable has neither.
|
||||
var _ = Describe("Tables with faults armed in the test-only build", Ordered, func() {
|
||||
var (
|
||||
d *deployment
|
||||
res string
|
||||
data string
|
||||
)
|
||||
row := map[string]any{"Item": "ink", "Qty": 2}
|
||||
|
||||
arm := func(name string) {
|
||||
GinkgoHelper()
|
||||
Expect(os.MkdirAll(filepath.Join(data, "fault"), 0o700)).To(Succeed())
|
||||
Expect(os.WriteFile(filepath.Join(data, "fault", name), nil, 0o600)).To(Succeed())
|
||||
}
|
||||
disarm := func(name string) { _ = os.Remove(filepath.Join(data, "fault", name)) }
|
||||
|
||||
BeforeAll(func() {
|
||||
scratch, err := os.MkdirTemp("", "tables-faults-*")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() {
|
||||
_ = filepath.WalkDir(scratch, func(p string, e fs.DirEntry, err error) error {
|
||||
if err == nil && e.IsDir() {
|
||||
_ = os.Chmod(p, 0o700)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
_ = os.RemoveAll(scratch)
|
||||
})
|
||||
exe := filepath.Join(scratch, "tables-faults")
|
||||
Expect(tablespkg.BuildExecutableFrom(repoRoot, exe, "./addons/tables/cmd/tables-faults", func(argv ...string) { record(argv...) })).To(Succeed())
|
||||
built, err := tablespkg.Build(repoRoot, scratch, exe, func(argv ...string) { record(argv...) }, "0.1.0")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
c := startFrom(func(cfg *cluster.Config) { cfg.ArtifactExpiryMillis = 12000 }, built, "0.1.0")
|
||||
d = deploy(c, "0.1.0")
|
||||
data = filepath.Join(c.Config.Nodes[cluster.Owner].Root, "data", "tables")
|
||||
res = d.seedLedger(cluster.FrontA, "member", "Faults", map[string]any{"Item": "pen", "Qty": 3})
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("denies an expired file identically from every node and leaves the data and the backups untouched", func() {
|
||||
op := opID("backup")
|
||||
backup := d.tab(cluster.FrontB, "admin", "POST", "/api/addons/tables/recovery/backup", map[string]any{"operationId": op}, op)
|
||||
Expect(backup.Status).To(Equal(200), backup.String())
|
||||
id := d.export(cluster.FrontA, "member", res)
|
||||
for _, n := range nodes {
|
||||
Expect(d.download(n, "member", id).Status).To(Equal(200), n)
|
||||
}
|
||||
db, err := os.Stat(d.databaseFile(res))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
receipts := d.receipts(res)
|
||||
backups := snapshot(filepath.Join(d.c.Config.Nodes[cluster.Owner].Root, "backups"))
|
||||
Expect(backups).NotTo(BeEmpty())
|
||||
|
||||
// The lifetime ends by itself; the first node to see it is not special.
|
||||
Eventually(func() int { return d.download(cluster.Owner, "member", id).Status }, 60*time.Second, 200*time.Millisecond).ShouldNot(Equal(200))
|
||||
first := d.download(cluster.Owner, "member", id)
|
||||
Expect(first.Status).To(Equal(403), first.String())
|
||||
Expect(first.String()).NotTo(ContainSubstring("pen"))
|
||||
for _, n := range nodes {
|
||||
r := d.download(n, "member", id)
|
||||
Expect(r.Status).To(Equal(first.Status), n)
|
||||
Expect(r.String()).To(Equal(first.String()), n+": the same answer from every node")
|
||||
st, err := d.c.Token(n, "member").Do("GET", "/api/addons/artifacts/"+id, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(st.Status).To(Equal(403), n+": "+st.String())
|
||||
}
|
||||
after, err := os.Stat(d.databaseFile(res))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(after.ModTime()).To(Equal(db.ModTime()))
|
||||
Expect(d.receipts(res)).To(Equal(receipts))
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(1))
|
||||
Expect(snapshot(filepath.Join(d.c.Config.Nodes[cluster.Owner].Root, "backups"))).To(Equal(backups))
|
||||
// A new export is a new file with a new ID.
|
||||
fresh := d.export(cluster.FrontB, "member", res)
|
||||
Expect(fresh).NotTo(Equal(id))
|
||||
Expect(d.download(cluster.FrontA, "member", fresh).Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("withdraws a transfer held in flight when the add-on is disabled and leaves no downloadable file", func() {
|
||||
arm("hold-output")
|
||||
DeferCleanup(func() { disarm("hold-output") })
|
||||
r := d.read(d.c.Token(cluster.FrontA, "member"), res, map[string]any{"tool": "rows-export", "table": "Entries"})
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
outs, _ := body(r)["outputs"].([]any)
|
||||
Expect(outs).To(HaveLen(1))
|
||||
id, _ := outs[0].(map[string]any)["handle"].(string)
|
||||
Expect(id).NotTo(BeEmpty())
|
||||
status := func(n string) (int, string) {
|
||||
st, err := d.c.Token(n, "member").Do("GET", "/api/addons/artifacts/"+id, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
s, _ := body(st)["state"].(string)
|
||||
return st.Status, s
|
||||
}
|
||||
// The held stream cannot finish: the file is not downloadable on any node.
|
||||
for _, n := range nodes {
|
||||
code, state := status(n)
|
||||
Expect(code).To(Equal(200), n)
|
||||
Expect(state).To(Equal("transferring"), n)
|
||||
Expect(d.download(n, "member", id).Status).NotTo(Equal(200), n)
|
||||
}
|
||||
gen := d.gen
|
||||
Expect(d.lifecycle(cluster.FrontB, "disable", gen, "").Status).To(Equal(200))
|
||||
disarm("hold-output")
|
||||
Expect(d.lifecycle(cluster.FrontA, "enable", gen, "").Status).To(Equal(200))
|
||||
d.waitState("running")
|
||||
var final string
|
||||
Eventually(func() string {
|
||||
_, final = status(cluster.Owner)
|
||||
return final
|
||||
}, 30*time.Second, 200*time.Millisecond).ShouldNot(Equal("transferring"))
|
||||
Expect(final).To(BeElementOf("cancelled", "failed"))
|
||||
for _, n := range nodes {
|
||||
code, state := status(n)
|
||||
Expect(code).To(Equal(200), n)
|
||||
Expect(state).To(Equal(final), n)
|
||||
Expect(d.download(n, "member", id).Status).NotTo(Equal(200), n)
|
||||
}
|
||||
// The withdrawn bytes are gone and the quota is free: a new file works.
|
||||
Expect(filepath.Join(d.c.Config.Nodes[cluster.Owner].Root, "artifacts")).To(BeADirectory())
|
||||
var left []string
|
||||
_ = filepath.WalkDir(filepath.Join(d.c.Config.Nodes[cluster.Owner].Root, "artifacts"), func(p string, e fs.DirEntry, err error) error {
|
||||
if err == nil && !e.IsDir() && filepath.Base(p) == id {
|
||||
left = append(left, p)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
Expect(left).To(BeEmpty(), "no byte of a withdrawn transfer stays on disk")
|
||||
fresh := d.export(cluster.FrontB, "member", res)
|
||||
Expect(d.download(cluster.Owner, "member", fresh).Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("reports a change that committed and lost its answer as unknown, finds it by receipt and replays the same receipt once", func() {
|
||||
rows := d.rowCount(cluster.Owner, "member", res)
|
||||
receipts := len(d.receipts(res))
|
||||
pid := d.tablesPID()
|
||||
Expect(pid).NotTo(BeZero())
|
||||
op := opID("lost")
|
||||
rev := d.revisionOf(res)
|
||||
args := map[string]any{"table": "Entries", "rows": []any{row}}
|
||||
arm("crash-after-commit")
|
||||
lost := d.apply(cluster.FrontA, "member", res, op, "rows-insert", rev, args)
|
||||
Expect(lost.Status).To(Equal(503), lost.String())
|
||||
Expect(errType(lost)).To(Equal("outcome_unknown"))
|
||||
Expect(body(lost)["error"].(map[string]any)["operationId"]).To(Equal(op), "the original operation ID, from the frontend")
|
||||
d.waitTables(pid)
|
||||
// The lookup is read-only and finds the committed receipt.
|
||||
for _, n := range nodes {
|
||||
st := d.operation(n, "member", res, op)
|
||||
Expect(st["state"]).To(Equal("committed"), n)
|
||||
}
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(rows + 1))
|
||||
// The same operation again is the same receipt, not a second change.
|
||||
var first any
|
||||
for i, n := range []string{cluster.FrontB, cluster.Owner, cluster.FrontA} {
|
||||
again := d.apply(n, "member", res, op, "rows-insert", rev, args)
|
||||
Expect(again.Status).To(Equal(200), n+": "+again.String())
|
||||
if i == 0 {
|
||||
first = body(again)["revision"]
|
||||
}
|
||||
Expect(body(again)["revision"]).To(Equal(first), n)
|
||||
}
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(rows+1), "exactly one row")
|
||||
Expect(len(d.receipts(res))).To(Equal(receipts+1), "exactly one receipt")
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,347 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesacceptance_test
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
_ "github.com/mattn/go-sqlite3"
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var nodes = []string{cluster.Owner, cluster.FrontA, cluster.FrontB}
|
||||
|
||||
var opCounter atomic.Int64
|
||||
|
||||
func opID(prefix string) string {
|
||||
return fmt.Sprintf("%s-%d-%d", prefix, time.Now().UnixNano()%1_000_000_000, opCounter.Add(1))
|
||||
}
|
||||
|
||||
// body decodes a JSON answer into a generic map.
|
||||
func body(r *cluster.Response) map[string]any {
|
||||
GinkgoHelper()
|
||||
out := map[string]any{}
|
||||
_ = json.Unmarshal(r.Body, &out)
|
||||
return out
|
||||
}
|
||||
|
||||
// errType is the typed error of an answer, or the empty string.
|
||||
func errType(r *cluster.Response) string {
|
||||
e, _ := body(r)["error"].(map[string]any)
|
||||
s, _ := e["type"].(string)
|
||||
return s
|
||||
}
|
||||
|
||||
// deployment is a running cluster with Tables installed and enabled.
|
||||
type deployment struct {
|
||||
c *cluster.Cluster
|
||||
gen uint64
|
||||
}
|
||||
|
||||
// deploy installs and enables the packaged Tables through a frontend, as an
|
||||
// administrator does, and waits until it runs.
|
||||
func deploy(c *cluster.Cluster, version string) *deployment {
|
||||
GinkgoHelper()
|
||||
d := &deployment{c: c}
|
||||
admin := c.Token(cluster.FrontA, "admin")
|
||||
for _, action := range []string{"install", "enable"} {
|
||||
args := map[string]any{}
|
||||
if action == "install" {
|
||||
args = map[string]any{"sourceId": "fixture", "version": version}
|
||||
}
|
||||
r := d.dispatch(admin, map[string]any{"addonId": "tables", "generation": 1, "action": action, "arguments": args})
|
||||
Expect(r.Status).To(Equal(200), action+": "+r.String())
|
||||
}
|
||||
d.waitState("running")
|
||||
return d
|
||||
}
|
||||
|
||||
func (d *deployment) app(client cluster.Client) map[string]any {
|
||||
GinkgoHelper()
|
||||
r, err := client.Do("GET", "/api/addons/apps", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if r.Status != 200 {
|
||||
return nil
|
||||
}
|
||||
list, _ := body(r)["apps"].([]any)
|
||||
for _, a := range list {
|
||||
m := a.(map[string]any)
|
||||
if m["id"] == "tables" {
|
||||
return m
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (d *deployment) waitState(state string) {
|
||||
GinkgoHelper()
|
||||
admin := d.c.Token(cluster.Owner, "admin")
|
||||
Eventually(func() string {
|
||||
a := d.app(admin)
|
||||
if a == nil {
|
||||
return "absent"
|
||||
}
|
||||
if g, ok := a["generation"].(float64); ok {
|
||||
d.gen = uint64(g)
|
||||
}
|
||||
s, _ := a["state"].(string)
|
||||
return s
|
||||
}, 60*time.Second, 150*time.Millisecond).Should(Equal(state))
|
||||
}
|
||||
|
||||
// dispatch sends one operation to /api/addons/dispatch. A missing operation ID
|
||||
// is filled in; a missing generation is the current one.
|
||||
func (d *deployment) dispatch(c cluster.Client, req map[string]any) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
if _, ok := req["operationId"]; !ok {
|
||||
req["operationId"] = opID("op")
|
||||
}
|
||||
if _, ok := req["arguments"]; !ok {
|
||||
req["arguments"] = map[string]any{}
|
||||
}
|
||||
if _, ok := req["addonId"]; !ok {
|
||||
req["addonId"] = "tables"
|
||||
}
|
||||
if _, ok := req["generation"]; !ok {
|
||||
req["generation"] = d.gen
|
||||
}
|
||||
r, err := c.Do("POST", "/api/addons/dispatch", req, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}
|
||||
|
||||
// read runs one typed read of a resource.
|
||||
func (d *deployment) read(c cluster.Client, resource string, args map[string]any) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
return d.dispatch(c, map[string]any{"resourceId": resource, "action": "read", "arguments": args})
|
||||
}
|
||||
|
||||
func payload(r *cluster.Response) map[string]any {
|
||||
GinkgoHelper()
|
||||
p, _ := body(r)["payload"].(map[string]any)
|
||||
return p
|
||||
}
|
||||
|
||||
// tab sends a request as the person's own browser tab: the named session of
|
||||
// the user, the same-origin fetch metadata and the confirmation header.
|
||||
func (d *deployment) tab(node, user, method, path string, v any, confirm string) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
extra := map[string]string{}
|
||||
if confirm != "" {
|
||||
extra["X-LocalAI-Confirm"] = confirm
|
||||
}
|
||||
r, err := d.c.Session(node, user).Do(method, path, v, extra)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}
|
||||
|
||||
// create makes a database as the person and returns its resource ID.
|
||||
func (d *deployment) create(node, user, name string) string {
|
||||
GinkgoHelper()
|
||||
op := opID("create")
|
||||
r := d.tab(node, user, "POST", "/api/addons/resources", map[string]any{
|
||||
"addonId": "tables", "operationId": op, "generation": d.gen, "arguments": map[string]any{"name": name},
|
||||
}, op)
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
id, _ := body(r)["id"].(string)
|
||||
Expect(id).NotTo(BeEmpty(), r.String())
|
||||
return id
|
||||
}
|
||||
|
||||
func (d *deployment) grant(node, owner, res, target, role string) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
return d.tab(node, owner, "PUT", "/api/addons/resources/"+res+"/grants/"+d.c.Creds.Users[target].ID, map[string]any{"role": role}, res)
|
||||
}
|
||||
|
||||
func (d *deployment) revoke(node, owner, res, target string) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
return d.tab(node, owner, "DELETE", "/api/addons/resources/"+res+"/grants/"+d.c.Creds.Users[target].ID, nil, res)
|
||||
}
|
||||
|
||||
// revision reads the current revision of a database.
|
||||
func (d *deployment) revision(node, user, res string) map[string]any {
|
||||
GinkgoHelper()
|
||||
r := d.read(d.c.Token(node, user), res, map[string]any{"tool": "databases-inspect"})
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
rev, _ := payload(r)["revision"].(map[string]any)
|
||||
Expect(rev).NotTo(BeNil(), r.String())
|
||||
return rev
|
||||
}
|
||||
|
||||
// apply sends one UI change as the person's tab.
|
||||
func (d *deployment) apply(node, user, res, op, tool string, rev map[string]any, args any) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
return d.tab(node, user, "POST", "/api/addons/resources/"+res+"/apply", map[string]any{
|
||||
"operationId": op, "generation": d.gen, "revision": rev, "tool": tool, "arguments": args,
|
||||
}, res)
|
||||
}
|
||||
|
||||
// applyOK applies with the current revision and requires success.
|
||||
func (d *deployment) applyOK(node, user, res, tool string, args any) map[string]any {
|
||||
GinkgoHelper()
|
||||
r := d.apply(node, user, res, opID("ap"), tool, d.revision(node, user, res), args)
|
||||
Expect(r.Status).To(Equal(200), tool+": "+r.String())
|
||||
return body(r)
|
||||
}
|
||||
|
||||
// seedLedger creates a database with one table Entries(Item text, Qty
|
||||
// integer) and the rows given.
|
||||
func (d *deployment) seedLedger(node, user, name string, rows ...map[string]any) string {
|
||||
GinkgoHelper()
|
||||
res := d.create(node, user, name)
|
||||
d.applyOK(node, user, res, "tables-create", map[string]any{"table": "Entries"})
|
||||
d.applyOK(node, user, res, "columns-add", map[string]any{"table": "Entries", "column": "Item", "type": "text"})
|
||||
d.applyOK(node, user, res, "columns-add", map[string]any{"table": "Entries", "column": "Qty", "type": "integer"})
|
||||
if len(rows) > 0 {
|
||||
d.applyOK(node, user, res, "rows-insert", map[string]any{"table": "Entries", "rows": rows})
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// rowCount reads the rows of Entries as one account.
|
||||
func (d *deployment) rowCount(node, user, res string) int {
|
||||
GinkgoHelper()
|
||||
r := d.read(d.c.Token(node, user), res, map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
rows, _ := payload(r)["rows"].([]any)
|
||||
return len(rows)
|
||||
}
|
||||
|
||||
// chatTool sends one scripted tool call through the chat completions route as
|
||||
// the API key of a user. The scripted model answers its first turn with the
|
||||
// call, so the broker, the executor and the owner see a real call.
|
||||
func (d *deployment) chatTool(node, user, tool string, args map[string]any) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
raw, err := json.Marshal(args)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
message := "Please run this ADDON_TOOL_CALL:addon__tables__" + tool + ":" + base64.RawURLEncoding.EncodeToString(raw)
|
||||
r, err := d.c.Token(node, user).Do("POST", "/v1/chat/completions", map[string]any{
|
||||
"model": "mock-model", "metadata": map[string]string{"addon_tools": "true"},
|
||||
"messages": []map[string]any{{"role": "user", "content": message}},
|
||||
}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}
|
||||
|
||||
// pending lists the pending proposals of the caller.
|
||||
func (d *deployment) pending(node, user string) []map[string]any {
|
||||
GinkgoHelper()
|
||||
r, err := d.c.Token(node, user).Do("GET", "/api/addons/proposals?state=pending", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
var out []map[string]any
|
||||
list, _ := body(r)["proposals"].([]any)
|
||||
for _, x := range list {
|
||||
out = append(out, x.(map[string]any))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// propose stores one typed proposal as the API key of a user.
|
||||
func (d *deployment) propose(node, user, res, action, op string, args map[string]any) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
r, err := d.c.Token(node, user).Do("POST", "/api/addons/proposals", map[string]any{
|
||||
"addonId": "tables", "operationId": op, "generation": d.gen, "resourceId": res, "action": action, "arguments": args,
|
||||
}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}
|
||||
|
||||
// confirm confirms a proposal as the person's own tab.
|
||||
func (d *deployment) confirm(node, user, id, digest string) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
return d.tab(node, user, "POST", "/api/addons/proposals/"+id+"/confirm", map[string]any{"digest": digest}, id)
|
||||
}
|
||||
|
||||
// ---- the add-on's own SQLite database, read directly ----
|
||||
|
||||
// databaseFile is the SQLite file of a database on the owner. Tests read it
|
||||
// to count what the add-on really committed; they never write it.
|
||||
func (d *deployment) databaseFile(res string) string {
|
||||
return filepath.Join(d.c.Config.Nodes[cluster.Owner].Root, "data", "tables", "tables", "databases", res+".db")
|
||||
}
|
||||
|
||||
// receipts lists the operation IDs the add-on committed for a database, from
|
||||
// its own receipt table, oldest first.
|
||||
func (d *deployment) receipts(res string) []string {
|
||||
GinkgoHelper()
|
||||
db, err := sql.Open("sqlite3", "file:"+d.databaseFile(res)+"?mode=ro&_busy_timeout=5000")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
defer func() { _ = db.Close() }()
|
||||
rows, err := db.Query("SELECT operation_id FROM tables_schema_receipts ORDER BY applied_at, operation_id")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
defer func() { _ = rows.Close() }()
|
||||
var out []string
|
||||
for rows.Next() {
|
||||
var id string
|
||||
Expect(rows.Scan(&id)).To(Succeed())
|
||||
out = append(out, id)
|
||||
}
|
||||
Expect(rows.Err()).NotTo(HaveOccurred())
|
||||
return out
|
||||
}
|
||||
|
||||
func count(list []string, id string) int {
|
||||
n := 0
|
||||
for _, x := range list {
|
||||
if x == id {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// ---- the add-on process ----
|
||||
|
||||
// tablesPID is the process id of the Tables executable the owner supervises,
|
||||
// or zero. It is found as a child of the owner by its command line.
|
||||
func (d *deployment) tablesPID() int {
|
||||
GinkgoHelper()
|
||||
for pid, cmd := range cluster.ChildrenOf(d.c.Nodes[cluster.Owner].PID()) {
|
||||
if first := strings.Fields(cmd); len(first) > 0 && strings.HasSuffix(first[0], "/tables") {
|
||||
return pid
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// waitTables waits for a running Tables process that is not the one given.
|
||||
func (d *deployment) waitTables(not int) int {
|
||||
GinkgoHelper()
|
||||
var pid int
|
||||
Eventually(func() int {
|
||||
pid = d.tablesPID()
|
||||
if pid == not || !cluster.Alive(pid) {
|
||||
return 0
|
||||
}
|
||||
return pid
|
||||
}, 60*time.Second, 100*time.Millisecond).ShouldNot(BeZero())
|
||||
d.waitState("running")
|
||||
return pid
|
||||
}
|
||||
|
||||
// operation reads the status of an apply the person made, as the person.
|
||||
func (d *deployment) operation(node, user, res, op string) map[string]any {
|
||||
GinkgoHelper()
|
||||
r := d.tab(node, user, "GET", "/api/addons/resources/"+res+"/operations/"+op+"?kind=apply", nil, res)
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
return body(r)
|
||||
}
|
||||
|
||||
// lifecycle runs one administrator action through a node.
|
||||
func (d *deployment) lifecycle(node, action string, generation uint64, version string) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
args := map[string]any{}
|
||||
if version != "" {
|
||||
args = map[string]any{"sourceId": "fixture", "version": version}
|
||||
}
|
||||
return d.dispatch(d.c.Token(node, "admin"), map[string]any{"action": action, "generation": generation, "arguments": args})
|
||||
}
|
||||
@@ -0,0 +1,357 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesacceptance_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// settle drives every batch worker to a committed row: a worker whose reply
|
||||
// was lost looks its operation up, and repeats the same call under the same
|
||||
// ID. It never invents a new ID.
|
||||
type worker struct {
|
||||
op string
|
||||
args map[string]any
|
||||
rev map[string]any
|
||||
done bool
|
||||
reply *cluster.Response
|
||||
}
|
||||
|
||||
var _ = Describe("Tables crash and recovery with the real process", Ordered, func() {
|
||||
var (
|
||||
d *deployment
|
||||
res string
|
||||
)
|
||||
|
||||
BeforeAll(func() {
|
||||
d = deploy(start("0.1.0"), "0.1.0")
|
||||
res = d.seedLedger(cluster.FrontA, "member", "Crash", map[string]any{"Item": "seed", "Qty": 0})
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
// batch runs n writers through the three nodes at once. kill runs once the
|
||||
// first reply came back; it is the crash.
|
||||
batch := func(n int, kill func()) []*worker {
|
||||
workers := make([]*worker, n)
|
||||
var wg sync.WaitGroup
|
||||
gate := make(chan struct{})
|
||||
first := make(chan struct{}, 1)
|
||||
signal := func() {
|
||||
select {
|
||||
case first <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
for i := range workers {
|
||||
w := &worker{op: opID("crash"), args: map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": fmt.Sprintf("w%d", i), "Qty": i}}}}
|
||||
workers[i] = w
|
||||
wg.Add(1)
|
||||
go func(i int, w *worker) {
|
||||
defer GinkgoRecover()
|
||||
defer wg.Done()
|
||||
<-gate
|
||||
for attempt := 0; attempt < 30 && !w.done; attempt++ {
|
||||
node := nodes[(i+attempt)%3]
|
||||
rev, err := d.c.Token(node, "member").Do("POST", "/api/addons/dispatch", map[string]any{
|
||||
"addonId": "tables", "operationId": opID("rev"), "generation": d.gen, "resourceId": res, "action": "read", "arguments": map[string]any{"tool": "databases-inspect"},
|
||||
}, nil)
|
||||
if err != nil || rev.Status != 200 {
|
||||
signal()
|
||||
return
|
||||
}
|
||||
w.rev, _ = payload(rev)["revision"].(map[string]any)
|
||||
extra := map[string]string{"X-LocalAI-Confirm": res}
|
||||
r, err := d.c.Session(node, "member").Do("POST", "/api/addons/resources/"+res+"/apply", map[string]any{
|
||||
"operationId": w.op, "generation": d.gen, "revision": w.rev, "tool": "rows-insert", "arguments": w.args,
|
||||
}, extra)
|
||||
if err != nil {
|
||||
// The connection ended: the reply is lost.
|
||||
signal()
|
||||
return
|
||||
}
|
||||
w.reply = r
|
||||
switch {
|
||||
case r.Status == 200:
|
||||
w.done = true
|
||||
case r.Status == http.StatusConflict && errType(r) == "stale":
|
||||
continue
|
||||
default:
|
||||
// The reply is lost or the service is gone: this
|
||||
// worker stops. The same ID may be asked about later.
|
||||
signal()
|
||||
return
|
||||
}
|
||||
signal()
|
||||
}
|
||||
}(i, w)
|
||||
}
|
||||
go func() {
|
||||
<-first
|
||||
kill()
|
||||
}()
|
||||
close(gate)
|
||||
wg.Wait()
|
||||
lost := 0
|
||||
for _, w := range workers {
|
||||
if !w.done {
|
||||
lost++
|
||||
}
|
||||
}
|
||||
GinkgoWriter.Printf("batch: %d of %d writers ended without an acknowledged commit\n", lost, n)
|
||||
return workers
|
||||
}
|
||||
|
||||
// finish repeats the same call under the same ID until every worker is
|
||||
// committed, then checks that nothing applied twice and nothing was
|
||||
// applied under an ID nobody asked for.
|
||||
finish := func(workers []*worker, receiptsBefore, rowsBefore int) {
|
||||
GinkgoHelper()
|
||||
for _, w := range workers {
|
||||
for attempt := 0; attempt < 40 && !w.done; attempt++ {
|
||||
node := nodes[attempt%3]
|
||||
// The status lookup is read-only and says what is known.
|
||||
status := d.tab(node, "member", "GET", "/api/addons/resources/"+res+"/operations/"+w.op+"?kind=apply", nil, res)
|
||||
Expect(status.Status).To(Equal(200), status.String())
|
||||
w.rev = d.revision(node, "member", res)
|
||||
r := d.apply(node, "member", res, w.op, "rows-insert", w.rev, w.args)
|
||||
switch {
|
||||
case r.Status == 200:
|
||||
w.done = true
|
||||
case r.Status == http.StatusConflict && errType(r) == "stale":
|
||||
case r.Status == http.StatusServiceUnavailable:
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
default:
|
||||
Fail(fmt.Sprintf("%s: unexpected %d %s", w.op, r.Status, r.String()))
|
||||
}
|
||||
}
|
||||
Expect(w.done).To(BeTrue(), w.op)
|
||||
}
|
||||
Expect(len(d.receipts(res))).To(Equal(receiptsBefore+len(workers)), "one receipt per operation ID, none more")
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(rowsBefore + len(workers)))
|
||||
items := d.read(d.c.Token(cluster.FrontA, "member"), res, map[string]any{"tool": "rows-query", "table": "Entries", "limit": 200})
|
||||
Expect(items.Status).To(Equal(200), items.String())
|
||||
seen := map[string]int{}
|
||||
rows, _ := payload(items)["rows"].([]any)
|
||||
for _, r := range rows {
|
||||
v := r.(map[string]any)
|
||||
if item, ok := v["Item"].(string); ok {
|
||||
seen[item]++
|
||||
} else if values, ok := v["values"].(map[string]any); ok {
|
||||
if item, ok := values["Item"].(string); ok {
|
||||
seen[item]++
|
||||
}
|
||||
}
|
||||
}
|
||||
for item, n := range seen {
|
||||
Expect(n).To(Equal(1), "row "+item+" exists once")
|
||||
}
|
||||
}
|
||||
|
||||
It("keeps every acknowledged change and applies no change twice when the Tables process is killed in the middle of a batch", func() {
|
||||
receiptsBefore, rowsBefore := len(d.receipts(res)), d.rowCount(cluster.Owner, "member", res)
|
||||
old := d.tablesPID()
|
||||
Expect(old).NotTo(BeZero())
|
||||
workers := batch(16, func() { _ = syscall.Kill(old, syscall.SIGKILL) })
|
||||
Expect(cluster.Alive(old) && d.tablesPID() == old).To(BeFalse(), "the killed process is gone")
|
||||
d.waitTables(old)
|
||||
finish(workers, receiptsBefore, rowsBefore)
|
||||
})
|
||||
|
||||
It("keeps every acknowledged change and applies no change twice when the owner is killed in the middle of a batch and restarted", func() {
|
||||
receiptsBefore, rowsBefore := len(d.receipts(res)), d.rowCount(cluster.Owner, "member", res)
|
||||
owner := d.c.Nodes[cluster.Owner]
|
||||
workers := batch(16, func() { owner.Kill() })
|
||||
// The add-on does not outlive its owner.
|
||||
Eventually(func() int { return d.tablesPID() }, 20*time.Second, 100*time.Millisecond).Should(BeZero())
|
||||
for _, n := range []string{cluster.FrontA, cluster.FrontB} {
|
||||
r := d.read(d.c.Token(n, "member"), res, map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(r.Status).To(Equal(503), n+": "+r.String())
|
||||
}
|
||||
Expect(owner.Start()).To(Succeed())
|
||||
Expect(owner.Ready(120 * time.Second)).To(Succeed())
|
||||
d.waitState("running")
|
||||
finish(workers, receiptsBefore, rowsBefore)
|
||||
})
|
||||
|
||||
It("blocks every other write while a confirmation is unresolved, keeps reading, and accepts writes again only after the owner role resolves it", func() {
|
||||
before := d.rowCount(cluster.Owner, "member", res)
|
||||
receiptsBefore := len(d.receipts(res))
|
||||
op := opID("frozen")
|
||||
p := d.propose(cluster.FrontA, "editor", res, "write", op, map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{map[string]any{"Item": "frozen", "Qty": 1}}})
|
||||
Expect(p.Status).To(Equal(200), p.String())
|
||||
digest, _ := body(p)["digest"].(string)
|
||||
|
||||
// The process stops where it stands: the apply is handed over and
|
||||
// never answered. Then it is killed. The owner cannot know whether
|
||||
// the change ran.
|
||||
pid := d.tablesPID()
|
||||
Expect(syscall.Kill(pid, syscall.SIGSTOP)).To(Succeed())
|
||||
var confirmed *cluster.Response
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer GinkgoRecover()
|
||||
defer close(done)
|
||||
confirmed = d.confirm(cluster.FrontB, "editor", op, digest)
|
||||
}()
|
||||
Eventually(func() string {
|
||||
r, err := d.c.Token(cluster.FrontA, "editor").Do("GET", "/api/addons/proposals/"+op, nil, nil)
|
||||
if err != nil || r.Status != 200 {
|
||||
return ""
|
||||
}
|
||||
s, _ := body(r)["state"].(string)
|
||||
return s
|
||||
}, 30*time.Second, 50*time.Millisecond).Should(Equal("executing"))
|
||||
Expect(syscall.Kill(pid, syscall.SIGKILL)).To(Succeed())
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(90 * time.Second):
|
||||
Fail("the confirmation did not end after the process was killed")
|
||||
}
|
||||
Expect(confirmed.Status).To(Equal(503), confirmed.String())
|
||||
Expect(errType(confirmed)).To(Equal("outcome_unknown"), confirmed.String())
|
||||
Expect(body(confirmed)["error"].(map[string]any)["operationId"]).To(Equal(op))
|
||||
d.waitTables(pid)
|
||||
|
||||
// Reads continue; every other write is blocked, for the owner role
|
||||
// with the operation to resolve and for the others without it.
|
||||
for _, n := range nodes {
|
||||
Expect(d.read(d.c.Token(n, "editor"), res, map[string]any{"tool": "rows-query", "table": "Entries"}).Status).To(Equal(200), n)
|
||||
}
|
||||
ownerWrite := d.apply(cluster.FrontB, "member", res, opID("blk"), "rows-insert", d.revisionOf(res), map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "blocked"}}})
|
||||
Expect(ownerWrite.Status).To(Equal(403), ownerWrite.String())
|
||||
Expect(errType(ownerWrite)).To(Equal("outcome_blocked"))
|
||||
Expect(ownerWrite.String()).To(ContainSubstring(op))
|
||||
other := d.propose(cluster.FrontA, "editor", res, "write", opID("blk"), map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{map[string]any{"Item": "blocked"}}})
|
||||
Expect(other.Status).To(Equal(403), other.String())
|
||||
Expect(other.String()).NotTo(ContainSubstring(op), "the operation ID of another person is not disclosed")
|
||||
|
||||
// The lookup finds no receipt: the process died before it committed.
|
||||
// That is not a proof, so the block stays until the owner resolves it.
|
||||
rec := d.tab(cluster.FrontA, "editor", "POST", "/api/addons/proposals/"+op+"/reconcile", map[string]any{}, op)
|
||||
Expect(rec.Status).To(Equal(200), rec.String())
|
||||
Expect(body(rec)["state"]).To(Equal("outcome-unknown"))
|
||||
Expect(d.apply(cluster.FrontB, "member", res, opID("blk"), "rows-insert", d.revisionOf(res), map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "blocked"}}}).Status).To(Equal(403))
|
||||
listed := d.tab(cluster.Owner, "member", "GET", "/api/addons/resources/"+res+"/proposals?state=unresolved", nil, res)
|
||||
Expect(listed.Status).To(Equal(200), listed.String())
|
||||
var entry map[string]any
|
||||
for _, e := range body(listed)["proposals"].([]any) {
|
||||
if m := e.(map[string]any); m["id"] == op {
|
||||
entry = m
|
||||
}
|
||||
}
|
||||
Expect(entry).NotTo(BeNil(), listed.String())
|
||||
Expect(d.tab(cluster.FrontB, "admin", "POST", "/api/addons/proposals/"+op+"/resolve", map[string]any{"digest": entry["digest"], "reason": "x"}, op).Status).To(Equal(403))
|
||||
ok := d.tab(cluster.FrontB, "member", "POST", "/api/addons/proposals/"+op+"/resolve", map[string]any{"digest": entry["digest"], "reason": "the process died before it committed"}, op)
|
||||
Expect(ok.Status).To(Equal(200), ok.String())
|
||||
Expect(body(ok)["state"]).To(Equal("abandoned-unknown"))
|
||||
// The unresolved change never ran, and writes are accepted again.
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(before))
|
||||
Expect(len(d.receipts(res))).To(Equal(receiptsBefore))
|
||||
after := d.apply(cluster.FrontA, "member", res, opID("after"), "rows-insert", d.revisionOf(res), map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "after"}}})
|
||||
Expect(after.Status).To(Equal(200), after.String())
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("Tables when the add-on is disabled, updated, removed and installed again", Ordered, func() {
|
||||
var (
|
||||
d *deployment
|
||||
res string
|
||||
)
|
||||
|
||||
BeforeAll(func() {
|
||||
d = deploy(start("0.1.0", "0.2.0"), "0.1.0")
|
||||
res = d.seedLedger(cluster.FrontA, "member", "Lifecycle", map[string]any{"Item": "kept", "Qty": 1})
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
Expect(d.grant(cluster.Owner, "member", res, "viewer", "viewer").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("denies the open tabs of a disabled add-on everywhere and serves them again, with the same data, when it is enabled", func() {
|
||||
gen := d.gen
|
||||
tabRev := d.revisionOf(res)
|
||||
Expect(d.lifecycle(cluster.FrontB, "disable", gen, "").Status).To(Equal(200))
|
||||
for _, n := range nodes {
|
||||
w := d.apply(n, "editor", res, opID("dis"), "rows-insert", tabRev, map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "x"}}})
|
||||
Expect(w.Status).To(BeElementOf(403, 503), n+": "+w.String())
|
||||
r := d.read(d.c.Token(n, "viewer"), res, map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(r.Status).To(BeElementOf(403, 503), n+": "+r.String())
|
||||
// A disabled add-on offers the model no tools: the turn is a
|
||||
// plain answer and no tool ran.
|
||||
chat := d.chatTool(n, "member", "rows-query", map[string]any{"resourceId": res, "table": "Entries"})
|
||||
Expect(chat.Status).To(Equal(200), n+": "+chat.String())
|
||||
Expect(chat.String()).To(ContainSubstring("This is a mocked response."))
|
||||
Expect(chat.String()).NotTo(ContainSubstring("kept"), n)
|
||||
}
|
||||
// A member does not see a disabled add-on; the administrator does.
|
||||
for _, n := range nodes {
|
||||
Expect(d.app(d.c.Session(n, "member"))).To(BeNil(), n)
|
||||
Expect(d.app(d.c.Session(n, "admin"))).NotTo(BeNil(), n)
|
||||
}
|
||||
Expect(d.lifecycle(cluster.FrontA, "enable", gen, "").Status).To(Equal(200))
|
||||
d.waitState("running")
|
||||
for _, n := range nodes {
|
||||
Expect(d.rowCount(n, "viewer", res)).To(Equal(1), n)
|
||||
}
|
||||
// Nothing the disabled add-on was asked left a proposal behind.
|
||||
Expect(d.pending(cluster.Owner, "member")).To(BeEmpty())
|
||||
Expect(d.pending(cluster.Owner, "editor")).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("keeps data and grants when the add-on is removed and installed again, and refuses what the old generation prepared", func() {
|
||||
oldGen := d.gen
|
||||
oldRev := d.revisionOf(res)
|
||||
held := d.propose(cluster.FrontA, "editor", res, "write", opID("old-gen"), map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{map[string]any{"Item": "held"}}})
|
||||
Expect(held.Status).To(Equal(200), held.String())
|
||||
heldID, _ := body(held)["id"].(string)
|
||||
heldDigest, _ := body(held)["digest"].(string)
|
||||
|
||||
Expect(d.lifecycle(cluster.FrontB, "disable", oldGen, "").Status).To(Equal(200))
|
||||
Expect(d.lifecycle(cluster.FrontA, "uninstall", 0, "").Status).To(Equal(200))
|
||||
for _, n := range nodes {
|
||||
r := d.read(d.c.Token(n, "viewer"), res, map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(r.Status).To(BeElementOf(403, 503), n+": "+r.String())
|
||||
}
|
||||
Expect(d.lifecycle(cluster.FrontA, "install", 0, "0.1.0").Status).To(Equal(200))
|
||||
cur := d.app(d.c.Token(cluster.FrontA, "admin"))
|
||||
Expect(cur).NotTo(BeNil())
|
||||
Expect(d.lifecycle(cluster.FrontB, "enable", uint64(cur["generation"].(float64)), "").Status).To(Equal(200))
|
||||
d.waitState("running")
|
||||
Expect(d.gen).NotTo(Equal(oldGen), "the new installation is a new generation")
|
||||
|
||||
// Data and grants outlived the removal, for every role and node.
|
||||
for _, n := range nodes {
|
||||
Expect(d.rowCount(n, "viewer", res)).To(Equal(1), n)
|
||||
Expect(d.rowCount(n, "editor", res)).To(Equal(1), n)
|
||||
Expect(d.rowCount(n, "member", res)).To(Equal(1), n)
|
||||
stranger := d.read(d.c.Token(n, "friend"), res, map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
Expect(stranger.Status).To(Equal(403), n)
|
||||
}
|
||||
// The tab and the proposal of the old generation do nothing.
|
||||
stale := d.tab(cluster.FrontB, "editor", "POST", "/api/addons/resources/"+res+"/apply", map[string]any{
|
||||
"operationId": opID("stale-gen"), "generation": oldGen, "revision": oldRev, "tool": "rows-insert", "arguments": map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "old"}}},
|
||||
}, res)
|
||||
Expect(stale.Status).To(Equal(409), stale.String())
|
||||
Expect(errType(stale)).To(Equal("stale"))
|
||||
got := d.confirm(cluster.FrontA, "editor", heldID, heldDigest)
|
||||
Expect(got.Status).To(Equal(403), got.String())
|
||||
Expect(errType(got)).To(Equal("addon_error"))
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(1))
|
||||
// The proposal of the old generation is still listed, unchanged and
|
||||
// never applied: it names the generation it was made for, which no
|
||||
// confirmation can reach any more, and it expires.
|
||||
left := d.pending(cluster.Owner, "editor")
|
||||
Expect(left).To(HaveLen(1))
|
||||
Expect(left[0]["id"]).To(Equal(heldID))
|
||||
Expect(left[0]["generation"]).To(BeNumerically("==", oldGen))
|
||||
again := d.confirm(cluster.FrontB, "editor", heldID, heldDigest)
|
||||
Expect(again.Status).To(Equal(403), again.String())
|
||||
// New work in the new generation succeeds.
|
||||
Expect(d.apply(cluster.FrontB, "editor", res, opID("new-gen"), "rows-insert", d.revisionOf(res), map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "new"}}}).Status).To(Equal(200))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,81 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesacceptance_test
|
||||
|
||||
import (
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// A request that the real add-on refuses as invalid (a table that does not
|
||||
// exist) is a deterministic refusal: the same typed 400 on every route and
|
||||
// every node, never an outage banner and never an unknown outcome, and
|
||||
// nothing is stored or applied.
|
||||
var _ = Describe("Tables refusals of invalid requests across the owner and two frontends", Ordered, func() {
|
||||
var (
|
||||
d *deployment
|
||||
res string
|
||||
)
|
||||
missing := map[string]any{"table": "NoSuchTable", "rows": []any{map[string]any{"Item": "x", "Qty": 1}}}
|
||||
|
||||
BeforeAll(func() {
|
||||
d = deploy(start("0.1.0"), "0.1.0")
|
||||
res = d.seedLedger(cluster.FrontA, "member", "Refusals", map[string]any{"Item": "pen", "Qty": 3})
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("answers the refusal as invalid_request with status 400 on every route and node and changes nothing", func() {
|
||||
receipts := len(d.receipts(res))
|
||||
for _, n := range nodes {
|
||||
read := d.read(d.c.Token(n, "member"), res, map[string]any{"tool": "rows-query", "table": "NoSuchTable"})
|
||||
Expect(read.Status).To(Equal(400), n+" read: "+read.String())
|
||||
Expect(errType(read)).To(Equal("invalid_request"), n)
|
||||
|
||||
relay, err := d.c.Token(n, "member").Do("POST", "/api/addons/relay", map[string]any{"kind": "json", "request": map[string]any{
|
||||
"addonId": "tables", "resourceId": res, "generation": d.gen, "operationId": opID("rel"), "action": "read",
|
||||
"arguments": map[string]any{"tool": "rows-query", "table": "NoSuchTable"}}}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(relay.Status).To(Equal(400), n+" relay: "+relay.String())
|
||||
Expect(errType(relay)).To(Equal("invalid_request"), n)
|
||||
|
||||
args := map[string]any{"tool": "rows-insert", "table": "NoSuchTable", "rows": missing["rows"]}
|
||||
proposed := d.propose(n, "editor", res, "write", opID("refused"), args)
|
||||
Expect(proposed.Status).To(Equal(400), n+" proposal: "+proposed.String())
|
||||
Expect(errType(proposed)).To(Equal("invalid_request"), n)
|
||||
|
||||
applied := d.apply(n, "editor", res, opID("refused-ap"), "rows-insert", d.revisionOf(res), missing)
|
||||
Expect(applied.Status).To(Equal(400), n+" apply: "+applied.String())
|
||||
Expect(errType(applied)).To(Equal("invalid_request"), n)
|
||||
|
||||
// The chat consumer gets it as a tool result and the model says
|
||||
// the request was invalid, never that the add-on is down.
|
||||
chat := d.chatTool(n, "editor", "rows-insert", map[string]any{"resourceId": res, "table": "NoSuchTable", "rows": missing["rows"]})
|
||||
Expect(chat.Status).To(Equal(200), n+" chat: "+chat.String())
|
||||
Expect(chat.String()).To(ContainSubstring("refused the request as invalid"), n)
|
||||
readChat := d.chatTool(n, "editor", "rows-query", map[string]any{"resourceId": res, "table": "NoSuchTable"})
|
||||
Expect(readChat.Status).To(Equal(200), n+" chat read: "+readChat.String())
|
||||
Expect(readChat.String()).To(ContainSubstring("refused the request as invalid"), n)
|
||||
}
|
||||
Expect(d.pending(cluster.Owner, "editor")).To(BeEmpty())
|
||||
Expect(len(d.receipts(res))).To(Equal(receipts))
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(1))
|
||||
})
|
||||
|
||||
It("does not consume the operation ID of a refused change", func() {
|
||||
op := opID("reuse")
|
||||
rev := d.revisionOf(res)
|
||||
refused := d.apply(cluster.FrontB, "editor", res, op, "rows-insert", rev, missing)
|
||||
Expect(refused.Status).To(Equal(400), refused.String())
|
||||
Expect(errType(refused)).To(Equal("invalid_request"))
|
||||
// A refusal is not an unknown outcome: the same ID applies the
|
||||
// corrected request, once.
|
||||
fixed := map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "ink", "Qty": 2}}}
|
||||
ok := d.apply(cluster.FrontA, "editor", res, op, "rows-insert", rev, fixed)
|
||||
Expect(ok.Status).To(Equal(200), ok.String())
|
||||
Expect(count(d.receipts(res), "ui-"+op) + count(d.receipts(res), op)).To(BeNumerically(">=", 0))
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(2))
|
||||
again := d.apply(cluster.Owner, "editor", res, op, "rows-insert", rev, fixed)
|
||||
Expect(again.Status).To(Equal(200), again.String())
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(2), "the repeat replays the receipt")
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,464 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesacceptance_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// export asks for the CSV of a table and waits until the owner holds the file.
|
||||
func (d *deployment) export(node, user, res string) (id string) {
|
||||
GinkgoHelper()
|
||||
r := d.read(d.c.Token(node, user), res, map[string]any{"tool": "rows-export", "table": "Entries"})
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
outs, _ := body(r)["outputs"].([]any)
|
||||
Expect(outs).To(HaveLen(1), r.String())
|
||||
out := outs[0].(map[string]any)
|
||||
id, _ = out["handle"].(string)
|
||||
Expect(out["name"]).To(Equal("Entries.csv"))
|
||||
Eventually(func(g Gomega) string {
|
||||
st, err := d.c.Token(cluster.FrontB, user).Do("GET", "/api/addons/artifacts/"+id, nil, nil)
|
||||
g.Expect(err).NotTo(HaveOccurred())
|
||||
g.Expect(st.Status).To(Equal(200), st.String())
|
||||
s, _ := body(st)["state"].(string)
|
||||
return s
|
||||
}, 30*time.Second, 100*time.Millisecond).Should(Equal("complete"))
|
||||
return id
|
||||
}
|
||||
|
||||
func (d *deployment) download(node, user, id string) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
r, err := d.c.Token(node, user).Do("GET", "/api/addons/artifacts/"+id+"/download", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}
|
||||
|
||||
var _ = Describe("Tables files across the owner and two frontends", Ordered, func() {
|
||||
var (
|
||||
d *deployment
|
||||
res string
|
||||
)
|
||||
|
||||
BeforeAll(func() {
|
||||
d = deploy(start("0.1.0"), "0.1.0")
|
||||
res = d.seedLedger(cluster.FrontA, "member", "Files", map[string]any{"Item": "pen", "Qty": 3}, map[string]any{"Item": "=SUM(A1:A9)", "Qty": 1})
|
||||
Expect(d.grant(cluster.Owner, "member", res, "viewer", "viewer").Status).To(Equal(200))
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("exports a CSV through a frontend and serves the same bytes from every node to its requester only", func() {
|
||||
id := d.export(cluster.FrontA, "member", res)
|
||||
var first []byte
|
||||
for _, n := range nodes {
|
||||
r := d.download(n, "member", id)
|
||||
Expect(r.Status).To(Equal(200), n+": "+r.String())
|
||||
Expect(r.Header.Get("Content-Disposition")).To(ContainSubstring("attachment"))
|
||||
Expect(r.Header.Get("Content-Type")).To(Equal("text/csv"))
|
||||
Expect(r.Header.Get("X-Content-Type-Options")).To(Equal("nosniff"))
|
||||
Expect(r.String()).To(ContainSubstring("pen"))
|
||||
// A formula is neutralised by the add-on, not left to the sheet.
|
||||
Expect(r.String()).To(ContainSubstring("'=SUM(A1:A9)"))
|
||||
if first == nil {
|
||||
first = r.Body
|
||||
}
|
||||
Expect(r.Body).To(Equal(first), n)
|
||||
// The session of the same person works too.
|
||||
s, err := d.c.Session(n, "member").Do("GET", "/api/addons/artifacts/"+id+"/download", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(s.Status).To(Equal(200), n)
|
||||
}
|
||||
// A person with a grant on the database does not get the file of
|
||||
// another person, and an unknown ID looks the same.
|
||||
unknown := d.download(cluster.FrontA, "editor", "a0123456789abcdef0123456789abcde")
|
||||
for _, n := range nodes {
|
||||
for _, who := range []string{"editor", "viewer", "friend", "admin"} {
|
||||
r := d.download(n, who, id)
|
||||
Expect(r.Status).To(Equal(403), who+" on "+n)
|
||||
Expect(r.String()).To(Equal(unknown.String()), who+" on "+n)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
It("stops the downloads of a person whose access to the database was revoked, on every node, and keeps the owner's", func() {
|
||||
id := d.export(cluster.FrontB, "viewer", res)
|
||||
for _, n := range nodes {
|
||||
Expect(d.download(n, "viewer", id).Status).To(Equal(200), n)
|
||||
}
|
||||
mine := d.export(cluster.Owner, "member", res)
|
||||
Expect(d.revoke(cluster.FrontA, "member", res, "viewer").Status).To(Equal(200))
|
||||
for _, n := range nodes {
|
||||
Expect(d.download(n, "viewer", id).Status).To(Equal(403), n)
|
||||
Expect(d.download(n, "member", mine).Status).To(Equal(200), n)
|
||||
}
|
||||
// A viewer who may not read the database cannot start an export.
|
||||
r := d.read(d.c.Token(cluster.FrontB, "viewer"), res, map[string]any{"tool": "rows-export", "table": "Entries"})
|
||||
Expect(r.Status).To(Equal(403), r.String())
|
||||
Expect(d.grant(cluster.Owner, "member", res, "viewer", "viewer").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("denies downloads while the add-on is disabled and serves a complete file again when it is enabled", func() {
|
||||
id := d.export(cluster.FrontA, "member", res)
|
||||
gen := d.gen
|
||||
Expect(d.lifecycle(cluster.FrontB, "disable", gen, "").Status).To(Equal(200))
|
||||
for _, n := range nodes {
|
||||
r := d.download(n, "member", id)
|
||||
Expect(r.Status).To(BeElementOf(403, 503), n+": "+r.String())
|
||||
Expect(r.String()).NotTo(ContainSubstring("pen"))
|
||||
}
|
||||
Expect(d.lifecycle(cluster.FrontA, "enable", gen, "").Status).To(Equal(200))
|
||||
d.waitState("running")
|
||||
for _, n := range nodes {
|
||||
Expect(d.download(n, "member", id).Status).To(Equal(200), n)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("Tables when the owner is down", Ordered, func() {
|
||||
It("fails every response type closed with the same typed answer from both frontends, runs nothing there and serves everything again after the restart", func() {
|
||||
d := deploy(start("0.1.0"), "0.1.0")
|
||||
res := d.seedLedger(cluster.FrontA, "member", "Outage", map[string]any{"Item": "pen", "Qty": 3})
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
file := d.export(cluster.FrontB, "member", res)
|
||||
app := d.app(d.c.Session(cluster.FrontA, "member"))
|
||||
entry := app["ui"].(map[string]any)["entry"].(string)
|
||||
held := d.propose(cluster.FrontA, "editor", res, "write", opID("held"), map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{map[string]any{"Item": "held"}}})
|
||||
Expect(held.Status).To(Equal(200), held.String())
|
||||
heldID, _ := body(held)["id"].(string)
|
||||
heldDigest, _ := body(held)["digest"].(string)
|
||||
statusOp := opID("stat")
|
||||
Expect(d.dispatch(d.c.Token(cluster.FrontA, "admin"), map[string]any{"operationId": statusOp, "action": "enable", "generation": d.gen}).Status).To(BeNumerically("<", 500))
|
||||
rows := d.rowCount(cluster.Owner, "member", res)
|
||||
receipts := len(d.receipts(res))
|
||||
file2 := d.databaseFile(res)
|
||||
before, err := os.Stat(file2)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
|
||||
owner := d.c.Nodes[cluster.Owner]
|
||||
ownerPID := owner.PID()
|
||||
owner.Kill()
|
||||
Eventually(func() int { return len(cluster.ChildrenOf(ownerPID)) }, 20*time.Second, 100*time.Millisecond).Should(BeZero())
|
||||
|
||||
type call struct {
|
||||
name string
|
||||
do func(n string) *cluster.Response
|
||||
}
|
||||
relay := func(kind string) func(n string) *cluster.Response {
|
||||
return func(n string) *cluster.Response {
|
||||
r, err := d.c.Token(n, "admin").Do("POST", "/api/addons/relay", map[string]any{"kind": kind, "request": map[string]any{
|
||||
"addonId": "tables", "generation": d.gen, "operationId": statusOp, "action": "operation-status", "arguments": map[string]any{}}}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}
|
||||
}
|
||||
proposalOp, applyOp := opID("down-p"), opID("down-a")
|
||||
calls := []call{
|
||||
{"json read", func(n string) *cluster.Response {
|
||||
return d.read(d.c.Token(n, "member"), res, map[string]any{"tool": "rows-query", "table": "Entries"})
|
||||
}},
|
||||
{"json relay", func(n string) *cluster.Response {
|
||||
r, err := d.c.Token(n, "member").Do("POST", "/api/addons/relay", map[string]any{"kind": "json", "request": map[string]any{
|
||||
"addonId": "tables", "resourceId": res, "generation": d.gen, "operationId": opID("rel"), "action": "read", "arguments": map[string]any{"tool": "rows-query", "table": "Entries"}}}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}},
|
||||
{"progress", relay("progress")},
|
||||
{"status", relay("status")},
|
||||
{"file status", func(n string) *cluster.Response {
|
||||
r, err := d.c.Token(n, "member").Do("GET", "/api/addons/artifacts/"+file, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}},
|
||||
{"file download", func(n string) *cluster.Response { return d.download(n, "member", file) }},
|
||||
{"asset", func(n string) *cluster.Response {
|
||||
r, err := d.c.Session(n, "member").Do("GET", entry, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}},
|
||||
{"apps", func(n string) *cluster.Response {
|
||||
r, err := d.c.Session(n, "member").Do("GET", "/api/addons/apps", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}},
|
||||
{"proposal", func(n string) *cluster.Response {
|
||||
return d.propose(n, "editor", res, "write", proposalOp, map[string]any{"tool": "rows-insert", "table": "Entries", "rows": []any{map[string]any{"Item": "x"}}})
|
||||
}},
|
||||
{"proposal list", func(n string) *cluster.Response {
|
||||
r, err := d.c.Token(n, "editor").Do("GET", "/api/addons/proposals?state=pending", nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}},
|
||||
{"confirmation", func(n string) *cluster.Response { return d.confirm(n, "editor", heldID, heldDigest) }},
|
||||
{"apply", func(n string) *cluster.Response {
|
||||
return d.apply(n, "member", res, applyOp, "rows-insert", map[string]any{"schema": 1, "data": 0}, map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "x"}}})
|
||||
}},
|
||||
{"chat tool", func(n string) *cluster.Response {
|
||||
return d.chatTool(n, "member", "rows-insert", map[string]any{"resourceId": res, "table": "Entries", "rows": []any{map[string]any{"Item": "x"}}})
|
||||
}},
|
||||
}
|
||||
for _, c := range calls {
|
||||
var bodies []string
|
||||
for _, n := range []string{cluster.FrontA, cluster.FrontB} {
|
||||
r := c.do(n)
|
||||
// The chat route reports the add-on tools as unavailable with
|
||||
// its own envelope, the same from both frontends: it never
|
||||
// serves a model turn that could invent an answer.
|
||||
Expect(r.Status).To(Equal(503), c.name+" on "+n+": "+r.String())
|
||||
Expect(r.String()).NotTo(ContainSubstring("pen"), c.name)
|
||||
bodies = append(bodies, r.String())
|
||||
}
|
||||
if len(bodies) == 2 {
|
||||
Expect(bodies[1]).To(Equal(bodies[0]), c.name+": the same typed answer from every frontend")
|
||||
}
|
||||
}
|
||||
for _, n := range []string{cluster.FrontA, cluster.FrontB} {
|
||||
Expect(cluster.ChildrenOf(d.c.Nodes[n].PID())).To(BeEmpty(), n+" runs nothing for the add-on")
|
||||
Expect(d.c.Config.Nodes[n].Root).NotTo(BeADirectory(), n+" keeps no add-on storage")
|
||||
}
|
||||
after, err := os.Stat(file2)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(after.ModTime()).To(Equal(before.ModTime()), "no database was touched while the owner was down")
|
||||
|
||||
Expect(owner.Start()).To(Succeed())
|
||||
Expect(owner.Ready(120 * time.Second)).To(Succeed())
|
||||
d.waitState("running")
|
||||
for _, n := range nodes {
|
||||
Expect(d.rowCount(n, "member", res)).To(Equal(rows), n)
|
||||
Expect(d.download(n, "member", file).Status).To(Equal(200), n)
|
||||
asset, err := d.c.Session(n, "member").Do("GET", entry, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(asset.Status).To(Equal(200), n)
|
||||
}
|
||||
Expect(len(d.receipts(res))).To(Equal(receipts), "nothing was applied by a failed call")
|
||||
// No call made while the owner was down stored a proposal: the editor
|
||||
// holds exactly the one it held before, whatever route the call took.
|
||||
pending := d.pending(cluster.Owner, "editor")
|
||||
Expect(pending).To(HaveLen(1))
|
||||
Expect(pending[0]["id"]).To(Equal(heldID))
|
||||
Expect(d.pending(cluster.Owner, "member")).To(BeEmpty())
|
||||
// The held proposal is still the editor's, and applies once.
|
||||
ok := d.confirm(cluster.FrontB, "editor", heldID, heldDigest)
|
||||
Expect(ok.Status).To(Equal(200), ok.String())
|
||||
Expect(body(ok)["state"]).To(Equal("applied"))
|
||||
Expect(d.rowCount(cluster.FrontA, "member", res)).To(Equal(rows + 1))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("Tables backup and restore through the administrator's tab", Ordered, func() {
|
||||
var (
|
||||
d *deployment
|
||||
res string
|
||||
)
|
||||
|
||||
BeforeAll(func() {
|
||||
d = deploy(start("0.1.0"), "0.1.0")
|
||||
res = d.seedLedger(cluster.FrontA, "member", "Restore", map[string]any{"Item": "one", "Qty": 1})
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
recovery := func(node, user, verb string, payload map[string]any) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
op := opID(verb)
|
||||
payload["operationId"] = op
|
||||
return d.tab(node, user, "POST", "/api/addons/tables/recovery/"+verb, payload, op)
|
||||
}
|
||||
backupID := func(r *cluster.Response) string {
|
||||
GinkgoHelper()
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
id, _ := body(r)["id"].(string)
|
||||
Expect(id).NotTo(BeEmpty())
|
||||
return id
|
||||
}
|
||||
stop := func(node string) {
|
||||
GinkgoHelper()
|
||||
op := opID("stop")
|
||||
r := d.tab(node, "admin", "POST", "/api/addons/apps/tables/stop", map[string]any{"operationId": op, "generation": d.gen}, op)
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
d.waitState("stopped")
|
||||
}
|
||||
startApp := func(node string) {
|
||||
GinkgoHelper()
|
||||
op := opID("start")
|
||||
r := d.tab(node, "admin", "POST", "/api/addons/apps/tables/start", map[string]any{"operationId": op, "generation": d.gen}, op)
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
d.waitState("running")
|
||||
}
|
||||
|
||||
It("is the administrator's own action on every node and never a key's, a member's or a tool's", func() {
|
||||
for _, n := range nodes {
|
||||
for _, user := range []string{"member", "editor"} {
|
||||
r := recovery(n, user, "backup", map[string]any{})
|
||||
Expect(r.Status).To(Equal(403), user+" on "+n+": "+r.String())
|
||||
}
|
||||
keyed, err := d.c.Token(n, "admin").Do("POST", "/api/addons/tables/recovery/backup", map[string]any{"operationId": opID("key")}, map[string]string{})
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(keyed.Status).To(Equal(403), n+": "+keyed.String())
|
||||
}
|
||||
})
|
||||
|
||||
It("refuses a corrupt backup and a restore that is not allowed, and leaves the data as it was", func() {
|
||||
good := backupID(recovery(cluster.FrontB, "admin", "backup", map[string]any{}))
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(1))
|
||||
// Data changes after the backup.
|
||||
d.applyOK(cluster.FrontA, "member", res, "rows-insert", map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "two", "Qty": 2}}})
|
||||
bad := backupID(recovery(cluster.FrontA, "admin", "backup", map[string]any{}))
|
||||
|
||||
// One byte of one file of the second backup is changed.
|
||||
dir := filepath.Join(d.c.Config.Nodes[cluster.Owner].Root, "backups", "tables", bad)
|
||||
var target string
|
||||
Expect(filepath.Walk(dir, func(p string, info os.FileInfo, e error) error {
|
||||
if e == nil && !info.IsDir() && filepath.Ext(p) == ".db" && target == "" {
|
||||
target = p
|
||||
}
|
||||
return e
|
||||
})).To(Succeed())
|
||||
Expect(target).NotTo(BeEmpty(), "the backup holds a database file")
|
||||
raw, err := os.ReadFile(target)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
raw[len(raw)/2] ^= 0xff
|
||||
Expect(os.Chmod(target, 0o600)).To(Succeed())
|
||||
Expect(os.WriteFile(target, raw, 0o600)).To(Succeed())
|
||||
|
||||
v := recovery(cluster.FrontB, "admin", "verify", map[string]any{"backupId": bad})
|
||||
Expect(errType(v)).To(Equal("backup_corrupt"), v.String())
|
||||
Expect(v.Status).NotTo(Equal(200))
|
||||
Expect(recovery(cluster.Owner, "admin", "verify", map[string]any{"backupId": good}).Status).To(Equal(200))
|
||||
|
||||
// Refusals before anything is touched.
|
||||
running := recovery(cluster.FrontA, "admin", "restore", map[string]any{"backupId": good, "confirm": "tables"})
|
||||
Expect(errType(running)).To(Equal("addon_running"), running.String())
|
||||
stop(cluster.FrontB)
|
||||
wrongConfirm := recovery(cluster.FrontB, "admin", "restore", map[string]any{"backupId": good, "confirm": "other"})
|
||||
Expect(errType(wrongConfirm)).To(Equal("confirmation_mismatch"), wrongConfirm.String())
|
||||
missing := recovery(cluster.FrontB, "admin", "restore", map[string]any{"backupId": "b20000101t000000z-00000000", "confirm": "tables"})
|
||||
Expect(errType(missing)).To(Equal("backup_not_found"), missing.String())
|
||||
corrupt := recovery(cluster.FrontA, "admin", "restore", map[string]any{"backupId": bad, "confirm": "tables"})
|
||||
Expect(errType(corrupt)).To(Equal("backup_corrupt"), corrupt.String())
|
||||
startApp(cluster.FrontA)
|
||||
Expect(d.rowCount(cluster.FrontB, "member", res)).To(Equal(2), "a refused restore changed nothing")
|
||||
})
|
||||
|
||||
It("restores a verified backup: the data goes back, the add-on starts again and access follows the current accounts", func() {
|
||||
good := ""
|
||||
list := d.tab(cluster.FrontA, "admin", "GET", "/api/addons/tables/recovery", nil, "")
|
||||
Expect(list.Status).To(Equal(200), list.String())
|
||||
for _, b := range body(list)["backups"].([]any) {
|
||||
m := b.(map[string]any)
|
||||
if m["verification"] == "verified" && m["verified"] == true {
|
||||
good = m["id"].(string)
|
||||
break
|
||||
}
|
||||
}
|
||||
Expect(good).NotTo(BeEmpty(), list.String())
|
||||
stop(cluster.Owner)
|
||||
r := recovery(cluster.FrontB, "admin", "restore", map[string]any{"backupId": good, "confirm": "tables"})
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
startApp(cluster.FrontA)
|
||||
for _, n := range nodes {
|
||||
Expect(d.rowCount(n, "member", res)).To(BeNumerically(">=", 1), n)
|
||||
}
|
||||
// The restored data is the data of the backup: the second row was
|
||||
// made after it and is gone, with the previous copy kept aside.
|
||||
Expect(d.rowCount(cluster.Owner, "member", res)).To(Equal(1))
|
||||
for _, n := range nodes {
|
||||
Expect(d.read(d.c.Token(n, "friend"), res, map[string]any{"tool": "rows-query", "table": "Entries"}).Status).To(Equal(403), n)
|
||||
}
|
||||
Expect(d.apply(cluster.FrontA, "member", res, opID("post-restore"), "rows-insert", d.revisionOf(res), map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "three"}}}).Status).To(Equal(200))
|
||||
_ = fmt.Sprint
|
||||
_ = http.StatusOK
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("Tables upgrade with an explicit check between two packaged versions", Ordered, func() {
|
||||
var (
|
||||
d *deployment
|
||||
res string
|
||||
)
|
||||
|
||||
BeforeAll(func() {
|
||||
d = deploy(start("0.1.0", "0.2.0"), "0.1.0")
|
||||
res = d.seedLedger(cluster.FrontA, "member", "Upgrade", map[string]any{"Item": "kept", "Qty": 1})
|
||||
Expect(d.grant(cluster.Owner, "member", res, "editor", "editor").Status).To(Equal(200))
|
||||
})
|
||||
|
||||
It("reports an update only on a check, shows a member no digest, refuses a key and applies one pinned candidate with the data and grants kept", func() {
|
||||
check := func(node, user string) *cluster.Response {
|
||||
op := opID("chk")
|
||||
return d.tab(node, user, "POST", "/api/addons/tables/update/check", map[string]any{"operationId": op}, op)
|
||||
}
|
||||
r := check(cluster.FrontB, "admin")
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
st := body(r)
|
||||
Expect(st["state"]).To(Equal("update-available"))
|
||||
Expect(st["installedVersion"]).To(Equal("0.1.0"))
|
||||
Expect(st["candidateVersion"]).To(Equal("0.2.0"))
|
||||
installed, _ := st["installedDigest"].(string)
|
||||
candidate, _ := st["candidateDigest"].(string)
|
||||
Expect(installed).To(Equal(packages["0.1.0"].ArchiveDigest))
|
||||
Expect(candidate).To(Equal(packages["0.2.0"].ArchiveDigest))
|
||||
// The same answer from every node, and nothing about the library or
|
||||
// the digests for a member.
|
||||
for _, n := range nodes {
|
||||
m := d.tab(n, "member", "GET", "/api/addons/tables/update", nil, "")
|
||||
Expect(m.Status).To(Equal(200), n)
|
||||
Expect(body(m)["state"]).To(Equal("update-available"))
|
||||
Expect(m.String()).NotTo(ContainSubstring(candidate))
|
||||
Expect(m.String()).NotTo(ContainSubstring("sourceId"))
|
||||
keyed, err := d.c.Token(n, "admin").Do("POST", "/api/addons/tables/update/check", map[string]any{"operationId": opID("k")}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(keyed.Status).To(Equal(403), n)
|
||||
asMember := check(n, "member")
|
||||
Expect(asMember.Status).To(Equal(403), n)
|
||||
}
|
||||
|
||||
genBefore := d.gen
|
||||
upgrade := func(node, op, expected, cand string) *cluster.Response {
|
||||
return d.tab(node, "admin", "POST", "/api/addons/tables/upgrade", map[string]any{
|
||||
"operationId": op, "expectedDigest": expected, "candidateDigest": cand, "generation": genBefore}, op)
|
||||
}
|
||||
stale := upgrade(cluster.FrontA, opID("up"), "sha256:"+fmt.Sprintf("%064d", 0), candidate)
|
||||
Expect(stale.Status).To(Equal(409), stale.String())
|
||||
Expect(errType(stale)).To(Equal("stale_installed"))
|
||||
changed := upgrade(cluster.FrontA, opID("up"), installed, "sha256:"+fmt.Sprintf("%064d", 1))
|
||||
Expect(changed.Status).To(Equal(409), changed.String())
|
||||
Expect(errType(changed)).To(Equal("candidate_changed"))
|
||||
Expect(d.app(d.c.Token(cluster.Owner, "admin"))["version"]).To(Equal("0.1.0"), "a refused pin changed nothing")
|
||||
|
||||
before := d.rowCount(cluster.Owner, "member", res)
|
||||
op := opID("up")
|
||||
done := upgrade(cluster.FrontB, op, installed, candidate)
|
||||
Expect(done.Status).To(Equal(200), done.String())
|
||||
d.waitState("running")
|
||||
for _, n := range nodes {
|
||||
a := d.app(d.c.Token(n, "admin"))
|
||||
Expect(a["version"]).To(Equal("0.2.0"), n)
|
||||
}
|
||||
// The same operation from another node is the recorded result.
|
||||
again := upgrade(cluster.FrontA, op, installed, candidate)
|
||||
Expect(again.Status).To(Equal(200), again.String())
|
||||
// Data, grants and the receipts survived, and the new generation works.
|
||||
for _, n := range nodes {
|
||||
Expect(d.rowCount(n, "member", res)).To(Equal(before), n)
|
||||
Expect(d.rowCount(n, "editor", res)).To(Equal(before), n)
|
||||
Expect(d.read(d.c.Token(n, "friend"), res, map[string]any{"tool": "rows-query", "table": "Entries"}).Status).To(Equal(403), n)
|
||||
}
|
||||
Expect(d.apply(cluster.FrontB, "editor", res, opID("post-up"), "rows-insert", d.revisionOf(res), map[string]any{"table": "Entries", "rows": []any{map[string]any{"Item": "new"}}}).Status).To(Equal(200))
|
||||
// A backup was made before the data was touched.
|
||||
rec := d.tab(cluster.FrontA, "admin", "GET", "/api/addons/tables/recovery", nil, "")
|
||||
Expect(rec.Status).To(Equal(200), rec.String())
|
||||
reasons := ""
|
||||
for _, b := range body(rec)["backups"].([]any) {
|
||||
reasons += b.(map[string]any)["reason"].(string) + " "
|
||||
}
|
||||
Expect(reasons).To(ContainSubstring("upgrade"))
|
||||
// And the check now says there is nothing newer.
|
||||
final := check(cluster.FrontA, "admin")
|
||||
Expect(body(final)["state"]).To(Equal("up-to-date"), final.String())
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,287 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
// Package tablesacceptance_test is the end-to-end acceptance of the Tables
|
||||
// add-on. It starts the foundation cluster (one owner and two forwarding
|
||||
// frontends as separate server processes over the real mTLS carrier, one
|
||||
// shared account store) and installs the real packaged Tables from a local
|
||||
// catalog: the package is built by scripts/addons/package.sh from
|
||||
// addons/tables, and its layers are the files the installer unpacks. Tests
|
||||
// use real named sessions and API keys, the public HTTP routes only and the
|
||||
// real SQLite databases of the add-on. Nothing is published anywhere.
|
||||
package tablesacceptance_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
"github.com/mudler/LocalAI/tests/addons/tablesacceptance/tablespkg"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
"github.com/onsi/ginkgo/v2/types"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestTablesAcceptance(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Tables acceptance")
|
||||
}
|
||||
|
||||
var (
|
||||
repoRoot string
|
||||
paths cluster.Paths
|
||||
packages = map[string]tablespkg.Release{}
|
||||
// commands records every command the suite itself ran, for the manifest.
|
||||
commandsMu sync.Mutex
|
||||
commands [][]string
|
||||
// traces maps a failed spec to the retained log file of the cluster.
|
||||
tracesMu sync.Mutex
|
||||
traces = map[string]string{}
|
||||
)
|
||||
|
||||
func record(cmd ...string) {
|
||||
commandsMu.Lock()
|
||||
defer commandsMu.Unlock()
|
||||
commands = append(commands, cmd)
|
||||
}
|
||||
|
||||
func run(dir string, env []string, name string, args ...string) {
|
||||
GinkgoHelper()
|
||||
record(append([]string{name}, args...)...)
|
||||
cmd := exec.Command(name, args...)
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(os.Environ(), env...)
|
||||
b, err := cmd.CombinedOutput()
|
||||
Expect(err).NotTo(HaveOccurred(), string(b))
|
||||
}
|
||||
|
||||
var _ = BeforeSuite(func() {
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
repoRoot = filepath.Clean(filepath.Join(filepath.Dir(file), "..", "..", ".."))
|
||||
_, err := evidenceLocation(os.Getenv("TABLES_ACCEPTANCE_EVIDENCE"), repoRoot)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
scratch, err := os.MkdirTemp("", "tables-acceptance-bin-*")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() {
|
||||
_ = filepath.WalkDir(scratch, func(p string, d fs.DirEntry, e error) error {
|
||||
if e == nil && d.IsDir() {
|
||||
_ = os.Chmod(p, 0o700)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
_ = os.RemoveAll(scratch)
|
||||
})
|
||||
build := func(env, out, pkgPath string, extra ...string) string {
|
||||
if p := os.Getenv(env); p != "" {
|
||||
return p
|
||||
}
|
||||
target := filepath.Join(scratch, out)
|
||||
args := append([]string{"build", "-p", "2", "-tags", "auth"}, extra...)
|
||||
args = append(args, "-o", target, pkgPath)
|
||||
run(repoRoot, []string{"CGO_ENABLED=1"}, "go", args...)
|
||||
return target
|
||||
}
|
||||
paths.Server = build("TABLES_ACCEPTANCE_SERVER", "ui-test-server", "./tests/e2e-ui")
|
||||
paths.Mock = os.Getenv("TABLES_ACCEPTANCE_MOCK")
|
||||
if paths.Mock == "" {
|
||||
paths.Mock = filepath.Join(repoRoot, "tests/e2e/mock-backend/mock-backend")
|
||||
}
|
||||
Expect(paths.Mock).To(BeAnExistingFile(), "build tests/e2e/mock-backend first")
|
||||
exe := os.Getenv("TABLES_ACCEPTANCE_TABLES")
|
||||
if exe == "" {
|
||||
exe = filepath.Join(scratch, "tables")
|
||||
Expect(tablespkg.BuildExecutable(repoRoot, exe, func(argv ...string) { record(argv...) })).To(Succeed())
|
||||
}
|
||||
// Two releases of the same platform: the second one is what the explicit
|
||||
// upgrade installs. Both are built by the real packaging entry point.
|
||||
built, err := tablespkg.Build(repoRoot, scratch, exe, func(argv ...string) { record(argv...) }, "0.1.0", "0.2.0")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
for v, r := range built {
|
||||
Expect(r.ImageDigest).To(MatchRegexp(`^sha256:[0-9a-f]{64}$`))
|
||||
packages[v] = r
|
||||
}
|
||||
})
|
||||
|
||||
// start brings a cluster up in a fresh directory and takes it down at the end
|
||||
// of the container. A failed spec keeps the logs of every node as a trace.
|
||||
func start(versions ...string) *cluster.Cluster {
|
||||
GinkgoHelper()
|
||||
return startFrom(nil, packages, versions...)
|
||||
}
|
||||
|
||||
// startFrom is start with releases of another build and a change of the
|
||||
// cluster configuration that is saved before any process starts.
|
||||
func startFrom(tune func(*cluster.Config), releases map[string]tablespkg.Release, versions ...string) *cluster.Cluster {
|
||||
GinkgoHelper()
|
||||
dir, err := os.MkdirTemp("", "tables-acceptance-*")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
c, err := cluster.New(dir, paths)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
if tune != nil {
|
||||
tune(c.Config)
|
||||
Expect(c.Config.Save()).To(Succeed())
|
||||
}
|
||||
for _, v := range versions {
|
||||
_, err := c.Library.AddArchive("tables", v, releases[v].Archive)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
}
|
||||
DeferCleanup(func() {
|
||||
retainTrace(c)
|
||||
c.Down()
|
||||
_ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, e error) error {
|
||||
if e == nil && d.IsDir() {
|
||||
_ = os.Chmod(p, 0o700)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
_ = os.RemoveAll(dir)
|
||||
})
|
||||
Expect(c.Up()).To(Succeed())
|
||||
return c
|
||||
}
|
||||
|
||||
// retainTrace copies the node logs of a failed spec to the log directory of
|
||||
// the run, so the evidence of a failure outlives the scratch directory.
|
||||
func retainTrace(c *cluster.Cluster) {
|
||||
if !CurrentSpecReport().Failed() {
|
||||
return
|
||||
}
|
||||
out := filepath.Join(repoRoot, ".cache/addons/logs/tables-acceptance-traces")
|
||||
if err := os.MkdirAll(out, 0o755); err != nil {
|
||||
return
|
||||
}
|
||||
name := sanitize(CurrentSpecReport().LeafNodeText) + ".log"
|
||||
var b strings.Builder
|
||||
for _, n := range cluster.Instances {
|
||||
fmt.Fprintf(&b, "==== %s ====\n%s\n", n, c.Nodes[n].Tail(200))
|
||||
}
|
||||
path := filepath.Join(out, name)
|
||||
if os.WriteFile(path, []byte(b.String()), 0o644) == nil {
|
||||
tracesMu.Lock()
|
||||
traces[CurrentSpecReport().LeafNodeText] = path
|
||||
tracesMu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
func sanitize(s string) string {
|
||||
var b strings.Builder
|
||||
for _, r := range strings.ToLower(s) {
|
||||
if r >= 'a' && r <= 'z' || r >= '0' && r <= '9' {
|
||||
b.WriteRune(r)
|
||||
} else {
|
||||
b.WriteByte('-')
|
||||
}
|
||||
}
|
||||
out := b.String()
|
||||
if len(out) > 80 {
|
||||
out = out[:80]
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ---- evidence manifest ----
|
||||
|
||||
// evidencePath is where the suite writes the acceptance evidence.
|
||||
func evidencePath() string {
|
||||
p, err := evidenceLocation(os.Getenv("TABLES_ACCEPTANCE_EVIDENCE"), repoRoot)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// evidenceLocation resolves where the manifest goes. A relative path would
|
||||
// resolve against the package directory of the test binary and leave the
|
||||
// manifest inside the tree, so it is refused.
|
||||
func evidenceLocation(env, root string) (string, error) {
|
||||
if env == "" {
|
||||
return filepath.Join(root, ".cache/addons/tables-acceptance-evidence.json"), nil
|
||||
}
|
||||
if !filepath.IsAbs(env) {
|
||||
return "", fmt.Errorf("TABLES_ACCEPTANCE_EVIDENCE must be an absolute path, got %q", env)
|
||||
}
|
||||
return env, nil
|
||||
}
|
||||
|
||||
type evidenceCase struct {
|
||||
Name string `json:"name"`
|
||||
State string `json:"state"`
|
||||
Millis int64 `json:"millis"`
|
||||
Trace string `json:"trace,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type evidence struct {
|
||||
Schema int `json:"schema"`
|
||||
Suite string `json:"suite"`
|
||||
Written string `json:"written"`
|
||||
Platform platform `json:"platform"`
|
||||
Commands [][]string `json:"commands"`
|
||||
Artifact artifact `json:"artifact"`
|
||||
Cases []evidenceCase `json:"cases"`
|
||||
// Browser holds the Playwright cases; the runner adds them after the
|
||||
// browser run, from its report.
|
||||
Browser []evidenceCase `json:"browser,omitempty"`
|
||||
Publication string `json:"publication"`
|
||||
}
|
||||
|
||||
type platform struct {
|
||||
OS string `json:"os"`
|
||||
Arch string `json:"arch"`
|
||||
Native bool `json:"native"`
|
||||
}
|
||||
|
||||
type artifact struct {
|
||||
Addon string `json:"addon"`
|
||||
Versions map[string]digest `json:"versions"`
|
||||
}
|
||||
|
||||
type digest struct {
|
||||
Archive string `json:"archive"`
|
||||
Image string `json:"image"`
|
||||
}
|
||||
|
||||
var _ = ReportAfterSuite("write the acceptance evidence manifest", func(report Report) {
|
||||
ev := evidence{
|
||||
Schema: 1, Suite: "tables-acceptance", Written: time.Now().UTC().Format(time.RFC3339),
|
||||
Platform: platform{OS: runtime.GOOS, Arch: runtime.GOARCH, Native: true},
|
||||
Artifact: artifact{Addon: "tables", Versions: map[string]digest{}},
|
||||
Publication: "none: the packages are built and installed locally and nothing is pushed, tagged or signed",
|
||||
}
|
||||
commandsMu.Lock()
|
||||
ev.Commands = append(ev.Commands, commands...)
|
||||
commandsMu.Unlock()
|
||||
ev.Commands = append(ev.Commands, []string{"go", "test", "-count=1", "-tags", "auth", "./tests/addons/tablesacceptance"})
|
||||
for v, p := range packages {
|
||||
ev.Artifact.Versions[v] = digest{Archive: p.ArchiveDigest, Image: p.ImageDigest}
|
||||
}
|
||||
tracesMu.Lock()
|
||||
defer tracesMu.Unlock()
|
||||
for _, s := range report.SpecReports {
|
||||
if s.LeafNodeType != types.NodeTypeIt {
|
||||
continue
|
||||
}
|
||||
c := evidenceCase{Name: s.LeafNodeText, State: s.State.String(), Millis: s.RunTime.Milliseconds(), Trace: traces[s.LeafNodeText]}
|
||||
if s.Failed() {
|
||||
c.Error = s.Failure.Message
|
||||
}
|
||||
ev.Cases = append(ev.Cases, c)
|
||||
}
|
||||
b, err := json.MarshalIndent(ev, "", " ")
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
_ = os.MkdirAll(filepath.Dir(evidencePath()), 0o755)
|
||||
tmp := evidencePath() + ".tmp"
|
||||
if os.WriteFile(tmp, b, 0o644) == nil {
|
||||
_ = os.Rename(tmp, evidencePath())
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,156 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
// Package tablespkg builds the installable Tables packages of the acceptance
|
||||
// suites with the real packaging entry point. It runs scripts/addons/package.sh
|
||||
// to make the OCI layout of one platform, checks every layer against its
|
||||
// digest and joins the layers into the plain archive a local catalog serves.
|
||||
// Nothing is signed, pushed or published.
|
||||
package tablespkg
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Release is one packaged version of Tables.
|
||||
type Release struct {
|
||||
Version string
|
||||
// Archive is the plain tar the catalog serves: the layers of the layout
|
||||
// of package.sh in one archive.
|
||||
Archive []byte
|
||||
// ArchiveDigest is the catalog digest of Archive.
|
||||
ArchiveDigest string
|
||||
// ImageDigest is the digest of the platform image in the layout.
|
||||
ImageDigest string
|
||||
}
|
||||
|
||||
// Recorder receives every command the builder runs.
|
||||
type Recorder func(argv ...string)
|
||||
|
||||
func run(root string, rec Recorder, env []string, name string, args ...string) error {
|
||||
if rec != nil {
|
||||
rec(append([]string{name}, args...)...)
|
||||
}
|
||||
cmd := exec.Command(name, args...) // #nosec G204 -- test harness starts the binaries it built
|
||||
cmd.Dir = root
|
||||
cmd.Env = append(cmd.Environ(), env...)
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s %s: %w\n%s", name, strings.Join(args, " "), err, out)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// BuildExecutable builds the Tables executable of this platform.
|
||||
func BuildExecutable(root, out string, rec Recorder) error {
|
||||
return BuildExecutableFrom(root, out, "./addons/tables/cmd/tables", rec)
|
||||
}
|
||||
|
||||
// BuildExecutableFrom builds another main package of the Tables module. The
|
||||
// fault suites use it for the test-only executable; nothing else does.
|
||||
func BuildExecutableFrom(root, out, pkg string, rec Recorder) error {
|
||||
return run(root, rec, []string{"CGO_ENABLED=1"}, "go", "build", "-p", "2", "-trimpath", "-o", out, pkg)
|
||||
}
|
||||
|
||||
// Build packages one executable under each version and returns the releases.
|
||||
// scratch must be an empty directory the caller removes.
|
||||
func Build(root, scratch, executable string, rec Recorder, versions ...string) (map[string]Release, error) {
|
||||
if _, err := os.Stat(filepath.Join(root, "addons/tables/ui/dist/ui/tables.js")); err != nil {
|
||||
if err := run(root, rec, nil, "node", "addons/tables/ui/build.mjs"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
out := map[string]Release{}
|
||||
for _, version := range versions {
|
||||
layout := filepath.Join(scratch, "layout-"+version)
|
||||
if err := run(root, rec, nil, "sh", "scripts/addons/package.sh", "--layout", layout, "--version", version,
|
||||
"--platform", runtime.GOOS+"/"+runtime.GOARCH, "--executable", executable, "--local-check"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
archive, image, err := mergeLayers(layout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
sum := sha256.Sum256(archive)
|
||||
out[version] = Release{Version: version, Archive: archive, ArchiveDigest: "sha256:" + hex.EncodeToString(sum[:]), ImageDigest: image}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// mergeLayers turns the layers of an OCI layout into one plain tar, in the
|
||||
// order the receipt of package.sh lists them.
|
||||
func mergeLayers(layout string) ([]byte, string, error) {
|
||||
raw, err := os.ReadFile(filepath.Join(layout, "receipt.json")) // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
var receipt struct {
|
||||
ManifestDigest string `json:"manifestDigest"`
|
||||
Publishable bool `json:"publishable"`
|
||||
Layers []struct {
|
||||
Digest string `json:"digest"`
|
||||
} `json:"layers"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &receipt); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if receipt.Publishable {
|
||||
return nil, "", fmt.Errorf("a local acceptance package must not be publishable")
|
||||
}
|
||||
var out bytes.Buffer
|
||||
tw := tar.NewWriter(&out)
|
||||
seen := map[string]bool{}
|
||||
for _, l := range receipt.Layers {
|
||||
hexDigest := strings.TrimPrefix(l.Digest, "sha256:")
|
||||
blob, err := os.ReadFile(filepath.Join(layout, "blobs", "sha256", hexDigest)) // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
sum := sha256.Sum256(blob)
|
||||
if hex.EncodeToString(sum[:]) != hexDigest {
|
||||
return nil, "", fmt.Errorf("layer %s does not match its digest", l.Digest)
|
||||
}
|
||||
zr, err := gzip.NewReader(bytes.NewReader(blob))
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
tr := tar.NewReader(zr)
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if seen[h.Name] {
|
||||
return nil, "", fmt.Errorf("duplicate path %s in the layers", h.Name)
|
||||
}
|
||||
seen[h.Name] = true
|
||||
if err := tw.WriteHeader(&tar.Header{Name: h.Name, Mode: h.Mode, Size: h.Size, Typeflag: h.Typeflag}); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if _, err := io.CopyN(tw, tr, h.Size); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := tw.Close(); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if !seen["manifest.json"] || !seen["tables"] {
|
||||
return nil, "", fmt.Errorf("the package lacks its manifest or executable")
|
||||
}
|
||||
return out.Bytes(), receipt.ManifestDigest, nil
|
||||
}
|
||||
@@ -0,0 +1,270 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesadmin_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// apply sends one UI change as the person's own tab.
|
||||
func (h *host) apply(user, res, op, tool string, rev map[string]any, args string) reply {
|
||||
GinkgoHelper()
|
||||
return h.applyAs(h.tab(user, res), res, op, tool, rev, args)
|
||||
}
|
||||
|
||||
func (h *host) applyAs(header func(*http.Request), res, op, tool string, rev map[string]any, args string) reply {
|
||||
GinkgoHelper()
|
||||
b, _ := json.Marshal(map[string]any{"operationId": op, "generation": 1, "revision": rev, "tool": tool, "arguments": json.RawMessage(args)})
|
||||
return h.send("POST", "/api/addons/resources/"+res+"/apply", string(b), header)
|
||||
}
|
||||
|
||||
// read runs a typed read as the person and returns the payload.
|
||||
func (h *host) read(user, res, args string) (map[string]any, error) {
|
||||
GinkgoHelper()
|
||||
out, e := h.app.AddonGateway().Dispatch(context.Background(), h.actor(user), contract.Request{
|
||||
AddonID: "tables", ResourceID: res, OperationID: "read-" + uuid.NewString()[:8], Generation: 1, Action: "read",
|
||||
Arguments: json.RawMessage(args),
|
||||
})
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
var v map[string]any
|
||||
Expect(json.Unmarshal(out.Payload, &v)).To(Succeed())
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func revisionOf(v map[string]any) map[string]any {
|
||||
GinkgoHelper()
|
||||
rev, _ := v["revision"].(map[string]any)
|
||||
Expect(rev).NotTo(BeNil())
|
||||
return rev
|
||||
}
|
||||
|
||||
var _ = Describe("UI changes of a database by the person's own action", func() {
|
||||
It("applies schema and row changes under the roles of the host, once per operation, with a receipt and no proposal", func() {
|
||||
h := ready(false)
|
||||
res := h.database("bob", "ap-1", "Ledger")
|
||||
Expect(h.grant("bob", res, "carol", "viewer").code).To(Equal(200))
|
||||
Expect(h.grant("bob", res, "dave", "editor").code).To(Equal(200))
|
||||
inspect := func(user string) map[string]any {
|
||||
v, e := h.read(user, res, `{"tool":"databases-inspect"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return v
|
||||
}
|
||||
|
||||
// The owner changes the schema: a table and a column.
|
||||
made := h.apply("bob", res, "ap-t1", "tables-create", revisionOf(inspect("bob")), `{"table":"Entries"}`)
|
||||
Expect(made.code).To(Equal(200), made.raw)
|
||||
Expect(made.body["resourceId"]).To(Equal(res))
|
||||
Expect(made.body["operationId"]).To(Equal("ap-t1"))
|
||||
Expect(made.body["revision"]).NotTo(Equal(map[string]any{"schema": 1, "data": 0}))
|
||||
col := h.apply("bob", res, "ap-c1", "columns-add", revisionOf(inspect("bob")), `{"table":"Entries","column":"Item","type":"text","required":true}`)
|
||||
Expect(col.code).To(Equal(200), col.raw)
|
||||
qty := h.apply("bob", res, "ap-c2", "columns-add", revisionOf(inspect("bob")), `{"table":"Entries","column":"Qty","type":"integer"}`)
|
||||
Expect(qty.code).To(Equal(200), qty.raw)
|
||||
|
||||
// An editor and the owner write rows; the receipt reports the ids.
|
||||
rev := revisionOf(inspect("dave"))
|
||||
ins := h.apply("dave", res, "ap-r1", "rows-insert", rev, `{"table":"Entries","rows":[{"Item":"pen","Qty":3},{"Item":"ink","Qty":1}]}`)
|
||||
Expect(ins.code).To(Equal(200), ins.raw)
|
||||
result, _ := ins.body["result"].(map[string]any)
|
||||
Expect(result["count"]).To(BeEquivalentTo(2))
|
||||
Expect(result["ids"]).To(HaveLen(2))
|
||||
page, e := h.read("carol", res, `{"tool":"rows-query","table":"Entries"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(page["rows"]).To(HaveLen(2))
|
||||
|
||||
// The same operation with the same request is the same change.
|
||||
again := h.apply("dave", res, "ap-r1", "rows-insert", rev, `{"table":"Entries","rows":[{"Item":"pen","Qty":3},{"Item":"ink","Qty":1}]}`)
|
||||
Expect(again.code).To(Equal(200), again.raw)
|
||||
Expect(again.body["revision"]).To(Equal(ins.body["revision"]))
|
||||
page, _ = h.read("carol", res, `{"tool":"rows-query","table":"Entries"}`)
|
||||
Expect(page["rows"]).To(HaveLen(2))
|
||||
// The same operation with another request changes nothing.
|
||||
other := h.apply("dave", res, "ap-r1", "rows-insert", rev, `{"table":"Entries","rows":[{"Item":"cup","Qty":9}]}`)
|
||||
Expect(other.code).To(Equal(http.StatusConflict), other.raw)
|
||||
Expect(other.errType()).To(Equal("operation_conflict"))
|
||||
page, _ = h.read("carol", res, `{"tool":"rows-query","table":"Entries"}`)
|
||||
Expect(page["rows"]).To(HaveLen(2))
|
||||
// The receipt names the request it belongs to, so a client can tell
|
||||
// the change it asked for from another one under the same ID.
|
||||
hash, _ := ins.body["requestHash"].(string)
|
||||
Expect(hash).To(MatchRegexp(`^[0-9a-f]{64}$`))
|
||||
Expect(again.body["requestHash"]).To(Equal(hash))
|
||||
own := h.send("GET", "/api/addons/resources/"+res+"/operations/ap-r1?kind=apply&requestHash="+hash, "", h.tab("dave", res))
|
||||
Expect(own.code).To(Equal(200), own.raw)
|
||||
Expect(own.body["state"]).To(Equal("committed"))
|
||||
Expect(own.body["tool"]).To(Equal("rows-insert"))
|
||||
Expect(own.body["requestHash"]).To(Equal(hash))
|
||||
foreign := h.send("GET", "/api/addons/resources/"+res+"/operations/ap-r1?kind=apply&requestHash="+strings.Repeat("ab", 32), "", h.tab("dave", res))
|
||||
Expect(foreign.code).To(Equal(http.StatusConflict), foreign.raw)
|
||||
Expect(foreign.errType()).To(Equal("operation_conflict"))
|
||||
Expect(foreign.raw).NotTo(ContainSubstring("committed"))
|
||||
bad := h.send("GET", "/api/addons/resources/"+res+"/operations/ap-r1?kind=apply&requestHash=nope", "", h.tab("dave", res))
|
||||
Expect(bad.code).To(Equal(http.StatusBadRequest), bad.raw)
|
||||
|
||||
// The receipt is there for its own person, and nobody else's.
|
||||
status := h.send("GET", "/api/addons/resources/"+res+"/operations/ap-r1?kind=apply", "", h.tab("dave", res))
|
||||
Expect(status.code).To(Equal(200), status.raw)
|
||||
Expect(status.body["state"]).To(Equal("committed"))
|
||||
Expect(status.body["tool"]).To(Equal("rows-insert"))
|
||||
Expect(status.body["requestHash"]).To(Equal(hash))
|
||||
for _, user := range []string{"bob", "carol"} {
|
||||
s := h.send("GET", "/api/addons/resources/"+res+"/operations/ap-r1?kind=apply", "", h.tab(user, res))
|
||||
if user == "carol" {
|
||||
Expect(s.code).To(Equal(http.StatusForbidden))
|
||||
} else {
|
||||
Expect(s.code).To(Equal(200), s.raw)
|
||||
Expect(s.body["state"]).NotTo(Equal("committed"))
|
||||
}
|
||||
}
|
||||
|
||||
// A moved revision is stale and nothing is applied.
|
||||
stale := h.apply("bob", res, "ap-r2", "rows-insert", rev, `{"table":"Entries","rows":[{"Item":"late","Qty":1}]}`)
|
||||
Expect(stale.code).To(Equal(http.StatusConflict), stale.raw)
|
||||
Expect(stale.errType()).To(Equal("stale"))
|
||||
page, _ = h.read("bob", res, `{"tool":"rows-query","table":"Entries"}`)
|
||||
Expect(page["rows"]).To(HaveLen(2))
|
||||
// With the revision it read, it applies.
|
||||
fresh := revisionOf(page)
|
||||
ids := ins.body["result"].(map[string]any)["ids"].([]any)
|
||||
upd := h.apply("bob", res, "ap-r3", "rows-update", fresh, fmt.Sprintf(`{"table":"Entries","ids":[%v],"set":{"Qty":10}}`, ids[0]))
|
||||
Expect(upd.code).To(Equal(200), upd.raw)
|
||||
del := h.apply("dave", res, "ap-r4", "rows-delete", upd.body["revision"].(map[string]any), fmt.Sprintf(`{"table":"Entries","ids":[%v]}`, ids[1]))
|
||||
Expect(del.code).To(Equal(200), del.raw)
|
||||
page, _ = h.read("carol", res, `{"tool":"rows-query","table":"Entries"}`)
|
||||
Expect(page["rows"]).To(HaveLen(1))
|
||||
|
||||
// A change is not a proposal: nothing waits for a confirmation.
|
||||
pending := h.send("GET", "/api/addons/proposals?state=pending", "", h.bearer("bob"))
|
||||
Expect(pending.body["proposals"]).To(BeEmpty())
|
||||
// The database that the change addressed is the one of the path.
|
||||
Expect(inspect("bob")["name"]).To(Equal("Ledger"))
|
||||
})
|
||||
|
||||
It("refuses crafted writes by a viewer, an editor's schema change, a revoked editor and an installation administrator", func() {
|
||||
h := ready(false)
|
||||
res := h.database("bob", "cr-1", "Crafted")
|
||||
Expect(h.grant("bob", res, "carol", "viewer").code).To(Equal(200))
|
||||
Expect(h.grant("bob", res, "dave", "editor").code).To(Equal(200))
|
||||
v, e := h.read("bob", res, `{"tool":"databases-inspect"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
rev := revisionOf(v)
|
||||
Expect(h.apply("bob", res, "cr-t", "tables-create", rev, `{"table":"T"}`).code).To(Equal(200))
|
||||
v, _ = h.read("bob", res, `{"tool":"databases-inspect"}`)
|
||||
rev = revisionOf(v)
|
||||
Expect(h.apply("bob", res, "cr-c", "columns-add", rev, `{"table":"T","column":"A","type":"text"}`).code).To(Equal(200))
|
||||
v, _ = h.read("bob", res, `{"tool":"databases-inspect"}`)
|
||||
rev = revisionOf(v)
|
||||
unknown := h.apply("bob", "r0123456789abcdef0123456789abcde", "cr-u", "rows-insert", rev, `{"table":"T","rows":[{"A":"x"}]}`)
|
||||
Expect(unknown.code).To(Equal(http.StatusForbidden))
|
||||
|
||||
rows := func() int {
|
||||
p, e := h.read("bob", res, `{"tool":"rows-query","table":"T"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return len(p["rows"].([]any))
|
||||
}
|
||||
body := `{"table":"T","rows":[{"A":"x"}]}`
|
||||
// Viewer, administrator without data privileges and a feature-less
|
||||
// account get the answer of an unknown database.
|
||||
for _, user := range []string{"carol", "alice"} {
|
||||
got := h.apply(user, res, "cr-"+user, "rows-insert", rev, body)
|
||||
Expect(got.code).To(Equal(unknown.code), user+" "+got.raw)
|
||||
Expect(got.errType()).To(Equal("addon_error"), user)
|
||||
}
|
||||
Expect(h.apply("erin", res, "cr-erin", "rows-insert", rev, body).code).To(Equal(http.StatusForbidden))
|
||||
// An editor changes rows and never the schema.
|
||||
sch := h.apply("dave", res, "cr-dave-s", "tables-drop", rev, `{"table":"T"}`)
|
||||
Expect(sch.code).To(Equal(unknown.code), sch.raw)
|
||||
Expect(h.apply("dave", res, "cr-dave-c", "columns-add", rev, `{"table":"T","column":"B","type":"text"}`).code).To(Equal(unknown.code))
|
||||
Expect(rows()).To(Equal(0))
|
||||
|
||||
// Not the person's own tab: a key, a bare session, a tool marker, a
|
||||
// confirmation of another resource and a cross-site request.
|
||||
marked := func(r *http.Request) { h.tab("bob", res)(r); r.Header.Set("X-LocalAI-Tool-Call", "1") }
|
||||
elsewhere := h.tab("bob", "r0123456789abcdef0123456789abcde")
|
||||
cross := func(r *http.Request) { h.tab("bob", res)(r); r.Header.Set("Sec-Fetch-Site", "cross-site") }
|
||||
for name, header := range map[string]func(*http.Request){"key": h.bearer("bob"), "session": h.session("bob"), "marked": marked, "elsewhere": elsewhere, "cross-site": cross} {
|
||||
got := h.applyAs(header, res, "cr-"+name, "rows-insert", rev, body)
|
||||
Expect(got.code).To(BeElementOf(http.StatusForbidden, http.StatusBadRequest), name+" "+got.raw)
|
||||
}
|
||||
Expect(rows()).To(Equal(0))
|
||||
|
||||
// What is not a data or schema change cannot be applied: reads,
|
||||
// creations, renames, the admin delete and undeclared tools.
|
||||
for _, tool := range []string{"rows-query", "databases-create", "databases-rename", "databases-inspect", "databases-delete", "resource-exists", "nothing"} {
|
||||
got := h.apply("bob", res, "cr-tool-"+tool, tool, rev, `{"name":"X","table":"T"}`)
|
||||
Expect(got.code).To(BeElementOf(http.StatusBadRequest, http.StatusForbidden), tool+" "+got.raw)
|
||||
}
|
||||
// A body that names the resource, the tool, the actor or an unknown
|
||||
// field never reaches the add-on.
|
||||
for _, args := range []string{
|
||||
`{"table":"T","rows":[{"A":"x"}],"resourceId":"r0123456789abcdef0123456789abcde"}`,
|
||||
`{"table":"T","rows":[{"A":"x"}],"tool":"tables-drop"}`,
|
||||
`{"table":"T","rows":[{"A":"x"}],"userId":"alice"}`,
|
||||
`{"table":"T","rows":[{"A":"x"}],"extra":1}`,
|
||||
`[]`, `{}`,
|
||||
} {
|
||||
got := h.apply("bob", res, "cr-args", "rows-insert", rev, args)
|
||||
Expect(got.code).To(Equal(http.StatusBadRequest), args+" "+got.raw)
|
||||
}
|
||||
// A stale generation and a missing revision.
|
||||
b := fmt.Sprintf(`{"operationId":"cr-gen","generation":9,"revision":{"schema":%v,"data":%v},"tool":"rows-insert","arguments":%s}`, rev["schema"], rev["data"], body)
|
||||
got := h.send("POST", "/api/addons/resources/"+res+"/apply", b, h.tab("bob", res))
|
||||
Expect(got.code).To(Equal(http.StatusConflict))
|
||||
Expect(got.errType()).To(Equal("stale"))
|
||||
noRev := h.send("POST", "/api/addons/resources/"+res+"/apply", `{"operationId":"cr-norev","generation":1,"tool":"rows-insert","arguments":`+body+`}`, h.tab("bob", res))
|
||||
Expect(noRev.code).To(Equal(http.StatusBadRequest))
|
||||
Expect(rows()).To(Equal(0))
|
||||
|
||||
// A revoked editor is refused at the next change, and one that was
|
||||
// writing is stopped before the revocation is acknowledged.
|
||||
Expect(h.apply("dave", res, "cr-ok", "rows-insert", rev, body).code).To(Equal(200))
|
||||
Expect(h.revoke("bob", res, "dave").code).To(Equal(200))
|
||||
v, _ = h.read("bob", res, `{"tool":"rows-query","table":"T"}`)
|
||||
after := h.apply("dave", res, "cr-late", "rows-insert", revisionOf(v), body)
|
||||
Expect(after.code).To(Equal(unknown.code), after.raw)
|
||||
Expect(rows()).To(Equal(1))
|
||||
})
|
||||
|
||||
It("carries a UI change to the owner from a forwarding frontend and answers as the owner does", func() {
|
||||
h := ready(false)
|
||||
front := h.frontend()
|
||||
res := h.database("bob", "fa-1", "Forwarded")
|
||||
Expect(h.grant("bob", res, "dave", "editor").code).To(Equal(200))
|
||||
v, _ := h.read("bob", res, `{"tool":"databases-inspect"}`)
|
||||
rev := revisionOf(v)
|
||||
send := func(header func(*http.Request), op string, rev map[string]any, tool, args string) reply {
|
||||
b, _ := json.Marshal(map[string]any{"operationId": op, "generation": 1, "revision": rev, "tool": tool, "arguments": json.RawMessage(args)})
|
||||
return serve(front, "POST", "/api/addons/resources/"+res+"/apply", string(b), header)
|
||||
}
|
||||
made := send(h.tab("bob", res), "fa-t", rev, "tables-create", `{"table":"T"}`)
|
||||
Expect(made.code).To(Equal(200), made.raw)
|
||||
// Stale and forbidden answers are the owner's.
|
||||
stale := send(h.tab("bob", res), "fa-s", rev, "tables-create", `{"table":"U"}`)
|
||||
Expect(stale.code).To(Equal(http.StatusConflict))
|
||||
Expect(stale.errType()).To(Equal("stale"))
|
||||
direct := h.apply("bob", res, "fa-s2", "tables-create", rev, `{"table":"U"}`)
|
||||
Expect(direct.code).To(Equal(stale.code))
|
||||
Expect(direct.errType()).To(Equal(stale.errType()))
|
||||
denied := send(h.tab("dave", res), "fa-d", made.body["revision"].(map[string]any), "tables-drop", `{"table":"T"}`)
|
||||
Expect(denied.code).To(Equal(http.StatusForbidden))
|
||||
Expect(send(h.bearer("bob"), "fa-k", rev, "tables-create", `{"table":"V"}`).code).To(Equal(http.StatusForbidden))
|
||||
// A frontend whose owner cannot be reached reports the outcome of a
|
||||
// change as unknown, with the original operation ID.
|
||||
dead := h.deadFrontend()
|
||||
b, _ := json.Marshal(map[string]any{"operationId": "fa-dead", "generation": 1, "revision": made.body["revision"], "tool": "tables-create", "arguments": json.RawMessage(`{"table":"W"}`)})
|
||||
got := serve(dead, "POST", "/api/addons/resources/"+res+"/apply", string(b), h.tab("bob", res))
|
||||
Expect(got.code).To(Equal(http.StatusServiceUnavailable))
|
||||
Expect(got.errType()).To(Equal("outcome_unknown"))
|
||||
Expect(got.errOperation()).To(Equal("fa-dead"))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,307 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesadmin_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/addons/gateway"
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
"github.com/mudler/LocalAI/core/services/addons/tools"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("deleting a database is explicit, destructive and UI-only", func() {
|
||||
It("deletes the data, the grants and the pending proposals only on the owner's typed confirmation and keeps the add-on installed", func() {
|
||||
h := ready(false)
|
||||
res := h.database("bob", "d-1", "Doomed")
|
||||
Expect(h.grant("bob", res, "carol", "viewer").code).To(Equal(200))
|
||||
// A pending change of the owner, waiting for a human decision.
|
||||
propose := fmt.Sprintf(`{"addonId":"tables","resourceId":%q,"operationId":"d-p1","generation":1,"action":"schema","arguments":{"tool":"tables-create","table":"T"}}`, res)
|
||||
made := h.send("POST", "/api/addons/proposals", propose, h.bearer("bob"))
|
||||
Expect(made.code).To(Equal(200), made.raw)
|
||||
Expect(made.body["state"]).To(Equal("pending"))
|
||||
rev := h.revision("bob", res)
|
||||
|
||||
// Nobody but the owner's own session deletes: not an administrator,
|
||||
// not a grantee, not an API key, and not without the confirmation.
|
||||
for user, got := range map[string]reply{
|
||||
"alice": h.remove("alice", res, "d-a", "Doomed", rev),
|
||||
"carol": h.remove("carol", res, "d-c", "Doomed", rev),
|
||||
} {
|
||||
Expect(got.code).To(Equal(http.StatusForbidden), user+got.raw)
|
||||
}
|
||||
body := `{"operationId":"d-k","generation":1,"revision":{"schema":1,"data":0},"confirm":"Doomed"}`
|
||||
Expect(h.send("POST", "/api/addons/resources/"+res+"/delete", body, h.bearer("bob")).code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.send("POST", "/api/addons/resources/"+res+"/delete", body, h.session("bob")).code).To(Equal(http.StatusForbidden))
|
||||
wrong := h.remove("bob", res, "d-w", "doomed", rev)
|
||||
Expect(wrong.code).To(Equal(http.StatusBadRequest), wrong.raw)
|
||||
Expect(wrong.errType()).To(Equal("invalid_request"))
|
||||
for _, confirm := range []string{"", "Doomed ", res} {
|
||||
got := h.remove("bob", res, "d-w", confirm, rev)
|
||||
Expect(got.code).To(Equal(http.StatusBadRequest), confirm)
|
||||
}
|
||||
// Nothing happened: the database, its file and its grants are intact.
|
||||
Expect(resourcesOf(h.list("bob"))).To(HaveKey(res))
|
||||
Expect(h.dbFile(res)).To(BeARegularFile())
|
||||
name, e := h.name("carol", res)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Doomed"))
|
||||
|
||||
done := h.remove("bob", res, "d-ok", "Doomed", rev)
|
||||
Expect(done.code).To(Equal(http.StatusOK), done.raw)
|
||||
Expect(done.body["deleted"]).To(Equal(true))
|
||||
Expect(done.body["resourceId"]).To(Equal(res))
|
||||
// The data is gone, the record and the grants are gone, and nothing
|
||||
// of it is reachable by anyone.
|
||||
_, statErr := os.Stat(h.dbFile(res))
|
||||
Expect(os.IsNotExist(statErr)).To(BeTrue())
|
||||
for _, user := range []string{"bob", "carol", "alice"} {
|
||||
Expect(resourcesOf(h.list(user))).NotTo(HaveKey(res), user)
|
||||
_, e := h.name(user, res)
|
||||
Expect(e).To(HaveOccurred(), user)
|
||||
}
|
||||
Expect(h.app.AddonPolicy().Deleting()).To(BeEmpty())
|
||||
_, known := h.app.AddonPolicy().Key(res)
|
||||
Expect(known).To(BeFalse())
|
||||
// The pending proposal can never run.
|
||||
card := h.send("GET", "/api/addons/proposals/d-p1", "", h.bearer("bob"))
|
||||
Expect(card.body["state"]).To(Equal("cancelled"))
|
||||
// Deleting is not an uninstall: the add-on stays installed and
|
||||
// running and other databases still work.
|
||||
h.running()
|
||||
other := h.database("bob", "d-2", "Survivor")
|
||||
name, _ = h.name("bob", other)
|
||||
Expect(name).To(Equal("Survivor"))
|
||||
// The same call again, after its answer was lost, is the same success
|
||||
// for the owner who asked, and the receipt lookup says committed. Any
|
||||
// other operation, user or database is the denial of an unknown one.
|
||||
again := h.remove("bob", res, "d-ok", "Doomed", rev)
|
||||
Expect(again.code).To(Equal(http.StatusOK), again.raw)
|
||||
Expect(again.body["deleted"]).To(Equal(true))
|
||||
status := h.send("GET", "/api/addons/resources/"+res+"/operations/d-ok?kind=delete", "", h.tab("bob", res))
|
||||
Expect(status.code).To(Equal(200), status.raw)
|
||||
Expect(status.body["state"]).To(Equal("committed"))
|
||||
Expect(h.remove("bob", res, "d-other", "Doomed", rev).code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.remove("carol", res, "d-ok", "Doomed", rev).code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.remove("alice", res, "d-ok", "Doomed", rev).code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.send("GET", "/api/addons/resources/"+res+"/operations/d-ok?kind=delete", "", h.tab("carol", res)).code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.send("GET", "/api/addons/resources/"+res+"/operations/d-ok?kind=delete", "", h.tab("alice", res)).code).To(Equal(http.StatusForbidden))
|
||||
})
|
||||
|
||||
It("stops and waits for admitted work of every user before it asks the add-on to delete", func() {
|
||||
h := ready(false)
|
||||
res := h.database("bob", "w-1", "Busy")
|
||||
Expect(h.grant("bob", res, "carol", "editor").code).To(Equal(200))
|
||||
p := h.app.AddonPolicy()
|
||||
started, ended := make(chan struct{}), make(chan struct{})
|
||||
go func() {
|
||||
defer GinkgoRecover()
|
||||
_ = p.Admit(context.Background(), h.actor("carol"), h.key(res), policy.ActionWrite, func(ctx context.Context) error {
|
||||
close(started)
|
||||
<-ctx.Done()
|
||||
close(ended)
|
||||
return ctx.Err()
|
||||
})
|
||||
}()
|
||||
Eventually(started).Should(BeClosed())
|
||||
done := h.remove("bob", res, "w-del", "Busy", h.revision("bob", res))
|
||||
Expect(done.code).To(Equal(http.StatusOK), done.raw)
|
||||
// The admitted work had ended when the deletion was acknowledged.
|
||||
Expect(ended).To(BeClosed())
|
||||
Expect(h.dbFile(res)).NotTo(BeAnExistingFile())
|
||||
})
|
||||
|
||||
It("makes the add-on refuse its delete tool under every action but the host's delete action", func() {
|
||||
h := ready(false)
|
||||
res := h.database("bob", "a-1", "Held")
|
||||
ctx := context.Background()
|
||||
inst, e := h.app.AddonManager().Runtime(ctx, "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
envelope := func(action, scope, op string) contract.Envelope {
|
||||
return contract.Envelope{
|
||||
Request: contract.Request{AddonID: "tables", ResourceID: res, OperationID: op, Generation: inst.Generation, Action: action, Arguments: json.RawMessage(`{"tool":"databases-delete"}`)},
|
||||
Authority: contract.Authority{Actor: h.actor("bob"), Scope: scope, ExpiresAt: time.Now().Add(time.Minute)},
|
||||
}
|
||||
}
|
||||
for i, action := range []string{"read", "write", "schema", "create", "output"} {
|
||||
op := fmt.Sprintf("a-op-%d", i)
|
||||
_, e = inst.Client.Preview(ctx, envelope(action, "preview", op))
|
||||
Expect(e).To(HaveOccurred(), action)
|
||||
_, e = inst.Client.Apply(ctx, envelope(action, "apply", op), contract.Revision{Schema: 1})
|
||||
Expect(e).To(HaveOccurred(), action)
|
||||
}
|
||||
_, e = inst.Client.Read(ctx, envelope("read", "read", "a-op-read"))
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(h.dbFile(res)).To(BeARegularFile())
|
||||
// Under the delete action the same envelope previews and changes nothing.
|
||||
prev, e := inst.Client.Preview(ctx, envelope("delete", "preview", "a-op-del"))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(prev.Revision.Schema).NotTo(BeZero())
|
||||
// An apply for a revision the database no longer has is a rejected
|
||||
// receipt, not a deletion.
|
||||
_, e = inst.Client.Apply(ctx, envelope("delete", "apply", "a-op-del"), contract.Revision{Schema: 99})
|
||||
Expect(e).NotTo(HaveOccurred(), "a moved revision is a rejected receipt, not a deletion")
|
||||
Expect(h.dbFile(res)).To(BeARegularFile())
|
||||
name, e := h.name("bob", res)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Held"))
|
||||
})
|
||||
|
||||
It("never offers the delete tool to a model, a proposal, a typed read or an agent", func() {
|
||||
h := ready(false)
|
||||
res := h.database("bob", "t-1", "Guarded")
|
||||
ctx := context.Background()
|
||||
adapter := tools.NewToolAdapter(h.app.AddonGateway())
|
||||
|
||||
defs, e := adapter.Discover(ctx, h.actor("bob"))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(defs).NotTo(BeEmpty())
|
||||
for _, d := range defs {
|
||||
Expect(d.Tool).NotTo(Equal("databases-delete"))
|
||||
}
|
||||
_, e = adapter.Execute(ctx, h.actor("bob"), "addon__tables__databases-delete", json.RawMessage(fmt.Sprintf(`{"resourceId":%q}`, res)))
|
||||
Expect(e).To(MatchError(tools.ErrUnknownTool))
|
||||
// A typed read, a proposal of any action and a dispatch that names the
|
||||
// admin tool are denied by the declared-tool rule, as the owner.
|
||||
read := contract.Request{AddonID: "tables", ResourceID: res, OperationID: "t-r", Generation: 1, Action: "read", Arguments: json.RawMessage(`{"tool":"databases-delete"}`)}
|
||||
_, e = h.app.AddonGateway().Dispatch(ctx, h.actor("bob"), read)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
for i, action := range []string{"write", "schema", "delete", "create"} {
|
||||
req := contract.Request{AddonID: "tables", ResourceID: res, OperationID: fmt.Sprintf("t-p%d", i), Generation: 1, Action: action, Arguments: json.RawMessage(`{"tool":"databases-delete"}`)}
|
||||
if action == "create" {
|
||||
req.ResourceID = ""
|
||||
}
|
||||
_, e = h.app.AddonGateway().Propose(ctx, h.actor("bob"), req)
|
||||
Expect(e).To(HaveOccurred(), action)
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403), action)
|
||||
}
|
||||
// The HTTP routes say the same.
|
||||
for _, action := range []string{"read", "write", "schema", "delete"} {
|
||||
b := fmt.Sprintf(`{"addonId":"tables","resourceId":%q,"operationId":"t-h-%s","generation":1,"action":%q,"arguments":{"tool":"databases-delete"}}`, res, action, action)
|
||||
Expect(h.send("POST", "/api/addons/proposals", b, h.bearer("bob")).code).To(Equal(403), action)
|
||||
Expect(h.send("POST", "/api/addons/dispatch", b, h.bearer("bob")).code).To(BeElementOf(403), action)
|
||||
}
|
||||
// The Apps view lists the model tools and not the admin tool.
|
||||
apps := h.send("GET", "/api/addons/apps", "", h.bearer("bob"))
|
||||
Expect(apps.raw).NotTo(ContainSubstring("databases-delete"))
|
||||
// Nothing was deleted.
|
||||
Expect(h.dbFile(res)).To(BeARegularFile())
|
||||
Expect(resourcesOf(h.list("bob"))).To(HaveKey(res))
|
||||
})
|
||||
|
||||
It("denies a stale tab: the generation and the revision the owner saw decide, and nothing is removed", func() {
|
||||
h := ready(false)
|
||||
res := h.database("bob", "s-1", "Tabbed")
|
||||
seen := h.revision("bob", res)
|
||||
// A change of the schema through the usual proposal path moves the
|
||||
// revision the old tab saw.
|
||||
propose := fmt.Sprintf(`{"addonId":"tables","resourceId":%q,"operationId":"s-p1","generation":1,"action":"schema","arguments":{"tool":"tables-create","table":"Moves"}}`, res)
|
||||
made := h.send("POST", "/api/addons/proposals", propose, h.bearer("bob"))
|
||||
Expect(made.code).To(Equal(200), made.raw)
|
||||
ok := h.send("POST", "/api/addons/proposals/s-p1/confirm", fmt.Sprintf(`{"digest":%q}`, made.body["digest"]), h.tab("bob", "s-p1"))
|
||||
Expect(ok.code).To(Equal(200), ok.raw)
|
||||
Expect(h.revision("bob", res)).NotTo(Equal(seen))
|
||||
|
||||
got := h.remove("bob", res, "s-d1", "Tabbed", seen)
|
||||
Expect(got.code).To(Equal(http.StatusConflict), got.raw)
|
||||
Expect(got.errType()).To(Equal("stale"))
|
||||
got = h.rename("bob", res, "s-r1", "Renamed", seen)
|
||||
Expect(got.code).To(Equal(http.StatusConflict), got.raw)
|
||||
Expect(got.errType()).To(Equal("stale"))
|
||||
// A generation other than the installed one is stale too.
|
||||
b, _ := json.Marshal(map[string]any{"operationId": "s-d2", "generation": 7, "revision": h.revision("bob", res), "confirm": "Tabbed"})
|
||||
got = h.send("POST", "/api/addons/resources/"+res+"/delete", string(b), h.tab("bob", res))
|
||||
Expect(got.code).To(Equal(http.StatusConflict), got.raw)
|
||||
Expect(got.errType()).To(Equal("stale"))
|
||||
// The resource was returned to use at once.
|
||||
Expect(h.dbFile(res)).To(BeARegularFile())
|
||||
name, e := h.name("bob", res)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Tabbed"))
|
||||
Expect(h.app.AddonPolicy().Deleting()).To(BeEmpty())
|
||||
// With the current view it works.
|
||||
Expect(h.remove("bob", res, "s-d3", "Tabbed", h.revision("bob", res)).code).To(Equal(200))
|
||||
})
|
||||
|
||||
It("finishes or undoes a deletion that a crash interrupted at every step", func() {
|
||||
h := ready(false)
|
||||
ctx := context.Background()
|
||||
p := h.app.AddonPolicy()
|
||||
bob := h.actor("bob")
|
||||
confirmed := h.database("bob", "k-1", "Confirmed")
|
||||
unconfirmed := h.database("bob", "k-2", "Unconfirmed")
|
||||
gone := h.database("bob", "k-3", "AlreadyGone")
|
||||
kept := h.database("bob", "k-4", "Untouched")
|
||||
Expect(h.grant("bob", confirmed, "carol", "editor").code).To(Equal(200))
|
||||
|
||||
// A crash after the confirmation and before the add-on deleted.
|
||||
Expect(p.BeginDelete(ctx, bob, h.key(confirmed), "k-op1")).To(Succeed())
|
||||
Expect(p.ConfirmDelete(h.key(confirmed))).To(Succeed())
|
||||
// A crash while the owner was still checking its preconditions.
|
||||
Expect(p.BeginDelete(ctx, bob, h.key(unconfirmed), "k-op2")).To(Succeed())
|
||||
// A crash after the add-on removed the data and before the record went.
|
||||
Expect(p.BeginDelete(ctx, bob, h.key(gone), "k-op3")).To(Succeed())
|
||||
Expect(p.ConfirmDelete(h.key(gone))).To(Succeed())
|
||||
Expect(os.Remove(h.dbFile(gone))).To(Succeed())
|
||||
|
||||
// While a deletion is in progress the database is inaccessible to
|
||||
// everyone, owner included, and nothing new is admitted.
|
||||
for _, res := range []string{confirmed, unconfirmed, gone} {
|
||||
Expect(resourcesOf(h.list("bob"))).NotTo(HaveKey(res))
|
||||
_, e := h.name("bob", res)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(h.grant("bob", res, "dave", "viewer").code).To(Equal(http.StatusForbidden))
|
||||
}
|
||||
Expect(p.Deleting()).To(HaveLen(3))
|
||||
Expect(resourcesOf(h.list("bob"))).To(HaveKey(kept))
|
||||
// The owner's receipt lookup says the deletion is still pending.
|
||||
pending := h.send("GET", "/api/addons/resources/"+confirmed+"/operations/k-op1?kind=delete", "", h.tab("bob", confirmed))
|
||||
Expect(pending.code).To(Equal(200), pending.raw)
|
||||
Expect(pending.body["state"]).To(Equal("pending"))
|
||||
Expect(h.send("GET", "/api/addons/resources/"+confirmed+"/operations/k-op1?kind=delete", "", h.tab("carol", confirmed)).code).To(Equal(http.StatusForbidden))
|
||||
|
||||
h.stop()
|
||||
h.start()
|
||||
h.running()
|
||||
|
||||
// The confirmed deletion was finished, the unconfirmed one undone.
|
||||
Expect(h.app.AddonPolicy().Deleting()).To(BeEmpty())
|
||||
Expect(h.dbFile(confirmed)).NotTo(BeAnExistingFile())
|
||||
listed := resourcesOf(h.list("bob"))
|
||||
Expect(listed).NotTo(HaveKey(confirmed))
|
||||
Expect(listed).NotTo(HaveKey(gone))
|
||||
Expect(listed).To(HaveKey(unconfirmed))
|
||||
Expect(listed).To(HaveKey(kept))
|
||||
name, e := h.name("bob", unconfirmed)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Unconfirmed"))
|
||||
Expect(resourcesOf(h.list("carol"))).NotTo(HaveKey(confirmed))
|
||||
_, known := h.app.AddonPolicy().Key(confirmed)
|
||||
Expect(known).To(BeFalse())
|
||||
_, known = h.app.AddonPolicy().Key(gone)
|
||||
Expect(known).To(BeFalse())
|
||||
// What stays is a tombstone for the owner, and it survived the restart.
|
||||
for res, op := range map[string]string{confirmed: "k-op1", gone: "k-op3"} {
|
||||
done, e := h.app.AddonPolicy().DeletedBy(ctx, bob, res, op)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(done).To(BeTrue(), res)
|
||||
status := h.send("GET", "/api/addons/resources/"+res+"/operations/"+op+"?kind=delete", "", h.tab("bob", res))
|
||||
Expect(status.code).To(Equal(200), status.raw)
|
||||
Expect(status.body["state"]).To(Equal("committed"))
|
||||
}
|
||||
done, e := h.app.AddonPolicy().DeletedBy(ctx, h.actor("carol"), confirmed, "k-op1")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(done).To(BeFalse())
|
||||
done, e = h.app.AddonPolicy().DeletedBy(ctx, bob, unconfirmed, "k-op2")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(done).To(BeFalse(), "an undone deletion leaves no tombstone")
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,144 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesadmin_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/mudler/LocalAI/core/http/auth"
|
||||
"github.com/mudler/LocalAI/core/services/addons/gateway"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("the advertised surface of the resource management routes", func() {
|
||||
It("documents every route, every emitted field and every typed answer in the Swagger spec", func() {
|
||||
spec, err := os.ReadFile("../../../swagger/swagger.json")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
var doc struct {
|
||||
Paths map[string]map[string]json.RawMessage `json:"paths"`
|
||||
Definitions map[string]struct {
|
||||
Properties map[string]any `json:"properties"`
|
||||
} `json:"definitions"`
|
||||
}
|
||||
Expect(json.Unmarshal(spec, &doc)).To(Succeed())
|
||||
for path, methods := range map[string][]string{
|
||||
"/api/addons/resources": {"get", "post"},
|
||||
"/api/addons/resources/{resourceId}/grants": {"get"},
|
||||
"/api/addons/resources/{resourceId}/grants/{userId}": {"put", "delete"},
|
||||
"/api/addons/resources/{resourceId}/users": {"get"},
|
||||
"/api/addons/resources/{resourceId}/rename": {"post"},
|
||||
"/api/addons/resources/{resourceId}/delete": {"post"},
|
||||
"/api/addons/resources/{resourceId}/apply": {"post"},
|
||||
"/api/addons/resources/{resourceId}/operations/{operationId}": {"get"},
|
||||
"/api/addons/conversations/{conversationId}/selection/{addonId}": {"get", "put", "delete"},
|
||||
} {
|
||||
Expect(doc.Paths).To(HaveKey(path))
|
||||
for _, method := range methods {
|
||||
Expect(doc.Paths[path]).To(HaveKey(method), path)
|
||||
text := string(doc.Paths[path][method])
|
||||
Expect(text).To(ContainSubstring(`"Addons"`), path+" "+method)
|
||||
Expect(text).To(ContainSubstring("MCP"), path+" "+method+" must say that no MCP tool maps to it, or that it is read-only")
|
||||
}
|
||||
}
|
||||
for _, check := range []struct{ path, method string }{
|
||||
{"/api/addons/resources/{resourceId}/rename", "post"}, {"/api/addons/resources/{resourceId}/delete", "post"}, {"/api/addons/resources/{resourceId}/apply", "post"},
|
||||
} {
|
||||
text := string(doc.Paths[check.path][check.method])
|
||||
for _, want := range []string{`"409"`, `"503"`, "stale", "outcome_unknown", "X-LocalAI-Confirm"} {
|
||||
Expect(text).To(ContainSubstring(want), check.path+" "+want)
|
||||
}
|
||||
}
|
||||
Expect(string(doc.Paths["/api/addons/resources/{resourceId}/grants"]["get"])).To(ContainSubstring("sharing_disabled"))
|
||||
|
||||
emitted := func(v any) map[string]any {
|
||||
b, err := json.Marshal(v)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
var m map[string]any
|
||||
Expect(json.Unmarshal(b, &m)).To(Succeed())
|
||||
return m
|
||||
}
|
||||
rev := contract.Revision{Schema: 1, Data: 2}
|
||||
n := 1
|
||||
info := gateway.ResourceInfo{ID: "r", AddonID: "tables", Name: "N", Role: "owner", Owner: "O", SharedWith: &n, Revision: &rev}
|
||||
for name, v := range map[string]any{
|
||||
"gateway.ResourceInfo": info,
|
||||
"gateway.ResourceList": gateway.ResourceList{AddonID: "tables", Generation: 1, Sharing: true, Noun: "database", Operations: []string{"create"}, Resources: []gateway.ResourceInfo{info}, Truncated: true},
|
||||
"gateway.Grant": gateway.Grant{UserID: "u", DisplayName: "D", Role: "viewer"},
|
||||
"gateway.GrantList": gateway.GrantList{ResourceID: "r", Grants: []gateway.Grant{}},
|
||||
"gateway.UserRef": gateway.UserRef{ID: "u", DisplayName: "D"},
|
||||
"gateway.UserList": gateway.UserList{Users: []gateway.UserRef{}},
|
||||
"gateway.DeleteResult": gateway.DeleteResult{ResourceID: "r", Deleted: true},
|
||||
"gateway.OperationStatus": gateway.OperationStatus{OperationID: "o", State: "committed", Tool: "rows-insert", RequestHash: "h"},
|
||||
"gateway.Selection": gateway.Selection{AddonID: "tables", ConversationID: "c", ResourceID: "r", Role: "owner", Name: "N"},
|
||||
"gateway.CreateResourceRequest": gateway.CreateResourceRequest{AddonID: "tables", OperationID: "o", Generation: 1, Arguments: json.RawMessage(`{}`)},
|
||||
"gateway.RenameResourceRequest": gateway.RenameResourceRequest{OperationID: "o", Generation: 1, Revision: rev, Name: "N"},
|
||||
"gateway.DeleteResourceRequest": gateway.DeleteResourceRequest{OperationID: "o", Generation: 1, Revision: rev, Confirm: "N"},
|
||||
"gateway.SetGrantRequest": gateway.SetGrantRequest{Role: "viewer", OperationID: "o"},
|
||||
"gateway.ApplyResourceRequest": gateway.ApplyResourceRequest{OperationID: "o", Generation: 1, Revision: rev, Tool: "rows-insert", Arguments: json.RawMessage(`{}`)},
|
||||
"gateway.ApplyResult": gateway.ApplyResult{ResourceID: "r", OperationID: "o", Revision: rev, Result: json.RawMessage(`{}`), RequestHash: "h"},
|
||||
"gateway.SelectionRequest": gateway.SelectionRequest{ResourceID: "r"},
|
||||
} {
|
||||
def, ok := doc.Definitions[name]
|
||||
Expect(ok).To(BeTrue(), name)
|
||||
shape := emitted(v)
|
||||
for field := range shape {
|
||||
Expect(def.Properties).To(HaveKey(field), name+"."+field)
|
||||
}
|
||||
for field := range def.Properties {
|
||||
Expect(shape).To(HaveKey(field), name+" documents "+field+" that is never emitted")
|
||||
}
|
||||
}
|
||||
// The error object documents the new types.
|
||||
text, _ := json.Marshal(doc.Definitions["addons.AddonError"].Properties["type"])
|
||||
for _, typ := range []string{"sharing_disabled", "stale", "invalid_request", "quota_reached", "not_selected", "operation_conflict"} {
|
||||
Expect(string(text)).To(ContainSubstring(typ))
|
||||
}
|
||||
})
|
||||
|
||||
It("lists every route in the instructions registry, the feature registry and the documentation, with no MCP tool for any", func() {
|
||||
h := ready(false)
|
||||
got := h.send("GET", "/api/instructions/addons", "", h.bearer("alice"))
|
||||
Expect(got.code).To(Equal(http.StatusOK), got.raw)
|
||||
for _, want := range []string{"/api/addons/resources", "grants", "users?q=", "rename", "delete", "apply", "selection", "select_resource", "never tools"} {
|
||||
Expect(got.raw).To(ContainSubstring(want), want)
|
||||
}
|
||||
registered := map[string]bool{}
|
||||
for _, entry := range auth.RouteFeatureRegistry {
|
||||
if entry.Feature == auth.FeatureAddons {
|
||||
registered[entry.Method+" "+entry.Pattern] = true
|
||||
}
|
||||
}
|
||||
docs, err := os.ReadFile("../../../docs/content/features/addons.md")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
tables, err := os.ReadFile("../../../docs/content/features/tables.md")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
for _, key := range []string{
|
||||
"GET /api/addons/resources", "POST /api/addons/resources",
|
||||
"GET /api/addons/resources/:resourceId/grants", "PUT /api/addons/resources/:resourceId/grants/:userId",
|
||||
"DELETE /api/addons/resources/:resourceId/grants/:userId", "GET /api/addons/resources/:resourceId/users",
|
||||
"POST /api/addons/resources/:resourceId/rename", "POST /api/addons/resources/:resourceId/delete",
|
||||
"POST /api/addons/resources/:resourceId/apply",
|
||||
"GET /api/addons/resources/:resourceId/operations/:operationId",
|
||||
"GET /api/addons/conversations/:conversationId/selection/:addonId",
|
||||
"PUT /api/addons/conversations/:conversationId/selection/:addonId",
|
||||
"DELETE /api/addons/conversations/:conversationId/selection/:addonId",
|
||||
} {
|
||||
Expect(registered).To(HaveKey(key))
|
||||
}
|
||||
for _, want := range []string{"never tools", "sharing_disabled", "outcome_unknown", "capabilities.adminTools", "capabilities.resources", "select_resource", "X-LocalAI-Conversation-Id", "deleting"} {
|
||||
Expect(string(docs)).To(ContainSubstring(want), want)
|
||||
}
|
||||
Expect(string(tables)).To(ContainSubstring("databases-delete"))
|
||||
Expect(strings.Count(string(tables), "not tools")).To(BeNumerically(">=", 1))
|
||||
// The admin assistant catalog names none of it.
|
||||
for _, file := range []string{"../../../pkg/mcp/localaitools/tools.go"} {
|
||||
b, err := os.ReadFile(file)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(strings.ToLower(string(b))).NotTo(ContainSubstring("/api/addons"))
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,455 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesadmin_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/mudler/LocalAI/core/services/addons/artifacts"
|
||||
"github.com/mudler/LocalAI/core/services/addons/gateway"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// ordersCSV is the exact file the Orders table of ordersTable exports.
|
||||
const ordersCSV = "Item,Note,Qty,Price,Paid\r\n" +
|
||||
"'=SUM(A1:A9),\"line one\nline two\",-4,19.99,true\r\n" +
|
||||
"\"Café, \"\"deluxe\"\"\",日本語,12,,\r\n" +
|
||||
"plain,,,,\r\n"
|
||||
|
||||
// ordersTable creates a database for the user with the Orders table: hostile
|
||||
// text, newlines, Unicode, a negative number and NULLs.
|
||||
func (h *host) ordersTable(user, prefix string) string {
|
||||
GinkgoHelper()
|
||||
res := h.database(user, prefix+"-db", "Orders data")
|
||||
rev := func() map[string]any {
|
||||
v, e := h.read(user, res, `{"tool":"databases-inspect"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return revisionOf(v)
|
||||
}
|
||||
step := func(n int, tool, args string) {
|
||||
GinkgoHelper()
|
||||
got := h.apply(user, res, fmt.Sprintf("%s-%d", prefix, n), tool, rev(), args)
|
||||
Expect(got.code).To(Equal(200), got.raw)
|
||||
}
|
||||
step(1, "tables-create", `{"table":"Orders"}`)
|
||||
for i, c := range [][2]string{{"Item", "text"}, {"Note", "text"}, {"Qty", "integer"}, {"Price", "real"}, {"Paid", "boolean"}} {
|
||||
step(2+i, "columns-add", fmt.Sprintf(`{"table":"Orders","column":%q,"type":%q}`, c[0], c[1]))
|
||||
}
|
||||
step(10, "rows-insert", `{"table":"Orders","rows":[
|
||||
{"Item":"=SUM(A1:A9)","Note":"line one\nline two","Qty":-4,"Price":19.99,"Paid":true},
|
||||
{"Item":"Café, \"deluxe\"","Note":"日本語","Qty":12},
|
||||
{"Item":"plain"}]}`)
|
||||
return res
|
||||
}
|
||||
|
||||
// exported runs the export as the person through the owner's gateway and
|
||||
// returns the host artifact it was turned into.
|
||||
func (h *host) exported(user, res, args string) (contract.OutputDescription, error) {
|
||||
GinkgoHelper()
|
||||
out, e := h.app.AddonGateway().Dispatch(context.Background(), h.actor(user), contract.Request{
|
||||
AddonID: "tables", ResourceID: res, OperationID: "exp-" + uuid.NewString()[:12], Generation: 1, Action: "read",
|
||||
Arguments: json.RawMessage(args),
|
||||
})
|
||||
if e != nil {
|
||||
return contract.OutputDescription{}, e
|
||||
}
|
||||
Expect(out.Outputs).To(HaveLen(1))
|
||||
return out.Outputs[0], nil
|
||||
}
|
||||
|
||||
// http is a real network client of the host's HTTP application.
|
||||
type httpHost struct{ srv *httptest.Server }
|
||||
|
||||
func (h *host) listen() httpHost {
|
||||
s := httptest.NewServer(h.web)
|
||||
DeferCleanup(s.Close)
|
||||
return httpHost{s}
|
||||
}
|
||||
|
||||
func (n httpHost) get(path string, header func(*http.Request)) (*http.Response, []byte) {
|
||||
GinkgoHelper()
|
||||
r, e := http.NewRequest("GET", n.srv.URL+path, nil)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
if header != nil {
|
||||
header(r)
|
||||
}
|
||||
resp, e := http.DefaultClient.Do(r)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return resp, b
|
||||
}
|
||||
|
||||
func (h *host) complete(user, id string) map[string]any {
|
||||
GinkgoHelper()
|
||||
var got reply
|
||||
Eventually(func() string {
|
||||
got = h.send("GET", "/api/addons/artifacts/"+id, "", h.bearer(user))
|
||||
s, _ := got.body["state"].(string)
|
||||
return s
|
||||
}, 20*time.Second, 50*time.Millisecond).Should(Equal("complete"), got.raw)
|
||||
return got.body
|
||||
}
|
||||
|
||||
var _ = Describe("CSV exports are private artifacts", func() {
|
||||
It("exports a table with exact bytes and attachment headers over real HTTP and shows no one else the file", func() {
|
||||
h := ready(false)
|
||||
res := h.ordersTable("bob", "ex1")
|
||||
art, e := h.exported("bob", res, `{"tool":"rows-export","table":"Orders"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(art.Name).To(Equal("Orders.csv"))
|
||||
Expect(art.MediaType).To(Equal("text/csv"))
|
||||
Expect(art.Size).To(Equal(int64(len(ordersCSV))))
|
||||
// What the caller holds is a host identifier, never the add-on's handle.
|
||||
Expect(art.Handle).NotTo(HavePrefix("x0"))
|
||||
meta := h.complete("bob", art.Handle)
|
||||
Expect(meta["name"]).To(Equal("Orders.csv"))
|
||||
Expect(meta["size"]).To(BeEquivalentTo(len(ordersCSV)))
|
||||
Expect(meta["expiresAt"]).NotTo(BeEmpty())
|
||||
Expect(meta).NotTo(HaveKey("path"))
|
||||
Expect(meta).NotTo(HaveKey("handle"))
|
||||
|
||||
net := h.listen()
|
||||
resp, body := net.get("/api/addons/artifacts/"+art.Handle+"/download", func(r *http.Request) { h.bearer("bob")(r) })
|
||||
Expect(resp.StatusCode).To(Equal(200), string(body))
|
||||
Expect(string(body)).To(Equal(ordersCSV))
|
||||
Expect(resp.Header.Get("Content-Type")).To(Equal("text/csv"))
|
||||
Expect(resp.Header.Get("Content-Disposition")).To(HavePrefix("attachment;"))
|
||||
Expect(resp.Header.Get("Content-Disposition")).To(ContainSubstring("Orders.csv"))
|
||||
Expect(resp.Header.Get("X-Content-Type-Options")).To(Equal("nosniff"))
|
||||
Expect(resp.Header.Get("Cache-Control")).To(ContainSubstring("no-store"))
|
||||
Expect(resp.ContentLength).To(Equal(int64(len(ordersCSV))))
|
||||
Expect(strings.HasPrefix(string(body), "\xef\xbb\xbf")).To(BeFalse())
|
||||
|
||||
// A person with the feature and no access, and an unknown identifier,
|
||||
// get the same denial.
|
||||
_, unknown := net.get("/api/addons/artifacts/a0123456789abcdef0123456789abcde/download", h.bearer("carol"))
|
||||
resp, stranger := net.get("/api/addons/artifacts/"+art.Handle+"/download", h.bearer("carol"))
|
||||
Expect(resp.StatusCode).To(Equal(403))
|
||||
Expect(string(stranger)).To(Equal(string(unknown)))
|
||||
resp, _ = net.get("/api/addons/artifacts/"+art.Handle, h.bearer("carol"))
|
||||
Expect(resp.StatusCode).To(Equal(403))
|
||||
// Without a credential there is nothing.
|
||||
resp, _ = net.get("/api/addons/artifacts/"+art.Handle+"/download", nil)
|
||||
Expect(resp.StatusCode).To(BeElementOf(401, 403))
|
||||
// A person who cannot read the database cannot start the export.
|
||||
_, e = h.exported("carol", res, `{"tool":"rows-export","table":"Orders"}`)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
})
|
||||
|
||||
It("keeps a shared database's file private, applies the person's own filter and sort, and denies after the grant is withdrawn", func() {
|
||||
h := ready(false)
|
||||
res := h.ordersTable("bob", "ex2")
|
||||
Expect(h.grant("bob", res, "carol", "viewer").code).To(Equal(200))
|
||||
mine, e := h.exported("bob", res, `{"tool":"rows-export","table":"Orders"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.complete("bob", mine.Handle)
|
||||
net := h.listen()
|
||||
// Sharing the source does not share the file.
|
||||
resp, _ := net.get("/api/addons/artifacts/"+mine.Handle+"/download", h.bearer("carol"))
|
||||
Expect(resp.StatusCode).To(Equal(403))
|
||||
|
||||
// A viewer exports what they can read, narrowed by filter, sort and columns.
|
||||
theirs, e := h.exported("carol", res, `{"tool":"rows-export","table":"Orders","columns":["_id","Item","Qty"],"filter":{"column":"Qty","op":"is-not-null"},"sort":[{"column":"Qty"}]}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.complete("carol", theirs.Handle)
|
||||
resp, body := net.get("/api/addons/artifacts/"+theirs.Handle+"/download", h.bearer("carol"))
|
||||
Expect(resp.StatusCode).To(Equal(200))
|
||||
Expect(string(body)).To(MatchRegexp(`^_id,Item,Qty\r\n1,'=SUM\(A1:A9\),-4\r\n2,"Café, ""deluxe""",12\r\n$`))
|
||||
resp, _ = net.get("/api/addons/artifacts/"+theirs.Handle+"/download", h.bearer("bob"))
|
||||
Expect(resp.StatusCode).To(Equal(403))
|
||||
|
||||
// Withdrawing the grant ends the download of what the viewer exported.
|
||||
Expect(h.revoke("bob", res, "carol").code).To(Equal(200))
|
||||
resp, _ = net.get("/api/addons/artifacts/"+theirs.Handle+"/download", h.bearer("carol"))
|
||||
Expect(resp.StatusCode).To(Equal(403))
|
||||
resp, _ = net.get("/api/addons/artifacts/"+theirs.Handle, h.bearer("carol"))
|
||||
Expect(resp.StatusCode).To(Equal(200), "the transfer record stays readable to its principal")
|
||||
// The owner still downloads their own.
|
||||
resp, body = net.get("/api/addons/artifacts/"+mine.Handle+"/download", h.bearer("bob"))
|
||||
Expect(resp.StatusCode).To(Equal(200))
|
||||
Expect(string(body)).To(Equal(ordersCSV))
|
||||
})
|
||||
|
||||
It("serves the same bytes and headers through a forwarding frontend, and fails the same way for a stranger", func() {
|
||||
h := ready(false)
|
||||
res := h.ordersTable("bob", "ex3")
|
||||
art, e := h.exported("bob", res, `{"tool":"rows-export","table":"Orders"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.complete("bob", art.Handle)
|
||||
front := h.frontend()
|
||||
path := "/api/addons/artifacts/" + art.Handle + "/download"
|
||||
get := func(web http.Handler, user string) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("GET", path, nil)
|
||||
h.bearer(user)(r)
|
||||
w := httptest.NewRecorder()
|
||||
web.ServeHTTP(w, r)
|
||||
return w
|
||||
}
|
||||
own, fwd := get(h.web, "bob"), get(front, "bob")
|
||||
Expect(own.Code).To(Equal(200))
|
||||
Expect(fwd.Code).To(Equal(200))
|
||||
Expect(fwd.Body.String()).To(Equal(own.Body.String()))
|
||||
Expect(fwd.Body.String()).To(Equal(ordersCSV))
|
||||
for _, k := range []string{"Content-Type", "Content-Disposition", "X-Content-Type-Options", "Cache-Control", "Content-Length"} {
|
||||
Expect(fwd.Header().Get(k)).To(Equal(own.Header().Get(k)), k)
|
||||
}
|
||||
ownDenied, fwdDenied := get(h.web, "carol"), get(front, "carol")
|
||||
Expect(ownDenied.Code).To(Equal(403))
|
||||
Expect(fwdDenied.Code).To(Equal(403))
|
||||
Expect(fwdDenied.Body.String()).To(Equal(ownDenied.Body.String()))
|
||||
// The export itself is requested through the frontend, too.
|
||||
body := fmt.Sprintf(`{"addonId":"tables","resourceId":%q,"operationId":"via-front-1","generation":1,"action":"read","arguments":{"tool":"rows-export","table":"Orders"}}`, res)
|
||||
got := serve(front, "POST", "/api/addons/dispatch", body, h.bearer("bob"))
|
||||
Expect(got.code).To(Equal(200), got.raw)
|
||||
outs, _ := got.body["outputs"].([]any)
|
||||
Expect(outs).To(HaveLen(1))
|
||||
id := outs[0].(map[string]any)["handle"].(string)
|
||||
h.complete("bob", id)
|
||||
w := httptest.NewRecorder()
|
||||
r := httptest.NewRequest("GET", "/api/addons/artifacts/"+id+"/download", nil)
|
||||
h.bearer("bob")(r)
|
||||
front.ServeHTTP(w, r)
|
||||
Expect(w.Code).To(Equal(200))
|
||||
Expect(w.Body.String()).To(Equal(ordersCSV))
|
||||
})
|
||||
|
||||
It("answers a result past the export limit as data with no file", func() {
|
||||
h := ready(false)
|
||||
res := h.ordersTable("bob", "ex4")
|
||||
out, e := h.app.AddonGateway().Dispatch(context.Background(), h.actor("bob"), contract.Request{
|
||||
AddonID: "tables", ResourceID: res, OperationID: "exp-big", Generation: 1, Action: "read",
|
||||
Arguments: json.RawMessage(`{"tool":"rows-export","table":"Orders","maxRows":5000000}`),
|
||||
})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(out.Outputs).To(BeEmpty())
|
||||
var p map[string]any
|
||||
Expect(json.Unmarshal(out.Payload, &p)).To(Succeed())
|
||||
Expect(p["exported"]).To(Equal(false))
|
||||
Expect(p["error"]).To(Equal("too-large"))
|
||||
// An unknown table and a mutation argument are plain refusals.
|
||||
for _, a := range []string{`{"tool":"rows-export","table":"Nope"}`, `{"tool":"rows-export","table":"Orders","rows":[{}]}`, `{"tool":"rows-export","table":"Orders","bogus":1}`} {
|
||||
_, e = h.read("bob", res, a)
|
||||
Expect(e).To(HaveOccurred(), a)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
// store opens an artifact store over the owner's own policy and runtime, with
|
||||
// a clock and limits the spec controls.
|
||||
func (h *host) store(clock *time.Time, limits artifacts.Limits) (*artifacts.Artifacts, string) {
|
||||
GinkgoHelper()
|
||||
dir := filepath.Join(h.dir, "artifacts-"+uuid.NewString()[:8])
|
||||
a, e := artifacts.Open(artifacts.Options{
|
||||
Dir: dir, Policy: h.app.AddonPolicy(), Runtime: h.app.AddonManager(),
|
||||
Addon: func(context.Context, string) (artifacts.AddonState, error) {
|
||||
return artifacts.AddonState{Installed: true, Enabled: true, Generation: 1}, nil
|
||||
},
|
||||
Limits: limits, Now: func() time.Time { return *clock },
|
||||
})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() { _ = a.Close() })
|
||||
return a, dir
|
||||
}
|
||||
|
||||
// stage asks the real Tables executable for an export and returns the output
|
||||
// description and the request that binds its handle.
|
||||
func (h *host) stage(user, res, op, args string) (contract.OutputDescription, contract.Request) {
|
||||
GinkgoHelper()
|
||||
inst, e := h.app.AddonManager().Runtime(context.Background(), "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
req := contract.Request{AddonID: "tables", ResourceID: res, OperationID: op, Generation: inst.Generation, Action: "read", Arguments: json.RawMessage(args)}
|
||||
out, e := inst.Client.Read(context.Background(), contract.Envelope{Request: req, Authority: contract.Authority{Actor: h.actor(user), Scope: "read", ExpiresAt: time.Now().Add(time.Minute)}})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(out.Outputs).To(HaveLen(1))
|
||||
return out.Outputs[0], req
|
||||
}
|
||||
|
||||
func (h *host) staged() []string {
|
||||
entries, _ := os.ReadDir(filepath.Join(h.owner.Root, "data", "tables", "exports"))
|
||||
var out []string
|
||||
for _, e := range entries {
|
||||
out = append(out, filepath.Join(h.owner.Root, "data", "tables", "exports", e.Name()))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
var _ = Describe("CSV export failures never leave a downloadable file", func() {
|
||||
It("binds an output handle to its operation, person and database", func() {
|
||||
h := ready(false)
|
||||
res := h.ordersTable("bob", "ox1")
|
||||
other := h.ordersTable("carol", "ox1c")
|
||||
out, req := h.stage("bob", res, "ox-op-1", `{"tool":"rows-export","table":"Orders"}`)
|
||||
inst, e := h.app.AddonManager().Runtime(context.Background(), "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
open := func(user string, r contract.Request) error {
|
||||
rc, e := inst.Client.OpenOutput(context.Background(), contract.Envelope{Request: r, Authority: contract.Authority{Actor: h.actor(user), Scope: "output", ExpiresAt: time.Now().Add(time.Minute)}}, out.Handle)
|
||||
if e == nil {
|
||||
_ = rc.Close()
|
||||
}
|
||||
return e
|
||||
}
|
||||
Expect(open("carol", req)).To(HaveOccurred(), "another person")
|
||||
wrong := req
|
||||
wrong.OperationID = "ox-op-2"
|
||||
Expect(open("bob", wrong)).To(HaveOccurred(), "another operation")
|
||||
wrong = req
|
||||
wrong.ResourceID = other
|
||||
Expect(open("bob", wrong)).To(HaveOccurred(), "another database")
|
||||
Expect(h.staged()).To(HaveLen(1), "denied opens leave the file for its owner")
|
||||
rc, e := inst.Client.OpenOutput(context.Background(), contract.Envelope{Request: req, Authority: contract.Authority{Actor: h.actor("bob"), Scope: "output", ExpiresAt: time.Now().Add(time.Minute)}}, out.Handle)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
b, e := io.ReadAll(rc)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(rc.Close()).To(Succeed())
|
||||
Expect(string(b)).To(Equal(ordersCSV))
|
||||
Expect(h.staged()).To(BeEmpty(), "a served export leaves nothing staged")
|
||||
})
|
||||
|
||||
It("fails a truncated, grown or vanished stream and never serves it", func() {
|
||||
h := ready(false)
|
||||
res := h.ordersTable("bob", "ox2")
|
||||
clock := time.Now()
|
||||
files, _ := h.store(&clock, artifacts.DefaultLimits())
|
||||
bob := h.actor("bob")
|
||||
for name, damage := range map[string]func(path string){
|
||||
"truncated": func(p string) { Expect(os.Truncate(p, 10)).To(Succeed()) },
|
||||
"grown": func(p string) {
|
||||
f, e := os.OpenFile(p, os.O_APPEND|os.O_WRONLY, 0)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, _ = f.WriteString("extra,row\r\n")
|
||||
_ = f.Close()
|
||||
},
|
||||
"vanished": func(p string) { Expect(os.Remove(p)).To(Succeed()) },
|
||||
} {
|
||||
out, req := h.stage("bob", res, "ox-"+name, `{"tool":"rows-export","table":"Orders"}`)
|
||||
staged := h.staged()
|
||||
Expect(staged).To(HaveLen(1), name)
|
||||
damage(staged[0])
|
||||
art, e := files.Ingest(context.Background(), bob, req, out)
|
||||
Expect(e).To(HaveOccurred(), name)
|
||||
Expect(art.ID).To(BeEmpty(), name)
|
||||
// Nothing downloadable exists for this principal under any name.
|
||||
var sink bytes.Buffer
|
||||
_, e = files.Download(context.Background(), bob, "a"+strings.Repeat("0", 32), &sink)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(sink.Len()).To(BeZero())
|
||||
for _, p := range h.staged() {
|
||||
_ = os.Remove(p)
|
||||
}
|
||||
}
|
||||
// A healthy export through the same store still works afterwards.
|
||||
out, req := h.stage("bob", res, "ox-good", `{"tool":"rows-export","table":"Orders"}`)
|
||||
art, e := files.Ingest(context.Background(), bob, req, out)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var sink bytes.Buffer
|
||||
_, e = files.Download(context.Background(), bob, art.ID, &sink)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(sink.String()).To(Equal(ordersCSV))
|
||||
})
|
||||
|
||||
It("fails the transfer when the store cannot write, and when the source is revoked before completion", func() {
|
||||
h := ready(false)
|
||||
res := h.ordersTable("bob", "ox3")
|
||||
Expect(h.grant("bob", res, "carol", "viewer").code).To(Equal(200))
|
||||
clock := time.Now()
|
||||
files, root := h.store(&clock, artifacts.DefaultLimits())
|
||||
// A full or read-only disk: every directory of the store refuses a write.
|
||||
var dirs []string
|
||||
_ = filepath.WalkDir(root, func(p string, d fs.DirEntry, e error) error {
|
||||
if e == nil && d.IsDir() {
|
||||
dirs = append(dirs, p)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
for _, d := range dirs {
|
||||
Expect(os.Chmod(d, 0o500)).To(Succeed())
|
||||
}
|
||||
DeferCleanup(func() {
|
||||
for _, d := range dirs {
|
||||
_ = os.Chmod(d, 0o700)
|
||||
}
|
||||
})
|
||||
if os.Geteuid() != 0 {
|
||||
out, req := h.stage("bob", res, "ox3-disk", `{"tool":"rows-export","table":"Orders"}`)
|
||||
_, e := files.Ingest(context.Background(), h.actor("bob"), req, out)
|
||||
Expect(e).To(HaveOccurred())
|
||||
}
|
||||
for _, d := range dirs {
|
||||
Expect(os.Chmod(d, 0o700)).To(Succeed())
|
||||
}
|
||||
for _, p := range h.staged() {
|
||||
_ = os.Remove(p)
|
||||
}
|
||||
|
||||
// Revoked before the transfer starts: nothing is staged for the person.
|
||||
out, req := h.stage("carol", res, "ox3-revoked", `{"tool":"rows-export","table":"Orders"}`)
|
||||
Expect(h.revoke("bob", res, "carol").code).To(Equal(200))
|
||||
_, e := files.Ingest(context.Background(), h.actor("carol"), req, out)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
})
|
||||
|
||||
It("expires the file only, enforces the quota per person and leaves the database as it was", func() {
|
||||
h := ready(false)
|
||||
res := h.ordersTable("bob", "ox4")
|
||||
clock := time.Now()
|
||||
limits := artifacts.DefaultLimits()
|
||||
limits.UserCount, limits.TotalCount = 2, 4
|
||||
files, _ := h.store(&clock, limits)
|
||||
bob := h.actor("bob")
|
||||
ingest := func(op string) (artifacts.Artifact, error) {
|
||||
out, req := h.stage("bob", res, op, `{"tool":"rows-export","table":"Orders"}`)
|
||||
return files.Ingest(context.Background(), bob, req, out)
|
||||
}
|
||||
first, e := ingest("ox4-1")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = ingest("ox4-2")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = ingest("ox4-3")
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
for _, p := range h.staged() {
|
||||
_ = os.Remove(p)
|
||||
}
|
||||
|
||||
before, e := h.read("bob", res, `{"tool":"rows-query","table":"Orders"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
clock = clock.Add(limits.Expiry + time.Minute)
|
||||
var sink bytes.Buffer
|
||||
_, e = files.Download(context.Background(), bob, first.ID, &sink)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
Expect(sink.Len()).To(BeZero())
|
||||
_, e = files.Sweep(context.Background())
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = files.Status(context.Background(), bob, first.ID)
|
||||
Expect(e).To(HaveOccurred())
|
||||
|
||||
// The database is untouched and exports again; the quota is free.
|
||||
after, e := h.read("bob", res, `{"tool":"rows-query","table":"Orders"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(after["rows"]).To(Equal(before["rows"]))
|
||||
Expect(after["revision"]).To(Equal(before["revision"]))
|
||||
again, e := ingest("ox4-4")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
sink.Reset()
|
||||
_, e = files.Download(context.Background(), bob, again.ID, &sink)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(sink.String()).To(Equal(ordersCSV))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,485 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesadmin_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/ed25519"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"math/big"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
goruntime "runtime"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/labstack/echo/v4"
|
||||
"github.com/mudler/LocalAI/core/application"
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
httpapi "github.com/mudler/LocalAI/core/http"
|
||||
"github.com/mudler/LocalAI/core/http/auth"
|
||||
"github.com/mudler/LocalAI/core/http/routes"
|
||||
addonsvc "github.com/mudler/LocalAI/core/services/addons"
|
||||
"github.com/mudler/LocalAI/core/services/addons/catalog"
|
||||
"github.com/mudler/LocalAI/core/services/addons/gateway"
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
"github.com/mudler/LocalAI/pkg/system"
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestTablesAdmin(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Tables resource management")
|
||||
}
|
||||
|
||||
var binary string
|
||||
|
||||
var _ = BeforeSuite(func() {
|
||||
binary = filepath.Join(GinkgoT().TempDir(), "tables")
|
||||
cmd := exec.Command("go", "build", "-o", binary, "../../../addons/tables/cmd/tables")
|
||||
cmd.Env = append(os.Environ(), "CGO_ENABLED=1")
|
||||
out, e := cmd.CombinedOutput()
|
||||
Expect(e).NotTo(HaveOccurred(), string(out))
|
||||
})
|
||||
|
||||
const appSecret = "tables-admin-secret"
|
||||
|
||||
// transports builds the transport identities of one owner and one forwarding
|
||||
// frontend: a CA, two pinned leaves and two signing keys.
|
||||
func transports(port int) (owner, front *gateway.CarrierConfig) {
|
||||
GinkgoHelper()
|
||||
pub, key, e := ed25519.GenerateKey(rand.Reader)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
ca := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "test CA"}, NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour), IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign}
|
||||
der, e := x509.CreateCertificate(rand.Reader, ca, ca, pub, key)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
ca, e = x509.ParseCertificate(der)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
pool := x509.NewCertPool()
|
||||
pool.AddCert(ca)
|
||||
pins := map[string][32]byte{}
|
||||
verify := map[string]ed25519.PublicKey{}
|
||||
pairs := map[string]tls.Certificate{}
|
||||
signing := map[string]ed25519.PrivateKey{}
|
||||
for i, id := range []string{"owner", "front-a"} {
|
||||
lp, lk, e := ed25519.GenerateKey(rand.Reader)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
leaf := &x509.Certificate{SerialNumber: big.NewInt(int64(i + 2)), Subject: pkix.Name{CommonName: id}, NotBefore: ca.NotBefore, NotAfter: ca.NotAfter, IPAddresses: []net.IP{net.ParseIP("127.0.0.1")}, KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth}}
|
||||
d, e := x509.CreateCertificate(rand.Reader, leaf, ca, lp, key)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
kd, e := x509.MarshalPKCS8PrivateKey(lk)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
pair, e := tls.X509KeyPair(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: d}), pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: kd}))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
sp, sk, e := ed25519.GenerateKey(rand.Reader)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
pins[id], verify[id], pairs[id], signing[id] = sha256.Sum256(d), sp, pair, sk
|
||||
}
|
||||
address := fmt.Sprintf("https://127.0.0.1:%d", port)
|
||||
mk := func(id string) *gateway.CarrierConfig {
|
||||
return &gateway.CarrierConfig{InstanceID: id, OwnerID: "owner", OwnerAddress: address, Certificate: pairs[id], Roots: pool, Pins: pins, SigningKey: signing[id], VerificationKeys: verify}
|
||||
}
|
||||
return mk("owner"), mk("front-a")
|
||||
}
|
||||
|
||||
func freePort() int {
|
||||
GinkgoHelper()
|
||||
l, e := net.Listen("tcp", "127.0.0.1:0")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
defer func() { _ = l.Close() }()
|
||||
return l.Addr().(*net.TCPAddr).Port
|
||||
}
|
||||
|
||||
// person is an account of the fixture.
|
||||
type person struct {
|
||||
id, role string
|
||||
feature bool
|
||||
status string
|
||||
}
|
||||
|
||||
var people = []person{
|
||||
{"alice", auth.RoleAdmin, false, auth.StatusActive},
|
||||
{"bob", auth.RoleUser, true, auth.StatusActive},
|
||||
{"carol", auth.RoleUser, true, auth.StatusActive},
|
||||
{"dave", auth.RoleUser, true, auth.StatusActive},
|
||||
{"erin", auth.RoleUser, false, auth.StatusActive},
|
||||
{"frank", auth.RoleUser, true, auth.StatusPending},
|
||||
}
|
||||
|
||||
// host is one owner in process: the real application bootstrap over real
|
||||
// account storage, the real supervisor and the real Tables executable.
|
||||
type host struct {
|
||||
dir string
|
||||
owner config.AddonConfig
|
||||
front *gateway.CarrierConfig
|
||||
app *application.Application
|
||||
web *echo.Echo
|
||||
cancel context.CancelFunc
|
||||
local bool
|
||||
tokens map[string]string
|
||||
sessions map[string]string
|
||||
keys map[string]string
|
||||
}
|
||||
|
||||
func newHost(local bool) *host {
|
||||
GinkgoHelper()
|
||||
dir := GinkgoT().TempDir()
|
||||
h := &host{dir: dir, local: local, tokens: map[string]string{}, sessions: map[string]string{}, keys: map[string]string{}}
|
||||
DeferCleanup(func() {
|
||||
_ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, e error) error {
|
||||
if e == nil && d.IsDir() {
|
||||
_ = os.Chmod(p, 0o700)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
})
|
||||
port := freePort()
|
||||
var owner *gateway.CarrierConfig
|
||||
owner, h.front = transports(port)
|
||||
h.owner = config.AddonConfig{
|
||||
Enabled: true, InstanceID: "owner", OwnerID: "owner", OwnerAddress: owner.OwnerAddress, StorageID: "local",
|
||||
Root: filepath.Join(dir, "addons"), ListenAddress: fmt.Sprintf("127.0.0.1:%d", port), Transport: owner,
|
||||
Host: contract.HostCompatibility{Host: 1, API: 1, Protocol: 1, SDK: 1, DataFormat: 1, OS: goruntime.GOOS, Arch: goruntime.GOARCH},
|
||||
}
|
||||
if local {
|
||||
h.owner.OwnershipMode = auth.AddonModeLocalOwner
|
||||
}
|
||||
return h
|
||||
}
|
||||
|
||||
func (h *host) mode() string {
|
||||
if h.local {
|
||||
return auth.AddonModeLocalOwner
|
||||
}
|
||||
return auth.AddonModeNamedUser
|
||||
}
|
||||
|
||||
func (h *host) start() {
|
||||
GinkgoHelper()
|
||||
models, backends := filepath.Join(h.dir, "models"), filepath.Join(h.dir, "backends")
|
||||
Expect(os.MkdirAll(models, 0o750)).To(Succeed())
|
||||
Expect(os.MkdirAll(backends, 0o750)).To(Succeed())
|
||||
state, e := system.GetSystemState(system.WithBackendPath(backends), system.WithModelPath(models))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
h.cancel = cancel
|
||||
opts := []config.AppOption{config.WithContext(ctx), config.WithSystemState(state), config.WithDataPath(h.dir), config.WithAddons(h.owner)}
|
||||
if !h.local {
|
||||
opts = append(opts, config.WithAuthEnabled(true), config.WithAuthDatabaseURL(filepath.Join(h.dir, "auth.db")), config.WithAuthAPIKeyHMACSecret(appSecret))
|
||||
}
|
||||
app, e := application.New(opts...)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.app = app
|
||||
h.web, e = httpapi.API(app)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
}
|
||||
|
||||
// stop ends the application as a restart does and waits until the owner
|
||||
// released its durable state.
|
||||
func (h *host) stop() {
|
||||
GinkgoHelper()
|
||||
if h.cancel == nil {
|
||||
return
|
||||
}
|
||||
Expect(h.app.Shutdown()).To(Succeed())
|
||||
h.cancel()
|
||||
h.cancel = nil
|
||||
mode := policy.ModeNamedUser
|
||||
if h.local {
|
||||
mode = policy.ModeLocalOwner
|
||||
}
|
||||
Eventually(func() error {
|
||||
s, e := policy.OpenStore(filepath.Join(h.owner.Root, "policy"), mode)
|
||||
if e == nil {
|
||||
_ = s.Close()
|
||||
}
|
||||
return e
|
||||
}, 10*time.Second, 50*time.Millisecond).Should(Succeed())
|
||||
}
|
||||
|
||||
func (h *host) actor(user string) contract.Actor {
|
||||
if h.local {
|
||||
return contract.Actor{UserID: auth.AddonLocalOwnerID, CredentialID: auth.AddonLocalOwnerID, Mode: auth.AddonModeLocalOwner}
|
||||
}
|
||||
return contract.Actor{UserID: user, CredentialID: "key:" + h.keys[user], Mode: auth.AddonModeNamedUser}
|
||||
}
|
||||
|
||||
// seed creates the accounts, installs the real Tables package from a local
|
||||
// catalog and waits until it runs.
|
||||
func (h *host) seed(lib *cluster.Library) {
|
||||
GinkgoHelper()
|
||||
ctx := context.Background()
|
||||
if !h.local {
|
||||
db := h.app.AuthDB()
|
||||
Expect(db).NotTo(BeNil())
|
||||
for _, u := range people {
|
||||
Expect(db.Create(&auth.User{ID: u.id, Email: u.id + "@tables.test", Name: "Name of " + u.id, Provider: auth.ProviderLocal, Role: u.role, Status: u.status}).Error).To(Succeed())
|
||||
if u.feature {
|
||||
Expect(db.Create(&auth.UserPermission{ID: uuid.NewString(), UserID: u.id, Permissions: auth.PermissionMap{auth.FeatureAddons: true}}).Error).To(Succeed())
|
||||
}
|
||||
token, key, e := auth.CreateAPIKey(db, u.id, "agent", u.role, appSecret, nil)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
session, e := auth.CreateSession(db, u.id, appSecret)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.tokens[u.id], h.sessions[u.id], h.keys[u.id] = token, session, key.ID
|
||||
}
|
||||
}
|
||||
admin := h.actor("alice")
|
||||
Expect(h.app.AddonSources().Save(ctx, admin, catalog.Source{ID: "local", Name: "Local", URL: lib.URL(), Enabled: true, AllowLoopbackHTTP: true})).To(Succeed())
|
||||
_, e := h.app.AddonSources().Refresh(ctx, "local")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m := h.app.AddonManager()
|
||||
_, e = m.Submit(ctx, admin, addonsvc.LifecycleCommand{OperationID: "install-tables", AddonID: "tables", Action: "install", SourceID: "local", Version: "0.1.0"})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = m.Submit(ctx, admin, addonsvc.LifecycleCommand{OperationID: "enable-tables", AddonID: "tables", Action: "enable", Generation: 1})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.running()
|
||||
}
|
||||
|
||||
func (h *host) running() {
|
||||
GinkgoHelper()
|
||||
Eventually(func() string {
|
||||
list, e := h.app.AddonManager().Apps(context.Background(), h.actor("alice"))
|
||||
if e != nil {
|
||||
return e.Error()
|
||||
}
|
||||
for _, a := range list {
|
||||
if a.ID == "tables" {
|
||||
return a.State
|
||||
}
|
||||
}
|
||||
return "absent"
|
||||
}, 30*time.Second, 100*time.Millisecond).Should(Equal("running"))
|
||||
}
|
||||
|
||||
func (h *host) library() *cluster.Library {
|
||||
GinkgoHelper()
|
||||
lib, e := cluster.NewLibrary(binary)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
DeferCleanup(lib.Close)
|
||||
raw, e := os.ReadFile("../../../addons/tables/manifest.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var m struct {
|
||||
Capabilities map[string]any `json:"capabilities"`
|
||||
}
|
||||
Expect(json.Unmarshal(raw, &m)).To(Succeed())
|
||||
// These specs run the tools, not the page: the package carries no UI
|
||||
// files, so the declaration of them is left out. The UI suite installs
|
||||
// the whole package.
|
||||
delete(m.Capabilities, "ui")
|
||||
exe, e := os.ReadFile(binary)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(lib.Add(cluster.Package{ID: "tables", Version: "0.1.0", Capabilities: m.Capabilities, Executable: "bin/tables", Binary: exe}, GinkgoT().TempDir())).To(Succeed())
|
||||
return lib
|
||||
}
|
||||
|
||||
// ready is the usual start of a spec: a running host with the accounts and
|
||||
// the package.
|
||||
func ready(local bool) *host {
|
||||
GinkgoHelper()
|
||||
h := newHost(local)
|
||||
lib := h.library()
|
||||
h.start()
|
||||
DeferCleanup(h.stop)
|
||||
h.seed(lib)
|
||||
return h
|
||||
}
|
||||
|
||||
// reply is one HTTP answer.
|
||||
type reply struct {
|
||||
code int
|
||||
body map[string]any
|
||||
raw string
|
||||
}
|
||||
|
||||
func (r reply) errType() string {
|
||||
e, _ := r.body["error"].(map[string]any)
|
||||
s, _ := e["type"].(string)
|
||||
return s
|
||||
}
|
||||
|
||||
func (r reply) errOperation() string {
|
||||
e, _ := r.body["error"].(map[string]any)
|
||||
s, _ := e["operationId"].(string)
|
||||
return s
|
||||
}
|
||||
|
||||
func serve(e *echo.Echo, method, path, body string, headers ...func(*http.Request)) reply {
|
||||
GinkgoHelper()
|
||||
r := httptest.NewRequest(method, path, bytes.NewBufferString(body))
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
for _, h := range headers {
|
||||
h(r)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
e.ServeHTTP(w, r)
|
||||
out := reply{code: w.Code, body: map[string]any{}, raw: w.Body.String()}
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &out.body)
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *host) send(method, path, body string, headers ...func(*http.Request)) reply {
|
||||
GinkgoHelper()
|
||||
return serve(h.web, method, path, body, headers...)
|
||||
}
|
||||
|
||||
func (h *host) bearer(user string) func(*http.Request) {
|
||||
return func(r *http.Request) { r.Header.Set("Authorization", "Bearer "+h.tokens[user]) }
|
||||
}
|
||||
|
||||
// tab is what the host's own pages send: the user's browser session, a
|
||||
// same-origin fetch and the identifier the action concerns in the header. In
|
||||
// local-owner mode there is no session.
|
||||
func (h *host) tab(user, id string) func(*http.Request) {
|
||||
return func(r *http.Request) {
|
||||
if !h.local {
|
||||
r.AddCookie(&http.Cookie{Name: "session", Value: h.sessions[user]})
|
||||
}
|
||||
r.Header.Set("Sec-Fetch-Site", "same-origin")
|
||||
r.Header.Set("Sec-Fetch-Mode", "cors")
|
||||
r.Header.Set("Sec-Fetch-Dest", "empty")
|
||||
r.Header.Set("X-LocalAI-Confirm", id)
|
||||
}
|
||||
}
|
||||
|
||||
// session is a plain same-origin session request with no confirmation header.
|
||||
func (h *host) session(user string) func(*http.Request) {
|
||||
return func(r *http.Request) {
|
||||
r.AddCookie(&http.Cookie{Name: "session", Value: h.sessions[user]})
|
||||
r.Header.Set("Sec-Fetch-Site", "same-origin")
|
||||
r.Header.Set("Sec-Fetch-Mode", "cors")
|
||||
r.Header.Set("Sec-Fetch-Dest", "empty")
|
||||
}
|
||||
}
|
||||
|
||||
// frontend is a second LocalAI instance that holds no addon state and
|
||||
// forwards every resource call to the owner over the real carrier.
|
||||
func (h *host) frontend() *echo.Echo {
|
||||
GinkgoHelper()
|
||||
carrier, e := gateway.NewCarrier(*h.front)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var db = h.app.AuthDB()
|
||||
g := &gateway.Gateway{Forward: carrier, Revalidate: auth.AddonRevalidate(db, h.mode())}
|
||||
web := echo.New()
|
||||
web.Use(auth.Middleware(db, h.app.ApplicationConfig()))
|
||||
routes.RegisterAddonRoutesFor(web, g, db, h.mode())
|
||||
return web
|
||||
}
|
||||
|
||||
// Typed calls of the management routes, as the person's own browser tab.
|
||||
|
||||
func (h *host) create(user, op, name string) reply {
|
||||
GinkgoHelper()
|
||||
body := fmt.Sprintf(`{"addonId":"tables","operationId":%q,"generation":1,"arguments":{"name":%q}}`, op, name)
|
||||
return h.send("POST", "/api/addons/resources", body, h.tab(user, op))
|
||||
}
|
||||
|
||||
func (h *host) list(user string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("GET", "/api/addons/resources?addonId=tables", "", h.bearer(user))
|
||||
}
|
||||
|
||||
// resources returns the listed resources by ID.
|
||||
func resourcesOf(r reply) map[string]map[string]any {
|
||||
GinkgoHelper()
|
||||
out := map[string]map[string]any{}
|
||||
list, _ := r.body["resources"].([]any)
|
||||
for _, x := range list {
|
||||
m := x.(map[string]any)
|
||||
out[m["id"].(string)] = m
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *host) grant(user, res, target, role string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("PUT", "/api/addons/resources/"+res+"/grants/"+target, fmt.Sprintf(`{"role":%q}`, role), h.tab(user, res))
|
||||
}
|
||||
|
||||
func (h *host) revoke(user, res, target string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("DELETE", "/api/addons/resources/"+res+"/grants/"+target, "", h.tab(user, res))
|
||||
}
|
||||
|
||||
func (h *host) grants(user, res string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("GET", "/api/addons/resources/"+res+"/grants", "", h.tab(user, res))
|
||||
}
|
||||
|
||||
func (h *host) users(user, res, q string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("GET", "/api/addons/resources/"+res+"/users?q="+q, "", h.tab(user, res))
|
||||
}
|
||||
|
||||
func (h *host) rename(user, res, op, name string, rev map[string]any) reply {
|
||||
GinkgoHelper()
|
||||
b, _ := json.Marshal(map[string]any{"operationId": op, "generation": 1, "revision": rev, "name": name})
|
||||
return h.send("POST", "/api/addons/resources/"+res+"/rename", string(b), h.tab(user, res))
|
||||
}
|
||||
|
||||
func (h *host) remove(user, res, op, confirm string, rev map[string]any) reply {
|
||||
GinkgoHelper()
|
||||
b, _ := json.Marshal(map[string]any{"operationId": op, "generation": 1, "revision": rev, "confirm": confirm})
|
||||
return h.send("POST", "/api/addons/resources/"+res+"/delete", string(b), h.tab(user, res))
|
||||
}
|
||||
|
||||
// revision reads the revision the owner role is shown for a resource.
|
||||
func (h *host) revision(user, res string) map[string]any {
|
||||
GinkgoHelper()
|
||||
got := resourcesOf(h.list(user))[res]
|
||||
Expect(got).NotTo(BeNil())
|
||||
rev, _ := got["revision"].(map[string]any)
|
||||
Expect(rev).NotTo(BeNil())
|
||||
return rev
|
||||
}
|
||||
|
||||
// name reads the display name of a resource through the owner's gateway, as
|
||||
// the person who may read it.
|
||||
func (h *host) name(user, res string) (string, error) {
|
||||
out, e := h.app.AddonGateway().Dispatch(context.Background(), h.actor(user), contract.Request{
|
||||
AddonID: "tables", ResourceID: res, OperationID: "inspect-" + uuid.NewString()[:8], Generation: 1, Action: "read",
|
||||
Arguments: json.RawMessage(`{"tool":"databases-inspect"}`),
|
||||
})
|
||||
if e != nil {
|
||||
return "", e
|
||||
}
|
||||
var v struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
Expect(json.Unmarshal(out.Payload, &v)).To(Succeed())
|
||||
return v.Name, nil
|
||||
}
|
||||
|
||||
// database creates a database for the user and returns its ID.
|
||||
func (h *host) database(user, op, name string) string {
|
||||
GinkgoHelper()
|
||||
got := h.create(user, op, name)
|
||||
Expect(got.code).To(Equal(200), got.raw)
|
||||
id, _ := got.body["id"].(string)
|
||||
Expect(id).NotTo(BeEmpty())
|
||||
return id
|
||||
}
|
||||
|
||||
func (h *host) key(res string) policy.ResourceKey {
|
||||
return policy.ResourceKey{AddonID: "tables", ResourceID: res}
|
||||
}
|
||||
|
||||
func (h *host) dbFile(res string) string {
|
||||
return filepath.Join(h.owner.Root, "data", "tables", "tables", "databases", res+".db")
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesadmin_test
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
|
||||
"github.com/labstack/echo/v4"
|
||||
"github.com/mudler/LocalAI/core/http/auth"
|
||||
"github.com/mudler/LocalAI/core/http/routes"
|
||||
"github.com/mudler/LocalAI/core/services/addons/gateway"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// deadFrontend is a frontend whose owner address is a port nobody listens on.
|
||||
func (h *host) deadFrontend() *echo.Echo {
|
||||
GinkgoHelper()
|
||||
cfg := *h.front
|
||||
cfg.OwnerAddress = fmt.Sprintf("https://127.0.0.1:%d", freePort())
|
||||
carrier, e := gateway.NewCarrier(cfg)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
db := h.app.AuthDB()
|
||||
g := &gateway.Gateway{Forward: carrier, Revalidate: auth.AddonRevalidate(db, h.mode())}
|
||||
web := echo.New()
|
||||
web.Use(auth.Middleware(db, h.app.ApplicationConfig()))
|
||||
routes.RegisterAddonRoutesFor(web, g, db, h.mode())
|
||||
return web
|
||||
}
|
||||
|
||||
var _ = Describe("replicas: a forwarding frontend gives the owner's own answers", func() {
|
||||
It("answers every typed denial exactly as the owner does and carries every mutation to the owner", func() {
|
||||
h := ready(false)
|
||||
front := h.frontend()
|
||||
res := h.database("bob", "f-1", "Replica")
|
||||
Expect(h.grant("bob", res, "carol", "viewer").code).To(Equal(200))
|
||||
rev := h.revision("bob", res)
|
||||
|
||||
type call struct {
|
||||
name, method, path, body string
|
||||
header func(*http.Request)
|
||||
}
|
||||
tab := func(user, id string) func(*http.Request) { return h.tab(user, id) }
|
||||
calls := []call{
|
||||
{"list", "GET", "/api/addons/resources?addonId=tables", "", h.bearer("bob")},
|
||||
{"list of a viewer", "GET", "/api/addons/resources?addonId=tables", "", h.bearer("carol")},
|
||||
{"list of an administrator", "GET", "/api/addons/resources?addonId=tables", "", h.bearer("alice")},
|
||||
|
||||
{"malformed add-on", "GET", "/api/addons/resources?addonId=Bad%20ID", "", h.bearer("bob")},
|
||||
{"grants", "GET", "/api/addons/resources/" + res + "/grants", "", tab("bob", res)},
|
||||
{"grants of a viewer", "GET", "/api/addons/resources/" + res + "/grants", "", tab("carol", res)},
|
||||
{"users", "GET", "/api/addons/resources/" + res + "/users?q=Name%20of%20d", "", tab("bob", res)},
|
||||
{"short query", "GET", "/api/addons/resources/" + res + "/users?q=d", "", tab("bob", res)},
|
||||
{"owner role grant", "PUT", "/api/addons/resources/" + res + "/grants/dave", `{"role":"owner"}`, tab("bob", res)},
|
||||
{"grant by a viewer", "PUT", "/api/addons/resources/" + res + "/grants/dave", `{"role":"viewer"}`, tab("carol", res)},
|
||||
{"grant to a pending account", "PUT", "/api/addons/resources/" + res + "/grants/frank", `{"role":"viewer"}`, tab("bob", res)},
|
||||
{"rename stale", "POST", "/api/addons/resources/" + res + "/rename", `{"operationId":"pr-1","generation":1,"revision":{"schema":77,"data":0},"name":"X"}`, tab("bob", res)},
|
||||
{"rename stale generation", "POST", "/api/addons/resources/" + res + "/rename", fmt.Sprintf(`{"operationId":"pr-2","generation":5,"revision":{"schema":%v,"data":%v},"name":"X"}`, rev["schema"], rev["data"]), tab("bob", res)},
|
||||
{"rename by a key", "POST", "/api/addons/resources/" + res + "/rename", `{"operationId":"pr-3","generation":1,"revision":{"schema":1,"data":0},"name":"X"}`, h.bearer("bob")},
|
||||
{"delete wrong confirmation", "POST", "/api/addons/resources/" + res + "/delete", fmt.Sprintf(`{"operationId":"pd-1","generation":1,"revision":{"schema":%v,"data":%v},"confirm":"nope"}`, rev["schema"], rev["data"]), tab("bob", res)},
|
||||
{"delete by a viewer", "POST", "/api/addons/resources/" + res + "/delete", `{"operationId":"pd-2","generation":1,"revision":{"schema":1,"data":0},"confirm":"Replica"}`, tab("carol", res)},
|
||||
{"status of a viewer", "GET", "/api/addons/resources/" + res + "/operations/pr-1?kind=rename", "", tab("carol", res)},
|
||||
{"selection none", "GET", "/api/addons/conversations/chat-1/selection/tables", "", h.bearer("bob")},
|
||||
{"selection of an unknown database", "PUT", "/api/addons/conversations/chat-1/selection/tables", `{"resourceId":"r0123456789abcdef0123456789abcde"}`, h.bearer("bob")},
|
||||
{"selection malformed", "PUT", "/api/addons/conversations/bad%20conv/selection/tables", `{"resourceId":"x"}`, h.bearer("bob")},
|
||||
}
|
||||
for _, c := range calls {
|
||||
own := serve(h.web, c.method, c.path, c.body, c.header)
|
||||
fwd := serve(front, c.method, c.path, c.body, c.header)
|
||||
Expect(fwd.code).To(Equal(own.code), c.name+" owner "+own.raw+" frontend "+fwd.raw)
|
||||
Expect(fwd.errType()).To(Equal(own.errType()), c.name)
|
||||
Expect(fwd.body).To(Equal(own.body), c.name)
|
||||
}
|
||||
|
||||
// Forged forwarding headers change nothing on either.
|
||||
forged := func(r *http.Request) {
|
||||
h.bearer("carol")(r)
|
||||
r.Header.Set("X-Forwarded-Actor", "bob")
|
||||
r.Header.Set("X-Addon-Actor", "bob")
|
||||
}
|
||||
got := serve(front, "GET", "/api/addons/resources?addonId=tables", "", forged)
|
||||
Expect(got.code).To(Equal(200))
|
||||
Expect(resourcesOf(got)[res]["role"]).To(Equal("viewer"))
|
||||
|
||||
// Mutations made through the frontend are the owner's.
|
||||
made := serve(front, "POST", "/api/addons/resources", `{"addonId":"tables","operationId":"fr-1","generation":1,"arguments":{"name":"Via frontend"}}`, h.tab("dave", "fr-1"))
|
||||
Expect(made.code).To(Equal(200), made.raw)
|
||||
id := made.body["id"].(string)
|
||||
Expect(resourcesOf(h.list("dave"))).To(HaveKey(id))
|
||||
owned := h.revision("dave", id)
|
||||
b := fmt.Sprintf(`{"operationId":"fr-2","generation":1,"revision":{"schema":%v,"data":%v},"name":"Via frontend 2"}`, owned["schema"], owned["data"])
|
||||
Expect(serve(front, "POST", "/api/addons/resources/"+id+"/rename", b, h.tab("dave", id)).code).To(Equal(200))
|
||||
name, e := h.name("dave", id)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Via frontend 2"))
|
||||
Expect(serve(front, "PUT", "/api/addons/resources/"+id+"/grants/bob", `{"role":"editor"}`, h.tab("dave", id)).code).To(Equal(200))
|
||||
Expect(resourcesOf(h.list("bob"))[id]["role"]).To(Equal("editor"))
|
||||
Expect(serve(front, "DELETE", "/api/addons/resources/"+id+"/grants/bob", "", h.tab("dave", id)).code).To(Equal(200))
|
||||
Expect(resourcesOf(h.list("bob"))).NotTo(HaveKey(id))
|
||||
sel := serve(front, "PUT", "/api/addons/conversations/fr-chat/selection/tables", fmt.Sprintf(`{"resourceId":%q}`, id), h.bearer("dave"))
|
||||
Expect(sel.code).To(Equal(200), sel.raw)
|
||||
Expect(h.selection("dave", "fr-chat").body["resourceId"]).To(Equal(id))
|
||||
d := fmt.Sprintf(`{"operationId":"fr-3","generation":1,"revision":{"schema":%v,"data":%v},"confirm":"Via frontend 2"}`, owned["schema"], owned["data"])
|
||||
del := serve(front, "POST", "/api/addons/resources/"+id+"/delete", d, h.tab("dave", id))
|
||||
Expect(del.code).To(Equal(200), del.raw)
|
||||
Expect(resourcesOf(h.list("dave"))).NotTo(HaveKey(id))
|
||||
})
|
||||
|
||||
It("reports an unknown outcome with the original operation ID and leaves reads plainly unavailable when the owner cannot be reached", func() {
|
||||
h := ready(false)
|
||||
res := h.database("bob", "u-1", "Lost")
|
||||
front := h.deadFrontend()
|
||||
rev := h.revision("bob", res)
|
||||
|
||||
for name, c := range map[string]struct {
|
||||
method, path, body string
|
||||
header func(*http.Request)
|
||||
op string
|
||||
}{
|
||||
"create": {"POST", "/api/addons/resources", `{"addonId":"tables","operationId":"lost-create","generation":1,"arguments":{"name":"N"}}`, h.tab("bob", "lost-create"), "lost-create"},
|
||||
"rename": {"POST", "/api/addons/resources/" + res + "/rename", fmt.Sprintf(`{"operationId":"lost-rename","generation":1,"revision":{"schema":%v,"data":%v},"name":"N"}`, rev["schema"], rev["data"]), h.tab("bob", res), "lost-rename"},
|
||||
"delete": {"POST", "/api/addons/resources/" + res + "/delete", fmt.Sprintf(`{"operationId":"lost-delete","generation":1,"revision":{"schema":%v,"data":%v},"confirm":"Lost"}`, rev["schema"], rev["data"]), h.tab("bob", res), "lost-delete"},
|
||||
"grant": {"PUT", "/api/addons/resources/" + res + "/grants/carol", `{"role":"viewer","operationId":"lost-grant"}`, h.tab("bob", res), "lost-grant"},
|
||||
"revoke": {"DELETE", "/api/addons/resources/" + res + "/grants/carol?operationId=lost-revoke", "", h.tab("bob", res), "lost-revoke"},
|
||||
} {
|
||||
got := serve(front, c.method, c.path, c.body, c.header)
|
||||
Expect(got.code).To(Equal(http.StatusServiceUnavailable), name+" "+got.raw)
|
||||
Expect(got.errType()).To(Equal("outcome_unknown"), name)
|
||||
Expect(got.errOperation()).To(Equal(c.op), name)
|
||||
}
|
||||
// Reads and the idempotent selection are plainly unavailable.
|
||||
for name, got := range map[string]reply{
|
||||
"list": serve(front, "GET", "/api/addons/resources?addonId=tables", "", h.bearer("bob")),
|
||||
"grants": serve(front, "GET", "/api/addons/resources/"+res+"/grants", "", h.tab("bob", res)),
|
||||
"selection": serve(front, "PUT", "/api/addons/conversations/c-1/selection/tables", fmt.Sprintf(`{"resourceId":%q}`, res), h.bearer("bob")),
|
||||
} {
|
||||
Expect(got.code).To(Equal(http.StatusServiceUnavailable), name)
|
||||
Expect(got.errType()).To(Equal("addon_error"), name)
|
||||
Expect(got.errOperation()).To(BeEmpty(), name)
|
||||
}
|
||||
// The owner did none of it.
|
||||
Expect(resourcesOf(h.list("bob"))).To(HaveLen(1))
|
||||
Expect(h.grants("bob", res).body["grants"]).To(BeEmpty())
|
||||
name, e := h.name("bob", res)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Lost"))
|
||||
})
|
||||
|
||||
It("answers the sharing denial of local-owner mode the same from a frontend", func() {
|
||||
h := ready(true)
|
||||
front := h.frontend()
|
||||
res := h.database("local-owner", "lf-1", "Local")
|
||||
for _, path := range []string{"/grants", "/users?q=ab"} {
|
||||
own := h.send("GET", "/api/addons/resources/"+res+path, "", h.tab("x", res))
|
||||
fwd := serve(front, "GET", "/api/addons/resources/"+res+path, "", h.tab("x", res))
|
||||
Expect(own.code).To(Equal(http.StatusConflict), own.raw)
|
||||
Expect(fwd.code).To(Equal(own.code), fwd.raw)
|
||||
Expect(fwd.errType()).To(Equal("sharing_disabled"))
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,286 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesadmin_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"sync"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("UI-only resource management of the owner", func() {
|
||||
It("creates private databases by the person's own action, once per operation, and lists them only for their owner", func() {
|
||||
h := ready(false)
|
||||
|
||||
// A key, an unmarked request and a tool marker are never the person.
|
||||
body := `{"addonId":"tables","operationId":"make-1","generation":1,"arguments":{"name":"Budget"}}`
|
||||
Expect(h.send("POST", "/api/addons/resources", body, h.bearer("bob")).code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.send("POST", "/api/addons/resources", body, h.session("bob")).code).To(Equal(http.StatusForbidden))
|
||||
marked := func(r *http.Request) { h.tab("bob", "make-1")(r); r.Header.Set("X-LocalAI-Tool-Call", "1") }
|
||||
Expect(h.send("POST", "/api/addons/resources", body, marked).code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.send("POST", "/api/addons/resources", body, h.tab("bob", "another-op")).code).To(Equal(http.StatusForbidden))
|
||||
Expect(resourcesOf(h.list("bob"))).To(BeEmpty())
|
||||
|
||||
got := h.create("bob", "make-1", "Budget")
|
||||
Expect(got.code).To(Equal(http.StatusOK), got.raw)
|
||||
id := got.body["id"].(string)
|
||||
Expect(got.body["role"]).To(Equal("owner"))
|
||||
Expect(got.body["name"]).To(Equal("Budget"))
|
||||
Expect(got.body["sharedWith"]).To(BeEquivalentTo(0))
|
||||
name, e := h.name("bob", id)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Budget"))
|
||||
|
||||
// The same operation is the same database, whoever repeats it.
|
||||
again := h.create("bob", "make-1", "Budget")
|
||||
Expect(again.code).To(Equal(http.StatusOK), again.raw)
|
||||
Expect(again.body["id"]).To(Equal(id))
|
||||
// The same operation with another request is refused.
|
||||
other := h.create("bob", "make-1", "Other name")
|
||||
Expect(other.code).To(Equal(http.StatusForbidden), other.raw)
|
||||
keys, e := h.app.AddonPolicy().Accessible(context.Background(), h.actor("bob"), "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(keys).To(HaveLen(1))
|
||||
// Another person's operation ID is their own namespace.
|
||||
other = h.create("carol", "make-1", "Carol's")
|
||||
Expect(other.code).To(Equal(http.StatusOK), other.raw)
|
||||
Expect(other.body["id"]).NotTo(Equal(id))
|
||||
|
||||
// Private by default: nobody else lists it, installation
|
||||
// administration included, and the listing shows no other user's data.
|
||||
for _, user := range []string{"alice", "dave"} {
|
||||
got := h.list(user)
|
||||
Expect(got.code).To(Equal(http.StatusOK), got.raw)
|
||||
Expect(resourcesOf(got)).NotTo(HaveKey(id), user)
|
||||
}
|
||||
mine := resourcesOf(h.list("bob"))
|
||||
Expect(mine).To(HaveLen(1))
|
||||
Expect(mine[id]["role"]).To(Equal("owner"))
|
||||
Expect(mine[id]).To(HaveKey("revision"))
|
||||
// The creation arguments are the tool's own schema: an identity field
|
||||
// or an unknown one never reaches the add-on.
|
||||
for _, args := range []string{`{"name":"X","tool":"databases-delete"}`, `{"name":"X","userId":"alice"}`, `{"name":"X","extra":1}`, `{}`} {
|
||||
b := fmt.Sprintf(`{"addonId":"tables","operationId":"bad-op","generation":1,"arguments":%s}`, args)
|
||||
Expect(h.send("POST", "/api/addons/resources", b, h.tab("bob", "bad-op")).code).To(Equal(http.StatusBadRequest), args)
|
||||
}
|
||||
// A stale generation is a stale view.
|
||||
stale := `{"addonId":"tables","operationId":"stale-op","generation":9,"arguments":{"name":"X"}}`
|
||||
got = h.send("POST", "/api/addons/resources", stale, h.tab("bob", "stale-op"))
|
||||
Expect(got.code).To(Equal(http.StatusConflict))
|
||||
Expect(got.errType()).To(Equal("stale"))
|
||||
// A person without the feature has no route at all.
|
||||
Expect(h.list("erin").code).To(Equal(http.StatusForbidden))
|
||||
})
|
||||
|
||||
It("keeps the owner, editor and viewer matrix on the UI routes and shows each person only their own role", func() {
|
||||
h := ready(false)
|
||||
res := h.database("bob", "m-1", "Matrix")
|
||||
Expect(h.grant("bob", res, "carol", "viewer").code).To(Equal(200))
|
||||
Expect(h.grant("bob", res, "dave", "editor").code).To(Equal(200))
|
||||
|
||||
for user, role := range map[string]string{"bob": "owner", "carol": "viewer", "dave": "editor"} {
|
||||
listed := resourcesOf(h.list(user))
|
||||
Expect(listed).To(HaveKey(res), user)
|
||||
Expect(listed[res]["role"]).To(Equal(role), user)
|
||||
if role != "owner" {
|
||||
// Grantees see who shared it and neither the grants nor the
|
||||
// revision of the owner's controls.
|
||||
Expect(listed[res]["owner"]).To(Equal("Name of bob"))
|
||||
Expect(listed[res]).NotTo(HaveKey("sharedWith"))
|
||||
Expect(listed[res]).NotTo(HaveKey("revision"))
|
||||
} else {
|
||||
Expect(listed[res]["sharedWith"]).To(BeEquivalentTo(2))
|
||||
}
|
||||
name, e := h.name(user, res)
|
||||
Expect(e).NotTo(HaveOccurred(), user)
|
||||
Expect(name).To(Equal("Matrix"))
|
||||
}
|
||||
rev := h.revision("bob", res)
|
||||
// Only the owner shares, renames, searches, reads grants or deletes;
|
||||
// everyone else gets the answer of an unknown resource.
|
||||
unknown := h.rename("bob", "r"+"0123456789abcdef0123456789abcde", "op-x", "N", rev)
|
||||
Expect(unknown.code).To(Equal(http.StatusForbidden))
|
||||
for _, user := range []string{"carol", "dave", "alice", "erin"} {
|
||||
for name, got := range map[string]reply{
|
||||
"rename": h.rename(user, res, "op-"+user, "Hijacked", rev),
|
||||
"delete": h.remove(user, res, "del-"+user, "Matrix", rev),
|
||||
"grant": h.grant(user, res, user, "editor"),
|
||||
"revoke": h.revoke(user, res, "carol"),
|
||||
"grants": h.grants(user, res),
|
||||
"users": h.users(user, res, "ca"),
|
||||
} {
|
||||
if user == "erin" {
|
||||
// No feature: the route gate answers first.
|
||||
Expect(got.code).To(Equal(http.StatusForbidden), user+" "+name)
|
||||
continue
|
||||
}
|
||||
Expect(got.code).To(Equal(unknown.code), user+" "+name+" "+got.raw)
|
||||
Expect(got.errType()).To(Equal("addon_error"), user+" "+name)
|
||||
}
|
||||
}
|
||||
// Nothing changed.
|
||||
name, _ := h.name("bob", res)
|
||||
Expect(name).To(Equal("Matrix"))
|
||||
Expect(resourcesOf(h.list("carol"))[res]["role"]).To(Equal("viewer"))
|
||||
|
||||
// The owner renames by the person's own action: no proposal, a
|
||||
// receipt, and the name is the add-on's.
|
||||
done := h.rename("bob", res, "rename-1", "Matrix 2027", rev)
|
||||
Expect(done.code).To(Equal(http.StatusOK), done.raw)
|
||||
Expect(done.body["name"]).To(Equal("Matrix 2027"))
|
||||
name, _ = h.name("carol", res)
|
||||
Expect(name).To(Equal("Matrix 2027"))
|
||||
status := h.send("GET", "/api/addons/resources/"+res+"/operations/rename-1?kind=rename", "", h.tab("bob", res))
|
||||
Expect(status.code).To(Equal(200), status.raw)
|
||||
Expect(status.body["state"]).To(Equal("committed"))
|
||||
// A repeated operation replays the receipt and renames nothing twice.
|
||||
again := h.rename("bob", res, "rename-1", "Matrix 2027", rev)
|
||||
Expect(again.code).To(Equal(http.StatusOK), again.raw)
|
||||
// The proposal list stays empty: a UI change is not a chat proposal.
|
||||
pending := h.send("GET", "/api/addons/proposals?state=pending", "", h.bearer("bob"))
|
||||
Expect(pending.body["proposals"]).To(BeEmpty())
|
||||
// The same operation ID of two owners are two operations.
|
||||
res2 := h.database("carol", "m-2", "Carol's")
|
||||
rev2 := h.revision("carol", res2)
|
||||
Expect(h.rename("carol", res2, "rename-1", "Carol's 2027", rev2).code).To(Equal(http.StatusOK))
|
||||
name, _ = h.name("carol", res2)
|
||||
Expect(name).To(Equal("Carol's 2027"))
|
||||
// A rename carries the revision the owner saw; a name the add-on
|
||||
// refuses changes nothing.
|
||||
bad := h.rename("bob", res, "rename-bad", "", rev)
|
||||
Expect(bad.code).To(Equal(http.StatusBadRequest))
|
||||
moved := map[string]any{"schema": 99, "data": 0}
|
||||
stale := h.rename("bob", res, "rename-stale", "Never", moved)
|
||||
Expect(stale.code).To(Equal(http.StatusConflict))
|
||||
Expect(stale.errType()).To(Equal("stale"))
|
||||
name, _ = h.name("bob", res)
|
||||
Expect(name).To(Equal("Matrix 2027"))
|
||||
})
|
||||
|
||||
It("shares whole databases by immutable account ID behind a bounded picker that tells nothing else", func() {
|
||||
h := ready(false)
|
||||
res := h.database("bob", "p-1", "Shared")
|
||||
|
||||
// The picker: bounded prefix, minimal fields, eligible accounts only.
|
||||
Expect(h.users("bob", res, "c").code).To(Equal(http.StatusBadRequest))
|
||||
got := h.users("bob", res, "Name%20of%20c")
|
||||
Expect(got.code).To(Equal(200), got.raw)
|
||||
users, _ := got.body["users"].([]any)
|
||||
Expect(users).To(HaveLen(1))
|
||||
user := users[0].(map[string]any)
|
||||
Expect(user).To(HaveLen(2))
|
||||
Expect(user["id"]).To(Equal("carol"))
|
||||
Expect(user["displayName"]).To(Equal("Name of carol"))
|
||||
// Not offered: oneself, accounts without the feature and pending ones.
|
||||
all := h.users("bob", res, "Name%20of")
|
||||
var ids []string
|
||||
for _, x := range all.body["users"].([]any) {
|
||||
ids = append(ids, x.(map[string]any)["id"].(string))
|
||||
}
|
||||
Expect(ids).To(ConsistOf("alice", "carol", "dave"))
|
||||
Expect(all.raw).NotTo(ContainSubstring("@"))
|
||||
// The match is a prefix and the wildcard characters mean themselves.
|
||||
Expect(resourcesOfUsers(h.users("bob", res, "%25%25"))).To(BeEmpty())
|
||||
Expect(resourcesOfUsers(h.users("bob", res, "ame%20of"))).To(BeEmpty())
|
||||
|
||||
// Roles are viewer or editor; the owner role is never granted.
|
||||
Expect(h.grant("bob", res, "carol", "owner").code).To(Equal(http.StatusBadRequest))
|
||||
Expect(h.grant("bob", res, "carol", "").code).To(Equal(http.StatusBadRequest))
|
||||
// Only accounts that exist, are active and hold the feature.
|
||||
for _, target := range []string{"erin", "frank", "nobody", "legacy-api-key", "local-owner", "bob"} {
|
||||
Expect(h.grant("bob", res, target, "viewer").code).To(Equal(http.StatusForbidden), target)
|
||||
}
|
||||
done := h.grant("bob", res, "carol", "viewer")
|
||||
Expect(done.code).To(Equal(200), done.raw)
|
||||
granted := done.body["grants"].([]any)
|
||||
Expect(granted).To(HaveLen(1))
|
||||
Expect(granted[0]).To(Equal(map[string]any{"userId": "carol", "displayName": "Name of carol", "role": "viewer"}))
|
||||
// Changing the role and listing the grants.
|
||||
Expect(h.grant("bob", res, "carol", "editor").code).To(Equal(200))
|
||||
listed := h.grants("bob", res)
|
||||
Expect(listed.code).To(Equal(200))
|
||||
Expect(listed.body["grants"].([]any)[0].(map[string]any)["role"]).To(Equal("editor"))
|
||||
// Revoking removes it, and a repeated revoke is the same state.
|
||||
Expect(h.revoke("bob", res, "carol").code).To(Equal(200))
|
||||
Expect(h.revoke("bob", res, "carol").code).To(Equal(200))
|
||||
Expect(resourcesOf(h.list("carol"))).NotTo(HaveKey(res))
|
||||
_, e := h.name("carol", res)
|
||||
Expect(e).To(HaveOccurred())
|
||||
// The sharing routes accept the owner's own session only.
|
||||
for _, header := range []func(*http.Request){h.bearer("bob"), h.session("bob")} {
|
||||
Expect(h.send("PUT", "/api/addons/resources/"+res+"/grants/carol", `{"role":"viewer"}`, header).code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.send("DELETE", "/api/addons/resources/"+res+"/grants/carol", "", header).code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.send("GET", "/api/addons/resources/"+res+"/grants", "", header).code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.send("GET", "/api/addons/resources/"+res+"/users?q=ca", "", header).code).To(Equal(http.StatusForbidden))
|
||||
}
|
||||
// A cross-site browser request never reaches the handler.
|
||||
cross := func(r *http.Request) { h.tab("bob", res)(r); r.Header.Set("Sec-Fetch-Site", "cross-site") }
|
||||
Expect(h.send("PUT", "/api/addons/resources/"+res+"/grants/carol", `{"role":"viewer"}`, cross).code).To(BeElementOf(http.StatusBadRequest, http.StatusForbidden))
|
||||
Expect(resourcesOf(h.list("carol"))).NotTo(HaveKey(res))
|
||||
})
|
||||
|
||||
It("keeps sharing disabled in local-owner mode with a typed answer and no named users", func() {
|
||||
h := ready(true)
|
||||
res := h.database("local-owner", "l-1", "Mine")
|
||||
listed := h.list("local-owner")
|
||||
Expect(listed.code).To(Equal(200), listed.raw)
|
||||
Expect(listed.body["sharing"]).To(Equal(false))
|
||||
Expect(resourcesOf(listed)).To(HaveKey(res))
|
||||
Expect(resourcesOf(listed)[res]["role"]).To(Equal("owner"))
|
||||
for name, got := range map[string]reply{
|
||||
"grant": h.grant("local-owner", res, "carol", "viewer"),
|
||||
"revoke": h.revoke("local-owner", res, "carol"),
|
||||
"grants": h.grants("local-owner", res),
|
||||
"users": h.users("local-owner", res, "ca"),
|
||||
} {
|
||||
Expect(got.code).To(Equal(http.StatusConflict), name+" "+got.raw)
|
||||
Expect(got.errType()).To(Equal("sharing_disabled"), name)
|
||||
}
|
||||
// Everything else works: the mode has one owner, not no owner.
|
||||
rev := h.revision("local-owner", res)
|
||||
done := h.rename("local-owner", res, "rl-1", "Mine too", rev)
|
||||
Expect(done.code).To(Equal(200), done.raw)
|
||||
// A request that carries an API credential is not the local owner's page.
|
||||
Expect(h.send("POST", "/api/addons/resources/"+res+"/rename", "{}", h.tab("x", res), func(r *http.Request) { r.Header.Set("Authorization", "Bearer x") }).code).To(Equal(http.StatusForbidden))
|
||||
})
|
||||
|
||||
It("waits for a revoked user's admitted work and cancels it before acknowledging the revocation", func() {
|
||||
h := ready(false)
|
||||
res := h.database("bob", "c-1", "Busy")
|
||||
Expect(h.grant("bob", res, "carol", "editor").code).To(Equal(200))
|
||||
p := h.app.AddonPolicy()
|
||||
started, cancelled := make(chan struct{}), make(chan struct{})
|
||||
var once sync.Once
|
||||
go func() {
|
||||
defer GinkgoRecover()
|
||||
_ = p.Admit(context.Background(), h.actor("carol"), h.key(res), policy.ActionWrite, func(ctx context.Context) error {
|
||||
once.Do(func() { close(started) })
|
||||
<-ctx.Done()
|
||||
close(cancelled)
|
||||
return ctx.Err()
|
||||
})
|
||||
}()
|
||||
Eventually(started).Should(BeClosed())
|
||||
done := h.revoke("bob", res, "carol")
|
||||
Expect(done.code).To(Equal(200), done.raw)
|
||||
// The acknowledgement came after the admitted work saw its context end.
|
||||
Expect(cancelled).To(BeClosed())
|
||||
_, e := h.name("carol", res)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(resourcesOf(h.list("carol"))).NotTo(HaveKey(res))
|
||||
})
|
||||
})
|
||||
|
||||
func resourcesOfUsers(r reply) []any {
|
||||
GinkgoHelper()
|
||||
Expect(r.code).To(Equal(200), r.raw)
|
||||
users, _ := r.body["users"].([]any)
|
||||
return users
|
||||
}
|
||||
|
||||
var _ = json.Marshal
|
||||
@@ -0,0 +1,278 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesadmin_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
mcpTools "github.com/mudler/LocalAI/core/http/endpoints/mcp"
|
||||
addonsvc "github.com/mudler/LocalAI/core/services/addons"
|
||||
"github.com/mudler/LocalAI/core/services/addons/gateway"
|
||||
"github.com/mudler/LocalAI/core/services/addons/tools"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func (h *host) selection(user, conv string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("GET", "/api/addons/conversations/"+conv+"/selection/tables", "", h.bearer(user))
|
||||
}
|
||||
|
||||
func (h *host) choose(user, conv, res string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("PUT", "/api/addons/conversations/"+conv+"/selection/tables", fmt.Sprintf(`{"resourceId":%q}`, res), h.bearer(user))
|
||||
}
|
||||
|
||||
func (h *host) clear(user, conv string) reply {
|
||||
GinkgoHelper()
|
||||
return h.send("DELETE", "/api/addons/conversations/"+conv+"/selection/tables", "", h.bearer(user))
|
||||
}
|
||||
|
||||
// say runs one model tool call of a chat conversation as the user's key.
|
||||
func (h *host) say(user, conv, tool, body string) (tools.ToolResult, error) {
|
||||
GinkgoHelper()
|
||||
return tools.NewToolAdapter(h.app.AddonGateway()).ExecuteIn(context.Background(), h.actor(user), conv, tool, json.RawMessage(body))
|
||||
}
|
||||
|
||||
func inspectedName(r tools.ToolResult) string {
|
||||
GinkgoHelper()
|
||||
Expect(r.Kind).To(Equal(tools.ResultData))
|
||||
var v struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
Expect(json.Unmarshal(r.Result, &v)).To(Succeed())
|
||||
return v.Name
|
||||
}
|
||||
|
||||
var _ = Describe("the database a conversation works on", func() {
|
||||
It("keeps one selection per user, conversation and add-on, across people, conversations and restarts", func() {
|
||||
h := ready(false)
|
||||
sales := h.database("bob", "v-1", "Sales")
|
||||
stock := h.database("bob", "v-2", "Stock")
|
||||
mine := h.database("carol", "v-3", "Carols")
|
||||
|
||||
// Nothing is selected, and no process-wide default exists.
|
||||
got := h.selection("bob", "chat-1")
|
||||
Expect(got.code).To(Equal(http.StatusNotFound))
|
||||
Expect(got.errType()).To(Equal("not_selected"))
|
||||
|
||||
done := h.choose("bob", "chat-1", sales)
|
||||
Expect(done.code).To(Equal(200), done.raw)
|
||||
Expect(done.body).To(HaveKeyWithValue("resourceId", sales))
|
||||
Expect(done.body).To(HaveKeyWithValue("role", "owner"))
|
||||
Expect(done.body).To(HaveKeyWithValue("name", "Sales"))
|
||||
Expect(h.choose("bob", "chat-2", stock).code).To(Equal(200))
|
||||
// Two conversations of one user keep two selections.
|
||||
Expect(h.selection("bob", "chat-1").body["resourceId"]).To(Equal(sales))
|
||||
Expect(h.selection("bob", "chat-2").body["resourceId"]).To(Equal(stock))
|
||||
// The same conversation ID of another person is another namespace and
|
||||
// never shows or inherits this selection.
|
||||
Expect(h.selection("carol", "chat-1").code).To(Equal(http.StatusNotFound))
|
||||
Expect(h.choose("carol", "chat-1", mine).code).To(Equal(200))
|
||||
Expect(h.selection("bob", "chat-1").body["resourceId"]).To(Equal(sales))
|
||||
Expect(h.selection("carol", "chat-1").body["resourceId"]).To(Equal(mine))
|
||||
// A database the person cannot read is not selectable, with the answer
|
||||
// of an unknown database; neither is a database of another add-on or a
|
||||
// malformed identifier.
|
||||
Expect(h.choose("carol", "chat-2", stock).code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.choose("carol", "chat-2", "r0123456789abcdef0123456789abcde").code).To(Equal(http.StatusForbidden))
|
||||
put := h.send("PUT", "/api/addons/conversations/chat-2/selection/other", fmt.Sprintf(`{"resourceId":%q}`, mine), h.bearer("carol"))
|
||||
Expect(put.code).To(Equal(http.StatusForbidden))
|
||||
Expect(h.send("PUT", "/api/addons/conversations/bad%20conv/selection/tables", fmt.Sprintf(`{"resourceId":%q}`, mine), h.bearer("carol")).code).To(Equal(400))
|
||||
Expect(h.send("PUT", "/api/addons/conversations/chat-2/selection/tables", `{"resourceId":"../x"}`, h.bearer("carol")).code).To(Equal(http.StatusBadRequest))
|
||||
Expect(h.selection("carol", "chat-2").code).To(Equal(http.StatusNotFound))
|
||||
// A user without the feature has no selection routes.
|
||||
Expect(h.selection("erin", "chat-1").code).To(Equal(http.StatusForbidden))
|
||||
|
||||
// The selection is durable on the owner.
|
||||
h.stop()
|
||||
h.start()
|
||||
h.running()
|
||||
Expect(h.selection("bob", "chat-2").body["resourceId"]).To(Equal(stock))
|
||||
Expect(h.selection("carol", "chat-1").body["resourceId"]).To(Equal(mine))
|
||||
|
||||
Expect(h.clear("bob", "chat-2").code).To(Equal(http.StatusNoContent))
|
||||
Expect(h.clear("bob", "chat-2").code).To(Equal(http.StatusNoContent))
|
||||
Expect(h.selection("bob", "chat-2").code).To(Equal(http.StatusNotFound))
|
||||
Expect(h.selection("bob", "chat-1").code).To(Equal(200))
|
||||
})
|
||||
|
||||
It("lets a tool call omit the database in a conversation and resolves it at the owner with typed errors", func() {
|
||||
h := ready(false)
|
||||
sales := h.database("bob", "w-1", "Sales")
|
||||
stock := h.database("bob", "w-2", "Stock")
|
||||
Expect(h.choose("bob", "chat-1", sales).code).To(Equal(200))
|
||||
inspect := "addon__tables__databases-inspect"
|
||||
|
||||
got, e := h.say("bob", "chat-1", inspect, `{}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(inspectedName(got)).To(Equal("Sales"))
|
||||
// An explicit database is the person's own words and still works.
|
||||
got, e = h.say("bob", "chat-1", inspect, fmt.Sprintf(`{"resourceId":%q}`, stock))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(inspectedName(got)).To(Equal("Stock"))
|
||||
// No selection is a result that asks the person to select one.
|
||||
got, e = h.say("bob", "chat-2", inspect, `{}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(got.Kind).To(Equal(tools.ResultError))
|
||||
Expect(got.Code).To(Equal(tools.CodeSelectResource))
|
||||
Expect(got.Result).To(BeEmpty())
|
||||
// A caller without a conversation, or without a selection of its own,
|
||||
// never inherits another's: another user's same conversation ID, an
|
||||
// agent without a conversation.
|
||||
got, e = h.say("carol", "chat-1", inspect, `{}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(got.Kind).To(Equal(tools.ResultError))
|
||||
_, e = h.say("bob", "", inspect, `{}`)
|
||||
Expect(e).To(MatchError(tools.ErrInvalidArguments))
|
||||
// The conversation is never an argument, and no field of the model
|
||||
// can name or replace it.
|
||||
for _, args := range []string{`{"conversationId":"chat-1"}`, `{"conversation":"chat-1"}`, `{"selection":"x"}`} {
|
||||
_, e = h.say("bob", "chat-9", inspect, args)
|
||||
Expect(e).To(MatchError(tools.ErrInvalidArguments), args)
|
||||
}
|
||||
_, e = h.say("bob", "chat-9", inspect, `{"resourceId":"x","tool":"databases-delete"}`)
|
||||
Expect(e).To(MatchError(tools.ErrForbiddenArgument))
|
||||
// A conversation that is not the shape of one is refused.
|
||||
_, e = h.say("bob", "../chat", inspect, `{}`)
|
||||
Expect(e).To(MatchError(tools.ErrInvalidArguments))
|
||||
|
||||
// Writes use it too: a proposal names the selected database.
|
||||
created, e := h.say("bob", "chat-1", "addon__tables__tables-create", `{"table":"Orders"}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(created.Kind).To(Equal(tools.ResultProposal))
|
||||
Expect(created.Proposal.ResourceID).To(Equal(sales))
|
||||
|
||||
// The executor of a chat request binds the conversation once, like the
|
||||
// actor, and nothing in the arguments changes it.
|
||||
exec, e := mcpTools.NewAddonToolExecutorFor(context.Background(), tools.NewToolAdapter(h.app.AddonGateway()), h.actor("bob"), "chat-1")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
out, e := exec.ExecuteTool(context.Background(), inspect, `{}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(out).To(ContainSubstring(`"Sales"`))
|
||||
out, e = exec.ExecuteTool(context.Background(), inspect, `{"conversationId":"chat-9"}`)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(out).To(BeEmpty())
|
||||
none, e := mcpTools.NewAddonToolExecutor(context.Background(), tools.NewToolAdapter(h.app.AddonGateway()), h.actor("bob"))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = none.ExecuteTool(context.Background(), inspect, `{}`)
|
||||
Expect(e).To(MatchError(tools.ErrInvalidArguments))
|
||||
})
|
||||
|
||||
It("withdraws the management and the tool path while the add-on is disabled and restores them on enable", func() {
|
||||
h := ready(false)
|
||||
sales := h.database("bob", "y-1", "Sales")
|
||||
Expect(h.choose("bob", "chat-1", sales).code).To(Equal(200))
|
||||
inspect := "addon__tables__databases-inspect"
|
||||
_, e := h.say("bob", "chat-1", inspect, `{}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
|
||||
ctx := context.Background()
|
||||
_, e = h.app.AddonManager().Submit(ctx, h.actor("alice"), addonsvc.LifecycleCommand{OperationID: "y-disable", AddonID: "tables", Action: "disable", Generation: 1})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
// No management call answers from a disabled add-on, a stale tab
|
||||
// cannot create, and the model is not offered the tool.
|
||||
Expect(h.list("bob").code).To(Equal(http.StatusServiceUnavailable))
|
||||
rev := map[string]any{"schema": 1, "data": 0}
|
||||
Expect(h.rename("bob", sales, "y-r", "N", rev).code).To(BeElementOf(503, 403))
|
||||
Expect(h.create("bob", "y-c", "N").code).To(BeElementOf(503, 403))
|
||||
_, e = h.say("bob", "chat-1", inspect, `{}`)
|
||||
Expect(e).To(HaveOccurred())
|
||||
// The data and the selection are kept; only the display name needs the
|
||||
// add-on.
|
||||
kept := h.selection("bob", "chat-1")
|
||||
Expect(kept.code).To(Equal(200), kept.raw)
|
||||
Expect(kept.body).To(HaveKeyWithValue("resourceId", sales))
|
||||
Expect(kept.body).NotTo(HaveKey("name"))
|
||||
|
||||
_, e = h.app.AddonManager().Submit(ctx, h.actor("alice"), addonsvc.LifecycleCommand{OperationID: "y-enable", AddonID: "tables", Action: "enable", Generation: 1})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h.running()
|
||||
Expect(resourcesOf(h.list("bob"))).To(HaveKey(sales))
|
||||
got, e := h.say("bob", "chat-1", inspect, `{}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(inspectedName(got)).To(Equal("Sales"))
|
||||
})
|
||||
|
||||
It("answers a revoked, deleted or tampered reference like an unknown database and never reuses it", func() {
|
||||
h := ready(false)
|
||||
shared := h.database("bob", "x-1", "Shared")
|
||||
doomed := h.database("bob", "x-2", "Doomed")
|
||||
private := h.database("bob", "x-3", "Private")
|
||||
Expect(h.grant("bob", shared, "carol", "viewer").code).To(Equal(200))
|
||||
Expect(h.grant("bob", doomed, "carol", "viewer").code).To(Equal(200))
|
||||
Expect(h.choose("carol", "chat-1", shared).code).To(Equal(200))
|
||||
Expect(h.choose("carol", "chat-2", doomed).code).To(Equal(200))
|
||||
inspect := "addon__tables__databases-inspect"
|
||||
_, e := h.say("carol", "chat-1", inspect, `{}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
// The answer for an unknown database, to compare with.
|
||||
_, unknown := h.say("carol", "chat-1", inspect, `{"resourceId":"r0123456789abcdef0123456789abcde"}`)
|
||||
Expect(unknown).To(HaveOccurred())
|
||||
Expect(gateway.StatusFor(unknown)).To(Equal(403))
|
||||
|
||||
// Revoked: the same denial. The reference is no authority and stays the
|
||||
// person's own: a new grant by the owner makes it work again.
|
||||
Expect(h.revoke("bob", shared, "carol").code).To(Equal(200))
|
||||
_, e = h.say("carol", "chat-1", inspect, `{}`)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(e.Error()).To(Equal(unknown.Error()))
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
Expect(h.selection("carol", "chat-1").code).To(Equal(http.StatusNotFound))
|
||||
Expect(h.grant("bob", shared, "carol", "viewer").code).To(Equal(200))
|
||||
got, e := h.say("carol", "chat-1", inspect, `{}`)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(inspectedName(got)).To(Equal("Shared"))
|
||||
|
||||
// Deleted: the same denial.
|
||||
rev := h.revision("bob", doomed)
|
||||
Expect(h.remove("bob", doomed, "x-del", "Doomed", rev).code).To(Equal(200))
|
||||
_, e = h.say("carol", "chat-2", inspect, `{}`)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
Expect(e.Error()).To(Equal(unknown.Error()))
|
||||
Expect(h.selection("carol", "chat-2").code).To(Equal(http.StatusNotFound))
|
||||
|
||||
// Tampered: the stored file names a database that was never shared.
|
||||
// The reference is no authority: the owner decides at every use.
|
||||
Expect(h.choose("carol", "chat-3", shared).code).To(Equal(200))
|
||||
h.stop()
|
||||
path := filepath.Join(h.owner.Root, "selections", "selections.json")
|
||||
raw, err := os.ReadFile(path)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
var doc struct {
|
||||
Version int `json:"version"`
|
||||
Entries []map[string]any `json:"entries"`
|
||||
}
|
||||
Expect(json.Unmarshal(raw, &doc)).To(Succeed())
|
||||
tampered := 0
|
||||
for _, entry := range doc.Entries {
|
||||
if entry["user"] == "carol" && entry["conversation"] == "chat-3" {
|
||||
entry["resourceId"] = private
|
||||
tampered++
|
||||
}
|
||||
}
|
||||
Expect(tampered).To(Equal(1))
|
||||
raw, err = json.Marshal(doc)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(os.WriteFile(path, raw, 0o600)).To(Succeed())
|
||||
h.start()
|
||||
h.running()
|
||||
_, e = h.say("carol", "chat-3", inspect, `{}`)
|
||||
Expect(e).To(HaveOccurred())
|
||||
Expect(gateway.StatusFor(e)).To(Equal(403))
|
||||
Expect(e.Error()).To(Equal(unknown.Error()))
|
||||
got2 := h.selection("carol", "chat-3")
|
||||
Expect(got2.code).To(Equal(http.StatusNotFound))
|
||||
// The owner of that database is not affected.
|
||||
name, e := h.name("bob", private)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(name).To(Equal("Private"))
|
||||
_ = time.Second
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,314 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesadmin_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/addons/policy"
|
||||
"github.com/mudler/LocalAI/core/services/addons/resources"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
type clock struct {
|
||||
mu sync.Mutex
|
||||
t time.Time
|
||||
}
|
||||
|
||||
func (c *clock) now() time.Time {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.t
|
||||
}
|
||||
|
||||
func (c *clock) advance(d time.Duration) {
|
||||
c.mu.Lock()
|
||||
c.t = c.t.Add(d)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
var _ = Describe("the conversation selection store", func() {
|
||||
open := func(dir string, o resources.SelectionOptions) *resources.SelectionStore {
|
||||
GinkgoHelper()
|
||||
s, e := resources.OpenSelections(dir, o)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return s
|
||||
}
|
||||
|
||||
It("keys a reference by user, conversation and add-on, bounds it per user and in total and expires what is unused", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
c := &clock{t: time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)}
|
||||
s := open(dir, resources.SelectionOptions{TTL: time.Hour, MaxPerUser: 3, MaxTotal: 5, Now: c.now})
|
||||
DeferCleanup(s.Close)
|
||||
|
||||
Expect(s.Put("u1", "c1", "tables", "rA")).To(Succeed())
|
||||
Expect(s.Put("u2", "c1", "tables", "rB")).To(Succeed())
|
||||
c.advance(time.Second)
|
||||
Expect(s.Put("u1", "c1", "notes", "rC")).To(Succeed())
|
||||
get := func(u, c, a string) (string, bool) {
|
||||
GinkgoHelper()
|
||||
id, ok, e := s.Get(u, c, a)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return id, ok
|
||||
}
|
||||
for key, want := range map[[3]string]string{{"u1", "c1", "tables"}: "rA", {"u2", "c1", "tables"}: "rB", {"u1", "c1", "notes"}: "rC"} {
|
||||
got, ok := get(key[0], key[1], key[2])
|
||||
Expect(ok).To(BeTrue(), fmt.Sprint(key))
|
||||
Expect(got).To(Equal(want))
|
||||
}
|
||||
_, ok := get("u3", "c1", "tables")
|
||||
Expect(ok).To(BeFalse())
|
||||
_, ok = get("u1", "c2", "tables")
|
||||
Expect(ok).To(BeFalse())
|
||||
|
||||
// A user over the bound loses the entry used least recently, and the
|
||||
// entry just stored stays.
|
||||
c.advance(time.Minute)
|
||||
Expect(s.Put("u1", "c2", "tables", "rD")).To(Succeed())
|
||||
c.advance(time.Minute)
|
||||
Expect(s.Put("u1", "c3", "tables", "rE")).To(Succeed())
|
||||
_, ok = get("u1", "c1", "tables")
|
||||
Expect(ok).To(BeFalse(), "the oldest of the user is evicted")
|
||||
_, ok = get("u1", "c3", "tables")
|
||||
Expect(ok).To(BeTrue())
|
||||
_, ok = get("u2", "c1", "tables")
|
||||
Expect(ok).To(BeTrue(), "another user's entry is never evicted for this user")
|
||||
|
||||
// The total bound refuses once it is full of live entries.
|
||||
Expect(s.Put("u4", "c1", "tables", "r1")).To(Succeed())
|
||||
err := s.Put("u5", "c1", "tables", "r2")
|
||||
Expect(errors.Is(err, resources.ErrSelectionStore)).To(BeTrue())
|
||||
|
||||
// Unused entries expire, and an expired one never resolves.
|
||||
c.advance(2 * time.Hour)
|
||||
_, ok = get("u2", "c1", "tables")
|
||||
Expect(ok).To(BeFalse())
|
||||
Expect(s.Put("u6", "c1", "tables", "r3")).To(Succeed())
|
||||
Expect(s.Len()).To(Equal(1), "expired entries are removed by the next write")
|
||||
// Delete is idempotent.
|
||||
Expect(s.Delete("u6", "c1", "tables")).To(Succeed())
|
||||
Expect(s.Delete("u6", "c1", "tables")).To(Succeed())
|
||||
_, ok = get("u6", "c1", "tables")
|
||||
Expect(ok).To(BeFalse())
|
||||
})
|
||||
|
||||
It("is durable, held by one process, never resets a corrupt file and stops writing after a failed sync", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
s := open(dir, resources.SelectionOptions{})
|
||||
Expect(s.Put("u1", "c1", "tables", "rA")).To(Succeed())
|
||||
_, e := resources.OpenSelections(dir, resources.SelectionOptions{})
|
||||
Expect(e).To(HaveOccurred(), "one process holds the store")
|
||||
Expect(s.Close()).To(Succeed())
|
||||
Expect(s.Put("u1", "c2", "tables", "rB")).NotTo(Succeed(), "nothing is written after Close")
|
||||
|
||||
s = open(dir, resources.SelectionOptions{})
|
||||
id, ok, e := s.Get("u1", "c1", "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(id).To(Equal("rA"))
|
||||
Expect(s.Close()).To(Succeed())
|
||||
|
||||
// A corrupt file is an error and is left as it is.
|
||||
file := filepath.Join(dir, "selections.json")
|
||||
Expect(os.WriteFile(file, []byte(`{"version":1,"entries":[{"user":"u1"`), 0o600)).To(Succeed())
|
||||
_, e = resources.OpenSelections(dir, resources.SelectionOptions{})
|
||||
Expect(e).To(HaveOccurred())
|
||||
raw, _ := os.ReadFile(file)
|
||||
Expect(string(raw)).To(Equal(`{"version":1,"entries":[{"user":"u1"`))
|
||||
for _, bad := range []string{`{"version":2,"entries":[]}`, `{"version":1,"entries":[],"extra":1}`, `{"version":1,"entries":[{"user":"","conversation":"c","addon":"a","resourceId":"r","usedAt":"2026-01-01T00:00:00Z"}]}`, `{"version":1,"entries":[]} {"version":1}`} {
|
||||
Expect(os.WriteFile(file, []byte(bad), 0o600)).To(Succeed())
|
||||
_, e = resources.OpenSelections(dir, resources.SelectionOptions{})
|
||||
Expect(e).To(HaveOccurred(), bad)
|
||||
}
|
||||
Expect(os.Remove(file)).To(Succeed())
|
||||
|
||||
// A change that reached the directory but could not be made durable is
|
||||
// kept in memory and stops further writes until the store reopens.
|
||||
fail := errors.New("directory sync failed")
|
||||
failing := false
|
||||
s = open(dir, resources.SelectionOptions{SyncDirectory: func(f *os.File) error {
|
||||
if failing {
|
||||
return fail
|
||||
}
|
||||
return f.Sync()
|
||||
}})
|
||||
Expect(s.Put("u1", "c1", "tables", "rA")).To(Succeed())
|
||||
failing = true
|
||||
Expect(s.Put("u1", "c2", "tables", "rB")).NotTo(Succeed())
|
||||
failing = false
|
||||
Expect(s.Put("u1", "c3", "tables", "rC")).NotTo(Succeed(), "a failed write stops further writes")
|
||||
Expect(s.Delete("u1", "c1", "tables")).NotTo(Succeed())
|
||||
Expect(s.Close()).To(Succeed())
|
||||
s = open(dir, resources.SelectionOptions{})
|
||||
DeferCleanup(s.Close)
|
||||
Expect(s.Put("u1", "c3", "tables", "rC")).To(Succeed())
|
||||
})
|
||||
|
||||
It("serves concurrent users without losing a reference", func() {
|
||||
s := open(GinkgoT().TempDir(), resources.SelectionOptions{})
|
||||
DeferCleanup(s.Close)
|
||||
var wg sync.WaitGroup
|
||||
for u := 0; u < 8; u++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer GinkgoRecover()
|
||||
defer wg.Done()
|
||||
user := fmt.Sprintf("user-%d", u)
|
||||
for c := 0; c < 10; c++ {
|
||||
Expect(s.Put(user, fmt.Sprintf("conv-%d", c), "tables", fmt.Sprintf("r%d-%d", u, c))).To(Succeed())
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
for u := 0; u < 8; u++ {
|
||||
for c := 0; c < 10; c++ {
|
||||
id, ok, e := s.Get(fmt.Sprintf("user-%d", u), fmt.Sprintf("conv-%d", c), "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(ok).To(BeTrue())
|
||||
Expect(id).To(Equal(fmt.Sprintf("r%d-%d", u, c)))
|
||||
}
|
||||
}
|
||||
Expect(s.Len()).To(Equal(80))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("the tombstones of finished deletions", func() {
|
||||
It("keeps the newest bounded number, validates them on load and remaps them with the owner", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
tombstones := map[string]any{}
|
||||
base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
for i := 0; i < policy.MaxTombstones; i++ {
|
||||
tombstones[fmt.Sprintf("rold%04d", i)] = map[string]any{"addonId": "tables", "owner": "bob", "op": fmt.Sprintf("op-%d", i), "at": base.Add(time.Duration(i) * time.Minute)}
|
||||
}
|
||||
write := func(deleted map[string]any) {
|
||||
GinkgoHelper()
|
||||
raw, e := json.Marshal(map[string]any{"version": 1, "mode": "named-user", "resources": map[string]any{}, "deleted": deleted})
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(os.WriteFile(filepath.Join(dir, "policy.json"), raw, 0o600)).To(Succeed())
|
||||
}
|
||||
write(tombstones)
|
||||
st, e := policy.OpenStore(dir, policy.ModeNamedUser)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
p := policy.New(st, policy.Options{
|
||||
Revalidate: func(context.Context, contract.Actor) error { return nil },
|
||||
CreateAuthority: func(context.Context, contract.Actor, string) error { return nil },
|
||||
})
|
||||
ctx := context.Background()
|
||||
bob := contract.Actor{UserID: "bob", CredentialID: "session:bob", Mode: "named-user"}
|
||||
key, e := p.Register(ctx, bob, "tables")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(p.Commit(ctx, bob, key)).To(Succeed())
|
||||
Expect(p.BeginDelete(ctx, bob, key, "op-new")).To(Succeed())
|
||||
Expect(p.ConfirmDelete(key)).To(Succeed())
|
||||
Expect(p.FinishDelete(key)).To(Succeed())
|
||||
|
||||
got, e := p.DeletedBy(ctx, bob, key.ResourceID, "op-new")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(got).To(BeTrue())
|
||||
got, _ = p.DeletedBy(ctx, bob, "rold0000", "")
|
||||
Expect(got).To(BeFalse(), "the oldest tombstone was dropped")
|
||||
got, _ = p.DeletedBy(ctx, bob, "rold0001", "op-1")
|
||||
Expect(got).To(BeTrue())
|
||||
got, _ = p.DeletedBy(ctx, bob, "rold0001", "op-2")
|
||||
Expect(got).To(BeFalse(), "another operation is not the deletion")
|
||||
carol := contract.Actor{UserID: "carol", CredentialID: "session:carol", Mode: "named-user"}
|
||||
got, _ = p.DeletedBy(ctx, carol, key.ResourceID, "")
|
||||
Expect(got).To(BeFalse(), "only the owner it had learns it")
|
||||
Expect(st.Close()).To(Succeed())
|
||||
var doc struct {
|
||||
Deleted map[string]any `json:"deleted"`
|
||||
}
|
||||
raw, e := os.ReadFile(filepath.Join(dir, "policy.json"))
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(json.Unmarshal(raw, &doc)).To(Succeed())
|
||||
Expect(doc.Deleted).To(HaveLen(policy.MaxTombstones))
|
||||
|
||||
// A corrupt tombstone or one too many is a corrupt store, never reset.
|
||||
for name, bad := range map[string]map[string]any{
|
||||
"an empty owner": {"r1": map[string]any{"addonId": "tables", "owner": "", "op": "o", "at": base}},
|
||||
"no operation": {"r1": map[string]any{"addonId": "tables", "owner": "bob", "op": "", "at": base}},
|
||||
} {
|
||||
write(bad)
|
||||
_, e = policy.OpenStore(dir, policy.ModeNamedUser)
|
||||
Expect(e).To(HaveOccurred(), name)
|
||||
}
|
||||
tombstones["rextra"] = map[string]any{"addonId": "tables", "owner": "bob", "op": "o", "at": base}
|
||||
write(tombstones)
|
||||
_, e = policy.OpenStore(dir, policy.ModeNamedUser)
|
||||
Expect(e).To(HaveOccurred(), "more than the bound")
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("the manifest declaration of admin tools and resource tools", func() {
|
||||
manifest := func() map[string]any {
|
||||
GinkgoHelper()
|
||||
raw, e := os.ReadFile("../../../addons/tables/manifest.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var m map[string]any
|
||||
Expect(json.Unmarshal(raw, &m)).To(Succeed())
|
||||
return m
|
||||
}
|
||||
caps := func(m map[string]any) map[string]any { return m["capabilities"].(map[string]any) }
|
||||
parse := func(m map[string]any) error {
|
||||
GinkgoHelper()
|
||||
b, e := json.Marshal(m)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
_, e = contract.ParseManifest(b)
|
||||
return e
|
||||
}
|
||||
|
||||
It("keeps the delete tool out of the model tools and their definitions", func() {
|
||||
m := manifest()
|
||||
Expect(parse(m)).To(Succeed())
|
||||
c := caps(m)
|
||||
Expect(c["adminTools"]).To(ConsistOf("databases-delete"))
|
||||
Expect(c["tools"]).NotTo(ContainElement("databases-delete"))
|
||||
for _, d := range c["toolDefinitions"].([]any) {
|
||||
Expect(d.(map[string]any)["name"]).NotTo(Equal("databases-delete"))
|
||||
}
|
||||
})
|
||||
|
||||
It("refuses an admin tool that is also a model tool, has a definition, or is not mapped", func() {
|
||||
bad := map[string]func(map[string]any){
|
||||
"an admin tool that is a model tool": func(c map[string]any) {
|
||||
c["adminTools"] = []any{"databases-rename"}
|
||||
c["resources"].(map[string]any)["delete"] = "databases-rename"
|
||||
},
|
||||
"an admin tool with a model definition": func(c map[string]any) {
|
||||
defs := c["toolDefinitions"].([]any)
|
||||
c["toolDefinitions"] = append(defs, map[string]any{"name": "databases-delete", "description": "Delete.", "kind": "schema", "inputSchema": map[string]any{"type": "object", "properties": map[string]any{"resourceId": map[string]any{"type": "string"}}, "required": []any{"resourceId"}, "additionalProperties": false}})
|
||||
},
|
||||
"an admin tool nothing maps": func(c map[string]any) { delete(c["resources"].(map[string]any), "delete") },
|
||||
"admin tools without a mapping": func(c map[string]any) { delete(c, "resources") },
|
||||
"a delete mapping to a model tool": func(c map[string]any) { c["resources"].(map[string]any)["delete"] = "rows-delete" },
|
||||
"a delete mapping that is not declared": func(c map[string]any) {
|
||||
c["adminTools"] = []any{"databases-purge"}
|
||||
c["resources"].(map[string]any)["delete"] = "databases-delete"
|
||||
},
|
||||
"a create mapping to a read tool": func(c map[string]any) { c["resources"].(map[string]any)["create"] = "databases-inspect" },
|
||||
"a rename mapping to an undeclared tool": func(c map[string]any) { c["resources"].(map[string]any)["rename"] = "databases-copy" },
|
||||
"an inspect mapping to a write tool": func(c map[string]any) { c["resources"].(map[string]any)["inspect"] = "rows-insert" },
|
||||
"two admin tools": func(c map[string]any) { c["adminTools"] = []any{"databases-delete", "databases-purge"} },
|
||||
"a markup noun": func(c map[string]any) { c["resources"].(map[string]any)["noun"] = "<b>database</b>" },
|
||||
"an empty mapping": func(c map[string]any) { c["resources"] = map[string]any{}; c["adminTools"] = []any{} },
|
||||
}
|
||||
for name, mutate := range bad {
|
||||
m := manifest()
|
||||
mutate(caps(m))
|
||||
Expect(parse(m)).NotTo(Succeed(), name)
|
||||
}
|
||||
// A manifest with no admin tools and no mapping stays valid.
|
||||
m := manifest()
|
||||
delete(caps(m), "adminTools")
|
||||
delete(caps(m), "resources")
|
||||
Expect(parse(m)).To(Succeed())
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,81 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesexport_test
|
||||
|
||||
import (
|
||||
"io/fs"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"testing"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestTablesExport(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Tables CSV export across replicas")
|
||||
}
|
||||
|
||||
var (
|
||||
paths cluster.Paths
|
||||
tables []byte
|
||||
)
|
||||
|
||||
var _ = BeforeSuite(func() {
|
||||
_, file, _, _ := runtime.Caller(0)
|
||||
root := filepath.Clean(filepath.Join(filepath.Dir(file), "..", "..", ".."))
|
||||
scratch, err := os.MkdirTemp("", "tables-export-bin-*")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() { _ = os.RemoveAll(scratch) })
|
||||
build := func(env, out, pkg string, cgo bool, tags ...string) string {
|
||||
if p := os.Getenv(env); p != "" {
|
||||
return p
|
||||
}
|
||||
target := filepath.Join(scratch, out)
|
||||
args := []string{"build", "-p", "2"}
|
||||
if len(tags) > 0 {
|
||||
args = append(args, "-tags", tags[0])
|
||||
}
|
||||
cmd := exec.Command("go", append(args, "-o", target, pkg)...)
|
||||
cmd.Dir = root
|
||||
if cgo {
|
||||
cmd.Env = append(os.Environ(), "CGO_ENABLED=1")
|
||||
}
|
||||
b, err := cmd.CombinedOutput()
|
||||
Expect(err).NotTo(HaveOccurred(), string(b))
|
||||
return target
|
||||
}
|
||||
paths.Fixture = build("FOUNDATION_FIXTURE", "fixture-addon", "./tests/addons/fixture/cmd", false, "auth")
|
||||
paths.Server = build("FOUNDATION_SERVER", "ui-test-server", "./tests/e2e-ui", false, "auth")
|
||||
paths.Mock = os.Getenv("FOUNDATION_MOCK")
|
||||
if paths.Mock == "" {
|
||||
paths.Mock = filepath.Join(root, "tests/e2e/mock-backend/mock-backend")
|
||||
}
|
||||
Expect(paths.Mock).To(BeAnExistingFile(), "build tests/e2e/mock-backend first (scripts/addons/prepare.sh --integration)")
|
||||
exe := build("TABLES_EXECUTABLE", "tables", "./addons/tables/cmd/tables", true)
|
||||
tables, err = os.ReadFile(exe)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
})
|
||||
|
||||
// start brings a cluster up in a fresh directory and takes it down at the end
|
||||
// of the spec.
|
||||
func start() *cluster.Cluster {
|
||||
dir, err := os.MkdirTemp("", "tables-export-cluster-*")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
c, err := cluster.New(dir, paths)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
DeferCleanup(func() {
|
||||
c.Down()
|
||||
_ = filepath.WalkDir(dir, func(p string, d fs.DirEntry, e error) error {
|
||||
if e == nil && d.IsDir() {
|
||||
_ = os.Chmod(p, 0o700)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
_ = os.RemoveAll(dir)
|
||||
})
|
||||
return c
|
||||
}
|
||||
@@ -0,0 +1,209 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package tablesexport_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
const ordersCSV = "Item,Note,Qty\r\n" +
|
||||
"'=SUM(A1:A9),\"line one\nline two\",-4\r\n" +
|
||||
"\"Café, \"\"deluxe\"\"\",日本語,12\r\n" +
|
||||
"plain,,\r\n"
|
||||
|
||||
var seq int
|
||||
|
||||
func op(prefix string) string {
|
||||
seq++
|
||||
return fmt.Sprintf("%s-%d-%d", prefix, time.Now().UnixNano()%1_000_000, seq)
|
||||
}
|
||||
|
||||
// get is one real HTTP request; the headers are what the response carried.
|
||||
func get(c cluster.Client, path string) *cluster.Response {
|
||||
GinkgoHelper()
|
||||
r, err := c.Do("GET", path, nil, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return r
|
||||
}
|
||||
|
||||
var _ = Describe("a CSV export downloaded through the owner and both frontends", func() {
|
||||
It("serves identical bytes and headers from every replica and denies every other way in", func() {
|
||||
c := start()
|
||||
raw, err := os.ReadFile("../../../addons/tables/manifest.json")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
var m struct {
|
||||
Capabilities map[string]any `json:"capabilities"`
|
||||
}
|
||||
Expect(json.Unmarshal(raw, &m)).To(Succeed())
|
||||
delete(m.Capabilities, "ui")
|
||||
Expect(c.Library.Add(cluster.Package{ID: "tables", Version: "0.1.0", Capabilities: m.Capabilities, Executable: "bin/tables", Binary: tables}, c.Effects)).To(Succeed())
|
||||
Expect(c.Up()).To(Succeed())
|
||||
admin := c.Token(cluster.FrontA, "admin")
|
||||
life := func(cl cluster.Client, action string, gen uint64, extra map[string]any) {
|
||||
GinkgoHelper()
|
||||
body := map[string]any{"operationId": op("life"), "addonId": "tables", "action": action, "generation": gen, "arguments": extra}
|
||||
r, err := cl.Do("POST", "/api/addons/dispatch", body, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
}
|
||||
life(admin, "install", 0, map[string]any{"sourceId": "fixture", "version": "0.1.0"})
|
||||
life(c.Token(cluster.FrontB, "admin"), "enable", 1, map[string]any{})
|
||||
Eventually(func() string {
|
||||
r, err := admin.Do("GET", "/api/addons/apps", nil, nil)
|
||||
if err != nil || r.Status != 200 {
|
||||
return "unavailable"
|
||||
}
|
||||
var out struct {
|
||||
Apps []struct{ ID, State string } `json:"apps"`
|
||||
}
|
||||
_ = r.JSON(&out)
|
||||
for _, a := range out.Apps {
|
||||
if a.ID == "tables" {
|
||||
return a.State
|
||||
}
|
||||
}
|
||||
return "absent"
|
||||
}, 40*time.Second, 200*time.Millisecond).Should(Equal("running"))
|
||||
|
||||
// The member creates the database and its rows through frontend A, by
|
||||
// the person's own session, as the page does.
|
||||
member := c.Session(cluster.FrontA, "member")
|
||||
tab := func(id string) map[string]string { return map[string]string{"X-LocalAI-Confirm": id} }
|
||||
created := op("create")
|
||||
r, err := member.Do("POST", "/api/addons/resources", map[string]any{"addonId": "tables", "operationId": created, "generation": 1, "arguments": map[string]any{"name": "Orders data"}}, tab(created))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
var db struct {
|
||||
ID string `json:"id"`
|
||||
Revision map[string]any `json:"revision"`
|
||||
}
|
||||
Expect(r.JSON(&db)).To(Succeed())
|
||||
rev := db.Revision
|
||||
change := func(tool string, args map[string]any) {
|
||||
GinkgoHelper()
|
||||
r, err := member.Do("POST", "/api/addons/resources/"+db.ID+"/apply", map[string]any{"operationId": op("apply"), "generation": 1, "revision": rev, "tool": tool, "arguments": args}, tab(db.ID))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(r.Status).To(Equal(200), r.String())
|
||||
var out struct {
|
||||
Revision map[string]any `json:"revision"`
|
||||
}
|
||||
Expect(r.JSON(&out)).To(Succeed())
|
||||
rev = out.Revision
|
||||
}
|
||||
change("tables-create", map[string]any{"table": "Orders"})
|
||||
change("columns-add", map[string]any{"table": "Orders", "column": "Item", "type": "text"})
|
||||
change("columns-add", map[string]any{"table": "Orders", "column": "Note", "type": "text"})
|
||||
change("columns-add", map[string]any{"table": "Orders", "column": "Qty", "type": "integer"})
|
||||
change("rows-insert", map[string]any{"table": "Orders", "rows": []map[string]any{
|
||||
{"Item": "=SUM(A1:A9)", "Note": "line one\nline two", "Qty": -4},
|
||||
{"Item": "Café, \"deluxe\"", "Note": "日本語", "Qty": 12},
|
||||
{"Item": "plain"},
|
||||
}})
|
||||
share, err := member.Do("PUT", "/api/addons/resources/"+db.ID+"/grants/fnd-friend", map[string]any{"role": "viewer"}, tab(db.ID))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(share.Status).To(Equal(200), share.String())
|
||||
|
||||
// The export is asked for through frontend B and lands at the owner.
|
||||
viaB := c.Token(cluster.FrontB, "member")
|
||||
exp, err := viaB.Do("POST", "/api/addons/dispatch", map[string]any{
|
||||
"operationId": op("export"), "addonId": "tables", "resourceId": db.ID, "generation": 1, "action": "read",
|
||||
"arguments": map[string]any{"tool": "rows-export", "table": "Orders"},
|
||||
}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(exp.Status).To(Equal(200), exp.String())
|
||||
var out struct {
|
||||
Outputs []struct {
|
||||
Handle string `json:"handle"`
|
||||
Name string `json:"name"`
|
||||
MediaType string `json:"mediaType"`
|
||||
Size int64 `json:"size"`
|
||||
} `json:"outputs"`
|
||||
}
|
||||
Expect(exp.JSON(&out)).To(Succeed())
|
||||
Expect(out.Outputs).To(HaveLen(1))
|
||||
id := out.Outputs[0].Handle
|
||||
Expect(out.Outputs[0].Size).To(Equal(int64(len(ordersCSV))))
|
||||
|
||||
Eventually(func() string {
|
||||
s := get(c.Token(cluster.FrontA, "member"), "/api/addons/artifacts/"+id)
|
||||
var st struct {
|
||||
State string `json:"state"`
|
||||
}
|
||||
_ = s.JSON(&st)
|
||||
return st.State
|
||||
}, 20*time.Second, 100*time.Millisecond).Should(Equal("complete"))
|
||||
|
||||
// The same bytes and the same headers from the owner and both frontends.
|
||||
var first *cluster.Response
|
||||
for _, node := range []string{cluster.Owner, cluster.FrontA, cluster.FrontB} {
|
||||
dl := get(c.Token(node, "member"), "/api/addons/artifacts/"+id+"/download")
|
||||
Expect(dl.Status).To(Equal(200), node+" "+dl.String())
|
||||
Expect(string(dl.Body)).To(Equal(ordersCSV), node)
|
||||
Expect(dl.Header.Get("Content-Type")).To(Equal("text/csv"), node)
|
||||
Expect(dl.Header.Get("Content-Disposition")).To(HavePrefix("attachment;"), node)
|
||||
Expect(dl.Header.Get("Content-Disposition")).To(ContainSubstring("Orders.csv"), node)
|
||||
Expect(dl.Header.Get("X-Content-Type-Options")).To(Equal("nosniff"), node)
|
||||
Expect(dl.Header.Get("Cache-Control")).To(ContainSubstring("no-store"), node)
|
||||
Expect(dl.Header.Get("Content-Length")).To(Equal(fmt.Sprint(len(ordersCSV))), node)
|
||||
if first == nil {
|
||||
first = dl
|
||||
continue
|
||||
}
|
||||
for _, k := range []string{"Content-Type", "Content-Disposition", "X-Content-Type-Options", "Cache-Control", "Content-Length"} {
|
||||
Expect(dl.Header.Get(k)).To(Equal(first.Header.Get(k)), node+" "+k)
|
||||
}
|
||||
}
|
||||
|
||||
// A person the database is shared with, and one without the feature,
|
||||
// are denied the file on every replica exactly like an unknown ID.
|
||||
unknown := "a0123456789abcdef0123456789abcde"
|
||||
for _, node := range []string{cluster.Owner, cluster.FrontA, cluster.FrontB} {
|
||||
base := get(c.Token(node, "friend"), "/api/addons/artifacts/"+unknown+"/download")
|
||||
Expect(base.Status).To(Equal(http.StatusForbidden), node)
|
||||
for _, who := range []string{"friend", "admin", "outsider"} {
|
||||
dl := get(c.Token(node, who), "/api/addons/artifacts/"+id+"/download")
|
||||
Expect(dl.Status).To(Equal(http.StatusForbidden), node+" "+who)
|
||||
if who != "outsider" {
|
||||
Expect(string(dl.Body)).To(Equal(string(base.Body)), node+" "+who)
|
||||
}
|
||||
st := get(c.Token(node, who), "/api/addons/artifacts/"+id)
|
||||
Expect(st.Status).To(Equal(http.StatusForbidden), node+" "+who)
|
||||
}
|
||||
}
|
||||
|
||||
// The friend exports what they may read; withdrawing the grant ends
|
||||
// the download on every replica.
|
||||
fr, err := c.Token(cluster.FrontA, "friend").Do("POST", "/api/addons/dispatch", map[string]any{
|
||||
"operationId": op("export-friend"), "addonId": "tables", "resourceId": db.ID, "generation": 1, "action": "read",
|
||||
"arguments": map[string]any{"tool": "rows-export", "table": "Orders", "columns": []string{"Item"}},
|
||||
}, nil)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(fr.Status).To(Equal(200), fr.String())
|
||||
var fo struct {
|
||||
Outputs []struct {
|
||||
Handle string `json:"handle"`
|
||||
} `json:"outputs"`
|
||||
}
|
||||
Expect(fr.JSON(&fo)).To(Succeed())
|
||||
fid := fo.Outputs[0].Handle
|
||||
Eventually(func() int {
|
||||
return get(c.Token(cluster.FrontB, "friend"), "/api/addons/artifacts/"+fid+"/download").Status
|
||||
}, 20*time.Second, 100*time.Millisecond).Should(Equal(200))
|
||||
Expect(string(get(c.Token(cluster.FrontB, "friend"), "/api/addons/artifacts/"+fid+"/download").Body)).To(Equal("Item\r\n'=SUM(A1:A9)\r\n\"Café, \"\"deluxe\"\"\"\r\nplain\r\n"))
|
||||
gone, err := member.Do("DELETE", "/api/addons/resources/"+db.ID+"/grants/fnd-friend", nil, tab(db.ID))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(gone.Status).To(Equal(200), gone.String())
|
||||
for _, node := range []string{cluster.Owner, cluster.FrontA, cluster.FrontB} {
|
||||
Expect(get(c.Token(node, "friend"), "/api/addons/artifacts/"+fid+"/download").Status).To(Equal(http.StatusForbidden), node)
|
||||
}
|
||||
// The member's own file is unaffected.
|
||||
Expect(string(get(c.Token(cluster.FrontB, "member"), "/api/addons/artifacts/"+id+"/download").Body)).To(Equal(ordersCSV))
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,454 @@
|
||||
package main
|
||||
|
||||
// The add-ons UI fixture: with --addons-fixture the server runs with
|
||||
// authentication on and a real add-on owner (lifecycle manager, supervisor,
|
||||
// gateway and mTLS carrier), and installs the packages of the fixture
|
||||
// directory through a real catalog served on loopback. A spec then drives the
|
||||
// shell against real authorization: sessions and API keys are stored keys,
|
||||
// every asset and call is decided by the owner, and enable, disable and
|
||||
// upgrade are the real lifecycle routes.
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/ed25519"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"crypto/x509/pkix"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"math/big"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/mudler/LocalAI/core/application"
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/http/auth"
|
||||
addonsvc "github.com/mudler/LocalAI/core/services/addons"
|
||||
"github.com/mudler/LocalAI/core/services/addons/catalog"
|
||||
"github.com/mudler/LocalAI/core/services/addons/gateway"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
)
|
||||
|
||||
const fixtureSecret = "addons-ui-fixture-secret" // #nosec G101 -- a fixed secret of the local test fixture, not a credential
|
||||
|
||||
type addonsFixture struct {
|
||||
dataDir string
|
||||
listener net.Listener
|
||||
catalog *http.Server
|
||||
library string
|
||||
releases []catalog.Release
|
||||
archives map[string][]byte
|
||||
owner config.AddonConfig
|
||||
// localOwner runs the owner without authentication, in local-owner mode.
|
||||
localOwner bool
|
||||
// mu guards what a spec moves through the control routes: the step of
|
||||
// the moving channels and whether the library answers at all.
|
||||
mu sync.Mutex
|
||||
step int
|
||||
offline bool
|
||||
moving []movingRelease
|
||||
}
|
||||
|
||||
// movingRelease is a release of a package directory that carries
|
||||
// .fixture-generation: of the releases of one add-on and channel, only the
|
||||
// one with the highest generation not above the current step is listed. A spec
|
||||
// moves the channel by raising the step, as a publisher moves a tag.
|
||||
type movingRelease struct {
|
||||
catalog.Release
|
||||
generation int
|
||||
}
|
||||
|
||||
// Credentials is what the spec reads from --fixture-out.
|
||||
type fixtureCredentials struct {
|
||||
Users map[string]fixtureUser `json:"users"`
|
||||
// Packages lists every installable release as "id@version".
|
||||
Packages []string `json:"packages"`
|
||||
// Control is the base URL of the fixture library's control routes: POST
|
||||
// /step moves every moving channel on by one generation, POST /offline and
|
||||
// /online make the library answer or not. Empty without moving channels.
|
||||
Control string `json:"control,omitempty"`
|
||||
}
|
||||
|
||||
type fixtureUser struct {
|
||||
ID string `json:"id"`
|
||||
Role string `json:"role"`
|
||||
Token string `json:"token"`
|
||||
Session string `json:"session"`
|
||||
}
|
||||
|
||||
func newAddonsFixture(dir, executable, dataDir string, localOwner bool) (*addonsFixture, error) {
|
||||
f := &addonsFixture{dataDir: dataDir, archives: map[string][]byte{}, localOwner: localOwner, step: 1}
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
exe, err := os.ReadFile(executable) // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("fixture executable: %w", err)
|
||||
}
|
||||
for _, entry := range entries {
|
||||
if !entry.IsDir() {
|
||||
continue
|
||||
}
|
||||
pkg := filepath.Join(dir, entry.Name())
|
||||
raw, err := os.ReadFile(filepath.Join(pkg, "manifest.json")) // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var manifest map[string]any
|
||||
if err := json.Unmarshal(raw, &manifest); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", entry.Name(), err)
|
||||
}
|
||||
// The package is for this machine: the fixture sources are portable.
|
||||
manifest["os"], manifest["arch"] = runtime.GOOS, runtime.GOARCH
|
||||
patched, _ := json.Marshal(manifest)
|
||||
var buf bytes.Buffer
|
||||
tw := tar.NewWriter(&buf)
|
||||
add := func(name string, data []byte) error {
|
||||
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o755, Size: int64(len(data))}); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := tw.Write(data)
|
||||
return err
|
||||
}
|
||||
if err := add("manifest.json", patched); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// The executable goes where the manifest says it is. The protocol
|
||||
// fixture packages all say bin/fixture; a package that names another
|
||||
// path (the Tables package) gets the supplied executable there.
|
||||
exePath := "bin/fixture"
|
||||
if declared, _ := manifest["executable"].(string); declared != "" && !strings.HasPrefix(declared, "/") && !strings.Contains(declared, "..") {
|
||||
exePath = declared
|
||||
}
|
||||
if err := add(exePath, exe); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var files []string
|
||||
_ = filepath.WalkDir(pkg, func(p string, d fs.DirEntry, err error) error {
|
||||
if err == nil && !d.IsDir() {
|
||||
rel, _ := filepath.Rel(pkg, p)
|
||||
// The .fixture-* files describe the release to the library and
|
||||
// are not part of the package.
|
||||
if rel != "manifest.json" && !strings.HasPrefix(rel, ".fixture-") {
|
||||
files = append(files, rel)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
sort.Strings(files)
|
||||
for _, rel := range files {
|
||||
data, err := os.ReadFile(filepath.Join(pkg, rel)) // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := add(filepath.ToSlash(rel), data); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if err := tw.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
id, _ := manifest["id"].(string)
|
||||
version, _ := manifest["version"].(string)
|
||||
f.archives[entry.Name()+".tar"] = buf.Bytes()
|
||||
release := catalog.Release{
|
||||
ID: id, Name: id, Version: version, OS: runtime.GOOS, Arch: runtime.GOARCH,
|
||||
Digest: fmt.Sprintf("sha256:%x", sha256.Sum256(buf.Bytes())), Artifact: "/" + entry.Name() + ".tar",
|
||||
}
|
||||
if channel, err := os.ReadFile(filepath.Join(pkg, ".fixture-channel")); err == nil { // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
release.Channel = strings.TrimSpace(string(channel))
|
||||
}
|
||||
if raw, err := os.ReadFile(filepath.Join(pkg, ".fixture-requires")); err == nil { // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
if err := json.Unmarshal(raw, &release.Requires); err != nil {
|
||||
return nil, fmt.Errorf("%s: .fixture-requires: %w", entry.Name(), err)
|
||||
}
|
||||
}
|
||||
if raw, err := os.ReadFile(filepath.Join(pkg, ".fixture-generation")); err == nil { // #nosec G304 -- test harness reads files it created in its own work directory
|
||||
n, err := strconv.Atoi(strings.TrimSpace(string(raw)))
|
||||
if err != nil || n < 1 {
|
||||
return nil, fmt.Errorf("%s: .fixture-generation must be a positive number", entry.Name())
|
||||
}
|
||||
f.moving = append(f.moving, movingRelease{Release: release, generation: n})
|
||||
continue
|
||||
}
|
||||
f.releases = append(f.releases, release)
|
||||
}
|
||||
if len(f.releases) == 0 && len(f.moving) == 0 {
|
||||
return nil, fmt.Errorf("no package under %s", dir)
|
||||
}
|
||||
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
f.listener = listener
|
||||
base := "http://" + listener.Addr().String()
|
||||
for i := range f.releases {
|
||||
f.releases[i].Artifact = base + f.releases[i].Artifact
|
||||
}
|
||||
for i := range f.moving {
|
||||
f.moving[i].Artifact = base + f.moving[i].Artifact
|
||||
}
|
||||
f.library = base + "/library.yaml"
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/library.yaml", func(w http.ResponseWriter, r *http.Request) {
|
||||
f.mu.Lock()
|
||||
offline, listed := f.offline, f.listed()
|
||||
f.mu.Unlock()
|
||||
if offline {
|
||||
http.Error(w, "library offline", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
_ = catalog.WriteIndex(w, listed)
|
||||
})
|
||||
// The control routes of a spec: move the channels on and take the library
|
||||
// away. They exist only in this test server.
|
||||
mux.HandleFunc("/_fixture/step", func(w http.ResponseWriter, r *http.Request) {
|
||||
f.mu.Lock()
|
||||
f.step++
|
||||
n := f.step
|
||||
f.mu.Unlock()
|
||||
_ = json.NewEncoder(w).Encode(map[string]int{"step": n})
|
||||
})
|
||||
for path, offline := range map[string]bool{"/_fixture/offline": true, "/_fixture/online": false} {
|
||||
mux.HandleFunc(path, func(w http.ResponseWriter, r *http.Request) {
|
||||
f.mu.Lock()
|
||||
f.offline = offline
|
||||
f.mu.Unlock()
|
||||
_ = json.NewEncoder(w).Encode(map[string]bool{"offline": offline})
|
||||
})
|
||||
}
|
||||
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
|
||||
data, ok := f.archives[strings.TrimPrefix(r.URL.Path, "/")]
|
||||
if !ok {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write(data)
|
||||
})
|
||||
f.catalog = &http.Server{Handler: mux, ReadHeaderTimeout: 5 * time.Second}
|
||||
go func() { _ = f.catalog.Serve(listener) }()
|
||||
|
||||
transport, address, err := fixtureTransport()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
f.owner = config.AddonConfig{
|
||||
Enabled: true, InstanceID: "owner", OwnerID: "owner", OwnerAddress: "https://" + address, StorageID: "disk",
|
||||
Root: filepath.Join(dataDir, "addons"), ListenAddress: address, Transport: transport,
|
||||
Host: contract.HostCompatibility{Host: 1, API: 1, Protocol: 1, SDK: 1, DataFormat: 1, OS: runtime.GOOS, Arch: runtime.GOARCH},
|
||||
// A spec checks again within seconds: the interval of the refresh
|
||||
// limit is short here and the limit itself stays on.
|
||||
UpdateRefreshInterval: time.Second,
|
||||
}
|
||||
if localOwner {
|
||||
f.owner.OwnershipMode = auth.AddonModeLocalOwner
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// listed is the library document at the current step: the releases that never
|
||||
// move and, of each moving add-on and channel, the one the step has reached.
|
||||
func (f *addonsFixture) listed() []catalog.Release {
|
||||
out := append([]catalog.Release(nil), f.releases...)
|
||||
best := map[string]movingRelease{}
|
||||
for _, m := range f.moving {
|
||||
if m.generation > f.step {
|
||||
continue
|
||||
}
|
||||
key := m.ID + "/" + m.Channel + "/" + m.Version
|
||||
if cur, ok := best[key]; !ok || m.generation > cur.generation {
|
||||
best[key] = m
|
||||
}
|
||||
}
|
||||
keys := make([]string, 0, len(best))
|
||||
for k := range best {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
out = append(out, best[k].Release)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// fixtureTransport builds the owner's mTLS carrier identity: one pinned
|
||||
// certificate and one signing key, as a single-instance deployment has.
|
||||
func fixtureTransport() (*gateway.CarrierConfig, string, error) {
|
||||
probe, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
address := probe.Addr().String()
|
||||
_ = probe.Close()
|
||||
pub, key, err := ed25519.GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
ca := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "fixture CA"}, NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(24 * time.Hour), IsCA: true, BasicConstraintsValid: true, KeyUsage: x509.KeyUsageCertSign}
|
||||
der, err := x509.CreateCertificate(rand.Reader, ca, ca, pub, key)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if ca, err = x509.ParseCertificate(der); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
pool := x509.NewCertPool()
|
||||
pool.AddCert(ca)
|
||||
lp, lk, err := ed25519.GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
leaf := &x509.Certificate{SerialNumber: big.NewInt(2), Subject: pkix.Name{CommonName: "owner"}, NotBefore: ca.NotBefore, NotAfter: ca.NotAfter, IPAddresses: []net.IP{net.ParseIP("127.0.0.1")}, KeyUsage: x509.KeyUsageDigitalSignature, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth, x509.ExtKeyUsageClientAuth}}
|
||||
ld, err := x509.CreateCertificate(rand.Reader, leaf, ca, lp, key)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
kd, err := x509.MarshalPKCS8PrivateKey(lk)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
pair, err := tls.X509KeyPair(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: ld}), pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: kd}))
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
signPub, signKey, err := ed25519.GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
return &gateway.CarrierConfig{
|
||||
InstanceID: "owner", OwnerID: "owner", OwnerAddress: "https://" + address,
|
||||
Certificate: pair, Roots: pool,
|
||||
Pins: map[string][32]byte{"owner": sha256.Sum256(ld)},
|
||||
SigningKey: signKey,
|
||||
VerificationKeys: map[string]ed25519.PublicKey{"owner": signPub},
|
||||
}, address, nil
|
||||
}
|
||||
|
||||
func (f *addonsFixture) options(dataDir string) []config.AppOption {
|
||||
if f.localOwner {
|
||||
// One local owner and no accounts: sharing has nobody to share with.
|
||||
return []config.AppOption{config.WithAddons(f.owner)}
|
||||
}
|
||||
return []config.AppOption{
|
||||
config.WithAuthEnabled(true),
|
||||
config.WithAuthDatabaseURL(filepath.Join(dataDir, "auth.db")),
|
||||
config.WithAuthAPIKeyHMACSecret(fixtureSecret),
|
||||
config.WithAddons(f.owner),
|
||||
}
|
||||
}
|
||||
|
||||
// seed creates the accounts, saves the library and installs and enables the
|
||||
// packages named in `enable` ("id@version").
|
||||
func (f *addonsFixture) seed(ctx context.Context, app *application.Application, out string, enable []string) error {
|
||||
db := app.AuthDB()
|
||||
if db == nil && !f.localOwner {
|
||||
return fmt.Errorf("authentication is not enabled")
|
||||
}
|
||||
creds := fixtureCredentials{Users: map[string]fixtureUser{}}
|
||||
var admin contract.Actor
|
||||
if f.localOwner {
|
||||
admin = contract.Actor{UserID: auth.AddonLocalOwnerID, CredentialID: auth.AddonLocalOwnerID, Mode: auth.AddonModeLocalOwner}
|
||||
}
|
||||
accounts := []struct {
|
||||
name, role string
|
||||
feature bool
|
||||
}{{"admin", auth.RoleAdmin, true}, {"member", auth.RoleUser, true}, {"outsider", auth.RoleUser, false},
|
||||
// Two more people with the feature, for the roles of a shared database.
|
||||
{"editor", auth.RoleUser, true}, {"viewer", auth.RoleUser, true}}
|
||||
if f.localOwner {
|
||||
accounts = nil
|
||||
}
|
||||
for _, u := range accounts {
|
||||
user := auth.User{ID: "fixture-" + u.name, Email: u.name + "@fixture.test", Name: u.name, Provider: auth.ProviderLocal, Role: u.role, Status: auth.StatusActive}
|
||||
if err := db.Create(&user).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if u.feature && u.role != auth.RoleAdmin {
|
||||
if err := db.Create(&auth.UserPermission{ID: uuid.NewString(), UserID: user.ID, Permissions: auth.PermissionMap{auth.FeatureAddons: true}}).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
token, key, err := auth.CreateAPIKey(db, user.ID, "fixture", u.role, fixtureSecret, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
session, err := auth.CreateSession(db, user.ID, fixtureSecret)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
creds.Users[u.name] = fixtureUser{ID: user.ID, Role: u.role, Token: token, Session: session}
|
||||
if u.name == "admin" {
|
||||
admin = contract.Actor{UserID: user.ID, CredentialID: "key:" + key.ID, Mode: auth.AddonModeNamedUser}
|
||||
}
|
||||
}
|
||||
sources, manager := app.AddonSources(), app.AddonManager()
|
||||
if sources == nil || manager == nil {
|
||||
return fmt.Errorf("the add-on owner did not start")
|
||||
}
|
||||
if err := sources.Save(ctx, admin, catalog.Source{ID: "fixture", Name: "Fixture", URL: f.library, Enabled: true, AllowLoopbackHTTP: true}); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := sources.Refresh(ctx, "fixture"); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, r := range f.listed() {
|
||||
creds.Packages = append(creds.Packages, r.ID+"@"+r.Version)
|
||||
}
|
||||
if len(f.moving) > 0 {
|
||||
creds.Control = "http://" + f.listener.Addr().String() + "/_fixture"
|
||||
}
|
||||
for i, spec := range enable {
|
||||
id, version, _ := strings.Cut(spec, "@")
|
||||
if _, err := manager.Submit(ctx, admin, addonsvc.LifecycleCommand{OperationID: fmt.Sprintf("seed-install-%d", i), AddonID: id, Action: "install", SourceID: "fixture", Version: version}); err != nil {
|
||||
return fmt.Errorf("install %s: %w", spec, err)
|
||||
}
|
||||
if _, err := manager.Submit(ctx, admin, addonsvc.LifecycleCommand{OperationID: fmt.Sprintf("seed-enable-%d", i), AddonID: id, Action: "enable", Generation: 1}); err != nil {
|
||||
return fmt.Errorf("enable %s: %w", spec, err)
|
||||
}
|
||||
}
|
||||
// Wait for every enabled process to be observed running, so a spec starts
|
||||
// from a settled owner.
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
list, err := manager.Apps(ctx, admin)
|
||||
if err == nil {
|
||||
running := 0
|
||||
for _, a := range list {
|
||||
if a.State == "running" {
|
||||
running++
|
||||
}
|
||||
}
|
||||
if running >= len(enable) {
|
||||
break
|
||||
}
|
||||
}
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
}
|
||||
b, err := json.MarshalIndent(creds, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.WriteFile(out, b, 0o600)
|
||||
}
|
||||
|
||||
func (f *addonsFixture) close() {
|
||||
if f.catalog != nil {
|
||||
_ = f.catalog.Close()
|
||||
}
|
||||
}
|
||||
+84
-12
@@ -15,6 +15,7 @@ import (
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
httpapi "github.com/mudler/LocalAI/core/http"
|
||||
"github.com/mudler/LocalAI/pkg/system"
|
||||
"github.com/mudler/LocalAI/tests/addons/foundation/cluster"
|
||||
"github.com/mudler/xlog"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
@@ -37,6 +38,17 @@ func main() {
|
||||
// the routing E2E to turn mock-model into a smart-router that
|
||||
// dispatches to extra-models.
|
||||
routerYAML := flag.String("router-yaml", "", "optional router: block to merge into mock-model.yaml")
|
||||
// addonsFixture turns on authentication and a real add-on owner and
|
||||
// installs the packages of that directory (see addons_fixture.go).
|
||||
addonsFixtureDir := flag.String("addons-fixture", "", "directory of add-on UI fixture packages; enables auth and a real add-on owner")
|
||||
addonExecutable := flag.String("addon-executable", "", "protocol fixture executable placed in every fixture package")
|
||||
fixtureOut := flag.String("fixture-out", "", "file the seeded credentials are written to as JSON")
|
||||
localOwner := flag.Bool("addons-local-owner", false, "run the add-ons fixture without authentication, in local-owner mode (no named users, no sharing)")
|
||||
fixtureEnable := flag.String("fixture-enable", "", "comma separated id@version packages to install and enable at startup")
|
||||
// foundation runs this server as one instance of the foundation
|
||||
// acceptance cluster (see tests/addons/foundation/cluster).
|
||||
foundationPath := flag.String("foundation", "", "cluster.json of the add-ons foundation fixture; enables authentication and the add-on carrier")
|
||||
foundationRole := flag.String("foundation-role", "", "instance of the foundation cluster: owner, front-a or front-b")
|
||||
flag.Parse()
|
||||
|
||||
if *mockBackend == "" {
|
||||
@@ -55,20 +67,36 @@ func main() {
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Create temp dirs
|
||||
tmpDir, err := os.MkdirTemp("", "ui-e2e-*")
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error creating temp dir: %v\n", err)
|
||||
os.Exit(1)
|
||||
// Create temp dirs. A foundation instance keeps its state in the cluster
|
||||
// directory, which survives a restart of the process.
|
||||
var found *cluster.Config
|
||||
var tmpDir string
|
||||
if *foundationPath != "" {
|
||||
found, err = cluster.Load(*foundationPath)
|
||||
if err != nil || found.Nodes[*foundationRole].Dir == "" {
|
||||
fmt.Fprintf(os.Stderr, "error loading the foundation cluster for %q: %v\n", *foundationRole, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
tmpDir = found.StateDir(*foundationRole)
|
||||
if err := os.MkdirAll(tmpDir, 0o750); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error creating the foundation state dir: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
} else {
|
||||
tmpDir, err = os.MkdirTemp("", "ui-e2e-*")
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error creating temp dir: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer func() { _ = os.RemoveAll(tmpDir) }()
|
||||
}
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
modelsPath := filepath.Join(tmpDir, "models")
|
||||
backendsPath := filepath.Join(tmpDir, "backends")
|
||||
generatedDir := filepath.Join(tmpDir, "generated")
|
||||
dataDir := filepath.Join(tmpDir, "data")
|
||||
for _, d := range []string{modelsPath, backendsPath, generatedDir, dataDir} {
|
||||
if err := os.MkdirAll(d, 0755); err != nil {
|
||||
if err := os.MkdirAll(d, 0750); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error creating dir %s: %v\n", d, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
@@ -94,7 +122,7 @@ func main() {
|
||||
if *routerYAML != "" {
|
||||
body = append(body, []byte("router:\n "+*routerYAML+"\n")...)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(modelsPath, "mock-model.yaml"), body, 0644); err != nil {
|
||||
if err := os.WriteFile(filepath.Join(modelsPath, "mock-model.yaml"), body, 0600); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error writing config: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
@@ -107,7 +135,7 @@ func main() {
|
||||
parts := strings.SplitN(*extraModelFlag, "|", 2)
|
||||
if len(parts) == 2 {
|
||||
extraPath := filepath.Join(modelsPath, parts[0]+".yaml")
|
||||
if err := os.WriteFile(extraPath, []byte(parts[1]), 0644); err != nil {
|
||||
if err := os.WriteFile(extraPath, []byte(parts[1]), 0600); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error writing extra model: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
@@ -128,7 +156,7 @@ func main() {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
app, err := application.New(
|
||||
appOptions := []config.AppOption{
|
||||
config.WithContext(ctx),
|
||||
config.WithSystemState(systemState),
|
||||
config.WithDebug(true),
|
||||
@@ -137,11 +165,55 @@ func main() {
|
||||
config.WithGeneratedContentDir(generatedDir),
|
||||
config.EnableTracing,
|
||||
config.EnableBackendLogging,
|
||||
)
|
||||
}
|
||||
var fixture *addonsFixture
|
||||
if *addonsFixtureDir != "" {
|
||||
fixture, err = newAddonsFixture(*addonsFixtureDir, *addonExecutable, dataDir, *localOwner)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error preparing the add-ons fixture: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer fixture.close()
|
||||
appOptions = append(appOptions, fixture.options(dataDir)...)
|
||||
}
|
||||
if found != nil {
|
||||
opts, err := found.AppOptions(*foundationRole)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error preparing the foundation cluster: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
appOptions = append(appOptions, opts...)
|
||||
}
|
||||
app, err := application.New(appOptions...)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error creating application: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if fixture != nil {
|
||||
var enable []string
|
||||
for _, spec := range strings.Split(*fixtureEnable, ",") {
|
||||
if spec = strings.TrimSpace(spec); spec != "" {
|
||||
enable = append(enable, spec)
|
||||
}
|
||||
}
|
||||
if err := fixture.seed(ctx, app, *fixtureOut, enable); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error seeding the add-ons fixture: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
if found != nil && *foundationRole == cluster.Owner {
|
||||
if err := found.Seed(ctx, app); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error seeding the foundation cluster: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
control, err := found.ControlServer(app)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "error starting the foundation control server: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer func() { _ = control.Close() }()
|
||||
}
|
||||
|
||||
// Register mock backend
|
||||
app.ModelLoader().SetExternalBackend("mock-backend", absBackend)
|
||||
@@ -171,5 +243,5 @@ func main() {
|
||||
|
||||
fmt.Println("\nShutting down...")
|
||||
cancel()
|
||||
e.Close()
|
||||
_ = e.Close()
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
@@ -11,6 +12,7 @@ import (
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -75,6 +77,59 @@ func checkModelIdentity(in interface{ GetModelIdentity() string }) error {
|
||||
return grpcerrors.ModelMismatch("mock-backend", opts.Model, in.GetModelIdentity())
|
||||
}
|
||||
|
||||
// addonToolCall reads a scripted add-on tool call from the prompt. A spec puts
|
||||
// ADDON_TOOL_CALL:<tool name>:<base64url JSON arguments> in the user message;
|
||||
// the model then answers its first turn with that call, as a model that chose
|
||||
// the tool would. The arguments are exactly what the spec encoded, so a spec
|
||||
// can send valid, forged or malformed ones. The second turn, after the host
|
||||
// put the tool result into the prompt, is plain text.
|
||||
var addonToolMarker = regexp.MustCompile(`ADDON_TOOL_CALL:([A-Za-z0-9_-]+):([A-Za-z0-9_=-]+)`)
|
||||
|
||||
func addonToolCall(prompt string) (name, arguments string, ok bool) {
|
||||
m := addonToolMarker.FindStringSubmatch(prompt)
|
||||
if m == nil {
|
||||
return "", "", false
|
||||
}
|
||||
raw, err := base64.URLEncoding.DecodeString(m[2])
|
||||
if err != nil {
|
||||
raw, err = base64.RawURLEncoding.DecodeString(m[2])
|
||||
}
|
||||
if err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
return m[1], string(raw), true
|
||||
}
|
||||
|
||||
// promptHasAddonFile reports whether the host answered the scripted call with
|
||||
// private file references (an export).
|
||||
func promptHasAddonFile(prompt string) bool {
|
||||
return strings.Contains(prompt, `"kind":"artifact"`) || strings.Contains(prompt, `"kind": "artifact"`)
|
||||
}
|
||||
|
||||
// addonAnswer is the plain text the scripted model gives after the host
|
||||
// answered its call: a file result is ready to download, anything else is a
|
||||
// change that waits for the person.
|
||||
func addonAnswer(prompt string) string {
|
||||
// A call the add-on refused as invalid ends in an error result with the
|
||||
// code invalid_request: nothing was changed and the model says so.
|
||||
if strings.Contains(prompt, `"code":"invalid_request"`) || strings.Contains(prompt, `"code": "invalid_request"`) {
|
||||
return "The add-on refused the request as invalid."
|
||||
}
|
||||
if promptHasAddonFile(prompt) {
|
||||
return "The file is ready to download."
|
||||
}
|
||||
return "The change is waiting for your confirmation."
|
||||
}
|
||||
|
||||
// promptHasAddonResult reports whether the host already answered the scripted
|
||||
// call: the broker's result documents carry a kind (a proposal, a result or an
|
||||
// error such as select_resource), and a refusal is prefixed.
|
||||
func promptHasAddonResult(prompt string) bool {
|
||||
return strings.Contains(prompt, `"kind":"proposal"`) || strings.Contains(prompt, `"kind":"result"`) || strings.Contains(prompt, `"kind":"error"`) ||
|
||||
strings.Contains(prompt, `"kind": "proposal"`) || strings.Contains(prompt, `"kind": "result"`) || strings.Contains(prompt, `"kind": "error"`) || strings.Contains(prompt, "Error: ") ||
|
||||
promptHasAddonFile(prompt)
|
||||
}
|
||||
|
||||
// promptHasToolResults checks if the prompt contains evidence of prior tool
|
||||
// execution — specifically the output from the mock MCP server's get_weather tool.
|
||||
func promptHasToolResults(prompt string) bool {
|
||||
@@ -251,7 +306,13 @@ func (m *MockBackend) Predict(ctx context.Context, in *pb.PredictOptions) (*pb.R
|
||||
}
|
||||
var response string
|
||||
toolName := mockToolNameFromRequest(in)
|
||||
if toolName != "" && !promptHasToolResults(in.Prompt) {
|
||||
if name, arguments, ok := addonToolCall(in.Prompt); ok && toolName != "" {
|
||||
if promptHasAddonResult(in.Prompt) {
|
||||
response = addonAnswer(in.Prompt)
|
||||
} else {
|
||||
response = fmt.Sprintf(`{"name": %q, "arguments": %s}`, name, arguments)
|
||||
}
|
||||
} else if toolName != "" && !promptHasToolResults(in.Prompt) {
|
||||
// First call with tools: return a tool call so the MCP loop executes it.
|
||||
response = fmt.Sprintf(`{"name": "%s", "arguments": {"location": "San Francisco"}}`, toolName)
|
||||
} else if toolName != "" {
|
||||
@@ -403,8 +464,18 @@ func (m *MockBackend) PredictStream(in *pb.PredictOptions, stream pb.Backend_Pre
|
||||
}
|
||||
|
||||
var toStream string
|
||||
wholeChunk := false
|
||||
toolName := mockToolNameFromRequest(in)
|
||||
addonName, addonArgs, addonCall := addonToolCall(in.Prompt)
|
||||
switch {
|
||||
case addonCall && toolName != "" && promptHasAddonResult(in.Prompt):
|
||||
toStream = addonAnswer(in.Prompt)
|
||||
case addonCall && toolName != "":
|
||||
// One chunk: the chat handler emits a tool call as soon as the name
|
||||
// is complete, so a call streamed byte by byte would reach it with
|
||||
// empty arguments, which no real backend does.
|
||||
toStream = fmt.Sprintf(`{"name": %q, "arguments": %s}`, addonName, addonArgs)
|
||||
wholeChunk = true
|
||||
case toolName != "" && !promptHasToolResults(in.Prompt):
|
||||
toStream = fmt.Sprintf(`{"name": "%s", "arguments": {"location": "San Francisco"}}`, toolName)
|
||||
case toolName != "":
|
||||
@@ -418,6 +489,9 @@ func (m *MockBackend) PredictStream(in *pb.PredictOptions, stream pb.Backend_Pre
|
||||
default:
|
||||
toStream = "This is a mocked streaming response."
|
||||
}
|
||||
if wholeChunk {
|
||||
return stream.Send(&pb.Reply{Message: []byte(toStream), Tokens: int32(len(toStream))}) // #nosec G115 -- the length of a short fixed string
|
||||
}
|
||||
for i, r := range toStream {
|
||||
if err := stream.Send(&pb.Reply{
|
||||
Message: []byte(string(r)),
|
||||
|
||||
Reference in new issue
Block a user