mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
feat(addons): installable add-ons foundation and the Tables add-on (#12702)
* feat(addons): add the add-on runtime, navigation and first-party add-ons Add the add-on runtime to LocalAI: manifests, the host operation broker, admin gating by account role, the add-on UI bundle builder and the SDK for add-on pages. Move the navigation to Chat, Models, Add-ons and Settings, with pinned add-ons in the sidebar and an Add-ons drawer. Ship the first-party add-ons (voice, face, fine-tuning, quantization) as packages for linux/amd64, linux/arm64 and darwin/arm64. Intel macOS (darwin/amd64) is enabled in the platform files. Squashed from the review branch. The tree is unchanged; the pre-squash history is kept on a local backup branch. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gosec): tighten file modes and annotate false positives Write job state, configs and imported models with mode 0600, clean the paths that are read from disk, and mark the route tables, the header name and the retry jitter as reviewed false positives. Assisted-by: Claude Code:claude-sonnet-5-5 gosec Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): keep API keys off recovery mutations and same-site cookies POST /api/addons/dispatch forwarded the recovery and update mutations to the owner for any administrator key, which skipped the browser-session rule of their own routes. Refuse those actions there. Also apply the same-origin rule to the legacy token cookie, not only the session cookie, because the global CSRF check lets same-site requests through. Do not fail the start of LocalAI when an add-on library cannot be refreshed; log it and keep the installed add-ons up. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): refuse a reinstall that would migrate retained data unbacked An upgrade takes a verified backup before the new package migrates the data. A reinstall over retained data has no running generation to stop, so the new package would migrate it without one. Refuse a reinstall whose data format differs from the retained one. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): release the manager lock while a release is fetched Submit held the state lock across the catalog lookup and the package download, so the apps list and status reads stalled for the whole install. Serialize Submit and Reconcile with their own lock and drop the state lock around the fetch only. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): kill an add-on only after three missed health probes One slow 100 ms probe killed a busy add-on and dropped the work it was serving. Require three misses in a row. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): move settings under the operate hub The Operate hub already has Settings as a tab, so the sidebar no longer shows a separate Settings entry. The /app/settings route stays, and the hub tab still reaches it. The navigation test now expects Chat, Models and Add-ons as the primary items. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): accept same-host browser requests that carry no fetch metadata Browsers do not send Sec-Fetch-* headers over plain HTTP on an address that is not localhost, so every add-on request from a LAN address was refused as cross-site. Keep refusing a Sec-Fetch-Site value other than same-origin. When the header is absent, accept the request only if its Origin, or failing that its Referer, names the same host and port as the request. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): show the server's reason when an add-on request is refused A 403 or 400 answer showed a generic message and hid the reason in the response body, so a refusal by the browser guard could not be told from a missing permission. Show error.message from the body as the server sent it, and keep the generic text when the body has none. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): run the add-on owner in the normal binary The owner was only enabled when an embedding host set the add-on config, so a normal run answered every add-on request as unavailable. Add --addons-enabled (default true) and --addons-ownership-mode (default named-user, or local-owner; anything else stops startup). The node is its own owner: state lives under the data path, and the carrier uses loopback keys made at each start and kept in memory. --addons-enabled=false turns the owner off completely. Named-user mode needs the account database. With authentication off the owner stays off and a warning says how to use local-owner. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): keep the loopback owner address stable across restarts The single-node loopback transport took a new ephemeral port at each start. The owner stores its address in its identity file and refuses to start when the address differs, so every start after the first failed with "invalid addon owner configuration". Choose the port once, keep it in the add-on state root (loopback-port, mode 0600) and reuse it. If the port is taken at a later start, stop with an error that names the port and --addons-enabled=false, because another port would break the identity check. The keys and certificate stay in memory. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): let add-on pages keep the host modules they were written against A page written for the web UI imports React, the router, i18next and a few shared components. The add-on builder refuses a bundle that holds a host file, so moving such a page into an add-on meant rewriting it. Add a compat mode to the shared builder. It answers those imports from a directory of stand-ins that read the SDK, bundles the dynamic import of the page code into the one output file, and collects the style sheets the page imports. A relative import that is not in the add-on's own source resolves to the file at the same path under the compat directory. Widen the SDK for these pages, without a version change: more host components (spinner, modal, lightbox, model picker) and hooks (models, resources, voice profiles, operations, MCP client), a style on icons, the icon names the pages draw, sdk.navigation.openApp to open a page of the app (only addresses under /app, and the way back as the only state) and sdk.portal. The shell passes the query string to the page. The stand-ins cover the router (the tab in the path, the model in the query), the translator over the host locale files, the core API calls through sdk.core.fetch, and byte copies of the pure helpers, with tests that keep the copies equal to the originals. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): ship Studio and Talk as add-ons instead of built-in pages Studio and Talk were compiled into the web bundle and always listed in the Add-ons drawer. They are add-ons now, like Voice, Face, fine-tuning and quantization: a person installs and removes them, the drawer lists them only while they are installed, and they are packaged the same way. Move the pages, their components, hooks and helpers into addons/studio and addons/talk. Each has a manifest, a UI bundle, a release platform matrix and a catalog template, and is built by the compat mode of the shared builder. The core API stays where it is: the pages call it with the signed-in person's session. The history keeps its browser storage keys. The unit tests of the moved helpers move with them. In the core, drop the two pages, the Create group of the sidebar (Chat joins Home and Models in the old layout), the built-in catalog entries (the catalog is empty) and the talk locale namespace. The old addresses (/app/studio, /app/talk and the generator links) redirect to the add-on pages, with the model as ?model=. The Home /studio command is offered only while the Studio add-on is installed and opens it. The release workflow and its tests now list seven add-ons. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): run the Studio and Talk specs against the add-on bundles The pages are add-ons now, so a spec that draws one installs the real built bundle first: the apps list names it and its module and style sheet are served from the build output, the way the add-on owner serves them. studio-addon.js holds that helper, and the Studio and Talk fixtures call it. Update the specs for the new addresses (/app/addons/studio/studio, the model as ?model=), the session permission the add-on route needs, the legacy redirects, the Home /studio command that appears only while the add-on is installed, and the sidebar without a Create group. Add specs for the add-ons themselves: every tab draws with no script error, the shell titles the page once, the old addresses say so when the add-on is not installed, a button of the page reaches a core page with the way back, and the drawer lists both only after they are installed. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): centre the icon of a built-in tile in its icon box The drawer drew the icon of a built-in tile (Studio's palette was one) in a bare 40px box. Only the placeholder box was laid out as a centred grid with the 20px size, so the real icon sat in the top-left corner at the size of the page text. Give the built-in tile the same classes as the placeholder box, and check in a browser that the icon is centred. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): describe Studio and Talk as add-ons Replace the built-in list (now empty) with the Studio and Talk add-on sections, the new targets of the old links, the Home command, and the seven packaged add-ons. Describe the SDK additions the pages use: components, hooks, icons, openApp and portal. Say on the Studio page that it is installed from the drawer. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): click the shell title in the Studio slash-key spec The add-on shell titles the page, and the Studio overview no longer draws a second title, but the generator tabs keep theirs, so a bare h1 locator matched two elements. Click the first, which is the shell's. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): read the loopback port file through os.Root The persisted port file was read with a path built from the add-on root, which gosec reports as a file inclusion through a variable. Open it through os.Root so the name cannot leave the root, even by a symbolic link, and require a regular file with no group or other access. A looser file now fails closed instead of being trusted. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): expect Settings under the Operate hub, not the sidebar Settings moved out of the sidebar into the Operate hub. The navigation spec still expected a sidebar link. Assert that the sidebar has no Settings entry, and that the Operate hub opens Settings as a tab. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): drop Settings from the 404 places Settings is a tab of the Operate hub and not a destination of its own, so the 404 page does not list it. Assert that it is absent. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): drive the media history and TTS specs through the Studio add-on The image, video, sound and speech generators are pages of the Studio add-on now, and the old addresses redirect to it. Install the add-on with the Studio fixtures, open its pages and use its compose card, so the specs run against the page that exists. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): count the first-run steps, not every list item The recommended models are a list inside the second step, so counting all list items read 3 only while the models were still loading. Count the step elements, which does not depend on the load order. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): fail the node refresh by state, not by call count The spec served the list to the first request and failed the rest. The first request is not always the one from the page, so the page could get the failure first and show no list. Serve the list until the page shows it, then fail the refreshes. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): default to local-owner when auth is off A server without authentication has no accounts, so named-user left the add-on owner off. Resolve the unset mode in one place: local-owner when authentication is off, named-user when it is on. A flag or environment value still wins, and an unknown value is still refused. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): register an official add-on library by default Add gallery/addons/index.yaml, a YAML index of the four first-party add-ons (fine-tuning, quantization, voice, face) in the library format. The entries come from addonpkg catalog over locally built linux/amd64 packages, so the digests pin those images. On the first start, when the owner has no sources.json, register the raw GitHub URL of that index on master as the "official" library and write the file. The file is then the record: a library the admin removes stays removed, and an existing file is never changed. --addons-official-library=false skips the default. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): document the ownership default and the official library Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): list Studio and Talk in the first-party index Add the Studio and Talk entries to gallery/addons/index.yaml next to the four existing ones. They come from addonpkg catalog over locally built linux/amd64 packages, so the digests pin those images. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): expect Studio and Talk in the official index The official index now lists six add-ons. Update the test that decodes it so it accepts Studio and Talk. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): add a real icon to each first-party add-on Each icon is a 256 by 256 PNG: a rounded square in one colour of the UI kit palette with a white outline glyph from the kit sprite. Fine-tuning uses sliders, quantization layers, voice waveform, face user, studio palette, talk phone and tables database. They were rendered with headless Chrome from the sprite paths. Assisted-by: Claude Code:claude-sonnet-5-5 chrome Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): ship a committed icon.png instead of the generated square The shared UI builder and the Tables builder copy ui/icon.png from the add-on source tree to the package unchanged when the file exists. They write the generated placeholder only when it does not. Add a fixture with an icon and a test for both cases. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): list the tables add-on in the first-party index Add the Tables entry to gallery/addons/index.yaml next to the six existing ones. It comes from addonpkg catalog over a locally built linux/amd64 package, so the digest pins that image. The official index test now expects seven add-ons. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): make the Voice add-on one page with tabs The Voice add-on had four pages: recognition, library, create and speech, and the drawer listed two of them. It now has one page, "voice", and one drawer entry. Recognition, Library, Text to speech and From a recording are tabs of that page, drawn with the same tab strip the recognition page already used. The tab is the first path segment (voice/library), as in the Studio add-on, and the form that adds a voice is a view of the Library tab (voice/library/new). The old page names stay declared in the manifest. Each one redirects to its tab and keeps the query string, so saved links still open. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): send the old voice routes to the Voice tabs /app/voice, /app/voice-library and /app/voice-library/new now open the matching tab of the Voice add-on page while the add-on is installed and running, and the core pages otherwise. The moved-page helper compares the first segment of the target with the declared pages, so a target can name a tab. The docs and the navigation spec list the new addresses. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): cover the Voice tabs and the old page redirects The add-on unit tests render the tabs instead of the four pages. The Playwright spec checks the tab strip, the address of each tab, the missing Library tab for a member, and that the old page addresses of the add-on redirect to their tab with the query string kept. The migration table test follows the new targets. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(studio): keep the model menu inside a narrow screen The Home model picker sizes its menu from the viewport, and assumes the picker sits at the page gutter. In the Studio composer card the field is further in, so on a 390px screen the menu of the Sound expansion picker ended 7px past the right edge. Size the menu from the field it opens under instead. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): build the add-on bundles once for the whole run Each worker built the Studio and Talk bundles into the same dist directory, and the build empties that directory first. A worker that started, or restarted after a failure, removed the files another worker was serving, so a Studio spec saw "the style sheet of the add-on could not be loaded". Build once in a global setup before the workers start. A run without the setup still builds per worker. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): register the worker when the add-node spec says so The mocked roster added the worker on the third read of /api/nodes. The shell and the page both read it, and which one asks first depends on load, so the page could take the new worker as part of the roster it opened with and never show it as found. The worker now joins after the page shows it is listening. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): hold the gallery answer until the skeleton is seen The spec delayed the answer by a fixed 800 ms and then looked for the skeleton. The assertion polls at growing intervals, so on a slow machine it could step over that window. The answer now waits until the skeleton has been seen, and the spec then checks that the skeleton goes away. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): give the PDF tag the time the other PDF specs have pdf.js and its worker load on first use. The other specs in this file wait up to ten seconds for the result of that load. The first one waited five, and missed the tag on a busy machine. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): build the Face and Fine-tuning bundles in the global setup The navigation spec serves the built Fine-tuning bundle, but nothing built it, so two specs failed on a fresh checkout when the dist folder was missing. The spec also built the Face bundle in a hook that every worker runs, and a build rewrites the folder the other workers serve. The global setup now builds Studio, Talk, Face and Fine-tuning once, before the workers start. The spec builds the two bundles itself only when it runs without the global setup. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): fix the Home answers in the stale pin spec The spec asserts that the page shows no alert. The Home page raises its own alert when the model list fails or a staging operation has failed, and both answers come from the shared test server. A slow reply or a failed install left there counted as an alert about the pin. The spec now fixes both answers, so the alert count tells only about the pin. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): read the server address in the add-node spec from the page The spec expected the register address to carry port 8089, so it failed on any other port. It now expects the origin the page was opened from. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): give the navigation routes spec its own catalog No page is built in today, so the catalog is empty and the three specs that expect the studio and talk entries failed on the list and on the pin seed. They passed only while Studio and Talk were built in. The spec now sets a catalog of two ungated entries for each example and restores the old one afterwards. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(cli): report a failed write of the MLX hostfile The tunnel callback ignored the error of os.WriteFile and set the hostfile variable even when the file was not written. It now logs the error and leaves the variable alone. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): discard the close error of the held listener errcheck flags the deferred Close of the listener that keeps the port busy. The test has no use for that error, so the defer drops it explicitly. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the add-on tool tests to Ginkgo The fine-tuning, quantization and add-on host library tests used t.Errorf, t.Fatal and t.Run, which the forbidigo rule rejects. They now use Describe, It and Gomega, with a suite bootstrap in each package. The cases and the assertions are the same. The tables in the host client and the tool schema tests keep one entry per case. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the host operation tests to Ginkgo The hostops tests used the testing package calls that forbidigo rejects. They now run as Ginkgo specs under one suite bootstrap. Every check and every per-operation case is kept. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the pin store tests to Ginkgo The pin service, file store, sweep and shared store contract tests used the testing package calls that forbidigo rejects. The store contract is now a function that registers its specs for each store, so the file store, the memory-only store and the database store run the same cases. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the remaining add-on tests to Ginkgo The built-in catalog, tool arguments, loopback carrier, host operation contract, packaging, application, config, CLI and endpoint tests used the testing package calls that forbidigo rejects. They now run as Ginkgo specs. The endpoint tests also drop the unchecked Close errors that errcheck reports when the auth tag is on. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): wait for the targeted Reinstall button before the install click The "generic deletion blocks targeted install at confirmation" spec dispatches a popstate to set the target node, then clicks Reinstall with a script click. The click can run before the targeted page has rendered or while the button is still disabled, so no install request is sent and the poll on the install count times out. It failed 1 run in 10 with two workers and on every retry in CI. The spec now waits for the button to be enabled first. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(addons): refresh the voice package digest in the first-party index The Voice add-on became one page with tabs and got a new icon, so the digest in gallery/addons/index.yaml no longer matched the build. Render the entry again with addonpkg catalog over a locally built linux/amd64 package; the main page is now voice. The other entries are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): wait for the target node's Reinstall button after the popstate The "generic deletion blocks targeted install at confirmation" spec dispatches a popstate to set the target node, then waits for a button named Reinstall and clicks it. Until React renders the targeted page, the page still draws the generic Reinstall button. It is enabled and has the same name, so the wait passed on it, the click started a global install, and no node install request was sent. The poll on the install count timed out with the confirm dialog still open. With the CPU throttled to 8x, the spec failed on 20 of 20 runs, and the button it clicked had the title "Reinstall". The spec now waits for the button titled "Reinstall on <node>". With the throttle it passes 30 of 30 runs. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): give the Studio hand-off spec a budget for the cold bundle load The "a result made through a hand-off records which result it came from" spec opens the Studio add-on straight from an address, twice. Each is a cold load: the page shows "Loading" until the add-on module has run, and the next assertion had the default five seconds. The load takes about two seconds on an idle run and seven to ten under CPU load, so the spec failed on a loaded machine with the hand-off note not yet drawn. With the CPU throttled to 6x and eight busy loops running, it failed on 21 of 30 runs. The first assertion after each load now waits up to 20 seconds, and the spec then passed 30 of 30 under the same load. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): carry the identity page styles in the Face add-on The Face page is drawn with the shared identity views, but its style sheet held only the clip and error rules. The page, card, model list, known-faces table, enrol sheet and distance scale classes had no rules. They looked right only when the host had already loaded the style sheet of its own Faces page, which a visit through the add-on drawer does not do. Opened that way, the cards had no gap, the model list showed bullets, and the install buttons ran into the names. The Face sheet now holds the same identity rules as the core sheet, so the page looks the same whichever way it is opened. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): scope the narrow-screen identity rules of the Voice add-on The rules for narrow screens (single-column pairs, smaller tool padding, compact rank rows) were copied into the Voice style sheet without their media query, so they applied at every width. They now sit in a max-width 720px query, and the reduced-motion rule in its own query, as in the core sheet. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): drop the repeated title from the Voice and Face pages The add-on shell already titles the page, so the recognition pages showed the name twice: "Voice" above "Voices", "Face" above "Faces". Keep the supporting line and the model picker, as the Studio page does, and remove the unused title strings. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): replace the Pinned sidebar section with an Apps entry The sidebar no longer lists pinned add-ons. It shows one Apps entry that opens the add-ons drawer, with a Show all link below it that opens the same drawer. Pins are used from the drawer, where a pinned tile keeps its pin and move buttons. The server pins API is unchanged. Remove the PinnedNav component and its locale title, rename the items.addons label to Apps, and update the e2e specs and the add-ons documentation to match. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): assert the single shell title on the Voice and Face pages The add-on shell now shows the one title, "Voice" or "Face", and the page no longer repeats it. The tests still looked for the h1 "Voices" and "Faces". Assert that the add-on page has exactly one h1 and that it is the shell title. The core pages keep their own "Voices" and "Faces" titles. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): drop the repeated title from the Voice Library and Text to speech pages The add-on shell already titles the page "Voice", so these two tabs showed a second heading below it ("Speech voices", "Speak text"). Keep the supporting line, as the Studio and recognition pages do, and remove the unused title strings. The e2e spec now checks that each tab has exactly one h1, the shell title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): list pinned apps under Apps in the sidebar A pinned app now shows as an indented child of the Apps entry, with its icon and title, in pin order. Show all stays the last child. The pins come from GET /api/addons/pins through the navigation context, and the apps from the add-on registry, which the drawer feeds on every list read, so the links follow the drawer. A pin without an installed app is not drawn, nothing extra shows when no app is pinned, and the children are hidden when the sidebar is collapsed to the rail. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): say that pinned apps are listed under Apps in the sidebar The sidebar lists the pinned apps between Apps and Show all, in pin order. The pins section and the navigation section said that pins do not appear in the sidebar. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): remove the Show all link under Apps in the sidebar The link repeated the Apps entry, which opens the same drawer, and looked redundant when no app was pinned. Apps is now the only add-on entry; pinned apps still list under it in pin order. The link, its locale key and the style comment are gone, and the e2e spec and the add-ons docs now say that Apps is the only entry. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show a first-run card on Apps when nothing is installed An administrator who opens Apps with no add-on installed now sees a short card with three steps: choose a library, browse and install, start the add-on. Its button opens the Browse tab. The card is hidden once an add-on is installed and for anyone who cannot install. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): hide the pinned apps in the icon rail When the window is narrower than 1024 px the sidebar shrinks to the icon rail by CSS alone, without the collapsed class. The pinned app links were still drawn there, with the indent meant for the full sidebar, so their icons sat to the right of the rail and were cut off. Hide the pinned app links in both rail states. The Apps entry keeps its centred icon. The e2e spec checks that the rail shows no pinned app and that every icon is centred at 1280 px collapsed and at 1000 px. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): stop the backend redirect test racing the installed list The test reads the redirected URL right after the path changes. The Backends page then loads the installed list and drops a selected backend that is not installed, so the backend parameter was sometimes already gone when the test read it. The mock server has no llama-cpp, and about 15 of 40 repeated runs failed on the backend assertion. The redirect itself was correct, including show_all. Serve the selected backend in the installed list, so the page keeps the selection and the assertions stay as strict as before. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): accept a same-host Referer or Origin for human confirmation Browsers send no Sec-Fetch-* headers over plain HTTP on an address that is not localhost, so creating a resource from a LAN address failed with "Creating a resource requires your own browser session". When Sec-Fetch-Site is present it must still be same-origin with an empty destination, as before. When it is absent, Origin or, failing that, Referer must name the request host and port. A destination or mode that is present must still be the expected one. This matches the rule in BrowserProtection. The credential, token cookie, API key and tool-context checks are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): keep hub entries off the Apps route in the sidebar The Build hub still owned /app/addons through its legacy Add-ons tab, so the sidebar lit both Apps and Build on the Apps page. A tab whose gate is closed now owns no route when the sidebar asks, so only the Apps entry is active there. The hubs keep their own routes, and the legacy navigation still gives /app/addons to the Build hub. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): label the pin control on app tiles and in the detail view The pin was a small icon-only button and hard to find. Each app tile now shows a button reading Pin, or Pinned with a filled pin, and the detail view has the same control in its header. Both toggle the same stored pin. The Apps grid lists pinned apps first, in pin order, then the others. The sidebar list and the server pins API are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): check the Voice From a recording tab has one title The Library and Text to speech tabs already assert that the add-on shell title is the only h1. Do the same for the From a recording tab, so a heading added there later fails the navigation spec. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): stop the add-on shell drawing a title over add-on pages The app decides what its page shows. The shell drew the add-on name as a header above every add-on page, so an add-on could not choose its own title. The shell now draws no header on a live, loading or frozen add-on page; each add-on titles its own pages. The unavailable and missing-page states keep the name, because no add-on page is drawn there. The runtime and shell specs now check that the shell adds no page title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): let each add-on draw its own page title The shell no longer titles add-on pages, so each add-on draws one h1 per page. Voice titles its tabs "Voices" (Recognition and From a recording), "Speech voices" (Library) and "Speak text" (Text to speech); "Add a speech voice" already had its own. Face draws "Faces", Studio "Studio" and Talk "Talk". Tables promotes the database list and database name from h2 to h1. Fine-tuning and Quantization already drew a title. The add-on unit tests and the navigation and Studio specs now assert exactly one h1 per page, with the page's own title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): add a kind to library entries and read unknown kinds tolerantly A library entry now has an optional kind: addon (the default when it is absent), backend or model. Every index written before kinds existed stays valid and its entries are add-ons. A reader keeps the entries of the kinds it knows and skips the others without reading their fields, so a newer kind never makes an older host refuse the whole index. The refresh result and the library state report how many entries were skipped and of which kinds, and the refresh logs a warning. A kind that is not a plain kind name still makes the index malformed. Search lists add-ons unless the caller asks for a kind, and Resolve returns add-ons only, so an add-on install cannot be pointed at a backend image. A library policy may name the signer with an anchored identityRegex instead of an exact identity, so builds from master and from tags both verify. An add-on install refuses a policy that carries one. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backends): install a backend from a library entry A library can now list backends. An entry of kind backend names the image of the backend gallery by digest, and POST /backends/library/install installs it on the local node. The gallery stays the source of truth and its install path is the one used: the library resolves the entry, takes the signature policy from the library, pins the image by digest and calls the same install as the gallery. Installing a name that came from a gallery, another library or by hand is refused and changes nothing. The signature policy of a backend library names the signer with an anchored identityRegex, so images signed by the merge workflow on master and on tags both verify. A pattern that is not anchored is refused. An install is bounded by a size limit per kind: add-ons keep 512 MiB and a backend image defaults to 8 GiB, set with --addons-backend-max-bytes. The limit reads the manifest after the signature check and before any layer is fetched. The metadata of an installed backend records the library it came from. scripts/backends/cmd/backendcatalog renders entries from backend/index.yaml and pins them against the registry. The pilot is the CPU build of local-store, in gallery/backends/index.yaml. The library is registered on the first start only with --addons-backend-library, which is off by default. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show the kind of library entries and install backends from them The Browse tab lists the entries of every kind this host reads and marks each one as an add-on or a backend. A backend installs through the backend route and not through the add-on lifecycle: the confirmation says that the backend gallery is not changed and that a backend runs native code, and a refusal of the host, such as a name installed from another origin, is shown with its own reason. The details give the kind, the alias and the signer pattern. The library editor can name the signer with a pattern. Without this a save would drop the pattern of a backend library. The Libraries tab reports the entries a refresh skipped because their kind is not read. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): describe backends in the library Document the entry kinds, how a reader skips a kind it does not know, the backend entry format, the signer identity pattern, the install route, the size limit of each kind, the pilot library and the command that renders it. The backend page links to it. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(mcp): install a backend from a library entry The backend library route installs a backend from an add-on library entry, and the admin REST surface and the MCP catalog must stay in step. Add the install_library_backend tool with an in-process and an HTTP client, the route map entry, the safety-rule listing and a skill recipe. Listing library entries stays on the add-on routes, which the assistant does not expose, so no list tool is added. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): carry the add-on library keys in every locale The add-on library and backend library strings existed only in English, so the other locales had no value for them. Add the missing keys of the addons namespace to de, es, id, it, ko, pt-BR and zh-CN. The values are the English text until a translation is contributed. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(mcp): install a backend from a library entry" This reverts commit7493cff727. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "docs(addons): describe backends in the library" This reverts commit57816ed7e6. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(ui): show the kind of library entries and install backends from them" This reverts commit49b2b151b7. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(backends): install a backend from a library entry" This reverts commit1d8d3e3421. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(addons): add a kind to library entries and read unknown kinds tolerantly" This reverts commit484813f856. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): drop the backend library keys from the locales Revert the part of39f77ae29that added strings for backend libraries: the kind labels, the backend install texts, the signer identity pattern and the skipped entries notice. The add-on library keys stay. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add an add-ons control to the chat header Add-on tools were on or off only through the /addons command, with a toast and no state a person could see. The chat header now has an Add-ons button in the style of the MCP dropdown. It lists each running add-on that ships tools with a switch, and a badge shows how many are on. The chat keeps the ids that are on (addonToolIds) next to the older boolean, and both are saved with the chat. A chat that has only the boolean means every add-on. The request sends addon_tools alone when every add-on is on, and adds addon_tool_ids when only some are. /addons turns all on or all off and uses the same state. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): let a chat request name the add-ons whose tools it uses addon_tools=true still offers the tools of every add-on the caller may use. A new addon_tool_ids metadata key, a comma-separated list of add-on ids, narrows that to some add-ons. The list counts only with the opt-in, and a list that names nothing offers no tool rather than all of them. The executor drops the tools of add-ons that are not listed and refuses a call to one as an unknown tool, so a model cannot reach a tool the person turned off. Both keys are consumed so they do not reach the backend. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): cover the per-add-on tool list and the chat header control Go tests cover the list form (the list filters the tools offered and the calls allowed) and the unchanged boolean form. Unit tests cover the chat add-on rules. Playwright specs cover the header control: listing only add-ons that ship tools, toggling each one, the count, /addons in step with the switches, and the request metadata. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): check each add-on page has one title when not enabled The shell names an add-on only where the add-on page cannot mount (not enabled, failed, no UI, bundle not loaded, no such page). Once the page mounts it draws its own title, including when it says it is not enabled on this server, and the shell draws none. The shell code already works this way; this change states the rule beside the unavailable state and pins it in the specs. The shell spec now expects exactly one title in the unavailable states. The navigation spec checks the Face page without a face model (the add-on title only) and the Face page of a stopped add-on (the shell title only). Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): upgrade add-ons on their own when the setting is on A new auto_upgrade_addons setting (flag LOCALAI_AUTO_UPGRADE_ADDONS, default off) lets the owner upgrade an installed add-on when its library lists a newer version. It follows the backend setting: a startup flag, a runtime setting, and the same schedule as the backend update check (a first pass shortly after start, then every six hours). Each pass fetches the library of every installed add-on, and for each available update builds the same pinned upgrade command an administrator sends and runs it through the same lifecycle path. The channel rules, the signer identity, the digest checks, the SDK compatibility check and the backup before the upgrade all apply unchanged. The only difference is the gate: the owner acts on its own setting, so no administrator is asked. A refused or failed upgrade changes nothing and is logged; a candidate that failed is not retried until the library lists another digest. With the setting off, nothing runs and upgrades stay manual. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add the Upgrade add-ons automatically toggle to Settings The toggle sits beside Upgrade backends automatically and is off by default. It saves the auto_upgrade_addons runtime setting. The English locale gets the label key, and the Settings test fixture lists the new field. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): cover the automatic upgrade setting and document it The setting defaults to off in the config, the runtime settings and the flag, and is read back from the saved settings file. With it on, a real owner upgrades an installed add-on through the manager; with it off, nothing changes, and a change of the setting at run time takes effect on the next pass. The manager specs show the upgrade goes through the same pinned path as the manual one, and that a corrupted package, a failed signature policy, an incompatible release, a missing update checker and a caller who is not an administrator each leave the installed version in place. The new specs are added to the lifecycle case inventory. The add-ons page describes the setting, its schedule and its default. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(addons): name the library entry identifier id The library entry of an add-on used addonId while the shared envelope and the model entries use id. Release now has the field ID, written as id, and every writer emits it: the index encoder, the package workflow templates, the catalog generator and gallery/addons/index.yaml. The digests in the index did not change. DecodeIndex still reads addonId as a legacy alias for one release, so an existing library and a library served by an older tool keep working. An entry that holds both names is refused. The Apps UI reads id and falls back to addonId. Assisted-by: Claude Code:claude-sonnet-5-5 go node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): use the id field in the auto-upgrade test release The auto-upgrade test arrived after the identifier rename and still named the release field AddonID, so the application tests did not build. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): annotate the auto-upgrade actor label for gosec The actor label looks like a credential to gosec G101. It is a fixed name for upgrades the owner starts itself. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(schema): persist the per-chat add-on tool fields The chat header persists addonTools and addonToolIds, but Conversation had no fields for them, so the server dropped them on save. Add both and update the generated swagger files by hand because swag is not installed here. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): count the add-ons auto-upgrade setting in the settings groups The setting sits in the Backends and galleries group, once. The page now lists 54 settings plus the three branding images, so the spec expects 57. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): pause the fake clock ahead of now in the clear-all spec The fake clock runs from install, so pausing at the current time failed with 'Cannot fast-forward to the past' on a slow runner. The save itself is synchronous and was not lost. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): hold the Studio capabilities reply longer in the skeleton spec 800 ms was shorter than a slow runner needs to render the page, so the skeleton was gone before the check. The product shows it. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(i18n): add the French add-ons locale Master added the fr locale after this branch forked. The add-ons unit test requires every locale to carry the add-ons namespace, so fr needs its own addons.json. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
mudler-agentandEttore Di Giacinto authored and GitHub committed
1 parent
099d6c3dd0
commit
e45dc5e9f7
964 files changed
+159560
-515
No files matched your search
@@ -0,0 +1,10 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package contract
|
||||
|
||||
import (
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestContract(t *testing.T) { RegisterFailHandler(Fail); RunSpecs(t, "Addon contracts") }
|
||||
@@ -0,0 +1,214 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package contract
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"slices"
|
||||
)
|
||||
|
||||
// Host slot identifiers. The list is closed: adding a slot is a host change
|
||||
// with its own review, never an addon declaration.
|
||||
const (
|
||||
SlotChatComposerActions = "chat.composer.actions"
|
||||
SlotChatMessageActions = "chat.message.actions"
|
||||
SlotAppNav = "app.nav"
|
||||
SlotSettingsSections = "settings.sections"
|
||||
)
|
||||
|
||||
// HostSlots are the slot identifiers a manifest may contribute to.
|
||||
var HostSlots = []string{SlotChatComposerActions, SlotChatMessageActions, SlotAppNav, SlotSettingsSections}
|
||||
|
||||
// Chat action placements. Each one is a host slot of its own.
|
||||
const (
|
||||
PlacementComposer = "composer"
|
||||
PlacementMessage = "message"
|
||||
)
|
||||
|
||||
// Renderer kinds an addon may declare for one of its own tools.
|
||||
const (
|
||||
RendererApprovalCard = "approval-card"
|
||||
RendererToolResult = "tool-result"
|
||||
)
|
||||
|
||||
// Bounds of the declarations of one addon.
|
||||
const (
|
||||
MaxChatActions = 16
|
||||
MaxSlots = 16
|
||||
MaxRenderers = 16
|
||||
MaxLabelChars = 60
|
||||
)
|
||||
|
||||
// IconNames are the semantic icon names of the host Icon allowlist. The host
|
||||
// SDK (core/http/react-ui/src/addons/sdk.js) carries the same list; a test
|
||||
// keeps both equal.
|
||||
var IconNames = []string{
|
||||
"puzzle", "file", "file-text", "folder", "table", "database", "search", "plus", "minus", "close", "check",
|
||||
"check-circle", "alert-circle", "warning", "info", "edit", "trash", "download", "upload", "refresh",
|
||||
"settings", "list", "grid", "chart-line", "clock", "calendar", "tag", "star", "link", "external-link",
|
||||
"copy", "eye", "lock", "user", "users", "message", "sparkles", "chevron-down", "chevron-up", "chevron-right",
|
||||
}
|
||||
|
||||
// componentName is the export name of a component of the addon module: a
|
||||
// PascalCase identifier, so it can never be a prototype key or a host name.
|
||||
var componentName = regexp.MustCompile(`^[A-Z][A-Za-z0-9]{0,63}$`)
|
||||
|
||||
// ValidComponentName reports whether s may name an exported component.
|
||||
func ValidComponentName(s string) bool { return componentName.MatchString(s) }
|
||||
|
||||
// ChatTarget is what a chat action triggers: exactly one of Tool (a tool the
|
||||
// addon declares) and Component (a component the addon module exports).
|
||||
type ChatTarget struct {
|
||||
Tool string `json:"tool,omitempty"`
|
||||
Component string `json:"component,omitempty"`
|
||||
}
|
||||
|
||||
// ChatAction is one chat action. A bare identifier (the form of the first
|
||||
// manifest version) declares only the ID and is never drawn: it has no
|
||||
// placement. A full declaration has a label, an icon, a placement and a target.
|
||||
type ChatAction struct {
|
||||
ID string
|
||||
Label *Localized
|
||||
Icon string
|
||||
Placement string
|
||||
Target *ChatTarget
|
||||
}
|
||||
|
||||
type chatActionWire struct {
|
||||
ID string `json:"id"`
|
||||
Label *Localized `json:"label,omitempty"`
|
||||
Icon string `json:"icon,omitempty"`
|
||||
Placement string `json:"placement,omitempty"`
|
||||
Target *ChatTarget `json:"target,omitempty"`
|
||||
}
|
||||
|
||||
// Declared reports whether the action is a full declaration.
|
||||
func (a ChatAction) Declared() bool {
|
||||
return a.Label != nil || a.Icon != "" || a.Placement != "" || a.Target != nil
|
||||
}
|
||||
|
||||
func (a ChatAction) MarshalJSON() ([]byte, error) {
|
||||
if !a.Declared() {
|
||||
return json.Marshal(a.ID)
|
||||
}
|
||||
return json.Marshal(chatActionWire{ID: a.ID, Label: a.Label, Icon: a.Icon, Placement: a.Placement, Target: a.Target})
|
||||
}
|
||||
|
||||
func (a *ChatAction) UnmarshalJSON(b []byte) error {
|
||||
b = bytes.TrimSpace(b)
|
||||
if len(b) > 0 && b[0] == '"' {
|
||||
var id string
|
||||
if e := json.Unmarshal(b, &id); e != nil {
|
||||
return e
|
||||
}
|
||||
*a = ChatAction{ID: id}
|
||||
return nil
|
||||
}
|
||||
var w chatActionWire
|
||||
d := json.NewDecoder(bytes.NewReader(b))
|
||||
d.DisallowUnknownFields()
|
||||
if e := d.Decode(&w); e != nil {
|
||||
return e
|
||||
}
|
||||
*a = ChatAction{ID: w.ID, Label: w.Label, Icon: w.Icon, Placement: w.Placement, Target: w.Target}
|
||||
return nil
|
||||
}
|
||||
|
||||
// SlotContribution places one exported component in a host slot.
|
||||
type SlotContribution struct {
|
||||
ID string `json:"id"`
|
||||
Slot string `json:"slot"`
|
||||
Component string `json:"component"`
|
||||
}
|
||||
|
||||
// Renderer draws the approval card or the result of one of the addon's own
|
||||
// tools. The host owns the frame around it.
|
||||
type Renderer struct {
|
||||
Kind string `json:"kind"`
|
||||
Tool string `json:"tool"`
|
||||
Component string `json:"component"`
|
||||
}
|
||||
|
||||
// ChatActionIDs returns the identifiers of every declared chat action.
|
||||
func (u UI) ChatActionIDs() []string {
|
||||
out := make([]string, 0, len(u.ChatActions))
|
||||
for _, a := range u.ChatActions {
|
||||
out = append(out, a.ID)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func validIconName(name string) bool { return slices.Contains(IconNames, name) }
|
||||
|
||||
// validateContributions checks the chat actions, slots and renderers of one
|
||||
// UI declaration against the addon's own tools.
|
||||
func validateContributions(u *UI, tools []string) error {
|
||||
if len(u.ChatActions) > MaxChatActions || len(u.Slots) > MaxSlots || len(u.Renderers) > MaxRenderers {
|
||||
return fmt.Errorf("too many UI contributions")
|
||||
}
|
||||
own := map[string]bool{}
|
||||
for _, t := range tools {
|
||||
own[t] = true
|
||||
}
|
||||
ids := map[string]bool{}
|
||||
for _, a := range u.ChatActions {
|
||||
if !namespace.MatchString(a.ID) || ids[a.ID] {
|
||||
return fmt.Errorf("invalid or duplicate chat action")
|
||||
}
|
||||
ids[a.ID] = true
|
||||
if !a.Declared() {
|
||||
continue
|
||||
}
|
||||
if a.Label == nil || !a.Label.valid(MaxLabelChars) {
|
||||
return fmt.Errorf("invalid chat action label")
|
||||
}
|
||||
if a.Placement != PlacementComposer && a.Placement != PlacementMessage {
|
||||
return fmt.Errorf("unknown chat action placement")
|
||||
}
|
||||
if !validIconName(a.Icon) && (a.Icon == "" || a.Icon != u.Icon) {
|
||||
return fmt.Errorf("chat action icon must be a host icon name or the packaged icon")
|
||||
}
|
||||
t := a.Target
|
||||
if t == nil || (t.Tool == "") == (t.Component == "") {
|
||||
return fmt.Errorf("chat action needs exactly one target")
|
||||
}
|
||||
if t.Tool != "" && !own[t.Tool] {
|
||||
return fmt.Errorf("chat action targets a tool the addon does not declare")
|
||||
}
|
||||
if t.Component != "" && !componentName.MatchString(t.Component) {
|
||||
return fmt.Errorf("invalid chat action component name")
|
||||
}
|
||||
}
|
||||
for _, s := range u.Slots {
|
||||
if !namespace.MatchString(s.ID) || ids[s.ID] {
|
||||
return fmt.Errorf("invalid or duplicate slot contribution")
|
||||
}
|
||||
ids[s.ID] = true
|
||||
if !slices.Contains(HostSlots, s.Slot) {
|
||||
return fmt.Errorf("unknown host slot")
|
||||
}
|
||||
if !componentName.MatchString(s.Component) {
|
||||
return fmt.Errorf("invalid slot component name")
|
||||
}
|
||||
}
|
||||
renderers := map[string]bool{}
|
||||
for _, r := range u.Renderers {
|
||||
if r.Kind != RendererApprovalCard && r.Kind != RendererToolResult {
|
||||
return fmt.Errorf("unknown renderer kind")
|
||||
}
|
||||
if !own[r.Tool] {
|
||||
return fmt.Errorf("renderer for a tool the addon does not declare")
|
||||
}
|
||||
if !componentName.MatchString(r.Component) {
|
||||
return fmt.Errorf("invalid renderer component name")
|
||||
}
|
||||
key := r.Kind + "\x00" + r.Tool
|
||||
if renderers[key] {
|
||||
return fmt.Errorf("duplicate renderer")
|
||||
}
|
||||
renderers[key] = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package contract
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"regexp"
|
||||
)
|
||||
|
||||
// The host operation channel is how an add-on process asks the host to do
|
||||
// something it cannot do itself. These are the wire types of that channel.
|
||||
// Every operation is registered by the host, allowed per add-on by a host
|
||||
// setting, and checked against the authority of a request the host is
|
||||
// serving for that add-on. The default is refuse.
|
||||
|
||||
// HostRequest names one host operation. OperationID is the ID of the request
|
||||
// the host is serving for the add-on; it never names a user.
|
||||
type HostRequest struct {
|
||||
OperationID string `json:"operationId"`
|
||||
Op string `json:"op"`
|
||||
Arguments json.RawMessage `json:"arguments,omitempty" swaggertype:"object"`
|
||||
}
|
||||
|
||||
// HostResponse carries a result or an error, never both.
|
||||
type HostResponse struct {
|
||||
Result json.RawMessage `json:"result,omitempty" swaggertype:"object"`
|
||||
Error *HostError `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// HostError is a refusal or a failure of a host operation. Code is one of the
|
||||
// Host* constants below.
|
||||
type HostError struct {
|
||||
Code string `json:"code"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
func (e *HostError) Error() string { return "host operation refused (" + e.Code + "): " + e.Message }
|
||||
|
||||
const (
|
||||
HostNoLiveOperation = "no_live_operation"
|
||||
HostScope = "scope"
|
||||
HostFeature = "feature"
|
||||
HostUnknownOp = "unknown_op"
|
||||
HostBadRequest = "bad_request"
|
||||
HostTooLarge = "too_large"
|
||||
HostNotFound = "not_found"
|
||||
HostConflict = "conflict"
|
||||
HostUnavailable = "unavailable"
|
||||
)
|
||||
|
||||
// HostCredentialHeader carries the host credential of the incarnation.
|
||||
// #nosec G101 -- this is a header name, not a credential value.
|
||||
const HostCredentialHeader = "X-Addon-Host-Credential"
|
||||
|
||||
// hostOpName is the brief's shape ^[a-z]+\.[a-z.]+$ without an empty word:
|
||||
// no trailing dot and no double dot.
|
||||
var hostOpName = regexp.MustCompile(`^[a-z]+(\.[a-z]+)+$`)
|
||||
|
||||
// ValidHostOp reports whether op has the shape of a host operation name:
|
||||
// lower-case words joined by dots, such as finetune.get.
|
||||
func ValidHostOp(op string) bool { return len(op) <= 64 && hostOpName.MatchString(op) }
|
||||
|
||||
// DecodeHostRequest decodes one request with the strict decoder: no unknown
|
||||
// field, no duplicate key, no trailing data, bounded size.
|
||||
func DecodeHostRequest(b []byte, max int64) (HostRequest, error) {
|
||||
var r HostRequest
|
||||
if err := strictJSON(b, &r, max); err != nil {
|
||||
return HostRequest{}, err
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package contract
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("host operation wire types", func() {
|
||||
It("round-trip", func() {
|
||||
req := HostRequest{OperationID: "op1", Op: "finetune.get", Arguments: json.RawMessage(`{"jobId":"j1"}`)}
|
||||
b, err := json.Marshal(req)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
got, err := DecodeHostRequest(b, 1<<20)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(got.OperationID).To(Equal("op1"))
|
||||
Expect(got.Op).To(Equal("finetune.get"))
|
||||
Expect(string(got.Arguments)).To(Equal(`{"jobId":"j1"}`))
|
||||
|
||||
resp := HostResponse{Error: &HostError{Code: "scope", Message: "no"}}
|
||||
b, _ = json.Marshal(resp)
|
||||
var back HostResponse
|
||||
Expect(json.Unmarshal(b, &back)).To(Succeed())
|
||||
Expect(back.Error).ToNot(BeNil())
|
||||
Expect(back.Error.Code).To(Equal("scope"))
|
||||
Expect(back.Error.Error()).ToNot(BeEmpty())
|
||||
|
||||
b, _ = json.Marshal(HostResponse{Result: json.RawMessage(`{"ok":true}`)})
|
||||
Expect(json.Unmarshal(b, &back)).To(Succeed())
|
||||
Expect(string(back.Result)).To(Equal(`{"ok":true}`))
|
||||
})
|
||||
|
||||
DescribeTable("the strict decoder refuses what it does not know",
|
||||
func(body string) {
|
||||
_, err := DecodeHostRequest([]byte(body), 1<<20)
|
||||
Expect(err).To(HaveOccurred(), "accepted")
|
||||
},
|
||||
Entry("unknown field", `{"operationId":"a","op":"finetune.get","arguments":{},"userId":"x"}`),
|
||||
Entry("duplicate key", `{"operationId":"a","operationId":"b","op":"finetune.get"}`),
|
||||
Entry("trailing", `{"operationId":"a","op":"finetune.get"} {}`),
|
||||
Entry("not an object", `[]`),
|
||||
Entry("empty", ``),
|
||||
)
|
||||
|
||||
It("refuses an oversized body", func() {
|
||||
_, err := DecodeHostRequest([]byte(`{"operationId":"a","op":"finetune.get"}`), 10)
|
||||
Expect(err).To(HaveOccurred(), "an oversized body was accepted")
|
||||
})
|
||||
|
||||
DescribeTable("an operation name has the dotted shape",
|
||||
func(op string, want bool) {
|
||||
Expect(ValidHostOp(op)).To(Equal(want), "%q", op)
|
||||
},
|
||||
Entry("finetune.get", "finetune.get", true),
|
||||
Entry("quantize.start", "quantize.start", true),
|
||||
Entry("finetune.a.b", "finetune.a.b", true),
|
||||
Entry("empty", "", false),
|
||||
Entry("no dot", "finetune", false),
|
||||
Entry("leading dot", ".get", false),
|
||||
Entry("uppercase", "Finetune.get", false),
|
||||
Entry("trailing space", "finetune.get ", false),
|
||||
Entry("slash", "finetune/get", false),
|
||||
Entry("trailing newline", "finetune.get\n", false),
|
||||
Entry("hyphen", "fine-tune.get", false),
|
||||
Entry("empty verb", "finetune.", false),
|
||||
Entry("too long", strings.Repeat("a", 70)+".get", false),
|
||||
)
|
||||
})
|
||||
@@ -0,0 +1,49 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package contract
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"reflect"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Limits are host configuration, not addon-controlled manifest claims. Durations
|
||||
// use Go time.Duration (nanoseconds in JSON). Zero never means unlimited.
|
||||
type Limits struct {
|
||||
ManifestBytes int64 `json:"manifestBytes"`
|
||||
EnvelopeBytes int64 `json:"envelopeBytes"`
|
||||
PayloadBytes int64 `json:"payloadBytes"`
|
||||
ReadTimeout time.Duration `json:"readTimeout"`
|
||||
PreviewTimeout time.Duration `json:"previewTimeout"`
|
||||
ApplyTimeout time.Duration `json:"applyTimeout"`
|
||||
ShutdownTimeout time.Duration `json:"shutdownTimeout"`
|
||||
BackoffInitial time.Duration `json:"backoffInitial"`
|
||||
BackoffMax time.Duration `json:"backoffMax"`
|
||||
BackoffAttempts int64 `json:"backoffAttempts"`
|
||||
BatchSize int64 `json:"batchSize"`
|
||||
QueryDepth int64 `json:"queryDepth"`
|
||||
QueryRows int64 `json:"queryRows"`
|
||||
QueryBytes int64 `json:"queryBytes"`
|
||||
ProposalExpiry time.Duration `json:"proposalExpiry"`
|
||||
ArtifactExpiry time.Duration `json:"artifactExpiry"`
|
||||
ArtifactBytes int64 `json:"artifactBytes"`
|
||||
ArtifactQuotaBytes int64 `json:"artifactQuotaBytes"`
|
||||
ArtifactCount int64 `json:"artifactCount"`
|
||||
Concurrency int64 `json:"concurrency"`
|
||||
}
|
||||
|
||||
func DefaultLimits() Limits {
|
||||
return Limits{ManifestBytes: 64 << 10, EnvelopeBytes: 2 << 20, PayloadBytes: 1 << 20, ReadTimeout: 30 * time.Second, PreviewTimeout: 30 * time.Second, ApplyTimeout: 60 * time.Second, ShutdownTimeout: 10 * time.Second, BackoffInitial: 250 * time.Millisecond, BackoffMax: 30 * time.Second, BackoffAttempts: 5, BatchSize: 100, QueryDepth: 8, QueryRows: 1000, QueryBytes: 1 << 20, ProposalExpiry: 5 * time.Minute, ArtifactExpiry: time.Hour, ArtifactBytes: 100 << 20, ArtifactQuotaBytes: 1 << 30, ArtifactCount: 100, Concurrency: 4}
|
||||
}
|
||||
func (l Limits) Validate() error {
|
||||
v := reflect.ValueOf(l)
|
||||
for i := 0; i < v.NumField(); i++ {
|
||||
if v.Field(i).Int() <= 0 {
|
||||
return fmt.Errorf("%s must be positive and finite", v.Type().Field(i).Name)
|
||||
}
|
||||
}
|
||||
if l.BackoffInitial > l.BackoffMax || l.ArtifactBytes > l.ArtifactQuotaBytes || l.PayloadBytes > l.EnvelopeBytes || l.QueryBytes > l.PayloadBytes || l.QueryDepth > 64 {
|
||||
return fmt.Errorf("inconsistent limits")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,506 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
// Package contract defines version-one addon manifests and runtime wire DTOs.
|
||||
package contract
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"path"
|
||||
"regexp"
|
||||
"strings"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
const SchemaVersion = 1
|
||||
|
||||
// HostSDKVersion is the add-on UI SDK version this host provides. Version 2
|
||||
// adds the shared UI primitives, the hooks, the viewer and sdk.core.fetch. An
|
||||
// add-on declares the range it accepts in ui.sdk; a host outside that range
|
||||
// refuses it at install.
|
||||
const HostSDKVersion uint64 = 2
|
||||
|
||||
// SDKRefusal is the install-time refusal of an add-on whose ui.sdk range does
|
||||
// not contain the host SDK version. Its text is meant for the administrator.
|
||||
type SDKRefusal struct{ Min, Max, Host uint64 }
|
||||
|
||||
func (e *SDKRefusal) Error() string {
|
||||
return fmt.Sprintf("the add-on needs SDK %d to %d and this host provides SDK %d", e.Min, e.Max, e.Host)
|
||||
}
|
||||
|
||||
// Bounds of the optional apps-surface fields. They are character counts, not
|
||||
// bytes, so a title in any script has the same room.
|
||||
const (
|
||||
MaxTitleChars = 80
|
||||
MaxDescriptionChars = 280
|
||||
MaxLanguages = 16
|
||||
// MaxIconBytes bounds a packaged icon. Package validation enforces it
|
||||
// against the real file.
|
||||
MaxIconBytes = 256 << 10
|
||||
maxAssetPath = 256
|
||||
)
|
||||
|
||||
// LaunchModeAlways is the only launch mode of version one: the owner runs the
|
||||
// addon for as long as it is enabled. An absent mode means the same.
|
||||
const LaunchModeAlways = "always"
|
||||
|
||||
// IconTypes maps the extensions an icon may have to their media type.
|
||||
var IconTypes = map[string]string{".png": "image/png", ".webp": "image/webp", ".jpg": "image/jpeg", ".jpeg": "image/jpeg"}
|
||||
|
||||
type VersionRange struct {
|
||||
Min uint64 `json:"min"`
|
||||
Max uint64 `json:"max"`
|
||||
}
|
||||
|
||||
func (r VersionRange) accepts(v uint64) bool {
|
||||
return r.Min > 0 && r.Max >= r.Min && v >= r.Min && v <= r.Max
|
||||
}
|
||||
|
||||
type HostCompatibility struct {
|
||||
Host uint64 `json:"host"`
|
||||
API uint64 `json:"api"`
|
||||
Protocol uint64 `json:"protocol"`
|
||||
SDK uint64 `json:"sdk"`
|
||||
DataFormat uint64 `json:"dataFormat"`
|
||||
OS string `json:"os"`
|
||||
Arch string `json:"arch"`
|
||||
}
|
||||
type Manifest struct {
|
||||
SchemaVersion int `json:"schemaVersion"`
|
||||
ID string `json:"id"`
|
||||
Version string `json:"version"`
|
||||
Executable string `json:"executable"`
|
||||
OS string `json:"os"`
|
||||
Arch string `json:"arch"`
|
||||
Host VersionRange `json:"host"`
|
||||
API VersionRange `json:"api"`
|
||||
Protocol VersionRange `json:"protocol"`
|
||||
DataFormat VersionRange `json:"dataFormat"`
|
||||
// LaunchMode is absent or "always": the owner starts the addon while it is
|
||||
// enabled. "on-demand" is reserved and rejected until it is implemented.
|
||||
LaunchMode string `json:"launchMode,omitempty"`
|
||||
Capabilities Capabilities `json:"capabilities"`
|
||||
}
|
||||
type Capabilities struct {
|
||||
Service bool `json:"service,omitempty"`
|
||||
Tools []string `json:"tools,omitempty"`
|
||||
// ToolDefinitions describe the tools a model may call: the schema of the
|
||||
// arguments and the host authority of each. A declared tool without a
|
||||
// definition is not offered to models.
|
||||
ToolDefinitions []ToolDefinition `json:"toolDefinitions,omitempty"`
|
||||
// AdminTools are tools that only the host's own UI-only resource
|
||||
// management endpoints may run. They are never in Tools, never have a
|
||||
// tool definition and never reach a model, a proposal, a typed dispatch
|
||||
// or an agent: the declared-tool rule of those paths reads Tools only.
|
||||
AdminTools []string `json:"adminTools,omitempty"`
|
||||
// Resources says which tools the UI-only resource management endpoints
|
||||
// use for this add-on's resources (create, rename, inspect, delete).
|
||||
Resources *ResourceTools `json:"resources,omitempty"`
|
||||
UI *UI `json:"ui,omitempty"`
|
||||
}
|
||||
|
||||
// ResourceTools maps the host's UI-only resource management operations to
|
||||
// tool names of the add-on. Every field is optional; an operation without a
|
||||
// tool is not offered for the add-on.
|
||||
type ResourceTools struct {
|
||||
// Noun is the singular word for one resource in the UI, such as database.
|
||||
Noun string `json:"noun,omitempty"`
|
||||
// Create is a tool of Tools whose definition has the kind create.
|
||||
Create string `json:"create,omitempty"`
|
||||
// Rename is a tool of Tools whose definition has the kind schema and takes
|
||||
// a name argument.
|
||||
Rename string `json:"rename,omitempty"`
|
||||
// Inspect is a tool of Tools whose definition has the kind read. Its
|
||||
// result is an object with a name string; the host shows it as the
|
||||
// display name of the resource.
|
||||
Inspect string `json:"inspect,omitempty"`
|
||||
// Delete is a tool of AdminTools that removes the resource and its
|
||||
// backing data. It is reachable only from the UI-only delete endpoint.
|
||||
Delete string `json:"delete,omitempty"`
|
||||
}
|
||||
|
||||
// MaxAdminTools bounds capabilities.adminTools.
|
||||
const MaxAdminTools = 4
|
||||
|
||||
// ResourceExistsTool is the read tool an add-on declares to say whether the
|
||||
// backing of one of its resources exists.
|
||||
const ResourceExistsTool = "resource-exists"
|
||||
|
||||
// validateResourceTools checks AdminTools and Resources against the declared
|
||||
// tools and their definitions.
|
||||
func validateResourceTools(c Capabilities) error {
|
||||
if len(c.AdminTools) > MaxAdminTools || !uniqueIDs(c.AdminTools) {
|
||||
return fmt.Errorf("invalid admin tools")
|
||||
}
|
||||
declared := map[string]bool{}
|
||||
for _, t := range c.Tools {
|
||||
declared[t] = true
|
||||
}
|
||||
for _, t := range c.AdminTools {
|
||||
if declared[t] {
|
||||
return fmt.Errorf("admin tool %q is also a model tool", t)
|
||||
}
|
||||
}
|
||||
kinds := map[string]string{}
|
||||
for _, d := range c.ToolDefinitions {
|
||||
kinds[d.Name] = d.Kind
|
||||
}
|
||||
r := c.Resources
|
||||
if r == nil {
|
||||
if len(c.AdminTools) > 0 {
|
||||
return fmt.Errorf("admin tools need a resources mapping")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if r.Create == "" && r.Rename == "" && r.Inspect == "" && r.Delete == "" {
|
||||
return fmt.Errorf("empty resources mapping")
|
||||
}
|
||||
if r.Noun != "" && !plainText(r.Noun, 40) {
|
||||
return fmt.Errorf("invalid resource noun")
|
||||
}
|
||||
for _, want := range []struct{ name, kind, label string }{
|
||||
{r.Create, ToolKindCreate, "create"}, {r.Rename, ToolKindSchema, "rename"}, {r.Inspect, ToolKindRead, "inspect"},
|
||||
} {
|
||||
if want.name == "" {
|
||||
continue
|
||||
}
|
||||
if !declared[want.name] || kinds[want.name] != want.kind {
|
||||
return fmt.Errorf("resources %s must name a declared tool of kind %s", want.label, want.kind)
|
||||
}
|
||||
}
|
||||
if (r.Create != "" || r.Delete != "") && !declared[ResourceExistsTool] {
|
||||
// The host settles an interrupted creation or deletion only on an
|
||||
// explicit answer of this tool. Without it a missing answer would be
|
||||
// read as a missing backing and its record dropped over live data.
|
||||
return fmt.Errorf("resources that are created or deleted must declare the %s tool", ResourceExistsTool)
|
||||
}
|
||||
if r.Delete == "" {
|
||||
if len(c.AdminTools) > 0 {
|
||||
return fmt.Errorf("admin tool is not mapped")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if len(c.AdminTools) != 1 || c.AdminTools[0] != r.Delete {
|
||||
return fmt.Errorf("resources delete must name the one admin tool")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type UI struct {
|
||||
Entry string `json:"entry"`
|
||||
CSS []string `json:"css,omitempty"`
|
||||
SDK VersionRange `json:"sdk"`
|
||||
Pages []string `json:"pages,omitempty"`
|
||||
Navigation []string `json:"navigation,omitempty"`
|
||||
// ChatActions are the chat actions: a bare identifier (inert) or a full
|
||||
// declaration with label, icon, placement and target.
|
||||
ChatActions []ChatAction `json:"chatActions,omitempty"`
|
||||
// Slots place exported components in closed host slots. Renderers draw
|
||||
// the approval card or the result of one of the addon's own tools.
|
||||
Slots []SlotContribution `json:"slots,omitempty"`
|
||||
Renderers []Renderer `json:"renderers,omitempty"`
|
||||
// MainPage is the route the Apps surface opens. It must be one of Pages;
|
||||
// without it the addon has no Open action.
|
||||
MainPage string `json:"mainPage,omitempty"`
|
||||
// Icon is a packaged immutable image, never a URL.
|
||||
Icon string `json:"icon,omitempty"`
|
||||
// Title and Description are plain text, either one string or a map of
|
||||
// BCP 47 language tag to string.
|
||||
Title *Localized `json:"title,omitempty"`
|
||||
Description *Localized `json:"description,omitempty"`
|
||||
}
|
||||
|
||||
// Localized is plain text that is either one string or a map of language tag
|
||||
// to string. Exactly one of Text and ByLanguage is set.
|
||||
type Localized struct {
|
||||
Text string
|
||||
ByLanguage map[string]string
|
||||
}
|
||||
|
||||
func (l Localized) MarshalJSON() ([]byte, error) {
|
||||
if l.ByLanguage != nil {
|
||||
return json.Marshal(l.ByLanguage)
|
||||
}
|
||||
return json.Marshal(l.Text)
|
||||
}
|
||||
func (l *Localized) UnmarshalJSON(b []byte) error {
|
||||
b = bytes.TrimSpace(b)
|
||||
var out Localized
|
||||
switch {
|
||||
case len(b) > 0 && b[0] == '"':
|
||||
if e := json.Unmarshal(b, &out.Text); e != nil {
|
||||
return e
|
||||
}
|
||||
case len(b) > 0 && b[0] == '{':
|
||||
out.ByLanguage = map[string]string{}
|
||||
if e := json.Unmarshal(b, &out.ByLanguage); e != nil {
|
||||
return e
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("localized text must be a string or a language map")
|
||||
}
|
||||
*l = out
|
||||
return nil
|
||||
}
|
||||
|
||||
var namespace = regexp.MustCompile(`^[a-z][a-z0-9-]{0,63}$`)
|
||||
var immutableVersion = regexp.MustCompile(`^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-[a-z][a-z0-9-]*(\.[a-z0-9-]+)*)?$`)
|
||||
var assetPath = regexp.MustCompile(`^[a-zA-Z0-9_./-]+$`)
|
||||
|
||||
// reservedAddonIDs are the first path segments the host itself serves under
|
||||
// /api/addons/. An add-on ID shares that position (/api/addons/<id>/assets/...),
|
||||
// so an add-on with one of these IDs would have its asset URLs captured by the
|
||||
// host route of the same name. A test derives the registered segments from the
|
||||
// real routes and requires each to be listed here.
|
||||
var reservedAddonIDs = []string{"apps", "proposals", "artifacts", "resources", "conversations", "dispatch", "relay", "catalog", "icon", "assets", "installed", "sources", "builtin", "pins"}
|
||||
|
||||
// ReservedAddonIDs returns a copy of the reserved add-on IDs.
|
||||
func ReservedAddonIDs() []string { return append([]string(nil), reservedAddonIDs...) }
|
||||
|
||||
// ReservedAddonID reports whether id is a path segment the host serves.
|
||||
func ReservedAddonID(id string) bool {
|
||||
for _, r := range reservedAddonIDs {
|
||||
if r == id {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// ValidAddonID reports whether id is a well-formed add-on ID that no host
|
||||
// route uses.
|
||||
func ValidAddonID(id string) bool { return namespace.MatchString(id) && !ReservedAddonID(id) }
|
||||
|
||||
func relativePath(s string) bool {
|
||||
return s != "" && s != "." && s != ".." && assetPath.MatchString(s) && !strings.HasPrefix(s, "/") && path.Clean(s) == s && !strings.HasPrefix(s, "../")
|
||||
}
|
||||
func uniqueIDs(ids []string) bool {
|
||||
seen := map[string]bool{}
|
||||
for _, id := range ids {
|
||||
if !namespace.MatchString(id) || seen[id] {
|
||||
return false
|
||||
}
|
||||
seen[id] = true
|
||||
}
|
||||
return true
|
||||
}
|
||||
func ParseManifest(data []byte) (Manifest, error) {
|
||||
var m Manifest
|
||||
if err := strictJSON(data, &m, DefaultLimits().ManifestBytes); err != nil {
|
||||
return m, err
|
||||
}
|
||||
// A raw tool schema is kept as the bytes a JSON encoder writes for it:
|
||||
// compact, with HTML-significant characters escaped. The supervisor sends
|
||||
// the manifest's capabilities to the addon and compares what the addon
|
||||
// echoes with deep equality, so a schema held with the indentation of the
|
||||
// file would never match its own echo.
|
||||
for i := range m.Capabilities.ToolDefinitions {
|
||||
c, err := json.Marshal(m.Capabilities.ToolDefinitions[i].InputSchema)
|
||||
if err != nil {
|
||||
return m, fmt.Errorf("invalid tool schema")
|
||||
}
|
||||
m.Capabilities.ToolDefinitions[i].InputSchema = c
|
||||
}
|
||||
// Parsing validates intrinsic constraints without imposing this machine's platform.
|
||||
h := HostCompatibility{Host: m.Host.Min, API: m.API.Min, Protocol: m.Protocol.Min, DataFormat: m.DataFormat.Min, OS: m.OS, Arch: m.Arch}
|
||||
if m.Capabilities.UI != nil {
|
||||
h.SDK = m.Capabilities.UI.SDK.Min
|
||||
}
|
||||
return m, ValidateManifest(m, h)
|
||||
}
|
||||
func ValidateManifest(m Manifest, h HostCompatibility) error {
|
||||
if ReservedAddonID(m.ID) {
|
||||
return fmt.Errorf("add-on ID %q is reserved for a host route", m.ID)
|
||||
}
|
||||
if m.SchemaVersion != SchemaVersion || !namespace.MatchString(m.ID) || !immutableVersion.MatchString(m.Version) || !relativePath(m.Executable) {
|
||||
return fmt.Errorf("invalid manifest identity, version or executable")
|
||||
}
|
||||
// on-demand is reserved: accepting it would promise behavior the owner does
|
||||
// not implement, so every value but the default is refused at install.
|
||||
if m.LaunchMode != "" && m.LaunchMode != LaunchModeAlways {
|
||||
return fmt.Errorf("unsupported launch mode")
|
||||
}
|
||||
if !namespace.MatchString(m.OS) || !namespace.MatchString(m.Arch) || m.OS != h.OS || m.Arch != h.Arch {
|
||||
return fmt.Errorf("incompatible platform")
|
||||
}
|
||||
if !m.Host.accepts(h.Host) || !m.API.accepts(h.API) || !m.Protocol.accepts(h.Protocol) || !m.DataFormat.accepts(h.DataFormat) {
|
||||
return fmt.Errorf("incompatible host, API, protocol or data format")
|
||||
}
|
||||
c := m.Capabilities
|
||||
if (!c.Service && len(c.Tools) == 0 && c.UI == nil) || !uniqueIDs(c.Tools) {
|
||||
return fmt.Errorf("invalid capabilities")
|
||||
}
|
||||
if e := validateToolDefinitions(c.ToolDefinitions, c.Tools); e != nil {
|
||||
return e
|
||||
}
|
||||
if e := validateResourceTools(c); e != nil {
|
||||
return e
|
||||
}
|
||||
if u := c.UI; u != nil {
|
||||
if u.SDK.Min > 0 && u.SDK.Max >= u.SDK.Min && !u.SDK.accepts(h.SDK) {
|
||||
return &SDKRefusal{Min: u.SDK.Min, Max: u.SDK.Max, Host: h.SDK}
|
||||
}
|
||||
if !relativePath(u.Entry) || !u.SDK.accepts(h.SDK) || !uniqueIDs(u.Pages) || !uniqueIDs(u.Navigation) {
|
||||
return fmt.Errorf("invalid or incompatible UI")
|
||||
}
|
||||
seen := map[string]bool{u.Entry: true}
|
||||
for _, css := range u.CSS {
|
||||
if !relativePath(css) || seen[css] {
|
||||
return fmt.Errorf("invalid UI asset")
|
||||
}
|
||||
seen[css] = true
|
||||
}
|
||||
if u.Icon != "" {
|
||||
if !ValidIconPath(u.Icon) || seen[u.Icon] {
|
||||
return fmt.Errorf("invalid UI icon")
|
||||
}
|
||||
}
|
||||
if u.Title != nil && !u.Title.valid(MaxTitleChars) || u.Description != nil && !u.Description.valid(MaxDescriptionChars) {
|
||||
return fmt.Errorf("invalid UI title or description")
|
||||
}
|
||||
pages := map[string]bool{}
|
||||
for _, p := range u.Pages {
|
||||
pages[p] = true
|
||||
}
|
||||
if u.MainPage != "" && !pages[u.MainPage] {
|
||||
return fmt.Errorf("main page requires declared page")
|
||||
}
|
||||
for _, n := range u.Navigation {
|
||||
if !pages[n] {
|
||||
return fmt.Errorf("navigation requires declared page")
|
||||
}
|
||||
}
|
||||
if e := validateContributions(u, c.Tools); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var languageTag = regexp.MustCompile(`^[A-Za-z]{2,3}(-[A-Za-z0-9]{2,8}){0,3}$`)
|
||||
|
||||
// ValidIconPath reports whether p is a package-relative path of an allowed
|
||||
// image type. A URL, an absolute path and a traversal never are.
|
||||
func ValidIconPath(p string) bool {
|
||||
return len(p) <= maxAssetPath && IconTypes[strings.ToLower(path.Ext(p))] != "" && relativePath(p)
|
||||
}
|
||||
|
||||
// ValidPlainText reports whether s is non-empty display text of at most max
|
||||
// characters with no markup, control characters or direction overrides.
|
||||
func ValidPlainText(s string, max int) bool { return plainText(s, max) }
|
||||
|
||||
// ValidName reports whether s is a page, tool or addon identifier.
|
||||
func ValidName(s string) bool { return namespace.MatchString(s) }
|
||||
|
||||
// plainText accepts text that cannot carry markup, control characters or
|
||||
// direction overrides, within max characters.
|
||||
func plainText(s string, max int) bool {
|
||||
n := utf8.RuneCountInString(s)
|
||||
if !utf8.ValidString(s) || n == 0 || n > max || strings.TrimSpace(s) != s {
|
||||
return false
|
||||
}
|
||||
for _, r := range s {
|
||||
if r == '<' || r == '>' || unicode.IsControl(r) || unicode.Is(unicode.Cf, r) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (l Localized) valid(max int) bool {
|
||||
if l.ByLanguage == nil {
|
||||
return plainText(l.Text, max)
|
||||
}
|
||||
if l.Text != "" || len(l.ByLanguage) == 0 || len(l.ByLanguage) > MaxLanguages {
|
||||
return false
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for tag, text := range l.ByLanguage {
|
||||
key := strings.ToLower(tag)
|
||||
if len(tag) > 35 || !languageTag.MatchString(tag) || seen[key] || !plainText(text, max) {
|
||||
return false
|
||||
}
|
||||
seen[key] = true
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// Default is the text a caller that does not choose a language gets: the
|
||||
// plain string, else the English entry, else the first tag in sorted order.
|
||||
func (l Localized) Default() string {
|
||||
if l.ByLanguage == nil {
|
||||
return l.Text
|
||||
}
|
||||
if t, ok := l.ByLanguage["en"]; ok {
|
||||
return t
|
||||
}
|
||||
best := ""
|
||||
for tag := range l.ByLanguage {
|
||||
if best == "" || tag < best {
|
||||
best = tag
|
||||
}
|
||||
}
|
||||
return l.ByLanguage[best]
|
||||
}
|
||||
|
||||
// strictJSON rejects duplicate keys as well as unknown fields. The token walk
|
||||
// also bounds nesting before the standard decoder materializes wire values.
|
||||
func strictJSON(b []byte, v any, max int64) error {
|
||||
if int64(len(b)) > max || len(bytes.TrimSpace(b)) == 0 || bytes.TrimSpace(b)[0] != '{' {
|
||||
return fmt.Errorf("invalid or oversized JSON object")
|
||||
}
|
||||
d := json.NewDecoder(bytes.NewReader(b))
|
||||
d.UseNumber()
|
||||
if err := walkJSON(d, 0); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := d.Token(); err != io.EOF {
|
||||
return fmt.Errorf("trailing JSON")
|
||||
}
|
||||
d = json.NewDecoder(bytes.NewReader(b))
|
||||
d.DisallowUnknownFields()
|
||||
return d.Decode(v)
|
||||
}
|
||||
func walkJSON(d *json.Decoder, depth int) error {
|
||||
if depth > 64 {
|
||||
return fmt.Errorf("JSON nesting exceeds limit")
|
||||
}
|
||||
t, err := d.Token()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
delim, ok := t.(json.Delim)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
switch delim {
|
||||
case '{':
|
||||
seen := map[string]bool{}
|
||||
for d.More() {
|
||||
key, e := d.Token()
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
k, ok := key.(string)
|
||||
if !ok || seen[k] {
|
||||
return fmt.Errorf("duplicate JSON key")
|
||||
}
|
||||
seen[k] = true
|
||||
if e = walkJSON(d, depth+1); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
case '[':
|
||||
for d.More() {
|
||||
if e := walkJSON(d, depth+1); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("invalid delimiter")
|
||||
}
|
||||
_, err = d.Token()
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,355 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package contract
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func fixture() []byte {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/service.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return b
|
||||
}
|
||||
func compatible() HostCompatibility {
|
||||
return HostCompatibility{Host: 1, API: 1, Protocol: 1, SDK: 1, DataFormat: 1, OS: "linux", Arch: "amd64"}
|
||||
}
|
||||
|
||||
var _ = Describe("Manifest", func() {
|
||||
It("refuses an add-on that needs a newer SDK than the host provides, and says so", func() {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/page.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m.Capabilities.UI.SDK = VersionRange{Min: 2, Max: 2}
|
||||
host := compatible() // SDK 1
|
||||
e = ValidateManifest(m, host)
|
||||
Expect(e).To(MatchError("the add-on needs SDK 2 to 2 and this host provides SDK 1"))
|
||||
})
|
||||
|
||||
It("accepts a range that contains the host SDK", func() {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/page.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m.Capabilities.UI.SDK = VersionRange{Min: 1, Max: 2}
|
||||
host := compatible()
|
||||
host.SDK = HostSDKVersion
|
||||
Expect(ValidateManifest(m, host)).To(Succeed())
|
||||
})
|
||||
|
||||
It("accepts a range that spans the host SDK", func() {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/page.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m.Capabilities.UI.SDK = VersionRange{Min: 1, Max: 3}
|
||||
host := compatible()
|
||||
host.SDK = HostSDKVersion
|
||||
Expect(ValidateManifest(m, host)).To(Succeed())
|
||||
})
|
||||
|
||||
It("refuses an add-on built for SDK 1 only on a version 2 host", func() {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/page.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m.Capabilities.UI.SDK = VersionRange{Min: 1, Max: 1}
|
||||
host := compatible()
|
||||
host.SDK = HostSDKVersion
|
||||
Expect(ValidateManifest(m, host)).To(MatchError("the add-on needs SDK 1 to 1 and this host provides SDK 2"))
|
||||
})
|
||||
|
||||
It("keeps the generic message for a range that is malformed", func() {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/page.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m.Capabilities.UI.SDK = VersionRange{Min: 3, Max: 2}
|
||||
Expect(ValidateManifest(m, compatible())).To(MatchError("invalid or incompatible UI"))
|
||||
})
|
||||
|
||||
It("accepts independent service tool page and chat capabilities", func() {
|
||||
for _, name := range []string{"service", "headless", "page", "chat"} {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/" + name + ".json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(ValidateManifest(m, compatible())).To(Succeed())
|
||||
}
|
||||
m, e := ParseManifest(fixture())
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
h := compatible()
|
||||
h.SDK = 0
|
||||
Expect(ValidateManifest(m, h)).To(Succeed())
|
||||
m.Capabilities = Capabilities{Tools: []string{"query"}}
|
||||
Expect(ValidateManifest(m, h)).To(Succeed())
|
||||
})
|
||||
It("requires the existence tool of an addon whose resources are created or deleted", func() {
|
||||
raw, e := os.ReadFile("../../../addons/tables/manifest.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
load := func() Manifest {
|
||||
m, e := ParseManifest(raw)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return m
|
||||
}
|
||||
without := func(m *Manifest) {
|
||||
var keep []string
|
||||
for _, t := range m.Capabilities.Tools {
|
||||
if t != "resource-exists" {
|
||||
keep = append(keep, t)
|
||||
}
|
||||
}
|
||||
m.Capabilities.Tools = keep
|
||||
}
|
||||
Expect(ValidateManifest(load(), compatible())).To(Succeed())
|
||||
|
||||
// Delete and create both need it: without an explicit answer the
|
||||
// host could not tell a missing backing from an add-on that cannot
|
||||
// answer, and would drop the record of data that is still there.
|
||||
m := load()
|
||||
without(&m)
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed())
|
||||
m = load()
|
||||
without(&m)
|
||||
m.Capabilities.Resources.Delete, m.Capabilities.AdminTools = "", nil
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), "create only")
|
||||
m = load()
|
||||
without(&m)
|
||||
m.Capabilities.Resources.Create = ""
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), "delete only")
|
||||
// A mapping that creates and deletes nothing needs no existence tool.
|
||||
m = load()
|
||||
without(&m)
|
||||
m.Capabilities.Resources.Create, m.Capabilities.Resources.Delete, m.Capabilities.AdminTools = "", "", nil
|
||||
Expect(ValidateManifest(m, compatible())).To(Succeed())
|
||||
})
|
||||
It("rejects invalid namespace IDs and executable paths", func() {
|
||||
for _, id := range []string{"", "../sample", "a/b", "A", "a.b", strings.Repeat("a", 65)} {
|
||||
m, _ := ParseManifest(fixture())
|
||||
m.ID = id
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), id)
|
||||
}
|
||||
for _, p := range []string{"", "/bin/x", "..", "../x", "bin/../x", "bin//x", "C:/x", `bin\x`, "https://x", "bin/%2e/x"} {
|
||||
m, _ := ParseManifest(fixture())
|
||||
m.Executable = p
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), p)
|
||||
}
|
||||
})
|
||||
It("rejects add-on IDs that the host serves as a route segment", func() {
|
||||
// Every first path segment the host uses under /api/addons/ would
|
||||
// otherwise capture the asset URL of an add-on with that ID.
|
||||
Expect(ReservedAddonIDs()).To(ConsistOf("apps", "proposals", "artifacts", "resources", "conversations", "dispatch", "relay", "catalog", "icon", "assets", "installed", "sources", "builtin", "pins"))
|
||||
for _, id := range ReservedAddonIDs() {
|
||||
m, _ := ParseManifest(fixture())
|
||||
m.ID = id
|
||||
e := ValidateManifest(m, compatible())
|
||||
Expect(e).To(HaveOccurred(), id)
|
||||
Expect(e.Error()).To(ContainSubstring("reserved"), id)
|
||||
Expect(ValidAddonID(id)).To(BeFalse(), id)
|
||||
}
|
||||
Expect(ValidAddonID("proposals-tool")).To(BeTrue())
|
||||
m, _ := ParseManifest(fixture())
|
||||
m.ID = "app"
|
||||
Expect(ValidateManifest(m, compatible())).To(Succeed())
|
||||
})
|
||||
It("rejects invalid versions ranges and capabilities", func() {
|
||||
for _, v := range []string{"", "latest", "development", "v1.0.0", "01.2.3", "1.2", "1.2.3+mutable", "1.2.3-a..b", "1.2.3-a."} {
|
||||
m, _ := ParseManifest(fixture())
|
||||
m.Version = v
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), v)
|
||||
}
|
||||
m, _ := ParseManifest(fixture())
|
||||
m.SchemaVersion = 2
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed())
|
||||
m, _ = ParseManifest(fixture())
|
||||
m.API = VersionRange{Min: 2, Max: 1}
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed())
|
||||
m, _ = ParseManifest(fixture())
|
||||
m.Capabilities = Capabilities{}
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed())
|
||||
m.Capabilities.Tools = []string{"query", "query"}
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed())
|
||||
})
|
||||
It("rejects unknown duplicate trailing and oversized manifest JSON", func() {
|
||||
for _, b := range [][]byte{[]byte(strings.Replace(string(fixture()), `"service":true`, `"service":true,"shell":true`, 1)), []byte(strings.Replace(string(fixture()), `"id":"sample"`, `"id":"sample","id":"other"`, 1)), append(fixture(), []byte(` {}`)...), []byte(`null`), []byte(strings.Repeat(" ", int(DefaultLimits().ManifestBytes)+1))} {
|
||||
_, e := ParseManifest(b)
|
||||
Expect(e).To(HaveOccurred())
|
||||
}
|
||||
})
|
||||
It("rejects incompatible host API protocol SDK data and platform fixtures", func() {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/incompatible.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var hosts []HostCompatibility
|
||||
Expect(json.Unmarshal(b, &hosts)).To(Succeed())
|
||||
page, e := os.ReadFile("../../../tests/addons/fixtures/contracts/page.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(page)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
for _, h := range hosts {
|
||||
Expect(ValidateManifest(m, h)).NotTo(Succeed())
|
||||
}
|
||||
})
|
||||
It("validates UI only when declared and rejects inconsistent contributions", func() {
|
||||
m, _ := ParseManifest(fixture())
|
||||
m.Capabilities.UI = &UI{Entry: "ui/main.js", SDK: VersionRange{Min: 1, Max: 1}, ChatActions: []ChatAction{{ID: "open"}}}
|
||||
Expect(ValidateManifest(m, compatible())).To(Succeed())
|
||||
for _, u := range []*UI{{}, {Entry: "../x", SDK: VersionRange{1, 1}}, {Entry: "ui/main.js", SDK: VersionRange{1, 1}, CSS: []string{"/x"}}, {Entry: "ui/main.js", SDK: VersionRange{1, 1}, Navigation: []string{"missing"}}, {Entry: "ui/main.js", SDK: VersionRange{1, 1}, ChatActions: []ChatAction{{ID: "open"}, {ID: "open"}}}} {
|
||||
m.Capabilities.UI = u
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed())
|
||||
}
|
||||
})
|
||||
It("accepts the apps surface fields and keeps headless manifests valid", func() {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/app.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(m.LaunchMode).To(Equal("always"))
|
||||
u := m.Capabilities.UI
|
||||
Expect(u.MainPage).To(Equal("home"))
|
||||
Expect(u.Icon).To(Equal("ui/icon.png"))
|
||||
Expect(u.Title.ByLanguage).To(Equal(map[string]string{"en": "Sample", "it": "Esempio"}))
|
||||
Expect(u.Description.Text).To(Equal("A sample addon."))
|
||||
// The canonical wire form is stable: decoding and encoding it again
|
||||
// gives the same bytes.
|
||||
out, e := json.Marshal(m)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var compact bytes.Buffer
|
||||
Expect(json.Compact(&compact, b)).To(Succeed())
|
||||
Expect(string(out)).To(Equal(compact.String()))
|
||||
// Every field is optional: a headless manifest and a UI without a
|
||||
// main page stay valid, and an explicit launch mode is not required.
|
||||
for _, name := range []string{"headless", "service", "page", "chat"} {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/" + name + ".json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred(), name)
|
||||
Expect(m.LaunchMode).To(BeEmpty())
|
||||
if m.Capabilities.UI != nil {
|
||||
Expect(m.Capabilities.UI.MainPage).To(BeEmpty())
|
||||
}
|
||||
}
|
||||
headless, e := ParseManifest(fixture())
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
headless.LaunchMode = "always"
|
||||
Expect(ValidateManifest(headless, compatible())).To(Succeed())
|
||||
})
|
||||
It("rejects unknown and reserved launch modes", func() {
|
||||
for _, mode := range []string{"on-demand", "ON-DEMAND", "Always", "never", "manual", " always", "always ", "lazy"} {
|
||||
m, _ := ParseManifest(fixture())
|
||||
m.LaunchMode = mode
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), mode)
|
||||
b, _ := json.Marshal(m)
|
||||
_, e := ParseManifest(b)
|
||||
Expect(e).To(HaveOccurred(), mode)
|
||||
}
|
||||
})
|
||||
It("rejects remote data absolute and traversing icon paths", func() {
|
||||
app := func() Manifest {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/app.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return m
|
||||
}
|
||||
for _, icon := range []string{"http://example.test/i.png", "https://example.test/i.png", "//example.test/i.png", "data:image/png;base64,AAAA", "/etc/icon.png", "../icon.png", "ui/../../icon.png", "ui//icon.png", "./ui/icon.png", `ui\icon.png`, "C:/icon.png", "ui/icon.png?x=1", "ui/icon.png#x", "ui/icon", "ui/icon.svg", "ui/icon.html", "ui/icon.js", "ui/", ".", "..", strings.Repeat("a/", 200) + "icon.png"} {
|
||||
m := app()
|
||||
m.Capabilities.UI.Icon = icon
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), icon)
|
||||
}
|
||||
// An icon cannot also be the module or a stylesheet.
|
||||
for _, icon := range []string{"ui/main.js", "ui/main.css"} {
|
||||
m := app()
|
||||
m.Capabilities.UI.Icon = icon
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), icon)
|
||||
}
|
||||
for _, icon := range []string{"ui/icon.png", "assets/icon.webp", "icon.jpg", "ui/icon.jpeg"} {
|
||||
m := app()
|
||||
m.Capabilities.UI.Icon = icon
|
||||
Expect(ValidateManifest(m, compatible())).To(Succeed(), icon)
|
||||
}
|
||||
})
|
||||
It("requires the main page to be one of the declared pages", func() {
|
||||
app := func() Manifest {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/app.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return m
|
||||
}
|
||||
for _, page := range []string{"missing", "Home", "../home", "/home", "home/", "home x"} {
|
||||
m := app()
|
||||
m.Capabilities.UI.MainPage = page
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), page)
|
||||
}
|
||||
// A main page without any declared page is not a route either.
|
||||
m := app()
|
||||
m.Capabilities.UI.Pages = nil
|
||||
m.Capabilities.UI.Navigation = nil
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed())
|
||||
m.Capabilities.UI.MainPage = ""
|
||||
Expect(ValidateManifest(m, compatible())).To(Succeed())
|
||||
// A main page need not be in the navigation.
|
||||
m = app()
|
||||
m.Capabilities.UI.MainPage = "settings"
|
||||
Expect(ValidateManifest(m, compatible())).To(Succeed())
|
||||
})
|
||||
It("bounds localized titles and descriptions and rejects markup", func() {
|
||||
app := func() Manifest {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/app.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
m, e := ParseManifest(b)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
return m
|
||||
}
|
||||
many := map[string]string{}
|
||||
for i := 0; i < 40; i++ {
|
||||
many[string(rune('a'+i%26))+string(rune('a'+i/26))+"-US"] = "x"
|
||||
}
|
||||
bad := []*Localized{
|
||||
{Text: ""}, {Text: strings.Repeat("a", 281)}, {Text: "<b>Sample</b>"}, {Text: "a > b"}, {Text: "line\nbreak"}, {Text: "bell\a"}, {Text: "bidi \u202e override"}, {Text: "\xff\xfe"},
|
||||
{ByLanguage: map[string]string{}}, {ByLanguage: many}, {ByLanguage: map[string]string{"en": ""}}, {ByLanguage: map[string]string{"en": strings.Repeat("a", 281)}},
|
||||
{ByLanguage: map[string]string{"": "x"}}, {ByLanguage: map[string]string{"not a tag": "x"}}, {ByLanguage: map[string]string{"en_US": "x"}}, {ByLanguage: map[string]string{"en": "x", "EN": "y"}}, {ByLanguage: map[string]string{"en": "<i>x</i>"}}, {ByLanguage: map[string]string{strings.Repeat("a", 40): "x"}},
|
||||
}
|
||||
for i, l := range bad {
|
||||
m := app()
|
||||
m.Capabilities.UI.Title = l
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), "title %d", i)
|
||||
m = app()
|
||||
m.Capabilities.UI.Description = l
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), "description %d", i)
|
||||
}
|
||||
m := app()
|
||||
m.Capabilities.UI.Title = &Localized{Text: strings.Repeat("a", 81)}
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed(), "title is limited to 80 characters")
|
||||
m.Capabilities.UI.Title = &Localized{ByLanguage: map[string]string{"en": strings.Repeat("a", 81)}}
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed())
|
||||
m = app()
|
||||
m.Capabilities.UI.Description = &Localized{Text: strings.Repeat("a", 280)}
|
||||
Expect(ValidateManifest(m, compatible())).To(Succeed())
|
||||
m.Capabilities.UI.Description = &Localized{Text: strings.Repeat("a", 281)}
|
||||
Expect(ValidateManifest(m, compatible())).NotTo(Succeed())
|
||||
m.Capabilities.UI.Title = &Localized{Text: strings.Repeat("\u00e8", 80)}
|
||||
m.Capabilities.UI.Description = nil
|
||||
Expect(ValidateManifest(m, compatible())).To(Succeed(), "bounds count characters")
|
||||
// Wire shapes: a string and a map decode, anything else is rejected.
|
||||
for _, w := range []string{`"title":7`, `"title":["a"]`, `"title":null,"x":1`, `"title":{"en":7}`, `"title":{"en":"a","en":"b"}`} {
|
||||
b, _ := os.ReadFile("../../../tests/addons/fixtures/contracts/app.json")
|
||||
c := strings.Replace(string(b), `"title": {
|
||||
"en": "Sample",
|
||||
"it": "Esempio"
|
||||
}`, w, 1)
|
||||
_, e := ParseManifest([]byte(c))
|
||||
Expect(e).To(HaveOccurred(), w)
|
||||
}
|
||||
})
|
||||
It("reserves the segments of the navigation routes", func() {
|
||||
Expect(ReservedAddonID("builtin")).To(BeTrue())
|
||||
Expect(ReservedAddonID("pins")).To(BeTrue())
|
||||
Expect(ValidAddonID("pins")).To(BeFalse())
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,223 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package contract
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"mime"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ErrRefused is the add-on's own refusal of a request it answered as
|
||||
// invalid: nothing was changed and repeating the same call cannot succeed.
|
||||
// It is not an outage and not an unknown outcome.
|
||||
var ErrRefused = errors.New("addon refused the request as invalid")
|
||||
|
||||
type Actor struct {
|
||||
UserID string `json:"userId"`
|
||||
CredentialID string `json:"credentialId"`
|
||||
Mode string `json:"mode"`
|
||||
}
|
||||
type Request struct {
|
||||
AddonID string `json:"addonId"`
|
||||
ResourceID string `json:"resourceId,omitempty"`
|
||||
OperationID string `json:"operationId"`
|
||||
Generation uint64 `json:"generation"`
|
||||
Action string `json:"action"`
|
||||
Arguments json.RawMessage `json:"arguments" swaggertype:"object"`
|
||||
}
|
||||
type Authority struct {
|
||||
Actor Actor `json:"actor"`
|
||||
Scope string `json:"scope"`
|
||||
ExpiresAt time.Time `json:"expiresAt"`
|
||||
}
|
||||
type Envelope struct {
|
||||
Request Request `json:"request"`
|
||||
Authority Authority `json:"authority"`
|
||||
}
|
||||
type OutputDescription struct {
|
||||
Handle string `json:"handle"`
|
||||
Name string `json:"name"`
|
||||
MediaType string `json:"mediaType"`
|
||||
Size int64 `json:"size"`
|
||||
}
|
||||
type Result struct {
|
||||
Payload json.RawMessage `json:"payload,omitempty" swaggertype:"object"`
|
||||
Outputs []OutputDescription `json:"outputs,omitempty"`
|
||||
}
|
||||
type Revision struct {
|
||||
Schema uint64 `json:"schema"`
|
||||
Data uint64 `json:"data"`
|
||||
}
|
||||
type Preview struct {
|
||||
OperationID string `json:"operationId"`
|
||||
Revision Revision `json:"revision"`
|
||||
ExpiresAt time.Time `json:"expiresAt"`
|
||||
Summary json.RawMessage `json:"summary" swaggertype:"object"`
|
||||
}
|
||||
type ReceiptState string
|
||||
|
||||
const (
|
||||
ReceiptPending ReceiptState = "pending"
|
||||
ReceiptCommitted ReceiptState = "committed"
|
||||
ReceiptRejected ReceiptState = "rejected"
|
||||
ReceiptUnknown ReceiptState = "unknown"
|
||||
)
|
||||
|
||||
type Receipt struct {
|
||||
OperationID string `json:"operationId"`
|
||||
State ReceiptState `json:"state"`
|
||||
Revision Revision `json:"revision"`
|
||||
Result Result `json:"result"`
|
||||
}
|
||||
type Runtime interface {
|
||||
Read(context.Context, Envelope) (Result, error)
|
||||
Preview(context.Context, Envelope) (Preview, error)
|
||||
Apply(context.Context, Envelope, Revision) (Receipt, error)
|
||||
Receipt(context.Context, string) (Receipt, error)
|
||||
OpenOutput(context.Context, Envelope, string) (io.ReadCloser, error)
|
||||
}
|
||||
|
||||
// UnmarshalJSON applies fixed default ingress bounds and structural validation.
|
||||
// Hosts with overrides decode using their transport's bounded decoder and call
|
||||
// Validate with their configured limits before admission.
|
||||
func (v *Envelope) UnmarshalJSON(b []byte) error {
|
||||
type wire Envelope
|
||||
var w wire
|
||||
if err := strictJSON(b, &w, DefaultLimits().EnvelopeBytes); err != nil {
|
||||
return err
|
||||
}
|
||||
candidate := Envelope(w)
|
||||
if err := candidate.Validate(time.Time{}, DefaultLimits()); err != nil {
|
||||
return err
|
||||
}
|
||||
*v = candidate
|
||||
return nil
|
||||
}
|
||||
func (v Envelope) Validate(now time.Time, l Limits) error {
|
||||
if err := l.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
r, a := v.Request, v.Authority
|
||||
if !namespace.MatchString(r.AddonID) || !namespace.MatchString(r.OperationID) || !namespace.MatchString(r.Action) || (r.ResourceID != "" && !namespace.MatchString(r.ResourceID)) || r.Generation == 0 {
|
||||
return fmt.Errorf("invalid request binding")
|
||||
}
|
||||
// Principal identifiers are opaque authentication-system IDs, not filesystem paths.
|
||||
if strings.TrimSpace(a.Actor.UserID) == "" || strings.TrimSpace(a.Actor.CredentialID) == "" || len(a.Actor.UserID) > 256 || len(a.Actor.CredentialID) > 256 || (a.Actor.Mode != "named-user" && a.Actor.Mode != "local-owner") || !a.ExpiresAt.After(now) {
|
||||
return fmt.Errorf("incomplete or expired authority")
|
||||
}
|
||||
switch a.Scope {
|
||||
case "read", "preview", "apply", "output":
|
||||
default:
|
||||
return fmt.Errorf("invalid authority scope")
|
||||
}
|
||||
var args map[string]json.RawMessage
|
||||
if err := strictJSON(r.Arguments, &args, l.PayloadBytes); err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if int64(len(b)) > l.EnvelopeBytes {
|
||||
return fmt.Errorf("oversized envelope")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (r Result) Validate(l Limits) error {
|
||||
if err := l.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(r.Payload) > 0 && (!json.Valid(r.Payload) || int64(len(r.Payload)) > l.PayloadBytes) {
|
||||
return fmt.Errorf("invalid result payload")
|
||||
}
|
||||
if int64(len(r.Outputs)) > l.ArtifactCount {
|
||||
return fmt.Errorf("too many outputs")
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
var total int64
|
||||
for _, o := range r.Outputs {
|
||||
if !namespace.MatchString(o.Handle) || seen[o.Handle] || !relativePath(o.Name) || strings.Contains(o.Name, "/") || o.Size < 0 || o.Size > l.ArtifactBytes {
|
||||
return fmt.Errorf("invalid output")
|
||||
}
|
||||
seen[o.Handle] = true
|
||||
if _, _, err := mime.ParseMediaType(o.MediaType); err != nil || !strings.Contains(o.MediaType, "/") {
|
||||
return fmt.Errorf("invalid media type")
|
||||
}
|
||||
if o.Size > l.ArtifactQuotaBytes-total {
|
||||
return fmt.Errorf("output quota exceeded")
|
||||
}
|
||||
total += o.Size
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (p Preview) Validate(now time.Time, l Limits) error {
|
||||
if err := l.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
if !namespace.MatchString(p.OperationID) || p.Revision.Schema == 0 || !p.ExpiresAt.After(now) {
|
||||
return fmt.Errorf("invalid preview")
|
||||
}
|
||||
if !json.Valid(p.Summary) || int64(len(p.Summary)) > l.PayloadBytes {
|
||||
return fmt.Errorf("invalid preview summary")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (r Receipt) Validate(l Limits) error {
|
||||
if !namespace.MatchString(r.OperationID) {
|
||||
return fmt.Errorf("invalid receipt operation ID")
|
||||
}
|
||||
switch r.State {
|
||||
case ReceiptPending, ReceiptCommitted, ReceiptRejected, ReceiptUnknown:
|
||||
default:
|
||||
return fmt.Errorf("invalid receipt state")
|
||||
}
|
||||
if r.State == ReceiptCommitted && r.Revision.Schema == 0 {
|
||||
return fmt.Errorf("committed receipt requires revision")
|
||||
}
|
||||
return r.Result.Validate(l)
|
||||
}
|
||||
|
||||
func (v *Result) UnmarshalJSON(b []byte) error {
|
||||
type wire Result
|
||||
var w wire
|
||||
if err := strictJSON(b, &w, DefaultLimits().EnvelopeBytes); err != nil {
|
||||
return err
|
||||
}
|
||||
candidate := Result(w)
|
||||
if err := candidate.Validate(DefaultLimits()); err != nil {
|
||||
return err
|
||||
}
|
||||
*v = candidate
|
||||
return nil
|
||||
}
|
||||
func (v *Preview) UnmarshalJSON(b []byte) error {
|
||||
type wire Preview
|
||||
var w wire
|
||||
if err := strictJSON(b, &w, DefaultLimits().EnvelopeBytes); err != nil {
|
||||
return err
|
||||
}
|
||||
candidate := Preview(w)
|
||||
if err := candidate.Validate(time.Time{}, DefaultLimits()); err != nil {
|
||||
return err
|
||||
}
|
||||
*v = candidate
|
||||
return nil
|
||||
}
|
||||
func (v *Receipt) UnmarshalJSON(b []byte) error {
|
||||
type wire Receipt
|
||||
var w wire
|
||||
if err := strictJSON(b, &w, DefaultLimits().EnvelopeBytes); err != nil {
|
||||
return err
|
||||
}
|
||||
candidate := Receipt(w)
|
||||
if err := candidate.Validate(DefaultLimits()); err != nil {
|
||||
return err
|
||||
}
|
||||
*v = candidate
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package contract
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func envelope() Envelope {
|
||||
return Envelope{Request: Request{AddonID: "sample", ResourceID: "resource-1", OperationID: "operation-1", Generation: 1, Action: "query", Arguments: json.RawMessage(`{"limit":1}`)}, Authority: Authority{Actor: Actor{UserID: "user-1", CredentialID: "key-1", Mode: "named-user"}, Scope: "read", ExpiresAt: time.Date(2030, 1, 1, 0, 0, 0, 0, time.UTC)}}
|
||||
}
|
||||
|
||||
var _ = Describe("Protocol", func() {
|
||||
It("uses canonical version one wire fixtures", func() {
|
||||
b, e := os.ReadFile("../../../tests/addons/fixtures/contracts/envelope.json")
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
var v Envelope
|
||||
Expect(v.UnmarshalJSON(b)).To(Succeed())
|
||||
Expect(v).To(Equal(envelope()))
|
||||
out, e := json.Marshal(v)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
Expect(string(out)).To(Equal(strings.TrimSpace(string(b))))
|
||||
for _, v := range []any{Preview{OperationID: "operation-1", Revision: Revision{Schema: 1, Data: 2}, ExpiresAt: envelope().Authority.ExpiresAt, Summary: json.RawMessage(`{"count":1}`)}, Receipt{OperationID: "operation-1", State: ReceiptCommitted, Revision: Revision{Schema: 1, Data: 3}, Result: Result{Payload: json.RawMessage(`{"count":1}`), Outputs: []OutputDescription{{Handle: "output-1", Name: "export.csv", MediaType: "text/csv", Size: 10}}}}} {
|
||||
b, e := json.Marshal(v)
|
||||
Expect(e).NotTo(HaveOccurred())
|
||||
copy := reflect.New(reflect.TypeOf(v))
|
||||
Expect(json.Unmarshal(b, copy.Interface())).To(Succeed())
|
||||
Expect(copy.Elem().Interface()).To(Equal(v))
|
||||
}
|
||||
})
|
||||
It("rejects incomplete expired or forged authority shapes", func() {
|
||||
now := time.Date(2029, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
for _, change := range []func(*Envelope){func(v *Envelope) { v.Authority.Actor.UserID = "" }, func(v *Envelope) { v.Authority.Actor.CredentialID = "" }, func(v *Envelope) { v.Authority.Actor.Mode = "anonymous" }, func(v *Envelope) { v.Authority.Scope = "admin" }, func(v *Envelope) { v.Authority.ExpiresAt = now }, func(v *Envelope) { v.Request.OperationID = "" }, func(v *Envelope) { v.Request.Generation = 0 }, func(v *Envelope) { v.Request.AddonID = "../x" }, func(v *Envelope) { v.Request.ResourceID = "/x" }, func(v *Envelope) { v.Request.Arguments = json.RawMessage(`null`) }} {
|
||||
v := envelope()
|
||||
change(&v)
|
||||
Expect(v.Validate(now, DefaultLimits())).NotTo(Succeed())
|
||||
}
|
||||
Expect(envelope().Validate(now, DefaultLimits())).To(Succeed())
|
||||
b, _ := json.Marshal(envelope())
|
||||
b = []byte(strings.Replace(string(b), `"scope":"read"`, `"scope":"read","approved":true`, 1))
|
||||
var v Envelope
|
||||
Expect(json.Unmarshal(b, &v)).NotTo(Succeed())
|
||||
})
|
||||
It("rejects unknown and oversized wire payloads", func() {
|
||||
for _, target := range []any{&Preview{}, &Receipt{}, &Result{}} {
|
||||
Expect(json.Unmarshal([]byte(`{"unexpected":true}`), target)).NotTo(Succeed())
|
||||
}
|
||||
|
||||
v := envelope()
|
||||
v.Request.Arguments = json.RawMessage(`{"x":"` + strings.Repeat("x", int(DefaultLimits().PayloadBytes)) + `"}`)
|
||||
Expect(v.Validate(time.Now(), DefaultLimits())).NotTo(Succeed())
|
||||
b, _ := json.Marshal(v)
|
||||
var other Envelope
|
||||
Expect(json.Unmarshal(b, &other)).NotTo(Succeed())
|
||||
for _, s := range []string{`null`, `{}`, `{"request":{},"request":{}}`, `{"authority":{"actor":{"unexpected":true}}}`} {
|
||||
Expect(json.Unmarshal([]byte(s), &other)).NotTo(Succeed())
|
||||
}
|
||||
})
|
||||
It("validates finite defaults overrides and cross field bounds", func() {
|
||||
l := DefaultLimits()
|
||||
Expect(l.Validate()).To(Succeed())
|
||||
rv := reflect.ValueOf(&l).Elem()
|
||||
for i := 0; i < rv.NumField(); i++ {
|
||||
old := rv.Field(i).Int()
|
||||
rv.Field(i).SetInt(0)
|
||||
Expect(l.Validate()).NotTo(Succeed(), rv.Type().Field(i).Name)
|
||||
rv.Field(i).SetInt(-1)
|
||||
Expect(l.Validate()).NotTo(Succeed())
|
||||
rv.Field(i).SetInt(old)
|
||||
}
|
||||
l.QueryRows = 50
|
||||
l.Concurrency = 2
|
||||
Expect(l.Validate()).To(Succeed())
|
||||
l.BackoffInitial = l.BackoffMax + 1
|
||||
Expect(l.Validate()).NotTo(Succeed())
|
||||
l = DefaultLimits()
|
||||
l.ArtifactBytes = l.ArtifactQuotaBytes + 1
|
||||
Expect(l.Validate()).NotTo(Succeed())
|
||||
l = DefaultLimits()
|
||||
l.PayloadBytes = l.EnvelopeBytes + 1
|
||||
Expect(l.Validate()).NotTo(Succeed())
|
||||
})
|
||||
It("validates preview receipt states and output descriptions", func() {
|
||||
p := Preview{OperationID: "operation-1", Revision: Revision{1, 1}, ExpiresAt: envelope().Authority.ExpiresAt, Summary: json.RawMessage(`{}`)}
|
||||
Expect(p.Validate(time.Now(), DefaultLimits())).To(Succeed())
|
||||
p.OperationID = ""
|
||||
Expect(p.Validate(time.Now(), DefaultLimits())).NotTo(Succeed())
|
||||
for _, s := range []ReceiptState{ReceiptPending, ReceiptCommitted, ReceiptRejected, ReceiptUnknown} {
|
||||
r := Receipt{OperationID: "operation-1", State: s, Revision: Revision{1, 1}}
|
||||
Expect(r.Validate(DefaultLimits())).To(Succeed())
|
||||
}
|
||||
r := Receipt{OperationID: "operation-1", State: "success"}
|
||||
Expect(r.Validate(DefaultLimits())).NotTo(Succeed())
|
||||
for _, o := range []OutputDescription{{Handle: "../file", Name: "x", MediaType: "text/csv", Size: 1}, {Handle: "output", Name: "../x", MediaType: "text/csv", Size: 1}, {Handle: "output", Name: "x", MediaType: "bad", Size: 1}, {Handle: "output", Name: "x", MediaType: "text/csv", Size: -1}} {
|
||||
Expect((Result{Outputs: []OutputDescription{o}}).Validate(DefaultLimits())).NotTo(Succeed())
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,272 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package contract
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Tool kinds. The kind says which host authority an add-on tool runs under; it
|
||||
// is the add-on's declaration, and the host still checks the caller's role
|
||||
// for that authority on every call.
|
||||
const (
|
||||
// ToolKindRead runs under the read action and returns data.
|
||||
ToolKindRead = "read"
|
||||
// ToolKindWrite proposes a data change under the write action.
|
||||
ToolKindWrite = "write"
|
||||
// ToolKindSchema proposes a schema change under the schema action.
|
||||
ToolKindSchema = "schema"
|
||||
// ToolKindCreate proposes the creation of a resource. It names no
|
||||
// resource: the host reserves one for the proposer.
|
||||
ToolKindCreate = "create"
|
||||
)
|
||||
|
||||
// ResourceArgument is the tool argument that names the resource a tool works
|
||||
// on. The host takes it out of the arguments and binds it to the request, so
|
||||
// the add-on never reads it from the arguments.
|
||||
const ResourceArgument = "resourceId"
|
||||
|
||||
// Bounds of the tool definitions of one manifest.
|
||||
const (
|
||||
MaxToolDefinitions = 32
|
||||
MaxToolDescriptionLen = 400
|
||||
MaxToolSchemaBytes = 8 << 10
|
||||
MaxToolDefinitionBytes = 32 << 10
|
||||
maxToolSchemaDepth = 6
|
||||
maxToolProperties = 32
|
||||
maxToolEnum = 32
|
||||
)
|
||||
|
||||
// ToolDefinition describes one tool the add-on offers to a model. It is
|
||||
// served by the host from the verified manifest: the model sees this schema
|
||||
// and nothing the add-on process says at run time.
|
||||
type ToolDefinition struct {
|
||||
// Name is one of capabilities.tools.
|
||||
Name string `json:"name"`
|
||||
// Description is plain text for the model.
|
||||
Description string `json:"description"`
|
||||
// Kind is read, write, schema or create.
|
||||
Kind string `json:"kind"`
|
||||
// InputSchema is the JSON schema of the arguments object. It is an
|
||||
// object schema with explicit properties, additionalProperties false at
|
||||
// the top level and the keywords the host allows only.
|
||||
InputSchema json.RawMessage `json:"inputSchema" swaggertype:"object"`
|
||||
}
|
||||
|
||||
var toolPropertyName = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9]{0,63}$`)
|
||||
|
||||
// forbiddenToolProperties are argument names that stand for identity,
|
||||
// authority or approval. A schema never declares them, and the host refuses
|
||||
// them in a call, whatever the schema says.
|
||||
var forbiddenToolProperties = map[string]bool{
|
||||
"tool": true, "actor": true, "user": true, "userid": true, "credential": true, "credentialid": true,
|
||||
"role": true, "mode": true, "scope": true, "authority": true, "approved": true, "approve": true,
|
||||
"approval": true, "confirm": true, "confirmed": true, "confirmation": true, "digest": true,
|
||||
"proposal": true, "proposalid": true, "operation": true, "operationid": true, "generation": true,
|
||||
"addon": true, "addonid": true, "action": true, "owner": true, "grant": true, "share": true,
|
||||
"admin": true, "session": true, "token": true, "apikey": true, "force": true, "yes": true,
|
||||
}
|
||||
|
||||
// ForbiddenToolProperty reports whether name is an argument name that no
|
||||
// tool may declare or receive. The comparison ignores case and the
|
||||
// separators - and _.
|
||||
func ForbiddenToolProperty(name string) bool {
|
||||
k := strings.ToLower(strings.TrimSpace(name))
|
||||
k = strings.NewReplacer("-", "", "_", "", ".", "").Replace(k)
|
||||
return forbiddenToolProperties[k]
|
||||
}
|
||||
|
||||
var schemaKeywords = map[string]bool{
|
||||
"type": true, "properties": true, "required": true, "items": true, "enum": true, "description": true,
|
||||
"minimum": true, "maximum": true, "minLength": true, "maxLength": true, "minItems": true, "maxItems": true,
|
||||
"additionalProperties": true, "default": true,
|
||||
}
|
||||
|
||||
var schemaTypes = map[string]bool{"object": true, "string": true, "integer": true, "number": true, "boolean": true, "array": true}
|
||||
|
||||
// validateToolDefinitions checks the definitions against the declared tools.
|
||||
func validateToolDefinitions(defs []ToolDefinition, tools []string) error {
|
||||
if len(defs) == 0 {
|
||||
return nil
|
||||
}
|
||||
if len(defs) > MaxToolDefinitions {
|
||||
return fmt.Errorf("too many tool definitions")
|
||||
}
|
||||
declared := map[string]bool{}
|
||||
for _, t := range tools {
|
||||
declared[t] = true
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
total := 0
|
||||
for _, d := range defs {
|
||||
if !declared[d.Name] || seen[d.Name] {
|
||||
return fmt.Errorf("tool definition names an undeclared or repeated tool")
|
||||
}
|
||||
seen[d.Name] = true
|
||||
if !plainText(d.Description, MaxToolDescriptionLen) {
|
||||
return fmt.Errorf("invalid tool description")
|
||||
}
|
||||
switch d.Kind {
|
||||
case ToolKindRead, ToolKindWrite, ToolKindSchema, ToolKindCreate:
|
||||
default:
|
||||
return fmt.Errorf("invalid tool kind")
|
||||
}
|
||||
if len(d.InputSchema) == 0 || len(d.InputSchema) > MaxToolSchemaBytes {
|
||||
return fmt.Errorf("tool schema is missing or too large")
|
||||
}
|
||||
total += len(d.InputSchema) + len(d.Description)
|
||||
if total > MaxToolDefinitionBytes {
|
||||
return fmt.Errorf("tool definitions are too large")
|
||||
}
|
||||
if err := ValidateToolSchema(d.InputSchema, d.Kind); err != nil {
|
||||
return fmt.Errorf("tool %s: %w", d.Name, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ValidateToolSchema checks the arguments schema of one tool: a bounded
|
||||
// object schema built only from the allowed keywords, with no identity or
|
||||
// approval property. A tool that works on a resource names it with the
|
||||
// resourceId argument; a create tool does not, and a read tool may leave it
|
||||
// out (see IsResourceless).
|
||||
func ValidateToolSchema(raw json.RawMessage, kind string) error {
|
||||
if err := strictJSON(raw, new(map[string]json.RawMessage), MaxToolSchemaBytes); err != nil {
|
||||
return err
|
||||
}
|
||||
var root map[string]any
|
||||
d := json.NewDecoder(bytes.NewReader(raw))
|
||||
d.UseNumber()
|
||||
if err := d.Decode(&root); err != nil {
|
||||
return err
|
||||
}
|
||||
if root["type"] != "object" {
|
||||
return fmt.Errorf("the arguments schema must be an object")
|
||||
}
|
||||
if v, ok := root["additionalProperties"]; !ok || v != false {
|
||||
return fmt.Errorf("the arguments schema must set additionalProperties to false")
|
||||
}
|
||||
if err := walkSchema(root, 0); err != nil {
|
||||
return err
|
||||
}
|
||||
props, _ := root["properties"].(map[string]any)
|
||||
_, has := props[ResourceArgument]
|
||||
// A create tool names no resource: the host reserves one. A read tool may
|
||||
// name none either, when it answers about the caller and not about a
|
||||
// resource; the host then scopes the result to the calling user. Every
|
||||
// other tool works on a resource and names it.
|
||||
switch {
|
||||
case kind == ToolKindCreate && has:
|
||||
return fmt.Errorf("%s is absent for a create tool", ResourceArgument)
|
||||
case kind != ToolKindCreate && kind != ToolKindRead && !has:
|
||||
return fmt.Errorf("%s is required for a tool of a resource", ResourceArgument)
|
||||
}
|
||||
req, _ := root["required"].([]any)
|
||||
required := map[string]bool{}
|
||||
for _, r := range req {
|
||||
s, ok := r.(string)
|
||||
if !ok {
|
||||
return fmt.Errorf("invalid required list")
|
||||
}
|
||||
if _, ok := props[s]; !ok {
|
||||
return fmt.Errorf("required names an undeclared property")
|
||||
}
|
||||
required[s] = true
|
||||
}
|
||||
if has && !required[ResourceArgument] {
|
||||
return fmt.Errorf("%s must be required", ResourceArgument)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func walkSchema(node map[string]any, depth int) error {
|
||||
if depth > maxToolSchemaDepth {
|
||||
return fmt.Errorf("schema nesting is too deep")
|
||||
}
|
||||
for k, v := range node {
|
||||
if !schemaKeywords[k] {
|
||||
return fmt.Errorf("schema keyword %q is not allowed", k)
|
||||
}
|
||||
switch k {
|
||||
case "type":
|
||||
s, ok := v.(string)
|
||||
if !ok || !schemaTypes[s] {
|
||||
return fmt.Errorf("invalid schema type")
|
||||
}
|
||||
case "description":
|
||||
s, ok := v.(string)
|
||||
if !ok || !plainText(s, MaxToolDescriptionLen) {
|
||||
return fmt.Errorf("invalid schema description")
|
||||
}
|
||||
case "enum":
|
||||
list, ok := v.([]any)
|
||||
if !ok || len(list) == 0 || len(list) > maxToolEnum {
|
||||
return fmt.Errorf("invalid enum")
|
||||
}
|
||||
for _, e := range list {
|
||||
if _, ok := e.(string); !ok {
|
||||
return fmt.Errorf("enum values must be strings")
|
||||
}
|
||||
}
|
||||
case "additionalProperties":
|
||||
if _, ok := v.(bool); !ok {
|
||||
return fmt.Errorf("additionalProperties must be a boolean")
|
||||
}
|
||||
case "required":
|
||||
if _, ok := v.([]any); !ok {
|
||||
return fmt.Errorf("invalid required list")
|
||||
}
|
||||
case "properties":
|
||||
props, ok := v.(map[string]any)
|
||||
if !ok || len(props) > maxToolProperties {
|
||||
return fmt.Errorf("invalid properties")
|
||||
}
|
||||
for name, p := range props {
|
||||
if !toolPropertyName.MatchString(name) {
|
||||
return fmt.Errorf("invalid property name")
|
||||
}
|
||||
// Identity and approval names are refused at every depth: a
|
||||
// nested object is still read as arguments by the add-on.
|
||||
if ForbiddenToolProperty(name) {
|
||||
return fmt.Errorf("property %q is reserved", name)
|
||||
}
|
||||
child, ok := p.(map[string]any)
|
||||
if !ok {
|
||||
return fmt.Errorf("invalid property schema")
|
||||
}
|
||||
if err := walkSchema(child, depth+1); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
case "items":
|
||||
child, ok := v.(map[string]any)
|
||||
if !ok {
|
||||
return fmt.Errorf("invalid items schema")
|
||||
}
|
||||
if err := walkSchema(child, depth+1); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// IsResourceless reports a tool definition that names no resource: a read tool
|
||||
// whose schema has no resourceId property. The host runs such a read without a
|
||||
// resource and the add-on answers for the calling user only.
|
||||
func IsResourceless(d ToolDefinition) bool {
|
||||
if d.Kind != ToolKindRead {
|
||||
return false
|
||||
}
|
||||
var root struct {
|
||||
Properties map[string]json.RawMessage `json:"properties"`
|
||||
}
|
||||
if json.Unmarshal(d.InputSchema, &root) != nil {
|
||||
return false
|
||||
}
|
||||
_, has := root.Properties[ResourceArgument]
|
||||
return !has
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package contract
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = Describe("Tool schemas and the resource argument", func() {
|
||||
const withoutResource = `{"type":"object","properties":{"limit":{"type":"integer"}},"additionalProperties":false}`
|
||||
const withResource = `{"type":"object","properties":{"resourceId":{"type":"string"}},"required":["resourceId"],"additionalProperties":false}`
|
||||
const resourceNotRequired = `{"type":"object","properties":{"resourceId":{"type":"string"}},"additionalProperties":false}`
|
||||
|
||||
It("accepts a read tool that names no resource", func() {
|
||||
Expect(ValidateToolSchema(json.RawMessage(withoutResource), ToolKindRead)).To(Succeed())
|
||||
})
|
||||
It("still accepts a read tool that requires its resource", func() {
|
||||
Expect(ValidateToolSchema(json.RawMessage(withResource), ToolKindRead)).To(Succeed())
|
||||
})
|
||||
It("refuses a read tool that offers the resource without requiring it", func() {
|
||||
Expect(ValidateToolSchema(json.RawMessage(resourceNotRequired), ToolKindRead)).NotTo(Succeed())
|
||||
})
|
||||
It("refuses a write or schema tool without a resource", func() {
|
||||
Expect(ValidateToolSchema(json.RawMessage(withoutResource), ToolKindWrite)).NotTo(Succeed())
|
||||
Expect(ValidateToolSchema(json.RawMessage(withoutResource), ToolKindSchema)).NotTo(Succeed())
|
||||
})
|
||||
It("still accepts a write or schema tool that requires its resource", func() {
|
||||
Expect(ValidateToolSchema(json.RawMessage(withResource), ToolKindWrite)).To(Succeed())
|
||||
Expect(ValidateToolSchema(json.RawMessage(withResource), ToolKindSchema)).To(Succeed())
|
||||
})
|
||||
It("keeps a create tool free of the resource argument", func() {
|
||||
Expect(ValidateToolSchema(json.RawMessage(withoutResource), ToolKindCreate)).To(Succeed())
|
||||
Expect(ValidateToolSchema(json.RawMessage(withResource), ToolKindCreate)).NotTo(Succeed())
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,209 @@
|
||||
# Add-on proposal flow model
|
||||
|
||||
A pure reducer that models the execution flow of one add-on proposal: Propose,
|
||||
Confirm, Cancel, execute, outcome-unknown handling, receipt lookup, the add-on
|
||||
hold and generation changes. It is the spec of that flow. It is a model only:
|
||||
no implementation code uses it yet, and no implementation task below is done.
|
||||
|
||||
Code locations are for branch `feat/addons-tables` at `4721e12d5`.
|
||||
|
||||
## Files
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `reduce.go` | State, event and effect types, the rule table, and `Reduce`. The spec. |
|
||||
| `traces.go` | Scenario traces as data: events, expected rule ids, expected effects, expected final state. |
|
||||
| `explore.go` | Breadth-first search over a finite event universe, the safety predicates and the liveness check. |
|
||||
| `*_test.go` | Trace tests, the invariant test, coverage tests (every state/event pair, every rule fires). |
|
||||
| `cmd/viewer`, `gen.go` | Generator of `viewer.html`. |
|
||||
| `viewer.html` | Generated step-through viewer. Do not edit it by hand. |
|
||||
|
||||
Run the model: `go test -race ./pkg/addons/model/proposalflow`.
|
||||
Regenerate the viewer after any change to `reduce.go` or `traces.go`:
|
||||
`go generate ./pkg/addons/model/proposalflow`.
|
||||
|
||||
The viewer is generated by a small Go command that runs the real reducer and
|
||||
embeds the result as JSON in an HTML template. The `modeling-flows-as-reducers`
|
||||
skill suggests esbuild, which bundles a JavaScript reducer. This reducer is Go,
|
||||
so there is nothing to bundle; the viewer replays the Go reducer at generation
|
||||
time instead of copying its logic.
|
||||
|
||||
## States
|
||||
|
||||
One constructor per situation. The model is finer than the stored state: some
|
||||
constructors share one durable `gateway.ProposalState` (the `Durable()` method).
|
||||
|
||||
| State | Durable | Why it exists |
|
||||
|---|---|---|
|
||||
| `Unproposed` | none | Start. A Propose that stores nothing (held, blocked, no expiry) stays here and is retryable. |
|
||||
| `Refused` | none | Propose refused: undeclared tool, read tool, role too low for the derived action, stale generation. |
|
||||
| `Pending` | pending | Stored, waiting for a human decision. |
|
||||
| `Admitting` | pending | Confirm accepted, waiting for the admission decision. Volatile. Needed so a second Confirm and a Cancel can race it. |
|
||||
| `ExecPrepared` | executing | Executing is durable, nothing was sent. A not-sent answer returns to `Pending` (finding a). |
|
||||
| `ExecSent` | executing | The request may have reached the add-on. Never goes back. |
|
||||
| `OutcomeUnknown` | outcome-unknown | Sent, no evidence, automatic lookups left (counter `Lookups`). Blocks writes. |
|
||||
| `UnknownManual` | outcome-unknown | Automatic lookups ran out. Blocks writes. Only a lookup answer or the owner leaves it (finding c). |
|
||||
| `Abandoned` | abandoned-unknown | Owner stopped waiting. No longer blocks. A late receipt still settles it. |
|
||||
| `Applied` | applied | Committed receipt. Terminal. |
|
||||
| `Rejected` | rejected | The add-on definitely did not apply (rejected receipt, refusal, conflict). Terminal. |
|
||||
| `NotApplied` | rejected | A lookup proved the operation never commits. Terminal. New (see findings). |
|
||||
| `Cancelled` | cancelled | Never ran: proposer, hold or lowered authority. Terminal. |
|
||||
| `Expired` | expired | Reached expiry unconfirmed. Terminal. |
|
||||
| `Stale` | rejected | Generation moved. Terminal. |
|
||||
| `Dropped` | none | Abandoned outcome removed after retention. Terminal. |
|
||||
|
||||
The action is not stored: `Prop.Action()` derives it from `Prop.Kind` (the tool
|
||||
kind), so a proposal cannot hold an action that disagrees with its tool.
|
||||
|
||||
## Events
|
||||
|
||||
`Propose`, `Confirm`, `Cancel`, `Tick`, `AdmitGranted`, `AdmitRefused`,
|
||||
`SendStarted`, `NotSent`, `ApplyResult` (committed, rejected, refused,
|
||||
conflict, unsettled, transport error, timeout), `LookupDue`, `ReceiptResult`
|
||||
(committed, rejected, not-committed, unsettled), `OwnerResolve`, `HoldAcquired`,
|
||||
`HoldReleased`, `GenerationChanged`, `AuthorityLowered`, `Restart`,
|
||||
`RetentionElapsed`.
|
||||
|
||||
Time enters as numbers in `Now` fields. I/O results enter as events. Effects
|
||||
are `Reply`, `Persist`, `Admit`, `Dispatch`, `CancelDispatch`, `CancelSend`,
|
||||
`LookupReceipt` and `Alarm`.
|
||||
|
||||
Two events are new contracts the shell must keep:
|
||||
|
||||
- `SendStarted`: the runtime client says the request is on the wire. Before it,
|
||||
a failure is `NotSent` and retryable. After it, a failure is unknown.
|
||||
- `CancelDispatch` then `NotSent` or `SendStarted`: a hold or a lowered
|
||||
authority never claims a cancellation it did not achieve.
|
||||
|
||||
## Invariants (checked by `TestInvariants`)
|
||||
|
||||
Safety, on every reachable state and step:
|
||||
|
||||
- S1 a closed proposal never changes, so it is never Applied and Cancelled.
|
||||
- S2 Applied only on a committed apply or receipt answer. A dispatch is never success.
|
||||
- S3 Admitting only from Pending, by the proposer, a human surface, the stored digest, before expiry, with a live role that covers the derived action.
|
||||
- S4 a schema action is never admitted for a non-owner.
|
||||
- S5 execution starts only from an admission for the proposal's own generation and before expiry.
|
||||
- S6 at most one dispatch, only on the admission step, always after `persist:executing`.
|
||||
- S7 once sent, never back to a state that can dispatch. `ExecPrepared` returns to `Pending` only on a retryable not-sent answer.
|
||||
- S8 an unknown outcome comes only from a send that may have happened, or from a restart. An admission refusal or a not-sent answer never makes one (finding a).
|
||||
- S9 a blocking situation is entered only by admission.
|
||||
- S10 Propose stores a proposal only with a role that covers the action derived from the tool kind, whatever action the caller claimed (finding b).
|
||||
- State predicates: a schema proposal is held only for an owner, every proposal has a mutating action, the lookup counter is bounded, an abandoned outcome does not block.
|
||||
|
||||
Liveness: from every reachable state a terminal state is reachable (reverse
|
||||
reachability over the search graph). `UnknownManual` has an exit.
|
||||
|
||||
## Findings the model makes checkable
|
||||
|
||||
Status: (a), (b) and (d) are fixed in code. (e) is partly fixed: a hold that cancels the dispatch before the first request byte now ends the proposal as cancelled (R35, R52). The eager generation checks (R57, R58) are not done.
|
||||
|
||||
| Finding | Model | Code today |
|
||||
|---|---|---|
|
||||
| (a) refusal before send must not be unknown | `ExecPrepared` + `NotSent{slots, invalid, withdrawn}` -> `Pending` (R34); `AdmitRefused` -> `Pending` (R29); S8. | `Broker.execute` maps every error that is not `ErrRefused` to `OutcomeUnknown` (`core/services/addons/policy/proposals.go:1329`). `Client.admit` (no slot, withdrawn), `Client.valid` and the pre-wire failures of `Client.send` all return the same `ErrRuntime` as a lost reply (`core/services/addons/runtime/client.go:63`, `:73`, `:82`, `:86`). The store has no `executing` -> `pending` transition. |
|
||||
| (b) action derived from tool kind | `Prop.Action()` from `Kind`; R4; S4, S10; `Claimed` never read. | `Broker.Propose` takes the action from `request.Action` (`proposals.go:695`). `gateway.AuthorizeTool` checks only that a named tool is declared, not its kind, and returns nil when the arguments name no tool (`core/services/addons/gateway/tools.go:74`). The record stores `Action` as a string from the caller (`proposals.go:766`). |
|
||||
| (c) unknown outcome must be resolvable | R43, R44, R45, R48, R61; `NotCommitted` answer -> `NotApplied` (R42); S8; liveness. | Lookups stop after a time window (`recovery.go:157`, `:168`), then only `Resolve` leaves. The contract has no "never commits" receipt answer (`pkg/addons/contract/protocol.go:62`), so a lookup cannot prove not-committed. A generation change makes every lookup fail until the window ends (`recovery.go:55`). |
|
||||
| (d) found while modelling | `ApplyResult{conflict}` -> `Rejected` (R38). | A 409 becomes `runtime.ErrConflict`, which wraps `ErrRuntime`, not `ErrRefused` (`runtime/client.go:99`, `supervisor.go:33`), so `execute` records an unknown outcome for a definite answer. |
|
||||
| (e) found while modelling | Hold on `ExecPrepared` and `ExecSent` (R52, R53); generation change (R57, R58). | `execute` checks `ctx.Err()` once before the call (`proposals.go:1305`). A hold between that check and the wire cancels the call and the proposal becomes unknown, though the send may not have happened. Generation is checked only at confirm (`proposals.go:1281`); a change does not make a pending proposal Stale eagerly. Today `RecoveryPort.Hold` cancels pending proposals (`addon_state.go:380`, `broker_manage.go:15`). |
|
||||
|
||||
## Rule map
|
||||
|
||||
The last column is the model trace that holds the rule today. When the
|
||||
implementation calls the model, the verifying test is a conformance test in
|
||||
`core/services/addons/policy` named `TestBrokerFollowsModel/<trace>`, which
|
||||
replays the trace against a real broker with fakes for the clock, store and
|
||||
runtime. Those tests do not exist yet. "Gap" means the code does not do what
|
||||
the rule says.
|
||||
|
||||
| Rule | Meaning (short) | Implementation location | Status | Model trace |
|
||||
|---|---|---|---|---|
|
||||
| R1 | Propose while held: nothing stored | `Policy.Admit` -> `errHeld`, `admission.go:198` | matches | propose-while-held-then-retry |
|
||||
| R2 | undeclared tool refused | `Broker.authorizeTool`, then `toolKind` | partial: only when the arguments name a tool | propose-undeclared-tool |
|
||||
| R3 | read tool is not a proposal | `Broker.proposalAction`: a read, create or undefined tool is refused | matches for a named tool; arguments without a tool keep the claimed action | propose-read-tool |
|
||||
| R4 | role below derived action refused | `Broker.Propose`: `Admit` with the action from `proposalAction`; a disagreeing claim is refused | matches | editor-schema-propose-refused |
|
||||
| R5 | stale generation at Propose | `proposals.go:738` | matches | propose-stale-generation |
|
||||
| R6 | Propose blocked by unresolved outcome | `proposals.go:731` | matches | propose-while-held-then-retry |
|
||||
| R7 | no usable expiry | `proposals.go:763` | matches | propose-while-held-then-retry |
|
||||
| R8 | stored as Pending | `Broker.Propose`, `insert`, with the derived action | matches | happy-path |
|
||||
| R9 | repeat returns stored proposal | `proposals.go:723`, `sameRequest` | matches | duplicate-confirm-after-applied |
|
||||
| R10 | Confirm by non-proposer | `Broker.own`, `proposals.go:1061` | matches | two-actors-racing-confirm |
|
||||
| R11 | non-human credential | `confirmCredential`, `proposals.go:1188` | matches | confirm-guards |
|
||||
| R12 | digest mismatch | `proposals.go:1197` | matches | confirm-guards |
|
||||
| R13 | Confirm after expiry | `expire`, `proposals.go:1218` | matches | confirm-guards |
|
||||
| R14 | live role below derived action | `Policy.Admit` -> `check`, with the stored (derived) action | matches | editor-confirm-of-schema-after-demotion |
|
||||
| R15 | Confirm accepted -> Admitting | `Confirm` calling `Admit`, `proposals.go:1237` | matches | happy-path |
|
||||
| R16 | Confirm while in admission | writer slot, `admission.go:220`; second call then finds state not pending, `proposals.go:1260` | matches (answer is `ErrProposalClosed`, model says busy) | two-actors-racing-confirm |
|
||||
| R17 | repeat Confirm of unresolved: lookup | `resolveForConfirm`, `proposals.go:1247` | matches | duplicate-confirm-while-unknown |
|
||||
| R18 | Confirm of Applied | `proposals.go:1201` | matches | duplicate-confirm-after-applied |
|
||||
| R19 | Confirm of Rejected, NotApplied, Stale | `proposals.go:1201` | matches (Stale and NotApplied are `rejected` in the store) | generation-change-while-pending |
|
||||
| R20 | Confirm of Cancelled or Dropped | `proposals.go:1216` | matches | cancel-races-confirm |
|
||||
| R21 | Confirm of Expired | `proposals.go:1212` | matches | expiry |
|
||||
| R22 | request for an unstored proposal | `Broker.own`, `proposals.go:1063` | matches | editor-schema-propose-refused |
|
||||
| R23 | Cancel by non-proposer | `proposals.go:1153` | matches | two-actors-racing-confirm |
|
||||
| R24 | Cancel Pending | `proposals.go:1160` | matches | cancel-pending |
|
||||
| R25 | Cancel during admission | `proposals.go:1160` then `execute` finds it closed, `proposals.go:1260` | matches | cancel-races-confirm |
|
||||
| R26 | Cancel of non-pending | `proposals.go:1160` -> `ErrProposalClosed` | matches | duplicate-confirm-after-applied |
|
||||
| R27 | Cancel of Cancelled | `proposals.go:1157` | matches | cancel-pending |
|
||||
| R28 | Tick past expiry | `Broker.sweep`, `proposals.go:900` | matches | expiry |
|
||||
| R29 | admission refused -> Pending | `Admit` errors leave the record pending; a refused runtime slot returns it through R34 | matches | retry-after-admission-refused |
|
||||
| R30 | admission after expiry | `execute` -> `expire`, `proposals.go:1271` | matches | expiry-seen-at-admission |
|
||||
| R31 | admission for another generation | `execute`, `proposals.go:1281` (stored as `rejected`) | matches | generation-seen-at-admission |
|
||||
| R32 | persist executing, then dispatch | `execute`, `proposals.go:1291` | matches | happy-path |
|
||||
| R33 | SendStarted | `runtime.Client.send` (`WroteHeaders`): a failure after it is not `ErrNotSent` | matches | happy-path |
|
||||
| R34 | NotSent retryable -> Pending | `execute`, `runtime.ErrNotSent` branch; `TestBrokerFollowsModel` | matches | retry-after-not-sent |
|
||||
| R35 | NotSent by hold or authority -> Cancelled | `execute`: context checked before the call, and a not-sent answer with a done context | partial (e): a hold that lands after the client began the send is still unknown | hold-during-prepared-dispatch |
|
||||
| R36 | NotSent superseded -> Stale | none | gap (e) | generation-change-while-prepared |
|
||||
| R37 | apply committed -> Applied | `execute`, `proposals.go:1335` | matches | happy-path |
|
||||
| R38 | apply rejected, refused, conflict -> Rejected | `execute`: `ErrRefused`, `ErrConflict` (`ErrProposalConflict`), rejected receipt | matches | addon-conflict-is-rejected |
|
||||
| R39 | no evidence after send -> OutcomeUnknown | `execute`, last error branch; a not-sent error never reaches it | matches | timeout-unknown-then-receipt-committed |
|
||||
| R40 | receipt committed -> Applied | `Reconcile`, `recovery.go:72` | matches | timeout-unknown-then-receipt-committed |
|
||||
| R41 | receipt rejected -> Rejected | `Reconcile`, `recovery.go:74` | matches | authority-lowered-after-send |
|
||||
| R42 | receipt not-committed -> NotApplied | none | gap (c): no such answer in the contract | hold-during-sent-request |
|
||||
| R43 | lookup unsettled, lookups left | `RetryUnresolved`, `recovery.go:188` (time window, not count) | matches in effect | timeout-unknown-then-receipt-committed |
|
||||
| R44 | lookups out -> UnknownManual | window end, `lookupWindow`, `recovery.go:168` (no state change) | gap (c): no distinct state | unknown-bounded-then-manual-resolution |
|
||||
| R45 | lookup timer | `RetryUnresolved`, `recovery.go:188` | matches | timeout-unknown-then-receipt-committed |
|
||||
| R46 | Resolve denied | `Resolve`, `recovery.go:260` | matches | unknown-bounded-then-manual-resolution |
|
||||
| R47 | Resolve while held | `Admit` -> `errHeld`, `recovery.go:284` | matches | unknown-bounded-then-manual-resolution |
|
||||
| R48 | Resolve -> Abandoned | `recovery.go:310` | matches | unknown-bounded-then-manual-resolution |
|
||||
| R49 | Resolve of a non-unresolved proposal | `recovery.go:320` | matches | cancel-pending |
|
||||
| R50 | Resolve of a settled proposal | `recovery.go:294` | matches | authority-lowered-after-send |
|
||||
| R51 | Hold on Pending | `RecoveryPort.Hold` -> `InvalidateResource`, `addon_state.go:380` | matches | hold-on-pending |
|
||||
| R52 | Hold on prepared dispatch | `HoldAddon` cancels the context, `addon_state.go:48` | partial (e) | hold-during-prepared-dispatch |
|
||||
| R53 | Hold on sent request | same | matches in effect | hold-during-sent-request |
|
||||
| R54 | authority lowered, Pending | `InvalidateProposals`, `proposals.go:551` | matches | authority-lowered-while-pending |
|
||||
| R55 | authority lowered, prepared | `change` revocation cancels admitted work, `admission.go:275` | partial (e) | authority-lowered-before-send |
|
||||
| R56 | authority lowered, sent | same | matches in effect | authority-lowered-after-send |
|
||||
| R57 | generation change, Pending | none eager; the check is at confirm | gap (e) | generation-change-while-pending |
|
||||
| R58 | generation change, prepared | none | gap (e) | generation-change-while-prepared |
|
||||
| R59 | restart in admission | admission is volatile | matches | restart-recovery |
|
||||
| R60 | restart with executing -> unknown | `Recover`, `recovery.go:126` | matches | restart-recovery |
|
||||
| R61 | retention drops Abandoned | `sweep`, `proposals.go:909` | matches | unknown-bounded-then-manual-resolution |
|
||||
| I1-I10 | ignored pairs, each with a reason in `Rules` | not applicable | n/a | I3, I4, I5, I7, I8 in traces; all by the search |
|
||||
| V1 | SendStarted outside a prepared dispatch | shell contract | new | search only |
|
||||
|
||||
## Not represented
|
||||
|
||||
These paths exist in the code and are outside the model. Each is a candidate
|
||||
for a later extension, not a failure of this one.
|
||||
|
||||
- Create proposals: the reservation, `admitCreate`, `settleCreates`,
|
||||
`resolveCreate` and `BackingExists` (`proposals.go:788`, `recovery.go:383`).
|
||||
- Restore from a backup (`afterRestore`, `addon_state.go:311`): pending ->
|
||||
cancelled and executing -> unknown. It needs an `AddonRestored` event.
|
||||
- Revision staleness: a preview refresh after a revision change. Only the
|
||||
add-on's conflict answer is modelled (`ApplyResult{conflict}`).
|
||||
- Quotas, retention pressure and `makeRoom` (`proposals.go:982`).
|
||||
- Cross-proposal blocking is an input (`Propose.Blocked`, `AdmitRefused{blocked}`),
|
||||
not a model of two proposals.
|
||||
- The gateway's `byState` rewrite of a 503 into an unknown outcome
|
||||
(`proposals.go:1370`). It reads the stored state, so with finding (a) it turns
|
||||
a not-sent failure of an executing proposal into "unknown" for the caller.
|
||||
- `Resolve` of an executing proposal after a crash and before `Recover`: the
|
||||
code allows it when no call is live; the model answers closed.
|
||||
- Per-user and per-resource scope. The model has one proposal, one proposer and
|
||||
one other user.
|
||||
|
||||
## Search result
|
||||
|
||||
62 reachable states, 4154 steps over the event universe, 37 terminal states, 0
|
||||
dead ends, 0 violations. Every rule id fires in the traces or the search.
|
||||
@@ -0,0 +1,117 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
// Command viewer writes a self-contained HTML page that steps through the
|
||||
// traces of the proposal-flow model. It runs the real reducer: the page
|
||||
// embeds the JSON this program computes, not a copy of the logic.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
_ "embed"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
pf "github.com/mudler/LocalAI/pkg/addons/model/proposalflow"
|
||||
)
|
||||
|
||||
//go:embed viewer.tmpl.html
|
||||
var page string
|
||||
|
||||
type stateJSON struct {
|
||||
Name string `json:"name"`
|
||||
Detail string `json:"detail"`
|
||||
Durable string `json:"durable"`
|
||||
}
|
||||
|
||||
type stepJSON struct {
|
||||
Event string `json:"event"`
|
||||
Type string `json:"type"`
|
||||
Rule string `json:"rule"`
|
||||
RuleKind string `json:"ruleKind"`
|
||||
RuleText string `json:"ruleText"`
|
||||
Before stateJSON `json:"before"`
|
||||
After stateJSON `json:"after"`
|
||||
Effects []string `json:"effects"`
|
||||
}
|
||||
|
||||
type traceJSON struct {
|
||||
Name string `json:"name"`
|
||||
About string `json:"about"`
|
||||
Final stateJSON `json:"final"`
|
||||
OK bool `json:"ok"`
|
||||
Bad []string `json:"bad"`
|
||||
Steps []stepJSON `json:"steps"`
|
||||
}
|
||||
|
||||
type ruleJSON struct {
|
||||
ID string `json:"id"`
|
||||
Kind string `json:"kind"`
|
||||
Text string `json:"text"`
|
||||
}
|
||||
|
||||
type exploreJSON struct {
|
||||
States int `json:"states"`
|
||||
Edges int `json:"edges"`
|
||||
Terminals int `json:"terminals"`
|
||||
DeadEnds []string `json:"deadEnds"`
|
||||
Violations []string `json:"violations"`
|
||||
ByName map[string]int `json:"byName"`
|
||||
}
|
||||
|
||||
type dataJSON struct {
|
||||
Traces []traceJSON `json:"traces"`
|
||||
Rules []ruleJSON `json:"rules"`
|
||||
Explore exploreJSON `json:"explore"`
|
||||
}
|
||||
|
||||
// nonNil keeps empty lists as [] in the JSON, not null.
|
||||
func nonNil(l []string) []string {
|
||||
if l == nil {
|
||||
return []string{}
|
||||
}
|
||||
return l
|
||||
}
|
||||
|
||||
func st(s pf.State) stateJSON {
|
||||
d := strings.TrimPrefix(fmt.Sprintf("%+v", s), "{")
|
||||
d = strings.TrimSuffix(d, "}")
|
||||
return stateJSON{Name: s.Name(), Detail: d, Durable: s.Durable()}
|
||||
}
|
||||
|
||||
func main() {
|
||||
out := flag.String("o", "viewer.html", "output file")
|
||||
flag.Parse()
|
||||
var data dataJSON
|
||||
for _, r := range pf.Rules {
|
||||
data.Rules = append(data.Rules, ruleJSON{ID: r.ID, Kind: string(r.Kind), Text: r.Text})
|
||||
}
|
||||
for _, tr := range pf.Traces() {
|
||||
bad := tr.Verify()
|
||||
t := traceJSON{Name: tr.Name, About: tr.About, Final: st(tr.Final), OK: len(bad) == 0, Bad: nonNil(bad)}
|
||||
for _, r := range pf.Run(pf.Unproposed{}, tr.Events()) {
|
||||
rule, _ := pf.RuleByID(r.Rule)
|
||||
t.Steps = append(t.Steps, stepJSON{
|
||||
Event: pf.Describe(r.Event), Type: r.Event.Name(), Rule: r.Rule, RuleKind: string(rule.Kind), RuleText: rule.Text,
|
||||
Before: st(r.Before), After: st(r.After), Effects: pf.EffectNames(r.Effects),
|
||||
})
|
||||
}
|
||||
data.Traces = append(data.Traces, t)
|
||||
}
|
||||
rep := pf.Explore()
|
||||
data.Explore = exploreJSON{States: rep.States, Edges: rep.Edges, Terminals: len(rep.Terminals), DeadEnds: nonNil(rep.DeadEnds), Violations: nonNil(rep.Violations), ByName: rep.ByName}
|
||||
var buf bytes.Buffer
|
||||
enc := json.NewEncoder(&buf)
|
||||
enc.SetEscapeHTML(true)
|
||||
if err := enc.Encode(data); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
html := strings.Replace(page, "/*__DATA__*/null", strings.TrimSpace(buf.String()), 1)
|
||||
if err := os.WriteFile(*out, []byte(html), 0o600); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Add-on proposal flow model</title>
|
||||
<style>
|
||||
:root{--pf-bg:#fbfaf7;--pf-fg:#1d1c1a;--pf-mute:#6b6862;--pf-line:#dcd8cf;--pf-card:#fff;--pf-acc:#1f5f8b;--pf-ok:#2b7a3d;--pf-bad:#b3261e;--pf-warn:#8a5a00;--pf-hi:#e8f1f8}
|
||||
@media (prefers-color-scheme:dark){:root:not([data-theme="light"]){--pf-bg:#161614;--pf-fg:#ecebe6;--pf-mute:#a09d95;--pf-line:#34332f;--pf-card:#1e1e1b;--pf-acc:#7cb7e0;--pf-ok:#7fd08f;--pf-bad:#f2958e;--pf-warn:#e0b25a;--pf-hi:#22313c}}
|
||||
body{margin:0;background:var(--pf-bg);color:var(--pf-fg);font:15px/1.5 system-ui,sans-serif}
|
||||
.pf-wrap{max-width:1100px;margin:0 auto;padding:16px}
|
||||
h1{font-size:20px;margin:4px 0 2px}
|
||||
.pf-sub{color:var(--pf-mute);margin:0 0 12px}
|
||||
.pf-grid{display:grid;grid-template-columns:260px 1fr;gap:16px}
|
||||
@media (max-width:760px){.pf-grid{grid-template-columns:1fr}}
|
||||
.pf-list{list-style:none;margin:0;padding:0;border:1px solid var(--pf-line);border-radius:8px;background:var(--pf-card);max-height:70vh;overflow:auto}
|
||||
.pf-list button{display:block;width:100%;text-align:left;border:0;border-bottom:1px solid var(--pf-line);background:none;color:inherit;padding:8px 10px;font:inherit;cursor:pointer}
|
||||
.pf-list button[aria-current="true"]{background:var(--pf-hi);font-weight:600}
|
||||
.pf-list li:last-child button{border-bottom:0}
|
||||
.pf-card{border:1px solid var(--pf-line);border-radius:8px;background:var(--pf-card);padding:12px;margin-bottom:12px}
|
||||
.pf-ctl{display:flex;gap:8px;align-items:center;flex-wrap:wrap;margin:8px 0}
|
||||
.pf-ctl button{font:inherit;padding:4px 12px;border:1px solid var(--pf-line);border-radius:6px;background:var(--pf-card);color:inherit;cursor:pointer}
|
||||
.pf-ctl button:disabled{opacity:.4;cursor:default}
|
||||
table{border-collapse:collapse;width:100%;font-size:13px}
|
||||
th,td{border-bottom:1px solid var(--pf-line);padding:5px 6px;text-align:left;vertical-align:top}
|
||||
tr.pf-cur{background:var(--pf-hi)}
|
||||
tr.pf-row{cursor:pointer}
|
||||
code,.pf-mono{font:12px/1.4 ui-monospace,monospace;overflow-wrap:anywhere}
|
||||
td.pf-fx{white-space:nowrap}
|
||||
.pf-rule{font-weight:700;color:var(--pf-acc)}
|
||||
.pf-rule[data-kind="ignored"]{color:var(--pf-mute)}
|
||||
.pf-rule[data-kind="violation"]{color:var(--pf-bad)}
|
||||
.pf-state{font-weight:600}
|
||||
.pf-why{border-left:3px solid var(--pf-acc);padding:4px 10px;margin:8px 0;background:var(--pf-hi)}
|
||||
.pf-pass{color:var(--pf-ok)}.pf-fail{color:var(--pf-bad)}
|
||||
.pf-tag{display:inline-block;border:1px solid var(--pf-line);border-radius:10px;padding:0 7px;margin:0 4px 2px 0;font-size:12px}
|
||||
details{margin-top:8px}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="pf-wrap">
|
||||
<h1>Add-on proposal flow model</h1>
|
||||
<p class="pf-sub">Generated by running the real reducer (<code>reduce.go</code>) over the traces. Regenerate with <code>go generate ./pkg/addons/model/proposalflow</code>.</p>
|
||||
<div class="pf-card" id="pf-explore"></div>
|
||||
<div class="pf-grid">
|
||||
<ul class="pf-list" id="pf-traces"></ul>
|
||||
<div>
|
||||
<div class="pf-card">
|
||||
<h2 id="pf-title" style="font-size:17px;margin:0"></h2>
|
||||
<p id="pf-about" class="pf-sub" style="margin:4px 0"></p>
|
||||
<div class="pf-ctl"><button id="pf-prev">Previous</button><button id="pf-next">Next</button><button id="pf-all">Show all</button><span id="pf-pos" class="pf-sub"></span></div>
|
||||
<div id="pf-why" class="pf-why"></div>
|
||||
<table><thead><tr><th>#</th><th>Event</th><th>Rule</th><th>Before</th><th>After</th><th>Effects</th></tr></thead><tbody id="pf-steps"></tbody></table>
|
||||
<p id="pf-final" class="pf-sub"></p>
|
||||
</div>
|
||||
<details class="pf-card"><summary>Rule table</summary><table id="pf-rules"></table></details>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<script>
|
||||
const DATA = /*__DATA__*/null;
|
||||
const $ = id => document.getElementById(id);
|
||||
let cur = 0, pos = 0, showAll = false;
|
||||
function esc(s){return String(s).replace(/[&<>"]/g,c=>({'&':'&','<':'<','>':'>','"':'"'}[c]));}
|
||||
function stateHtml(s){return '<span class="pf-state">'+esc(s.name)+'</span><div class="pf-mono">'+esc(s.detail)+'</div>';}
|
||||
function renderExplore(){
|
||||
const e = DATA.explore, bad = e.violations.length + e.deadEnds.length;
|
||||
const names = Object.keys(e.byName).sort().map(n=>'<span class="pf-tag">'+esc(n)+' '+e.byName[n]+'</span>').join('');
|
||||
$('pf-explore').innerHTML = '<b>Invariant search</b>: '+e.states+' states, '+e.edges+' steps, '+e.terminals+' terminal states, '+
|
||||
e.deadEnds.length+' dead ends, '+e.violations.length+' violations. <span class="'+(bad?'pf-fail':'pf-pass')+'">'+(bad?'FAILED':'passed')+'</span><div>'+names+'</div>';
|
||||
}
|
||||
function renderList(){
|
||||
$('pf-traces').innerHTML = DATA.traces.map((t,i)=>'<li><button data-i="'+i+'" '+(i===cur?'aria-current="true"':'')+'>'+esc(t.name)+' <span class="'+(t.ok?'pf-pass':'pf-fail')+'">'+(t.ok?'ok':'FAIL')+'</span></button></li>').join('');
|
||||
$('pf-traces').querySelectorAll('button').forEach(b=>b.onclick=()=>{cur=+b.dataset.i;pos=0;showAll=false;render();});
|
||||
}
|
||||
function render(){
|
||||
renderList();
|
||||
const t = DATA.traces[cur];
|
||||
$('pf-title').textContent = t.name;
|
||||
$('pf-about').textContent = t.about;
|
||||
const upto = showAll ? t.steps.length : pos + 1;
|
||||
$('pf-steps').innerHTML = t.steps.slice(0, upto).map((s,i)=>
|
||||
'<tr class="pf-row '+(i===pos&&!showAll?'pf-cur':'')+'" data-i="'+i+'"><td>'+(i+1)+'</td><td class="pf-mono">'+esc(s.event)+'</td><td><span class="pf-rule" data-kind="'+esc(s.ruleKind)+'">'+esc(s.rule)+'</span></td><td>'+stateHtml(s.before)+'</td><td>'+stateHtml(s.after)+'</td><td class="pf-mono pf-fx">'+(s.effects.length?s.effects.map(esc).join('<br>'):'none')+'</td></tr>').join('');
|
||||
$('pf-steps').querySelectorAll('tr').forEach(r=>r.onclick=()=>{pos=+r.dataset.i;showAll=false;render();});
|
||||
const s = t.steps[pos];
|
||||
$('pf-why').innerHTML = 'Step '+(pos+1)+' of '+t.steps.length+' went to <b>'+esc(s.after.name)+'</b>. Rule <b>'+esc(s.rule)+'</b> ('+esc(s.ruleKind)+'): '+esc(s.ruleText)+'. Event: <code>'+esc(s.event)+'</code>.';
|
||||
$('pf-pos').textContent = (pos+1)+' / '+t.steps.length;
|
||||
$('pf-prev').disabled = pos===0; $('pf-next').disabled = pos===t.steps.length-1;
|
||||
$('pf-final').innerHTML = 'Expected final state: <b>'+esc(t.final.name)+'</b> <span class="pf-mono">'+esc(t.final.detail)+'</span>. '+(t.ok?'<span class="pf-pass">The reducer matches the expected rules, effects and final state.</span>':'<span class="pf-fail">'+t.bad.map(esc).join('; ')+'</span>');
|
||||
}
|
||||
$('pf-prev').onclick=()=>{if(pos>0){pos--;showAll=false;render();}};
|
||||
$('pf-next').onclick=()=>{const t=DATA.traces[cur];if(pos<t.steps.length-1){pos++;showAll=false;render();}};
|
||||
$('pf-all').onclick=()=>{showAll=!showAll;render();};
|
||||
document.addEventListener('keydown',e=>{if(e.key==='ArrowRight')$('pf-next').click();if(e.key==='ArrowLeft')$('pf-prev').click();});
|
||||
$('pf-rules').innerHTML = '<tr><th>Id</th><th>Kind</th><th>Meaning</th></tr>'+DATA.rules.map(r=>'<tr><td><span class="pf-rule" data-kind="'+esc(r.kind)+'">'+esc(r.id)+'</span></td><td>'+esc(r.kind)+'</td><td>'+esc(r.text)+'</td></tr>').join('');
|
||||
renderExplore(); render();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,445 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
package proposalflow
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Terminal says the proposal reached an end that no event leaves. Abandoned is
|
||||
// not terminal: a late receipt still settles it.
|
||||
func Terminal(s State) bool {
|
||||
switch s.(type) {
|
||||
case Refused, Applied, Rejected, NotApplied, Cancelled, Expired, Stale, Dropped:
|
||||
return true
|
||||
case Unproposed, Pending, Admitting, ExecPrepared, ExecSent, OutcomeUnknown, UnknownManual, Abandoned:
|
||||
return false
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Blocking says the situation blocks conflicting writes on the resource.
|
||||
func Blocking(s State) bool {
|
||||
switch s.(type) {
|
||||
case ExecPrepared, ExecSent, OutcomeUnknown, UnknownManual:
|
||||
return true
|
||||
case Unproposed, Refused, Pending, Admitting, Abandoned, Applied, Rejected, NotApplied, Cancelled, Expired, Stale, Dropped:
|
||||
return false
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// PropOf returns the proposal data of a situation, if it carries any.
|
||||
func PropOf(s State) (Prop, bool) {
|
||||
switch st := s.(type) {
|
||||
case Pending:
|
||||
return st.P, true
|
||||
case Admitting:
|
||||
return st.P, true
|
||||
case ExecPrepared:
|
||||
return st.P, true
|
||||
case ExecSent:
|
||||
return st.P, true
|
||||
case OutcomeUnknown:
|
||||
return st.P, true
|
||||
case UnknownManual:
|
||||
return st.P, true
|
||||
case Abandoned:
|
||||
return st.P, true
|
||||
case Applied:
|
||||
return st.P, true
|
||||
case Rejected:
|
||||
return st.P, true
|
||||
case NotApplied:
|
||||
return st.P, true
|
||||
case Cancelled:
|
||||
return st.P, true
|
||||
case Expired:
|
||||
return st.P, true
|
||||
case Stale:
|
||||
return st.P, true
|
||||
case Dropped:
|
||||
return st.P, true
|
||||
case Unproposed, Refused:
|
||||
return Prop{}, false
|
||||
}
|
||||
return Prop{}, false
|
||||
}
|
||||
|
||||
// Key is a stable identity of a state value for the search.
|
||||
func Key(s State) string { return fmt.Sprintf("%s%+v", s.Name(), s) }
|
||||
|
||||
// EffectNames renders effects in their short stable form.
|
||||
func EffectNames(fx []Effect) []string {
|
||||
out := make([]string, len(fx))
|
||||
for i, f := range fx {
|
||||
out[i] = f.String()
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- invariants
|
||||
|
||||
// CheckState checks the safety predicates that hold in every reachable state.
|
||||
func CheckState(s State) []string {
|
||||
var bad []string
|
||||
if p, ok := PropOf(s); ok {
|
||||
// A schema action is only ever held by an owner's proposal. The action
|
||||
// is derived from Kind, so no state can disagree with its tool.
|
||||
if p.Kind == KindSchema && p.ProposerRole != RoleOwner {
|
||||
bad = append(bad, "schema proposal held for a non-owner: "+Key(s))
|
||||
}
|
||||
if p.Action() == ActionNone {
|
||||
bad = append(bad, "proposal without a mutating action: "+Key(s))
|
||||
}
|
||||
}
|
||||
if u, ok := s.(OutcomeUnknown); ok && (u.Lookups < 0 || u.Lookups >= MaxLookups) {
|
||||
bad = append(bad, fmt.Sprintf("lookup counter out of bound: %+v", u))
|
||||
}
|
||||
if _, ok := s.(Abandoned); ok && Blocking(s) {
|
||||
bad = append(bad, "an abandoned outcome must not block writes")
|
||||
}
|
||||
return bad
|
||||
}
|
||||
|
||||
func postSend(s State) bool {
|
||||
switch s.(type) {
|
||||
case ExecSent, OutcomeUnknown, UnknownManual, Abandoned:
|
||||
return true
|
||||
case Unproposed, Refused, Pending, Admitting, ExecPrepared, Applied, Rejected, NotApplied, Cancelled, Expired, Stale, Dropped:
|
||||
return false
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func executing(s State) bool {
|
||||
switch s.(type) {
|
||||
case ExecPrepared, ExecSent:
|
||||
return true
|
||||
case Unproposed, Refused, Pending, Admitting, OutcomeUnknown, UnknownManual, Abandoned, Applied, Rejected, NotApplied, Cancelled, Expired, Stale, Dropped:
|
||||
return false
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func isUnknown(s State) bool {
|
||||
switch s.(type) {
|
||||
case OutcomeUnknown, UnknownManual:
|
||||
return true
|
||||
case Unproposed, Refused, Pending, Admitting, ExecPrepared, ExecSent, Abandoned, Applied, Rejected, NotApplied, Cancelled, Expired, Stale, Dropped:
|
||||
return false
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// CheckTransition checks the safety predicates of one step.
|
||||
func CheckTransition(from State, e Event, to State, fx []Effect) []string {
|
||||
var bad []string
|
||||
add := func(format string, a ...any) {
|
||||
bad = append(bad, fmt.Sprintf(format, a...)+" ["+Key(from)+" --"+describe(e)+"--> "+Key(to)+"]")
|
||||
}
|
||||
// S1: a closed proposal never changes, so it is never Applied and
|
||||
// Cancelled. Only Abandoned is open, and it only ends.
|
||||
if Terminal(from) && Key(from) != Key(to) {
|
||||
add("S1 a terminal state changed")
|
||||
}
|
||||
// S2: Applied only on evidence of a commit. A dispatch is never success.
|
||||
if _, was := from.(Applied); !was {
|
||||
if _, now := to.(Applied); now {
|
||||
okEv := false
|
||||
switch ev := e.(type) {
|
||||
case ApplyResult:
|
||||
okEv = ev.Outcome == ApplyCommitted
|
||||
case ReceiptResult:
|
||||
okEv = ev.Answer == ReceiptCommitted
|
||||
}
|
||||
if !okEv {
|
||||
add("S2 Applied without a committed answer")
|
||||
}
|
||||
}
|
||||
}
|
||||
// S3: a proposal enters Admitting only by a confirmation of its own
|
||||
// proposer, from a human surface, with the stored digest, before expiry
|
||||
// and with a live role that covers the derived action.
|
||||
_, wasAdmitting := from.(Admitting)
|
||||
if _, now := to.(Admitting); now && !wasAdmitting {
|
||||
if _, was := from.(Pending); !was {
|
||||
add("S3 Admitting entered from outside Pending")
|
||||
}
|
||||
c, ok := e.(Confirm)
|
||||
p, _ := PropOf(to)
|
||||
if !ok || c.Actor != p.Proposer || !c.Human || !c.DigestOK || c.Now >= p.ExpiresAt || c.Role.rank() < p.Action().Required().rank() {
|
||||
add("S3 Admitting without a valid confirmation")
|
||||
}
|
||||
// S4: a schema action is never admitted for a non-owner.
|
||||
if p.Action() == ActionSchema && (!ok || c.Role != RoleOwner) {
|
||||
add("S4 schema action admitted for a non-owner")
|
||||
}
|
||||
}
|
||||
// S5: execution starts only from an admission for the proposal's own
|
||||
// generation, before expiry. A stale generation never executes.
|
||||
if _, now := to.(ExecPrepared); now {
|
||||
if _, was := from.(Admitting); !was {
|
||||
if _, again := from.(ExecPrepared); !again {
|
||||
add("S5 execution entered from outside Admitting")
|
||||
}
|
||||
} else {
|
||||
g, ok := e.(AdmitGranted)
|
||||
p, _ := PropOf(to)
|
||||
if !ok || g.RuntimeGen != p.Gen || g.Now >= p.ExpiresAt {
|
||||
add("S5 execution from a stale generation or expired proposal")
|
||||
}
|
||||
}
|
||||
}
|
||||
// S6: at most one dispatch, only on the step that persists executing, and
|
||||
// persist comes first.
|
||||
nd := 0
|
||||
for i, f := range fx {
|
||||
if _, ok := f.(Dispatch); ok {
|
||||
nd++
|
||||
if i == 0 || fx[i-1].String() != "persist:executing" {
|
||||
add("S6 dispatch not preceded by persist:executing")
|
||||
}
|
||||
if _, was := from.(Admitting); !was {
|
||||
add("S6 dispatch outside the admission step")
|
||||
}
|
||||
}
|
||||
}
|
||||
if nd > 1 {
|
||||
add("S6 more than one dispatch in a step")
|
||||
}
|
||||
// S7: once the request may have been sent it never goes back to a state
|
||||
// that can send again. This is what makes a second apply impossible.
|
||||
if postSend(from) {
|
||||
switch to.(type) {
|
||||
case Unproposed, Pending, Admitting, ExecPrepared:
|
||||
add("S7 a sent proposal returned to a state that can dispatch")
|
||||
case Refused, ExecSent, OutcomeUnknown, UnknownManual, Abandoned, Applied, Rejected, NotApplied, Cancelled, Expired, Stale, Dropped:
|
||||
}
|
||||
}
|
||||
if _, was := from.(ExecPrepared); was {
|
||||
if _, back := to.(Pending); back {
|
||||
n, ok := e.(NotSent)
|
||||
if !ok || (n.Cause != NotSentSlots && n.Cause != NotSentInvalid && n.Cause != NotSentWithdrawn) {
|
||||
add("S7 prepared returned to Pending without a retryable not-sent answer")
|
||||
}
|
||||
}
|
||||
}
|
||||
// S8: an unknown outcome comes only from a send that may have happened
|
||||
// (finding a), or from a restart that cannot tell. An admission refusal or
|
||||
// a not-sent answer never makes one.
|
||||
if isUnknown(to) && !isUnknown(from) {
|
||||
okSrc := false
|
||||
if executing(from) {
|
||||
switch ev := e.(type) {
|
||||
case ApplyResult:
|
||||
okSrc = ev.Outcome == ApplyUnsettled || ev.Outcome == ApplyTransportError || ev.Outcome == ApplyTimeout
|
||||
case Restart:
|
||||
okSrc = true
|
||||
}
|
||||
}
|
||||
if !okSrc {
|
||||
add("S8 unknown outcome without a send that may have reached the add-on")
|
||||
}
|
||||
}
|
||||
// S9: blocking states only exist after the dispatch step.
|
||||
if Blocking(to) && !Blocking(from) {
|
||||
if _, adm := from.(Admitting); !adm {
|
||||
add("S9 a blocking situation entered outside admission or recovery")
|
||||
}
|
||||
}
|
||||
// S10: Propose creates a state only with a role that covers the action
|
||||
// derived from the tool kind, whatever action the caller claimed.
|
||||
if _, was := from.(Unproposed); was {
|
||||
if _, now := to.(Pending); now {
|
||||
pr, _ := e.(Propose)
|
||||
if pr.Role.rank() < ActionOf(pr.Tool).Required().rank() || ActionOf(pr.Tool) == ActionNone {
|
||||
add("S10 proposal stored with a role below the derived action")
|
||||
}
|
||||
}
|
||||
}
|
||||
return bad
|
||||
}
|
||||
|
||||
func describe(e Event) string { return fmt.Sprintf("%s%+v", e.Name(), e) }
|
||||
|
||||
// Describe renders an event for people.
|
||||
func Describe(e Event) string { return describe(e) }
|
||||
|
||||
// ---------------------------------------------------------------- search
|
||||
|
||||
// Universe is every external event that can arrive in any state, with a small
|
||||
// finite set of parameters. Each parameter varies one at a time around a
|
||||
// base, plus the full role and tool grid for Propose.
|
||||
func Universe() []Event {
|
||||
const exp = 10
|
||||
var u []Event
|
||||
base := Propose{ID: "op", Actor: "alice", Role: RoleOwner, Tool: KindWrite, Claimed: ActionWrite, Gen: 1, RuntimeGen: 1, ExpiresAt: exp, Now: 0}
|
||||
for _, r := range []Role{RoleViewer, RoleEditor, RoleOwner} {
|
||||
for _, k := range []ToolKind{KindUndeclared, KindRead, KindWrite, KindSchema} {
|
||||
p := base
|
||||
p.Role, p.Tool = r, k
|
||||
u = append(u, p)
|
||||
}
|
||||
}
|
||||
for _, mod := range []func(*Propose){
|
||||
func(p *Propose) { p.RuntimeGen = 2 },
|
||||
func(p *Propose) { p.Held = true },
|
||||
func(p *Propose) { p.Blocked = true },
|
||||
func(p *Propose) { p.ExpiresAt = 0 },
|
||||
func(p *Propose) { p.Claimed = ActionSchema },
|
||||
} {
|
||||
p := base
|
||||
mod(&p)
|
||||
u = append(u, p)
|
||||
}
|
||||
cb := Confirm{Actor: "alice", Role: RoleOwner, Human: true, DigestOK: true, Now: 0}
|
||||
u = append(u, cb)
|
||||
for _, mod := range []func(*Confirm){
|
||||
func(c *Confirm) { c.Role = RoleEditor },
|
||||
func(c *Confirm) { c.Role = RoleViewer },
|
||||
func(c *Confirm) { c.Role = RoleNone },
|
||||
func(c *Confirm) { c.Human = false },
|
||||
func(c *Confirm) { c.DigestOK = false },
|
||||
func(c *Confirm) { c.Actor = "bob" },
|
||||
func(c *Confirm) { c.Now = exp },
|
||||
} {
|
||||
c := cb
|
||||
mod(&c)
|
||||
u = append(u, c)
|
||||
}
|
||||
u = append(u, Cancel{Actor: "alice"}, Cancel{Actor: "bob"}, Tick{Now: 0}, Tick{Now: exp})
|
||||
u = append(u, AdmitGranted{Now: 0, RuntimeGen: 1}, AdmitGranted{Now: 0, RuntimeGen: 2}, AdmitGranted{Now: exp, RuntimeGen: 1})
|
||||
for _, c := range []AdmitCause{CauseHeld, CauseForbidden, CauseUnavailable, CauseBlocked, CauseCaller} {
|
||||
u = append(u, AdmitRefused{Cause: c})
|
||||
}
|
||||
u = append(u, SendStarted{})
|
||||
for _, c := range []NotSentCause{NotSentSlots, NotSentInvalid, NotSentWithdrawn, NotSentHold, NotSentAuthority, NotSentSuperseded} {
|
||||
u = append(u, NotSent{Cause: c})
|
||||
}
|
||||
for _, o := range []ApplyOutcome{ApplyCommitted, ApplyRejected, ApplyRefused, ApplyConflict, ApplyUnsettled, ApplyTransportError, ApplyTimeout} {
|
||||
u = append(u, ApplyResult{Outcome: o})
|
||||
}
|
||||
u = append(u, LookupDue{})
|
||||
for _, a := range []ReceiptAnswer{ReceiptCommitted, ReceiptRejected, ReceiptNotCommitted, ReceiptUnsettled} {
|
||||
u = append(u, ReceiptResult{Answer: a})
|
||||
}
|
||||
ob := OwnerResolve{Role: RoleOwner, Human: true, Reason: true}
|
||||
u = append(u, ob)
|
||||
for _, mod := range []func(*OwnerResolve){
|
||||
func(o *OwnerResolve) { o.Role = RoleEditor },
|
||||
func(o *OwnerResolve) { o.Human = false },
|
||||
func(o *OwnerResolve) { o.Reason = false },
|
||||
func(o *OwnerResolve) { o.Held = true },
|
||||
} {
|
||||
o := ob
|
||||
mod(&o)
|
||||
u = append(u, o)
|
||||
}
|
||||
u = append(u, HoldAcquired{}, HoldReleased{}, GenerationChanged{New: 2}, AuthorityLowered{}, Restart{}, RetentionElapsed{})
|
||||
return u
|
||||
}
|
||||
|
||||
// Edge is one explored step.
|
||||
type Edge struct {
|
||||
From, To string
|
||||
Event string
|
||||
Rule string
|
||||
}
|
||||
|
||||
// Report is the result of the search.
|
||||
type Report struct {
|
||||
States int
|
||||
Edges int
|
||||
Terminals []string
|
||||
DeadEnds []string
|
||||
Violations []string
|
||||
RuleHits map[string]int
|
||||
// ByName counts reached situations per constructor name.
|
||||
ByName map[string]int
|
||||
// Path gives, per state key, the shortest event path from the start.
|
||||
Path map[string][]string
|
||||
}
|
||||
|
||||
// Explore searches every state reachable from Unproposed by the Universe, checks
|
||||
// the safety predicates on every state and step, and checks liveness: from
|
||||
// every reachable state a terminal state must be reachable. Counters are
|
||||
// bounded by MaxLookups and the finite Universe.
|
||||
func Explore() Report {
|
||||
type node struct {
|
||||
s State
|
||||
path []string
|
||||
}
|
||||
init := Unproposed{}
|
||||
seen := map[string]node{Key(init): {s: init}}
|
||||
queue := []node{{s: init}}
|
||||
succ := map[string]map[string]bool{}
|
||||
rep := Report{RuleHits: map[string]int{}, ByName: map[string]int{}, Path: map[string][]string{}}
|
||||
uni := Universe()
|
||||
for len(queue) > 0 {
|
||||
n := queue[0]
|
||||
queue = queue[1:]
|
||||
k := Key(n.s)
|
||||
for _, v := range CheckState(n.s) {
|
||||
rep.Violations = append(rep.Violations, v+" path="+strings.Join(n.path, " ; "))
|
||||
}
|
||||
succ[k] = map[string]bool{}
|
||||
for _, e := range uni {
|
||||
next, fx, rule := Reduce(n.s, e)
|
||||
rep.Edges++
|
||||
rep.RuleHits[rule]++
|
||||
if _, ok := RuleByID(rule); !ok {
|
||||
rep.Violations = append(rep.Violations, "unknown rule id "+rule)
|
||||
}
|
||||
for _, v := range CheckTransition(n.s, e, next, fx) {
|
||||
rep.Violations = append(rep.Violations, v+" path="+strings.Join(n.path, " ; "))
|
||||
}
|
||||
nk := Key(next)
|
||||
succ[k][nk] = true
|
||||
if _, ok := seen[nk]; !ok {
|
||||
nn := node{s: next, path: append(append([]string{}, n.path...), describe(e))}
|
||||
seen[nk] = nn
|
||||
queue = append(queue, nn)
|
||||
}
|
||||
}
|
||||
}
|
||||
// Liveness: reverse reachability from the terminal states.
|
||||
good := map[string]bool{}
|
||||
for k, n := range seen {
|
||||
if Terminal(n.s) {
|
||||
good[k] = true
|
||||
}
|
||||
}
|
||||
for changed := true; changed; {
|
||||
changed = false
|
||||
for k, outs := range succ {
|
||||
if good[k] {
|
||||
continue
|
||||
}
|
||||
for o := range outs {
|
||||
if good[o] {
|
||||
good[k] = true
|
||||
changed = true
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for k, n := range seen {
|
||||
rep.ByName[n.s.Name()]++
|
||||
rep.Path[k] = n.path
|
||||
if Terminal(n.s) {
|
||||
rep.Terminals = append(rep.Terminals, k)
|
||||
}
|
||||
if !good[k] {
|
||||
rep.DeadEnds = append(rep.DeadEnds, k+" path="+strings.Join(n.path, " ; "))
|
||||
}
|
||||
}
|
||||
rep.States = len(seen)
|
||||
sort.Strings(rep.Terminals)
|
||||
sort.Strings(rep.DeadEnds)
|
||||
sort.Strings(rep.Violations)
|
||||
return rep
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
package proposalflow
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = ginkgo.Describe("the invariant search", func() {
|
||||
// The search runs the breadth-first search from Unproposed over every
|
||||
// event of the Universe. It fails on any safety violation (with the event
|
||||
// path to it) and on any reachable state from which no terminal state can
|
||||
// be reached (a deadlock).
|
||||
ginkgo.It("finds no safety violation and no dead end", func() {
|
||||
rep := Explore()
|
||||
_, _ = fmt.Fprintf(ginkgo.GinkgoWriter, "states=%d edges=%d terminals=%d deadEnds=%d violations=%d\n", rep.States, rep.Edges, len(rep.Terminals), len(rep.DeadEnds), len(rep.Violations))
|
||||
Expect(rep.Violations).To(BeEmpty())
|
||||
Expect(rep.DeadEnds).To(BeEmpty(), "no event leads from a dead end to a terminal state")
|
||||
})
|
||||
|
||||
// This checks that the search is not vacuous.
|
||||
ginkgo.It("reaches every constructor", func() {
|
||||
rep := Explore()
|
||||
for _, s := range AllStates() {
|
||||
Expect(rep.ByName[s.Name()]).NotTo(BeZero(), "the search never reached %s", s.Name())
|
||||
}
|
||||
})
|
||||
|
||||
// This proves the checker can fail: a reducer that turns a not-sent
|
||||
// refusal into an unknown outcome (finding a) must violate S8. The check
|
||||
// runs the predicate on a hand-made step, so the real reducer is untouched.
|
||||
ginkgo.It("sees an injected bug", func() {
|
||||
p := prop(KindWrite, RoleOwner)
|
||||
catch := func(rule string, bad []string) {
|
||||
Expect(strings.Join(bad, "\n")).To(ContainSubstring(rule), "%s did not catch its violation", rule)
|
||||
}
|
||||
catch("S8", CheckTransition(ExecPrepared{P: p}, NotSent{NotSentSlots}, OutcomeUnknown{P: p}, nil))
|
||||
catch("S4", CheckTransition(Pending{P: prop(KindSchema, RoleOwner)}, Confirm{Actor: "alice", Role: RoleEditor, Human: true, DigestOK: true, Now: 1},
|
||||
Admitting{P: prop(KindSchema, RoleOwner), By: "alice"}, nil))
|
||||
catch("S5", CheckTransition(Admitting{P: p, By: "alice"}, AdmitGranted{Now: 1, RuntimeGen: 2}, ExecPrepared{P: p},
|
||||
[]Effect{Persist{To: "executing"}, Dispatch{}}))
|
||||
catch("S1", CheckTransition(Cancelled{P: p}, ApplyResult{ApplyCommitted}, Applied{P: p}, nil))
|
||||
catch("S7", CheckTransition(ExecSent{P: p}, NotSent{NotSentSlots}, Pending{P: p}, nil))
|
||||
})
|
||||
|
||||
// This is the liveness of finding (c): every blocking unknown situation
|
||||
// reaches a terminal state by events the owner or the environment can
|
||||
// send, including from UnknownManual with no lookups left.
|
||||
ginkgo.It("gives every unknown outcome an exit", func() {
|
||||
p := prop(KindWrite, RoleOwner)
|
||||
for _, s := range []State{OutcomeUnknown{P: p}, OutcomeUnknown{P: p, Lookups: 1}, UnknownManual{P: p}} {
|
||||
cur := s
|
||||
for _, e := range []Event{ownerResolve(), RetentionElapsed{}} {
|
||||
cur, _, _ = Reduce(cur, e)
|
||||
}
|
||||
Expect(Terminal(cur)).To(BeTrue(), "from %s the owner resolution and retention end in %s, not a terminal state", Key(s), Key(cur))
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,9 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
package proposalflow
|
||||
|
||||
// The step-through viewer is generated by running the real reducer over the
|
||||
// traces. Run `go generate ./pkg/addons/model/proposalflow` after any change
|
||||
// to reduce.go or traces.go.
|
||||
//
|
||||
//go:generate go run ./cmd/viewer -o viewer.html
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,130 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
package proposalflow
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// declaredTypes lists the type names in reduce.go that declare the marker
|
||||
// method, so a new constructor or event that is not in AllStates or AllEvents
|
||||
// fails the build of the tests.
|
||||
func declaredTypes(marker string) []string {
|
||||
ginkgo.GinkgoHelper()
|
||||
f, err := parser.ParseFile(token.NewFileSet(), "reduce.go", nil, 0)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
var out []string
|
||||
for _, d := range f.Decls {
|
||||
fn, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fn.Recv == nil || fn.Name.Name != marker {
|
||||
continue
|
||||
}
|
||||
if id, ok := fn.Recv.List[0].Type.(*ast.Ident); ok {
|
||||
out = append(out, id.Name)
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
var _ = ginkgo.Describe("the reducer", func() {
|
||||
ginkgo.It("lists every declared state and event", func() {
|
||||
var states, events []string
|
||||
for _, s := range AllStates() {
|
||||
states = append(states, s.Name())
|
||||
}
|
||||
for _, e := range AllEvents() {
|
||||
events = append(events, e.Name())
|
||||
}
|
||||
sort.Strings(states)
|
||||
sort.Strings(events)
|
||||
Expect(states).To(Equal(declaredTypes("isState")), "AllStates")
|
||||
Expect(events).To(Equal(declaredTypes("isEvent")), "AllEvents")
|
||||
})
|
||||
|
||||
// Every (state, event) pair, with every parameter variant of the event,
|
||||
// goes to the reducer. No pair may panic, and every pair must return a
|
||||
// listed rule id and a state of the model.
|
||||
ginkgo.It("handles every state and event pair", func() {
|
||||
p := prop(KindWrite, RoleOwner)
|
||||
states := []State{
|
||||
Unproposed{}, Refused{Why: RefuseUndeclaredTool}, Pending{P: p}, Admitting{P: p, By: "alice"}, ExecPrepared{P: p}, ExecSent{P: p},
|
||||
OutcomeUnknown{P: p, Lookups: 0}, OutcomeUnknown{P: p, Lookups: 1}, UnknownManual{P: p}, Abandoned{P: p}, Applied{P: p},
|
||||
Rejected{P: p, Why: RejectReceipt}, NotApplied{P: p}, Cancelled{P: p, Why: CancelByHold}, Expired{P: p}, Stale{P: p}, Dropped{P: p},
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, s := range states {
|
||||
seen[s.Name()] = true
|
||||
}
|
||||
for _, s := range AllStates() {
|
||||
Expect(seen[s.Name()]).To(BeTrue(), "no sample state for %s", s.Name())
|
||||
}
|
||||
events := Universe()
|
||||
named := map[string]bool{}
|
||||
for _, e := range events {
|
||||
named[e.Name()] = true
|
||||
}
|
||||
for _, e := range AllEvents() {
|
||||
Expect(named[e.Name()]).To(BeTrue(), "Universe has no %s", e.Name())
|
||||
}
|
||||
for _, s := range states {
|
||||
for _, e := range events {
|
||||
var next State
|
||||
var rule string
|
||||
Expect(func() { next, _, rule = Reduce(s, e) }).NotTo(Panic(), "%s x %s", s.Name(), Describe(e))
|
||||
_, ok := RuleByID(rule)
|
||||
Expect(ok).To(BeTrue(), "%s x %s: rule %q not in the table", s.Name(), e.Name(), rule)
|
||||
Expect(next).NotTo(BeNil(), "%s x %s: nil state", s.Name(), e.Name())
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
ginkgo.It("is pure", func() {
|
||||
p := prop(KindWrite, RoleOwner)
|
||||
for _, s := range []State{Pending{P: p}, ExecSent{P: p}, OutcomeUnknown{P: p}} {
|
||||
for _, e := range Universe() {
|
||||
n1, f1, r1 := Reduce(s, e)
|
||||
n2, f2, r2 := Reduce(s, e)
|
||||
Expect(Key(n1)).To(Equal(Key(n2)), "%s x %s", s.Name(), e.Name())
|
||||
Expect(r1).To(Equal(r2), "%s x %s", s.Name(), e.Name())
|
||||
Expect(strings.Join(EffectNames(f1), "|")).To(Equal(strings.Join(EffectNames(f2), "|")), "%s x %s", s.Name(), e.Name())
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
ginkgo.It("has a rule table without duplicates", func() {
|
||||
seen := map[string]bool{}
|
||||
for _, r := range Rules {
|
||||
Expect(seen[r.ID]).To(BeFalse(), "duplicate rule %s", r.ID)
|
||||
seen[r.ID] = true
|
||||
Expect(r.Text).NotTo(BeEmpty(), "rule %s has no text", r.ID)
|
||||
}
|
||||
})
|
||||
|
||||
ginkgo.It("takes the action from the tool kind", func() {
|
||||
Expect(ActionOf(KindSchema)).To(Equal(ActionSchema))
|
||||
Expect(ActionOf(KindWrite)).To(Equal(ActionWrite))
|
||||
Expect(ActionOf(KindRead)).To(Equal(ActionNone), "a read tool must not map to a mutating action")
|
||||
Expect(ActionOf(KindUndeclared)).To(Equal(ActionNone), "an undeclared tool must not map to a mutating action")
|
||||
// The claimed action of the request never changes the result.
|
||||
for _, claimed := range []Action{ActionNone, ActionWrite, ActionSchema} {
|
||||
for _, role := range []Role{RoleViewer, RoleEditor, RoleOwner} {
|
||||
a := propose(role, KindSchema)
|
||||
a.Claimed = claimed
|
||||
b := propose(role, KindSchema)
|
||||
b.Claimed = ActionWrite
|
||||
sa, _, ra := Reduce(Unproposed{}, a)
|
||||
sb, _, rb := Reduce(Unproposed{}, b)
|
||||
Expect(Key(sa)).To(Equal(Key(sb)), "the claimed action %v changed the outcome for %v", claimed, role)
|
||||
Expect(ra).To(Equal(rb))
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,15 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
package proposalflow
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestProposalFlow(t *testing.T) {
|
||||
RegisterFailHandler(ginkgo.Fail)
|
||||
ginkgo.RunSpecs(t, "Proposal flow model")
|
||||
}
|
||||
@@ -0,0 +1,514 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
package proposalflow
|
||||
|
||||
import "fmt"
|
||||
|
||||
// Step is one event of a trace with what the reducer must answer.
|
||||
type Step struct {
|
||||
Event Event
|
||||
Rule string
|
||||
Effects []string
|
||||
}
|
||||
|
||||
// Trace is a named scenario: a list of events with the expected rule ids and
|
||||
// effects in order, and the expected final state. The tests assert all three.
|
||||
// The viewer replays the same traces through the real reducer.
|
||||
type Trace struct {
|
||||
Name string
|
||||
About string
|
||||
Steps []Step
|
||||
Final State
|
||||
}
|
||||
|
||||
// StepResult is what the reducer really did for one step.
|
||||
type StepResult struct {
|
||||
Event Event
|
||||
Before State
|
||||
After State
|
||||
Rule string
|
||||
Effects []Effect
|
||||
}
|
||||
|
||||
// Run feeds the events through Reduce from the start state.
|
||||
func Run(start State, events []Event) []StepResult {
|
||||
out := make([]StepResult, 0, len(events))
|
||||
s := start
|
||||
for _, e := range events {
|
||||
next, fx, rule := Reduce(s, e)
|
||||
out = append(out, StepResult{Event: e, Before: s, After: next, Rule: rule, Effects: fx})
|
||||
s = next
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Events lists the events of a trace.
|
||||
func (t Trace) Events() []Event {
|
||||
ev := make([]Event, len(t.Steps))
|
||||
for i, s := range t.Steps {
|
||||
ev[i] = s.Event
|
||||
}
|
||||
return ev
|
||||
}
|
||||
|
||||
// Verify replays the trace and reports every mismatch with the expectation.
|
||||
func (t Trace) Verify() []string {
|
||||
var bad []string
|
||||
res := Run(Unproposed{}, t.Events())
|
||||
for i, r := range res {
|
||||
want := t.Steps[i]
|
||||
if r.Rule != want.Rule {
|
||||
bad = append(bad, fmt.Sprintf("%s step %d (%s): rule %s, want %s", t.Name, i+1, r.Event.Name(), r.Rule, want.Rule))
|
||||
}
|
||||
got := EffectNames(r.Effects)
|
||||
if len(got) != len(want.Effects) {
|
||||
bad = append(bad, fmt.Sprintf("%s step %d: effects %v, want %v", t.Name, i+1, got, want.Effects))
|
||||
continue
|
||||
}
|
||||
for j := range got {
|
||||
if got[j] != want.Effects[j] {
|
||||
bad = append(bad, fmt.Sprintf("%s step %d: effects %v, want %v", t.Name, i+1, got, want.Effects))
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(res) > 0 && Key(res[len(res)-1].After) != Key(t.Final) {
|
||||
bad = append(bad, fmt.Sprintf("%s: final %s, want %s", t.Name, Key(res[len(res)-1].After), Key(t.Final)))
|
||||
}
|
||||
return bad
|
||||
}
|
||||
|
||||
// Trace building blocks. The proposer is alice; bob is another user.
|
||||
|
||||
func prop(k ToolKind, r Role) Prop {
|
||||
return Prop{ID: "op1", Proposer: "alice", ProposerRole: r, Kind: k, Gen: 1, ExpiresAt: 10}
|
||||
}
|
||||
|
||||
func propose(r Role, k ToolKind) Propose {
|
||||
return Propose{ID: "op1", Actor: "alice", Role: r, Tool: k, Claimed: ActionWrite, Gen: 1, RuntimeGen: 1, ExpiresAt: 10, Now: 0}
|
||||
}
|
||||
|
||||
func confirm() Confirm {
|
||||
return Confirm{Actor: "alice", Role: RoleOwner, Human: true, DigestOK: true, Now: 1}
|
||||
}
|
||||
|
||||
func granted() AdmitGranted { return AdmitGranted{Now: 1, RuntimeGen: 1} }
|
||||
|
||||
func st(e Event, rule string, fx ...string) Step { return Step{Event: e, Rule: rule, Effects: fx} }
|
||||
|
||||
// ownerResolve is the resource owner at a human surface with a reason.
|
||||
func ownerResolve() OwnerResolve { return OwnerResolve{Role: RoleOwner, Human: true, Reason: true} }
|
||||
|
||||
// toSent is the common prefix that ends in a request on the wire.
|
||||
func toSent() []Step {
|
||||
return []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(SendStarted{}, "R33"),
|
||||
}
|
||||
}
|
||||
|
||||
func with(prefix []Step, more ...Step) []Step { return append(append([]Step{}, prefix...), more...) }
|
||||
|
||||
// Traces returns the scenario traces. They cover the three review findings:
|
||||
// (a) a refusal before the send is retryable, (b) the action comes from the
|
||||
// tool kind, (c) an unknown outcome always has a way out.
|
||||
func Traces() []Trace {
|
||||
w := prop(KindWrite, RoleOwner)
|
||||
sch := prop(KindSchema, RoleOwner)
|
||||
return []Trace{
|
||||
{
|
||||
Name: "happy-path",
|
||||
About: "propose, confirm, admit, send, committed: Applied",
|
||||
Steps: with(toSent(), st(ApplyResult{ApplyCommitted}, "R37", "persist:applied", "reply:applied")),
|
||||
Final: Applied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "retry-after-not-sent",
|
||||
About: "finding (a): the client refuses before the send (no free slot); the proposal returns to Pending and is retried, never OutcomeUnknown",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(NotSent{NotSentSlots}, "R34", "persist:pending", "reply:unavailable"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(SendStarted{}, "R33"),
|
||||
st(ApplyResult{ApplyCommitted}, "R37", "persist:applied", "reply:applied"),
|
||||
},
|
||||
Final: Applied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "retry-after-admission-refused",
|
||||
About: "finding (a): the admission says held; nothing was persisted or sent, so the proposal stays retryable",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(AdmitRefused{CauseHeld}, "R29", "reply:unavailable"),
|
||||
st(HoldReleased{}, "I4"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(AdmitRefused{CauseBlocked}, "R29", "reply:blocked"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(SendStarted{}, "R33"),
|
||||
st(ApplyResult{ApplyCommitted}, "R37", "persist:applied", "reply:applied"),
|
||||
},
|
||||
Final: Applied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "withdrawn-then-generation-moved",
|
||||
About: "finding (a): the client is withdrawn before the send (add-on replaced); retryable, then the generation check makes it Stale instead of unknown",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(NotSent{NotSentWithdrawn}, "R34", "persist:pending", "reply:unavailable"),
|
||||
st(GenerationChanged{New: 2}, "R57", "persist:rejected"),
|
||||
st(confirm(), "R19", "reply:rejected"),
|
||||
},
|
||||
Final: Stale{P: w},
|
||||
},
|
||||
{
|
||||
Name: "timeout-unknown-then-receipt-committed",
|
||||
About: "finding (c): the apply times out after the send; lookups run; the receipt says committed",
|
||||
Steps: with(toSent(),
|
||||
st(ApplyResult{ApplyTimeout}, "R39", "persist:outcome-unknown", "reply:outcome-unknown"),
|
||||
st(LookupDue{}, "R45", "lookup-receipt"),
|
||||
st(ReceiptResult{ReceiptUnsettled}, "R43", "persist:outcome-unknown"),
|
||||
st(LookupDue{}, "R45", "lookup-receipt"),
|
||||
st(ReceiptResult{ReceiptCommitted}, "R40", "persist:applied"),
|
||||
),
|
||||
Final: Applied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "unknown-bounded-then-manual-resolution",
|
||||
About: "finding (c): lookups stay unsettled up to the bound; UnknownManual; owner resolution abandons it; retention drops it. No state is stuck",
|
||||
Steps: with(toSent(),
|
||||
st(ApplyResult{ApplyTransportError}, "R39", "persist:outcome-unknown", "reply:outcome-unknown"),
|
||||
st(ReceiptResult{ReceiptUnsettled}, "R43", "persist:outcome-unknown"),
|
||||
st(ReceiptResult{ReceiptUnsettled}, "R44", "persist:outcome-unknown"),
|
||||
st(LookupDue{}, "I7"),
|
||||
st(confirm(), "R17", "lookup-receipt", "reply:outcome-unknown"),
|
||||
st(OwnerResolve{Role: RoleEditor, Human: true, Reason: true}, "R46", "reply:forbidden"),
|
||||
st(OwnerResolve{Role: RoleOwner, Human: true, Reason: true, Held: true}, "R47", "reply:unavailable"),
|
||||
st(ownerResolve(), "R48", "persist:abandoned-unknown", "lookup-receipt", "reply:resolved"),
|
||||
st(LookupDue{}, "R45", "lookup-receipt"),
|
||||
st(RetentionElapsed{}, "R61"),
|
||||
),
|
||||
Final: Dropped{P: w},
|
||||
},
|
||||
{
|
||||
Name: "late-receipt-after-abandon",
|
||||
About: "an abandoned outcome that later turns out committed is recorded as Applied, not hidden",
|
||||
Steps: with(toSent(),
|
||||
st(ApplyResult{ApplyTimeout}, "R39", "persist:outcome-unknown", "reply:outcome-unknown"),
|
||||
st(ownerResolve(), "R48", "persist:abandoned-unknown", "lookup-receipt", "reply:resolved"),
|
||||
st(ReceiptResult{ReceiptCommitted}, "R40", "persist:applied"),
|
||||
),
|
||||
Final: Applied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "two-actors-racing-confirm",
|
||||
About: "another user's confirm is forbidden; a second confirm by the proposer while admission runs is busy; a repeat during execution looks the receipt up and never applies twice",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(Confirm{Actor: "bob", Role: RoleOwner, Human: true, DigestOK: true, Now: 1}, "R10", "reply:forbidden"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(confirm(), "R16", "reply:busy"),
|
||||
st(Cancel{Actor: "bob"}, "R23", "reply:forbidden"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(confirm(), "R17", "lookup-receipt", "reply:outcome-unknown"),
|
||||
st(Cancel{Actor: "alice"}, "R26", "reply:closed"),
|
||||
st(SendStarted{}, "R33"),
|
||||
st(ApplyResult{ApplyCommitted}, "R37", "persist:applied", "reply:applied"),
|
||||
},
|
||||
Final: Applied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "cancel-races-confirm",
|
||||
About: "cancel wins while the confirmation waits for admission; the late admission answer finds the proposal closed and nothing runs",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(Cancel{Actor: "alice"}, "R25", "persist:cancelled", "reply:cancelled"),
|
||||
st(granted(), "I3"),
|
||||
st(confirm(), "R20", "reply:closed"),
|
||||
},
|
||||
Final: Cancelled{P: w, Why: CancelByProposer},
|
||||
},
|
||||
{
|
||||
Name: "hold-during-prepared-dispatch",
|
||||
About: "a hold arrives after executing is persisted but before the send: the dispatch is cancelled, nothing was sent, Cancelled",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(HoldAcquired{}, "R52", "cancel-dispatch:hold"),
|
||||
st(NotSent{NotSentHold}, "R35", "persist:cancelled", "reply:cancelled"),
|
||||
st(HoldReleased{}, "I2"),
|
||||
},
|
||||
Final: Cancelled{P: w, Why: CancelByHold},
|
||||
},
|
||||
{
|
||||
Name: "hold-too-late-to-cancel",
|
||||
About: "the hold races the send and loses: the request is on the wire, so the proposal waits for the result instead of claiming it was cancelled",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(HoldAcquired{}, "R52", "cancel-dispatch:hold"),
|
||||
st(SendStarted{}, "R33"),
|
||||
st(HoldAcquired{}, "R53", "cancel-send"),
|
||||
st(HoldReleased{}, "I4"),
|
||||
st(ApplyResult{ApplyCommitted}, "R37", "persist:applied", "reply:applied"),
|
||||
},
|
||||
Final: Applied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "hold-during-sent-request",
|
||||
About: "a hold aborts the request in flight; the aborted call is OutcomeUnknown, and a not-committed answer settles it",
|
||||
Steps: with(toSent(),
|
||||
st(HoldAcquired{}, "R53", "cancel-send"),
|
||||
st(ApplyResult{ApplyTransportError}, "R39", "persist:outcome-unknown", "reply:outcome-unknown"),
|
||||
st(LookupDue{}, "R45", "lookup-receipt"),
|
||||
st(ReceiptResult{ReceiptNotCommitted}, "R42", "persist:rejected"),
|
||||
),
|
||||
Final: NotApplied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "hold-on-pending",
|
||||
About: "a hold invalidates a pending proposal; a later confirm gets closed",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(HoldAcquired{}, "R51", "persist:cancelled"),
|
||||
st(confirm(), "R20", "reply:closed"),
|
||||
},
|
||||
Final: Cancelled{P: w, Why: CancelByHold},
|
||||
},
|
||||
{
|
||||
Name: "generation-change-while-pending",
|
||||
About: "the add-on moves to a new generation: the pending proposal is Stale and never executes",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(GenerationChanged{New: 2}, "R57", "persist:rejected"),
|
||||
st(confirm(), "R19", "reply:rejected"),
|
||||
st(Tick{Now: 20}, "I2"),
|
||||
},
|
||||
Final: Stale{P: w},
|
||||
},
|
||||
{
|
||||
Name: "generation-seen-at-admission",
|
||||
About: "the generation moved before the change event arrived: the admission check finds a runtime of another generation and the proposal is Stale",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(AdmitGranted{Now: 1, RuntimeGen: 2}, "R31", "persist:rejected", "reply:stale"),
|
||||
},
|
||||
Final: Stale{P: w},
|
||||
},
|
||||
{
|
||||
Name: "generation-change-while-prepared",
|
||||
About: "the generation moves after executing is persisted: the dispatch is cancelled and the not-sent answer makes it Stale",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(GenerationChanged{New: 2}, "R58", "cancel-dispatch:superseded"),
|
||||
st(NotSent{NotSentSuperseded}, "R36", "persist:rejected", "reply:stale"),
|
||||
},
|
||||
Final: Stale{P: w},
|
||||
},
|
||||
{
|
||||
Name: "duplicate-confirm-after-applied",
|
||||
About: "a repeated confirm, propose or cancel after Applied returns the stored answer and runs nothing",
|
||||
Steps: with(toSent(),
|
||||
st(ApplyResult{ApplyCommitted}, "R37", "persist:applied", "reply:applied"),
|
||||
st(confirm(), "R18", "reply:applied"),
|
||||
st(propose(RoleOwner, KindWrite), "R9", "reply:existing"),
|
||||
st(Cancel{Actor: "alice"}, "R26", "reply:closed"),
|
||||
),
|
||||
Final: Applied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "duplicate-confirm-while-unknown",
|
||||
About: "a repeated confirm of an unknown outcome looks the receipt up and never applies again",
|
||||
Steps: with(toSent(),
|
||||
st(ApplyResult{ApplyTimeout}, "R39", "persist:outcome-unknown", "reply:outcome-unknown"),
|
||||
st(confirm(), "R17", "lookup-receipt", "reply:outcome-unknown"),
|
||||
st(confirm(), "R17", "lookup-receipt", "reply:outcome-unknown"),
|
||||
),
|
||||
Final: OutcomeUnknown{P: w},
|
||||
},
|
||||
{
|
||||
Name: "editor-schema-propose-refused",
|
||||
About: "finding (b): an editor proposes a schema-kind tool while claiming the write action; the action comes from the tool kind, so it is Refused",
|
||||
Steps: []Step{
|
||||
st(propose(RoleEditor, KindSchema), "R4", "reply:forbidden"),
|
||||
st(Confirm{Actor: "alice", Role: RoleEditor, Human: true, DigestOK: true, Now: 1}, "R22", "reply:forbidden"),
|
||||
},
|
||||
Final: Refused{Why: RefuseInsufficientRole},
|
||||
},
|
||||
{
|
||||
Name: "editor-confirm-of-schema-after-demotion",
|
||||
About: "finding (b): an owner proposes a schema change, is lowered to editor, and the confirm is forbidden; the proposal stays Pending",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindSchema), "R8", "persist:pending", "reply:created"),
|
||||
st(Confirm{Actor: "alice", Role: RoleEditor, Human: true, DigestOK: true, Now: 1}, "R14", "reply:forbidden"),
|
||||
st(confirm(), "R15", "admit:schema"),
|
||||
},
|
||||
Final: Admitting{P: sch, By: "alice"},
|
||||
},
|
||||
{
|
||||
Name: "out-of-order-receipt",
|
||||
About: "a receipt reply for a proposal not yet sent is ignored; a committed receipt that beats the apply reply settles it, and the late apply error changes nothing",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(ReceiptResult{ReceiptCommitted}, "I3"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(SendStarted{}, "R33"),
|
||||
st(ReceiptResult{ReceiptUnsettled}, "I5"),
|
||||
st(ReceiptResult{ReceiptCommitted}, "R40", "persist:applied"),
|
||||
st(ApplyResult{ApplyTransportError}, "I3"),
|
||||
},
|
||||
Final: Applied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "late-apply-reply-after-timeout",
|
||||
About: "the apply reply arrives after the timeout was recorded: the committed answer still settles it",
|
||||
Steps: with(toSent(),
|
||||
st(ApplyResult{ApplyTimeout}, "R39", "persist:outcome-unknown", "reply:outcome-unknown"),
|
||||
st(ApplyResult{ApplyCommitted}, "R37", "persist:applied", "reply:applied"),
|
||||
st(ApplyResult{ApplyTimeout}, "I3"),
|
||||
),
|
||||
Final: Applied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "restart-recovery",
|
||||
About: "a restart in admission returns to Pending; a restart with an executing proposal is OutcomeUnknown and a lookup settles it",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(Restart{}, "R59"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(Restart{}, "R60", "persist:outcome-unknown"),
|
||||
st(LookupDue{}, "R45", "lookup-receipt"),
|
||||
st(ReceiptResult{ReceiptCommitted}, "R40", "persist:applied"),
|
||||
},
|
||||
Final: Applied{P: w},
|
||||
},
|
||||
{
|
||||
Name: "addon-conflict-is-rejected",
|
||||
About: "a 409 answer is the add-on's definite answer, so Rejected, not OutcomeUnknown",
|
||||
Steps: with(toSent(), st(ApplyResult{ApplyConflict}, "R38", "persist:rejected", "reply:rejected")),
|
||||
Final: Rejected{P: w, Why: RejectConflict},
|
||||
},
|
||||
{
|
||||
Name: "expiry",
|
||||
About: "a tick before expiry changes nothing; a tick at expiry expires it; a confirm then gets expired",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(Tick{Now: 5}, "I8"),
|
||||
st(Tick{Now: 10}, "R28", "persist:expired"),
|
||||
st(confirm(), "R21", "reply:expired"),
|
||||
},
|
||||
Final: Expired{P: w},
|
||||
},
|
||||
{
|
||||
Name: "propose-while-held-then-retry",
|
||||
About: "a propose during a hold stores nothing and is retried after it",
|
||||
Steps: []Step{
|
||||
st(Propose{ID: "op1", Actor: "alice", Role: RoleOwner, Tool: KindWrite, Claimed: ActionWrite, Gen: 1, RuntimeGen: 1, ExpiresAt: 10, Held: true}, "R1", "reply:unavailable"),
|
||||
st(Propose{ID: "op1", Actor: "alice", Role: RoleOwner, Tool: KindWrite, Claimed: ActionWrite, Gen: 1, RuntimeGen: 1, ExpiresAt: 10, Blocked: true}, "R6", "reply:blocked"),
|
||||
st(Propose{ID: "op1", Actor: "alice", Role: RoleOwner, Tool: KindWrite, Claimed: ActionWrite, Gen: 1, RuntimeGen: 1, ExpiresAt: 0}, "R7", "reply:unavailable"),
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
},
|
||||
Final: Pending{P: w},
|
||||
},
|
||||
{
|
||||
Name: "propose-undeclared-tool",
|
||||
About: "a tool the add-on did not declare is refused and nothing is stored",
|
||||
Steps: []Step{st(propose(RoleOwner, KindUndeclared), "R2", "reply:refused")},
|
||||
Final: Refused{Why: RefuseUndeclaredTool},
|
||||
},
|
||||
{
|
||||
Name: "propose-read-tool",
|
||||
About: "a read tool is never a proposal",
|
||||
Steps: []Step{st(propose(RoleOwner, KindRead), "R3", "reply:refused")},
|
||||
Final: Refused{Why: RefuseNotAMutation},
|
||||
},
|
||||
{
|
||||
Name: "propose-stale-generation",
|
||||
About: "a preview made against another generation than the runtime's is refused",
|
||||
Steps: []Step{st(Propose{ID: "op1", Actor: "alice", Role: RoleOwner, Tool: KindWrite, Claimed: ActionWrite, Gen: 1, RuntimeGen: 2, ExpiresAt: 10}, "R5", "reply:stale")},
|
||||
Final: Refused{Why: RefuseStaleGeneration},
|
||||
},
|
||||
{
|
||||
Name: "confirm-guards",
|
||||
About: "a confirm from a non-human credential or with a changed digest is refused and changes nothing; a confirm after expiry expires the proposal",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(Confirm{Actor: "alice", Role: RoleOwner, Human: false, DigestOK: true, Now: 1}, "R11", "reply:forbidden"),
|
||||
st(Confirm{Actor: "alice", Role: RoleOwner, Human: true, DigestOK: false, Now: 1}, "R12", "reply:changed"),
|
||||
st(Confirm{Actor: "alice", Role: RoleOwner, Human: true, DigestOK: true, Now: 10}, "R13", "persist:expired", "reply:expired"),
|
||||
},
|
||||
Final: Expired{P: w},
|
||||
},
|
||||
{
|
||||
Name: "expiry-seen-at-admission",
|
||||
About: "the clock passes expiry while the confirmation waits for admission: Expired, nothing runs",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(AdmitGranted{Now: 10, RuntimeGen: 1}, "R30", "persist:expired", "reply:expired"),
|
||||
},
|
||||
Final: Expired{P: w},
|
||||
},
|
||||
{
|
||||
Name: "cancel-pending",
|
||||
About: "the proposer cancels a pending proposal; an owner resolution and a confirm of it are closed",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(ownerResolve(), "R49", "reply:closed"),
|
||||
st(Cancel{Actor: "alice"}, "R24", "persist:cancelled", "reply:cancelled"),
|
||||
st(Cancel{Actor: "alice"}, "R27", "reply:cancelled"),
|
||||
st(confirm(), "R20", "reply:closed"),
|
||||
},
|
||||
Final: Cancelled{P: w, Why: CancelByProposer},
|
||||
},
|
||||
{
|
||||
Name: "authority-lowered-while-pending",
|
||||
About: "the proposer loses authority: the pending proposal is cancelled",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(AuthorityLowered{}, "R54", "persist:cancelled"),
|
||||
},
|
||||
Final: Cancelled{P: w, Why: CancelByAuthority},
|
||||
},
|
||||
{
|
||||
Name: "authority-lowered-before-send",
|
||||
About: "authority is lowered after executing is persisted but before the send: the dispatch is cancelled, nothing was sent",
|
||||
Steps: []Step{
|
||||
st(propose(RoleOwner, KindWrite), "R8", "persist:pending", "reply:created"),
|
||||
st(confirm(), "R15", "admit:write"),
|
||||
st(granted(), "R32", "persist:executing", "dispatch"),
|
||||
st(AuthorityLowered{}, "R55", "cancel-dispatch:authority-lowered"),
|
||||
st(NotSent{NotSentAuthority}, "R35", "persist:cancelled", "reply:cancelled"),
|
||||
},
|
||||
Final: Cancelled{P: w, Why: CancelByAuthority},
|
||||
},
|
||||
{
|
||||
Name: "authority-lowered-after-send",
|
||||
About: "authority is lowered while the request is on the wire: the call is aborted, the outcome is unknown, and a rejected receipt settles it",
|
||||
Steps: with(toSent(),
|
||||
st(AuthorityLowered{}, "R56", "cancel-send"),
|
||||
st(ApplyResult{ApplyTransportError}, "R39", "persist:outcome-unknown", "reply:outcome-unknown"),
|
||||
st(ReceiptResult{ReceiptRejected}, "R41", "persist:rejected"),
|
||||
st(ownerResolve(), "R50", "reply:resolved"),
|
||||
),
|
||||
Final: Rejected{P: w, Why: RejectReceipt},
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
package proposalflow
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
var _ = ginkgo.Describe("the scenario traces", func() {
|
||||
for _, tr := range Traces() {
|
||||
ginkgo.It("holds trace "+tr.Name, func() {
|
||||
Expect(tr.Verify()).To(BeEmpty())
|
||||
})
|
||||
}
|
||||
|
||||
ginkgo.It("has unique trace names", func() {
|
||||
seen := map[string]bool{}
|
||||
for _, tr := range Traces() {
|
||||
Expect(seen[tr.Name]).To(BeFalse(), "duplicate trace %s", tr.Name)
|
||||
seen[tr.Name] = true
|
||||
}
|
||||
})
|
||||
|
||||
// This names the scenarios the model must keep.
|
||||
ginkgo.It("keeps the required scenarios", func() {
|
||||
want := []string{
|
||||
"happy-path", "retry-after-not-sent", "retry-after-admission-refused", "timeout-unknown-then-receipt-committed",
|
||||
"unknown-bounded-then-manual-resolution", "two-actors-racing-confirm", "hold-during-prepared-dispatch",
|
||||
"hold-during-sent-request", "generation-change-while-pending", "duplicate-confirm-after-applied",
|
||||
"editor-schema-propose-refused", "out-of-order-receipt",
|
||||
}
|
||||
have := map[string]bool{}
|
||||
for _, tr := range Traces() {
|
||||
have[tr.Name] = true
|
||||
}
|
||||
for _, n := range want {
|
||||
Expect(have[n]).To(BeTrue(), "missing trace %s", n)
|
||||
}
|
||||
})
|
||||
|
||||
// The traces and the search together fire every rule id, so no rule is dead.
|
||||
ginkgo.It("fires every rule", func() {
|
||||
fired := map[string]bool{}
|
||||
for _, tr := range Traces() {
|
||||
for _, r := range Run(Unproposed{}, tr.Events()) {
|
||||
fired[r.Rule] = true
|
||||
}
|
||||
}
|
||||
for id := range Explore().RuleHits {
|
||||
fired[id] = true
|
||||
}
|
||||
var dead []string
|
||||
for _, r := range Rules {
|
||||
if !fired[r.ID] {
|
||||
dead = append(dead, r.ID)
|
||||
}
|
||||
}
|
||||
sort.Strings(dead)
|
||||
Expect(dead).To(BeEmpty(), "rules that never fire")
|
||||
})
|
||||
|
||||
// The hand-written traces, not only the search, exercise each transition
|
||||
// rule. A rule with no named trace is reported, not failed.
|
||||
ginkgo.It("reports transition rules that only the search fires", func() {
|
||||
fired := map[string]bool{}
|
||||
for _, tr := range Traces() {
|
||||
for _, r := range Run(Unproposed{}, tr.Events()) {
|
||||
fired[r.Rule] = true
|
||||
}
|
||||
}
|
||||
var missing []string
|
||||
for _, r := range Rules {
|
||||
if r.Kind == KindTransition && !fired[r.ID] {
|
||||
missing = append(missing, r.ID)
|
||||
}
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
_, _ = fmt.Fprintf(ginkgo.GinkgoWriter, "transition rules with no named trace (covered by the search only): %v\n", missing)
|
||||
}
|
||||
})
|
||||
})
|
||||
File diff suppressed because one or more lines are too long.
@@ -0,0 +1,311 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
v1 "github.com/google/go-containerregistry/pkg/v1"
|
||||
"github.com/google/go-containerregistry/pkg/v1/empty"
|
||||
"github.com/google/go-containerregistry/pkg/v1/layout"
|
||||
"github.com/google/go-containerregistry/pkg/v1/mutate"
|
||||
"github.com/google/go-containerregistry/pkg/v1/tarball"
|
||||
"github.com/google/go-containerregistry/pkg/v1/types"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
)
|
||||
|
||||
// hostFor is the compatibility the package is validated against. The numbers
|
||||
// are the ones the host runtime uses today; a package whose manifest range
|
||||
// does not contain them is not installable and must not be published.
|
||||
func hostFor(platform string) contract.HostCompatibility {
|
||||
os_, arch, _ := strings.Cut(platform, "/")
|
||||
return contract.HostCompatibility{Host: 1, API: 1, Protocol: 1, SDK: contract.HostSDKVersion, DataFormat: 1, OS: os_, Arch: arch}
|
||||
}
|
||||
|
||||
type fileEntry struct {
|
||||
name string
|
||||
data []byte
|
||||
mode int64
|
||||
}
|
||||
|
||||
// tarLayer writes entries in sorted order with fixed metadata so the same
|
||||
// inputs always give the same digest.
|
||||
func tarLayer(entries []fileEntry, role string) (v1.Layer, error) {
|
||||
sort.Slice(entries, func(i, j int) bool { return entries[i].name < entries[j].name })
|
||||
var b bytes.Buffer
|
||||
w := tar.NewWriter(&b)
|
||||
for _, e := range entries {
|
||||
h := &tar.Header{Typeflag: tar.TypeReg, Name: e.name, Mode: e.mode, Size: int64(len(e.data)), ModTime: time.Unix(0, 0), Format: tar.FormatPAX}
|
||||
if err := w.WriteHeader(h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, err := w.Write(e.data); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
data := b.Bytes()
|
||||
return tarball.LayerFromOpener(func() (io.ReadCloser, error) { return io.NopCloser(bytes.NewReader(data)), nil }, tarball.WithMediaType(types.OCILayer))
|
||||
}
|
||||
|
||||
// rawImage replaces the manifest bytes of an image, which the mutate package
|
||||
// cannot do for artifactType.
|
||||
type rawImage struct {
|
||||
v1.Image
|
||||
raw []byte
|
||||
}
|
||||
|
||||
func (r rawImage) RawManifest() ([]byte, error) { return r.raw, nil }
|
||||
func (r rawImage) Manifest() (*v1.Manifest, error) {
|
||||
return v1.ParseManifest(bytes.NewReader(r.raw))
|
||||
}
|
||||
func (r rawImage) Digest() (v1.Hash, error) {
|
||||
h, _, err := v1.SHA256(bytes.NewReader(r.raw))
|
||||
return h, err
|
||||
}
|
||||
func (r rawImage) Size() (int64, error) { return int64(len(r.raw)), nil }
|
||||
|
||||
func stampManifest(template []byte, version, platform string) ([]byte, error) {
|
||||
var m map[string]json.RawMessage
|
||||
if err := json.Unmarshal(template, &m); err != nil {
|
||||
return nil, fmt.Errorf("manifest template: %w", err)
|
||||
}
|
||||
os_, arch, _ := strings.Cut(platform, "/")
|
||||
for k, v := range map[string]string{"version": version, "os": os_, "arch": arch} {
|
||||
b, _ := json.Marshal(v)
|
||||
m[k] = b
|
||||
}
|
||||
out, err := json.MarshalIndent(m, "", " ")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return append(out, '\n'), nil
|
||||
}
|
||||
|
||||
func readRegular(p string) ([]byte, error) {
|
||||
info, err := os.Lstat(p)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return nil, fmt.Errorf("%s is not a regular file", p)
|
||||
}
|
||||
return os.ReadFile(p) // #nosec G304 -- the packaging tool reads paths the operator passes on its command line
|
||||
}
|
||||
|
||||
func cmdBuild(args []string) error {
|
||||
fs_ := flag.NewFlagSet("build", flag.ContinueOnError)
|
||||
out := fs_.String("layout", "", "output OCI layout directory (must not exist or be empty)")
|
||||
version := fs_.String("version", "", "release version, semver without the v prefix")
|
||||
platform := fs_.String("platform", "", "os/arch")
|
||||
config := fs_.String("config", "addons/tables/package/platforms.yaml", "platform matrix file")
|
||||
exe := fs_.String("executable", "", "built executable of the platform")
|
||||
runtimeDir := fs_.String("runtime-dir", "", "directory that holds the runtime files of the platform")
|
||||
revision := fs_.String("revision", "", "source commit, 40 hex characters")
|
||||
local := fs_.Bool("local-check", false, "allow a platform that is not enabled; the layout is marked unpublishable")
|
||||
root := fs_.String("root", ".", "repository root that the matrix paths are relative to")
|
||||
if err := fs_.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if *out == "" || *exe == "" {
|
||||
return errors.New("--layout and --executable are required")
|
||||
}
|
||||
if !versionRE.MatchString(*version) {
|
||||
return fmt.Errorf("version %q is not a semantic version without a v prefix or build metadata", *version)
|
||||
}
|
||||
if !platformRE.MatchString(*platform) {
|
||||
return fmt.Errorf("platform %q is not os/arch of linux or darwin on amd64 or arm64", *platform)
|
||||
}
|
||||
if *revision != "" && !shaRE.MatchString(*revision) {
|
||||
return errors.New("revision must be 40 lower-case hex characters")
|
||||
}
|
||||
cfg, err := loadPlatforms(*config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
entry, ok := cfg.find(*platform)
|
||||
if !ok {
|
||||
return fmt.Errorf("platform %s is not in the platform matrix", *platform)
|
||||
}
|
||||
if !entry.Enabled && !*local {
|
||||
return fmt.Errorf("platform %s is not enabled in the platform matrix", *platform)
|
||||
}
|
||||
if st, err := os.Stat(*out); err == nil {
|
||||
ents, _ := os.ReadDir(*out)
|
||||
if !st.IsDir() || len(ents) > 0 {
|
||||
return fmt.Errorf("layout %s exists and is not empty", *out)
|
||||
}
|
||||
}
|
||||
template, err := readRegular(filepath.Join(*root, cfg.Manifest))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
manifest, err := stampManifest(template, *version, *platform)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
m, err := contract.ParseManifest(manifest)
|
||||
if err != nil {
|
||||
return fmt.Errorf("manifest: %w", err)
|
||||
}
|
||||
if err := contract.ValidateManifest(m, hostFor(*platform)); err != nil {
|
||||
return fmt.Errorf("manifest: %w", err)
|
||||
}
|
||||
if m.ID != cfg.Addon || m.Executable != cfg.Executable {
|
||||
return errors.New("manifest id or executable differs from the platform matrix")
|
||||
}
|
||||
exeBytes, err := readRegular(*exe)
|
||||
if err != nil || len(exeBytes) == 0 {
|
||||
return fmt.Errorf("executable %s: missing, empty or not a regular file", *exe)
|
||||
}
|
||||
files := []fileEntry{{name: m.Executable, data: exeBytes, mode: 0o755}}
|
||||
for _, r := range entry.Runtime {
|
||||
if *runtimeDir == "" {
|
||||
return fmt.Errorf("runtime file %s is required by the platform and --runtime-dir is not set", r)
|
||||
}
|
||||
b, err := readRegular(filepath.Join(*runtimeDir, filepath.FromSlash(r)))
|
||||
if err != nil {
|
||||
return fmt.Errorf("runtime file %s: %w", r, err)
|
||||
}
|
||||
files = append(files, fileEntry{name: r, data: b, mode: 0o644})
|
||||
}
|
||||
type role struct {
|
||||
name string
|
||||
entries []fileEntry
|
||||
}
|
||||
roles := []role{{roleExecutable, files}}
|
||||
if u := m.Capabilities.UI; u != nil {
|
||||
if cfg.UI == "" {
|
||||
return errors.New("the manifest declares a UI and the platform matrix names no ui directory")
|
||||
}
|
||||
ui, err := collectUI(filepath.Join(*root, cfg.UI), path.Dir(u.Entry))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
declared := append([]string{u.Entry}, u.CSS...)
|
||||
if u.Icon != "" {
|
||||
declared = append(declared, u.Icon)
|
||||
}
|
||||
have := map[string]bool{}
|
||||
for _, f := range ui {
|
||||
have[f.name] = true
|
||||
}
|
||||
for _, d := range declared {
|
||||
if !have[d] {
|
||||
return fmt.Errorf("declared UI asset %s is missing from %s", d, cfg.UI)
|
||||
}
|
||||
}
|
||||
roles = append(roles, role{roleUI, ui})
|
||||
}
|
||||
roles = append(roles, role{roleManifest, []fileEntry{{name: "manifest.json", data: manifest, mode: 0o644}}})
|
||||
|
||||
img := mutate.ConfigMediaType(mutate.MediaType(empty.Image, types.OCIManifestSchema1), types.OCIConfigJSON)
|
||||
for _, r := range roles {
|
||||
l, err := tarLayer(r.entries, r.name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
img, err = mutate.Append(img, mutate.Addendum{Layer: l, Annotations: map[string]string{AnnotationLayer: r.name}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
osName, arch, _ := strings.Cut(*platform, "/")
|
||||
cf, err := img.ConfigFile()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cf.OS, cf.Architecture, cf.Created = osName, arch, v1.Time{}
|
||||
cf.History = nil
|
||||
if img, err = mutate.ConfigFile(img, cf); err != nil {
|
||||
return err
|
||||
}
|
||||
mf, err := img.Manifest()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
mf.ArtifactType = ArtifactType
|
||||
mf.Annotations = map[string]string{AnnotationID: m.ID, AnnotationVersion: *version, AnnotationOCIVer: *version, AnnotationPlatform: *platform}
|
||||
if *revision != "" {
|
||||
mf.Annotations[AnnotationRevision] = *revision
|
||||
}
|
||||
raw, err := json.Marshal(mf)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
final := rawImage{Image: img, raw: raw}
|
||||
digest, err := final.Digest()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(*out, 0o750); err != nil {
|
||||
return err
|
||||
}
|
||||
lp, err := layout.Write(*out, empty.Index)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := lp.AppendImage(final, layout.WithPlatform(v1.Platform{OS: osName, Architecture: arch}), layout.WithAnnotations(map[string]string{"org.opencontainers.image.ref.name": *version})); err != nil {
|
||||
return err
|
||||
}
|
||||
r := Receipt{Schema: 1, Addon: m.ID, Version: *version, Platform: *platform, Revision: *revision, Publishable: entry.Enabled && !*local, ManifestDigest: digest.String(), ConfigDigest: mf.Config.Digest.String()}
|
||||
for i, d := range mf.Layers {
|
||||
r.Layers = append(r.Layers, ReceiptLayer{Role: roles[i].name, Digest: d.Digest.String(), Size: d.Size, MediaType: string(d.MediaType)})
|
||||
}
|
||||
rb, err := json.MarshalIndent(r, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(*out, receiptFile), append(rb, '\n'), 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = fmt.Fprintf(stdout, "digest=%s\nplatform=%s\nversion=%s\npublishable=%t\n", digest, *platform, *version, r.Publishable)
|
||||
return nil
|
||||
}
|
||||
|
||||
// collectUI reads every file under the UI directory that sits below prefix
|
||||
// (the top directory of the declared entry). Links and special files are
|
||||
// refused so a build output cannot smuggle content from outside the tree.
|
||||
func collectUI(dir, prefix string) ([]fileEntry, error) {
|
||||
top := filepath.Join(dir, filepath.FromSlash(prefix))
|
||||
var out []fileEntry
|
||||
err := filepath.WalkDir(top, func(p string, d fs.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if d.IsDir() {
|
||||
return nil
|
||||
}
|
||||
if !d.Type().IsRegular() {
|
||||
return fmt.Errorf("UI file %s is not a regular file", p)
|
||||
}
|
||||
rel, err := filepath.Rel(dir, p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := os.ReadFile(p) // #nosec G122,G304 -- the tree is owned by this process and locked; the walk only normalises its own files
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
out = append(out, fileEntry{name: filepath.ToSlash(rel), data: b, mode: 0o644})
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("UI directory: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"github.com/google/go-containerregistry/pkg/v1"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
var ownerRepoRE = regexp.MustCompile(`^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$`)
|
||||
|
||||
// catalogRelease is the shape of one catalog entry the host reads. It repeats
|
||||
// the JSON names of catalog.Release on purpose: the generated file is checked
|
||||
// against the names, not against this program's idea of them.
|
||||
type catalogRelease struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description"`
|
||||
Version string `json:"version"`
|
||||
Channel string `json:"channel"`
|
||||
Artifact string `json:"artifact"`
|
||||
Digest string `json:"digest"`
|
||||
OS string `json:"os"`
|
||||
Arch string `json:"arch"`
|
||||
Policy struct {
|
||||
RequireSignature bool `json:"requireSignature"`
|
||||
Issuer string `json:"issuer"`
|
||||
Identity string `json:"identity"`
|
||||
} `json:"policy"`
|
||||
UI struct {
|
||||
Title string `json:"title"`
|
||||
Description string `json:"description"`
|
||||
MainPage string `json:"mainPage"`
|
||||
Icon string `json:"icon"`
|
||||
} `json:"ui"`
|
||||
}
|
||||
|
||||
// signerIdentity is the exact certificate identity of the add-on package
|
||||
// workflow for one ref. It names this workflow file and no other, so a
|
||||
// signature of the backend workflows never satisfies an add-on policy.
|
||||
func signerIdentity(ownerRepo, ref string) (string, error) {
|
||||
if !ownerRepoRE.MatchString(ownerRepo) {
|
||||
return "", fmt.Errorf("owner/repository %q is invalid", ownerRepo)
|
||||
}
|
||||
if err := validRef(ref); err != nil || !(strings.HasPrefix(ref, "refs/heads/") || strings.HasPrefix(ref, "refs/tags/")) {
|
||||
return "", fmt.Errorf("signer ref %q is invalid", ref)
|
||||
}
|
||||
return "https://github.com/" + ownerRepo + "/.github/workflows/" + SignerWorkflow + "@" + ref, nil
|
||||
}
|
||||
|
||||
func jsonString(s string) string {
|
||||
b, _ := json.Marshal(s)
|
||||
return string(b[1 : len(b)-1])
|
||||
}
|
||||
|
||||
// cmdCatalog renders the catalog entries of one merged layout from a
|
||||
// template. Every value comes from a verified layout or an explicit input;
|
||||
// nothing is looked up from a registry, so the entries pin the digests that
|
||||
// were built and verified.
|
||||
func cmdCatalog(args []string) error {
|
||||
fs := flag.NewFlagSet("catalog", flag.ContinueOnError)
|
||||
template := fs.String("template", "gallery/addons/tables.yaml.tmpl", "entry template")
|
||||
dir := fs.String("merged", "", "merged layout")
|
||||
config := fs.String("config", "", "platform matrix file")
|
||||
repository := fs.String("repository", "", "registry and repository the package was published to")
|
||||
tag := fs.String("tag", "", "immutable tag the artifact reference names: the version or sha-<commit>")
|
||||
channel := fs.String("channel", "", "channel name the entries are current for: latest, development or a branch channel")
|
||||
ownerRepo := fs.String("owner-repo", "", "GitHub owner/repository that ran the workflow")
|
||||
signerRef := fs.String("signer-ref", "", "ref of the signing workflow run")
|
||||
out := fs.String("out", "", "output file")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
repo, err := parseRepository(*repository)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !immutableTagRE.MatchString(*tag) {
|
||||
return fmt.Errorf("--tag %q is not an immutable version or commit tag", *tag)
|
||||
}
|
||||
if *dir == "" || *out == "" || *channel == "" {
|
||||
return errors.New("--merged, --channel and --out are required")
|
||||
}
|
||||
if *channel != aliasLatest && *channel != aliasDevelopment && !branchAliasRE.MatchString(*channel) {
|
||||
return fmt.Errorf("channel %q is not a channel alias", *channel)
|
||||
}
|
||||
identity, err := signerIdentity(*ownerRepo, *signerRef)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmpl, err := os.ReadFile(*template)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cfg, err := loadPlatforms(*config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
m, _, err := verifyMerged(*dir, cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
root, err := readRootIndex(*dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
raw, err := readBlob(*dir, root.Manifests[0].Digest, root.Manifests[0].Size)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var im v1.IndexManifest
|
||||
if err := json.Unmarshal(raw, &im); err != nil {
|
||||
return err
|
||||
}
|
||||
var releases []catalogRelease
|
||||
var doc bytes.Buffer
|
||||
for _, d := range im.Manifests {
|
||||
l, err := loadLegByDescriptor(*dir, d)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
platform := d.Platform.OS + "/" + d.Platform.Architecture
|
||||
var rec Receipt
|
||||
for _, r := range m.Legs {
|
||||
if r.Platform == platform {
|
||||
rec = r
|
||||
}
|
||||
}
|
||||
v, err := verifyLoaded(l, &rec, platform, m.Version, cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ui := v.Manifest.Capabilities.UI
|
||||
title, description, mainPage, icon := v.Manifest.ID, "", "", ""
|
||||
if ui != nil {
|
||||
mainPage, icon = ui.MainPage, ui.Icon
|
||||
if ui.Title != nil {
|
||||
title = ui.Title.Default()
|
||||
}
|
||||
if ui.Description != nil {
|
||||
description = ui.Description.Default()
|
||||
}
|
||||
}
|
||||
values := map[string]string{
|
||||
"addon": v.Manifest.ID, "version": m.Version, "tag": *tag, "channel": *channel,
|
||||
"repository": repo.Name(), "digest": d.Digest.String(),
|
||||
"os": d.Platform.OS, "arch": d.Platform.Architecture,
|
||||
"issuer": OIDCIssuer, "identity": identity,
|
||||
"title": title, "description": description, "main_page": mainPage, "icon": icon,
|
||||
}
|
||||
text := string(tmpl)
|
||||
for k, val := range values {
|
||||
text = strings.ReplaceAll(text, "{{"+k+"}}", jsonString(val))
|
||||
}
|
||||
if strings.Contains(text, "{{") {
|
||||
return fmt.Errorf("template has an unknown placeholder near %q", text[strings.Index(text, "{{"):min(len(text), strings.Index(text, "{{")+40)])
|
||||
}
|
||||
item := "- " + strings.ReplaceAll(strings.TrimRight(text, "\n"), "\n", "\n ") + "\n"
|
||||
parsed, err := readCatalogIndex([]byte(item))
|
||||
if err != nil || len(parsed) != 1 {
|
||||
return fmt.Errorf("rendered entry: %v", err)
|
||||
}
|
||||
r := parsed[0]
|
||||
if err := checkRelease(r, m.Version, d.Digest.String(), repo.Name(), *tag); err != nil {
|
||||
return err
|
||||
}
|
||||
releases = append(releases, r)
|
||||
doc.WriteString(item)
|
||||
}
|
||||
if _, err := readCatalogIndex(doc.Bytes()); err != nil {
|
||||
return fmt.Errorf("rendered index: %w", err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(*out), 0o750); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(*out, doc.Bytes(), 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = fmt.Fprintf(stdout, "entries=%d\nchannel=%s\n", len(releases), *channel)
|
||||
return nil
|
||||
}
|
||||
|
||||
// readCatalogIndex reads a generated library index: one YAML document that is
|
||||
// a block list of entries, with no unknown field. It is a second, independent
|
||||
// reader of the format on purpose; the host has its own, and the integration
|
||||
// test makes the host's reader judge the generated file.
|
||||
func readCatalogIndex(b []byte) ([]catalogRelease, error) {
|
||||
dec := yaml.NewDecoder(bytes.NewReader(b))
|
||||
var doc yaml.Node
|
||||
if err := dec.Decode(&doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := dec.Decode(new(yaml.Node)); !errors.Is(err, io.EOF) {
|
||||
return nil, errors.New("index holds more than one document")
|
||||
}
|
||||
if doc.Kind != yaml.DocumentNode || len(doc.Content) != 1 || doc.Content[0].Kind != yaml.SequenceNode || doc.Content[0].Style&yaml.FlowStyle != 0 {
|
||||
return nil, errors.New("index must be a YAML block list of entries")
|
||||
}
|
||||
var plain any
|
||||
if err := doc.Content[0].Decode(&plain); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
raw, err := json.Marshal(plain)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []catalogRelease
|
||||
if err := strictJSON(raw, &out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// checkRelease is the template contract: an entry pins the digest that was
|
||||
// built, names the package repository, and requires a signature of the
|
||||
// add-on workflow.
|
||||
func checkRelease(r catalogRelease, version, digest, repository, tag string) error {
|
||||
switch {
|
||||
case r.Digest != digest || !digestRE.MatchString(r.Digest):
|
||||
return errors.New("entry digest differs from the verified image digest")
|
||||
case r.Version != version:
|
||||
return errors.New("entry version differs from the package version")
|
||||
case r.Artifact != "oci://"+repository+":"+tag:
|
||||
return fmt.Errorf("entry artifact %q must be oci://%s:%s", r.Artifact, repository, tag)
|
||||
case !r.Policy.RequireSignature || r.Policy.Issuer != OIDCIssuer:
|
||||
return errors.New("entry must require a signature of the GitHub Actions OIDC issuer")
|
||||
case !strings.Contains(r.Policy.Identity, "/.github/workflows/"+SignerWorkflow+"@refs/"):
|
||||
return errors.New("entry identity must name the add-on package workflow")
|
||||
case r.OS == "" || r.Arch == "" || r.ID == "" || r.Channel == "":
|
||||
return errors.New("entry needs addon id, os, arch and channel")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// Channel and tag namespaces. They are disjoint by construction:
|
||||
//
|
||||
// latest, development reserved aliases, moved only by their own rule
|
||||
// <semver> immutable release version
|
||||
// sha-<12 hex> immutable source commit
|
||||
// branch-<label>-<hash12> branch channels, one per full original ref
|
||||
//
|
||||
// A branch tag always starts with "branch-", and no reserved alias, version
|
||||
// or commit tag does, so a branch name can never take over one of them.
|
||||
const (
|
||||
aliasLatest = "latest"
|
||||
aliasDevelopment = "development"
|
||||
developmentRef = "refs/heads/master"
|
||||
branchPrefix = "branch-"
|
||||
maxLabel = 40
|
||||
maxRef = 4096
|
||||
)
|
||||
|
||||
// Decision is what an event may publish.
|
||||
type Decision struct {
|
||||
Publish bool
|
||||
Reason string
|
||||
Immutable string // version or commit tag, never moves
|
||||
Aliases []string // moving tags, promoted only after signing
|
||||
Channel string // serialization key of the alias promotion
|
||||
SignerRef string // ref the signing identity carries
|
||||
Version string // release version when the event carries one
|
||||
}
|
||||
|
||||
// branchLabel normalizes a branch name the way metadata-action does for a
|
||||
// ref tag (lower case, characters outside the tag alphabet become a dash),
|
||||
// but only ever to a bounded, readable label. The label is never trusted for
|
||||
// identity: the hash of the full original ref is.
|
||||
func branchLabel(name string) string {
|
||||
var b strings.Builder
|
||||
dash := true
|
||||
for _, r := range strings.ToLower(name) {
|
||||
if r >= 'a' && r <= 'z' || r >= '0' && r <= '9' {
|
||||
b.WriteRune(r)
|
||||
dash = false
|
||||
continue
|
||||
}
|
||||
if !dash {
|
||||
b.WriteByte('-')
|
||||
dash = true
|
||||
}
|
||||
}
|
||||
label := strings.Trim(b.String(), "-")
|
||||
if len(label) > maxLabel {
|
||||
label = strings.Trim(label[:maxLabel], "-")
|
||||
}
|
||||
if label == "" {
|
||||
label = "x"
|
||||
}
|
||||
return label
|
||||
}
|
||||
|
||||
// devVersion is the version stamped into packages built from a branch. It is
|
||||
// a pre-release of 0.0.0, so it can never be taken for a release.
|
||||
func devVersion(sha string) string { return "0.0.0-dev." + sha[:12] }
|
||||
|
||||
func branchTag(ref string) string {
|
||||
sum := sha256.Sum256([]byte(ref))
|
||||
return branchPrefix + branchLabel(strings.TrimPrefix(ref, "refs/heads/")) + "-" + hex.EncodeToString(sum[:])[:12]
|
||||
}
|
||||
|
||||
func validRef(ref string) error {
|
||||
if ref == "" {
|
||||
return errors.New("empty ref")
|
||||
}
|
||||
if len(ref) > maxRef || !utf8.ValidString(ref) {
|
||||
return errors.New("ref is too long or not UTF-8")
|
||||
}
|
||||
for _, r := range ref {
|
||||
if r < 0x20 || r == 0x7f {
|
||||
return errors.New("ref holds a control character")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// decide applies the event policy. Pull requests and branch pushes never
|
||||
// publish, so a branch name cannot reach a registry. The default branch alone
|
||||
// publishes development. A tag push prepares the immutable version and never
|
||||
// moves an alias. Only a published, non-draft, stable release moves latest.
|
||||
func decide(event, ref, sha string, draft, prerelease bool) (Decision, error) {
|
||||
if err := validRef(ref); err != nil {
|
||||
return Decision{}, err
|
||||
}
|
||||
if !shaRE.MatchString(sha) {
|
||||
return Decision{}, errors.New("sha must be 40 lower-case hex characters")
|
||||
}
|
||||
commit := "sha-" + sha[:12]
|
||||
switch event {
|
||||
case "pull_request", "pull_request_target":
|
||||
return Decision{Reason: "pull request validation never publishes"}, nil
|
||||
case "push":
|
||||
switch {
|
||||
case ref == developmentRef:
|
||||
return Decision{Publish: true, Reason: "default branch publishes development", Immutable: commit, Aliases: []string{aliasDevelopment}, Channel: aliasDevelopment, SignerRef: ref, Version: devVersion(sha)}, nil
|
||||
case strings.HasPrefix(ref, "refs/heads/") && len(ref) > len("refs/heads/"):
|
||||
return Decision{Reason: "branch push never publishes; only master publishes development"}, nil
|
||||
case strings.HasPrefix(ref, "refs/tags/"):
|
||||
v, err := tagVersion(ref)
|
||||
if err != nil {
|
||||
return Decision{}, err
|
||||
}
|
||||
return Decision{Publish: true, Reason: "tag push prepares the immutable version only", Immutable: v, Channel: "version-" + v, SignerRef: ref, Version: v}, nil
|
||||
}
|
||||
return Decision{}, fmt.Errorf("unsupported ref %q", ref)
|
||||
case "release":
|
||||
if draft {
|
||||
return Decision{}, errors.New("a draft release never publishes")
|
||||
}
|
||||
v, err := tagVersion(ref)
|
||||
if err != nil {
|
||||
return Decision{}, err
|
||||
}
|
||||
d := Decision{Publish: true, Reason: "published release", Immutable: v, SignerRef: ref, Version: v, Channel: "version-" + v}
|
||||
if prerelease || strings.Contains(v, "-") {
|
||||
d.Reason = "pre-release prepares the immutable version only"
|
||||
return d, nil
|
||||
}
|
||||
d.Aliases = []string{aliasLatest}
|
||||
d.Channel = aliasLatest
|
||||
d.Reason = "stable release promotes latest after complete publication"
|
||||
return d, nil
|
||||
}
|
||||
return Decision{}, fmt.Errorf("event %q is not allowed to publish", event)
|
||||
}
|
||||
|
||||
func tagVersion(ref string) (string, error) {
|
||||
t, ok := strings.CutPrefix(ref, "refs/tags/")
|
||||
if !ok {
|
||||
return "", fmt.Errorf("%q is not a tag ref", ref)
|
||||
}
|
||||
v, ok := strings.CutPrefix(t, "v")
|
||||
if !ok || !versionRE.MatchString(v) {
|
||||
return "", fmt.Errorf("tag %q is not a v<semver> release tag", t)
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func cmdChannel(args []string) error {
|
||||
fs := flag.NewFlagSet("channel", flag.ContinueOnError)
|
||||
event := fs.String("event", "", "GitHub event name")
|
||||
ref := fs.String("ref", "", "full ref of the event")
|
||||
sha := fs.String("sha", "", "source commit")
|
||||
draft := fs.Bool("draft", false, "the release is a draft")
|
||||
pre := fs.Bool("prerelease", false, "the release is marked pre-release")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
d, err := decide(*event, *ref, *sha, *draft, *pre)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = fmt.Fprintf(stdout, "publish=%t\nreason=%s\nimmutable=%s\naliases=%s\nchannel=%s\nsigner_ref=%s\nversion=%s\n",
|
||||
d.Publish, d.Reason, d.Immutable, strings.Join(d.Aliases, ","), d.Channel, d.SignerRef, d.Version)
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// Constants of the package format. Anything that reads a package compares
|
||||
// against these, never against a value it found in the package.
|
||||
const (
|
||||
// ArtifactType marks an OCI manifest as a LocalAI add-on package.
|
||||
ArtifactType = "application/vnd.localai.addon.package.v1"
|
||||
// IndexArtifactType marks the image index that joins the platform images.
|
||||
IndexArtifactType = "application/vnd.localai.addon.package.index.v1"
|
||||
|
||||
AnnotationID = "io.localai.addon.id"
|
||||
AnnotationVersion = "io.localai.addon.version"
|
||||
AnnotationPlatform = "io.localai.addon.platform"
|
||||
AnnotationLayer = "io.localai.addon.layer"
|
||||
AnnotationRevision = "org.opencontainers.image.revision"
|
||||
AnnotationOCIVer = "org.opencontainers.image.version"
|
||||
|
||||
// SignerWorkflow is the workflow file whose OIDC identity signs packages.
|
||||
// It is deliberately not the backend merge workflow.
|
||||
SignerWorkflow = "addons-package.yml"
|
||||
OIDCIssuer = "https://token.actions.githubusercontent.com"
|
||||
)
|
||||
|
||||
var (
|
||||
versionRE = regexp.MustCompile(`^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-[0-9A-Za-z]+(\.[0-9A-Za-z-]+)*)?$`)
|
||||
platformRE = regexp.MustCompile(`^(linux|darwin)/(amd64|arm64)$`)
|
||||
addonRE = regexp.MustCompile(`^[a-z][a-z0-9-]{0,62}$`)
|
||||
shaRE = regexp.MustCompile(`^[0-9a-f]{40}$`)
|
||||
digestRE = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
|
||||
)
|
||||
|
||||
// PlatformEntry is one row of the release platform matrix. A platform ships
|
||||
// only while Enabled is true, and it is enabled only after its native runtime
|
||||
// check passes on a runner of that platform.
|
||||
type PlatformEntry struct {
|
||||
Platform string `yaml:"platform"`
|
||||
Enabled bool `yaml:"enabled"`
|
||||
Runner string `yaml:"runner"`
|
||||
NativeCheck string `yaml:"native_check"`
|
||||
Runtime []string `yaml:"runtime"`
|
||||
}
|
||||
|
||||
// PlatformsFile is addons/<id>/package/platforms.yaml.
|
||||
type PlatformsFile struct {
|
||||
Schema int `yaml:"schema"`
|
||||
Addon string `yaml:"addon"`
|
||||
Executable string `yaml:"executable"`
|
||||
Manifest string `yaml:"manifest"`
|
||||
UI string `yaml:"ui"`
|
||||
Platforms []PlatformEntry `yaml:"platforms"`
|
||||
}
|
||||
|
||||
// NativeCheckKind is the only native check that exists: the packaged
|
||||
// executable is extracted on the native runner and started.
|
||||
const NativeCheckKind = "executable-start"
|
||||
|
||||
var errNoPlatforms = errors.New("the release platform matrix has no enabled platform; enable a platform after its native check passes")
|
||||
|
||||
func loadPlatforms(file string) (*PlatformsFile, error) {
|
||||
if file == "" {
|
||||
return nil, errors.New("platform matrix file is required")
|
||||
}
|
||||
b, err := os.ReadFile(file) // #nosec G304 -- the packaging tool reads paths the operator passes on its command line
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("platform matrix: %w", err)
|
||||
}
|
||||
dec := yaml.NewDecoder(bytes.NewReader(b))
|
||||
dec.KnownFields(true)
|
||||
var p PlatformsFile
|
||||
if err := dec.Decode(&p); err != nil {
|
||||
return nil, fmt.Errorf("platform matrix: %w", err)
|
||||
}
|
||||
if p.Schema != 1 {
|
||||
return nil, fmt.Errorf("platform matrix: unsupported schema %d", p.Schema)
|
||||
}
|
||||
if !addonRE.MatchString(p.Addon) {
|
||||
return nil, fmt.Errorf("platform matrix: invalid addon id %q", p.Addon)
|
||||
}
|
||||
for _, f := range []string{p.Executable, p.Manifest} {
|
||||
if f == "" || !localPath(f) {
|
||||
return nil, fmt.Errorf("platform matrix: executable and manifest need a local path, got %q", f)
|
||||
}
|
||||
}
|
||||
if p.UI != "" && !localPath(p.UI) {
|
||||
return nil, fmt.Errorf("platform matrix: ui must be a local path, got %q", p.UI)
|
||||
}
|
||||
if len(p.Platforms) == 0 {
|
||||
return nil, errNoPlatforms
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, e := range p.Platforms {
|
||||
if !platformRE.MatchString(e.Platform) {
|
||||
return nil, fmt.Errorf("platform matrix: unsupported platform %q", e.Platform)
|
||||
}
|
||||
if seen[e.Platform] {
|
||||
return nil, fmt.Errorf("platform matrix: duplicate platform %q", e.Platform)
|
||||
}
|
||||
seen[e.Platform] = true
|
||||
if e.Enabled && (e.Runner == "" || e.NativeCheck != NativeCheckKind) {
|
||||
return nil, fmt.Errorf("platform matrix: enabled platform %q needs a runner and native_check %q", e.Platform, NativeCheckKind)
|
||||
}
|
||||
for _, r := range e.Runtime {
|
||||
if !localPath(r) || r == "manifest.json" || r == p.Executable {
|
||||
return nil, fmt.Errorf("platform matrix: invalid runtime file %q", r)
|
||||
}
|
||||
}
|
||||
}
|
||||
return &p, nil
|
||||
}
|
||||
|
||||
func localPath(p string) bool {
|
||||
return p != "" && !path.IsAbs(p) && path.Clean(p) == p && p != "." && p != ".." && !strings.HasPrefix(p, "../") && !strings.ContainsAny(p, "\\:\x00")
|
||||
}
|
||||
|
||||
func (p *PlatformsFile) enabled() ([]PlatformEntry, error) {
|
||||
var out []PlatformEntry
|
||||
for _, e := range p.Platforms {
|
||||
if e.Enabled {
|
||||
out = append(out, e)
|
||||
}
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, errNoPlatforms
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Platform < out[j].Platform })
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (p *PlatformsFile) find(platform string) (PlatformEntry, bool) {
|
||||
for _, e := range p.Platforms {
|
||||
if e.Platform == platform {
|
||||
return e, true
|
||||
}
|
||||
}
|
||||
return PlatformEntry{}, false
|
||||
}
|
||||
@@ -0,0 +1,342 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/google/go-containerregistry/pkg/registry"
|
||||
v1 "github.com/google/go-containerregistry/pkg/v1"
|
||||
"github.com/google/go-containerregistry/pkg/v1/types"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
func TestPackaging(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Add-on packaging")
|
||||
}
|
||||
|
||||
// pngBytes is the smallest valid signature of a PNG file; the checks read the
|
||||
// signature, not the pixels.
|
||||
var pngBytes = []byte("\x89PNG\r\n\x1a\n" + strings.Repeat("x", 32))
|
||||
|
||||
// project is a hermetic repository root: the real Tables manifest, a small
|
||||
// fake UI build and a platform matrix file.
|
||||
type project struct {
|
||||
root string
|
||||
config string
|
||||
}
|
||||
|
||||
func newProject(enabled map[string][]string) *project {
|
||||
GinkgoHelper()
|
||||
return newProjectFor("tables", enabled)
|
||||
}
|
||||
|
||||
// newProjectFor is newProject for the first-party add-on id: its real manifest
|
||||
// and a fake UI build named after it.
|
||||
func newProjectFor(id string, enabled map[string][]string) *project {
|
||||
GinkgoHelper()
|
||||
root := GinkgoT().TempDir()
|
||||
manifest, err := os.ReadFile("../../../addons/" + id + "/manifest.json")
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
write := func(rel string, b []byte) {
|
||||
p := filepath.Join(root, rel)
|
||||
Expect(os.MkdirAll(filepath.Dir(p), 0o755)).To(Succeed())
|
||||
Expect(os.WriteFile(p, b, 0o644)).To(Succeed())
|
||||
}
|
||||
write("addons/"+id+"/manifest.json", manifest)
|
||||
write("dist/ui/"+id+".js", []byte("export default function mount() {}\n"))
|
||||
write("dist/ui/"+id+".css", []byte(".x{}\n"))
|
||||
write("dist/ui/icon.png", pngBytes)
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "schema: 1\naddon: %s\nexecutable: %s\nmanifest: addons/%s/manifest.json\nui: dist\nplatforms:\n", id, id, id)
|
||||
for _, p := range []string{"linux/amd64", "linux/arm64", "darwin/arm64"} {
|
||||
rt, on := enabled[p]
|
||||
fmt.Fprintf(&b, " - platform: %s\n enabled: %t\n runner: runner-%s\n native_check: executable-start\n", p, on, strings.ReplaceAll(p, "/", "-"))
|
||||
if len(rt) > 0 {
|
||||
b.WriteString(" runtime:\n")
|
||||
for _, r := range rt {
|
||||
fmt.Fprintf(&b, " - %s\n", r)
|
||||
}
|
||||
}
|
||||
}
|
||||
write("platforms.yaml", []byte(b.String()))
|
||||
return &project{root: root, config: filepath.Join(root, "platforms.yaml")}
|
||||
}
|
||||
|
||||
// run executes an addonpkg subcommand and returns its output and error.
|
||||
func run(args ...string) (string, error) {
|
||||
var out bytes.Buffer
|
||||
err := Run(args, &out)
|
||||
return out.String(), err
|
||||
}
|
||||
|
||||
func exe(content string) string {
|
||||
GinkgoHelper()
|
||||
p := filepath.Join(GinkgoT().TempDir(), "tables")
|
||||
Expect(os.WriteFile(p, []byte(content), 0o755)).To(Succeed())
|
||||
return p
|
||||
}
|
||||
|
||||
func (p *project) build(platform, version, executable string, extra ...string) (string, error) {
|
||||
dir := filepath.Join(GinkgoT().TempDir(), strings.ReplaceAll(platform, "/", "-"))
|
||||
args := []string{"build", "--config", p.config, "--root", p.root, "--layout", dir, "--version", version, "--platform", platform, "--executable", executable}
|
||||
_, err := run(append(args, extra...)...)
|
||||
return dir, err
|
||||
}
|
||||
|
||||
func (p *project) mustBuild(platform, version, content string, extra ...string) string {
|
||||
GinkgoHelper()
|
||||
dir, err := p.build(platform, version, exe(content), extra...)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return dir
|
||||
}
|
||||
|
||||
func (p *project) verify(dir, platform, version string) error {
|
||||
_, err := run("verify", "--config", p.config, "--layout", dir, "--platform", platform, "--version", version)
|
||||
return err
|
||||
}
|
||||
|
||||
func expectCode(err error, code string) {
|
||||
GinkgoHelper()
|
||||
Expect(err).To(HaveOccurred())
|
||||
Expect(err.Error()).To(HavePrefix(code+":"), "got %v", err)
|
||||
}
|
||||
|
||||
// tfile is one entry of a rewritten layer.
|
||||
type tfile struct {
|
||||
name string
|
||||
data []byte
|
||||
mode int64
|
||||
typeflag byte
|
||||
link string
|
||||
}
|
||||
|
||||
type layerSpec struct {
|
||||
role string
|
||||
mediaType string
|
||||
files []tfile
|
||||
}
|
||||
|
||||
func sha(b []byte) (string, string) {
|
||||
s := sha256.Sum256(b)
|
||||
return hex.EncodeToString(s[:]), "sha256:" + hex.EncodeToString(s[:])
|
||||
}
|
||||
|
||||
func readSpec(dir string) []layerSpec {
|
||||
GinkgoHelper()
|
||||
l, _, err := loadLeg(dir)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
var out []layerSpec
|
||||
for _, d := range l.manifest.Layers {
|
||||
blob, err := readBlob(dir, d.Digest, d.Size)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
gz, err := gzip.NewReader(bytes.NewReader(blob))
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
tr := tar.NewReader(gz)
|
||||
ls := layerSpec{role: d.Annotations[AnnotationLayer], mediaType: string(d.MediaType)}
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
b, _ := io.ReadAll(tr)
|
||||
ls.files = append(ls.files, tfile{name: h.Name, data: b, mode: h.Mode, typeflag: h.Typeflag})
|
||||
}
|
||||
out = append(out, ls)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// rewrite replaces the layers of a layout and keeps manifest, config, index
|
||||
// and receipt consistent with them, so only the semantic defect is left.
|
||||
// edit may change the manifest, config and receipt too.
|
||||
func rewrite(dir string, specs []layerSpec, edit func(*v1.Manifest, *v1.ConfigFile, *Receipt)) {
|
||||
GinkgoHelper()
|
||||
l, rec, err := loadLeg(dir)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
put := func(b []byte) (v1.Hash, int64) {
|
||||
h, full := sha(b)
|
||||
Expect(os.WriteFile(filepath.Join(dir, "blobs", "sha256", h), b, 0o644)).To(Succeed())
|
||||
hash, err := v1.NewHash(full)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return hash, int64(len(b))
|
||||
}
|
||||
m := l.manifest
|
||||
m.Layers = nil
|
||||
rec.Layers = nil
|
||||
var diffIDs []v1.Hash
|
||||
for _, s := range specs {
|
||||
var tb bytes.Buffer
|
||||
tw := tar.NewWriter(&tb)
|
||||
for _, f := range s.files {
|
||||
h := &tar.Header{Name: f.name, Mode: f.mode, Size: int64(len(f.data)), Typeflag: f.typeflag, Linkname: f.link}
|
||||
if f.typeflag == 0 {
|
||||
h.Typeflag = tar.TypeReg
|
||||
}
|
||||
if h.Typeflag != tar.TypeReg {
|
||||
h.Size = 0
|
||||
}
|
||||
Expect(tw.WriteHeader(h)).To(Succeed())
|
||||
if h.Typeflag == tar.TypeReg {
|
||||
_, err := tw.Write(f.data)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
}
|
||||
}
|
||||
Expect(tw.Close()).To(Succeed())
|
||||
_, full := sha(tb.Bytes())
|
||||
diff, _ := v1.NewHash(full)
|
||||
diffIDs = append(diffIDs, diff)
|
||||
var gb bytes.Buffer
|
||||
gw := gzip.NewWriter(&gb)
|
||||
_, err := gw.Write(tb.Bytes())
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(gw.Close()).To(Succeed())
|
||||
hash, size := put(gb.Bytes())
|
||||
d := v1.Descriptor{MediaType: "application/vnd.oci.image.layer.v1.tar+gzip", Digest: hash, Size: size, Annotations: map[string]string{AnnotationLayer: s.role}}
|
||||
if s.mediaType != "" {
|
||||
d.MediaType = types.MediaType(s.mediaType)
|
||||
}
|
||||
m.Layers = append(m.Layers, d)
|
||||
rec.Layers = append(rec.Layers, ReceiptLayer{Role: s.role, Digest: hash.String(), Size: size, MediaType: string(d.MediaType)})
|
||||
}
|
||||
cfg := l.config
|
||||
cfg.RootFS.DiffIDs = diffIDs
|
||||
if edit != nil {
|
||||
edit(&m, &cfg, rec)
|
||||
}
|
||||
cb, err := json.Marshal(cfg)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
m.Config.Digest, m.Config.Size = put(cb)
|
||||
mb, err := json.Marshal(m)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
mh, ms := put(mb)
|
||||
rec.ManifestDigest, rec.ConfigDigest = mh.String(), m.Config.Digest.String()
|
||||
idx, err := readRootIndex(dir)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
idx.Manifests[0].Digest, idx.Manifests[0].Size = mh, ms
|
||||
ib, err := json.Marshal(idx)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(os.WriteFile(filepath.Join(dir, "index.json"), ib, 0o644)).To(Succeed())
|
||||
rb, err := json.Marshal(rec)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
Expect(os.WriteFile(filepath.Join(dir, receiptFile), rb, 0o644)).To(Succeed())
|
||||
}
|
||||
|
||||
func find(s []layerSpec, role string) *layerSpec {
|
||||
for i := range s {
|
||||
if s[i].role == role {
|
||||
return &s[i]
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (l *layerSpec) drop(name string) {
|
||||
var keep []tfile
|
||||
for _, f := range l.files {
|
||||
if f.name != name {
|
||||
keep = append(keep, f)
|
||||
}
|
||||
}
|
||||
l.files = keep
|
||||
}
|
||||
|
||||
func (l *layerSpec) get(name string) *tfile {
|
||||
for i := range l.files {
|
||||
if l.files[i].name == name {
|
||||
return &l.files[i]
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// countingRegistry is an in-process registry that counts write requests.
|
||||
type countingRegistry struct {
|
||||
mu sync.Mutex
|
||||
writes int
|
||||
server *httptest.Server
|
||||
host string
|
||||
}
|
||||
|
||||
func newRegistry() *countingRegistry {
|
||||
GinkgoHelper()
|
||||
r := &countingRegistry{}
|
||||
handler := registry.New(registry.Logger(log.New(io.Discard, "", 0)))
|
||||
r.server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
|
||||
if req.Method == http.MethodPut || req.Method == http.MethodPost || req.Method == http.MethodPatch {
|
||||
r.mu.Lock()
|
||||
r.writes++
|
||||
r.mu.Unlock()
|
||||
}
|
||||
handler.ServeHTTP(w, req)
|
||||
}))
|
||||
DeferCleanup(r.server.Close)
|
||||
r.host = strings.TrimPrefix(r.server.URL, "http://")
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *countingRegistry) writeCount() int {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.writes
|
||||
}
|
||||
|
||||
// repo returns a repository on this registry.
|
||||
func (r *countingRegistry) repo(name string) string { return r.host + "/" + name }
|
||||
|
||||
// merge builds one leg per enabled platform and joins them.
|
||||
func (p *project) merged(version, revision string, contents map[string]string) string {
|
||||
GinkgoHelper()
|
||||
var legs []string
|
||||
for _, plat := range []string{"linux/amd64", "linux/arm64"} {
|
||||
c := contents[plat]
|
||||
if c == "" {
|
||||
c = "executable of " + plat
|
||||
}
|
||||
legs = append(legs, p.mustBuild(plat, version, c, "--revision", revision))
|
||||
}
|
||||
out := filepath.Join(GinkgoT().TempDir(), "merged")
|
||||
args := append([]string{"merge", "--config", p.config, "--out", out, "--version", version, "--revision", revision}, legs...)
|
||||
_, err := run(args...)
|
||||
Expect(err).NotTo(HaveOccurred())
|
||||
return out
|
||||
}
|
||||
|
||||
// git makes a history of n empty commits and returns the git dir and shas.
|
||||
func gitHistory(n int) (string, []string) {
|
||||
GinkgoHelper()
|
||||
dir := GinkgoT().TempDir()
|
||||
env := append(os.Environ(), "GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_SYSTEM=/dev/null")
|
||||
git := func(args ...string) string {
|
||||
cmd := exec.Command("git", append([]string{"-C", dir, "-c", "user.name=t", "-c", "user.email=t@example.invalid", "-c", "commit.gpgsign=false"}, args...)...)
|
||||
cmd.Env = env
|
||||
out, err := cmd.CombinedOutput()
|
||||
Expect(err).NotTo(HaveOccurred(), string(out))
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
git("init", "-q")
|
||||
var shas []string
|
||||
for i := 0; i < n; i++ {
|
||||
git("commit", "-q", "--allow-empty", "-m", fmt.Sprintf("c%d", i))
|
||||
shas = append(shas, git("rev-parse", "HEAD"))
|
||||
}
|
||||
return filepath.Join(dir, ".git"), shas
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
v1 "github.com/google/go-containerregistry/pkg/v1"
|
||||
"github.com/google/go-containerregistry/pkg/v1/types"
|
||||
)
|
||||
|
||||
// Roles of the layers of a platform image, in order. The ui layer exists only
|
||||
// when the manifest declares a UI.
|
||||
const (
|
||||
roleExecutable = "executable"
|
||||
roleUI = "ui"
|
||||
roleManifest = "manifest"
|
||||
)
|
||||
|
||||
// pkgError carries a stable code so a test (and a reader of CI logs) can tell
|
||||
// which package rule failed.
|
||||
type pkgError struct{ code, msg string }
|
||||
|
||||
func (e *pkgError) Error() string { return e.code + ": " + e.msg }
|
||||
|
||||
func fail(code, format string, a ...any) error {
|
||||
return &pkgError{code: code, msg: fmt.Sprintf(format, a...)}
|
||||
}
|
||||
|
||||
// Receipt is the digest receipt written next to the layout.
|
||||
type Receipt struct {
|
||||
Schema int `json:"schema"`
|
||||
Addon string `json:"addon"`
|
||||
Version string `json:"version"`
|
||||
Platform string `json:"platform"`
|
||||
Revision string `json:"revision,omitempty"`
|
||||
Publishable bool `json:"publishable"`
|
||||
ManifestDigest string `json:"manifestDigest"`
|
||||
ConfigDigest string `json:"configDigest"`
|
||||
Layers []ReceiptLayer `json:"layers"`
|
||||
}
|
||||
|
||||
type ReceiptLayer struct {
|
||||
Role string `json:"role"`
|
||||
Digest string `json:"digest"`
|
||||
Size int64 `json:"size"`
|
||||
MediaType string `json:"mediaType"`
|
||||
}
|
||||
|
||||
const receiptFile = "receipt.json"
|
||||
|
||||
// leg is one platform image read from a layout, with every blob re-hashed.
|
||||
type leg struct {
|
||||
dir string
|
||||
descriptor v1.Descriptor
|
||||
manifest v1.Manifest
|
||||
rawMan []byte
|
||||
config v1.ConfigFile
|
||||
}
|
||||
|
||||
func strictJSON(b []byte, v any) error {
|
||||
d := json.NewDecoder(bytes.NewReader(b))
|
||||
d.DisallowUnknownFields()
|
||||
if err := d.Decode(v); err != nil {
|
||||
return err
|
||||
}
|
||||
if d.More() {
|
||||
return errors.New("trailing data")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// readBlob returns the bytes of a blob after checking digest and size.
|
||||
func readBlob(dir string, d v1.Hash, size int64) ([]byte, error) {
|
||||
if d.Algorithm != "sha256" || len(d.Hex) != 64 {
|
||||
return nil, fail("E_DIGEST", "unsupported digest %q", d.String())
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, "blobs", d.Algorithm, d.Hex)) // #nosec G304 -- the packaging tool reads paths the operator passes on its command line
|
||||
if err != nil {
|
||||
return nil, fail("E_DIGEST", "blob %s is missing", d)
|
||||
}
|
||||
sum := sha256.Sum256(b)
|
||||
if hex.EncodeToString(sum[:]) != d.Hex {
|
||||
return nil, fail("E_DIGEST", "blob %s does not hash to its digest", d)
|
||||
}
|
||||
if size >= 0 && int64(len(b)) != size {
|
||||
return nil, fail("E_DIGEST", "blob %s has size %d, descriptor says %d", d, len(b), size)
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
func readRootIndex(dir string) (*v1.IndexManifest, error) {
|
||||
marker, err := os.ReadFile(filepath.Join(dir, "oci-layout")) // #nosec G304 -- the packaging tool reads paths the operator passes on its command line
|
||||
if err != nil {
|
||||
return nil, fail("E_LAYOUT", "oci-layout file is missing")
|
||||
}
|
||||
var m struct {
|
||||
Version string `json:"imageLayoutVersion"`
|
||||
}
|
||||
if err := strictJSON(marker, &m); err != nil || m.Version != "1.0.0" {
|
||||
return nil, fail("E_LAYOUT", "unsupported oci-layout marker")
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, "index.json")) // #nosec G304 -- the packaging tool reads paths the operator passes on its command line
|
||||
if err != nil {
|
||||
return nil, fail("E_LAYOUT", "index.json is missing")
|
||||
}
|
||||
var idx v1.IndexManifest
|
||||
if err := strictJSON(b, &idx); err != nil {
|
||||
return nil, fail("E_LAYOUT", "index.json: %v", err)
|
||||
}
|
||||
if idx.SchemaVersion != 2 {
|
||||
return nil, fail("E_LAYOUT", "index.json schemaVersion %d", idx.SchemaVersion)
|
||||
}
|
||||
return &idx, nil
|
||||
}
|
||||
|
||||
// loadManifestBlob reads one image manifest and its config by digest.
|
||||
func loadLegByDescriptor(dir string, d v1.Descriptor) (*leg, error) {
|
||||
if d.MediaType != types.OCIManifestSchema1 {
|
||||
return nil, fail("E_LAYERS", "manifest media type %q is not an OCI image manifest", d.MediaType)
|
||||
}
|
||||
raw, err := readBlob(dir, d.Digest, d.Size)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
l := &leg{dir: dir, descriptor: d, rawMan: raw}
|
||||
if err := strictJSON(raw, &l.manifest); err != nil {
|
||||
return nil, fail("E_LAYERS", "image manifest: %v", err)
|
||||
}
|
||||
if l.manifest.MediaType != types.OCIManifestSchema1 {
|
||||
return nil, fail("E_LAYERS", "image manifest mediaType %q", l.manifest.MediaType)
|
||||
}
|
||||
cfg, err := readBlob(dir, l.manifest.Config.Digest, l.manifest.Config.Size)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if l.manifest.Config.MediaType != types.OCIConfigJSON {
|
||||
return nil, fail("E_LAYERS", "config media type %q", l.manifest.Config.MediaType)
|
||||
}
|
||||
if err := json.Unmarshal(cfg, &l.config); err != nil {
|
||||
return nil, fail("E_LAYERS", "image config: %v", err)
|
||||
}
|
||||
return l, nil
|
||||
}
|
||||
|
||||
func loadLeg(dir string) (*leg, *Receipt, error) {
|
||||
idx, err := readRootIndex(dir)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if len(idx.Manifests) != 1 {
|
||||
return nil, nil, fail("E_LAYOUT", "a platform layout holds exactly one image, found %d", len(idx.Manifests))
|
||||
}
|
||||
l, err := loadLegByDescriptor(dir, idx.Manifests[0])
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, receiptFile)) // #nosec G304 -- the packaging tool reads paths the operator passes on its command line
|
||||
if err != nil {
|
||||
return nil, nil, fail("E_DIGEST", "digest receipt is missing")
|
||||
}
|
||||
var r Receipt
|
||||
if err := strictJSON(b, &r); err != nil {
|
||||
return nil, nil, fail("E_DIGEST", "digest receipt: %v", err)
|
||||
}
|
||||
return l, &r, nil
|
||||
}
|
||||
|
||||
// copyBytes is io.Copy into a buffer; kept tiny so call sites read clearly.
|
||||
func readAll(r io.Reader, limit int64) ([]byte, error) {
|
||||
b, err := io.ReadAll(io.LimitReader(r, limit+1))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if int64(len(b)) > limit {
|
||||
return nil, errors.New("too large")
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
@@ -0,0 +1,349 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
v1 "github.com/google/go-containerregistry/pkg/v1"
|
||||
"github.com/google/go-containerregistry/pkg/v1/empty"
|
||||
"github.com/google/go-containerregistry/pkg/v1/layout"
|
||||
"github.com/google/go-containerregistry/pkg/v1/mutate"
|
||||
"github.com/google/go-containerregistry/pkg/v1/types"
|
||||
)
|
||||
|
||||
// MergedFile lists the legs a merged layout joined, with their receipts.
|
||||
const mergedFile = "merged.json"
|
||||
|
||||
type merged struct {
|
||||
Schema int `json:"schema"`
|
||||
Addon string `json:"addon"`
|
||||
Version string `json:"version"`
|
||||
Revision string `json:"revision,omitempty"`
|
||||
IndexDigest string `json:"indexDigest"`
|
||||
Legs []Receipt `json:"legs"`
|
||||
}
|
||||
|
||||
// rawIndex replaces the manifest bytes of an image index.
|
||||
type rawIndex struct {
|
||||
inner v1.ImageIndex
|
||||
raw []byte
|
||||
}
|
||||
|
||||
func (r rawIndex) MediaType() (types.MediaType, error) { return r.inner.MediaType() }
|
||||
func (r rawIndex) RawManifest() ([]byte, error) { return r.raw, nil }
|
||||
func (r rawIndex) IndexManifest() (*v1.IndexManifest, error) {
|
||||
return v1.ParseIndexManifest(bytes.NewReader(r.raw))
|
||||
}
|
||||
func (r rawIndex) Digest() (v1.Hash, error) {
|
||||
h, _, err := v1.SHA256(bytes.NewReader(r.raw))
|
||||
return h, err
|
||||
}
|
||||
func (r rawIndex) Size() (int64, error) { return int64(len(r.raw)), nil }
|
||||
func (r rawIndex) Image(h v1.Hash) (v1.Image, error) { return r.inner.Image(h) }
|
||||
func (r rawIndex) ImageIndex(h v1.Hash) (v1.ImageIndex, error) { return r.inner.ImageIndex(h) }
|
||||
|
||||
func cmdMatrix(args []string) error {
|
||||
fs := flag.NewFlagSet("matrix", flag.ContinueOnError)
|
||||
config := fs.String("config", "", "platform matrix file")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
cfg, err := loadPlatforms(*config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
en, err := cfg.enabled()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
type row struct {
|
||||
Platform string `json:"platform"`
|
||||
Runner string `json:"runner"`
|
||||
Slug string `json:"slug"`
|
||||
}
|
||||
var rows []row
|
||||
for _, e := range en {
|
||||
rows = append(rows, row{e.Platform, e.Runner, strings.ReplaceAll(e.Platform, "/", "-")})
|
||||
}
|
||||
b, err := json.Marshal(map[string]any{"include": rows})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = fmt.Fprintf(stdout, "%s\n", b)
|
||||
return nil
|
||||
}
|
||||
|
||||
// verifyMerged checks a merged layout against the platform matrix: exactly
|
||||
// the enabled platforms, one image each, the same version, every leg
|
||||
// publishable and valid.
|
||||
func verifyMerged(dir string, cfg *PlatformsFile) (*merged, v1.Hash, error) {
|
||||
var zero v1.Hash
|
||||
b, err := os.ReadFile(filepath.Join(dir, mergedFile)) // #nosec G304 -- the packaging tool reads paths the operator passes on its command line
|
||||
if err != nil {
|
||||
return nil, zero, fail("E_MERGE", "%s is missing", mergedFile)
|
||||
}
|
||||
var m merged
|
||||
if err := strictJSON(b, &m); err != nil {
|
||||
return nil, zero, fail("E_MERGE", "%s: %v", mergedFile, err)
|
||||
}
|
||||
idx, err := readRootIndex(dir)
|
||||
if err != nil {
|
||||
return nil, zero, err
|
||||
}
|
||||
if len(idx.Manifests) != 1 || idx.Manifests[0].MediaType != types.OCIImageIndex {
|
||||
return nil, zero, fail("E_MERGE", "a merged layout holds exactly one image index")
|
||||
}
|
||||
top := idx.Manifests[0]
|
||||
if top.Digest.String() != m.IndexDigest {
|
||||
return nil, zero, fail("E_DIGEST", "merge receipt names another index digest")
|
||||
}
|
||||
raw, err := readBlob(dir, top.Digest, top.Size)
|
||||
if err != nil {
|
||||
return nil, zero, err
|
||||
}
|
||||
var im v1.IndexManifest
|
||||
if err := strictJSON(raw, &im); err != nil {
|
||||
return nil, zero, fail("E_MERGE", "image index: %v", err)
|
||||
}
|
||||
if im.ArtifactType != IndexArtifactType || im.Annotations[AnnotationID] != cfg.Addon || im.Annotations[AnnotationVersion] != m.Version {
|
||||
return nil, zero, fail("E_ARTIFACT_TYPE", "image index is not a %s %s package index", cfg.Addon, m.Version)
|
||||
}
|
||||
enabled, err := cfg.enabled()
|
||||
if err != nil {
|
||||
return nil, zero, fail("E_PLATFORMS", "%v", err)
|
||||
}
|
||||
want := map[string]bool{}
|
||||
for _, e := range enabled {
|
||||
want[e.Platform] = true
|
||||
}
|
||||
got := map[string]bool{}
|
||||
receipts := map[string]Receipt{}
|
||||
for _, r := range m.Legs {
|
||||
receipts[r.Platform] = r
|
||||
}
|
||||
for _, d := range im.Manifests {
|
||||
if d.Platform == nil {
|
||||
return nil, zero, fail("E_PLATFORM", "index entry %s has no platform", d.Digest)
|
||||
}
|
||||
p := d.Platform.OS + "/" + d.Platform.Architecture
|
||||
if got[p] {
|
||||
return nil, zero, fail("E_PLATFORMS", "platform %s appears twice", p)
|
||||
}
|
||||
got[p] = true
|
||||
if !want[p] {
|
||||
return nil, zero, fail("E_PLATFORMS", "platform %s is not enabled in the platform matrix", p)
|
||||
}
|
||||
rec, ok := receipts[p]
|
||||
if !ok || !rec.Publishable {
|
||||
return nil, zero, fail("E_PLATFORMS", "platform %s has no publishable receipt", p)
|
||||
}
|
||||
l, err := loadLegByDescriptor(dir, d)
|
||||
if err != nil {
|
||||
return nil, zero, err
|
||||
}
|
||||
if _, err := verifyLoaded(l, &rec, p, m.Version, cfg); err != nil {
|
||||
return nil, zero, fmt.Errorf("%s: %w", p, err)
|
||||
}
|
||||
}
|
||||
for _, p := range sortedKeys(want) {
|
||||
if !got[p] {
|
||||
return nil, zero, fail("E_PLATFORMS", "platform %s is enabled and has no image: a partial index is never published", p)
|
||||
}
|
||||
}
|
||||
return &m, top.Digest, nil
|
||||
}
|
||||
|
||||
func cmdMerge(args []string) error {
|
||||
fs := flag.NewFlagSet("merge", flag.ContinueOnError)
|
||||
out := fs.String("out", "", "merged layout directory (must not exist or be empty)")
|
||||
config := fs.String("config", "", "platform matrix file")
|
||||
version := fs.String("version", "", "release version")
|
||||
revision := fs.String("revision", "", "source commit")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
legs := fs.Args()
|
||||
if *out == "" || *version == "" || len(legs) == 0 {
|
||||
return errors.New("--out, --version, --config and at least one leg layout are required")
|
||||
}
|
||||
cfg, err := loadPlatforms(*config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !versionRE.MatchString(*version) {
|
||||
return fmt.Errorf("version %q is not a semantic version", *version)
|
||||
}
|
||||
if ents, err := os.ReadDir(*out); err == nil && len(ents) > 0 {
|
||||
return errors.New("output layout is not empty")
|
||||
}
|
||||
type item struct {
|
||||
platform string
|
||||
img v1.Image
|
||||
rec Receipt
|
||||
}
|
||||
var items []item
|
||||
seen := map[string]bool{}
|
||||
for _, dir := range legs {
|
||||
_, rec, err := loadLeg(dir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s: %w", dir, err)
|
||||
}
|
||||
if seen[rec.Platform] {
|
||||
return fail("E_PLATFORMS", "platform %s appears in two legs", rec.Platform)
|
||||
}
|
||||
seen[rec.Platform] = true
|
||||
if _, err := verifyLeg(dir, rec.Platform, *version, cfg); err != nil {
|
||||
return fmt.Errorf("%s: %w", rec.Platform, err)
|
||||
}
|
||||
if !rec.Publishable {
|
||||
return fail("E_PLATFORMS", "leg %s is a local check layout, not publishable", rec.Platform)
|
||||
}
|
||||
lp, err := layout.FromPath(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ii, err := lp.ImageIndex()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
im, err := ii.IndexManifest()
|
||||
if err != nil || len(im.Manifests) != 1 {
|
||||
return fail("E_LAYOUT", "leg %s: unreadable index", dir)
|
||||
}
|
||||
img, err := ii.Image(im.Manifests[0].Digest)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
items = append(items, item{rec.Platform, img, *rec})
|
||||
}
|
||||
// Completeness is checked on the final index too (verifyMerged), but a
|
||||
// missing leg is reported before any output exists.
|
||||
enabled, err := cfg.enabled()
|
||||
if err != nil {
|
||||
return fail("E_PLATFORMS", "%v", err)
|
||||
}
|
||||
for _, e := range enabled {
|
||||
if !seen[e.Platform] {
|
||||
return fail("E_PLATFORMS", "platform %s is enabled and has no leg: a partial index is never created", e.Platform)
|
||||
}
|
||||
}
|
||||
for p := range seen {
|
||||
if _, ok := cfg.find(p); !ok {
|
||||
return fail("E_PLATFORMS", "leg platform %s is not in the platform matrix", p)
|
||||
}
|
||||
}
|
||||
sort.Slice(items, func(i, j int) bool { return items[i].platform < items[j].platform })
|
||||
var idx v1.ImageIndex = mutate.IndexMediaType(empty.Index, types.OCIImageIndex)
|
||||
var receipts []Receipt
|
||||
for _, it := range items {
|
||||
osName, arch, _ := strings.Cut(it.platform, "/")
|
||||
idx = mutate.AppendManifests(idx, mutate.IndexAddendum{Add: it.img, Descriptor: v1.Descriptor{Platform: &v1.Platform{OS: osName, Architecture: arch}}})
|
||||
receipts = append(receipts, it.rec)
|
||||
}
|
||||
im, err := idx.IndexManifest()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
im.ArtifactType = IndexArtifactType
|
||||
im.Annotations = map[string]string{AnnotationID: cfg.Addon, AnnotationVersion: *version, AnnotationOCIVer: *version}
|
||||
if *revision != "" {
|
||||
if !shaRE.MatchString(*revision) {
|
||||
return errors.New("revision must be 40 lower-case hex characters")
|
||||
}
|
||||
im.Annotations[AnnotationRevision] = *revision
|
||||
}
|
||||
raw, err := json.Marshal(im)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
final := rawIndex{inner: idx, raw: raw}
|
||||
digest, err := final.Digest()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(*out, 0o750); err != nil {
|
||||
return err
|
||||
}
|
||||
lp, err := layout.Write(*out, empty.Index)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := lp.AppendIndex(final); err != nil {
|
||||
return err
|
||||
}
|
||||
mb, err := json.MarshalIndent(merged{Schema: 1, Addon: cfg.Addon, Version: *version, Revision: *revision, IndexDigest: digest.String(), Legs: receipts}, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(*out, mergedFile), append(mb, '\n'), 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, _, err := verifyMerged(*out, cfg); err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = fmt.Fprintf(stdout, "index=%s\nplatforms=%d\nversion=%s\n", digest, len(items), *version)
|
||||
return nil
|
||||
}
|
||||
|
||||
// cmdNativeCheck extracts the verified package and starts its executable
|
||||
// with an argument the add-on protocol rejects. An executable that loads and
|
||||
// runs on this platform exits 2; a missing library, a wrong format or a crash
|
||||
// does not. The check refuses to run for a platform other than the host's, so
|
||||
// an advertised platform is only ever checked natively.
|
||||
func cmdNativeCheck(args []string) error {
|
||||
fs := flag.NewFlagSet("native-check", flag.ContinueOnError)
|
||||
dir := fs.String("layout", "", "platform OCI layout")
|
||||
platform := fs.String("platform", "", "platform of the layout")
|
||||
version := fs.String("version", "", "version of the layout")
|
||||
config := fs.String("config", "", "platform matrix file")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
host := runtime.GOOS + "/" + runtime.GOARCH
|
||||
if *platform != host {
|
||||
return fmt.Errorf("native check of %s needs a native runner; this host is %s", *platform, host)
|
||||
}
|
||||
cfg, err := loadPlatforms(*config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
v, err := verifyLeg(*dir, *platform, *version, cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp, err := os.MkdirTemp("", "addon-native-check-")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = os.RemoveAll(tmp) }()
|
||||
if err := writeExtracted(v, tmp); err != nil {
|
||||
return err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, filepath.Join(tmp, v.Manifest.Executable)) // #nosec G204 -- runs the add-on executable of the package under verification, from a private temporary directory
|
||||
cmd.Dir = tmp
|
||||
cmd.Stdin = nil
|
||||
var buf bytes.Buffer
|
||||
cmd.Stdout, cmd.Stderr = io.Discard, &buf
|
||||
err = cmd.Run()
|
||||
var ee *exec.ExitError
|
||||
if errors.As(err, &ee) && ee.ExitCode() == 2 {
|
||||
_, _ = fmt.Fprintf(stdout, "native-check=passed platform=%s\n", *platform)
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("native check failed: executable did not start and reject its arguments (%v): %s", err, strings.TrimSpace(buf.String()))
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,77 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// The release platforms of the packaged first-party add-ons, with the runner
|
||||
// that builds and checks each one natively.
|
||||
var releasePlatforms = map[string]string{
|
||||
"linux/amd64": "ubuntu-24.04",
|
||||
"linux/arm64": "ubuntu-24.04-arm",
|
||||
"darwin/arm64": "macos-14",
|
||||
"darwin/amd64": "macos-15-intel",
|
||||
}
|
||||
|
||||
// nativeChecked records the platforms whose native check has passed on a runner
|
||||
// of that platform. A task of group G adds a platform here in the same change
|
||||
// that enables it in the five files and shows the passing legs in its pull
|
||||
// request. Enabling without this record, or recording without enabling, fails.
|
||||
var nativeChecked = map[string]bool{"linux/amd64": true, "linux/arm64": true, "darwin/arm64": true, "darwin/amd64": true}
|
||||
|
||||
var packagedAddons = []string{"voice", "face", "fine-tuning", "quantization", "studio", "talk"}
|
||||
|
||||
var _ = Describe("the packaged add-ons", func() {
|
||||
for _, id := range packagedAddons {
|
||||
Context(id, func() {
|
||||
It("lists every release platform with its runner", func() {
|
||||
p, err := loadPlatforms(filepath.Join("..", "..", "..", "addons", id, "package", "platforms.yaml"))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(p.Addon).To(Equal(id))
|
||||
Expect(p.Executable).To(Equal(id))
|
||||
got := map[string]string{}
|
||||
for _, e := range p.Platforms {
|
||||
got[e.Platform] = e.Runner
|
||||
Expect(e.NativeCheck).To(Equal(NativeCheckKind), "%s: native check", e.Platform)
|
||||
}
|
||||
Expect(got).To(HaveLen(len(releasePlatforms)))
|
||||
for platform, runner := range releasePlatforms {
|
||||
Expect(got[platform]).To(Equal(runner), "platform %s", platform)
|
||||
}
|
||||
})
|
||||
|
||||
It("enables a platform exactly when its native check is recorded", func() {
|
||||
p, err := loadPlatforms(filepath.Join("..", "..", "..", "addons", id, "package", "platforms.yaml"))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
var enabled []string
|
||||
for _, e := range p.Platforms {
|
||||
if e.Enabled {
|
||||
enabled = append(enabled, e.Platform)
|
||||
}
|
||||
}
|
||||
var recorded []string
|
||||
for platform := range nativeChecked {
|
||||
recorded = append(recorded, platform)
|
||||
}
|
||||
sort.Strings(enabled)
|
||||
sort.Strings(recorded)
|
||||
Expect(enabled).To(Equal(recorded), "enabled platforms and recorded native checks differ")
|
||||
})
|
||||
|
||||
It("has a catalog template", func() {
|
||||
b, err := os.ReadFile(filepath.Join("..", "..", "..", "gallery", "addons", id+".yaml.tmpl"))
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
for _, want := range []string{`id: "{{addon}}"`, "requireSignature: true"} {
|
||||
Expect(strings.Contains(string(b), want)).To(BeTrue(), "template lacks %q", want)
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,390 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// trustedJobs are the only jobs that may hold registry credentials, the
|
||||
// OIDC token permission or an environment.
|
||||
var trustedJobs = map[string]bool{"publish": true, "promote": true}
|
||||
|
||||
// deniedInWorkflow are strings that tie a workflow to the backend image
|
||||
// pipeline. The add-on package workflow has its own registry input, its own
|
||||
// signer identity and its own jobs.
|
||||
var deniedInWorkflow = []string{
|
||||
"quay.io", "go-skynet", "backend_merge", "backend_build", "backend.yml", "backend_pr",
|
||||
"ci-cache", "docker.io", "dockerhub", "quayusername", "quaypassword", "dockerusername", "dockerpassword",
|
||||
}
|
||||
|
||||
// checkWorkflow applies the workflow policy to a parsed workflow and returns
|
||||
// every violation, each with a stable P_ code.
|
||||
func checkWorkflow(data []byte) []string {
|
||||
var doc map[string]any
|
||||
if err := yaml.Unmarshal(data, &doc); err != nil {
|
||||
return []string{"P_PARSE: " + err.Error()}
|
||||
}
|
||||
var out []string
|
||||
add := func(code, format string, a ...any) { out = append(out, code+": "+fmt.Sprintf(format, a...)) }
|
||||
text := func(v any) string { b, _ := yaml.Marshal(v); return string(b) }
|
||||
whole := text(doc)
|
||||
lower := strings.ToLower(whole)
|
||||
|
||||
on, _ := doc["on"].(map[string]any)
|
||||
if on == nil {
|
||||
on, _ = doc[fmt.Sprint(true)].(map[string]any)
|
||||
}
|
||||
if on == nil {
|
||||
add("P_TRIGGERS", "workflow has no on: mapping")
|
||||
}
|
||||
if _, ok := on["pull_request_target"]; ok || strings.Contains(lower, "pull_request_target") {
|
||||
add("P_PR_TARGET", "pull_request_target must not appear: it runs untrusted code with repository credentials")
|
||||
}
|
||||
for _, t := range []string{"push", "pull_request", "release"} {
|
||||
if _, ok := on[t]; !ok {
|
||||
add("P_TRIGGERS", "trigger %s is required", t)
|
||||
}
|
||||
}
|
||||
for _, d := range deniedInWorkflow {
|
||||
if strings.Contains(lower, d) {
|
||||
add("P_BACKEND", "workflow references the backend pipeline: %q", d)
|
||||
}
|
||||
}
|
||||
// A literal latest tag would let any run move the stable channel.
|
||||
for _, loc := range regexp.MustCompile(`latest`).FindAllStringIndex(whole, -1) {
|
||||
before, after := byte(' '), byte(' ')
|
||||
if loc[0] > 0 {
|
||||
before = whole[loc[0]-1]
|
||||
}
|
||||
if loc[1] < len(whole) {
|
||||
after = whole[loc[1]]
|
||||
}
|
||||
word := func(c byte) bool {
|
||||
return c == '-' || c == '_' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9'
|
||||
}
|
||||
if !word(before) && !word(after) {
|
||||
add("P_LATEST", "a literal latest tag appears; only scripts/addons/channel.sh may decide it")
|
||||
break
|
||||
}
|
||||
}
|
||||
perms, _ := doc["permissions"].(map[string]any)
|
||||
if len(perms) != 1 || perms["contents"] != "read" {
|
||||
add("P_PERMISSIONS", "workflow-level permissions must be exactly contents: read")
|
||||
}
|
||||
jobs, _ := doc["jobs"].(map[string]any)
|
||||
job := func(n string) map[string]any { m, _ := jobs[n].(map[string]any); return m }
|
||||
for _, n := range []string{"plan", "validate", "package", "publish", "promote"} {
|
||||
if job(n) == nil {
|
||||
add("P_JOBS", "job %s is required", n)
|
||||
}
|
||||
}
|
||||
for n, v := range jobs {
|
||||
j, _ := v.(map[string]any)
|
||||
t := text(j)
|
||||
if trustedJobs[n] {
|
||||
continue
|
||||
}
|
||||
if regexp.MustCompile(`secrets\.[A-Za-z0-9_]*`).FindString(strings.ReplaceAll(t, "secrets.GITHUB_TOKEN", "")) != "" {
|
||||
add("P_UNTRUSTED_SECRETS", "job %s reads secrets: only publish and promote may", n)
|
||||
}
|
||||
if _, ok := j["environment"]; ok {
|
||||
add("P_UNTRUSTED_SECRETS", "job %s names an environment: only publish and promote may", n)
|
||||
}
|
||||
if p, ok := j["permissions"].(map[string]any); ok {
|
||||
for k, val := range p {
|
||||
if val != "read" && val != "none" {
|
||||
add("P_UNTRUSTED_SECRETS", "job %s has %s: %v; untrusted jobs are read-only and never hold id-token", n, k, val)
|
||||
}
|
||||
}
|
||||
}
|
||||
if strings.Contains(t, "id-token") && !strings.Contains(t, "id-token: none") {
|
||||
add("P_UNTRUSTED_SECRETS", "job %s mentions id-token", n)
|
||||
}
|
||||
if strings.Contains(t, "docker/login-action") || strings.Contains(t, "cosign sign") || strings.Contains(t, "addonpkg publish") || strings.Contains(t, "addonpkg promote") {
|
||||
add("P_UNTRUSTED_SECRETS", "job %s logs in, signs or publishes: only publish and promote may", n)
|
||||
}
|
||||
}
|
||||
needs := func(n string) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
switch v := job(n)["needs"].(type) {
|
||||
case string:
|
||||
out[v] = true
|
||||
case []any:
|
||||
for _, x := range v {
|
||||
out[fmt.Sprint(x)] = true
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
pub, pro := job("publish"), job("promote")
|
||||
// reaches reports whether job from depends on job to, directly or
|
||||
// through other jobs. The release chain is validate -> package ->
|
||||
// publish -> promote, and each link must be structural.
|
||||
var reaches func(from, to string, seen map[string]bool) bool
|
||||
reaches = func(from, to string, seen map[string]bool) bool {
|
||||
if seen[from] {
|
||||
return false
|
||||
}
|
||||
seen[from] = true
|
||||
for n := range needs(from) {
|
||||
if n == to || reaches(n, to, seen) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
for _, n := range []string{"plan", "validate", "package"} {
|
||||
if !needs("publish")[n] {
|
||||
add("P_NEEDS", "publish must need %s: every check and platform leg must pass before anything is published", n)
|
||||
}
|
||||
}
|
||||
for _, n := range []string{"plan", "validate"} {
|
||||
if !needs("package")[n] {
|
||||
add("P_NEEDS", "package must need %s", n)
|
||||
}
|
||||
}
|
||||
if !needs("promote")["publish"] {
|
||||
add("P_NEEDS", "promote must need publish: aliases move only after signing")
|
||||
}
|
||||
if !reaches("promote", "validate", map[string]bool{}) {
|
||||
add("P_NEEDS", "promote must depend on validate through publish: latest moves only after complete validation")
|
||||
}
|
||||
// A job that can finish "successfully" or be bypassed after validate
|
||||
// failed would defeat the gate.
|
||||
if v := job("validate"); v != nil {
|
||||
if c, ok := v["continue-on-error"]; ok && fmt.Sprint(c) != "false" {
|
||||
add("P_VALIDATE_GATE", "validate must not set continue-on-error: a red validation would not stop the release")
|
||||
}
|
||||
if c, ok := v["if"]; ok {
|
||||
cond := fmt.Sprint(c)
|
||||
for _, bad := range []string{"always()", "cancelled()", "failure()"} {
|
||||
if strings.Contains(cond, bad) {
|
||||
add("P_VALIDATE_GATE", "validate must not use %s in its condition", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
vsteps, _ := v["steps"].([]any)
|
||||
for _, st := range vsteps {
|
||||
m, _ := st.(map[string]any)
|
||||
if c, ok := m["continue-on-error"]; ok && fmt.Sprint(c) != "false" {
|
||||
add("P_VALIDATE_GATE", "validate must not set continue-on-error on a step")
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, n := range []string{"package", "publish", "promote"} {
|
||||
cond := fmt.Sprint(job(n)["if"])
|
||||
for _, bad := range []string{"always()", "cancelled()", "failure()"} {
|
||||
if strings.Contains(cond, bad) {
|
||||
add("P_VALIDATE_GATE", "%s must not use %s: it would run after a failed validate", n, bad)
|
||||
}
|
||||
}
|
||||
if c, ok := job(n)["continue-on-error"]; ok && fmt.Sprint(c) != "false" {
|
||||
add("P_VALIDATE_GATE", "%s must not set continue-on-error", n)
|
||||
}
|
||||
}
|
||||
for _, n := range []string{"publish", "promote"} {
|
||||
j := job(n)
|
||||
cond := fmt.Sprint(j["if"])
|
||||
if !strings.Contains(cond, "needs.plan.outputs.publish == 'true'") {
|
||||
add("P_TRUSTED_GATE", "%s must run only when the plan says publish == 'true'", n)
|
||||
}
|
||||
for _, bad := range []string{"always()", "cancelled()", "failure()"} {
|
||||
if strings.Contains(cond, bad) {
|
||||
add("P_TRUSTED_GATE", "%s must not use %s: a failed leg would still publish", n, bad)
|
||||
}
|
||||
}
|
||||
if j != nil && j["environment"] == nil {
|
||||
add("P_TRUSTED_GATE", "%s must run in a protected environment", n)
|
||||
}
|
||||
}
|
||||
if p, _ := pub["permissions"].(map[string]any); p["id-token"] != "write" {
|
||||
add("P_SIGNER", "publish needs id-token: write for keyless signing")
|
||||
}
|
||||
if p, _ := pro["permissions"].(map[string]any); p["id-token"] == "write" {
|
||||
add("P_SIGNER", "promote must not hold id-token: write")
|
||||
}
|
||||
if pro != nil && strings.Contains(fmt.Sprint(pro["if"]), "needs.plan.outputs.aliases") == false {
|
||||
add("P_TRUSTED_GATE", "promote must run only when the plan names aliases (a tag push names none)")
|
||||
}
|
||||
// Matrix comes from the validated platform matrix, never a literal.
|
||||
if m, _ := job("package")["strategy"].(map[string]any); m == nil || !strings.Contains(text(m), "needs.plan.outputs.matrix") {
|
||||
add("P_MATRIX", "package matrix must come from needs.plan.outputs.matrix")
|
||||
}
|
||||
// Every job that works on one add-on runs once for each add-on, and no
|
||||
// step names one add-on by path.
|
||||
if strings.Contains(whole, "addons/tables/") {
|
||||
add("P_ADDON_PATH", "the workflow names addons/tables/ directly; use the add-on of the matrix leg")
|
||||
}
|
||||
for _, n := range []string{"validate", "publish", "promote"} {
|
||||
m, _ := job(n)["strategy"].(map[string]any)
|
||||
mm, _ := m["matrix"].(map[string]any)
|
||||
if _, ok := mm["addon"]; !ok {
|
||||
add("P_MATRIX", "%s must run once for each add-on: strategy.matrix.addon is missing", n)
|
||||
}
|
||||
}
|
||||
// Publication is per add-on: the groups, the repository and the digest
|
||||
// must not be shared between add-ons.
|
||||
for _, n := range []string{"publish", "promote"} {
|
||||
j := job(n)
|
||||
c, _ := j["concurrency"].(map[string]any)
|
||||
if !strings.Contains(fmt.Sprint(c["group"]), "matrix.addon") {
|
||||
add("P_CONCURRENCY", "%s must serialize per add-on: the concurrency group needs matrix.addon", n)
|
||||
}
|
||||
if t := text(j); !strings.Contains(t, `ADDON_OCI_REPOSITORY=$repo" >> "$GITHUB_ENV"`) || !strings.Contains(t, `${ADDON_OCI_BASE%/*}/$ADDON`) {
|
||||
add("P_REPOSITORY", "%s must choose the repository of the add-on: <namespace>/<id>, and the configured repository for tables", n)
|
||||
}
|
||||
}
|
||||
// Tables keeps the configured repository and every other add-on derives its
|
||||
// own from the namespace.
|
||||
for _, n := range []string{"publish", "promote"} {
|
||||
if !regexp.MustCompile(`if \[ "\$ADDON" = tables \]; then\s+repo="\$ADDON_OCI_BASE"\s+else\s+repo="\$\{ADDON_OCI_BASE%/\*\}/\$ADDON"\s+fi`).MatchString(text(job(n))) {
|
||||
add("P_REPOSITORY", "%s must keep ADDON_OCI_REPOSITORY for tables and derive <namespace>/<id> for every other add-on", n)
|
||||
}
|
||||
// A leg that is pushing or moving an alias must finish, and one
|
||||
// failing add-on must not stop another.
|
||||
st, _ := job(n)["strategy"].(map[string]any)
|
||||
if st["fail-fast"] != false {
|
||||
add("P_FAIL_FAST", "%s must set strategy.fail-fast: false", n)
|
||||
}
|
||||
}
|
||||
// The layouts of publish come from the legs of its own add-on only.
|
||||
{
|
||||
found := false
|
||||
pubSteps, _ := pub["steps"].([]any)
|
||||
for _, s := range pubSteps {
|
||||
st, _ := s.(map[string]any)
|
||||
if u, _ := st["uses"].(string); !strings.HasPrefix(u, "actions/download-artifact@") {
|
||||
continue
|
||||
}
|
||||
w, _ := st["with"].(map[string]any)
|
||||
found = true
|
||||
if pat := fmt.Sprint(w["pattern"]); pat != "layout-${{ matrix.addon }}-*" {
|
||||
add("P_ARTIFACTS", "publish must download only the layouts of its own add-on (layout-${{ matrix.addon }}-*), not %q", pat)
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
add("P_ARTIFACTS", "publish does not download the platform layouts")
|
||||
}
|
||||
}
|
||||
// Promote moves an alias only to a well-formed digest.
|
||||
if t := text(pro); !strings.Contains(t, "sha256:????????????????????????????????????????????????????????????????)") || !strings.Contains(t, "missing or malformed") {
|
||||
add("P_DIGEST", "promote must refuse a digest that is missing or not sha256:<64 characters>")
|
||||
}
|
||||
if _, ok := job("publish")["outputs"]; ok {
|
||||
add("P_ORDER", "publish runs once for each add-on and must not hand a job output to promote; it uploads the digest as an artifact")
|
||||
}
|
||||
// Step order inside publish: merge, push, sign, verify.
|
||||
steps, _ := pub["steps"].([]any)
|
||||
index := map[string]int{}
|
||||
for i, s := range steps {
|
||||
st, _ := s.(map[string]any)
|
||||
if id, ok := st["id"].(string); ok {
|
||||
index[id] = i
|
||||
}
|
||||
if strings.Contains(text(st), "addonpkg promote") {
|
||||
add("P_ORDER", "publish must not move aliases; promote does after signing")
|
||||
}
|
||||
}
|
||||
last := -1
|
||||
for _, id := range []string{"merge", "push", "sign", "verify"} {
|
||||
i, ok := index[id]
|
||||
if !ok {
|
||||
add("P_ORDER", "publish has no step with id %s", id)
|
||||
continue
|
||||
}
|
||||
if i < last {
|
||||
add("P_ORDER", "step %s runs before an earlier stage of merge, push, sign, verify", id)
|
||||
}
|
||||
last = i
|
||||
}
|
||||
if i, ok := index["sign"]; ok {
|
||||
s := text(steps[i])
|
||||
for _, need := range []string{"cosign sign", "--new-bundle-format", "--registry-referrers-mode=oci-1-1", "@${DIGEST}"} {
|
||||
if !strings.Contains(s, need) {
|
||||
add("P_SIGNER", "sign step lacks %q", need)
|
||||
}
|
||||
}
|
||||
}
|
||||
if i, ok := index["verify"]; ok {
|
||||
s := text(steps[i])
|
||||
for _, need := range []string{"cosign verify", "--certificate-identity ", "--certificate-oidc-issuer ", SignerWorkflow, "--new-bundle-format"} {
|
||||
if !strings.Contains(s, need) {
|
||||
add("P_SIGNER", "verify step lacks %q", need)
|
||||
}
|
||||
}
|
||||
if strings.Contains(s, "identity-regexp") || strings.Contains(s, ".*") {
|
||||
add("P_SIGNER", "verify step must pin an exact identity, not a pattern")
|
||||
}
|
||||
}
|
||||
// Promote: the alias command, serialized per channel.
|
||||
if !strings.Contains(text(pro), "addonpkg promote") {
|
||||
add("P_ORDER", "promote job does not run addonpkg promote")
|
||||
}
|
||||
conc, _ := pro["concurrency"].(map[string]any)
|
||||
if !strings.Contains(fmt.Sprint(conc["group"]), "needs.plan.outputs.channel") || conc["cancel-in-progress"] != false {
|
||||
add("P_CONCURRENCY", "promote must serialize per channel with cancel-in-progress: false")
|
||||
}
|
||||
conc, _ = pub["concurrency"].(map[string]any)
|
||||
if !strings.Contains(fmt.Sprint(conc["group"]), "needs.plan.outputs.immutable") || conc["cancel-in-progress"] != false {
|
||||
add("P_CONCURRENCY", "publish must serialize per immutable tag with cancel-in-progress: false")
|
||||
}
|
||||
// Plan fails closed on missing configuration.
|
||||
plan := text(job("plan"))
|
||||
for _, need := range []string{"scripts/addons/channel.sh", "addonpkg matrix", `-z "$ADDON_OCI_REPOSITORY"`, "has no enabled platform; refusing to publish"} {
|
||||
if !strings.Contains(plan, need) {
|
||||
add("P_FAIL_CLOSED", "plan job lacks %q", need)
|
||||
}
|
||||
}
|
||||
// Only the "no enabled platform" answer of the matrix command may drop an
|
||||
// add-on, and only on an event that publishes nothing. Any other error, such
|
||||
// as a missing or malformed platform file, fails the plan.
|
||||
if !regexp.MustCompile(`(?s)elif grep -q 'no enabled platform' "\$err"; then\s+if \[ "\$publish" = "true" \]; then\s+echo "add-on \$id has no enabled platform; refusing to publish" >&2\s+exit 1\s+fi\s+continue\s+else\s+.*?exit 1\s+fi`).MatchString(plan) {
|
||||
add("P_FAIL_CLOSED", "plan must exit 1 for an add-on without an enabled platform on a publishing event, skip it on others, and fail on any other matrix error")
|
||||
}
|
||||
// Checkout never takes code of a pull request head into a job that could
|
||||
// hold credentials, and never persists the token.
|
||||
for n, v := range jobs {
|
||||
j, _ := v.(map[string]any)
|
||||
steps, _ := j["steps"].([]any)
|
||||
for _, s := range steps {
|
||||
st, _ := s.(map[string]any)
|
||||
uses, _ := st["uses"].(string)
|
||||
if !strings.HasPrefix(uses, "actions/checkout@") {
|
||||
continue
|
||||
}
|
||||
w, _ := st["with"].(map[string]any)
|
||||
if strings.Contains(fmt.Sprint(w["ref"]), "pull_request.head") || strings.Contains(fmt.Sprint(w["ref"]), "head_ref") {
|
||||
add("P_CHECKOUT", "job %s checks out a pull request head", n)
|
||||
}
|
||||
if w["persist-credentials"] != false {
|
||||
add("P_CHECKOUT", "job %s must set persist-credentials: false", n)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cmdPolicy(args []string) error {
|
||||
fs := flag.NewFlagSet("policy", flag.ContinueOnError)
|
||||
file := fs.String("workflow", ".github/workflows/addons-package.yml", "workflow file")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := os.ReadFile(*file)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
problems := checkWorkflow(b)
|
||||
if len(problems) > 0 {
|
||||
return errors.New(strings.Join(problems, "\n"))
|
||||
}
|
||||
_, _ = fmt.Fprintf(stdout, "workflow-policy=passed\n")
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,528 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/google/go-containerregistry/pkg/authn"
|
||||
"github.com/google/go-containerregistry/pkg/name"
|
||||
v1 "github.com/google/go-containerregistry/pkg/v1"
|
||||
"github.com/google/go-containerregistry/pkg/v1/layout"
|
||||
"github.com/google/go-containerregistry/pkg/v1/remote"
|
||||
"github.com/google/go-containerregistry/pkg/v1/remote/transport"
|
||||
)
|
||||
|
||||
var (
|
||||
immutableTagRE = regexp.MustCompile(`^(sha-[0-9a-f]{12}|(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-[0-9A-Za-z]+(\.[0-9A-Za-z-]+)*)?)$`)
|
||||
branchAliasRE = regexp.MustCompile(`^branch-[a-z0-9]+(-[a-z0-9]+)*-[0-9a-f]{12}$`)
|
||||
)
|
||||
|
||||
// backendRepositoryPrefixes are repositories that belong to the backend
|
||||
// image pipeline. Add-on packages never go there, whatever the input says.
|
||||
var backendRepositoryPrefixes = []string{
|
||||
"quay.io/go-skynet/",
|
||||
"index.docker.io/localai/",
|
||||
"docker.io/localai/",
|
||||
"ghcr.io/mudler/localai",
|
||||
}
|
||||
|
||||
// addonRepositoryNamespace is the one repository under the backend registry
|
||||
// organization that belongs to add-on packages. It is the only exception to
|
||||
// backendRepositoryPrefixes.
|
||||
const addonRepositoryNamespace = "quay.io/go-skynet/local-ai-addons"
|
||||
|
||||
// parseRepository validates the registry and repository input. It has no
|
||||
// default: an empty value is an error, so a missing configuration variable
|
||||
// fails the run instead of publishing somewhere unintended.
|
||||
func parseRepository(s string) (name.Repository, error) {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
return name.Repository{}, errors.New("repository input is required: set the add-on registry and repository configuration")
|
||||
}
|
||||
if strings.ContainsAny(s, "@: ") && !strings.Contains(s, ":") {
|
||||
return name.Repository{}, errors.New("repository must not carry a tag or digest")
|
||||
}
|
||||
repo, err := name.NewRepository(s, name.StrictValidation)
|
||||
if err != nil {
|
||||
return name.Repository{}, fmt.Errorf("repository %q: %w", s, err)
|
||||
}
|
||||
full := strings.ToLower(repo.Name())
|
||||
if full == addonRepositoryNamespace || strings.HasPrefix(full, addonRepositoryNamespace+"/") {
|
||||
return repo, nil
|
||||
}
|
||||
for _, p := range backendRepositoryPrefixes {
|
||||
if strings.HasPrefix(full+"/", p) || strings.HasPrefix(full, p) {
|
||||
return name.Repository{}, fmt.Errorf("repository %q is a backend image repository; add-on packages use their own namespace", s)
|
||||
}
|
||||
}
|
||||
return repo, nil
|
||||
}
|
||||
|
||||
func isNotFound(err error) bool {
|
||||
var te *transport.Error
|
||||
return errors.As(err, &te) && te.StatusCode == 404
|
||||
}
|
||||
|
||||
func remoteOptions(ctx context.Context) []remote.Option {
|
||||
return []remote.Option{remote.WithContext(ctx), remote.WithAuthFromKeychain(authn.DefaultKeychain)}
|
||||
}
|
||||
|
||||
// guardImmutable refuses a tag that already names different content.
|
||||
func guardImmutable(ctx context.Context, tag name.Tag, digest v1.Hash) (exists bool, err error) {
|
||||
d, err := remote.Head(tag, remoteOptions(ctx)...)
|
||||
if err != nil {
|
||||
if isNotFound(err) {
|
||||
return false, nil
|
||||
}
|
||||
return false, fmt.Errorf("reading %s: %w", tag, err)
|
||||
}
|
||||
if d.Digest != digest {
|
||||
return true, fail("E_IMMUTABLE", "%s already names %s; an immutable tag is never overwritten with %s", tag.Identifier(), d.Digest, digest)
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// publishIndex pushes a verified merged layout: every platform image and the
|
||||
// index by digest first, then the immutable tag. Re-running it with the same
|
||||
// layout resumes: pushes of existing blobs are no-ops and the tag already
|
||||
// names the same digest.
|
||||
func publishIndex(ctx context.Context, dir string, cfg *PlatformsFile, repo name.Repository, immutable string, adopt bool) (v1.Hash, error) {
|
||||
var zero v1.Hash
|
||||
if !immutableTagRE.MatchString(immutable) {
|
||||
return zero, fail("E_RESERVED", "%q is not an immutable version or commit tag", immutable)
|
||||
}
|
||||
m, digest, err := verifyMerged(dir, cfg)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
if strings.HasPrefix(immutable, "sha-") {
|
||||
if len(m.Revision) != 40 || immutable != "sha-"+m.Revision[:12] {
|
||||
return zero, fail("E_RESERVED", "commit tag %q does not name the source commit of the layout", immutable)
|
||||
}
|
||||
} else if immutable != m.Version {
|
||||
return zero, fail("E_RESERVED", "version tag %q differs from the layout version %q", immutable, m.Version)
|
||||
}
|
||||
tag := repo.Tag(immutable)
|
||||
if _, err := guardImmutable(ctx, tag, digest); err != nil {
|
||||
if adopt && strings.HasPrefix(err.Error(), "E_IMMUTABLE") {
|
||||
// A rebuild of the same version and commit differs in bytes only
|
||||
// when the build is not reproducible. The published content is
|
||||
// kept, never replaced, after it passes the same completeness
|
||||
// rules; the workflow verifies its signature next.
|
||||
if d, aerr := adoptExisting(ctx, repo, tag, m, cfg); aerr == nil {
|
||||
return d, nil
|
||||
} else {
|
||||
return zero, fmt.Errorf("%w; cannot adopt: %v", err, aerr)
|
||||
}
|
||||
}
|
||||
return zero, err
|
||||
}
|
||||
lp, err := layout.FromPath(dir)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
root, err := lp.ImageIndex()
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
idx, err := root.ImageIndex(digest)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
opts := remoteOptions(ctx)
|
||||
if err := remote.WriteIndex(repo.Digest(digest.String()), idx, opts...); err != nil {
|
||||
return zero, fmt.Errorf("pushing %s: %w", digest, err)
|
||||
}
|
||||
// A second look closes the window between the first check and the tag.
|
||||
if _, err := guardImmutable(ctx, tag, digest); err != nil {
|
||||
return zero, err
|
||||
}
|
||||
if err := remote.Tag(tag, idx, opts...); err != nil {
|
||||
return zero, fmt.Errorf("tagging %s: %w", tag, err)
|
||||
}
|
||||
return digest, nil
|
||||
}
|
||||
|
||||
// adoptExisting accepts the content that an immutable tag already names when
|
||||
// it is a complete package index of the same add-on, version and commit.
|
||||
func adoptExisting(ctx context.Context, repo name.Repository, tag name.Tag, m *merged, cfg *PlatformsFile) (v1.Hash, error) {
|
||||
var zero v1.Hash
|
||||
d, err := remote.Get(tag, remoteOptions(ctx)...)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
idx, err := d.ImageIndex()
|
||||
if err != nil {
|
||||
return zero, errors.New("existing tag does not name an image index")
|
||||
}
|
||||
im, err := idx.IndexManifest()
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
a := im.Annotations
|
||||
if im.ArtifactType != IndexArtifactType || a[AnnotationID] != cfg.Addon || a[AnnotationVersion] != m.Version || a[AnnotationRevision] != m.Revision || m.Revision == "" {
|
||||
return zero, errors.New("existing index was not built from the same add-on, version and commit")
|
||||
}
|
||||
enabled, err := cfg.enabled()
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
have := map[string]bool{}
|
||||
for _, c := range im.Manifests {
|
||||
if c.Platform != nil {
|
||||
have[c.Platform.OS+"/"+c.Platform.Architecture] = true
|
||||
}
|
||||
}
|
||||
if len(have) != len(enabled) || len(im.Manifests) != len(enabled) {
|
||||
return zero, errors.New("existing index does not match the enabled platforms")
|
||||
}
|
||||
for _, e := range enabled {
|
||||
if !have[e.Platform] {
|
||||
return zero, fmt.Errorf("existing index has no %s image", e.Platform)
|
||||
}
|
||||
}
|
||||
return d.Digest, nil
|
||||
}
|
||||
|
||||
func cmdPublish(args []string) error {
|
||||
fs := flag.NewFlagSet("publish", flag.ContinueOnError)
|
||||
dir := fs.String("layout", "", "merged layout")
|
||||
repository := fs.String("repository", "", "registry and repository, for example registry.example/owner/addon-tables")
|
||||
immutable := fs.String("tag", "", "immutable version or commit tag")
|
||||
config := fs.String("config", "", "platform matrix file")
|
||||
adopt := fs.Bool("adopt-existing", false, "keep a published index of the same version and commit when a rebuild differs")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
repo, err := parseRepository(*repository)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cfg, err := loadPlatforms(*config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if *dir == "" {
|
||||
return errors.New("--layout is required")
|
||||
}
|
||||
ctx := context.Background()
|
||||
d, err := publishIndex(ctx, *dir, cfg, repo, *immutable, *adopt)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
children, err := childDigests(ctx, repo, d)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = fmt.Fprintf(stdout, "digest=%s\nreference=%s@%s\nchildren=%s\n", d, repo.Name(), d, strings.Join(children, ","))
|
||||
return nil
|
||||
}
|
||||
|
||||
// Promotion outcomes.
|
||||
const (
|
||||
promoted = "promoted"
|
||||
unchanged = "unchanged"
|
||||
stale = "stale"
|
||||
)
|
||||
|
||||
type aliasState struct {
|
||||
digest v1.Hash
|
||||
version string
|
||||
revision string
|
||||
}
|
||||
|
||||
func readAlias(ctx context.Context, repo name.Repository, alias string) (*aliasState, error) {
|
||||
ref := repo.Tag(alias)
|
||||
d, err := remote.Get(ref, remoteOptions(ctx)...)
|
||||
if err != nil {
|
||||
if isNotFound(err) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
st := &aliasState{digest: d.Digest}
|
||||
idx, err := d.ImageIndex()
|
||||
if err != nil {
|
||||
return nil, fail("E_STALE", "alias %s does not name an image index", alias)
|
||||
}
|
||||
im, err := idx.IndexManifest()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
st.version, st.revision = im.Annotations[AnnotationVersion], im.Annotations[AnnotationRevision]
|
||||
return st, nil
|
||||
}
|
||||
|
||||
func compareSemver(a, b string) int {
|
||||
pa, pb := splitSemver(a), splitSemver(b)
|
||||
for i := 0; i < 3; i++ {
|
||||
if pa.num[i] != pb.num[i] {
|
||||
if pa.num[i] < pb.num[i] {
|
||||
return -1
|
||||
}
|
||||
return 1
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case pa.pre == "" && pb.pre == "":
|
||||
return 0
|
||||
case pa.pre == "":
|
||||
return 1
|
||||
case pb.pre == "":
|
||||
return -1
|
||||
}
|
||||
as, bs := strings.Split(pa.pre, "."), strings.Split(pb.pre, ".")
|
||||
for i := 0; i < len(as) && i < len(bs); i++ {
|
||||
if as[i] == bs[i] {
|
||||
continue
|
||||
}
|
||||
an, aok := atoi(as[i])
|
||||
bn, bok := atoi(bs[i])
|
||||
switch {
|
||||
case aok && bok:
|
||||
if an < bn {
|
||||
return -1
|
||||
}
|
||||
return 1
|
||||
case aok:
|
||||
return -1
|
||||
case bok:
|
||||
return 1
|
||||
case as[i] < bs[i]:
|
||||
return -1
|
||||
default:
|
||||
return 1
|
||||
}
|
||||
}
|
||||
return len(as) - len(bs)
|
||||
}
|
||||
|
||||
type semver struct {
|
||||
num [3]int
|
||||
pre string
|
||||
}
|
||||
|
||||
func splitSemver(v string) semver {
|
||||
core, pre, _ := strings.Cut(v, "-")
|
||||
var s semver
|
||||
for i, p := range strings.SplitN(core, ".", 3) {
|
||||
n, _ := atoi(p)
|
||||
if i < 3 {
|
||||
s.num[i] = n
|
||||
}
|
||||
}
|
||||
s.pre = pre
|
||||
return s
|
||||
}
|
||||
|
||||
func atoi(s string) (int, bool) {
|
||||
if s == "" {
|
||||
return 0, false
|
||||
}
|
||||
n := 0
|
||||
for _, r := range s {
|
||||
if r < '0' || r > '9' || n > 1<<40 {
|
||||
return 0, false
|
||||
}
|
||||
n = n*10 + int(r-'0')
|
||||
}
|
||||
return n, true
|
||||
}
|
||||
|
||||
// errCommitUnreachable reports that git does not have one of the commits.
|
||||
// It is not the same as "not an ancestor": the history that would answer the
|
||||
// question is missing from the checkout.
|
||||
var errCommitUnreachable = errors.New("commit is not reachable in this checkout")
|
||||
|
||||
// isAncestor asks git whether candidate (the commit of this run) is an
|
||||
// ancestor of named (the commit an alias names). It returns
|
||||
// errCommitUnreachable when named is unknown to the repository, as happens
|
||||
// after a force-push. A missing candidate and any other git failure are plain
|
||||
// errors.
|
||||
func isAncestor(ctx context.Context, gitDir, candidate, named string) (bool, error) {
|
||||
if !shaRE.MatchString(candidate) || !shaRE.MatchString(named) {
|
||||
return false, errors.New("commits must be 40 hex characters")
|
||||
}
|
||||
var stderr bytes.Buffer
|
||||
cmd := exec.CommandContext(ctx, "git", "--git-dir", gitDir, "merge-base", "--is-ancestor", candidate, named) // #nosec G204 -- fixed git arguments; the revisions are 40-hex commit identifiers checked by the caller
|
||||
cmd.Stderr = &stderr
|
||||
err := cmd.Run()
|
||||
var ee *exec.ExitError
|
||||
switch {
|
||||
case err == nil:
|
||||
return true, nil
|
||||
case errors.As(err, &ee) && ee.ExitCode() == 1:
|
||||
return false, nil
|
||||
case errors.As(err, &ee) && ee.ExitCode() == 128 && strings.Contains(stderr.String(), "Not a valid commit name"):
|
||||
// Which side is missing? This run's commit must be in the checkout.
|
||||
if !commitExists(ctx, gitDir, candidate) {
|
||||
return false, fmt.Errorf("git merge-base: commit %s of this run is missing: %w", candidate, err)
|
||||
}
|
||||
return false, fmt.Errorf("%w: %s", errCommitUnreachable, named)
|
||||
}
|
||||
return false, fmt.Errorf("git merge-base: %w: %s", err, strings.TrimSpace(stderr.String()))
|
||||
}
|
||||
|
||||
func commitExists(ctx context.Context, gitDir, sha string) bool {
|
||||
return exec.CommandContext(ctx, "git", "--git-dir", gitDir, "cat-file", "-e", sha+"^{commit}").Run() == nil // #nosec G204 -- fixed git arguments; the revisions are 40-hex commit identifiers checked by the caller
|
||||
}
|
||||
|
||||
// promotionSource says which event asks for the promotion.
|
||||
type promotionSource struct {
|
||||
Ref string // full git ref of the event, for example refs/heads/feature/x
|
||||
Event string // GitHub event name
|
||||
}
|
||||
|
||||
// pushToOwnBranch reports whether the event is a push to the ref that owns
|
||||
// this branch channel. The channel name is derived from the ref, so the name
|
||||
// itself is the record of which ref the channel follows.
|
||||
func (s promotionSource) pushToOwnBranch(alias string) bool {
|
||||
return s.Event == "push" && strings.HasPrefix(s.Ref, "refs/heads/") && branchTag(s.Ref) == alias
|
||||
}
|
||||
|
||||
// promoteAlias moves one moving tag to a digest that was already published,
|
||||
// verified and signed. It never moves an alias backwards: a slower, older run
|
||||
// finds a newer alias and leaves it alone.
|
||||
func promoteAlias(ctx context.Context, repo name.Repository, alias string, digest v1.Hash, version, revision, gitDir string) (string, error) {
|
||||
return promoteAliasFrom(ctx, repo, alias, digest, version, revision, gitDir, promotionSource{})
|
||||
}
|
||||
|
||||
// promoteAliasFrom is promoteAlias with the event that asks for the move. A
|
||||
// branch channel whose current commit is gone from the checkout (the branch
|
||||
// was force-pushed) accepts only a push to its own ref as the new truth.
|
||||
func promoteAliasFrom(ctx context.Context, repo name.Repository, alias string, digest v1.Hash, version, revision, gitDir string, src promotionSource) (string, error) {
|
||||
switch {
|
||||
case alias == aliasLatest:
|
||||
if !versionRE.MatchString(version) || strings.Contains(version, "-") {
|
||||
return "", fail("E_RESERVED", "latest only follows a stable release version, got %q", version)
|
||||
}
|
||||
case alias == aliasDevelopment || branchAliasRE.MatchString(alias):
|
||||
if !shaRE.MatchString(revision) {
|
||||
return "", fail("E_RESERVED", "alias %s needs the source commit", alias)
|
||||
}
|
||||
default:
|
||||
return "", fail("E_RESERVED", "%q is not a channel alias", alias)
|
||||
}
|
||||
cur, err := readAlias(ctx, repo, alias)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
// The target must already exist in the registry: a promotion never
|
||||
// creates content, it only names content that was published and signed.
|
||||
target, err := remote.Get(repo.Digest(digest.String()), remoteOptions(ctx)...)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("target %s is not published: %w", digest, err)
|
||||
}
|
||||
if cur != nil {
|
||||
if cur.digest == digest {
|
||||
return unchanged, nil
|
||||
}
|
||||
if alias == aliasLatest {
|
||||
if cur.version != "" && version != "" {
|
||||
switch c := compareSemver(version, cur.version); {
|
||||
case c < 0:
|
||||
return stale, nil
|
||||
case c == 0:
|
||||
return "", fail("E_IMMUTABLE", "latest already names version %s with another digest %s", cur.version, cur.digest)
|
||||
}
|
||||
}
|
||||
} else if cur.revision != "" {
|
||||
if cur.revision == revision {
|
||||
return "", fail("E_IMMUTABLE", "channel %s already names commit %s with another digest %s", alias, revision, cur.digest)
|
||||
}
|
||||
// The new commit is older than the one the alias names: a slower
|
||||
// run. A descendant, or a diverged commit, is the new truth of
|
||||
// the branch. When the named commit is gone from the checkout
|
||||
// the branch was rewritten: only a push to the channel's own
|
||||
// branch may take it over. Everything else fails closed.
|
||||
older, err := isAncestor(ctx, gitDir, revision, cur.revision)
|
||||
switch {
|
||||
case errors.Is(err, errCommitUnreachable):
|
||||
if alias == aliasDevelopment || !src.pushToOwnBranch(alias) {
|
||||
return "", fail("E_UNREACHABLE", "channel %s names commit %s, which is not in this checkout; only a push to the channel's own branch may replace it", alias, cur.revision)
|
||||
}
|
||||
older = false
|
||||
case err != nil:
|
||||
return "", err
|
||||
}
|
||||
if older {
|
||||
return stale, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
idx, err := target.ImageIndex()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := remote.Tag(repo.Tag(alias), idx, remoteOptions(ctx)...); err != nil {
|
||||
return "", fmt.Errorf("tagging %s: %w", alias, err)
|
||||
}
|
||||
return promoted, nil
|
||||
}
|
||||
|
||||
func cmdPromote(args []string) error {
|
||||
fs := flag.NewFlagSet("promote", flag.ContinueOnError)
|
||||
repository := fs.String("repository", "", "registry and repository")
|
||||
alias := fs.String("alias", "", "latest, development or a branch channel tag")
|
||||
digest := fs.String("digest", "", "published index digest")
|
||||
version := fs.String("version", "", "release version of the digest")
|
||||
revision := fs.String("revision", "", "source commit of the digest")
|
||||
gitDir := fs.String("git-dir", ".git", "git directory with the history of both commits")
|
||||
ref := fs.String("ref", "", "git ref of the event that asks for the promotion")
|
||||
event := fs.String("event", "", "GitHub event name that asks for the promotion")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
repo, err := parseRepository(*repository)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !digestRE.MatchString(*digest) {
|
||||
return errors.New("--digest must be sha256:<64 hex>")
|
||||
}
|
||||
h, err := v1.NewHash(*digest)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||
defer cancel()
|
||||
if _, err := os.Stat(*gitDir); err != nil && *alias != aliasLatest {
|
||||
return fmt.Errorf("git directory: %w", err)
|
||||
}
|
||||
out, err := promoteAliasFrom(ctx, repo, *alias, h, *version, *revision, *gitDir, promotionSource{Ref: *ref, Event: *event})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = fmt.Fprintf(stdout, "result=%s\nalias=%s\ndigest=%s\n", out, *alias, h)
|
||||
return nil
|
||||
}
|
||||
|
||||
// childDigests lists the platform images of a published index, so the
|
||||
// workflow can verify each one the way an installer will.
|
||||
func childDigests(ctx context.Context, repo name.Repository, digest v1.Hash) ([]string, error) {
|
||||
d, err := remote.Get(repo.Digest(digest.String()), remoteOptions(ctx)...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
idx, err := d.ImageIndex()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
im, err := idx.IndexManifest()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []string
|
||||
for _, c := range im.Manifests {
|
||||
out = append(out, c.Digest.String())
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
// Package packaging builds, checks, merges and publishes first-party add-on
|
||||
// OCI packages. It is the code behind scripts/addons/package.sh,
|
||||
// scripts/addons/channel.sh and the add-on package workflow, so the rules
|
||||
// that workflow relies on are code that a test can run.
|
||||
//
|
||||
// Every operation fails closed: a missing input is an error, never a default.
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
)
|
||||
|
||||
// stdout receives the key=value result lines of the subcommands.
|
||||
var stdout io.Writer = os.Stdout
|
||||
|
||||
var commands = map[string]func(args []string) error{
|
||||
"matrix": cmdMatrix,
|
||||
"channel": cmdChannel,
|
||||
"build": cmdBuild,
|
||||
"verify": cmdVerify,
|
||||
"extract": cmdExtract,
|
||||
"native-check": cmdNativeCheck,
|
||||
"merge": cmdMerge,
|
||||
"publish": cmdPublish,
|
||||
"promote": cmdPromote,
|
||||
"catalog": cmdCatalog,
|
||||
"policy": cmdPolicy,
|
||||
}
|
||||
|
||||
// Run executes one subcommand of the addonpkg command line.
|
||||
func Run(args []string, out io.Writer) error {
|
||||
if len(args) == 0 {
|
||||
return fmt.Errorf("usage: addonpkg <subcommand> [flags]")
|
||||
}
|
||||
cmd, ok := commands[args[0]]
|
||||
if !ok {
|
||||
return fmt.Errorf("unknown subcommand %q", args[0])
|
||||
}
|
||||
previous := stdout
|
||||
stdout = out
|
||||
defer func() { stdout = previous }()
|
||||
return cmd(args[1:])
|
||||
}
|
||||
@@ -0,0 +1,373 @@
|
||||
// SPDX-License-Identifier: MIT
|
||||
package packaging
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/google/go-containerregistry/pkg/v1/types"
|
||||
"github.com/mudler/LocalAI/pkg/addons/contract"
|
||||
)
|
||||
|
||||
// Limits of one package, in line with the installer's defaults.
|
||||
const (
|
||||
maxPackageBytes = 512 << 20
|
||||
maxPackageFiles = 10000
|
||||
)
|
||||
|
||||
// secondReact lists markers that only appear in a bundled React runtime. The
|
||||
// host supplies React through the UI SDK; a package that carries its own
|
||||
// copy would run two Reacts on one page.
|
||||
var secondReact = []string{
|
||||
"__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED",
|
||||
"__CLIENT_INTERNALS_DO_NOT_USE_OR_WARN_USERS_THEY_CANNOT_UPGRADE",
|
||||
"react-dom.production",
|
||||
"react.production",
|
||||
"ReactCurrentDispatcher",
|
||||
"ReactDOMCurrentDispatcher",
|
||||
}
|
||||
|
||||
// extracted is one validated file of a package.
|
||||
type extracted struct {
|
||||
role string
|
||||
name string
|
||||
data []byte
|
||||
exec bool
|
||||
}
|
||||
|
||||
// Verified is what a successful verification learned about a leg.
|
||||
type Verified struct {
|
||||
Receipt Receipt
|
||||
Manifest contract.Manifest
|
||||
Files []extracted
|
||||
}
|
||||
|
||||
// openLayers decompresses and parses every layer with the installer's own
|
||||
// path rules: clean relative names, regular files and directories only, no
|
||||
// duplicate across layers, bounded totals. It also enforces package rules the
|
||||
// installer does not: no set-id or write bits and execute only where declared.
|
||||
func (l *leg) openLayers() ([]extracted, error) {
|
||||
var out []extracted
|
||||
seen := map[string]bool{}
|
||||
var total int64
|
||||
for i, d := range l.manifest.Layers {
|
||||
if d.MediaType != types.OCILayer {
|
||||
return nil, fail("E_LAYERS", "layer %d has media type %q, expected %q", i, d.MediaType, types.OCILayer)
|
||||
}
|
||||
role := d.Annotations[AnnotationLayer]
|
||||
blob, err := readBlob(l.dir, d.Digest, d.Size)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
gz, err := gzip.NewReader(bytes.NewReader(blob))
|
||||
if err != nil {
|
||||
return nil, fail("E_LAYERS", "layer %d is not gzip: %v", i, err)
|
||||
}
|
||||
t := tar.NewReader(gz)
|
||||
for {
|
||||
h, err := t.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fail("E_LAYERS", "layer %d: %v", i, err)
|
||||
}
|
||||
n := strings.TrimSuffix(h.Name, "/")
|
||||
if n == "" || n == "." || path.Clean(n) != n || strings.HasPrefix(n, "/") || n == ".." || strings.HasPrefix(n, "../") || strings.ContainsAny(n, "\\:\x00") || seen[n] {
|
||||
return nil, fail("E_PATH", "unsafe or duplicate path %q", h.Name)
|
||||
}
|
||||
seen[n] = true
|
||||
if h.Typeflag == tar.TypeDir {
|
||||
continue
|
||||
}
|
||||
if h.Typeflag != tar.TypeReg {
|
||||
return nil, fail("E_PATH", "%q is not a regular file", h.Name)
|
||||
}
|
||||
if h.Mode&0o7000 != 0 || h.Mode&0o022 != 0 {
|
||||
return nil, fail("E_MODE", "%q has mode %04o: set-id, sticky and group or world write bits are not allowed", n, h.Mode)
|
||||
}
|
||||
total += h.Size
|
||||
if h.Size < 0 || total > maxPackageBytes || len(out) >= maxPackageFiles {
|
||||
return nil, fail("E_LAYERS", "package exceeds its size or file limit")
|
||||
}
|
||||
data, err := readAll(t, h.Size)
|
||||
if err != nil || int64(len(data)) != h.Size {
|
||||
return nil, fail("E_LAYERS", "layer %d: short or oversized file %q", i, n)
|
||||
}
|
||||
out = append(out, extracted{role: role, name: n, data: data, exec: h.Mode&0o111 != 0})
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// verifyLeg applies every package rule to one platform layout.
|
||||
func verifyLeg(dir, platform, version string, cfg *PlatformsFile) (*Verified, error) {
|
||||
l, rec, err := loadLeg(dir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return verifyLoaded(l, rec, platform, version, cfg)
|
||||
}
|
||||
|
||||
func verifyLoaded(l *leg, rec *Receipt, platform, version string, cfg *PlatformsFile) (*Verified, error) {
|
||||
entry, ok := cfg.find(platform)
|
||||
if !ok {
|
||||
return nil, fail("E_PLATFORM", "platform %s is not in the platform matrix", platform)
|
||||
}
|
||||
// Platform: descriptor, annotation, config and receipt must all agree.
|
||||
osName, arch, _ := strings.Cut(platform, "/")
|
||||
p := l.descriptor.Platform
|
||||
if p == nil || p.OS != osName || p.Architecture != arch || l.config.OS != osName || l.config.Architecture != arch ||
|
||||
l.manifest.Annotations[AnnotationPlatform] != platform || rec.Platform != platform {
|
||||
return nil, fail("E_PLATFORM", "layout is not a %s image", platform)
|
||||
}
|
||||
if l.manifest.ArtifactType != ArtifactType {
|
||||
return nil, fail("E_ARTIFACT_TYPE", "artifactType is %q, expected %q", l.manifest.ArtifactType, ArtifactType)
|
||||
}
|
||||
if l.manifest.Annotations[AnnotationID] != cfg.Addon || l.manifest.Annotations[AnnotationVersion] != version || rec.Version != version || rec.Addon != cfg.Addon {
|
||||
return nil, fail("E_VERSION", "annotations or receipt do not name %s %s", cfg.Addon, version)
|
||||
}
|
||||
if rec.ManifestDigest != l.descriptor.Digest.String() || rec.ConfigDigest != l.manifest.Config.Digest.String() {
|
||||
return nil, fail("E_DIGEST", "receipt digests do not match the image")
|
||||
}
|
||||
if len(rec.Layers) != len(l.manifest.Layers) {
|
||||
return nil, fail("E_DIGEST", "receipt lists %d layers, image has %d", len(rec.Layers), len(l.manifest.Layers))
|
||||
}
|
||||
for i, d := range l.manifest.Layers {
|
||||
if rec.Layers[i].Digest != d.Digest.String() || rec.Layers[i].Size != d.Size || rec.Layers[i].MediaType != string(d.MediaType) || rec.Layers[i].Role != d.Annotations[AnnotationLayer] {
|
||||
return nil, fail("E_DIGEST", "receipt layer %d does not match the image", i)
|
||||
}
|
||||
}
|
||||
files, err := l.openLayers()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var manifestBytes []byte
|
||||
for _, f := range files {
|
||||
if f.name == "manifest.json" {
|
||||
manifestBytes = f.data
|
||||
}
|
||||
}
|
||||
if manifestBytes == nil {
|
||||
return nil, fail("E_MANIFEST", "manifest.json is missing")
|
||||
}
|
||||
m, err := contract.ParseManifest(manifestBytes)
|
||||
if err != nil {
|
||||
return nil, fail("E_MANIFEST", "%v", err)
|
||||
}
|
||||
if m.OS != osName || m.Arch != arch {
|
||||
return nil, fail("E_PLATFORM", "manifest targets %s/%s, image is %s", m.OS, m.Arch, platform)
|
||||
}
|
||||
if m.Version != version {
|
||||
return nil, fail("E_VERSION", "manifest version %q, expected %q", m.Version, version)
|
||||
}
|
||||
if m.ID != cfg.Addon {
|
||||
return nil, fail("E_MANIFEST", "manifest id %q, expected %q", m.ID, cfg.Addon)
|
||||
}
|
||||
if err := contract.ValidateManifest(m, hostFor(platform)); err != nil {
|
||||
return nil, fail("E_MANIFEST", "%v", err)
|
||||
}
|
||||
|
||||
// Layer roles: executable, ui when declared, manifest. Nothing else.
|
||||
want := []string{roleExecutable}
|
||||
if m.Capabilities.UI != nil {
|
||||
want = append(want, roleUI)
|
||||
}
|
||||
want = append(want, roleManifest)
|
||||
hasUI := false
|
||||
for _, d := range l.manifest.Layers {
|
||||
hasUI = hasUI || d.Annotations[AnnotationLayer] == roleUI
|
||||
}
|
||||
if m.Capabilities.UI != nil && !hasUI {
|
||||
return nil, fail("E_UI", "the manifest declares a UI and the image has no ui layer")
|
||||
}
|
||||
if len(l.manifest.Layers) != len(want) {
|
||||
return nil, fail("E_LAYERS", "image has %d layers, expected %d (%s)", len(l.manifest.Layers), len(want), strings.Join(want, ", "))
|
||||
}
|
||||
for i, d := range l.manifest.Layers {
|
||||
if d.Annotations[AnnotationLayer] != want[i] {
|
||||
return nil, fail("E_LAYERS", "layer %d has role %q, expected %q", i, d.Annotations[AnnotationLayer], want[i])
|
||||
}
|
||||
}
|
||||
byName := map[string]extracted{}
|
||||
for _, f := range files {
|
||||
byName[f.name] = f
|
||||
}
|
||||
// Entrypoint and runtime dependencies.
|
||||
exe, ok := byName[m.Executable]
|
||||
if !ok || exe.role != roleExecutable {
|
||||
return nil, fail("E_ENTRYPOINT", "executable %q is missing from the executable layer", m.Executable)
|
||||
}
|
||||
if !exe.exec || len(exe.data) == 0 {
|
||||
return nil, fail("E_MODE", "executable %q is empty or not executable", m.Executable)
|
||||
}
|
||||
allowed := map[string]string{m.Executable: roleExecutable, "manifest.json": roleManifest}
|
||||
for _, r := range entry.Runtime {
|
||||
allowed[r] = roleExecutable
|
||||
if f, ok := byName[r]; !ok || f.role != roleExecutable {
|
||||
return nil, fail("E_RUNTIME", "runtime file %q is missing", r)
|
||||
}
|
||||
}
|
||||
if u := m.Capabilities.UI; u != nil {
|
||||
for _, a := range append(append([]string{u.Entry}, u.CSS...), u.Icon) {
|
||||
if a == "" {
|
||||
continue
|
||||
}
|
||||
if f, ok := byName[a]; !ok || f.role != roleUI {
|
||||
return nil, fail("E_UI", "declared UI asset %q is missing", a)
|
||||
}
|
||||
}
|
||||
if u.Icon != "" {
|
||||
if err := checkIcon(u.Icon, byName[u.Icon].data); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
top := path.Dir(u.Entry) + "/"
|
||||
for _, f := range files {
|
||||
if f.role == roleUI && !strings.HasPrefix(f.name, top) {
|
||||
return nil, fail("E_LAYERS", "UI layer file %q is outside %s", f.name, top)
|
||||
}
|
||||
if f.role == roleUI && (strings.HasSuffix(f.name, ".js") || strings.HasSuffix(f.name, ".mjs")) {
|
||||
for _, marker := range secondReact {
|
||||
if bytes.Contains(f.data, []byte(marker)) {
|
||||
return nil, fail("E_REACT", "UI file %q bundles its own React runtime (%s); the host provides React", f.name, marker)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, f := range files {
|
||||
if r, ok := allowed[f.name]; ok {
|
||||
if f.role != r {
|
||||
return nil, fail("E_LAYERS", "file %q is in layer %q, expected %q", f.name, f.role, r)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if f.role != roleUI {
|
||||
return nil, fail("E_LAYERS", "unexpected file %q in the %s layer", f.name, f.role)
|
||||
}
|
||||
if m.Capabilities.UI == nil {
|
||||
return nil, fail("E_LAYERS", "unexpected UI file %q", f.name)
|
||||
}
|
||||
}
|
||||
for _, f := range files {
|
||||
if f.exec && f.name != m.Executable {
|
||||
return nil, fail("E_MODE", "%q is executable and is not the declared entrypoint", f.name)
|
||||
}
|
||||
}
|
||||
return &Verified{Receipt: *rec, Manifest: m, Files: files}, nil
|
||||
}
|
||||
|
||||
func cmdVerify(args []string) error {
|
||||
fs := flag.NewFlagSet("verify", flag.ContinueOnError)
|
||||
dir := fs.String("layout", "", "platform OCI layout")
|
||||
platform := fs.String("platform", "", "expected os/arch")
|
||||
version := fs.String("version", "", "expected version")
|
||||
config := fs.String("config", "addons/tables/package/platforms.yaml", "platform matrix file")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if *dir == "" || *platform == "" || *version == "" {
|
||||
return errors.New("--layout, --platform and --version are required")
|
||||
}
|
||||
cfg, err := loadPlatforms(*config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
v, err := verifyLeg(*dir, *platform, *version, cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, _ = fmt.Fprintf(stdout, "verified=%s %s %s digest=%s files=%d\n", cfg.Addon, *version, *platform, v.Receipt.ManifestDigest, len(v.Files))
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeExtracted restores a verified package to dest. The executable gets its
|
||||
// validated mode back; every other file is 0644.
|
||||
func writeExtracted(v *Verified, dest string) error {
|
||||
for _, f := range v.Files {
|
||||
p := filepath.Join(dest, filepath.FromSlash(f.name))
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o750); err != nil {
|
||||
return err
|
||||
}
|
||||
mode := os.FileMode(0o644)
|
||||
if f.exec {
|
||||
mode = 0o755
|
||||
}
|
||||
if err := os.WriteFile(p, f.data, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Chmod(p, mode); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func cmdExtract(args []string) error {
|
||||
fs := flag.NewFlagSet("extract", flag.ContinueOnError)
|
||||
dir := fs.String("layout", "", "platform OCI layout")
|
||||
platform := fs.String("platform", "", "expected os/arch")
|
||||
version := fs.String("version", "", "expected version")
|
||||
config := fs.String("config", "addons/tables/package/platforms.yaml", "platform matrix file")
|
||||
dest := fs.String("dest", "", "empty destination directory")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if *dest == "" {
|
||||
return errors.New("--dest is required")
|
||||
}
|
||||
if ents, err := os.ReadDir(*dest); err == nil && len(ents) > 0 {
|
||||
return errors.New("destination is not empty")
|
||||
}
|
||||
cfg, err := loadPlatforms(*config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
v, err := verifyLeg(*dir, *platform, *version, cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return writeExtracted(v, *dest)
|
||||
}
|
||||
|
||||
func sortedKeys[V any](m map[string]V) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
// checkIcon applies the installer's icon rule at build time, so that a
|
||||
// package whose icon the host would refuse is never published.
|
||||
func checkIcon(name string, b []byte) error {
|
||||
media := contract.IconTypes[strings.ToLower(path.Ext(name))]
|
||||
if media == "" || len(b) == 0 || int64(len(b)) > contract.MaxIconBytes {
|
||||
return fail("E_UI", "icon %q has an unsupported type or size", name)
|
||||
}
|
||||
ok := false
|
||||
switch media {
|
||||
case "image/png":
|
||||
ok = bytes.HasPrefix(b, []byte("\x89PNG\r\n\x1a\n"))
|
||||
case "image/jpeg":
|
||||
ok = bytes.HasPrefix(b, []byte{0xff, 0xd8, 0xff})
|
||||
case "image/webp":
|
||||
ok = len(b) >= 12 && string(b[:4]) == "RIFF" && string(b[8:12]) == "WEBP"
|
||||
}
|
||||
if !ok {
|
||||
return fail("E_UI", "icon %q does not have the signature of its type", name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
Reference in new issue
Block a user