mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
fix(ui): size model fit against the cluster and move node labels into the selector (#11765)
* fix(ui): move node labels into the scheduling selector field The scheduling page kept a node-label browser open above the rules whether or not anyone was writing one, while the field that actually needs labels, the rule's node selector, was two bare text inputs with no hint of what the cluster reports. The browser is gone. The selector's key input now completes against the label keys the cluster uses, and the value input offers only the values that key takes. The roster already loads for the page, so the suggestions cost no request, and a roster that fails to load costs the admin the hints and nothing else. Suggestions stay suggestions: a key no node reports yet still commits as typed, which is how an admin writes a rule before labelling the nodes for it. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): size model fit against the cluster, not the frontend The models page asked the frontend how much memory a model may occupy. In distributed mode the frontend is usually a GPU-less pod while every model runs on a worker, so a fleet of GPU nodes was told it could only run the smallest CPU build. The variant picker's fits flag and its auto-selection came from the same place, as did the hardware recommendations. The registry now reports the largest single healthy backend node. The largest node, not the fleet total: a model loads into one node, so four 16GB workers are not a home for a 40GB model. An operator-set VRAM budget caps a node's contribution, because the scheduler refuses a load above that ceiling anyway, and a GPU node beats a CPU node holding more system RAM. GET /api/resources and GET /api/models carry this as an additional cluster object. Their aggregate and ram fields keep reporting the frontend's own hardware, which is what the resource monitor shows. Variant selection judges backends against the union of the capabilities present in the cluster, the way backend discovery already did. Every path degrades to the local host: no cluster object in single-node mode, and none when the registry cannot be read, so a hiccup narrows the answer back to single-node behaviour rather than marking the whole catalog too large. The verdicts now name the node they belong to, since a model fits somewhere or nowhere. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
mudler's LocalAI [bot]andEttore Di Giacinto authored and GitHub committed
1 parent
0cdc31dcb3
commit
29899cd1e0
31 files changed
+1387
-356
No files matched your search
@@ -0,0 +1,94 @@
|
||||
package gallery_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/core/gallery"
|
||||
"github.com/mudler/LocalAI/pkg/system"
|
||||
)
|
||||
|
||||
// On a distributed controller the GPUs live on the workers, so a variant
|
||||
// picker sized against the controller tells admins a cluster of A100s can only
|
||||
// run the smallest CPU build.
|
||||
var _ = Describe("ClusterResolveEnv", func() {
|
||||
gib := func(n uint64) uint64 { return n * 1024 * 1024 * 1024 }
|
||||
|
||||
// The controller as Argus actually runs it: no GPU at all.
|
||||
var controller *system.SystemState
|
||||
|
||||
BeforeEach(func() {
|
||||
controller = system.NewCapabilityState("default")
|
||||
})
|
||||
|
||||
It("sizes models against the cluster reading rather than the controller", func() {
|
||||
env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13"})
|
||||
|
||||
Expect(env.AvailableMemory).To(Equal(gib(80)))
|
||||
})
|
||||
|
||||
It("accepts a CUDA backend that only the workers can run", func() {
|
||||
env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13"})
|
||||
|
||||
Expect(env.BackendCompatible).ToNot(BeNil())
|
||||
// A name carrying the cuda token is what the controller rejects today;
|
||||
// a bare engine name like "vllm" passes on any host and would prove
|
||||
// nothing about the union.
|
||||
Expect(env.BackendCompatible("cuda-13-vllm")).To(BeTrue())
|
||||
Expect(env.BackendCompatible("llama-cpp")).To(BeTrue())
|
||||
})
|
||||
|
||||
// The union must stay a filter, not an open door: a Linux NVIDIA fleet
|
||||
// still cannot run an Apple-only build.
|
||||
It("still rejects a backend no node in the cluster can run", func() {
|
||||
env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13"})
|
||||
|
||||
Expect(env.BackendCompatible("mlx")).To(BeFalse())
|
||||
})
|
||||
|
||||
It("accepts a backend that any one node in a mixed fleet can run", func() {
|
||||
env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13", "metal"})
|
||||
|
||||
Expect(env.BackendCompatible("mlx")).To(BeTrue())
|
||||
Expect(env.BackendCompatible("cuda-13-vllm")).To(BeTrue())
|
||||
})
|
||||
|
||||
// Ranking has to follow the hardware too, or a cluster of NVIDIA workers
|
||||
// gets offered the GGUF build over the vLLM one it should prefer.
|
||||
It("ranks engines by the workers' hardware, not the controller's", func() {
|
||||
env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13"})
|
||||
|
||||
Expect(env.EnginePreference).To(Equal(system.NewCapabilityState("nvidia-cuda-13").EnginePreferenceTokens()))
|
||||
})
|
||||
|
||||
// Every degradation path lands here, so it must be indistinguishable from
|
||||
// the single-node behavior that shipped before any of this existed.
|
||||
It("falls back to the host description when the cluster reports nothing", func() {
|
||||
host := gallery.HostResolveEnv(context.Background(), controller)
|
||||
env := gallery.ClusterResolveEnv(context.Background(), controller, 0, nil)
|
||||
|
||||
Expect(env.AvailableMemory).To(Equal(host.AvailableMemory))
|
||||
Expect(env.EnginePreference).To(Equal(host.EnginePreference))
|
||||
Expect(env.BackendCompatible("cuda-13-vllm")).To(Equal(host.BackendCompatible("cuda-13-vllm")))
|
||||
Expect(env.BackendCompatible("mlx")).To(Equal(host.BackendCompatible("mlx")))
|
||||
})
|
||||
|
||||
// A cluster that reports capabilities but no usable memory reading should
|
||||
// still gain the hardware view; only the size question falls back.
|
||||
It("keeps the host memory when only the memory reading is missing", func() {
|
||||
host := gallery.HostResolveEnv(context.Background(), controller)
|
||||
env := gallery.ClusterResolveEnv(context.Background(), controller, 0, []string{"nvidia-cuda-13"})
|
||||
|
||||
Expect(env.AvailableMemory).To(Equal(host.AvailableMemory))
|
||||
Expect(env.BackendCompatible("cuda-13-vllm")).To(BeTrue())
|
||||
})
|
||||
|
||||
It("keeps the probe wired so variant sizes are still measured", func() {
|
||||
env := gallery.ClusterResolveEnv(context.Background(), controller, gib(80), []string{"nvidia-cuda-13"})
|
||||
|
||||
Expect(env.ProbeMemory).ToNot(BeNil())
|
||||
Expect(env.ServingFeaturePreference).To(Equal(system.ServingFeaturePreferenceTokens()))
|
||||
})
|
||||
})
|
||||
@@ -938,3 +938,73 @@ func SafetyScanGalleryModel(galleryModel *GalleryModel) error {
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ClusterResolveEnv describes a CLUSTER to variant selection, where
|
||||
// HostResolveEnv describes one machine.
|
||||
//
|
||||
// It exists because a distributed controller is the wrong machine to ask. The
|
||||
// controller is typically a GPU-less pod while every model actually runs on a
|
||||
// worker, so a picker sized against it reports that a fleet of A100s can only
|
||||
// run the smallest CPU build, and auto-selection then installs exactly that.
|
||||
//
|
||||
// availableMemory is the largest single healthy node's budget, and capabilities
|
||||
// are the capability strings present in the cluster. Either may be empty: a
|
||||
// zero memory reading keeps the host's own figure and an empty capability list
|
||||
// keeps the host's own hardware verdict, so every degradation path lands back
|
||||
// on the single-node behavior rather than on a cluster described as having
|
||||
// nothing.
|
||||
func ClusterResolveEnv(ctx context.Context, systemState *system.SystemState, availableMemory uint64, capabilities []string) ResolveEnv {
|
||||
env := HostResolveEnv(ctx, systemState)
|
||||
|
||||
if availableMemory > 0 {
|
||||
env.AvailableMemory = availableMemory
|
||||
}
|
||||
if len(capabilities) == 0 {
|
||||
return env
|
||||
}
|
||||
|
||||
// One state pinned per capability, mirroring AvailableBackendsForCapabilities:
|
||||
// the controller's own detection must not leak into a worker's verdict, and
|
||||
// a forced capability on the controller image must not either.
|
||||
nodeStates := make([]*system.SystemState, 0, len(capabilities))
|
||||
for _, capability := range capabilities {
|
||||
nodeStates = append(nodeStates, system.NewCapabilityState(capability,
|
||||
system.WithBackendPath(systemState.Backend.BackendsPath)))
|
||||
}
|
||||
|
||||
hostCompatible := env.BackendCompatible
|
||||
// A union, because a variant only has to run SOMEWHERE. The controller
|
||||
// stays in the union so a cluster whose workers all went offline still
|
||||
// describes itself the way it did before distributed mode existed.
|
||||
env.BackendCompatible = func(backend string) bool {
|
||||
if hostCompatible != nil && hostCompatible(backend) {
|
||||
return true
|
||||
}
|
||||
for _, nodeState := range nodeStates {
|
||||
if nodeState.IsBackendCompatible(backend, "") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Ranking follows the same hardware as the filter. Left on the controller's
|
||||
// tokens, an NVIDIA fleet would be offered the GGUF build over the vLLM one
|
||||
// even though nothing filtered the vLLM build out.
|
||||
seen := make(map[string]struct{})
|
||||
preference := make([]string, 0, len(nodeStates))
|
||||
for _, nodeState := range nodeStates {
|
||||
for _, token := range nodeState.EnginePreferenceTokens() {
|
||||
if _, dup := seen[token]; dup {
|
||||
continue
|
||||
}
|
||||
seen[token] = struct{}{}
|
||||
preference = append(preference, token)
|
||||
}
|
||||
}
|
||||
if len(preference) > 0 {
|
||||
env.EnginePreference = preference
|
||||
}
|
||||
|
||||
return env
|
||||
}
|
||||
@@ -356,7 +356,7 @@ func (mgs *BackendEndpointService) UpgradeBackendEndpoint() echo.HandlerFunc {
|
||||
// local system state is the only thing worth filtering against.
|
||||
type ClusterCapabilityProvider func(ctx context.Context) ([]string, error)
|
||||
|
||||
// resolveClusterCapabilities reads the capabilities present in the cluster,
|
||||
// ResolveClusterCapabilities reads the capabilities present in the cluster,
|
||||
// degrading to the local-only listing on error.
|
||||
//
|
||||
// Every capability-filtered discovery endpoint shares this: on a distributed
|
||||
@@ -364,7 +364,7 @@ type ClusterCapabilityProvider func(ctx context.Context) ([]string, error)
|
||||
// (usually GPU-less) host hides GPU-only backends the cluster can actually
|
||||
// run. A registry hiccup must never blank the catalog, so a failure falls back
|
||||
// to the pre-existing local-only behavior rather than erroring the request.
|
||||
func resolveClusterCapabilities(ctx context.Context, provider ClusterCapabilityProvider) []string {
|
||||
func ResolveClusterCapabilities(ctx context.Context, provider ClusterCapabilityProvider) []string {
|
||||
if provider == nil {
|
||||
return nil
|
||||
}
|
||||
@@ -423,7 +423,7 @@ func installedInCluster(backend *gallery.GalleryBackend, clusterInstalled map[st
|
||||
// @Router /backends/available [get]
|
||||
func (mgs *BackendEndpointService) ListAvailableBackendsEndpoint(systemState *system.SystemState, clusterCapabilities ClusterCapabilityProvider, clusterInstalled ClusterInstalledProvider) echo.HandlerFunc {
|
||||
return func(c echo.Context) error {
|
||||
capabilities := resolveClusterCapabilities(c.Request().Context(), clusterCapabilities)
|
||||
capabilities := ResolveClusterCapabilities(c.Request().Context(), clusterCapabilities)
|
||||
|
||||
backends, err := gallery.AvailableBackendsForCapabilities(mgs.galleries, systemState, capabilities)
|
||||
if err != nil {
|
||||
|
||||
@@ -276,7 +276,7 @@ func DownloadExportedModelEndpoint(ftService *finetune.FineTuneService) echo.Han
|
||||
// ListFineTuneBackendsEndpoint returns installed backends tagged with "fine-tuning".
|
||||
func ListFineTuneBackendsEndpoint(appConfig *config.ApplicationConfig, clusterCapabilities ClusterCapabilityProvider, clusterInstalled ClusterInstalledProvider) echo.HandlerFunc {
|
||||
return func(c echo.Context) error {
|
||||
capabilities := resolveClusterCapabilities(c.Request().Context(), clusterCapabilities)
|
||||
capabilities := ResolveClusterCapabilities(c.Request().Context(), clusterCapabilities)
|
||||
installed := resolveClusterInstalled(c.Request().Context(), clusterInstalled)
|
||||
backends, err := gallery.AvailableBackendsForCapabilities(appConfig.BackendGalleries, appConfig.SystemState, capabilities)
|
||||
if err != nil {
|
||||
|
||||
@@ -195,7 +195,7 @@ func DownloadQuantizedModelEndpoint(qService *quantization.QuantizationService)
|
||||
// ListQuantizationBackendsEndpoint returns installed backends tagged with "quantization".
|
||||
func ListQuantizationBackendsEndpoint(appConfig *config.ApplicationConfig, clusterCapabilities ClusterCapabilityProvider, clusterInstalled ClusterInstalledProvider) echo.HandlerFunc {
|
||||
return func(c echo.Context) error {
|
||||
capabilities := resolveClusterCapabilities(c.Request().Context(), clusterCapabilities)
|
||||
capabilities := ResolveClusterCapabilities(c.Request().Context(), clusterCapabilities)
|
||||
installed := resolveClusterInstalled(c.Request().Context(), clusterInstalled)
|
||||
backends, err := gallery.AvailableBackendsForCapabilities(appConfig.BackendGalleries, appConfig.SystemState, capabilities)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
import { test, expect } from "./coverage-fixtures.js";
|
||||
|
||||
// On a distributed controller the models run on the workers, so every "will
|
||||
// this fit" answer on this page is about their hardware. The controller is
|
||||
// usually a GPU-less pod: sized against it, a cluster of A100s is told it can
|
||||
// only run the smallest CPU build.
|
||||
|
||||
const GB = 1024 * 1024 * 1024;
|
||||
|
||||
const MODELS = [
|
||||
{ name: "big-gpu-model", description: "Needs a real GPU", backend: "vllm", installed: false, tags: ["chat"] },
|
||||
];
|
||||
|
||||
// 40GB: far past the controller's 8GB of RAM, comfortably inside one 80GB card.
|
||||
const ESTIMATES = {
|
||||
"big-gpu-model": {
|
||||
sizeBytes: 40 * GB,
|
||||
sizeDisplay: "40.0 GB",
|
||||
estimates: { 8192: { vramBytes: 40 * GB, vramDisplay: "40.0 GB" } },
|
||||
},
|
||||
};
|
||||
|
||||
// The controller as Argus actually runs it: 8GB of system RAM, no GPU.
|
||||
const CONTROLLER_ONLY = {
|
||||
type: "ram",
|
||||
available: true,
|
||||
gpus: [],
|
||||
aggregate: { total_memory: 8 * GB, used_memory: 2 * GB, free_memory: 6 * GB, gpu_count: 0 },
|
||||
};
|
||||
|
||||
const WITH_CLUSTER = {
|
||||
...CONTROLLER_ONLY,
|
||||
cluster: {
|
||||
enabled: true,
|
||||
node_id: "n-1",
|
||||
node_name: "dgx-01",
|
||||
total_memory: 80 * GB,
|
||||
is_gpu: true,
|
||||
node_count: 4,
|
||||
},
|
||||
};
|
||||
|
||||
async function mockModels(page, resources) {
|
||||
await page.route("**/api/models*", (route) =>
|
||||
route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
models: MODELS,
|
||||
allBackends: ["vllm"],
|
||||
allTags: ["chat"],
|
||||
availableModels: MODELS.length,
|
||||
installedModels: 3,
|
||||
totalPages: 1,
|
||||
currentPage: 1,
|
||||
}),
|
||||
}),
|
||||
);
|
||||
await page.route("**/api/models/estimate/*", (route) => {
|
||||
const name = decodeURIComponent(new URL(route.request().url()).pathname.split("/").pop());
|
||||
return route.fulfill({ contentType: "application/json", body: JSON.stringify(ESTIMATES[name] || {}) });
|
||||
});
|
||||
await page.route("**/api/resources", (route) =>
|
||||
route.fulfill({ contentType: "application/json", body: JSON.stringify(resources) }),
|
||||
);
|
||||
}
|
||||
|
||||
const railItems = (page) => page.locator('[data-testid="discover-rail-item"]');
|
||||
const railItem = (page, name) => page.locator(`[data-entity="${name}"]`);
|
||||
const railReady = (page) => expect(railItems(page).first()).toBeVisible({ timeout: 20_000 });
|
||||
const PANE = '[data-testid="discover-pane"]';
|
||||
|
||||
test.describe("Models gallery - cluster-aware fit", () => {
|
||||
test("a model that only a worker can hold is not called too large", async ({ page }) => {
|
||||
await mockModels(page, WITH_CLUSTER);
|
||||
await page.goto("/app/models");
|
||||
|
||||
await railReady(page);
|
||||
|
||||
// The whole defect in one assertion: 40GB against a 4-node cluster whose
|
||||
// largest card holds 80GB.
|
||||
await expect(railItem(page, "big-gpu-model")).toContainText("fits", { timeout: 20_000 });
|
||||
await expect(railItem(page, "big-gpu-model")).not.toContainText("too large");
|
||||
});
|
||||
|
||||
test("the fit verdict names the node it belongs to", async ({ page }) => {
|
||||
await mockModels(page, WITH_CLUSTER);
|
||||
await page.goto("/app/models");
|
||||
|
||||
await railReady(page);
|
||||
await railItem(page, "big-gpu-model").click();
|
||||
// Wait for the detail itself: until it renders, the pane still holds the
|
||||
// zero-state hero, which names the node for its own reasons.
|
||||
await expect(page.locator(PANE).getByText("40.0 GB")).toBeVisible({ timeout: 20_000 });
|
||||
|
||||
// The headroom this model has is headroom SOMEWHERE, and the stat says
|
||||
// where rather than leaving it to read as this machine's.
|
||||
await expect(page.locator(PANE)).toContainText(/headroom on dgx-01/i);
|
||||
});
|
||||
|
||||
test("the host summary describes the cluster, not the controller", async ({ page }) => {
|
||||
await mockModels(page, WITH_CLUSTER);
|
||||
await page.goto("/app/models");
|
||||
|
||||
await railReady(page);
|
||||
// 80 GB is the cluster's best node; 8 GB is this pod's own RAM and must
|
||||
// not be what the page advertises.
|
||||
await expect(page.locator(".zero-pane__title")).toContainText("80 GB");
|
||||
await expect(page.locator(".zero-pane__title")).not.toContainText("8.00 GB");
|
||||
});
|
||||
|
||||
// Single-node behavior is the fallback every degradation path lands on, so
|
||||
// it has to stay exactly as it was.
|
||||
test("without a cluster the verdict is still the local host's", async ({ page }) => {
|
||||
await mockModels(page, CONTROLLER_ONLY);
|
||||
await page.goto("/app/models");
|
||||
|
||||
await railReady(page);
|
||||
await expect(railItem(page, "big-gpu-model")).toContainText("too large", { timeout: 20_000 });
|
||||
});
|
||||
});
|
||||
@@ -36,35 +36,80 @@ async function mockScheduling(page, { rules = [rule], nodeList = nodes } = {}) {
|
||||
}
|
||||
|
||||
test.describe('Scheduling page', () => {
|
||||
test('groups node labels, collapses the reference, filters forgivingly, and expands results', async ({ page }) => {
|
||||
// Node labels are only ever needed while writing a rule's node selector, so
|
||||
// they live in that field rather than in a card standing open above the
|
||||
// rules whether or not anyone is writing one.
|
||||
test('keeps no standing label browser on the page', async ({ page }) => {
|
||||
await mockScheduling(page)
|
||||
await page.goto('/app/scheduling')
|
||||
await expect(page.getByText('llama-3.3')).toBeVisible()
|
||||
|
||||
const reference = page.getByTestId('node-label-reference')
|
||||
await expect(reference.getByText('Falcon GPU')).toBeVisible()
|
||||
await expect(reference.getByText('No labels')).toBeVisible()
|
||||
await expect(reference.locator('.scheduling-node-card')).toHaveCount(5)
|
||||
await expect(reference.getByText('5 of 27 nodes')).toBeVisible()
|
||||
await expect(page.getByTestId('node-label-reference')).toHaveCount(0)
|
||||
await expect(page.getByRole('button', { name: /node labels/i })).toHaveCount(0)
|
||||
await expect(page.locator('.scheduling-node-card')).toHaveCount(0)
|
||||
// Falcon GPU is a node name, and nothing on this page has a reason to
|
||||
// enumerate node names until a selector is being filled.
|
||||
await expect(page.getByText('Falcon GPU')).toHaveCount(0)
|
||||
})
|
||||
|
||||
const toggle = page.getByRole('button', { name: /node labels/i })
|
||||
await expect(toggle).toHaveAttribute('aria-expanded', 'true')
|
||||
await toggle.click()
|
||||
await expect(toggle).toHaveAttribute('aria-expanded', 'false')
|
||||
await expect(reference.getByRole('searchbox')).toBeHidden()
|
||||
await toggle.click()
|
||||
test('suggests the cluster\'s own label keys and values as the selector is typed', async ({ page }) => {
|
||||
await mockScheduling(page)
|
||||
await page.goto('/app/scheduling')
|
||||
await page.getByRole('button', { name: 'Add Scheduling Rule' }).click()
|
||||
|
||||
await reference.getByRole('searchbox').fill('GPU.VENDOR=nvi')
|
||||
await expect(reference.locator('.scheduling-node-card')).toHaveCount(1)
|
||||
await expect(reference.getByText('Falcon GPU')).toBeVisible()
|
||||
const keyInput = page.getByRole('combobox', { name: 'Selector key' })
|
||||
await keyInput.click()
|
||||
const suggestions = page.getByTestId('label-suggestions')
|
||||
// Every key the cluster reports, before a single character is typed.
|
||||
await expect(suggestions.getByRole('option', { name: 'gpu.vendor' })).toBeVisible()
|
||||
await expect(suggestions.getByRole('option', { name: 'zone' })).toBeVisible()
|
||||
|
||||
await reference.getByRole('searchbox').fill('flcn')
|
||||
await expect(reference.locator('.scheduling-node-card')).toHaveCount(1)
|
||||
await expect(reference.getByText('Falcon GPU')).toBeVisible()
|
||||
await keyInput.fill('vend')
|
||||
await expect(suggestions.getByRole('option')).toHaveCount(1)
|
||||
await suggestions.getByRole('option', { name: 'gpu.vendor' }).click()
|
||||
await expect(keyInput).toHaveValue('gpu.vendor')
|
||||
|
||||
await reference.getByRole('searchbox').fill('')
|
||||
await reference.getByRole('button', { name: 'Show 20 more nodes' }).click()
|
||||
await expect(reference.locator('.scheduling-node-card')).toHaveCount(25)
|
||||
await expect(reference.getByText('25 of 27 nodes')).toBeVisible()
|
||||
// Values are scoped to the key being filled, so a selector cannot be built
|
||||
// out of a pair no node matches.
|
||||
const valueInput = page.getByRole('combobox', { name: 'Selector value' })
|
||||
await valueInput.click()
|
||||
await expect(suggestions.getByRole('option', { name: 'NVIDIA' })).toBeVisible()
|
||||
await expect(suggestions.getByRole('option', { name: 'amd' })).toBeVisible()
|
||||
await expect(suggestions.getByRole('option', { name: 'east' })).toHaveCount(0)
|
||||
|
||||
await valueInput.fill('nvi')
|
||||
await suggestions.getByRole('option', { name: 'NVIDIA' }).click()
|
||||
await expect(valueInput).toHaveValue('NVIDIA')
|
||||
})
|
||||
|
||||
test('picks a suggestion from the keyboard', async ({ page }) => {
|
||||
await mockScheduling(page)
|
||||
await page.goto('/app/scheduling')
|
||||
await page.getByRole('button', { name: 'Add Scheduling Rule' }).click()
|
||||
|
||||
const keyInput = page.getByRole('combobox', { name: 'Selector key' })
|
||||
await keyInput.fill('zon')
|
||||
await keyInput.press('ArrowDown')
|
||||
await keyInput.press('Enter')
|
||||
await expect(keyInput).toHaveValue('zone')
|
||||
// Enter picked the suggestion rather than committing the chip, so the
|
||||
// half-built pair is still in the inputs.
|
||||
await expect(page.getByLabel('Node selector').getByText('zone=', { exact: true })).toHaveCount(0)
|
||||
})
|
||||
|
||||
// The cluster's vocabulary is a suggestion, never a constraint: an admin
|
||||
// labelling nodes for a rule they are about to write must still be able to
|
||||
// type a key no node reports yet.
|
||||
test('still accepts a label the cluster has never reported', async ({ page }) => {
|
||||
await mockScheduling(page)
|
||||
await page.goto('/app/scheduling')
|
||||
await page.getByRole('button', { name: 'Add Scheduling Rule' }).click()
|
||||
|
||||
await page.getByRole('combobox', { name: 'Selector key' }).fill('tenant')
|
||||
await page.getByRole('combobox', { name: 'Selector value' }).fill('acme')
|
||||
await page.getByRole('button', { name: 'Add selector' }).click()
|
||||
|
||||
await expect(page.getByLabel('Node selector').getByText('tenant=acme', { exact: true })).toBeVisible()
|
||||
})
|
||||
|
||||
test('edits all fields with a locked model and preserves values after a failed save', async ({ page }) => {
|
||||
@@ -113,42 +158,29 @@ test.describe('Scheduling page', () => {
|
||||
await expect(page.getByRole('combobox', { name: '' }).first()).toBeEnabled()
|
||||
})
|
||||
|
||||
test('shows node loading, empty, no-match, and retry states independently from rules', async ({ page }) => {
|
||||
let attempts = 0
|
||||
// The roster feeds suggestions and nothing else now, so failing to load it
|
||||
// must cost the admin nothing but the hints.
|
||||
test('leaves the selector fully usable when the node roster fails to load', async ({ page }) => {
|
||||
await page.route('**/api/nodes/scheduling', route => route.fulfill({ status: 200, contentType: 'application/json', body: JSON.stringify([rule]) }))
|
||||
await page.route('**/api/nodes', async route => {
|
||||
attempts++
|
||||
if (attempts === 1) {
|
||||
await new Promise(resolve => setTimeout(resolve, 250))
|
||||
await route.fulfill({ status: 500, body: 'failed' })
|
||||
} else {
|
||||
await route.fulfill({ status: 200, contentType: 'application/json', body: '[]' })
|
||||
}
|
||||
})
|
||||
await page.route('**/api/nodes', route => route.fulfill({ status: 500, body: 'failed' }))
|
||||
await page.goto('/app/scheduling')
|
||||
await expect(page.getByText('Loading node labels…')).toBeVisible()
|
||||
await expect(page.getByText('llama-3.3')).toBeVisible()
|
||||
await expect(page.getByText('Could not load node labels.')).toBeVisible()
|
||||
await page.getByRole('button', { name: 'Retry loading node labels' }).click()
|
||||
await expect(page.getByText('No nodes are available yet.')).toBeVisible()
|
||||
|
||||
await page.unroute('**/api/nodes')
|
||||
await page.route('**/api/nodes', route => route.fulfill({ status: 200, contentType: 'application/json', body: JSON.stringify(nodes) }))
|
||||
await page.reload()
|
||||
await page.getByRole('searchbox', { name: 'Search node labels' }).fill('not-a-real-label')
|
||||
await expect(page.getByText('No nodes match your search.')).toBeVisible()
|
||||
// The rules still render: the roster is not on their path.
|
||||
await expect(page.getByText('llama-3.3')).toBeVisible()
|
||||
|
||||
await page.getByRole('button', { name: 'Add Scheduling Rule' }).click()
|
||||
await page.getByRole('combobox', { name: 'Selector key' }).fill('gpu.vendor')
|
||||
await page.getByRole('combobox', { name: 'Selector value' }).fill('nvidia')
|
||||
await page.getByRole('button', { name: 'Add selector' }).click()
|
||||
|
||||
await expect(page.getByLabel('Node selector').getByText('gpu.vendor=nvidia', { exact: true })).toBeVisible()
|
||||
})
|
||||
|
||||
test('uses one node column and accessible rule actions on a narrow viewport', async ({ page }) => {
|
||||
test('keeps rule actions reachable on a narrow viewport', async ({ page }) => {
|
||||
await page.setViewportSize({ width: 390, height: 844 })
|
||||
await mockScheduling(page, { nodeList: nodes.slice(0, 2) })
|
||||
await page.goto('/app/scheduling')
|
||||
|
||||
const cards = page.locator('.scheduling-node-card')
|
||||
const first = await cards.nth(0).boundingBox()
|
||||
const second = await cards.nth(1).boundingBox()
|
||||
expect(second.y).toBeGreaterThan(first.y + first.height - 1)
|
||||
|
||||
const actions = page.locator('.scheduling-rule-actions')
|
||||
await expect(actions.getByRole('button', { name: 'Edit llama-3.3' })).toBeVisible()
|
||||
await expect(actions.getByRole('button', { name: 'Delete llama-3.3' })).toBeVisible()
|
||||
|
||||
@@ -1 +1 @@
|
||||
519
|
||||
514
|
||||
@@ -142,7 +142,8 @@
|
||||
"sha256": "SHA256",
|
||||
"backToAll": "Alle Modelle",
|
||||
"vramAt": "VRAM bei {{context}}",
|
||||
"headroom": "Spielraum"
|
||||
"headroom": "Spielraum",
|
||||
"headroomOn": "Spielraum auf {{node}}"
|
||||
},
|
||||
"empty": {
|
||||
"title": "Keine Modelle gefunden",
|
||||
@@ -216,6 +217,8 @@
|
||||
"browsing": "Durchsuchen",
|
||||
"pickHint": "Wähle ein Modell, um die Details zu sehen.",
|
||||
"heroWithRam": "{{ram}} Systemspeicher, {{count}} Modelle in der Galerie.",
|
||||
"heroWithCluster": "{{vram}} auf {{node}}, dem größten von {{nodes}} Knoten, {{count}} Modelle in der Galerie.",
|
||||
"heroWithNode": "{{vram}} auf {{node}}, {{count}} Modelle in der Galerie.",
|
||||
"byUseCase": "Oder mit einem Anwendungsfall starten",
|
||||
"pickText": "Chat, Reasoning, Embeddings",
|
||||
"pickVision": "Bilder und Dokumente lesen",
|
||||
|
||||
@@ -152,7 +152,8 @@
|
||||
"sha256": "SHA256",
|
||||
"backToAll": "All models",
|
||||
"vramAt": "VRAM at {{context}}",
|
||||
"headroom": "Headroom"
|
||||
"headroom": "Headroom",
|
||||
"headroomOn": "Headroom on {{node}}"
|
||||
},
|
||||
"empty": {
|
||||
"title": "No models found",
|
||||
@@ -232,6 +233,8 @@
|
||||
"browsing": "Browsing",
|
||||
"pickHint": "Select a model to see its detail.",
|
||||
"heroWithRam": "{{ram}} of system memory, {{count}} models in the gallery.",
|
||||
"heroWithCluster": "{{vram}} on {{node}}, the largest of {{nodes}} nodes, {{count}} models in the gallery.",
|
||||
"heroWithNode": "{{vram}} on {{node}}, {{count}} models in the gallery.",
|
||||
"byUseCase": "Or start with a use case",
|
||||
"pickText": "Chat, reasoning, embeddings",
|
||||
"pickVision": "Read images and documents",
|
||||
|
||||
@@ -142,7 +142,8 @@
|
||||
"sha256": "SHA256",
|
||||
"backToAll": "Todos los modelos",
|
||||
"vramAt": "VRAM a {{context}}",
|
||||
"headroom": "Margen"
|
||||
"headroom": "Margen",
|
||||
"headroomOn": "Margen en {{node}}"
|
||||
},
|
||||
"empty": {
|
||||
"title": "No se encontraron modelos",
|
||||
@@ -216,6 +217,8 @@
|
||||
"browsing": "Explorando",
|
||||
"pickHint": "Selecciona un modelo para ver su detalle.",
|
||||
"heroWithRam": "{{ram}} de memoria del sistema, {{count}} modelos en la galería.",
|
||||
"heroWithCluster": "{{vram}} en {{node}}, el mayor de {{nodes}} nodos, {{count}} modelos en la galería.",
|
||||
"heroWithNode": "{{vram}} en {{node}}, {{count}} modelos en la galería.",
|
||||
"byUseCase": "O empieza por un caso de uso",
|
||||
"pickText": "Chat, razonamiento, embeddings",
|
||||
"pickVision": "Leer imágenes y documentos",
|
||||
|
||||
@@ -149,7 +149,8 @@
|
||||
"sha256": "SHA256",
|
||||
"backToAll": "Semua model",
|
||||
"vramAt": "VRAM pada {{context}}",
|
||||
"headroom": "Sisa ruang"
|
||||
"headroom": "Sisa ruang",
|
||||
"headroomOn": "Sisa ruang di {{node}}"
|
||||
},
|
||||
"empty": {
|
||||
"title": "Model tidak ditemukan",
|
||||
@@ -229,6 +230,8 @@
|
||||
"browsing": "Menjelajah",
|
||||
"pickHint": "Pilih model untuk melihat detailnya.",
|
||||
"heroWithRam": "Memori sistem {{ram}}, {{count}} model di galeri.",
|
||||
"heroWithCluster": "{{vram}} di {{node}}, terbesar dari {{nodes}} node, {{count}} model di galeri.",
|
||||
"heroWithNode": "{{vram}} di {{node}}, {{count}} model di galeri.",
|
||||
"byUseCase": "Atau mulai dari kasus penggunaan",
|
||||
"pickText": "Obrolan, penalaran, embedding",
|
||||
"pickVision": "Membaca gambar dan dokumen",
|
||||
|
||||
@@ -142,7 +142,8 @@
|
||||
"sha256": "SHA256",
|
||||
"backToAll": "Tutti i modelli",
|
||||
"vramAt": "VRAM a {{context}}",
|
||||
"headroom": "Margine"
|
||||
"headroom": "Margine",
|
||||
"headroomOn": "Margine su {{node}}"
|
||||
},
|
||||
"empty": {
|
||||
"title": "Nessun modello trovato",
|
||||
@@ -216,6 +217,8 @@
|
||||
"browsing": "Esplorazione",
|
||||
"pickHint": "Seleziona un modello per vederne i dettagli.",
|
||||
"heroWithRam": "{{ram}} di memoria di sistema, {{count}} modelli nella galleria.",
|
||||
"heroWithCluster": "{{vram}} su {{node}}, il più grande di {{nodes}} nodi, {{count}} modelli nella galleria.",
|
||||
"heroWithNode": "{{vram}} su {{node}}, {{count}} modelli nella galleria.",
|
||||
"byUseCase": "Oppure parti da un caso d’uso",
|
||||
"pickText": "Chat, ragionamento, embedding",
|
||||
"pickVision": "Leggere immagini e documenti",
|
||||
|
||||
@@ -148,7 +148,8 @@
|
||||
"sha256": "SHA256",
|
||||
"backToAll": "모든 모델",
|
||||
"vramAt": "{{context}}에서의 VRAM",
|
||||
"headroom": "여유 공간"
|
||||
"headroom": "여유 공간",
|
||||
"headroomOn": "{{node}}의 여유 공간"
|
||||
},
|
||||
"empty": {
|
||||
"title": "모델을 찾을 수 없습니다",
|
||||
@@ -200,6 +201,8 @@
|
||||
"browsing": "둘러보기",
|
||||
"pickHint": "모델을 선택하면 상세 정보가 표시됩니다.",
|
||||
"heroWithRam": "시스템 메모리 {{ram}}, 갤러리에 모델 {{count}}개.",
|
||||
"heroWithCluster": "{{nodes}}개 노드 중 가장 큰 {{node}}에 {{vram}}, 갤러리에 모델 {{count}}개.",
|
||||
"heroWithNode": "{{node}}에 {{vram}}, 갤러리에 모델 {{count}}개.",
|
||||
"byUseCase": "또는 용도로 시작하기",
|
||||
"pickText": "채팅, 추론, 임베딩",
|
||||
"pickVision": "이미지와 문서 읽기",
|
||||
|
||||
@@ -152,7 +152,8 @@
|
||||
"sha256": "SHA256",
|
||||
"backToAll": "Todos os modelos",
|
||||
"vramAt": "VRAM em {{context}}",
|
||||
"headroom": "Margem de sobra"
|
||||
"headroom": "Margem de sobra",
|
||||
"headroomOn": "Folga em {{node}}"
|
||||
},
|
||||
"empty": {
|
||||
"title": "Nenhum modelo encontrado",
|
||||
@@ -232,6 +233,8 @@
|
||||
"browsing": "Explorando",
|
||||
"pickHint": "Selecione um modelo para ver seus detalhes.",
|
||||
"heroWithRam": "{{ram}} de memória do sistema, {{count}} modelos na galeria.",
|
||||
"heroWithCluster": "{{vram}} em {{node}}, o maior de {{nodes}} nós, {{count}} modelos na galeria.",
|
||||
"heroWithNode": "{{vram}} em {{node}}, {{count}} modelos na galeria.",
|
||||
"byUseCase": "Ou comece por um caso de uso",
|
||||
"pickText": "Chat, raciocínio, embeddings",
|
||||
"pickVision": "Leia imagens e documentos",
|
||||
|
||||
@@ -142,7 +142,8 @@
|
||||
"sha256": "SHA256",
|
||||
"backToAll": "全部模型",
|
||||
"vramAt": "{{context}} 时显存",
|
||||
"headroom": "剩余显存"
|
||||
"headroom": "剩余显存",
|
||||
"headroomOn": "{{node}} 上的余量"
|
||||
},
|
||||
"empty": {
|
||||
"title": "未找到模型",
|
||||
@@ -216,6 +217,8 @@
|
||||
"browsing": "浏览中",
|
||||
"pickHint": "选择一个模型以查看详情。",
|
||||
"heroWithRam": "{{ram}} 系统内存，图库中有 {{count}} 个模型。",
|
||||
"heroWithCluster": "{{node}} 上 {{vram}}，为 {{nodes}} 个节点中最大，图库中有 {{count}} 个模型。",
|
||||
"heroWithNode": "{{node}} 上 {{vram}}，图库中有 {{count}} 个模型。",
|
||||
"byUseCase": "或从用途开始",
|
||||
"pickText": "对话、推理、向量",
|
||||
"pickVision": "读取图像与文档",
|
||||
|
||||
+66
-110
@@ -2697,116 +2697,82 @@ select.input {
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.scheduling-node-reference {
|
||||
margin-bottom: var(--spacing-md);
|
||||
overflow: hidden;
|
||||
/* Key-value chip builder (node selectors, node labels) */
|
||||
.kvchips__chips {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 4px;
|
||||
margin-bottom: var(--spacing-xs);
|
||||
}
|
||||
|
||||
.scheduling-node-reference__toggle {
|
||||
.kvchips__chip {
|
||||
align-items: center;
|
||||
background: transparent;
|
||||
border: 0;
|
||||
background: var(--color-bg-tertiary);
|
||||
border: 1px solid var(--color-border-subtle);
|
||||
border-radius: var(--radius-sm);
|
||||
display: inline-flex;
|
||||
font-family: var(--font-mono);
|
||||
font-size: 0.75rem;
|
||||
gap: 4px;
|
||||
padding: 2px 8px;
|
||||
}
|
||||
|
||||
.kvchips__chip-remove {
|
||||
background: none;
|
||||
border: none;
|
||||
color: var(--color-text-muted);
|
||||
cursor: pointer;
|
||||
font-size: 0.625rem;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.kvchips__row {
|
||||
align-items: stretch;
|
||||
display: flex;
|
||||
gap: var(--spacing-xs);
|
||||
position: relative;
|
||||
}
|
||||
|
||||
.kvchips__add {
|
||||
min-height: 36px;
|
||||
}
|
||||
|
||||
/* Anchored to the input row so the list covers what follows the field rather
|
||||
than pushing the rest of the form down as the user types. */
|
||||
.kvchips__suggestions {
|
||||
background: var(--color-bg-secondary);
|
||||
border: 1px solid var(--color-border);
|
||||
border-radius: var(--radius-sm);
|
||||
box-shadow: var(--shadow-md);
|
||||
left: 0;
|
||||
list-style: none;
|
||||
margin: 4px 0 0;
|
||||
max-height: 220px;
|
||||
overflow-y: auto;
|
||||
padding: 4px;
|
||||
position: absolute;
|
||||
right: 0;
|
||||
top: 100%;
|
||||
z-index: 20;
|
||||
}
|
||||
|
||||
.kvchips__suggestion {
|
||||
background: none;
|
||||
border: none;
|
||||
border-radius: var(--radius-sm);
|
||||
color: var(--color-text-primary);
|
||||
cursor: pointer;
|
||||
display: flex;
|
||||
font: inherit;
|
||||
font-weight: var(--font-weight-semibold);
|
||||
justify-content: space-between;
|
||||
padding: var(--spacing-md);
|
||||
display: block;
|
||||
font-family: var(--font-mono);
|
||||
font-size: 0.8125rem;
|
||||
padding: var(--spacing-xs) var(--spacing-sm);
|
||||
text-align: left;
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.scheduling-node-reference__toggle:focus-visible {
|
||||
outline: 2px solid var(--color-primary);
|
||||
outline-offset: -2px;
|
||||
}
|
||||
|
||||
.scheduling-node-reference__content {
|
||||
border-top: 1px solid var(--color-border-subtle);
|
||||
padding: var(--spacing-md);
|
||||
}
|
||||
|
||||
.scheduling-node-reference__content > .text-note {
|
||||
margin: 0 0 var(--spacing-sm);
|
||||
}
|
||||
|
||||
.scheduling-node-toolbar {
|
||||
align-items: center;
|
||||
display: flex;
|
||||
gap: var(--spacing-md);
|
||||
margin-bottom: var(--spacing-md);
|
||||
}
|
||||
|
||||
.scheduling-node-toolbar .input {
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.scheduling-node-toolbar .text-meta {
|
||||
flex: none;
|
||||
}
|
||||
|
||||
.scheduling-node-grid {
|
||||
display: grid;
|
||||
gap: var(--spacing-sm);
|
||||
grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
|
||||
}
|
||||
|
||||
.scheduling-node-card {
|
||||
.kvchips__suggestion:hover,
|
||||
.kvchips__suggestion--active {
|
||||
background: var(--color-bg-tertiary);
|
||||
border: 1px solid var(--color-border-subtle);
|
||||
border-radius: var(--radius-md);
|
||||
min-width: 0;
|
||||
padding: var(--spacing-sm);
|
||||
}
|
||||
|
||||
.scheduling-node-card__header {
|
||||
align-items: center;
|
||||
display: flex;
|
||||
gap: var(--spacing-sm);
|
||||
justify-content: space-between;
|
||||
margin-bottom: var(--spacing-xs);
|
||||
}
|
||||
|
||||
.scheduling-node-status {
|
||||
color: var(--color-text-muted);
|
||||
font-size: 0.75rem;
|
||||
text-transform: capitalize;
|
||||
}
|
||||
|
||||
.scheduling-node-status--online,
|
||||
.scheduling-node-status--ready,
|
||||
.scheduling-node-status--healthy {
|
||||
color: var(--color-success);
|
||||
}
|
||||
|
||||
.scheduling-node-labels {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
.scheduling-node-label {
|
||||
border: 1px solid var(--color-border-subtle);
|
||||
border-radius: var(--radius-sm);
|
||||
font-family: var(--font-mono);
|
||||
font-size: 0.75rem;
|
||||
overflow-wrap: anywhere;
|
||||
padding: 2px 6px;
|
||||
}
|
||||
|
||||
.scheduling-node-message {
|
||||
align-items: center;
|
||||
color: var(--color-text-muted);
|
||||
display: flex;
|
||||
gap: var(--spacing-sm);
|
||||
justify-content: center;
|
||||
margin: var(--spacing-md) 0;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.scheduling-show-more {
|
||||
margin-top: var(--spacing-md);
|
||||
}
|
||||
|
||||
.scheduling-rule-actions {
|
||||
@@ -2816,16 +2782,6 @@ select.input {
|
||||
}
|
||||
|
||||
@media (max-width: 640px) {
|
||||
.scheduling-node-grid {
|
||||
grid-template-columns: minmax(0, 1fr);
|
||||
}
|
||||
|
||||
.scheduling-node-toolbar {
|
||||
align-items: stretch;
|
||||
flex-direction: column;
|
||||
gap: var(--spacing-xs);
|
||||
}
|
||||
|
||||
.scheduling-rule-actions {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import { useState } from 'react'
|
||||
import { useState, useRef, useEffect } from 'react'
|
||||
|
||||
import { suggestKeys, suggestValues } from '../../utils/nodeLabelSuggestions'
|
||||
|
||||
/**
|
||||
* Controlled chip-builder for { key: value } maps. Replaces the prior
|
||||
@@ -11,6 +13,14 @@ import { useState } from 'react'
|
||||
* labels editor). The component just renders chips and a key/value input
|
||||
* row.
|
||||
*
|
||||
* With `suggestions` it also completes what the user types against the
|
||||
* vocabulary the cluster actually uses. That is where label discovery lives on
|
||||
* the scheduling page: labels only matter while a selector is being written,
|
||||
* so browsing them belongs in the field rather than in a card standing open
|
||||
* above the rules. The suggestions are never a constraint - a key no node
|
||||
* reports yet still commits as typed, which is the workflow of writing a rule
|
||||
* before labelling the nodes for it.
|
||||
*
|
||||
* Props:
|
||||
* pairs - current map of key -> value
|
||||
* onAdd(k,v) - called when the user adds a pair (parent handles dedup
|
||||
@@ -18,45 +28,119 @@ import { useState } from 'react'
|
||||
* onRemove(k) - called when a chip's × is clicked
|
||||
* placeholderKey, placeholderValue - input hints
|
||||
* ariaLabel - accessible name for the section
|
||||
* ariaLabelKey, ariaLabelValue - accessible names for the two inputs
|
||||
* addLabel - accessible name for the commit button
|
||||
* suggestions - label index from utils/nodeLabelSuggestions; omit for none
|
||||
*/
|
||||
export default function KeyValueChips({ pairs, onAdd, onRemove, placeholderKey = 'key', placeholderValue = 'value', ariaLabel }) {
|
||||
export default function KeyValueChips({
|
||||
pairs, onAdd, onRemove,
|
||||
placeholderKey = 'key', placeholderValue = 'value',
|
||||
ariaLabel, ariaLabelKey = 'Key', ariaLabelValue = 'Value',
|
||||
addLabel = 'Add', suggestions,
|
||||
}) {
|
||||
const [k, setK] = useState('')
|
||||
const [v, setV] = useState('')
|
||||
// Which input owns the open list, and which of its options is armed for
|
||||
// Enter. -1 means the user is typing free text and Enter should commit the
|
||||
// pair rather than pick anything.
|
||||
const [openField, setOpenField] = useState(null)
|
||||
const [active, setActive] = useState(-1)
|
||||
const rowRef = useRef(null)
|
||||
|
||||
const entries = pairs ? Object.entries(pairs) : []
|
||||
|
||||
const options = !suggestions || !openField
|
||||
? []
|
||||
: openField === 'key'
|
||||
? suggestKeys(suggestions, k, entries.map(([key]) => key))
|
||||
: suggestValues(suggestions, k.trim(), v)
|
||||
|
||||
// A click anywhere else is a dismissal. Without this the list survives the
|
||||
// user moving on to the rest of the form and covers it.
|
||||
useEffect(() => {
|
||||
if (!openField) return undefined
|
||||
const onDocumentPointerDown = (event) => {
|
||||
if (!rowRef.current?.contains(event.target)) setOpenField(null)
|
||||
}
|
||||
document.addEventListener('mousedown', onDocumentPointerDown)
|
||||
return () => document.removeEventListener('mousedown', onDocumentPointerDown)
|
||||
}, [openField])
|
||||
|
||||
const add = () => {
|
||||
const key = k.trim()
|
||||
if (!key) return
|
||||
onAdd(key, v.trim())
|
||||
setK(''); setV('')
|
||||
}
|
||||
const onKeyDown = (e) => {
|
||||
if (e.key === 'Enter') { e.preventDefault(); add() }
|
||||
setOpenField(null); setActive(-1)
|
||||
}
|
||||
|
||||
const entries = pairs ? Object.entries(pairs) : []
|
||||
const pick = (field, option) => {
|
||||
if (field === 'key') setK(option)
|
||||
else setV(option)
|
||||
setOpenField(null)
|
||||
setActive(-1)
|
||||
}
|
||||
|
||||
const onKeyDown = (field) => (e) => {
|
||||
const open = openField === field && options.length > 0
|
||||
if (e.key === 'ArrowDown' && open) {
|
||||
e.preventDefault()
|
||||
setActive(current => (current + 1) % options.length)
|
||||
return
|
||||
}
|
||||
if (e.key === 'ArrowUp' && open) {
|
||||
e.preventDefault()
|
||||
setActive(current => (current <= 0 ? options.length - 1 : current - 1))
|
||||
return
|
||||
}
|
||||
if (e.key === 'Escape' && openField) {
|
||||
e.preventDefault()
|
||||
setOpenField(null)
|
||||
setActive(-1)
|
||||
return
|
||||
}
|
||||
if (e.key === 'Enter') {
|
||||
e.preventDefault()
|
||||
// Enter completes the suggestion the user armed, and commits the pair
|
||||
// otherwise. Committing a half-typed key because a list happened to be
|
||||
// open is the error this ordering avoids.
|
||||
if (open && active >= 0) pick(field, options[active])
|
||||
else add()
|
||||
}
|
||||
}
|
||||
|
||||
const listId = 'kvchips-suggestions'
|
||||
const inputProps = (field, value, setValue, placeholder, label) => ({
|
||||
className: 'input flex-1',
|
||||
type: 'text',
|
||||
role: suggestions ? 'combobox' : undefined,
|
||||
'aria-expanded': suggestions ? openField === field : undefined,
|
||||
'aria-controls': suggestions && openField === field ? listId : undefined,
|
||||
'aria-autocomplete': suggestions ? 'list' : undefined,
|
||||
'aria-label': label,
|
||||
placeholder,
|
||||
value,
|
||||
onChange: (e) => {
|
||||
setValue(e.target.value)
|
||||
if (suggestions) { setOpenField(field); setActive(-1) }
|
||||
},
|
||||
onFocus: () => { if (suggestions) { setOpenField(field); setActive(-1) } },
|
||||
onKeyDown: onKeyDown(field),
|
||||
})
|
||||
|
||||
return (
|
||||
<div aria-label={ariaLabel}>
|
||||
{entries.length > 0 && (
|
||||
<div style={{ display: 'flex', flexWrap: 'wrap', gap: 4, marginBottom: 'var(--spacing-xs)' }}>
|
||||
<div className="kvchips__chips">
|
||||
{entries.map(([key, val]) => (
|
||||
<span key={key} style={{
|
||||
display: 'inline-flex', alignItems: 'center', gap: 4,
|
||||
fontSize: '0.75rem', padding: '2px 8px',
|
||||
borderRadius: 'var(--radius-sm)',
|
||||
background: 'var(--color-bg-tertiary)',
|
||||
border: '1px solid var(--color-border-subtle)',
|
||||
fontFamily: 'var(--font-mono)',
|
||||
}}>
|
||||
<span key={key} className="kvchips__chip">
|
||||
{key}={val}
|
||||
<button
|
||||
type="button"
|
||||
onClick={(e) => { e.stopPropagation(); onRemove(key) }}
|
||||
aria-label={`Remove ${key}`}
|
||||
title="Remove"
|
||||
style={{
|
||||
background: 'none', border: 'none', cursor: 'pointer',
|
||||
color: 'var(--color-text-muted)', fontSize: '0.625rem', padding: 0,
|
||||
}}
|
||||
className="kvchips__chip-remove"
|
||||
>
|
||||
<i className="fas fa-times" />
|
||||
</button>
|
||||
@@ -64,32 +148,35 @@ export default function KeyValueChips({ pairs, onAdd, onRemove, placeholderKey =
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
<div style={{ display: 'flex', gap: 'var(--spacing-xs)', alignItems: 'stretch' }}>
|
||||
<input
|
||||
className="input flex-1"
|
||||
type="text"
|
||||
placeholder={placeholderKey}
|
||||
value={k}
|
||||
onChange={e => setK(e.target.value)}
|
||||
onKeyDown={onKeyDown}
|
||||
/>
|
||||
<input
|
||||
className="input flex-1"
|
||||
type="text"
|
||||
placeholder={placeholderValue}
|
||||
value={v}
|
||||
onChange={e => setV(e.target.value)}
|
||||
onKeyDown={onKeyDown}
|
||||
/>
|
||||
<div className="kvchips__row" ref={rowRef}>
|
||||
<input {...inputProps('key', k, setK, placeholderKey, ariaLabelKey)} />
|
||||
<input {...inputProps('value', v, setV, placeholderValue, ariaLabelValue)} />
|
||||
<button
|
||||
type="button"
|
||||
className="btn btn-secondary btn-sm"
|
||||
className="btn btn-secondary btn-sm kvchips__add"
|
||||
onClick={add}
|
||||
disabled={!k.trim()}
|
||||
style={{ minHeight: 36 }}
|
||||
aria-label={addLabel}
|
||||
>
|
||||
<i className="fas fa-plus" /> Add
|
||||
</button>
|
||||
{options.length > 0 && (
|
||||
<ul className="kvchips__suggestions" id={listId} role="listbox" data-testid="label-suggestions">
|
||||
{options.map((option, index) => (
|
||||
<li key={option} role="option" aria-selected={index === active}>
|
||||
<button
|
||||
type="button"
|
||||
className={`kvchips__suggestion${index === active ? ' kvchips__suggestion--active' : ''}`}
|
||||
// mousedown, not click: the input's blur would otherwise
|
||||
// close the list before the click ever lands.
|
||||
onMouseDown={(e) => { e.preventDefault(); pick(openField, option) }}
|
||||
>
|
||||
{option}
|
||||
</button>
|
||||
</li>
|
||||
))}
|
||||
</ul>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
|
||||
+11
-2
@@ -1,6 +1,7 @@
|
||||
import { useState, useEffect } from 'react'
|
||||
import { modelsApi } from '../utils/api'
|
||||
import { useResources } from './useResources'
|
||||
import { modelBudget } from '../utils/modelBudget'
|
||||
|
||||
// Data-driven "recommended for your hardware" model picks. The gallery exposes
|
||||
// no popularity/download signal and the list response carries no size, so we:
|
||||
@@ -21,13 +22,21 @@ const DEFAULT_CTX = 4096
|
||||
export const isNvfp4Name = (name) => /nvfp4/i.test(name || '')
|
||||
|
||||
export function hasNvidiaGpu(resources) {
|
||||
// A distributed controller has no GPUs of its own, so the question is
|
||||
// whether any worker does. The registry reports the cluster's best node,
|
||||
// and it is that node these picks have to run on.
|
||||
if (resources?.cluster?.enabled) return !!resources.cluster.is_gpu
|
||||
return Array.isArray(resources?.gpus) &&
|
||||
resources.gpus.some(g => (g?.vendor || '').toLowerCase() === 'nvidia')
|
||||
}
|
||||
|
||||
export function recommendTier(resources) {
|
||||
const isGpu = resources?.type === 'gpu'
|
||||
const vram = resources?.aggregate?.total_memory || 0
|
||||
// Same reading the models page sizes against: the cluster's largest node in
|
||||
// distributed mode, the local host otherwise. Ranked against the controller,
|
||||
// a fleet of A100s was recommended the models a GPU-less pod could run.
|
||||
const budget = modelBudget(resources)
|
||||
const isGpu = budget.scope === 'cluster' ? budget.hasGpu : resources?.type === 'gpu'
|
||||
const vram = budget.totalMemory
|
||||
if (!isGpu || vram <= 0) return { id: 'cpu', vram: 0 }
|
||||
if (vram < 8 * GB) return { id: 'gpu-small', vram }
|
||||
if (vram < 24 * GB) return { id: 'gpu-mid', vram }
|
||||
|
||||
@@ -7,6 +7,7 @@ import { safeHref } from '../utils/url'
|
||||
import { useDebouncedCallback } from '../hooks/useDebounce'
|
||||
import { useOperations } from '../hooks/useOperations'
|
||||
import { useResources } from '../hooks/useResources'
|
||||
import { modelBudget } from '../utils/modelBudget'
|
||||
import SearchableSelect from '../components/SearchableSelect'
|
||||
import PageHeader from '../components/PageHeader'
|
||||
import GalleryLoader from '../components/GalleryLoader'
|
||||
@@ -208,12 +209,13 @@ export default function Models() {
|
||||
const [useCaseOpen, setUseCaseOpen] = useState(false)
|
||||
// Rail groups the user has folded away.
|
||||
const [collapsedGroups, setCollapsedGroups] = useState(() => new Set())
|
||||
// Total GPU memory for "fits" check
|
||||
const totalGpuMemory = resources?.aggregate?.total_memory || 0
|
||||
// gpu_count is 0 and gpus is null on a CPU-only host, where total_memory is
|
||||
// system RAM. The fits check has always used it either way; only the copy
|
||||
// has to stop calling it VRAM.
|
||||
const hasGpu = (resources?.aggregate?.gpu_count || 0) > 0 || (resources?.gpus?.length || 0) > 0
|
||||
// What every "will it fit" verdict on this page is measured against. In
|
||||
// distributed mode that is the cluster's largest node rather than the
|
||||
// controller serving the page, which is usually a GPU-less pod (see
|
||||
// modelBudget).
|
||||
const budget = modelBudget(resources)
|
||||
const totalGpuMemory = budget.totalMemory
|
||||
const hasGpu = budget.hasGpu
|
||||
|
||||
const fetchModels = useCallback(async (params = {}) => {
|
||||
try {
|
||||
@@ -865,6 +867,7 @@ export default function Models() {
|
||||
onPickContext={setContextSize}
|
||||
totalGpuMemory={totalGpuMemory}
|
||||
fitsGpu={fitsGpu}
|
||||
budgetNode={budget.scope === 'cluster' ? budget.nodeName : ''}
|
||||
installing={isInstalling(selectedName)}
|
||||
progress={getOperationProgress(selectedName)}
|
||||
onInstall={handleInstall}
|
||||
@@ -894,9 +897,16 @@ export default function Models() {
|
||||
the data did not support. */}
|
||||
{totalGpuMemory <= 0
|
||||
? t('shelves.heroNoGpu', { count: stats.total })
|
||||
: hasGpu
|
||||
? t('shelves.heroWithGpu', { vram: formatBytes(totalGpuMemory), count: stats.total })
|
||||
: t('shelves.heroWithRam', { ram: formatBytes(totalGpuMemory), count: stats.total })}
|
||||
: budget.scope === 'cluster'
|
||||
// Naming the node is the point: a cluster figure with
|
||||
// no owner reads as this machine's, which is the very
|
||||
// confusion the cluster reading exists to end.
|
||||
? t(budget.nodeCount > 1 ? 'shelves.heroWithCluster' : 'shelves.heroWithNode', {
|
||||
vram: formatBytes(totalGpuMemory), node: budget.nodeName, nodes: budget.nodeCount, count: stats.total,
|
||||
})
|
||||
: hasGpu
|
||||
? t('shelves.heroWithGpu', { vram: formatBytes(totalGpuMemory), count: stats.total })
|
||||
: t('shelves.heroWithRam', { ram: formatBytes(totalGpuMemory), count: stats.total })}
|
||||
</h2>
|
||||
<p className="zero-pane__text">{t('shelves.heroHint')}</p>
|
||||
</div>
|
||||
@@ -1413,7 +1423,7 @@ function VramByContext({ estimate, contextSize, onPickContext, totalGpuMemory, t
|
||||
// and hands the rest to ModelDetail, which already knows how to render an
|
||||
// entry's fields and is shared with the per-variant panel.
|
||||
function DiscoverDetail({
|
||||
model, estimate, contextSize, onPickContext, totalGpuMemory, fitsGpu,
|
||||
model, estimate, contextSize, onPickContext, totalGpuMemory, fitsGpu, budgetNode,
|
||||
installing, progress, onInstall, installedProfile, onOpen, onManage, onBack,
|
||||
expandedFiles, setExpandedFiles, variantData, variantDetails, onLoadVariantDetail, t,
|
||||
}) {
|
||||
@@ -1472,7 +1482,11 @@ function DiscoverDetail({
|
||||
{ label: t('detail.size'), value: sizeDisplay && sizeDisplay !== '0 B' ? sizeDisplay : '—' },
|
||||
{ label: t('detail.vramAt', { context: contextLabel }), value: vramBytes ? formatBytes(vramBytes) : '—' },
|
||||
{
|
||||
label: t('detail.headroom'),
|
||||
// Headroom is headroom somewhere. On a distributed controller that
|
||||
// somewhere is a worker, and an unqualified figure reads as this
|
||||
// machine's, which is the confusion the cluster reading exists to
|
||||
// end.
|
||||
label: budgetNode ? t('detail.headroomOn', { node: budgetNode }) : t('detail.headroom'),
|
||||
value: headroom === null ? '—' : (headroom < 0 ? '−' : '') + formatBytes(Math.abs(headroom)),
|
||||
tone: headroom === null ? undefined : headroom < 0 ? 'bad' : 'ok',
|
||||
},
|
||||
|
||||
@@ -7,6 +7,7 @@ import ConfirmDialog from '../components/ConfirmDialog'
|
||||
import ResponsiveTable from '../components/ResponsiveTable'
|
||||
import SearchableModelSelect from '../components/SearchableModelSelect'
|
||||
import KeyValueChips from '../components/nodes/KeyValueChips'
|
||||
import { labelIndex } from '../utils/nodeLabelSuggestions'
|
||||
|
||||
// Numeric input with quick-pick preset chips. Picked over a slider because
|
||||
// replica counts are exact specs (operator math), not fuzzy estimates. The
|
||||
@@ -65,7 +66,7 @@ function configMode(config) {
|
||||
return 'placement'
|
||||
}
|
||||
|
||||
function SchedulingForm({ initialConfig, onSave, onCancel }) {
|
||||
function SchedulingForm({ initialConfig, onSave, onCancel, labels }) {
|
||||
const [mode, setMode] = useState(() => configMode(initialConfig))
|
||||
const [modelName, setModelName] = useState(initialConfig?.model_name || '')
|
||||
// Selector is now a chip-builder map instead of a comma-separated string.
|
||||
@@ -174,6 +175,10 @@ function SchedulingForm({ initialConfig, onSave, onCancel }) {
|
||||
placeholderKey="key (e.g. gpu.vendor)"
|
||||
placeholderValue="value (e.g. nvidia)"
|
||||
ariaLabel="Node selector"
|
||||
ariaLabelKey="Selector key"
|
||||
ariaLabelValue="Selector value"
|
||||
addLabel="Add selector"
|
||||
suggestions={labels}
|
||||
/>
|
||||
<span className="text-meta d-block mt-xs">
|
||||
{mode === 'placement'
|
||||
@@ -285,139 +290,25 @@ function SchedulingForm({ initialConfig, onSave, onCancel }) {
|
||||
)
|
||||
}
|
||||
|
||||
const INITIAL_NODE_LIMIT = 5
|
||||
const NODE_LIMIT_STEP = 20
|
||||
|
||||
function fuzzyIncludes(text, term) {
|
||||
if (text.includes(term)) return true
|
||||
let termIndex = 0
|
||||
for (const character of text) {
|
||||
if (character === term[termIndex]) termIndex++
|
||||
if (termIndex === term.length) return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
function matchesNode(node, query) {
|
||||
const terms = query.toLocaleLowerCase().trim().split(/\s+/).filter(Boolean)
|
||||
if (!terms.length) return true
|
||||
const labels = Object.entries(node.labels || {})
|
||||
const haystack = [
|
||||
node.name,
|
||||
node.id,
|
||||
...labels.flatMap(([key, value]) => [key, String(value), `${key}=${value}`]),
|
||||
].filter(Boolean).join(' ').toLocaleLowerCase()
|
||||
return terms.every(term => fuzzyIncludes(haystack, term))
|
||||
}
|
||||
|
||||
function NodeLabelReference() {
|
||||
const [expanded, setExpanded] = useState(true)
|
||||
const [nodes, setNodes] = useState([])
|
||||
const [query, setQuery] = useState('')
|
||||
const [visibleLimit, setVisibleLimit] = useState(INITIAL_NODE_LIMIT)
|
||||
const [loading, setLoading] = useState(true)
|
||||
const [error, setError] = useState(false)
|
||||
|
||||
const fetchNodes = useCallback(async () => {
|
||||
setLoading(true)
|
||||
setError(false)
|
||||
try {
|
||||
const data = await nodesApi.list()
|
||||
setNodes(Array.isArray(data) ? data : [])
|
||||
} catch {
|
||||
setError(true)
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}, [])
|
||||
|
||||
useEffect(() => { fetchNodes() }, [fetchNodes])
|
||||
|
||||
const filtered = nodes.filter(node => matchesNode(node, query))
|
||||
const visible = filtered.slice(0, visibleLimit)
|
||||
const updateQuery = event => {
|
||||
setQuery(event.target.value)
|
||||
setVisibleLimit(INITIAL_NODE_LIMIT)
|
||||
}
|
||||
|
||||
return (
|
||||
<section className="card scheduling-node-reference" data-testid="node-label-reference">
|
||||
<button
|
||||
type="button"
|
||||
className="scheduling-node-reference__toggle"
|
||||
aria-expanded={expanded}
|
||||
aria-controls="scheduling-node-label-content"
|
||||
onClick={() => setExpanded(value => !value)}
|
||||
>
|
||||
<span><i className="fas fa-tags icon-before" aria-hidden="true" />Node labels</span>
|
||||
<i className={`fas fa-chevron-${expanded ? 'up' : 'down'}`} aria-hidden="true" />
|
||||
</button>
|
||||
{expanded && (
|
||||
<div id="scheduling-node-label-content" className="scheduling-node-reference__content">
|
||||
<p className="text-note">Browse labels available for node selectors without leaving this page.</p>
|
||||
{loading ? (
|
||||
<p className="scheduling-node-message" role="status">Loading node labels…</p>
|
||||
) : error ? (
|
||||
<div className="scheduling-node-message" role="alert">
|
||||
<span>Could not load node labels.</span>
|
||||
<button type="button" className="btn btn-secondary btn-sm" aria-label="Retry loading node labels" onClick={fetchNodes}>Retry</button>
|
||||
</div>
|
||||
) : nodes.length === 0 ? (
|
||||
<p className="scheduling-node-message">No nodes are available yet.</p>
|
||||
) : (
|
||||
<>
|
||||
<div className="scheduling-node-toolbar">
|
||||
<input
|
||||
type="search"
|
||||
className="input"
|
||||
aria-label="Search node labels"
|
||||
placeholder="Search node, label, or key=value…"
|
||||
value={query}
|
||||
onChange={updateQuery}
|
||||
/>
|
||||
<span className="text-meta" aria-live="polite">{Math.min(visibleLimit, filtered.length)} of {filtered.length} nodes</span>
|
||||
</div>
|
||||
{filtered.length === 0 ? (
|
||||
<p className="scheduling-node-message">No nodes match your search.</p>
|
||||
) : (
|
||||
<div className="scheduling-node-grid">
|
||||
{visible.map(node => {
|
||||
const labels = Object.entries(node.labels || {})
|
||||
return (
|
||||
<article className="scheduling-node-card" key={node.id || node.name}>
|
||||
<div className="scheduling-node-card__header">
|
||||
<strong>{node.name || node.id}</strong>
|
||||
<span className={`scheduling-node-status scheduling-node-status--${String(node.status || 'unknown').toLowerCase()}`}>{node.status || 'unknown'}</span>
|
||||
</div>
|
||||
{labels.length ? (
|
||||
<div className="scheduling-node-labels">
|
||||
{labels.map(([key, value]) => <span className="scheduling-node-label" key={key}>{key}={String(value)}</span>)}
|
||||
</div>
|
||||
) : <span className="text-note">No labels</span>}
|
||||
</article>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
)}
|
||||
{visibleLimit < filtered.length && (
|
||||
<button type="button" className="btn btn-secondary btn-sm scheduling-show-more" aria-label="Show 20 more nodes" onClick={() => setVisibleLimit(limit => limit + NODE_LIMIT_STEP)}>
|
||||
Show 20 more
|
||||
</button>
|
||||
)}
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
)
|
||||
}
|
||||
|
||||
export default function Scheduling() {
|
||||
const { addToast } = useOutletContext()
|
||||
const { t } = useTranslation('admin')
|
||||
const [schedulingConfigs, setSchedulingConfigs] = useState([])
|
||||
const [formState, setFormState] = useState(null)
|
||||
const [confirmDelete, setConfirmDelete] = useState(null)
|
||||
// The label vocabulary the selector field completes against. A roster that
|
||||
// will not load costs the admin the hints and nothing else, so the failure
|
||||
// is swallowed rather than surfaced: the field still commits whatever is
|
||||
// typed into it.
|
||||
const [labels, setLabels] = useState(() => labelIndex([]))
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false
|
||||
nodesApi.list()
|
||||
.then(data => { if (!cancelled) setLabels(labelIndex(Array.isArray(data) ? data : [])) })
|
||||
.catch(() => {})
|
||||
return () => { cancelled = true }
|
||||
}, [])
|
||||
|
||||
const fetchScheduling = useCallback(async () => {
|
||||
try {
|
||||
@@ -453,7 +344,6 @@ export default function Scheduling() {
|
||||
supporting={t('scheduling.subtitle')}
|
||||
/>
|
||||
<div>
|
||||
<NodeLabelReference />
|
||||
<button className="btn btn-primary btn-sm mb-md"
|
||||
onClick={() => setFormState(current => current?.kind === 'add' ? null : { kind: 'add' })}>
|
||||
<i className="fas fa-plus icon-before" />
|
||||
@@ -465,6 +355,7 @@ export default function Scheduling() {
|
||||
initialConfig={formState.kind === 'edit' ? formState.config : undefined}
|
||||
onSave={handleSave}
|
||||
onCancel={() => setFormState(null)}
|
||||
labels={labels}
|
||||
/>
|
||||
)}
|
||||
{schedulingConfigs.length === 0 && !formState ? (
|
||||
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
// modelBudget answers the one question every "will this model run here" verdict
|
||||
// on the models page is built from: how much memory a model may occupy, and
|
||||
// whose memory it is.
|
||||
//
|
||||
// In distributed mode that is NOT the host serving this page. The controller is
|
||||
// usually a GPU-less pod while every model runs on a worker, so sizing against
|
||||
// its own aggregate told admins that a cluster of A100s could only run the
|
||||
// smallest CPU build. The server reports the cluster's best single node in an
|
||||
// additional `cluster` block; the local aggregate stays untouched for the
|
||||
// resource monitor, which is genuinely about this host.
|
||||
//
|
||||
// The best single node, not the fleet total: a model loads into one node, so a
|
||||
// summed fleet of four 16GB cards would promise a 40GB model a home it does not
|
||||
// have.
|
||||
//
|
||||
// Every missing or unusable field falls back to the local reading, so a
|
||||
// controller that cannot reach its registry keeps behaving exactly as a
|
||||
// single-node install does.
|
||||
export function modelBudget(resources) {
|
||||
const cluster = resources?.cluster
|
||||
if (cluster?.enabled && cluster.total_memory > 0) {
|
||||
return {
|
||||
totalMemory: cluster.total_memory,
|
||||
hasGpu: !!cluster.is_gpu,
|
||||
nodeName: cluster.node_name || '',
|
||||
nodeCount: cluster.node_count || 0,
|
||||
scope: 'cluster',
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
totalMemory: resources?.aggregate?.total_memory || 0,
|
||||
// gpu_count is 0 and gpus is null on a CPU-only host, where total_memory is
|
||||
// system RAM. The fits check has always used it either way; only the copy
|
||||
// has to stop calling it VRAM.
|
||||
hasGpu: (resources?.aggregate?.gpu_count || 0) > 0 || (resources?.gpus?.length || 0) > 0,
|
||||
nodeName: '',
|
||||
nodeCount: 0,
|
||||
scope: 'local',
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import test from 'node:test'
|
||||
|
||||
import { modelBudget } from './modelBudget.js'
|
||||
|
||||
const GB = 1024 * 1024 * 1024
|
||||
|
||||
test('reports nothing to size against before the first reading arrives', () => {
|
||||
assert.deepEqual(modelBudget(null), {
|
||||
totalMemory: 0, hasGpu: false, nodeName: '', nodeCount: 0, scope: 'local',
|
||||
})
|
||||
})
|
||||
|
||||
test('reports the local aggregate on a single-node host', () => {
|
||||
assert.deepEqual(
|
||||
modelBudget({ aggregate: { total_memory: 12 * GB, gpu_count: 1 }, gpus: [{ index: 0 }] }),
|
||||
{ totalMemory: 12 * GB, hasGpu: true, nodeName: '', nodeCount: 0, scope: 'local' },
|
||||
)
|
||||
})
|
||||
|
||||
test('treats a CPU-only host as having no GPU', () => {
|
||||
const budget = modelBudget({ aggregate: { total_memory: 8 * GB, gpu_count: 0 }, gpus: [] })
|
||||
assert.equal(budget.hasGpu, false)
|
||||
assert.equal(budget.totalMemory, 8 * GB)
|
||||
})
|
||||
|
||||
// The defect: the controller's own 8GB must not decide what a fleet of A100s
|
||||
// can run.
|
||||
test('prefers the cluster reading over the controller it is served from', () => {
|
||||
assert.deepEqual(
|
||||
modelBudget({
|
||||
aggregate: { total_memory: 8 * GB, gpu_count: 0 },
|
||||
gpus: [],
|
||||
cluster: { enabled: true, node_name: 'dgx-01', total_memory: 80 * GB, is_gpu: true, node_count: 4 },
|
||||
}),
|
||||
{ totalMemory: 80 * GB, hasGpu: true, nodeName: 'dgx-01', nodeCount: 4, scope: 'cluster' },
|
||||
)
|
||||
})
|
||||
|
||||
test('falls back to the local reading when the cluster reports no usable memory', () => {
|
||||
const budget = modelBudget({
|
||||
aggregate: { total_memory: 8 * GB, gpu_count: 0 },
|
||||
cluster: { enabled: true, node_name: 'dgx-01', total_memory: 0, is_gpu: false, node_count: 0 },
|
||||
})
|
||||
assert.equal(budget.scope, 'local')
|
||||
assert.equal(budget.totalMemory, 8 * GB)
|
||||
})
|
||||
|
||||
test('ignores a cluster block that says distributed mode is off', () => {
|
||||
const budget = modelBudget({
|
||||
aggregate: { total_memory: 8 * GB },
|
||||
cluster: { enabled: false, total_memory: 80 * GB },
|
||||
})
|
||||
assert.equal(budget.scope, 'local')
|
||||
assert.equal(budget.totalMemory, 8 * GB)
|
||||
})
|
||||
@@ -0,0 +1,52 @@
|
||||
// The scheduling page used to browse node labels in a card that stood open
|
||||
// above the rules whether or not anyone was writing one. Labels are only ever
|
||||
// needed while filling a rule's node selector, so discovery moved into that
|
||||
// field: these helpers turn the node roster into what the field offers as the
|
||||
// user types.
|
||||
//
|
||||
// The roster is already fetched for the page, so this costs no request.
|
||||
|
||||
// labelIndex reduces the node roster to the label vocabulary the cluster
|
||||
// actually uses: every distinct key, and the values each key takes.
|
||||
//
|
||||
// Nodes carrying no labels are not an error, they simply contribute nothing.
|
||||
export function labelIndex(nodes) {
|
||||
const values = {}
|
||||
for (const node of Array.isArray(nodes) ? nodes : []) {
|
||||
for (const [key, value] of Object.entries(node?.labels || {})) {
|
||||
const seen = values[key] || (values[key] = [])
|
||||
const text = String(value)
|
||||
if (!seen.includes(text)) seen.push(text)
|
||||
}
|
||||
}
|
||||
for (const key of Object.keys(values)) values[key].sort()
|
||||
return { keys: Object.keys(values).sort(), values }
|
||||
}
|
||||
|
||||
// rank orders matches so what the user is most likely typing comes first: a
|
||||
// prefix match beats a match buried in the middle of the string.
|
||||
function rank(candidates, query) {
|
||||
const needle = query.trim().toLowerCase()
|
||||
if (!needle) return candidates
|
||||
return candidates
|
||||
.filter(candidate => candidate.toLowerCase().includes(needle))
|
||||
.sort((a, b) => {
|
||||
const ap = a.toLowerCase().startsWith(needle)
|
||||
const bp = b.toLowerCase().startsWith(needle)
|
||||
if (ap !== bp) return ap ? -1 : 1
|
||||
return a.localeCompare(b)
|
||||
})
|
||||
}
|
||||
|
||||
// suggestKeys offers the label keys matching what has been typed, minus the
|
||||
// ones this selector already carries: re-adding a key would silently overwrite
|
||||
// the pair the user just built.
|
||||
export function suggestKeys(index, query, exclude = []) {
|
||||
return rank(index.keys.filter(key => !exclude.includes(key)), query)
|
||||
}
|
||||
|
||||
// suggestValues offers only the values the key being filled actually takes, so
|
||||
// a selector cannot be built out of a pair no node in the cluster matches.
|
||||
export function suggestValues(index, key, query) {
|
||||
return rank(index.values[key] || [], query)
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import test from 'node:test'
|
||||
|
||||
import { labelIndex, suggestKeys, suggestValues } from './nodeLabelSuggestions.js'
|
||||
|
||||
const NODES = [
|
||||
{ id: 'n1', name: 'Falcon GPU', labels: { 'gpu.vendor': 'NVIDIA', zone: 'east' } },
|
||||
{ id: 'n2', name: 'Worker 2', labels: { 'gpu.vendor': 'amd', zone: 'west' } },
|
||||
{ id: 'n3', name: 'Worker 3', labels: {} },
|
||||
{ id: 'n4', name: 'Worker 4' },
|
||||
{ id: 'n5', name: 'Worker 5', labels: { 'gpu.vram': '24GB' } },
|
||||
]
|
||||
|
||||
test('collects every distinct label key across the cluster', () => {
|
||||
assert.deepEqual(labelIndex(NODES).keys, ['gpu.vendor', 'gpu.vram', 'zone'])
|
||||
})
|
||||
|
||||
test('survives a node list that has not loaded yet', () => {
|
||||
assert.deepEqual(labelIndex(null), { keys: [], values: {} })
|
||||
assert.deepEqual(labelIndex([]), { keys: [], values: {} })
|
||||
})
|
||||
|
||||
test('collects the values a key actually takes, deduplicated', () => {
|
||||
const index = labelIndex([...NODES, { id: 'n6', labels: { zone: 'east' } }])
|
||||
assert.deepEqual(index.values.zone, ['east', 'west'])
|
||||
})
|
||||
|
||||
test('offers every key before the user has typed anything', () => {
|
||||
assert.deepEqual(suggestKeys(labelIndex(NODES), ''), ['gpu.vendor', 'gpu.vram', 'zone'])
|
||||
})
|
||||
|
||||
test('matches a key anywhere in the string, ignoring case', () => {
|
||||
assert.deepEqual(suggestKeys(labelIndex(NODES), 'VEND'), ['gpu.vendor'])
|
||||
})
|
||||
|
||||
test('ranks keys that start with the query above keys that merely contain it', () => {
|
||||
const index = labelIndex([{ id: 'n1', labels: { 'node.zone': 'a', zone: 'b' } }])
|
||||
assert.deepEqual(suggestKeys(index, 'zone'), ['zone', 'node.zone'])
|
||||
})
|
||||
|
||||
// A key already in the selector is not a suggestion: adding it again would
|
||||
// silently overwrite the pair the user just built.
|
||||
test('drops keys the selector already carries', () => {
|
||||
assert.deepEqual(suggestKeys(labelIndex(NODES), '', ['gpu.vendor']), ['gpu.vram', 'zone'])
|
||||
})
|
||||
|
||||
test('offers only the values that belong to the key being filled', () => {
|
||||
assert.deepEqual(suggestValues(labelIndex(NODES), 'gpu.vendor', ''), ['NVIDIA', 'amd'])
|
||||
})
|
||||
|
||||
test('matches a value ignoring case, so the chip keeps the cluster spelling', () => {
|
||||
assert.deepEqual(suggestValues(labelIndex(NODES), 'gpu.vendor', 'nvi'), ['NVIDIA'])
|
||||
})
|
||||
|
||||
test('offers nothing for a key the cluster has never reported', () => {
|
||||
assert.deepEqual(suggestValues(labelIndex(NODES), 'made.up', ''), [])
|
||||
})
|
||||
@@ -0,0 +1,99 @@
|
||||
package routes
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/mudler/LocalAI/core/application"
|
||||
"github.com/mudler/LocalAI/core/gallery"
|
||||
"github.com/mudler/LocalAI/core/http/endpoints/localai"
|
||||
"github.com/mudler/LocalAI/core/services/nodes"
|
||||
"github.com/mudler/LocalAI/pkg/system"
|
||||
"github.com/mudler/xlog"
|
||||
)
|
||||
|
||||
// ClusterMemoryProvider reports the memory budget a model actually gets: the
|
||||
// largest single healthy node. It is nil in single-node mode, where the local
|
||||
// host is the only thing worth sizing against.
|
||||
type ClusterMemoryProvider func(ctx context.Context) (*nodes.ClusterMemory, error)
|
||||
|
||||
// ClusterMemoryProviderFor returns the memory source backing every surface that
|
||||
// answers "will this model fit", or nil in single-node mode.
|
||||
//
|
||||
// This is the sibling of ClusterCapabilityProviderFor and exists for the same
|
||||
// reason. Backend discovery already unions worker capabilities because the
|
||||
// controller is usually a GPU-less pod; the model gallery asks a second
|
||||
// question about the same hardware, "how big a model can run here", and
|
||||
// answering it from the controller's own RAM told admins that a cluster of
|
||||
// A100s could only run the smallest CPU build.
|
||||
func ClusterMemoryProviderFor(app *application.Application) ClusterMemoryProvider {
|
||||
if app == nil || !app.IsDistributed() || app.Distributed().Registry == nil {
|
||||
return nil
|
||||
}
|
||||
return app.Distributed().Registry.HealthyNodeMemory
|
||||
}
|
||||
|
||||
// resolveClusterMemory reads the cluster's memory budget, degrading to no
|
||||
// reading on error.
|
||||
//
|
||||
// Every caller treats a nil reading as "size against the local host exactly as
|
||||
// before", so a registry hiccup narrows the catalog back to single-node
|
||||
// behavior rather than marking every model as too large.
|
||||
func resolveClusterMemory(ctx context.Context, provider ClusterMemoryProvider) *nodes.ClusterMemory {
|
||||
if provider == nil {
|
||||
return nil
|
||||
}
|
||||
memory, err := provider(ctx)
|
||||
if err != nil {
|
||||
xlog.Warn("Could not read cluster memory, sizing models against the local system only", "error", err)
|
||||
return nil
|
||||
}
|
||||
return memory
|
||||
}
|
||||
|
||||
// clusterResourceBlock renders a cluster reading for the API surfaces that
|
||||
// carry one, or nil when there is nothing to report.
|
||||
//
|
||||
// It is an ADDITIONAL field rather than a rewrite of the local aggregate. The
|
||||
// resource monitor shows the controller's genuine own usage and must keep
|
||||
// doing so; only the model-sizing surfaces switch to this block, and a client
|
||||
// that has never heard of it behaves exactly as it did before.
|
||||
func clusterResourceBlock(memory *nodes.ClusterMemory) map[string]any {
|
||||
if memory == nil {
|
||||
return nil
|
||||
}
|
||||
return map[string]any{
|
||||
"enabled": true,
|
||||
"node_id": memory.NodeID,
|
||||
"node_name": memory.NodeName,
|
||||
"total_memory": memory.TotalMemory,
|
||||
"is_gpu": memory.IsGPU,
|
||||
"node_count": memory.NodeCount,
|
||||
}
|
||||
}
|
||||
|
||||
// hostModelEnv describes the local host to variant selection.
|
||||
func hostModelEnv(ctx context.Context, systemState *system.SystemState) gallery.ResolveEnv {
|
||||
return gallery.HostResolveEnv(ctx, systemState)
|
||||
}
|
||||
|
||||
// clusterModelEnv describes whichever machine actually runs models to variant
|
||||
// selection: the cluster's best node in distributed mode, the local host
|
||||
// otherwise.
|
||||
//
|
||||
// The two providers are read independently on purpose. A cluster that can
|
||||
// report its hardware but not a usable memory figure still deserves the
|
||||
// hardware verdict, so each half falls back on its own.
|
||||
func clusterModelEnv(ctx context.Context, systemState *system.SystemState, memory ClusterMemoryProvider, capabilities localai.ClusterCapabilityProvider) gallery.ResolveEnv {
|
||||
reading := resolveClusterMemory(ctx, memory)
|
||||
caps := localai.ResolveClusterCapabilities(ctx, capabilities)
|
||||
|
||||
if reading == nil && len(caps) == 0 {
|
||||
return hostModelEnv(ctx, systemState)
|
||||
}
|
||||
|
||||
var budget uint64
|
||||
if reading != nil {
|
||||
budget = reading.TotalMemory
|
||||
}
|
||||
return gallery.ClusterResolveEnv(ctx, systemState, budget, caps)
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
package routes
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/nodes"
|
||||
"github.com/mudler/LocalAI/pkg/system"
|
||||
)
|
||||
|
||||
// Every model-sizing surface on a distributed controller reads the cluster
|
||||
// through these seams, and each one degrades to the controller's own hardware
|
||||
// rather than to "nothing fits".
|
||||
var _ = Describe("cluster memory resolution", func() {
|
||||
gib := func(n uint64) uint64 { return n * 1024 * 1024 * 1024 }
|
||||
|
||||
reading := &nodes.ClusterMemory{
|
||||
NodeID: "n-1", NodeName: "dgx-01", TotalMemory: 80 * 1024 * 1024 * 1024,
|
||||
IsGPU: true, NodeCount: 4,
|
||||
}
|
||||
|
||||
Describe("resolveClusterMemory", func() {
|
||||
It("reports nothing in single-node mode", func() {
|
||||
Expect(resolveClusterMemory(context.Background(), nil)).To(BeNil())
|
||||
})
|
||||
|
||||
It("reports the provider's reading", func() {
|
||||
provider := func(context.Context) (*nodes.ClusterMemory, error) { return reading, nil }
|
||||
|
||||
Expect(resolveClusterMemory(context.Background(), provider)).To(Equal(reading))
|
||||
})
|
||||
|
||||
// A registry hiccup must never mark the whole catalog as too large.
|
||||
It("degrades to no reading when the registry errors", func() {
|
||||
provider := func(context.Context) (*nodes.ClusterMemory, error) {
|
||||
return nil, errors.New("connection refused")
|
||||
}
|
||||
|
||||
Expect(resolveClusterMemory(context.Background(), provider)).To(BeNil())
|
||||
})
|
||||
})
|
||||
|
||||
Describe("clusterResourceBlock", func() {
|
||||
It("reports nothing to serialize when there is no reading", func() {
|
||||
Expect(clusterResourceBlock(nil)).To(BeNil())
|
||||
})
|
||||
|
||||
// The node name travels with the number because "fits" is only ever
|
||||
// meaningful somewhere, and the UI says where.
|
||||
It("names the node the budget belongs to", func() {
|
||||
block := clusterResourceBlock(reading)
|
||||
|
||||
Expect(block).To(HaveKeyWithValue("enabled", true))
|
||||
Expect(block).To(HaveKeyWithValue("node_count", 4))
|
||||
Expect(block).To(HaveKeyWithValue("total_memory", gib(80)))
|
||||
Expect(block).To(HaveKeyWithValue("node_name", "dgx-01"))
|
||||
Expect(block).To(HaveKeyWithValue("node_id", "n-1"))
|
||||
Expect(block).To(HaveKeyWithValue("is_gpu", true))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("clusterModelEnv", func() {
|
||||
controller := system.NewCapabilityState("default")
|
||||
|
||||
It("describes the controller when no provider is wired", func() {
|
||||
env := clusterModelEnv(context.Background(), controller, nil, nil)
|
||||
|
||||
Expect(env.AvailableMemory).To(Equal(hostModelEnv(context.Background(), controller).AvailableMemory))
|
||||
Expect(env.BackendCompatible("cuda-13-vllm")).To(BeFalse())
|
||||
})
|
||||
|
||||
It("describes the cluster when both providers answer", func() {
|
||||
memProvider := func(context.Context) (*nodes.ClusterMemory, error) { return reading, nil }
|
||||
capProvider := func(context.Context) ([]string, error) {
|
||||
return []string{"nvidia-cuda-13"}, nil
|
||||
}
|
||||
|
||||
env := clusterModelEnv(context.Background(), controller, memProvider, capProvider)
|
||||
|
||||
Expect(env.AvailableMemory).To(Equal(gib(80)))
|
||||
Expect(env.BackendCompatible("cuda-13-vllm")).To(BeTrue())
|
||||
})
|
||||
|
||||
// Half an answer is still better than the controller's: a cluster that
|
||||
// reports hardware but no usable memory keeps the hardware verdict.
|
||||
It("uses what the cluster could answer when the memory reading is missing", func() {
|
||||
capProvider := func(context.Context) ([]string, error) {
|
||||
return []string{"nvidia-cuda-13"}, nil
|
||||
}
|
||||
|
||||
env := clusterModelEnv(context.Background(), controller, nil, capProvider)
|
||||
|
||||
Expect(env.BackendCompatible("cuda-13-vllm")).To(BeTrue())
|
||||
Expect(env.AvailableMemory).To(Equal(hostModelEnv(context.Background(), controller).AvailableMemory))
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -123,6 +123,14 @@ func getDirectorySize(path string) (int64, error) {
|
||||
// RegisterUIAPIRoutes registers JSON API routes for the web UI
|
||||
func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfig *config.ApplicationConfig, galleryService *galleryop.GalleryService, opcache *galleryop.OpCache, applicationInstance *application.Application, adminMiddleware echo.MiddlewareFunc) {
|
||||
|
||||
// Both are nil in single-node mode, which leaves every surface below
|
||||
// sizing models against the local host exactly as it always has. In
|
||||
// distributed mode the models run on the workers, so "how big a model fits"
|
||||
// and "which hardware can run it" are questions about them, not about this
|
||||
// usually GPU-less controller.
|
||||
clusterMemory := ClusterMemoryProviderFor(applicationInstance)
|
||||
clusterCapabilities := ClusterCapabilityProviderFor(applicationInstance)
|
||||
|
||||
// Operations API - Get all current operations (models + backends)
|
||||
app.GET("/api/operations", func(c echo.Context) error {
|
||||
processingData, taskTypes := opcache.GetStatus()
|
||||
@@ -772,7 +780,7 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model
|
||||
|
||||
ramInfo, _ := xsysinfo.GetSystemRAMInfo()
|
||||
|
||||
return c.JSON(200, map[string]any{
|
||||
listing := map[string]any{
|
||||
"models": modelsJSON,
|
||||
"repositories": appConfig.Galleries,
|
||||
"allTags": tags,
|
||||
@@ -788,7 +796,16 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model
|
||||
"totalPages": totalPages,
|
||||
"prevPage": prevPage,
|
||||
"nextPage": nextPage,
|
||||
})
|
||||
}
|
||||
|
||||
// The ram* fields above stay the controller's own, so nothing that
|
||||
// reads them changes meaning; a client sizing models reads this
|
||||
// instead, and it is absent entirely in single-node mode.
|
||||
if block := clusterResourceBlock(resolveClusterMemory(c.Request().Context(), clusterMemory)); block != nil {
|
||||
listing["cluster"] = block
|
||||
}
|
||||
|
||||
return c.JSON(200, listing)
|
||||
}, adminMiddleware)
|
||||
|
||||
// Returns installed models with their capability flags for UI filtering
|
||||
@@ -1000,7 +1017,7 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model
|
||||
|
||||
// The full, unpaginated list: a variant references another gallery
|
||||
// entry by name and that entry need not be anywhere near this one.
|
||||
env := gallery.HostResolveEnv(c.Request().Context(), appConfig.SystemState)
|
||||
env := clusterModelEnv(c.Request().Context(), appConfig.SystemState, clusterMemory, clusterCapabilities)
|
||||
view, err := gallery.DescribeVariants(models, model, env)
|
||||
if err != nil {
|
||||
// A malformed variant list must not break the picker; the entry
|
||||
@@ -1876,6 +1893,13 @@ func RegisterUIAPIRoutes(app *echo.Echo, cl *config.ModelConfigLoader, ml *model
|
||||
"watchdog_interval": watchdogInterval,
|
||||
}
|
||||
|
||||
// An additional field, never a rewrite of the local aggregate above:
|
||||
// the resource monitor reports this controller's genuine own usage, and
|
||||
// only the model-sizing surfaces read the cluster block.
|
||||
if block := clusterResourceBlock(resolveClusterMemory(c.Request().Context(), clusterMemory)); block != nil {
|
||||
response["cluster"] = block
|
||||
}
|
||||
|
||||
return c.JSON(200, response)
|
||||
}, adminMiddleware)
|
||||
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
package nodes
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// ClusterMemory reports the memory budget a model actually gets in a
|
||||
// distributed deployment: that of the single largest healthy backend node.
|
||||
//
|
||||
// The largest node, not the fleet total. A model loads into one node, so
|
||||
// summing a fleet of four 16GB cards into 64GB would tell an admin a 40GB
|
||||
// model fits when no node can ever hold it. Naming the node is part of the
|
||||
// answer for the same reason: "fits" is only meaningful somewhere.
|
||||
type ClusterMemory struct {
|
||||
NodeID string `json:"node_id"`
|
||||
NodeName string `json:"node_name"`
|
||||
TotalMemory uint64 `json:"total_memory"`
|
||||
IsGPU bool `json:"is_gpu"`
|
||||
NodeCount int `json:"node_count"`
|
||||
}
|
||||
|
||||
// HealthyNodeMemory reports the largest model budget any single healthy backend
|
||||
// node can offer, or nil when the cluster can answer nothing.
|
||||
//
|
||||
// A nil reading is not an error. It means the caller should size against
|
||||
// whatever it sized against before, which keeps a registry hiccup or an
|
||||
// empty cluster from marking the entire catalog as too large.
|
||||
//
|
||||
// Only healthy backend nodes count, the same predicate the scheduler places
|
||||
// against, so a drained worker stops advertising hardware the cluster cannot
|
||||
// currently use.
|
||||
func (r *NodeRegistry) HealthyNodeMemory(ctx context.Context) (*ClusterMemory, error) {
|
||||
var nodes []BackendNode
|
||||
if err := r.db.WithContext(ctx).
|
||||
Where("status = ? AND node_type = ?", StatusHealthy, NodeTypeBackend).
|
||||
Find(&nodes).Error; err != nil {
|
||||
return nil, fmt.Errorf("listing healthy backend node memory: %w", err)
|
||||
}
|
||||
|
||||
var best *ClusterMemory
|
||||
count := 0
|
||||
for _, node := range nodes {
|
||||
budget, isGPU := nodeModelBudget(node)
|
||||
if budget == 0 {
|
||||
continue
|
||||
}
|
||||
count++
|
||||
if best == nil || betterBudget(budget, isGPU, best.TotalMemory, best.IsGPU) {
|
||||
best = &ClusterMemory{
|
||||
NodeID: node.ID,
|
||||
NodeName: node.Name,
|
||||
TotalMemory: budget,
|
||||
IsGPU: isGPU,
|
||||
}
|
||||
}
|
||||
}
|
||||
if best == nil {
|
||||
return nil, nil
|
||||
}
|
||||
best.NodeCount = count
|
||||
return best, nil
|
||||
}
|
||||
|
||||
// nodeModelBudget reports how much memory a model may occupy on one node, the
|
||||
// per-node form of the same question core/gallery answers for a single host:
|
||||
// VRAM when the node has a GPU, system RAM otherwise.
|
||||
//
|
||||
// An operator-set VRAM budget wins over raw VRAM. The scheduler already refuses
|
||||
// a load above that ceiling, so sizing against the raw total would advertise a
|
||||
// fit the cluster then rejects.
|
||||
func nodeModelBudget(node BackendNode) (uint64, bool) {
|
||||
if node.TotalVRAM > 0 {
|
||||
if node.VRAMBudgetBytes > 0 && node.VRAMBudgetBytes < node.TotalVRAM {
|
||||
return node.VRAMBudgetBytes, true
|
||||
}
|
||||
return node.TotalVRAM, true
|
||||
}
|
||||
return node.TotalRAM, false
|
||||
}
|
||||
|
||||
// betterBudget ranks one node's budget against the incumbent's.
|
||||
//
|
||||
// A GPU node always beats a CPU node, however much system RAM the CPU node
|
||||
// holds: a 512GB CPU box will serve a 70B model at a speed nobody would pick
|
||||
// over a 24GB card, so reporting the CPU box as the cluster's capability would
|
||||
// recommend models the cluster cannot usefully run.
|
||||
func betterBudget(budget uint64, isGPU bool, bestBudget uint64, bestIsGPU bool) bool {
|
||||
if isGPU != bestIsGPU {
|
||||
return isGPU
|
||||
}
|
||||
return budget > bestBudget
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
package nodes
|
||||
|
||||
import (
|
||||
"context"
|
||||
"runtime"
|
||||
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
|
||||
"github.com/mudler/LocalAI/core/services/testutil"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// The model gallery on a GPU-less controller sizes every "does this fit"
|
||||
// verdict against what this reports, so the node it picks decides which models
|
||||
// admins are told they can run.
|
||||
var _ = Describe("NodeRegistry HealthyNodeMemory", func() {
|
||||
var (
|
||||
db *gorm.DB
|
||||
registry *NodeRegistry
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
if runtime.GOOS == "darwin" {
|
||||
Skip("testcontainers requires Docker, not available on macOS CI")
|
||||
}
|
||||
db = testutil.SetupTestDB()
|
||||
var err error
|
||||
registry, err = NewNodeRegistry(db)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
})
|
||||
|
||||
register := func(node *BackendNode) {
|
||||
Expect(registry.Register(context.Background(), node, true)).To(Succeed())
|
||||
}
|
||||
|
||||
It("reports the VRAM of the single GPU worker", func() {
|
||||
register(&BackendNode{
|
||||
Name: "gpu-worker", NodeType: NodeTypeBackend, Address: "10.0.0.1:50051",
|
||||
TotalVRAM: 24_000_000_000, TotalRAM: 64_000_000_000, GPUVendor: "nvidia",
|
||||
})
|
||||
|
||||
mem, err := registry.HealthyNodeMemory(context.Background())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(mem).ToNot(BeNil())
|
||||
Expect(mem.TotalMemory).To(Equal(uint64(24_000_000_000)))
|
||||
Expect(mem.NodeName).To(Equal("gpu-worker"))
|
||||
Expect(mem.IsGPU).To(BeTrue())
|
||||
Expect(mem.NodeCount).To(Equal(1))
|
||||
})
|
||||
|
||||
// A model has to load on ONE node, so the biggest single node is the
|
||||
// budget. Summing the fleet would promise a 40GB model fits on a cluster of
|
||||
// four 16GB cards that can never hold it.
|
||||
It("picks the largest single node rather than the fleet total", func() {
|
||||
register(&BackendNode{
|
||||
Name: "small", NodeType: NodeTypeBackend, Address: "10.0.0.1:50051",
|
||||
TotalVRAM: 16_000_000_000, GPUVendor: "nvidia",
|
||||
})
|
||||
register(&BackendNode{
|
||||
Name: "large", NodeType: NodeTypeBackend, Address: "10.0.0.2:50051",
|
||||
TotalVRAM: 48_000_000_000, GPUVendor: "nvidia",
|
||||
})
|
||||
register(&BackendNode{
|
||||
Name: "medium", NodeType: NodeTypeBackend, Address: "10.0.0.3:50051",
|
||||
TotalVRAM: 24_000_000_000, GPUVendor: "nvidia",
|
||||
})
|
||||
|
||||
mem, err := registry.HealthyNodeMemory(context.Background())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(mem).ToNot(BeNil())
|
||||
Expect(mem.TotalMemory).To(Equal(uint64(48_000_000_000)))
|
||||
Expect(mem.NodeName).To(Equal("large"))
|
||||
Expect(mem.NodeCount).To(Equal(3))
|
||||
})
|
||||
|
||||
// A CPU worker runs models out of system RAM, exactly as a single-node
|
||||
// LocalAI does when it finds no GPU.
|
||||
It("falls back to system RAM for a worker with no GPU", func() {
|
||||
register(&BackendNode{
|
||||
Name: "cpu-worker", NodeType: NodeTypeBackend, Address: "10.0.0.1:50051",
|
||||
TotalRAM: 128_000_000_000,
|
||||
})
|
||||
|
||||
mem, err := registry.HealthyNodeMemory(context.Background())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(mem).ToNot(BeNil())
|
||||
Expect(mem.TotalMemory).To(Equal(uint64(128_000_000_000)))
|
||||
Expect(mem.NodeName).To(Equal("cpu-worker"))
|
||||
Expect(mem.IsGPU).To(BeFalse())
|
||||
})
|
||||
|
||||
// A 24GB card beats a 512GB CPU box for anything a user would actually
|
||||
// serve, so a GPU node wins even when a CPU node reports more bytes.
|
||||
It("prefers a GPU node over a CPU node holding more system RAM", func() {
|
||||
register(&BackendNode{
|
||||
Name: "fat-cpu", NodeType: NodeTypeBackend, Address: "10.0.0.1:50051",
|
||||
TotalRAM: 512_000_000_000,
|
||||
})
|
||||
register(&BackendNode{
|
||||
Name: "gpu", NodeType: NodeTypeBackend, Address: "10.0.0.2:50051",
|
||||
TotalVRAM: 24_000_000_000, GPUVendor: "nvidia",
|
||||
})
|
||||
|
||||
mem, err := registry.HealthyNodeMemory(context.Background())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(mem).ToNot(BeNil())
|
||||
Expect(mem.NodeName).To(Equal("gpu"))
|
||||
Expect(mem.IsGPU).To(BeTrue())
|
||||
Expect(mem.TotalMemory).To(Equal(uint64(24_000_000_000)))
|
||||
})
|
||||
|
||||
// The same predicate the scheduler places against: an unhealthy worker
|
||||
// cannot take a load, so its hardware must not size the catalog either.
|
||||
It("ignores unhealthy nodes", func() {
|
||||
register(&BackendNode{
|
||||
Name: "healthy", NodeType: NodeTypeBackend, Address: "10.0.0.1:50051",
|
||||
TotalVRAM: 16_000_000_000, GPUVendor: "nvidia",
|
||||
})
|
||||
register(&BackendNode{
|
||||
Name: "offline", NodeType: NodeTypeBackend, Address: "10.0.0.2:50051",
|
||||
TotalVRAM: 80_000_000_000, GPUVendor: "nvidia",
|
||||
})
|
||||
offline, err := registry.GetByName(context.Background(), "offline")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(registry.MarkUnhealthy(context.Background(), offline.ID)).To(Succeed())
|
||||
|
||||
mem, err := registry.HealthyNodeMemory(context.Background())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(mem).ToNot(BeNil())
|
||||
Expect(mem.NodeName).To(Equal("healthy"))
|
||||
Expect(mem.NodeCount).To(Equal(1))
|
||||
})
|
||||
|
||||
// The scheduler refuses a load that exceeds an operator-set budget, so a
|
||||
// verdict sized against raw VRAM would promise a fit the cluster rejects.
|
||||
It("respects an operator-set VRAM budget", func() {
|
||||
register(&BackendNode{
|
||||
Name: "capped", NodeType: NodeTypeBackend, Address: "10.0.0.1:50051",
|
||||
TotalVRAM: 48_000_000_000, GPUVendor: "nvidia",
|
||||
VRAMBudget: "24GB", VRAMBudgetManuallySet: true,
|
||||
})
|
||||
|
||||
mem, err := registry.HealthyNodeMemory(context.Background())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(mem).ToNot(BeNil())
|
||||
Expect(mem.TotalMemory).To(Equal(uint64(24_000_000_000)))
|
||||
})
|
||||
|
||||
// Nothing to size against is not an error: the caller degrades to the
|
||||
// controller's own memory rather than blanking the catalog.
|
||||
It("returns no reading when the cluster has no healthy backend node", func() {
|
||||
mem, err := registry.HealthyNodeMemory(context.Background())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(mem).To(BeNil())
|
||||
})
|
||||
|
||||
// A worker that reports neither VRAM nor RAM tells us nothing, and a zero
|
||||
// budget would mark every model as too large.
|
||||
It("returns no reading when every healthy node reports zero memory", func() {
|
||||
register(&BackendNode{
|
||||
Name: "silent", NodeType: NodeTypeBackend, Address: "10.0.0.1:50051",
|
||||
})
|
||||
|
||||
mem, err := registry.HealthyNodeMemory(context.Background())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(mem).To(BeNil())
|
||||
})
|
||||
})
|
||||
@@ -486,6 +486,43 @@ Used by the WebUI and admin API consumers. Requires admin authentication.
|
||||
|
||||
The **Nodes** page in the React WebUI provides a visual overview of all registered workers, their statuses, and loaded models. The page opens with a one-line **cluster pulse** summarising node health and an **attention callout** that surfaces nodes needing action (for example pending approvals). Below that, a roster of **node panels** lists each worker with its inline model chips (no expand click needed), filtered by an **All / Backend / Agent** segmented control. Selecting a panel opens a dedicated **node detail page** at `/app/nodes/:id` with per-node metrics, models, and backend actions. Model scheduling lives on its own **Scheduling** page (separate nav item), not as a tab on the Nodes page.
|
||||
|
||||
### Model sizing in the WebUI
|
||||
|
||||
The model gallery answers "will this model run here" against the cluster, not
|
||||
against the frontend. A distributed frontend is usually a GPU-less pod, so
|
||||
sizing models against its own memory would report that a fleet of GPU workers
|
||||
can only run the smallest CPU build.
|
||||
|
||||
The budget is the **largest single healthy backend node**, not the sum of the
|
||||
fleet: a model loads into one node, so four 16GB workers do not add up to a home
|
||||
for a 40GB model. A node's operator-set VRAM budget caps its contribution, since
|
||||
the scheduler would refuse a load above that ceiling anyway, and a GPU node wins
|
||||
over a CPU node holding more system RAM. The gallery names the node its verdict
|
||||
belongs to ("Fits on dgx-01").
|
||||
|
||||
`GET /api/resources` and `GET /api/models` carry this as an additional `cluster`
|
||||
object; their existing `aggregate` and `ram*` fields keep reporting the
|
||||
frontend's own hardware, which is what the resource monitor shows. The object is
|
||||
absent in single-node mode, and also whenever the registry cannot be read, in
|
||||
which case every sizing surface falls back to the local host:
|
||||
|
||||
```json
|
||||
{
|
||||
"cluster": {
|
||||
"enabled": true,
|
||||
"node_id": "a1b2c3",
|
||||
"node_name": "dgx-01",
|
||||
"total_memory": 85899345920,
|
||||
"is_gpu": true,
|
||||
"node_count": 4
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Variant selection (`GET /api/models/variants/:id`) uses the same reading, and
|
||||
judges backend compatibility against the union of the capabilities present in
|
||||
the cluster, so a CUDA-only build is offered when any worker can run it.
|
||||
|
||||
### Model configuration revisions
|
||||
|
||||
Distributed mode assigns a `config_revision` to each validated model configuration. It hashes the persisted semantic configuration, including fields such as `context_size` and parallel settings. YAML formatting, comments, and map order do not change it.
|
||||
@@ -836,6 +873,12 @@ curl -X POST http://frontend:8080/api/nodes/scheduling \
|
||||
|
||||
Without a node selector, models can schedule on any healthy node (default behavior).
|
||||
|
||||
In the WebUI, the node selector field completes what you type against the labels
|
||||
your cluster actually reports: start typing a key and the matching label keys
|
||||
appear inline, then the value field offers only the values that key takes. A key
|
||||
no node reports yet is still accepted as typed, so you can write a rule before
|
||||
labelling the nodes for it.
|
||||
|
||||
### Replica Auto-Scaling
|
||||
|
||||
Control the number of model replicas across the cluster:
|
||||
|
||||
Reference in new issue
Block a user