Ettore Di Giacinto adbb76607d fix(distributed): offline stale unhealthy nodes ...

Allow automatic stale-node handling to advance unhealthy nodes to offline while preserving unhealthy status when automatic handling is disabled.

Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]

2026-09-27 03:05:13 +00:00

..

prefixcache

fix(distributed): preserve master protocol extensions

2026-09-27 03:05:13 +00:00

agent_control_test.go

feat(cluster): make a peer prove which replica it is

2026-09-27 03:05:13 +00:00

agent_control.go

feat(distributed): carry an agent cancel on the worker's own tunnel

2026-09-27 03:05:13 +00:00

agent_selector_test.go

feat(distributed): carry an agent cancel on the worker's own tunnel

2026-09-27 03:05:13 +00:00

agent_selector.go

feat(distributed): carry an agent cancel on the worker's own tunnel

2026-09-27 03:05:13 +00:00

alias_scheduling_test.go

feat(distributed): key scheduling rules by a model alias (#11771)

2026-08-29 09:16:50 +02:00

alias_scheduling.go

feat(distributed): key scheduling rules by a model alias (#11771)

2026-08-29 09:16:50 +02:00

authenticated_routes_test.go

fix(worker): pin the rune cut, answer unload honestly, drop the dead publisher

2026-09-27 03:05:12 +00:00

backend_client_factory_test.go

fix(cluster): make "no route" a condition of its own, and let it out of the package

2026-09-27 03:05:12 +00:00

cluster_memory.go

fix(ui): size model fit against the cluster and move node labels into the selector (#11765)

2026-08-28 22:57:35 +02:00

control_client_test.go

feat(distributed): let a worker name a broadcast the frontend decides on

2026-09-27 03:05:12 +00:00

control_client.go

fix(distributed): drop a departed node's cached HTTP clients

2026-09-27 03:05:13 +00:00

control_worker_fake_test.go

refactor(distributed): delete MessagingClient and shrink the NATS client to fan-out

2026-09-27 03:05:13 +00:00

departure_test.go

fix(distributed): read a departed agent tunnel as the routing fact it is

2026-09-27 03:05:13 +00:00

departure.go

fix(distributed): read a departed agent tunnel as the routing fact it is

2026-09-27 03:05:13 +00:00

disk_headroom_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

disk_headroom.go

fix(nodes): never schedule a model onto a node that cannot store it (#11054)

2026-07-23 00:03:21 +02:00

distributed_store_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

distributed_store.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

file_stager_http.go

fix(distributed): drop a departed node's cached HTTP clients

2026-09-27 03:05:13 +00:00

file_stager_release_test.go

fix(distributed): preserve master protocol extensions

2026-09-27 03:05:13 +00:00

file_stager_s3_test.go

test(distributed): pin the unreadable-request rule at all three file verbs

2026-09-27 03:05:12 +00:00

file_stager_s3.go

fix(distributed): preserve master protocol extensions

2026-09-27 03:05:13 +00:00

file_stager_verify_deadline_test.go

feat(cluster): reach every worker through its tunnel, never its address

2026-09-27 03:05:12 +00:00

file_stager.go

fix(distributed): drop a departed node's cached HTTP clients

2026-09-27 03:05:13 +00:00

file_staging_3d_test.go

fix(distributed): drop a departed node's cached HTTP clients

2026-09-27 03:05:13 +00:00

file_staging_animation_test.go

feat: Add kimodo.cpp and 3D animation API/UI (#12095)

2026-09-18 06:12:03 +01:00

file_staging_client.go

test(distributed): pin the unreadable-request rule at all three file verbs

2026-09-27 03:05:12 +00:00

file_staging_lifecycle_test.go

fix(distributed): pass raw JPEG base64 through staging (#11956)

2026-09-10 00:42:55 +02:00

file_staging_sound_detection_test.go

fix(distributed): preserve master protocol extensions

2026-09-27 03:05:13 +00:00

file_staging_tts_test.go

feat(tts): support multi-reference personalities

2026-09-11 21:56:08 +00:00

file_transfer_finalize_test.go

fix(distributed): preserve master protocol extensions

2026-09-27 03:05:13 +00:00

file_transfer_server_test.go

feat(cluster): reach every worker through its tunnel, never its address

2026-09-27 03:05:12 +00:00

file_transfer_server.go

feat(distributed): give agent workers a tunnel of their own

2026-09-27 03:05:12 +00:00

health_miss_bounds_test.go

fix(distributed): bound the health monitor's miss streaks to live rows

2026-09-27 03:05:13 +00:00

health_mock_test.go

fix(distributed): bound the health monitor's miss streaks to live rows

2026-09-27 03:05:13 +00:00

health_test.go

fix(distributed): offline stale unhealthy nodes

2026-09-27 03:05:13 +00:00

health.go

fix(distributed): offline stale unhealthy nodes

2026-09-27 03:05:13 +00:00

heartbeat_checkpoint_test.go

fix(distributed): preserve master protocol extensions

2026-09-27 03:05:13 +00:00

inflight_test.go

feat: Add kimodo.cpp and 3D animation API/UI (#12095)

2026-09-18 06:12:03 +01:00

inflight.go

docs(grpc): withdraw the lint-cost claim, and make the one nolint tidy-proof

2026-09-27 03:05:12 +00:00

install_progress_publisher_test.go

fix(worker): pin the rune cut, answer unload honestly, drop the dead publisher

2026-09-27 03:05:12 +00:00

install_progress_publisher.go

fix(worker): pin the rune cut, answer unload honestly, drop the dead publisher

2026-09-27 03:05:12 +00:00

interfaces.go

feat(distributed): carry an agent cancel on the worker's own tunnel

2026-09-27 03:05:13 +00:00

load_deadline.go

fix(distributed): scale the remote model-load deadline with checkpoint size (#11030)

2026-07-22 09:30:23 +02:00

load_job_phase.go

fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock (#11514)

2026-08-15 13:20:11 +02:00

load_job_runner.go

fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock (#11514)

2026-08-15 13:20:11 +02:00

local_stub_invalidator_test.go

feat(cluster): reach every worker through its tunnel, never its address

2026-09-27 03:05:12 +00:00

local_stub_invalidator.go

fix(distributed): reaper reaps live backends, ghost model stubs, in_flight leak, sidecar staging runaway (#11142)

2026-07-27 18:36:31 +02:00

managers_agent_node_test.go

feat(distributed): stop a backend on one route, whatever the worker is

2026-09-27 03:05:12 +00:00

managers_distributed_test.go

feat(distributed): stop a backend on one route, whatever the worker is

2026-09-27 03:05:12 +00:00

managers_distributed.go

feat(distributed): give agent workers a tunnel of their own

2026-09-27 03:05:12 +00:00

model_cleanup_test.go

fix(distributed): keep model replicas config-consistent (#11664)

2026-08-22 22:44:03 +02:00

model_cleanup.go

fix(distributed): keep model replicas config-consistent (#11664)

2026-08-22 22:44:03 +02:00

model_identity_e2e_test.go

fix(distributed): reject wrong-model requests at the backend (#10970)

2026-07-20 13:05:47 +02:00

model_load_job_test.go

fix(distributed): keep staging operations stable (#11663)

2026-08-21 20:08:47 +02:00

model_load_job.go

fix(distributed): keep staging operations stable (#11663)

2026-08-21 20:08:47 +02:00

model_loading_error.go

fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock (#11514)

2026-08-15 13:20:11 +02:00

model_router_test.go

fix(distributed): refuse a worker that cannot tunnel, and say why it was refused

2026-09-27 03:05:12 +00:00

model_router.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

nodes_suite_test.go

Revert "chore(tests): Avoid network, sleep and more during tests" (#11601)

2026-08-19 16:39:39 +02:00

pending_op_cleanup_test.go

feat(distributed): read worker absence from the database, not from a bus timeout

2026-09-27 03:05:12 +00:00

per_node_clients_test.go

fix(distributed): drop a departed node's cached HTTP clients

2026-09-27 03:05:13 +00:00

pinned_eviction_test.go

fix(distributed): exclude pinned models from cluster eviction and idle scale-down

2026-09-11 23:08:26 +00:00

probe_cache_test.go

fix(distributed): let a worker's own refusal be evidence about its backend

2026-09-27 03:05:12 +00:00

probe_cache.go

fix(distributed): read a departed agent tunnel as the routing fact it is

2026-09-27 03:05:13 +00:00

rebroadcast_test.go

refactor(distributed): move job and agent fan-out onto the PostgreSQL carrier

2026-09-27 03:05:13 +00:00

rebroadcast.go

refactor(distributed): move job and agent fan-out onto the PostgreSQL carrier

2026-09-27 03:05:13 +00:00

reconciler_abandoned_load_test.go

fix(distributed): stop reclaiming healthy reconciler-driven loads

2026-08-24 13:00:34 +00:00

reconciler_abandoned_load.go

fix(distributed): stop reclaiming healthy reconciler-driven loads

2026-08-24 13:00:34 +00:00

reconciler_alias_test.go

feat(distributed): key scheduling rules by a model alias (#11771)

2026-08-29 09:16:50 +02:00

reconciler_busy_probe_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

reconciler_inflight_leak_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

reconciler_prober_test.go

fix(cluster): stop a late request frame reading as the worker's verdict

2026-09-27 03:05:12 +00:00

reconciler_test.go

feat(distributed): stop a backend on one route, whatever the worker is

2026-09-27 03:05:12 +00:00

reconciler_worker_processes_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

reconciler.go

feat(distributed): read worker absence from the database, not from a bus timeout

2026-09-27 03:05:12 +00:00

registry_capability_test.go

fix(distributed): backend discovery hid GPU-only backends behind the controller's capability (#10947)

2026-07-19 07:53:46 +00:00

registry_clustermemory_test.go

fix(ui): size model fit against the cluster and move node labels into the selector (#11765)

2026-08-28 22:57:35 +02:00

registry_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

registry_vrambudget_test.go

feat(vram): per-node VRAM allocation budget (LOCALAI_VRAM_BUDGET) (#10833)

2026-07-15 09:58:45 +02:00

registry_wire_test.go

fix(distributed): refuse a worker that cannot tunnel, and say why it was refused

2026-09-27 03:05:12 +00:00

registry.go

feat(distributed): read worker absence from the database, not from a bus timeout

2026-09-27 03:05:12 +00:00

replicapicker.go

refactor(routing): extract replica picker into pkg/clusterrouting (#10123)

2026-06-01 09:38:55 +02:00

revision_eligibility_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

revision_error_detail_test.go

fix(distributed): name both revisions in the stale error

2026-08-23 19:53:42 +00:00

router_companionstage_test.go

fix(model-artifacts): materialize longcat-video on the controller, and support companion repos (#10949)

2026-07-19 12:01:36 +02:00

router_dirstage_test.go

fix(model-artifacts): materialize longcat-video on the controller, and support companion repos (#10949)

2026-07-19 12:01:36 +02:00

router_eviction_alias_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

router_eviction_selector_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

router_hardware_internal_test.go

fix(config): only inject llama.cpp serving options on the llama.cpp path (#10822)

2026-07-14 17:46:15 +02:00

router_liveness_test.go

feat(cluster): make a peer prove which replica it is

2026-09-27 03:05:13 +00:00

router_liveness.go

fix(distributed): let absence be decided by something, at all three call sites

2026-09-27 03:05:12 +00:00

router_load_budget_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

router_load_job_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

router_load_timeout_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

router_missingsource_test.go

fix(model-artifacts): materialize longcat-video on the controller, and support companion repos (#10949)

2026-07-19 12:01:36 +02:00

router_optionstage_test.go

fix(distributed): stage backend companion assets to remote nodes (#10330)

2026-06-14 16:42:59 +02:00

router_reap_load_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

router_reservation_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

router_revision_lifecycle_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

router_sharedmodels_test.go

feat(distributed): add LOCALAI_DISTRIBUTED_SHARED_MODELS to skip staging on shared volumes (#10556) (#10566)

2026-06-28 01:23:07 +02:00

router_sidecar_stage_test.go

fix(distributed): reaper reaps live backends, ghost model stubs, in_flight leak, sidecar staging runaway (#11142)

2026-07-27 18:36:31 +02:00

router_slot_uncertainty_test.go

fix(distributed): preserve master protocol extensions

2026-09-27 03:05:13 +00:00

router_staging_context_test.go

fix(distributed): drop a departed node's cached HTTP clients

2026-09-27 03:05:13 +00:00

router_staging_deadline_test.go

fix(distributed): drop a departed node's cached HTTP clients

2026-09-27 03:05:13 +00:00

router_test.go

fix(distributed): drop a departed node's cached HTTP clients

2026-09-27 03:05:13 +00:00

router_unnamed_replica_test.go

fix(distributed): refuse a worker that cannot tunnel, and say why it was refused

2026-09-27 03:05:12 +00:00

router_unreachable_worker_test.go

feat(distributed): stop workers listening, and stop them advertising

2026-09-27 03:05:12 +00:00

router_virtual_model_test.go

fix(distributed): resolve paths for virtual models (#11911)

2026-09-07 19:32:17 +02:00

router_vrambudget_test.go

feat(vram): per-node VRAM allocation budget (LOCALAI_VRAM_BUDGET) (#10833)

2026-07-15 09:58:45 +02:00

router.go

fix(distributed): read a departed agent tunnel as the routing fact it is

2026-09-27 03:05:13 +00:00

scheduling_seed_test.go

feat(nodes): add pluggable routing pipeline

2026-09-11 21:47:08 +00:00

scheduling_seed.go

feat(nodes): add pluggable routing pipeline

2026-09-11 21:47:08 +00:00

staging_keys_test.go

feat: add distributed mode (#9124)

2026-03-30 00:47:27 +02:00

staging_keys.go

fix(model-artifacts): materialize longcat-video on the controller, and support companion repos (#10949)

2026-07-19 12:01:36 +02:00

staging_progress_broadcast_test.go

feat(distributed): move the last nine fan-out families onto PostgreSQL

2026-09-27 03:05:13 +00:00

staging_progress_pg_test.go

feat(distributed): move the last nine fan-out families onto PostgreSQL

2026-09-27 03:05:13 +00:00

staging_progress.go

fix(distributed): read a departed agent tunnel as the routing fact it is

2026-09-27 03:05:13 +00:00

unloader_stale_rows_test.go

feat(distributed): stop a backend on one route, whatever the worker is

2026-09-27 03:05:12 +00:00

unloader_test.go

refactor(distributed): delete MessagingClient and shrink the NATS client to fan-out

2026-09-27 03:05:13 +00:00

unloader_upgrade_test.go

feat(distributed): stop a backend on one route, whatever the worker is

2026-09-27 03:05:12 +00:00

unloader.go

feat(distributed): let a worker name a broadcast the frontend decides on

2026-09-27 03:05:12 +00:00

worker_readiness_datapath_test.go

fix(distributed): survive a slow control-plane database (#11837)

2026-09-02 12:37:12 +02:00

worker_readiness_test.go

feat(distributed): take the backend worker off NATS entirely

2026-09-27 03:05:12 +00:00

worker_readiness.go

feat(distributed): take the backend worker off NATS entirely

2026-09-27 03:05:12 +00:00

wrapper_transport_test.go

fix(grpc): let the transport answer through the wrappers, not only past gRPC

2026-09-27 03:05:12 +00:00