Ettore Di Giacinto 5dc9ed707d fix(nodes): back off failed replica scale-ups per model ...

When a scale-up failed, the reconciler tried again on every busy or
pressure tick, which logged a warning and sent a new backend install each
time. The "attempt" field in that warning was the loop index, so it always
read 1 and hid the repeats. A model that no request has loaded yet logged
"no load info" at warn level on every tick.

Track consecutive scale-up failures per model in memory. After a failure,
wait 30s, doubling on each further failure up to 10 minutes, before the
next try. A successful scale-up resets the count. Log the failure count
and the wait. Mark the missing load record with ErrNoLoadInfo and log it
at debug level, because only a first request can fix it.

Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

2026-11-03 09:33:41 +00:00

..

prefixcache

feat(prefixcache): index reported KV residency

2026-09-11 21:53:33 +00:00

agent_rpc_nats_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

agent_rpc_nats.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

alias_scheduling_frontends_test.go

fix(distributed): converge model configs across frontends (#12558)

2026-10-08 09:56:12 +02:00

alias_scheduling_test.go

feat(distributed): key scheduling rules by a model alias (#11771)

2026-08-29 09:16:50 +02:00

alias_scheduling.go

fix(distributed): converge model configs across frontends (#12558)

2026-10-08 09:56:12 +02:00

backend_op_locks.go

feat(galleryop): limit concurrent backend operations (#12888)

2026-10-25 00:26:45 +02:00

chat_start_nats_test.go

feat(chats): server-side chats with steering, sub-agents and compaction (#12918)

2026-11-01 12:08:23 +01:00

chat_start_nats.go

feat(chats): server-side chats with steering, sub-agents and compaction (#12918)

2026-11-01 12:08:23 +01:00

client_factory_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

cluster_memory.go

fix(ui): size model fit against the cluster and move node labels into the selector (#11765)

2026-08-28 22:57:35 +02:00

control_errors_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

control_nats.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

declared_files.go

fix(distributed): stage the files a model install declares (#12309)

2026-09-28 08:34:56 +02:00

disk_headroom_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

disk_headroom.go

fix(nodes): never schedule a model onto a node that cannot store it (#11054)

2026-07-23 00:03:21 +02:00

distributed_store_test.go

feat: add distributed mode (#9124)

2026-03-30 00:47:27 +02:00

distributed_store.go

feat: add distributed mode (#9124)

2026-03-30 00:47:27 +02:00

env_test.go

fix(proxy): propagate model env and accept direct keys (#12940)

2026-10-27 10:27:45 +01:00

file_stager_dial_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

file_stager_http.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

file_stager_release_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

file_stager_s3.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

file_stager_verify_deadline_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

file_stager.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

file_staging_3d_test.go

feat: Add 3d generation UI/API and trellis2cpp backend (#10979)

2026-07-29 16:15:04 +02:00

file_staging_animation_test.go

feat: Add kimodo.cpp and 3D animation API/UI (#12095)

2026-09-18 06:12:03 +01:00

file_staging_client.go

feat(tts): stream Qwen text and add assistant preset (#12770)

2026-10-19 12:30:52 +02:00

file_staging_diarization_test.go

fix(nodes): stage dedicated diarization audio (#12474)

2026-10-04 13:14:00 +02:00

file_staging_lifecycle_test.go

fix(distributed): pass raw JPEG base64 through staging (#11956)

2026-09-10 00:42:55 +02:00

file_staging_sound_detection_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

file_staging_tts_test.go

feat(tts): support multi-reference personalities

2026-09-11 21:56:08 +00:00

file_transfer_finalize_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

file_transfer_server_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

file_transfer_server.go

fix(distributed): bound ephemeral staging (#11924)

2026-09-08 18:13:40 +02:00

health_mock_test.go

feat(backends): timestral.cpp forecast backend and /v1/forecast API (#12885)

2026-10-24 22:47:04 +02:00

health_test.go

fix(distributed): cascade-clean stale node_models rows + filter routing by healthy status (#9754)

2026-05-13 21:57:50 +02:00

health.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

heartbeat_checkpoint_test.go

feat(nodes): add fleet operations dashboard (#12046)

2026-09-15 00:57:14 +02:00

inflight_test.go

feat(backends): timestral.cpp forecast backend and /v1/forecast API (#12885)

2026-10-24 22:47:04 +02:00

inflight.go

feat(backends): timestral.cpp forecast backend and /v1/forecast API (#12885)

2026-10-24 22:47:04 +02:00

install_progress_publisher_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

install_progress_publisher.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

interfaces.go

fix(proxy): propagate model env and accept direct keys (#12940)

2026-10-27 10:27:45 +01:00

load_cancel.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

load_deadline.go

fix(distributed): scale the remote model-load deadline with checkpoint size (#11030)

2026-07-22 09:30:23 +02:00

load_job_generation_test.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

load_job_lease_test.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

load_job_phase.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

load_job_runner.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

load_job_sqlite_test.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

load_operation_control_conformance_test.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

load_worker_ops_test.go

fix(proxy): propagate model env and accept direct keys (#12940)

2026-10-27 10:27:45 +01:00

local_stub_invalidator_test.go

fix(distributed): keep model replicas config-consistent (#11664)

2026-08-22 22:44:03 +02:00

local_stub_invalidator.go

fix(distributed): reaper reaps live backends, ghost model stubs, in_flight leak, sidecar staging runaway (#11142)

2026-07-27 18:36:31 +02:00

managers_agent_node_test.go

test(nodes): give the DistributedBackendManager fixtures a system state (#12877)

2026-10-24 15:34:21 +02:00

managers_distributed_fanout_test.go

feat(galleryop): limit concurrent backend operations (#12888)

2026-10-25 00:26:45 +02:00

managers_distributed_test.go

test(nodes): give the DistributedBackendManager fixtures a system state (#12877)

2026-10-24 15:34:21 +02:00

managers_distributed.go

feat(galleryop): limit concurrent backend operations (#12888)

2026-10-25 00:26:45 +02:00

model_cleanup_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

model_cleanup.go

fix(distributed): keep model replicas config-consistent (#11664)

2026-08-22 22:44:03 +02:00

model_identity_e2e_test.go

fix(distributed): reject wrong-model requests at the backend (#10970)

2026-07-20 13:05:47 +02:00

model_load_job_test.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

model_load_job.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

model_loading_error.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

model_router_test.go

fix(distributed): exclude pinned models from cluster eviction and idle scale-down

2026-09-11 23:08:26 +00:00

model_router.go

fix(distributed): keep model replicas config-consistent (#11664)

2026-08-22 22:44:03 +02:00

nodes_suite_test.go

Revert "chore(tests): Avoid network, sleep and more during tests" (#11601)

2026-08-19 16:39:39 +02:00

noroute_reactions_test.go

test(nodes): give the DistributedBackendManager fixtures a system state (#12877)

2026-10-24 15:34:21 +02:00

pending_op_cleanup_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

pinned_eviction_test.go

fix(distributed): exclude pinned models from cluster eviction and idle scale-down

2026-09-11 23:08:26 +00:00

probe_cache_test.go

fix(distributed): route per request across loaded replicas + cache probeHealth (#9968)

2026-05-24 08:15:27 +00:00

probe_cache.go

fix(distributed): route per request across loaded replicas + cache probeHealth (#9968)

2026-05-24 08:15:27 +00:00

reconciler_abandoned_load_test.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

reconciler_abandoned_load.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

reconciler_alias_test.go

feat(distributed): key scheduling rules by a model alias (#11771)

2026-08-29 09:16:50 +02:00

reconciler_busy_probe_test.go

fix(distributed): reaper reaps live backends, ghost model stubs, in_flight leak, sidecar staging runaway (#11142)

2026-07-27 18:36:31 +02:00

reconciler_inflight_leak_test.go

fix(distributed): exclude pinned models from cluster eviction and idle scale-down

2026-09-11 23:08:26 +00:00

reconciler_prober_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

reconciler_scaleup_backoff_test.go

fix(nodes): back off failed replica scale-ups per model

2026-11-03 09:33:41 +00:00

reconciler_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

reconciler_worker_processes_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

reconciler.go

fix(nodes): back off failed replica scale-ups per model

2026-11-03 09:33:41 +00:00

registry_capability_test.go

fix(distributed): backend discovery hid GPU-only backends behind the controller's capability (#10947)

2026-07-19 07:53:46 +00:00

registry_clustermemory_test.go

fix(ui): size model fit against the cluster and move node labels into the selector (#11765)

2026-08-28 22:57:35 +02:00

registry_test.go

feat(nodes): add fleet operations dashboard (#12046)

2026-09-15 00:57:14 +02:00

registry_vrambudget_test.go

feat(vram): per-node VRAM allocation budget (LOCALAI_VRAM_BUDGET) (#10833)

2026-07-15 09:58:45 +02:00

registry.go

feat(galleryop): limit concurrent backend operations (#12888)

2026-10-25 00:26:45 +02:00

replicapicker.go

refactor(routing): extract replica picker into pkg/clusterrouting (#10123)

2026-06-01 09:38:55 +02:00

revision_eligibility_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

revision_error_detail_test.go

fix(distributed): name both revisions in the stale error

2026-08-23 19:53:42 +00:00

router_companionstage_test.go

fix(model-artifacts): materialize longcat-video on the controller, and support companion repos (#10949)

2026-07-19 12:01:36 +02:00

router_declared_files_stage_test.go

fix(distributed): stage the files a model install declares (#12309)

2026-09-28 08:34:56 +02:00

router_dirstage_test.go

fix(model-artifacts): materialize longcat-video on the controller, and support companion repos (#10949)

2026-07-19 12:01:36 +02:00

router_eviction_alias_test.go

feat(distributed): key scheduling rules by a model alias (#11771)

2026-08-29 09:16:50 +02:00

router_eviction_selector_test.go

fix(distributed): keep eviction inside the model's node selector

2026-08-24 13:18:35 +00:00

router_hardware_internal_test.go

fix(config): only inject llama.cpp serving options on the llama.cpp path (#10822)

2026-07-14 17:46:15 +02:00

router_liveness_route_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

router_liveness.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

router_load_budget_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

router_load_job_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

router_load_timeout_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

router_missingsource_test.go

fix(model-artifacts): materialize longcat-video on the controller, and support companion repos (#10949)

2026-07-19 12:01:36 +02:00

router_nats_liveness_test.go

fix(distributed): check a node answers before scheduling onto it

2026-08-23 20:44:43 +00:00

router_optionstage_test.go

fix(distributed): stage backend companion assets to remote nodes (#10330)

2026-06-14 16:42:59 +02:00

router_reap_load_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

router_reservation_test.go

fix(distributed): reaper reaps live backends, ghost model stubs, in_flight leak, sidecar staging runaway (#11142)

2026-07-27 18:36:31 +02:00

router_revision_lifecycle_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

router_sharedmodels_test.go

feat(distributed): add LOCALAI_DISTRIBUTED_SHARED_MODELS to skip staging on shared volumes (#10556) (#10566)

2026-06-28 01:23:07 +02:00

router_sidecar_stage_test.go

fix(distributed): reaper reaps live backends, ghost model stubs, in_flight leak, sidecar staging runaway (#11142)

2026-07-27 18:36:31 +02:00

router_slot_uncertainty_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

router_staging_context_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

router_staging_deadline_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

router_test.go

fix(proxy): propagate model env and accept direct keys (#12940)

2026-10-27 10:27:45 +01:00

router_virtual_model_test.go

fix(distributed): resolve paths for virtual models (#11911)

2026-09-07 19:32:17 +02:00

router_vrambudget_test.go

feat(vram): per-node VRAM allocation budget (LOCALAI_VRAM_BUDGET) (#10833)

2026-07-15 09:58:45 +02:00

router.go

fix(nodes): back off failed replica scale-ups per model

2026-11-03 09:33:41 +00:00

scheduling_seed_test.go

feat(nodes): add pluggable routing pipeline

2026-09-11 21:47:08 +00:00

scheduling_seed.go

feat(nodes): add pluggable routing pipeline

2026-09-11 21:47:08 +00:00

staging_keys_test.go

feat: add distributed mode (#9124)

2026-03-30 00:47:27 +02:00

staging_keys.go

fix(model-artifacts): materialize longcat-video on the controller, and support companion repos (#10949)

2026-07-19 12:01:36 +02:00

staging_progress_broadcast_test.go

fix(distributed): broadcast file-staging progress across replicas (#10440)

2026-06-22 09:28:07 +02:00

staging_progress.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

unload_addressed_test.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

unloader_ping_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

unloader_stale_rows_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

unloader_test.go

fix(distributed): bound, stop and cancel model loads with leases and worker operations (#12524)

2026-10-07 16:59:56 +02:00

unloader_upgrade_test.go

refactor(distributed): put the NATS-only paths behind interfaces (#12395)

2026-10-02 23:58:41 +02:00

unloader.go

fix(proxy): propagate model env and accept direct keys (#12940)

2026-10-27 10:27:45 +01:00

worker_readiness_datapath_test.go

fix(distributed): survive a slow control-plane database (#11837)

2026-09-02 12:37:12 +02:00

worker_readiness_test.go

fix(worker): give the worker a real health endpoint and a mode-aware HEALTHCHECK (#10999)

2026-07-20 23:07:27 +02:00

worker_readiness.go

fix(distributed): survive a slow control-plane database (#11837)

2026-09-02 12:37:12 +02:00