mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 18:09:05 -04:00
16033d562a
A gallery model install hung for 94 minutes with zero bytes transferred, no
error, no retry and no abort, leaving a partial tree frozen at 18G. The last
log line was the download starting, then silence:
14:06:19 INFO Downloading url=".../LongCat-Video-Avatar-1.5/resolve/<rev>/base_model/diffusion_pytorch_model-000..."
The retry machinery from #10985 was working (two retries fired at 14:05:01 and
14:06:14); the third attempt simply never returned. The install never
completed, the model config was never written, and nothing surfaced the
failure.
The stall watchdog added earlier wraps the response *body*, so it only starts
guarding once downloadClient.Do() has returned. The transport had no
ResponseHeaderTimeout, so a peer that completes the dial and TLS handshake,
reads the request, and then never sends a status line parks Do() for the
process lifetime. IdleConnTimeout governs pooled idle connections, not an
in-flight request. Both the body request and the HEAD that probes for Range
support were unguarded.
Bound the header wait at the transport, not the client: a client-level Timeout
would also bound the body and truncate multi-tens-of-GB downloads. The knob is
opt-in (WithResponseHeaderTimeout) rather than a default in HardenedTransport,
because a streaming endpoint may legitimately withhold headers until it has
something to say, and capping that would break the streaming clients that share
this constructor.
Also fix a classification trap this exposed: net/http reports a
ResponseHeaderTimeout as an error satisfying errors.Is(err,
context.DeadlineExceeded), which IsRetryable read as "the caller gave up" and
refused to retry. An explicit transient marking now outranks the cancellation
sentinels; a caller who genuinely gave up is still caught by the ctx.Err()
check. The resume probe's error is likewise marked transient, so a momentarily
wedged origin no longer turns a resumable download into a hard install failure.
Third defect found in this download path, after #10985 (read vs write errors
conflated) and #11026 (hash verification emitted no progress and an expired
deadline returned success).
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
94 lines
3.2 KiB
Go
94 lines
3.2 KiB
Go
package downloader
|
|
|
|
import (
|
|
"fmt"
|
|
"io"
|
|
"sync"
|
|
"time"
|
|
)
|
|
|
|
// DownloadStallTimeout bounds how long an in-flight download may receive no
|
|
// data before it is aborted. A silently-dropped TCP connection (no FIN/RST)
|
|
// would otherwise block the body read forever, freezing an install at N bytes
|
|
// until an external reaper kills it. Overridable (tests set it small); a value
|
|
// <= 0 disables the guard.
|
|
var DownloadStallTimeout = 60 * time.Second
|
|
|
|
// DownloadResponseHeaderTimeout bounds how long a download request may wait for
|
|
// the server's response headers. DownloadStallTimeout cannot cover this window:
|
|
// it wraps the response body, which does not exist until the client has a
|
|
// response. A peer that accepts the connection, reads the request and then
|
|
// never answers leaves http.Client.Do parked forever, freezing an install at
|
|
// zero bytes for the process lifetime.
|
|
//
|
|
// The value has to tolerate an origin legitimately slow to *start* answering (a
|
|
// CDN under load, cold object storage, a redirect chain) while still bounding a
|
|
// wedge. Two minutes is far above any plausible legitimate time-to-first-byte
|
|
// and twice the stall window, so a genuine wedge fails in bounded time and the
|
|
// retry machinery resumes from the .partial. Deliberately NOT a client-level
|
|
// Timeout: that would also bound the body and truncate multi-tens-of-GB
|
|
// downloads. Overridable (tests set it small); a value <= 0 disables the guard.
|
|
var DownloadResponseHeaderTimeout = 120 * time.Second
|
|
|
|
// idleTimeoutReader wraps a streaming ReadCloser and aborts reads that make no
|
|
// progress within timeout. A standard io.Copy blocks indefinitely on a Read
|
|
// against a dead-but-unclosed socket; nothing in the copy loop can interrupt a
|
|
// blocked syscall. The watchdog timer closes the underlying reader on expiry,
|
|
// which unblocks the in-flight Read with an error. Each read that returns data
|
|
// resets the idle clock, so a slow-but-steady transfer never trips the guard.
|
|
type idleTimeoutReader struct {
|
|
rc io.ReadCloser
|
|
timeout time.Duration
|
|
|
|
mu sync.Mutex
|
|
timer *time.Timer
|
|
fired bool
|
|
done bool
|
|
}
|
|
|
|
func newIdleTimeoutReader(rc io.ReadCloser, timeout time.Duration) *idleTimeoutReader {
|
|
r := &idleTimeoutReader{rc: rc, timeout: timeout}
|
|
r.timer = time.AfterFunc(timeout, r.onStall)
|
|
return r
|
|
}
|
|
|
|
// onStall fires when no data has arrived within the timeout. Closing the
|
|
// underlying reader is what unblocks a Read parked in the kernel.
|
|
func (r *idleTimeoutReader) onStall() {
|
|
r.mu.Lock()
|
|
if r.done {
|
|
r.mu.Unlock()
|
|
return
|
|
}
|
|
r.fired = true
|
|
r.mu.Unlock()
|
|
_ = r.rc.Close()
|
|
}
|
|
|
|
func (r *idleTimeoutReader) Read(p []byte) (int, error) {
|
|
n, err := r.rc.Read(p)
|
|
if n > 0 {
|
|
r.timer.Reset(r.timeout)
|
|
}
|
|
if err != nil {
|
|
r.mu.Lock()
|
|
fired := r.fired
|
|
r.mu.Unlock()
|
|
if fired {
|
|
// Translate the "use of closed connection" the watchdog induced
|
|
// into an actionable stall error. This is not context.Canceled,
|
|
// so the caller keeps the .partial file for a later resume.
|
|
return n, fmt.Errorf("download stalled: no data received for %s", r.timeout)
|
|
}
|
|
}
|
|
return n, err
|
|
}
|
|
|
|
func (r *idleTimeoutReader) Close() error {
|
|
r.mu.Lock()
|
|
r.done = true
|
|
r.mu.Unlock()
|
|
r.timer.Stop()
|
|
return r.rc.Close()
|
|
}
|