mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 18:09:05 -04:00
16033d562a
A gallery model install hung for 94 minutes with zero bytes transferred, no
error, no retry and no abort, leaving a partial tree frozen at 18G. The last
log line was the download starting, then silence:
14:06:19 INFO Downloading url=".../LongCat-Video-Avatar-1.5/resolve/<rev>/base_model/diffusion_pytorch_model-000..."
The retry machinery from #10985 was working (two retries fired at 14:05:01 and
14:06:14); the third attempt simply never returned. The install never
completed, the model config was never written, and nothing surfaced the
failure.
The stall watchdog added earlier wraps the response *body*, so it only starts
guarding once downloadClient.Do() has returned. The transport had no
ResponseHeaderTimeout, so a peer that completes the dial and TLS handshake,
reads the request, and then never sends a status line parks Do() for the
process lifetime. IdleConnTimeout governs pooled idle connections, not an
in-flight request. Both the body request and the HEAD that probes for Range
support were unguarded.
Bound the header wait at the transport, not the client: a client-level Timeout
would also bound the body and truncate multi-tens-of-GB downloads. The knob is
opt-in (WithResponseHeaderTimeout) rather than a default in HardenedTransport,
because a streaming endpoint may legitimately withhold headers until it has
something to say, and capping that would break the streaming clients that share
this constructor.
Also fix a classification trap this exposed: net/http reports a
ResponseHeaderTimeout as an error satisfying errors.Is(err,
context.DeadlineExceeded), which IsRetryable read as "the caller gave up" and
refused to retry. An explicit transient marking now outranks the cancellation
sentinels; a caller who genuinely gave up is still caught by the ctx.Err()
check. The resume probe's error is likewise marked transient, so a momentarily
wedged origin no longer turns a resumable download into a hard install failure.
Third defect found in this download path, after #10985 (read vs write errors
conflated) and #11026 (hash verification emitted no progress and an expired
deadline returned success).
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
43 lines
1.3 KiB
Go
43 lines
1.3 KiB
Go
package downloader
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
|
|
. "github.com/onsi/ginkgo/v2"
|
|
. "github.com/onsi/gomega"
|
|
)
|
|
|
|
var _ = Describe("IsRetryable classification", func() {
|
|
// net/http surfaces a ResponseHeaderTimeout as an error satisfying
|
|
// errors.Is(err, context.DeadlineExceeded). That is our own transport guard
|
|
// aborting a wedged peer, not the caller giving up, so an explicit
|
|
// transient marking has to outrank the deadline sentinel.
|
|
wedged := func() error {
|
|
return fmt.Errorf("net/http: timeout awaiting response headers: %w", context.DeadlineExceeded)
|
|
}
|
|
|
|
It("retries a transient failure that reports itself as a deadline", func() {
|
|
Expect(IsRetryable(context.Background(), asTransient(wedged()))).To(BeTrue())
|
|
})
|
|
|
|
It("still refuses to retry once the caller's context is done", func() {
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
cancel()
|
|
Expect(IsRetryable(ctx, asTransient(wedged()))).To(BeFalse())
|
|
})
|
|
|
|
It("still refuses to retry a deliberate user abort", func() {
|
|
Expect(IsRetryable(context.Background(), asTransient(ErrUserCancelled))).To(BeFalse())
|
|
})
|
|
|
|
It("does not retry an unmarked failure", func() {
|
|
Expect(IsRetryable(context.Background(), errors.New("checksum mismatch"))).To(BeFalse())
|
|
})
|
|
|
|
It("leaves a nil error unretryable", func() {
|
|
Expect(IsRetryable(context.Background(), nil)).To(BeFalse())
|
|
})
|
|
})
|