Compare commits

...

12 Commits

Author SHA1 Message Date
Flaminel
bd28c7ab05 Fix missing notifications for new strike types (#112) 2025-04-08 22:20:51 +03:00
Flaminel
720279df65 Update README.md 2025-04-08 18:14:01 +03:00
Flaminel
2d4ec648b8 Update README.md 2025-04-06 18:10:46 +03:00
Flaminel
704fdaca4a Add cleanup for slow downloads (#110) 2025-04-06 13:28:05 +03:00
Flaminel
b134136e51 Update README.md 2025-03-29 01:11:41 +02:00
Flaminel
5ca717d7e0 Update README.md 2025-03-27 19:53:57 +02:00
Flaminel
7068ee5e5a Update README.md 2025-03-26 13:30:55 +02:00
Flaminel
9f770473e5 Remove Transmission downloads cache (#105) 2025-03-26 00:26:10 +02:00
Flaminel
5fe0f5750a Fix qBit queued items being processed (#102) 2025-03-21 23:06:31 +02:00
Flaminel
b8ce225ccc Fix Deluge service crashing when download is not found (#97) 2025-03-20 00:09:58 +02:00
Flaminel
f21f7388b7 Add download client customizable url base (#43) 2025-03-20 00:09:24 +02:00
Flaminel
a1354f231a Add base path support for arrs (#96) 2025-03-20 00:08:51 +02:00
49 changed files with 1039 additions and 268 deletions

View File

@@ -18,7 +18,7 @@ body:
required: true
- label: Ensured I am using the latest version.
required: true
- label: Enabled debug logging.
- label: Enabled verbose logging.
required: true
- type: textarea
id: what-happened

View File

@@ -18,7 +18,7 @@ body:
required: true
- label: Ensured I am using the latest version.
required: true
- label: Enabled debug logging.
- label: Enabled verbose logging.
required: true
- type: textarea
id: description

View File

@@ -12,6 +12,7 @@ cleanuperr was created primarily to address malicious files, such as `*.lnk` or
> **Features:**
> - Strike system to mark stalled or downloads stuck in metadata downloading.
> - Remove and block downloads that reached a maximum number of strikes.
> - Remove and block downloads that have a low download speed or high estimated completion time.
> - Remove downloads blocked by qBittorrent or by cleanuperr's **content blocker**.
> - Trigger a search for downloads removed from the *arrs.
> - Clean up downloads that have been seeding for a certain amount of time.
@@ -91,11 +92,11 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
#### 2. **Queue cleaner** will:
- Run every 5 minutes (or configured cron, or right after `content blocker`).
- Process all items in the *arr queue.
- Check each queue item if it is **stalled (download speed is 0)**, **stuck in metadata downloading** or **failed to be imported**.
- Check each queue item if it is **stalled (download speed is 0)**, **stuck in metadata downloading**, **failed to be imported** or **slow**.
- If it is, the item receives a **strike** and will continue to accumulate strikes every time it meets any of these conditions.
- Check each queue item if it meets one of the following condition in the download client:
- **Marked as completed, but 0 bytes have been downloaded** (due to files being blocked by qBittorrent or the **content blocker**).
- All associated files of are marked as **unwanted/skipped**.
- All associated files are marked as **unwanted/skipped/do not download**.
- If the item **DOES NOT** match the above criteria, it will be skipped.
- If the item **DOES** match the criteria or has received the **maximum number of strikes**:
- It will be removed from the *arr's queue and blocked.
@@ -168,39 +169,62 @@ services:
- ./cleanuperr/logs:/var/logs
- ./cleanuperr/ignored.txt:/ignored.txt
environment:
# general settings
- TZ=America/New_York
- DRY_RUN=false
- HTTP_MAX_RETRIES=0
- HTTP_TIMEOUT=100
# logging
- LOGGING__LOGLEVEL=Information
- LOGGING__FILE__ENABLED=false
- LOGGING__FILE__PATH=/var/logs/
- LOGGING__ENHANCED=true
# job triggers
- TRIGGERS__QUEUECLEANER=0 0/5 * * * ?
- TRIGGERS__CONTENTBLOCKER=0 0/5 * * * ?
- TRIGGERS__DOWNLOADCLEANER=0 0 * * * ?
# queue cleaner
- QUEUECLEANER__ENABLED=true
- QUEUECLEANER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- QUEUECLEANER__RUNSEQUENTIALLY=true
# failed imports
- QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE=false
- QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE=false
# - QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__0=title mismatch
# - QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1=manual import required
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__0=title mismatch
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1=manual import required
# stalled downloads
- QUEUECLEANER__STALLED_MAX_STRIKES=5
- QUEUECLEANER__STALLED_RESET_STRIKES_ON_PROGRESS=false
- QUEUECLEANER__STALLED_IGNORE_PRIVATE=false
- QUEUECLEANER__STALLED_DELETE_PRIVATE=false
# slow downloads
- QUEUECLEANER__SLOW_MAX_STRIKES=5
- QUEUECLEANER__SLOW_RESET_STRIKES_ON_PROGRESS=true
- QUEUECLEANER__SLOW_IGNORE_PRIVATE=false
- QUEUECLEANER__SLOW_DELETE_PRIVATE=false
- QUEUECLEANER__SLOW_MIN_SPEED=1MB
- QUEUECLEANER__SLOW_MAX_TIME=20
- QUEUECLEANER__SLOW_IGNORE_ABOVE_SIZE=60GB
# content blocker
- CONTENTBLOCKER__ENABLED=true
- CONTENTBLOCKER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- CONTENTBLOCKER__IGNORE_PRIVATE=false
- CONTENTBLOCKER__DELETE_PRIVATE=false
# download cleaner
- DOWNLOADCLEANER__ENABLED=true
- DOWNLOADCLEANER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- DOWNLOADCLEANER__DELETE_PRIVATE=false
# categories to seed until max ratio or min seed time has been reached
- DOWNLOADCLEANER__CATEGORIES__0__NAME=tv-sonarr
- DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO=-1
- DOWNLOADCLEANER__CATEGORIES__0__MIN_SEED_TIME=0
@@ -212,17 +236,22 @@ services:
- DOWNLOAD_CLIENT=none
# OR
# - DOWNLOAD_CLIENT=disabled
# OR
# - DOWNLOAD_CLIENT=qBittorrent
# - QBITTORRENT__URL=http://localhost:8080
# - QBITTORRENT__URL_BASE=myCustomPath
# - QBITTORRENT__USERNAME=user
# - QBITTORRENT__PASSWORD=pass
# OR
# - DOWNLOAD_CLIENT=deluge
# - DELUGE__URL_BASE=myCustomPath
# - DELUGE__URL=http://localhost:8112
# - DELUGE__PASSWORD=testing
# OR
# - DOWNLOAD_CLIENT=transmission
# - TRANSMISSION__URL=http://localhost:9091
# - TRANSMISSION__URL_BASE=myCustomPath
# - TRANSMISSION__USERNAME=test
# - TRANSMISSION__PASSWORD=testing
@@ -253,6 +282,7 @@ services:
- NOTIFIARR__ON_IMPORT_FAILED_STRIKE=true
- NOTIFIARR__ON_STALLED_STRIKE=true
- NOTIFIARR__ON_SLOW_STRIKE=true
- NOTIFIARR__ON_QUEUE_ITEM_DELETED=true
- NOTIFIARR__ON_DOWNLOAD_CLEANED=true
- NOTIFIARR__API_KEY=notifiarr_secret
@@ -268,7 +298,21 @@ services:
> [!TIP]
> ### Run as a Windows Service
> Check out this stackoverflow answer on how to do it: https://stackoverflow.com/a/15719678
> 1. Download latest nssm build from `https://nssm.cc/builds`.
> 2. Unzip `nssm.exe` in `C:\example\directory`.
> 3. Open a terminal with Administrator rights and execute these commands:
> ```
> nssm.exe install Cleanuperr "C:\example\directory\cleanuperr.exe"
> nssm.exe set Cleanuperr AppDirectory "C:\example\directory\"
> nssm.exe set Cleanuperr AppStdout "C:\example\directory\cleanuperr.log"
> nssm.exe set Cleanuperr AppStderr "C:\example\directory\cleanuperr.crash.log"
> nssm.exe set Cleanuperr AppRotateFiles 1
> nssm.exe set Cleanuperr AppRotateOnline 1
> nssm.exe set Cleanuperr AppRotateBytes 10485760
> nssm.exe set Cleanuperr AppRotateFiles 10
> nssm.exe set Cleanuperr Start SERVICE_AUTO_START
> nssm.exe start Cleanuperr
> ```
### <img src="https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/linux.svg" height="20" style="vertical-align: middle;"> <span style="vertical-align: middle;">Linux</span>
@@ -296,7 +340,7 @@ services:
> [!IMPORTANT]
> Some people have experienced problems when trying to execute cleanuperr on MacOS because the system actively blocked the file for not being signed.
> As per [this](), you may need to also execute this command:
> As per [this comment](https://stackoverflow.com/a/77907937), you may need to also execute this command:
> ```
> codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime /example/directory/cleanuperr
> ```

View File

@@ -14,7 +14,7 @@ deployment:
value: "false"
- name: LOGGING__LOGLEVEL
value: Debug
value: Verbose
- name: LOGGING__FILE__ENABLED
value: "true"
- name: LOGGING__FILE__PATH

View File

@@ -1,4 +1,5 @@
using Common.Exceptions;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadClient;
@@ -8,6 +9,9 @@ public sealed record DelugeConfig : IConfig
public Uri? Url { get; init; }
[ConfigurationKeyName("URL_BASE")]
public string UrlBase { get; init; } = string.Empty;
public string? Password { get; init; }
public void Validate()

View File

@@ -1,4 +1,5 @@
using Common.Exceptions;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadClient;
@@ -8,6 +9,9 @@ public sealed class QBitConfig : IConfig
public Uri? Url { get; init; }
[ConfigurationKeyName("URL_BASE")]
public string UrlBase { get; init; } = string.Empty;
public string? Username { get; init; }
public string? Password { get; init; }

View File

@@ -1,4 +1,5 @@
using Common.Exceptions;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadClient;
@@ -8,6 +9,9 @@ public record TransmissionConfig : IConfig
public Uri? Url { get; init; }
[ConfigurationKeyName("URL_BASE")]
public string UrlBase { get; init; } = "transmission";
public string? Username { get; init; }
public string? Password { get; init; }

View File

@@ -10,13 +10,16 @@ public abstract record NotificationConfig
[ConfigurationKeyName("ON_STALLED_STRIKE")]
public bool OnStalledStrike { get; init; }
[ConfigurationKeyName("ON_SLOW_STRIKE")]
public bool OnSlowStrike { get; init; }
[ConfigurationKeyName("ON_QUEUE_ITEM_DELETED")]
public bool OnQueueItemDeleted { get; init; }
[ConfigurationKeyName("ON_DOWNLOAD_CLEANED")]
public bool OnDownloadCleaned { get; init; }
public bool IsEnabled => OnImportFailedStrike || OnStalledStrike || OnQueueItemDeleted || OnDownloadCleaned;
public bool IsEnabled => OnImportFailedStrike || OnStalledStrike || OnSlowStrike || OnQueueItemDeleted || OnDownloadCleaned;
public abstract bool IsValid();
}

View File

@@ -1,4 +1,5 @@
using Common.Exceptions;
using Common.CustomDataTypes;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.QueueCleaner;
@@ -37,17 +38,74 @@ public sealed record QueueCleanerConfig : IJobConfig, IIgnoredDownloadsConfig
[ConfigurationKeyName("STALLED_DELETE_PRIVATE")]
public bool StalledDeletePrivate { get; init; }
[ConfigurationKeyName("SLOW_MAX_STRIKES")]
public ushort SlowMaxStrikes { get; init; }
[ConfigurationKeyName("SLOW_RESET_STRIKES_ON_PROGRESS")]
public bool SlowResetStrikesOnProgress { get; init; }
[ConfigurationKeyName("SLOW_IGNORE_PRIVATE")]
public bool SlowIgnorePrivate { get; init; }
[ConfigurationKeyName("SLOW_DELETE_PRIVATE")]
public bool SlowDeletePrivate { get; init; }
[ConfigurationKeyName("SLOW_MIN_SPEED")]
public string SlowMinSpeed { get; init; } = string.Empty;
public ByteSize SlowMinSpeedByteSize => string.IsNullOrEmpty(SlowMinSpeed) ? new ByteSize(0) : ByteSize.Parse(SlowMinSpeed);
[ConfigurationKeyName("SLOW_MAX_TIME")]
public double SlowMaxTime { get; init; }
[ConfigurationKeyName("SLOW_IGNORE_ABOVE_SIZE")]
public string SlowIgnoreAboveSize { get; init; } = string.Empty;
public ByteSize? SlowIgnoreAboveSizeByteSize => string.IsNullOrEmpty(SlowIgnoreAboveSize) ? null : ByteSize.Parse(SlowIgnoreAboveSize);
public void Validate()
{
if (ImportFailedMaxStrikes is > 0 and < 3)
{
throw new ValidationException("the minimum value for IMPORT_FAILED_MAX_STRIKES must be 3");
throw new ValidationException($"the minimum value for {SectionName}__IMPORT_FAILED_MAX_STRIKES must be 3");
}
if (StalledMaxStrikes is > 0 and < 3)
{
throw new ValidationException("the minimum value for STALLED_MAX_STRIKES must be 3");
throw new ValidationException($"the minimum value for {SectionName}__STALLED_MAX_STRIKES must be 3");
}
if (SlowMaxStrikes is > 0 and < 3)
{
throw new ValidationException($"the minimum value for {SectionName}__SLOW_MAX_STRIKES must be 3");
}
if (SlowMaxStrikes > 0)
{
bool isSlowSpeedSet = !string.IsNullOrEmpty(SlowMinSpeed);
if (isSlowSpeedSet && ByteSize.TryParse(SlowMinSpeed, out _) is false)
{
throw new ValidationException($"invalid value for {SectionName}__SLOW_MIN_SPEED");
}
if (SlowMaxTime < 0)
{
throw new ValidationException($"invalid value for {SectionName}__SLOW_MAX_TIME");
}
if (!isSlowSpeedSet && SlowMaxTime is 0)
{
throw new ValidationException($"either {SectionName}__SLOW_MIN_SPEED or {SectionName}__SLOW_MAX_STRIKES must be set");
}
bool isSlowIgnoreAboveSizeSet = !string.IsNullOrEmpty(SlowIgnoreAboveSize);
if (isSlowIgnoreAboveSizeSet && ByteSize.TryParse(SlowIgnoreAboveSize, out _) is false)
{
throw new ValidationException($"invalid value for {SectionName}__SLOW_IGNORE_ABOVE_SIZE");
}
}
}
}

View File

@@ -0,0 +1,115 @@
using System.Globalization;
namespace Common.CustomDataTypes;
public readonly struct ByteSize : IComparable<ByteSize>, IEquatable<ByteSize>
{
public long Bytes { get; }
private const long BytesPerKB = 1024;
private const long BytesPerMB = 1024 * 1024;
private const long BytesPerGB = 1024 * 1024 * 1024;
public ByteSize(long bytes)
{
if (bytes < 0)
{
throw new ArgumentOutOfRangeException(nameof(bytes), "bytes can not be negative");
}
Bytes = bytes;
}
public static ByteSize FromKilobytes(double kilobytes) => new((long)(kilobytes * BytesPerKB));
public static ByteSize FromMegabytes(double megabytes) => new((long)(megabytes * BytesPerMB));
public static ByteSize FromGigabytes(double gigabytes) => new((long)(gigabytes * BytesPerGB));
public static ByteSize Parse(string input)
{
if (string.IsNullOrWhiteSpace(input))
{
throw new ArgumentNullException(nameof(input));
}
input = input.Trim().ToUpperInvariant();
double value;
if (input.EndsWith("KB", StringComparison.InvariantCultureIgnoreCase))
{
value = double.Parse(input[..^2], CultureInfo.InvariantCulture);
return FromKilobytes(value);
}
if (input.EndsWith("MB", StringComparison.InvariantCultureIgnoreCase))
{
value = double.Parse(input[..^2], CultureInfo.InvariantCulture);
return FromMegabytes(value);
}
if (input.EndsWith("GB", StringComparison.InvariantCultureIgnoreCase))
{
value = double.Parse(input[..^2], CultureInfo.InvariantCulture);
return FromGigabytes(value);
}
throw new FormatException("invalid size format | only KB, MB and GB are supported");
}
public static bool TryParse(string? input, out ByteSize? result)
{
result = default;
if (string.IsNullOrWhiteSpace(input))
{
return false;
}
input = input.Trim().ToUpperInvariant();
if (input.EndsWith("KB", StringComparison.InvariantCultureIgnoreCase) &&
double.TryParse(input[..^2], NumberStyles.Float, CultureInfo.InvariantCulture, out double kb))
{
result = FromKilobytes(kb);
return true;
}
if (input.EndsWith("MB", StringComparison.InvariantCultureIgnoreCase) &&
double.TryParse(input[..^2], NumberStyles.Float, CultureInfo.InvariantCulture, out double mb))
{
result = FromMegabytes(mb);
return true;
}
if (input.EndsWith("GB", StringComparison.InvariantCultureIgnoreCase) &&
double.TryParse(input[..^2], NumberStyles.Float, CultureInfo.InvariantCulture, out double gb))
{
result = FromGigabytes(gb);
return true;
}
return false;
}
public override string ToString() =>
Bytes switch
{
>= BytesPerGB => $"{Bytes / (double)BytesPerGB:0.##} GB",
>= BytesPerMB => $"{Bytes / (double)BytesPerMB:0.##} MB",
_ => $"{Bytes / (double)BytesPerKB:0.##} KB"
};
public int CompareTo(ByteSize other) => Bytes.CompareTo(other.Bytes);
public bool Equals(ByteSize other) => Bytes == other.Bytes;
public override bool Equals(object? obj) => obj is ByteSize other && Equals(other);
public override int GetHashCode() => Bytes.GetHashCode();
public static bool operator ==(ByteSize left, ByteSize right) => left.Equals(right);
public static bool operator !=(ByteSize left, ByteSize right) => !(left == right);
public static bool operator <(ByteSize left, ByteSize right) => left.Bytes < right.Bytes;
public static bool operator >(ByteSize left, ByteSize right) => left.Bytes > right.Bytes;
public static bool operator <=(ByteSize left, ByteSize right) => left.Bytes <= right.Bytes;
public static bool operator >=(ByteSize left, ByteSize right) => left.Bytes >= right.Bytes;
public static ByteSize operator +(ByteSize left, ByteSize right) => new(left.Bytes + right.Bytes);
public static ByteSize operator -(ByteSize left, ByteSize right) => new(Math.Max(left.Bytes - right.Bytes, 0));
}

View File

@@ -0,0 +1,66 @@
using System.Text;
namespace Common.CustomDataTypes;
public readonly struct SmartTimeSpan : IComparable<SmartTimeSpan>, IEquatable<SmartTimeSpan>
{
public TimeSpan Time { get; }
public SmartTimeSpan(TimeSpan time)
{
Time = time;
}
public override string ToString()
{
if (Time == TimeSpan.Zero)
{
return "0 seconds";
}
StringBuilder sb = new();
if (Time.Days > 0)
{
sb.Append($"{Time.Days} day{(Time.Days > 1 ? "s" : "")} ");
}
if (Time.Hours > 0)
{
sb.Append($"{Time.Hours} hour{(Time.Hours > 1 ? "s" : "")} ");
}
if (Time.Minutes > 0)
{
sb.Append($"{Time.Minutes} minute{(Time.Minutes > 1 ? "s" : "")} ");
}
if (Time.Seconds > 0)
{
sb.Append($"{Time.Seconds} second{(Time.Seconds > 1 ? "s" : "")}");
}
return sb.ToString().TrimEnd();
}
public static SmartTimeSpan FromMinutes(double minutes) => new(TimeSpan.FromMinutes(minutes));
public static SmartTimeSpan FromSeconds(double seconds) => new(TimeSpan.FromSeconds(seconds));
public static SmartTimeSpan FromHours(double hours) => new(TimeSpan.FromHours(hours));
public static SmartTimeSpan FromDays(double days) => new(TimeSpan.FromDays(days));
public int CompareTo(SmartTimeSpan other) => Time.CompareTo(other.Time);
public bool Equals(SmartTimeSpan other) => Time.Equals(other.Time);
public override bool Equals(object? obj) => obj is SmartTimeSpan other && Equals(other);
public override int GetHashCode() => Time.GetHashCode();
public static bool operator ==(SmartTimeSpan left, SmartTimeSpan right) => left.Equals(right);
public static bool operator !=(SmartTimeSpan left, SmartTimeSpan right) => !left.Equals(right);
public static bool operator <(SmartTimeSpan left, SmartTimeSpan right) => left.Time < right.Time;
public static bool operator >(SmartTimeSpan left, SmartTimeSpan right) => left.Time > right.Time;
public static bool operator <=(SmartTimeSpan left, SmartTimeSpan right) => left.Time <= right.Time;
public static bool operator >=(SmartTimeSpan left, SmartTimeSpan right) => left.Time >= right.Time;
public static SmartTimeSpan operator +(SmartTimeSpan left, SmartTimeSpan right) => new(left.Time + right.Time);
public static SmartTimeSpan operator -(SmartTimeSpan left, SmartTimeSpan right) => new(left.Time - right.Time);
}

View File

@@ -2,7 +2,13 @@
public enum DeleteReason
{
None,
Stalled,
ImportFailed,
AllFilesBlocked
DownloadingMetadata,
SlowSpeed,
SlowTime,
AllFilesSkipped,
AllFilesSkippedByQBit,
AllFilesBlocked,
}

View File

@@ -3,5 +3,8 @@
public enum StrikeType
{
Stalled,
ImportFailed
DownloadingMetadata,
ImportFailed,
SlowSpeed,
SlowTime,
}

View File

@@ -1,6 +1,6 @@
namespace Domain.Models.Cache;
public sealed record CacheItem
public sealed record StalledCacheItem
{
/// <summary>
/// The amount of bytes that have been downloaded.

View File

@@ -2,7 +2,7 @@
namespace Domain.Models.Deluge.Response;
public sealed record TorrentStatus
public sealed record DownloadStatus
{
public string? Hash { get; init; }
@@ -12,8 +12,14 @@ public sealed record TorrentStatus
public ulong Eta { get; init; }
[JsonProperty("download_payload_rate")]
public long DownloadSpeed { get; init; }
public bool Private { get; init; }
[JsonProperty("total_size")]
public long Size { get; init; }
[JsonProperty("total_done")]
public long TotalDone { get; init; }

View File

@@ -25,6 +25,7 @@ public static class MainDI
{
config.AddConsumer<NotificationConsumer<FailedImportStrikeNotification>>();
config.AddConsumer<NotificationConsumer<StalledStrikeNotification>>();
config.AddConsumer<NotificationConsumer<SlowStrikeNotification>>();
config.AddConsumer<NotificationConsumer<QueueItemDeletedNotification>>();
config.AddConsumer<NotificationConsumer<DownloadCleanedNotification>>();
@@ -34,6 +35,7 @@ public static class MainDI
{
e.ConfigureConsumer<NotificationConsumer<FailedImportStrikeNotification>>(context);
e.ConfigureConsumer<NotificationConsumer<StalledStrikeNotification>>(context);
e.ConfigureConsumer<NotificationConsumer<SlowStrikeNotification>>(context);
e.ConfigureConsumer<NotificationConsumer<QueueItemDeletedNotification>>(context);
e.ConfigureConsumer<NotificationConsumer<DownloadCleanedNotification>>(context);
e.ConcurrentMessageLimit = 1;

View File

@@ -3,7 +3,7 @@
"HTTP_MAX_RETRIES": 0,
"HTTP_TIMEOUT": 10,
"Logging": {
"LogLevel": "Debug",
"LogLevel": "Verbose",
"Enhanced": true,
"File": {
"Enabled": false,
@@ -34,7 +34,14 @@
"STALLED_MAX_STRIKES": 5,
"STALLED_RESET_STRIKES_ON_PROGRESS": true,
"STALLED_IGNORE_PRIVATE": true,
"STALLED_DELETE_PRIVATE": false
"STALLED_DELETE_PRIVATE": false,
"SLOW_MAX_STRIKES": 5,
"SLOW_RESET_STRIKES_ON_PROGRESS": true,
"SLOW_IGNORE_PRIVATE": false,
"SLOW_DELETE_PRIVATE": false,
"SLOW_MIN_SPEED": "1MB",
"SLOW_MAX_TIME": 20,
"SLOW_IGNORE_ABOVE_SIZE": "4GB"
},
"DownloadCleaner": {
"Enabled": false,
@@ -52,15 +59,18 @@
"DOWNLOAD_CLIENT": "qbittorrent",
"qBittorrent": {
"Url": "http://localhost:8080",
"URL_BASE": "",
"Username": "test",
"Password": "testing"
},
"Deluge": {
"Url": "http://localhost:8112",
"URL_BASE": "",
"Password": "testing"
},
"Transmission": {
"Url": "http://localhost:9091",
"URL_BASE": "transmission",
"Username": "test",
"Password": "testing"
},
@@ -107,6 +117,7 @@
"Notifiarr": {
"ON_IMPORT_FAILED_STRIKE": true,
"ON_STALLED_STRIKE": true,
"ON_SLOW_STRIKE": true,
"ON_QUEUE_ITEM_DELETED": true,
"ON_DOWNLOAD_CLEANED": true,
"API_KEY": "",

View File

@@ -42,15 +42,18 @@
"DOWNLOAD_CLIENT": "none",
"qBittorrent": {
"Url": "http://localhost:8080",
"URL_BASE": "",
"Username": "",
"Password": ""
},
"Deluge": {
"Url": "http://localhost:8112",
"URL_BASE": "",
"Password": "testing"
},
"Transmission": {
"Url": "http://localhost:9091",
"URL_BASE": "transmission",
"Username": "test",
"Password": "testing"
},
@@ -97,6 +100,7 @@
"Notifiarr": {
"ON_IMPORT_FAILED_STRIKE": false,
"ON_STALLED_STRIKE": false,
"ON_SLOW_STRIKE": false,
"ON_QUEUE_ITEM_DELETED": false,
"ON_DOWNLOAD_CLEANED": false,
"API_KEY": "",

View File

@@ -39,7 +39,7 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
});
// Act
sut.ResetStrikesOnProgress("test-hash", 100);
sut.ResetStalledStrikesOnProgress("test-hash", 100);
// Assert
_fixture.Cache.ReceivedCalls().ShouldBeEmpty();
@@ -50,19 +50,19 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
{
// Arrange
const string hash = "test-hash";
CacheItem cacheItem = new CacheItem { Downloaded = 100 };
StalledCacheItem stalledCacheItem = new StalledCacheItem { Downloaded = 100 };
_fixture.Cache.TryGetValue(Arg.Any<object>(), out Arg.Any<object?>())
.Returns(x =>
{
x[1] = cacheItem;
x[1] = stalledCacheItem;
return true;
});
TestDownloadService sut = _fixture.CreateSut();
// Act
sut.ResetStrikesOnProgress(hash, 200);
sut.ResetStalledStrikesOnProgress(hash, 200);
// Assert
_fixture.Cache.Received(1).Remove(CacheKeys.Strike(StrikeType.Stalled, hash));
@@ -73,20 +73,20 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
{
// Arrange
const string hash = "test-hash";
CacheItem cacheItem = new CacheItem { Downloaded = 200 };
StalledCacheItem stalledCacheItem = new StalledCacheItem { Downloaded = 200 };
_fixture.Cache
.TryGetValue(Arg.Any<object>(), out Arg.Any<object?>())
.Returns(x =>
{
x[1] = cacheItem;
x[1] = stalledCacheItem;
return true;
});
TestDownloadService sut = _fixture.CreateSut();
// Act
sut.ResetStrikesOnProgress(hash, 100);
sut.ResetStalledStrikesOnProgress(hash, 100);
// Assert
_fixture.Cache.DidNotReceive().Remove(Arg.Any<object>());
@@ -98,27 +98,6 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
public StrikeAndCheckLimitTests(DownloadServiceFixture fixture) : base(fixture)
{
}
[Fact]
public async Task ShouldDelegateCallToStriker()
{
// Arrange
const string hash = "test-hash";
const string itemName = "test-item";
_fixture.Striker.StrikeAndCheckLimit(hash, itemName, 3, StrikeType.Stalled)
.Returns(true);
TestDownloadService sut = _fixture.CreateSut();
// Act
bool result = await sut.StrikeAndCheckLimit(hash, itemName);
// Assert
result.ShouldBeTrue();
await _fixture.Striker
.Received(1)
.StrikeAndCheckLimit(hash, itemName, 3, StrikeType.Stalled);
}
}
public class ShouldCleanDownloadTests : DownloadServiceTests

View File

@@ -3,6 +3,7 @@ using System.Text.RegularExpressions;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Domain.Enums;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.DownloadClient;
@@ -35,7 +36,7 @@ public class TestDownloadService : DownloadService
public override void Dispose() { }
public override Task LoginAsync() => Task.CompletedTask;
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads) => Task.FromResult(new StalledResult());
public override Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads) => Task.FromResult(new DownloadCheckResult());
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType,
ConcurrentBag<string> patterns, ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads) => Task.FromResult(new BlockFilesResult());
public override Task DeleteDownload(string hash) => Task.CompletedTask;
@@ -44,7 +45,6 @@ public class TestDownloadService : DownloadService
IReadOnlyList<string> ignoredDownloads) => Task.CompletedTask;
// Expose protected methods for testing
public new void ResetStrikesOnProgress(string hash, long downloaded) => base.ResetStrikesOnProgress(hash, downloaded);
public new Task<bool> StrikeAndCheckLimit(string hash, string itemName) => base.StrikeAndCheckLimit(hash, itemName);
public new void ResetStalledStrikesOnProgress(string hash, long downloaded) => base.ResetStalledStrikesOnProgress(hash, downloaded);
public new SeedingCheckResult ShouldCleanDownload(double ratio, TimeSpan seedingTime, Category category) => base.ShouldCleanDownload(ratio, seedingTime, category);
}

View File

@@ -4,7 +4,7 @@ namespace Infrastructure.Extensions;
public static class DelugeExtensions
{
public static bool ShouldIgnore(this TorrentStatus download, IReadOnlyList<string> ignoredDownloads)
public static bool ShouldIgnore(this DownloadStatus download, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{

View File

@@ -10,7 +10,7 @@ public static class CacheKeys
public static string BlocklistPatterns(InstanceType instanceType) => $"{instanceType.ToString()}_patterns";
public static string BlocklistRegexes(InstanceType instanceType) => $"{instanceType.ToString()}_regexes";
public static string Item(string hash) => $"item_{hash}";
public static string StrikeItem(string hash, StrikeType strikeType) => $"item_{hash}_{strikeType.ToString()}";
public static string IgnoredDownloads(string name) => $"{name}_ignored";
}

View File

@@ -13,7 +13,7 @@
<ItemGroup>
<PackageReference Include="FLM.QBittorrent" Version="1.0.0" />
<PackageReference Include="FLM.Transmission" Version="1.0.2" />
<PackageReference Include="FLM.Transmission" Version="1.0.3" />
<PackageReference Include="Mapster" Version="7.4.0" />
<PackageReference Include="MassTransit" Version="8.3.6" />
<PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="9.0.2" />

View File

@@ -43,9 +43,11 @@ public abstract class ArrClient : IArrClient
public virtual async Task<QueueListResponse> GetQueueItemsAsync(ArrInstance arrInstance, int page)
{
Uri uri = new(arrInstance.Url, GetQueueUrlPath(page));
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/{GetQueueUrlPath().TrimStart('/')}";
uriBuilder.Query = GetQueueUrlQuery(page);
using HttpRequestMessage request = new(HttpMethod.Get, uri);
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);
@@ -56,7 +58,7 @@ public abstract class ArrClient : IArrClient
}
catch
{
_logger.LogError("queue list failed | {uri}", uri);
_logger.LogError("queue list failed | {uri}", uriBuilder.Uri);
throw;
}
@@ -65,7 +67,7 @@ public abstract class ArrClient : IArrClient
if (queueResponse is null)
{
throw new Exception($"unrecognized queue list response | {uri} | {responseBody}");
throw new Exception($"unrecognized queue list response | {uriBuilder.Uri} | {responseBody}");
}
return queueResponse;
@@ -112,13 +114,20 @@ public abstract class ArrClient : IArrClient
return false;
}
public virtual async Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record, bool removeFromClient)
public virtual async Task DeleteQueueItemAsync(
ArrInstance arrInstance,
QueueRecord record,
bool removeFromClient,
DeleteReason deleteReason
)
{
Uri uri = new(arrInstance.Url, GetQueueDeleteUrlPath(record.Id, removeFromClient));
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/{GetQueueDeleteUrlPath(record.Id).TrimStart('/')}";
uriBuilder.Query = GetQueueDeleteUrlQuery(removeFromClient);
try
{
using HttpRequestMessage request = new(HttpMethod.Delete, uri);
using HttpRequestMessage request = new(HttpMethod.Delete, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
HttpResponseMessage? response = await _dryRunInterceptor.InterceptAsync<HttpResponseMessage>(SendRequestAsync, request);
@@ -126,15 +135,16 @@ public abstract class ArrClient : IArrClient
_logger.LogInformation(
removeFromClient
? "queue item deleted | {url} | {title}"
: "queue item removed from arr | {url} | {title}",
? "queue item deleted with reason {reason} | {url} | {title}"
: "queue item removed from arr with reason {reason} | {url} | {title}",
deleteReason.ToString(),
arrInstance.Url,
record.Title
);
}
catch
{
_logger.LogError("queue delete failed | {uri} | {title}", uri, record.Title);
_logger.LogError("queue delete failed | {uri} | {title}", uriBuilder.Uri, record.Title);
throw;
}
}
@@ -152,9 +162,13 @@ public abstract class ArrClient : IArrClient
return true;
}
protected abstract string GetQueueUrlPath(int page);
protected abstract string GetQueueUrlPath();
protected abstract string GetQueueDeleteUrlPath(long recordId, bool removeFromClient);
protected abstract string GetQueueUrlQuery(int page);
protected abstract string GetQueueDeleteUrlPath(long recordId);
protected abstract string GetQueueDeleteUrlQuery(bool removeFromClient);
protected virtual void SetApiKey(HttpRequestMessage request, string apiKey)
{

View File

@@ -11,7 +11,7 @@ public interface IArrClient
Task<bool> ShouldRemoveFromQueue(InstanceType instanceType, QueueRecord record, bool isPrivateDownload);
Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record, bool removeFromClient);
Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record, bool removeFromClient, DeleteReason deleteReason);
Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items);

View File

@@ -27,29 +27,42 @@ public class LidarrClient : ArrClient, ILidarrClient
{
}
protected override string GetQueueUrlPath(int page)
protected override string GetQueueUrlPath()
{
return $"/api/v1/queue?page={page}&pageSize=200&includeUnknownArtistItems=true&includeArtist=true&includeAlbum=true";
return "/api/v1/queue";
}
protected override string GetQueueDeleteUrlPath(long recordId, bool removeFromClient)
protected override string GetQueueUrlQuery(int page)
{
string path = $"/api/v1/queue/{recordId}?blocklist=true&skipRedownload=true&changeCategory=false";
return $"page={page}&pageSize=200&includeUnknownArtistItems=true&includeArtist=true&includeAlbum=true";
}
path += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
protected override string GetQueueDeleteUrlPath(long recordId)
{
return $"/api/v1/queue/{recordId}";
}
return path;
protected override string GetQueueDeleteUrlQuery(bool removeFromClient)
{
string query = "blocklist=true&skipRedownload=true&changeCategory=false";
query += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return query;
}
public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items)
{
if (items?.Count is null or 0) return;
if (items?.Count is null or 0)
{
return;
}
Uri uri = new(arrInstance.Url, "/api/v1/command");
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v1/command";
foreach (var command in GetSearchCommands(items))
{
using HttpRequestMessage request = new(HttpMethod.Post, uri);
using HttpRequestMessage request = new(HttpMethod.Post, uriBuilder.Uri);
request.Content = new StringContent(
JsonConvert.SerializeObject(command, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }),
Encoding.UTF8,
@@ -132,8 +145,11 @@ public class LidarrClient : ArrClient, ILidarrClient
private async Task<List<Album>?> GetAlbumsAsync(ArrInstance arrInstance, List<long> albumIds)
{
Uri uri = new(arrInstance.Url, $"api/v1/album?{string.Join('&', albumIds.Select(x => $"albumIds={x}"))}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v1/album";
uriBuilder.Query = string.Join('&', albumIds.Select(x => $"albumIds={x}"));
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using var response = await _httpClient.SendAsync(request);

View File

@@ -27,18 +27,27 @@ public class RadarrClient : ArrClient, IRadarrClient
{
}
protected override string GetQueueUrlPath(int page)
protected override string GetQueueUrlPath()
{
return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownMovieItems=true&includeMovie=true";
return "/api/v3/queue";
}
protected override string GetQueueDeleteUrlPath(long recordId, bool removeFromClient)
protected override string GetQueueUrlQuery(int page)
{
string path = $"/api/v3/queue/{recordId}?blocklist=true&skipRedownload=true&changeCategory=false";
path += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return $"page={page}&pageSize=200&includeUnknownMovieItems=true&includeMovie=true";
}
return path;
protected override string GetQueueDeleteUrlPath(long recordId)
{
return $"/api/v3/queue/{recordId}";
}
protected override string GetQueueDeleteUrlQuery(bool removeFromClient)
{
string query = "blocklist=true&skipRedownload=true&changeCategory=false";
query += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return query;
}
public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items)
@@ -50,14 +59,16 @@ public class RadarrClient : ArrClient, IRadarrClient
List<long> ids = items.Select(item => item.Id).ToList();
Uri uri = new(arrInstance.Url, "/api/v3/command");
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/command";
RadarrCommand command = new()
{
Name = "MoviesSearch",
MovieIds = ids,
};
using HttpRequestMessage request = new(HttpMethod.Post, uri);
using HttpRequestMessage request = new(HttpMethod.Post, uriBuilder.Uri);
request.Content = new StringContent(
JsonConvert.SerializeObject(command),
Encoding.UTF8,
@@ -135,8 +146,10 @@ public class RadarrClient : ArrClient, IRadarrClient
private async Task<Movie?> GetMovie(ArrInstance arrInstance, long movieId)
{
Uri uri = new(arrInstance.Url, $"api/v3/movie/{movieId}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/movie/{movieId}";
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);

View File

@@ -28,18 +28,27 @@ public class SonarrClient : ArrClient, ISonarrClient
{
}
protected override string GetQueueUrlPath(int page)
protected override string GetQueueUrlPath()
{
return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownSeriesItems=true&includeSeries=true&includeEpisode=true";
return "/api/v3/queue";
}
protected override string GetQueueDeleteUrlPath(long recordId, bool removeFromClient)
protected override string GetQueueUrlQuery(int page)
{
string path = $"/api/v3/queue/{recordId}?blocklist=true&skipRedownload=true&changeCategory=false";
return $"page={page}&pageSize=200&includeUnknownSeriesItems=true&includeSeries=true&includeEpisode=true";
}
path += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
protected override string GetQueueDeleteUrlPath(long recordId)
{
return $"/api/v3/queue/{recordId}";
}
return path;
protected override string GetQueueDeleteUrlQuery(bool removeFromClient)
{
string query = "blocklist=true&skipRedownload=true&changeCategory=false";
query += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return query;
}
public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items)
@@ -49,11 +58,12 @@ public class SonarrClient : ArrClient, ISonarrClient
return;
}
Uri uri = new(arrInstance.Url, "/api/v3/command");
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/command";
foreach (SonarrCommand command in GetSearchCommands(items.Cast<SonarrSearchItem>().ToHashSet()))
{
using HttpRequestMessage request = new(HttpMethod.Post, uri);
using HttpRequestMessage request = new(HttpMethod.Post, uriBuilder.Uri);
request.Content = new StringContent(
JsonConvert.SerializeObject(command, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }),
Encoding.UTF8,
@@ -199,8 +209,11 @@ public class SonarrClient : ArrClient, ISonarrClient
private async Task<List<Episode>?> GetEpisodesAsync(ArrInstance arrInstance, List<long> episodeIds)
{
Uri uri = new(arrInstance.Url, $"api/v3/episode?{string.Join('&', episodeIds.Select(x => $"episodeIds={x}"))}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/episode";
uriBuilder.Query = string.Join('&', episodeIds.Select(x => $"episodeIds={x}"));
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);
@@ -212,8 +225,10 @@ public class SonarrClient : ArrClient, ISonarrClient
private async Task<Series?> GetSeriesAsync(ArrInstance arrInstance, long seriesId)
{
Uri uri = new(arrInstance.Url, $"api/v3/series/{seriesId}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/series/{seriesId}";
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);

View File

@@ -142,7 +142,7 @@ public sealed class ContentBlocker : GenericHandler
removeFromClient = false;
}
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient);
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient, DeleteReason.AllFilesBlocked);
await _notifier.NotifyQueueItemDeleted(removeFromClient, DeleteReason.AllFilesBlocked);
}
});

View File

@@ -27,12 +27,15 @@ public sealed class DelugeClient
"label",
"seeding_time",
"ratio",
"trackers"
"trackers",
"download_payload_rate",
"total_size"
];
public DelugeClient(IOptions<DelugeConfig> config, IHttpClientFactory httpClientFactory)
{
_config = config.Value;
_config.Validate();
_httpClient = httpClientFactory.CreateClient(nameof(DelugeService));
}
@@ -77,18 +80,31 @@ public sealed class DelugeClient
return torrents.FirstOrDefault();
}
public async Task<TorrentStatus?> GetTorrentStatus(string hash)
public async Task<DownloadStatus?> GetTorrentStatus(string hash)
{
return await SendRequest<TorrentStatus?>(
"web.get_torrent_status",
hash,
Fields
);
try
{
return await SendRequest<DownloadStatus?>(
"web.get_torrent_status",
hash,
Fields
);
}
catch (DelugeClientException e)
{
// Deluge returns an error when the torrent is not found
if (e.Message == "AttributeError: 'NoneType' object has no attribute 'call'")
{
return null;
}
throw;
}
}
public async Task<List<TorrentStatus>?> GetStatusForAllTorrents()
public async Task<List<DownloadStatus>?> GetStatusForAllTorrents()
{
Dictionary<string, TorrentStatus>? downloads = await SendRequest<Dictionary<string, TorrentStatus>?>(
Dictionary<string, DownloadStatus>? downloads = await SendRequest<Dictionary<string, DownloadStatus>?>(
"core.get_torrents_status",
"",
Fields
@@ -121,8 +137,12 @@ public sealed class DelugeClient
{
StringContent content = new StringContent(json);
content.Headers.ContentType = new MediaTypeWithQualityHeaderValue("application/json");
var responseMessage = await _httpClient.PostAsync(new Uri(_config.Url, "/json"), content);
UriBuilder uriBuilder = new(_config.Url);
uriBuilder.Path = string.IsNullOrEmpty(_config.UrlBase)
? $"{uriBuilder.Path.TrimEnd('/')}/json"
: $"{uriBuilder.Path.TrimEnd('/')}/{_config.UrlBase.TrimStart('/').TrimEnd('/')}/json";
var responseMessage = await _httpClient.PostAsync(uriBuilder.Uri, content);
responseMessage.EnsureSuccessStatusCode();
var responseJson = await responseMessage.Content.ReadAsStringAsync();

View File

@@ -1,10 +1,12 @@
using System.Collections.Concurrent;
using System.Globalization;
using System.Text.RegularExpressions;
using Common.Attributes;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Common.CustomDataTypes;
using Domain.Enums;
using Domain.Models.Deluge.Response;
using Infrastructure.Extensions;
@@ -50,14 +52,14 @@ public class DelugeService : DownloadService, IDelugeService
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
public override async Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
hash = hash.ToLowerInvariant();
DelugeContents? contents = null;
StalledResult result = new();
DownloadCheckResult result = new();
TorrentStatus? download = await _client.GetTorrentStatus(hash);
DownloadStatus? download = await _client.GetTorrentStatus(hash);
if (download?.Hash is null)
{
@@ -65,6 +67,8 @@ public class DelugeService : DownloadService, IDelugeService
return result;
}
result.IsPrivate = download.Private;
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
@@ -79,6 +83,7 @@ public class DelugeService : DownloadService, IDelugeService
{
_logger.LogDebug(exception, "failed to find torrent {hash} in the download client", hash);
}
bool shouldRemove = contents?.Contents?.Count > 0;
@@ -92,17 +97,15 @@ public class DelugeService : DownloadService, IDelugeService
if (shouldRemove)
{
result.DeleteReason = DeleteReason.AllFilesBlocked;
// remove if all files are unwanted
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesSkipped;
return result;
}
result.ShouldRemove = shouldRemove || await IsItemStuckAndShouldRemove(download);
result.IsPrivate = download.Private;
// remove if download is stuck
(result.ShouldRemove, result.DeleteReason) = await EvaluateDownloadRemoval(download);
if (!shouldRemove && result.ShouldRemove)
{
result.DeleteReason = DeleteReason.Stalled;
}
return result;
}
@@ -114,7 +117,7 @@ public class DelugeService : DownloadService, IDelugeService
{
hash = hash.ToLowerInvariant();
TorrentStatus? download = await _client.GetTorrentStatus(hash);
DownloadStatus? download = await _client.GetTorrentStatus(hash);
BlockFilesResult result = new();
if (download?.Hash is null)
@@ -123,9 +126,6 @@ public class DelugeService : DownloadService, IDelugeService
return result;
}
var ceva = await _client.GetTorrentExtended(hash);
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
@@ -222,7 +222,7 @@ public class DelugeService : DownloadService, IDelugeService
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads)
{
foreach (TorrentStatus download in downloads)
foreach (DownloadStatus download in downloads)
{
if (string.IsNullOrEmpty(download.Hash))
{
@@ -295,33 +295,90 @@ public class DelugeService : DownloadService, IDelugeService
await _client.ChangeFilesPriority(hash, sortedPriorities);
}
private async Task<bool> IsItemStuckAndShouldRemove(TorrentStatus status)
private async Task<(bool, DeleteReason)> EvaluateDownloadRemoval(DownloadStatus status)
{
(bool ShouldRemove, DeleteReason Reason) result = await CheckIfSlow(status);
if (result.ShouldRemove)
{
return result;
}
return await CheckIfStuck(status);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfSlow(DownloadStatus download)
{
if (_queueCleanerConfig.SlowMaxStrikes is 0)
{
return (false, DeleteReason.None);
}
if (download.State is null || !download.State.Equals("Downloading", StringComparison.InvariantCultureIgnoreCase))
{
return (false, DeleteReason.None);
}
if (download.DownloadSpeed <= 0)
{
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.SlowIgnorePrivate && download.Private)
{
// ignore private trackers
_logger.LogDebug("skip slow check | download is private | {name}", download.Name);
return (false, DeleteReason.None);
}
if (download.Size > (_queueCleanerConfig.SlowIgnoreAboveSizeByteSize?.Bytes ?? long.MaxValue))
{
_logger.LogDebug("skip slow check | download is too large | {name}", download.Name);
return (false, DeleteReason.None);
}
ByteSize minSpeed = _queueCleanerConfig.SlowMinSpeedByteSize;
ByteSize currentSpeed = new ByteSize(download.DownloadSpeed);
SmartTimeSpan maxTime = SmartTimeSpan.FromHours(_queueCleanerConfig.SlowMaxTime);
SmartTimeSpan currentTime = SmartTimeSpan.FromSeconds(download.Eta);
return await CheckIfSlow(
download.Hash!,
download.Name!,
minSpeed,
currentSpeed,
maxTime,
currentTime
);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfStuck(DownloadStatus status)
{
if (_queueCleanerConfig.StalledMaxStrikes is 0)
{
return false;
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.StalledIgnorePrivate && status.Private)
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", status.Name);
return false;
return (false, DeleteReason.None);
}
if (status.State is null || !status.State.Equals("Downloading", StringComparison.InvariantCultureIgnoreCase))
{
return false;
return (false, DeleteReason.None);
}
if (status.Eta > 0)
{
return false;
return (false, DeleteReason.None);
}
ResetStrikesOnProgress(status.Hash!, status.TotalDone);
return await StrikeAndCheckLimit(status.Hash!, status.Name!);
ResetStalledStrikesOnProgress(status.Hash!, status.TotalDone);
return (await _striker.StrikeAndCheckLimit(status.Hash!, status.Name!, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled), DeleteReason.Stalled);
}
private static void ProcessFiles(Dictionary<string, DelugeFileOrDirectory>? contents, Action<string, DelugeFileOrDirectory> processFile)

View File

@@ -2,7 +2,7 @@
namespace Infrastructure.Verticals.DownloadClient;
public sealed record StalledResult
public sealed record DownloadCheckResult
{
/// <summary>
/// True if the download should be removed; otherwise false.

View File

@@ -3,6 +3,7 @@ using System.Text.RegularExpressions;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Common.CustomDataTypes;
using Common.Helpers;
using Domain.Enums;
using Domain.Models.Cache;
@@ -60,7 +61,7 @@ public abstract class DownloadService : IDownloadService
public abstract Task LoginAsync();
public abstract Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads);
public abstract Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads);
/// <inheritdoc/>
public abstract Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
@@ -78,32 +79,104 @@ public abstract class DownloadService : IDownloadService
public abstract Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads);
protected void ResetStrikesOnProgress(string hash, long downloaded)
protected void ResetStalledStrikesOnProgress(string hash, long downloaded)
{
if (!_queueCleanerConfig.StalledResetStrikesOnProgress)
{
return;
}
if (_cache.TryGetValue(CacheKeys.Item(hash), out CacheItem? cachedItem) && cachedItem is not null && downloaded > cachedItem.Downloaded)
if (_cache.TryGetValue(CacheKeys.StrikeItem(hash, StrikeType.Stalled), out StalledCacheItem? cachedItem) &&
cachedItem is not null && downloaded > cachedItem.Downloaded)
{
// cache item found
_cache.Remove(CacheKeys.Strike(StrikeType.Stalled, hash));
_logger.LogDebug("resetting strikes for {hash} due to progress", hash);
_logger.LogDebug("resetting stalled strikes for {hash} due to progress", hash);
}
_cache.Set(CacheKeys.Item(hash), new CacheItem { Downloaded = downloaded }, _cacheOptions);
_cache.Set(CacheKeys.StrikeItem(hash, StrikeType.Stalled), new StalledCacheItem { Downloaded = downloaded }, _cacheOptions);
}
protected void ResetSlowSpeedStrikesOnProgress(string downloadName, string hash)
{
if (!_queueCleanerConfig.SlowResetStrikesOnProgress)
{
return;
}
string key = CacheKeys.Strike(StrikeType.SlowSpeed, hash);
if (!_cache.TryGetValue(key, out object? value) || value is null)
{
return;
}
_cache.Remove(key);
_logger.LogDebug("resetting slow speed strikes due to progress | {name}", downloadName);
}
protected void ResetSlowTimeStrikesOnProgress(string downloadName, string hash)
{
if (!_queueCleanerConfig.SlowResetStrikesOnProgress)
{
return;
}
string key = CacheKeys.Strike(StrikeType.SlowTime, hash);
if (!_cache.TryGetValue(key, out object? value) || value is null)
{
return;
}
_cache.Remove(key);
_logger.LogDebug("resetting slow time strikes due to progress | {name}", downloadName);
}
/// <summary>
/// Strikes an item and checks if the limit has been reached.
/// </summary>
/// <param name="hash">The torrent hash.</param>
/// <param name="itemName">The name or title of the item.</param>
/// <returns>True if the limit has been reached; otherwise, false.</returns>
protected async Task<bool> StrikeAndCheckLimit(string hash, string itemName)
protected async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfSlow(
string downloadHash,
string downloadName,
ByteSize minSpeed,
ByteSize currentSpeed,
SmartTimeSpan maxTime,
SmartTimeSpan currentTime
)
{
return await _striker.StrikeAndCheckLimit(hash, itemName, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled);
if (minSpeed.Bytes > 0 && currentSpeed < minSpeed)
{
_logger.LogTrace("slow speed | {speed}/s | {name}", currentSpeed.ToString(), downloadName);
bool shouldRemove = await _striker
.StrikeAndCheckLimit(downloadHash, downloadName, _queueCleanerConfig.SlowMaxStrikes, StrikeType.SlowSpeed);
if (shouldRemove)
{
return (true, DeleteReason.SlowSpeed);
}
}
else
{
ResetSlowSpeedStrikesOnProgress(downloadName, downloadHash);
}
if (maxTime.Time > TimeSpan.Zero && currentTime > maxTime)
{
_logger.LogTrace("slow estimated time | {time} | {name}", currentTime.ToString(), downloadName);
bool shouldRemove = await _striker
.StrikeAndCheckLimit(downloadHash, downloadName, _queueCleanerConfig.SlowMaxStrikes, StrikeType.SlowTime);
if (shouldRemove)
{
return (true, DeleteReason.SlowTime);
}
}
else
{
ResetSlowTimeStrikesOnProgress(downloadName, downloadHash);
}
return (false, DeleteReason.None);
}
protected SeedingCheckResult ShouldCleanDownload(double ratio, TimeSpan seedingTime, Category category)

View File

@@ -28,7 +28,7 @@ public class DummyDownloadService : DownloadService
return Task.CompletedTask;
}
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
public override Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
throw new NotImplementedException();
}

View File

@@ -15,7 +15,7 @@ public interface IDownloadService : IDisposable
/// </summary>
/// <param name="hash">The download hash.</param>
/// <param name="ignoredDownloads">Downloads to ignore from processing.</param>
public Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads);
public Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads);
/// <summary>
/// Blocks unwanted files from being fully downloaded.

View File

@@ -5,6 +5,7 @@ using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Common.CustomDataTypes;
using Common.Helpers;
using Domain.Enums;
using Infrastructure.Extensions;
@@ -45,7 +46,11 @@ public class QBitService : DownloadService, IQBitService
{
_config = config.Value;
_config.Validate();
_client = new(httpClientFactory.CreateClient(Constants.HttpClientWithRetryName), _config.Url);
UriBuilder uriBuilder = new(_config.Url);
uriBuilder.Path = string.IsNullOrEmpty(_config.UrlBase)
? uriBuilder.Path
: $"{uriBuilder.Path.TrimEnd('/')}/{_config.UrlBase.TrimStart('/')}";
_client = new(httpClientFactory.CreateClient(Constants.HttpClientWithRetryName), uriBuilder.Uri);
}
public override async Task LoginAsync()
@@ -59,9 +64,9 @@ public class QBitService : DownloadService, IQBitService
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
public override async Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
StalledResult result = new();
DownloadCheckResult result = new();
TorrentInfo? download = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
.FirstOrDefault();
@@ -92,30 +97,25 @@ public class QBitService : DownloadService, IQBitService
bool.TryParse(dictValue?.ToString(), out bool boolValue)
&& boolValue;
// if all files were blocked by qBittorrent
if (download is { CompletionOn: not null, Downloaded: null or 0 })
{
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesBlocked;
return result;
}
IReadOnlyList<TorrentContent>? files = await _client.GetTorrentContentsAsync(hash);
// if all files are marked as skip
if (files?.Count is > 0 && files.All(x => x.Priority is TorrentContentPriority.Skip))
{
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesBlocked;
// if all files were blocked by qBittorrent
if (download is { CompletionOn: not null, Downloaded: null or 0 })
{
result.DeleteReason = DeleteReason.AllFilesSkippedByQBit;
return result;
}
// remove if all files are unwanted
result.DeleteReason = DeleteReason.AllFilesSkipped;
return result;
}
result.ShouldRemove = await IsItemStuckAndShouldRemove(download, result.IsPrivate);
if (result.ShouldRemove)
{
result.DeleteReason = DeleteReason.Stalled;
}
(result.ShouldRemove, result.DeleteReason) = await EvaluateDownloadRemoval(download, result.IsPrivate);
return result;
}
@@ -333,30 +333,96 @@ public class QBitService : DownloadService, IQBitService
_client.Dispose();
}
private async Task<bool> IsItemStuckAndShouldRemove(TorrentInfo torrent, bool isPrivate)
private async Task<(bool, DeleteReason)> EvaluateDownloadRemoval(TorrentInfo torrent, bool isPrivate)
{
(bool ShouldRemove, DeleteReason Reason) result = await CheckIfSlow(torrent, isPrivate);
if (result.ShouldRemove)
{
return result;
}
return await CheckIfStuck(torrent, isPrivate);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfSlow(TorrentInfo download, bool isPrivate)
{
if (_queueCleanerConfig.SlowMaxStrikes is 0)
{
return (false, DeleteReason.None);
}
if (download.State is not (TorrentState.Downloading or TorrentState.ForcedDownload))
{
return (false, DeleteReason.None);
}
if (download.DownloadSpeed <= 0)
{
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.SlowIgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip slow check | download is private | {name}", download.Name);
return (false, DeleteReason.None);
}
if (download.Size > (_queueCleanerConfig.SlowIgnoreAboveSizeByteSize?.Bytes ?? long.MaxValue))
{
_logger.LogDebug("skip slow check | download is too large | {name}", download.Name);
return (false, DeleteReason.None);
}
ByteSize minSpeed = _queueCleanerConfig.SlowMinSpeedByteSize;
ByteSize currentSpeed = new ByteSize(download.DownloadSpeed);
SmartTimeSpan maxTime = SmartTimeSpan.FromHours(_queueCleanerConfig.SlowMaxTime);
SmartTimeSpan currentTime = new SmartTimeSpan(download.EstimatedTime ?? TimeSpan.Zero);
return await CheckIfSlow(
download.Hash,
download.Name,
minSpeed,
currentSpeed,
maxTime,
currentTime
);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfStuck(TorrentInfo torrent, bool isPrivate)
{
if (_queueCleanerConfig.StalledMaxStrikes is 0)
{
return false;
}
if (_queueCleanerConfig.StalledIgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", torrent.Name);
return false;
return (false, DeleteReason.None);
}
if (torrent.State is not TorrentState.StalledDownload and not TorrentState.FetchingMetadata
and not TorrentState.ForcedFetchingMetadata)
{
// ignore other states
return false;
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.StalledIgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", torrent.Name);
return (false, DeleteReason.None);
}
if (torrent.State is TorrentState.StalledDownload)
{
_logger.LogTrace("stalled download | {name}", torrent.Name);
ResetStalledStrikesOnProgress(torrent.Hash, torrent.Downloaded ?? 0);
return (await _striker.StrikeAndCheckLimit(torrent.Hash, torrent.Name, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled), DeleteReason.Stalled);
}
ResetStrikesOnProgress(torrent.Hash, torrent.Downloaded ?? 0);
return await StrikeAndCheckLimit(torrent.Hash, torrent.Name);
_logger.LogTrace("downloading metadata | {name}", torrent.Name);
return (await _striker.StrikeAndCheckLimit(torrent.Hash, torrent.Name, _queueCleanerConfig.StalledMaxStrikes, StrikeType.DownloadingMetadata), DeleteReason.DownloadingMetadata);
}
private async Task<IReadOnlyList<TorrentTracker>> GetTrackersAsync(string hash)

View File

@@ -5,6 +5,7 @@ using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Common.CustomDataTypes;
using Common.Helpers;
using Domain.Enums;
using Infrastructure.Extensions;
@@ -26,7 +27,6 @@ public class TransmissionService : DownloadService, ITransmissionService
{
private readonly TransmissionConfig _config;
private readonly Client _client;
private TorrentInfo[]? _torrentsCache;
private static readonly string[] Fields =
[
@@ -42,7 +42,9 @@ public class TransmissionService : DownloadService, ITransmissionService
TorrentFields.DOWNLOAD_DIR,
TorrentFields.SECONDS_SEEDING,
TorrentFields.UPLOAD_RATIO,
TorrentFields.TRACKERS
TorrentFields.TRACKERS,
TorrentFields.RATE_DOWNLOAD,
TorrentFields.TOTAL_SIZE
];
public TransmissionService(
@@ -64,9 +66,13 @@ public class TransmissionService : DownloadService, ITransmissionService
{
_config = config.Value;
_config.Validate();
UriBuilder uriBuilder = new(_config.Url);
uriBuilder.Path = string.IsNullOrEmpty(_config.UrlBase)
? $"{uriBuilder.Path.TrimEnd('/')}/rpc"
: $"{uriBuilder.Path.TrimEnd('/')}/{_config.UrlBase.TrimStart('/').TrimEnd('/')}/rpc";
_client = new(
httpClientFactory.CreateClient(Constants.HttpClientWithRetryName),
new Uri(_config.Url, "/transmission/rpc").ToString(),
uriBuilder.Uri.ToString(),
login: _config.Username,
password: _config.Password
);
@@ -78,9 +84,9 @@ public class TransmissionService : DownloadService, ITransmissionService
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
public override async Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
StalledResult result = new();
DownloadCheckResult result = new();
TorrentInfo? download = await GetTorrentAsync(hash);
if (download is null)
@@ -115,17 +121,15 @@ public class TransmissionService : DownloadService, ITransmissionService
if (shouldRemove)
{
// remove if all files are unwanted
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesBlocked;
return result;
}
// remove if all files are unwanted or download is stuck
result.ShouldRemove = shouldRemove || await IsItemStuckAndShouldRemove(download);
// remove if download is stuck
(result.ShouldRemove, result.DeleteReason) = await EvaluateDownloadRemoval(download);
if (!shouldRemove && result.ShouldRemove)
{
result.DeleteReason = DeleteReason.Stalled;
}
return result;
}
@@ -334,60 +338,96 @@ public class TransmissionService : DownloadService, ITransmissionService
});
}
private async Task<bool> IsItemStuckAndShouldRemove(TorrentInfo torrent)
private async Task<(bool, DeleteReason)> EvaluateDownloadRemoval(TorrentInfo torrent)
{
(bool ShouldRemove, DeleteReason Reason) result = await CheckIfSlow(torrent);
if (result.ShouldRemove)
{
return result;
}
return await CheckIfStuck(torrent);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfSlow(TorrentInfo download)
{
if (_queueCleanerConfig.SlowMaxStrikes is 0)
{
return (false, DeleteReason.None);
}
if (download.Status is not 4)
{
// not in downloading state
return (false, DeleteReason.None);
}
if (download.RateDownload <= 0)
{
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.SlowIgnorePrivate && download.IsPrivate is true)
{
// ignore private trackers
_logger.LogDebug("skip slow check | download is private | {name}", download.Name);
return (false, DeleteReason.None);
}
if (download.TotalSize > (_queueCleanerConfig.SlowIgnoreAboveSizeByteSize?.Bytes ?? long.MaxValue))
{
_logger.LogDebug("skip slow check | download is too large | {name}", download.Name);
return (false, DeleteReason.None);
}
ByteSize minSpeed = _queueCleanerConfig.SlowMinSpeedByteSize;
ByteSize currentSpeed = new ByteSize(download.RateDownload ?? long.MaxValue);
SmartTimeSpan maxTime = SmartTimeSpan.FromHours(_queueCleanerConfig.SlowMaxTime);
SmartTimeSpan currentTime = SmartTimeSpan.FromSeconds(download.Eta ?? 0);
return await CheckIfSlow(
download.HashString!,
download.Name!,
minSpeed,
currentSpeed,
maxTime,
currentTime
);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfStuck(TorrentInfo download)
{
if (_queueCleanerConfig.StalledMaxStrikes is 0)
{
return false;
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.StalledIgnorePrivate && (torrent.IsPrivate ?? false))
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", torrent.Name);
return false;
}
if (torrent.Status is not 4)
if (download.Status is not 4)
{
// not in downloading state
return false;
}
if (torrent.Eta > 0)
{
return false;
return (false, DeleteReason.None);
}
ResetStrikesOnProgress(torrent.HashString!, torrent.DownloadedEver ?? 0);
return await StrikeAndCheckLimit(torrent.HashString!, torrent.Name!);
if (download.RateDownload > 0 || download.Eta > 0)
{
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.StalledIgnorePrivate && (download.IsPrivate ?? false))
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", download.Name);
return (false, DeleteReason.None);
}
ResetStalledStrikesOnProgress(download.HashString!, download.DownloadedEver ?? 0);
return (await _striker.StrikeAndCheckLimit(download.HashString!, download.Name!, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled), DeleteReason.Stalled);
}
private async Task<TorrentInfo?> GetTorrentAsync(string hash)
{
TorrentInfo? torrent = _torrentsCache?
.FirstOrDefault(x => x.HashString.Equals(hash, StringComparison.InvariantCultureIgnoreCase));
if (_torrentsCache is null || torrent is null)
{
// refresh cache
_torrentsCache = (await _client.TorrentGetAsync(Fields))
?.Torrents;
}
if (_torrentsCache?.Length is null or 0)
{
_logger.LogDebug("could not list torrents | {url}", _config.Url);
}
torrent = _torrentsCache?.FirstOrDefault(x => x.HashString.Equals(hash, StringComparison.InvariantCultureIgnoreCase));
if (torrent is null)
{
_logger.LogDebug("could not find torrent | {hash} | {url}", hash, _config.Url);
}
return torrent;
}
private async Task<TorrentInfo?> GetTorrentAsync(string hash) =>
(await _client.TorrentGetAsync(Fields, hash))
?.Torrents
?.FirstOrDefault();
}

View File

@@ -27,6 +27,9 @@ public sealed class NotificationConsumer<T> : IConsumer<T> where T : Notificatio
case StalledStrikeNotification stalledMessage:
await _notificationService.Notify(stalledMessage);
break;
case SlowStrikeNotification slowMessage:
await _notificationService.Notify(slowMessage);
break;
case QueueItemDeletedNotification queueItemDeleteMessage:
await _notificationService.Notify(queueItemDeleteMessage);
break;

View File

@@ -6,6 +6,8 @@ public interface INotificationFactory
List<INotificationProvider> OnStalledStrikeEnabled();
List<INotificationProvider> OnSlowStrikeEnabled();
List<INotificationProvider> OnQueueItemDeletedEnabled();
List<INotificationProvider> OnDownloadCleanedEnabled();

View File

@@ -12,6 +12,8 @@ public interface INotificationProvider
Task OnFailedImportStrike(FailedImportStrikeNotification notification);
Task OnStalledStrike(StalledStrikeNotification notification);
Task OnSlowStrike(SlowStrikeNotification notification);
Task OnQueueItemDeleted(QueueItemDeletedNotification notification);

View File

@@ -0,0 +1,5 @@
namespace Infrastructure.Verticals.Notifications.Models;
public sealed record SlowStrikeNotification : ArrNotification
{
}

View File

@@ -32,6 +32,11 @@ public class NotifiarrProvider : NotificationProvider
await _proxy.SendNotification(BuildPayload(notification, WarningColor), _config);
}
public override async Task OnSlowStrike(SlowStrikeNotification notification)
{
await _proxy.SendNotification(BuildPayload(notification, WarningColor), _config);
}
public override async Task OnQueueItemDeleted(QueueItemDeletedNotification notification)
{
await _proxy.SendNotification(BuildPayload(notification, ImportantColor), _config);

View File

@@ -24,6 +24,11 @@ public class NotificationFactory : INotificationFactory
ActiveProviders()
.Where(n => n.Config.OnStalledStrike)
.ToList();
public List<INotificationProvider> OnSlowStrikeEnabled() =>
ActiveProviders()
.Where(n => n.Config.OnSlowStrike)
.ToList();
public List<INotificationProvider> OnQueueItemDeletedEnabled() =>
ActiveProviders()

View File

@@ -18,6 +18,8 @@ public abstract class NotificationProvider : INotificationProvider
public abstract Task OnFailedImportStrike(FailedImportStrikeNotification notification);
public abstract Task OnStalledStrike(StalledStrikeNotification notification);
public abstract Task OnSlowStrike(SlowStrikeNotification notification);
public abstract Task OnQueueItemDeleted(QueueItemDeletedNotification notification);

View File

@@ -48,11 +48,16 @@ public class NotificationPublisher : INotificationPublisher
switch (strikeType)
{
case StrikeType.Stalled:
case StrikeType.DownloadingMetadata:
await _dryRunInterceptor.InterceptAsync(Notify<StalledStrikeNotification>, notification.Adapt<StalledStrikeNotification>());
break;
case StrikeType.ImportFailed:
await _dryRunInterceptor.InterceptAsync(Notify<FailedImportStrikeNotification>, notification.Adapt<FailedImportStrikeNotification>());
break;
case StrikeType.SlowSpeed:
case StrikeType.SlowTime:
await _dryRunInterceptor.InterceptAsync(Notify<SlowStrikeNotification>, notification.Adapt<SlowStrikeNotification>());
break;
}
}
catch (Exception ex)

View File

@@ -44,6 +44,21 @@ public class NotificationService
}
}
public async Task Notify(SlowStrikeNotification notification)
{
foreach (INotificationProvider provider in _notificationFactory.OnSlowStrikeEnabled())
{
try
{
await provider.OnSlowStrike(notification);
}
catch (Exception exception)
{
_logger.LogWarning(exception, "failed to send notification | provider {provider}", provider.Name);
}
}
}
public async Task Notify(QueueItemDeletedNotification notification)
{
foreach (INotificationProvider provider in _notificationFactory.OnQueueItemDeletedEnabled())

View File

@@ -77,6 +77,8 @@ public sealed class QueueCleaner : GenericHandler
QueueRecord record = group.First();
_logger.LogTrace("processing | {title} | {id}", record.Title, record.DownloadId);
if (!arrClient.IsRecordValid(record))
{
continue;
@@ -91,7 +93,7 @@ public sealed class QueueCleaner : GenericHandler
// push record to context
ContextProvider.Set(nameof(QueueRecord), record);
StalledResult stalledCheckResult = new();
DownloadCheckResult downloadCheckResult = new();
if (record.Protocol is "torrent" && _downloadClientConfig.DownloadClient is not Common.Enums.DownloadClient.Disabled)
{
@@ -102,14 +104,14 @@ public sealed class QueueCleaner : GenericHandler
}
// stalled download check
stalledCheckResult = await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId, ignoredDownloads);
downloadCheckResult = await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId, ignoredDownloads);
}
// failed import check
bool shouldRemoveFromArr = await arrClient.ShouldRemoveFromQueue(instanceType, record, stalledCheckResult.IsPrivate);
DeleteReason deleteReason = stalledCheckResult.ShouldRemove ? stalledCheckResult.DeleteReason : DeleteReason.ImportFailed;
bool shouldRemoveFromArr = await arrClient.ShouldRemoveFromQueue(instanceType, record, downloadCheckResult.IsPrivate);
DeleteReason deleteReason = downloadCheckResult.ShouldRemove ? downloadCheckResult.DeleteReason : DeleteReason.ImportFailed;
if (!shouldRemoveFromArr && !stalledCheckResult.ShouldRemove)
if (!shouldRemoveFromArr && !downloadCheckResult.ShouldRemove)
{
_logger.LogInformation("skip | {title}", record.Title);
continue;
@@ -119,20 +121,26 @@ public sealed class QueueCleaner : GenericHandler
bool removeFromClient = true;
if (stalledCheckResult.IsPrivate)
if (downloadCheckResult.IsPrivate)
{
if (stalledCheckResult.ShouldRemove && !_config.StalledDeletePrivate)
{
removeFromClient = false;
}
bool isStalledWithoutPruneFlag =
downloadCheckResult.DeleteReason is DeleteReason.Stalled &&
!_config.StalledDeletePrivate;
bool isSlowWithoutPruneFlag =
downloadCheckResult.DeleteReason is DeleteReason.SlowSpeed or DeleteReason.SlowTime &&
!_config.SlowDeletePrivate;
bool shouldKeepDueToDeleteRules = downloadCheckResult.ShouldRemove && (isStalledWithoutPruneFlag || isSlowWithoutPruneFlag);
bool shouldKeepDueToImportRules = shouldRemoveFromArr && !_config.ImportFailedDeletePrivate;
if (shouldRemoveFromArr && !_config.ImportFailedDeletePrivate)
if (shouldKeepDueToDeleteRules || shouldKeepDueToImportRules)
{
removeFromClient = false;
}
}
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient);
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient, deleteReason);
await _notifier.NotifyQueueItemDeleted(removeFromClient, deleteReason);
}
});

View File

@@ -178,7 +178,7 @@ services:
- TZ=Europe/Bucharest
- DRY_RUN=false
- LOGGING__LOGLEVEL=Debug
- LOGGING__LOGLEVEL=Verbose
- LOGGING__FILE__ENABLED=true
- LOGGING__FILE__PATH=/var/logs
- LOGGING__ENHANCED=true
@@ -193,14 +193,25 @@ services:
- QUEUECLEANER__ENABLED=true
- QUEUECLEANER__IGNORED_DOWNLOADS_PATH=/ignored
- QUEUECLEANER__RUNSEQUENTIALLY=true
- QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE=true
- QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE=false
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__0=file is a sample
- QUEUECLEANER__STALLED_MAX_STRIKES=5
- QUEUECLEANER__STALLED_RESET_STRIKES_ON_PROGRESS=true
- QUEUECLEANER__STALLED_IGNORE_PRIVATE=true
- QUEUECLEANER__STALLED_DELETE_PRIVATE=false
- QUEUECLEANER__SLOW_MAX_STRIKES=5
- QUEUECLEANER__SLOW_RESET_STRIKES_ON_PROGRESS=true
- QUEUECLEANER__SLOW_IGNORE_PRIVATE=false
- QUEUECLEANER__SLOW_DELETE_PRIVATE=false
- QUEUECLEANER__SLOW_MIN_SPEED=1MB
- QUEUECLEANER__SLOW_MAX_TIME=20
- QUEUECLEANER__SLOW_IGNORE_ABOVE_SIZE=1KB
- CONTENTBLOCKER__ENABLED=true
- CONTENTBLOCKER__IGNORED_DOWNLOADS_PATH=/ignored
- CONTENTBLOCKER__IGNORE_PRIVATE=true
@@ -251,12 +262,13 @@ services:
- LIDARR__INSTANCES__0__URL=http://lidarr:8686
- LIDARR__INSTANCES__0__APIKEY=7f677cfdc074414397af53dd633860c5
# - NOTIFIARR__ON_IMPORT_FAILED_STRIKE=true
# - NOTIFIARR__ON_STALLED_STRIKE=true
# - NOTIFIARR__ON_QUEUE_ITEM_DELETED=true
# - NOTIFIARR__ON_DOWNLOAD_CLEANED=true
# - NOTIFIARR__API_KEY=notifiarr_secret
# - NOTIFIARR__CHANNEL_ID=discord_channel_id
- NOTIFIARR__ON_IMPORT_FAILED_STRIKE=true
- NOTIFIARR__ON_STALLED_STRIKE=true
- NOTIFIARR__ON_SLOW_STRIKE=true
- NOTIFIARR__ON_QUEUE_ITEM_DELETED=true
- NOTIFIARR__ON_DOWNLOAD_CLEANED=true
- NOTIFIARR__API_KEY=notifiarr_secret
- NOTIFIARR__CHANNEL_ID=discord_channel_id
volumes:
- ./data/cleanuperr/logs:/var/logs
- ./data/cleanuperr/ignored_downloads:/ignored

View File

@@ -160,7 +160,7 @@
> If not set to `0`, the minimum value is `3`.
#### **`QUEUECLEANER__STALLED_RESET_STRIKES_ON_PROGRESS`**
- Controls whether to remove strikes if any download progress was made since last checked.
- Controls whether to remove the given strikes if any download progress was made since last checked.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
@@ -174,7 +174,7 @@
- Required: No.
#### **`QUEUECLEANER__STALLED_DELETE_PRIVATE`**
- Controls whether to delete stalled private downloads from the download client.
- Controls whether stalled downloads from private trackers should be removed from the download client.
- Has no effect if `QUEUECLEANER__STALLED_IGNORE_PRIVATE` is `true`.
- Type: Boolean
- Possible values: `true`, `false`
@@ -184,6 +184,65 @@
> [!WARNING]
> Setting `QUEUECLEANER__STALLED_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
#### **`QUEUECLEANER__SLOW_MAX_STRIKES`**
- Number of strikes before removing a slow download.
- Set to `0` to never remove slow downloads.
- A strike is given when an item is slow.
- Type: Integer
- Default: `0`
- Required: No.
> [!NOTE]
> If not set to `0`, the minimum value is `3`.
#### **`QUEUECLEANER__SLOW_RESET_STRIKES_ON_PROGRESS`**
- Controls whether to remove the given strikes if the download speed or estimated time are not slow anymore.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
#### **`QUEUECLEANER__SLOW_IGNORE_PRIVATE`**
- Controls whether to ignore slow downloads from private trackers.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
#### **`QUEUECLEANER__SLOW_DELETE_PRIVATE`**
- Controls whether slow downloads from private trackers should be removed from the download client.
- Has no effect if `QUEUECLEANER__SLOW_IGNORE_PRIVATE` is `true`.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
> [!WARNING]
> Setting `QUEUECLEANER__SLOW_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
#### **`QUEUECLEANER__SLOW_MIN_SPEED`**
- The minimum speed a download should have.
- Downloads receive strikes if their speed falls bellow this value.
- If not specified, downloads will not receive strikes for slow download speed.
- Type: String.
- Default: Empty.
- Required: No.
- Value examples: `1.5KB`, `400KB`, `2MB`
#### **`QUEUECLEANER__SLOW_MAX_TIME`**
- The maximum estimated hours a download should take to finish.
- Downloads receive strikes if their estimated finish time is above this value.
- If not specified (or `0`), downloads will not receive strikes for slow estimated finish time.
- Type: Integer.
- Default: `0`.
- Required: No.
#### **`QUEUECLEANER__SLOW_IGNORE_ABOVE_SIZE`**
- Downloads above this size will not be removed for being slow.
- Type: String.
- Default: Empty.
- Required: No.
- Value examples: `10KB`, `200MB`, `3GB`.
#
### Content Blocker settings
@@ -378,6 +437,12 @@
- Default: `http://localhost:8080`.
- Required: No.
#### **`QBITTORRENT__URL_BASE`**
- Adds a prefix to the qBittorrent url, such as `[QBITTORRENT__URL]/[QBITTORRENT__URL_BASE]/api`.
- Type: String.
- Default: Empty.
- Required: No.
#### **`QBITTORRENT__USERNAME`**
- Username for qBittorrent authentication.
- Type: String.
@@ -396,6 +461,12 @@
- Default: `http://localhost:8112`.
- Required: No.
#### **`DELUGE__URL_BASE`**
- Adds a prefix to the deluge json url, such as `[DELUGE__URL]/[DELUGE__URL_BASE]/json`.
- Type: String.
- Default: Empty.
- Required: No.
#### **`DELUGE__PASSWORD`**
- Password for Deluge authentication.
- Type: String.
@@ -408,6 +479,12 @@
- Default: `http://localhost:9091`.
- Required: No.
#### **`TRANSMISSION__URL_BASE`**
- Adds a prefix to the Transmission rpc url, such as `[TRANSMISSION__URL]/[TRANSMISSION__URL_BASE]/rpc`.
- Type: String.
- Default: `transmission`.
- Required: No.
#### **`TRANSMISSION__USERNAME`**
- Username for Transmission authentication.
- Type: String.
@@ -598,7 +675,14 @@
- Required: No.
#### **`NOTIFIARR__ON_STALLED_STRIKE`**
- Controls whether to notify when an item receives a stalled download strike.
- Controls whether to notify when an item receives a stalled download strike. This includes strikes for being stuck while downloading metadata.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
#### **`NOTIFIARR__ON_SLOW_STRIKE`**
- Controls whether to notify when an item receives a slow download strike. This includes strikes for having a low download speed or slow estimated finish time.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`