diff --git a/README.md b/README.md index 4e972405..6348fa2f 100644 --- a/README.md +++ b/README.md @@ -8,9 +8,19 @@ The tool supports both qBittorrent's built-in exclusion features and its own blo Refer to the [Environment variables](#Environment-variables) section for detailed configuration instructions and the [Setup](#Setup) section for an in-depth explanation of the cleanup process. +## Key features +- Marks unwanted files as skip/unwanted in the download client. +- Automatically strikes stalled or stuck downloads. +- Removes and blocks downloads that reached the maximum number of strikes or are marked as unwanted by the download client or by cleanuperr and triggers a search for removed downloads. + ## Important note -Only the **latest versions** of qBittorrent, Deluge, Sonarr etc. are supported, or earlier versions that have the same API as the latest version. +Only the **latest versions** of the following apps are supported, or earlier versions that have the same API as the latest version: +- qBittorrent +- Deluge +- Transmission +- Sonarr +- Radarr This tool is actively developed and still a work in progress. Join the Discord server if you want to reach out to me quickly (or just stay updated on new releases) so we can squash those pesky bugs together: @@ -28,12 +38,14 @@ This tool is actively developed and still a work in progress. Join the Discord s 2. **Queue cleaner** will: - Run every 5 minutes (or configured cron). - Process all items in the *arr queue. + - Check each queue item if it is **stalled (download speed is 0)**, **stuck in matadata downloading** or **failed to be imported**. + - If it is, the item receives a **strike** and will continue to accumulate strikes every time it meets any of these conditions. - Check each queue item if it meets one of the following condition in the download client: - **Marked as completed, but 0 bytes have been downloaded** (due to files being blocked by qBittorrent or the **content blocker**). - All associated files of are marked as **unwanted/skipped**. - If the item **DOES NOT** match the above criteria, it will be skipped. - - If the item **DOES** match the criteria: - - It will be removed from the *arr's queue. + - If the item **DOES** match the criteria or has received the **maximum number of strikes**: + - It will be removed from the *arr's queue and blocked. - It will be deleted from the download client. - A new search will be triggered for the *arr item. @@ -78,6 +90,8 @@ services: - QUEUECLEANER__ENABLED=true - QUEUECLEANER__RUNSEQUENTIALLY=true + - QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5 + - QUEUECLEANER__STALLED_MAX_STRIKES=5 - CONTENTBLOCKER__ENABLED=true - CONTENTBLOCKER__BLACKLIST__ENABLED=true @@ -91,21 +105,25 @@ services: - QBITTORRENT__USERNAME=user - QBITTORRENT__PASSWORD=pass # OR + # - DOWNLOAD_CLIENT=deluge # - DELUGE__URL=http://localhost:8112 # - DELUGE__PASSWORD=testing # OR + # - DOWNLOAD_CLIENT=transmission # - TRANSMISSION__URL=http://localhost:9091 # - TRANSMISSION__USERNAME=test # - TRANSMISSION__PASSWORD=testing - SONARR__ENABLED=true - SONARR__SEARCHTYPE=Episode + - SONARR__STALLED_MAX_STRIKES=5 - SONARR__INSTANCES__0__URL=http://localhost:8989 - SONARR__INSTANCES__0__APIKEY=secret1 - SONARR__INSTANCES__1__URL=http://localhost:8990 - SONARR__INSTANCES__1__APIKEY=secret2 - RADARR__ENABLED=true + - RADARR__STALLED_MAX_STRIKES=5 - RADARR__INSTANCES__0__URL=http://localhost:7878 - RADARR__INSTANCES__0__APIKEY=secret3 - RADARR__INSTANCES__1__URL=http://localhost:7879 @@ -123,11 +141,13 @@ services: | LOGGING__FILE__PATH | No | Directory where to save the log files | empty | | LOGGING__ENHANCED | No | Enhance logs whenever possible
A more detailed description is provided [here](variables.md#LOGGING__ENHANCED) | true | ||||| -| TRIGGERS__QUEUECLEANER | Yes if queue cleaner is enabled | [Quartz cron trigger](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) | 0 0/5 * * * ? | -| TRIGGERS__CONTENTBLOCKER | Yes if content blocker is enabled | [Quartz cron trigger](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) | 0 0/5 * * * ? | +| TRIGGERS__QUEUECLEANER | Yes if queue cleaner is enabled | [Quartz cron trigger](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html)
Can be a max of 1h interval | 0 0/5 * * * ? | +| TRIGGERS__CONTENTBLOCKER | Yes if content blocker is enabled | [Quartz cron trigger](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html)
Can be a max of 1h interval | 0 0/5 * * * ? | ||||| | QUEUECLEANER__ENABLED | No | Enable or disable the queue cleaner | true | | QUEUECLEANER__RUNSEQUENTIALLY | No | If set to true, the queue cleaner will run after the content blocker instead of running in parallel, streamlining the cleaning process | true | +| QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES | No | After how many strikes should a failed import be removed
0 means never | 0 | +| QUEUECLEANER__STALLED_MAX_STRIKES | No | After how many strikes should a stalled download be removed
0 means never | 0 | ||||| | CONTENTBLOCKER__ENABLED | No | Enable or disable the content blocker | false | | CONTENTBLOCKER__BLACKLIST__ENABLED | Yes if content blocker is enabled and whitelist is not enabled | Enable or disable the blacklist | false | @@ -149,12 +169,12 @@ services: ||||| | SONARR__ENABLED | No | Enable or disable Sonarr cleanup | true | | SONARR__SEARCHTYPE | No | What to search for after removing a queue item
Can be `Episode`, `Season` or `Series` | `Episode` | -| SONARR__INSTANCES__0__URL | Yes | First Sonarr instance url | http://localhost:8989 | -| SONARR__INSTANCES__0__APIKEY | Yes | First Sonarr instance API key | empty | +| SONARR__INSTANCES__0__URL | No | First Sonarr instance url | http://localhost:8989 | +| SONARR__INSTANCES__0__APIKEY | No | First Sonarr instance API key | empty | ||||| | RADARR__ENABLED | No | Enable or disable Radarr cleanup | false | -| RADARR__INSTANCES__0__URL | Yes | First Radarr instance url | http://localhost:8989 | -| RADARR__INSTANCES__0__APIKEY | Yes | First Radarr instance API key | empty | +| RADARR__INSTANCES__0__URL | No | First Radarr instance url | http://localhost:8989 | +| RADARR__INSTANCES__0__APIKEY | No | First Radarr instance API key | empty | # ### To be noted @@ -186,4 +206,10 @@ SONARR__INSTANCES____APIKEY ### Run as a Windows Service -Check out this stackoverflow answer on how to do it: https://stackoverflow.com/a/15719678 \ No newline at end of file +Check out this stackoverflow answer on how to do it: https://stackoverflow.com/a/15719678 + +## Credits +Special thanks for inspiration go to: +- [ThijmenGThN/swaparr](https://github.com/ThijmenGThN/swaparr) +- [ManiMatter/decluttarr](https://github.com/ManiMatter/decluttarr) +- [PaeyMoopy/sonarr-radarr-queue-cleaner](https://github.com/PaeyMoopy/sonarr-radarr-queue-cleaner) \ No newline at end of file diff --git a/code/Common/Common.csproj b/code/Common/Common.csproj index 395c6260..3290e709 100644 --- a/code/Common/Common.csproj +++ b/code/Common/Common.csproj @@ -7,7 +7,8 @@ - + + diff --git a/code/Common/Configuration/Arr/ArrConfig.cs b/code/Common/Configuration/Arr/ArrConfig.cs index 4ecf19a1..51f539bf 100644 --- a/code/Common/Configuration/Arr/ArrConfig.cs +++ b/code/Common/Configuration/Arr/ArrConfig.cs @@ -1,4 +1,4 @@ -namespace Common.Configuration.Arr; +namespace Common.Configuration.Arr; public abstract record ArrConfig { diff --git a/code/Common/Configuration/QueueCleaner/QueueCleanerConfig.cs b/code/Common/Configuration/QueueCleaner/QueueCleanerConfig.cs index c382cf9c..c3539799 100644 --- a/code/Common/Configuration/QueueCleaner/QueueCleanerConfig.cs +++ b/code/Common/Configuration/QueueCleaner/QueueCleanerConfig.cs @@ -1,4 +1,6 @@ -namespace Common.Configuration.QueueCleaner; +using Microsoft.Extensions.Configuration; + +namespace Common.Configuration.QueueCleaner; public sealed record QueueCleanerConfig : IJobConfig { @@ -7,6 +9,12 @@ public sealed record QueueCleanerConfig : IJobConfig public required bool Enabled { get; init; } public required bool RunSequentially { get; init; } + + [ConfigurationKeyName("IMPORT_FAILED_MAX_STRIKES")] + public ushort ImportFailedMaxStrikes { get; init; } + + [ConfigurationKeyName("STALLED_MAX_STRIKES")] + public ushort StalledMaxStrikes { get; init; } public void Validate() { diff --git a/code/Domain/Domain.csproj b/code/Domain/Domain.csproj index 75d3466b..de7d774e 100644 --- a/code/Domain/Domain.csproj +++ b/code/Domain/Domain.csproj @@ -7,7 +7,11 @@ - + + + + + diff --git a/code/Domain/Enums/StrikeType.cs b/code/Domain/Enums/StrikeType.cs new file mode 100644 index 00000000..1a04c151 --- /dev/null +++ b/code/Domain/Enums/StrikeType.cs @@ -0,0 +1,7 @@ +namespace Domain.Enums; + +public enum StrikeType +{ + Stalled, + ImportFailed +} \ No newline at end of file diff --git a/code/Domain/Models/Arr/Queue/QueueListResponse.cs b/code/Domain/Models/Arr/Queue/QueueListResponse.cs index 685bb4e0..7198e573 100644 --- a/code/Domain/Models/Arr/Queue/QueueListResponse.cs +++ b/code/Domain/Models/Arr/Queue/QueueListResponse.cs @@ -1,4 +1,4 @@ -namespace Domain.Arr.Queue; +namespace Domain.Models.Arr.Queue; public record QueueListResponse { diff --git a/code/Domain/Models/Arr/Queue/QueueRecord.cs b/code/Domain/Models/Arr/Queue/QueueRecord.cs index 298fad0a..0158cdad 100644 --- a/code/Domain/Models/Arr/Queue/QueueRecord.cs +++ b/code/Domain/Models/Arr/Queue/QueueRecord.cs @@ -1,4 +1,4 @@ -namespace Domain.Arr.Queue; +namespace Domain.Models.Arr.Queue; public record QueueRecord { diff --git a/code/Domain/Models/Arr/SonarrSearchItem.cs b/code/Domain/Models/Arr/SonarrSearchItem.cs index 4ac7eeaa..541d0280 100644 --- a/code/Domain/Models/Arr/SonarrSearchItem.cs +++ b/code/Domain/Models/Arr/SonarrSearchItem.cs @@ -1,9 +1,13 @@ -namespace Domain.Models.Arr; +using Common.Configuration.Arr; + +namespace Domain.Models.Arr; public sealed class SonarrSearchItem : SearchItem { public long SeriesId { get; set; } + public SonarrSearchType SearchType { get; set; } + public override bool Equals(object? obj) { if (obj is not SonarrSearchItem other) diff --git a/code/Domain/Models/Deluge/Response/DelugeMinimalStatus.cs b/code/Domain/Models/Deluge/Response/DelugeMinimalStatus.cs deleted file mode 100644 index 220a6fd4..00000000 --- a/code/Domain/Models/Deluge/Response/DelugeMinimalStatus.cs +++ /dev/null @@ -1,6 +0,0 @@ -namespace Domain.Models.Deluge.Response; - -public sealed record DelugeMinimalStatus -{ - public string? Hash { get; set; } -} \ No newline at end of file diff --git a/code/Domain/Models/Deluge/Response/TorrentStatus.cs b/code/Domain/Models/Deluge/Response/TorrentStatus.cs new file mode 100644 index 00000000..97d22a43 --- /dev/null +++ b/code/Domain/Models/Deluge/Response/TorrentStatus.cs @@ -0,0 +1,12 @@ +namespace Domain.Models.Deluge.Response; + +public sealed record TorrentStatus +{ + public string? Hash { get; set; } + + public string? State { get; set; } + + public string? Name { get; set; } + + public ulong Eta { get; set; } +} \ No newline at end of file diff --git a/code/Domain/Models/Sonarr/SonarrCommand.cs b/code/Domain/Models/Sonarr/SonarrCommand.cs index 0ff1f3ae..a0f8cc4b 100644 --- a/code/Domain/Models/Sonarr/SonarrCommand.cs +++ b/code/Domain/Models/Sonarr/SonarrCommand.cs @@ -1,4 +1,6 @@ -namespace Domain.Models.Sonarr; +using Common.Configuration.Arr; + +namespace Domain.Models.Sonarr; public sealed record SonarrCommand { @@ -9,4 +11,6 @@ public sealed record SonarrCommand public long? SeasonNumber { get; set; } public List? EpisodeIds { get; set; } + + public SonarrSearchType SearchType { get; set; } } \ No newline at end of file diff --git a/code/Executable/DependencyInjection/ConfigurationDI.cs b/code/Executable/DependencyInjection/ConfigurationDI.cs index d955f3b0..f237985b 100644 --- a/code/Executable/DependencyInjection/ConfigurationDI.cs +++ b/code/Executable/DependencyInjection/ConfigurationDI.cs @@ -3,6 +3,7 @@ using Common.Configuration.Arr; using Common.Configuration.ContentBlocker; using Common.Configuration.DownloadClient; using Common.Configuration.Logging; +using Common.Configuration.QueueCleaner; using Domain.Enums; namespace Executable.DependencyInjection; @@ -11,6 +12,7 @@ public static class ConfigurationDI { public static IServiceCollection AddConfiguration(this IServiceCollection services, IConfiguration configuration) => services + .Configure(configuration.GetSection(QueueCleanerConfig.SectionName)) .Configure(configuration.GetSection(ContentBlockerConfig.SectionName)) .Configure(configuration.GetSection(QBitConfig.SectionName)) .Configure(configuration.GetSection(DelugeConfig.SectionName)) diff --git a/code/Executable/DependencyInjection/LoggingDI.cs b/code/Executable/DependencyInjection/LoggingDI.cs index 5efbd6c6..0de177b4 100644 --- a/code/Executable/DependencyInjection/LoggingDI.cs +++ b/code/Executable/DependencyInjection/LoggingDI.cs @@ -52,6 +52,7 @@ public static class LoggingDI Log.Logger = logConfig .MinimumLevel.Is(level) .MinimumLevel.Override("Microsoft.Hosting.Lifetime", LogEventLevel.Information) + .MinimumLevel.Override("Microsoft.Extensions.Http", LogEventLevel.Warning) .MinimumLevel.Override("Quartz", LogEventLevel.Warning) .MinimumLevel.Override("System.Net.Http.HttpClient", LogEventLevel.Error) .WriteTo.Console(new ExpressionTemplate(consoleOutputTemplate.Replace("PAD", padding.ToString()))) diff --git a/code/Executable/DependencyInjection/MainDI.cs b/code/Executable/DependencyInjection/MainDI.cs index fe8bd46b..0e6d9c5c 100644 --- a/code/Executable/DependencyInjection/MainDI.cs +++ b/code/Executable/DependencyInjection/MainDI.cs @@ -19,6 +19,7 @@ public static class MainDI .AddLogging(builder => builder.ClearProviders().AddConsole()) .AddHttpClients() .AddConfiguration(configuration) + .AddMemoryCache() .AddServices() .AddQuartzServices(configuration); diff --git a/code/Executable/DependencyInjection/QuartzDI.cs b/code/Executable/DependencyInjection/QuartzDI.cs index 32a18bf9..534d9a8b 100644 --- a/code/Executable/DependencyInjection/QuartzDI.cs +++ b/code/Executable/DependencyInjection/QuartzDI.cs @@ -6,6 +6,7 @@ using Infrastructure.Verticals.ContentBlocker; using Infrastructure.Verticals.Jobs; using Infrastructure.Verticals.QueueCleaner; using Quartz; +using Quartz.Spi; namespace Executable.DependencyInjection; @@ -94,7 +95,20 @@ public static class QuartzDI { return; } + + var triggerObj = (IOperableTrigger)TriggerBuilder.Create() + .WithIdentity("ExampleTrigger") + .StartNow() + .WithCronSchedule(trigger) + .Build(); + var nextFireTimes = TriggerUtils.ComputeFireTimes(triggerObj, null, 2); + + if (nextFireTimes[1] - nextFireTimes[0] > TimeSpan.FromHours(1)) + { + throw new Exception($"{trigger} should have a fire time of maximum 1 hour"); + } + q.AddTrigger(opts => { opts.ForJob(typeName) diff --git a/code/Executable/DependencyInjection/ServicesDI.cs b/code/Executable/DependencyInjection/ServicesDI.cs index a5a62bbf..ed9ec24d 100644 --- a/code/Executable/DependencyInjection/ServicesDI.cs +++ b/code/Executable/DependencyInjection/ServicesDI.cs @@ -1,10 +1,10 @@ -using Executable.Jobs; -using Infrastructure.Verticals.Arr; +using Infrastructure.Verticals.Arr; using Infrastructure.Verticals.ContentBlocker; using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.DownloadClient.Deluge; using Infrastructure.Verticals.DownloadClient.QBittorrent; using Infrastructure.Verticals.DownloadClient.Transmission; +using Infrastructure.Verticals.ItemStriker; using Infrastructure.Verticals.QueueCleaner; namespace Executable.DependencyInjection; @@ -23,5 +23,6 @@ public static class ServicesDI .AddTransient() .AddTransient() .AddTransient() - .AddSingleton(); + .AddSingleton() + .AddSingleton(); } \ No newline at end of file diff --git a/code/Executable/Executable.csproj b/code/Executable/Executable.csproj index 3a1ecc4a..023e9807 100644 --- a/code/Executable/Executable.csproj +++ b/code/Executable/Executable.csproj @@ -9,15 +9,15 @@ - - + + - + - - + + diff --git a/code/Executable/Jobs/GenericJob.cs b/code/Executable/Jobs/GenericJob.cs index 8fc8adc0..48dc7272 100644 --- a/code/Executable/Jobs/GenericJob.cs +++ b/code/Executable/Jobs/GenericJob.cs @@ -11,7 +11,6 @@ public sealed class GenericJob : IJob private readonly ILogger> _logger; private readonly T _handler; - public GenericJob(ILogger> logger, T handler) { _logger = logger; diff --git a/code/Executable/appsettings.Development.json b/code/Executable/appsettings.Development.json index aea90457..f0c76faa 100644 --- a/code/Executable/appsettings.Development.json +++ b/code/Executable/appsettings.Development.json @@ -24,7 +24,9 @@ }, "QueueCleaner": { "Enabled": true, - "RunSequentially": true + "RunSequentially": true, + "IMPORT_FAILED_MAX_STRIKES": 5, + "STALLED_MAX_STRIKES": 5 }, "DOWNLOAD_CLIENT": "qbittorrent", "qBittorrent": { diff --git a/code/Executable/appsettings.json b/code/Executable/appsettings.json index 4c1011a3..cd298b77 100644 --- a/code/Executable/appsettings.json +++ b/code/Executable/appsettings.json @@ -24,7 +24,9 @@ }, "QueueCleaner": { "Enabled": true, - "RunSequentially": true + "RunSequentially": true, + "IMPORT_FAILED_MAX_STRIKES": 5, + "STALLED_MAX_STRIKES": 5 }, "DOWNLOAD_CLIENT": "qbittorrent", "qBittorrent": { diff --git a/code/Infrastructure/Infrastructure.csproj b/code/Infrastructure/Infrastructure.csproj index 69df953d..9e876b0b 100644 --- a/code/Infrastructure/Infrastructure.csproj +++ b/code/Infrastructure/Infrastructure.csproj @@ -13,7 +13,8 @@ - + + diff --git a/code/Infrastructure/Verticals/Arr/ArrClient.cs b/code/Infrastructure/Verticals/Arr/ArrClient.cs index a95bc46c..ac03ce46 100644 --- a/code/Infrastructure/Verticals/Arr/ArrClient.cs +++ b/code/Infrastructure/Verticals/Arr/ArrClient.cs @@ -1,8 +1,10 @@ -using Common.Configuration; -using Common.Configuration.Arr; +using Common.Configuration.Arr; using Common.Configuration.Logging; -using Domain.Arr.Queue; +using Common.Configuration.QueueCleaner; +using Domain.Enums; using Domain.Models.Arr; +using Domain.Models.Arr.Queue; +using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using Newtonsoft.Json; @@ -14,17 +16,28 @@ public abstract class ArrClient protected readonly ILogger _logger; protected readonly HttpClient _httpClient; protected readonly LoggingConfig _loggingConfig; + protected readonly QueueCleanerConfig _queueCleanerConfig; + protected readonly Striker _striker; - protected ArrClient(ILogger logger, IHttpClientFactory httpClientFactory, IOptions loggingConfig) + protected ArrClient( + ILogger logger, + IHttpClientFactory httpClientFactory, + IOptions loggingConfig, + IOptions queueCleanerConfig, + Striker striker + ) { _logger = logger; + _striker = striker; _httpClient = httpClientFactory.CreateClient(); _loggingConfig = loggingConfig.Value; + _queueCleanerConfig = queueCleanerConfig.Value; + _striker = striker; } public virtual async Task GetQueueItemsAsync(ArrInstance arrInstance, int page) { - Uri uri = new(arrInstance.Url, $"/api/v3/queue?page={page}&pageSize=200&sortKey=timeleft"); + Uri uri = new(arrInstance.Url, GetQueueUrlPath(page)); using HttpRequestMessage request = new(HttpMethod.Get, uri); SetApiKey(request, arrInstance.ApiKey); @@ -52,6 +65,28 @@ public abstract class ArrClient return queueResponse; } + public virtual bool ShouldRemoveFromQueue(QueueRecord record) + { + bool hasWarn() => record.TrackedDownloadStatus + .Equals("warning", StringComparison.InvariantCultureIgnoreCase); + bool isImportBlocked() => record.TrackedDownloadState + .Equals("importBlocked", StringComparison.InvariantCultureIgnoreCase); + bool isImportPending() => record.TrackedDownloadState + .Equals("importPending", StringComparison.InvariantCultureIgnoreCase); + + if (hasWarn() && (isImportBlocked() || isImportPending())) + { + return _striker.StrikeAndCheckLimit( + record.DownloadId, + record.Title, + _queueCleanerConfig.ImportFailedMaxStrikes, + StrikeType.ImportFailed + ); + } + + return false; + } + public virtual async Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord queueRecord) { Uri uri = new(arrInstance.Url, $"/api/v3/queue/{queueRecord.Id}?removeFromClient=true&blocklist=true&skipRedownload=true&changeCategory=false"); @@ -76,6 +111,25 @@ public abstract class ArrClient public abstract Task RefreshItemsAsync(ArrInstance arrInstance, ArrConfig config, HashSet? items); + public virtual bool IsRecordValid(QueueRecord record) + { + if (string.IsNullOrEmpty(record.DownloadId)) + { + _logger.LogDebug("skip | download id is null for {title}", record.Title); + return false; + } + + if (record.DownloadId.Equals(record.Title, StringComparison.InvariantCultureIgnoreCase)) + { + _logger.LogDebug("skip | item is not ready yet | {title}", record.Title); + return false; + } + + return true; + } + + protected abstract string GetQueueUrlPath(int page); + protected virtual void SetApiKey(HttpRequestMessage request, string apiKey) { request.Headers.Add("x-api-key", apiKey); diff --git a/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs b/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs index d5c01def..9f87a6c5 100644 --- a/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs +++ b/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs @@ -1,6 +1,6 @@ using Common.Configuration; using Common.Configuration.Arr; -using Domain.Arr.Queue; +using Domain.Models.Arr.Queue; using Microsoft.Extensions.Logging; namespace Infrastructure.Verticals.Arr; diff --git a/code/Infrastructure/Verticals/Arr/RadarrClient.cs b/code/Infrastructure/Verticals/Arr/RadarrClient.cs index a768a35a..a5bc7573 100644 --- a/code/Infrastructure/Verticals/Arr/RadarrClient.cs +++ b/code/Infrastructure/Verticals/Arr/RadarrClient.cs @@ -1,8 +1,12 @@ using System.Text; using Common.Configuration.Arr; using Common.Configuration.Logging; +using Common.Configuration.QueueCleaner; using Domain.Models.Arr; +using Domain.Models.Arr.Queue; using Domain.Models.Radarr; +using Infrastructure.Verticals.ItemStriker; +using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using Newtonsoft.Json; @@ -14,11 +18,18 @@ public sealed class RadarrClient : ArrClient public RadarrClient( ILogger logger, IHttpClientFactory httpClientFactory, - IOptions loggingConfig - ) : base(logger, httpClientFactory, loggingConfig) + IOptions loggingConfig, + IOptions queueCleanerConfig, + Striker striker + ) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker) { } + protected override string GetQueueUrlPath(int page) + { + return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownMovieItems=true&includeMovie=true"; + } + public override async Task RefreshItemsAsync(ArrInstance arrInstance, ArrConfig config, HashSet? items) { if (items?.Count is null or 0) @@ -59,6 +70,17 @@ public sealed class RadarrClient : ArrClient } } + public override bool IsRecordValid(QueueRecord record) + { + if (record.MovieId is 0) + { + _logger.LogDebug("skip | item information missing | {title}", record.Title); + return false; + } + + return base.IsRecordValid(record); + } + private static string GetSearchLog(Uri instanceUrl, RadarrCommand command, bool success, string? logContext) { string status = success ? "triggered" : "failed"; diff --git a/code/Infrastructure/Verticals/Arr/SonarrClient.cs b/code/Infrastructure/Verticals/Arr/SonarrClient.cs index 307eb155..2f6f119a 100644 --- a/code/Infrastructure/Verticals/Arr/SonarrClient.cs +++ b/code/Infrastructure/Verticals/Arr/SonarrClient.cs @@ -1,8 +1,12 @@ using System.Text; using Common.Configuration.Arr; using Common.Configuration.Logging; +using Common.Configuration.QueueCleaner; using Domain.Models.Arr; +using Domain.Models.Arr.Queue; using Domain.Models.Sonarr; +using Infrastructure.Verticals.ItemStriker; +using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using Newtonsoft.Json; @@ -14,10 +18,17 @@ public sealed class SonarrClient : ArrClient public SonarrClient( ILogger logger, IHttpClientFactory httpClientFactory, - IOptions loggingConfig - ) : base(logger, httpClientFactory, loggingConfig) + IOptions loggingConfig, + IOptions queueCleanerConfig, + Striker striker + ) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker) { } + + protected override string GetQueueUrlPath(int page) + { + return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownSeriesItems=true&includeSeries=true"; + } public override async Task RefreshItemsAsync(ArrInstance arrInstance, ArrConfig config, HashSet? items) { @@ -26,11 +37,9 @@ public sealed class SonarrClient : ArrClient return; } - SonarrConfig sonarrConfig = (SonarrConfig)config; - Uri uri = new(arrInstance.Url, "/api/v3/command"); - foreach (SonarrCommand command in GetSearchCommands(sonarrConfig.SearchType, items)) + foreach (SonarrCommand command in GetSearchCommands(items.Cast().ToHashSet())) { using HttpRequestMessage request = new(HttpMethod.Post, uri); request.Content = new StringContent( @@ -41,22 +50,33 @@ public sealed class SonarrClient : ArrClient SetApiKey(request, arrInstance.ApiKey); using HttpResponseMessage response = await _httpClient.SendAsync(request); - string? logContext = await ComputeCommandLogContextAsync(arrInstance, command, sonarrConfig.SearchType); + string? logContext = await ComputeCommandLogContextAsync(arrInstance, command, command.SearchType); try { response.EnsureSuccessStatusCode(); - _logger.LogInformation("{log}", GetSearchLog(sonarrConfig.SearchType, arrInstance.Url, command, true, logContext)); + _logger.LogInformation("{log}", GetSearchLog(command.SearchType, arrInstance.Url, command, true, logContext)); } catch { - _logger.LogError("{log}", GetSearchLog(sonarrConfig.SearchType, arrInstance.Url, command, false, logContext)); + _logger.LogError("{log}", GetSearchLog(command.SearchType, arrInstance.Url, command, false, logContext)); throw; } } } + public override bool IsRecordValid(QueueRecord record) + { + if (record.EpisodeId is 0 || record.SeriesId is 0) + { + _logger.LogDebug("skip | item information missing | {title}", record.Title); + return false; + } + + return base.IsRecordValid(record); + } + private static string GetSearchLog( SonarrSearchType searchType, Uri instanceUrl, @@ -191,7 +211,7 @@ public sealed class SonarrClient : ArrClient return JsonConvert.DeserializeObject(responseBody); } - private List GetSearchCommands(SonarrSearchType searchType, HashSet items) + private List GetSearchCommands(HashSet items) { const string episodeSearch = "EpisodeSearch"; const string seasonSearch = "SeasonSearch"; @@ -199,13 +219,13 @@ public sealed class SonarrClient : ArrClient List commands = new(); - foreach (SearchItem item in items) + foreach (SonarrSearchItem item in items) { - SonarrCommand command = searchType is SonarrSearchType.Episode + SonarrCommand command = item.SearchType is SonarrSearchType.Episode ? commands.FirstOrDefault() ?? new() { Name = episodeSearch, EpisodeIds = new() } : new(); - switch (searchType) + switch (item.SearchType) { case SonarrSearchType.Episode when command.EpisodeIds is null: command.EpisodeIds = [item.Id]; @@ -227,15 +247,16 @@ public sealed class SonarrClient : ArrClient break; default: - throw new ArgumentOutOfRangeException(nameof(searchType), searchType, null); + throw new ArgumentOutOfRangeException(nameof(item.SearchType), item.SearchType, null); } - if (searchType is SonarrSearchType.Episode && commands.Count > 0) + if (item.SearchType is SonarrSearchType.Episode && commands.Count > 0) { // only one command will be generated for episodes search continue; } + command.SearchType = item.SearchType; commands.Add(command); } diff --git a/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs b/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs index 880530f8..7c5d1d9a 100644 --- a/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs +++ b/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs @@ -1,7 +1,7 @@ using Common.Configuration; using Common.Configuration.Arr; -using Domain.Arr.Queue; using Domain.Enums; +using Domain.Models.Arr.Queue; using Infrastructure.Verticals.Arr; using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.Jobs; @@ -58,12 +58,4 @@ public sealed class ContentBlocker : GenericHandler } }); } - - private ArrClient GetClient(InstanceType type) => - type switch - { - InstanceType.Sonarr => _sonarrClient, - InstanceType.Radarr => _radarrClient, - _ => throw new NotImplementedException($"instance type {type} is not yet supported") - }; } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs index 484fd8f5..6e7134a9 100644 --- a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs @@ -1,44 +1,46 @@ -using Common.Configuration; using Common.Configuration.DownloadClient; +using Common.Configuration.QueueCleaner; using Domain.Models.Deluge.Response; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; namespace Infrastructure.Verticals.DownloadClient.Deluge; -public sealed class DelugeService : IDownloadService +public sealed class DelugeService : DownloadServiceBase { - private readonly ILogger _logger; private readonly DelugeClient _client; - private readonly FilenameEvaluator _filenameEvaluator; public DelugeService( ILogger logger, IOptions config, IHttpClientFactory httpClientFactory, - FilenameEvaluator filenameEvaluator - ) + IOptions queueCleanerConfig, + FilenameEvaluator filenameEvaluator, + Striker striker + ) : base(logger, queueCleanerConfig, filenameEvaluator, striker) { - _logger = logger; config.Value.Validate(); _client = new (config, httpClientFactory); - _filenameEvaluator = filenameEvaluator; } - public async Task LoginAsync() + public override async Task LoginAsync() { await _client.LoginAsync(); } - public async Task ShouldRemoveFromArrQueueAsync(string hash) + public override async Task ShouldRemoveFromArrQueueAsync(string hash) { hash = hash.ToLowerInvariant(); DelugeContents? contents = null; + + TorrentStatus? status = await GetTorrentStatus(hash); - if (!await HasMinimalStatus(hash)) + if (status?.Hash is null) { + _logger.LogDebug("failed to find torrent {hash} in the download client", hash); return false; } @@ -51,13 +53,7 @@ public sealed class DelugeService : IDownloadService _logger.LogDebug(exception, "failed to find torrent {hash} in the download client", hash); } - // if no files found, torrent might be stuck in Downloading metadata - if (contents?.Contents?.Count is null or 0) - { - return false; - } - - bool shouldRemove = true; + bool shouldRemove = contents?.Contents?.Count > 0; ProcessFiles(contents.Contents, (_, file) => { @@ -67,15 +63,18 @@ public sealed class DelugeService : IDownloadService } }); - return shouldRemove; + return shouldRemove || IsItemStuckAndShouldRemove(status); } - public async Task BlockUnwantedFilesAsync(string hash) + public override async Task BlockUnwantedFilesAsync(string hash) { hash = hash.ToLowerInvariant(); - if (!await HasMinimalStatus(hash)) + TorrentStatus? status = await GetTorrentStatus(hash); + + if (status?.Hash is null) { + _logger.LogDebug("failed to find torrent {hash} in the download client", hash); return; } @@ -126,22 +125,29 @@ public sealed class DelugeService : IDownloadService await _client.ChangeFilesPriority(hash, sortedPriorities); } - - private async Task HasMinimalStatus(string hash) + + private bool IsItemStuckAndShouldRemove(TorrentStatus status) { - DelugeMinimalStatus? status = await _client.SendRequest( - "web.get_torrent_status", - hash, - new[] { "hash" } - ); - - if (status?.Hash is null) + if (status.State is null || !status.State.Equals("Downloading", StringComparison.InvariantCultureIgnoreCase)) { - _logger.LogDebug("failed to find torrent {hash} in the download client", hash); return false; } - return true; + if (status.Eta > 0) + { + return false; + } + + return StrikeAndCheckLimit(status.Hash!, status.Name!); + } + + private async Task GetTorrentStatus(string hash) + { + return await _client.SendRequest( + "web.get_torrent_status", + hash, + new[] { "hash", "state", "name", "eta" } + ); } private static void ProcessFiles(Dictionary contents, Action processFile) @@ -161,7 +167,7 @@ public sealed class DelugeService : IDownloadService } } - public void Dispose() + public override void Dispose() { } } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs b/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs new file mode 100644 index 00000000..6ef71845 --- /dev/null +++ b/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs @@ -0,0 +1,42 @@ +using Common.Configuration.QueueCleaner; +using Domain.Enums; +using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.ItemStriker; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; + +namespace Infrastructure.Verticals.DownloadClient; + +public abstract class DownloadServiceBase : IDownloadService +{ + protected readonly ILogger _logger; + protected readonly QueueCleanerConfig _queueCleanerConfig; + protected readonly FilenameEvaluator _filenameEvaluator; + protected readonly Striker _striker; + + protected DownloadServiceBase( + ILogger logger, + IOptions queueCleanerConfig, + FilenameEvaluator filenameEvaluator, + Striker striker + ) + { + _logger = logger; + _queueCleanerConfig = queueCleanerConfig.Value; + _filenameEvaluator = filenameEvaluator; + _striker = striker; + } + + public abstract void Dispose(); + + public abstract Task LoginAsync(); + + public abstract Task ShouldRemoveFromArrQueueAsync(string hash); + + public abstract Task BlockUnwantedFilesAsync(string hash); + + protected bool StrikeAndCheckLimit(string hash, string itemName) + { + return _striker.StrikeAndCheckLimit(hash, itemName, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled); + } +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs b/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs index 7a09f605..d6b0de55 100644 --- a/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs @@ -1,33 +1,32 @@ -using Common.Configuration; -using Common.Configuration.DownloadClient; +using Common.Configuration.DownloadClient; +using Common.Configuration.QueueCleaner; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using QBittorrent.Client; namespace Infrastructure.Verticals.DownloadClient.QBittorrent; -public sealed class QBitService : IDownloadService +public sealed class QBitService : DownloadServiceBase { - private readonly ILogger _logger; private readonly QBitConfig _config; private readonly QBittorrentClient _client; - private readonly FilenameEvaluator _filenameEvaluator; public QBitService( ILogger logger, IOptions config, - FilenameEvaluator filenameEvaluator - ) + IOptions queueCleanerConfig, + FilenameEvaluator filenameEvaluator, + Striker striker + ) : base(logger, queueCleanerConfig, filenameEvaluator, striker) { - _logger = logger; _config = config.Value; _config.Validate(); _client = new(_config.Url); - _filenameEvaluator = filenameEvaluator; } - public async Task LoginAsync() + public override async Task LoginAsync() { if (string.IsNullOrEmpty(_config.Username) && string.IsNullOrEmpty(_config.Password)) { @@ -37,13 +36,14 @@ public sealed class QBitService : IDownloadService await _client.LoginAsync(_config.Username, _config.Password); } - public async Task ShouldRemoveFromArrQueueAsync(string hash) + public override async Task ShouldRemoveFromArrQueueAsync(string hash) { TorrentInfo? torrent = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] })) .FirstOrDefault(); if (torrent is null) { + _logger.LogDebug("failed to find torrent {hash} in the download client", hash); return false; } @@ -55,17 +55,16 @@ public sealed class QBitService : IDownloadService IReadOnlyList? files = await _client.GetTorrentContentsAsync(hash); - // if no files found, torrent might be stuck in Downloading metadata - if (files?.Count is null or 0) + // if all files are marked as skip + if (files?.Count is > 0 && files.All(x => x.Priority is TorrentContentPriority.Skip)) { - return false; + return true; } - // if all files are marked as skip - return files.All(x => x.Priority is TorrentContentPriority.Skip); + return IsItemStuckAndShouldRemove(torrent); } - public async Task BlockUnwantedFilesAsync(string hash) + public override async Task BlockUnwantedFilesAsync(string hash) { IReadOnlyList? files = await _client.GetTorrentContentsAsync(hash); @@ -91,8 +90,20 @@ public sealed class QBitService : IDownloadService } } - public void Dispose() + public override void Dispose() { _client.Dispose(); } + + private bool IsItemStuckAndShouldRemove(TorrentInfo torrent) + { + if (torrent.State is not TorrentState.StalledDownload and not TorrentState.FetchingMetadata + and not TorrentState.ForcedFetchingMetadata) + { + // ignore other states + return false; + } + + return StrikeAndCheckLimit(torrent.Hash, torrent.Name); + } } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs b/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs index 764d4e19..801968eb 100644 --- a/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs @@ -1,6 +1,7 @@ -using Common.Configuration; -using Common.Configuration.DownloadClient; +using Common.Configuration.DownloadClient; +using Common.Configuration.QueueCleaner; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using Transmission.API.RPC; @@ -9,21 +10,20 @@ using Transmission.API.RPC.Entity; namespace Infrastructure.Verticals.DownloadClient.Transmission; -public sealed class TransmissionService : IDownloadService +public sealed class TransmissionService : DownloadServiceBase { - private readonly ILogger _logger; private readonly TransmissionConfig _config; private readonly Client _client; - private readonly FilenameEvaluator _filenameEvaluator; private TorrentInfo[]? _torrentsCache; public TransmissionService( ILogger logger, IOptions config, - FilenameEvaluator filenameEvaluator - ) + IOptions queueCleanerConfig, + FilenameEvaluator filenameEvaluator, + Striker striker + ) : base(logger, queueCleanerConfig, filenameEvaluator, striker) { - _logger = logger; _config = config.Value; _config.Validate(); _client = new( @@ -31,44 +31,45 @@ public sealed class TransmissionService : IDownloadService login: _config.Username, password: _config.Password ); - _filenameEvaluator = filenameEvaluator; } - public async Task LoginAsync() + public override async Task LoginAsync() { await _client.GetSessionInformationAsync(); } - public async Task ShouldRemoveFromArrQueueAsync(string hash) + public override async Task ShouldRemoveFromArrQueueAsync(string hash) { TorrentInfo? torrent = await GetTorrentAsync(hash); - // if no files found, torrent might be stuck in Downloading metadata - if (torrent?.FileStats?.Length is null or 0) + if (torrent is null) { + _logger.LogDebug("failed to find torrent {hash} in the download client", hash); return false; } + + bool shouldRemove = torrent.FileStats?.Length > 0; foreach (TransmissionTorrentFileStats? stats in torrent.FileStats ?? []) { if (!stats.Wanted.HasValue) { // if any files stats are missing, do not remove - return false; + shouldRemove = false; } if (stats.Wanted.HasValue && stats.Wanted.Value) { // if any files are wanted, do not remove - return false; + shouldRemove = false; } } // remove if all files are unwanted - return true; + return shouldRemove || IsItemStuckAndShouldRemove(torrent); } - public async Task BlockUnwantedFilesAsync(string hash) + public override async Task BlockUnwantedFilesAsync(string hash) { TorrentInfo? torrent = await GetTorrentAsync(hash); @@ -108,10 +109,26 @@ public sealed class TransmissionService : IDownloadService FilesUnwanted = unwantedFiles.ToArray(), }); } - - public void Dispose() + + public override void Dispose() { } + + private bool IsItemStuckAndShouldRemove(TorrentInfo torrent) + { + if (torrent.Status is not 4) + { + // not in downloading state + return false; + } + + if (torrent.Eta > 0) + { + return false; + } + + return StrikeAndCheckLimit(torrent.HashString!, torrent.Name!); + } private async Task GetTorrentAsync(string hash) { @@ -120,7 +137,15 @@ public sealed class TransmissionService : IDownloadService if (_torrentsCache is null || torrent is null) { - string[] fields = [TorrentFields.FILES, TorrentFields.FILE_STATS, TorrentFields.HASH_STRING, TorrentFields.ID]; + string[] fields = [ + TorrentFields.FILES, + TorrentFields.FILE_STATS, + TorrentFields.HASH_STRING, + TorrentFields.ID, + TorrentFields.ETA, + TorrentFields.NAME, + TorrentFields.STATUS + ]; // refresh cache _torrentsCache = (await _client.TorrentGetAsync(fields)) diff --git a/code/Infrastructure/Verticals/ItemStriker/Striker.cs b/code/Infrastructure/Verticals/ItemStriker/Striker.cs new file mode 100644 index 00000000..d07a3ae4 --- /dev/null +++ b/code/Infrastructure/Verticals/ItemStriker/Striker.cs @@ -0,0 +1,57 @@ +using Domain.Enums; +using Microsoft.Extensions.Caching.Memory; +using Microsoft.Extensions.Logging; + +namespace Infrastructure.Verticals.ItemStriker; + +public class Striker +{ + private readonly ILogger _logger; + private readonly IMemoryCache _cache; + private readonly MemoryCacheEntryOptions _cacheOptions; + + public Striker(ILogger logger, IMemoryCache cache) + { + _logger = logger; + _cache = cache; + _cacheOptions = new MemoryCacheEntryOptions() + .SetSlidingExpiration(TimeSpan.FromHours(2)); + } + + public bool StrikeAndCheckLimit(string hash, string itemName, ushort maxStrikes, StrikeType strikeType) + { + if (maxStrikes is 0) + { + return false; + } + + string key = $"{strikeType.ToString()}_{hash}"; + + if (!_cache.TryGetValue(key, out int? strikeCount)) + { + strikeCount = 1; + } + else + { + ++strikeCount; + } + + _logger.LogDebug("item on strike number {strike} | reason {reason} | {name}", strikeCount, strikeType.ToString(), itemName); + _cache.Set(key, strikeCount, _cacheOptions); + + if (strikeCount < maxStrikes) + { + return false; + } + + if (strikeCount > maxStrikes) + { + _logger.LogWarning("blocked item keeps coming back | {name}", itemName); + _logger.LogWarning("be sure to enable \"Reject Blocklisted Torrent Hashes While Grabbing\" on your indexers to reject blocked items"); + } + + _logger.LogInformation("removing item with max strikes | reason {reason} | {name}", strikeType.ToString(), itemName); + + return true; + } +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Jobs/GenericHandler.cs b/code/Infrastructure/Verticals/Jobs/GenericHandler.cs index e089b0e7..864abd0c 100644 --- a/code/Infrastructure/Verticals/Jobs/GenericHandler.cs +++ b/code/Infrastructure/Verticals/Jobs/GenericHandler.cs @@ -1,10 +1,11 @@ using Common.Configuration.Arr; -using Domain.Arr.Queue; using Domain.Enums; using Domain.Models.Arr; +using Domain.Models.Arr.Queue; using Infrastructure.Verticals.Arr; using Infrastructure.Verticals.DownloadClient; using Microsoft.Extensions.Logging; +using Newtonsoft.Json; namespace Infrastructure.Verticals.Jobs; @@ -88,18 +89,27 @@ public abstract class GenericHandler : IDisposable _ => throw new NotImplementedException($"instance type {type} is not yet supported") }; - protected SearchItem GetRecordSearchItem(InstanceType type, QueueRecord record) => - type switch + protected SearchItem GetRecordSearchItem(InstanceType type, QueueRecord record, bool isPack = false) + { + return type switch { - InstanceType.Sonarr when _sonarrConfig.SearchType is SonarrSearchType.Episode => new SonarrSearchItem + InstanceType.Sonarr when _sonarrConfig.SearchType is SonarrSearchType.Episode && !isPack => new SonarrSearchItem { Id = record.EpisodeId, - SeriesId = record.SeriesId + SeriesId = record.SeriesId, + SearchType = SonarrSearchType.Episode + }, + InstanceType.Sonarr when _sonarrConfig.SearchType is SonarrSearchType.Episode && isPack => new SonarrSearchItem + { + Id = record.SeasonNumber, + SeriesId = record.SeriesId, + SearchType = SonarrSearchType.Season }, InstanceType.Sonarr when _sonarrConfig.SearchType is SonarrSearchType.Season => new SonarrSearchItem { Id = record.SeasonNumber, - SeriesId = record.SeriesId + SeriesId = record.SeriesId, + SearchType = SonarrSearchType.Series }, InstanceType.Sonarr when _sonarrConfig.SearchType is SonarrSearchType.Series => new SonarrSearchItem { @@ -111,4 +121,5 @@ public abstract class GenericHandler : IDisposable }, _ => throw new NotImplementedException($"instance type {type} is not yet supported") }; + } } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs b/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs index e9e7ae15..ec7098ed 100644 --- a/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs +++ b/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs @@ -1,8 +1,8 @@ -using Common.Configuration; using Common.Configuration.Arr; -using Domain.Arr.Queue; +using Common.Configuration.QueueCleaner; using Domain.Enums; using Domain.Models.Arr; +using Domain.Models.Arr.Queue; using Infrastructure.Verticals.Arr; using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.Jobs; @@ -29,34 +29,45 @@ public sealed class QueueCleaner : GenericHandler { HashSet itemsToBeRefreshed = []; ArrClient arrClient = GetClient(instanceType); + ArrConfig arrConfig = GetConfig(instanceType); await _arrArrQueueIterator.Iterate(arrClient, instance, async items => { - foreach (QueueRecord record in items) + var groups = items + .GroupBy(x => x.DownloadId) + .ToList(); + + foreach (var group in groups) { + if (group.Any(x => !arrClient.IsRecordValid(x))) + { + continue; + } + + QueueRecord record = group.First(); + if (record.Protocol is not "torrent") { continue; } - if (string.IsNullOrEmpty(record.DownloadId)) + if (!arrClient.IsRecordValid(record)) { - _logger.LogDebug("skip | download id is null for {title}", record.Title); continue; } - if (!await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId)) + if (!arrClient.ShouldRemoveFromQueue(record) && !await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId)) { _logger.LogInformation("skip | {title}", record.Title); continue; } - - itemsToBeRefreshed.Add(GetRecordSearchItem(instanceType, record)); + + itemsToBeRefreshed.Add(GetRecordSearchItem(instanceType, record, group.Count() > 1)); await arrClient.DeleteQueueItemAsync(instance, record); } }); - await arrClient.RefreshItemsAsync(instance, GetConfig(instanceType), itemsToBeRefreshed); + await arrClient.RefreshItemsAsync(instance, arrConfig, itemsToBeRefreshed); } } \ No newline at end of file diff --git a/code/test/data/nginx/sonarr.xml b/code/test/data/nginx/sonarr.xml index 344d8fbf..0ec28cd4 100644 --- a/code/test/data/nginx/sonarr.xml +++ b/code/test/data/nginx/sonarr.xml @@ -44,6 +44,17 @@ Sat, 24 Sep 2022 22:02:13 -0300 + + Top.Gear.S23E03.720p.x265.HDTV.HEVC.-.YSTEAM + Test + 4138858110 + magnet:?xt=urn:btih:cf92cf859b110af0ad3d94b846e006828417b193&dn=TPG.2303.720p.x265.yourserie.com.mkv + + 174674a88c8947f6f5057ac3f81efde384ed216c2de43564ec450f2cb4677554 + + Sat, 24 Sep 2022 22:02:13 -0300 + + Top.Gear.S23E01.720p.x265.HDTV.HEVC.-.YSTEAM Test @@ -65,5 +76,16 @@ Sat, 24 Sep 2022 22:02:13 -0300 + + + Sherlock.S01.1080p.BluRay.DD5.1.x264-DON + Test + 4138858110 + http://nginx/custom/sonarr_bad_pack.torrent + + 174674a88c8947f6f9057ac3f82efde384ed216cade43564ec45gf2cb4677554 + + Sat, 24 Sep 2022 22:02:13 -0300 + \ No newline at end of file diff --git a/code/test/data/nginx/sonarr_bad_pack.torrent b/code/test/data/nginx/sonarr_bad_pack.torrent new file mode 100644 index 00000000..c920381a --- /dev/null +++ b/code/test/data/nginx/sonarr_bad_pack.torrent @@ -0,0 +1 @@ +d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1734129464e4:infod5:filesld6:lengthi7e4:pathl47:Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkveee4:name40:Sherlock.S01.1080p.BluRay.DD5.1.x264-DON12:piece lengthi262144e6:pieces20:/˜ŽrÎèçƒlY€„·°|¶7ee \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/fa800a7d7c443a2c3561d1f8f393c089036dade1.torrent b/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/fa800a7d7c443a2c3561d1f8f393c089036dade1.torrent new file mode 100644 index 00000000..baae24e3 --- /dev/null +++ b/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/fa800a7d7c443a2c3561d1f8f393c089036dade1.torrent @@ -0,0 +1 @@ +d10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1734129464e4:infod5:filesld6:lengthi7e4:pathl47:Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkveee4:name40:Sherlock.S01.1080p.BluRay.DD5.1.x264-DON12:piece lengthi262144e6:pieces20:/˜ŽrÎèçƒlY€„·°|¶7ee \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/queue b/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/queue index a528236f..e5d0f16d 100644 --- a/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/queue +++ b/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/queue @@ -1,5 +1 @@ -2b2ec156461d77bc48b8fe4d62cede50dcdff8e0 -a4a1d1dd1db25763caa8f5e4d25ad72ef304094b -b72541215214be2a1d96ef6b29ca1305f5e5e1f6 -59ab2bc053430fe53e06a93e2eadb7acb6a6bf2c 11cece7f8721c484126b66f609d52738ff1bbf1e diff --git a/code/test/data/qbittorrent-bad/config/qBittorrent/GeoDB/dbip-country-lite.mmdb b/code/test/data/qbittorrent-bad/config/qBittorrent/GeoDB/dbip-country-lite.mmdb index 5e65de0b..5157c0fd 100644 Binary files a/code/test/data/qbittorrent-bad/config/qBittorrent/GeoDB/dbip-country-lite.mmdb and b/code/test/data/qbittorrent-bad/config/qBittorrent/GeoDB/dbip-country-lite.mmdb differ diff --git a/code/test/data/qbittorrent-bad/config/qBittorrent/qBittorrent-data.conf b/code/test/data/qbittorrent-bad/config/qBittorrent/qBittorrent-data.conf index 66611c87..25fa8584 100644 --- a/code/test/data/qbittorrent-bad/config/qBittorrent/qBittorrent-data.conf +++ b/code/test/data/qbittorrent-bad/config/qBittorrent/qBittorrent-data.conf @@ -1,2 +1,2 @@ [Stats] -AllStats=@Variant(\0\0\0\x1c\0\0\0\x2\0\0\0\x12\0\x41\0l\0l\0t\0i\0m\0\x65\0\x44\0L\0\0\0\x4\0\0\0\0\0\x61\xc0\xdf\0\0\0\x12\0\x41\0l\0l\0t\0i\0m\0\x65\0U\0L\0\0\0\x4\0\0\0\0\0\x9b\xf9\x8a) +AllStats=@Variant(\0\0\0\x1c\0\0\0\x2\0\0\0\x12\0\x41\0l\0l\0t\0i\0m\0\x65\0U\0L\0\0\0\x4\0\0\0\0\0\x9dm\x4\0\0\0\x12\0\x41\0l\0l\0t\0i\0m\0\x65\0\x44\0L\0\0\0\x4\0\0\0\0\0\x62_.) diff --git a/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkv b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkv new file mode 100644 index 00000000..d6def41f --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkv @@ -0,0 +1 @@ +episode \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkv b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkv new file mode 100644 index 00000000..d6def41f --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkv @@ -0,0 +1 @@ +episode \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkv b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkv new file mode 100644 index 00000000..d6def41f --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkv @@ -0,0 +1 @@ +episode \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/downloads/sonarr_bad_pack.torrent b/code/test/data/qbittorrent-bad/downloads/sonarr_bad_pack.torrent new file mode 100644 index 00000000..c920381a --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/sonarr_bad_pack.torrent @@ -0,0 +1 @@ +d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1734129464e4:infod5:filesld6:lengthi7e4:pathl47:Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkveee4:name40:Sherlock.S01.1080p.BluRay.DD5.1.x264-DON12:piece lengthi262144e6:pieces20:/˜ŽrÎèçƒlY€„·°|¶7ee \ No newline at end of file diff --git a/code/test/data/sonarr/config/Sentry/07ADDC43B5669C4F6DB64F2EF2B23B3FEEDFE865/.session b/code/test/data/sonarr/config/Sentry/07ADDC43B5669C4F6DB64F2EF2B23B3FEEDFE865/.session new file mode 100644 index 00000000..0644f275 --- /dev/null +++ b/code/test/data/sonarr/config/Sentry/07ADDC43B5669C4F6DB64F2EF2B23B3FEEDFE865/.session @@ -0,0 +1 @@ +{"update":{"sid":"4ee000d424144e078e7f3ef208e30647","did":"1df9f2cc-17dc-4130-9753-9b694f82f1b5","init":true,"started":"2024-12-13T22:41:57.8197572+00:00","timestamp":"2024-12-13T22:41:57.8202577+00:00","seq":0,"duration":0,"errors":0,"attrs":{"release":"4.0.10.2544-main","environment":"main"}}} \ No newline at end of file diff --git a/code/test/data/sonarr/config/logs.db b/code/test/data/sonarr/config/logs.db index 6999d2eb..1198ce52 100644 Binary files a/code/test/data/sonarr/config/logs.db and b/code/test/data/sonarr/config/logs.db differ diff --git a/code/test/data/sonarr/config/logs.db-shm b/code/test/data/sonarr/config/logs.db-shm index 9617b44b..9558dac0 100644 Binary files a/code/test/data/sonarr/config/logs.db-shm and b/code/test/data/sonarr/config/logs.db-shm differ diff --git a/code/test/data/sonarr/config/logs.db-wal b/code/test/data/sonarr/config/logs.db-wal index 25ae7297..443f505f 100644 Binary files a/code/test/data/sonarr/config/logs.db-wal and b/code/test/data/sonarr/config/logs.db-wal differ diff --git a/code/test/data/sonarr/config/sonarr.db b/code/test/data/sonarr/config/sonarr.db index 5db88d2c..7802f0f1 100644 Binary files a/code/test/data/sonarr/config/sonarr.db and b/code/test/data/sonarr/config/sonarr.db differ diff --git a/code/test/data/sonarr/config/sonarr.db-shm b/code/test/data/sonarr/config/sonarr.db-shm index be74b3a2..f5c93f42 100644 Binary files a/code/test/data/sonarr/config/sonarr.db-shm and b/code/test/data/sonarr/config/sonarr.db-shm differ diff --git a/code/test/data/sonarr/config/sonarr.db-wal b/code/test/data/sonarr/config/sonarr.db-wal index 475ce064..ebd30864 100644 Binary files a/code/test/data/sonarr/config/sonarr.db-wal and b/code/test/data/sonarr/config/sonarr.db-wal differ diff --git a/code/test/data/sonarr/config/sonarr.pid b/code/test/data/sonarr/config/sonarr.pid index bc768da7..70e1a64c 100644 --- a/code/test/data/sonarr/config/sonarr.pid +++ b/code/test/data/sonarr/config/sonarr.pid @@ -1 +1 @@ -146 \ No newline at end of file +144 \ No newline at end of file diff --git a/code/test/docker-compose.yml b/code/test/docker-compose.yml index 910eab91..e7fc73f5 100644 --- a/code/test/docker-compose.yml +++ b/code/test/docker-compose.yml @@ -172,7 +172,7 @@ services: container_name: cleanuperr environment: - LOGGING__LOGLEVEL=Debug - - LOGGING__FILE__ENABLED=false + - LOGGING__FILE__ENABLED=true - LOGGING__FILE__PATH=/var/logs - LOGGING__ENHANCED=true @@ -181,6 +181,8 @@ services: - QUEUECLEANER__ENABLED=true - QUEUECLEANER__RUNSEQUENTIALLY=true + - QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5 + - QUEUECLEANER__STALLED_MAX_STRIKES=5 - CONTENTBLOCKER__ENABLED=true - CONTENTBLOCKER__BLACKLIST__ENABLED=true