diff --git a/README.md b/README.md index 2ce667ed..d82f8203 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,7 @@ This tool is actively developed and still a work in progress, so using the `late 2. **Queue cleaner** will: - Run every 5 minutes (or configured cron, or right after `content blocker`). - Process all items in the *arr queue. - - Check each queue item if it is **stalled (download speed is 0)**, **stuck in matadata downloading** or **failed to be imported**. + - Check each queue item if it is **stalled (download speed is 0)**, **stuck in metadata downloading** or **failed to be imported**. - If it is, the item receives a **strike** and will continue to accumulate strikes every time it meets any of these conditions. - Check each queue item if it meets one of the following condition in the download client: - **Marked as completed, but 0 bytes have been downloaded** (due to files being blocked by qBittorrent or the **content blocker**). @@ -71,6 +71,9 @@ This tool is actively developed and still a work in progress, so using the `late - It will be removed from the *arr's queue and blocked. - It will be deleted from the download client. - A new search will be triggered for the *arr item. +3. **Download cleaner** will: + - Run every hour (or configured cron). + - Automatically clean up downloads that have been seeding for a certain amount of time. # Setup @@ -114,6 +117,8 @@ services: volumes: - ./cleanuperr/logs:/var/logs environment: + - DRY_RUN=false + - LOGGING__LOGLEVEL=Information - LOGGING__FILE__ENABLED=false - LOGGING__FILE__PATH=/var/logs/ @@ -121,6 +126,7 @@ services: - TRIGGERS__QUEUECLEANER=0 0/5 * * * ? - TRIGGERS__CONTENTBLOCKER=0 0/5 * * * ? + - TRIGGERS__DOWNLOADCLEANER=0 0 * * * ? - QUEUECLEANER__ENABLED=true - QUEUECLEANER__RUNSEQUENTIALLY=true @@ -138,6 +144,17 @@ services: - CONTENTBLOCKER__IGNORE_PRIVATE=false - CONTENTBLOCKER__DELETE_PRIVATE=false + - DOWNLOADCLEANER__ENABLED=true + - DOWNLOADCLEANER__DELETE_PRIVATE=false + - DOWNLOADCLEANER__CATEGORIES__0__NAME=tv-sonarr + - DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO=-1 + - DOWNLOADCLEANER__CATEGORIES__0__MIN_SEED_TIME=0 + - DOWNLOADCLEANER__CATEGORIES__0__MAX_SEED_TIME=240 + - DOWNLOADCLEANER__CATEGORIES__1__NAME=radarr + - DOWNLOADCLEANER__CATEGORIES__1__MAX_RATIO=-1 + - DOWNLOADCLEANER__CATEGORIES__1__MIN_SEED_TIME=0 + - DOWNLOADCLEANER__CATEGORIES__1__MAX_SEED_TIME=240 + - DOWNLOAD_CLIENT=none # OR # - DOWNLOAD_CLIENT=qBittorrent @@ -179,139 +196,25 @@ services: - LIDARR__INSTANCES__1__URL=http://radarr:8687 - LIDARR__INSTANCES__1__APIKEY=secret6 - # - NOTIFIARR__ON_IMPORT_FAILED_STRIKE=false - # - NOTIFIARR__ON_STALLED_STRIKE=false - # - NOTIFIARR__ON_QUEUE_ITEM_DELETE=false - # - NOTIFIARR__API_KEY=notifiarr_secret - # - NOTIFIARR__CHANNEL_ID=discord_channel_id + - NOTIFIARR__ON_IMPORT_FAILED_STRIKE=true + - NOTIFIARR__ON_STALLED_STRIKE=true + - NOTIFIARR__ON_QUEUE_ITEM_DELETED=true + - NOTIFIARR__ON_DOWNLOAD_CLEANED=true + - NOTIFIARR__API_KEY=notifiarr_secret + - NOTIFIARR__CHANNEL_ID=discord_channel_id ``` ## Environment variables -### General variables -
- Click here - -| Variable | Required | Description | Default value | -|---|---|---|---| -| LOGGING__LOGLEVEL | No | Can be `Verbose`, `Debug`, `Information`, `Warning`, `Error` or `Fatal`. | `Information` | -| LOGGING__FILE__ENABLED | No | Enable or disable logging to file. | false | -| LOGGING__FILE__PATH | No | Directory where to save the log files. | empty | -| LOGGING__ENHANCED | No | Enhance logs whenever possible.
A more detailed description is provided. [here](variables.md#LOGGING__ENHANCED) | true | -||||| -| TRIGGERS__QUEUECLEANER | Yes if queue cleaner is enabled | - [Quartz cron trigger](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html).
- Can be a max of 6h interval.
- **Is ignored if `QUEUECLEANER__RUNSEQUENTIALLY=true` and `CONTENTBLOCKER__ENABLED=true`**. | 0 0/5 * * * ? | -| TRIGGERS__CONTENTBLOCKER | Yes if content blocker is enabled | - [Quartz cron trigger](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html).
- Can be a max of 6h interval. | 0 0/5 * * * ? | -||||| -| QUEUECLEANER__ENABLED | No | Enable or disable the queue cleaner. | true | -| QUEUECLEANER__RUNSEQUENTIALLY | No | If set to true, the queue cleaner will run after the content blocker instead of running in parallel, streamlining the cleaning process. | true | -| QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES | No | - After how many strikes should a failed import be removed.
- 0 means never. | 0 | -| QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE | No | Whether to ignore failed imports from private trackers. | false | -| QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE | No | - Whether to delete failed imports of private downloads from the download client.
- Does not have any effect if `QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE` is `true`.
- **Set this to `true` if you don't care about seeding, ratio, H&R and potentially losing your tracker account.** | false | -| QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__0 | No | - First pattern to look for when an import is failed.
- If the specified message pattern is found, the item is skipped. | empty | -| QUEUECLEANER__STALLED_MAX_STRIKES | No | - After how many strikes should a stalled download be removed.
- 0 means never. | 0 | -| QUEUECLEANER__STALLED_RESET_STRIKES_ON_PROGRESS | No | Whether to remove strikes if any download progress was made since last checked. | false | -| QUEUECLEANER__STALLED_IGNORE_PRIVATE | No | Whether to ignore stalled downloads from private trackers. | false | -| QUEUECLEANER__STALLED_DELETE_PRIVATE | No | - Whether to delete stalled private downloads from the download client.
- Does not have any effect if `QUEUECLEANER__STALLED_IGNORE_PRIVATE` is `true`.
- **Set this to `true` if you don't care about seeding, ratio, H&R and potentially losing your tracker account.** | false | -||||| -| CONTENTBLOCKER__ENABLED | No | Enable or disable the content blocker. | false | -| CONTENTBLOCKER__IGNORE_PRIVATE | No | Whether to ignore downloads from private trackers. | false | -| CONTENTBLOCKER__DELETE_PRIVATE | No | - Whether to delete private downloads that have all files blocked from the download client.
- Does not have any effect if `CONTENTBLOCKER__IGNORE_PRIVATE` is `true`.
- **Set this to `true` if you don't care about seeding, ratio, H&R and potentially losing your tracker account.** | false | -
- -### Download client variables -
- Click here - -| Variable | Required | Description | Default value | -|---|---|---|---| -| DOWNLOAD_CLIENT | No | Download client that is used by *arrs
Can be `qbittorrent`, `deluge`, `transmission` or `none` | `none` | -| QBITTORRENT__URL | No | qBittorrent instance url | http://localhost:8112 | -| QBITTORRENT__USERNAME | No | qBittorrent user | empty | -| QBITTORRENT__PASSWORD | No | qBittorrent password | empty | -||||| -| DELUGE__URL | No | Deluge instance url | http://localhost:8080 | -| DELUGE__PASSWORD | No | Deluge password | empty | -||||| -| TRANSMISSION__URL | No | Transmission instance url | http://localhost:9091 | -| TRANSMISSION__USERNAME | No | Transmission user | empty | -| TRANSMISSION__PASSWORD | No | Transmission password | empty | -
- -### Arr variables -
- Click here - -| Variable | Required | Description | Default value | -|---|---|---|---| -| SONARR__ENABLED | No | Enable or disable Sonarr cleanup | false | -| SONARR__BLOCK__TYPE | No | Block type
Can be `blacklist` or `whitelist` | `blacklist` | -| SONARR__BLOCK__PATH | No | Path to the blocklist (local file or url)
Needs to be json compatible | empty | -| SONARR__SEARCHTYPE | No | What to search for after removing a queue item
Can be `Episode`, `Season` or `Series` | `Episode` | -| SONARR__INSTANCES__0__URL | No | First Sonarr instance url | http://localhost:8989 | -| SONARR__INSTANCES__0__APIKEY | No | First Sonarr instance API key | empty | -||||| -| RADARR__ENABLED | No | Enable or disable Radarr cleanup | false | -| RADARR__BLOCK__TYPE | No | Block type
Can be `blacklist` or `whitelist` | `blacklist` | -| RADARR__BLOCK__PATH | No | Path to the blocklist (local file or url)
Needs to be json compatible | empty | -| RADARR__INSTANCES__0__URL | No | First Radarr instance url | http://localhost:7878 | -| RADARR__INSTANCES__0__APIKEY | No | First Radarr instance API key | empty | -||||| -| LIDARR__ENABLED | No | Enable or disable LIDARR cleanup | false | -| LIDARR__BLOCK__TYPE | No | Block type
Can be `blacklist` or `whitelist` | `blacklist` | -| LIDARR__BLOCK__PATH | No | Path to the blocklist (local file or url)
Needs to be json compatible | empty | -| LIDARR__INSTANCES__0__URL | No | First LIDARR instance url | http://localhost:8686 | -| LIDARR__INSTANCES__0__APIKEY | No | First LIDARR instance API key | empty | -
- -### Notifications variables -
- Click here - -| Variable | Required | Description | Default value | -|---|---|---|---| -| NOTIFIARR__ON_IMPORT_FAILED_STRIKE | No | Notify on failed import strike. | false | -| NOTIFIARR__ON_STALLED_STRIKE | No | Notify on stalled download strike. | false | -| NOTIFIARR__ON_QUEUE_ITEM_DELETE | No | Notify on deleting a queue item. | false | -| NOTIFIARR__API_KEY | No | Notifiarr API key.
Requires Notifiarr's `Passthrough` integration to work. | empty | -| NOTIFIARR__CHANNEL_ID | No | Discord channel id for notifications. | empty | -
- - -### Advanced variables -
- Click here - -| Variable | Required | Description | Default value | -|---|---|---|---| -| HTTP_MAX_RETRIES | No | The number of times to retry a failed HTTP call (to *arrs, download clients etc.) | 0 | -| HTTP_TIMEOUT | No | The number of seconds to wait before failing an HTTP call (to *arrs, download clients etc.) | 100 | -
- -# - -> [!NOTE] -> 1. The queue cleaner and content blocker can be enabled or disabled separately, if you want to run only one of them. -> 2. Only one download client can be enabled at a time. If you have more than one download client, you should deploy multiple instances of cleanuperr. -> 3. The blocklists (blacklist/whitelist) should have a single pattern on each line and supports the following: -> ``` -> *example // file name ends with "example" -> example* // file name starts with "example" -> *example* // file name has "example" in the name -> example // file name is exactly the word "example" -> regex: // regex that needs to be marked at the start of the line with "regex:" -> ``` -> 4. Multiple Sonarr/Radarr/Lidarr instances can be specified using this format, where `` starts from `0`: -> ``` -> SONARR__INSTANCES____URL -> SONARR__INSTANCES____APIKEY -> ``` -> 5. Multiple failed import patterns can be specified using this format, where `` starts from 0: -> ``` -> QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__ -> ``` -> 6. [This blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist) and [this whitelist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist) can be used for Sonarr and Radarr, but they are not suitable for other *arrs. - -# +Jump to: +- [General settings](variables.md#general-settings) +- [Queue Cleaner settings](variables.md#queue-cleaner-settings) +- [Content Blocker settings](variables.md#content-blocker-settings) +- [Download Cleaner settings](variables.md#download-cleaner-settings) +- [Download Client settings](variables.md#download-client-settings) +- [Arr settings](variables.md#arr-settings) +- [Notification settings](variables.md#notification-settings) +- [Advanced settings](variables.md#advanced-settings) ### Binaries (if you're not using Docker) @@ -328,9 +231,10 @@ Special thanks for inspiration go to: - [ThijmenGThN/swaparr](https://github.com/ThijmenGThN/swaparr) - [ManiMatter/decluttarr](https://github.com/ManiMatter/decluttarr) - [PaeyMoopy/sonarr-radarr-queue-cleaner](https://github.com/PaeyMoopy/sonarr-radarr-queue-cleaner) -- [Sonarr](https://github.com/Sonarr/Sonarr) & [Radarr](https://github.com/Radarr/Radarr) for the logo +- [Sonarr](https://github.com/Sonarr/Sonarr) & [Radarr](https://github.com/Radarr/Radarr) # Buy me a coffee If I made your life just a tiny bit easier, consider buying me a coffee! -Buy Me A Coffee \ No newline at end of file +Buy Me A Coffee + diff --git a/code/Common/Attributes/DryRunSafeguardAttribute.cs b/code/Common/Attributes/DryRunSafeguardAttribute.cs new file mode 100644 index 00000000..2f5fa091 --- /dev/null +++ b/code/Common/Attributes/DryRunSafeguardAttribute.cs @@ -0,0 +1,6 @@ +namespace Common.Attributes; + +[AttributeUsage(AttributeTargets.Method, Inherited = true)] +public class DryRunSafeguardAttribute : Attribute +{ +} \ No newline at end of file diff --git a/code/Common/Common.csproj b/code/Common/Common.csproj index 3290e709..2b07fd7f 100644 --- a/code/Common/Common.csproj +++ b/code/Common/Common.csproj @@ -7,7 +7,7 @@ - + diff --git a/code/Common/Configuration/Arr/ArrConfig.cs b/code/Common/Configuration/Arr/ArrConfig.cs index 19713760..802c5c96 100644 --- a/code/Common/Configuration/Arr/ArrConfig.cs +++ b/code/Common/Configuration/Arr/ArrConfig.cs @@ -11,9 +11,9 @@ public abstract record ArrConfig public required List Instances { get; init; } } -public record Block +public readonly record struct Block { - public BlocklistType Type { get; set; } + public BlocklistType Type { get; init; } - public string? Path { get; set; } + public string? Path { get; init; } } \ No newline at end of file diff --git a/code/Common/Configuration/DownloadCleaner/Category.cs b/code/Common/Configuration/DownloadCleaner/Category.cs new file mode 100644 index 00000000..67d12c56 --- /dev/null +++ b/code/Common/Configuration/DownloadCleaner/Category.cs @@ -0,0 +1,45 @@ +using Common.Exceptions; +using Microsoft.Extensions.Configuration; + +namespace Common.Configuration.DownloadCleaner; + +public sealed record Category : IConfig +{ + public required string Name { get; init; } + + /// + /// Max ratio before removing a download. + /// + [ConfigurationKeyName("MAX_RATIO")] + public required double MaxRatio { get; init; } = -1; + + /// + /// Min number of hours to seed before removing a download, if the ratio has been met. + /// + [ConfigurationKeyName("MIN_SEED_TIME")] + public required double MinSeedTime { get; init; } = 0; + + /// + /// Number of hours to seed before removing a download. + /// + [ConfigurationKeyName("MAX_SEED_TIME")] + public required double MaxSeedTime { get; init; } = -1; + + public void Validate() + { + if (string.IsNullOrWhiteSpace(Name)) + { + throw new ValidationException($"{nameof(Name)} can not be empty"); + } + + if (MaxRatio < 0 && MaxSeedTime < 0) + { + throw new ValidationException($"both {nameof(MaxRatio)} and {nameof(MaxSeedTime)} are disabled"); + } + + if (MinSeedTime < 0) + { + throw new ValidationException($"{nameof(MinSeedTime)} can not be negative"); + } + } +} \ No newline at end of file diff --git a/code/Common/Configuration/DownloadCleaner/DownloadCleanerConfig.cs b/code/Common/Configuration/DownloadCleaner/DownloadCleanerConfig.cs new file mode 100644 index 00000000..7f08fdbe --- /dev/null +++ b/code/Common/Configuration/DownloadCleaner/DownloadCleanerConfig.cs @@ -0,0 +1,36 @@ +using Common.Exceptions; +using Microsoft.Extensions.Configuration; + +namespace Common.Configuration.DownloadCleaner; + +public sealed record DownloadCleanerConfig : IJobConfig +{ + public const string SectionName = "DownloadCleaner"; + + public bool Enabled { get; init; } + + public List? Categories { get; init; } + + [ConfigurationKeyName("DELETE_PRIVATE")] + public bool DeletePrivate { get; set; } + + public void Validate() + { + if (!Enabled) + { + return; + } + + if (Categories?.Count is null or 0) + { + throw new ValidationException("no categories configured"); + } + + if (Categories?.GroupBy(x => x.Name).Any(x => x.Count() > 1) is true) + { + throw new ValidationException("duplicated categories found"); + } + + Categories?.ForEach(x => x.Validate()); + } +} \ No newline at end of file diff --git a/code/Common/Configuration/DownloadClient/DelugeConfig.cs b/code/Common/Configuration/DownloadClient/DelugeConfig.cs index 18044208..be16ac02 100644 --- a/code/Common/Configuration/DownloadClient/DelugeConfig.cs +++ b/code/Common/Configuration/DownloadClient/DelugeConfig.cs @@ -1,4 +1,6 @@ -namespace Common.Configuration.DownloadClient; +using Common.Exceptions; + +namespace Common.Configuration.DownloadClient; public sealed record DelugeConfig : IConfig { @@ -12,7 +14,7 @@ public sealed record DelugeConfig : IConfig { if (Url is null) { - throw new ArgumentNullException(nameof(Url)); + throw new ValidationException($"{nameof(Url)} is empty"); } } } \ No newline at end of file diff --git a/code/Common/Configuration/DownloadClient/QBitConfig.cs b/code/Common/Configuration/DownloadClient/QBitConfig.cs index b2b3cbea..afdbc1ef 100644 --- a/code/Common/Configuration/DownloadClient/QBitConfig.cs +++ b/code/Common/Configuration/DownloadClient/QBitConfig.cs @@ -1,4 +1,6 @@ -namespace Common.Configuration.DownloadClient; +using Common.Exceptions; + +namespace Common.Configuration.DownloadClient; public sealed class QBitConfig : IConfig { @@ -14,7 +16,7 @@ public sealed class QBitConfig : IConfig { if (Url is null) { - throw new ArgumentNullException(nameof(Url)); + throw new ValidationException($"{nameof(Url)} is empty"); } } } \ No newline at end of file diff --git a/code/Common/Configuration/DownloadClient/TransmissionConfig.cs b/code/Common/Configuration/DownloadClient/TransmissionConfig.cs index 30e94b47..c029b10d 100644 --- a/code/Common/Configuration/DownloadClient/TransmissionConfig.cs +++ b/code/Common/Configuration/DownloadClient/TransmissionConfig.cs @@ -1,4 +1,6 @@ -namespace Common.Configuration.DownloadClient; +using Common.Exceptions; + +namespace Common.Configuration.DownloadClient; public record TransmissionConfig : IConfig { @@ -14,7 +16,7 @@ public record TransmissionConfig : IConfig { if (Url is null) { - throw new ArgumentNullException(nameof(Url)); + throw new ValidationException($"{nameof(Url)} is empty"); } } } \ No newline at end of file diff --git a/code/Common/Configuration/General/DryRunConfig.cs b/code/Common/Configuration/General/DryRunConfig.cs new file mode 100644 index 00000000..22b9c419 --- /dev/null +++ b/code/Common/Configuration/General/DryRunConfig.cs @@ -0,0 +1,9 @@ +using Microsoft.Extensions.Configuration; + +namespace Common.Configuration.General; + +public sealed record DryRunConfig +{ + [ConfigurationKeyName("DRY_RUN")] + public bool IsDryRun { get; init; } +} \ No newline at end of file diff --git a/code/Common/Configuration/General/HttpConfig.cs b/code/Common/Configuration/General/HttpConfig.cs index 35c51a00..a6029e03 100644 --- a/code/Common/Configuration/General/HttpConfig.cs +++ b/code/Common/Configuration/General/HttpConfig.cs @@ -1,8 +1,9 @@ -using Microsoft.Extensions.Configuration; +using Common.Exceptions; +using Microsoft.Extensions.Configuration; namespace Common.Configuration.General; -public class HttpConfig : IConfig +public sealed record HttpConfig : IConfig { [ConfigurationKeyName("HTTP_MAX_RETRIES")] public ushort MaxRetries { get; init; } @@ -14,7 +15,7 @@ public class HttpConfig : IConfig { if (Timeout is 0) { - throw new ArgumentException("HTTP_TIMEOUT must be greater than 0"); + throw new ValidationException("HTTP_TIMEOUT must be greater than 0"); } } } \ No newline at end of file diff --git a/code/Common/Configuration/General/TriggersConfig.cs b/code/Common/Configuration/General/TriggersConfig.cs index f157eeb3..77aee797 100644 --- a/code/Common/Configuration/General/TriggersConfig.cs +++ b/code/Common/Configuration/General/TriggersConfig.cs @@ -7,4 +7,6 @@ public sealed class TriggersConfig public required string QueueCleaner { get; init; } public required string ContentBlocker { get; init; } + + public required string DownloadCleaner { get; init; } } \ No newline at end of file diff --git a/code/Common/Configuration/Notification/NotificationConfig.cs b/code/Common/Configuration/Notification/NotificationConfig.cs index 2fc4574b..18e74156 100644 --- a/code/Common/Configuration/Notification/NotificationConfig.cs +++ b/code/Common/Configuration/Notification/NotificationConfig.cs @@ -10,10 +10,13 @@ public abstract record NotificationConfig [ConfigurationKeyName("ON_STALLED_STRIKE")] public bool OnStalledStrike { get; init; } - [ConfigurationKeyName("ON_QUEUE_ITEM_DELETE")] - public bool OnQueueItemDelete { get; init; } + [ConfigurationKeyName("ON_QUEUE_ITEM_DELETED")] + public bool OnQueueItemDeleted { get; init; } + + [ConfigurationKeyName("ON_DOWNLOAD_CLEANED")] + public bool OnDownloadCleaned { get; init; } - public bool IsEnabled => OnImportFailedStrike || OnStalledStrike || OnQueueItemDelete; + public bool IsEnabled => OnImportFailedStrike || OnStalledStrike || OnQueueItemDeleted || OnDownloadCleaned; public abstract bool IsValid(); } \ No newline at end of file diff --git a/code/Common/Exceptions/ValidationException.cs b/code/Common/Exceptions/ValidationException.cs new file mode 100644 index 00000000..87074adc --- /dev/null +++ b/code/Common/Exceptions/ValidationException.cs @@ -0,0 +1,12 @@ +namespace Common.Exceptions; + +public sealed class ValidationException : Exception +{ + public ValidationException() + { + } + + public ValidationException(string message) : base(message) + { + } +} \ No newline at end of file diff --git a/code/Domain/Enums/CleanReason.cs b/code/Domain/Enums/CleanReason.cs new file mode 100644 index 00000000..d9f218f0 --- /dev/null +++ b/code/Domain/Enums/CleanReason.cs @@ -0,0 +1,8 @@ +namespace Domain.Enums; + +public enum CleanReason +{ + None, + MaxRatioReached, + MaxSeedTimeReached, +} \ No newline at end of file diff --git a/code/Domain/Models/Deluge/Response/TorrentStatus.cs b/code/Domain/Models/Deluge/Response/TorrentStatus.cs index 85cf39f6..5ad65a50 100644 --- a/code/Domain/Models/Deluge/Response/TorrentStatus.cs +++ b/code/Domain/Models/Deluge/Response/TorrentStatus.cs @@ -16,4 +16,11 @@ public sealed record TorrentStatus [JsonProperty("total_done")] public long TotalDone { get; init; } + + public string? Label { get; init; } + + [JsonProperty("seeding_time")] + public long SeedingTime { get; init; } + + public float Ratio { get; init; } } \ No newline at end of file diff --git a/code/Executable/DependencyInjection/ConfigurationDI.cs b/code/Executable/DependencyInjection/ConfigurationDI.cs index efd351e4..6bccee3d 100644 --- a/code/Executable/DependencyInjection/ConfigurationDI.cs +++ b/code/Executable/DependencyInjection/ConfigurationDI.cs @@ -1,6 +1,8 @@ using Common.Configuration.Arr; using Common.Configuration.ContentBlocker; +using Common.Configuration.DownloadCleaner; using Common.Configuration.DownloadClient; +using Common.Configuration.General; using Common.Configuration.Logging; using Common.Configuration.QueueCleaner; @@ -10,8 +12,10 @@ public static class ConfigurationDI { public static IServiceCollection AddConfiguration(this IServiceCollection services, IConfiguration configuration) => services + .Configure(configuration) .Configure(configuration.GetSection(QueueCleanerConfig.SectionName)) .Configure(configuration.GetSection(ContentBlockerConfig.SectionName)) + .Configure(configuration.GetSection(DownloadCleanerConfig.SectionName)) .Configure(configuration) .Configure(configuration.GetSection(QBitConfig.SectionName)) .Configure(configuration.GetSection(DelugeConfig.SectionName)) diff --git a/code/Executable/DependencyInjection/LoggingDI.cs b/code/Executable/DependencyInjection/LoggingDI.cs index dc33821c..eb1e4ad2 100644 --- a/code/Executable/DependencyInjection/LoggingDI.cs +++ b/code/Executable/DependencyInjection/LoggingDI.cs @@ -1,6 +1,7 @@ using Common.Configuration.Logging; using Domain.Enums; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.DownloadCleaner; using Infrastructure.Verticals.QueueCleaner; using Serilog; using Serilog.Events; @@ -33,7 +34,7 @@ public static class LoggingDI const string consoleOutputTemplate = $"[{{@t:yyyy-MM-dd HH:mm:ss.fff}} {{@l:u3}}]{jobNameTemplate}{instanceNameTemplate} {{@m}}\n{{@x}}"; const string fileOutputTemplate = $"{{@t:yyyy-MM-dd HH:mm:ss.fff zzz}} [{{@l:u3}}]{jobNameTemplate}{instanceNameTemplate} {{@m:lj}}\n{{@x}}"; LogEventLevel level = LogEventLevel.Information; - List names = [nameof(ContentBlocker), nameof(QueueCleaner)]; + List names = [nameof(ContentBlocker), nameof(QueueCleaner), nameof(DownloadCleaner)]; int jobPadding = names.Max(x => x.Length) + 2; names = [InstanceType.Sonarr.ToString(), InstanceType.Radarr.ToString(), InstanceType.Lidarr.ToString()]; int arrPadding = names.Max(x => x.Length) + 2; diff --git a/code/Executable/DependencyInjection/MainDI.cs b/code/Executable/DependencyInjection/MainDI.cs index 7e6d531f..2bb12588 100644 --- a/code/Executable/DependencyInjection/MainDI.cs +++ b/code/Executable/DependencyInjection/MainDI.cs @@ -1,6 +1,8 @@ using System.Net; +using Castle.DynamicProxy; using Common.Configuration.General; using Common.Helpers; +using Infrastructure.Interceptors; using Infrastructure.Verticals.DownloadClient.Deluge; using Infrastructure.Verticals.Notifications.Consumers; using Infrastructure.Verticals.Notifications.Models; @@ -25,7 +27,8 @@ public static class MainDI { config.AddConsumer>(); config.AddConsumer>(); - config.AddConsumer>(); + config.AddConsumer>(); + config.AddConsumer>(); config.UsingInMemory((context, cfg) => { @@ -33,12 +36,14 @@ public static class MainDI { e.ConfigureConsumer>(context); e.ConfigureConsumer>(context); - e.ConfigureConsumer>(context); + e.ConfigureConsumer>(context); + e.ConfigureConsumer>(context); e.ConcurrentMessageLimit = 1; e.PrefetchCount = 1; }); }); - }); + }) + .AddDryRunInterceptor(); private static IServiceCollection AddHttpClients(this IServiceCollection services, IConfiguration configuration) { @@ -86,4 +91,31 @@ public static class MainDI .OrResult(response => !response.IsSuccessStatusCode && response.StatusCode != HttpStatusCode.Unauthorized) .WaitAndRetryAsync(config.MaxRetries, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))) ); + + private static IServiceCollection AddDryRunInterceptor(this IServiceCollection services) + { + services + .Where(s => s.ServiceType != typeof(IDryRunService) && typeof(IDryRunService).IsAssignableFrom(s.ServiceType)) + .ToList() + .ForEach(service => + { + services.Decorate(service.ServiceType, (target, svc) => + { + ProxyGenerator proxyGenerator = new(); + DryRunAsyncInterceptor interceptor = svc.GetRequiredService(); + + object implementation = proxyGenerator.CreateClassProxyWithTarget( + service.ServiceType, + target, + interceptor + ); + + ((IInterceptedService)target).Proxy = implementation; + + return implementation; + }); + }); + + return services; + } } \ No newline at end of file diff --git a/code/Executable/DependencyInjection/QuartzDI.cs b/code/Executable/DependencyInjection/QuartzDI.cs index c33988a8..73315faf 100644 --- a/code/Executable/DependencyInjection/QuartzDI.cs +++ b/code/Executable/DependencyInjection/QuartzDI.cs @@ -1,10 +1,12 @@ using Common.Configuration; using Common.Configuration.ContentBlocker; +using Common.Configuration.DownloadCleaner; using Common.Configuration.General; using Common.Configuration.QueueCleaner; using Common.Helpers; using Executable.Jobs; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.DownloadCleaner; using Infrastructure.Verticals.Jobs; using Infrastructure.Verticals.QueueCleaner; using Quartz; @@ -59,6 +61,12 @@ public static class QuartzDI { q.AddJob(queueCleanerConfig, triggersConfig.QueueCleaner); } + + DownloadCleanerConfig? downloadCleanerConfig = configuration + .GetRequiredSection(DownloadCleanerConfig.SectionName) + .Get(); + + q.AddJob(downloadCleanerConfig, triggersConfig.DownloadCleaner); } private static void AddJob( @@ -109,7 +117,7 @@ public static class QuartzDI if (triggerValue > Constants.TriggerMaxLimit) { - throw new Exception($"{trigger} should have a fire time of maximum 1 hour"); + throw new Exception($"{trigger} should have a fire time of maximum {Constants.TriggerMaxLimit.TotalHours} hours"); } if (triggerValue > StaticConfiguration.TriggerValue) diff --git a/code/Executable/DependencyInjection/ServicesDI.cs b/code/Executable/DependencyInjection/ServicesDI.cs index 73fa6182..b1730228 100644 --- a/code/Executable/DependencyInjection/ServicesDI.cs +++ b/code/Executable/DependencyInjection/ServicesDI.cs @@ -1,5 +1,7 @@ -using Infrastructure.Verticals.Arr; +using Infrastructure.Interceptors; +using Infrastructure.Verticals.Arr; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.DownloadCleaner; using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.DownloadClient.Deluge; using Infrastructure.Verticals.DownloadClient.QBittorrent; @@ -13,12 +15,14 @@ public static class ServicesDI { public static IServiceCollection AddServices(this IServiceCollection services) => services + .AddTransient() .AddTransient() .AddTransient() .AddTransient() .AddTransient() .AddTransient() - .AddTransient() + .AddTransient() + .AddTransient() .AddTransient() .AddTransient() .AddTransient() @@ -26,5 +30,5 @@ public static class ServicesDI .AddTransient() .AddTransient() .AddSingleton() - .AddSingleton(); + .AddSingleton(); } \ No newline at end of file diff --git a/code/Executable/Executable.csproj b/code/Executable/Executable.csproj index 326c094d..79c6b329 100644 --- a/code/Executable/Executable.csproj +++ b/code/Executable/Executable.csproj @@ -10,9 +10,9 @@ - - - + + + diff --git a/code/Executable/Jobs/GenericJob.cs b/code/Executable/Jobs/GenericJob.cs index 48dc7272..12181fe6 100644 --- a/code/Executable/Jobs/GenericJob.cs +++ b/code/Executable/Jobs/GenericJob.cs @@ -6,7 +6,7 @@ namespace Executable.Jobs; [DisallowConcurrentExecution] public sealed class GenericJob : IJob - where T : GenericHandler + where T : IHandler { private readonly ILogger> _logger; private readonly T _handler; diff --git a/code/Executable/appsettings.Development.json b/code/Executable/appsettings.Development.json index 65f88e6f..27dc69be 100644 --- a/code/Executable/appsettings.Development.json +++ b/code/Executable/appsettings.Development.json @@ -1,4 +1,5 @@ { + "DRY_RUN": true, "HTTP_MAX_RETRIES": 0, "HTTP_TIMEOUT": 10, "Logging": { @@ -11,7 +12,8 @@ }, "Triggers": { "QueueCleaner": "0/10 * * * * ?", - "ContentBlocker": "0/10 * * * * ?" + "ContentBlocker": "0/10 * * * * ?", + "DownloadCleaner": "0/10 * * * * ?" }, "ContentBlocker": { "Enabled": true, @@ -32,6 +34,18 @@ "STALLED_IGNORE_PRIVATE": true, "STALLED_DELETE_PRIVATE": false }, + "DownloadCleaner": { + "Enabled": false, + "DELETE_PRIVATE": false, + "CATEGORIES": [ + { + "Name": "tv-sonarr", + "MAX_RATIO": -1, + "MIN_SEED_TIME": 0, + "MAX_SEED_TIME": -1 + } + ] + }, "DOWNLOAD_CLIENT": "qbittorrent", "qBittorrent": { "Url": "http://localhost:8080", @@ -90,7 +104,8 @@ "Notifiarr": { "ON_IMPORT_FAILED_STRIKE": true, "ON_STALLED_STRIKE": true, - "ON_QUEUE_ITEM_DELETE": true, + "ON_QUEUE_ITEM_DELETED": true, + "ON_DOWNLOAD_CLEANED": true, "API_KEY": "", "CHANNEL_ID": "" } diff --git a/code/Executable/appsettings.json b/code/Executable/appsettings.json index 18bfe546..cf834a4a 100644 --- a/code/Executable/appsettings.json +++ b/code/Executable/appsettings.json @@ -1,4 +1,5 @@ { + "DRY_RUN": false, "HTTP_MAX_RETRIES": 0, "HTTP_TIMEOUT": 100, "Logging": { @@ -11,7 +12,8 @@ }, "Triggers": { "QueueCleaner": "0 0/5 * * * ?", - "ContentBlocker": "0 0/5 * * * ?" + "ContentBlocker": "0 0/5 * * * ?", + "DownloadCleaner": "0 0 * * * ?" }, "ContentBlocker": { "Enabled": false, @@ -29,6 +31,11 @@ "STALLED_IGNORE_PRIVATE": false, "STALLED_DELETE_PRIVATE": false }, + "DownloadCleaner": { + "Enabled": false, + "DELETE_PRIVATE": false, + "CATEGORIES": [] + }, "DOWNLOAD_CLIENT": "none", "qBittorrent": { "Url": "http://localhost:8080", @@ -87,7 +94,8 @@ "Notifiarr": { "ON_IMPORT_FAILED_STRIKE": false, "ON_STALLED_STRIKE": false, - "ON_QUEUE_ITEM_DELETE": false, + "ON_QUEUE_ITEM_DELETED": false, + "ON_DOWNLOAD_CLEANED": false, "API_KEY": "", "CHANNEL_ID": "" } diff --git a/code/Infrastructure.Tests/Infrastructure.Tests.csproj b/code/Infrastructure.Tests/Infrastructure.Tests.csproj new file mode 100644 index 00000000..0d647924 --- /dev/null +++ b/code/Infrastructure.Tests/Infrastructure.Tests.csproj @@ -0,0 +1,33 @@ + + + + net9.0 + enable + enable + false + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + + + + diff --git a/code/Infrastructure.Tests/Verticals/ContentBlocker/FilenameEvaluatorFixture.cs b/code/Infrastructure.Tests/Verticals/ContentBlocker/FilenameEvaluatorFixture.cs new file mode 100644 index 00000000..45d5546e --- /dev/null +++ b/code/Infrastructure.Tests/Verticals/ContentBlocker/FilenameEvaluatorFixture.cs @@ -0,0 +1,20 @@ +using Infrastructure.Verticals.ContentBlocker; +using Microsoft.Extensions.Logging; +using NSubstitute; + +namespace Infrastructure.Tests.Verticals.ContentBlocker; + +public class FilenameEvaluatorFixture +{ + public ILogger Logger { get; } + + public FilenameEvaluatorFixture() + { + Logger = Substitute.For>(); + } + + public FilenameEvaluator CreateSut() + { + return new FilenameEvaluator(Logger); + } +} \ No newline at end of file diff --git a/code/Infrastructure.Tests/Verticals/ContentBlocker/FilenameEvaluatorTests.cs b/code/Infrastructure.Tests/Verticals/ContentBlocker/FilenameEvaluatorTests.cs new file mode 100644 index 00000000..f295eb5c --- /dev/null +++ b/code/Infrastructure.Tests/Verticals/ContentBlocker/FilenameEvaluatorTests.cs @@ -0,0 +1,219 @@ +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.ContentBlocker; +using Shouldly; + +namespace Infrastructure.Tests.Verticals.ContentBlocker; + +public class FilenameEvaluatorTests : IClassFixture +{ + private readonly FilenameEvaluatorFixture _fixture; + + public FilenameEvaluatorTests(FilenameEvaluatorFixture fixture) + { + _fixture = fixture; + } + + public class PatternTests : FilenameEvaluatorTests + { + public PatternTests(FilenameEvaluatorFixture fixture) : base(fixture) { } + + [Fact] + public void WhenNoPatterns_ShouldReturnTrue() + { + // Arrange + var sut = _fixture.CreateSut(); + var patterns = new ConcurrentBag(); + var regexes = new ConcurrentBag(); + + // Act + var result = sut.IsValid("test.txt", BlocklistType.Blacklist, patterns, regexes); + + // Assert + result.ShouldBeTrue(); + } + + [Theory] + [InlineData("test.txt", "test.txt", true)] // Exact match + [InlineData("test.txt", "*.txt", true)] // End wildcard + [InlineData("test.txt", "test.*", true)] // Start wildcard + [InlineData("test.txt", "*test*", true)] // Both wildcards + [InlineData("test.txt", "other.txt", false)] // No match + public void Blacklist_ShouldMatchPatterns(string filename, string pattern, bool shouldBeBlocked) + { + // Arrange + var sut = _fixture.CreateSut(); + var patterns = new ConcurrentBag { pattern }; + var regexes = new ConcurrentBag(); + + // Act + var result = sut.IsValid(filename, BlocklistType.Blacklist, patterns, regexes); + + // Assert + result.ShouldBe(!shouldBeBlocked); + } + + [Theory] + [InlineData("test.txt", "test.txt", true)] // Exact match + [InlineData("test.txt", "*.txt", true)] // End wildcard + [InlineData("test.txt", "test.*", true)] // Start wildcard + [InlineData("test.txt", "*test*", true)] // Both wildcards + [InlineData("test.txt", "other.txt", false)] // No match + public void Whitelist_ShouldMatchPatterns(string filename, string pattern, bool shouldBeAllowed) + { + // Arrange + var sut = _fixture.CreateSut(); + var patterns = new ConcurrentBag { pattern }; + var regexes = new ConcurrentBag(); + + // Act + var result = sut.IsValid(filename, BlocklistType.Whitelist, patterns, regexes); + + // Assert + result.ShouldBe(shouldBeAllowed); + } + + [Theory] + [InlineData("TEST.TXT", "test.txt")] + [InlineData("test.txt", "TEST.TXT")] + public void ShouldBeCaseInsensitive(string filename, string pattern) + { + // Arrange + var sut = _fixture.CreateSut(); + var patterns = new ConcurrentBag { pattern }; + var regexes = new ConcurrentBag(); + + // Act + var result = sut.IsValid(filename, BlocklistType.Blacklist, patterns, regexes); + + // Assert + result.ShouldBeFalse(); + } + + [Fact] + public void MultiplePatterns_ShouldMatchAny() + { + // Arrange + var sut = _fixture.CreateSut(); + var patterns = new ConcurrentBag + { + "other.txt", + "*.pdf", + "test.*" + }; + var regexes = new ConcurrentBag(); + + // Act + var result = sut.IsValid("test.txt", BlocklistType.Blacklist, patterns, regexes); + + // Assert + result.ShouldBeFalse(); + } + } + + public class RegexTests : FilenameEvaluatorTests + { + public RegexTests(FilenameEvaluatorFixture fixture) : base(fixture) { } + + [Fact] + public void WhenNoRegexes_ShouldReturnTrue() + { + // Arrange + var sut = _fixture.CreateSut(); + var patterns = new ConcurrentBag(); + var regexes = new ConcurrentBag(); + + // Act + var result = sut.IsValid("test.txt", BlocklistType.Blacklist, patterns, regexes); + + // Assert + result.ShouldBeTrue(); + } + + [Theory] + [InlineData(@"test\d+\.txt", "test123.txt", true)] + [InlineData(@"test\d+\.txt", "test.txt", false)] + public void Blacklist_ShouldMatchRegexes(string pattern, string filename, bool shouldBeBlocked) + { + // Arrange + var sut = _fixture.CreateSut(); + var patterns = new ConcurrentBag(); + var regexes = new ConcurrentBag { new Regex(pattern, RegexOptions.IgnoreCase) }; + + // Act + var result = sut.IsValid(filename, BlocklistType.Blacklist, patterns, regexes); + + // Assert + result.ShouldBe(!shouldBeBlocked); + } + + [Theory] + [InlineData(@"test\d+\.txt", "test123.txt", true)] + [InlineData(@"test\d+\.txt", "test.txt", false)] + public void Whitelist_ShouldMatchRegexes(string pattern, string filename, bool shouldBeAllowed) + { + // Arrange + var sut = _fixture.CreateSut(); + var patterns = new ConcurrentBag(); + var regexes = new ConcurrentBag { new Regex(pattern, RegexOptions.IgnoreCase) }; + + // Act + var result = sut.IsValid(filename, BlocklistType.Whitelist, patterns, regexes); + + // Assert + result.ShouldBe(shouldBeAllowed); + } + + [Theory] + [InlineData(@"TEST\d+\.TXT", "test123.txt")] + [InlineData(@"test\d+\.txt", "TEST123.TXT")] + public void ShouldBeCaseInsensitive(string pattern, string filename) + { + // Arrange + var sut = _fixture.CreateSut(); + var patterns = new ConcurrentBag(); + var regexes = new ConcurrentBag { new Regex(pattern, RegexOptions.IgnoreCase) }; + + // Act + var result = sut.IsValid(filename, BlocklistType.Blacklist, patterns, regexes); + + // Assert + result.ShouldBeFalse(); + } + } + + public class CombinedTests : FilenameEvaluatorTests + { + public CombinedTests(FilenameEvaluatorFixture fixture) : base(fixture) { } + + [Fact] + public void WhenBothPatternsAndRegexes_ShouldMatchBoth() + { + // Arrange + var sut = _fixture.CreateSut(); + var patterns = new ConcurrentBag { "*.txt" }; + var regexes = new ConcurrentBag { new Regex(@"test\d+", RegexOptions.IgnoreCase) }; + + // Act + var result = sut.IsValid("test123.txt", BlocklistType.Blacklist, patterns, regexes); + + // Assert + result.ShouldBeFalse(); + } + + [Fact] + public void WhenPatternMatchesButRegexDoesNot_ShouldReturnFalse() + { + // Arrange + var sut = _fixture.CreateSut(); + var patterns = new ConcurrentBag { "*.txt" }; + var regexes = new ConcurrentBag { new Regex(@"test\d+", RegexOptions.IgnoreCase) }; + + // Act + var result = sut.IsValid("other.txt", BlocklistType.Whitelist, patterns, regexes); + + // Assert + result.ShouldBeFalse(); + } + } +} \ No newline at end of file diff --git a/code/Infrastructure.Tests/Verticals/DownloadClient/DownloadServiceFixture.cs b/code/Infrastructure.Tests/Verticals/DownloadClient/DownloadServiceFixture.cs new file mode 100644 index 00000000..bdab1877 --- /dev/null +++ b/code/Infrastructure.Tests/Verticals/DownloadClient/DownloadServiceFixture.cs @@ -0,0 +1,74 @@ +using Common.Configuration.ContentBlocker; +using Common.Configuration.DownloadCleaner; +using Common.Configuration.QueueCleaner; +using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.DownloadClient; +using Infrastructure.Verticals.ItemStriker; +using Infrastructure.Verticals.Notifications; +using Microsoft.Extensions.Caching.Memory; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using NSubstitute; + +namespace Infrastructure.Tests.Verticals.DownloadClient; + +public class DownloadServiceFixture : IDisposable +{ + public ILogger Logger { get; set; } + public IMemoryCache Cache { get; set; } + public IStriker Striker { get; set; } + + public DownloadServiceFixture() + { + Logger = Substitute.For>(); + Cache = Substitute.For(); + Striker = Substitute.For(); + } + + public TestDownloadService CreateSut( + QueueCleanerConfig? queueCleanerConfig = null, + ContentBlockerConfig? contentBlockerConfig = null + ) + { + queueCleanerConfig ??= new QueueCleanerConfig + { + Enabled = true, + RunSequentially = true, + StalledResetStrikesOnProgress = true, + StalledMaxStrikes = 3 + }; + + var queueCleanerOptions = Substitute.For>(); + queueCleanerOptions.Value.Returns(queueCleanerConfig); + + contentBlockerConfig ??= new ContentBlockerConfig + { + Enabled = true + }; + + var contentBlockerOptions = Substitute.For>(); + contentBlockerOptions.Value.Returns(contentBlockerConfig); + + var downloadCleanerOptions = Substitute.For>(); + downloadCleanerOptions.Value.Returns(new DownloadCleanerConfig()); + + var filenameEvaluator = Substitute.For(); + var notifier = Substitute.For(); + + return new TestDownloadService( + Logger, + queueCleanerOptions, + contentBlockerOptions, + downloadCleanerOptions, + Cache, + filenameEvaluator, + Striker, + notifier + ); + } + + public void Dispose() + { + // Cleanup if needed + } +} \ No newline at end of file diff --git a/code/Infrastructure.Tests/Verticals/DownloadClient/DownloadServiceTests.cs b/code/Infrastructure.Tests/Verticals/DownloadClient/DownloadServiceTests.cs new file mode 100644 index 00000000..9fcba545 --- /dev/null +++ b/code/Infrastructure.Tests/Verticals/DownloadClient/DownloadServiceTests.cs @@ -0,0 +1,235 @@ +using Common.Configuration.DownloadCleaner; +using Domain.Enums; +using Domain.Models.Cache; +using Infrastructure.Helpers; +using Infrastructure.Verticals.Context; +using Infrastructure.Verticals.DownloadClient; +using NSubstitute; +using NSubstitute.ClearExtensions; +using Shouldly; + +namespace Infrastructure.Tests.Verticals.DownloadClient; + +public class DownloadServiceTests : IClassFixture +{ + private readonly DownloadServiceFixture _fixture; + + public DownloadServiceTests(DownloadServiceFixture fixture) + { + _fixture = fixture; + _fixture.Cache.ClearSubstitute(); + _fixture.Striker.ClearSubstitute(); + } + + public class ResetStrikesOnProgressTests : DownloadServiceTests + { + public ResetStrikesOnProgressTests(DownloadServiceFixture fixture) : base(fixture) + { + } + + [Fact] + public void WhenStalledStrikeDisabled_ShouldNotResetStrikes() + { + // Arrange + TestDownloadService sut = _fixture.CreateSut(queueCleanerConfig: new() + { + Enabled = true, + RunSequentially = true, + StalledResetStrikesOnProgress = false, + }); + + // Act + sut.ResetStrikesOnProgress("test-hash", 100); + + // Assert + _fixture.Cache.ReceivedCalls().ShouldBeEmpty(); + } + + [Fact] + public void WhenProgressMade_ShouldResetStrikes() + { + // Arrange + const string hash = "test-hash"; + CacheItem cacheItem = new CacheItem { Downloaded = 100 }; + + _fixture.Cache.TryGetValue(Arg.Any(), out Arg.Any()) + .Returns(x => + { + x[1] = cacheItem; + return true; + }); + + TestDownloadService sut = _fixture.CreateSut(); + + // Act + sut.ResetStrikesOnProgress(hash, 200); + + // Assert + _fixture.Cache.Received(1).Remove(CacheKeys.Strike(StrikeType.Stalled, hash)); + } + + [Fact] + public void WhenNoProgress_ShouldNotResetStrikes() + { + // Arrange + const string hash = "test-hash"; + CacheItem cacheItem = new CacheItem { Downloaded = 200 }; + + _fixture.Cache + .TryGetValue(Arg.Any(), out Arg.Any()) + .Returns(x => + { + x[1] = cacheItem; + return true; + }); + + TestDownloadService sut = _fixture.CreateSut(); + + // Act + sut.ResetStrikesOnProgress(hash, 100); + + // Assert + _fixture.Cache.DidNotReceive().Remove(Arg.Any()); + } + } + + public class StrikeAndCheckLimitTests : DownloadServiceTests + { + public StrikeAndCheckLimitTests(DownloadServiceFixture fixture) : base(fixture) + { + } + + [Fact] + public async Task ShouldDelegateCallToStriker() + { + // Arrange + const string hash = "test-hash"; + const string itemName = "test-item"; + _fixture.Striker.StrikeAndCheckLimit(hash, itemName, 3, StrikeType.Stalled) + .Returns(true); + + TestDownloadService sut = _fixture.CreateSut(); + + // Act + bool result = await sut.StrikeAndCheckLimit(hash, itemName); + + // Assert + result.ShouldBeTrue(); + await _fixture.Striker + .Received(1) + .StrikeAndCheckLimit(hash, itemName, 3, StrikeType.Stalled); + } + } + + public class ShouldCleanDownloadTests : DownloadServiceTests + { + public ShouldCleanDownloadTests(DownloadServiceFixture fixture) : base(fixture) + { + ContextProvider.Set("downloadName", "test-download"); + } + + [Fact] + public void WhenRatioAndMinSeedTimeReached_ShouldReturnTrue() + { + // Arrange + Category category = new() + { + Name = "test", + MaxRatio = 1.0, + MinSeedTime = 1, + MaxSeedTime = -1 + }; + const double ratio = 1.5; + TimeSpan seedingTime = TimeSpan.FromHours(2); + + TestDownloadService sut = _fixture.CreateSut(); + + // Act + var result = sut.ShouldCleanDownload(ratio, seedingTime, category); + + // Assert + result.ShouldSatisfyAllConditions( + () => result.ShouldClean.ShouldBeTrue(), + () => result.Reason.ShouldBe(CleanReason.MaxRatioReached) + ); + } + + [Fact] + public void WhenRatioReachedAndMinSeedTimeNotReached_ShouldReturnFalse() + { + // Arrange + Category category = new() + { + Name = "test", + MaxRatio = 1.0, + MinSeedTime = 3, + MaxSeedTime = -1 + }; + const double ratio = 1.5; + TimeSpan seedingTime = TimeSpan.FromHours(2); + + TestDownloadService sut = _fixture.CreateSut(); + + // Act + var result = sut.ShouldCleanDownload(ratio, seedingTime, category); + + // Assert + result.ShouldSatisfyAllConditions( + () => result.ShouldClean.ShouldBeFalse(), + () => result.Reason.ShouldBe(CleanReason.None) + ); + } + + [Fact] + public void WhenMaxSeedTimeReached_ShouldReturnTrue() + { + // Arrange + Category category = new() + { + Name = "test", + MaxRatio = -1, + MinSeedTime = 0, + MaxSeedTime = 1 + }; + const double ratio = 0.5; + TimeSpan seedingTime = TimeSpan.FromHours(2); + + TestDownloadService sut = _fixture.CreateSut(); + + // Act + SeedingCheckResult result = sut.ShouldCleanDownload(ratio, seedingTime, category); + + // Assert + result.ShouldSatisfyAllConditions( + () => result.ShouldClean.ShouldBeTrue(), + () => result.Reason.ShouldBe(CleanReason.MaxSeedTimeReached) + ); + } + + [Fact] + public void WhenNeitherConditionMet_ShouldReturnFalse() + { + // Arrange + Category category = new() + { + Name = "test", + MaxRatio = 2.0, + MinSeedTime = 0, + MaxSeedTime = 3 + }; + const double ratio = 1.0; + TimeSpan seedingTime = TimeSpan.FromHours(1); + + TestDownloadService sut = _fixture.CreateSut(); + + // Act + var result = sut.ShouldCleanDownload(ratio, seedingTime, category); + + // Assert + result.ShouldSatisfyAllConditions( + () => result.ShouldClean.ShouldBeFalse(), + () => result.Reason.ShouldBe(CleanReason.None) + ); + } + } +} \ No newline at end of file diff --git a/code/Infrastructure.Tests/Verticals/DownloadClient/TestDownloadService.cs b/code/Infrastructure.Tests/Verticals/DownloadClient/TestDownloadService.cs new file mode 100644 index 00000000..022f113c --- /dev/null +++ b/code/Infrastructure.Tests/Verticals/DownloadClient/TestDownloadService.cs @@ -0,0 +1,45 @@ +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.ContentBlocker; +using Common.Configuration.DownloadCleaner; +using Common.Configuration.QueueCleaner; +using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.DownloadClient; +using Infrastructure.Verticals.ItemStriker; +using Infrastructure.Verticals.Notifications; +using Microsoft.Extensions.Caching.Memory; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; + +namespace Infrastructure.Tests.Verticals.DownloadClient; + +public class TestDownloadService : DownloadService +{ + public TestDownloadService( + ILogger logger, + IOptions queueCleanerConfig, + IOptions contentBlockerConfig, + IOptions downloadCleanerConfig, + IMemoryCache cache, + IFilenameEvaluator filenameEvaluator, + IStriker striker, + NotificationPublisher notifier) + : base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, + cache, filenameEvaluator, striker, notifier) + { + } + + public override void Dispose() { } + public override Task LoginAsync() => Task.CompletedTask; + public override Task ShouldRemoveFromArrQueueAsync(string hash) => Task.FromResult(new StalledResult()); + public override Task BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType, + ConcurrentBag patterns, ConcurrentBag regexes) => Task.FromResult(new BlockFilesResult()); + public override Task DeleteDownload(string hash) => Task.CompletedTask; + public override Task?> GetAllDownloadsToBeCleaned(List categories) => Task.FromResult?>(null); + public override Task CleanDownloads(List downloads, List categoriesToClean, HashSet excludedHashes) => Task.CompletedTask; + + // Expose protected methods for testing + public new void ResetStrikesOnProgress(string hash, long downloaded) => base.ResetStrikesOnProgress(hash, downloaded); + public new Task StrikeAndCheckLimit(string hash, string itemName) => base.StrikeAndCheckLimit(hash, itemName); + public new SeedingCheckResult ShouldCleanDownload(double ratio, TimeSpan seedingTime, Category category) => base.ShouldCleanDownload(ratio, seedingTime, category); +} \ No newline at end of file diff --git a/code/Infrastructure/Infrastructure.csproj b/code/Infrastructure/Infrastructure.csproj index ed985e29..09e6876b 100644 --- a/code/Infrastructure/Infrastructure.csproj +++ b/code/Infrastructure/Infrastructure.csproj @@ -12,13 +12,15 @@ + - - + + + diff --git a/code/Infrastructure/Interceptors/DryRunInterceptor.cs b/code/Infrastructure/Interceptors/DryRunInterceptor.cs new file mode 100644 index 00000000..df632ea6 --- /dev/null +++ b/code/Infrastructure/Interceptors/DryRunInterceptor.cs @@ -0,0 +1,49 @@ +using System.Reflection; +using Castle.DynamicProxy; +using Common.Attributes; +using Common.Configuration.General; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; + +namespace Infrastructure.Interceptors; + +public class DryRunAsyncInterceptor : AsyncInterceptorBase +{ + private readonly ILogger _logger; + private readonly DryRunConfig _config; + + public DryRunAsyncInterceptor(ILogger logger, IOptions config) + { + _logger = logger; + _config = config.Value; + } + + protected override async Task InterceptAsync(IInvocation invocation, IInvocationProceedInfo proceedInfo, Func proceed) + { + MethodInfo? method = invocation.MethodInvocationTarget ?? invocation.Method; + if (IsDryRun(method)) + { + _logger.LogInformation("[DRY RUN] skipping method: {name}", method.Name); + return; + } + + await proceed(invocation, proceedInfo); + } + + protected override async Task InterceptAsync(IInvocation invocation, IInvocationProceedInfo proceedInfo, Func> proceed) + { + MethodInfo? method = invocation.MethodInvocationTarget ?? invocation.Method; + if (IsDryRun(method)) + { + _logger.LogInformation("[DRY RUN] skipping method: {name}", method.Name); + return default!; + } + + return await proceed(invocation, proceedInfo); + } + + private bool IsDryRun(MethodInfo method) + { + return method.GetCustomAttributes(typeof(DryRunSafeguardAttribute), true).Any() && _config.IsDryRun; + } +} diff --git a/code/Infrastructure/Interceptors/IDryRunService.cs b/code/Infrastructure/Interceptors/IDryRunService.cs new file mode 100644 index 00000000..003d1b67 --- /dev/null +++ b/code/Infrastructure/Interceptors/IDryRunService.cs @@ -0,0 +1,5 @@ +namespace Infrastructure.Interceptors; + +public interface IDryRunService : IInterceptedService +{ +} \ No newline at end of file diff --git a/code/Infrastructure/Interceptors/IInterceptedService.cs b/code/Infrastructure/Interceptors/IInterceptedService.cs new file mode 100644 index 00000000..3d70429a --- /dev/null +++ b/code/Infrastructure/Interceptors/IInterceptedService.cs @@ -0,0 +1,6 @@ +namespace Infrastructure.Interceptors; + +public interface IInterceptedService +{ + public object Proxy { get; set; } +} \ No newline at end of file diff --git a/code/Infrastructure/Interceptors/InterceptedService.cs b/code/Infrastructure/Interceptors/InterceptedService.cs new file mode 100644 index 00000000..af92f8df --- /dev/null +++ b/code/Infrastructure/Interceptors/InterceptedService.cs @@ -0,0 +1,21 @@ +namespace Infrastructure.Interceptors; + +public class InterceptedService : IInterceptedService +{ + private object? _proxy; + + public object Proxy + { + get + { + if (_proxy is null) + { + throw new Exception("Proxy is not set"); + } + + return _proxy; + } + + set => _proxy = value; + } +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Arr/ArrClient.cs b/code/Infrastructure/Verticals/Arr/ArrClient.cs index 7baa2fe5..3a1ee1b9 100644 --- a/code/Infrastructure/Verticals/Arr/ArrClient.cs +++ b/code/Infrastructure/Verticals/Arr/ArrClient.cs @@ -1,3 +1,4 @@ +using Common.Attributes; using Common.Configuration.Arr; using Common.Configuration.Logging; using Common.Configuration.QueueCleaner; @@ -5,6 +6,8 @@ using Common.Helpers; using Domain.Enums; using Domain.Models.Arr; using Domain.Models.Arr.Queue; +using Infrastructure.Interceptors; +using Infrastructure.Verticals.Arr.Interfaces; using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; @@ -12,24 +15,30 @@ using Newtonsoft.Json; namespace Infrastructure.Verticals.Arr; -public abstract class ArrClient +public abstract class ArrClient : InterceptedService, IArrClient, IDryRunService { protected readonly ILogger _logger; protected readonly HttpClient _httpClient; protected readonly LoggingConfig _loggingConfig; protected readonly QueueCleanerConfig _queueCleanerConfig; - protected readonly Striker _striker; + protected readonly IStriker _striker; + + /// + /// Constructor to be used by interceptors. + /// + protected ArrClient() + { + } protected ArrClient( ILogger logger, IHttpClientFactory httpClientFactory, IOptions loggingConfig, IOptions queueCleanerConfig, - Striker striker + IStriker striker ) { _logger = logger; - _striker = striker; _httpClient = httpClientFactory.CreateClient(Constants.HttpClientWithRetryName); _loggingConfig = loggingConfig.Value; _queueCleanerConfig = queueCleanerConfig.Value; @@ -110,16 +119,14 @@ public abstract class ArrClient public virtual async Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record, bool removeFromClient) { Uri uri = new(arrInstance.Url, GetQueueDeleteUrlPath(record.Id, removeFromClient)); - - using HttpRequestMessage request = new(HttpMethod.Delete, uri); - SetApiKey(request, arrInstance.ApiKey); - - using HttpResponseMessage response = await _httpClient.SendAsync(request); try { - response.EnsureSuccessStatusCode(); + using HttpRequestMessage request = new(HttpMethod.Delete, uri); + SetApiKey(request, arrInstance.ApiKey); + using var _ = await ((ArrClient)Proxy).SendRequestAsync(request); + _logger.LogInformation( removeFromClient ? "queue item deleted | {url} | {title}" @@ -157,6 +164,16 @@ public abstract class ArrClient request.Headers.Add("x-api-key", apiKey); } + [DryRunSafeguard] + protected virtual async Task SendRequestAsync(HttpRequestMessage request) + { + HttpResponseMessage response = await _httpClient.SendAsync(request); + + response.EnsureSuccessStatusCode(); + + return response; + } + private bool HasIgnoredPatterns(QueueRecord record) { if (_queueCleanerConfig.ImportFailedIgnorePatterns?.Count is null or 0) diff --git a/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs b/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs index 9f87a6c5..2ea9b14a 100644 --- a/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs +++ b/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs @@ -1,6 +1,7 @@ using Common.Configuration; using Common.Configuration.Arr; using Domain.Models.Arr.Queue; +using Infrastructure.Verticals.Arr.Interfaces; using Microsoft.Extensions.Logging; namespace Infrastructure.Verticals.Arr; @@ -14,7 +15,7 @@ public sealed class ArrQueueIterator _logger = logger; } - public async Task Iterate(ArrClient arrClient, ArrInstance arrInstance, Func, Task> action) + public async Task Iterate(IArrClient arrClient, ArrInstance arrInstance, Func, Task> action) { const ushort maxPage = 100; ushort page = 1; diff --git a/code/Infrastructure/Verticals/Arr/Interfaces/IArrClient.cs b/code/Infrastructure/Verticals/Arr/Interfaces/IArrClient.cs new file mode 100644 index 00000000..4435a054 --- /dev/null +++ b/code/Infrastructure/Verticals/Arr/Interfaces/IArrClient.cs @@ -0,0 +1,19 @@ +using Common.Configuration.Arr; +using Domain.Enums; +using Domain.Models.Arr; +using Domain.Models.Arr.Queue; + +namespace Infrastructure.Verticals.Arr.Interfaces; + +public interface IArrClient +{ + Task GetQueueItemsAsync(ArrInstance arrInstance, int page); + + Task ShouldRemoveFromQueue(InstanceType instanceType, QueueRecord record, bool isPrivateDownload); + + Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record, bool removeFromClient); + + Task RefreshItemsAsync(ArrInstance arrInstance, HashSet? items); + + bool IsRecordValid(QueueRecord record); +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Arr/Interfaces/ILidarrClient.cs b/code/Infrastructure/Verticals/Arr/Interfaces/ILidarrClient.cs new file mode 100644 index 00000000..9a5cb3b0 --- /dev/null +++ b/code/Infrastructure/Verticals/Arr/Interfaces/ILidarrClient.cs @@ -0,0 +1,5 @@ +namespace Infrastructure.Verticals.Arr.Interfaces; + +public interface ILidarrClient : IArrClient +{ +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Arr/Interfaces/IRadarrClient.cs b/code/Infrastructure/Verticals/Arr/Interfaces/IRadarrClient.cs new file mode 100644 index 00000000..71b0cff0 --- /dev/null +++ b/code/Infrastructure/Verticals/Arr/Interfaces/IRadarrClient.cs @@ -0,0 +1,5 @@ +namespace Infrastructure.Verticals.Arr.Interfaces; + +public interface IRadarrClient : IArrClient +{ +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Arr/Interfaces/ISonarrClient.cs b/code/Infrastructure/Verticals/Arr/Interfaces/ISonarrClient.cs new file mode 100644 index 00000000..7863f7f7 --- /dev/null +++ b/code/Infrastructure/Verticals/Arr/Interfaces/ISonarrClient.cs @@ -0,0 +1,5 @@ +namespace Infrastructure.Verticals.Arr.Interfaces; + +public interface ISonarrClient : IArrClient +{ +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Arr/LidarrClient.cs b/code/Infrastructure/Verticals/Arr/LidarrClient.cs index ea495183..fb8171f6 100644 --- a/code/Infrastructure/Verticals/Arr/LidarrClient.cs +++ b/code/Infrastructure/Verticals/Arr/LidarrClient.cs @@ -5,6 +5,7 @@ using Common.Configuration.QueueCleaner; using Domain.Models.Arr; using Domain.Models.Arr.Queue; using Domain.Models.Lidarr; +using Infrastructure.Verticals.Arr.Interfaces; using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; @@ -12,14 +13,19 @@ using Newtonsoft.Json; namespace Infrastructure.Verticals.Arr; -public sealed class LidarrClient : ArrClient +public class LidarrClient : ArrClient, ILidarrClient { + /// + public LidarrClient() + { + } + public LidarrClient( ILogger logger, IHttpClientFactory httpClientFactory, IOptions loggingConfig, IOptions queueCleanerConfig, - Striker striker + IStriker striker ) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker) { } @@ -54,13 +60,12 @@ public sealed class LidarrClient : ArrClient ); SetApiKey(request, arrInstance.ApiKey); - using var response = await _httpClient.SendAsync(request); string? logContext = await ComputeCommandLogContextAsync(arrInstance, command); try { - response.EnsureSuccessStatusCode(); - + using var _ = await ((LidarrClient)Proxy).SendRequestAsync(request); + _logger.LogInformation("{log}", GetSearchLog(arrInstance.Url, command, true, logContext)); } catch diff --git a/code/Infrastructure/Verticals/Arr/RadarrClient.cs b/code/Infrastructure/Verticals/Arr/RadarrClient.cs index 488b2b09..e0e31925 100644 --- a/code/Infrastructure/Verticals/Arr/RadarrClient.cs +++ b/code/Infrastructure/Verticals/Arr/RadarrClient.cs @@ -5,6 +5,7 @@ using Common.Configuration.QueueCleaner; using Domain.Models.Arr; using Domain.Models.Arr.Queue; using Domain.Models.Radarr; +using Infrastructure.Verticals.Arr.Interfaces; using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; @@ -12,14 +13,19 @@ using Newtonsoft.Json; namespace Infrastructure.Verticals.Arr; -public sealed class RadarrClient : ArrClient +public class RadarrClient : ArrClient, IRadarrClient { + /// + public RadarrClient() + { + } + public RadarrClient( ILogger logger, IHttpClientFactory httpClientFactory, IOptions loggingConfig, IOptions queueCleanerConfig, - Striker striker + IStriker striker ) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker) { } @@ -62,12 +68,11 @@ public sealed class RadarrClient : ArrClient ); SetApiKey(request, arrInstance.ApiKey); - using HttpResponseMessage response = await _httpClient.SendAsync(request); string? logContext = await ComputeCommandLogContextAsync(arrInstance, command); try { - response.EnsureSuccessStatusCode(); + using var _ = await ((RadarrClient)Proxy).SendRequestAsync(request); _logger.LogInformation("{log}", GetSearchLog(arrInstance.Url, command, true, logContext)); } diff --git a/code/Infrastructure/Verticals/Arr/SonarrClient.cs b/code/Infrastructure/Verticals/Arr/SonarrClient.cs index 20ad6513..93156696 100644 --- a/code/Infrastructure/Verticals/Arr/SonarrClient.cs +++ b/code/Infrastructure/Verticals/Arr/SonarrClient.cs @@ -5,6 +5,7 @@ using Common.Configuration.QueueCleaner; using Domain.Models.Arr; using Domain.Models.Arr.Queue; using Domain.Models.Sonarr; +using Infrastructure.Verticals.Arr.Interfaces; using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; @@ -13,14 +14,19 @@ using Series = Domain.Models.Sonarr.Series; namespace Infrastructure.Verticals.Arr; -public sealed class SonarrClient : ArrClient +public class SonarrClient : ArrClient, ISonarrClient { + /// + public SonarrClient() + { + } + public SonarrClient( ILogger logger, IHttpClientFactory httpClientFactory, IOptions loggingConfig, IOptions queueCleanerConfig, - Striker striker + IStriker striker ) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker) { } @@ -58,13 +64,12 @@ public sealed class SonarrClient : ArrClient ); SetApiKey(request, arrInstance.ApiKey); - using HttpResponseMessage response = await _httpClient.SendAsync(request); string? logContext = await ComputeCommandLogContextAsync(arrInstance, command, command.SearchType); try { - response.EnsureSuccessStatusCode(); - + using var _ = await ((SonarrClient)Proxy).SendRequestAsync(request); + _logger.LogInformation("{log}", GetSearchLog(command.SearchType, arrInstance.Url, command, true, logContext)); } catch diff --git a/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs b/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs index d70f9062..8f6284b5 100644 --- a/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs +++ b/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs @@ -7,6 +7,7 @@ using Domain.Enums; using Domain.Models.Arr; using Domain.Models.Arr.Queue; using Infrastructure.Verticals.Arr; +using Infrastructure.Verticals.Arr.Interfaces; using Infrastructure.Verticals.Context; using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.Jobs; @@ -36,7 +37,6 @@ public sealed class ContentBlocker : GenericHandler BlocklistProvider blocklistProvider, DownloadServiceFactory downloadServiceFactory, NotificationPublisher notifier - ) : base( logger, downloadClientConfig, sonarrConfig, radarrConfig, lidarrConfig, @@ -76,7 +76,7 @@ public sealed class ContentBlocker : GenericHandler using var _ = LogContext.PushProperty("InstanceName", instanceType.ToString()); HashSet itemsToBeRefreshed = []; - ArrClient arrClient = GetClient(instanceType); + IArrClient arrClient = GetClient(instanceType); BlocklistType blocklistType = _blocklistProvider.GetBlocklistType(instanceType); ConcurrentBag patterns = _blocklistProvider.GetPatterns(instanceType); ConcurrentBag regexes = _blocklistProvider.GetRegexes(instanceType); @@ -131,7 +131,7 @@ public sealed class ContentBlocker : GenericHandler } await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient); - await _notifier.NotifyQueueItemDelete(removeFromClient, DeleteReason.AllFilesBlocked); + await _notifier.NotifyQueueItemDeleted(removeFromClient, DeleteReason.AllFilesBlocked); } }); diff --git a/code/Infrastructure/Verticals/ContentBlocker/FilenameEvaluator.cs b/code/Infrastructure/Verticals/ContentBlocker/FilenameEvaluator.cs index 23b1f452..be7738e1 100644 --- a/code/Infrastructure/Verticals/ContentBlocker/FilenameEvaluator.cs +++ b/code/Infrastructure/Verticals/ContentBlocker/FilenameEvaluator.cs @@ -5,7 +5,7 @@ using Microsoft.Extensions.Logging; namespace Infrastructure.Verticals.ContentBlocker; -public sealed class FilenameEvaluator +public class FilenameEvaluator : IFilenameEvaluator { private readonly ILogger _logger; @@ -31,7 +31,6 @@ public sealed class FilenameEvaluator { BlocklistType.Blacklist => !patterns.Any(pattern => MatchesPattern(filename, pattern)), BlocklistType.Whitelist => patterns.Any(pattern => MatchesPattern(filename, pattern)), - _ => true }; } @@ -46,7 +45,6 @@ public sealed class FilenameEvaluator { BlocklistType.Blacklist => !regexes.Any(regex => regex.IsMatch(filename)), BlocklistType.Whitelist => regexes.Any(regex => regex.IsMatch(filename)), - _ => true }; } @@ -76,6 +74,6 @@ public sealed class FilenameEvaluator ); } - return filename == pattern; + return filename.Equals(pattern, StringComparison.InvariantCultureIgnoreCase); } } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/ContentBlocker/IFilenameEvaluator.cs b/code/Infrastructure/Verticals/ContentBlocker/IFilenameEvaluator.cs new file mode 100644 index 00000000..91e69c00 --- /dev/null +++ b/code/Infrastructure/Verticals/ContentBlocker/IFilenameEvaluator.cs @@ -0,0 +1,10 @@ +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.ContentBlocker; + +namespace Infrastructure.Verticals.ContentBlocker; + +public interface IFilenameEvaluator +{ + bool IsValid(string filename, BlocklistType type, ConcurrentBag patterns, ConcurrentBag regexes); +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Context/ContextProvider.cs b/code/Infrastructure/Verticals/Context/ContextProvider.cs index 071b91e0..89adecba 100644 --- a/code/Infrastructure/Verticals/Context/ContextProvider.cs +++ b/code/Infrastructure/Verticals/Context/ContextProvider.cs @@ -17,8 +17,8 @@ public static class ContextProvider return _asyncLocalDict.Value?.TryGetValue(key, out object? value) is true ? value : null; } - public static T? Get(string key) where T : class + public static T Get(string key) where T : class { - return Get(key) as T; + return Get(key) as T ?? throw new Exception($"failed to get \"{key}\" from context"); } } diff --git a/code/Infrastructure/Verticals/DownloadCleaner/DownloadCleaner.cs b/code/Infrastructure/Verticals/DownloadCleaner/DownloadCleaner.cs new file mode 100644 index 00000000..27b3a566 --- /dev/null +++ b/code/Infrastructure/Verticals/DownloadCleaner/DownloadCleaner.cs @@ -0,0 +1,99 @@ +using Common.Configuration.Arr; +using Common.Configuration.DownloadCleaner; +using Common.Configuration.DownloadClient; +using Domain.Enums; +using Domain.Models.Arr.Queue; +using Infrastructure.Verticals.Arr; +using Infrastructure.Verticals.Arr.Interfaces; +using Infrastructure.Verticals.DownloadClient; +using Infrastructure.Verticals.Jobs; +using Infrastructure.Verticals.Notifications; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using Serilog.Context; + +namespace Infrastructure.Verticals.DownloadCleaner; + +public sealed class DownloadCleaner : GenericHandler +{ + private readonly DownloadCleanerConfig _config; + private readonly HashSet _excludedHashes = []; + + public DownloadCleaner( + ILogger logger, + IOptions config, + IOptions downloadClientConfig, + IOptions sonarrConfig, + IOptions radarrConfig, + IOptions lidarrConfig, + SonarrClient sonarrClient, + RadarrClient radarrClient, + LidarrClient lidarrClient, + ArrQueueIterator arrArrQueueIterator, + DownloadServiceFactory downloadServiceFactory, + NotificationPublisher notifier + ) : base( + logger, downloadClientConfig, + sonarrConfig, radarrConfig, lidarrConfig, + sonarrClient, radarrClient, lidarrClient, + arrArrQueueIterator, downloadServiceFactory, + notifier + ) + { + _config = config.Value; + _config.Validate(); + } + + public override async Task ExecuteAsync() + { + if (_config.Categories?.Count is null or 0) + { + _logger.LogWarning("no categories configured"); + return; + } + + await _downloadService.LoginAsync(); + + List? downloads = await _downloadService.GetAllDownloadsToBeCleaned(_config.Categories); + + if (downloads?.Count is null or 0) + { + _logger.LogDebug("no downloads found in the download client"); + return; + } + + // wait for the downloads to appear in the arr queue + await Task.Delay(10 * 1000); + + await ProcessArrConfigAsync(_sonarrConfig, InstanceType.Sonarr, true); + await ProcessArrConfigAsync(_radarrConfig, InstanceType.Radarr, true); + await ProcessArrConfigAsync(_lidarrConfig, InstanceType.Lidarr, true); + + await _downloadService.CleanDownloads(downloads, _config.Categories, _excludedHashes); + } + + protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType) + { + using var _ = LogContext.PushProperty("InstanceName", instanceType.ToString()); + + IArrClient arrClient = GetClient(instanceType); + + await _arrArrQueueIterator.Iterate(arrClient, instance, async items => + { + var groups = items + .Where(x => !string.IsNullOrEmpty(x.DownloadId)) + .GroupBy(x => x.DownloadId) + .ToList(); + + foreach (QueueRecord record in groups.Select(group => group.First())) + { + _excludedHashes.Add(record.DownloadId.ToLowerInvariant()); + } + }); + } + + public override void Dispose() + { + _downloadService.Dispose(); + } +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeClient.cs b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeClient.cs index aad8b31b..dd29288e 100644 --- a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeClient.cs +++ b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeClient.cs @@ -62,6 +62,26 @@ public sealed class DelugeClient await ListTorrentsExtended(new Dictionary { { "hash", hash } }); return torrents.FirstOrDefault(); } + + public async Task GetTorrentStatus(string hash) + { + return await SendRequest( + "web.get_torrent_status", + hash, + new[] { "hash", "state", "name", "eta", "private", "total_done", "label", "seeding_time", "ratio" } + ); + } + + public async Task?> GetStatusForAllTorrents() + { + Dictionary? downloads = await SendRequest?>( + "core.get_torrents_status", + "", + new[] { "hash", "state", "name", "eta", "private", "total_done", "label", "seeding_time", "ratio" } + ); + + return downloads?.Values.ToList(); + } public async Task GetTorrentFiles(string hash) { @@ -78,9 +98,9 @@ public sealed class DelugeClient await SendRequest>("core.set_torrent_options", hash, filePriorities); } - public async Task> DeleteTorrent(string hash) + public async Task DeleteTorrents(List hashes) { - return await SendRequest>("core.remove_torrents", new List { hash }, true); + await SendRequest>("core.remove_torrents", hashes, true); } private async Task PostJson(String json) diff --git a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs index 067b2b70..58924608 100644 --- a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs @@ -1,21 +1,31 @@ using System.Collections.Concurrent; using System.Text.RegularExpressions; +using Common.Attributes; using Common.Configuration.ContentBlocker; +using Common.Configuration.DownloadCleaner; using Common.Configuration.DownloadClient; using Common.Configuration.QueueCleaner; using Domain.Enums; using Domain.Models.Deluge.Response; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.Context; using Infrastructure.Verticals.ItemStriker; +using Infrastructure.Verticals.Notifications; +using MassTransit.Configuration; using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; namespace Infrastructure.Verticals.DownloadClient.Deluge; -public sealed class DelugeService : DownloadServiceBase +public class DelugeService : DownloadService, IDelugeService { private readonly DelugeClient _client; + + /// + public DelugeService() + { + } public DelugeService( ILogger logger, @@ -23,10 +33,12 @@ public sealed class DelugeService : DownloadServiceBase IHttpClientFactory httpClientFactory, IOptions queueCleanerConfig, IOptions contentBlockerConfig, + IOptions downloadCleanerConfig, IMemoryCache cache, - FilenameEvaluator filenameEvaluator, - Striker striker - ) : base(logger, queueCleanerConfig, contentBlockerConfig, cache, filenameEvaluator, striker) + IFilenameEvaluator filenameEvaluator, + IStriker striker, + NotificationPublisher notifier + ) : base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache, filenameEvaluator, striker, notifier) { config.Value.Validate(); _client = new (config, httpClientFactory); @@ -45,7 +57,7 @@ public sealed class DelugeService : DownloadServiceBase DelugeContents? contents = null; StalledResult result = new(); - TorrentStatus? status = await GetTorrentStatus(hash); + TorrentStatus? status = await _client.GetTorrentStatus(hash); if (status?.Hash is null) { @@ -98,7 +110,7 @@ public sealed class DelugeService : DownloadServiceBase { hash = hash.ToLowerInvariant(); - TorrentStatus? status = await GetTorrentStatus(hash); + TorrentStatus? status = await _client.GetTorrentStatus(hash); BlockFilesResult result = new(); if (status?.Hash is null) @@ -178,17 +190,89 @@ public sealed class DelugeService : DownloadServiceBase return result; } - await _client.ChangeFilesPriority(hash, sortedPriorities); + await ((DelugeService)Proxy).ChangeFilesPriority(hash, sortedPriorities); return result; } + + public override async Task?> GetAllDownloadsToBeCleaned(List categories) + { + return (await _client.GetStatusForAllTorrents()) + ?.Where(x => !string.IsNullOrEmpty(x.Hash)) + .Where(x => x.State?.Equals("seeding", StringComparison.InvariantCultureIgnoreCase) is true) + .Where(x => categories.Any(cat => cat.Name.Equals(x.Label, StringComparison.InvariantCultureIgnoreCase))) + .Cast() + .ToList(); + } + + /// + public override async Task CleanDownloads(List downloads, List categoriesToClean, HashSet excludedHashes) + { + foreach (TorrentStatus download in downloads) + { + if (string.IsNullOrEmpty(download.Hash)) + { + continue; + } + + Category? category = categoriesToClean + .FirstOrDefault(x => x.Name.Equals(download.Label, StringComparison.InvariantCultureIgnoreCase)); + + if (category is null) + { + continue; + } + + if (excludedHashes.Any(x => x.Equals(download.Hash, StringComparison.InvariantCultureIgnoreCase))) + { + _logger.LogDebug("skip | download is used by an arr | {name}", download.Name); + continue; + } + + if (!_downloadCleanerConfig.DeletePrivate && download.Private) + { + _logger.LogDebug("skip | download is private | {name}", download.Name); + continue; + } + + ContextProvider.Set("downloadName", download.Name); + ContextProvider.Set("hash", download.Hash); + + TimeSpan seedingTime = TimeSpan.FromSeconds(download.SeedingTime); + SeedingCheckResult result = ShouldCleanDownload(download.Ratio, seedingTime, category); + + if (!result.ShouldClean) + { + continue; + } + + await ((DelugeService)Proxy).DeleteDownload(download.Hash); + + _logger.LogInformation( + "download cleaned | {reason} reached | {name}", + result.Reason is CleanReason.MaxRatioReached + ? "MAX_RATIO & MIN_SEED_TIME" + : "MAX_SEED_TIME", + download.Name + ); + + await _notifier.NotifyDownloadCleaned(download.Ratio, seedingTime, category.Name, result.Reason); + } + } /// - public override async Task Delete(string hash) + [DryRunSafeguard] + public override async Task DeleteDownload(string hash) { hash = hash.ToLowerInvariant(); - await _client.DeleteTorrent(hash); + await _client.DeleteTorrents([hash]); + } + + [DryRunSafeguard] + protected virtual async Task ChangeFilesPriority(string hash, List sortedPriorities) + { + await _client.ChangeFilesPriority(hash, sortedPriorities); } private async Task IsItemStuckAndShouldRemove(TorrentStatus status) @@ -219,15 +303,6 @@ public sealed class DelugeService : DownloadServiceBase return await StrikeAndCheckLimit(status.Hash!, status.Name!); } - - private async Task GetTorrentStatus(string hash) - { - return await _client.SendRequest( - "web.get_torrent_status", - hash, - new[] { "hash", "state", "name", "eta", "private", "total_done" } - ); - } private static void ProcessFiles(Dictionary? contents, Action processFile) { diff --git a/code/Infrastructure/Verticals/DownloadClient/Deluge/IDelugeService.cs b/code/Infrastructure/Verticals/DownloadClient/Deluge/IDelugeService.cs new file mode 100644 index 00000000..0c516fb2 --- /dev/null +++ b/code/Infrastructure/Verticals/DownloadClient/Deluge/IDelugeService.cs @@ -0,0 +1,5 @@ +namespace Infrastructure.Verticals.DownloadClient.Deluge; + +public interface IDelugeService : IDownloadService +{ +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/DownloadService.cs b/code/Infrastructure/Verticals/DownloadClient/DownloadService.cs new file mode 100644 index 00000000..e32ed70b --- /dev/null +++ b/code/Infrastructure/Verticals/DownloadClient/DownloadService.cs @@ -0,0 +1,183 @@ +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.ContentBlocker; +using Common.Configuration.DownloadCleaner; +using Common.Configuration.QueueCleaner; +using Common.Helpers; +using Domain.Enums; +using Domain.Models.Cache; +using Infrastructure.Helpers; +using Infrastructure.Interceptors; +using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.Context; +using Infrastructure.Verticals.ItemStriker; +using Infrastructure.Verticals.Notifications; +using Microsoft.Extensions.Caching.Memory; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; + +namespace Infrastructure.Verticals.DownloadClient; + +public abstract class DownloadService : InterceptedService, IDownloadService +{ + protected readonly ILogger _logger; + protected readonly QueueCleanerConfig _queueCleanerConfig; + protected readonly ContentBlockerConfig _contentBlockerConfig; + protected readonly DownloadCleanerConfig _downloadCleanerConfig; + protected readonly IMemoryCache _cache; + protected readonly IFilenameEvaluator _filenameEvaluator; + protected readonly IStriker _striker; + protected readonly MemoryCacheEntryOptions _cacheOptions; + protected readonly NotificationPublisher _notifier; + + /// + /// Constructor to be used by interceptors. + /// + protected DownloadService() + { + } + + protected DownloadService( + ILogger logger, + IOptions queueCleanerConfig, + IOptions contentBlockerConfig, + IOptions downloadCleanerConfig, + IMemoryCache cache, + IFilenameEvaluator filenameEvaluator, + IStriker striker, + NotificationPublisher notifier) + { + _logger = logger; + _queueCleanerConfig = queueCleanerConfig.Value; + _contentBlockerConfig = contentBlockerConfig.Value; + _downloadCleanerConfig = downloadCleanerConfig.Value; + _cache = cache; + _filenameEvaluator = filenameEvaluator; + _striker = striker; + _notifier = notifier; + _cacheOptions = new MemoryCacheEntryOptions() + .SetSlidingExpiration(StaticConfiguration.TriggerValue + Constants.CacheLimitBuffer); + } + + public abstract void Dispose(); + + public abstract Task LoginAsync(); + + public abstract Task ShouldRemoveFromArrQueueAsync(string hash); + + /// + public abstract Task BlockUnwantedFilesAsync( + string hash, + BlocklistType blocklistType, + ConcurrentBag patterns, + ConcurrentBag regexes + ); + + /// + public abstract Task DeleteDownload(string hash); + + /// + public abstract Task?> GetAllDownloadsToBeCleaned(List categories); + + /// + public abstract Task CleanDownloads(List downloads, List categoriesToClean, HashSet excludedHashes); + + protected void ResetStrikesOnProgress(string hash, long downloaded) + { + if (!_queueCleanerConfig.StalledResetStrikesOnProgress) + { + return; + } + + if (_cache.TryGetValue(CacheKeys.Item(hash), out CacheItem? cachedItem) && cachedItem is not null && downloaded > cachedItem.Downloaded) + { + // cache item found + _cache.Remove(CacheKeys.Strike(StrikeType.Stalled, hash)); + _logger.LogDebug("resetting strikes for {hash} due to progress", hash); + } + + _cache.Set(CacheKeys.Item(hash), new CacheItem { Downloaded = downloaded }, _cacheOptions); + } + + /// + /// Strikes an item and checks if the limit has been reached. + /// + /// The torrent hash. + /// The name or title of the item. + /// True if the limit has been reached; otherwise, false. + protected async Task StrikeAndCheckLimit(string hash, string itemName) + { + return await _striker.StrikeAndCheckLimit(hash, itemName, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled); + } + + protected SeedingCheckResult ShouldCleanDownload(double ratio, TimeSpan seedingTime, Category category) + { + // check ratio + if (DownloadReachedRatio(ratio, seedingTime, category)) + { + return new() + { + ShouldClean = true, + Reason = CleanReason.MaxRatioReached + }; + } + + // check max seed time + if (DownloadReachedMaxSeedTime(seedingTime, category)) + { + return new() + { + ShouldClean = true, + Reason = CleanReason.MaxSeedTimeReached + }; + } + + return new(); + } + + private bool DownloadReachedRatio(double ratio, TimeSpan seedingTime, Category category) + { + if (category.MaxRatio < 0) + { + return false; + } + + string downloadName = ContextProvider.Get("downloadName"); + TimeSpan minSeedingTime = TimeSpan.FromHours(category.MinSeedTime); + + if (category.MinSeedTime > 0 && seedingTime < minSeedingTime) + { + _logger.LogDebug("skip | download has not reached MIN_SEED_TIME | {name}", downloadName); + return false; + } + + if (ratio < category.MaxRatio) + { + _logger.LogDebug("skip | download has not reached MAX_RATIO | {name}", downloadName); + return false; + } + + // max ration is 0 or reached + return true; + } + + private bool DownloadReachedMaxSeedTime(TimeSpan seedingTime, Category category) + { + if (category.MaxSeedTime < 0) + { + return false; + } + + string downloadName = ContextProvider.Get("downloadName"); + TimeSpan maxSeedingTime = TimeSpan.FromHours(category.MaxSeedTime); + + if (category.MaxSeedTime > 0 && seedingTime < maxSeedingTime) + { + _logger.LogDebug("skip | download has not reached MAX_SEED_TIME | {name}", downloadName); + return false; + } + + // max seed time is 0 or reached + return true; + } +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs b/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs deleted file mode 100644 index 1d896cb3..00000000 --- a/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs +++ /dev/null @@ -1,90 +0,0 @@ -using System.Collections.Concurrent; -using System.Text.RegularExpressions; -using Common.Configuration.ContentBlocker; -using Common.Configuration.QueueCleaner; -using Common.Helpers; -using Domain.Enums; -using Domain.Models.Cache; -using Infrastructure.Helpers; -using Infrastructure.Verticals.ContentBlocker; -using Infrastructure.Verticals.ItemStriker; -using Microsoft.Extensions.Caching.Memory; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Options; - -namespace Infrastructure.Verticals.DownloadClient; - -public abstract class DownloadServiceBase : IDownloadService -{ - protected readonly ILogger _logger; - protected readonly QueueCleanerConfig _queueCleanerConfig; - protected readonly ContentBlockerConfig _contentBlockerConfig; - protected readonly IMemoryCache _cache; - protected readonly FilenameEvaluator _filenameEvaluator; - protected readonly Striker _striker; - protected readonly MemoryCacheEntryOptions _cacheOptions; - - protected DownloadServiceBase( - ILogger logger, - IOptions queueCleanerConfig, - IOptions contentBlockerConfig, - IMemoryCache cache, - FilenameEvaluator filenameEvaluator, - Striker striker - ) - { - _logger = logger; - _queueCleanerConfig = queueCleanerConfig.Value; - _contentBlockerConfig = contentBlockerConfig.Value; - _cache = cache; - _filenameEvaluator = filenameEvaluator; - _striker = striker; - _cacheOptions = new MemoryCacheEntryOptions() - .SetSlidingExpiration(StaticConfiguration.TriggerValue + Constants.CacheLimitBuffer); - } - - public abstract void Dispose(); - - public abstract Task LoginAsync(); - - public abstract Task ShouldRemoveFromArrQueueAsync(string hash); - - /// - public abstract Task BlockUnwantedFilesAsync( - string hash, - BlocklistType blocklistType, - ConcurrentBag patterns, - ConcurrentBag regexes - ); - - /// - public abstract Task Delete(string hash); - - protected void ResetStrikesOnProgress(string hash, long downloaded) - { - if (!_queueCleanerConfig.StalledResetStrikesOnProgress) - { - return; - } - - if (_cache.TryGetValue(CacheKeys.Item(hash), out CacheItem? cachedItem) && cachedItem is not null && downloaded > cachedItem.Downloaded) - { - // cache item found - _cache.Remove(CacheKeys.Strike(StrikeType.Stalled, hash)); - _logger.LogDebug("resetting strikes for {hash} due to progress", hash); - } - - _cache.Set(CacheKeys.Item(hash), new CacheItem { Downloaded = downloaded }, _cacheOptions); - } - - /// - /// Strikes an item and checks if the limit has been reached. - /// - /// The torrent hash. - /// The name or title of the item. - /// True if the limit has been reached; otherwise, false. - protected async Task StrikeAndCheckLimit(string hash, string itemName) - { - return await _striker.StrikeAndCheckLimit(hash, itemName, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled); - } -} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/DummyDownloadService.cs b/code/Infrastructure/Verticals/DownloadClient/DummyDownloadService.cs index 41f11bee..413c6675 100644 --- a/code/Infrastructure/Verticals/DownloadClient/DummyDownloadService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/DummyDownloadService.cs @@ -1,18 +1,20 @@ using System.Collections.Concurrent; using System.Text.RegularExpressions; using Common.Configuration.ContentBlocker; +using Common.Configuration.DownloadCleaner; using Common.Configuration.QueueCleaner; using Infrastructure.Verticals.ContentBlocker; using Infrastructure.Verticals.ItemStriker; +using Infrastructure.Verticals.Notifications; using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; namespace Infrastructure.Verticals.DownloadClient; -public sealed class DummyDownloadService : DownloadServiceBase +public sealed class DummyDownloadService : DownloadService { - public DummyDownloadService(ILogger logger, IOptions queueCleanerConfig, IOptions contentBlockerConfig, IMemoryCache cache, FilenameEvaluator filenameEvaluator, Striker striker) : base(logger, queueCleanerConfig, contentBlockerConfig, cache, filenameEvaluator, striker) + public DummyDownloadService(ILogger logger, IOptions queueCleanerConfig, IOptions contentBlockerConfig, IOptions downloadCleanerConfig, IMemoryCache cache, IFilenameEvaluator filenameEvaluator, IStriker striker, NotificationPublisher notifier) : base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache, filenameEvaluator, striker, notifier) { } @@ -35,7 +37,17 @@ public sealed class DummyDownloadService : DownloadServiceBase throw new NotImplementedException(); } - public override Task Delete(string hash) + public override Task?> GetAllDownloadsToBeCleaned(List categories) + { + throw new NotImplementedException(); + } + + public override Task CleanDownloads(List downloads, List categoriesToClean, HashSet excludedHashes) + { + throw new NotImplementedException(); + } + + public override Task DeleteDownload(string hash) { throw new NotImplementedException(); } diff --git a/code/Infrastructure/Verticals/DownloadClient/IDownloadService.cs b/code/Infrastructure/Verticals/DownloadClient/IDownloadService.cs index 641239fd..d5975e8d 100644 --- a/code/Infrastructure/Verticals/DownloadClient/IDownloadService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/IDownloadService.cs @@ -1,10 +1,12 @@ using System.Collections.Concurrent; using System.Text.RegularExpressions; using Common.Configuration.ContentBlocker; +using Common.Configuration.DownloadCleaner; +using Infrastructure.Interceptors; namespace Infrastructure.Verticals.DownloadClient; -public interface IDownloadService : IDisposable +public interface IDownloadService : IDisposable, IDryRunService { public Task LoginAsync(); @@ -29,8 +31,23 @@ public interface IDownloadService : IDisposable ConcurrentBag regexes ); + /// + /// Fetches all downloads. + /// + /// The categories by which to filter the downloads. + /// A list of downloads for the provided categories. + Task?> GetAllDownloadsToBeCleaned(List categories); + + /// + /// Cleans the downloads. + /// + /// + /// The categories that should be cleaned. + /// The hashes that should not be cleaned. + public abstract Task CleanDownloads(List downloads, List categoriesToClean, HashSet excludedHashes); + /// /// Deletes a download item. /// - public Task Delete(string hash); + public Task DeleteDownload(string hash); } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/QBittorrent/IQBitService.cs b/code/Infrastructure/Verticals/DownloadClient/QBittorrent/IQBitService.cs new file mode 100644 index 00000000..fa89196d --- /dev/null +++ b/code/Infrastructure/Verticals/DownloadClient/QBittorrent/IQBitService.cs @@ -0,0 +1,5 @@ +namespace Infrastructure.Verticals.DownloadClient.QBittorrent; + +public interface IQBitService : IDownloadService +{ +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs b/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs index d082856f..f1c2d670 100644 --- a/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs @@ -1,34 +1,46 @@ using System.Collections.Concurrent; using System.Text.RegularExpressions; +using Common.Attributes; using Common.Configuration.ContentBlocker; +using Common.Configuration.DownloadCleaner; using Common.Configuration.DownloadClient; using Common.Configuration.QueueCleaner; using Common.Helpers; using Domain.Enums; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.Context; using Infrastructure.Verticals.ItemStriker; +using Infrastructure.Verticals.Notifications; using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using QBittorrent.Client; +using Category = Common.Configuration.DownloadCleaner.Category; namespace Infrastructure.Verticals.DownloadClient.QBittorrent; -public sealed class QBitService : DownloadServiceBase +public class QBitService : DownloadService, IQBitService { private readonly QBitConfig _config; private readonly QBittorrentClient _client; + /// + public QBitService() + { + } + public QBitService( ILogger logger, IHttpClientFactory httpClientFactory, IOptions config, IOptions queueCleanerConfig, IOptions contentBlockerConfig, + IOptions downloadCleanerConfig, IMemoryCache cache, - FilenameEvaluator filenameEvaluator, - Striker striker - ) : base(logger, queueCleanerConfig, contentBlockerConfig, cache, filenameEvaluator, striker) + IFilenameEvaluator filenameEvaluator, + IStriker striker, + NotificationPublisher notifier + ) : base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache, filenameEvaluator, striker, notifier) { _config = config.Value; _config.Validate(); @@ -188,17 +200,98 @@ public sealed class QBitService : DownloadServiceBase foreach (int fileIndex in unwantedFiles) { - await _client.SetFilePriorityAsync(hash, fileIndex, TorrentContentPriority.Skip); + await ((QBitService)Proxy).SkipFile(hash, fileIndex); } return result; } + + /// + public override async Task?> GetAllDownloadsToBeCleaned(List categories) => + (await _client.GetTorrentListAsync(new() + { + Filter = TorrentListFilter.Seeding + })) + ?.Where(x => !string.IsNullOrEmpty(x.Hash)) + .Where(x => categories.Any(cat => cat.Name.Equals(x.Category, StringComparison.InvariantCultureIgnoreCase))) + .Cast() + .ToList(); /// - public override async Task Delete(string hash) + public override async Task CleanDownloads(List downloads, List categoriesToClean, HashSet excludedHashes) + { + foreach (TorrentInfo download in downloads) + { + if (string.IsNullOrEmpty(download.Hash)) + { + continue; + } + + Category? category = categoriesToClean + .FirstOrDefault(x => download.Category.Equals(x.Name, StringComparison.InvariantCultureIgnoreCase)); + + if (category is null) + { + continue; + } + + if (excludedHashes.Any(x => x.Equals(download.Hash, StringComparison.InvariantCultureIgnoreCase))) + { + _logger.LogDebug("skip | download is used by an arr | {name}", download.Name); + continue; + } + + if (!_downloadCleanerConfig.DeletePrivate) + { + TorrentProperties? torrentProperties = await _client.GetTorrentPropertiesAsync(download.Hash); + + bool isPrivate = torrentProperties.AdditionalData.TryGetValue("is_private", out var dictValue) && + bool.TryParse(dictValue?.ToString(), out bool boolValue) + && boolValue; + + if (isPrivate) + { + _logger.LogDebug("skip | download is private | {name}", download.Name); + continue; + } + } + + ContextProvider.Set("downloadName", download.Name); + ContextProvider.Set("hash", download.Hash); + + SeedingCheckResult result = ShouldCleanDownload(download.Ratio, download.SeedingTime ?? TimeSpan.Zero, category); + + if (!result.ShouldClean) + { + continue; + } + + await ((QBitService)Proxy).DeleteDownload(download.Hash); + + _logger.LogInformation( + "download cleaned | {reason} reached | {name}", + result.Reason is CleanReason.MaxRatioReached + ? "MAX_RATIO & MIN_SEED_TIME" + : "MAX_SEED_TIME", + download.Name + ); + + await _notifier.NotifyDownloadCleaned(download.Ratio, download.SeedingTime ?? TimeSpan.Zero, category.Name, result.Reason); + } + } + + /// + [DryRunSafeguard] + public override async Task DeleteDownload(string hash) { await _client.DeleteAsync(hash, deleteDownloadedData: true); } + + [DryRunSafeguard] + protected virtual async Task SkipFile(string hash, int fileIndex) + { + await _client.SetFilePriorityAsync(hash, fileIndex, TorrentContentPriority.Skip); + } public override void Dispose() { diff --git a/code/Infrastructure/Verticals/DownloadClient/SeedingCheckResult.cs b/code/Infrastructure/Verticals/DownloadClient/SeedingCheckResult.cs new file mode 100644 index 00000000..5fa65492 --- /dev/null +++ b/code/Infrastructure/Verticals/DownloadClient/SeedingCheckResult.cs @@ -0,0 +1,9 @@ +using Domain.Enums; + +namespace Infrastructure.Verticals.DownloadClient; + +public sealed record SeedingCheckResult +{ + public bool ShouldClean { get; set; } + public CleanReason Reason { get; set; } +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/Transmission/ITransmissionService.cs b/code/Infrastructure/Verticals/DownloadClient/Transmission/ITransmissionService.cs new file mode 100644 index 00000000..230e19f9 --- /dev/null +++ b/code/Infrastructure/Verticals/DownloadClient/Transmission/ITransmissionService.cs @@ -0,0 +1,5 @@ +namespace Infrastructure.Verticals.DownloadClient.Transmission; + +public interface ITransmissionService : IDownloadService +{ +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs b/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs index 92c88a7a..660060d6 100644 --- a/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs @@ -1,12 +1,16 @@ using System.Collections.Concurrent; using System.Text.RegularExpressions; +using Common.Attributes; using Common.Configuration.ContentBlocker; +using Common.Configuration.DownloadCleaner; using Common.Configuration.DownloadClient; using Common.Configuration.QueueCleaner; using Common.Helpers; using Domain.Enums; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.Context; using Infrastructure.Verticals.ItemStriker; +using Infrastructure.Verticals.Notifications; using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; @@ -16,22 +20,29 @@ using Transmission.API.RPC.Entity; namespace Infrastructure.Verticals.DownloadClient.Transmission; -public sealed class TransmissionService : DownloadServiceBase +public class TransmissionService : DownloadService, ITransmissionService { private readonly TransmissionConfig _config; private readonly Client _client; private TorrentInfo[]? _torrentsCache; + /// + public TransmissionService() + { + } + public TransmissionService( IHttpClientFactory httpClientFactory, ILogger logger, IOptions config, IOptions queueCleanerConfig, IOptions contentBlockerConfig, + IOptions downloadCleanerConfig, IMemoryCache cache, - FilenameEvaluator filenameEvaluator, - Striker striker - ) : base(logger, queueCleanerConfig, contentBlockerConfig, cache, filenameEvaluator, striker) + IFilenameEvaluator filenameEvaluator, + IStriker striker, + NotificationPublisher notifier + ) : base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache, filenameEvaluator, striker, notifier) { _config = config.Value; _config.Validate(); @@ -164,16 +175,96 @@ public sealed class TransmissionService : DownloadServiceBase _logger.LogDebug("changing priorities | torrent {hash}", hash); - await _client.TorrentSetAsync(new TorrentSettings - { - Ids = [ torrent.Id ], - FilesUnwanted = unwantedFiles.ToArray(), - }); + await ((TransmissionService)Proxy).SetUnwantedFiles(torrent.Id, unwantedFiles.ToArray()); return result; } - public override async Task Delete(string hash) + /// + public override async Task?> GetAllDownloadsToBeCleaned(List categories) + { + string[] fields = [ + TorrentFields.FILES, + TorrentFields.FILE_STATS, + TorrentFields.HASH_STRING, + TorrentFields.ID, + TorrentFields.ETA, + TorrentFields.NAME, + TorrentFields.STATUS, + TorrentFields.IS_PRIVATE, + TorrentFields.DOWNLOADED_EVER, + TorrentFields.DOWNLOAD_DIR, + TorrentFields.SECONDS_SEEDING, + TorrentFields.UPLOAD_RATIO + ]; + + return (await _client.TorrentGetAsync(fields)) + ?.Torrents + ?.Where(x => !string.IsNullOrEmpty(x.HashString)) + .Where(x => x.Status is 5 or 6) + .Where(x => categories + .Any(cat => x.DownloadDir?.EndsWith(cat.Name, StringComparison.InvariantCultureIgnoreCase) is true) + ) + .Cast() + .ToList(); + } + + /// + public override async Task CleanDownloads(List downloads, List categoriesToClean, HashSet excludedHashes) + { + foreach (TorrentInfo download in downloads) + { + if (string.IsNullOrEmpty(download.HashString)) + { + continue; + } + + Category? category = categoriesToClean + .FirstOrDefault(x => download.DownloadDir?.EndsWith(x.Name, StringComparison.InvariantCultureIgnoreCase) is true); + + if (category is null) + { + continue; + } + + if (excludedHashes.Any(x => x.Equals(download.HashString, StringComparison.InvariantCultureIgnoreCase))) + { + _logger.LogDebug("skip | download is used by an arr | {name}", download.Name); + continue; + } + + if (!_downloadCleanerConfig.DeletePrivate && download.IsPrivate is true) + { + _logger.LogDebug("skip | download is private | {name}", download.Name); + continue; + } + + ContextProvider.Set("downloadName", download.Name); + ContextProvider.Set("hash", download.HashString); + + TimeSpan seedingTime = TimeSpan.FromSeconds(download.SecondsSeeding ?? 0); + SeedingCheckResult result = ShouldCleanDownload(download.uploadRatio ?? 0, seedingTime, category); + + if (!result.ShouldClean) + { + continue; + } + + await ((TransmissionService)Proxy).RemoveDownloadAsync(download.Id); + + _logger.LogInformation( + "download cleaned | {reason} reached | {name}", + result.Reason is CleanReason.MaxRatioReached + ? "MAX_RATIO & MIN_SEED_TIME" + : "MAX_SEED_TIME", + download.Name + ); + + await _notifier.NotifyDownloadCleaned(download.uploadRatio ?? 0, seedingTime, category.Name, result.Reason); + } + } + + public override async Task DeleteDownload(string hash) { TorrentInfo? torrent = await GetTorrentAsync(hash); @@ -189,6 +280,22 @@ public sealed class TransmissionService : DownloadServiceBase { } + [DryRunSafeguard] + protected virtual async Task RemoveDownloadAsync(long downloadId) + { + await _client.TorrentRemoveAsync([downloadId], true); + } + + [DryRunSafeguard] + protected virtual async Task SetUnwantedFiles(long downloadId, long[] unwantedFiles) + { + await _client.TorrentSetAsync(new TorrentSettings + { + Ids = [downloadId], + FilesUnwanted = unwantedFiles, + }); + } + private async Task IsItemStuckAndShouldRemove(TorrentInfo torrent) { if (_queueCleanerConfig.StalledMaxStrikes is 0) diff --git a/code/Infrastructure/Verticals/ItemStriker/IStriker.cs b/code/Infrastructure/Verticals/ItemStriker/IStriker.cs new file mode 100644 index 00000000..5973bbac --- /dev/null +++ b/code/Infrastructure/Verticals/ItemStriker/IStriker.cs @@ -0,0 +1,8 @@ +using Domain.Enums; + +namespace Infrastructure.Verticals.ItemStriker; + +public interface IStriker +{ + Task StrikeAndCheckLimit(string hash, string itemName, ushort maxStrikes, StrikeType strikeType); +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/ItemStriker/Striker.cs b/code/Infrastructure/Verticals/ItemStriker/Striker.cs index c1ce8ec7..84f8b29d 100644 --- a/code/Infrastructure/Verticals/ItemStriker/Striker.cs +++ b/code/Infrastructure/Verticals/ItemStriker/Striker.cs @@ -8,7 +8,7 @@ using Microsoft.Extensions.Logging; namespace Infrastructure.Verticals.ItemStriker; -public class Striker +public sealed class Striker : IStriker { private readonly ILogger _logger; private readonly IMemoryCache _cache; diff --git a/code/Infrastructure/Verticals/Jobs/GenericHandler.cs b/code/Infrastructure/Verticals/Jobs/GenericHandler.cs index dc40f0ee..5a69cb0b 100644 --- a/code/Infrastructure/Verticals/Jobs/GenericHandler.cs +++ b/code/Infrastructure/Verticals/Jobs/GenericHandler.cs @@ -4,6 +4,7 @@ using Domain.Enums; using Domain.Models.Arr; using Domain.Models.Arr.Queue; using Infrastructure.Verticals.Arr; +using Infrastructure.Verticals.Arr.Interfaces; using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.Notifications; using Microsoft.Extensions.Logging; @@ -11,16 +12,16 @@ using Microsoft.Extensions.Options; namespace Infrastructure.Verticals.Jobs; -public abstract class GenericHandler : IDisposable +public abstract class GenericHandler : IHandler, IDisposable { protected readonly ILogger _logger; protected readonly DownloadClientConfig _downloadClientConfig; protected readonly SonarrConfig _sonarrConfig; protected readonly RadarrConfig _radarrConfig; protected readonly LidarrConfig _lidarrConfig; - protected readonly SonarrClient _sonarrClient; - protected readonly RadarrClient _radarrClient; - protected readonly LidarrClient _lidarrClient; + protected readonly ISonarrClient _sonarrClient; + protected readonly IRadarrClient _radarrClient; + protected readonly ILidarrClient _lidarrClient; protected readonly ArrQueueIterator _arrArrQueueIterator; protected readonly IDownloadService _downloadService; protected readonly NotificationPublisher _notifier; @@ -31,9 +32,9 @@ public abstract class GenericHandler : IDisposable IOptions sonarrConfig, IOptions radarrConfig, IOptions lidarrConfig, - SonarrClient sonarrClient, - RadarrClient radarrClient, - LidarrClient lidarrClient, + ISonarrClient sonarrClient, + IRadarrClient radarrClient, + ILidarrClient lidarrClient, ArrQueueIterator arrArrQueueIterator, DownloadServiceFactory downloadServiceFactory, NotificationPublisher notifier @@ -68,7 +69,7 @@ public abstract class GenericHandler : IDisposable protected abstract Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType); - private async Task ProcessArrConfigAsync(ArrConfig config, InstanceType instanceType) + protected async Task ProcessArrConfigAsync(ArrConfig config, InstanceType instanceType, bool throwOnFailure = false) { if (!config.Enabled) { @@ -84,11 +85,16 @@ public abstract class GenericHandler : IDisposable catch (Exception exception) { _logger.LogError(exception, "failed to clean {type} instance | {url}", instanceType, arrInstance.Url); + + if (throwOnFailure) + { + throw; + } } } } - protected ArrClient GetClient(InstanceType type) => + protected IArrClient GetClient(InstanceType type) => type switch { InstanceType.Sonarr => _sonarrClient, diff --git a/code/Infrastructure/Verticals/Jobs/IHandler.cs b/code/Infrastructure/Verticals/Jobs/IHandler.cs new file mode 100644 index 00000000..560241a6 --- /dev/null +++ b/code/Infrastructure/Verticals/Jobs/IHandler.cs @@ -0,0 +1,6 @@ +namespace Infrastructure.Verticals.Jobs; + +public interface IHandler +{ + Task ExecuteAsync(); +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/Consumers/NotificationConsumer.cs b/code/Infrastructure/Verticals/Notifications/Consumers/NotificationConsumer.cs index 166155ee..297ce561 100644 --- a/code/Infrastructure/Verticals/Notifications/Consumers/NotificationConsumer.cs +++ b/code/Infrastructure/Verticals/Notifications/Consumers/NotificationConsumer.cs @@ -27,9 +27,12 @@ public sealed class NotificationConsumer : IConsumer where T : Notificatio case StalledStrikeNotification stalledMessage: await _notificationService.Notify(stalledMessage); break; - case QueueItemDeleteNotification queueItemDeleteMessage: + case QueueItemDeletedNotification queueItemDeleteMessage: await _notificationService.Notify(queueItemDeleteMessage); break; + case DownloadCleanedNotification downloadCleanedNotification: + await _notificationService.Notify(downloadCleanedNotification); + break; default: throw new NotImplementedException(); } diff --git a/code/Infrastructure/Verticals/Notifications/INotificationFactory.cs b/code/Infrastructure/Verticals/Notifications/INotificationFactory.cs index bfb26355..5cf3d2eb 100644 --- a/code/Infrastructure/Verticals/Notifications/INotificationFactory.cs +++ b/code/Infrastructure/Verticals/Notifications/INotificationFactory.cs @@ -6,5 +6,7 @@ public interface INotificationFactory List OnStalledStrikeEnabled(); - List OnQueueItemDeleteEnabled(); + List OnQueueItemDeletedEnabled(); + + List OnDownloadCleanedEnabled(); } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/INotificationProvider.cs b/code/Infrastructure/Verticals/Notifications/INotificationProvider.cs index 1dc00b64..ea692538 100644 --- a/code/Infrastructure/Verticals/Notifications/INotificationProvider.cs +++ b/code/Infrastructure/Verticals/Notifications/INotificationProvider.cs @@ -13,5 +13,7 @@ public interface INotificationProvider Task OnStalledStrike(StalledStrikeNotification notification); - Task OnQueueItemDelete(QueueItemDeleteNotification notification); + Task OnQueueItemDeleted(QueueItemDeletedNotification notification); + + Task OnDownloadCleaned(DownloadCleanedNotification notification); } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/Models/ArrNotification.cs b/code/Infrastructure/Verticals/Notifications/Models/ArrNotification.cs new file mode 100644 index 00000000..43886ed9 --- /dev/null +++ b/code/Infrastructure/Verticals/Notifications/Models/ArrNotification.cs @@ -0,0 +1,14 @@ +using Domain.Enums; + +namespace Infrastructure.Verticals.Notifications.Models; + +public record ArrNotification : Notification +{ + public required InstanceType InstanceType { get; init; } + + public required Uri InstanceUrl { get; init; } + + public required string Hash { get; init; } + + public Uri? Image { get; init; } +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/Models/DownloadCleanedNotification.cs b/code/Infrastructure/Verticals/Notifications/Models/DownloadCleanedNotification.cs new file mode 100644 index 00000000..1203ce6c --- /dev/null +++ b/code/Infrastructure/Verticals/Notifications/Models/DownloadCleanedNotification.cs @@ -0,0 +1,5 @@ +namespace Infrastructure.Verticals.Notifications.Models; + +public sealed record DownloadCleanedNotification : Notification +{ +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/Models/FailedImportStrikeNotification.cs b/code/Infrastructure/Verticals/Notifications/Models/FailedImportStrikeNotification.cs index b8ac5fa5..3699bf12 100644 --- a/code/Infrastructure/Verticals/Notifications/Models/FailedImportStrikeNotification.cs +++ b/code/Infrastructure/Verticals/Notifications/Models/FailedImportStrikeNotification.cs @@ -1,5 +1,5 @@ namespace Infrastructure.Verticals.Notifications.Models; -public sealed record FailedImportStrikeNotification : Notification +public sealed record FailedImportStrikeNotification : ArrNotification { } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/Models/Notification.cs b/code/Infrastructure/Verticals/Notifications/Models/Notification.cs index 910a6107..2a93022d 100644 --- a/code/Infrastructure/Verticals/Notifications/Models/Notification.cs +++ b/code/Infrastructure/Verticals/Notifications/Models/Notification.cs @@ -1,20 +1,12 @@ -using Domain.Enums; +namespace Infrastructure.Verticals.Notifications.Models; -namespace Infrastructure.Verticals.Notifications.Models; - -public record Notification +public abstract record Notification { - public required InstanceType InstanceType { get; init; } - - public required Uri InstanceUrl { get; init; } - - public required string Hash { get; init; } - public required string Title { get; init; } public required string Description { get; init; } - public Uri? Image { get; init; } - public List? Fields { get; init; } + + public NotificationLevel Level { get; init; } } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/Models/NotificationLevel.cs b/code/Infrastructure/Verticals/Notifications/Models/NotificationLevel.cs new file mode 100644 index 00000000..a2106128 --- /dev/null +++ b/code/Infrastructure/Verticals/Notifications/Models/NotificationLevel.cs @@ -0,0 +1,9 @@ +namespace Infrastructure.Verticals.Notifications.Models; + +public enum NotificationLevel +{ + Test, + Information, + Warning, + Important +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/Models/QueueItemDeleteNotification.cs b/code/Infrastructure/Verticals/Notifications/Models/QueueItemDeleteNotification.cs deleted file mode 100644 index 0a5b2cab..00000000 --- a/code/Infrastructure/Verticals/Notifications/Models/QueueItemDeleteNotification.cs +++ /dev/null @@ -1,5 +0,0 @@ -namespace Infrastructure.Verticals.Notifications.Models; - -public sealed record QueueItemDeleteNotification : Notification -{ -} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/Models/QueueItemDeletedNotification.cs b/code/Infrastructure/Verticals/Notifications/Models/QueueItemDeletedNotification.cs new file mode 100644 index 00000000..5af2de3e --- /dev/null +++ b/code/Infrastructure/Verticals/Notifications/Models/QueueItemDeletedNotification.cs @@ -0,0 +1,5 @@ +namespace Infrastructure.Verticals.Notifications.Models; + +public sealed record QueueItemDeletedNotification : ArrNotification +{ +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/Models/StalledStrikeNotification.cs b/code/Infrastructure/Verticals/Notifications/Models/StalledStrikeNotification.cs index 74f17ba8..f194bc57 100644 --- a/code/Infrastructure/Verticals/Notifications/Models/StalledStrikeNotification.cs +++ b/code/Infrastructure/Verticals/Notifications/Models/StalledStrikeNotification.cs @@ -1,5 +1,5 @@ namespace Infrastructure.Verticals.Notifications.Models; -public sealed record StalledStrikeNotification : Notification +public sealed record StalledStrikeNotification : ArrNotification { } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/Notifiarr/NotifiarrProvider.cs b/code/Infrastructure/Verticals/Notifications/Notifiarr/NotifiarrProvider.cs index dd189fb9..6ad8c1bc 100644 --- a/code/Infrastructure/Verticals/Notifications/Notifiarr/NotifiarrProvider.cs +++ b/code/Infrastructure/Verticals/Notifications/Notifiarr/NotifiarrProvider.cs @@ -1,4 +1,3 @@ -using Domain.Enums; using Infrastructure.Verticals.Notifications.Models; using Mapster; using Microsoft.Extensions.Options; @@ -12,6 +11,7 @@ public class NotifiarrProvider : NotificationProvider private const string WarningColor = "f0ad4e"; private const string ImportantColor = "bb2124"; + private const string Logo = "https://github.com/flmorg/cleanuperr/blob/main/Logo/48.png?raw=true"; public NotifiarrProvider(IOptions config, INotifiarrProxy proxy) : base(config) @@ -32,12 +32,17 @@ public class NotifiarrProvider : NotificationProvider await _proxy.SendNotification(BuildPayload(notification, WarningColor), _config); } - public override async Task OnQueueItemDelete(QueueItemDeleteNotification notification) + public override async Task OnQueueItemDeleted(QueueItemDeletedNotification notification) { await _proxy.SendNotification(BuildPayload(notification, ImportantColor), _config); } - private NotifiarrPayload BuildPayload(Notification notification, string color) + public override async Task OnDownloadCleaned(DownloadCleanedNotification notification) + { + await _proxy.SendNotification(BuildPayload(notification), _config); + } + + private NotifiarrPayload BuildPayload(ArrNotification notification, string color) { NotifiarrPayload payload = new() { @@ -47,7 +52,7 @@ public class NotifiarrProvider : NotificationProvider Text = new() { Title = notification.Title, - Icon = "https://github.com/flmorg/cleanuperr/blob/main/Logo/48.png?raw=true", + Icon = Logo, Description = notification.Description, Fields = new() { @@ -62,7 +67,7 @@ public class NotifiarrProvider : NotificationProvider }, Images = new() { - Thumbnail = new Uri("https://github.com/flmorg/cleanuperr/blob/main/Logo/48.png?raw=true"), + Thumbnail = new Uri(Logo), Image = notification.Image } } @@ -72,4 +77,32 @@ public class NotifiarrProvider : NotificationProvider return payload; } + + private NotifiarrPayload BuildPayload(DownloadCleanedNotification notification) + { + NotifiarrPayload payload = new() + { + Discord = new() + { + Color = ImportantColor, + Text = new() + { + Title = notification.Title, + Icon = Logo, + Description = notification.Description, + Fields = notification.Fields?.Adapt>() ?? [] + }, + Ids = new Ids + { + Channel = _config.ChannelId + }, + Images = new() + { + Thumbnail = new Uri(Logo) + } + } + }; + + return payload; + } } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/NotificationFactory.cs b/code/Infrastructure/Verticals/Notifications/NotificationFactory.cs index bc942527..10c5ba05 100644 --- a/code/Infrastructure/Verticals/Notifications/NotificationFactory.cs +++ b/code/Infrastructure/Verticals/Notifications/NotificationFactory.cs @@ -25,8 +25,13 @@ public class NotificationFactory : INotificationFactory .Where(n => n.Config.OnStalledStrike) .ToList(); - public List OnQueueItemDeleteEnabled() => + public List OnQueueItemDeletedEnabled() => ActiveProviders() - .Where(n => n.Config.OnQueueItemDelete) + .Where(n => n.Config.OnQueueItemDeleted) + .ToList(); + + public List OnDownloadCleanedEnabled() => + ActiveProviders() + .Where(n => n.Config.OnDownloadCleaned) .ToList(); } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/NotificationProvider.cs b/code/Infrastructure/Verticals/Notifications/NotificationProvider.cs index 749ba4d2..e1b6cc7b 100644 --- a/code/Infrastructure/Verticals/Notifications/NotificationProvider.cs +++ b/code/Infrastructure/Verticals/Notifications/NotificationProvider.cs @@ -19,5 +19,7 @@ public abstract class NotificationProvider : INotificationProvider public abstract Task OnStalledStrike(StalledStrikeNotification notification); - public abstract Task OnQueueItemDelete(QueueItemDeleteNotification notification); + public abstract Task OnQueueItemDeleted(QueueItemDeletedNotification notification); + + public abstract Task OnDownloadCleaned(DownloadCleanedNotification notification); } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Notifications/NotificationPublisher.cs b/code/Infrastructure/Verticals/Notifications/NotificationPublisher.cs index b8b486b6..7bbe2d68 100644 --- a/code/Infrastructure/Verticals/Notifications/NotificationPublisher.cs +++ b/code/Infrastructure/Verticals/Notifications/NotificationPublisher.cs @@ -1,6 +1,9 @@ -using Common.Configuration.Arr; +using System.Globalization; +using Common.Attributes; +using Common.Configuration.Arr; using Domain.Enums; using Domain.Models.Arr.Queue; +using Infrastructure.Interceptors; using Infrastructure.Verticals.Context; using Infrastructure.Verticals.Notifications.Models; using Mapster; @@ -9,27 +12,35 @@ using Microsoft.Extensions.Logging; namespace Infrastructure.Verticals.Notifications; -public sealed class NotificationPublisher +public class NotificationPublisher : InterceptedService, IDryRunService { private readonly ILogger _logger; private readonly IBus _messageBus; - + + /// + /// Constructor to be used by interceptors. + /// + public NotificationPublisher() + { + } + public NotificationPublisher(ILogger logger, IBus messageBus) { _logger = logger; _messageBus = messageBus; } - public async Task NotifyStrike(StrikeType strikeType, int strikeCount) + [DryRunSafeguard] + public virtual async Task NotifyStrike(StrikeType strikeType, int strikeCount) { try { - QueueRecord record = GetRecordFromContext(); - InstanceType instanceType = GetInstanceTypeFromContext(); - Uri instanceUrl = GetInstanceUrlFromContext(); - Uri? imageUrl = GetImageFromContext(record, instanceType); + QueueRecord record = ContextProvider.Get(nameof(QueueRecord)); + InstanceType instanceType = (InstanceType)ContextProvider.Get(nameof(InstanceType)); + Uri instanceUrl = ContextProvider.Get(nameof(ArrInstance) + nameof(ArrInstance.Url)); + Uri imageUrl = GetImageFromContext(record, instanceType); - Notification notification = new() + ArrNotification notification = new() { InstanceType = instanceType, InstanceUrl = instanceUrl, @@ -56,14 +67,15 @@ public sealed class NotificationPublisher } } - public async Task NotifyQueueItemDelete(bool removeFromClient, DeleteReason reason) + [DryRunSafeguard] + public virtual async Task NotifyQueueItemDeleted(bool removeFromClient, DeleteReason reason) { - QueueRecord record = GetRecordFromContext(); - InstanceType instanceType = GetInstanceTypeFromContext(); - Uri instanceUrl = GetInstanceUrlFromContext(); - Uri? imageUrl = GetImageFromContext(record, instanceType); + QueueRecord record = ContextProvider.Get(nameof(QueueRecord)); + InstanceType instanceType = (InstanceType)ContextProvider.Get(nameof(InstanceType)); + Uri instanceUrl = ContextProvider.Get(nameof(ArrInstance) + nameof(ArrInstance.Url)); + Uri imageUrl = GetImageFromContext(record, instanceType); - Notification notification = new() + QueueItemDeletedNotification notification = new() { InstanceType = instanceType, InstanceUrl = instanceUrl, @@ -74,20 +86,29 @@ public sealed class NotificationPublisher Fields = [new() { Title = "Removed from download client?", Text = removeFromClient ? "Yes" : "No" }] }; - await _messageBus.Publish(notification.Adapt()); + await _messageBus.Publish(notification); } - private static QueueRecord GetRecordFromContext() => - ContextProvider.Get(nameof(QueueRecord)) ?? throw new Exception("failed to get record from context"); + [DryRunSafeguard] + public virtual async Task NotifyDownloadCleaned(double ratio, TimeSpan seedingTime, string categoryName, CleanReason reason) + { + DownloadCleanedNotification notification = new() + { + Title = $"Cleaned item from download client with reason: {reason}", + Description = ContextProvider.Get("downloadName"), + Fields = + [ + new() { Title = "Hash", Text = ContextProvider.Get("hash").ToLowerInvariant() }, + new() { Title = "Category", Text = categoryName.ToLowerInvariant() }, + new() { Title = "Ratio", Text = $"{ratio.ToString(CultureInfo.InvariantCulture)}%" }, + new() { Title = "Seeding hours", Text = $"{Math.Round(seedingTime.TotalHours, 0).ToString(CultureInfo.InvariantCulture)}h" } + ], + Level = NotificationLevel.Important + }; + + await _messageBus.Publish(notification); + } - private static InstanceType GetInstanceTypeFromContext() => - (InstanceType)(ContextProvider.Get(nameof(InstanceType)) ?? - throw new Exception("failed to get instance type from context")); - - private static Uri GetInstanceUrlFromContext() => - ContextProvider.Get(nameof(ArrInstance) + nameof(ArrInstance.Url)) ?? - throw new Exception("failed to get instance url from context"); - private static Uri GetImageFromContext(QueueRecord record, InstanceType instanceType) => instanceType switch { diff --git a/code/Infrastructure/Verticals/Notifications/NotificationService.cs b/code/Infrastructure/Verticals/Notifications/NotificationService.cs index a6c11596..fbe8fd60 100644 --- a/code/Infrastructure/Verticals/Notifications/NotificationService.cs +++ b/code/Infrastructure/Verticals/Notifications/NotificationService.cs @@ -44,13 +44,28 @@ public class NotificationService } } - public async Task Notify(QueueItemDeleteNotification notification) + public async Task Notify(QueueItemDeletedNotification notification) { - foreach (INotificationProvider provider in _notificationFactory.OnQueueItemDeleteEnabled()) + foreach (INotificationProvider provider in _notificationFactory.OnQueueItemDeletedEnabled()) { try { - await provider.OnQueueItemDelete(notification); + await provider.OnQueueItemDeleted(notification); + } + catch (Exception exception) + { + _logger.LogWarning(exception, "failed to send notification | provider {provider}", provider.Name); + } + } + } + + public async Task Notify(DownloadCleanedNotification notification) + { + foreach (INotificationProvider provider in _notificationFactory.OnDownloadCleanedEnabled()) + { + try + { + await provider.OnDownloadCleaned(notification); } catch (Exception exception) { diff --git a/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs b/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs index 35d01bab..cdd49ad7 100644 --- a/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs +++ b/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs @@ -5,6 +5,7 @@ using Domain.Enums; using Domain.Models.Arr; using Domain.Models.Arr.Queue; using Infrastructure.Verticals.Arr; +using Infrastructure.Verticals.Arr.Interfaces; using Infrastructure.Verticals.Context; using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.Jobs; @@ -48,7 +49,7 @@ public sealed class QueueCleaner : GenericHandler using var _ = LogContext.PushProperty("InstanceName", instanceType.ToString()); HashSet itemsToBeRefreshed = []; - ArrClient arrClient = GetClient(instanceType); + IArrClient arrClient = GetClient(instanceType); // push to context ContextProvider.Set(nameof(ArrInstance) + nameof(ArrInstance.Url), instance.Url); @@ -113,7 +114,7 @@ public sealed class QueueCleaner : GenericHandler } await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient); - await _notifier.NotifyQueueItemDelete(removeFromClient, deleteReason); + await _notifier.NotifyQueueItemDeleted(removeFromClient, deleteReason); } }); diff --git a/code/cleanuperr.sln b/code/cleanuperr.sln index 32aaf4e5..e89bd128 100644 --- a/code/cleanuperr.sln +++ b/code/cleanuperr.sln @@ -8,6 +8,8 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Infrastructure", "Infrastru EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Domain", "Domain\Domain.csproj", "{8871592A-B260-4B15-8EF8-6AB24480DE5D}" EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Infrastructure.Tests", "Infrastructure.Tests\Infrastructure.Tests.csproj", "{F2DBB5FD-8D93-45D0-B211-5E9A8C14B684}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -30,5 +32,9 @@ Global {8871592A-B260-4B15-8EF8-6AB24480DE5D}.Debug|Any CPU.Build.0 = Debug|Any CPU {8871592A-B260-4B15-8EF8-6AB24480DE5D}.Release|Any CPU.ActiveCfg = Release|Any CPU {8871592A-B260-4B15-8EF8-6AB24480DE5D}.Release|Any CPU.Build.0 = Release|Any CPU + {F2DBB5FD-8D93-45D0-B211-5E9A8C14B684}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {F2DBB5FD-8D93-45D0-B211-5E9A8C14B684}.Debug|Any CPU.Build.0 = Debug|Any CPU + {F2DBB5FD-8D93-45D0-B211-5E9A8C14B684}.Release|Any CPU.ActiveCfg = Release|Any CPU + {F2DBB5FD-8D93-45D0-B211-5E9A8C14B684}.Release|Any CPU.Build.0 = Release|Any CPU EndGlobalSection EndGlobal diff --git a/code/test/data/transmission/config/bandwidth-groups.json b/code/test/data/transmission/config/bandwidth-groups.json deleted file mode 100644 index 2c63c085..00000000 --- a/code/test/data/transmission/config/bandwidth-groups.json +++ /dev/null @@ -1,2 +0,0 @@ -{ -} diff --git a/code/test/docker-compose.yml b/code/test/docker-compose.yml index d7cc0a09..572a04d0 100644 --- a/code/test/docker-compose.yml +++ b/code/test/docker-compose.yml @@ -148,6 +148,8 @@ services: - ./data/lidarr/config:/config - ./data/lidarr/music:/music - ./data/qbittorrent/downloads:/downloads + # - ./data/deluge/downloads:/downloads + # - ./data/transmission/downloads:/downloads ports: - 8686:8686 restart: unless-stopped @@ -163,6 +165,8 @@ services: - ./data/readarr/config:/config - ./data/readarr/books:/books - ./data/qbittorrent/downloads:/downloads + # - ./data/deluge/downloads:/downloads + # - ./data/transmission/downloads:/downloads ports: - 8787:8787 restart: unless-stopped @@ -171,6 +175,8 @@ services: image: ghcr.io/flmorg/cleanuperr:latest container_name: cleanuperr environment: + - DRY_RUN=false + - LOGGING__LOGLEVEL=Debug - LOGGING__FILE__ENABLED=true - LOGGING__FILE__PATH=/var/logs @@ -181,6 +187,7 @@ services: - TRIGGERS__QUEUECLEANER=0/30 * * * * ? - TRIGGERS__CONTENTBLOCKER=0/30 * * * * ? + - TRIGGERS__DOWNLOADCLEANER=0/30 * * * * ? - QUEUECLEANER__ENABLED=true - QUEUECLEANER__RUNSEQUENTIALLY=true @@ -196,17 +203,28 @@ services: - CONTENTBLOCKER__IGNORE_PRIVATE=true - CONTENTBLOCKER__DELETE_PRIVATE=false + - DOWNLOADCLEANER__ENABLED=true + - DOWNLOADCLEANER__DELETE_PRIVATE=false + - DOWNLOADCLEANER__CATEGORIES__0__NAME=tv-sonarr + - DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO=-1 + - DOWNLOADCLEANER__CATEGORIES__0__MIN_SEED_TIME=0 + - DOWNLOADCLEANER__CATEGORIES__0__MAX_SEED_TIME=0.01 + - DOWNLOADCLEANER__CATEGORIES__1__NAME=radarr + - DOWNLOADCLEANER__CATEGORIES__1__MAX_RATIO=-1 + - DOWNLOADCLEANER__CATEGORIES__1__MIN_SEED_TIME=0 + - DOWNLOADCLEANER__CATEGORIES__1__MAX_SEED_TIME=0.01 + - DOWNLOAD_CLIENT=qbittorrent - QBITTORRENT__URL=http://qbittorrent:8080 - QBITTORRENT__USERNAME=test - QBITTORRENT__PASSWORD=testing # OR # - DOWNLOAD_CLIENT=deluge - # - DELUGE__URL=http://localhost:8112 + # - DELUGE__URL=http://deluge:8112 # - DELUGE__PASSWORD=testing # OR # - DOWNLOAD_CLIENT=transmission - # - TRANSMISSION__URL=http://localhost:9091 + # - TRANSMISSION__URL=http://transmission:9091 # - TRANSMISSION__USERNAME=test # - TRANSMISSION__PASSWORD=testing @@ -231,7 +249,7 @@ services: # - NOTIFIARR__ON_IMPORT_FAILED_STRIKE=true # - NOTIFIARR__ON_STALLED_STRIKE=true - # - NOTIFIARR__ON_QUEUE_ITEM_DELETE=true + # - NOTIFIARR__ON_QUEUE_ITEM_DELETED=true # - NOTIFIARR__API_KEY=notifiarr_secret # - NOTIFIARR__CHANNEL_ID=discord_channel_id volumes: diff --git a/variables.md b/variables.md index 653b7904..8d64c038 100644 --- a/variables.md +++ b/variables.md @@ -1,11 +1,524 @@ -## LOGGING__ENHANCED +## Table of contents +- [General settings](variables.md#general-settings) +- [Queue Cleaner settings](variables.md#queue-cleaner-settings) +- [Content Blocker settings](variables.md#content-blocker-settings) +- [Download Cleaner settings](variables.md#download-cleaner-settings) +- [Download Client settings](variables.md#download-client-settings) +- [Arr settings](variables.md#arr-settings) +- [Notification settings](variables.md#notification-settings) +- [Advanced settings](variables.md#advanced-settings) -Some logs may contain information that is hard to read. Enhancing these logs usually comes with the cost of additional calls to the APIs. +# -If enabled, logs like this +### General settings -```movie search triggered | http://localhost:7878/ | movie ids: 1, 2``` +**`DRY_RUN`** +- When enabled, simulates irreversible operations (like deletions and notifications) without making actual changes. +- Type: Boolean. +- Possible values: `true`, `false`. +- Default: `false`. +- Required: No. -will transform into +**`LOGGING__LOGLEVEL`** +- Controls the detail level of application logs. +- Type: String. +- Possible values: `Verbose`, `Debug`, `Information`, `Warning`, `Error`, `Fatal`. +- Default: `Information`. +- Required: No. -```movie search triggered | http://localhost:7878/ | [Speak No Evil][The Wild Robot]``` \ No newline at end of file +**`LOGGING__FILE__ENABLED`** +- Enables logging to a file. +- Type: Boolean. +- Possible values: `true`, `false`. +- Default: `false`. +- Required: No. + +**`LOGGING__FILE__PATH`** +- Directory where log files will be saved. +- Type: String. +- Default: Empty. +- Required: No. + +**`LOGGING__ENHANCED`** +- Provides more detailed descriptions in logs whenever possible. +- Type: Boolean. +- Possible values: `true`, `false`. +- Default: `true`. +- Required: No. + + +# + +### Queue Cleaner settings + +**`TRIGGERS__QUEUECLEANER`** +- Cron schedule for the queue cleaner job. +- Type: String - [Quartz cron format](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html). +- Default: `0 0/5 * * * ?` (every 5 minutes). +- Required: Yes if queue cleaner is enabled. + +> [!NOTE] +> - Maximum interval is 6 hours. +> - Ignored if `QUEUECLEANER__RUNSEQUENTIALLY=true` and `CONTENTBLOCKER__ENABLED=true`. + +**`QUEUECLEANER__ENABLED`** +- Enables or disables the queue cleaning functionality. +- When enabled, processes all items in the *arr queue. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `true` +- Required: No. + +**`QUEUECLEANER__RUNSEQUENTIALLY`** +- Controls whether queue cleaner runs after content blocker instead of in parallel. +- When `true`, streamlines the cleaning process by running immediately after content blocker. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `true` +- Required: No. + +**`QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES`** +- Number of strikes before removing a failed import. +- Set to `0` to never remove failed imports. +- A strike is given when an item is stalled, stuck in metadata downloading, or failed to be imported. +- Type: Integer +- Possible values: `0` or greater +- Default: `0` +- Required: No. + +**`QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE`** +- Controls whether to ignore failed imports from private trackers. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE`** +- Controls whether to delete failed imports from private trackers from the download client. +- Has no effect if `QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE` is `true`. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +> [!WARNING] +> Setting `QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account. + +**`QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS`** +- Patterns to look for in failed import messages that should be ignored. +- Multiple patterns can be specified using incrementing numbers starting from 0. +- Type: String array +- Default: Empty. +- Required: No. +- Example: +```yaml +QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__0: "title mismatch" +QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required" +``` + +**`QUEUECLEANER__STALLED_MAX_STRIKES`** +- Number of strikes before removing a stalled download. +- Set to `0` to never remove stalled downloads. +- A strike is given when download speed is 0. +- Type: Integer +- Possible values: `0` or greater +- Default: `0` +- Required: No. + +**`QUEUECLEANER__STALLED_RESET_STRIKES_ON_PROGRESS`** +- Controls whether to remove strikes if any download progress was made since last checked. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`QUEUECLEANER__STALLED_IGNORE_PRIVATE`** +- Controls whether to ignore stalled downloads from private trackers. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`QUEUECLEANER__STALLED_DELETE_PRIVATE`** +- Controls whether to delete stalled private downloads from the download client. +- Has no effect if `QUEUECLEANER__STALLED_IGNORE_PRIVATE` is `true`. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +> [!WARNING] +> Setting `QUEUECLEANER__STALLED_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account. + +# + +### Content Blocker settings + +**`TRIGGERS__CONTENTBLOCKER`** +- Cron schedule for the content blocker job. +- Type: String - [Quartz cron format](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html). +- Default: `0 0/5 * * * ?` (every 5 minutes). +- Required: No. + +> [!NOTE] +> - Maximum interval is 6 hours. + +**`CONTENTBLOCKER__ENABLED`** +- Enables or disables the content blocker functionality. +- When enabled, processes all items in the *arr queue and marks unwanted files. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`CONTENTBLOCKER__IGNORE_PRIVATE`** +- Controls whether to ignore downloads from private trackers. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`CONTENTBLOCKER__DELETE_PRIVATE`** +- Controls whether to delete private downloads that have all files blocked from the download client. +- Has no effect if `CONTENTBLOCKER__IGNORE_PRIVATE` is `true`. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +> [!WARNING] +> Setting `CONTENTBLOCKER__DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account. + +# + +### Download Cleaner settings + +**`TRIGGERS__DOWNLOADCLEANER`** +- Cron schedule for the download cleaner job. +- Type: String - [Quartz cron format](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html). +- Default: `0 0 * * * ?` (every hour). +- Required: No. + +> [!NOTE] +> - Maximum interval is 6 hours. + +**`DOWNLOADCLEANER__ENABLED`** +- Enables or disables the download cleaner functionality. +- When enabled, automatically cleans up downloads that have been seeding for a certain amount of time. +- Type: Boolean. +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`DOWNLOADCLEANER__DELETE_PRIVATE`** +- Controls whether to delete private downloads. +- Type: Boolean. +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +> [!WARNING] +> Setting `DOWNLOADCLEANER__DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account. + +**`DOWNLOADCLEANER__CATEGORIES__0__NAME`** +- Name of the category to clean. +- Type: String. +- Default: Empty. +- Required: No. + +> [!NOTE] +> The category name must match the category that was set in the *arr. +> For qBittorrent, the category name is the name of the download category. +> For Deluge, the category name is the name of the label. +> For Transmission, the category name is the name of the download location. + +**`DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO`** +- Maximum ratio to reach before removing a download. +- Type: Decimal. +- Possible values: `-1` or greater (`-1` means no limit or disabled). +- Default: `-1` +- Required: No. + +**`DOWNLOADCLEANER__CATEGORIES__0__MIN_SEED_TIME`** +- Minimum number of hours to seed before removing a download, if the ratio has been met. +- Used with `MAX_RATIO` to ensure a minimum seed time. +- Type: Decimal. +- Possible values: `0` or greater. +- Default: `0` +- Required: No. + +**`DOWNLOADCLEANER__CATEGORIES__0__MAX_SEED_TIME`** +- Maximum number of hours to seed before removing a download. +- Type: Decimal. +- Possible values: `-1` or greater (`-1` means no limit or disabled). +- Default: `-1` +- Required: No. + +> [!NOTE] +> A download is cleaned when any of (`MAX_RATIO` & `MIN_SEED_TIME`) or `MAX_SEED_TIME` is reached. + +> [!NOTE] +> Multiple categories can be specified using this format, where `` starts from 0: +> ```yaml +> DOWNLOADCLEANER__CATEGORIES____NAME +> DOWNLOADCLEANER__CATEGORIES____MAX_RATIO +> DOWNLOADCLEANER__CATEGORIES____MIN_SEED_TIME +> DOWNLOADCLEANER__CATEGORIES____MAX_SEED_TIME +> ``` + +# + +### Download Client settings + +**`DOWNLOAD_CLIENT`** +- Specifies which download client is used by *arrs. +- Type: String. +- Possible values: `none`, `qbittorrent`, `deluge`, `transmission`. +- Default: `none` +- Required: No. + +> [!NOTE] +> Only one download client can be enabled at a time. If you have more than one download client, you should deploy multiple instances of cleanuperr. + +**`QBITTORRENT__URL`** +- URL of the qBittorrent instance. +- Type: String. +- Default: `http://localhost:8080`. +- Required: No. + +**`QBITTORRENT__USERNAME`** +- Username for qBittorrent authentication. +- Type: String. +- Default: Empty. +- Required: No. + +**`QBITTORRENT__PASSWORD`** +- Password for qBittorrent authentication. +- Type: String. +- Default: Empty. +- Required: No. + +**`DELUGE__URL`** +- URL of the Deluge instance. +- Type: String. +- Default: `http://localhost:8112`. +- Required: No. + +**`DELUGE__PASSWORD`** +- Password for Deluge authentication. +- Type: String. +- Default: Empty. +- Required: No. + +**`TRANSMISSION__URL`** +- URL of the Transmission instance. +- Type: String. +- Default: `http://localhost:9091`. +- Required: No. + +**`TRANSMISSION__USERNAME`** +- Username for Transmission authentication. +- Type: String. +- Default: Empty. +- Required: No. + +**`TRANSMISSION__PASSWORD`** +- Password for Transmission authentication. +- Type: String. +- Default: Empty. +- Required: No. + +# + +### Arr settings + +**`SONARR__ENABLED`** +- Enables or disables Sonarr cleanup. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`SONARR__BLOCK__TYPE`** +- Determines how file blocking works for Sonarr. +- Type: String +- Possible values: `blacklist`, `whitelist` +- Default: `blacklist` +- Required: No. + +**`SONARR__BLOCK__PATH`** +- Path to the blocklist file (local file or URL). +- Must be JSON compatible. +- Type: String +- Default: Empty. +- Required: No. + +**`SONARR__SEARCHTYPE`** +- Determines what to search for after removing a queue item. +- Type: String +- Possible values: `Episode`, `Season`, `Series` +- Default: `Episode` +- Required: No. + +**`SONARR__INSTANCES__0__URL`** +- URL of the Sonarr instance. +- Type: String +- Default: `http://localhost:8989` +- Required: No. + +**`SONARR__INSTANCES__0__APIKEY`** +- API key for the Sonarr instance. +- Type: String +- Default: Empty. +- Required: No. + +**`RADARR__ENABLED`** +- Enables or disables Radarr cleanup. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`RADARR__BLOCK__TYPE`** +- Determines how file blocking works for Radarr. +- Type: String +- Possible values: `blacklist`, `whitelist` +- Default: `blacklist` +- Required: No. + +**`RADARR__BLOCK__PATH`** +- Path to the blocklist file (local file or URL). +- Must be JSON compatible. +- Type: String +- Default: Empty. +- Required: No. + +**`RADARR__INSTANCES__0__URL`** +- URL of the Radarr instance. +- Type: String +- Default: `http://localhost:7878` +- Required: No. + +**`RADARR__INSTANCES__0__APIKEY`** +- API key for the Radarr instance. +- Type: String +- Default: Empty. +- Required: No. + +**`LIDARR__ENABLED`** +- Enables or disables Lidarr cleanup. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`LIDARR__BLOCK__TYPE`** +- Determines how file blocking works for Lidarr. +- Type: String +- Possible values: `blacklist`, `whitelist` +- Default: `blacklist` +- Required: No. + +**`LIDARR__BLOCK__PATH`** +- Path to the blocklist file (local file or URL). +- Must be JSON compatible. +- Type: String +- Default: Empty. +- Required: No. + +**`LIDARR__INSTANCES__0__URL`** +- URL of the Lidarr instance. +- Type: String +- Default: `http://localhost:8686` +- Required: No. + +**`LIDARR__INSTANCES__0__APIKEY`** +- API key for the Lidarr instance. +- Type: String +- Default: Empty. +- Required: No. + +> [!NOTE] +> Multiple instances can be specified for each *arr using this format, where `` starts from 0: +> ```yaml +> __INSTANCES____URL +> __INSTANCES____APIKEY +> ``` + +> [!NOTE] +> The blocklists (blacklist/whitelist) support the following patterns: +> ``` +> *example // file name ends with "example" +> example* // file name starts with "example" +> *example* // file name has "example" in the name +> example // file name is exactly the word "example" +> regex: // regex that needs to be marked at the start of the line with "regex:" +> ``` + +> [!NOTE] +> [This blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist) and [this whitelist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist) can be used for Sonarr and Radarr, but they are not suitable for other *arrs. + +# + +### Notification settings + +**`NOTIFIARR__API_KEY`** +- Notifiarr API key for sending notifications. +- Requires Notifiarr's [`Passthrough`](https://notifiarr.wiki/en/Website/Integrations/Passthrough) integration to work. +- Type: String +- Default: Empty. +- Required: No. + +**`NOTIFIARR__CHANNEL_ID`** +- Discord channel ID where notifications will be sent. +- Type: String +- Default: Empty. +- Required: No. + +**`NOTIFIARR__ON_IMPORT_FAILED_STRIKE`** +- Controls whether to notify when an item receives a failed import strike. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`NOTIFIARR__ON_STALLED_STRIKE`** +- Controls whether to notify when an item receives a stalled download strike. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`NOTIFIARR__ON_QUEUE_ITEM_DELETED`** +- Controls whether to notify when a queue item is deleted. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +**`NOTIFIARR__ON_DOWNLOAD_CLEANED`** +- Controls whether to notify when a download is cleaned. +- Type: Boolean +- Possible values: `true`, `false` +- Default: `false` +- Required: No. + +# + +### Advanced settings + +**`HTTP_MAX_RETRIES`** +- The number of times to retry a failed HTTP call. +- Applies to calls to *arrs, download clients, and other services. +- Type: Integer +- Possible values: `0` or greater +- Default: `0` +- Required: No. + +**`HTTP_TIMEOUT`** +- The number of seconds to wait before failing an HTTP call. +- Applies to calls to *arrs, download clients, and other services. +- Type: Integer +- Possible values: Greater than `0` +- Default: `100` +- Required: No. \ No newline at end of file