diff --git a/.gitignore b/.gitignore index ab2eb321..de8db2a1 100644 --- a/.gitignore +++ b/.gitignore @@ -105,7 +105,6 @@ _NCrunch_* _TeamCity* # Sonarr -config.xml nzbdrone.log*txt UpdateLogs/ *workspace.xml diff --git a/README.md b/README.md index 82478e5f..db0c625b 100644 --- a/README.md +++ b/README.md @@ -21,8 +21,9 @@ Only the **latest versions** of the following apps are supported, or earlier ver - Transmission - Sonarr - Radarr +- Lidarr -This tool is actively developed and still a work in progress. Join the Discord server if you want to reach out to me quickly (or just stay updated on new releases) so we can squash those pesky bugs together: +This tool is actively developed and still a work in progress, so using the `latest` Docker tag may result in breaking changes. Join the Discord server if you want to reach out to me quickly (or just stay updated on new releases) so we can squash those pesky bugs together: > https://discord.gg/sWggpnmGNY @@ -35,8 +36,12 @@ This tool is actively developed and still a work in progress. Join the Discord s - Mark the files that were found in the queue as **unwanted/skipped** if: - They **are listed in the blacklist**, or - They **are not included in the whitelist**. + - If **all files** of a download **are unwanted**: + - It will be removed from the *arr's queue and blocked. + - It will be deleted from the download client. + - A new search will be triggered for the *arr item. 2. **Queue cleaner** will: - - Run every 5 minutes (or configured cron). + - Run every 5 minutes (or configured cron, or right after `content blocker`). - Process all items in the *arr queue. - Check each queue item if it is **stalled (download speed is 0)**, **stuck in matadata downloading** or **failed to be imported**. - If it is, the item receives a **strike** and will continue to accumulate strikes every time it meets any of these conditions. @@ -63,8 +68,8 @@ This tool is actively developed and still a work in progress. Join the Discord s ## Using cleanuperr's blocklist (works with all supported download clients) -1. Set both `QUEUECLEANER__ENABLED` and `CONTENTBLOCKER_ENABLED` to `true` in your environment variables. -2. Configure and enable either a **blacklist** or a **whitelist** as described in the [Environment variables](#Environment-variables) section. +1. Set both `QUEUECLEANER__ENABLED` and `CONTENTBLOCKER__ENABLED` to `true` in your environment variables. +2. Configure and enable either a **blacklist** or a **whitelist** as described in the [Arr variables](#Arr-variables) section. 3. Once configured, cleanuperr will perform the following tasks: - Execute the **content blocker** job, as explained in the [How it works](#how-it-works) section. - Execute the **queue cleaner** job, as explained in the [How it works](#how-it-works) section. @@ -108,16 +113,13 @@ services: - CONTENTBLOCKER__ENABLED=true - CONTENTBLOCKER__IGNORE_PRIVATE=true - - CONTENTBLOCKER__BLACKLIST__ENABLED=true - - CONTENTBLOCKER__BLACKLIST__PATH=https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist - # OR - # - CONTENTBLOCKER__WHITELIST__ENABLED=true - # - CONTENTBLOCKER__WHITELIST__PATH=https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist - - DOWNLOAD_CLIENT=qBittorrent - - QBITTORRENT__URL=http://localhost:8080 - - QBITTORRENT__USERNAME=user - - QBITTORRENT__PASSWORD=pass + - DOWNLOAD_CLIENT=none + # OR + # - DOWNLOAD_CLIENT=qBittorrent + # - QBITTORRENT__URL=http://localhost:8080 + # - QBITTORRENT__USERNAME=user + # - QBITTORRENT__PASSWORD=pass # OR # - DOWNLOAD_CLIENT=deluge # - DELUGE__URL=http://localhost:8112 @@ -127,26 +129,40 @@ services: # - TRANSMISSION__URL=http://localhost:9091 # - TRANSMISSION__USERNAME=test # - TRANSMISSION__PASSWORD=testing - # OR - # - DOWNLOAD_CLIENT=none - SONARR__ENABLED=true - SONARR__SEARCHTYPE=Episode + - SONARR__BLOCK__TYPE=blacklist + - SONARR__BLOCK__PATH=https://example.com/path/to/file.txt - SONARR__INSTANCES__0__URL=http://localhost:8989 - SONARR__INSTANCES__0__APIKEY=secret1 - SONARR__INSTANCES__1__URL=http://localhost:8990 - SONARR__INSTANCES__1__APIKEY=secret2 - RADARR__ENABLED=true + - RADARR__BLOCK__TYPE=blacklist + - RADARR__BLOCK__PATH=https://example.com/path/to/file.txt - RADARR__INSTANCES__0__URL=http://localhost:7878 - RADARR__INSTANCES__0__APIKEY=secret3 - RADARR__INSTANCES__1__URL=http://localhost:7879 - RADARR__INSTANCES__1__APIKEY=secret4 + + - LIDARR__ENABLED=true + - LIDARR__BLOCK__TYPE=blacklist + - LIDARR__BLOCK__PATH=https://example.com/path/to/file.txt + - LIDARR__INSTANCES__0__URL=http://radarr:8686 + - LIDARR__INSTANCES__0__APIKEY=secret5 + - LIDARR__INSTANCES__1__URL=http://radarr:8687 + - LIDARR__INSTANCES__1__APIKEY=secret6 image: ghcr.io/flmorg/cleanuperr:latest restart: unless-stopped ``` -### Environment variables +## Environment variables + +### General variables +
+ Click here | Variable | Required | Description | Default value | |---|---|---|---| @@ -168,12 +184,15 @@ services: ||||| | CONTENTBLOCKER__ENABLED | No | Enable or disable the content blocker | false | | CONTENTBLOCKER__IGNORE_PRIVATE | No | Whether to ignore downloads from private trackers | false | -| CONTENTBLOCKER__BLACKLIST__ENABLED | Yes if content blocker is enabled and whitelist is not enabled | Enable or disable the blacklist | false | -| CONTENTBLOCKER__BLACKLIST__PATH | Yes if blacklist is enabled | Path to the blacklist (local file or url)
Needs to be json compatible | empty | -| CONTENTBLOCKER__WHITELIST__ENABLED | Yes if content blocker is enabled and blacklist is not enabled | Enable or disable the whitelist | false | -| CONTENTBLOCKER__WHITELIST__PATH | Yes if whitelist is enabled | Path to the whitelist (local file or url)
Needs to be json compatible | empty | -||||| -| DOWNLOAD_CLIENT | No | Download client that is used by *arrs
Can be `qbittorrent`, `deluge`, `transmission` or `none` | `qbittorrent` | +
+ +### Download client variables +
+ Click here + +| Variable | Required | Description | Default value | +|---|---|---|---| +| DOWNLOAD_CLIENT | No | Download client that is used by *arrs
Can be `qbittorrent`, `deluge`, `transmission` or `none` | `none` | | QBITTORRENT__URL | No | qBittorrent instance url | http://localhost:8112 | | QBITTORRENT__USERNAME | No | qBittorrent user | empty | | QBITTORRENT__PASSWORD | No | qBittorrent password | empty | @@ -184,42 +203,68 @@ services: | TRANSMISSION__URL | No | Transmission instance url | http://localhost:9091 | | TRANSMISSION__USERNAME | No | Transmission user | empty | | TRANSMISSION__PASSWORD | No | Transmission password | empty | -||||| -| SONARR__ENABLED | No | Enable or disable Sonarr cleanup | true | +
+ +### Arr variables +
+ Click here + +| Variable | Required | Description | Default value | +|---|---|---|---| +| SONARR__ENABLED | No | Enable or disable Sonarr cleanup | false | +| SONARR__BLOCK__TYPE | No | Block type
Can be `blacklist` or `whitelist` | `blacklist` | +| SONARR__BLOCK__PATH | No | Path to the blocklist (local file or url)
Needs to be json compatible | empty | | SONARR__SEARCHTYPE | No | What to search for after removing a queue item
Can be `Episode`, `Season` or `Series` | `Episode` | | SONARR__INSTANCES__0__URL | No | First Sonarr instance url | http://localhost:8989 | | SONARR__INSTANCES__0__APIKEY | No | First Sonarr instance API key | empty | ||||| | RADARR__ENABLED | No | Enable or disable Radarr cleanup | false | +| RADARR__BLOCK__TYPE | No | Block type
Can be `blacklist` or `whitelist` | `blacklist` | +| RADARR__BLOCK__PATH | No | Path to the blocklist (local file or url)
Needs to be json compatible | empty | | RADARR__INSTANCES__0__URL | No | First Radarr instance url | http://localhost:8989 | | RADARR__INSTANCES__0__APIKEY | No | First Radarr instance API key | empty | ||||| +| LIDARR__ENABLED | No | Enable or disable LIDARR cleanup | false | +| LIDARR__BLOCK__TYPE | No | Block type
Can be `blacklist` or `whitelist` | `blacklist` | +| LIDARR__BLOCK__PATH | No | Path to the blocklist (local file or url)
Needs to be json compatible | empty | +| LIDARR__INSTANCES__0__URL | No | First LIDARR instance url | http://localhost:8989 | +| LIDARR__INSTANCES__0__APIKEY | No | First LIDARR instance API key | empty | +
+ +### Advanced variables +
+ Click here + +| Variable | Required | Description | Default value | +|---|---|---|---| | HTTP_MAX_RETRIES | No | The number of times to retry a failed HTTP call (to *arrs, download clients etc.) | 0 | | HTTP_TIMEOUT | No | The number of seconds to wait before failing an HTTP call (to *arrs, download clients etc.) | 100 | +
+ # + ### To be noted -1. The blacklist and the whitelist can not be both enabled at the same time. -2. The queue cleaner and content blocker can be enabled or disabled separately, if you want to run only one of them. -3. Only one download client can be enabled at a time. If you have more than one download client, you should deploy multiple instances of cleanuperr. -4. The blocklists (blacklist/whitelist) should have a single pattern on each line and supports the following: +1. The queue cleaner and content blocker can be enabled or disabled separately, if you want to run only one of them. +2. Only one download client can be enabled at a time. If you have more than one download client, you should deploy multiple instances of cleanuperr. +3. The blocklists (blacklist/whitelist) should have a single pattern on each line and supports the following: ``` -*example // file name ends with "example" -example* // file name starts with "example" -*example* // file name has "example" in the name -example // file name is exactly the word "example" +*example // file name ends with "example" +example* // file name starts with "example" +*example* // file name has "example" in the name +example // file name is exactly the word "example" regex: // regex that needs to be marked at the start of the line with "regex:" ``` -5. Multiple Sonarr/Radarr instances can be specified using this format, where `` starts from 0: +4. Multiple Sonarr/Radarr/Lidarr instances can be specified using this format, where `` starts from `0`: ``` SONARR__INSTANCES____URL SONARR__INSTANCES____APIKEY ``` -6. Multiple failed import patterns can be specified using this format, where `` starts from 0: +5. Multiple failed import patterns can be specified using this format, where `` starts from 0: ``` QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__ ``` - +6. [This blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist) and [this whitelist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist) can be used for Sonarr and Radarr, but they are not suitable for other *arrs. # ### Binaries (if you're not using Docker) diff --git a/code/Common/Configuration/Arr/ArrConfig.cs b/code/Common/Configuration/Arr/ArrConfig.cs index 51f539bf..19713760 100644 --- a/code/Common/Configuration/Arr/ArrConfig.cs +++ b/code/Common/Configuration/Arr/ArrConfig.cs @@ -1,8 +1,19 @@ +using Common.Configuration.ContentBlocker; + namespace Common.Configuration.Arr; public abstract record ArrConfig { public required bool Enabled { get; init; } + + public Block Block { get; init; } = new(); public required List Instances { get; init; } +} + +public record Block +{ + public BlocklistType Type { get; set; } + + public string? Path { get; set; } } \ No newline at end of file diff --git a/code/Common/Configuration/Arr/LidarrConfig.cs b/code/Common/Configuration/Arr/LidarrConfig.cs new file mode 100644 index 00000000..33be0ec0 --- /dev/null +++ b/code/Common/Configuration/Arr/LidarrConfig.cs @@ -0,0 +1,6 @@ +namespace Common.Configuration.Arr; + +public sealed record LidarrConfig : ArrConfig +{ + public const string SectionName = "Lidarr"; +} \ No newline at end of file diff --git a/code/Domain/Enums/BlocklistType.cs b/code/Common/Configuration/ContentBlocker/BlocklistType.cs similarity index 53% rename from code/Domain/Enums/BlocklistType.cs rename to code/Common/Configuration/ContentBlocker/BlocklistType.cs index 99be1d59..fa0cee07 100644 --- a/code/Domain/Enums/BlocklistType.cs +++ b/code/Common/Configuration/ContentBlocker/BlocklistType.cs @@ -1,4 +1,4 @@ -namespace Domain.Enums; +namespace Common.Configuration.ContentBlocker; public enum BlocklistType { diff --git a/code/Common/Configuration/ContentBlocker/ContentBlockerConfig.cs b/code/Common/Configuration/ContentBlocker/ContentBlockerConfig.cs index 5d57f58a..fd882340 100644 --- a/code/Common/Configuration/ContentBlocker/ContentBlockerConfig.cs +++ b/code/Common/Configuration/ContentBlocker/ContentBlockerConfig.cs @@ -1,4 +1,4 @@ -using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.Configuration; namespace Common.Configuration.ContentBlocker; @@ -11,35 +11,7 @@ public sealed record ContentBlockerConfig : IJobConfig [ConfigurationKeyName("IGNORE_PRIVATE")] public bool IgnorePrivate { get; init; } - public PatternConfig? Blacklist { get; init; } - - public PatternConfig? Whitelist { get; init; } - public void Validate() { - if (!Enabled) - { - return; - } - - if (Blacklist is null && Whitelist is null) - { - throw new Exception("content blocker is enabled, but both blacklist and whitelist are missing"); - } - - if (Blacklist?.Enabled is true && Whitelist?.Enabled is true) - { - throw new Exception("only one exclusion (blacklist/whitelist) list is allowed"); - } - - if (Blacklist?.Enabled is true && string.IsNullOrEmpty(Blacklist.Path)) - { - throw new Exception("blacklist path is required"); - } - - if (Whitelist?.Enabled is true && string.IsNullOrEmpty(Whitelist.Path)) - { - throw new Exception("blacklist path is required"); - } } } \ No newline at end of file diff --git a/code/Common/Configuration/ContentBlocker/PatternConfig.cs b/code/Common/Configuration/ContentBlocker/PatternConfig.cs deleted file mode 100644 index 2dc63d90..00000000 --- a/code/Common/Configuration/ContentBlocker/PatternConfig.cs +++ /dev/null @@ -1,8 +0,0 @@ -namespace Common.Configuration.ContentBlocker; - -public sealed record PatternConfig -{ - public bool Enabled { get; init; } - - public string? Path { get; init; } -} \ No newline at end of file diff --git a/code/Common/Configuration/DownloadClient/DownloadClientConfig.cs b/code/Common/Configuration/DownloadClient/DownloadClientConfig.cs index ebbe0467..73b17087 100644 --- a/code/Common/Configuration/DownloadClient/DownloadClientConfig.cs +++ b/code/Common/Configuration/DownloadClient/DownloadClientConfig.cs @@ -5,5 +5,5 @@ namespace Common.Configuration.DownloadClient; public sealed record DownloadClientConfig { [ConfigurationKeyName("DOWNLOAD_CLIENT")] - public Enums.DownloadClient DownloadClient { get; init; } = Enums.DownloadClient.QBittorrent; + public Enums.DownloadClient DownloadClient { get; init; } = Enums.DownloadClient.None; } \ No newline at end of file diff --git a/code/Domain/Models/Arr/Blocking/BlockedItem.cs b/code/Domain/Models/Arr/Blocking/BlockedItem.cs new file mode 100644 index 00000000..b7d94687 --- /dev/null +++ b/code/Domain/Models/Arr/Blocking/BlockedItem.cs @@ -0,0 +1,8 @@ +namespace Domain.Models.Arr.Blocking; + +public record BlockedItem +{ + public required string Hash { get; init; } + + public required Uri InstanceUrl { get; init; } +} \ No newline at end of file diff --git a/code/Domain/Models/Arr/Blocking/LidarrBlockedItem.cs b/code/Domain/Models/Arr/Blocking/LidarrBlockedItem.cs new file mode 100644 index 00000000..6a2b59af --- /dev/null +++ b/code/Domain/Models/Arr/Blocking/LidarrBlockedItem.cs @@ -0,0 +1,8 @@ +namespace Domain.Models.Arr.Blocking; + +public sealed record LidarrBlockedItem : BlockedItem +{ + public required long AlbumId { get; init; } + + public required long ArtistId { get; init; } +} \ No newline at end of file diff --git a/code/Domain/Models/Arr/Blocking/RadarrBlockedItem.cs b/code/Domain/Models/Arr/Blocking/RadarrBlockedItem.cs new file mode 100644 index 00000000..16532dab --- /dev/null +++ b/code/Domain/Models/Arr/Blocking/RadarrBlockedItem.cs @@ -0,0 +1,6 @@ +namespace Domain.Models.Arr.Blocking; + +public sealed record RadarrBlockedItem : BlockedItem +{ + public required long MovieId { get; init; } +} \ No newline at end of file diff --git a/code/Domain/Models/Arr/Blocking/SonarrBlockedItem.cs b/code/Domain/Models/Arr/Blocking/SonarrBlockedItem.cs new file mode 100644 index 00000000..a42b0acb --- /dev/null +++ b/code/Domain/Models/Arr/Blocking/SonarrBlockedItem.cs @@ -0,0 +1,10 @@ +namespace Domain.Models.Arr.Blocking; + +public sealed record SonarrBlockedItem : BlockedItem +{ + public required long EpisodeId { get; init; } + + public required long SeasonNumber { get; init; } + + public required long SeriesId { get; init; } +} \ No newline at end of file diff --git a/code/Domain/Models/Arr/Queue/QueueRecord.cs b/code/Domain/Models/Arr/Queue/QueueRecord.cs index b5a940aa..c66f5980 100644 --- a/code/Domain/Models/Arr/Queue/QueueRecord.cs +++ b/code/Domain/Models/Arr/Queue/QueueRecord.cs @@ -2,10 +2,20 @@ namespace Domain.Models.Arr.Queue; public sealed record QueueRecord { - public int SeriesId { get; init; } - public int EpisodeId { get; init; } - public int SeasonNumber { get; init; } - public int MovieId { get; init; } + // Sonarr + public long SeriesId { get; init; } + public long EpisodeId { get; init; } + public long SeasonNumber { get; init; } + + // Radarr + public long MovieId { get; init; } + + // Lidarr + public long ArtistId { get; init; } + + public long AlbumId { get; init; } + + // common public required string Title { get; init; } public string Status { get; init; } public string TrackedDownloadStatus { get; init; } @@ -13,5 +23,5 @@ public sealed record QueueRecord public List? StatusMessages { get; init; } public required string DownloadId { get; init; } public required string Protocol { get; init; } - public required int Id { get; init; } + public required long Id { get; init; } } \ No newline at end of file diff --git a/code/Domain/Models/Lidarr/Album.cs b/code/Domain/Models/Lidarr/Album.cs new file mode 100644 index 00000000..bfd0f67d --- /dev/null +++ b/code/Domain/Models/Lidarr/Album.cs @@ -0,0 +1,12 @@ +namespace Domain.Models.Lidarr; + +public sealed record Album +{ + public long Id { get; set; } + + public string Title { get; set; } + + public long ArtistId { get; set; } + + public Artist Artist { get; set; } +} \ No newline at end of file diff --git a/code/Domain/Models/Lidarr/Artist.cs b/code/Domain/Models/Lidarr/Artist.cs new file mode 100644 index 00000000..483c0ad3 --- /dev/null +++ b/code/Domain/Models/Lidarr/Artist.cs @@ -0,0 +1,8 @@ +namespace Domain.Models.Lidarr; + +public sealed record Artist +{ + public long Id { get; set; } + + public string ArtistName { get; set; } +} \ No newline at end of file diff --git a/code/Domain/Models/Lidarr/LidarrCommand.cs b/code/Domain/Models/Lidarr/LidarrCommand.cs new file mode 100644 index 00000000..63c140c5 --- /dev/null +++ b/code/Domain/Models/Lidarr/LidarrCommand.cs @@ -0,0 +1,10 @@ +namespace Domain.Models.Lidarr; + +public sealed record LidarrCommand +{ + public string Name { get; set; } + + public List AlbumIds { get; set; } + + public long ArtistId { get; set; } +} \ No newline at end of file diff --git a/code/Executable/DependencyInjection/ConfigurationDI.cs b/code/Executable/DependencyInjection/ConfigurationDI.cs index 57fae7f4..efd351e4 100644 --- a/code/Executable/DependencyInjection/ConfigurationDI.cs +++ b/code/Executable/DependencyInjection/ConfigurationDI.cs @@ -1,10 +1,8 @@ -using Common.Configuration; -using Common.Configuration.Arr; +using Common.Configuration.Arr; using Common.Configuration.ContentBlocker; using Common.Configuration.DownloadClient; using Common.Configuration.Logging; using Common.Configuration.QueueCleaner; -using Domain.Enums; namespace Executable.DependencyInjection; @@ -20,5 +18,6 @@ public static class ConfigurationDI .Configure(configuration.GetSection(TransmissionConfig.SectionName)) .Configure(configuration.GetSection(SonarrConfig.SectionName)) .Configure(configuration.GetSection(RadarrConfig.SectionName)) + .Configure(configuration.GetSection(LidarrConfig.SectionName)) .Configure(configuration.GetSection(LoggingConfig.SectionName)); } \ No newline at end of file diff --git a/code/Executable/DependencyInjection/LoggingDI.cs b/code/Executable/DependencyInjection/LoggingDI.cs index 0de177b4..96cf8db2 100644 --- a/code/Executable/DependencyInjection/LoggingDI.cs +++ b/code/Executable/DependencyInjection/LoggingDI.cs @@ -1,4 +1,5 @@ using Common.Configuration.Logging; +using Domain.Enums; using Infrastructure.Verticals.ContentBlocker; using Infrastructure.Verticals.QueueCleaner; using Serilog; @@ -27,11 +28,22 @@ public static class LoggingDI } LoggerConfiguration logConfig = new(); - const string consoleOutputTemplate = "[{@t:yyyy-MM-dd HH:mm:ss.fff} {@l:u3}]{#if JobName is not null} {Concat('[',JobName,']'),PAD}{#end} {@m}\n{@x}"; - const string fileOutputTemplate = "{@t:yyyy-MM-dd HH:mm:ss.fff zzz} [{@l:u3}]{#if JobName is not null} {Concat('[',JobName,']'),PAD}{#end} {@m:lj}\n{@x}"; + const string jobNameTemplate = "{#if JobName is not null} {Concat('[',JobName,']'),JOB_PAD}{#end}"; + const string instanceNameTemplate = "{#if InstanceName is not null} {Concat('[',InstanceName,']'),ARR_PAD}"; + const string consoleOutputTemplate = $"[{{@t:yyyy-MM-dd HH:mm:ss.fff}} {{@l:u3}}]{jobNameTemplate}{instanceNameTemplate}{{#end}} {{@m}}\n{{@x}}"; + const string fileOutputTemplate = $"{{@t:yyyy-MM-dd HH:mm:ss.fff zzz}} [{{@l:u3}}]{jobNameTemplate}{instanceNameTemplate} {{@m:lj}}\n{{@x}}"; LogEventLevel level = LogEventLevel.Information; - List jobNames = [nameof(ContentBlocker), nameof(QueueCleaner)]; - int padding = jobNames.Max(x => x.Length) + 2; + List names = [nameof(ContentBlocker), nameof(QueueCleaner)]; + int jobPadding = names.Max(x => x.Length) + 2; + names = [InstanceType.Sonarr.ToString(), InstanceType.Radarr.ToString(), InstanceType.Lidarr.ToString()]; + int arrPadding = names.Max(x => x.Length) + 2; + + string consoleTemplate = consoleOutputTemplate + .Replace("JOB_PAD", jobPadding.ToString()) + .Replace("ARR_PAD", arrPadding.ToString()); + string fileTemplate = fileOutputTemplate + .Replace("JOB_PAD", jobPadding.ToString()) + .Replace("ARR_PAD", arrPadding.ToString()); if (config is not null) { @@ -41,7 +53,7 @@ public static class LoggingDI { logConfig.WriteTo.File( path: Path.Combine(config.File.Path, "cleanuperr-.txt"), - formatter: new ExpressionTemplate(fileOutputTemplate.Replace("PAD", padding.ToString())), + formatter: new ExpressionTemplate(fileTemplate), fileSizeLimitBytes: 10L * 1024 * 1024, rollingInterval: RollingInterval.Day, rollOnFileSizeLimit: true @@ -55,7 +67,7 @@ public static class LoggingDI .MinimumLevel.Override("Microsoft.Extensions.Http", LogEventLevel.Warning) .MinimumLevel.Override("Quartz", LogEventLevel.Warning) .MinimumLevel.Override("System.Net.Http.HttpClient", LogEventLevel.Error) - .WriteTo.Console(new ExpressionTemplate(consoleOutputTemplate.Replace("PAD", padding.ToString()))) + .WriteTo.Console(new ExpressionTemplate(consoleTemplate)) .Enrich.FromLogContext() .Enrich.WithProperty("ApplicationName", "cleanuperr") .CreateLogger(); diff --git a/code/Executable/DependencyInjection/ServicesDI.cs b/code/Executable/DependencyInjection/ServicesDI.cs index 7c5503f8..73fa6182 100644 --- a/code/Executable/DependencyInjection/ServicesDI.cs +++ b/code/Executable/DependencyInjection/ServicesDI.cs @@ -15,6 +15,7 @@ public static class ServicesDI services .AddTransient() .AddTransient() + .AddTransient() .AddTransient() .AddTransient() .AddTransient() diff --git a/code/Executable/appsettings.Development.json b/code/Executable/appsettings.Development.json index e25e77b9..fe798844 100644 --- a/code/Executable/appsettings.Development.json +++ b/code/Executable/appsettings.Development.json @@ -15,15 +15,7 @@ }, "ContentBlocker": { "Enabled": true, - "IGNORE_PRIVATE": true, - "Blacklist": { - "Enabled": false, - "Path": "https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist" - }, - "Whitelist": { - "Enabled": false, - "Path": "https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist" - } + "IGNORE_PRIVATE": true }, "QueueCleaner": { "Enabled": true, @@ -54,19 +46,40 @@ "Sonarr": { "Enabled": true, "SearchType": "Episode", + "Block": { + "Type": "blacklist", + "Path": "https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist" + }, "Instances": [ { "Url": "http://localhost:8989", - "ApiKey": "96736c3eb3144936b8f1d62d27be8cee" + "ApiKey": "425d1e713f0c405cbbf359ac0502c1f4" } ] }, "Radarr": { "Enabled": true, + "Block": { + "Type": "blacklist", + "Path": "https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist" + }, "Instances": [ { "Url": "http://localhost:7878", - "ApiKey": "705b553732ab4167ab23909305d60600" + "ApiKey": "8b7454f668e54c5b8f44f56f93969761" + } + ] + }, + "Lidarr": { + "Enabled": true, + "Block": { + "Type": "blacklist", + "Path": "https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist" + }, + "Instances": [ + { + "Url": "http://localhost:8686", + "ApiKey": "7f677cfdc074414397af53dd633860c5" } ] } diff --git a/code/Executable/appsettings.json b/code/Executable/appsettings.json index a558261d..e30f3753 100644 --- a/code/Executable/appsettings.json +++ b/code/Executable/appsettings.json @@ -15,15 +15,7 @@ }, "ContentBlocker": { "Enabled": false, - "IGNORE_PRIVATE": false, - "Blacklist": { - "Enabled": false, - "Path": "" - }, - "Whitelist": { - "Enabled": false, - "Path": "" - } + "IGNORE_PRIVATE": false }, "QueueCleaner": { "Enabled": true, @@ -34,7 +26,7 @@ "STALLED_MAX_STRIKES": 5, "STALLED_IGNORE_PRIVATE": false }, - "DOWNLOAD_CLIENT": "qbittorrent", + "DOWNLOAD_CLIENT": "none", "qBittorrent": { "Url": "http://localhost:8080", "Username": "", @@ -50,8 +42,12 @@ "Password": "testing" }, "Sonarr": { - "Enabled": true, + "Enabled": false, "SearchType": "Episode", + "Block": { + "Type": "blacklist", + "Path": "" + }, "Instances": [ { "Url": "http://localhost:8989", @@ -61,11 +57,28 @@ }, "Radarr": { "Enabled": false, + "Block": { + "Type": "blacklist", + "Path": "" + }, "Instances": [ { "Url": "http://localhost:7878", "ApiKey": "" } ] + }, + "Lidarr": { + "Enabled": false, + "Block": { + "Type": "blacklist", + "Path": "" + }, + "Instances": [ + { + "Url": "http://localhost:8686", + "ApiKey": "" + } + ] } } diff --git a/code/Infrastructure/Verticals/Arr/ArrClient.cs b/code/Infrastructure/Verticals/Arr/ArrClient.cs index 62871dff..797015b3 100644 --- a/code/Infrastructure/Verticals/Arr/ArrClient.cs +++ b/code/Infrastructure/Verticals/Arr/ArrClient.cs @@ -101,9 +101,9 @@ public abstract class ArrClient return false; } - public virtual async Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord queueRecord) + public virtual async Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record) { - Uri uri = new(arrInstance.Url, $"/api/v3/queue/{queueRecord.Id}?removeFromClient=true&blocklist=true&skipRedownload=true&changeCategory=false"); + Uri uri = new(arrInstance.Url, GetQueueDeleteUrlPath(record.Id)); using HttpRequestMessage request = new(HttpMethod.Delete, uri); SetApiKey(request, arrInstance.ApiKey); @@ -114,16 +114,16 @@ public abstract class ArrClient { response.EnsureSuccessStatusCode(); - _logger.LogInformation("queue item deleted | {url} | {title}", arrInstance.Url, queueRecord.Title); + _logger.LogInformation("queue item deleted | {url} | {title}", arrInstance.Url, record.Title); } catch { - _logger.LogError("queue delete failed | {uri} | {title}", uri, queueRecord.Title); + _logger.LogError("queue delete failed | {uri} | {title}", uri, record.Title); throw; } } - public abstract Task RefreshItemsAsync(ArrInstance arrInstance, ArrConfig config, HashSet? items); + public abstract Task RefreshItemsAsync(ArrInstance arrInstance, HashSet? items); public virtual bool IsRecordValid(QueueRecord record) { @@ -143,6 +143,8 @@ public abstract class ArrClient } protected abstract string GetQueueUrlPath(int page); + + protected abstract string GetQueueDeleteUrlPath(long recordId); protected virtual void SetApiKey(HttpRequestMessage request, string apiKey) { diff --git a/code/Infrastructure/Verticals/Arr/LidarrClient.cs b/code/Infrastructure/Verticals/Arr/LidarrClient.cs new file mode 100644 index 00000000..540d3b8d --- /dev/null +++ b/code/Infrastructure/Verticals/Arr/LidarrClient.cs @@ -0,0 +1,145 @@ +using System.Text; +using Common.Configuration.Arr; +using Common.Configuration.Logging; +using Common.Configuration.QueueCleaner; +using Domain.Models.Arr; +using Domain.Models.Arr.Queue; +using Domain.Models.Lidarr; +using Infrastructure.Verticals.ItemStriker; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using Newtonsoft.Json; + +namespace Infrastructure.Verticals.Arr; + +public sealed class LidarrClient : ArrClient +{ + public LidarrClient( + ILogger logger, + IHttpClientFactory httpClientFactory, + IOptions loggingConfig, + IOptions queueCleanerConfig, + Striker striker + ) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker) + { + } + + protected override string GetQueueUrlPath(int page) + { + return $"/api/v1/queue?page={page}&pageSize=200&includeUnknownArtistItems=true&includeArtist=true&includeAlbum=true"; + } + + protected override string GetQueueDeleteUrlPath(long recordId) + { + return $"/api/v1/queue/{recordId}?removeFromClient=true&blocklist=true&skipRedownload=true&changeCategory=false"; + } + + public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet? items) + { + if (items?.Count is null or 0) return; + + Uri uri = new(arrInstance.Url, "/api/v1/command"); + + foreach (var command in GetSearchCommands(items)) + { + using HttpRequestMessage request = new(HttpMethod.Post, uri); + request.Content = new StringContent( + JsonConvert.SerializeObject(command, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }), + Encoding.UTF8, + "application/json" + ); + SetApiKey(request, arrInstance.ApiKey); + + using var response = await _httpClient.SendAsync(request); + string? logContext = await ComputeCommandLogContextAsync(arrInstance, command); + + try + { + response.EnsureSuccessStatusCode(); + + _logger.LogInformation("{log}", GetSearchLog(arrInstance.Url, command, true, logContext)); + } + catch + { + _logger.LogError("{log}", GetSearchLog(arrInstance.Url, command, false, logContext)); + throw; + } + } + } + + public override bool IsRecordValid(QueueRecord record) + { + if (record.ArtistId is 0 || record.AlbumId is 0) + { + _logger.LogDebug("skip | artist id and/or album id missing | {title}", record.Title); + return false; + } + + return base.IsRecordValid(record); + } + + private static string GetSearchLog( + Uri instanceUrl, + LidarrCommand command, + bool success, + string? logContext + ) + { + string status = success ? "triggered" : "failed"; + + return $"album search {status} | {instanceUrl} | {logContext ?? $"albums: {string.Join(',', command.AlbumIds)}"}"; + } + + private async Task ComputeCommandLogContextAsync(ArrInstance arrInstance, LidarrCommand command) + { + try + { + if (!_loggingConfig.Enhanced) return null; + + StringBuilder log = new(); + + var albums = await GetAlbumsAsync(arrInstance, command.AlbumIds); + + if (albums?.Count is null or 0) return null; + + var groups = albums + .GroupBy(x => x.Artist.Id) + .ToList(); + + foreach (var group in groups) + { + var first = group.First(); + + log.Append($"[{first.Artist.ArtistName} albums {string.Join(',', group.Select(x => x.Title).ToList())}]"); + } + + return log.ToString(); + } + catch (Exception exception) + { + _logger.LogDebug(exception, "failed to compute log context"); + } + + return null; + } + + private async Task?> GetAlbumsAsync(ArrInstance arrInstance, List albumIds) + { + Uri uri = new(arrInstance.Url, $"api/v1/album?{string.Join('&', albumIds.Select(x => $"albumIds={x}"))}"); + using HttpRequestMessage request = new(HttpMethod.Get, uri); + SetApiKey(request, arrInstance.ApiKey); + + using var response = await _httpClient.SendAsync(request); + response.EnsureSuccessStatusCode(); + + string responseBody = await response.Content.ReadAsStringAsync(); + return JsonConvert.DeserializeObject>(responseBody); + } + + private List GetSearchCommands(HashSet items) + { + const string albumSearch = "AlbumSearch"; + + return [new LidarrCommand { Name = albumSearch, AlbumIds = items.Select(i => i.Id).ToList() }]; + } +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Arr/RadarrClient.cs b/code/Infrastructure/Verticals/Arr/RadarrClient.cs index a5bc7573..a223bdcf 100644 --- a/code/Infrastructure/Verticals/Arr/RadarrClient.cs +++ b/code/Infrastructure/Verticals/Arr/RadarrClient.cs @@ -6,7 +6,6 @@ using Domain.Models.Arr; using Domain.Models.Arr.Queue; using Domain.Models.Radarr; using Infrastructure.Verticals.ItemStriker; -using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using Newtonsoft.Json; @@ -30,7 +29,12 @@ public sealed class RadarrClient : ArrClient return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownMovieItems=true&includeMovie=true"; } - public override async Task RefreshItemsAsync(ArrInstance arrInstance, ArrConfig config, HashSet? items) + protected override string GetQueueDeleteUrlPath(long recordId) + { + return $"/api/v3/queue/{recordId}?removeFromClient=true&blocklist=true&skipRedownload=true&changeCategory=false"; + } + + public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet? items) { if (items?.Count is null or 0) { @@ -74,7 +78,7 @@ public sealed class RadarrClient : ArrClient { if (record.MovieId is 0) { - _logger.LogDebug("skip | item information missing | {title}", record.Title); + _logger.LogDebug("skip | movie id missing | {title}", record.Title); return false; } diff --git a/code/Infrastructure/Verticals/Arr/SonarrClient.cs b/code/Infrastructure/Verticals/Arr/SonarrClient.cs index 2f6f119a..d78da82a 100644 --- a/code/Infrastructure/Verticals/Arr/SonarrClient.cs +++ b/code/Infrastructure/Verticals/Arr/SonarrClient.cs @@ -6,7 +6,6 @@ using Domain.Models.Arr; using Domain.Models.Arr.Queue; using Domain.Models.Sonarr; using Infrastructure.Verticals.ItemStriker; -using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using Newtonsoft.Json; @@ -29,8 +28,13 @@ public sealed class SonarrClient : ArrClient { return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownSeriesItems=true&includeSeries=true"; } + + protected override string GetQueueDeleteUrlPath(long recordId) + { + return $"/api/v3/queue/{recordId}?removeFromClient=true&blocklist=true&skipRedownload=true&changeCategory=false"; + } - public override async Task RefreshItemsAsync(ArrInstance arrInstance, ArrConfig config, HashSet? items) + public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet? items) { if (items?.Count is null or 0) { @@ -70,7 +74,7 @@ public sealed class SonarrClient : ArrClient { if (record.EpisodeId is 0 || record.SeriesId is 0) { - _logger.LogDebug("skip | item information missing | {title}", record.Title); + _logger.LogDebug("skip | episode id and/or series id missing | {title}", record.Title); return false; } diff --git a/code/Infrastructure/Verticals/ContentBlocker/BlocklistProvider.cs b/code/Infrastructure/Verticals/ContentBlocker/BlocklistProvider.cs index 7d0e4de6..fa368638 100644 --- a/code/Infrastructure/Verticals/ContentBlocker/BlocklistProvider.cs +++ b/code/Infrastructure/Verticals/ContentBlocker/BlocklistProvider.cs @@ -1,9 +1,11 @@ -using System.Collections.Concurrent; +using System.Collections.Concurrent; using System.Diagnostics; using System.Text.RegularExpressions; +using Common.Configuration.Arr; using Common.Configuration.ContentBlocker; using Common.Helpers; using Domain.Enums; +using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; @@ -12,78 +14,98 @@ namespace Infrastructure.Verticals.ContentBlocker; public sealed class BlocklistProvider { private readonly ILogger _logger; - private readonly ContentBlockerConfig _config; + private readonly SonarrConfig _sonarrConfig; + private readonly RadarrConfig _radarrConfig; + private readonly LidarrConfig _lidarrConfig; private readonly HttpClient _httpClient; - - public BlocklistType BlocklistType { get; } + private readonly IMemoryCache _cache; + private bool _initialized; - public ConcurrentBag Patterns { get; } = []; - - public ConcurrentBag Regexes { get; } = []; + private const string Type = "type"; + private const string Patterns = "patterns"; + private const string Regexes = "regexes"; public BlocklistProvider( ILogger logger, - IOptions config, - IHttpClientFactory httpClientFactory) + IOptions sonarrConfig, + IOptions radarrConfig, + IOptions lidarrConfig, + IMemoryCache cache, + IHttpClientFactory httpClientFactory + ) { _logger = logger; - _config = config.Value; + _sonarrConfig = sonarrConfig.Value; + _radarrConfig = radarrConfig.Value; + _lidarrConfig = lidarrConfig.Value; + _cache = cache; _httpClient = httpClientFactory.CreateClient(Constants.HttpClientWithRetryName); - - _config.Validate(); - - if (_config.Blacklist?.Enabled is true) - { - BlocklistType = BlocklistType.Blacklist; - } - - if (_config.Whitelist?.Enabled is true) - { - BlocklistType = BlocklistType.Whitelist; - } } - public async Task LoadBlocklistAsync() + public async Task LoadBlocklistsAsync() { - if (Patterns.Count > 0 || Regexes.Count > 0) + if (_initialized) { - _logger.LogDebug("blocklist already loaded"); + _logger.LogDebug("blocklists already loaded"); return; } try { - await LoadPatternsAndRegexesAsync(); + await LoadPatternsAndRegexesAsync(_sonarrConfig.Block.Type, _sonarrConfig.Block.Path, InstanceType.Sonarr); + await LoadPatternsAndRegexesAsync(_radarrConfig.Block.Type, _radarrConfig.Block.Path, InstanceType.Radarr); + await LoadPatternsAndRegexesAsync(_lidarrConfig.Block.Type, _lidarrConfig.Block.Path, InstanceType.Lidarr); + + _initialized = true; } catch { - _logger.LogError("failed to load {type}", BlocklistType.ToString()); + _logger.LogError("failed to load blocklists"); throw; } } - private async Task LoadPatternsAndRegexesAsync() + public BlocklistType GetBlocklistType(InstanceType instanceType) { - string[] patterns; + _cache.TryGetValue($"{instanceType.ToString()}_{Type}", out BlocklistType? blocklistType); + + return blocklistType ?? BlocklistType.Blacklist; + } + + public ConcurrentBag GetPatterns(InstanceType instanceType) + { + _cache.TryGetValue($"{instanceType.ToString()}_{Patterns}", out ConcurrentBag? patterns); + + return patterns ?? []; + } + + public ConcurrentBag GetRegexes(InstanceType instanceType) + { + _cache.TryGetValue($"{instanceType.ToString()}_{Regexes}", out ConcurrentBag? regexes); - if (BlocklistType is BlocklistType.Blacklist) + return regexes ?? []; + } + + private async Task LoadPatternsAndRegexesAsync(BlocklistType blocklistType, string? blocklistPath, InstanceType instanceType) + { + if (string.IsNullOrEmpty(blocklistPath)) { - patterns = await ReadContentAsync(_config.Blacklist.Path); - } - else - { - patterns = await ReadContentAsync(_config.Whitelist.Path); + return; } + string[] filePatterns = await ReadContentAsync(blocklistPath); + long startTime = Stopwatch.GetTimestamp(); ParallelOptions options = new() { MaxDegreeOfParallelism = 5 }; const string regexId = "regex:"; + ConcurrentBag patterns = []; + ConcurrentBag regexes = []; - Parallel.ForEach(patterns, options, pattern => + Parallel.ForEach(filePatterns, options, pattern => { if (!pattern.StartsWith(regexId)) { - Patterns.Add(pattern); + patterns.Add(pattern); return; } @@ -92,7 +114,7 @@ public sealed class BlocklistProvider try { Regex regex = new(pattern, RegexOptions.Compiled); - Regexes.Add(regex); + regexes.Add(regex); } catch (ArgumentException) { @@ -101,10 +123,14 @@ public sealed class BlocklistProvider }); TimeSpan elapsed = Stopwatch.GetElapsedTime(startTime); + + _cache.Set($"{instanceType.ToString()}_{Type}", blocklistType); + _cache.Set($"{instanceType.ToString()}_{Patterns}", patterns); + _cache.Set($"{instanceType.ToString()}_{Regexes}", regexes); - _logger.LogDebug("loaded {count} patterns", Patterns.Count); - _logger.LogDebug("loaded {count} regexes", Regexes.Count); - _logger.LogDebug("blocklist loaded in {elapsed} ms", elapsed.TotalMilliseconds); + _logger.LogDebug("loaded {count} patterns", patterns.Count); + _logger.LogDebug("loaded {count} regexes", regexes.Count); + _logger.LogDebug("blocklist loaded in {elapsed} ms | {path}", elapsed.TotalMilliseconds, blocklistPath); } private async Task ReadContentAsync(string path) diff --git a/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs b/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs index aefe2fb1..793e82a1 100644 --- a/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs +++ b/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs @@ -1,12 +1,17 @@ -using Common.Configuration.Arr; +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.Arr; +using Common.Configuration.ContentBlocker; using Common.Configuration.DownloadClient; using Domain.Enums; +using Domain.Models.Arr; using Domain.Models.Arr.Queue; using Infrastructure.Verticals.Arr; using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.Jobs; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; +using Serilog.Context; namespace Infrastructure.Verticals.ContentBlocker; @@ -19,12 +24,19 @@ public sealed class ContentBlocker : GenericHandler IOptions downloadClientConfig, IOptions sonarrConfig, IOptions radarrConfig, + IOptions lidarrConfig, SonarrClient sonarrClient, RadarrClient radarrClient, + LidarrClient lidarrClient, ArrQueueIterator arrArrQueueIterator, BlocklistProvider blocklistProvider, DownloadServiceFactory downloadServiceFactory - ) : base(logger, downloadClientConfig, sonarrConfig.Value, radarrConfig.Value, sonarrClient, radarrClient, arrArrQueueIterator, downloadServiceFactory) + ) : base( + logger, downloadClientConfig, + sonarrConfig, radarrConfig, lidarrConfig, + sonarrClient, radarrClient, lidarrClient, + arrArrQueueIterator, downloadServiceFactory + ) { _blocklistProvider = blocklistProvider; } @@ -37,18 +49,40 @@ public sealed class ContentBlocker : GenericHandler return; } - await _blocklistProvider.LoadBlocklistAsync(); + bool blocklistIsConfigured = _sonarrConfig.Enabled && !string.IsNullOrEmpty(_sonarrConfig.Block.Path) || + _radarrConfig.Enabled && !string.IsNullOrEmpty(_radarrConfig.Block.Path) || + _lidarrConfig.Enabled && !string.IsNullOrEmpty(_lidarrConfig.Block.Path); + + if (!blocklistIsConfigured) + { + _logger.LogWarning("no blocklist is configured"); + return; + } + + await _blocklistProvider.LoadBlocklistsAsync(); await base.ExecuteAsync(); } protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType) { + using var _ = LogContext.PushProperty("InstanceName", instanceType.ToString()); + + HashSet itemsToBeRefreshed = []; ArrClient arrClient = GetClient(instanceType); + BlocklistType blocklistType = _blocklistProvider.GetBlocklistType(instanceType); + ConcurrentBag patterns = _blocklistProvider.GetPatterns(instanceType); + ConcurrentBag regexes = _blocklistProvider.GetRegexes(instanceType); await _arrArrQueueIterator.Iterate(arrClient, instance, async items => { - foreach (QueueRecord record in items) + var groups = items + .GroupBy(x => x.DownloadId) + .ToList(); + + foreach (var group in groups) { + QueueRecord record = group.First(); + if (record.Protocol is not "torrent") { continue; @@ -61,8 +95,19 @@ public sealed class ContentBlocker : GenericHandler } _logger.LogDebug("searching unwanted files for {title}", record.Title); - await _downloadService.BlockUnwantedFilesAsync(record.DownloadId); + + if (!await _downloadService.BlockUnwantedFilesAsync(record.DownloadId, blocklistType, patterns, regexes)) + { + continue; + } + + _logger.LogDebug("all files are marked as unwanted | {hash}", record.Title); + + itemsToBeRefreshed.Add(GetRecordSearchItem(instanceType, record, group.Count() > 1)); + await arrClient.DeleteQueueItemAsync(instance, record); } }); + + await arrClient.RefreshItemsAsync(instance, itemsToBeRefreshed); } } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/ContentBlocker/FilenameEvaluator.cs b/code/Infrastructure/Verticals/ContentBlocker/FilenameEvaluator.cs index ef17eddc..23b1f452 100644 --- a/code/Infrastructure/Verticals/ContentBlocker/FilenameEvaluator.cs +++ b/code/Infrastructure/Verticals/ContentBlocker/FilenameEvaluator.cs @@ -1,4 +1,6 @@ -using Domain.Enums; +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.ContentBlocker; using Microsoft.Extensions.Logging; namespace Infrastructure.Verticals.ContentBlocker; @@ -6,46 +8,44 @@ namespace Infrastructure.Verticals.ContentBlocker; public sealed class FilenameEvaluator { private readonly ILogger _logger; - private readonly BlocklistProvider _blocklistProvider; - public FilenameEvaluator(ILogger logger, BlocklistProvider blocklistProvider) + public FilenameEvaluator(ILogger logger) { _logger = logger; - _blocklistProvider = blocklistProvider; } // TODO create unit tests - public bool IsValid(string filename) + public bool IsValid(string filename, BlocklistType type, ConcurrentBag patterns, ConcurrentBag regexes) { - return IsValidAgainstPatterns(filename) && IsValidAgainstRegexes(filename); + return IsValidAgainstPatterns(filename, type, patterns) && IsValidAgainstRegexes(filename, type, regexes); } - private bool IsValidAgainstPatterns(string filename) + private static bool IsValidAgainstPatterns(string filename, BlocklistType type, ConcurrentBag patterns) { - if (_blocklistProvider.Patterns.Count is 0) + if (patterns.Count is 0) { return true; } - return _blocklistProvider.BlocklistType switch + return type switch { - BlocklistType.Blacklist => !_blocklistProvider.Patterns.Any(pattern => MatchesPattern(filename, pattern)), - BlocklistType.Whitelist => _blocklistProvider.Patterns.Any(pattern => MatchesPattern(filename, pattern)), + BlocklistType.Blacklist => !patterns.Any(pattern => MatchesPattern(filename, pattern)), + BlocklistType.Whitelist => patterns.Any(pattern => MatchesPattern(filename, pattern)), _ => true }; } - private bool IsValidAgainstRegexes(string filename) + private static bool IsValidAgainstRegexes(string filename, BlocklistType type, ConcurrentBag regexes) { - if (_blocklistProvider.Regexes.Count is 0) + if (regexes.Count is 0) { return true; } - return _blocklistProvider.BlocklistType switch + return type switch { - BlocklistType.Blacklist => !_blocklistProvider.Regexes.Any(regex => regex.IsMatch(filename)), - BlocklistType.Whitelist => _blocklistProvider.Regexes.Any(regex => regex.IsMatch(filename)), + BlocklistType.Blacklist => !regexes.Any(regex => regex.IsMatch(filename)), + BlocklistType.Whitelist => regexes.Any(regex => regex.IsMatch(filename)), _ => true }; } diff --git a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeClient.cs b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeClient.cs index 45194f94..aad8b31b 100644 --- a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeClient.cs +++ b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeClient.cs @@ -78,6 +78,11 @@ public sealed class DelugeClient await SendRequest>("core.set_torrent_options", hash, filePriorities); } + public async Task> DeleteTorrent(string hash) + { + return await SendRequest>("core.remove_torrents", new List { hash }, true); + } + private async Task PostJson(String json) { StringContent content = new StringContent(json); diff --git a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs index 62862db2..b2f95bca 100644 --- a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs @@ -1,3 +1,6 @@ +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.ContentBlocker; using Common.Configuration.DownloadClient; using Common.Configuration.QueueCleaner; using Domain.Models.Deluge.Response; @@ -30,6 +33,7 @@ public sealed class DelugeService : DownloadServiceBase await _client.LoginAsync(); } + /// public override async Task ShouldRemoveFromArrQueueAsync(string hash) { hash = hash.ToLowerInvariant(); @@ -70,7 +74,13 @@ public sealed class DelugeService : DownloadServiceBase return result; } - public override async Task BlockUnwantedFilesAsync(string hash) + /// + public override async Task BlockUnwantedFilesAsync( + string hash, + BlocklistType blocklistType, + ConcurrentBag patterns, + ConcurrentBag regexes + ) { hash = hash.ToLowerInvariant(); @@ -79,14 +89,14 @@ public sealed class DelugeService : DownloadServiceBase if (status?.Hash is null) { _logger.LogDebug("failed to find torrent {hash} in the download client", hash); - return; + return false; } if (_queueCleanerConfig.StalledIgnorePrivate && status.Private) { // ignore private trackers _logger.LogDebug("skip files check | download is private | {name}", status.Name); - return; + return false; } DelugeContents? contents = null; @@ -102,18 +112,27 @@ public sealed class DelugeService : DownloadServiceBase if (contents is null) { - return; + return false; } Dictionary priorities = []; bool hasPriorityUpdates = false; + long totalFiles = 0; + long totalUnwantedFiles = 0; ProcessFiles(contents.Contents, (name, file) => { + totalFiles++; int priority = file.Priority; - if (file.Priority is not 0 && !_filenameEvaluator.IsValid(name)) + if (file.Priority is 0) { + totalUnwantedFiles++; + } + + if (file.Priority is not 0 && !_filenameEvaluator.IsValid(name, blocklistType, patterns, regexes)) + { + totalUnwantedFiles++; priority = 0; hasPriorityUpdates = true; _logger.LogInformation("unwanted file found | {file}", file.Path); @@ -124,7 +143,7 @@ public sealed class DelugeService : DownloadServiceBase if (!hasPriorityUpdates) { - return; + return false; } _logger.LogDebug("changing priorities | torrent {hash}", hash); @@ -134,7 +153,23 @@ public sealed class DelugeService : DownloadServiceBase .Select(x => x.Value) .ToList(); + if (totalUnwantedFiles == totalFiles) + { + // Skip marking files as unwanted. The download will be removed completely. + return true; + } + await _client.ChangeFilesPriority(hash, sortedPriorities); + + return false; + } + + /// + public override async Task Delete(string hash) + { + hash = hash.ToLowerInvariant(); + + await _client.DeleteTorrent(hash); } private bool IsItemStuckAndShouldRemove(TorrentStatus status) @@ -173,8 +208,13 @@ public sealed class DelugeService : DownloadServiceBase ); } - private static void ProcessFiles(Dictionary contents, Action processFile) + private static void ProcessFiles(Dictionary? contents, Action processFile) { + if (contents is null) + { + return; + } + foreach (var (name, data) in contents) { switch (data.Type) diff --git a/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs b/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs index d8ec316d..0089a9f2 100644 --- a/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs +++ b/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs @@ -1,4 +1,7 @@ -using Common.Configuration.QueueCleaner; +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.ContentBlocker; +using Common.Configuration.QueueCleaner; using Domain.Enums; using Infrastructure.Verticals.ContentBlocker; using Infrastructure.Verticals.ItemStriker; @@ -33,8 +36,23 @@ public abstract class DownloadServiceBase : IDownloadService public abstract Task ShouldRemoveFromArrQueueAsync(string hash); - public abstract Task BlockUnwantedFilesAsync(string hash); + /// + public abstract Task BlockUnwantedFilesAsync( + string hash, + BlocklistType blocklistType, + ConcurrentBag patterns, + ConcurrentBag regexes + ); + /// + public abstract Task Delete(string hash); + + /// + /// Strikes an item and checks if the limit has been reached. + /// + /// The torrent hash. + /// The name or title of the item. + /// True if the limit has been reached; otherwise, false. protected bool StrikeAndCheckLimit(string hash, string itemName) { return _striker.StrikeAndCheckLimit(hash, itemName, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled); diff --git a/code/Infrastructure/Verticals/DownloadClient/DummyDownloadService.cs b/code/Infrastructure/Verticals/DownloadClient/DummyDownloadService.cs index eda4f1f1..4e0ac327 100644 --- a/code/Infrastructure/Verticals/DownloadClient/DummyDownloadService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/DummyDownloadService.cs @@ -1,4 +1,7 @@ -using Common.Configuration.QueueCleaner; +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.ContentBlocker; +using Common.Configuration.QueueCleaner; using Infrastructure.Verticals.ContentBlocker; using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; @@ -26,7 +29,12 @@ public sealed class DummyDownloadService : DownloadServiceBase throw new NotImplementedException(); } - public override Task BlockUnwantedFilesAsync(string hash) + public override Task BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType, ConcurrentBag patterns, ConcurrentBag regexes) + { + throw new NotImplementedException(); + } + + public override Task Delete(string hash) { throw new NotImplementedException(); } diff --git a/code/Infrastructure/Verticals/DownloadClient/IDownloadService.cs b/code/Infrastructure/Verticals/DownloadClient/IDownloadService.cs index 3f42b62f..45218b58 100644 --- a/code/Infrastructure/Verticals/DownloadClient/IDownloadService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/IDownloadService.cs @@ -1,10 +1,36 @@ -namespace Infrastructure.Verticals.DownloadClient; +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.ContentBlocker; + +namespace Infrastructure.Verticals.DownloadClient; public interface IDownloadService : IDisposable { public Task LoginAsync(); + /// + /// Checks whether the download should be removed from the *arr queue. + /// + /// The download hash. public Task ShouldRemoveFromArrQueueAsync(string hash); - public Task BlockUnwantedFilesAsync(string hash); + /// + /// Blocks unwanted files from being fully downloaded. + /// + /// The torrent hash. + /// The . + /// The patterns to test the files against. + /// The regexes to test the files against. + /// True if all files have been blocked; otherwise false. + public Task BlockUnwantedFilesAsync( + string hash, + BlocklistType blocklistType, + ConcurrentBag patterns, + ConcurrentBag regexes + ); + + /// + /// Deletes a download item. + /// + public Task Delete(string hash); } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs b/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs index 2d2ea7aa..0f717660 100644 --- a/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs @@ -1,3 +1,6 @@ +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.ContentBlocker; using Common.Configuration.DownloadClient; using Common.Configuration.QueueCleaner; using Common.Helpers; @@ -38,6 +41,7 @@ public sealed class QBitService : DownloadServiceBase await _client.LoginAsync(_config.Username, _config.Password); } + /// public override async Task ShouldRemoveFromArrQueueAsync(string hash) { RemoveResult result = new(); @@ -83,7 +87,13 @@ public sealed class QBitService : DownloadServiceBase return result; } - public override async Task BlockUnwantedFilesAsync(string hash) + /// + public override async Task BlockUnwantedFilesAsync( + string hash, + BlocklistType blocklistType, + ConcurrentBag patterns, + ConcurrentBag regexes + ) { TorrentInfo? torrent = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] })) .FirstOrDefault(); @@ -91,7 +101,7 @@ public sealed class QBitService : DownloadServiceBase if (torrent is null) { _logger.LogDebug("failed to find torrent {hash} in the download client", hash); - return; + return false; } TorrentProperties? torrentProperties = await _client.GetTorrentPropertiesAsync(hash); @@ -99,7 +109,7 @@ public sealed class QBitService : DownloadServiceBase if (torrentProperties is null) { _logger.LogDebug("failed to find torrent properties {hash} in the download client", hash); - return; + return false; } bool isPrivate = torrentProperties.AdditionalData.TryGetValue("is_private", out var dictValue) && @@ -110,15 +120,19 @@ public sealed class QBitService : DownloadServiceBase { // ignore private trackers _logger.LogDebug("skip files check | download is private | {name}", torrent.Name); - return; + return false; } IReadOnlyList? files = await _client.GetTorrentContentsAsync(hash); if (files is null) { - return; + return false; } + + List unwantedFiles = []; + long totalFiles = 0; + long totalUnwantedFiles = 0; foreach (TorrentContent file in files) { @@ -127,14 +141,47 @@ public sealed class QBitService : DownloadServiceBase continue; } - if (file.Priority is TorrentContentPriority.Skip || _filenameEvaluator.IsValid(file.Name)) + totalFiles++; + + if (file.Priority is TorrentContentPriority.Skip) + { + totalUnwantedFiles++; + continue; + } + + if (_filenameEvaluator.IsValid(file.Name, blocklistType, patterns, regexes)) { continue; } _logger.LogInformation("unwanted file found | {file}", file.Name); - await _client.SetFilePriorityAsync(hash, file.Index.Value, TorrentContentPriority.Skip); + unwantedFiles.Add(file.Index.Value); + totalUnwantedFiles++; } + + if (unwantedFiles.Count is 0) + { + return false; + } + + if (totalUnwantedFiles == totalFiles) + { + // Skip marking files as unwanted. The download will be removed completely. + return true; + } + + foreach (int fileIndex in unwantedFiles) + { + await _client.SetFilePriorityAsync(hash, fileIndex, TorrentContentPriority.Skip); + } + + return false; + } + + /// + public override async Task Delete(string hash) + { + await _client.DeleteAsync(hash, deleteDownloadedData: true); } public override void Dispose() diff --git a/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs b/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs index 0640f9d5..fe146d5c 100644 --- a/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs @@ -1,4 +1,7 @@ -using Common.Configuration.DownloadClient; +using System.Collections.Concurrent; +using System.Text.RegularExpressions; +using Common.Configuration.ContentBlocker; +using Common.Configuration.DownloadClient; using Common.Configuration.QueueCleaner; using Common.Helpers; using Infrastructure.Verticals.ContentBlocker; @@ -41,6 +44,7 @@ public sealed class TransmissionService : DownloadServiceBase await _client.GetSessionInformationAsync(); } + /// public override async Task ShouldRemoveFromArrQueueAsync(string hash) { RemoveResult result = new(); @@ -76,23 +80,31 @@ public sealed class TransmissionService : DownloadServiceBase return result; } - public override async Task BlockUnwantedFilesAsync(string hash) + /// + public override async Task BlockUnwantedFilesAsync( + string hash, + BlocklistType blocklistType, + ConcurrentBag patterns, + ConcurrentBag regexes + ) { TorrentInfo? torrent = await GetTorrentAsync(hash); if (torrent?.FileStats is null || torrent.Files is null) { - return; + return false; } if (_queueCleanerConfig.StalledIgnorePrivate && (torrent.IsPrivate ?? false)) { // ignore private trackers _logger.LogDebug("skip files check | download is private | {name}", torrent.Name); - return; + return false; } List unwantedFiles = []; + long totalFiles = 0; + long totalUnwantedFiles = 0; for (int i = 0; i < torrent.Files.Length; i++) { @@ -100,19 +112,34 @@ public sealed class TransmissionService : DownloadServiceBase { continue; } + + totalFiles++; - if (!torrent.FileStats[i].Wanted.Value || _filenameEvaluator.IsValid(torrent.Files[i].Name)) + if (!torrent.FileStats[i].Wanted.Value) + { + totalUnwantedFiles++; + continue; + } + + if (_filenameEvaluator.IsValid(torrent.Files[i].Name, blocklistType, patterns, regexes)) { continue; } _logger.LogInformation("unwanted file found | {file}", torrent.Files[i].Name); unwantedFiles.Add(i); + totalUnwantedFiles++; } if (unwantedFiles.Count is 0) { - return; + return false; + } + + if (totalUnwantedFiles == totalFiles) + { + // Skip marking files as unwanted. The download will be removed completely. + return true; } _logger.LogDebug("changing priorities | torrent {hash}", hash); @@ -122,6 +149,20 @@ public sealed class TransmissionService : DownloadServiceBase Ids = [ torrent.Id ], FilesUnwanted = unwantedFiles.ToArray(), }); + + return false; + } + + public override async Task Delete(string hash) + { + TorrentInfo? torrent = await GetTorrentAsync(hash); + + if (torrent is null) + { + return; + } + + await _client.TorrentRemoveAsync([torrent.Id], true); } public override void Dispose() diff --git a/code/Infrastructure/Verticals/ItemStriker/Striker.cs b/code/Infrastructure/Verticals/ItemStriker/Striker.cs index a30b262a..f814260e 100644 --- a/code/Infrastructure/Verticals/ItemStriker/Striker.cs +++ b/code/Infrastructure/Verticals/ItemStriker/Striker.cs @@ -37,7 +37,7 @@ public class Striker ++strikeCount; } - _logger.LogDebug("item on strike number {strike} | reason {reason} | {name}", strikeCount, strikeType.ToString(), itemName); + _logger.LogInformation("item on strike number {strike} | reason {reason} | {name}", strikeCount, strikeType.ToString(), itemName); _cache.Set(key, strikeCount, _cacheOptions); if (strikeCount < maxStrikes) diff --git a/code/Infrastructure/Verticals/Jobs/GenericHandler.cs b/code/Infrastructure/Verticals/Jobs/GenericHandler.cs index f40df8e5..83e4dc54 100644 --- a/code/Infrastructure/Verticals/Jobs/GenericHandler.cs +++ b/code/Infrastructure/Verticals/Jobs/GenericHandler.cs @@ -1,4 +1,4 @@ -using Common.Configuration.Arr; +using Common.Configuration.Arr; using Common.Configuration.DownloadClient; using Domain.Enums; using Domain.Models.Arr; @@ -16,28 +16,34 @@ public abstract class GenericHandler : IDisposable protected readonly DownloadClientConfig _downloadClientConfig; protected readonly SonarrConfig _sonarrConfig; protected readonly RadarrConfig _radarrConfig; + protected readonly LidarrConfig _lidarrConfig; protected readonly SonarrClient _sonarrClient; protected readonly RadarrClient _radarrClient; + protected readonly LidarrClient _lidarrClient; protected readonly ArrQueueIterator _arrArrQueueIterator; protected readonly IDownloadService _downloadService; protected GenericHandler( ILogger logger, IOptions downloadClientConfig, - SonarrConfig sonarrConfig, - RadarrConfig radarrConfig, + IOptions sonarrConfig, + IOptions radarrConfig, + IOptions lidarrConfig, SonarrClient sonarrClient, RadarrClient radarrClient, + LidarrClient lidarrClient, ArrQueueIterator arrArrQueueIterator, DownloadServiceFactory downloadServiceFactory ) { _logger = logger; _downloadClientConfig = downloadClientConfig.Value; - _sonarrConfig = sonarrConfig; - _radarrConfig = radarrConfig; + _sonarrConfig = sonarrConfig.Value; + _radarrConfig = radarrConfig.Value; + _lidarrConfig = lidarrConfig.Value; _sonarrClient = sonarrClient; _radarrClient = radarrClient; + _lidarrClient = lidarrClient; _arrArrQueueIterator = arrArrQueueIterator; _downloadService = downloadServiceFactory.CreateDownloadClient(); } @@ -48,6 +54,7 @@ public abstract class GenericHandler : IDisposable await ProcessArrConfigAsync(_sonarrConfig, InstanceType.Sonarr); await ProcessArrConfigAsync(_radarrConfig, InstanceType.Radarr); + await ProcessArrConfigAsync(_lidarrConfig, InstanceType.Lidarr); } public virtual void Dispose() @@ -82,17 +89,10 @@ public abstract class GenericHandler : IDisposable { InstanceType.Sonarr => _sonarrClient, InstanceType.Radarr => _radarrClient, + InstanceType.Lidarr => _lidarrClient, _ => throw new NotImplementedException($"instance type {type} is not yet supported") }; - protected ArrConfig GetConfig(InstanceType type) => - type switch - { - InstanceType.Sonarr => _sonarrConfig, - InstanceType.Radarr => _radarrConfig, - _ => throw new NotImplementedException($"instance type {type} is not yet supported") - }; - protected SearchItem GetRecordSearchItem(InstanceType type, QueueRecord record, bool isPack = false) { return type switch @@ -117,11 +117,15 @@ public abstract class GenericHandler : IDisposable }, InstanceType.Sonarr when _sonarrConfig.SearchType is SonarrSearchType.Series => new SonarrSearchItem { - Id = record.SeriesId, + Id = record.SeriesId }, InstanceType.Radarr => new SearchItem { - Id = record.MovieId, + Id = record.MovieId + }, + InstanceType.Lidarr => new SearchItem + { + Id = record.AlbumId }, _ => throw new NotImplementedException($"instance type {type} is not yet supported") }; diff --git a/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs b/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs index 91e30e86..77fc347b 100644 --- a/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs +++ b/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs @@ -8,6 +8,7 @@ using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.Jobs; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; +using Serilog.Context; namespace Infrastructure.Verticals.QueueCleaner; @@ -18,19 +19,27 @@ public sealed class QueueCleaner : GenericHandler IOptions downloadClientConfig, IOptions sonarrConfig, IOptions radarrConfig, + IOptions lidarrConfig, SonarrClient sonarrClient, RadarrClient radarrClient, + LidarrClient lidarrClient, ArrQueueIterator arrArrQueueIterator, DownloadServiceFactory downloadServiceFactory - ) : base(logger, downloadClientConfig, sonarrConfig.Value, radarrConfig.Value, sonarrClient, radarrClient, arrArrQueueIterator, downloadServiceFactory) + ) : base( + logger, downloadClientConfig, + sonarrConfig, radarrConfig, lidarrConfig, + sonarrClient, radarrClient, lidarrClient, + arrArrQueueIterator, downloadServiceFactory + ) { } protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType) { + using var _ = LogContext.PushProperty("InstanceName", instanceType.ToString()); + HashSet itemsToBeRefreshed = []; ArrClient arrClient = GetClient(instanceType); - ArrConfig arrConfig = GetConfig(instanceType); await _arrArrQueueIterator.Iterate(arrClient, instance, async items => { @@ -73,11 +82,10 @@ public sealed class QueueCleaner : GenericHandler } itemsToBeRefreshed.Add(GetRecordSearchItem(instanceType, record, group.Count() > 1)); - await arrClient.DeleteQueueItemAsync(instance, record); } }); - await arrClient.RefreshItemsAsync(instance, arrConfig, itemsToBeRefreshed); + await arrClient.RefreshItemsAsync(instance, itemsToBeRefreshed); } } \ No newline at end of file diff --git a/code/test/data/cleanuperr/config/music_whitelist b/code/test/data/cleanuperr/config/music_whitelist new file mode 100644 index 00000000..bf9e782d --- /dev/null +++ b/code/test/data/cleanuperr/config/music_whitelist @@ -0,0 +1 @@ +*.mp3 \ No newline at end of file diff --git a/code/test/data/cleanuperr/config/blacklist b/code/test/data/cleanuperr/config/video_blacklist similarity index 100% rename from code/test/data/cleanuperr/config/blacklist rename to code/test/data/cleanuperr/config/video_blacklist diff --git a/code/test/data/cleanuperr/config/whitelist b/code/test/data/cleanuperr/config/video_whitelist similarity index 100% rename from code/test/data/cleanuperr/config/whitelist rename to code/test/data/cleanuperr/config/video_whitelist diff --git a/code/test/data/deluge/config/label.conf b/code/test/data/deluge/config/label.conf index ca8d6e45..4c2a8226 100644 --- a/code/test/data/deluge/config/label.conf +++ b/code/test/data/deluge/config/label.conf @@ -3,6 +3,24 @@ "format": 1 }{ "labels": { + "lidarr": { + "apply_max": false, + "apply_move_completed": false, + "apply_queue": false, + "auto_add": false, + "auto_add_trackers": [], + "is_auto_managed": false, + "max_connections": -1, + "max_download_speed": -1, + "max_upload_slots": -1, + "max_upload_speed": -1, + "move_completed": false, + "move_completed_path": "", + "prioritize_first_last": false, + "remove_at_ratio": false, + "stop_at_ratio": false, + "stop_ratio": 2.0 + }, "radarr": { "apply_max": false, "apply_move_completed": false, diff --git a/code/test/data/lidarr/config/config.xml b/code/test/data/lidarr/config/config.xml new file mode 100644 index 00000000..440f8489 --- /dev/null +++ b/code/test/data/lidarr/config/config.xml @@ -0,0 +1,17 @@ + + * + 8686 + 6868 + False + True + 7f677cfdc074414397af53dd633860c5 + Forms + Enabled + master + debug + + + + Lidarr + Docker + \ No newline at end of file diff --git a/code/test/data/lidarr/config/lidarr.db b/code/test/data/lidarr/config/lidarr.db index 1d92f2c2..e6e78cd7 100644 Binary files a/code/test/data/lidarr/config/lidarr.db and b/code/test/data/lidarr/config/lidarr.db differ diff --git a/code/test/data/lidarr/config/logs.db-shm b/code/test/data/lidarr/config/lidarr.db-shm similarity index 98% rename from code/test/data/lidarr/config/logs.db-shm rename to code/test/data/lidarr/config/lidarr.db-shm index f5f573d5..d89118b0 100644 Binary files a/code/test/data/lidarr/config/logs.db-shm and b/code/test/data/lidarr/config/lidarr.db-shm differ diff --git a/code/test/data/lidarr/config/lidarr.db-wal b/code/test/data/lidarr/config/lidarr.db-wal new file mode 100644 index 00000000..0b53c171 Binary files /dev/null and b/code/test/data/lidarr/config/lidarr.db-wal differ diff --git a/code/test/data/lidarr/config/logs.db b/code/test/data/lidarr/config/logs.db index e8250213..c245a098 100644 Binary files a/code/test/data/lidarr/config/logs.db and b/code/test/data/lidarr/config/logs.db differ diff --git a/code/test/data/lidarr/config/logs.db-wal b/code/test/data/lidarr/config/logs.db-wal deleted file mode 100644 index 5f90fb80..00000000 Binary files a/code/test/data/lidarr/config/logs.db-wal and /dev/null differ diff --git a/code/test/data/nginx/lidarr.xml b/code/test/data/nginx/lidarr.xml new file mode 100644 index 00000000..e9cbbe5e --- /dev/null +++ b/code/test/data/nginx/lidarr.xml @@ -0,0 +1,38 @@ + + + Test feed + http://nginx/custom/sonarr.xml + + Test + + en-CA + Test + Tue, 5 Nov 2024 22:02:13 -0400 + Tue, 5 Nov 2024 22:02:13 -0400 + https://validator.w3.org/feed/docs/rss2.html + 30 + + + + Coldplay-Ghost Stories-(Deluxe Edition)-2014-MTD + Test + 104857600 + http://nginx/custom/lidarr_bad_single.torrent + + 174674a88c8947f6f9057a23f81efde384ed216cade43564ec450f2cb4677554 + + Sat, 24 Sep 2022 22:02:13 -0300 + + + + Coldplay-Everyday.Life-2019-C4 + Test + 104857600 + http://nginx/custom/lidarr_bad_pack.torrent + + 174674a88c8947f689057ac3f81efde384ed216cade43564ec450f2cb4677554 + + Sat, 24 Sep 2022 22:02:13 -0300 + + + \ No newline at end of file diff --git a/code/test/data/nginx/lidarr_bad_pack.torrent b/code/test/data/nginx/lidarr_bad_pack.torrent new file mode 100644 index 00000000..91d01ea2 --- /dev/null +++ b/code/test/data/nginx/lidarr_bad_pack.torrent @@ -0,0 +1 @@ +d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1736513625e4:infod5:filesld6:lengthi640e4:pathl27:coldplay-everyday_life.zipxeed6:lengthi640e4:pathl27:coldplay-everyday_life2.mp3eee4:name31:Coldplay-Everyday.Life-2019-C4/12:piece lengthi262144e6:pieces20:#Ú¯§Ñ4OduÎoÎÛ€¹Þ[=~ee \ No newline at end of file diff --git a/code/test/data/nginx/lidarr_bad_single.torrent b/code/test/data/nginx/lidarr_bad_single.torrent new file mode 100644 index 00000000..c829a7f3 --- /dev/null +++ b/code/test/data/nginx/lidarr_bad_single.torrent @@ -0,0 +1 @@ +d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1736513638e4:infod5:filesld6:lengthi640e4:pathl35:001-coldplay-always_in_my_head.zipxeee4:name49:Coldplay-Ghost Stories-(Deluxe Edition)-2014-MTD/12:piece lengthi262144e6:pieces20:·iV9qæ “Ý)-xÖ©'¦Èò«ee \ No newline at end of file diff --git a/code/test/data/nginx/radarr_bad_nested.torrent b/code/test/data/nginx/radarr_bad_nested.torrent index 8f738b02..b6e1106f 100644 --- a/code/test/data/nginx/radarr_bad_nested.torrent +++ b/code/test/data/nginx/radarr_bad_nested.torrent @@ -1 +1 @@ -d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1731931728e4:infod5:filesld6:lengthi2604e4:pathl4:Dir15:Dir1111:test11.zipxeed6:lengthi2604e4:pathl4:Dir110:sample.txteed6:lengthi2604e4:pathl4:Dir210:test2.zipxeed6:lengthi2604e4:pathl68:Speak.No.Evil.2024.2160p.MA.WEB-DL.DDP5.1.Atmos.H.265-HHWEB.mkv.zipxeed6:lengthi2604e4:pathl9:test.zipxeee4:name59:Speak.No.Evil.2024.2160p.MA.WEB-DL.DDP5.1.Atmos.H.265-HHWEB12:piece lengthi262144e6:pieces20:w¤ŸÌ³RÇþ'6Fíoð}ä°ee \ No newline at end of file +d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1731931728e4:infod5:filesld6:lengthi2604e4:pathl4:Dir15:Dir1111:test11.zipxeed6:lengthi2604e4:pathl4:Dir110:sample.txteed6:lengthi2604e4:pathl4:Dir210:test2.zipxeed6:lengthi2604e4:pathl68:Speak.No.Evil.2024.2160p.MA.WEB-DL.DDP5.1.Atmos.H.265-HHWEB.mkv.zipxeed6:lengthi2604e4:pathl9:test.zipxeee4:name59:Speak.No.Evil.2024.2160p.MA.WEB-DL.DDP5.1.Atmos.H.265-HHWEB12:piece lengthi262144e6:pieces20:w¤ŸÌ³RÇþ'6Fíoð}ä°7:privatei1eee \ No newline at end of file diff --git a/code/test/data/nginx/sonarr_bad_nested.torrent b/code/test/data/nginx/sonarr_bad_nested.torrent index 654d40c2..778aac72 100644 --- a/code/test/data/nginx/sonarr_bad_nested.torrent +++ b/code/test/data/nginx/sonarr_bad_nested.torrent @@ -1 +1 @@ -d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1731931738e4:infod5:filesld6:lengthi2604e4:pathl89:Agatha.All.Along.S01E01.Seekest.Thou.the.Road.2160p.APPS.WEB-DL.DDP5.1.Atmos. - Copy.zipxeed6:lengthi2604e4:pathl4:Dir15:Dir1111:test11.zipxeed6:lengthi2604e4:pathl4:Dir110:sample.txteed6:lengthi2604e4:pathl4:Dir210:test2.zipxeed6:lengthi2604e4:pathl9:test.zipxeee4:name88:Agatha.All.Along.S01E01.Seekest.Thou.the.Road.2160p.APPS.WEB-DL.DDP5.1.Atmos.H.265-VARYG12:piece lengthi262144e6:pieces20:w¤ŸÌ³RÇþ'6Fíoð}ä°ee \ No newline at end of file +d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1731931738e4:infod5:filesld6:lengthi2604e4:pathl89:Agatha.All.Along.S01E01.Seekest.Thou.the.Road.2160p.APPS.WEB-DL.DDP5.1.Atmos. - Copy.zipxeed6:lengthi2604e4:pathl4:Dir15:Dir1111:test11.zipxeed6:lengthi2604e4:pathl4:Dir110:sample.txteed6:lengthi2604e4:pathl4:Dir210:test2.zipxeed6:lengthi2604e4:pathl9:test.zipxeee4:name88:Agatha.All.Along.S01E01.Seekest.Thou.the.Road.2160p.APPS.WEB-DL.DDP5.1.Atmos.H.265-VARYG12:piece lengthi262144e6:pieces20:w¤ŸÌ³RÇþ'6Fíoð}ä°7:privatei1eee \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/downloads/Coldplay-Everyday.Life-2019-C4/coldplay-everyday_life.zipx b/code/test/data/qbittorrent-bad/downloads/Coldplay-Everyday.Life-2019-C4/coldplay-everyday_life.zipx new file mode 100644 index 00000000..946a6916 --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/Coldplay-Everyday.Life-2019-C4/coldplay-everyday_life.zipx @@ -0,0 +1 @@ +testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/downloads/Coldplay-Everyday.Life-2019-C4/coldplay-everyday_life2.mp3 b/code/test/data/qbittorrent-bad/downloads/Coldplay-Everyday.Life-2019-C4/coldplay-everyday_life2.mp3 new file mode 100644 index 00000000..946a6916 --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/Coldplay-Everyday.Life-2019-C4/coldplay-everyday_life2.mp3 @@ -0,0 +1 @@ +testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/downloads/Coldplay-Ghost Stories-(Deluxe Edition)-2014-MTD/001-coldplay-always_in_my_head.zipx b/code/test/data/qbittorrent-bad/downloads/Coldplay-Ghost Stories-(Deluxe Edition)-2014-MTD/001-coldplay-always_in_my_head.zipx new file mode 100644 index 00000000..946a6916 --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/Coldplay-Ghost Stories-(Deluxe Edition)-2014-MTD/001-coldplay-always_in_my_head.zipx @@ -0,0 +1 @@ +testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/downloads/lidarr_bad_pack.torrent b/code/test/data/qbittorrent-bad/downloads/lidarr_bad_pack.torrent new file mode 100644 index 00000000..91d01ea2 --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/lidarr_bad_pack.torrent @@ -0,0 +1 @@ +d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1736513625e4:infod5:filesld6:lengthi640e4:pathl27:coldplay-everyday_life.zipxeed6:lengthi640e4:pathl27:coldplay-everyday_life2.mp3eee4:name31:Coldplay-Everyday.Life-2019-C4/12:piece lengthi262144e6:pieces20:#Ú¯§Ñ4OduÎoÎÛ€¹Þ[=~ee \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/downloads/lidarr_bad_single.torrent b/code/test/data/qbittorrent-bad/downloads/lidarr_bad_single.torrent new file mode 100644 index 00000000..c829a7f3 --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/lidarr_bad_single.torrent @@ -0,0 +1 @@ +d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1736513638e4:infod5:filesld6:lengthi640e4:pathl35:001-coldplay-always_in_my_head.zipxeee4:name49:Coldplay-Ghost Stories-(Deluxe Edition)-2014-MTD/12:piece lengthi262144e6:pieces20:·iV9qæ “Ý)-xÖ©'¦Èò«ee \ No newline at end of file diff --git a/code/test/data/qbittorrent/config/qBittorrent/categories.json b/code/test/data/qbittorrent/config/qBittorrent/categories.json index 19e2935e..64666a90 100644 --- a/code/test/data/qbittorrent/config/qBittorrent/categories.json +++ b/code/test/data/qbittorrent/config/qBittorrent/categories.json @@ -1,4 +1,7 @@ { + "lidarr": { + "save_path": "" + }, "radarr": { "save_path": "" }, diff --git a/code/test/data/radarr/config/config.xml b/code/test/data/radarr/config/config.xml new file mode 100644 index 00000000..0bf6aa0c --- /dev/null +++ b/code/test/data/radarr/config/config.xml @@ -0,0 +1,17 @@ + + * + 7878 + 9898 + False + True + 8b7454f668e54c5b8f44f56f93969761 + Forms + Enabled + master + debug + + + + Radarr + Docker + \ No newline at end of file diff --git a/code/test/data/readarr/config/config.xml b/code/test/data/readarr/config/config.xml new file mode 100644 index 00000000..103b7ba7 --- /dev/null +++ b/code/test/data/readarr/config/config.xml @@ -0,0 +1,17 @@ + + * + 8787 + 6868 + False + True + 53388ac405894ef2ac6b82f907f481aa + Forms + Enabled + develop + debug + + + + Readarr + Docker + \ No newline at end of file diff --git a/code/test/data/sonarr/config/config.xml b/code/test/data/sonarr/config/config.xml new file mode 100644 index 00000000..53d50b5c --- /dev/null +++ b/code/test/data/sonarr/config/config.xml @@ -0,0 +1,17 @@ + + * + 8989 + 9898 + False + True + 425d1e713f0c405cbbf359ac0502c1f4 + Forms + Enabled + main + debug + + + + Sonarr + Docker + \ No newline at end of file diff --git a/code/test/docker-compose.yml b/code/test/docker-compose.yml index 6e581b9c..17c7861e 100644 --- a/code/test/docker-compose.yml +++ b/code/test/docker-compose.yml @@ -8,10 +8,10 @@ # ctorrent -t -u "http://tracker:6969/announce" -s example.torrent file_name # api keys -# sonarr: 96736c3eb3144936b8f1d62d27be8cee -# radarr: 705b553732ab4167ab23909305d60600 -# lidarr: 4bd467b8702a4ecf94f737922dac6481 -# readarr: 51c053efbea34bad90120d5c2237aa85 +# sonarr: 425d1e713f0c405cbbf359ac0502c1f4 +# radarr: 8b7454f668e54c5b8f44f56f93969761 +# lidarr: 7f677cfdc074414397af53dd633860c5 +# readarr: 53388ac405894ef2ac6b82f907f481aa services: qbittorrent: @@ -192,32 +192,39 @@ services: - CONTENTBLOCKER__ENABLED=true - CONTENTBLOCKER__IGNORE_PRIVATE=true - - CONTENTBLOCKER__BLACKLIST__ENABLED=true - - CONTENTBLOCKER__BLACKLIST__PATH=https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist - # OR - # - CONTENTBLOCKER__WHITELIST__ENABLED=true - # - CONTENTBLOCKER__WHITELIST__PATH=https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist - DOWNLOAD_CLIENT=qbittorrent - QBITTORRENT__URL=http://qbittorrent:8080 - QBITTORRENT__USERNAME=test - QBITTORRENT__PASSWORD=testing # OR + # - DOWNLOAD_CLIENT=deluge # - DELUGE__URL=http://localhost:8112 # - DELUGE__PASSWORD=testing # OR + # - DOWNLOAD_CLIENT=transmission # - TRANSMISSION__URL=http://localhost:9091 # - TRANSMISSION__USERNAME=test # - TRANSMISSION__PASSWORD=testing - SONARR__ENABLED=true - SONARR__SEARCHTYPE=Episode + - SONARR__BLOCK__TYPE=blacklist + - SONARR__BLOCK__PATH=https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist - SONARR__INSTANCES__0__URL=http://sonarr:8989 - - SONARR__INSTANCES__0__APIKEY=96736c3eb3144936b8f1d62d27be8cee + - SONARR__INSTANCES__0__APIKEY=425d1e713f0c405cbbf359ac0502c1f4 - RADARR__ENABLED=true + - RADARR__BLOCK__TYPE=blacklist + - RADARR__BLOCK__PATH=https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist - RADARR__INSTANCES__0__URL=http://radarr:7878 - - RADARR__INSTANCES__0__APIKEY=705b553732ab4167ab23909305d60600 + - RADARR__INSTANCES__0__APIKEY=8b7454f668e54c5b8f44f56f93969761 + + - LIDARR__ENABLED=true + - LIDARR__BLOCK__TYPE=blacklist + - LIDARR__BLOCK__PATH=https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist # TODO + - LIDARR__INSTANCES__0__URL=http://lidarr:8686 + - LIDARR__INSTANCES__0__APIKEY=7f677cfdc074414397af53dd633860c5 volumes: - ./data/cleanuperr/logs:/var/logs restart: unless-stopped