Compare commits

...

14 Commits

Author SHA1 Message Date
Flaminel
89a6eaf0ce Disable cleanup on torrent items if download client is not configured (#85) 2025-03-10 00:13:40 +02:00
Flaminel
027c4a0f4d Add option to ignore specific downloads (#79) 2025-03-09 23:38:27 +02:00
Flaminel
81990c6768 fixed missing README link 2025-03-03 22:37:22 +02:00
Flaminel
ba02aa0e49 Fix notifications failing when poster image is not set (#78) 2025-03-02 22:48:21 +02:00
Flaminel
5adbdbd920 Fix weird time zone display name on startup (#70) 2025-02-25 21:32:19 +02:00
Flaminel
b3b211d956 Add configurable time zone (#69) 2025-02-24 23:21:44 +02:00
Flaminel
279bd6d82d Fix Deluge timeout not being configurable (#68) 2025-02-24 18:32:44 +02:00
Flaminel
5dced28228 fixed errors on download cleaner when download client is none (#67) 2025-02-24 12:43:06 +02:00
Flaminel
51bdaf64e4 Fix interceptor memory leaks (#66) 2025-02-23 17:50:08 +02:00
Flaminel
9c8e0ebedc updated README 2025-02-18 13:16:49 +02:00
Flaminel
e1bea8a8c8 updated README 2025-02-17 23:59:36 +02:00
Marius Nechifor
a6d3820104 Improve Transmission category detection (#62) 2025-02-17 02:48:27 +02:00
Flaminel
36c793a5fb updated chart values 2025-02-16 17:43:35 +02:00
Flaminel
aade8a91c3 fixed dummy download service 2025-02-16 12:17:31 +02:00
49 changed files with 914 additions and 371 deletions

View File

@@ -8,12 +8,42 @@ cleanuperr is a tool for automating the cleanup of unwanted or blocked files in
cleanuperr was created primarily to address malicious files, such as `*.lnk` or `*.zipx`, that were getting stuck in Sonarr/Radarr and required manual intervention. Some of the reddit posts that made cleanuperr come to life can be found [here](https://www.reddit.com/r/sonarr/comments/1gqnx16/psa_sonarr_downloaded_a_virus/), [here](https://www.reddit.com/r/sonarr/comments/1gqwklr/sonar_downloaded_a_mkv_file_which_looked_like_a/), [here](https://www.reddit.com/r/sonarr/comments/1gpw2wa/downloaded_waiting_to_import/) and [here](https://www.reddit.com/r/sonarr/comments/1gpi344/downloads_not_importing_no_files_found/).
The tool supports both qBittorrent's built-in exclusion features and its own blocklist-based system. Binaries for all platforms are provided, along with Docker images for easy deployment.
> [!IMPORTANT]
> **Features:**
> - Strike system to mark stalled or downloads stuck in metadata downloading.
> - Remove and block downloads that reached a maximum number of strikes.
> - Remove downloads blocked by qBittorrent or by cleanuperr's **content blocker**.
> - Trigger a search for downloads removed from the *arrs.
> - Clean up downloads that have been seeding for a certain amount of time.
> - Notify on strike or download removal.
#
cleanuperr supports both qBittorrent's built-in exclusion features and its own blocklist-based system. Binaries for all platforms are provided, along with Docker images for easy deployment.
> [!WARNING]
> This tool is actively developed and still a work in progress, so using the `latest` Docker tag may result in breaking changes. Join the Discord server if you want to reach out to me quickly (or just stay updated on new releases) so we can squash those pesky bugs together:
>
> https://discord.gg/sWggpnmGNY
## Table of contents:
- [Naming choice](README.md#naming-choice)
- [Quick Start](README.md#quick-start)
- [How it works](README.md#how-it-works)
- [Setup](README.md#setup)
- [Usage](README.md#usage)
- [Docker Compose](README.md#docker-compose-yaml)
- [Environment Variables](README.md#environment-variables)
- [Binaries](README.md#binaries-if-youre-not-using-docker)
- [Credits](README.md#credits)
## Naming choice
I've had people asking why it's `cleanuperr` and not `cleanuparr` and that I should change it. This name was intentional.
I've seen a few discussions on this type of naming and I've decided that I didn't deserve the `arr` moniker since `cleanuperr` is not a fork of `NZB.Drone` and it does not have any affiliation with the arrs. I still wanted to keep the naming style close enough though, to suggest a correlation between them.
## Quick Start
> [!NOTE]
> ### Quick Start
>
> 1. **Docker (Recommended)**
> Pull the Docker image from `ghcr.io/flmorg/cleanuperr:latest`.
@@ -27,10 +57,6 @@ The tool supports both qBittorrent's built-in exclusion features and its own blo
> [!TIP]
> Refer to the [Environment variables](#Environment-variables) section for detailed configuration instructions and the [Setup](#Setup) section for an in-depth explanation of the cleanup process.
## Key features
- Marks unwanted files as skip/unwanted in the download client.
- Automatically strikes stalled or stuck downloads.
- Removes and blocks downloads that reached the maximum number of strikes or are marked as unwanted by the download client or by cleanuperr and triggers a search for removed downloads.
> [!IMPORTANT]
> Only the **latest versions** of the following apps are supported, or earlier versions that have the same API as the latest version:
@@ -41,10 +67,6 @@ The tool supports both qBittorrent's built-in exclusion features and its own blo
> - Radarr
> - Lidarr
This tool is actively developed and still a work in progress, so using the `latest` Docker tag may result in breaking changes. Join the Discord server if you want to reach out to me quickly (or just stay updated on new releases) so we can squash those pesky bugs together:
> https://discord.gg/sWggpnmGNY
# How it works
1. **Content blocker** will:
@@ -90,7 +112,7 @@ This tool is actively developed and still a work in progress, so using the `late
## Using cleanuperr's blocklist (works with all supported download clients)
1. Set both `QUEUECLEANER__ENABLED` and `CONTENTBLOCKER__ENABLED` to `true` in your environment variables.
2. Configure and enable either a **blacklist** or a **whitelist** as described in the [Arr variables](#Arr-variables) section.
2. Configure and enable either a **blacklist** or a **whitelist** as described in the [Arr variables](variables.md#Arr-settings) section.
3. Once configured, cleanuperr will perform the following tasks:
- Execute the **content blocker** job, as explained in the [How it works](#how-it-works) section.
- Execute the **queue cleaner** job, as explained in the [How it works](#how-it-works) section.
@@ -102,7 +124,8 @@ This tool is actively developed and still a work in progress, so using the `late
3. Optionally set failed import message patterns to ignore using `QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__<NUMBER>`.
4. Set `DOWNLOAD_CLIENT` to `none`.
**No other action involving a download client would work (e.g. content blocking, removing stalled downloads, excluding private trackers).**
> [!WARNING]
> When `DOWNLOAD_CLIENT=none`, no other action involving a download client would work (e.g. content blocking, removing stalled downloads, excluding private trackers).
## Usage
@@ -116,7 +139,9 @@ services:
restart: unless-stopped
volumes:
- ./cleanuperr/logs:/var/logs
- ./cleanuperr/ignored.txt:/ignored.txt
environment:
- TZ=America/New_York
- DRY_RUN=false
- LOGGING__LOGLEVEL=Information
@@ -129,6 +154,7 @@ services:
- TRIGGERS__DOWNLOADCLEANER=0 0 * * * ?
- QUEUECLEANER__ENABLED=true
- QUEUECLEANER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- QUEUECLEANER__RUNSEQUENTIALLY=true
- QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE=false
@@ -141,10 +167,12 @@ services:
- QUEUECLEANER__STALLED_DELETE_PRIVATE=false
- CONTENTBLOCKER__ENABLED=true
- CONTENTBLOCKER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- CONTENTBLOCKER__IGNORE_PRIVATE=false
- CONTENTBLOCKER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__ENABLED=true
- DOWNLOADCLEANER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- DOWNLOADCLEANER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__CATEGORIES__0__NAME=tv-sonarr
- DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO=-1

View File

@@ -10,6 +10,9 @@ deployment:
repository: ghcr.io/flmorg/cleanuperr
tag: latest
env:
- name: DRY_RUN
value: "false"
- name: LOGGING__LOGLEVEL
value: Debug
- name: LOGGING__FILE__ENABLED
@@ -18,6 +21,7 @@ deployment:
value: /var/logs
- name: LOGGING__ENHANCED
value: "true"
- name: TRIGGERS__QUEUECLEANER
value: 0 0/5 * * * ?
- name: TRIGGERS__CONTENTBLOCKER
@@ -47,6 +51,9 @@ deployment:
- name: CONTENTBLOCKER__DELETE_PRIVATE
value: "false"
- name: DOWNLOADCLEANER__ENABLED
value: "false"
- name: DOWNLOAD_CLIENT
value: qbittorrent
- name: QBITTORRENT__URL
@@ -75,6 +82,17 @@ deployment:
value: http://service.radarr-low-res.svc.cluster.local
- name: RADARR__INSTANCES__1__URL
value: http://service.radarr-high-res.svc.cluster.local
- name: NOTIFIARR__ON_IMPORT_FAILED_STRIKE
value: "true"
- name: NOTIFIARR__ON_STALLED_STRIKE
value: "true"
- name: NOTIFIARR__ON_QUEUE_ITEM_DELETED
value: "true"
- name: NOTIFIARR__ON_DOWNLOAD_CLEANED
value: "true"
- name: NOTIFIARR__CHANNEL_ID
value: "1340708411259748413"
envFromSecret:
- secretName: qbit-auth
envs:
@@ -94,6 +112,10 @@ deployment:
key: RDRL_API_KEY
- name: RADARR__INSTANCES__1__APIKEY
key: RDRH_API_KEY
- secretName: notifiarr-auth
envs:
- name: NOTIFIARR__API_KEY
key: API_KEY
resources:
requests:
cpu: 0m
@@ -133,4 +155,8 @@ vaultSecrets:
path: secrets/sonarr
templates:
SNRL_API_KEY: "{% .Secrets.low_api_key %}"
SNRH_API_KEY: "{% .Secrets.high_api_key %}"
SNRH_API_KEY: "{% .Secrets.high_api_key %}"
- name: notifiarr-auth
path: secrets/notifiarr
templates:
API_KEY: "{% .Secrets.passthrough_api_key %}"

View File

@@ -2,7 +2,7 @@
namespace Common.Configuration.ContentBlocker;
public sealed record ContentBlockerConfig : IJobConfig
public sealed record ContentBlockerConfig : IJobConfig, IIgnoredDownloadsConfig
{
public const string SectionName = "ContentBlocker";
@@ -13,6 +13,9 @@ public sealed record ContentBlockerConfig : IJobConfig
[ConfigurationKeyName("DELETE_PRIVATE")]
public bool DeletePrivate { get; init; }
[ConfigurationKeyName("IGNORED_DOWNLOADS_PATH")]
public string? IgnoredDownloadsPath { get; init; }
public void Validate()
{

View File

@@ -3,7 +3,7 @@ using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadCleaner;
public sealed record DownloadCleanerConfig : IJobConfig
public sealed record DownloadCleanerConfig : IJobConfig, IIgnoredDownloadsConfig
{
public const string SectionName = "DownloadCleaner";
@@ -12,7 +12,10 @@ public sealed record DownloadCleanerConfig : IJobConfig
public List<Category>? Categories { get; init; }
[ConfigurationKeyName("DELETE_PRIVATE")]
public bool DeletePrivate { get; set; }
public bool DeletePrivate { get; init; }
[ConfigurationKeyName("IGNORED_DOWNLOADS_PATH")]
public string? IgnoredDownloadsPath { get; init; }
public void Validate()
{

View File

@@ -0,0 +1,6 @@
namespace Common.Configuration;
public interface IIgnoredDownloadsConfig
{
string? IgnoredDownloadsPath { get; }
}

View File

@@ -2,7 +2,7 @@
namespace Common.Configuration.QueueCleaner;
public sealed record QueueCleanerConfig : IJobConfig
public sealed record QueueCleanerConfig : IJobConfig, IIgnoredDownloadsConfig
{
public const string SectionName = "QueueCleaner";
@@ -10,6 +10,9 @@ public sealed record QueueCleanerConfig : IJobConfig
public required bool RunSequentially { get; init; }
[ConfigurationKeyName("IGNORED_DOWNLOADS_PATH")]
public string? IgnoredDownloadsPath { get; init; }
[ConfigurationKeyName("IMPORT_FAILED_MAX_STRIKES")]
public ushort ImportFailedMaxStrikes { get; init; }

View File

@@ -23,4 +23,11 @@ public sealed record TorrentStatus
public long SeedingTime { get; init; }
public float Ratio { get; init; }
public required IReadOnlyList<Tracker> Trackers { get; init; }
}
public sealed record Tracker
{
public required Uri Url { get; init; }
}

View File

@@ -1,8 +1,6 @@
using System.Net;
using Castle.DynamicProxy;
using Common.Configuration.General;
using Common.Helpers;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.DownloadClient.Deluge;
using Infrastructure.Verticals.Notifications.Consumers;
using Infrastructure.Verticals.Notifications.Models;
@@ -42,8 +40,7 @@ public static class MainDI
e.PrefetchCount = 1;
});
});
})
.AddDryRunInterceptor();
});
private static IServiceCollection AddHttpClients(this IServiceCollection services, IConfiguration configuration)
{
@@ -65,7 +62,7 @@ public static class MainDI
services
.AddHttpClient(nameof(DelugeService), x =>
{
x.Timeout = TimeSpan.FromSeconds(5);
x.Timeout = TimeSpan.FromSeconds(config.Timeout);
})
.ConfigurePrimaryHttpMessageHandler(_ =>
{
@@ -91,31 +88,4 @@ public static class MainDI
.OrResult(response => !response.IsSuccessStatusCode && response.StatusCode != HttpStatusCode.Unauthorized)
.WaitAndRetryAsync(config.MaxRetries, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)))
);
private static IServiceCollection AddDryRunInterceptor(this IServiceCollection services)
{
services
.Where(s => s.ServiceType != typeof(IDryRunService) && typeof(IDryRunService).IsAssignableFrom(s.ServiceType))
.ToList()
.ForEach(service =>
{
services.Decorate(service.ServiceType, (target, svc) =>
{
ProxyGenerator proxyGenerator = new();
DryRunAsyncInterceptor interceptor = svc.GetRequiredService<DryRunAsyncInterceptor>();
object implementation = proxyGenerator.CreateClassProxyWithTarget(
service.ServiceType,
target,
interceptor
);
((IInterceptedService)target).Proxy = implementation;
return implementation;
});
});
return services;
}
}

View File

@@ -10,7 +10,7 @@ public static class NotificationsDI
.Configure<NotifiarrConfig>(configuration.GetSection(NotifiarrConfig.SectionName))
.AddTransient<INotifiarrProxy, NotifiarrProxy>()
.AddTransient<INotificationProvider, NotifiarrProvider>()
.AddTransient<NotificationPublisher>()
.AddTransient<INotificationPublisher, NotificationPublisher>()
.AddTransient<INotificationFactory, NotificationFactory>()
.AddTransient<NotificationService>();
}

View File

@@ -1,4 +1,8 @@
using Infrastructure.Interceptors;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Infrastructure.Interceptors;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.DownloadCleaner;
@@ -15,7 +19,7 @@ public static class ServicesDI
{
public static IServiceCollection AddServices(this IServiceCollection services) =>
services
.AddTransient<DryRunAsyncInterceptor>()
.AddTransient<IDryRunInterceptor, DryRunInterceptor>()
.AddTransient<SonarrClient>()
.AddTransient<RadarrClient>()
.AddTransient<LidarrClient>()
@@ -30,5 +34,8 @@ public static class ServicesDI
.AddTransient<ArrQueueIterator>()
.AddTransient<DownloadServiceFactory>()
.AddSingleton<BlocklistProvider>()
.AddSingleton<IStriker, Striker>();
.AddSingleton<IStriker, Striker>()
.AddSingleton<IgnoredDownloadsProvider<QueueCleanerConfig>>()
.AddSingleton<IgnoredDownloadsProvider<ContentBlockerConfig>>()
.AddSingleton<IgnoredDownloadsProvider<DownloadCleanerConfig>>();
}

View File

@@ -0,0 +1,23 @@
using System.Reflection;
namespace Executable;
public static class HostExtensions
{
public static IHost Init(this IHost host)
{
ILogger<Program> logger = host.Services.GetRequiredService<ILogger<Program>>();
Version? version = Assembly.GetExecutingAssembly().GetName().Version;
logger.LogInformation(
version is null
? "cleanuperr version not detected"
: $"cleanuperr v{version.Major}.{version.Minor}.{version.Build}"
);
logger.LogInformation("timezone: {tz}", TimeZoneInfo.Local.DisplayName);
return host;
}
}

View File

@@ -1,4 +1,4 @@
using System.Reflection;
using Executable;
using Executable.DependencyInjection;
var builder = Host.CreateApplicationBuilder(args);
@@ -7,15 +7,6 @@ builder.Services.AddInfrastructure(builder.Configuration);
builder.Logging.AddLogging(builder.Configuration);
var host = builder.Build();
var logger = host.Services.GetRequiredService<ILogger<Program>>();
var version = Assembly.GetExecutingAssembly().GetName().Version;
logger.LogInformation(
version is null
? "cleanuperr version not detected"
: $"cleanuperr v{version.Major}.{version.Minor}.{version.Build}"
);
host.Init();
host.Run();

View File

@@ -18,11 +18,13 @@
"ContentBlocker": {
"Enabled": true,
"IGNORE_PRIVATE": true,
"DELETE_PRIVATE": false
"DELETE_PRIVATE": false,
"IGNORED_DOWNLOADS_PATH": "../test/data/cleanuperr/ignored_downloads"
},
"QueueCleaner": {
"Enabled": true,
"RunSequentially": true,
"IGNORED_DOWNLOADS_PATH": "../test/data/cleanuperr/ignored_downloads",
"IMPORT_FAILED_MAX_STRIKES": 5,
"IMPORT_FAILED_IGNORE_PRIVATE": true,
"IMPORT_FAILED_DELETE_PRIVATE": false,
@@ -44,7 +46,8 @@
"MIN_SEED_TIME": 0,
"MAX_SEED_TIME": -1
}
]
],
"IGNORED_DOWNLOADS_PATH": "../test/data/cleanuperr/ignored_downloads"
},
"DOWNLOAD_CLIENT": "qbittorrent",
"qBittorrent": {

View File

@@ -17,11 +17,13 @@
},
"ContentBlocker": {
"Enabled": false,
"IGNORE_PRIVATE": false
"IGNORE_PRIVATE": false,
"IGNORED_DOWNLOADS_PATH": ""
},
"QueueCleaner": {
"Enabled": true,
"RunSequentially": true,
"IGNORED_DOWNLOADS_PATH": "",
"IMPORT_FAILED_MAX_STRIKES": 0,
"IMPORT_FAILED_IGNORE_PRIVATE": false,
"IMPORT_FAILED_DELETE_PRIVATE": false,
@@ -34,7 +36,8 @@
"DownloadCleaner": {
"Enabled": false,
"DELETE_PRIVATE": false,
"CATEGORIES": []
"CATEGORIES": [],
"IGNORED_DOWNLOADS_PATH": ""
},
"DOWNLOAD_CLIENT": "none",
"qBittorrent": {

View File

@@ -1,6 +1,7 @@
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.DownloadClient;
using Infrastructure.Verticals.ItemStriker;
@@ -53,7 +54,8 @@ public class DownloadServiceFixture : IDisposable
downloadCleanerOptions.Value.Returns(new DownloadCleanerConfig());
var filenameEvaluator = Substitute.For<IFilenameEvaluator>();
var notifier = Substitute.For<NotificationPublisher>();
var notifier = Substitute.For<INotificationPublisher>();
var dryRunInterceptor = Substitute.For<IDryRunInterceptor>();
return new TestDownloadService(
Logger,
@@ -63,7 +65,8 @@ public class DownloadServiceFixture : IDisposable
Cache,
filenameEvaluator,
Striker,
notifier
notifier,
dryRunInterceptor
);
}

View File

@@ -3,6 +3,7 @@ using System.Text.RegularExpressions;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.DownloadClient;
using Infrastructure.Verticals.ItemStriker;
@@ -23,20 +24,24 @@ public class TestDownloadService : DownloadService
IMemoryCache cache,
IFilenameEvaluator filenameEvaluator,
IStriker striker,
NotificationPublisher notifier)
: base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig,
cache, filenameEvaluator, striker, notifier)
INotificationPublisher notifier,
IDryRunInterceptor dryRunInterceptor
) : base(
logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache,
filenameEvaluator, striker, notifier, dryRunInterceptor
)
{
}
public override void Dispose() { }
public override Task LoginAsync() => Task.CompletedTask;
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash) => Task.FromResult(new StalledResult());
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType,
ConcurrentBag<string> patterns, ConcurrentBag<Regex> regexes) => Task.FromResult(new BlockFilesResult());
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads) => Task.FromResult(new StalledResult());
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType,
ConcurrentBag<string> patterns, ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads) => Task.FromResult(new BlockFilesResult());
public override Task DeleteDownload(string hash) => Task.CompletedTask;
public override Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories) => Task.FromResult<List<object>?>(null);
public override Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes) => Task.CompletedTask;
public override Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads) => Task.CompletedTask;
// Expose protected methods for testing
public new void ResetStrikesOnProgress(string hash, long downloaded) => base.ResetStrikesOnProgress(hash, downloaded);

View File

@@ -0,0 +1,29 @@
using Domain.Models.Deluge.Response;
namespace Infrastructure.Extensions;
public static class DelugeExtensions
{
public static bool ShouldIgnore(this TorrentStatus download, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (download.Hash?.Equals(value, StringComparison.InvariantCultureIgnoreCase) ?? false)
{
return true;
}
if (download.Label?.Equals(value, StringComparison.InvariantCultureIgnoreCase) ?? false)
{
return true;
}
if (download.Trackers.Any(x => x.Url.Host.EndsWith(value, StringComparison.InvariantCultureIgnoreCase)))
{
return true;
}
}
return false;
}
}

View File

@@ -0,0 +1,42 @@
using QBittorrent.Client;
namespace Infrastructure.Extensions;
public static class QBitExtensions
{
public static bool ShouldIgnore(this TorrentInfo download, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (download.Hash.Equals(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
if (download.Category.Equals(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
if (download.Tags.Contains(value, StringComparer.InvariantCultureIgnoreCase))
{
return true;
}
}
return false;
}
public static bool ShouldIgnore(this TorrentTracker tracker, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (tracker.Url.Host.EndsWith(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
}
return false;
}
}

View File

@@ -0,0 +1,42 @@
using Transmission.API.RPC.Entity;
namespace Infrastructure.Extensions;
public static class TransmissionExtensions
{
public static bool ShouldIgnore(this TorrentInfo download, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (download.HashString?.Equals(value, StringComparison.InvariantCultureIgnoreCase) ?? false)
{
return true;
}
if (download.GetCategory().Equals(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
bool? hasIgnoredTracker = download.Trackers?
.Any(x => new Uri(x.Announce!).Host.EndsWith(value, StringComparison.InvariantCultureIgnoreCase));
if (hasIgnoredTracker is true)
{
return true;
}
}
return false;
}
public static string GetCategory(this TorrentInfo download)
{
if (string.IsNullOrEmpty(download.DownloadDir))
{
return string.Empty;
}
return Path.GetFileName(Path.TrimEndingDirectorySeparator(download.DownloadDir));
}
}

View File

@@ -11,4 +11,6 @@ public static class CacheKeys
public static string BlocklistRegexes(InstanceType instanceType) => $"{instanceType.ToString()}_regexes";
public static string Item(string hash) => $"item_{hash}";
public static string IgnoredDownloads(string name) => $"{name}_ignored";
}

View File

@@ -12,7 +12,6 @@
</ItemGroup>
<ItemGroup>
<PackageReference Include="Castle.Core.AsyncInterceptor" Version="2.1.0" />
<PackageReference Include="FLM.QBittorrent" Version="1.0.0" />
<PackageReference Include="FLM.Transmission" Version="1.0.2" />
<PackageReference Include="Mapster" Version="7.4.0" />

View File

@@ -1,5 +1,4 @@
using System.Reflection;
using Castle.DynamicProxy;
using Common.Attributes;
using Common.Configuration.General;
using Microsoft.Extensions.Logging;
@@ -7,41 +6,70 @@ using Microsoft.Extensions.Options;
namespace Infrastructure.Interceptors;
public class DryRunAsyncInterceptor : AsyncInterceptorBase
public class DryRunInterceptor : IDryRunInterceptor
{
private readonly ILogger<DryRunAsyncInterceptor> _logger;
private readonly ILogger<DryRunInterceptor> _logger;
private readonly DryRunConfig _config;
public DryRunAsyncInterceptor(ILogger<DryRunAsyncInterceptor> logger, IOptions<DryRunConfig> config)
public DryRunInterceptor(ILogger<DryRunInterceptor> logger, IOptions<DryRunConfig> config)
{
_logger = logger;
_config = config.Value;
}
protected override async Task InterceptAsync(IInvocation invocation, IInvocationProceedInfo proceedInfo, Func<IInvocation, IInvocationProceedInfo, Task> proceed)
public void Intercept(Action action)
{
MethodInfo? method = invocation.MethodInvocationTarget ?? invocation.Method;
if (IsDryRun(method))
MethodInfo methodInfo = action.Method;
if (IsDryRun(methodInfo))
{
_logger.LogInformation("[DRY RUN] skipping method: {name}", method.Name);
_logger.LogInformation("[DRY RUN] skipping method: {name}", methodInfo.Name);
return;
}
await proceed(invocation, proceedInfo);
action();
}
protected override async Task<TResult> InterceptAsync<TResult>(IInvocation invocation, IInvocationProceedInfo proceedInfo, Func<IInvocation, IInvocationProceedInfo, Task<TResult>> proceed)
public Task InterceptAsync(Delegate action, params object[] parameters)
{
MethodInfo? method = invocation.MethodInvocationTarget ?? invocation.Method;
if (IsDryRun(method))
MethodInfo methodInfo = action.Method;
if (IsDryRun(methodInfo))
{
_logger.LogInformation("[DRY RUN] skipping method: {name}", method.Name);
return default!;
_logger.LogInformation("[DRY RUN] skipping method: {name}", methodInfo.Name);
return Task.CompletedTask;
}
return await proceed(invocation, proceedInfo);
}
object? result = action.DynamicInvoke(parameters);
if (result is Task task)
{
return task;
}
return Task.CompletedTask;
}
public Task<T?> InterceptAsync<T>(Delegate action, params object[] parameters)
{
MethodInfo methodInfo = action.Method;
if (IsDryRun(methodInfo))
{
_logger.LogInformation("[DRY RUN] skipping method: {name}", methodInfo.Name);
return Task.FromResult(default(T));
}
object? result = action.DynamicInvoke(parameters);
if (result is Task<T?> task)
{
return task;
}
return Task.FromResult(default(T));
}
private bool IsDryRun(MethodInfo method)
{
return method.GetCustomAttributes(typeof(DryRunSafeguardAttribute), true).Any() && _config.IsDryRun;

View File

@@ -0,0 +1,10 @@
namespace Infrastructure.Interceptors;
public interface IDryRunInterceptor
{
void Intercept(Action action);
Task InterceptAsync(Delegate action, params object[] parameters);
Task<T?> InterceptAsync<T>(Delegate action, params object[] parameters);
}

View File

@@ -1,5 +0,0 @@
namespace Infrastructure.Interceptors;
public interface IDryRunService : IInterceptedService
{
}

View File

@@ -1,6 +0,0 @@
namespace Infrastructure.Interceptors;
public interface IInterceptedService
{
public object Proxy { get; set; }
}

View File

@@ -1,21 +0,0 @@
namespace Infrastructure.Interceptors;
public class InterceptedService : IInterceptedService
{
private object? _proxy;
public object Proxy
{
get
{
if (_proxy is null)
{
throw new Exception("Proxy is not set");
}
return _proxy;
}
set => _proxy = value;
}
}

View File

@@ -0,0 +1,82 @@
using Common.Configuration;
using Infrastructure.Helpers;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
namespace Infrastructure.Providers;
public sealed class IgnoredDownloadsProvider<T>
where T : IIgnoredDownloadsConfig
{
private readonly ILogger<IgnoredDownloadsProvider<T>> _logger;
private IIgnoredDownloadsConfig _config;
private readonly IMemoryCache _cache;
private DateTime _lastModified = DateTime.MinValue;
public IgnoredDownloadsProvider(ILogger<IgnoredDownloadsProvider<T>> logger, IOptionsMonitor<T> config, IMemoryCache cache)
{
_config = config.CurrentValue;
config.OnChange((newValue) => _config = newValue);
_logger = logger;
_cache = cache;
if (string.IsNullOrEmpty(_config.IgnoredDownloadsPath))
{
return;
}
if (!File.Exists(_config.IgnoredDownloadsPath))
{
throw new FileNotFoundException("file not found", _config.IgnoredDownloadsPath);
}
}
public async Task<IReadOnlyList<string>> GetIgnoredDownloads()
{
if (string.IsNullOrEmpty(_config.IgnoredDownloadsPath))
{
return Array.Empty<string>();
}
FileInfo fileInfo = new(_config.IgnoredDownloadsPath);
if (fileInfo.LastWriteTime > _lastModified ||
!_cache.TryGetValue(CacheKeys.IgnoredDownloads(typeof(T).Name), out IReadOnlyList<string>? ignoredDownloads) ||
ignoredDownloads is null)
{
_lastModified = fileInfo.LastWriteTime;
return await LoadFile();
}
return ignoredDownloads;
}
private async Task<IReadOnlyList<string>> LoadFile()
{
try
{
if (string.IsNullOrEmpty(_config.IgnoredDownloadsPath))
{
return Array.Empty<string>();
}
string[] ignoredDownloads = (await File.ReadAllLinesAsync(_config.IgnoredDownloadsPath))
.Where(x => !string.IsNullOrWhiteSpace(x))
.ToArray();
_cache.Set(CacheKeys.IgnoredDownloads(typeof(T).Name), ignoredDownloads);
_logger.LogInformation("ignored downloads reloaded");
return ignoredDownloads;
}
catch (Exception exception)
{
_logger.LogError(exception, "error while reading ignored downloads file | {file}", _config.IgnoredDownloadsPath);
}
return Array.Empty<string>();
}
}

View File

@@ -15,27 +15,22 @@ using Newtonsoft.Json;
namespace Infrastructure.Verticals.Arr;
public abstract class ArrClient : InterceptedService, IArrClient, IDryRunService
public abstract class ArrClient : IArrClient
{
protected readonly ILogger<ArrClient> _logger;
protected readonly HttpClient _httpClient;
protected readonly LoggingConfig _loggingConfig;
protected readonly QueueCleanerConfig _queueCleanerConfig;
protected readonly IStriker _striker;
/// <summary>
/// Constructor to be used by interceptors.
/// </summary>
protected ArrClient()
{
}
protected readonly IDryRunInterceptor _dryRunInterceptor;
protected ArrClient(
ILogger<ArrClient> logger,
IHttpClientFactory httpClientFactory,
IOptions<LoggingConfig> loggingConfig,
IOptions<QueueCleanerConfig> queueCleanerConfig,
IStriker striker
IStriker striker,
IDryRunInterceptor dryRunInterceptor
)
{
_logger = logger;
@@ -43,6 +38,7 @@ public abstract class ArrClient : InterceptedService, IArrClient, IDryRunService
_loggingConfig = loggingConfig.Value;
_queueCleanerConfig = queueCleanerConfig.Value;
_striker = striker;
_dryRunInterceptor = dryRunInterceptor;
}
public virtual async Task<QueueListResponse> GetQueueItemsAsync(ArrInstance arrInstance, int page)
@@ -125,7 +121,8 @@ public abstract class ArrClient : InterceptedService, IArrClient, IDryRunService
using HttpRequestMessage request = new(HttpMethod.Delete, uri);
SetApiKey(request, arrInstance.ApiKey);
using var _ = await ((ArrClient)Proxy).SendRequestAsync(request);
HttpResponseMessage? response = await _dryRunInterceptor.InterceptAsync<HttpResponseMessage>(SendRequestAsync, request);
response?.Dispose();
_logger.LogInformation(
removeFromClient

View File

@@ -5,6 +5,7 @@ using Common.Configuration.QueueCleaner;
using Domain.Models.Arr;
using Domain.Models.Arr.Queue;
using Domain.Models.Lidarr;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.ItemStriker;
using Microsoft.Extensions.Logging;
@@ -15,18 +16,14 @@ namespace Infrastructure.Verticals.Arr;
public class LidarrClient : ArrClient, ILidarrClient
{
/// <inheritdoc/>
public LidarrClient()
{
}
public LidarrClient(
ILogger<LidarrClient> logger,
IHttpClientFactory httpClientFactory,
IOptions<LoggingConfig> loggingConfig,
IOptions<QueueCleanerConfig> queueCleanerConfig,
IStriker striker
) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker)
IStriker striker,
IDryRunInterceptor dryRunInterceptor
) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker, dryRunInterceptor)
{
}
@@ -64,7 +61,8 @@ public class LidarrClient : ArrClient, ILidarrClient
try
{
using var _ = await ((LidarrClient)Proxy).SendRequestAsync(request);
HttpResponseMessage? response = await _dryRunInterceptor.InterceptAsync<HttpResponseMessage>(SendRequestAsync, request);
response?.Dispose();
_logger.LogInformation("{log}", GetSearchLog(arrInstance.Url, command, true, logContext));
}

View File

@@ -5,6 +5,7 @@ using Common.Configuration.QueueCleaner;
using Domain.Models.Arr;
using Domain.Models.Arr.Queue;
using Domain.Models.Radarr;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.ItemStriker;
using Microsoft.Extensions.Logging;
@@ -15,18 +16,14 @@ namespace Infrastructure.Verticals.Arr;
public class RadarrClient : ArrClient, IRadarrClient
{
/// <inheritdoc/>
public RadarrClient()
{
}
public RadarrClient(
ILogger<ArrClient> logger,
IHttpClientFactory httpClientFactory,
IOptions<LoggingConfig> loggingConfig,
IOptions<QueueCleanerConfig> queueCleanerConfig,
IStriker striker
) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker)
IStriker striker,
IDryRunInterceptor dryRunInterceptor
) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker, dryRunInterceptor)
{
}
@@ -72,7 +69,8 @@ public class RadarrClient : ArrClient, IRadarrClient
try
{
using var _ = await ((RadarrClient)Proxy).SendRequestAsync(request);
HttpResponseMessage? response = await _dryRunInterceptor.InterceptAsync<HttpResponseMessage>(SendRequestAsync, request);
response?.Dispose();
_logger.LogInformation("{log}", GetSearchLog(arrInstance.Url, command, true, logContext));
}

View File

@@ -5,6 +5,7 @@ using Common.Configuration.QueueCleaner;
using Domain.Models.Arr;
using Domain.Models.Arr.Queue;
using Domain.Models.Sonarr;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.ItemStriker;
using Microsoft.Extensions.Logging;
@@ -16,18 +17,14 @@ namespace Infrastructure.Verticals.Arr;
public class SonarrClient : ArrClient, ISonarrClient
{
/// <inheritdoc/>
public SonarrClient()
{
}
public SonarrClient(
ILogger<SonarrClient> logger,
IHttpClientFactory httpClientFactory,
IOptions<LoggingConfig> loggingConfig,
IOptions<QueueCleanerConfig> queueCleanerConfig,
IStriker striker
) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker)
IStriker striker,
IDryRunInterceptor dryRunInterceptor
) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker, dryRunInterceptor)
{
}
@@ -68,8 +65,9 @@ public class SonarrClient : ArrClient, ISonarrClient
try
{
using var _ = await ((SonarrClient)Proxy).SendRequestAsync(request);
HttpResponseMessage? response = await _dryRunInterceptor.InterceptAsync<HttpResponseMessage>(SendRequestAsync, request);
response?.Dispose();
_logger.LogInformation("{log}", GetSearchLog(command.SearchType, arrInstance.Url, command, true, logContext));
}
catch

View File

@@ -6,6 +6,7 @@ using Common.Configuration.DownloadClient;
using Domain.Enums;
using Domain.Models.Arr;
using Domain.Models.Arr.Queue;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.Context;
@@ -22,7 +23,8 @@ public sealed class ContentBlocker : GenericHandler
{
private readonly ContentBlockerConfig _config;
private readonly BlocklistProvider _blocklistProvider;
private readonly IgnoredDownloadsProvider<ContentBlockerConfig> _ignoredDownloadsProvider;
public ContentBlocker(
ILogger<ContentBlocker> logger,
IOptions<ContentBlockerConfig> config,
@@ -36,7 +38,8 @@ public sealed class ContentBlocker : GenericHandler
ArrQueueIterator arrArrQueueIterator,
BlocklistProvider blocklistProvider,
DownloadServiceFactory downloadServiceFactory,
NotificationPublisher notifier
INotificationPublisher notifier,
IgnoredDownloadsProvider<ContentBlockerConfig> ignoredDownloadsProvider
) : base(
logger, downloadClientConfig,
sonarrConfig, radarrConfig, lidarrConfig,
@@ -47,6 +50,7 @@ public sealed class ContentBlocker : GenericHandler
{
_config = config.Value;
_blocklistProvider = blocklistProvider;
_ignoredDownloadsProvider = ignoredDownloadsProvider;
}
public override async Task ExecuteAsync()
@@ -73,6 +77,8 @@ public sealed class ContentBlocker : GenericHandler
protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType)
{
IReadOnlyList<string> ignoredDownloads = await _ignoredDownloadsProvider.GetIgnoredDownloads();
using var _ = LogContext.PushProperty("InstanceName", instanceType.ToString());
HashSet<SearchItem> itemsToBeRefreshed = [];
@@ -106,13 +112,19 @@ public sealed class ContentBlocker : GenericHandler
continue;
}
if (ignoredDownloads.Contains(record.DownloadId, StringComparer.InvariantCultureIgnoreCase))
{
_logger.LogInformation("skip | {title} | ignored", record.Title);
continue;
}
// push record to context
ContextProvider.Set(nameof(QueueRecord), record);
_logger.LogDebug("searching unwanted files for {title}", record.Title);
BlockFilesResult result = await _downloadService
.BlockUnwantedFilesAsync(record.DownloadId, blocklistType, patterns, regexes);
.BlockUnwantedFilesAsync(record.DownloadId, blocklistType, patterns, regexes, ignoredDownloads);
if (!result.ShouldRemove)
{

View File

@@ -3,6 +3,7 @@ using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Domain.Enums;
using Domain.Models.Arr.Queue;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.DownloadClient;
@@ -17,6 +18,7 @@ namespace Infrastructure.Verticals.DownloadCleaner;
public sealed class DownloadCleaner : GenericHandler
{
private readonly DownloadCleanerConfig _config;
private readonly IgnoredDownloadsProvider<DownloadCleanerConfig> _ignoredDownloadsProvider;
private readonly HashSet<string> _excludedHashes = [];
public DownloadCleaner(
@@ -31,7 +33,8 @@ public sealed class DownloadCleaner : GenericHandler
LidarrClient lidarrClient,
ArrQueueIterator arrArrQueueIterator,
DownloadServiceFactory downloadServiceFactory,
NotificationPublisher notifier
INotificationPublisher notifier,
IgnoredDownloadsProvider<DownloadCleanerConfig> ignoredDownloadsProvider
) : base(
logger, downloadClientConfig,
sonarrConfig, radarrConfig, lidarrConfig,
@@ -42,16 +45,25 @@ public sealed class DownloadCleaner : GenericHandler
{
_config = config.Value;
_config.Validate();
_ignoredDownloadsProvider = ignoredDownloadsProvider;
}
public override async Task ExecuteAsync()
{
if (_downloadClientConfig.DownloadClient is Common.Enums.DownloadClient.None)
{
_logger.LogWarning("download client is set to none");
return;
}
if (_config.Categories?.Count is null or 0)
{
_logger.LogWarning("no categories configured");
return;
}
IReadOnlyList<string> ignoredDownloads = await _ignoredDownloadsProvider.GetIgnoredDownloads();
await _downloadService.LoginAsync();
List<object>? downloads = await _downloadService.GetAllDownloadsToBeCleaned(_config.Categories);
@@ -69,7 +81,7 @@ public sealed class DownloadCleaner : GenericHandler
await ProcessArrConfigAsync(_radarrConfig, InstanceType.Radarr, true);
await ProcessArrConfigAsync(_lidarrConfig, InstanceType.Lidarr, true);
await _downloadService.CleanDownloads(downloads, _config.Categories, _excludedHashes);
await _downloadService.CleanDownloads(downloads, _config.Categories, _excludedHashes, ignoredDownloads);
}
protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType)

View File

@@ -16,6 +16,20 @@ public sealed class DelugeClient
private readonly DelugeConfig _config;
private readonly HttpClient _httpClient;
private static readonly IReadOnlyList<string> Fields =
[
"hash",
"state",
"name",
"eta",
"private",
"total_done",
"label",
"seeding_time",
"ratio",
"trackers"
];
public DelugeClient(IOptions<DelugeConfig> config, IHttpClientFactory httpClientFactory)
{
_config = config.Value;
@@ -68,7 +82,7 @@ public sealed class DelugeClient
return await SendRequest<TorrentStatus?>(
"web.get_torrent_status",
hash,
new[] { "hash", "state", "name", "eta", "private", "total_done", "label", "seeding_time", "ratio" }
Fields
);
}
@@ -77,7 +91,7 @@ public sealed class DelugeClient
Dictionary<string, TorrentStatus>? downloads = await SendRequest<Dictionary<string, TorrentStatus>?>(
"core.get_torrents_status",
"",
new[] { "hash", "state", "name", "eta", "private", "total_done", "label", "seeding_time", "ratio" }
Fields
);
return downloads?.Values.ToList();

View File

@@ -7,11 +7,12 @@ using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Domain.Enums;
using Domain.Models.Deluge.Response;
using Infrastructure.Extensions;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.Context;
using Infrastructure.Verticals.ItemStriker;
using Infrastructure.Verticals.Notifications;
using MassTransit.Configuration;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
@@ -22,11 +23,6 @@ public class DelugeService : DownloadService, IDelugeService
{
private readonly DelugeClient _client;
/// <inheritdoc/>
public DelugeService()
{
}
public DelugeService(
ILogger<DelugeService> logger,
IOptions<DelugeConfig> config,
@@ -37,8 +33,12 @@ public class DelugeService : DownloadService, IDelugeService
IMemoryCache cache,
IFilenameEvaluator filenameEvaluator,
IStriker striker,
NotificationPublisher notifier
) : base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache, filenameEvaluator, striker, notifier)
INotificationPublisher notifier,
IDryRunInterceptor dryRunInterceptor
) : base(
logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache,
filenameEvaluator, striker, notifier, dryRunInterceptor
)
{
config.Value.Validate();
_client = new (config, httpClientFactory);
@@ -50,20 +50,26 @@ public class DelugeService : DownloadService, IDelugeService
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
hash = hash.ToLowerInvariant();
DelugeContents? contents = null;
StalledResult result = new();
TorrentStatus? status = await _client.GetTorrentStatus(hash);
TorrentStatus? download = await _client.GetTorrentStatus(hash);
if (status?.Hash is null)
if (download?.Hash is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
try
{
@@ -89,8 +95,8 @@ public class DelugeService : DownloadService, IDelugeService
result.DeleteReason = DeleteReason.AllFilesBlocked;
}
result.ShouldRemove = shouldRemove || await IsItemStuckAndShouldRemove(status);
result.IsPrivate = status.Private;
result.ShouldRemove = shouldRemove || await IsItemStuckAndShouldRemove(download);
result.IsPrivate = download.Private;
if (!shouldRemove && result.ShouldRemove)
{
@@ -101,30 +107,37 @@ public class DelugeService : DownloadService, IDelugeService
}
/// <inheritdoc/>
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
)
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads)
{
hash = hash.ToLowerInvariant();
TorrentStatus? status = await _client.GetTorrentStatus(hash);
TorrentStatus? download = await _client.GetTorrentStatus(hash);
BlockFilesResult result = new();
if (status?.Hash is null)
if (download?.Hash is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
result.IsPrivate = status.Private;
if (_contentBlockerConfig.IgnorePrivate && status.Private)
var ceva = await _client.GetTorrentExtended(hash);
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
result.IsPrivate = download.Private;
if (_contentBlockerConfig.IgnorePrivate && download.Private)
{
// ignore private trackers
_logger.LogDebug("skip files check | download is private | {name}", status.Name);
_logger.LogDebug("skip files check | download is private | {name}", download.Name);
return result;
}
@@ -190,7 +203,7 @@ public class DelugeService : DownloadService, IDelugeService
return result;
}
await ((DelugeService)Proxy).ChangeFilesPriority(hash, sortedPriorities);
await _dryRunInterceptor.InterceptAsync(ChangeFilesPriority, hash, sortedPriorities);
return result;
}
@@ -206,7 +219,8 @@ public class DelugeService : DownloadService, IDelugeService
}
/// <inheritdoc/>
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads)
{
foreach (TorrentStatus download in downloads)
{
@@ -214,6 +228,12 @@ public class DelugeService : DownloadService, IDelugeService
{
continue;
}
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
continue;
}
Category? category = categoriesToClean
.FirstOrDefault(x => x.Name.Equals(download.Label, StringComparison.InvariantCultureIgnoreCase));
@@ -245,8 +265,8 @@ public class DelugeService : DownloadService, IDelugeService
{
continue;
}
await ((DelugeService)Proxy).DeleteDownload(download.Hash);
await _dryRunInterceptor.InterceptAsync(DeleteDownload, download.Hash);
_logger.LogInformation(
"download cleaned | {reason} reached | {name}",

View File

@@ -18,7 +18,7 @@ using Microsoft.Extensions.Options;
namespace Infrastructure.Verticals.DownloadClient;
public abstract class DownloadService : InterceptedService, IDownloadService
public abstract class DownloadService : IDownloadService
{
protected readonly ILogger<DownloadService> _logger;
protected readonly QueueCleanerConfig _queueCleanerConfig;
@@ -28,15 +28,9 @@ public abstract class DownloadService : InterceptedService, IDownloadService
protected readonly IFilenameEvaluator _filenameEvaluator;
protected readonly IStriker _striker;
protected readonly MemoryCacheEntryOptions _cacheOptions;
protected readonly NotificationPublisher _notifier;
protected readonly INotificationPublisher _notifier;
protected readonly IDryRunInterceptor _dryRunInterceptor;
/// <summary>
/// Constructor to be used by interceptors.
/// </summary>
protected DownloadService()
{
}
protected DownloadService(
ILogger<DownloadService> logger,
IOptions<QueueCleanerConfig> queueCleanerConfig,
@@ -45,7 +39,9 @@ public abstract class DownloadService : InterceptedService, IDownloadService
IMemoryCache cache,
IFilenameEvaluator filenameEvaluator,
IStriker striker,
NotificationPublisher notifier)
INotificationPublisher notifier,
IDryRunInterceptor dryRunInterceptor
)
{
_logger = logger;
_queueCleanerConfig = queueCleanerConfig.Value;
@@ -55,6 +51,7 @@ public abstract class DownloadService : InterceptedService, IDownloadService
_filenameEvaluator = filenameEvaluator;
_striker = striker;
_notifier = notifier;
_dryRunInterceptor = dryRunInterceptor;
_cacheOptions = new MemoryCacheEntryOptions()
.SetSlidingExpiration(StaticConfiguration.TriggerValue + Constants.CacheLimitBuffer);
}
@@ -63,15 +60,13 @@ public abstract class DownloadService : InterceptedService, IDownloadService
public abstract Task LoginAsync();
public abstract Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash);
public abstract Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads);
/// <inheritdoc/>
public abstract Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public abstract Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
);
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads);
/// <inheritdoc/>
public abstract Task DeleteDownload(string hash);
@@ -80,7 +75,8 @@ public abstract class DownloadService : InterceptedService, IDownloadService
public abstract Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories);
/// <inheritdoc/>
public abstract Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes);
public abstract Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads);
protected void ResetStrikesOnProgress(string hash, long downloaded)
{
@@ -134,7 +130,7 @@ public abstract class DownloadService : InterceptedService, IDownloadService
return new();
}
private bool DownloadReachedRatio(double ratio, TimeSpan seedingTime, Category category)
{
if (category.MaxRatio < 0)

View File

@@ -3,6 +3,7 @@ using System.Text.RegularExpressions;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.ItemStriker;
using Infrastructure.Verticals.Notifications;
@@ -12,9 +13,9 @@ using Microsoft.Extensions.Options;
namespace Infrastructure.Verticals.DownloadClient;
public sealed class DummyDownloadService : DownloadService
public class DummyDownloadService : DownloadService
{
public DummyDownloadService(ILogger<DownloadService> logger, IOptions<QueueCleanerConfig> queueCleanerConfig, IOptions<ContentBlockerConfig> contentBlockerConfig, IOptions<DownloadCleanerConfig> downloadCleanerConfig, IMemoryCache cache, IFilenameEvaluator filenameEvaluator, IStriker striker, NotificationPublisher notifier) : base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache, filenameEvaluator, striker, notifier)
public DummyDownloadService(ILogger<DownloadService> logger, IOptions<QueueCleanerConfig> queueCleanerConfig, IOptions<ContentBlockerConfig> contentBlockerConfig, IOptions<DownloadCleanerConfig> downloadCleanerConfig, IMemoryCache cache, IFilenameEvaluator filenameEvaluator, IStriker striker, INotificationPublisher notifier, IDryRunInterceptor dryRunInterceptor) : base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache, filenameEvaluator, striker, notifier, dryRunInterceptor)
{
}
@@ -27,12 +28,13 @@ public sealed class DummyDownloadService : DownloadService
return Task.CompletedTask;
}
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
throw new NotImplementedException();
}
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType, ConcurrentBag<string> patterns, ConcurrentBag<Regex> regexes)
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType, ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads)
{
throw new NotImplementedException();
}
@@ -42,7 +44,8 @@ public sealed class DummyDownloadService : DownloadService
throw new NotImplementedException();
}
public override Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads)
{
throw new NotImplementedException();
}

View File

@@ -6,7 +6,7 @@ using Infrastructure.Interceptors;
namespace Infrastructure.Verticals.DownloadClient;
public interface IDownloadService : IDisposable, IDryRunService
public interface IDownloadService : IDisposable
{
public Task LoginAsync();
@@ -14,7 +14,8 @@ public interface IDownloadService : IDisposable, IDryRunService
/// Checks whether the download should be removed from the *arr queue.
/// </summary>
/// <param name="hash">The download hash.</param>
public Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash);
/// <param name="ignoredDownloads">Downloads to ignore from processing.</param>
public Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads);
/// <summary>
/// Blocks unwanted files from being fully downloaded.
@@ -23,12 +24,13 @@ public interface IDownloadService : IDisposable, IDryRunService
/// <param name="blocklistType">The <see cref="BlocklistType"/>.</param>
/// <param name="patterns">The patterns to test the files against.</param>
/// <param name="regexes">The regexes to test the files against.</param>
/// <param name="ignoredDownloads">Downloads to ignore from processing.</param>
/// <returns>True if all files have been blocked; otherwise false.</returns>
public Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
ConcurrentBag<Regex> regexes,
IReadOnlyList<string> ignoredDownloads
);
/// <summary>
@@ -37,14 +39,16 @@ public interface IDownloadService : IDisposable, IDryRunService
/// <param name="categories">The categories by which to filter the downloads.</param>
/// <returns>A list of downloads for the provided categories.</returns>
Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories);
/// <summary>
/// Cleans the downloads.
/// </summary>
/// <param name="downloads"></param>
/// <param name="categoriesToClean">The categories that should be cleaned.</param>
/// <param name="excludedHashes">The hashes that should not be cleaned.</param>
public abstract Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes);
/// <param name="ignoredDownloads">Downloads to ignore from processing.</param>
public abstract Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads);
/// <summary>
/// Deletes a download item.

View File

@@ -1,5 +1,5 @@
namespace Infrastructure.Verticals.DownloadClient.QBittorrent;
public interface IQBitService : IDownloadService
public interface IQBitService : IDownloadService, IDisposable
{
}

View File

@@ -7,6 +7,8 @@ using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Common.Helpers;
using Domain.Enums;
using Infrastructure.Extensions;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.Context;
using Infrastructure.Verticals.ItemStriker;
@@ -24,11 +26,6 @@ public class QBitService : DownloadService, IQBitService
private readonly QBitConfig _config;
private readonly QBittorrentClient _client;
/// <inheritdoc/>
public QBitService()
{
}
public QBitService(
ILogger<QBitService> logger,
IHttpClientFactory httpClientFactory,
@@ -39,8 +36,12 @@ public class QBitService : DownloadService, IQBitService
IMemoryCache cache,
IFilenameEvaluator filenameEvaluator,
IStriker striker,
NotificationPublisher notifier
) : base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache, filenameEvaluator, striker, notifier)
INotificationPublisher notifier,
IDryRunInterceptor dryRunInterceptor
) : base(
logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache,
filenameEvaluator, striker, notifier, dryRunInterceptor
)
{
_config = config.Value;
_config.Validate();
@@ -58,18 +59,27 @@ public class QBitService : DownloadService, IQBitService
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
StalledResult result = new();
TorrentInfo? torrent = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
TorrentInfo? download = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
.FirstOrDefault();
if (torrent is null)
if (download is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
IReadOnlyList<TorrentTracker> trackers = await GetTrackersAsync(hash);
if (ignoredDownloads.Count > 0 &&
(download.ShouldIgnore(ignoredDownloads) || trackers.Any(x => x.ShouldIgnore(ignoredDownloads)) is true))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
TorrentProperties? torrentProperties = await _client.GetTorrentPropertiesAsync(hash);
if (torrentProperties is null)
@@ -83,7 +93,7 @@ public class QBitService : DownloadService, IQBitService
&& boolValue;
// if all files were blocked by qBittorrent
if (torrent is { CompletionOn: not null, Downloaded: null or 0 })
if (download is { CompletionOn: not null, Downloaded: null or 0 })
{
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesBlocked;
@@ -100,7 +110,7 @@ public class QBitService : DownloadService, IQBitService
return result;
}
result.ShouldRemove = await IsItemStuckAndShouldRemove(torrent, result.IsPrivate);
result.ShouldRemove = await IsItemStuckAndShouldRemove(download, result.IsPrivate);
if (result.ShouldRemove)
{
@@ -111,23 +121,32 @@ public class QBitService : DownloadService, IQBitService
}
/// <inheritdoc/>
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
ConcurrentBag<Regex> regexes,
IReadOnlyList<string> ignoredDownloads
)
{
TorrentInfo? torrent = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
TorrentInfo? download = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
.FirstOrDefault();
BlockFilesResult result = new();
if (torrent is null)
if (download is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
IReadOnlyList<TorrentTracker> trackers = await GetTrackersAsync(hash);
if (ignoredDownloads.Count > 0 &&
(download.ShouldIgnore(ignoredDownloads) || trackers.Any(x => x.ShouldIgnore(ignoredDownloads)) is true))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
TorrentProperties? torrentProperties = await _client.GetTorrentPropertiesAsync(hash);
if (torrentProperties is null)
@@ -145,7 +164,7 @@ public class QBitService : DownloadService, IQBitService
if (_contentBlockerConfig.IgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip files check | download is private | {name}", torrent.Name);
_logger.LogDebug("skip files check | download is private | {name}", download.Name);
return result;
}
@@ -200,7 +219,7 @@ public class QBitService : DownloadService, IQBitService
foreach (int fileIndex in unwantedFiles)
{
await ((QBitService)Proxy).SkipFile(hash, fileIndex);
await _dryRunInterceptor.InterceptAsync(SkipFile, hash, fileIndex);
}
return result;
@@ -218,7 +237,8 @@ public class QBitService : DownloadService, IQBitService
.ToList();
/// <inheritdoc/>
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads)
{
foreach (TorrentInfo download in downloads)
{
@@ -227,6 +247,15 @@ public class QBitService : DownloadService, IQBitService
continue;
}
IReadOnlyList<TorrentTracker> trackers = await GetTrackersAsync(download.Hash);
if (ignoredDownloads.Count > 0 &&
(download.ShouldIgnore(ignoredDownloads) || trackers.Any(x => x.ShouldIgnore(ignoredDownloads)) is true))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
continue;
}
Category? category = categoriesToClean
.FirstOrDefault(x => download.Category.Equals(x.Name, StringComparison.InvariantCultureIgnoreCase));
@@ -272,7 +301,7 @@ public class QBitService : DownloadService, IQBitService
continue;
}
await ((QBitService)Proxy).DeleteDownload(download.Hash);
await _dryRunInterceptor.InterceptAsync(DeleteDownload, download.Hash);
_logger.LogInformation(
"download cleaned | {reason} reached | {name}",
@@ -303,7 +332,7 @@ public class QBitService : DownloadService, IQBitService
{
_client.Dispose();
}
private async Task<bool> IsItemStuckAndShouldRemove(TorrentInfo torrent, bool isPrivate)
{
if (_queueCleanerConfig.StalledMaxStrikes is 0)
@@ -329,4 +358,11 @@ public class QBitService : DownloadService, IQBitService
return await StrikeAndCheckLimit(torrent.Hash, torrent.Name);
}
private async Task<IReadOnlyList<TorrentTracker>> GetTrackersAsync(string hash)
{
return (await _client.GetTorrentTrackersAsync(hash))
.Where(x => !x.Url.ToString().Contains("**"))
.ToList();
}
}

View File

@@ -7,6 +7,8 @@ using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Common.Helpers;
using Domain.Enums;
using Infrastructure.Extensions;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.Context;
using Infrastructure.Verticals.ItemStriker;
@@ -26,11 +28,23 @@ public class TransmissionService : DownloadService, ITransmissionService
private readonly Client _client;
private TorrentInfo[]? _torrentsCache;
/// <inheritdoc/>
public TransmissionService()
{
}
private static readonly string[] Fields =
[
TorrentFields.FILES,
TorrentFields.FILE_STATS,
TorrentFields.HASH_STRING,
TorrentFields.ID,
TorrentFields.ETA,
TorrentFields.NAME,
TorrentFields.STATUS,
TorrentFields.IS_PRIVATE,
TorrentFields.DOWNLOADED_EVER,
TorrentFields.DOWNLOAD_DIR,
TorrentFields.SECONDS_SEEDING,
TorrentFields.UPLOAD_RATIO,
TorrentFields.TRACKERS
];
public TransmissionService(
IHttpClientFactory httpClientFactory,
ILogger<TransmissionService> logger,
@@ -41,8 +55,12 @@ public class TransmissionService : DownloadService, ITransmissionService
IMemoryCache cache,
IFilenameEvaluator filenameEvaluator,
IStriker striker,
NotificationPublisher notifier
) : base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache, filenameEvaluator, striker, notifier)
INotificationPublisher notifier,
IDryRunInterceptor dryRunInterceptor
) : base(
logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache,
filenameEvaluator, striker, notifier, dryRunInterceptor
)
{
_config = config.Value;
_config.Validate();
@@ -60,21 +78,27 @@ public class TransmissionService : DownloadService, ITransmissionService
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
StalledResult result = new();
TorrentInfo? torrent = await GetTorrentAsync(hash);
TorrentInfo? download = await GetTorrentAsync(hash);
if (torrent is null)
if (download is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
bool shouldRemove = torrent.FileStats?.Length > 0;
result.IsPrivate = torrent.IsPrivate ?? false;
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogDebug("skip | download is ignored | {name}", download.Name);
return result;
}
bool shouldRemove = download.FileStats?.Length > 0;
result.IsPrivate = download.IsPrivate ?? false;
foreach (TransmissionTorrentFileStats? stats in torrent.FileStats ?? [])
foreach (TransmissionTorrentFileStats? stats in download.FileStats ?? [])
{
if (!stats.Wanted.HasValue)
{
@@ -95,7 +119,7 @@ public class TransmissionService : DownloadService, ITransmissionService
}
// remove if all files are unwanted or download is stuck
result.ShouldRemove = shouldRemove || await IsItemStuckAndShouldRemove(torrent);
result.ShouldRemove = shouldRemove || await IsItemStuckAndShouldRemove(download);
if (!shouldRemove && result.ShouldRemove)
{
@@ -106,28 +130,32 @@ public class TransmissionService : DownloadService, ITransmissionService
}
/// <inheritdoc/>
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
)
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads)
{
TorrentInfo? torrent = await GetTorrentAsync(hash);
TorrentInfo? download = await GetTorrentAsync(hash);
BlockFilesResult result = new();
if (torrent?.FileStats is null || torrent.Files is null)
if (download?.FileStats is null || download.Files is null)
{
return result;
}
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogDebug("skip | download is ignored | {name}", download.Name);
return result;
}
bool isPrivate = torrent.IsPrivate ?? false;
bool isPrivate = download.IsPrivate ?? false;
result.IsPrivate = isPrivate;
if (_contentBlockerConfig.IgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip files check | download is private | {name}", torrent.Name);
_logger.LogDebug("skip files check | download is private | {name}", download.Name);
return result;
}
@@ -135,27 +163,27 @@ public class TransmissionService : DownloadService, ITransmissionService
long totalFiles = 0;
long totalUnwantedFiles = 0;
for (int i = 0; i < torrent.Files.Length; i++)
for (int i = 0; i < download.Files.Length; i++)
{
if (torrent.FileStats?[i].Wanted == null)
if (download.FileStats?[i].Wanted == null)
{
continue;
}
totalFiles++;
if (!torrent.FileStats[i].Wanted.Value)
if (!download.FileStats[i].Wanted.Value)
{
totalUnwantedFiles++;
continue;
}
if (_filenameEvaluator.IsValid(torrent.Files[i].Name, blocklistType, patterns, regexes))
if (_filenameEvaluator.IsValid(download.Files[i].Name, blocklistType, patterns, regexes))
{
continue;
}
_logger.LogInformation("unwanted file found | {file}", torrent.Files[i].Name);
_logger.LogInformation("unwanted file found | {file}", download.Files[i].Name);
unwantedFiles.Add(i);
totalUnwantedFiles++;
}
@@ -174,8 +202,8 @@ public class TransmissionService : DownloadService, ITransmissionService
}
_logger.LogDebug("changing priorities | torrent {hash}", hash);
await ((TransmissionService)Proxy).SetUnwantedFiles(torrent.Id, unwantedFiles.ToArray());
await _dryRunInterceptor.InterceptAsync(SetUnwantedFiles, download.Id, unwantedFiles.ToArray());
return result;
}
@@ -183,34 +211,29 @@ public class TransmissionService : DownloadService, ITransmissionService
/// <inheritdoc/>
public override async Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories)
{
string[] fields = [
TorrentFields.FILES,
TorrentFields.FILE_STATS,
TorrentFields.HASH_STRING,
TorrentFields.ID,
TorrentFields.ETA,
TorrentFields.NAME,
TorrentFields.STATUS,
TorrentFields.IS_PRIVATE,
TorrentFields.DOWNLOADED_EVER,
TorrentFields.DOWNLOAD_DIR,
TorrentFields.SECONDS_SEEDING,
TorrentFields.UPLOAD_RATIO
];
return (await _client.TorrentGetAsync(fields))
return (await _client.TorrentGetAsync(Fields))
?.Torrents
?.Where(x => !string.IsNullOrEmpty(x.HashString))
.Where(x => x.Status is 5 or 6)
.Where(x => categories
.Any(cat => x.DownloadDir?.EndsWith(cat.Name, StringComparison.InvariantCultureIgnoreCase) is true)
.Any(cat =>
{
if (x.DownloadDir is null)
{
return false;
}
return Path.GetFileName(Path.TrimEndingDirectorySeparator(x.DownloadDir))
.Equals(cat.Name, StringComparison.InvariantCultureIgnoreCase);
})
)
.Cast<object>()
.ToList();
}
/// <inheritdoc/>
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads)
{
foreach (TorrentInfo download in downloads)
{
@@ -218,10 +241,25 @@ public class TransmissionService : DownloadService, ITransmissionService
{
continue;
}
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogDebug("skip | download is ignored | {name}", download.Name);
continue;
}
Category? category = categoriesToClean
.FirstOrDefault(x => download.DownloadDir?.EndsWith(x.Name, StringComparison.InvariantCultureIgnoreCase) is true);
.FirstOrDefault(x =>
{
if (download.DownloadDir is null)
{
return false;
}
return Path.GetFileName(Path.TrimEndingDirectorySeparator(download.DownloadDir))
.Equals(x.Name, StringComparison.InvariantCultureIgnoreCase);
});
if (category is null)
{
continue;
@@ -250,7 +288,7 @@ public class TransmissionService : DownloadService, ITransmissionService
continue;
}
await ((TransmissionService)Proxy).RemoveDownloadAsync(download.Id);
await _dryRunInterceptor.InterceptAsync(RemoveDownloadAsync, download.Id);
_logger.LogInformation(
"download cleaned | {reason} reached | {name}",
@@ -333,20 +371,8 @@ public class TransmissionService : DownloadService, ITransmissionService
if (_torrentsCache is null || torrent is null)
{
string[] fields = [
TorrentFields.FILES,
TorrentFields.FILE_STATS,
TorrentFields.HASH_STRING,
TorrentFields.ID,
TorrentFields.ETA,
TorrentFields.NAME,
TorrentFields.STATUS,
TorrentFields.IS_PRIVATE,
TorrentFields.DOWNLOADED_EVER
];
// refresh cache
_torrentsCache = (await _client.TorrentGetAsync(fields))
_torrentsCache = (await _client.TorrentGetAsync(Fields))
?.Torrents;
}

View File

@@ -1,6 +1,7 @@
using Common.Helpers;
using Domain.Enums;
using Infrastructure.Helpers;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.Context;
using Infrastructure.Verticals.Notifications;
using Microsoft.Extensions.Caching.Memory;
@@ -13,13 +14,15 @@ public sealed class Striker : IStriker
private readonly ILogger<Striker> _logger;
private readonly IMemoryCache _cache;
private readonly MemoryCacheEntryOptions _cacheOptions;
private readonly NotificationPublisher _notifier;
private readonly INotificationPublisher _notifier;
private readonly IDryRunInterceptor _dryRunInterceptor;
public Striker(ILogger<Striker> logger, IMemoryCache cache, NotificationPublisher notifier)
public Striker(ILogger<Striker> logger, IMemoryCache cache, INotificationPublisher notifier, IDryRunInterceptor dryRunInterceptor)
{
_logger = logger;
_cache = cache;
_notifier = notifier;
_dryRunInterceptor = dryRunInterceptor;
_cacheOptions = new MemoryCacheEntryOptions()
.SetSlidingExpiration(StaticConfiguration.TriggerValue + Constants.CacheLimitBuffer);
}

View File

@@ -24,7 +24,7 @@ public abstract class GenericHandler : IHandler, IDisposable
protected readonly ILidarrClient _lidarrClient;
protected readonly ArrQueueIterator _arrArrQueueIterator;
protected readonly IDownloadService _downloadService;
protected readonly NotificationPublisher _notifier;
protected readonly INotificationPublisher _notifier;
protected GenericHandler(
ILogger<GenericHandler> logger,
@@ -37,7 +37,7 @@ public abstract class GenericHandler : IHandler, IDisposable
ILidarrClient lidarrClient,
ArrQueueIterator arrArrQueueIterator,
DownloadServiceFactory downloadServiceFactory,
NotificationPublisher notifier
INotificationPublisher notifier
)
{
_logger = logger;

View File

@@ -0,0 +1,12 @@
using Domain.Enums;
namespace Infrastructure.Verticals.Notifications;
public interface INotificationPublisher
{
Task NotifyStrike(StrikeType strikeType, int strikeCount);
Task NotifyQueueItemDeleted(bool removeFromClient, DeleteReason reason);
Task NotifyDownloadCleaned(double ratio, TimeSpan seedingTime, string categoryName, CleanReason reason);
}

View File

@@ -12,25 +12,19 @@ using Microsoft.Extensions.Logging;
namespace Infrastructure.Verticals.Notifications;
public class NotificationPublisher : InterceptedService, IDryRunService
public class NotificationPublisher : INotificationPublisher
{
private readonly ILogger<NotificationPublisher> _logger;
private readonly ILogger<INotificationPublisher> _logger;
private readonly IBus _messageBus;
/// <summary>
/// Constructor to be used by interceptors.
/// </summary>
public NotificationPublisher()
{
}
public NotificationPublisher(ILogger<NotificationPublisher> logger, IBus messageBus)
private readonly IDryRunInterceptor _dryRunInterceptor;
public NotificationPublisher(ILogger<INotificationPublisher> logger, IBus messageBus, IDryRunInterceptor dryRunInterceptor)
{
_logger = logger;
_messageBus = messageBus;
_dryRunInterceptor = dryRunInterceptor;
}
[DryRunSafeguard]
public virtual async Task NotifyStrike(StrikeType strikeType, int strikeCount)
{
try
@@ -38,7 +32,7 @@ public class NotificationPublisher : InterceptedService, IDryRunService
QueueRecord record = ContextProvider.Get<QueueRecord>(nameof(QueueRecord));
InstanceType instanceType = (InstanceType)ContextProvider.Get<object>(nameof(InstanceType));
Uri instanceUrl = ContextProvider.Get<Uri>(nameof(ArrInstance) + nameof(ArrInstance.Url));
Uri imageUrl = GetImageFromContext(record, instanceType);
Uri? imageUrl = GetImageFromContext(record, instanceType);
ArrNotification notification = new()
{
@@ -54,10 +48,10 @@ public class NotificationPublisher : InterceptedService, IDryRunService
switch (strikeType)
{
case StrikeType.Stalled:
await _messageBus.Publish(notification.Adapt<StalledStrikeNotification>());
await _dryRunInterceptor.InterceptAsync(Notify<StalledStrikeNotification>, notification.Adapt<StalledStrikeNotification>());
break;
case StrikeType.ImportFailed:
await _messageBus.Publish(notification.Adapt<FailedImportStrikeNotification>());
await _dryRunInterceptor.InterceptAsync(Notify<FailedImportStrikeNotification>, notification.Adapt<FailedImportStrikeNotification>());
break;
}
}
@@ -67,54 +61,84 @@ public class NotificationPublisher : InterceptedService, IDryRunService
}
}
[DryRunSafeguard]
public virtual async Task NotifyQueueItemDeleted(bool removeFromClient, DeleteReason reason)
{
QueueRecord record = ContextProvider.Get<QueueRecord>(nameof(QueueRecord));
InstanceType instanceType = (InstanceType)ContextProvider.Get<object>(nameof(InstanceType));
Uri instanceUrl = ContextProvider.Get<Uri>(nameof(ArrInstance) + nameof(ArrInstance.Url));
Uri imageUrl = GetImageFromContext(record, instanceType);
QueueItemDeletedNotification notification = new()
try
{
InstanceType = instanceType,
InstanceUrl = instanceUrl,
Hash = record.DownloadId.ToLowerInvariant(),
Title = $"Deleting item from queue with reason: {reason}",
Description = record.Title,
Image = imageUrl,
Fields = [new() { Title = "Removed from download client?", Text = removeFromClient ? "Yes" : "No" }]
};
await _messageBus.Publish(notification);
QueueRecord record = ContextProvider.Get<QueueRecord>(nameof(QueueRecord));
InstanceType instanceType = (InstanceType)ContextProvider.Get<object>(nameof(InstanceType));
Uri instanceUrl = ContextProvider.Get<Uri>(nameof(ArrInstance) + nameof(ArrInstance.Url));
Uri? imageUrl = GetImageFromContext(record, instanceType);
QueueItemDeletedNotification notification = new()
{
InstanceType = instanceType,
InstanceUrl = instanceUrl,
Hash = record.DownloadId.ToLowerInvariant(),
Title = $"Deleting item from queue with reason: {reason}",
Description = record.Title,
Image = imageUrl,
Fields = [new() { Title = "Removed from download client?", Text = removeFromClient ? "Yes" : "No" }]
};
await _dryRunInterceptor.InterceptAsync(Notify<QueueItemDeletedNotification>, notification);
}
catch (Exception ex)
{
_logger.LogError(ex, "failed to notify queue item deleted");
}
}
public virtual async Task NotifyDownloadCleaned(double ratio, TimeSpan seedingTime, string categoryName, CleanReason reason)
{
try
{
DownloadCleanedNotification notification = new()
{
Title = $"Cleaned item from download client with reason: {reason}",
Description = ContextProvider.Get<string>("downloadName"),
Fields =
[
new() { Title = "Hash", Text = ContextProvider.Get<string>("hash").ToLowerInvariant() },
new() { Title = "Category", Text = categoryName.ToLowerInvariant() },
new() { Title = "Ratio", Text = $"{ratio.ToString(CultureInfo.InvariantCulture)}%" },
new()
{
Title = "Seeding hours", Text = $"{Math.Round(seedingTime.TotalHours, 0).ToString(CultureInfo.InvariantCulture)}h"
}
],
Level = NotificationLevel.Important
};
await _dryRunInterceptor.InterceptAsync(Notify<DownloadCleanedNotification>, notification);
}
catch (Exception ex)
{
_logger.LogError(ex, "failed to notify download cleaned");
}
}
[DryRunSafeguard]
public virtual async Task NotifyDownloadCleaned(double ratio, TimeSpan seedingTime, string categoryName, CleanReason reason)
private Task Notify<T>(T message) where T: notnull
{
DownloadCleanedNotification notification = new()
{
Title = $"Cleaned item from download client with reason: {reason}",
Description = ContextProvider.Get<string>("downloadName"),
Fields =
[
new() { Title = "Hash", Text = ContextProvider.Get<string>("hash").ToLowerInvariant() },
new() { Title = "Category", Text = categoryName.ToLowerInvariant() },
new() { Title = "Ratio", Text = $"{ratio.ToString(CultureInfo.InvariantCulture)}%" },
new() { Title = "Seeding hours", Text = $"{Math.Round(seedingTime.TotalHours, 0).ToString(CultureInfo.InvariantCulture)}h" }
],
Level = NotificationLevel.Important
};
await _messageBus.Publish(notification);
return _messageBus.Publish(message);
}
private static Uri GetImageFromContext(QueueRecord record, InstanceType instanceType) =>
instanceType switch
private Uri? GetImageFromContext(QueueRecord record, InstanceType instanceType)
{
Uri? image = instanceType switch
{
InstanceType.Sonarr => record.Series!.Images.FirstOrDefault(x => x.CoverType == "poster")?.RemoteUrl,
InstanceType.Radarr => record.Movie!.Images.FirstOrDefault(x => x.CoverType == "poster")?.RemoteUrl,
InstanceType.Lidarr => record.Album!.Images.FirstOrDefault(x => x.CoverType == "cover")?.Url,
_ => throw new ArgumentOutOfRangeException(nameof(instanceType))
} ?? throw new Exception("failed to get image url from context");
};
if (image is null)
{
_logger.LogWarning("no poster found for {title}", record.Title);
}
return image;
}
}

View File

@@ -4,6 +4,7 @@ using Common.Configuration.QueueCleaner;
using Domain.Enums;
using Domain.Models.Arr;
using Domain.Models.Arr.Queue;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.Context;
@@ -19,7 +20,8 @@ namespace Infrastructure.Verticals.QueueCleaner;
public sealed class QueueCleaner : GenericHandler
{
private readonly QueueCleanerConfig _config;
private readonly IgnoredDownloadsProvider<QueueCleanerConfig> _ignoredDownloadsProvider;
public QueueCleaner(
ILogger<QueueCleaner> logger,
IOptions<QueueCleanerConfig> config,
@@ -32,7 +34,8 @@ public sealed class QueueCleaner : GenericHandler
LidarrClient lidarrClient,
ArrQueueIterator arrArrQueueIterator,
DownloadServiceFactory downloadServiceFactory,
NotificationPublisher notifier
INotificationPublisher notifier,
IgnoredDownloadsProvider<QueueCleanerConfig> ignoredDownloadsProvider
) : base(
logger, downloadClientConfig,
sonarrConfig, radarrConfig, lidarrConfig,
@@ -42,10 +45,13 @@ public sealed class QueueCleaner : GenericHandler
)
{
_config = config.Value;
_ignoredDownloadsProvider = ignoredDownloadsProvider;
}
protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType)
{
IReadOnlyList<string> ignoredDownloads = await _ignoredDownloadsProvider.GetIgnoredDownloads();
using var _ = LogContext.PushProperty("InstanceName", instanceType.ToString());
HashSet<SearchItem> itemsToBeRefreshed = [];
@@ -75,15 +81,27 @@ public sealed class QueueCleaner : GenericHandler
continue;
}
if (ignoredDownloads.Contains(record.DownloadId, StringComparer.InvariantCultureIgnoreCase))
{
_logger.LogInformation("skip | {title} | ignored", record.Title);
continue;
}
// push record to context
ContextProvider.Set(nameof(QueueRecord), record);
StalledResult stalledCheckResult = new();
if (_downloadClientConfig.DownloadClient is not Common.Enums.DownloadClient.None && record.Protocol is "torrent")
if (record.Protocol is "torrent")
{
if (_downloadClientConfig.DownloadClient is Common.Enums.DownloadClient.None)
{
_logger.LogWarning("skip | download client is not configured | {title}", record.Title);
continue;
}
// stalled download check
stalledCheckResult = await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId);
stalledCheckResult = await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId, ignoredDownloads);
}
// failed import check

View File

@@ -0,0 +1 @@
ignored

View File

@@ -175,6 +175,7 @@ services:
image: ghcr.io/flmorg/cleanuperr:latest
container_name: cleanuperr
environment:
- TZ=Europe/Bucharest
- DRY_RUN=false
- LOGGING__LOGLEVEL=Debug
@@ -190,6 +191,7 @@ services:
- TRIGGERS__DOWNLOADCLEANER=0/30 * * * * ?
- QUEUECLEANER__ENABLED=true
- QUEUECLEANER__IGNORED_DOWNLOADS_PATH=/ignored
- QUEUECLEANER__RUNSEQUENTIALLY=true
- QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE=true
@@ -200,10 +202,12 @@ services:
- QUEUECLEANER__STALLED_DELETE_PRIVATE=false
- CONTENTBLOCKER__ENABLED=true
- CONTENTBLOCKER__IGNORED_DOWNLOADS_PATH=/ignored
- CONTENTBLOCKER__IGNORE_PRIVATE=true
- CONTENTBLOCKER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__ENABLED=true
- DOWNLOADCLEANER__IGNORED_DOWNLOADS_PATH=/ignored
- DOWNLOADCLEANER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__CATEGORIES__0__NAME=tv-sonarr
- DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO=-1
@@ -250,10 +254,12 @@ services:
# - NOTIFIARR__ON_IMPORT_FAILED_STRIKE=true
# - NOTIFIARR__ON_STALLED_STRIKE=true
# - NOTIFIARR__ON_QUEUE_ITEM_DELETED=true
# - NOTIFIARR__ON_DOWNLOAD_CLEANED=true
# - NOTIFIARR__API_KEY=notifiarr_secret
# - NOTIFIARR__CHANNEL_ID=discord_channel_id
volumes:
- ./data/cleanuperr/logs:/var/logs
- ./data/cleanuperr/ignored_downloads:/ignored
restart: unless-stopped
depends_on:
- qbittorrent

View File

@@ -12,6 +12,13 @@
### General settings
**`TZ`**
- The time zone to use.
- Type: String.
- Possible values: Any valid timezone.
- Default: `UTC`.
- Required: No.
**`DRY_RUN`**
- When enabled, simulates irreversible operations (like deletions and notifications) without making actual changes.
- Type: Boolean.
@@ -69,6 +76,31 @@
- Default: `true`
- Required: No.
**`QUEUECLEANER__IGNORED_DOWNLOADS_PATH`**
- Local path to the file containing ignored downloads.
- If the contents of the file are changed, they will be reloaded on the next job run.
- Accepted values:
- torrent hash
- qBitTorrent tag or category
- Deluge label
- Transmission category (last directory from the save location)
- torrent tracker domain
- Each value needs to be on a new line.
- Type: String.
- Default: Empty.
- Required: No.
- Example: `/ignored.txt`.
- Example of file contents:
```
fa800a7d7c443a2c3561d1f8f393c089036dade1
tv-sonarr
qbit-tag
mytracker.com
...
```
>[!IMPORTANT]
> Some people have experienced problems using Docker where the mounted file would not update inside the container if it was modified on the host. This is a Docker configuration problem and can not be solved by cleanuperr.
**`QUEUECLEANER__RUNSEQUENTIALLY`**
- Controls whether queue cleaner runs after content blocker instead of in parallel.
- When `true`, streamlines the cleaning process by running immediately after content blocker.
@@ -171,6 +203,31 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Default: `false`
- Required: No.
**`CONTENTBLOCKER__IGNORED_DOWNLOADS_PATH`**
- Local path to the file containing ignored downloads.
- If the contents of the file are changed, they will be reloaded on the next job run.
- Accepted values:
- torrent hash
- qBitTorrent tag or category
- Deluge label
- Transmission category (last directory from the save location)
- torrent tracker domain
- Each value needs to be on a new line.
- Type: String.
- Default: Empty.
- Required: No.
- Example: `/ignored.txt`.
- Example of file contents:
```
fa800a7d7c443a2c3561d1f8f393c089036dade1
tv-sonarr
qbit-tag
mytracker.com
...
```
>[!IMPORTANT]
> Some people have experienced problems using Docker where the mounted file would not update inside the container if it was modified on the host. This is a Docker configuration problem and can not be solved by cleanuperr.
**`CONTENTBLOCKER__IGNORE_PRIVATE`**
- Controls whether to ignore downloads from private trackers.
- Type: Boolean
@@ -210,6 +267,31 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Default: `false`
- Required: No.
**`DOWNLOADCLEANER__IGNORED_DOWNLOADS_PATH`**
- Local path to the file containing ignored downloads.
- If the contents of the file are changed, they will be reloaded on the next job run.
- Accepted values:
- torrent hash
- qBitTorrent tag or category
- Deluge label
- Transmission category (last directory from the save location)
- torrent tracker domain
- Each value needs to be on a new line.
- Type: String.
- Default: Empty.
- Required: No.
- Example: `/ignored.txt`.
- Example of file contents:
```
fa800a7d7c443a2c3561d1f8f393c089036dade1
tv-sonarr
qbit-tag
mytracker.com
...
```
>[!IMPORTANT]
> Some people have experienced problems using Docker where the mounted file would not update inside the container if it was modified on the host. This is a Docker configuration problem and can not be solved by cleanuperr.
**`DOWNLOADCLEANER__DELETE_PRIVATE`**
- Controls whether to delete private downloads.
- Type: Boolean.
@@ -230,7 +312,7 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
> The category name must match the category that was set in the *arr.
> For qBittorrent, the category name is the name of the download category.
> For Deluge, the category name is the name of the label.
> For Transmission, the category name is the name of the download location.
> For Transmission, the category name is the last directory from the save location.
**`DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO`**
- Maximum ratio to reach before removing a download.