Compare commits

...

25 Commits

Author SHA1 Message Date
Flaminel
704fdaca4a Add cleanup for slow downloads (#110) 2025-04-06 13:28:05 +03:00
Flaminel
b134136e51 Update README.md 2025-03-29 01:11:41 +02:00
Flaminel
5ca717d7e0 Update README.md 2025-03-27 19:53:57 +02:00
Flaminel
7068ee5e5a Update README.md 2025-03-26 13:30:55 +02:00
Flaminel
9f770473e5 Remove Transmission downloads cache (#105) 2025-03-26 00:26:10 +02:00
Flaminel
5fe0f5750a Fix qBit queued items being processed (#102) 2025-03-21 23:06:31 +02:00
Flaminel
b8ce225ccc Fix Deluge service crashing when download is not found (#97) 2025-03-20 00:09:58 +02:00
Flaminel
f21f7388b7 Add download client customizable url base (#43) 2025-03-20 00:09:24 +02:00
Flaminel
a1354f231a Add base path support for arrs (#96) 2025-03-20 00:08:51 +02:00
Flaminel
4bc1c33e81 Add option to explicitly disable the download client (#93) 2025-03-19 16:02:46 +02:00
Flaminel
32bcbab523 added docs for FreeBSD 2025-03-19 01:26:04 +02:00
Flaminel
b94ae21e11 update permissive blacklist 2025-03-13 10:16:52 +02:00
Flaminel
a92ebd75c2 Update docs (#88) 2025-03-11 23:42:21 +02:00
Flaminel
e6d3929fc9 Restrict max strikes to a minimum value (#87) 2025-03-11 23:35:07 +02:00
Flaminel
a68e13af35 Fix notifications when poster is not found (#89) 2025-03-11 23:34:44 +02:00
Flaminel
324c3ace8f Fix multiple runs on queue cleaner when download cleaner is enabled (#90) 2025-03-11 23:34:27 +02:00
Flaminel
3a9d5d9085 Fix patterns being loaded for disabled arrs (#80) 2025-03-11 23:18:34 +02:00
Flaminel
89a6eaf0ce Disable cleanup on torrent items if download client is not configured (#85) 2025-03-10 00:13:40 +02:00
Flaminel
027c4a0f4d Add option to ignore specific downloads (#79) 2025-03-09 23:38:27 +02:00
Flaminel
81990c6768 fixed missing README link 2025-03-03 22:37:22 +02:00
Flaminel
ba02aa0e49 Fix notifications failing when poster image is not set (#78) 2025-03-02 22:48:21 +02:00
Flaminel
5adbdbd920 Fix weird time zone display name on startup (#70) 2025-02-25 21:32:19 +02:00
Flaminel
b3b211d956 Add configurable time zone (#69) 2025-02-24 23:21:44 +02:00
Flaminel
279bd6d82d Fix Deluge timeout not being configurable (#68) 2025-02-24 18:32:44 +02:00
Flaminel
5dced28228 fixed errors on download cleaner when download client is none (#67) 2025-02-24 12:43:06 +02:00
57 changed files with 1948 additions and 623 deletions

View File

@@ -18,7 +18,7 @@ body:
required: true
- label: Ensured I am using the latest version.
required: true
- label: Enabled debug logging.
- label: Enabled verbose logging.
required: true
- type: textarea
id: what-happened

View File

@@ -18,7 +18,7 @@ body:
required: true
- label: Ensured I am using the latest version.
required: true
- label: Enabled debug logging.
- label: Enabled verbose logging.
required: true
- type: textarea
id: description

188
README.md
View File

@@ -16,6 +16,7 @@ cleanuperr was created primarily to address malicious files, such as `*.lnk` or
> - Trigger a search for downloads removed from the *arrs.
> - Clean up downloads that have been seeding for a certain amount of time.
> - Notify on strike or download removal.
> - Ignore certain torrent hashes, categories, tags or trackers from processing.
cleanuperr supports both qBittorrent's built-in exclusion features and its own blocklist-based system. Binaries for all platforms are provided, along with Docker images for easy deployment.
@@ -25,15 +26,22 @@ cleanuperr supports both qBittorrent's built-in exclusion features and its own b
> https://discord.gg/sWggpnmGNY
## Table of contents:
- [Naming choice](README.md#naming-choice)
- [Quick Start](README.md#quick-start)
- [How it works](README.md#how-it-works)
- [Setup](README.md#setup)
- [Usage](README.md#usage)
- [Docker Compose](README.md#docker-compose-yaml)
- [Environment Variables](README.md#environment-variables)
- [Binaries](README.md#binaries-if-youre-not-using-docker)
- [Credits](README.md#credits)
- [Naming choice](#naming-choice)
- [Quick Start](#quick-start)
- [How it works](#how-it-works)
- [Content blocker](#1-content-blocker-will)
- [Queue cleaner](#2-queue-cleaner-will)
- [Download cleaner](#3-download-cleaner-will)
- [Setup](#setup-examples)
- [Usage](#usage)
- [Docker](#docker)
- [Environment Variables](#environment-variables)
- [Docker Compose](#docker-compose-example)
- [Windows](#windows)
- [Linux](#linux)
- [MacOS](#macos)
- [FreeBSD](#freebsd)
- [Credits](#credits)
## Naming choice
@@ -52,10 +60,10 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
> Use the Unraid Community App.
>
> 3. **Manual Installation (if you're not using Docker)**
> More details [here](#binaries-if-youre-not-using-docker).
> Go to [Windows](#windows), [Linux](#linux) or [MacOS](#macos).
> [!TIP]
> Refer to the [Environment variables](#Environment-variables) section for detailed configuration instructions and the [Setup](#Setup) section for an in-depth explanation of the cleanup process.
> Refer to the [Environment variables](#environment-variables) section for detailed configuration instructions and the [Setup examples](#setup-examples) section for an in-depth explanation of the cleanup process.
> [!IMPORTANT]
@@ -69,7 +77,7 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
# How it works
1. **Content blocker** will:
#### 1. **Content blocker** will:
- Run every 5 minutes (or configured cron).
- Process all items in the *arr queue.
- Find the corresponding item from the download client for each queue item.
@@ -80,24 +88,24 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
- It will be removed from the *arr's queue and blocked.
- It will be deleted from the download client.
- A new search will be triggered for the *arr item.
2. **Queue cleaner** will:
#### 2. **Queue cleaner** will:
- Run every 5 minutes (or configured cron, or right after `content blocker`).
- Process all items in the *arr queue.
- Check each queue item if it is **stalled (download speed is 0)**, **stuck in metadata downloading** or **failed to be imported**.
- If it is, the item receives a **strike** and will continue to accumulate strikes every time it meets any of these conditions.
- Check each queue item if it meets one of the following condition in the download client:
- **Marked as completed, but 0 bytes have been downloaded** (due to files being blocked by qBittorrent or the **content blocker**).
- All associated files of are marked as **unwanted/skipped**.
- All associated files are marked as **unwanted/skipped/do not download**.
- If the item **DOES NOT** match the above criteria, it will be skipped.
- If the item **DOES** match the criteria or has received the **maximum number of strikes**:
- It will be removed from the *arr's queue and blocked.
- It will be deleted from the download client.
- A new search will be triggered for the *arr item.
3. **Download cleaner** will:
#### 3. **Download cleaner** will:
- Run every hour (or configured cron).
- Automatically clean up downloads that have been seeding for a certain amount of time.
# Setup
# Setup examples
## Using qBittorrent's built-in feature (works only with qBittorrent)
@@ -112,7 +120,7 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
## Using cleanuperr's blocklist (works with all supported download clients)
1. Set both `QUEUECLEANER__ENABLED` and `CONTENTBLOCKER__ENABLED` to `true` in your environment variables.
2. Configure and enable either a **blacklist** or a **whitelist** as described in the [Arr variables](#Arr-variables) section.
2. Configure and enable either a **blacklist** or a **whitelist** as described in the [Arr variables](variables.md#Arr-settings) section.
3. Once configured, cleanuperr will perform the following tasks:
- Execute the **content blocker** job, as explained in the [How it works](#how-it-works) section.
- Execute the **queue cleaner** job, as explained in the [How it works](#how-it-works) section.
@@ -129,7 +137,26 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
## Usage
### Docker compose yaml
### <img src="https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/docker.svg" height="20" style="vertical-align: middle;"> <span style="vertical-align: middle;">Docker</span>
### **Environment variables**
**Jump to:**
- [General settings](variables.md#general-settings)
- [Queue Cleaner settings](variables.md#queue-cleaner-settings)
- [Content Blocker settings](variables.md#content-blocker-settings)
- [Download Cleaner settings](variables.md#download-cleaner-settings)
- [Download Client settings](variables.md#download-client-settings)
- [Arr settings](variables.md#arr-settings)
- [Notification settings](variables.md#notification-settings)
- [Advanced settings](variables.md#advanced-settings)
### Docker compose example
> [!NOTE]
>
> This example contains all settings and should be modified to fit your needs.
```
version: "3.3"
@@ -139,7 +166,9 @@ services:
restart: unless-stopped
volumes:
- ./cleanuperr/logs:/var/logs
- ./cleanuperr/ignored.txt:/ignored.txt
environment:
- TZ=America/New_York
- DRY_RUN=false
- LOGGING__LOGLEVEL=Information
@@ -152,6 +181,7 @@ services:
- TRIGGERS__DOWNLOADCLEANER=0 0 * * * ?
- QUEUECLEANER__ENABLED=true
- QUEUECLEANER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- QUEUECLEANER__RUNSEQUENTIALLY=true
- QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE=false
@@ -164,10 +194,12 @@ services:
- QUEUECLEANER__STALLED_DELETE_PRIVATE=false
- CONTENTBLOCKER__ENABLED=true
- CONTENTBLOCKER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- CONTENTBLOCKER__IGNORE_PRIVATE=false
- CONTENTBLOCKER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__ENABLED=true
- DOWNLOADCLEANER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- DOWNLOADCLEANER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__CATEGORIES__0__NAME=tv-sonarr
- DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO=-1
@@ -182,15 +214,18 @@ services:
# OR
# - DOWNLOAD_CLIENT=qBittorrent
# - QBITTORRENT__URL=http://localhost:8080
# - QBITTORRENT__URL_BASE=myCustomPath
# - QBITTORRENT__USERNAME=user
# - QBITTORRENT__PASSWORD=pass
# OR
# - DOWNLOAD_CLIENT=deluge
# - DELUGE__URL_BASE=myCustomPath
# - DELUGE__URL=http://localhost:8112
# - DELUGE__PASSWORD=testing
# OR
# - DOWNLOAD_CLIENT=transmission
# - TRANSMISSION__URL=http://localhost:9091
# - TRANSMISSION__URL_BASE=myCustomPath
# - TRANSMISSION__USERNAME=test
# - TRANSMISSION__PASSWORD=testing
@@ -227,27 +262,109 @@ services:
- NOTIFIARR__CHANNEL_ID=discord_channel_id
```
## Environment variables
### <img src="https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/windows.svg" height="20" style="vertical-align: middle;"> <span style="vertical-align: middle;">Windows</span>
Jump to:
- [General settings](variables.md#general-settings)
- [Queue Cleaner settings](variables.md#queue-cleaner-settings)
- [Content Blocker settings](variables.md#content-blocker-settings)
- [Download Cleaner settings](variables.md#download-cleaner-settings)
- [Download Client settings](variables.md#download-client-settings)
- [Arr settings](variables.md#arr-settings)
- [Notification settings](variables.md#notification-settings)
- [Advanced settings](variables.md#advanced-settings)
### Binaries (if you're not using Docker)
1. Download the binaries from [releases](https://github.com/flmorg/cleanuperr/releases).
2. Extract them from the zip file.
3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [above](#environment-variables).
1. Download the zip file from [releases](https://github.com/flmorg/cleanuperr/releases).
2. Extract the zip file into `C:\example\directory`.
3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [here](#environment-variables).
4. Execute `cleanuperr.exe`.
> [!TIP]
> ### Run as a Windows Service
> Check out this stackoverflow answer on how to do it: https://stackoverflow.com/a/15719678
> 1. Download latest nssm build from `https://nssm.cc/builds`.
> 2. Unzip `nssm.exe` in `C:\example\directory`.
> 3. Open a terminal with Administrator rights and execute these commands:
> ```
> nssm.exe install Cleanuperr "C:\example\directory\cleanuperr.exe"
> nssm.exe set Cleanuperr AppDirectory "C:\example\directory\"
> nssm.exe set Cleanuperr AppStdout "C:\example\directory\cleanuperr.log"
> nssm.exe set Cleanuperr AppStderr "C:\example\directory\cleanuperr.crash.log"
> nssm.exe set Cleanuperr AppRotateFiles 1
> nssm.exe set Cleanuperr AppRotateOnline 1
> nssm.exe set Cleanuperr AppRotateBytes 10485760
> nssm.exe set Cleanuperr AppRotateFiles 10
> nssm.exe set Cleanuperr Start SERVICE_AUTO_START
> nssm.exe start Cleanuperr
> ```
### <img src="https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/linux.svg" height="20" style="vertical-align: middle;"> <span style="vertical-align: middle;">Linux</span>
1. Download the zip file from [releases](https://github.com/flmorg/cleanuperr/releases).
2. Extract the zip file into `/example/directory`.
3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [here](#environment-variables).
4. Open a terminal and execute these commands:
```
cd /example/directory
chmod +x cleanuperr
./cleanuperr
```
### <img src="https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/apple.svg" height="20" style="vertical-align: middle;"> <span style="vertical-align: middle;">MacOS</span>
1. Download the zip file from [releases](https://github.com/flmorg/cleanuperr/releases).
2. Extract the zip file into `/example/directory`.
3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [here](#environment-variables).
4. Open a terminal and execute these commands:
```
cd /example/directory
chmod +x cleanuperr
./cleanuperr
```
> [!IMPORTANT]
> Some people have experienced problems when trying to execute cleanuperr on MacOS because the system actively blocked the file for not being signed.
> As per [this comment](https://stackoverflow.com/a/77907937), you may need to also execute this command:
> ```
> codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime /example/directory/cleanuperr
> ```
### <img src="https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/freebsd.svg" height="20" style="vertical-align: middle;"> <span style="vertical-align: middle;">FreeBSD</span>
1. Installation:
```
# install dependencies
pkg install -y git icu libinotify libunwind wget
# set up the dotnet SDK
cd ~
wget -q https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/download/v9.0.104-amd64-freebsd-14/dotnet-sdk-9.0.104-freebsd-x64.tar.gz
export DOTNET_ROOT=$(pwd)/.dotnet
mkdir -p "$DOTNET_ROOT" && tar zxf dotnet-sdk-9.0.104-freebsd-x64.tar.gz -C "$DOTNET_ROOT"
export PATH=$PATH:$DOTNET_ROOT:$DOTNET_ROOT/tools
# download NuGet dependencies
mkdir -p /tmp/nuget
wget -q -P /tmp/nuget/ https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/download/v9.0.104-amd64-freebsd-14/Microsoft.AspNetCore.App.Runtime.freebsd-x64.9.0.3.nupkg
wget -q -P /tmp/nuget/ https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/download/v9.0.104-amd64-freebsd-14/Microsoft.NETCore.App.Host.freebsd-x64.9.0.3.nupkg
wget -q -P /tmp/nuget/ https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/download/v9.0.104-amd64-freebsd-14/Microsoft.NETCore.App.Runtime.freebsd-x64.9.0.3.nupkg
# add NuGet source
dotnet nuget add source /tmp/nuget --name tmp
# add GitHub NuGet source
# a PAT (Personal Access Token) can be generated here https://github.com/settings/tokens
dotnet nuget add source --username <YOUR_USERNAME> --password <YOUR_PERSONAL_ACCESS_TOKEN> --store-password-in-clear-text --name flmorg https://nuget.pkg.github.com/flmorg/index.json
```
2. Building:
```
# clone the project
git clone https://github.com/flmorg/cleanuperr.git
cd cleanuperr
# build and publish the app
dotnet publish code/Executable/Executable.csproj -c Release --self-contained -o artifacts /p:PublishSingleFile=true
# move the files to permanent destination
mv artifacts/cleanuperr /example/directory/
mv artifacts/appsettings.json /example/directory/
```
3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [here](#environment-variables).
4. Run the app:
```
cd /example/directory
chmod +x cleanuperr
./cleanuperr
```
# Credits
Special thanks for inspiration go to:
@@ -260,4 +377,3 @@ Special thanks for inspiration go to:
If I made your life just a tiny bit easier, consider buying me a coffee!
<a href="https://buymeacoffee.com/flaminel" target="_blank"><img src="https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png" alt="Buy Me A Coffee" style="height: 41px !important;width: 174px !important;box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;-webkit-box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;" ></a>

View File

@@ -1,4 +1,5 @@
*.apk
*.arj
*.bat
*.bin
*.bmp

View File

@@ -14,7 +14,7 @@ deployment:
value: "false"
- name: LOGGING__LOGLEVEL
value: Debug
value: Verbose
- name: LOGGING__FILE__ENABLED
value: "true"
- name: LOGGING__FILE__PATH

View File

@@ -2,7 +2,7 @@
namespace Common.Configuration.ContentBlocker;
public sealed record ContentBlockerConfig : IJobConfig
public sealed record ContentBlockerConfig : IJobConfig, IIgnoredDownloadsConfig
{
public const string SectionName = "ContentBlocker";
@@ -13,6 +13,9 @@ public sealed record ContentBlockerConfig : IJobConfig
[ConfigurationKeyName("DELETE_PRIVATE")]
public bool DeletePrivate { get; init; }
[ConfigurationKeyName("IGNORED_DOWNLOADS_PATH")]
public string? IgnoredDownloadsPath { get; init; }
public void Validate()
{

View File

@@ -3,7 +3,7 @@ using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadCleaner;
public sealed record DownloadCleanerConfig : IJobConfig
public sealed record DownloadCleanerConfig : IJobConfig, IIgnoredDownloadsConfig
{
public const string SectionName = "DownloadCleaner";
@@ -12,7 +12,10 @@ public sealed record DownloadCleanerConfig : IJobConfig
public List<Category>? Categories { get; init; }
[ConfigurationKeyName("DELETE_PRIVATE")]
public bool DeletePrivate { get; set; }
public bool DeletePrivate { get; init; }
[ConfigurationKeyName("IGNORED_DOWNLOADS_PATH")]
public string? IgnoredDownloadsPath { get; init; }
public void Validate()
{

View File

@@ -1,4 +1,5 @@
using Common.Exceptions;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadClient;
@@ -8,6 +9,9 @@ public sealed record DelugeConfig : IConfig
public Uri? Url { get; init; }
[ConfigurationKeyName("URL_BASE")]
public string UrlBase { get; init; } = string.Empty;
public string? Password { get; init; }
public void Validate()

View File

@@ -1,4 +1,5 @@
using Common.Exceptions;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadClient;
@@ -8,6 +9,9 @@ public sealed class QBitConfig : IConfig
public Uri? Url { get; init; }
[ConfigurationKeyName("URL_BASE")]
public string UrlBase { get; init; } = string.Empty;
public string? Username { get; init; }
public string? Password { get; init; }

View File

@@ -1,4 +1,5 @@
using Common.Exceptions;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadClient;
@@ -8,6 +9,9 @@ public record TransmissionConfig : IConfig
public Uri? Url { get; init; }
[ConfigurationKeyName("URL_BASE")]
public string UrlBase { get; init; } = "transmission";
public string? Username { get; init; }
public string? Password { get; init; }

View File

@@ -0,0 +1,6 @@
namespace Common.Configuration;
public interface IIgnoredDownloadsConfig
{
string? IgnoredDownloadsPath { get; }
}

View File

@@ -1,8 +1,10 @@
using Microsoft.Extensions.Configuration;
using Common.CustomDataTypes;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.QueueCleaner;
public sealed record QueueCleanerConfig : IJobConfig
public sealed record QueueCleanerConfig : IJobConfig, IIgnoredDownloadsConfig
{
public const string SectionName = "QueueCleaner";
@@ -10,6 +12,9 @@ public sealed record QueueCleanerConfig : IJobConfig
public required bool RunSequentially { get; init; }
[ConfigurationKeyName("IGNORED_DOWNLOADS_PATH")]
public string? IgnoredDownloadsPath { get; init; }
[ConfigurationKeyName("IMPORT_FAILED_MAX_STRIKES")]
public ushort ImportFailedMaxStrikes { get; init; }
@@ -33,8 +38,74 @@ public sealed record QueueCleanerConfig : IJobConfig
[ConfigurationKeyName("STALLED_DELETE_PRIVATE")]
public bool StalledDeletePrivate { get; init; }
[ConfigurationKeyName("SLOW_MAX_STRIKES")]
public ushort SlowMaxStrikes { get; init; }
[ConfigurationKeyName("SLOW_RESET_STRIKES_ON_PROGRESS")]
public bool SlowResetStrikesOnProgress { get; init; }
[ConfigurationKeyName("SLOW_IGNORE_PRIVATE")]
public bool SlowIgnorePrivate { get; init; }
[ConfigurationKeyName("SLOW_DELETE_PRIVATE")]
public bool SlowDeletePrivate { get; init; }
[ConfigurationKeyName("SLOW_MIN_SPEED")]
public string SlowMinSpeed { get; init; } = string.Empty;
public ByteSize SlowMinSpeedByteSize => string.IsNullOrEmpty(SlowMinSpeed) ? new ByteSize(0) : ByteSize.Parse(SlowMinSpeed);
[ConfigurationKeyName("SLOW_MAX_TIME")]
public double SlowMaxTime { get; init; }
[ConfigurationKeyName("SLOW_IGNORE_ABOVE_SIZE")]
public string SlowIgnoreAboveSize { get; init; } = string.Empty;
public ByteSize? SlowIgnoreAboveSizeByteSize => string.IsNullOrEmpty(SlowIgnoreAboveSize) ? null : ByteSize.Parse(SlowIgnoreAboveSize);
public void Validate()
{
if (ImportFailedMaxStrikes is > 0 and < 3)
{
throw new ValidationException($"the minimum value for {SectionName}__IMPORT_FAILED_MAX_STRIKES must be 3");
}
if (StalledMaxStrikes is > 0 and < 3)
{
throw new ValidationException($"the minimum value for {SectionName}__STALLED_MAX_STRIKES must be 3");
}
if (SlowMaxStrikes is > 0 and < 3)
{
throw new ValidationException($"the minimum value for {SectionName}__SLOW_MAX_STRIKES must be 3");
}
if (SlowMaxStrikes > 0)
{
bool isSlowSpeedSet = !string.IsNullOrEmpty(SlowMinSpeed);
if (isSlowSpeedSet && ByteSize.TryParse(SlowMinSpeed, out _) is false)
{
throw new ValidationException($"invalid value for {SectionName}__SLOW_MIN_SPEED");
}
if (SlowMaxTime < 0)
{
throw new ValidationException($"invalid value for {SectionName}__SLOW_MAX_TIME");
}
if (!isSlowSpeedSet && SlowMaxTime is 0)
{
throw new ValidationException($"either {SectionName}__SLOW_MIN_SPEED or {SectionName}__SLOW_MAX_STRIKES must be set");
}
bool isSlowIgnoreAboveSizeSet = !string.IsNullOrEmpty(SlowIgnoreAboveSize);
if (isSlowIgnoreAboveSizeSet && ByteSize.TryParse(SlowIgnoreAboveSize, out _) is false)
{
throw new ValidationException($"invalid value for {SectionName}__SLOW_IGNORE_ABOVE_SIZE");
}
}
}
}

View File

@@ -0,0 +1,115 @@
using System.Globalization;
namespace Common.CustomDataTypes;
public readonly struct ByteSize : IComparable<ByteSize>, IEquatable<ByteSize>
{
public long Bytes { get; }
private const long BytesPerKB = 1024;
private const long BytesPerMB = 1024 * 1024;
private const long BytesPerGB = 1024 * 1024 * 1024;
public ByteSize(long bytes)
{
if (bytes < 0)
{
throw new ArgumentOutOfRangeException(nameof(bytes), "bytes can not be negative");
}
Bytes = bytes;
}
public static ByteSize FromKilobytes(double kilobytes) => new((long)(kilobytes * BytesPerKB));
public static ByteSize FromMegabytes(double megabytes) => new((long)(megabytes * BytesPerMB));
public static ByteSize FromGigabytes(double gigabytes) => new((long)(gigabytes * BytesPerGB));
public static ByteSize Parse(string input)
{
if (string.IsNullOrWhiteSpace(input))
{
throw new ArgumentNullException(nameof(input));
}
input = input.Trim().ToUpperInvariant();
double value;
if (input.EndsWith("KB", StringComparison.InvariantCultureIgnoreCase))
{
value = double.Parse(input[..^2], CultureInfo.InvariantCulture);
return FromKilobytes(value);
}
if (input.EndsWith("MB", StringComparison.InvariantCultureIgnoreCase))
{
value = double.Parse(input[..^2], CultureInfo.InvariantCulture);
return FromMegabytes(value);
}
if (input.EndsWith("GB", StringComparison.InvariantCultureIgnoreCase))
{
value = double.Parse(input[..^2], CultureInfo.InvariantCulture);
return FromGigabytes(value);
}
throw new FormatException("invalid size format | only KB, MB and GB are supported");
}
public static bool TryParse(string? input, out ByteSize? result)
{
result = default;
if (string.IsNullOrWhiteSpace(input))
{
return false;
}
input = input.Trim().ToUpperInvariant();
if (input.EndsWith("KB", StringComparison.InvariantCultureIgnoreCase) &&
double.TryParse(input[..^2], NumberStyles.Float, CultureInfo.InvariantCulture, out double kb))
{
result = FromKilobytes(kb);
return true;
}
if (input.EndsWith("MB", StringComparison.InvariantCultureIgnoreCase) &&
double.TryParse(input[..^2], NumberStyles.Float, CultureInfo.InvariantCulture, out double mb))
{
result = FromMegabytes(mb);
return true;
}
if (input.EndsWith("GB", StringComparison.InvariantCultureIgnoreCase) &&
double.TryParse(input[..^2], NumberStyles.Float, CultureInfo.InvariantCulture, out double gb))
{
result = FromGigabytes(gb);
return true;
}
return false;
}
public override string ToString() =>
Bytes switch
{
>= BytesPerGB => $"{Bytes / (double)BytesPerGB:0.##} GB",
>= BytesPerMB => $"{Bytes / (double)BytesPerMB:0.##} MB",
_ => $"{Bytes / (double)BytesPerKB:0.##} KB"
};
public int CompareTo(ByteSize other) => Bytes.CompareTo(other.Bytes);
public bool Equals(ByteSize other) => Bytes == other.Bytes;
public override bool Equals(object? obj) => obj is ByteSize other && Equals(other);
public override int GetHashCode() => Bytes.GetHashCode();
public static bool operator ==(ByteSize left, ByteSize right) => left.Equals(right);
public static bool operator !=(ByteSize left, ByteSize right) => !(left == right);
public static bool operator <(ByteSize left, ByteSize right) => left.Bytes < right.Bytes;
public static bool operator >(ByteSize left, ByteSize right) => left.Bytes > right.Bytes;
public static bool operator <=(ByteSize left, ByteSize right) => left.Bytes <= right.Bytes;
public static bool operator >=(ByteSize left, ByteSize right) => left.Bytes >= right.Bytes;
public static ByteSize operator +(ByteSize left, ByteSize right) => new(left.Bytes + right.Bytes);
public static ByteSize operator -(ByteSize left, ByteSize right) => new(Math.Max(left.Bytes - right.Bytes, 0));
}

View File

@@ -0,0 +1,66 @@
using System.Text;
namespace Common.CustomDataTypes;
public readonly struct SmartTimeSpan : IComparable<SmartTimeSpan>, IEquatable<SmartTimeSpan>
{
public TimeSpan Time { get; }
public SmartTimeSpan(TimeSpan time)
{
Time = time;
}
public override string ToString()
{
if (Time == TimeSpan.Zero)
{
return "0 seconds";
}
StringBuilder sb = new();
if (Time.Days > 0)
{
sb.Append($"{Time.Days} day{(Time.Days > 1 ? "s" : "")} ");
}
if (Time.Hours > 0)
{
sb.Append($"{Time.Hours} hour{(Time.Hours > 1 ? "s" : "")} ");
}
if (Time.Minutes > 0)
{
sb.Append($"{Time.Minutes} minute{(Time.Minutes > 1 ? "s" : "")} ");
}
if (Time.Seconds > 0)
{
sb.Append($"{Time.Seconds} second{(Time.Seconds > 1 ? "s" : "")}");
}
return sb.ToString().TrimEnd();
}
public static SmartTimeSpan FromMinutes(double minutes) => new(TimeSpan.FromMinutes(minutes));
public static SmartTimeSpan FromSeconds(double seconds) => new(TimeSpan.FromSeconds(seconds));
public static SmartTimeSpan FromHours(double hours) => new(TimeSpan.FromHours(hours));
public static SmartTimeSpan FromDays(double days) => new(TimeSpan.FromDays(days));
public int CompareTo(SmartTimeSpan other) => Time.CompareTo(other.Time);
public bool Equals(SmartTimeSpan other) => Time.Equals(other.Time);
public override bool Equals(object? obj) => obj is SmartTimeSpan other && Equals(other);
public override int GetHashCode() => Time.GetHashCode();
public static bool operator ==(SmartTimeSpan left, SmartTimeSpan right) => left.Equals(right);
public static bool operator !=(SmartTimeSpan left, SmartTimeSpan right) => !left.Equals(right);
public static bool operator <(SmartTimeSpan left, SmartTimeSpan right) => left.Time < right.Time;
public static bool operator >(SmartTimeSpan left, SmartTimeSpan right) => left.Time > right.Time;
public static bool operator <=(SmartTimeSpan left, SmartTimeSpan right) => left.Time <= right.Time;
public static bool operator >=(SmartTimeSpan left, SmartTimeSpan right) => left.Time >= right.Time;
public static SmartTimeSpan operator +(SmartTimeSpan left, SmartTimeSpan right) => new(left.Time + right.Time);
public static SmartTimeSpan operator -(SmartTimeSpan left, SmartTimeSpan right) => new(left.Time - right.Time);
}

View File

@@ -5,5 +5,6 @@ public enum DownloadClient
QBittorrent,
Deluge,
Transmission,
None
None,
Disabled
}

View File

@@ -2,7 +2,13 @@
public enum DeleteReason
{
None,
Stalled,
ImportFailed,
AllFilesBlocked
DownloadingMetadata,
SlowSpeed,
SlowTime,
AllFilesSkipped,
AllFilesSkippedByQBit,
AllFilesBlocked,
}

View File

@@ -3,5 +3,8 @@
public enum StrikeType
{
Stalled,
ImportFailed
DownloadingMetadata,
ImportFailed,
SlowSpeed,
SlowTime,
}

View File

@@ -1,6 +1,6 @@
namespace Domain.Models.Cache;
public sealed record CacheItem
public sealed record StalledCacheItem
{
/// <summary>
/// The amount of bytes that have been downloaded.

View File

@@ -2,7 +2,7 @@
namespace Domain.Models.Deluge.Response;
public sealed record TorrentStatus
public sealed record DownloadStatus
{
public string? Hash { get; init; }
@@ -12,8 +12,14 @@ public sealed record TorrentStatus
public ulong Eta { get; init; }
[JsonProperty("download_payload_rate")]
public long DownloadSpeed { get; init; }
public bool Private { get; init; }
[JsonProperty("total_size")]
public long Size { get; init; }
[JsonProperty("total_done")]
public long TotalDone { get; init; }
@@ -23,4 +29,11 @@ public sealed record TorrentStatus
public long SeedingTime { get; init; }
public float Ratio { get; init; }
public required IReadOnlyList<Tracker> Trackers { get; init; }
}
public sealed record Tracker
{
public required Uri Url { get; init; }
}

View File

@@ -62,7 +62,7 @@ public static class MainDI
services
.AddHttpClient(nameof(DelugeService), x =>
{
x.Timeout = TimeSpan.FromSeconds(5);
x.Timeout = TimeSpan.FromSeconds(config.Timeout);
})
.ConfigurePrimaryHttpMessageHandler(_ =>
{

View File

@@ -55,7 +55,7 @@ public static class QuartzDI
if (contentBlockerConfig?.Enabled is true && queueCleanerConfig is { Enabled: true, RunSequentially: true })
{
q.AddJob<QueueCleaner>(queueCleanerConfig, string.Empty);
q.AddJobListener(new JobChainingListener(nameof(QueueCleaner)));
q.AddJobListener(new JobChainingListener(nameof(ContentBlocker), nameof(QueueCleaner)));
}
else
{

View File

@@ -1,4 +1,8 @@
using Infrastructure.Interceptors;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Infrastructure.Interceptors;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.DownloadCleaner;
@@ -30,5 +34,8 @@ public static class ServicesDI
.AddTransient<ArrQueueIterator>()
.AddTransient<DownloadServiceFactory>()
.AddSingleton<BlocklistProvider>()
.AddSingleton<IStriker, Striker>();
.AddSingleton<IStriker, Striker>()
.AddSingleton<IgnoredDownloadsProvider<QueueCleanerConfig>>()
.AddSingleton<IgnoredDownloadsProvider<ContentBlockerConfig>>()
.AddSingleton<IgnoredDownloadsProvider<DownloadCleanerConfig>>();
}

View File

@@ -0,0 +1,23 @@
using System.Reflection;
namespace Executable;
public static class HostExtensions
{
public static IHost Init(this IHost host)
{
ILogger<Program> logger = host.Services.GetRequiredService<ILogger<Program>>();
Version? version = Assembly.GetExecutingAssembly().GetName().Version;
logger.LogInformation(
version is null
? "cleanuperr version not detected"
: $"cleanuperr v{version.Major}.{version.Minor}.{version.Build}"
);
logger.LogInformation("timezone: {tz}", TimeZoneInfo.Local.DisplayName);
return host;
}
}

View File

@@ -1,4 +1,4 @@
using System.Reflection;
using Executable;
using Executable.DependencyInjection;
var builder = Host.CreateApplicationBuilder(args);
@@ -7,15 +7,6 @@ builder.Services.AddInfrastructure(builder.Configuration);
builder.Logging.AddLogging(builder.Configuration);
var host = builder.Build();
var logger = host.Services.GetRequiredService<ILogger<Program>>();
var version = Assembly.GetExecutingAssembly().GetName().Version;
logger.LogInformation(
version is null
? "cleanuperr version not detected"
: $"cleanuperr v{version.Major}.{version.Minor}.{version.Build}"
);
host.Init();
host.Run();

View File

@@ -3,7 +3,7 @@
"HTTP_MAX_RETRIES": 0,
"HTTP_TIMEOUT": 10,
"Logging": {
"LogLevel": "Debug",
"LogLevel": "Verbose",
"Enhanced": true,
"File": {
"Enabled": false,
@@ -18,11 +18,13 @@
"ContentBlocker": {
"Enabled": true,
"IGNORE_PRIVATE": true,
"DELETE_PRIVATE": false
"DELETE_PRIVATE": false,
"IGNORED_DOWNLOADS_PATH": "../test/data/cleanuperr/ignored_downloads"
},
"QueueCleaner": {
"Enabled": true,
"RunSequentially": true,
"IGNORED_DOWNLOADS_PATH": "../test/data/cleanuperr/ignored_downloads",
"IMPORT_FAILED_MAX_STRIKES": 5,
"IMPORT_FAILED_IGNORE_PRIVATE": true,
"IMPORT_FAILED_DELETE_PRIVATE": false,
@@ -32,7 +34,14 @@
"STALLED_MAX_STRIKES": 5,
"STALLED_RESET_STRIKES_ON_PROGRESS": true,
"STALLED_IGNORE_PRIVATE": true,
"STALLED_DELETE_PRIVATE": false
"STALLED_DELETE_PRIVATE": false,
"SLOW_MAX_STRIKES": 5,
"SLOW_RESET_STRIKES_ON_PROGRESS": true,
"SLOW_IGNORE_PRIVATE": false,
"SLOW_DELETE_PRIVATE": false,
"SLOW_MIN_SPEED": "1MB",
"SLOW_MAX_TIME": 20,
"SLOW_IGNORE_ABOVE_SIZE": "4GB"
},
"DownloadCleaner": {
"Enabled": false,
@@ -44,20 +53,24 @@
"MIN_SEED_TIME": 0,
"MAX_SEED_TIME": -1
}
]
],
"IGNORED_DOWNLOADS_PATH": "../test/data/cleanuperr/ignored_downloads"
},
"DOWNLOAD_CLIENT": "qbittorrent",
"qBittorrent": {
"Url": "http://localhost:8080",
"URL_BASE": "",
"Username": "test",
"Password": "testing"
},
"Deluge": {
"Url": "http://localhost:8112",
"URL_BASE": "",
"Password": "testing"
},
"Transmission": {
"Url": "http://localhost:9091",
"URL_BASE": "transmission",
"Username": "test",
"Password": "testing"
},

View File

@@ -17,11 +17,13 @@
},
"ContentBlocker": {
"Enabled": false,
"IGNORE_PRIVATE": false
"IGNORE_PRIVATE": false,
"IGNORED_DOWNLOADS_PATH": ""
},
"QueueCleaner": {
"Enabled": true,
"RunSequentially": true,
"IGNORED_DOWNLOADS_PATH": "",
"IMPORT_FAILED_MAX_STRIKES": 0,
"IMPORT_FAILED_IGNORE_PRIVATE": false,
"IMPORT_FAILED_DELETE_PRIVATE": false,
@@ -34,20 +36,24 @@
"DownloadCleaner": {
"Enabled": false,
"DELETE_PRIVATE": false,
"CATEGORIES": []
"CATEGORIES": [],
"IGNORED_DOWNLOADS_PATH": ""
},
"DOWNLOAD_CLIENT": "none",
"qBittorrent": {
"Url": "http://localhost:8080",
"URL_BASE": "",
"Username": "",
"Password": ""
},
"Deluge": {
"Url": "http://localhost:8112",
"URL_BASE": "",
"Password": "testing"
},
"Transmission": {
"Url": "http://localhost:9091",
"URL_BASE": "transmission",
"Username": "test",
"Password": "testing"
},

View File

@@ -39,7 +39,7 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
});
// Act
sut.ResetStrikesOnProgress("test-hash", 100);
sut.ResetStalledStrikesOnProgress("test-hash", 100);
// Assert
_fixture.Cache.ReceivedCalls().ShouldBeEmpty();
@@ -50,19 +50,19 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
{
// Arrange
const string hash = "test-hash";
CacheItem cacheItem = new CacheItem { Downloaded = 100 };
StalledCacheItem stalledCacheItem = new StalledCacheItem { Downloaded = 100 };
_fixture.Cache.TryGetValue(Arg.Any<object>(), out Arg.Any<object?>())
.Returns(x =>
{
x[1] = cacheItem;
x[1] = stalledCacheItem;
return true;
});
TestDownloadService sut = _fixture.CreateSut();
// Act
sut.ResetStrikesOnProgress(hash, 200);
sut.ResetStalledStrikesOnProgress(hash, 200);
// Assert
_fixture.Cache.Received(1).Remove(CacheKeys.Strike(StrikeType.Stalled, hash));
@@ -73,20 +73,20 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
{
// Arrange
const string hash = "test-hash";
CacheItem cacheItem = new CacheItem { Downloaded = 200 };
StalledCacheItem stalledCacheItem = new StalledCacheItem { Downloaded = 200 };
_fixture.Cache
.TryGetValue(Arg.Any<object>(), out Arg.Any<object?>())
.Returns(x =>
{
x[1] = cacheItem;
x[1] = stalledCacheItem;
return true;
});
TestDownloadService sut = _fixture.CreateSut();
// Act
sut.ResetStrikesOnProgress(hash, 100);
sut.ResetStalledStrikesOnProgress(hash, 100);
// Assert
_fixture.Cache.DidNotReceive().Remove(Arg.Any<object>());
@@ -98,27 +98,6 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
public StrikeAndCheckLimitTests(DownloadServiceFixture fixture) : base(fixture)
{
}
[Fact]
public async Task ShouldDelegateCallToStriker()
{
// Arrange
const string hash = "test-hash";
const string itemName = "test-item";
_fixture.Striker.StrikeAndCheckLimit(hash, itemName, 3, StrikeType.Stalled)
.Returns(true);
TestDownloadService sut = _fixture.CreateSut();
// Act
bool result = await sut.StrikeAndCheckLimit(hash, itemName);
// Assert
result.ShouldBeTrue();
await _fixture.Striker
.Received(1)
.StrikeAndCheckLimit(hash, itemName, 3, StrikeType.Stalled);
}
}
public class ShouldCleanDownloadTests : DownloadServiceTests

View File

@@ -3,6 +3,7 @@ using System.Text.RegularExpressions;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Domain.Enums;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.DownloadClient;
@@ -35,15 +36,15 @@ public class TestDownloadService : DownloadService
public override void Dispose() { }
public override Task LoginAsync() => Task.CompletedTask;
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash) => Task.FromResult(new StalledResult());
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType,
ConcurrentBag<string> patterns, ConcurrentBag<Regex> regexes) => Task.FromResult(new BlockFilesResult());
public override Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads) => Task.FromResult(new DownloadCheckResult());
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType,
ConcurrentBag<string> patterns, ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads) => Task.FromResult(new BlockFilesResult());
public override Task DeleteDownload(string hash) => Task.CompletedTask;
public override Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories) => Task.FromResult<List<object>?>(null);
public override Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes) => Task.CompletedTask;
public override Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads) => Task.CompletedTask;
// Expose protected methods for testing
public new void ResetStrikesOnProgress(string hash, long downloaded) => base.ResetStrikesOnProgress(hash, downloaded);
public new Task<bool> StrikeAndCheckLimit(string hash, string itemName) => base.StrikeAndCheckLimit(hash, itemName);
public new void ResetStalledStrikesOnProgress(string hash, long downloaded) => base.ResetStalledStrikesOnProgress(hash, downloaded);
public new SeedingCheckResult ShouldCleanDownload(double ratio, TimeSpan seedingTime, Category category) => base.ShouldCleanDownload(ratio, seedingTime, category);
}

View File

@@ -0,0 +1,29 @@
using Domain.Models.Deluge.Response;
namespace Infrastructure.Extensions;
public static class DelugeExtensions
{
public static bool ShouldIgnore(this DownloadStatus download, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (download.Hash?.Equals(value, StringComparison.InvariantCultureIgnoreCase) ?? false)
{
return true;
}
if (download.Label?.Equals(value, StringComparison.InvariantCultureIgnoreCase) ?? false)
{
return true;
}
if (download.Trackers.Any(x => x.Url.Host.EndsWith(value, StringComparison.InvariantCultureIgnoreCase)))
{
return true;
}
}
return false;
}
}

View File

@@ -0,0 +1,42 @@
using QBittorrent.Client;
namespace Infrastructure.Extensions;
public static class QBitExtensions
{
public static bool ShouldIgnore(this TorrentInfo download, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (download.Hash.Equals(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
if (download.Category.Equals(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
if (download.Tags.Contains(value, StringComparer.InvariantCultureIgnoreCase))
{
return true;
}
}
return false;
}
public static bool ShouldIgnore(this TorrentTracker tracker, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (tracker.Url.Host.EndsWith(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
}
return false;
}
}

View File

@@ -0,0 +1,42 @@
using Transmission.API.RPC.Entity;
namespace Infrastructure.Extensions;
public static class TransmissionExtensions
{
public static bool ShouldIgnore(this TorrentInfo download, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (download.HashString?.Equals(value, StringComparison.InvariantCultureIgnoreCase) ?? false)
{
return true;
}
if (download.GetCategory().Equals(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
bool? hasIgnoredTracker = download.Trackers?
.Any(x => new Uri(x.Announce!).Host.EndsWith(value, StringComparison.InvariantCultureIgnoreCase));
if (hasIgnoredTracker is true)
{
return true;
}
}
return false;
}
public static string GetCategory(this TorrentInfo download)
{
if (string.IsNullOrEmpty(download.DownloadDir))
{
return string.Empty;
}
return Path.GetFileName(Path.TrimEndingDirectorySeparator(download.DownloadDir));
}
}

View File

@@ -10,5 +10,7 @@ public static class CacheKeys
public static string BlocklistPatterns(InstanceType instanceType) => $"{instanceType.ToString()}_patterns";
public static string BlocklistRegexes(InstanceType instanceType) => $"{instanceType.ToString()}_regexes";
public static string Item(string hash) => $"item_{hash}";
public static string StrikeItem(string hash, StrikeType strikeType) => $"item_{hash}_{strikeType.ToString()}";
public static string IgnoredDownloads(string name) => $"{name}_ignored";
}

View File

@@ -13,7 +13,7 @@
<ItemGroup>
<PackageReference Include="FLM.QBittorrent" Version="1.0.0" />
<PackageReference Include="FLM.Transmission" Version="1.0.2" />
<PackageReference Include="FLM.Transmission" Version="1.0.3" />
<PackageReference Include="Mapster" Version="7.4.0" />
<PackageReference Include="MassTransit" Version="8.3.6" />
<PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="9.0.2" />

View File

@@ -0,0 +1,82 @@
using Common.Configuration;
using Infrastructure.Helpers;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
namespace Infrastructure.Providers;
public sealed class IgnoredDownloadsProvider<T>
where T : IIgnoredDownloadsConfig
{
private readonly ILogger<IgnoredDownloadsProvider<T>> _logger;
private IIgnoredDownloadsConfig _config;
private readonly IMemoryCache _cache;
private DateTime _lastModified = DateTime.MinValue;
public IgnoredDownloadsProvider(ILogger<IgnoredDownloadsProvider<T>> logger, IOptionsMonitor<T> config, IMemoryCache cache)
{
_config = config.CurrentValue;
config.OnChange((newValue) => _config = newValue);
_logger = logger;
_cache = cache;
if (string.IsNullOrEmpty(_config.IgnoredDownloadsPath))
{
return;
}
if (!File.Exists(_config.IgnoredDownloadsPath))
{
throw new FileNotFoundException("file not found", _config.IgnoredDownloadsPath);
}
}
public async Task<IReadOnlyList<string>> GetIgnoredDownloads()
{
if (string.IsNullOrEmpty(_config.IgnoredDownloadsPath))
{
return Array.Empty<string>();
}
FileInfo fileInfo = new(_config.IgnoredDownloadsPath);
if (fileInfo.LastWriteTime > _lastModified ||
!_cache.TryGetValue(CacheKeys.IgnoredDownloads(typeof(T).Name), out IReadOnlyList<string>? ignoredDownloads) ||
ignoredDownloads is null)
{
_lastModified = fileInfo.LastWriteTime;
return await LoadFile();
}
return ignoredDownloads;
}
private async Task<IReadOnlyList<string>> LoadFile()
{
try
{
if (string.IsNullOrEmpty(_config.IgnoredDownloadsPath))
{
return Array.Empty<string>();
}
string[] ignoredDownloads = (await File.ReadAllLinesAsync(_config.IgnoredDownloadsPath))
.Where(x => !string.IsNullOrWhiteSpace(x))
.ToArray();
_cache.Set(CacheKeys.IgnoredDownloads(typeof(T).Name), ignoredDownloads);
_logger.LogInformation("ignored downloads reloaded");
return ignoredDownloads;
}
catch (Exception exception)
{
_logger.LogError(exception, "error while reading ignored downloads file | {file}", _config.IgnoredDownloadsPath);
}
return Array.Empty<string>();
}
}

View File

@@ -43,9 +43,11 @@ public abstract class ArrClient : IArrClient
public virtual async Task<QueueListResponse> GetQueueItemsAsync(ArrInstance arrInstance, int page)
{
Uri uri = new(arrInstance.Url, GetQueueUrlPath(page));
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/{GetQueueUrlPath().TrimStart('/')}";
uriBuilder.Query = GetQueueUrlQuery(page);
using HttpRequestMessage request = new(HttpMethod.Get, uri);
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);
@@ -56,7 +58,7 @@ public abstract class ArrClient : IArrClient
}
catch
{
_logger.LogError("queue list failed | {uri}", uri);
_logger.LogError("queue list failed | {uri}", uriBuilder.Uri);
throw;
}
@@ -65,7 +67,7 @@ public abstract class ArrClient : IArrClient
if (queueResponse is null)
{
throw new Exception($"unrecognized queue list response | {uri} | {responseBody}");
throw new Exception($"unrecognized queue list response | {uriBuilder.Uri} | {responseBody}");
}
return queueResponse;
@@ -112,13 +114,20 @@ public abstract class ArrClient : IArrClient
return false;
}
public virtual async Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record, bool removeFromClient)
public virtual async Task DeleteQueueItemAsync(
ArrInstance arrInstance,
QueueRecord record,
bool removeFromClient,
DeleteReason deleteReason
)
{
Uri uri = new(arrInstance.Url, GetQueueDeleteUrlPath(record.Id, removeFromClient));
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/{GetQueueDeleteUrlPath(record.Id).TrimStart('/')}";
uriBuilder.Query = GetQueueDeleteUrlQuery(removeFromClient);
try
{
using HttpRequestMessage request = new(HttpMethod.Delete, uri);
using HttpRequestMessage request = new(HttpMethod.Delete, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
HttpResponseMessage? response = await _dryRunInterceptor.InterceptAsync<HttpResponseMessage>(SendRequestAsync, request);
@@ -126,15 +135,16 @@ public abstract class ArrClient : IArrClient
_logger.LogInformation(
removeFromClient
? "queue item deleted | {url} | {title}"
: "queue item removed from arr | {url} | {title}",
? "queue item deleted with reason {reason} | {url} | {title}"
: "queue item removed from arr with reason {reason} | {url} | {title}",
deleteReason.ToString(),
arrInstance.Url,
record.Title
);
}
catch
{
_logger.LogError("queue delete failed | {uri} | {title}", uri, record.Title);
_logger.LogError("queue delete failed | {uri} | {title}", uriBuilder.Uri, record.Title);
throw;
}
}
@@ -152,9 +162,13 @@ public abstract class ArrClient : IArrClient
return true;
}
protected abstract string GetQueueUrlPath(int page);
protected abstract string GetQueueUrlPath();
protected abstract string GetQueueDeleteUrlPath(long recordId, bool removeFromClient);
protected abstract string GetQueueUrlQuery(int page);
protected abstract string GetQueueDeleteUrlPath(long recordId);
protected abstract string GetQueueDeleteUrlQuery(bool removeFromClient);
protected virtual void SetApiKey(HttpRequestMessage request, string apiKey)
{

View File

@@ -11,7 +11,7 @@ public interface IArrClient
Task<bool> ShouldRemoveFromQueue(InstanceType instanceType, QueueRecord record, bool isPrivateDownload);
Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record, bool removeFromClient);
Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record, bool removeFromClient, DeleteReason deleteReason);
Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items);

View File

@@ -27,29 +27,42 @@ public class LidarrClient : ArrClient, ILidarrClient
{
}
protected override string GetQueueUrlPath(int page)
protected override string GetQueueUrlPath()
{
return $"/api/v1/queue?page={page}&pageSize=200&includeUnknownArtistItems=true&includeArtist=true&includeAlbum=true";
return "/api/v1/queue";
}
protected override string GetQueueDeleteUrlPath(long recordId, bool removeFromClient)
protected override string GetQueueUrlQuery(int page)
{
string path = $"/api/v1/queue/{recordId}?blocklist=true&skipRedownload=true&changeCategory=false";
return $"page={page}&pageSize=200&includeUnknownArtistItems=true&includeArtist=true&includeAlbum=true";
}
path += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
protected override string GetQueueDeleteUrlPath(long recordId)
{
return $"/api/v1/queue/{recordId}";
}
return path;
protected override string GetQueueDeleteUrlQuery(bool removeFromClient)
{
string query = "blocklist=true&skipRedownload=true&changeCategory=false";
query += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return query;
}
public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items)
{
if (items?.Count is null or 0) return;
if (items?.Count is null or 0)
{
return;
}
Uri uri = new(arrInstance.Url, "/api/v1/command");
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v1/command";
foreach (var command in GetSearchCommands(items))
{
using HttpRequestMessage request = new(HttpMethod.Post, uri);
using HttpRequestMessage request = new(HttpMethod.Post, uriBuilder.Uri);
request.Content = new StringContent(
JsonConvert.SerializeObject(command, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }),
Encoding.UTF8,
@@ -132,8 +145,11 @@ public class LidarrClient : ArrClient, ILidarrClient
private async Task<List<Album>?> GetAlbumsAsync(ArrInstance arrInstance, List<long> albumIds)
{
Uri uri = new(arrInstance.Url, $"api/v1/album?{string.Join('&', albumIds.Select(x => $"albumIds={x}"))}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v1/album";
uriBuilder.Query = string.Join('&', albumIds.Select(x => $"albumIds={x}"));
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using var response = await _httpClient.SendAsync(request);

View File

@@ -27,18 +27,27 @@ public class RadarrClient : ArrClient, IRadarrClient
{
}
protected override string GetQueueUrlPath(int page)
protected override string GetQueueUrlPath()
{
return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownMovieItems=true&includeMovie=true";
return "/api/v3/queue";
}
protected override string GetQueueDeleteUrlPath(long recordId, bool removeFromClient)
protected override string GetQueueUrlQuery(int page)
{
string path = $"/api/v3/queue/{recordId}?blocklist=true&skipRedownload=true&changeCategory=false";
path += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return $"page={page}&pageSize=200&includeUnknownMovieItems=true&includeMovie=true";
}
return path;
protected override string GetQueueDeleteUrlPath(long recordId)
{
return $"/api/v3/queue/{recordId}";
}
protected override string GetQueueDeleteUrlQuery(bool removeFromClient)
{
string query = "blocklist=true&skipRedownload=true&changeCategory=false";
query += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return query;
}
public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items)
@@ -50,14 +59,16 @@ public class RadarrClient : ArrClient, IRadarrClient
List<long> ids = items.Select(item => item.Id).ToList();
Uri uri = new(arrInstance.Url, "/api/v3/command");
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/command";
RadarrCommand command = new()
{
Name = "MoviesSearch",
MovieIds = ids,
};
using HttpRequestMessage request = new(HttpMethod.Post, uri);
using HttpRequestMessage request = new(HttpMethod.Post, uriBuilder.Uri);
request.Content = new StringContent(
JsonConvert.SerializeObject(command),
Encoding.UTF8,
@@ -135,8 +146,10 @@ public class RadarrClient : ArrClient, IRadarrClient
private async Task<Movie?> GetMovie(ArrInstance arrInstance, long movieId)
{
Uri uri = new(arrInstance.Url, $"api/v3/movie/{movieId}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/movie/{movieId}";
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);

View File

@@ -28,18 +28,27 @@ public class SonarrClient : ArrClient, ISonarrClient
{
}
protected override string GetQueueUrlPath(int page)
protected override string GetQueueUrlPath()
{
return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownSeriesItems=true&includeSeries=true&includeEpisode=true";
return "/api/v3/queue";
}
protected override string GetQueueDeleteUrlPath(long recordId, bool removeFromClient)
protected override string GetQueueUrlQuery(int page)
{
string path = $"/api/v3/queue/{recordId}?blocklist=true&skipRedownload=true&changeCategory=false";
return $"page={page}&pageSize=200&includeUnknownSeriesItems=true&includeSeries=true&includeEpisode=true";
}
path += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
protected override string GetQueueDeleteUrlPath(long recordId)
{
return $"/api/v3/queue/{recordId}";
}
return path;
protected override string GetQueueDeleteUrlQuery(bool removeFromClient)
{
string query = "blocklist=true&skipRedownload=true&changeCategory=false";
query += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return query;
}
public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items)
@@ -49,11 +58,12 @@ public class SonarrClient : ArrClient, ISonarrClient
return;
}
Uri uri = new(arrInstance.Url, "/api/v3/command");
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/command";
foreach (SonarrCommand command in GetSearchCommands(items.Cast<SonarrSearchItem>().ToHashSet()))
{
using HttpRequestMessage request = new(HttpMethod.Post, uri);
using HttpRequestMessage request = new(HttpMethod.Post, uriBuilder.Uri);
request.Content = new StringContent(
JsonConvert.SerializeObject(command, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }),
Encoding.UTF8,
@@ -199,8 +209,11 @@ public class SonarrClient : ArrClient, ISonarrClient
private async Task<List<Episode>?> GetEpisodesAsync(ArrInstance arrInstance, List<long> episodeIds)
{
Uri uri = new(arrInstance.Url, $"api/v3/episode?{string.Join('&', episodeIds.Select(x => $"episodeIds={x}"))}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/episode";
uriBuilder.Query = string.Join('&', episodeIds.Select(x => $"episodeIds={x}"));
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);
@@ -212,8 +225,10 @@ public class SonarrClient : ArrClient, ISonarrClient
private async Task<Series?> GetSeriesAsync(ArrInstance arrInstance, long seriesId)
{
Uri uri = new(arrInstance.Url, $"api/v3/series/{seriesId}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/series/{seriesId}";
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);

View File

@@ -49,9 +49,9 @@ public sealed class BlocklistProvider
try
{
await LoadPatternsAndRegexesAsync(_sonarrConfig.Block.Type, _sonarrConfig.Block.Path, InstanceType.Sonarr);
await LoadPatternsAndRegexesAsync(_radarrConfig.Block.Type, _radarrConfig.Block.Path, InstanceType.Radarr);
await LoadPatternsAndRegexesAsync(_lidarrConfig.Block.Type, _lidarrConfig.Block.Path, InstanceType.Lidarr);
await LoadPatternsAndRegexesAsync(_sonarrConfig, InstanceType.Sonarr);
await LoadPatternsAndRegexesAsync(_radarrConfig, InstanceType.Radarr);
await LoadPatternsAndRegexesAsync(_lidarrConfig, InstanceType.Lidarr);
_initialized = true;
}
@@ -83,14 +83,19 @@ public sealed class BlocklistProvider
return regexes ?? [];
}
private async Task LoadPatternsAndRegexesAsync(BlocklistType blocklistType, string? blocklistPath, InstanceType instanceType)
private async Task LoadPatternsAndRegexesAsync(ArrConfig arrConfig, InstanceType instanceType)
{
if (string.IsNullOrEmpty(blocklistPath))
if (!arrConfig.Enabled)
{
return;
}
string[] filePatterns = await ReadContentAsync(blocklistPath);
if (string.IsNullOrEmpty(arrConfig.Block.Path))
{
return;
}
string[] filePatterns = await ReadContentAsync(arrConfig.Block.Path);
long startTime = Stopwatch.GetTimestamp();
ParallelOptions options = new() { MaxDegreeOfParallelism = 5 };
@@ -121,13 +126,13 @@ public sealed class BlocklistProvider
TimeSpan elapsed = Stopwatch.GetElapsedTime(startTime);
_cache.Set(CacheKeys.BlocklistType(instanceType), blocklistType);
_cache.Set(CacheKeys.BlocklistType(instanceType), arrConfig.Block.Type);
_cache.Set(CacheKeys.BlocklistPatterns(instanceType), patterns);
_cache.Set(CacheKeys.BlocklistRegexes(instanceType), regexes);
_logger.LogDebug("loaded {count} patterns", patterns.Count);
_logger.LogDebug("loaded {count} regexes", regexes.Count);
_logger.LogDebug("blocklist loaded in {elapsed} ms | {path}", elapsed.TotalMilliseconds, blocklistPath);
_logger.LogDebug("blocklist loaded in {elapsed} ms | {path}", elapsed.TotalMilliseconds, arrConfig.Block.Path);
}
private async Task<string[]> ReadContentAsync(string path)

View File

@@ -6,6 +6,7 @@ using Common.Configuration.DownloadClient;
using Domain.Enums;
using Domain.Models.Arr;
using Domain.Models.Arr.Queue;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.Context;
@@ -22,7 +23,8 @@ public sealed class ContentBlocker : GenericHandler
{
private readonly ContentBlockerConfig _config;
private readonly BlocklistProvider _blocklistProvider;
private readonly IgnoredDownloadsProvider<ContentBlockerConfig> _ignoredDownloadsProvider;
public ContentBlocker(
ILogger<ContentBlocker> logger,
IOptions<ContentBlockerConfig> config,
@@ -36,7 +38,8 @@ public sealed class ContentBlocker : GenericHandler
ArrQueueIterator arrArrQueueIterator,
BlocklistProvider blocklistProvider,
DownloadServiceFactory downloadServiceFactory,
INotificationPublisher notifier
INotificationPublisher notifier,
IgnoredDownloadsProvider<ContentBlockerConfig> ignoredDownloadsProvider
) : base(
logger, downloadClientConfig,
sonarrConfig, radarrConfig, lidarrConfig,
@@ -47,13 +50,14 @@ public sealed class ContentBlocker : GenericHandler
{
_config = config.Value;
_blocklistProvider = blocklistProvider;
_ignoredDownloadsProvider = ignoredDownloadsProvider;
}
public override async Task ExecuteAsync()
{
if (_downloadClientConfig.DownloadClient is Common.Enums.DownloadClient.None)
if (_downloadClientConfig.DownloadClient is Common.Enums.DownloadClient.None or Common.Enums.DownloadClient.Disabled)
{
_logger.LogWarning("download client is set to none");
_logger.LogWarning("download client is not set");
return;
}
@@ -73,6 +77,8 @@ public sealed class ContentBlocker : GenericHandler
protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType)
{
IReadOnlyList<string> ignoredDownloads = await _ignoredDownloadsProvider.GetIgnoredDownloads();
using var _ = LogContext.PushProperty("InstanceName", instanceType.ToString());
HashSet<SearchItem> itemsToBeRefreshed = [];
@@ -106,13 +112,19 @@ public sealed class ContentBlocker : GenericHandler
continue;
}
if (ignoredDownloads.Contains(record.DownloadId, StringComparer.InvariantCultureIgnoreCase))
{
_logger.LogInformation("skip | {title} | ignored", record.Title);
continue;
}
// push record to context
ContextProvider.Set(nameof(QueueRecord), record);
_logger.LogDebug("searching unwanted files for {title}", record.Title);
BlockFilesResult result = await _downloadService
.BlockUnwantedFilesAsync(record.DownloadId, blocklistType, patterns, regexes);
.BlockUnwantedFilesAsync(record.DownloadId, blocklistType, patterns, regexes, ignoredDownloads);
if (!result.ShouldRemove)
{
@@ -130,7 +142,7 @@ public sealed class ContentBlocker : GenericHandler
removeFromClient = false;
}
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient);
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient, DeleteReason.AllFilesBlocked);
await _notifier.NotifyQueueItemDeleted(removeFromClient, DeleteReason.AllFilesBlocked);
}
});

View File

@@ -3,6 +3,7 @@ using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Domain.Enums;
using Domain.Models.Arr.Queue;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.DownloadClient;
@@ -17,6 +18,7 @@ namespace Infrastructure.Verticals.DownloadCleaner;
public sealed class DownloadCleaner : GenericHandler
{
private readonly DownloadCleanerConfig _config;
private readonly IgnoredDownloadsProvider<DownloadCleanerConfig> _ignoredDownloadsProvider;
private readonly HashSet<string> _excludedHashes = [];
public DownloadCleaner(
@@ -31,7 +33,8 @@ public sealed class DownloadCleaner : GenericHandler
LidarrClient lidarrClient,
ArrQueueIterator arrArrQueueIterator,
DownloadServiceFactory downloadServiceFactory,
INotificationPublisher notifier
INotificationPublisher notifier,
IgnoredDownloadsProvider<DownloadCleanerConfig> ignoredDownloadsProvider
) : base(
logger, downloadClientConfig,
sonarrConfig, radarrConfig, lidarrConfig,
@@ -42,16 +45,25 @@ public sealed class DownloadCleaner : GenericHandler
{
_config = config.Value;
_config.Validate();
_ignoredDownloadsProvider = ignoredDownloadsProvider;
}
public override async Task ExecuteAsync()
{
if (_downloadClientConfig.DownloadClient is Common.Enums.DownloadClient.None or Common.Enums.DownloadClient.Disabled)
{
_logger.LogWarning("download client is not set");
return;
}
if (_config.Categories?.Count is null or 0)
{
_logger.LogWarning("no categories configured");
return;
}
IReadOnlyList<string> ignoredDownloads = await _ignoredDownloadsProvider.GetIgnoredDownloads();
await _downloadService.LoginAsync();
List<object>? downloads = await _downloadService.GetAllDownloadsToBeCleaned(_config.Categories);
@@ -69,7 +81,7 @@ public sealed class DownloadCleaner : GenericHandler
await ProcessArrConfigAsync(_radarrConfig, InstanceType.Radarr, true);
await ProcessArrConfigAsync(_lidarrConfig, InstanceType.Lidarr, true);
await _downloadService.CleanDownloads(downloads, _config.Categories, _excludedHashes);
await _downloadService.CleanDownloads(downloads, _config.Categories, _excludedHashes, ignoredDownloads);
}
protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType)

View File

@@ -16,9 +16,26 @@ public sealed class DelugeClient
private readonly DelugeConfig _config;
private readonly HttpClient _httpClient;
private static readonly IReadOnlyList<string> Fields =
[
"hash",
"state",
"name",
"eta",
"private",
"total_done",
"label",
"seeding_time",
"ratio",
"trackers",
"download_payload_rate",
"total_size"
];
public DelugeClient(IOptions<DelugeConfig> config, IHttpClientFactory httpClientFactory)
{
_config = config.Value;
_config.Validate();
_httpClient = httpClientFactory.CreateClient(nameof(DelugeService));
}
@@ -63,21 +80,34 @@ public sealed class DelugeClient
return torrents.FirstOrDefault();
}
public async Task<TorrentStatus?> GetTorrentStatus(string hash)
public async Task<DownloadStatus?> GetTorrentStatus(string hash)
{
return await SendRequest<TorrentStatus?>(
"web.get_torrent_status",
hash,
new[] { "hash", "state", "name", "eta", "private", "total_done", "label", "seeding_time", "ratio" }
);
try
{
return await SendRequest<DownloadStatus?>(
"web.get_torrent_status",
hash,
Fields
);
}
catch (DelugeClientException e)
{
// Deluge returns an error when the torrent is not found
if (e.Message == "AttributeError: 'NoneType' object has no attribute 'call'")
{
return null;
}
throw;
}
}
public async Task<List<TorrentStatus>?> GetStatusForAllTorrents()
public async Task<List<DownloadStatus>?> GetStatusForAllTorrents()
{
Dictionary<string, TorrentStatus>? downloads = await SendRequest<Dictionary<string, TorrentStatus>?>(
Dictionary<string, DownloadStatus>? downloads = await SendRequest<Dictionary<string, DownloadStatus>?>(
"core.get_torrents_status",
"",
new[] { "hash", "state", "name", "eta", "private", "total_done", "label", "seeding_time", "ratio" }
Fields
);
return downloads?.Values.ToList();
@@ -107,8 +137,12 @@ public sealed class DelugeClient
{
StringContent content = new StringContent(json);
content.Headers.ContentType = new MediaTypeWithQualityHeaderValue("application/json");
var responseMessage = await _httpClient.PostAsync(new Uri(_config.Url, "/json"), content);
UriBuilder uriBuilder = new(_config.Url);
uriBuilder.Path = string.IsNullOrEmpty(_config.UrlBase)
? $"{uriBuilder.Path.TrimEnd('/')}/json"
: $"{uriBuilder.Path.TrimEnd('/')}/{_config.UrlBase.TrimStart('/').TrimEnd('/')}/json";
var responseMessage = await _httpClient.PostAsync(uriBuilder.Uri, content);
responseMessage.EnsureSuccessStatusCode();
var responseJson = await responseMessage.Content.ReadAsStringAsync();

View File

@@ -1,12 +1,15 @@
using System.Collections.Concurrent;
using System.Globalization;
using System.Text.RegularExpressions;
using Common.Attributes;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Common.CustomDataTypes;
using Domain.Enums;
using Domain.Models.Deluge.Response;
using Infrastructure.Extensions;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.Context;
@@ -49,20 +52,28 @@ public class DelugeService : DownloadService, IDelugeService
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override async Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
hash = hash.ToLowerInvariant();
DelugeContents? contents = null;
StalledResult result = new();
DownloadCheckResult result = new();
TorrentStatus? status = await _client.GetTorrentStatus(hash);
DownloadStatus? download = await _client.GetTorrentStatus(hash);
if (status?.Hash is null)
if (download?.Hash is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
result.IsPrivate = download.Private;
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
try
{
@@ -72,6 +83,7 @@ public class DelugeService : DownloadService, IDelugeService
{
_logger.LogDebug(exception, "failed to find torrent {hash} in the download client", hash);
}
bool shouldRemove = contents?.Contents?.Count > 0;
@@ -85,45 +97,47 @@ public class DelugeService : DownloadService, IDelugeService
if (shouldRemove)
{
result.DeleteReason = DeleteReason.AllFilesBlocked;
// remove if all files are unwanted
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesSkipped;
return result;
}
result.ShouldRemove = shouldRemove || await IsItemStuckAndShouldRemove(status);
result.IsPrivate = status.Private;
// remove if download is stuck
(result.ShouldRemove, result.DeleteReason) = await EvaluateDownloadRemoval(download);
if (!shouldRemove && result.ShouldRemove)
{
result.DeleteReason = DeleteReason.Stalled;
}
return result;
}
/// <inheritdoc/>
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
)
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads)
{
hash = hash.ToLowerInvariant();
TorrentStatus? status = await _client.GetTorrentStatus(hash);
DownloadStatus? download = await _client.GetTorrentStatus(hash);
BlockFilesResult result = new();
if (status?.Hash is null)
if (download?.Hash is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
result.IsPrivate = status.Private;
if (_contentBlockerConfig.IgnorePrivate && status.Private)
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
result.IsPrivate = download.Private;
if (_contentBlockerConfig.IgnorePrivate && download.Private)
{
// ignore private trackers
_logger.LogDebug("skip files check | download is private | {name}", status.Name);
_logger.LogDebug("skip files check | download is private | {name}", download.Name);
return result;
}
@@ -205,14 +219,21 @@ public class DelugeService : DownloadService, IDelugeService
}
/// <inheritdoc/>
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads)
{
foreach (TorrentStatus download in downloads)
foreach (DownloadStatus download in downloads)
{
if (string.IsNullOrEmpty(download.Hash))
{
continue;
}
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
continue;
}
Category? category = categoriesToClean
.FirstOrDefault(x => x.Name.Equals(download.Label, StringComparison.InvariantCultureIgnoreCase));
@@ -274,33 +295,90 @@ public class DelugeService : DownloadService, IDelugeService
await _client.ChangeFilesPriority(hash, sortedPriorities);
}
private async Task<bool> IsItemStuckAndShouldRemove(TorrentStatus status)
private async Task<(bool, DeleteReason)> EvaluateDownloadRemoval(DownloadStatus status)
{
(bool ShouldRemove, DeleteReason Reason) result = await CheckIfSlow(status);
if (result.ShouldRemove)
{
return result;
}
return await CheckIfStuck(status);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfSlow(DownloadStatus download)
{
if (_queueCleanerConfig.SlowMaxStrikes is 0)
{
return (false, DeleteReason.None);
}
if (download.State is null || !download.State.Equals("Downloading", StringComparison.InvariantCultureIgnoreCase))
{
return (false, DeleteReason.None);
}
if (download.DownloadSpeed <= 0)
{
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.SlowIgnorePrivate && download.Private)
{
// ignore private trackers
_logger.LogDebug("skip slow check | download is private | {name}", download.Name);
return (false, DeleteReason.None);
}
if (download.Size > (_queueCleanerConfig.SlowIgnoreAboveSizeByteSize?.Bytes ?? long.MaxValue))
{
_logger.LogDebug("skip slow check | download is too large | {name}", download.Name);
return (false, DeleteReason.None);
}
ByteSize minSpeed = _queueCleanerConfig.SlowMinSpeedByteSize;
ByteSize currentSpeed = new ByteSize(download.DownloadSpeed);
SmartTimeSpan maxTime = SmartTimeSpan.FromHours(_queueCleanerConfig.SlowMaxTime);
SmartTimeSpan currentTime = SmartTimeSpan.FromSeconds(download.Eta);
return await CheckIfSlow(
download.Hash!,
download.Name!,
minSpeed,
currentSpeed,
maxTime,
currentTime
);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfStuck(DownloadStatus status)
{
if (_queueCleanerConfig.StalledMaxStrikes is 0)
{
return false;
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.StalledIgnorePrivate && status.Private)
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", status.Name);
return false;
return (false, DeleteReason.None);
}
if (status.State is null || !status.State.Equals("Downloading", StringComparison.InvariantCultureIgnoreCase))
{
return false;
return (false, DeleteReason.None);
}
if (status.Eta > 0)
{
return false;
return (false, DeleteReason.None);
}
ResetStrikesOnProgress(status.Hash!, status.TotalDone);
return await StrikeAndCheckLimit(status.Hash!, status.Name!);
ResetStalledStrikesOnProgress(status.Hash!, status.TotalDone);
return (await _striker.StrikeAndCheckLimit(status.Hash!, status.Name!, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled), DeleteReason.Stalled);
}
private static void ProcessFiles(Dictionary<string, DelugeFileOrDirectory>? contents, Action<string, DelugeFileOrDirectory> processFile)

View File

@@ -2,7 +2,7 @@
namespace Infrastructure.Verticals.DownloadClient;
public sealed record StalledResult
public sealed record DownloadCheckResult
{
/// <summary>
/// True if the download should be removed; otherwise false.

View File

@@ -3,6 +3,7 @@ using System.Text.RegularExpressions;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Common.CustomDataTypes;
using Common.Helpers;
using Domain.Enums;
using Domain.Models.Cache;
@@ -60,15 +61,13 @@ public abstract class DownloadService : IDownloadService
public abstract Task LoginAsync();
public abstract Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash);
public abstract Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads);
/// <inheritdoc/>
public abstract Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public abstract Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
);
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads);
/// <inheritdoc/>
public abstract Task DeleteDownload(string hash);
@@ -77,34 +76,107 @@ public abstract class DownloadService : IDownloadService
public abstract Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories);
/// <inheritdoc/>
public abstract Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes);
public abstract Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads);
protected void ResetStrikesOnProgress(string hash, long downloaded)
protected void ResetStalledStrikesOnProgress(string hash, long downloaded)
{
if (!_queueCleanerConfig.StalledResetStrikesOnProgress)
{
return;
}
if (_cache.TryGetValue(CacheKeys.Item(hash), out CacheItem? cachedItem) && cachedItem is not null && downloaded > cachedItem.Downloaded)
if (_cache.TryGetValue(CacheKeys.StrikeItem(hash, StrikeType.Stalled), out StalledCacheItem? cachedItem) &&
cachedItem is not null && downloaded > cachedItem.Downloaded)
{
// cache item found
_cache.Remove(CacheKeys.Strike(StrikeType.Stalled, hash));
_logger.LogDebug("resetting strikes for {hash} due to progress", hash);
_logger.LogDebug("resetting stalled strikes for {hash} due to progress", hash);
}
_cache.Set(CacheKeys.Item(hash), new CacheItem { Downloaded = downloaded }, _cacheOptions);
_cache.Set(CacheKeys.StrikeItem(hash, StrikeType.Stalled), new StalledCacheItem { Downloaded = downloaded }, _cacheOptions);
}
protected void ResetSlowSpeedStrikesOnProgress(string downloadName, string hash)
{
if (!_queueCleanerConfig.SlowResetStrikesOnProgress)
{
return;
}
string key = CacheKeys.Strike(StrikeType.SlowSpeed, hash);
if (!_cache.TryGetValue(key, out object? value) || value is null)
{
return;
}
_cache.Remove(key);
_logger.LogDebug("resetting slow speed strikes due to progress | {name}", downloadName);
}
protected void ResetSlowTimeStrikesOnProgress(string downloadName, string hash)
{
if (!_queueCleanerConfig.SlowResetStrikesOnProgress)
{
return;
}
string key = CacheKeys.Strike(StrikeType.SlowTime, hash);
if (!_cache.TryGetValue(key, out object? value) || value is null)
{
return;
}
_cache.Remove(key);
_logger.LogDebug("resetting slow time strikes due to progress | {name}", downloadName);
}
/// <summary>
/// Strikes an item and checks if the limit has been reached.
/// </summary>
/// <param name="hash">The torrent hash.</param>
/// <param name="itemName">The name or title of the item.</param>
/// <returns>True if the limit has been reached; otherwise, false.</returns>
protected async Task<bool> StrikeAndCheckLimit(string hash, string itemName)
protected async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfSlow(
string downloadHash,
string downloadName,
ByteSize minSpeed,
ByteSize currentSpeed,
SmartTimeSpan maxTime,
SmartTimeSpan currentTime
)
{
return await _striker.StrikeAndCheckLimit(hash, itemName, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled);
if (minSpeed.Bytes > 0 && currentSpeed < minSpeed)
{
_logger.LogTrace("slow speed | {speed}/s | {name}", currentSpeed.ToString(), downloadName);
bool shouldRemove = await _striker
.StrikeAndCheckLimit(downloadHash, downloadName, _queueCleanerConfig.SlowMaxStrikes, StrikeType.SlowSpeed);
if (shouldRemove)
{
return (true, DeleteReason.SlowSpeed);
}
}
else
{
ResetSlowSpeedStrikesOnProgress(downloadName, downloadHash);
}
if (maxTime.Time > TimeSpan.Zero && currentTime > maxTime)
{
_logger.LogTrace("slow estimated time | {time} | {name}", currentTime.ToString(), downloadName);
bool shouldRemove = await _striker
.StrikeAndCheckLimit(downloadHash, downloadName, _queueCleanerConfig.SlowMaxStrikes, StrikeType.SlowTime);
if (shouldRemove)
{
return (true, DeleteReason.SlowTime);
}
}
else
{
ResetSlowTimeStrikesOnProgress(downloadName, downloadHash);
}
return (false, DeleteReason.None);
}
protected SeedingCheckResult ShouldCleanDownload(double ratio, TimeSpan seedingTime, Category category)
@@ -131,7 +203,7 @@ public abstract class DownloadService : IDownloadService
return new();
}
private bool DownloadReachedRatio(double ratio, TimeSpan seedingTime, Category category)
{
if (category.MaxRatio < 0)

View File

@@ -25,6 +25,7 @@ public sealed class DownloadServiceFactory
Common.Enums.DownloadClient.Deluge => _serviceProvider.GetRequiredService<DelugeService>(),
Common.Enums.DownloadClient.Transmission => _serviceProvider.GetRequiredService<TransmissionService>(),
Common.Enums.DownloadClient.None => _serviceProvider.GetRequiredService<DummyDownloadService>(),
Common.Enums.DownloadClient.Disabled => _serviceProvider.GetRequiredService<DummyDownloadService>(),
_ => throw new ArgumentOutOfRangeException()
};
}

View File

@@ -28,12 +28,13 @@ public class DummyDownloadService : DownloadService
return Task.CompletedTask;
}
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
throw new NotImplementedException();
}
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType, ConcurrentBag<string> patterns, ConcurrentBag<Regex> regexes)
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType, ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads)
{
throw new NotImplementedException();
}
@@ -43,7 +44,8 @@ public class DummyDownloadService : DownloadService
throw new NotImplementedException();
}
public override Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads)
{
throw new NotImplementedException();
}

View File

@@ -14,7 +14,8 @@ public interface IDownloadService : IDisposable
/// Checks whether the download should be removed from the *arr queue.
/// </summary>
/// <param name="hash">The download hash.</param>
public Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash);
/// <param name="ignoredDownloads">Downloads to ignore from processing.</param>
public Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads);
/// <summary>
/// Blocks unwanted files from being fully downloaded.
@@ -23,12 +24,13 @@ public interface IDownloadService : IDisposable
/// <param name="blocklistType">The <see cref="BlocklistType"/>.</param>
/// <param name="patterns">The patterns to test the files against.</param>
/// <param name="regexes">The regexes to test the files against.</param>
/// <param name="ignoredDownloads">Downloads to ignore from processing.</param>
/// <returns>True if all files have been blocked; otherwise false.</returns>
public Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
ConcurrentBag<Regex> regexes,
IReadOnlyList<string> ignoredDownloads
);
/// <summary>
@@ -37,14 +39,16 @@ public interface IDownloadService : IDisposable
/// <param name="categories">The categories by which to filter the downloads.</param>
/// <returns>A list of downloads for the provided categories.</returns>
Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories);
/// <summary>
/// Cleans the downloads.
/// </summary>
/// <param name="downloads"></param>
/// <param name="categoriesToClean">The categories that should be cleaned.</param>
/// <param name="excludedHashes">The hashes that should not be cleaned.</param>
public abstract Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes);
/// <param name="ignoredDownloads">Downloads to ignore from processing.</param>
public abstract Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads);
/// <summary>
/// Deletes a download item.

View File

@@ -5,8 +5,10 @@ using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Common.CustomDataTypes;
using Common.Helpers;
using Domain.Enums;
using Infrastructure.Extensions;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.Context;
@@ -44,7 +46,11 @@ public class QBitService : DownloadService, IQBitService
{
_config = config.Value;
_config.Validate();
_client = new(httpClientFactory.CreateClient(Constants.HttpClientWithRetryName), _config.Url);
UriBuilder uriBuilder = new(_config.Url);
uriBuilder.Path = string.IsNullOrEmpty(_config.UrlBase)
? uriBuilder.Path
: $"{uriBuilder.Path.TrimEnd('/')}/{_config.UrlBase.TrimStart('/')}";
_client = new(httpClientFactory.CreateClient(Constants.HttpClientWithRetryName), uriBuilder.Uri);
}
public override async Task LoginAsync()
@@ -58,18 +64,27 @@ public class QBitService : DownloadService, IQBitService
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override async Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
StalledResult result = new();
TorrentInfo? torrent = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
DownloadCheckResult result = new();
TorrentInfo? download = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
.FirstOrDefault();
if (torrent is null)
if (download is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
IReadOnlyList<TorrentTracker> trackers = await GetTrackersAsync(hash);
if (ignoredDownloads.Count > 0 &&
(download.ShouldIgnore(ignoredDownloads) || trackers.Any(x => x.ShouldIgnore(ignoredDownloads)) is true))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
TorrentProperties? torrentProperties = await _client.GetTorrentPropertiesAsync(hash);
if (torrentProperties is null)
@@ -82,52 +97,56 @@ public class QBitService : DownloadService, IQBitService
bool.TryParse(dictValue?.ToString(), out bool boolValue)
&& boolValue;
// if all files were blocked by qBittorrent
if (torrent is { CompletionOn: not null, Downloaded: null or 0 })
{
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesBlocked;
return result;
}
IReadOnlyList<TorrentContent>? files = await _client.GetTorrentContentsAsync(hash);
// if all files are marked as skip
if (files?.Count is > 0 && files.All(x => x.Priority is TorrentContentPriority.Skip))
{
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesBlocked;
// if all files were blocked by qBittorrent
if (download is { CompletionOn: not null, Downloaded: null or 0 })
{
result.DeleteReason = DeleteReason.AllFilesSkippedByQBit;
return result;
}
// remove if all files are unwanted
result.DeleteReason = DeleteReason.AllFilesSkipped;
return result;
}
result.ShouldRemove = await IsItemStuckAndShouldRemove(torrent, result.IsPrivate);
if (result.ShouldRemove)
{
result.DeleteReason = DeleteReason.Stalled;
}
(result.ShouldRemove, result.DeleteReason) = await EvaluateDownloadRemoval(download, result.IsPrivate);
return result;
}
/// <inheritdoc/>
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
ConcurrentBag<Regex> regexes,
IReadOnlyList<string> ignoredDownloads
)
{
TorrentInfo? torrent = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
TorrentInfo? download = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
.FirstOrDefault();
BlockFilesResult result = new();
if (torrent is null)
if (download is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
IReadOnlyList<TorrentTracker> trackers = await GetTrackersAsync(hash);
if (ignoredDownloads.Count > 0 &&
(download.ShouldIgnore(ignoredDownloads) || trackers.Any(x => x.ShouldIgnore(ignoredDownloads)) is true))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
TorrentProperties? torrentProperties = await _client.GetTorrentPropertiesAsync(hash);
if (torrentProperties is null)
@@ -145,7 +164,7 @@ public class QBitService : DownloadService, IQBitService
if (_contentBlockerConfig.IgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip files check | download is private | {name}", torrent.Name);
_logger.LogDebug("skip files check | download is private | {name}", download.Name);
return result;
}
@@ -218,7 +237,8 @@ public class QBitService : DownloadService, IQBitService
.ToList();
/// <inheritdoc/>
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads)
{
foreach (TorrentInfo download in downloads)
{
@@ -227,6 +247,15 @@ public class QBitService : DownloadService, IQBitService
continue;
}
IReadOnlyList<TorrentTracker> trackers = await GetTrackersAsync(download.Hash);
if (ignoredDownloads.Count > 0 &&
(download.ShouldIgnore(ignoredDownloads) || trackers.Any(x => x.ShouldIgnore(ignoredDownloads)) is true))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
continue;
}
Category? category = categoriesToClean
.FirstOrDefault(x => download.Category.Equals(x.Name, StringComparison.InvariantCultureIgnoreCase));
@@ -303,30 +332,103 @@ public class QBitService : DownloadService, IQBitService
{
_client.Dispose();
}
private async Task<bool> IsItemStuckAndShouldRemove(TorrentInfo torrent, bool isPrivate)
private async Task<(bool, DeleteReason)> EvaluateDownloadRemoval(TorrentInfo torrent, bool isPrivate)
{
(bool ShouldRemove, DeleteReason Reason) result = await CheckIfSlow(torrent, isPrivate);
if (result.ShouldRemove)
{
return result;
}
return await CheckIfStuck(torrent, isPrivate);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfSlow(TorrentInfo download, bool isPrivate)
{
if (_queueCleanerConfig.SlowMaxStrikes is 0)
{
return (false, DeleteReason.None);
}
if (download.State is not (TorrentState.Downloading or TorrentState.ForcedDownload))
{
return (false, DeleteReason.None);
}
if (download.DownloadSpeed <= 0)
{
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.SlowIgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip slow check | download is private | {name}", download.Name);
return (false, DeleteReason.None);
}
if (download.Size > (_queueCleanerConfig.SlowIgnoreAboveSizeByteSize?.Bytes ?? long.MaxValue))
{
_logger.LogDebug("skip slow check | download is too large | {name}", download.Name);
return (false, DeleteReason.None);
}
ByteSize minSpeed = _queueCleanerConfig.SlowMinSpeedByteSize;
ByteSize currentSpeed = new ByteSize(download.DownloadSpeed);
SmartTimeSpan maxTime = SmartTimeSpan.FromHours(_queueCleanerConfig.SlowMaxTime);
SmartTimeSpan currentTime = new SmartTimeSpan(download.EstimatedTime ?? TimeSpan.Zero);
return await CheckIfSlow(
download.Hash,
download.Name,
minSpeed,
currentSpeed,
maxTime,
currentTime
);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfStuck(TorrentInfo torrent, bool isPrivate)
{
if (_queueCleanerConfig.StalledMaxStrikes is 0)
{
return false;
}
if (_queueCleanerConfig.StalledIgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", torrent.Name);
return false;
return (false, DeleteReason.None);
}
if (torrent.State is not TorrentState.StalledDownload and not TorrentState.FetchingMetadata
and not TorrentState.ForcedFetchingMetadata)
{
// ignore other states
return false;
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.StalledIgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", torrent.Name);
return (false, DeleteReason.None);
}
if (torrent.State is TorrentState.StalledDownload)
{
_logger.LogTrace("stalled download | {name}", torrent.Name);
ResetStalledStrikesOnProgress(torrent.Hash, torrent.Downloaded ?? 0);
return (await _striker.StrikeAndCheckLimit(torrent.Hash, torrent.Name, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled), DeleteReason.Stalled);
}
ResetStrikesOnProgress(torrent.Hash, torrent.Downloaded ?? 0);
_logger.LogTrace("downloading metadata | {name}", torrent.Name);
return (await _striker.StrikeAndCheckLimit(torrent.Hash, torrent.Name, _queueCleanerConfig.StalledMaxStrikes, StrikeType.DownloadingMetadata), DeleteReason.DownloadingMetadata);
}
return await StrikeAndCheckLimit(torrent.Hash, torrent.Name);
private async Task<IReadOnlyList<TorrentTracker>> GetTrackersAsync(string hash)
{
return (await _client.GetTorrentTrackersAsync(hash))
.Where(x => !x.Url.ToString().Contains("**"))
.ToList();
}
}

View File

@@ -5,8 +5,10 @@ using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Common.CustomDataTypes;
using Common.Helpers;
using Domain.Enums;
using Infrastructure.Extensions;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.Context;
@@ -25,7 +27,25 @@ public class TransmissionService : DownloadService, ITransmissionService
{
private readonly TransmissionConfig _config;
private readonly Client _client;
private TorrentInfo[]? _torrentsCache;
private static readonly string[] Fields =
[
TorrentFields.FILES,
TorrentFields.FILE_STATS,
TorrentFields.HASH_STRING,
TorrentFields.ID,
TorrentFields.ETA,
TorrentFields.NAME,
TorrentFields.STATUS,
TorrentFields.IS_PRIVATE,
TorrentFields.DOWNLOADED_EVER,
TorrentFields.DOWNLOAD_DIR,
TorrentFields.SECONDS_SEEDING,
TorrentFields.UPLOAD_RATIO,
TorrentFields.TRACKERS,
TorrentFields.RATE_DOWNLOAD,
TorrentFields.TOTAL_SIZE
];
public TransmissionService(
IHttpClientFactory httpClientFactory,
@@ -46,9 +66,13 @@ public class TransmissionService : DownloadService, ITransmissionService
{
_config = config.Value;
_config.Validate();
UriBuilder uriBuilder = new(_config.Url);
uriBuilder.Path = string.IsNullOrEmpty(_config.UrlBase)
? $"{uriBuilder.Path.TrimEnd('/')}/rpc"
: $"{uriBuilder.Path.TrimEnd('/')}/{_config.UrlBase.TrimStart('/').TrimEnd('/')}/rpc";
_client = new(
httpClientFactory.CreateClient(Constants.HttpClientWithRetryName),
new Uri(_config.Url, "/transmission/rpc").ToString(),
uriBuilder.Uri.ToString(),
login: _config.Username,
password: _config.Password
);
@@ -60,21 +84,27 @@ public class TransmissionService : DownloadService, ITransmissionService
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override async Task<DownloadCheckResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
StalledResult result = new();
TorrentInfo? torrent = await GetTorrentAsync(hash);
DownloadCheckResult result = new();
TorrentInfo? download = await GetTorrentAsync(hash);
if (torrent is null)
if (download is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
bool shouldRemove = torrent.FileStats?.Length > 0;
result.IsPrivate = torrent.IsPrivate ?? false;
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogDebug("skip | download is ignored | {name}", download.Name);
return result;
}
bool shouldRemove = download.FileStats?.Length > 0;
result.IsPrivate = download.IsPrivate ?? false;
foreach (TransmissionTorrentFileStats? stats in torrent.FileStats ?? [])
foreach (TransmissionTorrentFileStats? stats in download.FileStats ?? [])
{
if (!stats.Wanted.HasValue)
{
@@ -91,43 +121,45 @@ public class TransmissionService : DownloadService, ITransmissionService
if (shouldRemove)
{
// remove if all files are unwanted
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesBlocked;
return result;
}
// remove if all files are unwanted or download is stuck
result.ShouldRemove = shouldRemove || await IsItemStuckAndShouldRemove(torrent);
// remove if download is stuck
(result.ShouldRemove, result.DeleteReason) = await EvaluateDownloadRemoval(download);
if (!shouldRemove && result.ShouldRemove)
{
result.DeleteReason = DeleteReason.Stalled;
}
return result;
}
/// <inheritdoc/>
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
)
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads)
{
TorrentInfo? torrent = await GetTorrentAsync(hash);
TorrentInfo? download = await GetTorrentAsync(hash);
BlockFilesResult result = new();
if (torrent?.FileStats is null || torrent.Files is null)
if (download?.FileStats is null || download.Files is null)
{
return result;
}
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogDebug("skip | download is ignored | {name}", download.Name);
return result;
}
bool isPrivate = torrent.IsPrivate ?? false;
bool isPrivate = download.IsPrivate ?? false;
result.IsPrivate = isPrivate;
if (_contentBlockerConfig.IgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip files check | download is private | {name}", torrent.Name);
_logger.LogDebug("skip files check | download is private | {name}", download.Name);
return result;
}
@@ -135,27 +167,27 @@ public class TransmissionService : DownloadService, ITransmissionService
long totalFiles = 0;
long totalUnwantedFiles = 0;
for (int i = 0; i < torrent.Files.Length; i++)
for (int i = 0; i < download.Files.Length; i++)
{
if (torrent.FileStats?[i].Wanted == null)
if (download.FileStats?[i].Wanted == null)
{
continue;
}
totalFiles++;
if (!torrent.FileStats[i].Wanted.Value)
if (!download.FileStats[i].Wanted.Value)
{
totalUnwantedFiles++;
continue;
}
if (_filenameEvaluator.IsValid(torrent.Files[i].Name, blocklistType, patterns, regexes))
if (_filenameEvaluator.IsValid(download.Files[i].Name, blocklistType, patterns, regexes))
{
continue;
}
_logger.LogInformation("unwanted file found | {file}", torrent.Files[i].Name);
_logger.LogInformation("unwanted file found | {file}", download.Files[i].Name);
unwantedFiles.Add(i);
totalUnwantedFiles++;
}
@@ -175,7 +207,7 @@ public class TransmissionService : DownloadService, ITransmissionService
_logger.LogDebug("changing priorities | torrent {hash}", hash);
await _dryRunInterceptor.InterceptAsync(SetUnwantedFiles, torrent.Id, unwantedFiles.ToArray());
await _dryRunInterceptor.InterceptAsync(SetUnwantedFiles, download.Id, unwantedFiles.ToArray());
return result;
}
@@ -183,22 +215,7 @@ public class TransmissionService : DownloadService, ITransmissionService
/// <inheritdoc/>
public override async Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories)
{
string[] fields = [
TorrentFields.FILES,
TorrentFields.FILE_STATS,
TorrentFields.HASH_STRING,
TorrentFields.ID,
TorrentFields.ETA,
TorrentFields.NAME,
TorrentFields.STATUS,
TorrentFields.IS_PRIVATE,
TorrentFields.DOWNLOADED_EVER,
TorrentFields.DOWNLOAD_DIR,
TorrentFields.SECONDS_SEEDING,
TorrentFields.UPLOAD_RATIO
];
return (await _client.TorrentGetAsync(fields))
return (await _client.TorrentGetAsync(Fields))
?.Torrents
?.Where(x => !string.IsNullOrEmpty(x.HashString))
.Where(x => x.Status is 5 or 6)
@@ -219,7 +236,8 @@ public class TransmissionService : DownloadService, ITransmissionService
}
/// <inheritdoc/>
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads)
{
foreach (TorrentInfo download in downloads)
{
@@ -227,6 +245,12 @@ public class TransmissionService : DownloadService, ITransmissionService
{
continue;
}
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogDebug("skip | download is ignored | {name}", download.Name);
continue;
}
Category? category = categoriesToClean
.FirstOrDefault(x =>
@@ -314,72 +338,96 @@ public class TransmissionService : DownloadService, ITransmissionService
});
}
private async Task<bool> IsItemStuckAndShouldRemove(TorrentInfo torrent)
private async Task<(bool, DeleteReason)> EvaluateDownloadRemoval(TorrentInfo torrent)
{
(bool ShouldRemove, DeleteReason Reason) result = await CheckIfSlow(torrent);
if (result.ShouldRemove)
{
return result;
}
return await CheckIfStuck(torrent);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfSlow(TorrentInfo download)
{
if (_queueCleanerConfig.SlowMaxStrikes is 0)
{
return (false, DeleteReason.None);
}
if (download.Status is not 4)
{
// not in downloading state
return (false, DeleteReason.None);
}
if (download.RateDownload <= 0)
{
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.SlowIgnorePrivate && download.IsPrivate is true)
{
// ignore private trackers
_logger.LogDebug("skip slow check | download is private | {name}", download.Name);
return (false, DeleteReason.None);
}
if (download.TotalSize > (_queueCleanerConfig.SlowIgnoreAboveSizeByteSize?.Bytes ?? long.MaxValue))
{
_logger.LogDebug("skip slow check | download is too large | {name}", download.Name);
return (false, DeleteReason.None);
}
ByteSize minSpeed = _queueCleanerConfig.SlowMinSpeedByteSize;
ByteSize currentSpeed = new ByteSize(download.RateDownload ?? long.MaxValue);
SmartTimeSpan maxTime = SmartTimeSpan.FromHours(_queueCleanerConfig.SlowMaxTime);
SmartTimeSpan currentTime = SmartTimeSpan.FromSeconds(download.Eta ?? 0);
return await CheckIfSlow(
download.HashString!,
download.Name!,
minSpeed,
currentSpeed,
maxTime,
currentTime
);
}
private async Task<(bool ShouldRemove, DeleteReason Reason)> CheckIfStuck(TorrentInfo download)
{
if (_queueCleanerConfig.StalledMaxStrikes is 0)
{
return false;
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.StalledIgnorePrivate && (torrent.IsPrivate ?? false))
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", torrent.Name);
return false;
}
if (torrent.Status is not 4)
if (download.Status is not 4)
{
// not in downloading state
return false;
}
if (torrent.Eta > 0)
{
return false;
return (false, DeleteReason.None);
}
ResetStrikesOnProgress(torrent.HashString!, torrent.DownloadedEver ?? 0);
return await StrikeAndCheckLimit(torrent.HashString!, torrent.Name!);
if (download.RateDownload > 0 || download.Eta > 0)
{
return (false, DeleteReason.None);
}
if (_queueCleanerConfig.StalledIgnorePrivate && (download.IsPrivate ?? false))
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", download.Name);
return (false, DeleteReason.None);
}
ResetStalledStrikesOnProgress(download.HashString!, download.DownloadedEver ?? 0);
return (await _striker.StrikeAndCheckLimit(download.HashString!, download.Name!, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled), DeleteReason.Stalled);
}
private async Task<TorrentInfo?> GetTorrentAsync(string hash)
{
TorrentInfo? torrent = _torrentsCache?
.FirstOrDefault(x => x.HashString.Equals(hash, StringComparison.InvariantCultureIgnoreCase));
if (_torrentsCache is null || torrent is null)
{
string[] fields = [
TorrentFields.FILES,
TorrentFields.FILE_STATS,
TorrentFields.HASH_STRING,
TorrentFields.ID,
TorrentFields.ETA,
TorrentFields.NAME,
TorrentFields.STATUS,
TorrentFields.IS_PRIVATE,
TorrentFields.DOWNLOADED_EVER
];
// refresh cache
_torrentsCache = (await _client.TorrentGetAsync(fields))
?.Torrents;
}
if (_torrentsCache?.Length is null or 0)
{
_logger.LogDebug("could not list torrents | {url}", _config.Url);
}
torrent = _torrentsCache?.FirstOrDefault(x => x.HashString.Equals(hash, StringComparison.InvariantCultureIgnoreCase));
if (torrent is null)
{
_logger.LogDebug("could not find torrent | {hash} | {url}", hash, _config.Url);
}
return torrent;
}
private async Task<TorrentInfo?> GetTorrentAsync(string hash) =>
(await _client.TorrentGetAsync(Fields, hash))
?.Torrents
?.FirstOrDefault();
}

View File

@@ -4,10 +4,12 @@ namespace Infrastructure.Verticals.Jobs;
public class JobChainingListener : IJobListener
{
private readonly string _firstJobName;
private readonly string _nextJobName;
public JobChainingListener(string nextJobName)
public JobChainingListener(string firstJobName, string nextJobName)
{
_firstJobName = firstJobName;
_nextJobName = nextJobName;
}
@@ -19,7 +21,7 @@ public class JobChainingListener : IJobListener
public async Task JobWasExecuted(IJobExecutionContext context, JobExecutionException? jobException, CancellationToken cancellationToken)
{
if (string.IsNullOrEmpty(_nextJobName) || context.JobDetail.Key.Name == _nextJobName)
if (string.IsNullOrEmpty(_nextJobName) || context.JobDetail.Key.Name == _nextJobName || context.JobDetail.Key.Name != _firstJobName)
{
return;
}

View File

@@ -32,7 +32,7 @@ public class NotificationPublisher : INotificationPublisher
QueueRecord record = ContextProvider.Get<QueueRecord>(nameof(QueueRecord));
InstanceType instanceType = (InstanceType)ContextProvider.Get<object>(nameof(InstanceType));
Uri instanceUrl = ContextProvider.Get<Uri>(nameof(ArrInstance) + nameof(ArrInstance.Url));
Uri imageUrl = GetImageFromContext(record, instanceType);
Uri? imageUrl = GetImageFromContext(record, instanceType);
ArrNotification notification = new()
{
@@ -63,42 +63,59 @@ public class NotificationPublisher : INotificationPublisher
public virtual async Task NotifyQueueItemDeleted(bool removeFromClient, DeleteReason reason)
{
QueueRecord record = ContextProvider.Get<QueueRecord>(nameof(QueueRecord));
InstanceType instanceType = (InstanceType)ContextProvider.Get<object>(nameof(InstanceType));
Uri instanceUrl = ContextProvider.Get<Uri>(nameof(ArrInstance) + nameof(ArrInstance.Url));
Uri imageUrl = GetImageFromContext(record, instanceType);
QueueItemDeletedNotification notification = new()
try
{
InstanceType = instanceType,
InstanceUrl = instanceUrl,
Hash = record.DownloadId.ToLowerInvariant(),
Title = $"Deleting item from queue with reason: {reason}",
Description = record.Title,
Image = imageUrl,
Fields = [new() { Title = "Removed from download client?", Text = removeFromClient ? "Yes" : "No" }]
};
await _dryRunInterceptor.InterceptAsync(Notify<QueueItemDeletedNotification>, notification);
QueueRecord record = ContextProvider.Get<QueueRecord>(nameof(QueueRecord));
InstanceType instanceType = (InstanceType)ContextProvider.Get<object>(nameof(InstanceType));
Uri instanceUrl = ContextProvider.Get<Uri>(nameof(ArrInstance) + nameof(ArrInstance.Url));
Uri? imageUrl = GetImageFromContext(record, instanceType);
QueueItemDeletedNotification notification = new()
{
InstanceType = instanceType,
InstanceUrl = instanceUrl,
Hash = record.DownloadId.ToLowerInvariant(),
Title = $"Deleting item from queue with reason: {reason}",
Description = record.Title,
Image = imageUrl,
Fields = [new() { Title = "Removed from download client?", Text = removeFromClient ? "Yes" : "No" }]
};
await _dryRunInterceptor.InterceptAsync(Notify<QueueItemDeletedNotification>, notification);
}
catch (Exception ex)
{
_logger.LogError(ex, "failed to notify queue item deleted");
}
}
public virtual async Task NotifyDownloadCleaned(double ratio, TimeSpan seedingTime, string categoryName, CleanReason reason)
{
DownloadCleanedNotification notification = new()
try
{
Title = $"Cleaned item from download client with reason: {reason}",
Description = ContextProvider.Get<string>("downloadName"),
Fields =
[
new() { Title = "Hash", Text = ContextProvider.Get<string>("hash").ToLowerInvariant() },
new() { Title = "Category", Text = categoryName.ToLowerInvariant() },
new() { Title = "Ratio", Text = $"{ratio.ToString(CultureInfo.InvariantCulture)}%" },
new() { Title = "Seeding hours", Text = $"{Math.Round(seedingTime.TotalHours, 0).ToString(CultureInfo.InvariantCulture)}h" }
],
Level = NotificationLevel.Important
};
DownloadCleanedNotification notification = new()
{
Title = $"Cleaned item from download client with reason: {reason}",
Description = ContextProvider.Get<string>("downloadName"),
Fields =
[
new() { Title = "Hash", Text = ContextProvider.Get<string>("hash").ToLowerInvariant() },
new() { Title = "Category", Text = categoryName.ToLowerInvariant() },
new() { Title = "Ratio", Text = $"{ratio.ToString(CultureInfo.InvariantCulture)}%" },
new()
{
Title = "Seeding hours", Text = $"{Math.Round(seedingTime.TotalHours, 0).ToString(CultureInfo.InvariantCulture)}h"
}
],
Level = NotificationLevel.Important
};
await _dryRunInterceptor.InterceptAsync(Notify<DownloadCleanedNotification>, notification);
await _dryRunInterceptor.InterceptAsync(Notify<DownloadCleanedNotification>, notification);
}
catch (Exception ex)
{
_logger.LogError(ex, "failed to notify download cleaned");
}
}
[DryRunSafeguard]
@@ -107,12 +124,21 @@ public class NotificationPublisher : INotificationPublisher
return _messageBus.Publish(message);
}
private static Uri GetImageFromContext(QueueRecord record, InstanceType instanceType) =>
instanceType switch
private Uri? GetImageFromContext(QueueRecord record, InstanceType instanceType)
{
Uri? image = instanceType switch
{
InstanceType.Sonarr => record.Series!.Images.FirstOrDefault(x => x.CoverType == "poster")?.RemoteUrl,
InstanceType.Radarr => record.Movie!.Images.FirstOrDefault(x => x.CoverType == "poster")?.RemoteUrl,
InstanceType.Lidarr => record.Album!.Images.FirstOrDefault(x => x.CoverType == "cover")?.Url,
InstanceType.Sonarr => record.Series?.Images?.FirstOrDefault(x => x.CoverType == "poster")?.RemoteUrl,
InstanceType.Radarr => record.Movie?.Images?.FirstOrDefault(x => x.CoverType == "poster")?.RemoteUrl,
InstanceType.Lidarr => record.Album?.Images?.FirstOrDefault(x => x.CoverType == "cover")?.Url,
_ => throw new ArgumentOutOfRangeException(nameof(instanceType))
} ?? throw new Exception("failed to get image url from context");
};
if (image is null)
{
_logger.LogWarning("no poster found for {title}", record.Title);
}
return image;
}
}

View File

@@ -4,6 +4,7 @@ using Common.Configuration.QueueCleaner;
using Domain.Enums;
using Domain.Models.Arr;
using Domain.Models.Arr.Queue;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.Context;
@@ -19,7 +20,8 @@ namespace Infrastructure.Verticals.QueueCleaner;
public sealed class QueueCleaner : GenericHandler
{
private readonly QueueCleanerConfig _config;
private readonly IgnoredDownloadsProvider<QueueCleanerConfig> _ignoredDownloadsProvider;
public QueueCleaner(
ILogger<QueueCleaner> logger,
IOptions<QueueCleanerConfig> config,
@@ -32,7 +34,8 @@ public sealed class QueueCleaner : GenericHandler
LidarrClient lidarrClient,
ArrQueueIterator arrArrQueueIterator,
DownloadServiceFactory downloadServiceFactory,
INotificationPublisher notifier
INotificationPublisher notifier,
IgnoredDownloadsProvider<QueueCleanerConfig> ignoredDownloadsProvider
) : base(
logger, downloadClientConfig,
sonarrConfig, radarrConfig, lidarrConfig,
@@ -42,10 +45,14 @@ public sealed class QueueCleaner : GenericHandler
)
{
_config = config.Value;
_config.Validate();
_ignoredDownloadsProvider = ignoredDownloadsProvider;
}
protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType)
{
IReadOnlyList<string> ignoredDownloads = await _ignoredDownloadsProvider.GetIgnoredDownloads();
using var _ = LogContext.PushProperty("InstanceName", instanceType.ToString());
HashSet<SearchItem> itemsToBeRefreshed = [];
@@ -70,27 +77,41 @@ public sealed class QueueCleaner : GenericHandler
QueueRecord record = group.First();
_logger.LogTrace("processing | {title} | {id}", record.Title, record.DownloadId);
if (!arrClient.IsRecordValid(record))
{
continue;
}
if (ignoredDownloads.Contains(record.DownloadId, StringComparer.InvariantCultureIgnoreCase))
{
_logger.LogInformation("skip | {title} | ignored", record.Title);
continue;
}
// push record to context
ContextProvider.Set(nameof(QueueRecord), record);
StalledResult stalledCheckResult = new();
DownloadCheckResult downloadCheckResult = new();
if (_downloadClientConfig.DownloadClient is not Common.Enums.DownloadClient.None && record.Protocol is "torrent")
if (record.Protocol is "torrent" && _downloadClientConfig.DownloadClient is not Common.Enums.DownloadClient.Disabled)
{
if (_downloadClientConfig.DownloadClient is Common.Enums.DownloadClient.None)
{
_logger.LogWarning("skip | download client is not configured | {title}", record.Title);
continue;
}
// stalled download check
stalledCheckResult = await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId);
downloadCheckResult = await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId, ignoredDownloads);
}
// failed import check
bool shouldRemoveFromArr = await arrClient.ShouldRemoveFromQueue(instanceType, record, stalledCheckResult.IsPrivate);
DeleteReason deleteReason = stalledCheckResult.ShouldRemove ? stalledCheckResult.DeleteReason : DeleteReason.ImportFailed;
bool shouldRemoveFromArr = await arrClient.ShouldRemoveFromQueue(instanceType, record, downloadCheckResult.IsPrivate);
DeleteReason deleteReason = downloadCheckResult.ShouldRemove ? downloadCheckResult.DeleteReason : DeleteReason.ImportFailed;
if (!shouldRemoveFromArr && !stalledCheckResult.ShouldRemove)
if (!shouldRemoveFromArr && !downloadCheckResult.ShouldRemove)
{
_logger.LogInformation("skip | {title}", record.Title);
continue;
@@ -100,20 +121,26 @@ public sealed class QueueCleaner : GenericHandler
bool removeFromClient = true;
if (stalledCheckResult.IsPrivate)
if (downloadCheckResult.IsPrivate)
{
if (stalledCheckResult.ShouldRemove && !_config.StalledDeletePrivate)
{
removeFromClient = false;
}
bool isStalledWithoutPruneFlag =
downloadCheckResult.DeleteReason is DeleteReason.Stalled &&
!_config.StalledDeletePrivate;
bool isSlowWithoutPruneFlag =
downloadCheckResult.DeleteReason is DeleteReason.SlowSpeed or DeleteReason.SlowTime &&
!_config.SlowDeletePrivate;
bool shouldKeepDueToDeleteRules = downloadCheckResult.ShouldRemove && (isStalledWithoutPruneFlag || isSlowWithoutPruneFlag);
bool shouldKeepDueToImportRules = shouldRemoveFromArr && !_config.ImportFailedDeletePrivate;
if (shouldRemoveFromArr && !_config.ImportFailedDeletePrivate)
if (shouldKeepDueToDeleteRules || shouldKeepDueToImportRules)
{
removeFromClient = false;
}
}
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient);
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient, deleteReason);
await _notifier.NotifyQueueItemDeleted(removeFromClient, deleteReason);
}
});

View File

@@ -0,0 +1 @@
ignored

View File

@@ -175,9 +175,10 @@ services:
image: ghcr.io/flmorg/cleanuperr:latest
container_name: cleanuperr
environment:
- TZ=Europe/Bucharest
- DRY_RUN=false
- LOGGING__LOGLEVEL=Debug
- LOGGING__LOGLEVEL=Verbose
- LOGGING__FILE__ENABLED=true
- LOGGING__FILE__PATH=/var/logs
- LOGGING__ENHANCED=true
@@ -190,20 +191,34 @@ services:
- TRIGGERS__DOWNLOADCLEANER=0/30 * * * * ?
- QUEUECLEANER__ENABLED=true
- QUEUECLEANER__IGNORED_DOWNLOADS_PATH=/ignored
- QUEUECLEANER__RUNSEQUENTIALLY=true
- QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE=true
- QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE=false
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__0=file is a sample
- QUEUECLEANER__STALLED_MAX_STRIKES=5
- QUEUECLEANER__STALLED_RESET_STRIKES_ON_PROGRESS=true
- QUEUECLEANER__STALLED_IGNORE_PRIVATE=true
- QUEUECLEANER__STALLED_DELETE_PRIVATE=false
- QUEUECLEANER__SLOW_MAX_STRIKES=5
- QUEUECLEANER__SLOW_RESET_STRIKES_ON_PROGRESS=true
- QUEUECLEANER__SLOW_IGNORE_PRIVATE=false
- QUEUECLEANER__SLOW_DELETE_PRIVATE=false
- QUEUECLEANER__SLOW_MIN_SPEED=1MB
- QUEUECLEANER__SLOW_MAX_TIME=20
- QUEUECLEANER__SLOW_IGNORE_ABOVE_SIZE=1KB
- CONTENTBLOCKER__ENABLED=true
- CONTENTBLOCKER__IGNORED_DOWNLOADS_PATH=/ignored
- CONTENTBLOCKER__IGNORE_PRIVATE=true
- CONTENTBLOCKER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__ENABLED=true
- DOWNLOADCLEANER__IGNORED_DOWNLOADS_PATH=/ignored
- DOWNLOADCLEANER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__CATEGORIES__0__NAME=tv-sonarr
- DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO=-1
@@ -255,6 +270,7 @@ services:
# - NOTIFIARR__CHANNEL_ID=discord_channel_id
volumes:
- ./data/cleanuperr/logs:/var/logs
- ./data/cleanuperr/ignored_downloads:/ignored
restart: unless-stopped
depends_on:
- qbittorrent

View File

@@ -1,45 +1,52 @@
## Table of contents
- [General settings](variables.md#general-settings)
- [Queue Cleaner settings](variables.md#queue-cleaner-settings)
- [Content Blocker settings](variables.md#content-blocker-settings)
- [Download Cleaner settings](variables.md#download-cleaner-settings)
- [Download Client settings](variables.md#download-client-settings)
- [Arr settings](variables.md#arr-settings)
- [Notification settings](variables.md#notification-settings)
- [Advanced settings](variables.md#advanced-settings)
- [General settings](#general-settings)
- [Queue Cleaner settings](#queue-cleaner-settings)
- [Content Blocker settings](#content-blocker-settings)
- [Download Cleaner settings](#download-cleaner-settings)
- [Download Client settings](#download-client-settings)
- [Arr settings](#arr-settings)
- [Notification settings](#notification-settings)
- [Advanced settings](#advanced-settings)
#
### General settings
**`DRY_RUN`**
#### **`TZ`**
- The time zone to use.
- Type: String.
- Possible values: Any valid timezone.
- Default: `UTC`.
- Required: No.
#### **`DRY_RUN`**
- When enabled, simulates irreversible operations (like deletions and notifications) without making actual changes.
- Type: Boolean.
- Possible values: `true`, `false`.
- Default: `false`.
- Required: No.
**`LOGGING__LOGLEVEL`**
#### **`LOGGING__LOGLEVEL`**
- Controls the detail level of application logs.
- Type: String.
- Possible values: `Verbose`, `Debug`, `Information`, `Warning`, `Error`, `Fatal`.
- Default: `Information`.
- Required: No.
**`LOGGING__FILE__ENABLED`**
#### **`LOGGING__FILE__ENABLED`**
- Enables logging to a file.
- Type: Boolean.
- Possible values: `true`, `false`.
- Default: `false`.
- Required: No.
**`LOGGING__FILE__PATH`**
#### **`LOGGING__FILE__PATH`**
- Directory where log files will be saved.
- Type: String.
- Default: Empty.
- Required: No.
**`LOGGING__ENHANCED`**
#### **`LOGGING__ENHANCED`**
- Provides more detailed descriptions in logs whenever possible.
- Type: Boolean.
- Possible values: `true`, `false`.
@@ -51,7 +58,7 @@
### Queue Cleaner settings
**`TRIGGERS__QUEUECLEANER`**
#### **`TRIGGERS__QUEUECLEANER`**
- Cron schedule for the queue cleaner job.
- Type: String - [Quartz cron format](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html).
- Default: `0 0/5 * * * ?` (every 5 minutes).
@@ -59,9 +66,9 @@
> [!NOTE]
> - Maximum interval is 6 hours.
> - Ignored if `QUEUECLEANER__RUNSEQUENTIALLY=true` and `CONTENTBLOCKER__ENABLED=true`.
> - Is ignored if `QUEUECLEANER__RUNSEQUENTIALLY=true` and `CONTENTBLOCKER__ENABLED=true`.
**`QUEUECLEANER__ENABLED`**
#### **`QUEUECLEANER__ENABLED`**
- Enables or disables the queue cleaning functionality.
- When enabled, processes all items in the *arr queue.
- Type: Boolean
@@ -69,7 +76,32 @@
- Default: `true`
- Required: No.
**`QUEUECLEANER__RUNSEQUENTIALLY`**
#### **`QUEUECLEANER__IGNORED_DOWNLOADS_PATH`**
- Local path to the file containing ignored downloads.
- If the contents of the file are changed, they will be reloaded on the next job run.
- Accepted values:
- torrent hash
- qBitTorrent tag or category
- Deluge label
- Transmission category (last directory from the save location)
- torrent tracker domain
- Each value needs to be on a new line.
- Type: String.
- Default: Empty.
- Required: No.
- Example: `/ignored.txt`.
- Example of file contents:
```
fa800a7d7c443a2c3561d1f8f393c089036dade1
tv-sonarr
qbit-tag
mytracker.com
...
```
>[!IMPORTANT]
> Some people have experienced problems using Docker where the mounted file would not update inside the container if it was modified on the host. This is a Docker configuration problem and can not be solved by cleanuperr.
#### **`QUEUECLEANER__RUNSEQUENTIALLY`**
- Controls whether queue cleaner runs after content blocker instead of in parallel.
- When `true`, streamlines the cleaning process by running immediately after content blocker.
- Type: Boolean
@@ -77,23 +109,24 @@
- Default: `true`
- Required: No.
**`QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES`**
#### **`QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES`**
- Number of strikes before removing a failed import.
- Set to `0` to never remove failed imports.
- A strike is given when an item is stalled, stuck in metadata downloading, or failed to be imported.
- A strike is given when an item fails to be imported.
- Type: Integer
- Possible values: `0` or greater
- Default: `0`
- Required: No.
> [!NOTE]
> If not set to `0`, the minimum value is `3`.
**`QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE`**
#### **`QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE`**
- Controls whether to ignore failed imports from private trackers.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE`**
#### **`QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE`**
- Controls whether to delete failed imports from private trackers from the download client.
- Has no effect if `QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE` is `true`.
- Type: Boolean
@@ -102,45 +135,46 @@
- Required: No.
> [!WARNING]
> Setting `QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account.
> Setting `QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
**`QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS`**
#### **`QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS`**
- Patterns to look for in failed import messages that should be ignored.
- Multiple patterns can be specified using incrementing numbers starting from 0.
- Type: String array
- Default: Empty.
- Required: No.
- Example:
```yaml
QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__0: "title mismatch"
QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
```
```yaml
QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__0: "title mismatch"
QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
```
**`QUEUECLEANER__STALLED_MAX_STRIKES`**
#### **`QUEUECLEANER__STALLED_MAX_STRIKES`**
- Number of strikes before removing a stalled download.
- Set to `0` to never remove stalled downloads.
- A strike is given when download speed is 0.
- A strike is given when an item is stalled (not downloading) or stuck while downloading metadata.
- Type: Integer
- Possible values: `0` or greater
- Default: `0`
- Required: No.
> [!NOTE]
> If not set to `0`, the minimum value is `3`.
**`QUEUECLEANER__STALLED_RESET_STRIKES_ON_PROGRESS`**
- Controls whether to remove strikes if any download progress was made since last checked.
#### **`QUEUECLEANER__STALLED_RESET_STRIKES_ON_PROGRESS`**
- Controls whether to remove the given strikes if any download progress was made since last checked.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`QUEUECLEANER__STALLED_IGNORE_PRIVATE`**
#### **`QUEUECLEANER__STALLED_IGNORE_PRIVATE`**
- Controls whether to ignore stalled downloads from private trackers.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`QUEUECLEANER__STALLED_DELETE_PRIVATE`**
- Controls whether to delete stalled private downloads from the download client.
#### **`QUEUECLEANER__STALLED_DELETE_PRIVATE`**
- Controls whether stalled downloads from private trackers should be removed from the download client.
- Has no effect if `QUEUECLEANER__STALLED_IGNORE_PRIVATE` is `true`.
- Type: Boolean
- Possible values: `true`, `false`
@@ -148,13 +182,72 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Required: No.
> [!WARNING]
> Setting `QUEUECLEANER__STALLED_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account.
> Setting `QUEUECLEANER__STALLED_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
#### **`QUEUECLEANER__SLOW_MAX_STRIKES`**
- Number of strikes before removing a slow download.
- Set to `0` to never remove slow downloads.
- A strike is given when an item is slow.
- Type: Integer
- Default: `0`
- Required: No.
> [!NOTE]
> If not set to `0`, the minimum value is `3`.
#### **`QUEUECLEANER__SLOW_RESET_STRIKES_ON_PROGRESS`**
- Controls whether to remove the given strikes if the download speed or estimated time are not slow anymore.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
#### **`QUEUECLEANER__SLOW_IGNORE_PRIVATE`**
- Controls whether to ignore slow downloads from private trackers.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
#### **`QUEUECLEANER__SLOW_DELETE_PRIVATE`**
- Controls whether slow downloads from private trackers should be removed from the download client.
- Has no effect if `QUEUECLEANER__SLOW_IGNORE_PRIVATE` is `true`.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
> [!WARNING]
> Setting `QUEUECLEANER__SLOW_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
#### **`QUEUECLEANER__SLOW_MIN_SPEED`**
- The minimum speed a download should have.
- Downloads receive strikes if their speed falls bellow this value.
- If not specified, downloads will not receive strikes for slow download speed.
- Type: String.
- Default: Empty.
- Required: No.
- Value examples: `1.5KB`, `400KB`, `2MB`
#### **`QUEUECLEANER__SLOW_MAX_TIME`**
- The maximum estimated hours a download should take to finish.
- Downloads receive strikes if their estimated finish time is above this value.
- If not specified (or `0`), downloads will not receive strikes for slow estimated finish time.
- Type: Integer.
- Default: `0`.
- Required: No.
#### **`QUEUECLEANER__SLOW_IGNORE_ABOVE_SIZE`**
- Downloads above this size will not be removed for being slow.
- Type: String.
- Default: Empty.
- Required: No.
- Value examples: `10KB`, `200MB`, `3GB`.
#
### Content Blocker settings
**`TRIGGERS__CONTENTBLOCKER`**
#### **`TRIGGERS__CONTENTBLOCKER`**
- Cron schedule for the content blocker job.
- Type: String - [Quartz cron format](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html).
- Default: `0 0/5 * * * ?` (every 5 minutes).
@@ -163,7 +256,7 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
> [!NOTE]
> - Maximum interval is 6 hours.
**`CONTENTBLOCKER__ENABLED`**
#### **`CONTENTBLOCKER__ENABLED`**
- Enables or disables the content blocker functionality.
- When enabled, processes all items in the *arr queue and marks unwanted files.
- Type: Boolean
@@ -171,14 +264,39 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Default: `false`
- Required: No.
**`CONTENTBLOCKER__IGNORE_PRIVATE`**
#### **`CONTENTBLOCKER__IGNORED_DOWNLOADS_PATH`**
- Local path to the file containing ignored downloads.
- If the contents of the file are changed, they will be reloaded on the next job run.
- Accepted values:
- torrent hash
- qBitTorrent tag or category
- Deluge label
- Transmission category (last directory from the save location)
- torrent tracker domain
- Each value needs to be on a new line.
- Type: String.
- Default: Empty.
- Required: No.
- Example: `/ignored.txt`.
- Example of file contents:
```
fa800a7d7c443a2c3561d1f8f393c089036dade1
tv-sonarr
qbit-tag
mytracker.com
...
```
>[!IMPORTANT]
> Some people have experienced problems using Docker where the mounted file would not update inside the container if it was modified on the host. This is a Docker configuration problem and can not be solved by cleanuperr.
#### **`CONTENTBLOCKER__IGNORE_PRIVATE`**
- Controls whether to ignore downloads from private trackers.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`CONTENTBLOCKER__DELETE_PRIVATE`**
#### **`CONTENTBLOCKER__DELETE_PRIVATE`**
- Controls whether to delete private downloads that have all files blocked from the download client.
- Has no effect if `CONTENTBLOCKER__IGNORE_PRIVATE` is `true`.
- Type: Boolean
@@ -187,13 +305,13 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Required: No.
> [!WARNING]
> Setting `CONTENTBLOCKER__DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account.
> Setting `CONTENTBLOCKER__DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
#
### Download Cleaner settings
**`TRIGGERS__DOWNLOADCLEANER`**
#### **`TRIGGERS__DOWNLOADCLEANER`**
- Cron schedule for the download cleaner job.
- Type: String - [Quartz cron format](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html).
- Default: `0 0 * * * ?` (every hour).
@@ -202,7 +320,7 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
> [!NOTE]
> - Maximum interval is 6 hours.
**`DOWNLOADCLEANER__ENABLED`**
#### **`DOWNLOADCLEANER__ENABLED`**
- Enables or disables the download cleaner functionality.
- When enabled, automatically cleans up downloads that have been seeding for a certain amount of time.
- Type: Boolean.
@@ -210,7 +328,32 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Default: `false`
- Required: No.
**`DOWNLOADCLEANER__DELETE_PRIVATE`**
#### **`DOWNLOADCLEANER__IGNORED_DOWNLOADS_PATH`**
- Local path to the file containing ignored downloads.
- If the contents of the file are changed, they will be reloaded on the next job run.
- Accepted values:
- torrent hash
- qBitTorrent tag or category
- Deluge label
- Transmission category (last directory from the save location)
- torrent tracker domain
- Each value needs to be on a new line.
- Type: String.
- Default: Empty.
- Required: No.
- Example: `/ignored.txt`.
- Example of file contents:
```
fa800a7d7c443a2c3561d1f8f393c089036dade1
tv-sonarr
qbit-tag
mytracker.com
...
```
>[!IMPORTANT]
> Some people have experienced problems using Docker where the mounted file would not update inside the container if it was modified on the host. This is a Docker configuration problem and can not be solved by cleanuperr.
#### **`DOWNLOADCLEANER__DELETE_PRIVATE`**
- Controls whether to delete private downloads.
- Type: Boolean.
- Possible values: `true`, `false`
@@ -218,9 +361,9 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Required: No.
> [!WARNING]
> Setting `DOWNLOADCLEANER__DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account.
> Setting `DOWNLOADCLEANER__DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
**`DOWNLOADCLEANER__CATEGORIES__0__NAME`**
#### **`DOWNLOADCLEANER__CATEGORIES__0__NAME`**
- Name of the category to clean.
- Type: String.
- Default: Empty.
@@ -228,18 +371,21 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
> [!NOTE]
> The category name must match the category that was set in the *arr.
>
> For qBittorrent, the category name is the name of the download category.
>
> For Deluge, the category name is the name of the label.
> For Transmission, the category name is the name of the download location.
>
> For Transmission, the category name is the last directory from the save location.
**`DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO`**
#### **`DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO`**
- Maximum ratio to reach before removing a download.
- Type: Decimal.
- Possible values: `-1` or greater (`-1` means no limit or disabled).
- Default: `-1`
- Required: No.
**`DOWNLOADCLEANER__CATEGORIES__0__MIN_SEED_TIME`**
#### **`DOWNLOADCLEANER__CATEGORIES__0__MIN_SEED_TIME`**
- Minimum number of hours to seed before removing a download, if the ratio has been met.
- Used with `MAX_RATIO` to ensure a minimum seed time.
- Type: Decimal.
@@ -247,7 +393,7 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Default: `0`
- Required: No.
**`DOWNLOADCLEANER__CATEGORIES__0__MAX_SEED_TIME`**
#### **`DOWNLOADCLEANER__CATEGORIES__0__MAX_SEED_TIME`**
- Maximum number of hours to seed before removing a download.
- Type: Decimal.
- Possible values: `-1` or greater (`-1` means no limit or disabled).
@@ -270,59 +416,82 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
### Download Client settings
**`DOWNLOAD_CLIENT`**
#### **`DOWNLOAD_CLIENT`**
- Specifies which download client is used by *arrs.
- Type: String.
- Possible values: `none`, `qbittorrent`, `deluge`, `transmission`.
- Possible values: `none`, `qbittorrent`, `deluge`, `transmission`, `disabled`.
- Default: `none`
- Required: No.
> [!NOTE]
> Only one download client can be enabled at a time. If you have more than one download client, you should deploy multiple instances of cleanuperr.
**`QBITTORRENT__URL`**
> [!IMPORTANT]
> When the download client is set to `disabled`, the queue cleaner will be able to remove items that are failed to be imported even if there is no download client configured. This means that all downloads, including private ones, will be completely removed.
>
> Setting `DOWNLOAD_CLIENT=disabled` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
#### **`QBITTORRENT__URL`**
- URL of the qBittorrent instance.
- Type: String.
- Default: `http://localhost:8080`.
- Required: No.
**`QBITTORRENT__USERNAME`**
#### **`QBITTORRENT__URL_BASE`**
- Adds a prefix to the qBittorrent url, such as `[QBITTORRENT__URL]/[QBITTORRENT__URL_BASE]/api`.
- Type: String.
- Default: Empty.
- Required: No.
#### **`QBITTORRENT__USERNAME`**
- Username for qBittorrent authentication.
- Type: String.
- Default: Empty.
- Required: No.
**`QBITTORRENT__PASSWORD`**
#### **`QBITTORRENT__PASSWORD`**
- Password for qBittorrent authentication.
- Type: String.
- Default: Empty.
- Required: No.
**`DELUGE__URL`**
#### **`DELUGE__URL`**
- URL of the Deluge instance.
- Type: String.
- Default: `http://localhost:8112`.
- Required: No.
**`DELUGE__PASSWORD`**
#### **`DELUGE__URL_BASE`**
- Adds a prefix to the deluge json url, such as `[DELUGE__URL]/[DELUGE__URL_BASE]/json`.
- Type: String.
- Default: Empty.
- Required: No.
#### **`DELUGE__PASSWORD`**
- Password for Deluge authentication.
- Type: String.
- Default: Empty.
- Required: No.
**`TRANSMISSION__URL`**
#### **`TRANSMISSION__URL`**
- URL of the Transmission instance.
- Type: String.
- Default: `http://localhost:9091`.
- Required: No.
**`TRANSMISSION__USERNAME`**
#### **`TRANSMISSION__URL_BASE`**
- Adds a prefix to the Transmission rpc url, such as `[TRANSMISSION__URL]/[TRANSMISSION__URL_BASE]/rpc`.
- Type: String.
- Default: `transmission`.
- Required: No.
#### **`TRANSMISSION__USERNAME`**
- Username for Transmission authentication.
- Type: String.
- Default: Empty.
- Required: No.
**`TRANSMISSION__PASSWORD`**
#### **`TRANSMISSION__PASSWORD`**
- Password for Transmission authentication.
- Type: String.
- Default: Empty.
@@ -332,112 +501,6 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
### Arr settings
**`SONARR__ENABLED`**
- Enables or disables Sonarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`SONARR__BLOCK__TYPE`**
- Determines how file blocking works for Sonarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
**`SONARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
**`SONARR__SEARCHTYPE`**
- Determines what to search for after removing a queue item.
- Type: String
- Possible values: `Episode`, `Season`, `Series`
- Default: `Episode`
- Required: No.
**`SONARR__INSTANCES__0__URL`**
- URL of the Sonarr instance.
- Type: String
- Default: `http://localhost:8989`
- Required: No.
**`SONARR__INSTANCES__0__APIKEY`**
- API key for the Sonarr instance.
- Type: String
- Default: Empty.
- Required: No.
**`RADARR__ENABLED`**
- Enables or disables Radarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`RADARR__BLOCK__TYPE`**
- Determines how file blocking works for Radarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
**`RADARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
**`RADARR__INSTANCES__0__URL`**
- URL of the Radarr instance.
- Type: String
- Default: `http://localhost:7878`
- Required: No.
**`RADARR__INSTANCES__0__APIKEY`**
- API key for the Radarr instance.
- Type: String
- Default: Empty.
- Required: No.
**`LIDARR__ENABLED`**
- Enables or disables Lidarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`LIDARR__BLOCK__TYPE`**
- Determines how file blocking works for Lidarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
**`LIDARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
**`LIDARR__INSTANCES__0__URL`**
- URL of the Lidarr instance.
- Type: String
- Default: `http://localhost:8686`
- Required: No.
**`LIDARR__INSTANCES__0__APIKEY`**
- API key for the Lidarr instance.
- Type: String
- Default: Empty.
- Required: No.
> [!NOTE]
> Multiple instances can be specified for each *arr using this format, where `<NUMBER>` starts from 0:
> ```yaml
@@ -445,8 +508,29 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
> <ARR>__INSTANCES__<NUMBER>__APIKEY
> ```
#### **`SONARR__ENABLED`**
- Enables or disables Sonarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
#### **`SONARR__BLOCK__TYPE`**
- Determines how file blocking works for Sonarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
#### **`SONARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
> [!NOTE]
> The blocklists (blacklist/whitelist) support the following patterns:
> The blocklists support the following patterns:
> ```
> *example // file name ends with "example"
> example* // file name starts with "example"
@@ -456,47 +540,155 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
> ```
> [!NOTE]
> [This blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist) and [this whitelist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist) can be used for Sonarr and Radarr, but they are not suitable for other *arrs.
> [This blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist), [this permissive blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist_permissive) and [this whitelist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist) can be used for Sonarr and Radarr.
#### **`SONARR__SEARCHTYPE`**
- Determines what to search for after removing a queue item.
- Type: String
- Possible values: `Episode`, `Season`, `Series`
- Default: `Episode`
- Required: No.
#### **`SONARR__INSTANCES__0__URL`**
- URL of the Sonarr instance.
- Type: String
- Default: `http://localhost:8989`
- Required: No.
#### **`SONARR__INSTANCES__0__APIKEY`**
- API key for the Sonarr instance.
- Type: String
- Default: Empty.
- Required: No.
#### **`RADARR__ENABLED`**
- Enables or disables Radarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
#### **`RADARR__BLOCK__TYPE`**
- Determines how file blocking works for Radarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
#### **`RADARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
> [!NOTE]
> The blocklists support the following patterns:
> ```
> *example // file name ends with "example"
> example* // file name starts with "example"
> *example* // file name has "example" in the name
> example // file name is exactly the word "example"
> regex:<ANY_REGEX> // regex that needs to be marked at the start of the line with "regex:"
> ```
> [!NOTE]
> [This blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist), [this permissive blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist_permissive) and [this whitelist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist) can be used for Sonarr and Radarr.
#### **`RADARR__INSTANCES__0__URL`**
- URL of the Radarr instance.
- Type: String
- Default: `http://localhost:7878`
- Required: No.
#### **`RADARR__INSTANCES__0__APIKEY`**
- API key for the Radarr instance.
- Type: String
- Default: Empty.
- Required: No.
#### **`LIDARR__ENABLED`**
- Enables or disables Lidarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
#### **`LIDARR__BLOCK__TYPE`**
- Determines how file blocking works for Lidarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
#### **`LIDARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
> [!NOTE]
> The blocklists support the following patterns:
> ```
> *example // file name ends with "example"
> example* // file name starts with "example"
> *example* // file name has "example" in the name
> example // file name is exactly the word "example"
> regex:<ANY_REGEX> // regex that needs to be marked at the start of the line with "regex:"
> ```
#### **`LIDARR__INSTANCES__0__URL`**
- URL of the Lidarr instance.
- Type: String
- Default: `http://localhost:8686`
- Required: No.
#### **`LIDARR__INSTANCES__0__APIKEY`**
- API key for the Lidarr instance.
- Type: String
- Default: Empty.
- Required: No.
#
### Notification settings
**`NOTIFIARR__API_KEY`**
#### **`NOTIFIARR__API_KEY`**
- Notifiarr API key for sending notifications.
- Requires Notifiarr's [`Passthrough`](https://notifiarr.wiki/en/Website/Integrations/Passthrough) integration to work.
- Type: String
- Default: Empty.
- Required: No.
**`NOTIFIARR__CHANNEL_ID`**
#### **`NOTIFIARR__CHANNEL_ID`**
- Discord channel ID where notifications will be sent.
- Type: String
- Default: Empty.
- Required: No.
**`NOTIFIARR__ON_IMPORT_FAILED_STRIKE`**
#### **`NOTIFIARR__ON_IMPORT_FAILED_STRIKE`**
- Controls whether to notify when an item receives a failed import strike.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`NOTIFIARR__ON_STALLED_STRIKE`**
#### **`NOTIFIARR__ON_STALLED_STRIKE`**
- Controls whether to notify when an item receives a stalled download strike.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`NOTIFIARR__ON_QUEUE_ITEM_DELETED`**
#### **`NOTIFIARR__ON_QUEUE_ITEM_DELETED`**
- Controls whether to notify when a queue item is deleted.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`NOTIFIARR__ON_DOWNLOAD_CLEANED`**
#### **`NOTIFIARR__ON_DOWNLOAD_CLEANED`**
- Controls whether to notify when a download is cleaned.
- Type: Boolean
- Possible values: `true`, `false`
@@ -507,7 +699,7 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
### Advanced settings
**`HTTP_MAX_RETRIES`**
#### **`HTTP_MAX_RETRIES`**
- The number of times to retry a failed HTTP call.
- Applies to calls to *arrs, download clients, and other services.
- Type: Integer
@@ -515,10 +707,10 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Default: `0`
- Required: No.
**`HTTP_TIMEOUT`**
#### **`HTTP_TIMEOUT`**
- The number of seconds to wait before failing an HTTP call.
- Applies to calls to *arrs, download clients, and other services.
- Type: Integer
- Possible values: Greater than `0`
- Possible values: Greater than `0`.
- Default: `100`
- Required: No.