Trigger queue cleaner sequentially (#14)

* added option to run queue cleaner after content blocker

* updated readme to clearly state what the jobs do

README.md

@@ -10,36 +10,52 @@ Refer to the [Environment variables](#Environment-variables) section for detaile
 
 ## Important note
 
-Only the <b>latest versions</b> of qBittorrent, Deluge, Sonarr etc. are supported, or earlier versions that have the same API as the latest version.
+Only the **latest versions** of qBittorrent, Deluge, Sonarr etc. are supported, or earlier versions that have the same API as the latest version.
 
-This tool is actively developed and still a work in progress. Join the Discord server if you want to reach out to me quickly (or just stay updated on new releases) so we can squash those pesky bugs together: https://discord.gg/cJYPs9Bt
+This tool is actively developed and still a work in progress. Join the Discord server if you want to reach out to me quickly (or just stay updated on new releases) so we can squash those pesky bugs together:
+
+> https://discord.gg/cJYPs9Bt
+
+# How it works
+
+1. **Content blocker** will:
+   - Run every 5 minutes (or configured cron).
+   - Process all items in the *arr queue.
+   - Find the corresponding item from the download client for each queue item.
+   - Mark the files that were found in the queue as **unwanted/skipped** if:
+     - They **are listed in the blacklist**, or
+     - They **are not included in the whitelist**.
+2. **Queue cleaner** will:
+   - Run every 5 minutes (or configured cron).
+   - Process all items in the *arr queue.
+   - Check each queue item if it meets one of the following condition in the download client:
+     - **Marked as completed, but 0 bytes have been downloaded** (due to files being blocked by qBittorrent or the **content blocker**).
+     - All associated files of are marked as **unwanted/skipped**.
+   - If the item **DOES NOT** match the above criteria, it will be skipped.
+   - If the item **DOES** match the criteria:
+     - It will be removed from the *arr's queue.
+     - It will be deleted from the download client.
+     - A new search will be triggered for the *arr item.
 
 # Setup
 
 ## Using qBittorrent's built-in feature (works only with qBittorrent)
 
-1. Go to qBittorrent -> Options -> Downloads -> make sure `Excluded file names` is checked -> Set an exclusion list.
-   - [blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist)
-   - [permissive blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist_permissive)
+1. Go to qBittorrent -> Options -> Downloads -> make sure `Excluded file names` is checked -> Paste an exclusion list that you have copied.
+   - [blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist), or
+   - [permissive blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist_permissive), or
    - create your own
-2. Start cleanuperr with `QUEUECLEANER__ENABLED` set to `true`.
-3. cleanuperr will execute a queue cleaner cron job at every 5 minutes that will:
-   1. go through all items from Sonarr/Radarr's queue.
-   2. each a queue item is checked: 
-      - if it has been <b>marked as completed and 0 bytes have been downloaded</b> (because qBittorrent blocked the files).
-      - if all its files are skipped.
-   3. if the item <b>IS NOT</b> as described, it is skipped.
-   4. if the item <b>IS</b> as described, it is removed from Sonarr/Radarr's queue, removed from qBittorrent and a search is triggered for the show/movie.
+2. qBittorrent will block files from being downloaded. In the case of malicious content, **nothing is downloaded and the torrent is marked as complete**.
+3. Start **cleanuperr** with `QUEUECLEANER__ENABLED` set to `true`.
+4. The **queue cleaner** will perform a cleanup process as described in the [How it works](#how-it-works) section.
 
 ## Using cleanuperr's blocklist (works with all supported download clients)
 
-1. Start cleanuperr with both `QUEUECLEANER_ENABLED` and `CONTENTBLOCKER_ENABLED` set to `true`.
-2. Be sure to set and enable a blacklist or a whitelist as described in the [Environment variables](#Environment-variables) section.
-3. cleanuperr with execute the following jobs:
-   - the same queue cleaner as described [here](#Using-qBittorrents-built-in-feature)
-   - a content blocker cron job at every 5 minutes that will mark files as unwanted/skipped if:
-      - they are in the blacklist.
-      - they are not in the whitelist.
+1. Set both `QUEUECLEANER_ENABLED` and `CONTENTBLOCKER_ENABLED` to `true` in your environment variables.
+2. Configure and enable either a **blacklist** or a **whitelist** as described in the [Environment variables](#Environment-variables) section.
+3. Once configured, cleanuperr will perform the following tasks:
+   - Execute the **content blocker** job, as explained in the [How it works](#how-it-works) section.
+   - Execute the **queue cleaner** job, as explained in the [How it works](#how-it-works) section.
 
 ## Usage
 
@@ -79,6 +95,7 @@ services:
       - TRIGGERS__CONTENTBLOCKER=0 0/5 * * * ?
 
       - QUEUECLEANER__ENABLED=true
+      - QUEUECLEANER__RUNSEQUENTIALLY=true
 
       - CONTENTBLOCKER__ENABLED=true
       - CONTENTBLOCKER__BLACKLIST__ENABLED=true
@@ -126,6 +143,7 @@ services:
 | TRIGGERS__CONTENTBLOCKER | Yes if content blocker is enabled | [Quartz cron trigger](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) | 0 0/5 * * * ? |
 |||||
 | QUEUECLEANER__ENABLED | No | Enable or disable the queue cleaner | true |
+| QUEUECLEANER__RUNSEQUENTIALLY | No | If set to true, the queue cleaner will run after the content blocker instead of running in parallel, streamlining the cleaning process | true |
 |||||
 | CONTENTBLOCKER__ENABLED | No | Enable or disable the content blocker | false |
 | CONTENTBLOCKER__BLACKLIST__ENABLED | Yes if content blocker is enabled and whitelist is not enabled | Enable or disable the blacklist | false |
@@ -181,7 +199,7 @@ SONARR__INSTANCES__<NUMBER>__APIKEY
 
 1. Download the binaries from [releases](https://github.com/flmorg/cleanuperr/releases).
 2. Extract them from the zip file.
-3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [above](/README.md#environment-variables).
+3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [above](#environment-variables).
 
 ### Run as a Windows Service
 

code/Common/Configuration/QueueCleaner/QueueCleanerConfig.cs

@@ -5,6 +5,8 @@ public sealed record QueueCleanerConfig : IJobConfig
     public const string SectionName = "QueueCleaner";
     
     public required bool Enabled { get; init; }
+    
+    public required bool RunSequentially { get; init; }
 
     public void Validate()
     {

code/Executable/DependencyInjection/QuartzDI.cs

@@ -37,32 +37,33 @@ public static class QuartzDI
         TriggersConfig triggersConfig
     )
     {
-        q.AddJob<QueueCleaner, QueueCleanerConfig>(
-            configuration,
-            QueueCleanerConfig.SectionName,
-            triggersConfig.QueueCleaner
-        );
+        ContentBlockerConfig? contentBlockerConfig = configuration
+            .GetRequiredSection(ContentBlockerConfig.SectionName)
+            .Get<ContentBlockerConfig>();
         
-        q.AddJob<ContentBlocker, ContentBlockerConfig>(
-            configuration,
-            ContentBlockerConfig.SectionName,
-            triggersConfig.ContentBlocker
-        );
+        q.AddJob<ContentBlocker>(contentBlockerConfig, triggersConfig.ContentBlocker);
+        
+        QueueCleanerConfig? queueCleanerConfig = configuration
+            .GetRequiredSection(QueueCleanerConfig.SectionName)
+            .Get<QueueCleanerConfig>();
+
+        if (contentBlockerConfig?.Enabled is true && queueCleanerConfig is { Enabled: true, RunSequentially: true })
+        {
+            q.AddJob<QueueCleaner>(queueCleanerConfig, string.Empty);
+            q.AddJobListener(new JobChainingListener(nameof(QueueCleaner)));
+        }
+        else
+        {
+            q.AddJob<QueueCleaner>(queueCleanerConfig, triggersConfig.QueueCleaner);
+        }
     }
     
-    private static void AddJob<T, TConfig>(
+    private static void AddJob<T>(
         this IServiceCollectionQuartzConfigurator q,
-        IConfiguration configuration,
-        string configSectionName,
+        IJobConfig? config,
         string trigger
-    )
-        where T: GenericHandler
-        where TConfig : IJobConfig
+    ) where T: GenericHandler
     {
-        IJobConfig? config = configuration
-            .GetRequiredSection(configSectionName)
-            .Get<TConfig>();
-        
         string typeName = typeof(T).Name;
         
         if (config is null)
@@ -75,11 +76,25 @@ public static class QuartzDI
             return;
         }
 
+        bool hasTrigger = trigger.Length > 0;
+
         q.AddJob<GenericJob<T>>(opts =>
         {
             opts.WithIdentity(typeName);
+
+            if (!hasTrigger)
+            {
+                // jobs with no triggers need to be stored durably
+                opts.StoreDurably();
+            }
         });
 
+        // skip empty triggers
+        if (!hasTrigger)
+        {
+            return;
+        }
+
         q.AddTrigger(opts =>
         {
             opts.ForJob(typeName)

code/Executable/appsettings.Development.json

@@ -12,7 +12,7 @@
     "ContentBlocker": "0/10 * * * * ?"
   },
   "ContentBlocker": {
-    "Enabled": false,
+    "Enabled": true,
     "Blacklist": {
       "Enabled": false,
       "Path": "https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist"
@@ -23,7 +23,8 @@
     }
   },
   "QueueCleaner": {
-    "Enabled": true
+    "Enabled": true,
+    "RunSequentially": true
   },
   "qBittorrent": {
     "Enabled": true,

code/Executable/appsettings.json

@@ -23,7 +23,8 @@
     }
   },
   "QueueCleaner": {
-    "Enabled": true
+    "Enabled": true,
+    "RunSequentially": true
   },
   "qBittorrent": {
     "Enabled": true,

code/Infrastructure/Infrastructure.csproj

@@ -16,6 +16,7 @@
       <PackageReference Include="Microsoft.Extensions.Http" Version="8.0.0" />
       <PackageReference Include="Microsoft.Extensions.Options" Version="8.0.1" />
       <PackageReference Include="QBittorrent.Client" Version="1.9.24285.1" />
+      <PackageReference Include="Quartz" Version="3.13.1" />
     </ItemGroup>
 
 </Project>

code/Infrastructure/Verticals/Jobs/JobChainingListener.cs

@@ -0,0 +1,35 @@
+using Quartz;
+
+namespace Infrastructure.Verticals.Jobs;
+
+public class JobChainingListener : IJobListener
+{
+    private readonly string _nextJobName;
+
+    public JobChainingListener(string nextJobName)
+    {
+        _nextJobName = nextJobName;
+    }
+
+    public string Name => nameof(JobChainingListener);
+
+    public Task JobExecutionVetoed(IJobExecutionContext context, CancellationToken cancellationToken) => Task.CompletedTask;
+
+    public Task JobToBeExecuted(IJobExecutionContext context, CancellationToken cancellationToken) => Task.CompletedTask;
+
+    public async Task JobWasExecuted(IJobExecutionContext context, JobExecutionException? jobException, CancellationToken cancellationToken)
+    {
+        if (string.IsNullOrEmpty(_nextJobName))
+        {
+            return;
+        }
+        
+        IScheduler scheduler = context.Scheduler;
+        JobKey nextJobKey = new(_nextJobName);
+
+        if (await scheduler.CheckExists(nextJobKey, cancellationToken))
+        {
+            await scheduler.TriggerJob(nextJobKey, cancellationToken);
+        }
+    }
+}

code/test/docker-compose.yml

@@ -177,6 +177,7 @@ services:
       - TRIGGERS__CONTENTBLOCKER=0/30 * * * * ?
 
       - QUEUECLEANER__ENABLED=true
+      - QUEUECLEANER__RUNSEQUENTIALLY=true
 
       - CONTENTBLOCKER__ENABLED=true
       - CONTENTBLOCKER__BLACKLIST__ENABLED=true