Compare commits

...

100 Commits

Author SHA1 Message Date
Robert McRackan
68ad627159 update dependencies 2022-04-30 21:03:40 -04:00
Robert McRackan
878a5dd36c Libary import got a complete overhaul. On a library of 1,200 titles: initial scan is 80-85% faster. Subsequent imports are 60-70% faster 2022-04-29 16:35:49 -04:00
Robert McRackan
7c144b8277 Bug fix #234 : chapters were no longer included in the m4b file 2022-04-27 11:31:13 -04:00
Robert McRackan
bca8c3865b Expose a way to insert ad hoc library books to grid 2022-04-26 16:37:13 -04:00
Robert McRackan
58102acd35 Trivial refactoring 2022-04-26 16:34:59 -04:00
Robert McRackan
5e577843f7 Fixing genre metatag is conditional upon AllowLibationFixup setting 2022-04-26 09:49:21 -04:00
Robert McRackan
e1d549cead update dependencies 2022-04-26 09:27:13 -04:00
Robert McRackan
323b8f2fb9 minor refactors 2022-04-26 08:18:35 -04:00
Robert McRackan
3dcbcf42ed Fix 2 for issue #202 2022-04-25 22:21:36 -04:00
Robert McRackan
825078abc6 New feature: metadata correction in converted output files 2022-04-25 13:34:22 -04:00
Robert McRackan
6be44966ad * enhancement #202 : use audible category for file's genre metatag. Thanks @MBucari ! 2022-04-25 13:23:43 -04:00
rmcrackan
66da138556 Merge pull request #233 from Mbucari/master
Update Libation to work with new AAXClean.Codecs
2022-04-25 13:16:38 -04:00
Michael Bucari-Tovo
e5dd4b856e Update Libation to work with new AAXClean.Codecs 2022-04-24 19:40:34 -06:00
Robert McRackan
5caa9c5687 Improved logging for import errors 2022-04-16 16:36:49 -04:00
Robert McRackan
c8c0ffeb0d Bug fix #231 : Some books without categories are not getting imported 2022-04-15 16:30:43 -04:00
Robert McRackan
bfceb58d6b Merge branch 'master' of https://github.com/rmcrackan/Libation 2022-04-12 09:16:10 -04:00
Robert McRackan
2e4c4cf5f7 bug fix #228 : recover from corrupt BookTags.json 2022-04-12 09:16:02 -04:00
rmcrackan
23966c9b00 Update README.md 2022-04-11 10:06:12 -04:00
rmcrackan
ef73d2243d add login details 2022-04-11 10:03:08 -04:00
Robert McRackan
c95feebd39 Add images for 'alternate login' readme 2022-04-11 09:56:59 -04:00
Robert McRackan
d6601fed83 Bug fix #225 : SaferEnumerateFiles will skip files with UnauthorizedAccessException 2022-04-08 09:11:36 -04:00
Robert McRackan
962e379642 Added debugging around file move and file delete 2022-03-30 09:52:39 -04:00
Robert McRackan
cbc61f5a2d Files and folders cannot end with dots 2022-03-25 13:51:29 -04:00
Robert McRackan
2eaac6acc2 Bug fix #210 : if attempting to paginate more than 10,000 titles : "Implied library size is unsupported" 2022-03-17 16:14:12 -04:00
Robert McRackan
03b458765c Bug fix: getting library had errors for libraries with over 10k titles 2022-03-17 13:06:55 -04:00
Robert McRackan
c8b4bc6361 Address issue #199 : Rare users can be unable to handle library scan batch size of 250. When timeout occurs, retry with batch size of 50 2022-03-06 14:21:21 -05:00
rmcrackan
d9b5725ff1 Update README.md
CLI manual change to settings file no longer needed
2022-03-03 16:59:31 -05:00
Robert McRackan
0a0f60192b Better exception logging with Serilog.Exceptions library 2022-03-03 16:54:43 -05:00
Robert McRackan
424d939c15 Update dependencies 2022-03-03 13:02:15 -05:00
Robert McRackan
87f13ff8ed Incr. ver. 2022-02-21 10:26:58 -05:00
Robert McRackan
1e24df626a Add error recovery around FileLocations.json to handle file corruption 2022-02-21 10:24:56 -05:00
Robert McRackan
0312786721 Add description to exports #197 2022-02-14 13:44:54 -05:00
Robert McRackan
1f8a5b256e Bug fix: Defensive FirstOrDefault. #194 2022-02-03 08:53:41 -05:00
rmcrackan
426391f01c Update README.md
Forgot to remove .nfo from readme
2022-01-30 09:55:34 -05:00
Robert McRackan
c296bff47f Bug fix #181 : if audible gives a null picture id then once that image is searched for, no further cover art was downloaded 2022-01-12 22:20:58 -05:00
Robert McRackan
6b649cf4ca Rare duplicates are making their way into the db. Defensive FirstOrDefault added to address bug #184 2022-01-11 07:34:42 -05:00
Robert McRackan
5103240a76 Bugfix: Latest AudibleAPI addresses #175 . Thank you again @mkb79 2022-01-04 11:25:49 -05:00
rmcrackan
c2418b10f6 Update README.md 2022-01-04 10:53:53 -05:00
Robert McRackan
d705c23472 Bug fix #158 : troublesome check is unnecessary anyway 2021-12-07 16:06:15 -05:00
Robert McRackan
de45d008c7 Bug fix #167 : folders with leading or trailing whitespace will break file saving. Including paths created from templates 2021-12-07 09:24:36 -05:00
Robert McRackan
c267332027 update dependencies 2021-12-06 15:12:44 -05:00
Robert McRackan
4829e85faf Bug fix #163 , #171 : pre-audible uk logins were failing. Thanks @mkb79 ! 2021-12-06 13:14:06 -05:00
Robert McRackan
2acb9ca7e5 Bug fix: Spent hours hunting down why database files weren't closing correctly. New to EF Core 6 "SQLite: Connections are pooled" " This results in database files being kept open by the process even after the ADO.NET connection object is closed." wtf microsoft?! 2021-12-06 12:00:12 -05:00
Robert McRackan
b260554a2a Bug fix #173 : error when importing the same book from multiple accounts during the same import 2021-12-06 11:05:46 -05:00
Robert McRackan
41a4055cd9 init default settings 2021-12-03 14:47:21 -05:00
Robert McRackan
c6e9ba9bf9 new user: init settings 2021-12-02 16:47:35 -05:00
Robert McRackan
5059333b38 Bug fix: First click on Liberated icon shows 'File not found: <temppath>'. Second click opens correct final path. #164 2021-11-30 09:54:32 -05:00
Robert McRackan
b4015030cf tl;dr text on alt login. Issue #160 2021-11-29 11:12:26 -05:00
Robert McRackan
7f5cf8f018 New config setting: ShowImportedStats -- "Show number of newly imported titles? When unchecked, no pop-up will appear after library scan." 2021-11-29 11:06:23 -05:00
rmcrackan
2c9ccd9c78 Update README.md
TOC
2021-11-29 09:05:52 -05:00
rmcrackan
cebf218db4 Update README.md
Add 'Installation'
2021-11-29 09:03:51 -05:00
Robert McRackan
530b44a0e6 Bug fix: in paths, double slashes are not allowed *except* at beginning. eg: \\192.168.0.1 (issue #157 ) 2021-11-24 13:42:11 -05:00
Robert McRackan
b3dc5a7054 Upgrade to .net6 2021-11-24 12:59:02 -05:00
Robert McRackan
2567ccb44c Enhancement: add if-series conditional logic to custom file naming. Issue #151 2021-11-11 16:43:44 -05:00
Robert McRackan
e67eac92fd Bug fix for issue #152 : individual episodes were ignored when targeted directly (as opposed to targeting the parent series) 2021-11-09 21:27:44 -05:00
Robert McRackan
6e84fd97f1 Accounts dialog: more user-friendly validation 2021-11-08 08:55:07 -05:00
Robert McRackan
9a458bf3dc Protect against blank settings.json 2021-11-05 15:25:31 -04:00
Robert McRackan
283a46e1e2 Add debugging for issue #149 2021-11-05 13:42:47 -04:00
Robert McRackan
6ff2859c39 Update dependencies 2021-11-04 16:13:43 -04:00
Robert McRackan
e8df4952fc update dependencies 2021-11-03 16:51:40 -04:00
rmcrackan
b19e1e8a30 Update README for Custom File Naming 2021-11-02 22:20:51 -04:00
Robert McRackan
a3cf6ac40d Custom file naming: manual testing complete 2021-11-02 21:53:58 -04:00
Robert McRackan
ab450c37c4 Custom File Naming complete. Final testing remains 2021-11-02 17:05:29 -04:00
Robert McRackan
c837fefbdd template file naming: code complete. Clean up and testing remain 2021-11-02 14:26:11 -04:00
Robert McRackan
46b120ee41 Bug fix: slashes in template values (eg: title) breaks file management #145 2021-11-01 11:42:05 -04:00
Robert McRackan
cae8ca7ef3 Template error and warning checks return specific errors/warnings in addition to bools 2021-10-29 17:05:57 -04:00
Robert McRackan
904665da7f Bug fix: #143 2021-10-28 20:58:32 -04:00
Robert McRackan
2478c61df6 Bugfix: template validation was opposite #142 2021-10-28 14:38:01 -04:00
Robert McRackan
288ed75b5d increm ver 2021-10-27 20:46:02 -04:00
Robert McRackan
ad5efbd9a9 Bug fix for #141 2021-10-27 20:45:21 -04:00
Robert McRackan
7eb7b2a0f9 better formatting for download/decrypt ETA 2021-10-27 17:05:17 -04:00
Robert McRackan
d0051c0f02 Add templates to settings dialog incl load validate save. Edit buttons are in place but currently do nothing 2021-10-27 16:51:31 -04:00
Robert McRackan
d20517063e Settings: single screen => tabs 2021-10-27 15:50:41 -04:00
Robert McRackan
bcca69a102 Bug fix. Wrong template referenced 2021-10-26 16:35:08 -04:00
Robert McRackan
35f8c05106 File naming is Configuration driven: Configuration, AudioFileStorageExt, Templates, TemplateTags, 2021-10-26 16:18:27 -04:00
Robert McRackan
a3d38e082d Path.GetInvalidPathChars() acts differently in C# interactive vs live code 2021-10-26 13:54:37 -04:00
Robert McRackan
b2e956e70b Update dependencies 2021-10-26 13:06:24 -04:00
Robert McRackan
e5119357b2 File naming is fully template driven 2021-10-22 17:06:42 -04:00
Robert McRackan
b42ff827d5 GetStandardizedExtension unit tests 2021-10-22 13:09:05 -04:00
Robert McRackan
68da9779da Expose internal to Test projects 2021-10-22 11:07:18 -04:00
Robert McRackan
8e358d8f04 expose library book to multipart decrypter for file naming 2021-10-21 16:43:49 -04:00
Robert McRackan
0a986238bc Defensive FirstOrDefault 2021-10-21 15:39:53 -04:00
Robert McRackan
d636ceed8e File naming stuff is (finally) centralized under AudioFileStorageExt 2021-10-21 14:38:59 -04:00
Robert McRackan
e4fc104afe Naming logic for all new files can now originate from domian logic 2021-10-20 10:56:07 -04:00
Robert McRackan
87e3075fb3 Rename InternalUtilities to AudibleUtilities 2021-10-19 10:22:42 -04:00
Robert McRackan
ab44823c05 Bug fix: update book series. Defensive FirstOrDefault 2021-10-18 20:40:56 -04:00
Robert McRackan
2767f04621 split AaxcDownload single and multi 2021-10-18 14:41:57 -04:00
Robert McRackan
0f1ff0aa10 minor refactor 2021-10-18 13:56:12 -04:00
Robert McRackan
c1af253300 fix partial rollback 2021-10-18 13:44:40 -04:00
Robert McRackan
d08962cffa Refactor valid path/filename. Centralize validaion. Universal templating is one step closer 2021-10-18 13:36:55 -04:00
Robert McRackan
7720110460 Bug fix: tag filters stopped working on 8/22 2021-10-15 14:45:35 -04:00
Robert McRackan
dfa5829cbd Safe(r)Delete, Safe(r)Move : could have infinite loop of exceptions. Fixed. Limit 3 2021-10-12 17:05:01 -04:00
Robert McRackan
648b84ee55 All audible-related file naming terminates at FileUtility
File extensions: Dinah.Core => Libation FileUtility
2021-10-12 14:48:32 -04:00
Robert McRackan
6a81b9b02d more LibationFileManager rename. and bug fix 2021-10-11 17:10:37 -04:00
Robert McRackan
c43e03b228 FileManager: separate generic from Libation-specific 2021-10-11 16:06:50 -04:00
Robert McRackan
1de7edd9df Chapter splitting: file names need more leading zeros when qty >100 2021-10-09 14:20:21 -04:00
Robert McRackan
df90094884 Replaced another id dependency with cache. Now safe for multi-file audiobooks. Also safe for current session not trying to move files created in a previous session or a parallel session of a different title 2021-10-08 21:34:42 -04:00
Robert McRackan
c9a6c8fd35 Merge branch 'master' of https://github.com/rmcrackan/Libation 2021-10-08 11:48:05 -04:00
Robert McRackan
d0b78cc501 New and moved files Upsert themselves in FileManager.FilePathCache 2021-10-08 11:47:54 -04:00
rmcrackan
0b7bc4d938 Update README.md
List all `libationcli export` options
2021-10-07 08:59:13 -04:00
134 changed files with 4492 additions and 1790 deletions

View File

@@ -1,12 +1,16 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0</TargetFramework>
<TargetFramework>net6.0-windows</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AAXClean" Version="0.1.9" />
<PackageReference Include="Dinah.Core" Version="1.1.1.2" />
<PackageReference Include="AAXClean" Version="0.3.2" />
<PackageReference Include="AAXClean.Codecs" Version="0.1.10" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\FileManager\FileManager.csproj" />
</ItemGroup>
</Project>

View File

@@ -0,0 +1,95 @@
using System;
using AAXClean;
using Dinah.Core.Net.Http;
namespace AaxDecrypter
{
public abstract class AaxcDownloadConvertBase : AudiobookDownloadBase
{
public event EventHandler<AppleTags> RetrievedMetadata;
protected OutputFormat OutputFormat { get; }
protected AaxFile AaxFile;
protected AaxcDownloadConvertBase(string outFileName, string cacheDirectory, DownloadLicense dlLic, OutputFormat outputFormat)
: base(outFileName, cacheDirectory, dlLic)
{
OutputFormat = outputFormat;
}
/// <summary>Setting cover art by this method will insert the art into the audiobook metadata</summary>
public override void SetCoverArt(byte[] coverArt)
{
base.SetCoverArt(coverArt);
if (coverArt is not null && AaxFile?.AppleTags is not null)
AaxFile.AppleTags.Cover = coverArt;
}
protected bool Step_GetMetadata()
{
AaxFile = new AaxFile(InputFileStream);
OnRetrievedTitle(AaxFile.AppleTags.TitleSansUnabridged);
OnRetrievedAuthors(AaxFile.AppleTags.FirstAuthor ?? "[unknown]");
OnRetrievedNarrators(AaxFile.AppleTags.Narrator ?? "[unknown]");
OnRetrievedCoverArt(AaxFile.AppleTags.Cover);
RetrievedMetadata?.Invoke(this, AaxFile.AppleTags);
return !IsCanceled;
}
protected DownloadProgress Step_DownloadAudiobook_Start()
{
var zeroProgress = new DownloadProgress
{
BytesReceived = 0,
ProgressPercentage = 0,
TotalBytesToReceive = InputFileStream.Length
};
OnDecryptProgressUpdate(zeroProgress);
AaxFile.SetDecryptionKey(DownloadLicense.AudibleKey, DownloadLicense.AudibleIV);
return zeroProgress;
}
protected void Step_DownloadAudiobook_End(DownloadProgress zeroProgress)
{
AaxFile.Close();
CloseInputFileStream();
OnDecryptProgressUpdate(zeroProgress);
}
protected void AaxFile_ConversionProgressUpdate(object sender, ConversionProgressEventArgs e)
{
var duration = AaxFile.Duration;
var remainingSecsToProcess = (duration - e.ProcessPosition).TotalSeconds;
var estTimeRemaining = remainingSecsToProcess / e.ProcessSpeed;
if (double.IsNormal(estTimeRemaining))
OnDecryptTimeRemaining(TimeSpan.FromSeconds(estTimeRemaining));
var progressPercent = e.ProcessPosition.TotalSeconds / duration.TotalSeconds;
OnDecryptProgressUpdate(
new DownloadProgress
{
ProgressPercentage = 100 * progressPercent,
BytesReceived = (long)(InputFileStream.Length * progressPercent),
TotalBytesToReceive = InputFileStream.Length
});
}
public override void Cancel()
{
IsCanceled = true;
AaxFile?.Cancel();
AaxFile?.Dispose();
CloseInputFileStream();
}
}
}

View File

@@ -1,225 +0,0 @@
using AAXClean;
using Dinah.Core;
using Dinah.Core.IO;
using Dinah.Core.Net.Http;
using Dinah.Core.StepRunner;
using System;
using System.IO;
using System.Linq;
namespace AaxDecrypter
{
public class AaxcDownloadConverter : AudiobookDownloadBase
{
const int MAX_FILENAME_LENGTH = 255;
private static readonly TimeSpan minChapterLength = TimeSpan.FromSeconds(3);
protected override StepSequence steps { get; }
private AaxFile aaxFile;
private OutputFormat OutputFormat { get; }
public AaxcDownloadConverter(string outFileName, string cacheDirectory, DownloadLicense dlLic, OutputFormat outputFormat, bool splitFileByChapters)
:base(outFileName, cacheDirectory, dlLic)
{
OutputFormat = outputFormat;
steps = new StepSequence
{
Name = "Download and Convert Aaxc To " + OutputFormat,
["Step 1: Get Aaxc Metadata"] = Step1_GetMetadata,
["Step 2: Download Decrypted Audiobook"] = splitFileByChapters
? Step2_DownloadAudiobookAsMultipleFilesPerChapter
: Step2_DownloadAudiobookAsSingleFile,
["Step 3: Create Cue"] = splitFileByChapters
? () => true
: Step3_CreateCue,
["Step 4: Cleanup"] = Step4_Cleanup,
};
}
/// <summary>
/// Setting cover art by this method will insert the art into the audiobook metadata
/// </summary>
public override void SetCoverArt(byte[] coverArt)
{
base.SetCoverArt(coverArt);
aaxFile?.AppleTags.SetCoverArt(coverArt);
}
protected override bool Step1_GetMetadata()
{
aaxFile = new AaxFile(InputFileStream);
OnRetrievedTitle(aaxFile.AppleTags.TitleSansUnabridged);
OnRetrievedAuthors(aaxFile.AppleTags.FirstAuthor ?? "[unknown]");
OnRetrievedNarrators(aaxFile.AppleTags.Narrator ?? "[unknown]");
OnRetrievedCoverArt(aaxFile.AppleTags.Cover);
return !isCanceled;
}
protected override bool Step2_DownloadAudiobookAsSingleFile()
{
var zeroProgress = Step2_Start();
if (File.Exists(outputFileName))
FileExt.SafeDelete(outputFileName);
var outputFile = File.Open(outputFileName, FileMode.OpenOrCreate, FileAccess.ReadWrite);
aaxFile.ConversionProgressUpdate += AaxFile_ConversionProgressUpdate;
var decryptionResult = OutputFormat == OutputFormat.M4b ? aaxFile.ConvertToMp4a(outputFile, downloadLicense.ChapterInfo) : aaxFile.ConvertToMp3(outputFile);
aaxFile.ConversionProgressUpdate -= AaxFile_ConversionProgressUpdate;
downloadLicense.ChapterInfo = aaxFile.Chapters;
Step2_End(zeroProgress);
return decryptionResult == ConversionResult.NoErrorsDetected && !isCanceled;
}
private bool Step2_DownloadAudiobookAsMultipleFilesPerChapter()
{
var zeroProgress = Step2_Start();
var chapters = downloadLicense.ChapterInfo.Chapters.ToList();
//Ensure split files are at least minChapterLength in duration.
var splitChapters = new ChapterInfo();
var runningTotal = TimeSpan.Zero;
string title = "";
for (int i = 0; i < chapters.Count; i++)
{
if (runningTotal == TimeSpan.Zero)
title = chapters[i].Title;
runningTotal += chapters[i].Duration;
if (runningTotal >= minChapterLength)
{
splitChapters.AddChapter(title, runningTotal);
runningTotal = TimeSpan.Zero;
}
}
aaxFile.ConversionProgressUpdate += AaxFile_ConversionProgressUpdate;
if(OutputFormat == OutputFormat.M4b)
ConvertToMultiMp4b(splitChapters);
else
ConvertToMultiMp3(splitChapters);
aaxFile.ConversionProgressUpdate -= AaxFile_ConversionProgressUpdate;
Step2_End(zeroProgress);
return !isCanceled;
}
private DownloadProgress Step2_Start()
{
var zeroProgress = new DownloadProgress
{
BytesReceived = 0,
ProgressPercentage = 0,
TotalBytesToReceive = InputFileStream.Length
};
OnDecryptProgressUpdate(zeroProgress);
aaxFile.SetDecryptionKey(downloadLicense.AudibleKey, downloadLicense.AudibleIV);
return zeroProgress;
}
private void Step2_End(DownloadProgress zeroProgress)
{
aaxFile.Close();
CloseInputFileStream();
OnDecryptProgressUpdate(zeroProgress);
}
private void ConvertToMultiMp4b(ChapterInfo splitChapters)
{
var chapterCount = 0;
aaxFile.ConvertToMultiMp4a(splitChapters, newSplitCallback =>
{
var fileName = GetMultipartFileName(outputFileName, ++chapterCount, newSplitCallback.Chapter.Title);
if (File.Exists(fileName))
FileExt.SafeDelete(fileName);
newSplitCallback.OutputFile = File.Open(fileName, FileMode.OpenOrCreate);
});
}
private void ConvertToMultiMp3(ChapterInfo splitChapters)
{
var chapterCount = 0;
aaxFile.ConvertToMultiMp3(splitChapters, newSplitCallback =>
{
var fileName = GetMultipartFileName(outputFileName, ++chapterCount, newSplitCallback.Chapter.Title);
if (File.Exists(fileName))
FileExt.SafeDelete(fileName);
newSplitCallback.OutputFile = File.Open(fileName, FileMode.OpenOrCreate);
newSplitCallback.LameConfig.ID3.Track = chapterCount.ToString();
});
}
private static string GetMultipartFileName(string baseFileName, int chapterCount, string chapterTitle)
{
string extension = Path.GetExtension(baseFileName);
var fileNameChars = $"{Path.GetFileNameWithoutExtension(baseFileName)} - {chapterCount:D2} - {chapterTitle}".ToCharArray();
//Replace illegal path characters with spaces.
for (int i = 0; i <fileNameChars.Length; i++)
{
foreach (var illegal in Path.GetInvalidFileNameChars())
{
if (fileNameChars[i] == illegal)
{
fileNameChars[i] = ' ';
break;
}
}
}
var fileName = new string(fileNameChars).Truncate(MAX_FILENAME_LENGTH - extension.Length);
return Path.Combine(Path.GetDirectoryName(baseFileName), fileName + extension);
}
private void AaxFile_ConversionProgressUpdate(object sender, ConversionProgressEventArgs e)
{
var duration = aaxFile.Duration;
double remainingSecsToProcess = (duration - e.ProcessPosition).TotalSeconds;
double estTimeRemaining = remainingSecsToProcess / e.ProcessSpeed;
if (double.IsNormal(estTimeRemaining))
OnDecryptTimeRemaining(TimeSpan.FromSeconds(estTimeRemaining));
double progressPercent = e.ProcessPosition.TotalSeconds / duration.TotalSeconds;
OnDecryptProgressUpdate(
new DownloadProgress
{
ProgressPercentage = 100 * progressPercent,
BytesReceived = (long)(InputFileStream.Length * progressPercent),
TotalBytesToReceive = InputFileStream.Length
});
}
public override void Cancel()
{
isCanceled = true;
aaxFile?.Cancel();
aaxFile?.Dispose();
CloseInputFileStream();
}
protected override int GetSpeedup(TimeSpan elapsed)
=> (int)(aaxFile.Duration.TotalSeconds / (long)elapsed.TotalSeconds);
}
}

View File

@@ -0,0 +1,143 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using AAXClean;
using AAXClean.Codecs;
using Dinah.Core.StepRunner;
using FileManager;
namespace AaxDecrypter
{
public class AaxcDownloadMultiConverter : AaxcDownloadConvertBase
{
protected override StepSequence Steps { get; }
private Func<MultiConvertFileProperties, string> multipartFileNameCallback { get; }
private static TimeSpan minChapterLength { get; } = TimeSpan.FromSeconds(3);
private List<string> multiPartFilePaths { get; } = new List<string>();
public AaxcDownloadMultiConverter(string outFileName, string cacheDirectory, DownloadLicense dlLic, OutputFormat outputFormat,
Func<MultiConvertFileProperties, string> multipartFileNameCallback = null)
: base(outFileName, cacheDirectory, dlLic, outputFormat)
{
Steps = new StepSequence
{
Name = "Download and Convert Aaxc To " + OutputFormat,
["Step 1: Get Aaxc Metadata"] = Step_GetMetadata,
["Step 2: Download Decrypted Audiobook"] = Step_DownloadAudiobookAsMultipleFilesPerChapter,
["Step 3: Cleanup"] = Step_Cleanup,
};
this.multipartFileNameCallback = multipartFileNameCallback ?? MultiConvertFileProperties.DefaultMultipartFilename;
}
/*
https://github.com/rmcrackan/Libation/pull/127#issuecomment-939088489
If the chapter truly is empty, that is, 0 audio frames in length, then yes it is ignored.
If the chapter is shorter than 3 seconds long but still has some audio frames, those frames are combined with the following chapter and not split into a new file.
I also implemented file naming by chapter title. When 2 or more consecutive chapters are combined, the first of the combined chapter's title is used in the file name. For example, given an audiobook with the following chapters:
00:00:00 - 00:00:02 | Part 1
00:00:02 - 00:35:00 | Chapter 1
00:35:02 - 01:02:00 | Chapter 2
01:02:00 - 01:02:02 | Part 2
01:02:02 - 01:41:00 | Chapter 3
01:41:00 - 02:05:00 | Chapter 4
The book will be split into the following files:
00:00:00 - 00:35:00 | Book - 01 - Part 1.m4b
00:35:00 - 01:02:00 | Book - 02 - Chapter 2.m4b
01:02:00 - 01:41:00 | Book - 03 - Part 2.m4b
01:41:00 - 02:05:00 | Book - 04 - Chapter 4.m4b
That naming may not be desirable for everyone, but it's an easy change to instead use the last of the combined chapter's title in the file name.
*/
private bool Step_DownloadAudiobookAsMultipleFilesPerChapter()
{
var zeroProgress = Step_DownloadAudiobook_Start();
var chapters = DownloadLicense.ChapterInfo.Chapters.ToList();
// Ensure split files are at least minChapterLength in duration.
var splitChapters = new ChapterInfo();
var runningTotal = TimeSpan.Zero;
string title = "";
for (int i = 0; i < chapters.Count; i++)
{
if (runningTotal == TimeSpan.Zero)
title = chapters[i].Title;
runningTotal += chapters[i].Duration;
if (runningTotal >= minChapterLength)
{
splitChapters.AddChapter(title, runningTotal);
runningTotal = TimeSpan.Zero;
}
}
// reset, just in case
multiPartFilePaths.Clear();
AaxFile.ConversionProgressUpdate += AaxFile_ConversionProgressUpdate;
if (OutputFormat == OutputFormat.M4b)
ConvertToMultiMp4a(splitChapters);
else
ConvertToMultiMp3(splitChapters);
AaxFile.ConversionProgressUpdate -= AaxFile_ConversionProgressUpdate;
Step_DownloadAudiobook_End(zeroProgress);
var success = !IsCanceled;
if (success)
foreach (var path in multiPartFilePaths)
OnFileCreated(path);
return success;
}
private void ConvertToMultiMp4a(ChapterInfo splitChapters)
{
var chapterCount = 0;
AaxFile.ConvertToMultiMp4a(splitChapters, newSplitCallback =>
createOutputFileStream(++chapterCount, splitChapters, newSplitCallback)
);
}
private void ConvertToMultiMp3(ChapterInfo splitChapters)
{
var chapterCount = 0;
AaxFile.ConvertToMultiMp3(splitChapters, newSplitCallback =>
{
createOutputFileStream(++chapterCount, splitChapters, newSplitCallback);
((NAudio.Lame.LameConfig)newSplitCallback.UserState).ID3.Track = chapterCount.ToString();
});
}
private void createOutputFileStream(int currentChapter, ChapterInfo splitChapters, NewSplitCallback newSplitCallback)
{
var fileName = multipartFileNameCallback(new()
{
OutputFileName = OutputFileName,
PartsPosition = currentChapter,
PartsTotal = splitChapters.Count,
Title = newSplitCallback?.Chapter?.Title
});
fileName = FileUtility.GetValidFilename(fileName);
multiPartFilePaths.Add(fileName);
FileUtility.SaferDelete(fileName);
newSplitCallback.OutputFile = File.Open(fileName, FileMode.OpenOrCreate);
}
}
}

View File

@@ -0,0 +1,54 @@
using System;
using System.IO;
using AAXClean;
using AAXClean.Codecs;
using Dinah.Core.StepRunner;
using FileManager;
namespace AaxDecrypter
{
public class AaxcDownloadSingleConverter : AaxcDownloadConvertBase
{
protected override StepSequence Steps { get; }
public AaxcDownloadSingleConverter(string outFileName, string cacheDirectory, DownloadLicense dlLic, OutputFormat outputFormat)
: base(outFileName, cacheDirectory, dlLic, outputFormat)
{
Steps = new StepSequence
{
Name = "Download and Convert Aaxc To " + OutputFormat,
["Step 1: Get Aaxc Metadata"] = Step_GetMetadata,
["Step 2: Download Decrypted Audiobook"] = Step_DownloadAudiobookAsSingleFile,
["Step 3: Create Cue"] = Step_CreateCue,
["Step 4: Cleanup"] = Step_Cleanup,
};
}
private bool Step_DownloadAudiobookAsSingleFile()
{
var zeroProgress = Step_DownloadAudiobook_Start();
FileUtility.SaferDelete(OutputFileName);
var outputFile = File.Open(OutputFileName, FileMode.OpenOrCreate, FileAccess.ReadWrite);
AaxFile.ConversionProgressUpdate += AaxFile_ConversionProgressUpdate;
var decryptionResult
= OutputFormat == OutputFormat.M4b
? AaxFile.ConvertToMp4a(outputFile, DownloadLicense.ChapterInfo)
: AaxFile.ConvertToMp3(outputFile);
AaxFile.ConversionProgressUpdate -= AaxFile_ConversionProgressUpdate;
DownloadLicense.ChapterInfo = AaxFile.Chapters;
Step_DownloadAudiobook_End(zeroProgress);
var success = decryptionResult == ConversionResult.NoErrorsDetected && !IsCanceled;
if (success)
base.OnFileCreated(OutputFileName);
return success;
}
}
}

View File

@@ -1,13 +1,9 @@
using Dinah.Core;
using Dinah.Core.IO;
using System;
using System.IO;
using Dinah.Core;
using Dinah.Core.Net.Http;
using Dinah.Core.StepRunner;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using FileManager;
namespace AaxDecrypter
{
@@ -21,81 +17,74 @@ namespace AaxDecrypter
public event EventHandler<byte[]> RetrievedCoverArt;
public event EventHandler<DownloadProgress> DecryptProgressUpdate;
public event EventHandler<TimeSpan> DecryptTimeRemaining;
public event EventHandler<string> FileCreated;
public string AppName { get; set; }
protected bool isCanceled { get; set; }
protected string outputFileName { get; }
protected string cacheDir { get; }
protected DownloadLicense downloadLicense { get; }
protected bool IsCanceled { get; set; }
protected string OutputFileName { get; private set; }
protected DownloadLicense DownloadLicense { get; }
protected NetworkFileStream InputFileStream => (nfsPersister ??= OpenNetworkFileStream()).NetworkFileStream;
// Don't give the property a 'set'. This should have to be an obvious choice; not accidental
protected void SetOutputFileName(string newOutputFileName) => OutputFileName = newOutputFileName;
protected abstract StepSequence steps { get; }
protected abstract StepSequence Steps { get; }
private NetworkFileStreamPersister nfsPersister;
private string jsonDownloadState => Path.Combine(cacheDir, Path.GetFileNameWithoutExtension(outputFileName) + ".json");
private string tempFile => PathLib.ReplaceExtension(jsonDownloadState, ".tmp");
private string jsonDownloadState { get; }
private string tempFile => Path.ChangeExtension(jsonDownloadState, ".tmp");
public AudiobookDownloadBase(string outFileName, string cacheDirectory, DownloadLicense dlLic)
protected AudiobookDownloadBase(string outFileName, string cacheDirectory, DownloadLicense dlLic)
{
AppName = GetType().Name;
OutputFileName = ArgumentValidator.EnsureNotNullOrWhiteSpace(outFileName, nameof(outFileName));
ArgumentValidator.EnsureNotNullOrWhiteSpace(outFileName, nameof(outFileName));
outputFileName = outFileName;
var outDir = Path.GetDirectoryName(outputFileName);
var outDir = Path.GetDirectoryName(OutputFileName);
if (!Directory.Exists(outDir))
throw new ArgumentNullException(nameof(outDir), "Directory does not exist");
if (File.Exists(outputFileName))
File.Delete(outputFileName);
throw new DirectoryNotFoundException($"Directory does not exist: {nameof(outDir)}");
if (!Directory.Exists(cacheDirectory))
throw new ArgumentNullException(nameof(cacheDirectory), "Directory does not exist");
cacheDir = cacheDirectory;
throw new DirectoryNotFoundException($"Directory does not exist: {nameof(cacheDirectory)}");
jsonDownloadState = Path.Combine(cacheDirectory, Path.ChangeExtension(OutputFileName, ".json"));
downloadLicense = ArgumentValidator.EnsureNotNull(dlLic, nameof(dlLic));
DownloadLicense = ArgumentValidator.EnsureNotNull(dlLic, nameof(dlLic));
// delete file after validation is complete
FileUtility.SaferDelete(OutputFileName);
}
public abstract void Cancel();
protected abstract int GetSpeedup(TimeSpan elapsed);
protected abstract bool Step2_DownloadAudiobookAsSingleFile();
protected abstract bool Step1_GetMetadata();
public virtual void SetCoverArt(byte[] coverArt)
{
if (coverArt is null) return;
OnRetrievedCoverArt(coverArt);
if (coverArt is not null)
OnRetrievedCoverArt(coverArt);
}
public bool Run()
{
var (IsSuccess, Elapsed) = steps.Run();
var (IsSuccess, Elapsed) = Steps.Run();
if (!IsSuccess)
{
Console.WriteLine("WARNING-Conversion failed");
return false;
}
Serilog.Log.Logger.Error("Conversion failed");
//Serilog.Log.Logger.Information($"Speedup is {GetSpeedup(Elapsed)}x realtime.");
return true;
}
return IsSuccess;
}
protected void OnRetrievedTitle(string title)
protected void OnRetrievedTitle(string title)
=> RetrievedTitle?.Invoke(this, title);
protected void OnRetrievedAuthors(string authors)
protected void OnRetrievedAuthors(string authors)
=> RetrievedAuthors?.Invoke(this, authors);
protected void OnRetrievedNarrators(string narrators)
protected void OnRetrievedNarrators(string narrators)
=> RetrievedNarrators?.Invoke(this, narrators);
protected void OnRetrievedCoverArt(byte[] coverArt)
protected void OnRetrievedCoverArt(byte[] coverArt)
=> RetrievedCoverArt?.Invoke(this, coverArt);
protected void OnDecryptProgressUpdate(DownloadProgress downloadProgress)
=> DecryptProgressUpdate?.Invoke(this, downloadProgress);
protected void OnDecryptTimeRemaining(TimeSpan timeRemaining)
protected void OnDecryptTimeRemaining(TimeSpan timeRemaining)
=> DecryptTimeRemaining?.Invoke(this, timeRemaining);
protected void OnFileCreated(string path)
=> FileCreated?.Invoke(this, path);
protected void CloseInputFileStream()
{
@@ -103,62 +92,59 @@ namespace AaxDecrypter
nfsPersister?.Dispose();
}
protected bool Step3_CreateCue()
protected bool Step_CreateCue()
{
// not a critical step. its failure should not prevent future steps from running
try
{
File.WriteAllText(PathLib.ReplaceExtension(outputFileName, ".cue"), Cue.CreateContents(Path.GetFileName(outputFileName), downloadLicense.ChapterInfo));
var path = Path.ChangeExtension(OutputFileName, ".cue");
path = FileUtility.GetValidFilename(path);
File.WriteAllText(path, Cue.CreateContents(Path.GetFileName(OutputFileName), DownloadLicense.ChapterInfo));
OnFileCreated(path);
}
catch (Exception ex)
{
Serilog.Log.Logger.Error(ex, $"{nameof(Step3_CreateCue)}. FAILED");
Serilog.Log.Logger.Error(ex, $"{nameof(Step_CreateCue)}. FAILED");
}
return !isCanceled;
return !IsCanceled;
}
protected bool Step4_Cleanup()
protected bool Step_Cleanup()
{
FileExt.SafeDelete(jsonDownloadState);
FileExt.SafeDelete(tempFile);
return !isCanceled;
FileUtility.SaferDelete(jsonDownloadState);
FileUtility.SaferDelete(tempFile);
return !IsCanceled;
}
private NetworkFileStreamPersister OpenNetworkFileStream()
{
NetworkFileStreamPersister nfsp;
if (!File.Exists(jsonDownloadState))
return NewNetworkFilePersister();
if (File.Exists(jsonDownloadState))
try
{
try
{
nfsp = new NetworkFileStreamPersister(jsonDownloadState);
//If More than ~1 hour has elapsed since getting the download url, it will expire.
//The new url will be to the same file.
nfsp.NetworkFileStream.SetUriForSameFile(new Uri(downloadLicense.DownloadUrl));
}
catch
{
FileExt.SafeDelete(jsonDownloadState);
FileExt.SafeDelete(tempFile);
nfsp = NewNetworkFilePersister();
}
var nfsp = new NetworkFileStreamPersister(jsonDownloadState);
// If More than ~1 hour has elapsed since getting the download url, it will expire.
// The new url will be to the same file.
nfsp.NetworkFileStream.SetUriForSameFile(new Uri(DownloadLicense.DownloadUrl));
return nfsp;
}
else
catch
{
nfsp = NewNetworkFilePersister();
FileUtility.SaferDelete(jsonDownloadState);
FileUtility.SaferDelete(tempFile);
return NewNetworkFilePersister();
}
return nfsp;
}
private NetworkFileStreamPersister NewNetworkFilePersister()
{
var headers = new System.Net.WebHeaderCollection
{
{ "User-Agent", downloadLicense.UserAgent }
{ "User-Agent", DownloadLicense.UserAgent }
};
var networkFileStream = new NetworkFileStream(tempFile, new Uri(downloadLicense.DownloadUrl), 0, headers);
var networkFileStream = new NetworkFileStream(tempFile, new Uri(DownloadLicense.DownloadUrl), 0, headers);
return new NetworkFileStreamPersister(networkFileStream, jsonDownloadState);
}
}

View File

@@ -0,0 +1,25 @@
using System;
using System.IO;
using FileManager;
namespace AaxDecrypter
{
public class MultiConvertFileProperties
{
public string OutputFileName { get; set; }
public int PartsPosition { get; set; }
public int PartsTotal { get; set; }
public string Title { get; set; }
public static string DefaultMultipartFilename(MultiConvertFileProperties multiConvertFileProperties)
{
var template = Path.ChangeExtension(multiConvertFileProperties.OutputFileName, null) + " - <ch# 0> - <title>" + Path.GetExtension(multiConvertFileProperties.OutputFileName);
var fileNamingTemplate = new FileNamingTemplate(template) { IllegalCharacterReplacements = " " };
fileNamingTemplate.AddParameterReplacement("ch# 0", FileUtility.GetSequenceFormatted(multiConvertFileProperties.PartsPosition, multiConvertFileProperties.PartsTotal));
fileNamingTemplate.AddParameterReplacement("title", multiConvertFileProperties.Title ?? "");
return fileNamingTemplate.GetFilePath();
}
}
}

View File

@@ -193,9 +193,6 @@ namespace AaxDecrypter
if (response.StatusCode != HttpStatusCode.PartialContent)
throw new WebException($"Server at {Uri.Host} responded with unexpected status code: {response.StatusCode}.");
if (response.Headers.GetValues("Accept-Ranges").FirstOrDefault(r => r.EqualsInsensitive("bytes")) is null)
throw new WebException($"Server at {Uri.Host} does not support Http ranges");
//Content length is the length of the range request, and it is only equal
//to the complete file length if requesting Range: bytes=0-
if (WritePosition == 0)

View File

@@ -1,56 +1,47 @@
using Dinah.Core.IO;
using System;
using System.Threading;
using Dinah.Core.Net.Http;
using Dinah.Core.StepRunner;
using System;
using System.IO;
using System.Linq;
using System.Threading;
using FileManager;
namespace AaxDecrypter
{
public class UnencryptedAudiobookDownloader : AudiobookDownloadBase
{
protected override StepSequence steps { get; }
protected override StepSequence Steps { get; }
public UnencryptedAudiobookDownloader(string outFileName, string cacheDirectory, DownloadLicense dlLic)
: base(outFileName, cacheDirectory, dlLic)
{
steps = new StepSequence
Steps = new StepSequence
{
Name = "Download Mp3 Audiobook",
["Step 1: Get Mp3 Metadata"] = Step1_GetMetadata,
["Step 2: Download Audiobook"] = Step2_DownloadAudiobookAsSingleFile,
["Step 3: Create Cue"] = Step3_CreateCue,
["Step 4: Cleanup"] = Step4_Cleanup,
["Step 1: Get Mp3 Metadata"] = Step_GetMetadata,
["Step 2: Download Audiobook"] = Step_DownloadAudiobookAsSingleFile,
["Step 3: Create Cue"] = Step_CreateCue,
["Step 4: Cleanup"] = Step_Cleanup,
};
}
public override void Cancel()
{
isCanceled = true;
IsCanceled = true;
CloseInputFileStream();
}
protected override int GetSpeedup(TimeSpan elapsed)
{
//Not implemented
return 0;
}
protected override bool Step1_GetMetadata()
protected bool Step_GetMetadata()
{
OnRetrievedCoverArt(null);
return !isCanceled;
return !IsCanceled;
}
protected override bool Step2_DownloadAudiobookAsSingleFile()
private bool Step_DownloadAudiobookAsSingleFile()
{
DateTime startTime = DateTime.Now;
//MUST put InputFileStream.Length first, because it starts background downloader.
// MUST put InputFileStream.Length first, because it starts background downloader.
while (InputFileStream.Length > InputFileStream.WritePosition && !InputFileStream.IsCancelled)
{
@@ -75,12 +66,11 @@ namespace AaxDecrypter
CloseInputFileStream();
if (File.Exists(outputFileName))
FileExt.SafeDelete(outputFileName);
var realOutputFileName = FileUtility.SaferMoveToValidPath(InputFileStream.SaveFilePath, OutputFileName);
SetOutputFileName(realOutputFileName);
OnFileCreated(realOutputFileName);
FileExt.SafeMove(InputFileStream.SaveFilePath, outputFileName);
return !isCanceled;
return !IsCanceled;
}
}
}

View File

@@ -2,8 +2,8 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0</TargetFramework>
<Version>6.2.3.0</Version>
<TargetFramework>net6.0-windows</TargetFramework>
<Version>7.0.0.1</Version>
</PropertyGroup>
<ItemGroup>
@@ -11,11 +11,11 @@
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Octokit" Version="0.50.0" />
<PackageReference Include="Octokit" Version="0.51.0" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\InternalUtilities\InternalUtilities.csproj" />
<ProjectReference Include="..\AudibleUtilities\AudibleUtilities.csproj" />
</ItemGroup>
</Project>

View File

@@ -3,11 +3,11 @@ using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using AudibleUtilities;
using Dinah.Core;
using Dinah.Core.IO;
using Dinah.Core.Logging;
using FileManager;
using InternalUtilities;
using LibationFileManager;
using Newtonsoft.Json.Linq;
using Serilog;
@@ -50,15 +50,48 @@ namespace AppScaffolding
public static void RunPostConfigMigrations(Configuration config)
{
AudibleApiStorage.EnsureAccountsSettingsFileExists();
PopulateMissingConfigValues(config);
//
// migrations go below here
//
Migrations.migrate_to_v5_2_0__post_config(config);
Migrations.migrate_to_v5_7_1(config);
Migrations.migrate_to_v6_1_2(config);
Migrations.migrate_to_v6_2_0(config);
Migrations.migrate_to_v6_6_9(config);
}
public static void PopulateMissingConfigValues(Configuration config)
{
config.InProgress ??= Configuration.WinTemp;
if (!config.Exists(nameof(config.AllowLibationFixup)))
config.AllowLibationFixup = true;
if (!config.Exists(nameof(config.DecryptToLossy)))
config.DecryptToLossy = false;
if (!config.Exists(nameof(config.BadBook)))
config.BadBook = Configuration.BadBookAction.Ask;
if (!config.Exists(nameof(config.DownloadEpisodes)))
config.DownloadEpisodes = true;
if (!config.Exists(nameof(config.ImportEpisodes)))
config.ImportEpisodes = true;
if (!config.Exists(nameof(config.SplitFilesByChapter)))
config.SplitFilesByChapter = false;
if (!config.Exists(nameof(config.FolderTemplate)))
config.FolderTemplate = Templates.Folder.DefaultTemplate;
if (!config.Exists(nameof(config.FileTemplate)))
config.FileTemplate = Templates.File.DefaultTemplate;
if (!config.Exists(nameof(config.ChapterFileTemplate)))
config.ChapterFileTemplate = Templates.ChapterFile.DefaultTemplate;
if (!config.Exists(nameof(config.ShowImportedStats)))
config.ShowImportedStats = true;
}
/// <summary>Initialize logging. Run after migration</summary>
@@ -74,29 +107,12 @@ namespace AppScaffolding
if (config.GetObject("Serilog") != null)
return;
// "Serilog": {
// "MinimumLevel": "Information"
// "WriteTo": [
// {
// "Name": "Console"
// },
// {
// "Name": "File",
// "Args": {
// "rollingInterval": "Day",
// "outputTemplate": ...
// }
// }
// ],
// "Using": [ "Dinah.Core" ],
// "Enrich": [ "WithCaller" ]
// }
var serilogObj = new JObject
{
{ "MinimumLevel", "Information" },
{ "WriteTo", new JArray
{
new JObject { {"Name", "Console" } },
// new JObject { {"Name", "Console" } }, // this has caused more problems than it's solved
new JObject
{
{ "Name", "File" },
@@ -111,14 +127,16 @@ namespace AppScaffolding
// output example: 2019-11-26 08:48:40.224 -05:00 [DBG] Begin Libation
// - with class and method info: "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level:u3}] (at {Caller}) {Message:lj}{NewLine}{Exception}";
// output example: 2019-11-26 08:48:40.224 -05:00 [DBG] (at LibationWinForms.Program.init()) Begin Libation
{ "outputTemplate", "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level:u3}] (at {Caller}) {Message:lj}{NewLine}{Exception}" }
// {Properties:j} needed for expanded exception logging
{ "outputTemplate", "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level:u3}] (at {Caller}) {Message:lj}{NewLine}{Exception} {Properties:j}" }
}
}
}
}
},
{ "Using", new JArray{ "Dinah.Core" } }, // dll's name, NOT namespace
{ "Enrich", new JArray{ "WithCaller" } },
// better exception logging with: Serilog.Exceptions library -- WithExceptionDetails
{ "Using", new JArray{ "Dinah.Core", "Serilog.Exceptions" } }, // dll's name, NOT namespace
{ "Enrich", new JArray{ "WithCaller", "WithExceptionDetails" } },
};
config.SetObject("Serilog", serilogObj);
}
@@ -133,9 +151,9 @@ namespace AppScaffolding
// capture most Console.WriteLine() and write to serilog. See below tests for details.
// Some dependencies print helpful info via Console.WriteLine. We'd like to log it.
//
// Serilog also writes to Console so this might be asking for trouble. ie: infinite loops.
// SerilogTextWriter needs to be more robust and tested. Esp the Write() methods.
// Empirical testing so far has shown no issues.
// If Serilog also writes to Console, this might be asking for trouble. ie: infinite loops.
// To use that way, SerilogTextWriter needs to be more robust and tested. Esp the Write() methods.
// However, empirical testing so far has shown no issues.
Console.SetOut(new MultiTextWriter(origOut, new SerilogTextWriter()));
#region Console => Serilog tests
@@ -211,11 +229,11 @@ namespace AppScaffolding
config.InProgress,
DownloadsInProgressDir = AudibleFileStorage.DownloadsInProgress,
DownloadsInProgressFiles = Directory.EnumerateFiles(AudibleFileStorage.DownloadsInProgress).Count(),
AudibleFileStorage.DownloadsInProgressDirectory,
DownloadsInProgressFiles = FileManager.FileUtility.SaferEnumerateFiles(AudibleFileStorage.DownloadsInProgressDirectory).Count(),
DecryptInProgressDir = AudibleFileStorage.DecryptInProgress,
DecryptInProgressFiles = Directory.EnumerateFiles(AudibleFileStorage.DecryptInProgress).Count(),
AudibleFileStorage.DecryptInProgressDirectory,
DecryptInProgressFiles = FileManager.FileUtility.SaferEnumerateFiles(AudibleFileStorage.DecryptInProgressDirectory).Count(),
});
}
@@ -311,39 +329,53 @@ namespace AppScaffolding
"WinTemp" => Path.GetFullPath(Path.Combine(Path.GetTempPath(), "Libation")),
_ => path
};
public static void migrate_to_v5_2_0__post_config(Configuration config)
{
if (!config.Exists(nameof(config.AllowLibationFixup)))
config.AllowLibationFixup = true;
if (!config.Exists(nameof(config.DecryptToLossy)))
config.DecryptToLossy = false;
}
#endregion
// add config.BadBook
public static void migrate_to_v5_7_1(Configuration config)
public static void migrate_to_v6_6_9(Configuration config)
{
if (!config.Exists(nameof(config.BadBook)))
config.BadBook = Configuration.BadBookAction.Ask;
}
var writeToPath = $"Serilog.WriteTo";
// add config.DownloadEpisodes , config.ImportEpisodes
public static void migrate_to_v6_1_2(Configuration config)
{
if (!config.Exists(nameof(config.DownloadEpisodes)))
config.DownloadEpisodes = true;
// remove WriteTo[].Name == Console
{
if (UNSAFE_MigrationHelper.Settings_TryGetArrayLength(writeToPath, out var length1))
{
for (var i = length1 - 1; i >= 0; i--)
{
var exists = UNSAFE_MigrationHelper.Settings_TryGetFromJsonPath($"{writeToPath}[{i}].Name", out var value);
if (!config.Exists(nameof(config.ImportEpisodes)))
config.ImportEpisodes = true;
}
if (exists && value == "Console")
UNSAFE_MigrationHelper.Settings_RemoveFromArray(writeToPath, i);
}
}
}
// add config.SplitFilesByChapter
public static void migrate_to_v6_2_0(Configuration config)
{
if (!config.Exists(nameof(config.SplitFilesByChapter)))
config.SplitFilesByChapter = false;
// add Serilog.Exceptions -- WithExceptionDetails
{
// outputTemplate should contain "{Properties:j}"
{
// re-calculate. previous loop may have changed the length
if (UNSAFE_MigrationHelper.Settings_TryGetArrayLength(writeToPath, out var length2))
{
var propertyName = "outputTemplate";
for (var i = 0; i < length2; i++)
{
var jsonPath = $"{writeToPath}[{i}].Args";
var exists = UNSAFE_MigrationHelper.Settings_TryGetFromJsonPath($"{jsonPath}.{propertyName}", out var value);
var newValue = "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level:u3}] (at {Caller}) {Message:lj}{NewLine}{Exception} {Properties:j}";
if (exists && value != newValue)
UNSAFE_MigrationHelper.Settings_SetWithJsonPath(jsonPath, propertyName, newValue);
}
}
}
// Serilog.Using must include "Serilog.Exceptions"
UNSAFE_MigrationHelper.Settings_AddUniqueToArray("Serilog.Using", "Serilog.Exceptions");
// Serilog.Enrich must include "WithExceptionDetails"
UNSAFE_MigrationHelper.Settings_AddUniqueToArray("Serilog.Enrich", "WithExceptionDetails");
}
}
}
}

View File

@@ -113,6 +113,108 @@ namespace AppScaffolding
return success;
}
public static bool Settings_JsonPathIsType(string jsonPath, JTokenType jTokenType)
{
JToken val = null;
process_SettingsJson(jObj => val = jObj.SelectToken(jsonPath), false);
return val?.Type == jTokenType;
}
public static bool Settings_TryGetFromJsonPath(string jsonPath, out string value)
{
JToken val = null;
process_SettingsJson(jObj => val = jObj.SelectToken(jsonPath), false);
if (val?.Type == JTokenType.String)
{
value = val.Value<string>();
return true;
}
else
{
value = null;
return false;
}
}
public static void Settings_SetWithJsonPath(string jsonPath, string propertyName, string newValue)
{
if (!Settings_TryGetFromJsonPath($"{jsonPath}.{propertyName}", out _))
return;
process_SettingsJson(jObj =>
{
var token = jObj.SelectToken(jsonPath);
if (token is null
|| token is not JObject o
|| o[propertyName] is null)
return;
var oldValue = token.Value<string>(propertyName);
if (oldValue != newValue)
token[propertyName] = newValue;
});
}
public static bool Settings_TryGetArrayLength(string jsonPath, out int length)
{
length = 0;
if (!Settings_JsonPathIsType(jsonPath, JTokenType.Array))
return false;
JArray array = null;
process_SettingsJson(jObj => array = (JArray)jObj.SelectToken(jsonPath));
length = array.Count;
return true;
}
public static void Settings_AddToArray(string jsonPath, string newValue)
{
if (!Settings_JsonPathIsType(jsonPath, JTokenType.Array))
return;
process_SettingsJson(jObj =>
{
var array = (JArray)jObj.SelectToken(jsonPath);
array.Add(newValue);
});
}
/// <summary>Do not add if already exists</summary>
public static void Settings_AddUniqueToArray(string arrayPath, string newValue)
{
if (!Settings_TryGetArrayLength(arrayPath, out var qty))
return;
for (var i = 0; i < qty; i++)
{
var exists = Settings_TryGetFromJsonPath($"{arrayPath}[{i}]", out var value);
if (exists && value == newValue)
return;
}
Settings_AddToArray(arrayPath, newValue);
}
/// <summary>only remove if not exists</summary>
public static void Settings_RemoveFromArray(string jsonPath, int position)
{
if (!Settings_JsonPathIsType(jsonPath, JTokenType.Array))
return;
process_SettingsJson(jObj =>
{
var array = (JArray)jObj.SelectToken(jsonPath);
if (position < array.Count)
array.RemoveAt(position);
});
}
/// <summary>only insert if not exists</summary>
public static void Settings_Insert(string key, string value)
=> process_SettingsJson(jObj => jObj.TryAdd(key, value));

View File

@@ -1,12 +1,12 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0</TargetFramework>
<TargetFramework>net6.0-windows</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="CsvHelper" Version="27.1.1" />
<PackageReference Include="NPOI" Version="2.5.4" />
<PackageReference Include="CsvHelper" Version="27.2.1" />
<PackageReference Include="NPOI" Version="2.5.6" />
</ItemGroup>
<ItemGroup>

View File

@@ -1,7 +1,7 @@
using System;
using System.Collections.Generic;
using DataLayer;
using FileManager;
using LibationFileManager;
namespace ApplicationServices
{

View File

@@ -3,10 +3,11 @@ using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using AudibleApi;
using AudibleUtilities;
using DataLayer;
using Dinah.Core;
using DtoImporterService;
using InternalUtilities;
using LibationFileManager;
using Serilog;
using static DtoImporterService.PerfLogger;
@@ -43,7 +44,7 @@ namespace ApplicationServices
}
catch (AudibleApi.Authentication.LoginFailedException lfEx)
{
lfEx.SaveFiles(FileManager.Configuration.Instance.LibationFiles);
lfEx.SaveFiles(Configuration.Instance.LibationFiles);
// nuget Serilog.Exceptions would automatically log custom properties
// However, it comes with a scary warning when used with EntityFrameworkCore which I'm not yet ready to implement:
@@ -88,6 +89,9 @@ namespace ApplicationServices
var totalCount = importItems.Count;
Log.Logger.Information($"GetAllLibraryItems: Total count {totalCount}");
if (totalCount == 0)
return default;
Log.Logger.Information("Begin long-running import");
logTime($"pre {nameof(importIntoDbAsync)}");
var newCount = await importIntoDbAsync(importItems);
@@ -98,7 +102,7 @@ namespace ApplicationServices
}
catch (AudibleApi.Authentication.LoginFailedException lfEx)
{
lfEx.SaveFiles(FileManager.Configuration.Instance.LibationFiles);
lfEx.SaveFiles(Configuration.Instance.LibationFiles);
// nuget Serilog.Exceptions would automatically log custom properties
// However, it comes with a scary warning when used with EntityFrameworkCore which I'm not yet ready to implement:
@@ -155,7 +159,7 @@ namespace ApplicationServices
logTime($"pre scanAccountAsync {account.AccountName}");
var dtoItems = await apiExtended.GetLibraryValidatedAsync(libraryResponseGroups, FileManager.Configuration.Instance.ImportEpisodes);
var dtoItems = await apiExtended.GetLibraryValidatedAsync(libraryResponseGroups, Configuration.Instance.ImportEpisodes);
logTime($"post scanAccountAsync {account.AccountName} qty: {dtoItems.Count}");
@@ -163,25 +167,46 @@ namespace ApplicationServices
}
private static async Task<int> importIntoDbAsync(List<ImportItem> importItems)
{
logTime("importIntoDbAsync -- pre db");
using var context = DbContexts.GetContext();
var libraryBookImporter = new LibraryBookImporter(context);
var newCount = await Task.Run(() => libraryBookImporter.Import(importItems));
logTime("importIntoDbAsync -- post Import()");
int qtyChanges = saveChanges(context);
logTime("importIntoDbAsync -- post SaveChanges");
if (qtyChanges > 0)
await Task.Run(() => finalizeLibrarySizeChange());
logTime("importIntoDbAsync -- post finalizeLibrarySizeChange");
return newCount;
}
private static int saveChanges(LibationContext context)
{
logTime("importIntoDbAsync -- pre db");
using var context = DbContexts.GetContext();
var libraryImporter = new LibraryBookImporter(context);
var newCount = await Task.Run(() => libraryImporter.Import(importItems));
logTime("importIntoDbAsync -- post Import()");
var qtyChanges = context.SaveChanges();
logTime("importIntoDbAsync -- post SaveChanges");
try
{
return context.SaveChanges();
}
catch (Microsoft.EntityFrameworkCore.DbUpdateException ex)
{
// DbUpdateException exceptions can wreck serilog. Condense it until we can find a better solution. I suspect the culpret is the "WithExceptionDetails" serilog extension
if (qtyChanges > 0)
await Task.Run(() => finalizeLibrarySizeChange());
logTime("importIntoDbAsync -- post finalizeLibrarySizeChange");
static string format(Exception ex) => $"\r\nMessage: {ex.Message}\r\nStack Trace:\r\n{ex.StackTrace}";
return newCount;
}
#endregion
var msg = "Microsoft.EntityFrameworkCore.DbUpdateException";
if (ex.InnerException is null)
throw new Exception($"{msg}{format(ex)}");
throw new Exception(
$"{msg}{format(ex)}",
new Exception($"Inner Exception{format(ex.InnerException)}"));
}
}
#endregion
#region remove books
public static Task<List<LibraryBook>> RemoveBooksAsync(List<string> idsToRemove) => Task.Run(() => removeBooks(idsToRemove));
#region remove books
public static Task<List<LibraryBook>> RemoveBooksAsync(List<string> idsToRemove) => Task.Run(() => removeBooks(idsToRemove));
private static List<LibraryBook> removeBooks(List<string> idsToRemove)
{
using var context = DbContexts.GetContext();
@@ -228,6 +253,7 @@ namespace ApplicationServices
if (qtyChanges > 0)
{
SearchEngineCommands.UpdateLiberatedStatus(book);
SearchEngineCommands.UpdateBookTags(book);
BookUserDefinedItemCommitted?.Invoke(null, book.AudibleProductId);
}
@@ -244,7 +270,7 @@ namespace ApplicationServices
// must be here instead of in db layer due to AaxcExists
public static LiberatedStatus Liberated_Status(Book book)
=> book.Audio_Exists ? book.UserDefinedItem.BookStatus
: FileManager.AudibleFileStorage.AaxcExists(book.AudibleProductId) ? LiberatedStatus.PartialDownload
: AudibleFileStorage.AaxcExists(book.AudibleProductId) ? LiberatedStatus.PartialDownload
: LiberatedStatus.NotLiberated;
// exists here for feature predictability. It makes sense for this to be where Liberated_Status is

View File

@@ -44,6 +44,9 @@ namespace ApplicationServices
[Name("Length In Minutes")]
public int LengthInMinutes { get; set; }
[Name("Description")]
public string Description { get; set; }
[Name("Publisher")]
public string Publisher { get; set; }
@@ -111,6 +114,7 @@ namespace ApplicationServices
AuthorNames = a.Book.AuthorNames,
NarratorNames = a.Book.NarratorNames,
LengthInMinutes = a.Book.LengthInMinutes,
Description = a.Book.Description,
Publisher = a.Book.Publisher,
HasPdf = a.Book.HasPdf,
SeriesNames = a.Book.SeriesNames,
@@ -180,6 +184,7 @@ namespace ApplicationServices
nameof (ExportDto.AuthorNames),
nameof (ExportDto.NarratorNames),
nameof (ExportDto.LengthInMinutes),
nameof (ExportDto.Description),
nameof (ExportDto.Publisher),
nameof (ExportDto.HasPdf),
nameof (ExportDto.SeriesNames),
@@ -233,6 +238,7 @@ namespace ApplicationServices
row.CreateCell(col++).SetCellValue(dto.AuthorNames);
row.CreateCell(col++).SetCellValue(dto.NarratorNames);
row.CreateCell(col++).SetCellValue(dto.LengthInMinutes);
row.CreateCell(col++).SetCellValue(dto.Description);
row.CreateCell(col++).SetCellValue(dto.Publisher);
row.CreateCell(col++).SetCellValue(dto.HasPdf);
row.CreateCell(col++).SetCellValue(dto.SeriesNames);

View File

@@ -6,7 +6,7 @@ using AudibleApi.Authorization;
using Dinah.Core;
using Newtonsoft.Json;
namespace InternalUtilities
namespace AudibleUtilities
{
public class Account : IUpdatable
{

View File

@@ -6,7 +6,7 @@ using AudibleApi.Authorization;
using Dinah.Core;
using Newtonsoft.Json;
namespace InternalUtilities
namespace AudibleUtilities
{
// 'AccountsSettings' is intentionally NOT IEnumerable<> so that properties can be added/extended
// from newtonsoft (https://www.newtonsoft.com/json/help/html/SerializationGuide.htm):

View File

@@ -3,7 +3,7 @@ using AudibleApi.Authorization;
using Dinah.Core.IO;
using Newtonsoft.Json;
namespace InternalUtilities
namespace AudibleUtilities
{
public class AccountsSettingsPersister : JsonFilePersister<AccountsSettings>
{

View File

@@ -8,7 +8,7 @@ using Dinah.Core;
using Polly;
using Polly.Retry;
namespace InternalUtilities
namespace AudibleUtilities
{
/// <summary>USE THIS from within Libation. It wraps the call with correct JSONPath</summary>
public class ApiExtended
@@ -127,11 +127,18 @@ namespace InternalUtilities
// items = AudibleApi.Common.Converter.FromJson<List<Item>>(System.IO.File.ReadAllText(library_json));
//}
#endif
Serilog.Log.Logger.Debug("Begin initial library scan");
if (!items.Any())
items = await Api.GetAllLibraryItemsAsync(responseGroups);
Serilog.Log.Logger.Debug("Initial library scan complete. Begin episode scan");
await manageEpisodesAsync(items, importEpisodes);
Serilog.Log.Logger.Debug("Episode scan complete");
#if DEBUG
//System.IO.File.WriteAllText(library_json, AudibleApi.Common.Converter.ToJson(items));
#endif
@@ -166,8 +173,7 @@ namespace InternalUtilities
Serilog.Log.Logger.Information($"{parents.Count} series of shows/podcasts found");
// remove episode parents. even if the following stuff fails, these will still be removed from the collection.
// also must happen before processing children because children abuses this flag
// remove episode parents. even if the following stuff fails, these will still be removed from the collection
items.RemoveAll(i => i.IsEpisodes);
if (importEpisodes)
@@ -192,6 +198,14 @@ namespace InternalUtilities
{
var children = await getEpisodeChildrenAsync(parent);
// actual individual episode, not the parent of a series.
// for now I'm keeping it inside this method since it fits the work flow, incl. importEpisodes logic
if (!children.Any())
{
results.Add(parent);
continue;
}
foreach (var child in children)
{
// use parent's 'DateAdded'. DateAdded is just a convenience prop for: PurchaseDate.UtcDateTime
@@ -202,7 +216,8 @@ namespace InternalUtilities
new Series
{
Asin = parent.Asin,
Sequence = parent.Relationships.Single(r => r.Asin == child.Asin).Sort.ToString(),
// This should properly be Single() not FirstOrDefault(), but FirstOrDefault is defensive for malformed data from audible
Sequence = parent.Relationships.FirstOrDefault(r => r.Asin == child.Asin).Sort.ToString(),
Title = parent.TitleWithSubtitle
}
};

View File

@@ -1,9 +1,9 @@
using System;
using System.IO;
using FileManager;
using LibationFileManager;
using Newtonsoft.Json;
namespace InternalUtilities
namespace AudibleUtilities
{
public static class AudibleApiStorage
{

View File

@@ -3,7 +3,7 @@ using System.Collections.Generic;
using System.Linq;
using AudibleApi.Common;
namespace InternalUtilities
namespace AudibleUtilities
{
public interface IValidator
{

View File

@@ -0,0 +1,15 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net6.0-windows</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AudibleApi" Version="2.7.3.1" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\LibationFileManager\LibationFileManager.csproj" />
</ItemGroup>
</Project>

View File

@@ -0,0 +1 @@
[assembly: System.Runtime.CompilerServices.InternalsVisibleTo(nameof(AudibleUtilities) + ".Tests")]

View File

@@ -9,8 +9,8 @@ namespace DataLayer.Configurations
{
entity.HasKey(bc => new { bc.BookId, bc.ContributorId, bc.Role });
entity.HasIndex(b => b.BookId);
entity.HasIndex(b => b.ContributorId);
entity.HasIndex(bc => bc.BookId);
entity.HasIndex(bc => bc.ContributorId);
entity
.HasOne(bc => bc.Book)

View File

@@ -21,12 +21,12 @@ namespace DataLayer.Configurations
// - update LibraryBook import code
// - would likely challenge assumptions throughout Libation which have been true up until now
entity.HasKey(b => b.BookId);
entity.HasKey(lb => lb.BookId);
entity
.HasOne(le => le.Book)
.HasOne(lb => lb.Book)
.WithOne()
.HasForeignKey<LibraryBook>(le => le.BookId);
.HasForeignKey<LibraryBook>(lb => lb.BookId);
}
}
}

View File

@@ -7,10 +7,10 @@ namespace DataLayer.Configurations
{
public void Configure(EntityTypeBuilder<SeriesBook> entity)
{
entity.HasKey(bc => new { bc.SeriesId, bc.BookId });
entity.HasKey(sb => new { sb.SeriesId, sb.BookId });
entity.HasIndex(b => b.SeriesId);
entity.HasIndex(b => b.BookId);
entity.HasIndex(sb => sb.SeriesId);
entity.HasIndex(sb => sb.BookId);
entity
.HasOne(sb => sb.Series)

View File

@@ -7,8 +7,8 @@ namespace DataLayer.Configurations
{
public void Configure(EntityTypeBuilder<Series> entity)
{
entity.HasKey(b => b.SeriesId);
entity.HasIndex(b => b.AudibleSeriesId);
entity.HasKey(s => s.SeriesId);
entity.HasIndex(s => s.AudibleSeriesId);
entity
.Metadata

View File

@@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0</TargetFramework>
<TargetFramework>net6.0-windows</TargetFramework>
</PropertyGroup>
<PropertyGroup>
@@ -12,20 +12,20 @@
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Dinah.EntityFrameworkCore" Version="1.0.5.2" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="5.0.10">
<PackageReference Include="Dinah.EntityFrameworkCore" Version="4.0.0.3" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="6.0.4">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="5.0.10" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="5.0.10">
<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="6.0.4" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="6.0.4">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\FileManager\FileManager.csproj" />
<ProjectReference Include="..\LibationFileManager\LibationFileManager.csproj" />
</ItemGroup>
<ItemGroup>

View File

@@ -185,6 +185,9 @@ namespace DataLayer
{
get
{
if (_seriesLink is null)
return "";
// first: alphabetical by name
var withNames = _seriesLink
.Where(s => !string.IsNullOrWhiteSpace(s.Series.Name))

View File

@@ -18,7 +18,7 @@ namespace DataLayer
public class Category
{
// Empty is a special case. use private ctor w/o validation
public static Category GetEmpty() => new Category { CategoryId = -1, AudibleCategoryId = "", Name = "" };
public static Category GetEmpty() => new() { CategoryId = -1, AudibleCategoryId = "", Name = "" };
internal int CategoryId { get; private set; }
public string AudibleCategoryId { get; private set; }

View File

@@ -7,7 +7,7 @@ namespace DataLayer
public class Contributor
{
// Empty is a special case. use private ctor w/o validation
public static Contributor GetEmpty() => new Contributor { ContributorId = -1, Name = "" };
public static Contributor GetEmpty() => new() { ContributorId = -1, Name = "" };
// contributors search links are just name with url-encoding. space can be + or %20
// author search link: /search?searchAuthor=Robert+Bevan
@@ -31,16 +31,12 @@ namespace DataLayer
public string AudibleContributorId { get; private set; }
private Contributor() { }
public Contributor(string name)
public Contributor(string name, string audibleContributorId = null)
{
ArgumentValidator.EnsureNotNullOrWhiteSpace(name, nameof(name));
Name = ArgumentValidator.EnsureNotNullOrWhiteSpace(name, nameof(name));
_booksLink = new HashSet<BookContributor>();
Name = name;
}
public Contributor(string name, string audibleContributorId) : this(name)
{
// don't overwrite with null or whitespace but not an error
if (!string.IsNullOrWhiteSpace(audibleContributorId))
AudibleContributorId = audibleContributorId;

View File

@@ -33,7 +33,7 @@ namespace DataLayer
// import previously saved tags
ArgumentValidator.EnsureNotNullOrWhiteSpace(book.AudibleProductId, nameof(book.AudibleProductId));
Tags = FileManager.TagsPersistence.GetTags(book.AudibleProductId);
Tags = LibationFileManager.TagsPersistence.GetTags(book.AudibleProductId);
}
#region Tags

View File

@@ -19,7 +19,8 @@ namespace DataLayer
public static Book GetBook(this IQueryable<Book> books, string productId)
=> books
.GetBooks()
.SingleOrDefault(b => b.AudibleProductId == productId);
// 'Single' is more accurate but 'First' is faster and less error prone
.FirstOrDefault(b => b.AudibleProductId == productId);
/// <summary>This is still IQueryable. YOU MUST CALL ToList() YOURSELF</summary>
public static IQueryable<Book> GetBooks(this IQueryable<Book> books, Expression<Func<Book, bool>> predicate)

View File

@@ -24,7 +24,7 @@ namespace DataLayer
.Select(t => (t.Book.AudibleProductId, t.Tags))
.ToList();
FileManager.TagsPersistence.Save(tagsCollection);
LibationFileManager.TagsPersistence.Save(tagsCollection);
}
}
}

View File

@@ -1,38 +0,0 @@
using System;
using System.Collections.Generic;
using System.Linq;
namespace DataLayer.Utilities
{
public static class LocalDatabaseInfo
{
public static List<string> GetLocalDBInstances()
{
// Start the child process.
using var p = new System.Diagnostics.Process
{
StartInfo = new System.Diagnostics.ProcessStartInfo
{
UseShellExecute = false,
RedirectStandardOutput = true,
FileName = "cmd.exe",
Arguments = "/C sqllocaldb info",
CreateNoWindow = true,
WindowStyle = System.Diagnostics.ProcessWindowStyle.Hidden
}
};
p.Start();
var output = p.StandardOutput.ReadToEnd();
p.WaitForExit();
// if LocalDb is not installed then it will return that 'sqllocaldb' is not recognized as an internal or external command operable program or batch file
return string.IsNullOrWhiteSpace(output) || output.Contains("not recognized")
? new List<string>()
: output
.Split(new string[] { Environment.NewLine }, StringSplitOptions.None)
.Select(i => i.Trim())
.Where(i => !string.IsNullOrEmpty(i))
.ToList();
}
}
}

View File

@@ -2,64 +2,55 @@
using System.Collections.Generic;
using System.Linq;
using AudibleApi.Common;
using AudibleUtilities;
using DataLayer;
using InternalUtilities;
using Dinah.Core.Collections.Generic;
namespace DtoImporterService
{
public class BookImporter : ItemsImporterBase
{
public BookImporter(LibationContext context) : base(context) { }
protected override IValidator Validator => new BookValidator();
public override IEnumerable<Exception> Validate(IEnumerable<ImportItem> importItems) => new BookValidator().Validate(importItems.Select(i => i.DtoItem));
public Dictionary<string, Book> Cache { get; private set; } = new();
private ContributorImporter contributorImporter { get; }
private SeriesImporter seriesImporter { get; }
private CategoryImporter categoryImporter { get; }
public BookImporter(LibationContext context) : base(context)
{
contributorImporter = new ContributorImporter(DbContext);
seriesImporter = new SeriesImporter(DbContext);
categoryImporter = new CategoryImporter(DbContext);
}
protected override int DoImport(IEnumerable<ImportItem> importItems)
{
// pre-req.s
new ContributorImporter(DbContext).Import(importItems);
new SeriesImporter(DbContext).Import(importItems);
new CategoryImporter(DbContext).Import(importItems);
// get distinct
var productIds = importItems.Select(i => i.DtoItem.ProductId).Distinct().ToList();
// load db existing => .Local
loadLocal_books(productIds);
contributorImporter.Import(importItems);
seriesImporter.Import(importItems);
categoryImporter.Import(importItems);
// load db existing => hash table
loadLocal_books(importItems);
// upsert
var qtyNew = upsertBooks(importItems);
return qtyNew;
}
private void loadLocal_books(List<string> productIds)
private void loadLocal_books(IEnumerable<ImportItem> importItems)
{
// if this context has already loaded books, don't need to reload them. vestige from when context was long-lived. in practice, we now typically use a fresh context. this is quick though so no harm in leaving it.
var localProductIds = DbContext.Books.Local.Select(b => b.AudibleProductId).ToList();
var remainingProductIds = productIds
.Except(localProductIds)
// get distinct
var productIds = importItems
.Select(i => i.DtoItem.ProductId)
.Distinct()
.ToList();
#region // explanation of DbContext.Books.GetBooks(b => remainingProductIds.Contains(b.AudibleProductId)).ToList();
/*
articles suggest loading to Local with
context.Books.Load();
we want Books and associated fields
context.Books.GetBooks(b => remainingProductIds.Contains(b.AudibleProductId)).ToList();
this is emulating Load() but with also getting associated fields
from: Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions
// Summary:
// Enumerates the query. When using Entity Framework, this causes the results of
// the query to be loaded into the associated context. This is equivalent to calling
// ToList and then throwing away the list (without the overhead of actually creating
// the list).
public static void Load<TSource>([NotNullAttribute] this IQueryable<TSource> source);
*/
#endregion
// GetBooks() eager loads Series, category, et al
if (remainingProductIds.Any())
DbContext.Books.GetBooks(b => remainingProductIds.Contains(b.AudibleProductId)).ToList();
Cache = DbContext.Books
.GetBooks(b => productIds.Contains(b.AudibleProductId))
.ToDictionarySafe(b => b.AudibleProductId);
}
private int upsertBooks(IEnumerable<ImportItem> importItems)
@@ -68,8 +59,7 @@ namespace DtoImporterService
foreach (var item in importItems)
{
var book = DbContext.Books.Local.SingleOrDefault(p => p.AudibleProductId == item.DtoItem.ProductId);
if (book is null)
if (!Cache.TryGetValue(item.DtoItem.ProductId, out var book))
{
book = createNewBook(item);
qtyNew++;
@@ -94,7 +84,7 @@ namespace DtoImporterService
// nested logic is required so order of names is retained. else, contributors may appear in the order they were inserted into the db
var authors = item
.Authors
.Select(a => DbContext.Contributors.Local.Single(c => a.Name == c.Name))
.Select(a => contributorImporter.Cache[a.Name])
.ToList();
var narrators
@@ -104,7 +94,7 @@ namespace DtoImporterService
// nested logic is required so order of names is retained. else, contributors may appear in the order they were inserted into the db
: item
.Narrators
.Select(n => DbContext.Contributors.Local.Single(c => n.Name == c.Name))
.Select(n => contributorImporter.Cache[n.Name])
.ToList();
// categories are laid out for a breadcrumb. category is 1st, subcategory is 2nd
@@ -118,24 +108,44 @@ namespace DtoImporterService
// 2+
: item.Categories[1].CategoryId;
var category = DbContext.Categories.Local.SingleOrDefault(c => c.AudibleCategoryId == lastCategory);
var category = categoryImporter.Cache[lastCategory];
var book = DbContext.Books.Add(new Book(
new AudibleProductId(item.ProductId),
item.TitleWithSubtitle,
item.Description,
item.LengthInMinutes,
contentType,
authors,
narrators,
category,
importItem.LocaleName)
).Entity;
Book book;
try
{
book = DbContext.Books.Add(new Book(
new AudibleProductId(item.ProductId),
item.TitleWithSubtitle,
item.Description,
item.LengthInMinutes,
contentType,
authors,
narrators,
category,
importItem.LocaleName)
).Entity;
Cache.Add(book.AudibleProductId, book);
}
catch (Exception ex)
{
Serilog.Log.Logger.Error(ex, "Error adding book. {@DebugInfo}", new {
item.ProductId,
item.TitleWithSubtitle,
item.Description,
item.LengthInMinutes,
contentType,
QtyAuthors = authors?.Count,
QtyNarrators = narrators?.Count,
Category = category?.Name,
importItem.LocaleName
});
throw;
}
var publisherName = item.Publisher;
if (!string.IsNullOrWhiteSpace(publisherName))
{
var publisher = DbContext.Contributors.Local.Single(c => publisherName == c.Name);
var publisher = contributorImporter.Cache[publisherName];
book.ReplacePublisher(publisher);
}
@@ -152,7 +162,9 @@ namespace DtoImporterService
var item = importItem.DtoItem;
// set/update book-specific info which may have changed
book.PictureId = item.PictureId;
if (item.PictureId is not null)
book.PictureId = item.PictureId;
book.UpdateProductRating(item.Product_OverallStars, item.Product_PerformanceStars, item.Product_StoryStars);
// important to update user-specific info. this will have changed if user has rated/reviewed the book since last library import
@@ -164,7 +176,7 @@ namespace DtoImporterService
{
foreach (var seriesEntry in item.Series)
{
var series = DbContext.Series.Local.Single(s => seriesEntry.SeriesId == s.AudibleSeriesId);
var series = seriesImporter.Cache[seriesEntry.SeriesId];
book.UpsertSeries(series, seriesEntry.Sequence);
}
}

View File

@@ -2,16 +2,19 @@
using System.Collections.Generic;
using System.Linq;
using AudibleApi.Common;
using AudibleUtilities;
using DataLayer;
using InternalUtilities;
using Dinah.Core.Collections.Generic;
namespace DtoImporterService
{
public class CategoryImporter : ItemsImporterBase
{
public CategoryImporter(LibationContext context) : base(context) { }
protected override IValidator Validator => new CategoryValidator();
public override IEnumerable<Exception> Validate(IEnumerable<ImportItem> importItems) => new CategoryValidator().Validate(importItems.Select(i => i.DtoItem));
public Dictionary<string, Category> Cache { get; private set; } = new();
public CategoryImporter(LibationContext context) : base(context) { }
protected override int DoImport(IEnumerable<ImportItem> importItems)
{
@@ -19,7 +22,9 @@ namespace DtoImporterService
var categoryIds = importItems
.Select(i => i.DtoItem)
.GetCategoriesDistinct()
.Select(c => c.CategoryId).ToList();
.Select(c => c.CategoryId)
.Distinct()
.ToList();
// load db existing => .Local
loadLocal_categories(categoryIds);
@@ -35,17 +40,13 @@ namespace DtoImporterService
private void loadLocal_categories(List<string> categoryIds)
{
var localIds = DbContext.Categories.Local.Select(c => c.AudibleCategoryId).ToList();
var remainingCategoryIds = categoryIds
.Distinct()
.Except(localIds)
.ToList();
// must include default/empty/missing
categoryIds.Add(Category.GetEmpty().AudibleCategoryId);
// load existing => local
// remember to include default/empty/missing
var emptyName = Contributor.GetEmpty().Name;
if (remainingCategoryIds.Any())
DbContext.Categories.Where(c => remainingCategoryIds.Contains(c.AudibleCategoryId) || c.Name == emptyName).ToList();
Cache = DbContext.Categories
.Where(c => categoryIds.Contains(c.AudibleCategoryId))
.ToDictionarySafe(c => c.AudibleCategoryId);
}
// only use after loading contributors => local
@@ -66,14 +67,11 @@ namespace DtoImporterService
Category parentCategory = null;
if (i == 1)
// should be "Single()" but user is getting a strange error
parentCategory = DbContext.Categories.Local.FirstOrDefault(c => c.AudibleCategoryId == pair[0].CategoryId);
Cache.TryGetValue(pair[0].CategoryId, out parentCategory);
// should be "SingleOrDefault()" but user is getting a strange error
var category = DbContext.Categories.Local.FirstOrDefault(c => c.AudibleCategoryId == id);
if (category is null)
if (!Cache.TryGetValue(id, out var category))
{
category = DbContext.Categories.Add(new Category(new AudibleCategoryId(id), name)).Entity;
category = addCategory(id, name);
qtyNew++;
}
@@ -83,5 +81,24 @@ namespace DtoImporterService
return qtyNew;
}
private Category addCategory(string id, string name)
{
try
{
var category = new Category(new AudibleCategoryId(id), name);
var entityEntry = DbContext.Categories.Add(category);
var entity = entityEntry.Entity;
Cache.Add(entity.AudibleCategoryId, entity);
return entity;
}
catch (Exception ex)
{
Serilog.Log.Logger.Error(ex, "Error adding category. {@DebugInfo}", new { id, name });
throw;
}
}
}
}

View File

@@ -2,16 +2,19 @@
using System.Collections.Generic;
using System.Linq;
using AudibleApi.Common;
using AudibleUtilities;
using DataLayer;
using InternalUtilities;
using Dinah.Core.Collections.Generic;
namespace DtoImporterService
{
public class ContributorImporter : ItemsImporterBase
{
public ContributorImporter(LibationContext context) : base(context) { }
protected override IValidator Validator => new ContributorValidator();
public override IEnumerable<Exception> Validate(IEnumerable<ImportItem> importItems) => new ContributorValidator().Validate(importItems.Select(i => i.DtoItem));
public Dictionary<string, Contributor> Cache { get; private set; } = new();
public ContributorImporter(LibationContext context) : base(context) { }
protected override int DoImport(IEnumerable<ImportItem> importItems)
{
@@ -50,58 +53,61 @@ namespace DtoImporterService
// must include default/empty/missing
contributorNames.Add(Contributor.GetEmpty().Name);
//// BAD: very inefficient
// var x = context.Contributors.Local.Where(c => !contribNames.Contains(c.Name));
// GOOD: Except() is efficient. Due to hashing, it's close to O(n)
var localNames = DbContext.Contributors.Local.Select(c => c.Name).ToList();
var remainingContribNames = contributorNames
.Distinct()
.Except(localNames)
.ToList();
// load existing => local
if (remainingContribNames.Any())
DbContext.Contributors.Where(c => remainingContribNames.Contains(c.Name)).ToList();
Cache = DbContext.Contributors
.Where(c => contributorNames.Contains(c.Name))
.ToDictionarySafe(c => c.Name);
}
// only use after loading contributors => local
private int upsertPeople(List<Person> people)
{
var localNames = DbContext.Contributors.Local.Select(c => c.Name).ToList();
var newPeople = people
.Select(p => p.Name)
.Distinct()
.Except(localNames)
.ToList();
var hash = people
// new people only
.Where(p => !Cache.ContainsKey(p.Name))
// remove duplicates by Name. first in wins
.ToDictionarySafe(p => p.Name);
var groupby = people.GroupBy(
p => p.Name,
p => p,
(key, g) => new { Name = key, People = g.ToList() }
);
foreach (var name in newPeople)
foreach (var kvp in hash)
{
var p = groupby.Single(g => g.Name == name).People.First();
DbContext.Contributors.Add(new Contributor(p.Name, p.Asin));
var person = kvp.Value;
addContributor(person.Name, person.Asin);
}
return newPeople.Count;
return hash.Count;
}
// only use after loading contributors => local
private int upsertPublishers(List<string> publishers)
{
var localNames = DbContext.Contributors.Local.Select(c => c.Name).ToList();
var newPublishers = publishers
.Distinct()
.Except(localNames)
.ToList();
var hash = publishers
// new publishers only
.Where(p => !Cache.ContainsKey(p))
// remove duplicates
.ToHashSet();
foreach (var pub in newPublishers)
DbContext.Contributors.Add(new Contributor(pub));
foreach (var pub in hash)
addContributor(pub);
return newPublishers.Count;
return hash.Count;
}
}
private Contributor addContributor(string name, string id = null)
{
try
{
var newContrib = new Contributor(name);
var entityEntry = DbContext.Contributors.Add(newContrib);
var entity = entityEntry.Entity;
Cache.Add(entity.Name, entity);
return entity;
}
catch (Exception ex)
{
Serilog.Log.Logger.Error(ex, "Error adding contributor. {@DebugInfo}", new { name, id });
throw;
}
}
}
}

View File

@@ -1,12 +1,12 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0</TargetFramework>
<TargetFramework>net6.0-windows</TargetFramework>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\AudibleUtilities\AudibleUtilities.csproj" />
<ProjectReference Include="..\DataLayer\DataLayer.csproj" />
<ProjectReference Include="..\InternalUtilities\InternalUtilities.csproj" />
</ItemGroup>
</Project>

View File

@@ -1,9 +1,9 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AudibleUtilities;
using DataLayer;
using Dinah.Core;
using InternalUtilities;
namespace DtoImporterService
{
@@ -53,5 +53,9 @@ namespace DtoImporterService
public abstract class ItemsImporterBase : ImporterBase<IEnumerable<ImportItem>>
{
protected ItemsImporterBase(LibationContext context) : base(context) { }
protected abstract IValidator Validator { get; }
public sealed override IEnumerable<Exception> Validate(IEnumerable<ImportItem> importItems)
=> Validator.Validate(importItems.Select(i => i.DtoItem));
}
}

View File

@@ -1,20 +1,26 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AudibleUtilities;
using DataLayer;
using InternalUtilities;
using Dinah.Core.Collections.Generic;
namespace DtoImporterService
{
public class LibraryBookImporter : ItemsImporterBase
{
public LibraryBookImporter(LibationContext context) : base(context) { }
protected override IValidator Validator => new LibraryValidator();
public override IEnumerable<Exception> Validate(IEnumerable<ImportItem> importItems) => new LibraryValidator().Validate(importItems.Select(i => i.DtoItem));
private BookImporter bookImporter { get; }
public LibraryBookImporter(LibationContext context) : base(context)
{
bookImporter = new BookImporter(DbContext);
}
protected override int DoImport(IEnumerable<ImportItem> importItems)
{
new BookImporter(DbContext).Import(importItems);
bookImporter.Import(importItems);
var qtyNew = upsertLibraryBooks(importItems);
return qtyNew;
@@ -35,18 +41,32 @@ namespace DtoImporterService
// CURRENT SOLUTION: don't re-insert
var currentLibraryProductIds = DbContext.LibraryBooks.Select(l => l.Book.AudibleProductId).ToList();
var newItems = importItems.Where(dto => !currentLibraryProductIds.Contains(dto.DtoItem.ProductId)).ToList();
var newItems = importItems
.Where(dto => !currentLibraryProductIds.Contains(dto.DtoItem.ProductId))
.ToList();
// if 2 accounts try to import the same book in the same transaction: error since we're only tracking and pulling by asin.
// just use the first
var hash = newItems.ToDictionarySafe(dto => dto.DtoItem.ProductId);
foreach (var kvp in hash)
{
var newItem = kvp.Value;
foreach (var newItem in newItems)
{
var libraryBook = new LibraryBook(
DbContext.Books.Local.Single(b => b.AudibleProductId == newItem.DtoItem.ProductId),
bookImporter.Cache[newItem.DtoItem.ProductId],
newItem.DtoItem.DateAdded,
newItem.AccountId);
DbContext.LibraryBooks.Add(libraryBook);
try
{
DbContext.LibraryBooks.Add(libraryBook);
}
catch (Exception ex)
{
Serilog.Log.Logger.Error(ex, "Error adding library book. {@DebugInfo}", new { libraryBook.Book, libraryBook.Account });
}
}
var qtyNew = newItems.Count;
var qtyNew = hash.Count;
return qtyNew;
}
}

View File

@@ -8,8 +8,8 @@ namespace DtoImporterService
public record timeLogEntry(string msg, long totalElapsed, long delta);
public static class PerfLogger
{
private static Stopwatch sw = new Stopwatch();
private static List<timeLogEntry> __log { get; } = new List<timeLogEntry> { new("begin", 0, 0) };
private static Stopwatch sw { get; } = new();
private static List<timeLogEntry> __log { get; } = new() { new("begin", 0, 0) };
public static void logTime(string s)
{

View File

@@ -2,16 +2,19 @@
using System.Collections.Generic;
using System.Linq;
using AudibleApi.Common;
using AudibleUtilities;
using DataLayer;
using InternalUtilities;
using Dinah.Core.Collections.Generic;
namespace DtoImporterService
{
public class SeriesImporter : ItemsImporterBase
{
public SeriesImporter(LibationContext context) : base(context) { }
protected override IValidator Validator => new SeriesValidator();
public override IEnumerable<Exception> Validate(IEnumerable<ImportItem> importItems) => new SeriesValidator().Validate(importItems.Select(i => i.DtoItem));
public Dictionary<string, DataLayer.Series> Cache { get; private set; } = new();
public SeriesImporter(LibationContext context) : base(context) { }
protected override int DoImport(IEnumerable<ImportItem> importItems)
{
@@ -31,15 +34,12 @@ namespace DtoImporterService
private void loadLocal_series(List<AudibleApi.Common.Series> series)
{
var seriesIds = series.Select(s => s.SeriesId).ToList();
var localIds = DbContext.Series.Local.Select(s => s.AudibleSeriesId).ToList();
var remainingSeriesIds = seriesIds
.Distinct()
.Except(localIds)
.ToList();
var seriesIds = series.Select(s => s.SeriesId).Distinct().ToList();
if (remainingSeriesIds.Any())
DbContext.Series.Where(s => remainingSeriesIds.Contains(s.AudibleSeriesId)).ToList();
if (seriesIds.Any())
Cache = DbContext.Series
.Where(s => seriesIds.Contains(s.AudibleSeriesId))
.ToDictionarySafe(s => s.AudibleSeriesId);
}
private int upsertSeries(List<AudibleApi.Common.Series> requestedSeries)
@@ -48,10 +48,10 @@ namespace DtoImporterService
foreach (var s in requestedSeries)
{
var series = DbContext.Series.Local.FirstOrDefault(c => c.AudibleSeriesId == s.SeriesId);
if (series is null)
// AudibleApi.Common.Series.SeriesId == DataLayer.AudibleSeriesId
if (!Cache.TryGetValue(s.SeriesId, out var series))
{
series = DbContext.Series.Add(new DataLayer.Series(new AudibleSeriesId(s.SeriesId))).Entity;
series = addSeries(s.SeriesId);
qtyNew++;
}
series.UpdateName(s.SeriesName);
@@ -59,5 +59,24 @@ namespace DtoImporterService
return qtyNew;
}
private DataLayer.Series addSeries(string seriesId)
{
try
{
var series = new DataLayer.Series(new AudibleSeriesId(seriesId));
var entityEntry = DbContext.Series.Add(series);
var entity = entityEntry.Entity;
Cache.Add(entity.AudibleSeriesId, entity);
return entity;
}
catch (Exception ex)
{
Serilog.Log.Logger.Error(ex, "Error adding series. {@DebugInfo}", new { seriesId });
throw;
}
}
}
}

View File

@@ -0,0 +1,37 @@
* Local (eg DbContext.Books.Local): indexes/hashes PK and nothing else. Local.Find(PK) is fast. All other searches (eg FirstOrDefault) have awful performance. It deceptively *feels* like we get this partially for free since added/modified entries live here.
* live db: for all importers, fields used for lookup are indexed
Using BookImporter as an example: since AudibleProductId is indexed, hitting the live db is much faster than using Local. Fastest is putting all in a local hash table
Note: GetBook/GetBooks eager loads Series, category, et al
for 1,200 iterations
* load to LocalView
DbContext.Books.Local.FirstOrDefault(p => p.AudibleProductId == item.DtoItem.ProductId)
27,125 ms
* read from live db
DbContext.Books.GetBook(item.DtoItem.ProductId)
12,224 ms
* load to hash table: Dictionary<string, Book>
dictionary[item.DtoItem.ProductId];
1 ms (yes: ONE)
With hashtable, somehow memory usage was not significantly affected
-----------------------------------
why were we using Local to begin with?
articles suggest loading to Local with
context.Books.Load();
this loads this table but not associated fields
we want Books and associated fields
context.Books.GetBooks(b => remainingProductIds.Contains(b.AudibleProductId)).ToList();
this is emulating Load() but with also getting associated fields
from: Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions
// Summary:
// Enumerates the query. When using Entity Framework, this causes the results of
// the query to be loaded into the associated context. This is equivalent to calling
// ToList and then throwing away the list (without the overhead of actually creating
// the list).
public static void Load<TSource>([NotNullAttribute] this IQueryable<TSource> source);

View File

@@ -11,30 +11,33 @@ namespace FileLiberator
public event EventHandler<byte[]> CoverImageDiscovered;
public abstract void Cancel();
protected void OnRequestCoverArt(Action<byte[]> setCoverArtDel)
{
Serilog.Log.Logger.Debug("Event fired {@DebugInfo}", new { Name = nameof(RequestCoverArt) });
RequestCoverArt?.Invoke(this, setCoverArtDel);
}
protected void OnTitleDiscovered(string title)
protected void OnTitleDiscovered(string title) => OnTitleDiscovered(null, title);
protected void OnTitleDiscovered(object _, string title)
{
Serilog.Log.Logger.Debug("Event fired {@DebugInfo}", new { Name = nameof(TitleDiscovered), Title = title });
TitleDiscovered?.Invoke(this, title);
}
protected void OnAuthorsDiscovered(string authors)
protected void OnAuthorsDiscovered(string authors) => OnAuthorsDiscovered(null, authors);
protected void OnAuthorsDiscovered(object _, string authors)
{
Serilog.Log.Logger.Debug("Event fired {@DebugInfo}", new { Name = nameof(AuthorsDiscovered), Authors = authors });
AuthorsDiscovered?.Invoke(this, authors);
}
protected void OnNarratorsDiscovered(string narrators)
protected void OnNarratorsDiscovered(string narrators) => OnNarratorsDiscovered(null, narrators);
protected void OnNarratorsDiscovered(object _, string narrators)
{
Serilog.Log.Logger.Debug("Event fired {@DebugInfo}", new { Name = nameof(NarratorsDiscovered), Narrators = narrators });
NarratorsDiscovered?.Invoke(this, narrators);
}
protected void OnRequestCoverArt(Action<byte[]> setCoverArtDel)
{
Serilog.Log.Logger.Debug("Event fired {@DebugInfo}", new { Name = nameof(RequestCoverArt) });
RequestCoverArt?.Invoke(this, setCoverArtDel);
}
protected void OnCoverImageDiscovered(byte[] coverImage)
{
Serilog.Log.Logger.Debug("Event fired {@DebugInfo}", new { Name = nameof(CoverImageDiscovered), CoverImageBytes = coverImage?.Length });

View File

@@ -0,0 +1,56 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using DataLayer;
using Dinah.Core;
using FileManager;
using LibationFileManager;
namespace FileLiberator
{
public static class AudioFileStorageExt
{
private class MultipartRenamer
{
private LibraryBook libraryBook { get; }
internal MultipartRenamer(LibraryBook libraryBook) => this.libraryBook = libraryBook;
internal string MultipartFilename(AaxDecrypter.MultiConvertFileProperties props)
=> Templates.ChapterFile.GetFilename(libraryBook.ToDto(), props);
}
public static Func<AaxDecrypter.MultiConvertFileProperties, string> CreateMultipartRenamerFunc(this AudioFileStorage _, LibraryBook libraryBook)
=> new MultipartRenamer(libraryBook).MultipartFilename;
/// <summary>
/// DownloadDecryptBook:
/// File path for where to move files into.
/// Path: directory nested inside of Books directory
/// File name: n/a
/// </summary>
public static string GetDestinationDirectory(this AudioFileStorage _, LibraryBook libraryBook)
=> Templates.Folder.GetFilename(libraryBook.ToDto());
/// <summary>
/// DownloadDecryptBook:
/// Path: in progress directory.
/// File name: final file name.
/// </summary>
public static string GetInProgressFilename(this AudioFileStorage _, LibraryBook libraryBook, string extension)
=> Templates.File.GetFilename(libraryBook.ToDto(), AudibleFileStorage.DecryptInProgressDirectory, extension);
/// <summary>
/// PDF: audio file does not exist
/// </summary>
public static string GetBooksDirectoryFilename(this AudioFileStorage _, LibraryBook libraryBook, string extension)
=> Templates.File.GetFilename(libraryBook.ToDto(), AudibleFileStorage.BooksDirectory, extension);
/// <summary>
/// PDF: audio file already exists
/// </summary>
public static string GetCustomDirFilename(this AudioFileStorage _, LibraryBook libraryBook, string dirFullPath, string extension)
=> Templates.File.GetFilename(libraryBook.ToDto(), dirFullPath, extension);
}
}

View File

@@ -1,14 +1,14 @@
using AAXClean;
using System;
using System.IO;
using System.Threading.Tasks;
using AAXClean;
using AAXClean.Codecs;
using DataLayer;
using Dinah.Core;
using Dinah.Core.ErrorHandling;
using Dinah.Core.IO;
using Dinah.Core.Net.Http;
using FileManager;
using System;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using LibationFileManager;
namespace FileLiberator
{
@@ -17,7 +17,7 @@ namespace FileLiberator
private Mp4File m4bBook;
private long fileSize;
private static string Mp3FileName(string m4bPath) => m4bPath is null ? string.Empty : PathLib.ReplaceExtension(m4bPath, ".mp3");
private static string Mp3FileName(string m4bPath) => Path.ChangeExtension(m4bPath ?? "", ".mp3");
public override void Cancel() => m4bBook?.Cancel();
@@ -52,9 +52,9 @@ namespace FileLiberator
m4bBook.InputStream.Close();
mp3File.Close();
var mp3Path = Mp3FileName(m4bPath);
FileExt.SafeMove(mp3File.Name, mp3Path);
var proposedMp3Path = Mp3FileName(m4bPath);
var realMp3Path = FileUtility.SaferMoveToValidPath(mp3File.Name, proposedMp3Path);
OnFileCreated(libraryBook, realMp3Path);
var statusHandler = new StatusHandler();
@@ -73,8 +73,8 @@ namespace FileLiberator
private void M4bBook_ConversionProgressUpdate(object sender, ConversionProgressEventArgs e)
{
var duration = m4bBook.Duration;
double remainingSecsToProcess = (duration - e.ProcessPosition).TotalSeconds;
double estTimeRemaining = remainingSecsToProcess / e.ProcessSpeed;
var remainingSecsToProcess = (duration - e.ProcessPosition).TotalSeconds;
var estTimeRemaining = remainingSecsToProcess / e.ProcessSpeed;
if (double.IsNormal(estTimeRemaining))
OnStreamingTimeRemaining(TimeSpan.FromSeconds(estTimeRemaining));

View File

@@ -9,6 +9,7 @@ using DataLayer;
using Dinah.Core;
using Dinah.Core.ErrorHandling;
using FileManager;
using LibationFileManager;
namespace FileLiberator
{
@@ -16,8 +17,27 @@ namespace FileLiberator
{
private AudiobookDownloadBase abDownloader;
public override bool Validate(LibraryBook libraryBook) => !libraryBook.Book.Audio_Exists;
public override void Cancel() => abDownloader?.Cancel();
public override async Task<StatusHandler> ProcessAsync(LibraryBook libraryBook)
{
var entries = new List<FilePathCache.CacheEntry>();
// these only work so minimally b/c CacheEntry is a record.
// in case of parallel decrypts, only capture the ones for this book id.
// if user somehow starts multiple decrypts of the same book in parallel: on their own head be it
void FilePathCache_Inserted(object sender, FilePathCache.CacheEntry e)
{
if (e.Id.EqualsInsensitive(libraryBook.Book.AudibleProductId))
entries.Add(e);
}
void FilePathCache_Removed(object sender, FilePathCache.CacheEntry e)
{
if (e.Id.EqualsInsensitive(libraryBook.Book.AudibleProductId))
entries.Remove(e);
}
OnBegin(libraryBook);
try
@@ -25,16 +45,29 @@ namespace FileLiberator
if (libraryBook.Book.Audio_Exists)
return new StatusHandler { "Cannot find decrypt. Final audio file already exists" };
var outputAudioFilename = await downloadAudiobookAsync(AudibleFileStorage.DownloadsInProgress, AudibleFileStorage.DecryptInProgress, libraryBook);
bool success = false;
try
{
FilePathCache.Inserted += FilePathCache_Inserted;
FilePathCache.Removed += FilePathCache_Removed;
success = await downloadAudiobookAsync(libraryBook);
}
finally
{
FilePathCache.Inserted -= FilePathCache_Inserted;
FilePathCache.Removed -= FilePathCache_Removed;
}
// decrypt failed
if (outputAudioFilename is null)
if (!success)
return new StatusHandler { "Decrypt failed" };
// moves files and returns dest dir
var moveResults = MoveFilesToBooksDir(libraryBook.Book, outputAudioFilename);
// moves new files from temp dir to final dest
var movedAudioFile = moveFilesToBooksDir(libraryBook, entries);
if (!moveResults.movedAudioFile)
// decrypt failed
if (!movedAudioFile)
return new StatusHandler { "Cannot find final audio file after decryption" };
libraryBook.Book.UserDefinedItem.BookStatus = LiberatedStatus.Liberated;
@@ -47,13 +80,13 @@ namespace FileLiberator
}
}
private async Task<string> downloadAudiobookAsync(string cacheDir, string destinationDir, LibraryBook libraryBook)
private async Task<bool> downloadAudiobookAsync(LibraryBook libraryBook)
{
OnStreamingBegin($"Begin decrypting {libraryBook}");
try
{
validate(libraryBook);
downloadValidation(libraryBook);
var api = await libraryBook.GetApiAsync();
var contentLic = await api.GetDownloadLicenseAsync(libraryBook.Book.AudibleProductId);
@@ -81,28 +114,39 @@ namespace FileLiberator
foreach (var chap in contentLic.ContentMetadata?.ChapterInfo?.Chapters)
audiobookDlLic.ChapterInfo.AddChapter(chap.Title, TimeSpan.FromMilliseconds(chap.LengthMs));
}
var outFileName = Path.Combine(destinationDir, $"{PathLib.ToPathSafeString(libraryBook.Book.Title)} [{libraryBook.Book.AudibleProductId}].{outputFormat.ToString().ToLower()}");
abDownloader = contentLic.DrmType == AudibleApi.Common.DrmType.Adrm
? new AaxcDownloadConverter(outFileName, cacheDir, audiobookDlLic, outputFormat, Configuration.Instance.SplitFilesByChapter)
: new UnencryptedAudiobookDownloader(outFileName, cacheDir, audiobookDlLic);
abDownloader.AppName = "Libation";
abDownloader.DecryptProgressUpdate += (_, progress) => OnStreamingProgressChanged(progress);
abDownloader.DecryptTimeRemaining += (_, remaining) => OnStreamingTimeRemaining(remaining);
abDownloader.RetrievedTitle += (_, title) => OnTitleDiscovered(title);
abDownloader.RetrievedAuthors += (_, authors) => OnAuthorsDiscovered(authors);
abDownloader.RetrievedNarrators += (_, narrators) => OnNarratorsDiscovered(narrators);
var outFileName = AudibleFileStorage.Audio.GetInProgressFilename(libraryBook, outputFormat.ToString().ToLower());
var cacheDir = AudibleFileStorage.DownloadsInProgressDirectory;
if (contentLic.DrmType != AudibleApi.Common.DrmType.Adrm)
abDownloader = new UnencryptedAudiobookDownloader(outFileName, cacheDir, audiobookDlLic);
else
{
AaxcDownloadConvertBase converter
= Configuration.Instance.SplitFilesByChapter ? new AaxcDownloadMultiConverter(
outFileName, cacheDir, audiobookDlLic, outputFormat,
AudibleFileStorage.Audio.CreateMultipartRenamerFunc(libraryBook))
: new AaxcDownloadSingleConverter(outFileName, cacheDir, audiobookDlLic, outputFormat);
if (Configuration.Instance.AllowLibationFixup)
converter.RetrievedMetadata += (_, tags) => tags.Generes = string.Join(", ", libraryBook.Book.CategoriesNames);
abDownloader = converter;
}
abDownloader.DecryptProgressUpdate += OnStreamingProgressChanged;
abDownloader.DecryptTimeRemaining += OnStreamingTimeRemaining;
abDownloader.RetrievedTitle += OnTitleDiscovered;
abDownloader.RetrievedAuthors += OnAuthorsDiscovered;
abDownloader.RetrievedNarrators += OnNarratorsDiscovered;
abDownloader.RetrievedCoverArt += AaxcDownloader_RetrievedCoverArt;
abDownloader.FileCreated += (_, path) => OnFileCreated(libraryBook, path);
// REAL WORK DONE HERE
var success = await Task.Run(abDownloader.Run);
// decrypt failed
if (!success)
return null;
return outFileName;
return success;
}
finally
{
@@ -110,76 +154,7 @@ namespace FileLiberator
}
}
private void AaxcDownloader_RetrievedCoverArt(object sender, byte[] e)
{
if (e is null && Configuration.Instance.AllowLibationFixup)
{
OnRequestCoverArt(abDownloader.SetCoverArt);
}
if (e is not null)
{
OnCoverImageDiscovered(e);
}
}
private static (string destinationDir, bool movedAudioFile) MoveFilesToBooksDir(Book product, string outputAudioFilename)
{
// create final directory. move each file into it. MOVE AUDIO FILE LAST
// new dir: safetitle_limit50char + " [" + productId + "]"
// TODO make this method handle multiple audio files or a single audio file.
var destinationDir = AudibleFileStorage.Audio.GetDestDir(product.Title, product.AudibleProductId);
Directory.CreateDirectory(destinationDir);
var sortedFiles = getProductFilesSorted(product, outputAudioFilename);
var musicFileExt = Path.GetExtension(outputAudioFilename).Trim('.');
// audio filename: safetitle_limit50char + " [" + productId + "]." + audio_ext
var audioFileName = FileUtility.GetValidFilename(destinationDir, product.Title, musicFileExt, product.AudibleProductId);
bool movedAudioFile = false;
foreach (var f in sortedFiles)
{
var dest
= AudibleFileStorage.Audio.IsFileTypeMatch(f)//f.Extension.Equals($".{musicFileExt}", StringComparison.OrdinalIgnoreCase)
? Path.Join(destinationDir, f.Name)
// non-audio filename: safetitle_limit50char + " [" + productId + "][" + audio_ext + "]." + non_audio_ext
: FileUtility.GetValidFilename(destinationDir, product.Title, f.Extension, product.AudibleProductId, musicFileExt);
if (Path.GetExtension(dest).Trim('.').ToLower() == "cue")
Cue.UpdateFileName(f, audioFileName);
File.Move(f.FullName, dest);
movedAudioFile |= AudibleFileStorage.Audio.IsFileTypeMatch(f);
}
AudibleFileStorage.Audio.Refresh();
return (destinationDir, movedAudioFile);
}
private static List<FileInfo> getProductFilesSorted(Book product, string outputAudioFilename)
{
// files are: temp path\author\[asin].ext
var m4bDir = new FileInfo(outputAudioFilename).Directory;
var files = m4bDir
.EnumerateFiles()
.Where(f => f.Name.ContainsInsensitive(product.AudibleProductId))
.ToList();
// move audio files to the end of the collection so these files are moved last
var musicFiles = files.Where(f => AudibleFileStorage.Audio.IsFileTypeMatch(f));
var sortedFiles = files
.Except(musicFiles)
.Concat(musicFiles)
.ToList();
return sortedFiles;
}
private static void validate(LibraryBook libraryBook)
private static void downloadValidation(LibraryBook libraryBook)
{
string errorString(string field)
=> $"{errorTitle()}\r\nCannot download book. {field} is not known. Try re-importing the account which owns this book.";
@@ -201,11 +176,45 @@ namespace FileLiberator
throw new Exception(errorString("Locale"));
}
public override bool Validate(LibraryBook libraryBook) => !libraryBook.Book.Audio_Exists;
public override void Cancel()
private void AaxcDownloader_RetrievedCoverArt(object _, byte[] e)
{
abDownloader?.Cancel();
if (e is not null)
OnCoverImageDiscovered(e);
else if (Configuration.Instance.AllowLibationFixup)
OnRequestCoverArt(abDownloader.SetCoverArt);
}
}
/// <summary>Move new files to 'Books' directory</summary>
/// <returns>True if audiobook file(s) were successfully created and can be located on disk. Else false.</returns>
private static bool moveFilesToBooksDir(LibraryBook libraryBook, List<FilePathCache.CacheEntry> entries)
{
// create final directory. move each file into it
var destinationDir = AudibleFileStorage.Audio.GetDestinationDirectory(libraryBook);
Directory.CreateDirectory(destinationDir);
FilePathCache.CacheEntry getFirstAudio() => entries.FirstOrDefault(f => f.FileType == FileType.Audio);
if (getFirstAudio() == default)
return false;
for (var i = 0; i < entries.Count; i++)
{
var entry = entries[i];
var realDest = FileUtility.SaferMoveToValidPath(entry.Path, Path.Combine(destinationDir, Path.GetFileName(entry.Path)));
FilePathCache.Insert(libraryBook.Book.AudibleProductId, realDest);
// propogate corrected path. Must update cache with corrected path. Also want updated path for cue file (after this for-loop)
entries[i] = entry with { Path = realDest };
}
var cue = entries.FirstOrDefault(f => f.FileType == FileType.Cue);
if (cue != default)
Cue.UpdateFileName(cue.Path, getFirstAudio().Path);
AudibleFileStorage.Audio.Refresh();
return true;
}
}
}

View File

@@ -12,14 +12,14 @@ namespace FileLiberator
{
var client = new HttpClient();
var progress = new Progress<DownloadProgress>();
progress.ProgressChanged += (_, e) => OnStreamingProgressChanged(e);
var progress = new Progress<DownloadProgress>(OnStreamingProgressChanged);
OnStreamingBegin(proposedDownloadFilePath);
try
{
var actualDownloadedFilePath = await client.DownloadFileAsync(downloadUrl, proposedDownloadFilePath, progress);
OnFileCreated("Upgrade", actualDownloadedFilePath);
return actualDownloadedFilePath;
}
finally

View File

@@ -8,6 +8,7 @@ using DataLayer;
using Dinah.Core.ErrorHandling;
using Dinah.Core.Net.Http;
using FileManager;
using LibationFileManager;
namespace FileLiberator
{
@@ -39,19 +40,14 @@ namespace FileLiberator
private static string getProposedDownloadFilePath(LibraryBook libraryBook)
{
var extension = Path.GetExtension(getdownloadUrl(libraryBook));
// if audio file exists, get it's dir. else return base Book dir
var existingPath = Path.GetDirectoryName(AudibleFileStorage.Audio.GetPath(libraryBook.Book.AudibleProductId));
var file = getdownloadUrl(libraryBook);
if (existingPath is not null)
return AudibleFileStorage.Audio.GetCustomDirFilename(libraryBook, existingPath, extension);
if (existingPath != null)
return Path.Combine(existingPath, Path.GetFileName(file));
var full = FileUtility.GetValidFilename(
AudibleFileStorage.PdfStorageDirectory,
libraryBook.Book.Title,
Path.GetExtension(file),
libraryBook.Book.AudibleProductId);
return full;
return AudibleFileStorage.Audio.GetBooksDirectoryFilename(libraryBook, extension);
}
private static string getdownloadUrl(LibraryBook libraryBook)
@@ -66,12 +62,13 @@ namespace FileLiberator
var api = await libraryBook.GetApiAsync();
var downloadUrl = await api.GetPdfDownloadLinkAsync(libraryBook.Book.AudibleProductId);
var progress = new Progress<DownloadProgress>();
progress.ProgressChanged += (_, e) => OnStreamingProgressChanged(e);
var progress = new Progress<DownloadProgress>(OnStreamingProgressChanged);
var client = new HttpClient();
var actualDownloadedFilePath = await client.DownloadFileAsync(downloadUrl, proposedDownloadFilePath, progress);
OnFileCreated(libraryBook, actualDownloadedFilePath);
OnStatusUpdate(actualDownloadedFilePath);
return actualDownloadedFilePath;
}

View File

@@ -1,14 +1,13 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0</TargetFramework>
<TargetFramework>net6.0-windows</TargetFramework>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\AaxDecrypter\AaxDecrypter.csproj" />
<ProjectReference Include="..\DataLayer\DataLayer.csproj" />
<ProjectReference Include="..\FileManager\FileManager.csproj" />
<ProjectReference Include="..\InternalUtilities\InternalUtilities.csproj" />
<ProjectReference Include="..\AudibleUtilities\AudibleUtilities.csproj" />
</ItemGroup>
</Project>

View File

@@ -27,7 +27,7 @@ namespace FileLiberator
public IEnumerable<LibraryBook> GetValidLibraryBooks(IEnumerable<LibraryBook> library)
=> library.Where(libraryBook =>
Validate(libraryBook)
&& (libraryBook.Book.ContentType != ContentType.Episode || FileManager.Configuration.Instance.DownloadEpisodes)
&& (libraryBook.Book.ContentType != ContentType.Episode || LibationFileManager.Configuration.Instance.DownloadEpisodes)
);
public async Task<StatusHandler> ProcessSingleAsync(LibraryBook libraryBook, bool validate)

View File

@@ -9,6 +9,8 @@ namespace FileLiberator
public event EventHandler<DownloadProgress> StreamingProgressChanged;
public event EventHandler<TimeSpan> StreamingTimeRemaining;
public event EventHandler<string> StreamingCompleted;
/// <summary>Fired when a file is successfully saved to disk</summary>
public event EventHandler<(string id, string path)> FileCreated;
protected void OnStreamingBegin(string filePath)
{
@@ -16,12 +18,14 @@ namespace FileLiberator
StreamingBegin?.Invoke(this, filePath);
}
protected void OnStreamingProgressChanged(DownloadProgress progress)
protected void OnStreamingProgressChanged(DownloadProgress progress) => OnStreamingProgressChanged(null, progress);
protected void OnStreamingProgressChanged(object _, DownloadProgress progress)
{
StreamingProgressChanged?.Invoke(this, progress);
}
protected void OnStreamingTimeRemaining(TimeSpan timeRemaining)
protected void OnStreamingTimeRemaining(TimeSpan timeRemaining) => OnStreamingTimeRemaining(null, timeRemaining);
protected void OnStreamingTimeRemaining(object _, TimeSpan timeRemaining)
{
StreamingTimeRemaining?.Invoke(this, timeRemaining);
}
@@ -30,8 +34,14 @@ namespace FileLiberator
{
Serilog.Log.Logger.Debug("Event fired {@DebugInfo}", new { Name = nameof(StreamingCompleted), Message = filePath });
StreamingCompleted?.Invoke(this, filePath);
}
//TODO: Update file cache
protected void OnFileCreated(DataLayer.LibraryBook libraryBook, string path) => OnFileCreated(libraryBook.Book.AudibleProductId, path);
protected void OnFileCreated(string id, string path)
{
Serilog.Log.Logger.Information("File created {@DebugInfo}", new { Name = nameof(FileCreated), id, path });
LibationFileManager.FilePathCache.Insert(id, path);
FileCreated?.Invoke(this, (id, path));
}
}
}

View File

@@ -4,6 +4,7 @@ using System.Linq;
using System.Threading.Tasks;
using DataLayer;
using Dinah.Core;
using LibationFileManager;
namespace FileLiberator
{
@@ -19,8 +20,24 @@ namespace FileLiberator
public static async Task<AudibleApi.Api> GetApiAsync(this LibraryBook libraryBook)
{
var apiExtended = await InternalUtilities.ApiExtended.CreateAsync(libraryBook.Account, libraryBook.Book.Locale);
var apiExtended = await AudibleUtilities.ApiExtended.CreateAsync(libraryBook.Account, libraryBook.Book.Locale);
return apiExtended.Api;
}
public static LibraryBookDto ToDto(this LibraryBook libraryBook) => new()
{
Account = libraryBook.Account,
AudibleProductId = libraryBook.Book.AudibleProductId,
Title = libraryBook.Book.Title ?? "",
Locale = libraryBook.Book.Locale,
Authors = libraryBook.Book.Authors.Select(c => c.Name).ToList(),
Narrators = libraryBook.Book.Narrators.Select(c => c.Name).ToList(),
SeriesName = libraryBook.Book.SeriesLink.FirstOrDefault()?.Series.Name,
SeriesNumber = libraryBook.Book.SeriesLink.FirstOrDefault()?.Order
};
}
}

View File

@@ -0,0 +1 @@
[assembly: System.Runtime.CompilerServices.InternalsVisibleTo(nameof(FileLiberator) + ".Tests")]

View File

@@ -1,149 +0,0 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using Dinah.Core;
using Dinah.Core.Collections.Generic;
namespace FileManager
{
public enum FileType { Unknown, Audio, AAXC, PDF }
public abstract class AudibleFileStorage : Enumeration<AudibleFileStorage>
{
protected abstract string[] Extensions { get; }
public abstract string StorageDirectory { get; }
public static string DownloadsInProgress => Directory.CreateDirectory(Path.Combine(Configuration.Instance.InProgress, "DownloadsInProgress")).FullName;
public static string DecryptInProgress => Directory.CreateDirectory(Path.Combine(Configuration.Instance.InProgress, "DecryptInProgress")).FullName;
public static string PdfStorageDirectory => BooksDirectory;
private static AaxcFileStorage AAXC { get; } = new AaxcFileStorage();
public static bool AaxcExists(string productId) => AAXC.Exists(productId);
#region static
public static AudioFileStorage Audio { get; } = new AudioFileStorage();
public static string BooksDirectory
{
get
{
if (string.IsNullOrWhiteSpace(Configuration.Instance.Books))
Configuration.Instance.Books = Path.Combine(Configuration.UserProfile, "Books");
return Directory.CreateDirectory(Configuration.Instance.Books).FullName;
}
}
private static object bookDirectoryFilesLocker { get; } = new();
internal static BackgroundFileSystem BookDirectoryFiles { get; set; }
#endregion
#region instance
public FileType FileType => (FileType)Value;
protected IEnumerable<string> extensions_noDots { get; }
private string extAggr { get; }
protected AudibleFileStorage(FileType fileType) : base((int)fileType, fileType.ToString())
{
extensions_noDots = Extensions.Select(ext => ext.ToLower().Trim('.')).ToList();
extAggr = extensions_noDots.Aggregate((a, b) => $"{a}|{b}");
BookDirectoryFiles ??= new BackgroundFileSystem(BooksDirectory, "*.*", SearchOption.AllDirectories);
}
protected string GetFilePath(string productId)
{
var cachedFile = FilePathCache.GetPath(productId, FileType);
if (cachedFile != null)
return cachedFile;
var regex = new Regex($@"{productId}.*?\.({extAggr})$", RegexOptions.IgnoreCase);
string firstOrNull;
if (StorageDirectory == BooksDirectory)
{
//If user changed the BooksDirectory, reinitialize.
if (StorageDirectory != BookDirectoryFiles.RootDirectory)
{
lock (bookDirectoryFilesLocker)
{
if (StorageDirectory != BookDirectoryFiles.RootDirectory)
{
BookDirectoryFiles = new BackgroundFileSystem(StorageDirectory, "*.*", SearchOption.AllDirectories);
}
}
}
firstOrNull = BookDirectoryFiles.FindFile(regex);
}
else
{
firstOrNull =
Directory
.EnumerateFiles(StorageDirectory, "*.*", SearchOption.AllDirectories)
.FirstOrDefault(s => regex.IsMatch(s));
}
if (firstOrNull is null)
return null;
FilePathCache.Upsert(productId, FileType, firstOrNull);
return firstOrNull;
}
#endregion
}
public class AudioFileStorage : AudibleFileStorage
{
protected override string[] Extensions { get; } = new[] { "m4b", "mp3", "aac", "mp4", "m4a", "ogg", "flac" };
// we always want to use the latest config value, therefore
// - DO use 'get' arrow "=>"
// - do NOT use assign "="
public override string StorageDirectory => BooksDirectory;
public AudioFileStorage() : base(FileType.Audio) { }
public void Refresh() => BookDirectoryFiles.RefreshFiles();
public string GetDestDir(string title, string asin)
{
// to prevent the paths from getting too long, we don't need after the 1st ":" for the folder
var underscoreIndex = title.IndexOf(':');
var titleDir
= underscoreIndex < 4
? title
: title.Substring(0, underscoreIndex);
var finalDir = FileUtility.GetValidFilename(StorageDirectory, titleDir, null, asin);
return finalDir;
}
public bool IsFileTypeMatch(FileInfo fileInfo)
=> extensions_noDots.ContainsInsensative(fileInfo.Extension.Trim('.'));
public string GetPath(string productId) => GetFilePath(productId);
}
public class AaxcFileStorage : AudibleFileStorage
{
protected override string[] Extensions { get; } = new[] { "aaxc" };
// we always want to use the latest config value, therefore
// - DO use 'get' arrow "=>"
// - do NOT use assign "="
public override string StorageDirectory => DownloadsInProgress;
public AaxcFileStorage() : base(FileType.AAXC) { }
/// <summary>
/// Example for full books:
/// Search recursively in _books directory. Full book exists if either are true
/// - a directory name has the product id and an audio file is immediately inside
/// - any audio filename contains the product id
/// </summary>
public bool Exists(string productId) => GetFilePath(productId) != null;
}
}

View File

@@ -8,13 +8,9 @@ using System.Threading.Tasks;
namespace FileManager
{
/// <summary>
/// Tracks actual locations of files. This is especially useful for clicking button to navigate to the book's files.
///
/// Note: this is no longer how Libation manages "Liberated" state. That is not statefully managed in the database.
/// This paradigm is what allows users to manually choose to not download books. Also allows them to manually toggle
/// this state and download again.
/// Tracks actual locations of files.
/// </summary>
internal class BackgroundFileSystem
public class BackgroundFileSystem
{
public string RootDirectory { get; private set; }
public string SearchPattern { get; private set; }
@@ -47,7 +43,7 @@ namespace FileManager
lock (fsCacheLocker)
{
fsCache.Clear();
fsCache.AddRange(Directory.EnumerateFiles(RootDirectory, SearchPattern, SearchOption));
fsCache.AddRange(FileUtility.SaferEnumerateFiles(RootDirectory, SearchPattern, SearchOption));
}
}
@@ -56,7 +52,7 @@ namespace FileManager
Stop();
lock (fsCacheLocker)
fsCache.AddRange(Directory.EnumerateFiles(RootDirectory, SearchPattern, SearchOption));
fsCache.AddRange(FileUtility.SaferEnumerateFiles(RootDirectory, SearchPattern, SearchOption));
directoryChangesEvents = new BlockingCollection<FileSystemEventArgs>();
fileSystemWatcher = new FileSystemWatcher(RootDirectory)
@@ -139,7 +135,7 @@ namespace FileManager
private void AddPath(string path)
{
if (File.GetAttributes(path).HasFlag(FileAttributes.Directory))
AddUniqueFiles(Directory.EnumerateFiles(path, SearchPattern, SearchOption));
AddUniqueFiles(FileUtility.SaferEnumerateFiles(path, SearchPattern, SearchOption));
else
AddUniqueFile(path);
}

View File

@@ -1,13 +1,12 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Dinah.Core" Version="1.1.1.2" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="5.0.0" />
<PackageReference Include="Polly" Version="7.2.2" />
<PackageReference Include="Dinah.Core" Version="4.0.6.1" />
<PackageReference Include="Polly" Version="7.2.3" />
</ItemGroup>
</Project>

View File

@@ -0,0 +1,64 @@
using System;
using System.Collections.Generic;
using System.Linq;
using Dinah.Core;
namespace FileManager
{
/// <summary>Get valid filename. Advanced features incl. parameterized template</summary>
public class FileNamingTemplate
{
/// <summary>Proposed full file path. May contain optional html-styled template tags. Eg: &lt;name&gt;</summary>
public string Template { get; }
/// <param name="template">Proposed file name with optional html-styled template tags.</param>
public FileNamingTemplate(string template) => Template = ArgumentValidator.EnsureNotNullOrWhiteSpace(template, nameof(template));
/// <summary>Optional step 1: Replace html-styled template tags with parameters. Eg {"name", "Bill Gates"} => /&lt;name&gt;/ => /Bill Gates/</summary>
public Dictionary<string, object> ParameterReplacements { get; } = new Dictionary<string, object>();
/// <summary>Convenience method</summary>
public void AddParameterReplacement(string key, object value)
// using .Add() instead of "[key] = value" will make unintended overwriting throw exception
=> ParameterReplacements.Add(key, value);
/// <summary>If set, truncate each parameter replacement to this many characters. Default 50</summary>
public int? ParameterMaxSize { get; set; } = 50;
/// <summary>Optional step 2: Replace all illegal characters with this. Default=<see cref="string.Empty"/></summary>
public string IllegalCharacterReplacements { get; set; }
/// <summary>Generate a valid path for this file or directory</summary>
public string GetFilePath()
{
var filename = Template;
foreach (var r in ParameterReplacements)
filename = filename.Replace($"<{formatKey(r.Key)}>", formatValue(r.Value));
return FileUtility.GetValidFilename(filename, IllegalCharacterReplacements);
}
private static string formatKey(string key)
=> key
.Replace("<", "")
.Replace(">", "");
private string formatValue(object value)
{
if (value is null)
return "";
// Other illegal characters will be taken care of later. Must take care of slashes now so params can't introduce new folders.
// Esp important for file templates.
var val = value
.ToString()
.Replace($"{System.IO.Path.DirectorySeparatorChar}", IllegalCharacterReplacements)
.Replace($"{System.IO.Path.AltDirectorySeparatorChar}", IllegalCharacterReplacements);
return
ParameterMaxSize.HasValue && ParameterMaxSize.Value > 0
? val.Truncate(ParameterMaxSize.Value)
: val;
}
}
}

View File

@@ -1,95 +0,0 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Dinah.Core.Collections.Immutable;
using Newtonsoft.Json;
namespace FileManager
{
public static class FilePathCache
{
private const string FILENAME = "FileLocations.json";
internal class CacheEntry
{
public string Id { get; set; }
public FileType FileType { get; set; }
public string Path { get; set; }
}
private static Cache<CacheEntry> cache { get; } = new Cache<CacheEntry>();
private static string jsonFile => Path.Combine(Configuration.Instance.LibationFiles, FILENAME);
static FilePathCache()
{
// load json into memory. if file doesn't exist, nothing to do. save() will create if needed
if (File.Exists(jsonFile))
{
var list = JsonConvert.DeserializeObject<List<CacheEntry>>(File.ReadAllText(jsonFile));
cache = new Cache<CacheEntry>(list);
}
}
public static bool Exists(string id, FileType type) => GetPath(id, type) != null;
public static string GetPath(string id, FileType type)
{
var entry = cache.SingleOrDefault(i => i.Id == id && i.FileType == type);
if (entry == null)
return null;
if (!File.Exists(entry.Path))
{
remove(entry);
return null;
}
return entry.Path;
}
private static void remove(CacheEntry entry)
{
cache.Remove(entry);
save();
}
public static void Upsert(string id, FileType type, string path)
{
if (!File.Exists(path))
throw new FileNotFoundException("Cannot add path to cache. File not found");
var entry = cache.SingleOrDefault(i => i.Id == id && i.FileType == type);
if (entry is null)
cache.Add(new CacheEntry { Id = id, FileType = type, Path = path });
else
entry.Path = path;
save();
}
// cache is thread-safe and lock free. but file saving is not
private static object locker { get; } = new object();
private static void save()
{
// create json if not exists
static void resave() => File.WriteAllText(jsonFile, JsonConvert.SerializeObject(cache.ToList(), Formatting.Indented));
lock (locker)
{
try { resave(); }
catch (IOException)
{
try { resave(); }
catch (IOException ex)
{
Serilog.Log.Logger.Error(ex, $"Error saving {FILENAME}");
throw;
}
}
}
}
}
}

View File

@@ -1,52 +1,272 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using Dinah.Core;
using Polly;
using Polly.Retry;
namespace FileManager
{
public static class FileUtility
{
public static string GetValidFilename(string dirFullPath, string filename, string extension, params string[] metadataSuffixes)
/// <summary>
/// "txt" => ".txt"
/// <br />".txt" => ".txt"
/// <br />null or whitespace => ""
/// </summary>
public static string GetStandardizedExtension(string extension)
=> string.IsNullOrWhiteSpace(extension)
? (extension ?? "")?.Trim()
: '.' + extension.Trim().Trim('.');
/// <summary>
/// Return position with correct number of leading zeros.
/// <br />- 2 of 9 => "2"
/// <br />- 2 of 90 => "02"
/// <br />- 2 of 900 => "002"
/// </summary>
/// <param name="position">position in sequence. The 'x' in 'x of y'</param>
/// <param name="total">total qty in sequence. The 'y' in 'x of y'</param>
public static string GetSequenceFormatted(int position, int total)
{
if (string.IsNullOrWhiteSpace(dirFullPath))
throw new ArgumentException($"{nameof(dirFullPath)} may not be null or whitespace", nameof(dirFullPath));
ArgumentValidator.EnsureGreaterThan(position, nameof(position), 0);
ArgumentValidator.EnsureGreaterThan(total, nameof(total), 0);
if (position > total)
throw new ArgumentException($"{position} may not be greater than {total}");
// file max length = 255. dir max len = 247
return position.ToString().PadLeft(total.ToString().Length, '0');
}
// sanitize
filename = getAsciiTag(filename);
// manage length
if (filename.Length > 50)
filename = filename.Substring(0, 50) + "[...]";
private const int MAX_FILENAME_LENGTH = 255;
private const int MAX_DIRECTORY_LENGTH = 247;
// append id. it is 10 or 14 char in the common cases
if (metadataSuffixes != null && metadataSuffixes.Length > 0)
filename += " [" + string.Join("][", metadataSuffixes) + "]";
/// <summary>
/// Ensure valid file name path:
/// <br/>- remove invalid chars
/// <br/>- ensure uniqueness
/// <br/>- enforce max file length
/// </summary>
public static string GetValidFilename(string path, string illegalCharacterReplacements = "")
{
ArgumentValidator.EnsureNotNull(path, nameof(path));
// this method may also be used for directory names, so no guarantee of extension
if (!string.IsNullOrWhiteSpace(extension))
extension = '.' + extension.Trim('.');
// remove invalid chars
path = GetSafePath(path, illegalCharacterReplacements);
// ensure uniqueness and check lengths
var dir = Path.GetDirectoryName(path);
dir = dir.Truncate(MAX_DIRECTORY_LENGTH);
var filename = Path.GetFileNameWithoutExtension(path);
var fileStem = Path.Combine(dir, filename);
var extension = Path.GetExtension(path);
var fullfilename = fileStem.Truncate(MAX_FILENAME_LENGTH - extension.Length) + extension;
fullfilename = removeInvalidWhitespace(fullfilename);
// ensure uniqueness
var fullfilename = Path.Combine(dirFullPath, filename + extension);
var i = 0;
while (File.Exists(fullfilename))
fullfilename = Path.Combine(dirFullPath, filename + $" ({++i})" + extension);
{
var increm = $" ({++i})";
fullfilename = fileStem.Truncate(MAX_FILENAME_LENGTH - increm.Length - extension.Length) + increm + extension;
}
return fullfilename;
}
private static string getAsciiTag(string property)
// GetInvalidFileNameChars contains everything in GetInvalidPathChars plus ':', '*', '?', '\\', '/'
/// <summary>Use with file name, not full path. Valid path charaters which are invalid file name characters will be replaced: ':', '\\', '/'</summary>
public static string GetSafeFileName(string str, string illegalCharacterReplacements = "")
=> string.Join(illegalCharacterReplacements ?? "", str.Split(Path.GetInvalidFileNameChars()));
/// <summary>Use with full path, not file name. Valid path charaters which are invalid file name characters will be retained: '\\', '/'</summary>
public static string GetSafePath(string path, string illegalCharacterReplacements = "")
{
if (property == null)
return "";
ArgumentValidator.EnsureNotNull(path, nameof(path));
// omit characters which are invalid. EXCEPTION: change colon to underscore
property = property.Replace(':', '_');
path = replaceInvalidChars(path, illegalCharacterReplacements);
path = standardizeSlashes(path);
path = replaceColons(path, illegalCharacterReplacements);
path = removeDoubleSlashes(path);
// GetInvalidFileNameChars contains everything in GetInvalidPathChars plus ':', '*', '?', '\\', '/'
foreach (var ch in Path.GetInvalidFileNameChars())
property = property.Replace(ch.ToString(), "");
return property;
return path;
}
private static char[] invalidChars { get; } = Path.GetInvalidPathChars().Union(new[] {
'*', '?',
// these are weird. If you run Path.GetInvalidPathChars() in Visual Studio's "C# Interactive", then these characters are included.
// In live code, Path.GetInvalidPathChars() does not include them
'"', '<', '>'
}).ToArray();
private static string replaceInvalidChars(string path, string illegalCharacterReplacements)
=> string.Join(illegalCharacterReplacements ?? "", path.Split(invalidChars));
private static string standardizeSlashes(string path)
=> path.Replace(Path.AltDirectorySeparatorChar, Path.DirectorySeparatorChar);
private static string replaceColons(string path, string illegalCharacterReplacements)
{
// replace all colons except within the first 2 chars
var builder = new System.Text.StringBuilder();
for (var i = 0; i < path.Length; i++)
{
var c = path[i];
if (i >= 2 && c == ':')
builder.Append(illegalCharacterReplacements);
else
builder.Append(c);
}
return builder.ToString();
}
private static string removeDoubleSlashes(string path)
{
if (path.Length < 2)
return path;
// exception: don't try to condense the initial dbl bk slashes in a path. eg: \\192.168.0.1
var remainder = path[1..];
var dblSeparator = $"{Path.DirectorySeparatorChar}{Path.DirectorySeparatorChar}";
while (remainder.Contains(dblSeparator))
remainder = remainder.Replace(dblSeparator, $"{Path.DirectorySeparatorChar}");
return path[0] + remainder;
}
private static string removeInvalidWhitespace_pattern { get; } = $@"[\s\.]*\{Path.DirectorySeparatorChar}\s*";
private static Regex removeInvalidWhitespace_regex { get; } = new(removeInvalidWhitespace_pattern, RegexOptions.Compiled | RegexOptions.IgnorePatternWhitespace);
/// <summary>no part of the path may begin or end in whitespace</summary>
private static string removeInvalidWhitespace(string fullfilename)
{
// no whitespace at beginning or end
// replace whitespace around path slashes
// regex (with space added for clarity)
// \s* \\ \s* => \
// no ending dots. beginning dots are valid
// regex is easier by ending with separator
fullfilename += Path.DirectorySeparatorChar;
fullfilename = removeInvalidWhitespace_regex.Replace(fullfilename, Path.DirectorySeparatorChar.ToString());
// take seperator back off
fullfilename = RemoveLastCharacter(fullfilename);
fullfilename = removeDoubleSlashes(fullfilename);
return fullfilename;
}
public static string RemoveLastCharacter(this string str) => string.IsNullOrEmpty(str) ? str : str[..^1];
/// <summary>
/// Move file.
/// <br/>- Ensure valid file name path: remove invalid chars, ensure uniqueness, enforce max file length
/// <br/>- Perform <see cref="SaferMove"/>
/// <br/>- Return valid path
/// </summary>
public static string SaferMoveToValidPath(string source, string destination)
{
destination = GetValidFilename(destination);
SaferMove(source, destination);
return destination;
}
private static int maxRetryAttempts { get; } = 3;
private static TimeSpan pauseBetweenFailures { get; } = TimeSpan.FromMilliseconds(100);
private static RetryPolicy retryPolicy { get; } =
Policy
.Handle<Exception>()
.WaitAndRetry(maxRetryAttempts, i => pauseBetweenFailures);
/// <summary>Delete file. No error when source does not exist. Retry up to 3 times before throwing exception.</summary>
public static void SaferDelete(string source)
=> retryPolicy.Execute(() =>
{
try
{
if (!File.Exists(source))
{
Serilog.Log.Logger.Debug("No file to delete: {@DebugText}", new { source });
return;
}
Serilog.Log.Logger.Debug("Attempt to delete file: {@DebugText}", new { source });
File.Delete(source);
Serilog.Log.Logger.Information("File successfully deleted: {@DebugText}", new { source });
}
catch (Exception e)
{
Serilog.Log.Logger.Error(e, "Failed to delete file: {@DebugText}", new { source });
throw;
}
});
/// <summary>Move file. No error when source does not exist. Retry up to 3 times before throwing exception.</summary>
public static void SaferMove(string source, string destination)
=> retryPolicy.Execute(() =>
{
try
{
if (!File.Exists(source))
{
Serilog.Log.Logger.Debug("No file to move: {@DebugText}", new { source });
return;
}
SaferDelete(destination);
var dir = Path.GetDirectoryName(destination);
Serilog.Log.Logger.Debug("Attempt to create directory: {@DebugText}", new { dir });
Directory.CreateDirectory(dir);
Serilog.Log.Logger.Debug("Attempt to move file: {@DebugText}", new { source, destination });
File.Move(source, destination);
Serilog.Log.Logger.Information("File successfully moved: {@DebugText}", new { source, destination });
}
catch (Exception e)
{
Serilog.Log.Logger.Error(e, "Failed to move file: {@DebugText}", new { source, destination });
throw;
}
});
/// <summary>
/// A safer way to get all the files in a directory and sub directory without crashing on UnauthorizedException or PathTooLongException
/// </summary>
/// <param name="rootPath">Starting directory</param>
/// <param name="patternMatch">Filename pattern match</param>
/// <param name="searchOption">Search subdirectories or only top level directory for files</param>
/// <returns>List of files</returns>
public static IEnumerable<string> SaferEnumerateFiles(string path, string searchPattern = "*", SearchOption searchOption = SearchOption.TopDirectoryOnly)
{
var foundFiles = Enumerable.Empty<string>();
if (searchOption == SearchOption.AllDirectories)
{
try
{
IEnumerable<string> subDirs = Directory.EnumerateDirectories(path);
// Add files in subdirectories recursively to the list
foreach (string dir in subDirs)
foundFiles = foundFiles.Concat(SaferEnumerateFiles(dir, searchPattern, searchOption));
}
catch (UnauthorizedAccessException) { }
catch (PathTooLongException) { }
}
try
{
// Add files from the current directory
foundFiles = foundFiles.Concat(Directory.EnumerateFiles(path, searchPattern));
}
catch (UnauthorizedAccessException) { }
return foundFiles;
}
}
}

View File

@@ -30,8 +30,7 @@ namespace FileManager
if (IsReadOnly)
return;
File.WriteAllText(Filepath, "{}");
System.Threading.Thread.Sleep(100);
createNewFile();
}
public string GetString(string propertyName)
@@ -205,9 +204,39 @@ namespace FileManager
private JObject readFile()
{
if (!File.Exists(Filepath))
{
var msg = "Unrecoverable error. Settings file cannot be found";
var ex = new FileNotFoundException(msg, Filepath);
Serilog.Log.Logger.Error(ex, msg);
throw ex;
}
var settingsJsonContents = File.ReadAllText(Filepath);
if (string.IsNullOrWhiteSpace(settingsJsonContents))
{
createNewFile();
settingsJsonContents = File.ReadAllText(Filepath);
}
var jObject = JsonConvert.DeserializeObject<JObject>(settingsJsonContents);
if (jObject is null)
{
var msg = "Unrecoverable error. Unable to read settings from Settings file";
var ex = new NullReferenceException(msg);
Serilog.Log.Logger.Error(ex, msg);
throw ex;
}
return jObject;
}
private void createNewFile()
{
File.WriteAllText(Filepath, "{}");
System.Threading.Thread.Sleep(100);
}
}
}

View File

@@ -0,0 +1 @@
[assembly: System.Runtime.CompilerServices.InternalsVisibleTo(nameof(FileManager) + ".Tests")]

View File

@@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0</TargetFramework>
<TargetFramework>net6.0</TargetFramework>
</PropertyGroup>
</Project>

View File

@@ -1,15 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AudibleApi" Version="2.2.0.1" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\FileManager\FileManager.csproj" />
</ItemGroup>
</Project>

View File

@@ -1,7 +1,7 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 16
VisualStudioVersion = 16.0.28803.156
# Visual Studio Version 17
VisualStudioVersion = 17.0.31903.59
MinimumVisualStudioVersion = 10.0.40219.1
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "_Solution Items", "_Solution Items", "{03C8835F-936C-4AF7-87AE-FF92BDBE8B9B}"
ProjectSection(SolutionItems) = preProject
@@ -23,13 +23,13 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "2 Utilities (domain ignoran
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "AaxDecrypter", "AaxDecrypter\AaxDecrypter.csproj", "{8BD8E012-F44F-4EE2-A234-D66C14D5FE4B}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FileManager", "FileManager\FileManager.csproj", "{1AE65B61-9C05-4C80-ABFF-48F16E22FDF1}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "LibationFileManager", "LibationFileManager\LibationFileManager.csproj", "{1AE65B61-9C05-4C80-ABFF-48F16E22FDF1}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "DataLayer", "DataLayer\DataLayer.csproj", "{59A10DF3-63EC-43F1-A3BF-4000CFA118D2}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FileLiberator", "FileLiberator\FileLiberator.csproj", "{393B5B27-D15C-4F77-9457-FA14BA8F3C73}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "InternalUtilities", "InternalUtilities\InternalUtilities.csproj", "{06882742-27A6-4347-97D9-56162CEC9C11}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "AudibleUtilities", "AudibleUtilities\AudibleUtilities.csproj", "{06882742-27A6-4347-97D9-56162CEC9C11}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "3 Domain Internal Utilities (db ignorant)", "3 Domain Internal Utilities (db ignorant)", "{F0CBB7A7-D3FB-41FF-8F47-CF3F6A592249}"
EndProject
@@ -46,8 +46,6 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "ApplicationServices", "Appl
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "_Tests", "_Tests", "{67E66E82-5532-4440-AFB3-9FB1DF9DEF53}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "InternalUtilities.Tests", "_Tests\InternalUtilities.Tests\InternalUtilities.Tests.csproj", "{8447C956-B03E-4F59-9DD4-877793B849D9}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "LibationSearchEngine.Tests", "_Tests\LibationSearchEngine.Tests\LibationSearchEngine.Tests.csproj", "{C5B21768-C7C9-4FCB-AC1E-187B223D5A98}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Hoopla", "Hoopla\Hoopla.csproj", "{D8F56E5A-3E65-41A6-B7E7-C4515A264B1F}"
@@ -56,6 +54,16 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "LibationCli", "LibationCli\
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "AppScaffolding", "AppScaffolding\AppScaffolding.csproj", "{595E7C4D-506D-486D-98B7-5FDDF398D033}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FileManager", "FileManager\FileManager.csproj", "{E86014F9-E4B3-4CD4-A210-2B3DB571DD86}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "AudibleUtilities.Tests", "_Tests\AudibleUtilities.Tests\AudibleUtilities.Tests.csproj", "{788294BE-0D8E-40D4-9CEE-67896FBB52CE}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FileLiberator.Tests", "_Tests\FileLiberator.Tests\FileLiberator.Tests.csproj", "{5B8FC827-BF58-4CB1-A59E-BDEB9C62A05E}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FileManager.Tests", "_Tests\FileManager.Tests\FileManager.Tests.csproj", "{F2E04270-4551-41C4-99FF-E7125BED708C}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "LibationFileManager.Tests", "_Tests\LibationFileManager.Tests\LibationFileManager.Tests.csproj", "{EB781571-8548-477E-82AD-FB9FAB548D2F}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
@@ -98,10 +106,6 @@ Global
{B95650EA-25F0-449E-BA5D-99126BC5D730}.Debug|Any CPU.Build.0 = Debug|Any CPU
{B95650EA-25F0-449E-BA5D-99126BC5D730}.Release|Any CPU.ActiveCfg = Release|Any CPU
{B95650EA-25F0-449E-BA5D-99126BC5D730}.Release|Any CPU.Build.0 = Release|Any CPU
{8447C956-B03E-4F59-9DD4-877793B849D9}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{8447C956-B03E-4F59-9DD4-877793B849D9}.Debug|Any CPU.Build.0 = Debug|Any CPU
{8447C956-B03E-4F59-9DD4-877793B849D9}.Release|Any CPU.ActiveCfg = Release|Any CPU
{8447C956-B03E-4F59-9DD4-877793B849D9}.Release|Any CPU.Build.0 = Release|Any CPU
{C5B21768-C7C9-4FCB-AC1E-187B223D5A98}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{C5B21768-C7C9-4FCB-AC1E-187B223D5A98}.Debug|Any CPU.Build.0 = Debug|Any CPU
{C5B21768-C7C9-4FCB-AC1E-187B223D5A98}.Release|Any CPU.ActiveCfg = Release|Any CPU
@@ -118,6 +122,26 @@ Global
{595E7C4D-506D-486D-98B7-5FDDF398D033}.Debug|Any CPU.Build.0 = Debug|Any CPU
{595E7C4D-506D-486D-98B7-5FDDF398D033}.Release|Any CPU.ActiveCfg = Release|Any CPU
{595E7C4D-506D-486D-98B7-5FDDF398D033}.Release|Any CPU.Build.0 = Release|Any CPU
{E86014F9-E4B3-4CD4-A210-2B3DB571DD86}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{E86014F9-E4B3-4CD4-A210-2B3DB571DD86}.Debug|Any CPU.Build.0 = Debug|Any CPU
{E86014F9-E4B3-4CD4-A210-2B3DB571DD86}.Release|Any CPU.ActiveCfg = Release|Any CPU
{E86014F9-E4B3-4CD4-A210-2B3DB571DD86}.Release|Any CPU.Build.0 = Release|Any CPU
{788294BE-0D8E-40D4-9CEE-67896FBB52CE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{788294BE-0D8E-40D4-9CEE-67896FBB52CE}.Debug|Any CPU.Build.0 = Debug|Any CPU
{788294BE-0D8E-40D4-9CEE-67896FBB52CE}.Release|Any CPU.ActiveCfg = Release|Any CPU
{788294BE-0D8E-40D4-9CEE-67896FBB52CE}.Release|Any CPU.Build.0 = Release|Any CPU
{5B8FC827-BF58-4CB1-A59E-BDEB9C62A05E}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{5B8FC827-BF58-4CB1-A59E-BDEB9C62A05E}.Debug|Any CPU.Build.0 = Debug|Any CPU
{5B8FC827-BF58-4CB1-A59E-BDEB9C62A05E}.Release|Any CPU.ActiveCfg = Release|Any CPU
{5B8FC827-BF58-4CB1-A59E-BDEB9C62A05E}.Release|Any CPU.Build.0 = Release|Any CPU
{F2E04270-4551-41C4-99FF-E7125BED708C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{F2E04270-4551-41C4-99FF-E7125BED708C}.Debug|Any CPU.Build.0 = Debug|Any CPU
{F2E04270-4551-41C4-99FF-E7125BED708C}.Release|Any CPU.ActiveCfg = Release|Any CPU
{F2E04270-4551-41C4-99FF-E7125BED708C}.Release|Any CPU.Build.0 = Release|Any CPU
{EB781571-8548-477E-82AD-FB9FAB548D2F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{EB781571-8548-477E-82AD-FB9FAB548D2F}.Debug|Any CPU.Build.0 = Debug|Any CPU
{EB781571-8548-477E-82AD-FB9FAB548D2F}.Release|Any CPU.ActiveCfg = Release|Any CPU
{EB781571-8548-477E-82AD-FB9FAB548D2F}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
@@ -132,11 +156,15 @@ Global
{635F00E1-AAD1-45F7-BEB7-D909AD33B9F6} = {8679CAC8-9164-4007-BDD2-F004810EDA14}
{401865F5-1942-4713-B230-04544C0A97B0} = {41CDCC73-9B81-49DD-9570-C54406E852AF}
{B95650EA-25F0-449E-BA5D-99126BC5D730} = {41CDCC73-9B81-49DD-9570-C54406E852AF}
{8447C956-B03E-4F59-9DD4-877793B849D9} = {67E66E82-5532-4440-AFB3-9FB1DF9DEF53}
{C5B21768-C7C9-4FCB-AC1E-187B223D5A98} = {67E66E82-5532-4440-AFB3-9FB1DF9DEF53}
{D8F56E5A-3E65-41A6-B7E7-C4515A264B1F} = {7FBBB086-0807-4998-85BF-6D1A49C8AD05}
{428163C3-D558-4914-B570-A92069521877} = {8679CAC8-9164-4007-BDD2-F004810EDA14}
{595E7C4D-506D-486D-98B7-5FDDF398D033} = {8679CAC8-9164-4007-BDD2-F004810EDA14}
{E86014F9-E4B3-4CD4-A210-2B3DB571DD86} = {43E3ACB3-E0BC-4370-8DBB-E3720C8C8FD1}
{788294BE-0D8E-40D4-9CEE-67896FBB52CE} = {67E66E82-5532-4440-AFB3-9FB1DF9DEF53}
{5B8FC827-BF58-4CB1-A59E-BDEB9C62A05E} = {67E66E82-5532-4440-AFB3-9FB1DF9DEF53}
{F2E04270-4551-41C4-99FF-E7125BED708C} = {67E66E82-5532-4440-AFB3-9FB1DF9DEF53}
{EB781571-8548-477E-82AD-FB9FAB548D2F} = {67E66E82-5532-4440-AFB3-9FB1DF9DEF53}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {615E00ED-BAEF-4E8E-A92A-9B82D87942A9}

View File

@@ -3,7 +3,7 @@
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net5.0</TargetFramework>
<TargetFramework>net6.0-windows</TargetFramework>
<PublishTrimmed>true</PublishTrimmed>
<PublishReadyToRun>true</PublishReadyToRun>

View File

@@ -3,8 +3,8 @@ using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using ApplicationServices;
using AudibleUtilities;
using CommandLine;
using InternalUtilities;
namespace LibationCli
{

View File

@@ -3,8 +3,8 @@ using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using ApplicationServices;
using AudibleUtilities;
using CommandLine;
using InternalUtilities;
namespace LibationCli
{
@@ -35,7 +35,8 @@ namespace LibationCli
var (TotalBooksProcessed, NewBooksAdded) = await LibraryCommands.ImportAccountAsync((a) => ApiExtended.CreateAsync(a), _accounts);
Console.WriteLine("Scan complete.");
Console.WriteLine($"Total processed: {TotalBooksProcessed}\r\nNew: {NewBooksAdded}");
Console.WriteLine($"Total processed: {TotalBooksProcessed}");
Console.WriteLine($"New: {NewBooksAdded}");
}
private Account[] getAccounts()

View File

@@ -0,0 +1,106 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using FileManager;
namespace LibationFileManager
{
public abstract class AudibleFileStorage
{
protected abstract string GetFilePathCustom(string productId);
#region static
public static string DownloadsInProgressDirectory => Directory.CreateDirectory(Path.Combine(Configuration.Instance.InProgress, "DownloadsInProgress")).FullName;
public static string DecryptInProgressDirectory => Directory.CreateDirectory(Path.Combine(Configuration.Instance.InProgress, "DecryptInProgress")).FullName;
private static AaxcFileStorage AAXC { get; } = new AaxcFileStorage();
public static bool AaxcExists(string productId) => AAXC.Exists(productId);
public static AudioFileStorage Audio { get; } = new AudioFileStorage();
public static string BooksDirectory
{
get
{
if (string.IsNullOrWhiteSpace(Configuration.Instance.Books))
Configuration.Instance.Books = Path.Combine(Configuration.UserProfile, "Books");
return Directory.CreateDirectory(Configuration.Instance.Books).FullName;
}
}
#endregion
#region instance
private FileType FileType { get; }
private string regexTemplate { get; }
protected AudibleFileStorage(FileType fileType)
{
FileType = fileType;
var extAggr = FileTypes.GetExtensions(FileType).Aggregate((a, b) => $"{a}|{b}");
regexTemplate = $@"{{0}}.*?\.({extAggr})$";
}
protected string GetFilePath(string productId)
{
// primary lookup
var cachedFile = FilePathCache.GetFirstPath(productId, FileType);
if (cachedFile is not null && File.Exists(cachedFile))
return cachedFile;
// secondary lookup attempt
var firstOrNull = GetFilePathCustom(productId);
if (firstOrNull is not null)
FilePathCache.Insert(productId, firstOrNull);
return firstOrNull;
}
protected Regex GetBookSearchRegex(string productId)
{
var pattern = string.Format(regexTemplate, productId);
return new Regex(pattern, RegexOptions.IgnoreCase);
}
#endregion
}
internal class AaxcFileStorage : AudibleFileStorage
{
internal AaxcFileStorage() : base(FileType.AAXC) { }
protected override string GetFilePathCustom(string productId)
{
var regex = GetBookSearchRegex(productId);
return FileUtility
.SaferEnumerateFiles(DownloadsInProgressDirectory, "*.*", SearchOption.AllDirectories)
.FirstOrDefault(s => regex.IsMatch(s));
}
public bool Exists(string productId) => GetFilePath(productId) != null;
}
public class AudioFileStorage : AudibleFileStorage
{
internal AudioFileStorage() : base(FileType.Audio)
=> BookDirectoryFiles ??= new BackgroundFileSystem(BooksDirectory, "*.*", SearchOption.AllDirectories);
private static BackgroundFileSystem BookDirectoryFiles { get; set; }
private static object bookDirectoryFilesLocker { get; } = new();
protected override string GetFilePathCustom(string productId)
{
// If user changed the BooksDirectory: reinitialize
lock (bookDirectoryFilesLocker)
if (BooksDirectory != BookDirectoryFiles.RootDirectory)
BookDirectoryFiles = new BackgroundFileSystem(BooksDirectory, "*.*", SearchOption.AllDirectories);
var regex = GetBookSearchRegex(productId);
return BookDirectoryFiles.FindFile(regex);
}
public void Refresh() => BookDirectoryFiles.RefreshFiles();
public string GetPath(string productId) => GetFilePath(productId);
}
}

View File

@@ -5,13 +5,14 @@ using System.IO;
using System.Linq;
using Dinah.Core;
using Dinah.Core.Logging;
using FileManager;
using Microsoft.Extensions.Configuration;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using Serilog;
using Serilog.Events;
namespace FileManager
namespace LibationFileManager
{
public class Configuration
{
@@ -132,6 +133,13 @@ namespace FileManager
set => persistentDictionary.SetString(nameof(BadBook), value.ToString());
}
[Description("Show number of newly imported titles? When unchecked, no pop-up will appear after library scan.")]
public bool ShowImportedStats
{
get => persistentDictionary.GetNonString<bool>(nameof(ShowImportedStats));
set => persistentDictionary.SetNonString(nameof(ShowImportedStats), value);
}
[Description("Import episodes? (eg: podcasts) When unchecked, episodes will not be imported into Libation.")]
public bool ImportEpisodes
{
@@ -144,8 +152,40 @@ namespace FileManager
{
get => persistentDictionary.GetNonString<bool>(nameof(DownloadEpisodes));
set => persistentDictionary.SetNonString(nameof(DownloadEpisodes), value);
}
#region templates: custom file naming
[Description("How to format the folders in which files will be saved")]
public string FolderTemplate
{
get => getTemplate(nameof(FolderTemplate), Templates.Folder);
set => setTemplate(nameof(FolderTemplate), Templates.Folder, value);
}
[Description("How to format the saved pdf and audio files")]
public string FileTemplate
{
get => getTemplate(nameof(FileTemplate), Templates.File);
set => setTemplate(nameof(FileTemplate), Templates.File, value);
}
[Description("How to format the saved audio files when split by chapters")]
public string ChapterFileTemplate
{
get => getTemplate(nameof(ChapterFileTemplate), Templates.ChapterFile);
set => setTemplate(nameof(ChapterFileTemplate), Templates.ChapterFile, value);
}
private string getTemplate(string settingName, Templates templ) => templ.GetValid(persistentDictionary.GetString(settingName));
private void setTemplate(string settingName, Templates templ, string newValue)
{
var template = newValue?.Trim();
if (templ.IsValid(template))
persistentDictionary.SetString(settingName, template);
}
#endregion
#endregion
#region known directories
@@ -230,7 +270,7 @@ namespace FileManager
var valueWasChanged = persistentDictionary.SetWithJsonPath("Serilog", "MinimumLevel", value.ToString());
if (!valueWasChanged)
{
Log.Logger.Information("LogLevel.set attempt. No change");
Log.Logger.Debug("LogLevel.set attempt. No change");
return;
}

View File

@@ -0,0 +1,115 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Dinah.Core.Collections.Immutable;
using Newtonsoft.Json;
namespace LibationFileManager
{
public static class FilePathCache
{
public record CacheEntry(string Id, FileType FileType, string Path);
private const string FILENAME = "FileLocations.json";
public static event EventHandler<CacheEntry> Inserted;
public static event EventHandler<CacheEntry> Removed;
private static Cache<CacheEntry> cache { get; } = new Cache<CacheEntry>();
private static string jsonFile => Path.Combine(Configuration.Instance.LibationFiles, FILENAME);
static FilePathCache()
{
// load json into memory. if file doesn't exist, nothing to do. save() will create if needed
if (File.Exists(jsonFile))
{
var list = JsonConvert.DeserializeObject<List<CacheEntry>>(File.ReadAllText(jsonFile));
// file exists but deser is null. this will never happen when file is healthy
if (list is null)
{
lock (locker)
{
Serilog.Log.Logger.Error("Error deserializing file. Wrong format. Possibly corrupt. Deleting file. {@DebugInfo}", new { jsonFile });
File.Delete(jsonFile);
return;
}
}
cache = new Cache<CacheEntry>(list);
}
}
public static bool Exists(string id, FileType type) => GetFirstPath(id, type) != null;
public static List<(FileType fileType, string path)> GetFiles(string id)
=> getEntries(entry => entry.Id == id)
.Select(entry => (entry.FileType, entry.Path))
.ToList();
public static string GetFirstPath(string id, FileType type)
=> getEntries(entry => entry.Id == id && entry.FileType == type)
?.FirstOrDefault()
?.Path;
private static List<CacheEntry> getEntries(Func<CacheEntry, bool> predicate)
{
var entries = cache.Where(predicate).ToList();
if (entries is null || !entries.Any())
return null;
remove(entries.Where(e => !File.Exists(e.Path)).ToList());
return entries;
}
private static void remove(List<CacheEntry> entries)
{
if (entries is null)
return;
lock (locker)
{
foreach (var entry in entries)
{
cache.Remove(entry);
Removed?.Invoke(null, entry);
}
save();
}
}
public static void Insert(string id, string path)
{
var type = FileTypes.GetFileTypeFromPath(path);
var entry = new CacheEntry(id, type, path);
cache.Add(entry);
Inserted?.Invoke(null, entry);
save();
}
// cache is thread-safe and lock free. but file saving is not
private static object locker { get; } = new object();
private static void save()
{
// create json if not exists
static void resave() => File.WriteAllText(jsonFile, JsonConvert.SerializeObject(cache.ToList(), Formatting.Indented));
lock (locker)
{
try { resave(); }
catch (IOException)
{
try { resave(); }
catch (IOException ex)
{
Serilog.Log.Logger.Error(ex, $"Error saving {FILENAME}");
throw;
}
}
}
}
}
}

View File

@@ -0,0 +1,39 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
namespace LibationFileManager
{
public enum FileType { Unknown, Audio, AAXC, PDF, Zip, Cue }
public static class FileTypes
{
private static Dictionary<string, FileType> dic => new()
{
["aaxc"] = FileType.AAXC,
["cue"] = FileType.Cue,
["pdf"] = FileType.PDF,
["zip"] = FileType.Zip,
["aac"] = FileType.Audio,
["flac"] = FileType.Audio,
["m4a"] = FileType.Audio,
["m4b"] = FileType.Audio,
["mp3"] = FileType.Audio,
["mp4"] = FileType.Audio,
["ogg"] = FileType.Audio,
};
public static FileType GetFileTypeFromPath(string path)
=> dic.TryGetValue(Path.GetExtension(path).ToLower().Trim('.'), out var fileType)
? fileType
: FileType.Unknown;
public static List<string> GetExtensions(FileType fileType)
=> dic
.Where(kvp => kvp.Value == fileType)
.Select(kvp => kvp.Key)
.ToList();
}
}

View File

@@ -0,0 +1,17 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net6.0-windows</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="6.0.0" />
<PackageReference Include="Serilog.Exceptions" Version="8.1.0" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\AaxDecrypter\AaxDecrypter.csproj" />
<ProjectReference Include="..\FileManager\FileManager.csproj" />
</ItemGroup>
</Project>

View File

@@ -0,0 +1,29 @@
using System;
using System.Collections.Generic;
using System.Linq;
namespace LibationFileManager
{
public class BookDto
{
public string AudibleProductId { get; set; }
public string Title { get; set; }
public string Locale { get; set; }
public IEnumerable<string> Authors { get; set; }
public string AuthorNames => string.Join(", ", Authors);
public string FirstAuthor => Authors.FirstOrDefault();
public IEnumerable<string> Narrators { get; set; }
public string NarratorNames => string.Join(", ", Narrators);
public string FirstNarrator => Narrators.FirstOrDefault();
public string SeriesName { get; set; }
public string SeriesNumber { get; set; }
}
public class LibraryBookDto : BookDto
{
public string Account { get; set; }
}
}

View File

@@ -6,7 +6,7 @@ using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;
namespace FileManager
namespace LibationFileManager
{
public enum PictureSize { _80x80 = 80, _300x300 = 300, _500x500 = 500 }
public class PictureCachedEventArgs : EventArgs
@@ -73,16 +73,10 @@ namespace FileManager
if (!cache.ContainsKey(def) || cache[def] == null)
{
var path = getPath(def);
byte[] bytes;
if (File.Exists(path))
bytes = File.ReadAllBytes(path);
else
{
bytes = downloadBytes(def);
saveFile(def, bytes);
}
var bytes
= File.Exists(path)
? File.ReadAllBytes(path)
: downloadBytes(def);
cache[def] = bytes;
}
return cache[def];
@@ -104,7 +98,6 @@ namespace FileManager
continue;
var bytes = downloadBytes(def);
saveFile(def, bytes);
lock (cacheLocker)
cache[def] = bytes;
@@ -115,14 +108,24 @@ namespace FileManager
private static HttpClient imageDownloadClient { get; } = new HttpClient();
private static byte[] downloadBytes(PictureDefinition def)
{
var sz = (int)def.Size;
return imageDownloadClient.GetByteArrayAsync("ht" + $"tps://images-na.ssl-images-amazon.com/images/I/{def.PictureId}._SL{sz}_.jpg").Result;
}
if (def.PictureId is null)
return getDefaultImage(def.Size);
private static void saveFile(PictureDefinition def, byte[] bytes)
{
var path = getPath(def);
File.WriteAllBytes(path, bytes);
try
{
var sz = (int)def.Size;
var bytes = imageDownloadClient.GetByteArrayAsync("ht" + $"tps://images-na.ssl-images-amazon.com/images/I/{def.PictureId}._SL{sz}_.jpg").Result;
// save image file. make sure to not save default image
var path = getPath(def);
File.WriteAllBytes(path, bytes);
return bytes;
}
catch
{
return getDefaultImage(def.Size);
}
}
}
}

View File

@@ -5,7 +5,7 @@ using System.Linq;
using Dinah.Core.Collections.Generic;
using Newtonsoft.Json;
namespace FileManager
namespace LibationFileManager
{
public static class QuickFilters
{

View File

@@ -1,11 +1,11 @@
using System.IO;
namespace FileManager
namespace LibationFileManager
{
public static class SqliteStorage
{
// not customizable. don't move to config
private static string databasePath => Path.Combine(Configuration.Instance.LibationFiles, "LibationContext.db");
public static string ConnectionString => $"Data Source={databasePath};Foreign Keys=False;";
public static string ConnectionString => $"Data Source={databasePath};Foreign Keys=False;Pooling=False;";
}
}

View File

@@ -6,7 +6,7 @@ using Newtonsoft.Json;
using Polly;
using Polly.Retry;
namespace FileManager
namespace LibationFileManager
{
/// <summary>
/// Tags must also be stored in db for search performance. Stored in json file to survive a db reset.
@@ -54,11 +54,17 @@ namespace FileManager
private static void ensureCache()
{
if (cache is null)
lock (locker)
cache = !File.Exists(TagsFile)
? new Dictionary<string, string>()
: JsonConvert.DeserializeObject<Dictionary<string, string>>(File.ReadAllText(TagsFile));
if (cache is not null)
return;
lock (locker)
{
if (File.Exists(TagsFile))
cache = JsonConvert.DeserializeObject<Dictionary<string, string>>(File.ReadAllText(TagsFile));
// if file doesn't exist. or if file is corrupt and deserialize returns null
cache ??= new Dictionary<string, string>();
}
}
}
}

View File

@@ -0,0 +1,44 @@
using System;
using System.Collections.Generic;
using System.Linq;
using Dinah.Core;
namespace LibationFileManager
{
public sealed class TemplateTags : Enumeration<TemplateTags>
{
public string TagName => DisplayName;
public string Description { get; }
public bool IsChapterOnly { get; }
private static int value = 0;
private TemplateTags(string tagName, string description, bool isChapterOnly = false) : base(value++, tagName)
{
Description = description;
IsChapterOnly = isChapterOnly;
}
// putting these first is the incredibly lazy way to make them show up first in the EditTemplateDialog
public static TemplateTags ChCount { get; } = new TemplateTags("ch count", "Number of chapters", true);
public static TemplateTags ChTitle { get; } = new TemplateTags("ch title", "Chapter title", true);
public static TemplateTags ChNumber { get; } = new TemplateTags("ch#", "Chapter #", true);
public static TemplateTags ChNumber0 { get; } = new TemplateTags("ch# 0", "Chapter # with leading zeros", true);
public static TemplateTags Id { get; } = new TemplateTags("id", "Audible ID");
public static TemplateTags Title { get; } = new TemplateTags("title", "Full title");
public static TemplateTags TitleShort { get; } = new TemplateTags("title short", "Title. Stop at first colon");
public static TemplateTags Author { get; } = new TemplateTags("author", "Author(s)");
public static TemplateTags FirstAuthor { get; } = new TemplateTags("first author", "First author");
public static TemplateTags Narrator { get; } = new TemplateTags("narrator", "Narrator(s)");
public static TemplateTags FirstNarrator { get; } = new TemplateTags("first narrator", "First narrator");
public static TemplateTags Series { get; } = new TemplateTags("series", "Name of series");
// can't also have a leading zeros version. Too many weird edge cases. Eg: "1-4"
public static TemplateTags SeriesNumber { get; } = new TemplateTags("series#", "Number order in series");
public static TemplateTags Account { get; } = new TemplateTags("account", "Audible account of this book");
public static TemplateTags Locale { get; } = new TemplateTags("locale", "Region/country");
// Special case. Isn't mapped to a replacement in Templates.cs
// Included here for display by EditTemplateDialog
public static TemplateTags IfSeries { get; } = new TemplateTags("if series->...<-if series", "Only include if part of a series");
}
}

View File

@@ -0,0 +1,284 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using Dinah.Core;
using FileManager;
namespace LibationFileManager
{
public abstract class Templates
{
protected static string[] Valid => Array.Empty<string>();
public const string ERROR_NULL_IS_INVALID = "Null template is invalid.";
public const string ERROR_FULL_PATH_IS_INVALID = @"No colons or full paths allowed. Eg: should not start with C:\";
public const string ERROR_INVALID_FILE_NAME_CHAR = @"Only file name friendly characters allowed. Eg: no colons or slashes";
public const string WARNING_EMPTY = "Template is empty.";
public const string WARNING_WHITE_SPACE = "Template is white space.";
public const string WARNING_NO_TAGS = "Should use tags. Eg: <title>";
public const string WARNING_HAS_CHAPTER_TAGS = "Chapter tags should only be used in the template used for naming files which are split by chapter. Eg: <ch title>";
public const string WARNING_NO_CHAPTER_NUMBER_TAG = "Should include chapter number tag in template used for naming files which are split by chapter. Ie: <ch#> or <ch# 0>";
public static FolderTemplate Folder { get; } = new FolderTemplate();
public static FileTemplate File { get; } = new FileTemplate();
public static ChapterFileTemplate ChapterFile { get; } = new ChapterFileTemplate();
public abstract string Name { get; }
public abstract string Description { get; }
public abstract string DefaultTemplate { get; }
protected abstract bool IsChapterized { get; }
protected Templates() { }
#region validation
internal string GetValid(string configValue)
{
var value = configValue?.Trim();
return IsValid(value) ? value : DefaultTemplate;
}
public abstract IEnumerable<string> GetErrors(string template);
public bool IsValid(string template) => !GetErrors(template).Any();
public abstract IEnumerable<string> GetWarnings(string template);
public bool HasWarnings(string template) => GetWarnings(template).Any();
protected static string[] GetFileErrors(string template)
{
// File name only; not path. all other path chars are valid enough to pass this check and will be handled on final save.
// null is invalid. whitespace is valid but not recommended
if (template is null)
return new[] { ERROR_NULL_IS_INVALID };
if (template.Contains(':')
|| template.Contains(Path.DirectorySeparatorChar)
|| template.Contains(Path.AltDirectorySeparatorChar)
)
return new[] { ERROR_INVALID_FILE_NAME_CHAR };
return Valid;
}
protected IEnumerable<string> GetStandardWarnings(string template)
{
var warnings = GetErrors(template).ToList();
if (template is null)
return warnings;
if (string.IsNullOrEmpty(template))
warnings.Add(WARNING_EMPTY);
else if (string.IsNullOrWhiteSpace(template))
warnings.Add(WARNING_WHITE_SPACE);
if (TagCount(template) == 0)
warnings.Add(WARNING_NO_TAGS);
if (!IsChapterized && ContainsChapterOnlyTags(template))
warnings.Add(WARNING_HAS_CHAPTER_TAGS);
return warnings;
}
internal int TagCount(string template)
=> GetTemplateTags()
// for <id><id> == 1, use:
// .Count(t => template.Contains($"<{t.TagName}>"))
// .Sum() impl: <id><id> == 2
.Sum(t => template.Split($"<{t.TagName}>").Length - 1);
internal static bool ContainsChapterOnlyTags(string template)
=> TemplateTags.GetAll()
.Where(t => t.IsChapterOnly)
.Any(t => ContainsTag(template, t.TagName));
internal static bool ContainsTag(string template, string tag) => template.Contains($"<{tag}>");
#endregion
#region to file name
/// <summary>
/// EditTemplateDialog: Get template generated filename for portion of path
/// </summary>
public string GetPortionFilename(LibraryBookDto libraryBookDto, string template)
=> string.IsNullOrWhiteSpace(template)
? ""
: getFileNamingTemplate(libraryBookDto, template, null, null)
.GetFilePath();
private static Regex ifSeriesRegex { get; } = new Regex("<if series->(.*?)<-if series>", RegexOptions.Compiled | RegexOptions.IgnoreCase);
internal static FileNamingTemplate getFileNamingTemplate(LibraryBookDto libraryBookDto, string template, string dirFullPath, string extension)
{
ArgumentValidator.EnsureNotNullOrWhiteSpace(template, nameof(template));
ArgumentValidator.EnsureNotNull(libraryBookDto, nameof(libraryBookDto));
dirFullPath = dirFullPath?.Trim() ?? "";
// for non-series, remove <if series-> and <-if series> tags and everything in between
// for series, remove <if series-> and <-if series> tags, what's in between will remain
template = ifSeriesRegex.Replace(
template,
string.IsNullOrWhiteSpace(libraryBookDto.SeriesName) ? "" : "$1");
var t = template + FileUtility.GetStandardizedExtension(extension);
var fullfilename = dirFullPath == "" ? t : Path.Combine(dirFullPath, t);
var fileNamingTemplate = new FileNamingTemplate(fullfilename) { IllegalCharacterReplacements = "_" };
var title = libraryBookDto.Title ?? "";
var titleShort = title.IndexOf(':') < 1 ? title : title.Substring(0, title.IndexOf(':'));
fileNamingTemplate.AddParameterReplacement(TemplateTags.Id, libraryBookDto.AudibleProductId);
fileNamingTemplate.AddParameterReplacement(TemplateTags.Title, title);
fileNamingTemplate.AddParameterReplacement(TemplateTags.TitleShort, titleShort);
fileNamingTemplate.AddParameterReplacement(TemplateTags.Author, libraryBookDto.AuthorNames);
fileNamingTemplate.AddParameterReplacement(TemplateTags.FirstAuthor, libraryBookDto.FirstAuthor);
fileNamingTemplate.AddParameterReplacement(TemplateTags.Narrator, libraryBookDto.NarratorNames);
fileNamingTemplate.AddParameterReplacement(TemplateTags.FirstNarrator, libraryBookDto.FirstNarrator);
fileNamingTemplate.AddParameterReplacement(TemplateTags.Series, libraryBookDto.SeriesName);
fileNamingTemplate.AddParameterReplacement(TemplateTags.SeriesNumber, libraryBookDto.SeriesNumber);
fileNamingTemplate.AddParameterReplacement(TemplateTags.Account, libraryBookDto.Account);
fileNamingTemplate.AddParameterReplacement(TemplateTags.Locale, libraryBookDto.Locale);
return fileNamingTemplate;
}
#endregion
public IEnumerable<TemplateTags> GetTemplateTags()
=> TemplateTags.GetAll()
// yeah, this line is a little funky but it works when you think through it. also: trust the unit tests
.Where(t => IsChapterized || !t.IsChapterOnly);
public string Sanitize(string template)
{
var value = template ?? "";
// don't use alt slash
value = value.Replace(Path.AltDirectorySeparatorChar, Path.DirectorySeparatorChar);
// don't allow double slashes
var sing = $"{Path.DirectorySeparatorChar}";
var dbl = $"{Path.DirectorySeparatorChar}{Path.DirectorySeparatorChar}";
while (value.Contains(dbl))
value = value.Replace(dbl, sing);
// trim. don't start or end with slash
while (true)
{
var start = value.Length;
value = value
.Trim()
.Trim(Path.DirectorySeparatorChar);
var end = value.Length;
if (start == end)
break;
}
return value;
}
public class FolderTemplate : Templates
{
public override string Name => "Folder Template";
public override string Description => Configuration.GetDescription(nameof(Configuration.FolderTemplate));
public override string DefaultTemplate { get; } = "<title short> [<id>]";
protected override bool IsChapterized { get; } = false;
internal FolderTemplate() : base() { }
#region validation
public override IEnumerable<string> GetErrors(string template)
{
// null is invalid. whitespace is valid but not recommended
if (template is null)
return new[] { ERROR_NULL_IS_INVALID };
// must be relative. no colons. all other path chars are valid enough to pass this check and will be handled on final save.
if (template.Contains(':'))
return new[] { ERROR_FULL_PATH_IS_INVALID };
return Valid;
}
public override IEnumerable<string> GetWarnings(string template) => GetStandardWarnings(template);
#endregion
#region to file name
/// <summary>USES LIVE CONFIGURATION VALUES</summary>
public string GetFilename(LibraryBookDto libraryBookDto)
=> getFileNamingTemplate(libraryBookDto, Configuration.Instance.FolderTemplate, AudibleFileStorage.BooksDirectory, null)
.GetFilePath();
#endregion
}
public class FileTemplate : Templates
{
public override string Name => "File Template";
public override string Description => Configuration.GetDescription(nameof(Configuration.FileTemplate));
public override string DefaultTemplate { get; } = "<title> [<id>]";
protected override bool IsChapterized { get; } = false;
internal FileTemplate() : base() { }
#region validation
public override IEnumerable<string> GetErrors(string template) => GetFileErrors(template);
public override IEnumerable<string> GetWarnings(string template) => GetStandardWarnings(template);
#endregion
#region to file name
/// <summary>USES LIVE CONFIGURATION VALUES</summary>
public string GetFilename(LibraryBookDto libraryBookDto, string dirFullPath, string extension)
=> getFileNamingTemplate(libraryBookDto, Configuration.Instance.FileTemplate, dirFullPath, extension)
.GetFilePath();
#endregion
}
public class ChapterFileTemplate : Templates
{
public override string Name => "Chapter File Template";
public override string Description => Configuration.GetDescription(nameof(Configuration.ChapterFileTemplate));
public override string DefaultTemplate { get; } = "<title> [<id>] - <ch# 0> - <ch title>";
protected override bool IsChapterized { get; } = true;
internal ChapterFileTemplate() : base() { }
#region validation
public override IEnumerable<string> GetErrors(string template) => GetFileErrors(template);
public override IEnumerable<string> GetWarnings(string template)
{
var warnings = GetStandardWarnings(template).ToList();
if (template is null)
return warnings;
// recommended to incl. <ch#> or <ch# 0>
if (!ContainsTag(template, TemplateTags.ChNumber.TagName) && !ContainsTag(template, TemplateTags.ChNumber0.TagName))
warnings.Add(WARNING_NO_CHAPTER_NUMBER_TAG);
return warnings;
}
#endregion
#region to file name
/// <summary>USES LIVE CONFIGURATION VALUES</summary>
public string GetFilename(LibraryBookDto libraryBookDto, AaxDecrypter.MultiConvertFileProperties props)
=> GetPortionFilename(libraryBookDto, Configuration.Instance.ChapterFileTemplate, props, AudibleFileStorage.DecryptInProgressDirectory);
public string GetPortionFilename(LibraryBookDto libraryBookDto, string template, AaxDecrypter.MultiConvertFileProperties props, string fullDirPath)
{
var fileNamingTemplate = getFileNamingTemplate(libraryBookDto, template, fullDirPath, Path.GetExtension(props.OutputFileName));
fileNamingTemplate.AddParameterReplacement(TemplateTags.ChCount, props.PartsTotal);
fileNamingTemplate.AddParameterReplacement(TemplateTags.ChNumber, props.PartsPosition);
fileNamingTemplate.AddParameterReplacement(TemplateTags.ChNumber0, FileUtility.GetSequenceFormatted(props.PartsPosition, props.PartsTotal));
fileNamingTemplate.AddParameterReplacement(TemplateTags.ChTitle, props.Title ?? "");
return fileNamingTemplate.GetFilePath();
}
#endregion
}
}
}

View File

@@ -0,0 +1,13 @@
using System;
using System.Collections.Generic;
using System.Linq;
using FileManager;
namespace LibationFileManager
{
public static class UtilityExtensions
{
public static void AddParameterReplacement(this FileNamingTemplate fileNamingTemplate, TemplateTags templateTags, object value)
=> fileNamingTemplate.AddParameterReplacement(templateTags.TagName, value);
}
}

View File

@@ -0,0 +1 @@
[assembly: System.Runtime.CompilerServices.InternalsVisibleTo(nameof(LibationFileManager) + ".Tests")]

View File

@@ -1,7 +1,7 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net5.0</TargetFramework>
<TargetFramework>net6.0-windows</TargetFramework>
</PropertyGroup>
<ItemGroup>
@@ -14,7 +14,6 @@
<ItemGroup>
<ProjectReference Include="..\DataLayer\DataLayer.csproj" />
<ProjectReference Include="..\FileManager\FileManager.csproj" />
</ItemGroup>
</Project>

View File

@@ -5,7 +5,7 @@ using System.Linq;
using System.Text.RegularExpressions;
using DataLayer;
using Dinah.Core;
using FileManager;
using LibationFileManager;
using Lucene.Net.Analysis.Standard;
using Lucene.Net.Documents;
using Lucene.Net.Index;

View File

@@ -0,0 +1 @@
[assembly: System.Runtime.CompilerServices.InternalsVisibleTo(nameof(LibationSearchEngine) + ".Tests")]

View File

@@ -1,6 +1,6 @@
using DataLayer;
using FileManager;
using System;
using System;
using DataLayer;
using LibationFileManager;
namespace LibationWinForms.BookLiberation
{

View File

@@ -2,6 +2,7 @@
using DataLayer;
using Dinah.Core.Net.Http;
using Dinah.Core.Threading;
using LibationFileManager;
using LibationWinForms.BookLiberation.BaseForms;
namespace LibationWinForms.BookLiberation
@@ -13,30 +14,25 @@ namespace LibationWinForms.BookLiberation
private Func<byte[]> GetCoverArtDelegate;
// book info
private string title;
private string authorNames;
private string narratorNames;
#region Processable event handler overrides
public override void Processable_Begin(object sender, LibraryBook libraryBook)
{
base.Processable_Begin(sender, libraryBook);
GetCoverArtDelegate = () => FileManager.PictureStorage.GetPictureSynchronously(
new FileManager.PictureDefinition(
GetCoverArtDelegate = () => PictureStorage.GetPictureSynchronously(
new PictureDefinition(
libraryBook.Book.PictureId,
FileManager.PictureSize._500x500));
PictureSize._500x500));
//Set default values from library
AudioDecodable_TitleDiscovered(sender, libraryBook.Book.Title);
AudioDecodable_AuthorsDiscovered(sender, string.Join(", ", libraryBook.Book.Authors));
AudioDecodable_NarratorsDiscovered(sender, string.Join(", ", libraryBook.Book.NarratorNames));
AudioDecodable_AuthorsDiscovered(sender, libraryBook.Book.AuthorNames);
AudioDecodable_NarratorsDiscovered(sender, libraryBook.Book.NarratorNames);
AudioDecodable_CoverImageDiscovered(sender,
FileManager.PictureStorage.GetPicture(
new FileManager.PictureDefinition(
PictureStorage.GetPicture(
new PictureDefinition(
libraryBook.Book.PictureId,
FileManager.PictureSize._80x80)).bytes);
PictureSize._80x80)).bytes);
}
#endregion
@@ -59,14 +55,23 @@ namespace LibationWinForms.BookLiberation
updateRemainingTime((int)timeRemaining.TotalSeconds);
}
private void updateRemainingTime(int remaining)
=> remainingTimeLbl.UIThreadAsync(() => remainingTimeLbl.Text = $"ETA:\r\n{formatTime(remaining)}");
private string formatTime(int seconds)
{
var timeSpan = new TimeSpan(0, 0, seconds);
return
timeSpan.TotalHours >= 1 ? $"{timeSpan:%h}h {timeSpan:mm}m {timeSpan:ss}s"
: timeSpan.TotalMinutes >= 1 ? $"{timeSpan:%m}m {timeSpan:ss}s"
: $"{seconds} sec";
}
#endregion
#region AudioDecodable event handlers
public override void AudioDecodable_RequestCoverArt(object sender, Action<byte[]> setCoverArtDelegate)
{
base.AudioDecodable_RequestCoverArt(sender, setCoverArtDelegate);
setCoverArtDelegate(GetCoverArtDelegate?.Invoke());
}
private string title;
private string authorNames;
private string narratorNames;
public override void AudioDecodable_TitleDiscovered(object sender, string title)
{
@@ -90,18 +95,20 @@ namespace LibationWinForms.BookLiberation
updateBookInfo();
}
private void updateBookInfo()
=> bookInfoLbl.UIThreadAsync(() => bookInfoLbl.Text = $"{title}\r\nBy {authorNames}\r\nNarrated by {narratorNames}");
public override void AudioDecodable_RequestCoverArt(object sender, Action<byte[]> setCoverArtDelegate)
{
base.AudioDecodable_RequestCoverArt(sender, setCoverArtDelegate);
setCoverArtDelegate(GetCoverArtDelegate?.Invoke());
}
public override void AudioDecodable_CoverImageDiscovered(object sender, byte[] coverArt)
{
base.AudioDecodable_CoverImageDiscovered(sender, coverArt);
pictureBox1.UIThreadAsync(() => pictureBox1.Image = Dinah.Core.Drawing.ImageReader.ToImage(coverArt));
}
#endregion
// thread-safe UI updates
private void updateBookInfo()
=> bookInfoLbl.UIThreadAsync(() => bookInfoLbl.Text = $"{title}\r\nBy {authorNames}\r\nNarrated by {narratorNames}");
private void updateRemainingTime(int remaining)
=> remainingTimeLbl.UIThreadAsync(() => remainingTimeLbl.Text = $"ETA:\r\n{remaining} sec");
}
}

View File

@@ -1,6 +1,6 @@
using DataLayer;
using FileManager;
using System;
using System;
using DataLayer;
using LibationFileManager;
namespace LibationWinForms.BookLiberation
{

View File

@@ -152,11 +152,12 @@ namespace LibationWinForms.BookLiberation.BaseForms
#endregion
#region AudioDecodable event handlers
public virtual void AudioDecodable_RequestCoverArt(object sender, Action<byte[]> setCoverArtDelegate) { }
public virtual void AudioDecodable_TitleDiscovered(object sender, string title) { }
public virtual void AudioDecodable_AuthorsDiscovered(object sender, string authors) { }
public virtual void AudioDecodable_NarratorsDiscovered(object sender, string narrators) { }
public virtual void AudioDecodable_CoverImageDiscovered(object sender, byte[] coverArt) { }
public virtual void AudioDecodable_RequestCoverArt(object sender, Action<byte[]> setCoverArtDelegate) { }
#endregion
}
}

View File

@@ -1,11 +1,11 @@
using DataLayer;
using Dinah.Core;
using FileLiberator;
using LibationWinForms.BookLiberation.BaseForms;
using System;
using System.Linq;
using System;
using System.Threading.Tasks;
using System.Windows.Forms;
using DataLayer;
using Dinah.Core;
using FileLiberator;
using LibationFileManager;
using LibationWinForms.BookLiberation.BaseForms;
namespace LibationWinForms.BookLiberation
{
@@ -218,12 +218,12 @@ namespace LibationWinForms.BookLiberation
{
LogMe.Error("ERROR. All books have not been processed. Most recent book: processing failed");
DialogResult? dialogResult = FileManager.Configuration.Instance.BadBook switch
DialogResult? dialogResult = Configuration.Instance.BadBook switch
{
FileManager.Configuration.BadBookAction.Abort => DialogResult.Abort,
FileManager.Configuration.BadBookAction.Retry => DialogResult.Retry,
FileManager.Configuration.BadBookAction.Ignore => DialogResult.Ignore,
FileManager.Configuration.BadBookAction.Ask => null,
Configuration.BadBookAction.Abort => DialogResult.Abort,
Configuration.BadBookAction.Retry => DialogResult.Retry,
Configuration.BadBookAction.Ignore => DialogResult.Ignore,
Configuration.BadBookAction.Ask => null,
_ => null
};

View File

@@ -1,9 +1,9 @@
using AudibleApi;
using InternalUtilities;
using System;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Windows.Forms;
using AudibleApi;
using AudibleUtilities;
namespace LibationWinForms.Dialogs
{
@@ -112,6 +112,9 @@ namespace LibationWinForms.Dialogs
{
try
{
if (!inputIsValid())
return;
// without transaction, accounts persister will write ANY EDIT immediately to file
using var persister = AudibleApiStorage.GetAccountsSettingsPersister();
@@ -129,6 +132,28 @@ namespace LibationWinForms.Dialogs
}
}
private bool inputIsValid()
{
var dtos = getRowDtos();
foreach (var dto in dtos)
{
if (string.IsNullOrWhiteSpace(dto.AccountId))
{
MessageBox.Show("Account id cannot be blank. Please enter an account id for all accounts.", "Blank account", MessageBoxButtons.OK, MessageBoxIcon.Error);
return false;
}
if (string.IsNullOrWhiteSpace(dto.LocaleName))
{
MessageBox.Show("Please select a locale (i.e.: country or region) for all accounts.", "Blank region", MessageBoxButtons.OK, MessageBoxIcon.Error);
return false;
}
}
return true;
}
private void persist(AccountsSettings accountsSettings)
{
var existingAccounts = accountsSettings.Accounts;
@@ -152,11 +177,6 @@ namespace LibationWinForms.Dialogs
// upsert each. validation occurs through Account and AccountsSettings
foreach (var dto in dtos)
{
if (string.IsNullOrWhiteSpace(dto.AccountId))
throw new Exception("Please enter an account id for all accounts");
if (string.IsNullOrWhiteSpace(dto.LocaleName))
throw new Exception("Please select a locale (i.e.: country or region) for all accounts");
var acct = accountsSettings.Upsert(dto.AccountId, dto.LocaleName);
acct.LibraryScan = dto.LibraryScan;
acct.AccountName

View File

@@ -4,6 +4,7 @@ using System.Linq;
using System.Windows.Forms;
using DataLayer;
using Dinah.Core;
using LibationFileManager;
namespace LibationWinForms.Dialogs
{
@@ -39,7 +40,7 @@ namespace LibationWinForms.Dialogs
{
this.Text = Book.Title;
(var isDefault, var picture) = FileManager.PictureStorage.GetPicture(new FileManager.PictureDefinition(Book.PictureId, FileManager.PictureSize._80x80));
(_, var picture) = PictureStorage.GetPicture(new PictureDefinition(Book.PictureId, PictureSize._80x80));
this.coverPb.Image = Dinah.Core.Drawing.ImageReader.ToImage(picture);
var t = @$"

View File

@@ -1,8 +1,8 @@
using FileManager;
using System;
using System;
using System.Linq;
using System.Collections.Generic;
using System.Windows.Forms;
using LibationFileManager;
namespace LibationWinForms.Dialogs
{

View File

@@ -1,9 +1,9 @@
using Dinah.Core;
using FileManager;
using System;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Windows.Forms;
using Dinah.Core;
using LibationFileManager;
namespace LibationWinForms.Dialogs
{

View File

@@ -1,7 +1,7 @@
using FileManager;
using System;
using System;
using System.Linq;
using System.Windows.Forms;
using LibationFileManager;
namespace LibationWinForms.Dialogs
{

View File

@@ -0,0 +1,197 @@
namespace LibationWinForms.Dialogs
{
partial class EditTemplateDialog
{
/// <summary>
/// Required designer variable.
/// </summary>
private System.ComponentModel.IContainer components = null;
/// <summary>
/// Clean up any resources being used.
/// </summary>
/// <param name="disposing">true if managed resources should be disposed; otherwise, false.</param>
protected override void Dispose(bool disposing)
{
if (disposing && (components != null))
{
components.Dispose();
}
base.Dispose(disposing);
}
#region Windows Form Designer generated code
/// <summary>
/// Required method for Designer support - do not modify
/// the contents of this method with the code editor.
/// </summary>
private void InitializeComponent()
{
this.saveBtn = new System.Windows.Forms.Button();
this.cancelBtn = new System.Windows.Forms.Button();
this.templateTb = new System.Windows.Forms.TextBox();
this.templateLbl = new System.Windows.Forms.Label();
this.resetToDefaultBtn = new System.Windows.Forms.Button();
this.listView1 = new System.Windows.Forms.ListView();
this.columnHeader1 = new System.Windows.Forms.ColumnHeader();
this.columnHeader2 = new System.Windows.Forms.ColumnHeader();
this.richTextBox1 = new System.Windows.Forms.RichTextBox();
this.warningsLbl = new System.Windows.Forms.Label();
this.exampleLbl = new System.Windows.Forms.Label();
this.SuspendLayout();
//
// saveBtn
//
this.saveBtn.Anchor = ((System.Windows.Forms.AnchorStyles)((System.Windows.Forms.AnchorStyles.Bottom | System.Windows.Forms.AnchorStyles.Right)));
this.saveBtn.Location = new System.Drawing.Point(714, 345);
this.saveBtn.Margin = new System.Windows.Forms.Padding(4, 3, 4, 3);
this.saveBtn.Name = "saveBtn";
this.saveBtn.Size = new System.Drawing.Size(88, 27);
this.saveBtn.TabIndex = 98;
this.saveBtn.Text = "Save";
this.saveBtn.UseVisualStyleBackColor = true;
this.saveBtn.Click += new System.EventHandler(this.saveBtn_Click);
//
// cancelBtn
//
this.cancelBtn.Anchor = ((System.Windows.Forms.AnchorStyles)((System.Windows.Forms.AnchorStyles.Bottom | System.Windows.Forms.AnchorStyles.Right)));
this.cancelBtn.DialogResult = System.Windows.Forms.DialogResult.Cancel;
this.cancelBtn.Location = new System.Drawing.Point(832, 345);
this.cancelBtn.Margin = new System.Windows.Forms.Padding(4, 3, 4, 3);
this.cancelBtn.Name = "cancelBtn";
this.cancelBtn.Size = new System.Drawing.Size(88, 27);
this.cancelBtn.TabIndex = 99;
this.cancelBtn.Text = "Cancel";
this.cancelBtn.UseVisualStyleBackColor = true;
this.cancelBtn.Click += new System.EventHandler(this.cancelBtn_Click);
//
// templateTb
//
this.templateTb.Anchor = ((System.Windows.Forms.AnchorStyles)(((System.Windows.Forms.AnchorStyles.Top | System.Windows.Forms.AnchorStyles.Left)
| System.Windows.Forms.AnchorStyles.Right)));
this.templateTb.Location = new System.Drawing.Point(12, 27);
this.templateTb.Name = "templateTb";
this.templateTb.Size = new System.Drawing.Size(779, 23);
this.templateTb.TabIndex = 1;
this.templateTb.TextChanged += new System.EventHandler(this.templateTb_TextChanged);
//
// templateLbl
//
this.templateLbl.AutoSize = true;
this.templateLbl.Location = new System.Drawing.Point(12, 9);
this.templateLbl.Name = "templateLbl";
this.templateLbl.Size = new System.Drawing.Size(89, 15);
this.templateLbl.TabIndex = 0;
this.templateLbl.Text = "[template desc]";
//
// resetToDefaultBtn
//
this.resetToDefaultBtn.Anchor = ((System.Windows.Forms.AnchorStyles)((System.Windows.Forms.AnchorStyles.Top | System.Windows.Forms.AnchorStyles.Right)));
this.resetToDefaultBtn.Location = new System.Drawing.Point(797, 26);
this.resetToDefaultBtn.Name = "resetToDefaultBtn";
this.resetToDefaultBtn.Size = new System.Drawing.Size(124, 23);
this.resetToDefaultBtn.TabIndex = 2;
this.resetToDefaultBtn.Text = "Reset to default";
this.resetToDefaultBtn.UseVisualStyleBackColor = true;
this.resetToDefaultBtn.Click += new System.EventHandler(this.resetToDefaultBtn_Click);
//
// listView1
//
this.listView1.Anchor = ((System.Windows.Forms.AnchorStyles)(((System.Windows.Forms.AnchorStyles.Top | System.Windows.Forms.AnchorStyles.Bottom)
| System.Windows.Forms.AnchorStyles.Left)));
this.listView1.Columns.AddRange(new System.Windows.Forms.ColumnHeader[] {
this.columnHeader1,
this.columnHeader2});
this.listView1.HideSelection = false;
this.listView1.Location = new System.Drawing.Point(12, 56);
this.listView1.Name = "listView1";
this.listView1.Size = new System.Drawing.Size(328, 283);
this.listView1.TabIndex = 3;
this.listView1.UseCompatibleStateImageBehavior = false;
this.listView1.View = System.Windows.Forms.View.Details;
//
// columnHeader1
//
this.columnHeader1.Text = "Tag";
this.columnHeader1.Width = 137;
//
// columnHeader2
//
this.columnHeader2.Text = "Description";
this.columnHeader2.Width = 170;
//
// richTextBox1
//
this.richTextBox1.Anchor = ((System.Windows.Forms.AnchorStyles)((((System.Windows.Forms.AnchorStyles.Top | System.Windows.Forms.AnchorStyles.Bottom)
| System.Windows.Forms.AnchorStyles.Left)
| System.Windows.Forms.AnchorStyles.Right)));
this.richTextBox1.BorderStyle = System.Windows.Forms.BorderStyle.FixedSingle;
this.richTextBox1.Location = new System.Drawing.Point(346, 74);
this.richTextBox1.Name = "richTextBox1";
this.richTextBox1.ReadOnly = true;
this.richTextBox1.Size = new System.Drawing.Size(574, 185);
this.richTextBox1.TabIndex = 5;
this.richTextBox1.Text = "";
//
// warningsLbl
//
this.warningsLbl.Anchor = ((System.Windows.Forms.AnchorStyles)((System.Windows.Forms.AnchorStyles.Bottom | System.Windows.Forms.AnchorStyles.Left)));
this.warningsLbl.Font = new System.Drawing.Font("Segoe UI", 9F, System.Drawing.FontStyle.Bold, System.Drawing.GraphicsUnit.Point);
this.warningsLbl.ForeColor = System.Drawing.Color.Firebrick;
this.warningsLbl.Location = new System.Drawing.Point(346, 262);
this.warningsLbl.Name = "warningsLbl";
this.warningsLbl.Size = new System.Drawing.Size(574, 77);
this.warningsLbl.TabIndex = 6;
this.warningsLbl.Text = "[warnings]";
//
// exampleLbl
//
this.exampleLbl.AutoSize = true;
this.exampleLbl.Location = new System.Drawing.Point(346, 56);
this.exampleLbl.Name = "exampleLbl";
this.exampleLbl.Size = new System.Drawing.Size(55, 15);
this.exampleLbl.TabIndex = 4;
this.exampleLbl.Text = "Example:";
//
// EditTemplateDialog
//
this.AcceptButton = this.saveBtn;
this.AutoScaleDimensions = new System.Drawing.SizeF(7F, 15F);
this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font;
this.CancelButton = this.cancelBtn;
this.ClientSize = new System.Drawing.Size(933, 388);
this.Controls.Add(this.exampleLbl);
this.Controls.Add(this.warningsLbl);
this.Controls.Add(this.richTextBox1);
this.Controls.Add(this.listView1);
this.Controls.Add(this.resetToDefaultBtn);
this.Controls.Add(this.templateLbl);
this.Controls.Add(this.templateTb);
this.Controls.Add(this.cancelBtn);
this.Controls.Add(this.saveBtn);
this.FormBorderStyle = System.Windows.Forms.FormBorderStyle.FixedToolWindow;
this.Margin = new System.Windows.Forms.Padding(4, 3, 4, 3);
this.Name = "EditTemplateDialog";
this.StartPosition = System.Windows.Forms.FormStartPosition.CenterParent;
this.Text = "Edit Template";
this.Load += new System.EventHandler(this.EditTemplateDialog_Load);
this.ResumeLayout(false);
this.PerformLayout();
}
#endregion
private System.Windows.Forms.Button saveBtn;
private System.Windows.Forms.Button cancelBtn;
private System.Windows.Forms.TextBox templateTb;
private System.Windows.Forms.Label templateLbl;
private System.Windows.Forms.Button resetToDefaultBtn;
private System.Windows.Forms.ListView listView1;
private System.Windows.Forms.ColumnHeader columnHeader1;
private System.Windows.Forms.ColumnHeader columnHeader2;
private System.Windows.Forms.RichTextBox richTextBox1;
private System.Windows.Forms.Label warningsLbl;
private System.Windows.Forms.Label exampleLbl;
}
}

View File

@@ -0,0 +1,173 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.IO;
using System.Windows.Forms;
using Dinah.Core;
using LibationFileManager;
namespace LibationWinForms.Dialogs
{
public partial class EditTemplateDialog : Form
{
// final value. post-validity check
public string TemplateText { get; private set; }
// hold the work-in-progress value. not guaranteed to be valid
private string _workingTemplateText;
private string workingTemplateText
{
get => _workingTemplateText;
set => _workingTemplateText = template.Sanitize(value);
}
private void resetTextBox(string value) => this.templateTb.Text = workingTemplateText = value;
private Configuration config { get; } = Configuration.Instance;
private Templates template { get; }
private string inputTemplateText { get; }
public EditTemplateDialog() => InitializeComponent();
public EditTemplateDialog(Templates template, string inputTemplateText) : this()
{
this.template = ArgumentValidator.EnsureNotNull(template, nameof(template));
this.inputTemplateText = inputTemplateText ?? "";
}
private void EditTemplateDialog_Load(object sender, EventArgs e)
{
if (this.DesignMode)
return;
if (template is null)
{
MessageBoxAlertAdmin.Show($"Programming error. {nameof(EditTemplateDialog)} was not created correctly", "Edit template error", new NullReferenceException($"{nameof(template)} is null"));
return;
}
warningsLbl.Text = "";
this.Text = $"Edit {template.Name}";
this.templateLbl.Text = template.Description;
resetTextBox(inputTemplateText);
// populate list view
foreach (var tag in template.GetTemplateTags())
listView1.Items.Add(new ListViewItem(new[] { $"<{tag.TagName}>", tag.Description }));
}
private void resetToDefaultBtn_Click(object sender, EventArgs e) => resetTextBox(template.DefaultTemplate);
private void templateTb_TextChanged(object sender, EventArgs e)
{
workingTemplateText = templateTb.Text;
var isFolder = template == Templates.Folder;
var libraryBookDto = new LibraryBookDto
{
Account = "my account",
AudibleProductId = "123456789",
Title = "A Study in Scarlet: A Sherlock Holmes Novel",
Locale = "us",
Authors = new List<string> { "Arthur Conan Doyle", "Stephen Fry - introductions" },
Narrators = new List<string> { "Stephen Fry" },
SeriesName = "Sherlock Holmes",
SeriesNumber = "1"
};
var chapterName = "A Flight for Life";
var chapterNumber = 4;
var chaptersTotal = 10;
var books = config.Books;
var folder = Templates.Folder.GetPortionFilename(
libraryBookDto,
isFolder ? workingTemplateText : config.FolderTemplate);
var file
= template == Templates.ChapterFile
? Templates.ChapterFile.GetPortionFilename(
libraryBookDto,
workingTemplateText,
new() { OutputFileName = "", PartsPosition = chapterNumber, PartsTotal = chaptersTotal, Title = chapterName },
"")
: Templates.File.GetPortionFilename(
libraryBookDto,
isFolder ? config.FileTemplate : workingTemplateText);
var ext = config.DecryptToLossy ? "mp3" : "m4b";
const char ZERO_WIDTH_SPACE = '\u200B';
var sing = $"{Path.DirectorySeparatorChar}";
// result: can wrap long paths. eg:
// |-- LINE WRAP BOUNDARIES --|
// \books\author with a very <= normal line break on space between words
// long name\narrator narrator
// \title <= line break on the zero-with space we added before slashes
string slashWrap(string val) => val.Replace(sing, $"{ZERO_WIDTH_SPACE}{sing}");
warningsLbl.Text
= !template.HasWarnings(workingTemplateText)
? ""
: "Warning:\r\n" +
template
.GetWarnings(workingTemplateText)
.Select(err => $"- {err}")
.Aggregate((a, b) => $"{a}\r\n{b}");
var bold = new System.Drawing.Font(richTextBox1.Font, System.Drawing.FontStyle.Bold);
var reg = new System.Drawing.Font(richTextBox1.Font, System.Drawing.FontStyle.Regular);
richTextBox1.Clear();
richTextBox1.SelectionFont = reg;
richTextBox1.AppendText(slashWrap(books));
richTextBox1.AppendText(sing);
if (isFolder)
richTextBox1.SelectionFont = bold;
richTextBox1.AppendText(slashWrap(folder));
if (isFolder)
richTextBox1.SelectionFont = reg;
richTextBox1.AppendText(sing);
if (!isFolder)
richTextBox1.SelectionFont = bold;
richTextBox1.AppendText(file);
if (!isFolder)
richTextBox1.SelectionFont = reg;
richTextBox1.AppendText($".{ext}");
}
private void saveBtn_Click(object sender, EventArgs e)
{
if (!template.IsValid(workingTemplateText))
{
var errors = template
.GetErrors(workingTemplateText)
.Select(err => $"- {err}")
.Aggregate((a, b) => $"{a}\r\n{b}");
MessageBox.Show($"This template text is not valid. Errors:\r\n{errors}", "Invalid", MessageBoxButtons.OK, MessageBoxIcon.Error);
return;
}
TemplateText = workingTemplateText;
this.DialogResult = DialogResult.OK;
this.Close();
}
private void cancelBtn_Click(object sender, EventArgs e)
{
this.DialogResult = DialogResult.Cancel;
this.Close();
}
}
}

View File

@@ -0,0 +1,60 @@
<root>
<xsd:schema id="root" xmlns="" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata">
<xsd:import namespace="http://www.w3.org/XML/1998/namespace" />
<xsd:element name="root" msdata:IsDataSet="true">
<xsd:complexType>
<xsd:choice maxOccurs="unbounded">
<xsd:element name="metadata">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="value" type="xsd:string" minOccurs="0" />
</xsd:sequence>
<xsd:attribute name="name" use="required" type="xsd:string" />
<xsd:attribute name="type" type="xsd:string" />
<xsd:attribute name="mimetype" type="xsd:string" />
<xsd:attribute ref="xml:space" />
</xsd:complexType>
</xsd:element>
<xsd:element name="assembly">
<xsd:complexType>
<xsd:attribute name="alias" type="xsd:string" />
<xsd:attribute name="name" type="xsd:string" />
</xsd:complexType>
</xsd:element>
<xsd:element name="data">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
<xsd:element name="comment" type="xsd:string" minOccurs="0" msdata:Ordinal="2" />
</xsd:sequence>
<xsd:attribute name="name" type="xsd:string" use="required" msdata:Ordinal="1" />
<xsd:attribute name="type" type="xsd:string" msdata:Ordinal="3" />
<xsd:attribute name="mimetype" type="xsd:string" msdata:Ordinal="4" />
<xsd:attribute ref="xml:space" />
</xsd:complexType>
</xsd:element>
<xsd:element name="resheader">
<xsd:complexType>
<xsd:sequence>
<xsd:element name="value" type="xsd:string" minOccurs="0" msdata:Ordinal="1" />
</xsd:sequence>
<xsd:attribute name="name" type="xsd:string" use="required" />
</xsd:complexType>
</xsd:element>
</xsd:choice>
</xsd:complexType>
</xsd:element>
</xsd:schema>
<resheader name="resmimetype">
<value>text/microsoft-resx</value>
</resheader>
<resheader name="version">
<value>2.0</value>
</resheader>
<resheader name="reader">
<value>System.Resources.ResXResourceReader, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
</resheader>
<resheader name="writer">
<value>System.Resources.ResXResourceWriter, System.Windows.Forms, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</value>
</resheader>
</root>

Some files were not shown because too many files have changed in this diff Show More