Parsing HTML belongs before the data reaches the data layer, not in it. Move the
flattening to AudibleUtilities, where the rest of the Item translation lives, so
Book.Description holds plain text and DataLayer no longer references
HtmlAgilityPack.
Import now updates a description as well as creating one, which it never did
before: books imported while descriptions were stored as raw HTML are still
holding markup, and a re-scan is what clears it. A scan that reports no summary
leaves the stored one alone.
This also cleans up the CSV/Excel export, which was emitting raw HTML too.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Libation never had a copyright of its own to fall back on: it only cleaned up
whatever the .aaxc file already embedded. Widevine downloads carry no cprt at
all, so they came out blank.
The copyright line is only returned by the product_details response group, so
request it on the full library scan, store it on Book, and use it as the tag
fallback. Existing libraries pick it up on their next scan, and Audible still
returns null for plenty of titles.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Audible's .aaxc files often ship 01-Jan-2000 in rldt and 2000 in the year atom
instead of the real release date, and the file's tag beat the catalog's date.
Override that one value when the catalog knows better, leaving every other
embedded date alone; the comment records the alternatives considered.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Book.Description is publisher_summary verbatim, which is HTML. Audible's .aaxc
files embed an already-flattened copy, so the fallback only fired on Widevine
downloads - whose DASH files carry no description tags at all - and wrote raw
<p> markup into the file's comment and long-description atoms.
Move the grid's existing HtmlAgilityPack flattening into a shared helper that
also de-entitizes, and use it for the tag fallback. Extract the tag-filling body
of the RetrievedMetadata handler so it can be exercised directly by a test.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Reported in issue #1973: a scheduled liberate run re-requested a content license
for the same 59 titles every 15 minutes, 1397 refused requests in six hours,
because nothing about a failed PDF was remembered and the PDF step asked Audible
afresh every time.
The two download paths have been converging for a while - a PDF is now named and
placed by the audiobook path's own logic, and verified like one - and every part
of this bug lives where that stopped short.
Both steps ask Audible for the same license. The request asks for pdf_url
alongside the content reference, and LicenseInfo dropped it, so DownloadPdf
turned round and requested an identical license to read the field the first
response had already returned. Carry PdfUrl on LicenseInfo and give both steps
one ILicensedDownload contract: a license may be supplied to a step, and the one
a step ended up using is published for the next step for the same title. The CLI
and the GUI queue hand it on, so a title costs one license request per run
however many steps want something from it. A carried license is retried once
with a fresh one if it does not work, since Audible's links are signed and a long
decrypt can run between the two steps.
Where the audiobook step obtained no license there is nothing to hand on and the
supplement step does not run, which deletes a bug rather than guarding it:
Completed fires from a finally, so a refused audio download was followed at once
by a PDF request that reproduced the refusal.
Error now means the same for a PDF as for a book. The audiobook step has always
skipped LiberatedStatus.Error through AudioExists, and NeedsPdfDownload agrees,
but DownloadPdf selected on PdfExists and so retried an errored PDF forever. A
license that is granted and carries no pdf_url - the 'No PDF URL available' in
the report - is Audible saying the title has no PDF, and is written off that same
way instead of failing identically on every run. It stays resettable by
everything that resets a book: --force, a named title, Set PDF Not Downloaded.
Refusals now reach ProcessSingleAsync, which has always recorded them for
whichever step throws one; DownloadPdf swallowed everything and recorded nothing.
It keeps swallowing what the classifier does not recognise, which is what stopped
a missing PDF from taking the app down with it.
A bulk CLI run leaves alone the titles the last scan did not find, by the same
Downloadable rule every multi-title path in the app already uses, and the PDF
back-fill pass waits on a refused title just as the first pass does. --force and
a named title still attempt everything.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
A storefront that no longer lists a title still answers a request for it: HTTP 200,
total_results 1, and a product carrying an asin and a few always-returned flags.
Nothing between the request and the file noticed, so a re-download replaced a
metadata file written while the title was still listed - the only copy of that data -
with the placeholder.
Fetch the product before touching the destination, and leave the file alone when the
product carries no title. Reported in issue #1947, where a Canada-only title produced
{"asin":...,"asset_details":[],"is_preview_enabled":false,"is_vvab":false,"rating":{...}}
against every other storefront.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
A license denial left no trace: BookStatus stayed NotLiberated, so every
liberate run asked again. Only the GUI's bad-book dialog could mark a title
Error, and license denials take their own path and never reach that dialog, so
a headless install had no way at all to stop the retries. A cron schedule then
re-requested the same refused licenses every run and printed the same warning
block for each one, which is both wasted API traffic and the log noise reported
in issue #1947.
Record the refusal instead, with a wait that doubles per consecutive failure:
one day for an eligibility denial (up to 30), six hours for a title Audible has
no audio for yet such as an unreleased preorder (up to 7 days), one hour when
the denial names GenericError, which the GUI already reads as an outage
(up to 12). Nothing is permanent - every kind is attempted again on its own.
Only failures attributable to Audible are recorded. A dropped connection, a
decrypt error or a full disk keeps being retried on the next run as before.
Naming a title, --force, and setting a download status all clear the record:
asking for a title explicitly overrides the wait.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Records every successful audiobook download in the library database (a new
DownloadHistory table) and, when the user opts in, stops downloading once the
rolling 24 hour window is full.
The history lives in the database rather than a file under LibationFiles
because in Docker only the database is on a volume; a file there is discarded
on every container restart.
The limit is checked immediately before each book downloads, never at queueing
time, so a full queue stays full and the user can raise or disable the limit
mid-run. When nothing in the queue can proceed the queue pauses and re-checks
every 15 seconds, recomputing settings, history and clock from scratch, so a
queue left running for days drip-feeds itself as downloads age out. The CLI
never waits: it skips covered titles and reports a count.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
All processables are now created with an instance of Configuration, and they use that instance's settings.
Added Configuration.CreateEphemeralCopy() to clone Configuration without persistence.
- Remove instance queue. This is a database, after all, and is designed to be accessed and written to concurrently
- Reduce the number of calls to DbContexts.Create()
- Ensure that no LibationContext remains open across an await boundary. Multithread context access is the most likely culprit for past issues.
- Make all Update UserDefinedItem methods asynchronous.
- Add `LIBATION_FILES_DIR` environment variable to specify LibationFiles directory instead of appsettings.json
- OptionsBase supports overriding setting
- Added `EphemeralSettings` which are loaded from Settings.json once and can be modified with the `--override` command parameter
- Added `get-setting` command
- Prints (editable) settings and their values. Prints specified settings, or all settings if none specified
- `--listEnumValues` option will list all names for a speficied enum-type settings. If no setting names are specified, prints all enum values for all enum settings.
- Prints in a text-based table or bare with `-b` switch
- Added `get-license` command which requests a content license and prints it as a json to stdout
- Improved `liberate` command
- Added `-force` option to force liberation without validation.
- Added support to download with a license file supplied to stdin
- Improve startup performance when downloading explicit ASIN(s)
- Fix long-standing bug where cover art was not being downloading
- Add Book.IsSpatial property and add it to search index
- Read audio format of actual output files and store it in UserDefinedItem. Now works with MP3s.
- Store last downloaded audio file version
- Add IsSpatial, file version, and Audio Format to library exports and to template tags. Updated docs.
- Add last downloaded audio file version and format info to the Last Downloaded tab
- Migrated the DB
- Update AAXClean with some bug fixes
- Fixed error converting xHE-AAC audio files to mp3 when splitting by chapter (or trimming the audible branding from the beginning of the file)
- Improve mp3 ID# tags support. Chapter titles are now preserved.
- Add support for reading EC-3 and AC-4 audio format metadata
- Add more null safety
- Fix possible FilePathCache race condition
- Add MoveFilesToBooksDir progress reporting
- All metadata is now downloaded in parallel with other post-success tasks.
- Improve download resuming and file cleanup reliability
- The downloader creates temp files with a UUID filename and does not insert them into the FilePathCache. Created files only receive their final file names when they are moved into the Books folder. This is to prepare for a future plan re naming templates
- Add template tag support for multiple series
- Add series ID and contributor ID to template tags
- <first author> and <first narrator> are now name types with name formatter support
- Properly import contributor IDs into database
- Updated docs
