File.WriteAllText truncates the destination before writing, so an interrupted
write leaves a half-written or empty Settings.json, and the in-process lock added
in the previous commit cannot help a reader in another process - the GUI and the
CLI share this file.
Route every write through Dinah.Core.IO.AtomicFileWriter, which writes a sibling
temp file, flushes to disk and renames it over the destination. Validate the temp
file parses as json before the swap, the same way JsonFilePersister<T> already
saves AccountsSettings.json, so a bad payload leaves the existing file untouched.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Configuration.Instance is a process-wide singleton read and written from the UI
thread, BackgroundWorker callbacks and download workers at the same time, but
PersistentDictionary guarded only its file writes. Its two Dictionary caches were
touched without synchronization, so concurrent inserts eventually corrupted them
and threw "Operations that change non-concurrent collections must have exclusive
access" (issue #1959, reported from MainVM.UpdateCountsBw_Completed reading
AutoDownloadEpisodes while a second GetCounts pass ran).
Reads on the file were unguarded too: Exists/GetJObject could observe a partially
written Settings.json, and readFile responds to empty contents by rewriting the
file, so a reader racing a writer could scramble the settings on disk.
Take one lock across each operation's cache and file access, and keep logging
outside it by having writeFile report whether it rewrote the file.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
'Set Download status to Not Downloaded' moves both statuses together and was the only
way to reset a PDF, so a user who wanted a PDF re-fetched also queued the audiobook
for a fresh download - which then rewrote that title's other files. Reported in
issue #1947.
Add a PDF-only pair beside it, shown only for a selection that has a PDF, since for
anything else the existing pair is already audio-only.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
That the file existed said only that the server sent a body, and an Audible error is
a 200 with a JSON body like any other response. Dinah's downloader renames by
Content-Disposition, so such a body landed in the book's folder under whatever
Audible called it and the title was recorded as having its PDF.
Check the payload: a file named .pdf must carry the PDF header, and nothing may begin
with the opening character of a JSON or markup document. A rejected download is
deleted rather than left in the library, which also lets the empty-folder cleanup
run, and the file is added to the path cache only once it has passed.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
A storefront that no longer lists a title still answers a request for it: HTTP 200,
total_results 1, and a product carrying an asin and a few always-returned flags.
Nothing between the request and the file noticed, so a re-download replaced a
metadata file written while the title was still listed - the only copy of that data -
with the placeholder.
Fetch the product before touching the destination, and leave the file alone when the
product carries no title. Reported in issue #1947, where a Canada-only title produced
{"asin":...,"asset_details":[],"is_preview_enabled":false,"is_vvab":false,"rating":{...}}
against every other storefront.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
<title short> stops at the first colon, so it shortens Audible titles that
contain one just as readily as it drops Audible's subtitle, and distinct books
then collapse onto the same name. A colon cannot be searched for: the analyzer
discards punctuation and Lucene reads a colon in a query as a field separator.
Two bool index fields find the affected books instead.
Document how the two title tags differ, since <audible title> already drops
Audible's subtitle without ever cutting a title, and how to audit for names
that actually collide in a spreadsheet export.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
The dialog names the account in full because it is shown to whoever owns it, but
log files get attached to public issue reports, which is why the codebase has
MaskedLogEntry. Naming the account in the log the same way the dialog does would
have put real email addresses into every shared log.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
This reverts 6daaf33d. Master is 13.7.8 and the next release is the 0.0.1
increment from it, so the original 13.7.9 references were correct.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Both grids restored the last good filter by recursing into the filter handler, which
never terminated once the search index rather than the query was the problem: the
restore fails the same way, and the retry uses the same filter. The user got an
endless run of dialogs, each of them blaming a filter string that was fine. Only an
empty last-good filter broke the loop, because that short-circuits before reaching
the search engine.
The fallback is now a bounded sequence -- last good filter, then no filter -- and
the message distinguishes an index Libation cannot reach from a query it cannot
parse. Only the first failure is reported, so restoring is quiet. A malformed query
never surfaces as an IO-family exception, which QueryFailureShapeTests pins against
the real engine, so a typo is never mistaken for index trouble or made to trigger a
rebuild.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
The two path assertions added with the PDF fix compared a path the test built
itself against one that had been through LongPath, which on Windows prefixes a
drive-rooted path with \\?\ so paths past the 260 character limit work. Linux
adds no prefix, so this only showed up on the Windows job.
Normalising both sides is not just about the false failure. The inequality
assertion guarding 'the PDF was saved loose in the Books directory' compared a
raw temp path against a prefixed one, so on Windows it passed on the prefix
alone and would not have caught the bug it exists to catch.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Windows CI caught real over-reach. When another holder has write.lock, Windows
raises the sharing violation before Lucene can turn it into a
LockObtainFailedException, so it arrives as a plain IOException. Repairing anything
that is not a recognised lock conflict then meant deleting the index the other
holder was using -- exactly the second-instance case the retry exists for.
An IOException naming Lucene's write lock now counts as a lock conflict. Matching
the file name rather than the message wording keeps it working on non-English
Windows. The end-to-end test asserts the property instead of the exception type,
since the type legitimately differs by platform.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Releasing the write.lock part way through the retry budget raced with Lucene 3's
own lock bookkeeping: on Windows a competing Obtain left a handle on the file, so
Release and the temp directory cleanup both failed with a sharing violation. Hold
the lock for the whole budget instead and assert what actually matters, that a lock
conflict is retried and leaves the index files alone.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
A PDF is fetched through the same license request as the audiobook, so following
a refusal with a PDF request reproduced, through the PDF, exactly the per-run
refusal the wait exists to stop. The follow-up pass now skips the titles the
first pass deliberately left alone as well as the ones it attempted.
Also stop a failed PDF download leaving an empty folder in the library: a
PDF-only download is the one case that has to create the book's folder before it
has anything to put in it, so it now removes a folder it created and did not
fill. GetProposedDownloadFilePath goes back to being a pure path computation.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Troubleshooting gains an entry for missing and misplaced PDFs, including the
naming-template cause, since a library with no <id> in its folder and file
templates is one Libation cannot recognise the output of at all. The CLI
reference notes what a plain liberate run now covers, and that the
Audiobookshelf upload is deliberately not part of the PDF back-fill.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
A plain 'libationcli liberate' iterates the titles DownloadDecryptBook selects,
and that step selects on '!AudioExists'. A title needing nothing but its PDF was
therefore never reached by the verb documented as 'book and pdf backups' - only
'liberate --pdf' picked it up. For a library that was liberated before its PDFs
were, that is every title with a PDF.
Give the bulk run an optional second pass and have liberate use it for PDFs, the
way the app's Liberate All always has. Skipped when the first pass stopped early
so a run cut short by its download limit does not carry on doing other work, and
titles the first pass attempted are excluded by product id rather than by asking
Validate again, so a step that just failed is not immediately retried.
Left alone: the Audiobookshelf upload stays tied to a fresh liberation. Its
Validate passes for any liberated title, so including it here would walk the
whole library on the next run. 'abs upload' already exists for that.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
getProposedDownloadFilePath looked for the book's audio file and fell back to
the Books directory itself when it found none. That lookup matches on the
product id appearing in the path, so it finds nothing for a library whose
folder and file templates omit <id>, and nothing for a title marked downloaded
whose files are not on this machine. Those PDFs landed in the library root,
where they also shared one namespace and so could collide with each other.
Fall back to the folder template instead - the same folder the audiobook itself
would go in - and create it, since nothing else does on the PDF-only path.
Also give MockLibraryBook a three-field version: ToVersionString formats to at
least three fields, so the two-field default threw as soon as anything rendered
a naming template for a mock book.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Ported from #1949. The reporter's log paused auto-scan on a second account that
had never been logged in, while the dialog blamed an expired session and named no
account, so there was nothing to act on.
AccountCredentialStatus tells a never-registered account apart from one holding an
expired access token, by looking for a refresh token to renew from. AutoScanRunner
now hands the AuthenticationRequiredException to the notification so the prompt can
name the account, which means digging that exception back out of the wrappers the
scan adds on the way up. Same distinction in the log line and in the exception
message ApiExtended throws when interactive login is unavailable, which is what the
CLI and Docker users see.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Ported from #1949, which surfaces the manual recovery steps the maintainer had
been giving out by hand instead of leaving the user with a raw Lucene error.
Adapted to the failure now being contained: with the exception no longer escaping
into the library change, the scan-failure catch blocks #1949 hooked would never
see it, and hooking only those would still miss every other trigger -- removing
books is what crashed the GUI. So the guard moves from AppScaffolding into
SearchEngineCommands next to the update commands it protects, and raises
UpdateFailed from there. Both GUIs subscribe, so any trigger is covered, and the
event carries the exception rather than needing #1949's StackTrace string sniffing
to find it. The dialog is shown once per session: a damaged index fails on every
library change and these steps only need following once.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Fold the pdf-only exclusion into HonorsDeferredRetries instead of also checking
the processable type in the run loop, split the user-facing message building
into its own file next to the store, and leave GC.Collect on the success path
where it was.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Adds a Features page covering the wait schedule, what clears it, and what the
CLI and app show, plus cross-links from the CLI reference, the daily download
limit page and troubleshooting. Troubleshooting also gains an entry for a log
too large to attach to a bug report, which is how the reporter in issue #1947
found this.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Lucene 3's base-36 filename formatter overruns its buffer when segments.gen names
an absurd generation, so the rebuild path has to survive an IndexOutOfRangeException
as well as the IOException shapes. Damage does not always announce itself as IO.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Asserting on the generated JSON alone would pass just as happily with a
misspelled sink argument, which Serilog ignores in silence - and silently not
rolling is the bug. These build a real logger from Libation's own config and
write until it rolls, including a test that pins the old unbounded behaviour so
a future change to the defaults cannot quietly restore it.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
The classifier's inputs are the actual denials from the log attached to issue
#1947: owned titles on an inactive account, a Plus title no longer in the
catalog, and an unreleased preorder Audible has no audio for.
The backoff tests also caught a real overflow: first * 2^n exceeds a TimeSpan
long before the cap matters.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
A license denial left no trace: BookStatus stayed NotLiberated, so every
liberate run asked again. Only the GUI's bad-book dialog could mark a title
Error, and license denials take their own path and never reach that dialog, so
a headless install had no way at all to stop the retries. A cron schedule then
re-requested the same refused licenses every run and printed the same warning
block for each one, which is both wasted API traffic and the log noise reported
in issue #1947.
Record the refusal instead, with a wait that doubles per consecutive failure:
one day for an eligibility denial (up to 30), six hours for a title Audible has
no audio for yet such as an unreleased preorder (up to 7 days), one hour when
the denial names GenericError, which the GUI already reads as an outage
(up to 12). Nothing is permanent - every kind is attempted again on its own.
Only failures attributable to Audible are recorded. A dropped connection, a
decrypt error or a full disk keeps being retried on the next run as before.
Naming a title, --force, and setting a download status all clear the record:
asking for a title explicitly overrides the wait.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
A truncated or zero-length segments file is reported by Lucene 3 as a plain
IOException ("read past EOF") rather than a CorruptIndexException, so it was
misclassified as a write.lock conflict: CreateNewIndex burned its whole backoff
budget and rethrew, and the delete-and-rebuild recovery never ran. Passing
create/overwrite to IndexWriter does not repair it either, because
IndexFileDeleter reads every segments_* file in the directory and tolerates only
missing ones, so a single unreadable segments file -- even a stale one from an
older commit -- leaves the index permanently unopenable. The user's only cure
was deleting the SearchEngine folder by hand.
Retries are now reserved for genuine lock conflicts (LockObtainFailedException,
which derives from IOException, and UnauthorizedAccessException), and any other
open failure gets one delete-and-rebuild pass before giving up with a message
that says which folder to remove. The query path recovers too, since
IsRecoverableCorruptIndexException now recognizes the truncated-segments
signature.
Search index updates are also no longer allowed to fail the library change that
triggered them. Both events fire after the database is committed, so an escaping
exception reported a successful scan as "Error importing library" and, being the
first subscriber, stopped the handlers that refresh the grid and backup counts.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
The default Serilog config set rollingInterval only, so Serilog's own defaults
applied: no size-based roll and a 1 GB ceiling after which the sink silently
stops writing. A busy install (many accounts scanned several times an hour)
reaches tens of MB in a month, past the point where the log can be attached to
a bug report.
Add fileSizeLimitBytes, rollOnFileSizeLimit and retainedFileCountLimit to the
default File sink, and fill in whichever of the three an existing Settings.json
is missing so installs that already have a Serilog section benefit too. Only
absent keys are written, so a hand-tuned config is left alone.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Reflect community testing: once the subscription is inactive, Libation
cannot obtain a license and will not download those books.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Drop the series-parent reason, which no caller can reach: parents are expanded
into their children before anything is queued. Replace the enum and its three
switches with a record carrying the label and the advice, which is all the
switches were mapping to.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
"Absent from your last library scan (run Scan, ...): 2" buries the number behind a
parenthetical. Lead with the label and count, then the advice.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
The menu item is enabled from the grid's display status, so it can be clicked for
episodes the queue will reject - episodes absent from the last scan are the usual
case. Pre-filtering the children threw that reason away and left the queue with an
empty request it could only answer in general terms.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Names the local function ProcessOrStopAsync so it does not shadow
Processable.TryProcessAsync, which means something else entirely, and asserts
the whole sentence a stopped run prints rather than a fragment of it.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Gives the three liberate options their own section on the command line page,
and cross-references them from the daily download limit page so the two
limits are not confused for each other.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
Two levels. The pure tests supply a run's history directly and check the
decision to stop: the count and size thresholds, the always-allow-one rule,
and that titles another process downloaded, or that this run attempted
without downloading, are not counted.
The loop tests drive the real run loop against a real library database with
a processable that records downloads the way DownloadDecryptBook does, which
is as close to a limited run as is possible without an Audible account. They
pin the behavior that matters at the boundary: five books under a limit of
two download two, and a run whose books end exactly at its limit reports
nothing, because nothing was cut short.
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
The multi-book branch of QueueDownloadDecryptAsync returned false with no log
entry and no message whenever UnLiberated() came back empty, so a request
Libation understood and declined looked exactly like a dead button. Callers that
pre-filter with UnLiberated() land here with an empty list, which is the common
way to hit it.
Classify each title that cannot be queued and report the breakdown: already
downloaded, previously failed, absent from the last scan, or a series parent with
no audio of its own. Log it either way; show it only when a person is waiting,
so the automatic post-scan download stays silent.
Fixes#1940
Co-authored-by: rmcrackan <rmcrackan@gmail.com>
