mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-08-02 07:46:48 -04:00
816a735cde
Fixes #1138 Fixes #1141 ## Problem Two language editions of one book resolve to the same canonical title, so they render to the same path and the second gets a `_1` collision suffix. Audiobookshelf treats a folder as exactly one library item, so the pair becomes a single book with both files as tracks and a summed runtime. Shelfmark already parses and displays the language. It just never reached the template engine. ## `{Language}` template variable A template like `{Author}/{Title}{ (Language)}/{Author} - {Title}` now yields: ``` /library/J K Rowling/Harry Potter (sv)/J K Rowling - Harry Potter.m4b /library/J K Rowling/Harry Potter/J K Rowling - Harry Potter.m4b ``` The untagged edition's path is byte-identical to today, so no existing layout shifts. Three details worth flagging: **The value is casefolded.** On a case-insensitive filesystem `(SV)` and `(sv)` would collapse back into one folder, reintroducing the exact collision being fixed. **Values meaning "we don't know" render nothing** rather than producing `Project Hail Mary (unknown)` folders. Anna's Archive reports that string literally (`direct_download.py`, `language = detected or "unknown"`). **The frontend wasn't sending the release language at all**, so the token would have stayed empty for exactly the audiobook sources in the report. Prowlarr and AudiobookBay do not put language in `extra` the way `direct_download` does, hence the payload plumbing. It reads `release.language`, never `book.language` — the latter is the provider's canonical edition and would mislabel a translation, with a regression test for that specifically. Not gated to audiobooks: Calibre-Web-Automated stages ingested files by basename and discards folder structure, so the rename (filename) template is the only lever those users have. Verified that form works: `J K Rowling - Harry Potter (sv).epub`. ## Language consolidation (#1141) Three release sources each carried their own alias map, all resolving to the same ISO 639-1 codes, alongside a bundled database that only one of them used. Adding a language meant editing three places. Aliases now live in `data/book-languages.json` beside the code and name they belong to, and `shelfmark/core/languages.py` resolves any of them — two-letter code, ISO 639-2 three-letter in either the bibliographic or terminological form, or English name. Prowlarr and AudiobookBay drop their tables. Direct Download keeps its own path-parsing heuristics, including the ambiguous short codes that collide with English words (`de`, `en`, `no`, `in`), and takes only the alias data. This also closes a coverage gap. MyAnonamouse offers 62 languages; Prowlarr mapped 37, and an unmapped code is *dropped* rather than passed through, so the other 25 carried no language at all — leaving `{Language}` empty and the collision unfixed for Latin, Farsi, Tamil, Urdu and the rest. Seven languages MAM offers had no database entry at all: Bosnian, Burmese, Estonian, Icelandic, Manx, Scottish Gaelic, Sanskrit. Also fixes the Traditional Chinese code, which used a U+2011 non-breaking hyphen. Nothing compares against the ASCII spelling today so it was latent, but it would silently defeat the first thing that did. ## Validation Verified end to end against a live Prowlarr and MyAnonamouse, not just unit tests. A real search returning both an English and a Swedish edition, through the actual `queue_release` → `DownloadTask` → naming path: ``` STEP 1 real MAM search -> 37 releases, languages: ['en', 'sv'] STEP 3 queue_release -> task.language='sv' STEP 4 build_metadata_dict -> metadata['Language']='sv' STEP 5 build_library_path -> /library/J K Rowling/Harry Potter (sv)/... two language editions resolve to DIFFERENT folders: True ``` The refactor is pinned by a snapshot of both per-source maps taken *before* they were deleted. All 131 aliases are asserted to still resolve to the same code, one parametrised test each, so a regression names the specific alias. Also verified: the filename-only template, the retry round-trip (`serialize_task_for_retry` → `_restore_task_from_retry_payload`, plus a legacy payload with no `language` key), and placeholder handling. Added a `KNOWN_TOKENS` ordering invariant test — `find_placeholder()` does a substring `.find()` in list order and nothing protected that contract, so a future token in the wrong position could silently shadow an existing one. And a lockstep guard on the frontend, since `KNOWN_TOKENS` is hand-duplicated in TypeScript. **One caveat worth stating.** Three MAM codes are confirmed by observation (`ENG`→`en`, `SWE`→`sv`, `MAL`→`ml`, the last from a real `[MAL / EPUB]` Tagore release). The remaining ~59 are derived from ISO 639-2 rather than observed, because MAM's catalogue is overwhelmingly English — enabling 27 extra languages still yielded only one non-English hit across 258 results. Mitigated rather than closed: both 639-2 variants are present for every language where they differ, and a wrong alias is an unused entry while a missing one loses the language. Happy to correct any code a maintainer knows differs. ## Test results 2056 Python tests pass (up from 1906). Frontend typecheck, lint, format and 126 unit tests pass. Pre-existing failures on my machine, unchanged by this branch and unrelated: `tests/bypass/` needs `seleniumbase`, and `tests/config/test_entrypoint_permissions.py` uses bash-4 syntax that macOS bash 3.2 rejects. --------- Co-authored-by: delize <4028612+delize@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz>
80 lines
4.8 KiB
JSON
80 lines
4.8 KiB
JSON
[
|
||
{ "language": "English", "code": "en", "aliases": ["eng"] },
|
||
{ "language": "Chinese", "code": "zh", "aliases": ["chi", "zho"] },
|
||
{ "language": "Russian", "code": "ru", "aliases": ["rus"] },
|
||
{ "language": "Spanish", "code": "es", "aliases": ["spa"] },
|
||
{ "language": "French", "code": "fr", "aliases": ["fra", "fre"] },
|
||
{ "language": "German", "code": "de", "aliases": ["deu", "ger"] },
|
||
{ "language": "Italian", "code": "it", "aliases": ["ita"] },
|
||
{ "language": "Portuguese", "code": "pt", "aliases": ["por"] },
|
||
{ "language": "Polish", "code": "pl", "aliases": ["pol"] },
|
||
{ "language": "Bulgarian", "code": "bg", "aliases": ["bul"] },
|
||
{ "language": "Dutch", "code": "nl", "aliases": ["dut", "nld"] },
|
||
{ "language": "Japanese", "code": "ja", "aliases": ["jap", "jpn"] },
|
||
{ "language": "Arabic", "code": "ar", "aliases": ["ara"] },
|
||
{ "language": "Hebrew", "code": "he", "aliases": ["heb"] },
|
||
{ "language": "Turkish", "code": "tr", "aliases": ["tur"] },
|
||
{ "language": "Hungarian", "code": "hu", "aliases": ["hun"] },
|
||
{ "language": "Latin", "code": "la", "aliases": ["lat"] },
|
||
{ "language": "Czech", "code": "cs", "aliases": ["ces", "cze"] },
|
||
{ "language": "Korean", "code": "ko", "aliases": ["kor"] },
|
||
{ "language": "Ukrainian", "code": "uk", "aliases": ["ukr"] },
|
||
{ "language": "Indonesian", "code": "id", "aliases": ["ind"] },
|
||
{ "language": "Romanian", "code": "ro", "aliases": ["rom", "ron"] },
|
||
{ "language": "Swedish", "code": "sv", "aliases": ["swe"] },
|
||
{ "language": "Greek", "code": "el", "aliases": ["ell", "gre"] },
|
||
{ "language": "Lithuanian", "code": "lt", "aliases": ["lit"] },
|
||
{ "language": "Bangla", "code": "bn", "aliases": ["ben", "bengali"] },
|
||
{ "language": "Traditional Chinese", "code": "zh-Hant", "aliases": ["zh‑Hant"] },
|
||
{ "language": "Afrikaans", "code": "af", "aliases": ["afr"] },
|
||
{ "language": "Catalan", "code": "ca", "aliases": ["cat"] },
|
||
{ "language": "Danish", "code": "da", "aliases": ["dan"] },
|
||
{ "language": "Thai", "code": "th", "aliases": ["tha"] },
|
||
{ "language": "Hindi", "code": "hi", "aliases": ["hin"] },
|
||
{ "language": "Irish", "code": "ga", "aliases": ["gle"] },
|
||
{ "language": "Latvian", "code": "lv", "aliases": ["lav"] },
|
||
{ "language": "Tibetan", "code": "bo" },
|
||
{ "language": "Kannada", "code": "kn", "aliases": ["kan"] },
|
||
{ "language": "Serbian", "code": "sr", "aliases": ["srp"] },
|
||
{ "language": "Persian", "code": "fa", "aliases": ["farsi", "fas", "per"] },
|
||
{ "language": "Croatian", "code": "hr", "aliases": ["hrv"] },
|
||
{ "language": "Slovak", "code": "sk" },
|
||
{ "language": "Javanese", "code": "jv", "aliases": ["jav"] },
|
||
{ "language": "Vietnamese", "code": "vi", "aliases": ["vie"] },
|
||
{ "language": "Urdu", "code": "ur", "aliases": ["urd"] },
|
||
{ "language": "Finnish", "code": "fi", "aliases": ["fin"] },
|
||
{ "language": "Norwegian", "code": "no", "aliases": ["nor"] },
|
||
{ "language": "Kinyarwanda", "code": "rw" },
|
||
{ "language": "Tamil", "code": "ta", "aliases": ["tam"] },
|
||
{ "language": "Belarusian", "code": "be" },
|
||
{ "language": "Kazakh", "code": "kk" },
|
||
{ "language": "Mongolian", "code": "mn" },
|
||
{ "language": "Georgian", "code": "ka" },
|
||
{ "language": "Slovenian", "code": "sl", "aliases": ["slv"] },
|
||
{ "language": "Esperanto", "code": "eo" },
|
||
{ "language": "Galician", "code": "gl" },
|
||
{ "language": "Marathi", "code": "mr", "aliases": ["mar"] },
|
||
{ "language": "Filipino", "code": "fil", "aliases": ["tagalog", "tgl"] },
|
||
{ "language": "Gujarati", "code": "gu", "aliases": ["guj"] },
|
||
{ "language": "Malayalam", "code": "ml", "aliases": ["mal"] },
|
||
{ "language": "Kyrgyz", "code": "ky" },
|
||
{ "language": "Azerbaijani", "code": "az" },
|
||
{ "language": "Quechua", "code": "qu" },
|
||
{ "language": "Swahili", "code": "sw" },
|
||
{ "language": "Bashkir", "code": "ba" },
|
||
{ "language": "Punjabi", "code": "pa", "aliases": ["pan"] },
|
||
{ "language": "Malay", "code": "ms", "aliases": ["may", "msa"] },
|
||
{ "language": "Telugu", "code": "te", "aliases": ["tel"] },
|
||
{ "language": "Albanian", "code": "sq" },
|
||
{ "language": "Uyghur", "code": "ug" },
|
||
{ "language": "Armenian", "code": "hy" },
|
||
{ "language": "Shan", "code": "shn" },
|
||
{ "language": "Bosnian", "code": "bs", "aliases": ["bos"] },
|
||
{ "language": "Burmese", "code": "my", "aliases": ["bur", "mya"] },
|
||
{ "language": "Estonian", "code": "et", "aliases": ["est"] },
|
||
{ "language": "Icelandic", "code": "is", "aliases": ["ice", "isl"] },
|
||
{ "language": "Manx", "code": "gv", "aliases": ["glv"] },
|
||
{ "language": "Scottish Gaelic", "code": "gd", "aliases": ["gla"] },
|
||
{ "language": "Sanskrit", "code": "sa", "aliases": ["san"] }
|
||
]
|