mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-08-02 07:46:48 -04:00
Fixes #1138 Fixes #1141 ## Problem Two language editions of one book resolve to the same canonical title, so they render to the same path and the second gets a `_1` collision suffix. Audiobookshelf treats a folder as exactly one library item, so the pair becomes a single book with both files as tracks and a summed runtime. Shelfmark already parses and displays the language. It just never reached the template engine. ## `{Language}` template variable A template like `{Author}/{Title}{ (Language)}/{Author} - {Title}` now yields: ``` /library/J K Rowling/Harry Potter (sv)/J K Rowling - Harry Potter.m4b /library/J K Rowling/Harry Potter/J K Rowling - Harry Potter.m4b ``` The untagged edition's path is byte-identical to today, so no existing layout shifts. Three details worth flagging: **The value is casefolded.** On a case-insensitive filesystem `(SV)` and `(sv)` would collapse back into one folder, reintroducing the exact collision being fixed. **Values meaning "we don't know" render nothing** rather than producing `Project Hail Mary (unknown)` folders. Anna's Archive reports that string literally (`direct_download.py`, `language = detected or "unknown"`). **The frontend wasn't sending the release language at all**, so the token would have stayed empty for exactly the audiobook sources in the report. Prowlarr and AudiobookBay do not put language in `extra` the way `direct_download` does, hence the payload plumbing. It reads `release.language`, never `book.language` — the latter is the provider's canonical edition and would mislabel a translation, with a regression test for that specifically. Not gated to audiobooks: Calibre-Web-Automated stages ingested files by basename and discards folder structure, so the rename (filename) template is the only lever those users have. Verified that form works: `J K Rowling - Harry Potter (sv).epub`. ## Language consolidation (#1141) Three release sources each carried their own alias map, all resolving to the same ISO 639-1 codes, alongside a bundled database that only one of them used. Adding a language meant editing three places. Aliases now live in `data/book-languages.json` beside the code and name they belong to, and `shelfmark/core/languages.py` resolves any of them — two-letter code, ISO 639-2 three-letter in either the bibliographic or terminological form, or English name. Prowlarr and AudiobookBay drop their tables. Direct Download keeps its own path-parsing heuristics, including the ambiguous short codes that collide with English words (`de`, `en`, `no`, `in`), and takes only the alias data. This also closes a coverage gap. MyAnonamouse offers 62 languages; Prowlarr mapped 37, and an unmapped code is *dropped* rather than passed through, so the other 25 carried no language at all — leaving `{Language}` empty and the collision unfixed for Latin, Farsi, Tamil, Urdu and the rest. Seven languages MAM offers had no database entry at all: Bosnian, Burmese, Estonian, Icelandic, Manx, Scottish Gaelic, Sanskrit. Also fixes the Traditional Chinese code, which used a U+2011 non-breaking hyphen. Nothing compares against the ASCII spelling today so it was latent, but it would silently defeat the first thing that did. ## Validation Verified end to end against a live Prowlarr and MyAnonamouse, not just unit tests. A real search returning both an English and a Swedish edition, through the actual `queue_release` → `DownloadTask` → naming path: ``` STEP 1 real MAM search -> 37 releases, languages: ['en', 'sv'] STEP 3 queue_release -> task.language='sv' STEP 4 build_metadata_dict -> metadata['Language']='sv' STEP 5 build_library_path -> /library/J K Rowling/Harry Potter (sv)/... two language editions resolve to DIFFERENT folders: True ``` The refactor is pinned by a snapshot of both per-source maps taken *before* they were deleted. All 131 aliases are asserted to still resolve to the same code, one parametrised test each, so a regression names the specific alias. Also verified: the filename-only template, the retry round-trip (`serialize_task_for_retry` → `_restore_task_from_retry_payload`, plus a legacy payload with no `language` key), and placeholder handling. Added a `KNOWN_TOKENS` ordering invariant test — `find_placeholder()` does a substring `.find()` in list order and nothing protected that contract, so a future token in the wrong position could silently shadow an existing one. And a lockstep guard on the frontend, since `KNOWN_TOKENS` is hand-duplicated in TypeScript. **One caveat worth stating.** Three MAM codes are confirmed by observation (`ENG`→`en`, `SWE`→`sv`, `MAL`→`ml`, the last from a real `[MAL / EPUB]` Tagore release). The remaining ~59 are derived from ISO 639-2 rather than observed, because MAM's catalogue is overwhelmingly English — enabling 27 extra languages still yielded only one non-English hit across 258 results. Mitigated rather than closed: both 639-2 variants are present for every language where they differ, and a wrong alias is an unused entry while a missing one loses the language. Happy to correct any code a maintainer knows differs. ## Test results 2056 Python tests pass (up from 1906). Frontend typecheck, lint, format and 126 unit tests pass. Pre-existing failures on my machine, unchanged by this branch and unrelated: `tests/bypass/` needs `seleniumbase`, and `tests/config/test_entrypoint_permissions.py` uses bash-4 syntax that macOS bash 3.2 rejects. --------- Co-authored-by: delize <4028612+delize@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz>
177 lines
6.9 KiB
Python
177 lines
6.9 KiB
Python
"""API tests for the frontend config endpoint."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import importlib
|
|
from pathlib import Path
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
|
|
|
|
@pytest.fixture(scope="module")
|
|
def main_module():
|
|
"""Import `shelfmark.main` with background startup disabled."""
|
|
with patch("shelfmark.download.orchestrator.start"):
|
|
import shelfmark.main as main
|
|
|
|
importlib.reload(main)
|
|
return main
|
|
|
|
|
|
@pytest.fixture
|
|
def client(main_module):
|
|
return main_module.app.test_client()
|
|
|
|
|
|
def _set_session(client, *, user_id: str, db_user_id: int, is_admin: bool) -> None:
|
|
with client.session_transaction() as sess:
|
|
sess["user_id"] = user_id
|
|
sess["db_user_id"] = db_user_id
|
|
sess["is_admin"] = is_admin
|
|
|
|
|
|
def test_config_endpoint_uses_user_scope_and_runtime_flags(main_module, client):
|
|
_set_session(client, user_id="reader-1", db_user_id=42, is_admin=False)
|
|
|
|
calls: list[tuple[str, int | None]] = []
|
|
|
|
def fake_get(key, default=None, user_id=None):
|
|
calls.append((key, user_id))
|
|
values = {
|
|
"SHOW_RELEASE_SOURCE_LINKS": False,
|
|
"SHOW_COMBINED_SELECTOR": False,
|
|
"SEARCH_MODE": "universal",
|
|
"SEARCH_PAGE_TITLE": "Custom Shelfmark",
|
|
"METADATA_PROVIDER": "openlibrary",
|
|
"METADATA_PROVIDER_AUDIOBOOK": "",
|
|
"DEFAULT_RELEASE_SOURCE": "prowlarr",
|
|
"DEFAULT_RELEASE_SOURCE_AUDIOBOOK": "audiobookbay",
|
|
"DOWNLOAD_TO_BROWSER_CONTENT_TYPES": ["book", "audiobook"],
|
|
"AUTO_OPEN_DOWNLOADS_SIDEBAR": False,
|
|
"HARDCOVER_AUTO_REMOVE_ON_DOWNLOAD": True,
|
|
"AA_DEFAULT_SORT": "newest",
|
|
}
|
|
return values.get(key, default)
|
|
|
|
with (
|
|
patch.object(main_module.app_config, "get", side_effect=fake_get),
|
|
patch("shelfmark.config.env._is_config_dir_writable", return_value=True),
|
|
patch("shelfmark.core.onboarding.is_onboarding_complete", return_value=True),
|
|
patch("shelfmark.metadata_providers.get_provider_sort_options", return_value=["sort-a"]),
|
|
patch("shelfmark.metadata_providers.get_provider_search_fields", return_value=["field-a"]),
|
|
patch("shelfmark.metadata_providers.get_provider_default_sort", return_value="relevance"),
|
|
):
|
|
resp = client.get("/api/config")
|
|
|
|
assert resp.status_code == 200
|
|
data = resp.get_json()
|
|
assert data["show_release_source_links"] is False
|
|
assert data["show_combined_selector"] is False
|
|
assert data["search_mode"] == "universal"
|
|
assert data["search_page_title"] == "Custom Shelfmark"
|
|
assert data["metadata_sort_options"] == ["sort-a"]
|
|
assert data["metadata_search_fields"] == ["field-a"]
|
|
assert data["default_release_source"] == "prowlarr"
|
|
assert data["default_release_source_audiobook"] == "audiobookbay"
|
|
assert data["download_to_browser_content_types"] == ["book", "audiobook"]
|
|
assert data["settings_enabled"] is True
|
|
assert data["metadata_default_sort"] == "relevance"
|
|
|
|
assert ("SHOW_RELEASE_SOURCE_LINKS", None) in calls
|
|
assert ("SHOW_COMBINED_SELECTOR", 42) in calls
|
|
assert ("DOWNLOAD_TO_BROWSER_CONTENT_TYPES", 42) in calls
|
|
|
|
|
|
def test_config_endpoint_falls_back_to_audiobook_metadata_provider(main_module, client):
|
|
_set_session(client, user_id="reader-2", db_user_id=77, is_admin=False)
|
|
|
|
provider_calls: list[str] = []
|
|
|
|
def fake_get(key, default=None, user_id=None):
|
|
values = {
|
|
"METADATA_PROVIDER": "",
|
|
"METADATA_PROVIDER_AUDIOBOOK": "audiobook-search",
|
|
"SHOW_RELEASE_SOURCE_LINKS": True,
|
|
}
|
|
return values.get(key, default)
|
|
|
|
def sort_options(provider: str):
|
|
provider_calls.append(provider)
|
|
return [f"{provider}-sort"]
|
|
|
|
def search_fields(provider: str):
|
|
provider_calls.append(provider)
|
|
return [f"{provider}-field"]
|
|
|
|
def default_sort(provider: str):
|
|
provider_calls.append(provider)
|
|
return f"{provider}-default"
|
|
|
|
with (
|
|
patch.object(main_module.app_config, "get", side_effect=fake_get),
|
|
patch("shelfmark.config.env._is_config_dir_writable", return_value=True),
|
|
patch("shelfmark.core.onboarding.is_onboarding_complete", return_value=True),
|
|
patch("shelfmark.metadata_providers.get_provider_sort_options", side_effect=sort_options),
|
|
patch("shelfmark.metadata_providers.get_provider_search_fields", side_effect=search_fields),
|
|
patch("shelfmark.metadata_providers.get_provider_default_sort", side_effect=default_sort),
|
|
):
|
|
resp = client.get("/api/config")
|
|
|
|
assert resp.status_code == 200
|
|
data = resp.get_json()
|
|
assert data["metadata_sort_options"] == ["audiobook-search-sort"]
|
|
assert data["metadata_search_fields"] == ["audiobook-search-field"]
|
|
assert data["metadata_default_sort"] == "audiobook-search-default"
|
|
assert provider_calls == ["audiobook-search", "audiobook-search", "audiobook-search"]
|
|
|
|
|
|
def test_frontend_dist_resolves_from_repo_root(main_module):
|
|
expected_project_root = Path(main_module.__file__).resolve().parent.parent
|
|
|
|
assert main_module.PROJECT_ROOT == expected_project_root
|
|
assert main_module.FRONTEND_DIST == expected_project_root / "frontend-dist"
|
|
|
|
|
|
def test_config_endpoint_serves_languages_without_resolution_aliases(main_module, client):
|
|
"""book_languages is a client contract, not a dump of the language data file.
|
|
|
|
data/book-languages.json also carries the aliases used to resolve a source's
|
|
spelling of a language to a code. Those are server-side only: the frontend
|
|
Language type is {code, language}, and shipping the aliases inflated every
|
|
config response by around 40%.
|
|
"""
|
|
_set_session(client, user_id="reader-1", db_user_id=1, is_admin=False)
|
|
|
|
with (
|
|
patch("shelfmark.config.env._is_config_dir_writable", return_value=True),
|
|
patch("shelfmark.core.onboarding.is_onboarding_complete", return_value=True),
|
|
):
|
|
resp = client.get("/api/config")
|
|
|
|
assert resp.status_code == 200
|
|
languages = resp.get_json()["book_languages"]
|
|
|
|
assert languages, "no languages served"
|
|
offending = [entry for entry in languages if set(entry) != {"code", "language"}]
|
|
assert offending == [], f"unexpected keys leaked to clients: {offending[:3]}"
|
|
|
|
|
|
def test_language_data_file_is_only_read_by_the_shared_module(main_module):
|
|
"""Reading data/book-languages.json anywhere else reintroduces the drift the
|
|
shared module exists to prevent, and bypasses the alias handling."""
|
|
del main_module
|
|
|
|
repo_root = Path(__file__).resolve().parents[2]
|
|
allowed = {Path("shelfmark/core/languages.py")}
|
|
|
|
offenders = []
|
|
for path in (repo_root / "shelfmark").rglob("*.py"):
|
|
relative = path.relative_to(repo_root)
|
|
if relative in allowed:
|
|
continue
|
|
if "book-languages" in path.read_text(encoding="utf-8"):
|
|
offenders.append(str(relative))
|
|
|
|
assert offenders == [], f"should use shelfmark.core.languages instead: {offenders}"
|