Move the rendering of the script log "More" link from the backend's `stage_log` content to the frontend. The backend now provides a `has_script_log` flag, allowing the frontend to dynamically display the link and fetch the script log on demand.
Includes a database migration to clean up old embedded links in existing `stage_log` entries. This improves separation of concerns, simplifies `stage_log` data, and allows for more flexible frontend rendering.
* Add SafeUnpickler to guard against pickle-attacks
* Harden pickle unpickler with explicit allowlist
The previous `SafeUnpickler` allowed any class from `sabnzbd.*` by wildcard, which could still enable deserialization attacks if a "gadget class" (e.g., with a malicious `__del__` method) was present within our own package.
This commit renames the class to `RestrictedUnpickler` and changes its logic to only allow classes explicitly defined in `_SAFE_GLOBALS`. This significantly enhances security by preventing the unpickling of any unlisted classes, including those from within `sabnzbd`. Adds `os.stat_result` and `sabnzbd.nzb.*` classes to the allowlist for compatibility.
* Migrate web interface from CherryPy to Uvicorn/Starlette
Squashed rebase of feature/uvicorn (34 commits) onto develop, reconciled
with ~3 months of intervening develop changes.
Replaces the CherryPy webserver and request handling with Uvicorn/Starlette
across the API, web interface, RSS, config pages and related modules.
Reconciliation with develop during the rebase:
- api.py: kept develop's security/behaviour fixes (orphan path-traversal
guard, expanded log redaction incl. host_whitelist and
remote_label_replacement, get_dconfig single-return, get_retryable_jobs,
connections default, translated NNTP test errors) on top of the Starlette
request/response rewrite.
- interface.py: ported the RSS route handlers to develop's DB-backed
RSSRepository API (process_feed, rss_repository / find_job_by_url /
clear_feed / clear_downloaded / flag_downloaded).
- misc.py: kept develop's hachoir-based get_media_duration.
- requirements.txt: dropped the CherryPy stack, adopted develop's newer pins.
Also applied ruff --fix (PEP 604 unions, builtin generics) to align with
develop's lint config.
Verified: ruff check, black --check, and the affected test suites
(9413 passed, 1 skipped) all pass.
* Update starlette/uvicorn versions
* Fix race issues in global rss state
* Fix test race in server shutdown
The uvicorn migration turned /shutdown (and the shutdown API) into fire-and-forget: it spawned shutdown_program() in a background thread and replied immediately, whereas develop ran it synchronously and only replied once halt() had persisted all state. Because the module-scoped test teardown doesn't wait for the process to exit, the next module's clean_cache_dir wiped the shared cache dir (and reused the fixed port) while the previous instance was still saving state and holding the port — producing the three intermittent failures (deleted sabnzbd.log → "File log disabled or not found"; un-persisted [sorters] → KeyError; stale instance → missing wizard .quoteBlock).
* Fix robots and description, add favicon
* Remove remains of http basic auth
* Setup Starlette once configuration is available, fix static file relative cwd and url_base config
* abort_and_show_error when webserver fails to start
* Guard stopping webserver that never started
* Delegate XFF handling to ProxyHeadersMiddleware
* Merged params at request.state.params instead of modifying private apis
* Both shutdown routes share implementation and do not block event loop
* Run sync handlers via run_in_threadpool and facilitate eventual migration to async
* Pool database connections
* Online backup of database due to WAL changes
* Fix exception on None request.client (test clients or unix sockets)
* Fix flakey tests due to process not fully shutting down
* Restore X-Frame-Options behaviour via middleware
* Fix set_config_default with multiple keywords
* Remove broken logging call
* Restore api logging functionality
* Cache-Control: no-store
* Login only via POST
* Remove 401 (basic-auth) and add 404 handling via redirect
* Fix crash when shutdown not an int
* Use BaseRedirectResponse helper
* Remove trailing slashes from wizard routes
* URL helper, absolute URLs everywhere, fixes issues with nested navigation
* Fix scheduler adding multiple daysofweek
* Restore CherryPy api behaviour merging body with query params (body wins)
* Clearer documentation of get_request_params and request_params
* First stage supporting gradual api async
* Fix rss ajax consuming flash
* Restore access log functionality
* Hostname check in middleware
* Request logging in middleware
* Param parsing in middleware
* Security checks in middleware
* secured_expose is now purely route registration
* Lookup api handler once per request
* Fix flakey alert dialogs
* Trigger restart via BackgroundTask
* Restore CherryPy first param wins and get/post consistency
* Remove dead code
* Secure cookies based on protocol the client used
* Fix various issues with port_is_free
1. port_is_free answered the wrong question. It connect-probed ("is something answering?") rather than bind-probed ("can I bind?"). A port could report free and then kill startup at uvicorn's bind().
2. The bind-all remap crossed address families. :: was mapped to 127.0.0.1, probing IPv4 for an IPv6 bind — a regression against portend, which maps :: → ::1.
3. The call sites passed the wrong host. browserhost is a client-reachable address; the thing that has to be bindable is web_host.
4. Errors were swallowed. A bare except OSError hid gaierror, so an unresolvable host reported "free".
5. find_free_port had a port-0 trap. Under a bind-probe, currentport=0 always succeeds and returned 0 — the old failure sentinel. Now guarded, and None instead of 0.
6. Ports 80/443 were misdiagnosed. EACCES was folded into "occupied", producing ten futile probes and a panic claiming another program held the port. PermissionError now propagates to a dedicated panic explaining the actual remedies.
7. The tests were largely tautological. Three tests covering one branch, an IPv6 test with no IPv6 in it, a timeout test that never engaged the timeout, TOCTOU-prone fixed-range probes, no SO_REUSEADDR on the helper listener, and nothing asserting the property that matters — that "free" implies bindable.
8. A portability bug I introduced, then fixed. I'd baked Linux SO_REUSEADDR overlap semantics into four assertions; macOS differs. Now platform-aware, with the IPv6 regression re-covered by checking the socket family directly.
* Claim the bind address for uvicorn on startup, resolves "49" in err handling from cherrypy
* Rename function BaseRedirectResponse to base_redirect_response
* Restore error response on change web directory
* Add missing typings
* Fix return type of retry job for future types
* A better fix for xdist compatibility - test overwrote db_path
* Secure session cookies (rss flash)
* Inline or remove some functions
* Retry job futuretype behaviour
* Sneak a worksteal fix in
* Test and fix retry_job futuretype behaviour
* Verify RAR5 passwords when header encryption is not used
* Make flow a bit closer to rarfile
* Spelling
* Add note to trusting header verification
* Spelling
* Delegate to rarfile 4.3 where possible and cache rar*_s2k
* Explicit types instead of TypeVar
* Remove duplicate test and add a header encryption with correct password
* Verify after parse
* Rename method and explicit cache size
* Remove dead code
* Clearer verify after parse
* Performance early exit
* Clarify early exit
* Update comment, usually same password and salt
* Copy parse doc
* Track files during cleanup to prevent removing unrelated files
* Remove redundant path normalization
The `os.path.abspath` function already includes path normalization,
rendering the explicit call to `os.path.normpath` redundant
when used in conjunction. Removing it simplifies the code
without changing its behavior.
* Support extracting tar files
* Set fail_msg when Python is too old
* Update error to "Unpacking failed, TAR support requires Python 3.12 or later", set per set/file, and don't regard as a failure if unsupported
* Update sabnzbd/newsunpack.py
Co-authored-by: Safihre <safihre@sabnzbd.org>
* Use UNWANTED_FILE_PERMISSIONS (stat.S_ISUID and stat.S_ISGID) are already removed by tar_filter but this is cleaner
* TAR extraction test including non-executable check
* ruff
* Skip executable check on Windows
* Remove helloworld.sh
* get_unique_filename
* Test path traversal, owner/group and permissions
* Fix duplicate file issues with one_folder
* Rename filter
---------
Co-authored-by: Safihre <safihre@sabnzbd.org>
* Match to previous RSS behaviour
* Cleanup logic due to normalisation
* Tests comparing 4.5.5 outcome
* Remove comment - fixed since 4.5.5
* Consistent types for prio, rule, season, and episode
* Rename matching_rule_index
* Fix type warnings
* Replace duplicate nzo_ids in history database and use uuid4
* Remove SABnzbd_nzo on reuse
* Keep SABnzbd_nzo_ prefix for future jobs
* Try loading from multiple normal and future paths
* Purege NZO_FILE
* Rename NZO_FILE
* Allow failed migration to rollback
Closes#1574
Add tests for long paths
Make sure long path is >260
Add rar test file with invalid Windows filenames
Add rar_unpack tests for unicode and passworded sets
Simplify Unrar command building
Add test for rar_invalid_windows
Remove check for 260 chars in rar_unpack
Should never happen anymore
Let Unrar rename invalid filenames
Check full path output if rar_unpack
Add helper for check
Correct test_rar_unpack_invalid_windows_filenames
Apply changes also to Direct Unpacker
Extend testing to make sure full paths are tested
Add tests for long paths inside rar
Unrar auto-rename message is different on Linux
* Deobfuscate ON by default
* Correct data test set: filename that is not obfuscated. Except test_par2file
* Correct data test set: filename that is not obfuscated. Except test_par2file
* Commented out test in test_par2file.py
* Commented out test in test_par2file.py
* assert for unicode_rar 我喜欢编程 now working too
---------
Co-authored-by: sander <san.d.erjonkers+github@gmail.com>
* replace series/date/movie sorters with a generic sorter
* fix test_eval_sort on windoze
* unbreak and de-uglify the fix
* add special setting for season pack sorting
* remove unused import
* replace series/date/movie sorters with a generic sorter
* fix test_eval_sort on windoze
* unbreak and de-uglify the fix
* add special setting for season pack sorting
* remove unused import
* correct type for sort_type entries
* standardize ui
* add visual hints for drag-n-drop
* move presets directly below sort string field
* replace hex with ascii letters to avoid random occurences of (cd|e)[0-9]+
* Some styling things
---------
Co-authored-by: Safihre <safihre@sabnzbd.org>
* detect fully encrypted rars
* debug.warning working, nzo.fail_msg alas is overwritten
* a bit of clean-up
* a bit of clean-up
* the real clean-up
* no intermediate variable
Co-authored-by: Safihre <safihre@sabnzbd.org>
* Shorter message
Co-authored-by: Safihre <safihre@sabnzbd.org>
* more clean-up
* unittest
Co-authored-by: sander <san.d.erjonkers+github@gmail.com>
Co-authored-by: Safihre <safihre@sabnzbd.org>
* record new files generated based on par2
* record new files generated based on par2
* test first par2 based renaming, then deobfuscate obfuscated names
* remove commented-out line
* corrected contents zip-file
* try again, github
* try again, github
* correct_extension: basics, including unittest
* correct_extension: basics, including unittest
* correct_extension: puremagic into requirements.txt
* correct_extension: introduce a main for testing from CLI
* correct_extension: parse all parameters on CLI as files
* correct_extension: parse all parameters on CLI as files
* correct_extension: CLI parameter "-p" for privacy output
* correct_extension: has_common_extension() and most_likely_extension()
* correct_extension: has_common_extension() and most_likely_extension()
* correct_extension: add extension if file has no commonly used extension
* correct_extension: Black happy ... hopefully
* correct_extension: Black happy ... hopefully
* correct_extension: process feedback, mainly the extenions lists ^H^H^H^ tuples
* correct_extension: process feedback, mainly the extenions lists ^H^H^H^ tuples
* correct_extension: process feedback, mainly the extenions lists ^H^H^H^ tuples
* correct_extension: process feedback, mainly the extenions lists ^H^H^H^ tuples
* correct_extension: cleaned up
* correct_extension: cleaned up ... github-black now happy?
* correct_extension: cleaned up ... github-black now happy?
* correct_extension: cleaned up ... github-black now happy?
* correct_extension: cleaned up ... github-black now happy?
* correct_extension: cleaned up ... github-black now happy?
* correct_extension: easier if-then-logic, check if new_extension_to_add is filled.
* correct_extension: if puremagic does recoging txt or nzb, check ourselves
* correct_extension: if puremagic does recoging txt or nzb, check ourselves
* correct_extension: only files!
* correct_extension: only files!
* correct_extension: rNN files not common extension, plus easier testing
* correct_extension: clean-up ... no more boolean extension_too
* correct_extension: requirements.txt, solved a TODO, and use get_ext()
* correct_extension: a comment added
* correct_extension: correct typing, correct txt and nzb extension
* correct_extension: extensions always with dots, bug fix in what_is_most_likely_extension()
* correct_extension: back on track?
* correct_extension: back on track?
* correct_extension: better comments
* Save all interface values if useGlobalOptions is true
* Try to fix the tests
* New test test
* Another test test
* Remove default value for interface_settings
* fix deprecation warning in sabnews regex
* enable text, xml returns from get_api_result
* add api tests
* add functional api tests
* add tavern.yaml files to test data
* explicitly add lxml to work around pip dependency issues
* prevent pytest from picking up the tavern files
* Revert "fix deprecation warning in sabnews regex"
This reverts commit 4f0b7131e7.
* address minor issues
* integrate fixtures into conftest
* black :/
* harden queue repair test
* try a workaround for extremely slow test runs on windoze
* Correct server detection in functional tests
* move scripts dir inside SAB_CACHE_DIR
* also relocate the generated script
Co-authored-by: Safihre <safihre@sabnzbd.org>
Safihre