Files
sabnzbd/tests/data/tavern/api_queue_format.yaml
mnightingale 35d5355f49 Replace CherryPy with Uvicorn and Starlette (#3550)
* Migrate web interface from CherryPy to Uvicorn/Starlette

Squashed rebase of feature/uvicorn (34 commits) onto develop, reconciled
with ~3 months of intervening develop changes.

Replaces the CherryPy webserver and request handling with Uvicorn/Starlette
across the API, web interface, RSS, config pages and related modules.

Reconciliation with develop during the rebase:
- api.py: kept develop's security/behaviour fixes (orphan path-traversal
  guard, expanded log redaction incl. host_whitelist and
  remote_label_replacement, get_dconfig single-return, get_retryable_jobs,
  connections default, translated NNTP test errors) on top of the Starlette
  request/response rewrite.
- interface.py: ported the RSS route handlers to develop's DB-backed
  RSSRepository API (process_feed, rss_repository / find_job_by_url /
  clear_feed / clear_downloaded / flag_downloaded).
- misc.py: kept develop's hachoir-based get_media_duration.
- requirements.txt: dropped the CherryPy stack, adopted develop's newer pins.

Also applied ruff --fix (PEP 604 unions, builtin generics) to align with
develop's lint config.

Verified: ruff check, black --check, and the affected test suites
(9413 passed, 1 skipped) all pass.

* Update starlette/uvicorn versions

* Fix race issues in global rss state

* Fix test race in server shutdown

The uvicorn migration turned /shutdown (and the shutdown API) into fire-and-forget: it spawned shutdown_program() in a background thread and replied immediately, whereas develop ran it synchronously and only replied once halt() had persisted all state. Because the module-scoped test teardown doesn't wait for the process to exit, the next module's clean_cache_dir wiped the shared cache dir (and reused the fixed port) while the previous instance was still saving state and holding the port — producing the three intermittent failures (deleted sabnzbd.log → "File log disabled or not found"; un-persisted [sorters] → KeyError; stale instance → missing wizard .quoteBlock).

* Fix robots and description, add favicon

* Remove remains of http basic auth

* Setup Starlette once configuration is available, fix static file relative cwd and url_base config

* abort_and_show_error when webserver fails to start

* Guard stopping webserver that never started

* Delegate XFF handling to ProxyHeadersMiddleware

* Merged params at request.state.params instead of modifying private apis

* Both shutdown routes share implementation and do not block event loop

* Run sync handlers via run_in_threadpool and facilitate eventual migration to async

* Pool database connections

* Online backup of database due to WAL changes

* Fix exception on None request.client (test clients or unix sockets)

* Fix flakey tests due to process not fully shutting down

* Restore X-Frame-Options behaviour via middleware

* Fix set_config_default with multiple keywords

* Remove broken logging call

* Restore api logging functionality

* Cache-Control: no-store

* Login only via POST

* Remove 401 (basic-auth) and add 404 handling via redirect

* Fix crash when shutdown not an int

* Use BaseRedirectResponse helper

* Remove trailing slashes from wizard routes

* URL helper, absolute URLs everywhere, fixes issues with nested navigation

* Fix scheduler adding multiple daysofweek

* Restore CherryPy api behaviour merging body with query params (body wins)

* Clearer documentation of get_request_params and request_params

* First stage supporting gradual api async

* Fix rss ajax consuming flash

* Restore access log functionality

* Hostname check in middleware

* Request logging in middleware

* Param parsing in middleware

* Security checks in middleware

* secured_expose is now purely route registration

* Lookup api handler once per request

* Fix flakey alert dialogs

* Trigger restart via BackgroundTask

* Restore CherryPy first param wins and get/post consistency

* Remove dead code

* Secure cookies based on protocol the client used

* Fix various issues with port_is_free

1. port_is_free answered the wrong question. It connect-probed ("is something answering?") rather than bind-probed ("can I bind?"). A port could report free and then kill startup at uvicorn's bind().
2. The bind-all remap crossed address families. :: was mapped to 127.0.0.1, probing IPv4 for an IPv6 bind — a regression against portend, which maps :: → ::1.
3. The call sites passed the wrong host. browserhost is a client-reachable address; the thing that has to be bindable is web_host.
4. Errors were swallowed. A bare except OSError hid gaierror, so an unresolvable host reported "free".
5. find_free_port had a port-0 trap. Under a bind-probe, currentport=0 always succeeds and returned 0 — the old failure sentinel. Now guarded, and None instead of 0.
6. Ports 80/443 were misdiagnosed. EACCES was folded into "occupied", producing ten futile probes and a panic claiming another program held the port. PermissionError now propagates to a dedicated panic explaining the actual remedies.
7. The tests were largely tautological. Three tests covering one branch, an IPv6 test with no IPv6 in it, a timeout test that never engaged the timeout, TOCTOU-prone fixed-range probes, no SO_REUSEADDR on the helper listener, and nothing asserting the property that matters — that "free" implies bindable.
8. A portability bug I introduced, then fixed. I'd baked Linux SO_REUSEADDR overlap semantics into four assertions; macOS differs. Now platform-aware, with the IPv6 regression re-covered by checking the socket family directly.

* Claim the bind address for uvicorn on startup, resolves "49" in err handling from cherrypy

* Rename function BaseRedirectResponse to base_redirect_response

* Restore error response on change web directory

* Add missing typings

* Fix return type of retry job for future types

* A better fix for xdist compatibility - test overwrote db_path

* Secure session cookies (rss flash)

* Inline or remove some functions

* Retry job futuretype behaviour

* Sneak a worksteal fix in

* Test and fix retry_job futuretype behaviour
2026-08-11 13:27:51 +01:00

166 lines
5.7 KiB
YAML
Executable File

---
test_name: Check general queue format (json output)
strict:
- json:on
stages:
- name: queue format single entry
request:
url: "http://{SAB_HOST}:{SAB_PORT}/api"
method: GET
params:
mode: queue
apikey: "{SAB_APIKEY}"
limit: 1
response:
status_code: 200
headers:
content-type: !re_match "application/json"
content-type: !re_search "charset=(UTF|utf)-8"
cache-control: "no-store"
pragma: "no-cache"
access-control-allow-origin: "*"
json:
queue:
version: "{SAB_VERSION}"
paused: !anybool
pause_int: !re_match "[0-9]*"
paused_all: !anybool
diskspace1: !re_match "[0-9.]*"
diskspace2: !re_match "[0-9.]*"
diskspace1_norm: !re_search "[0-9][0-9.]*.?(\ [A-Z])?"
diskspace2_norm: !re_search "[0-9][0-9.]*.?(\ [A-Z])?"
diskspacetotal1: !re_match "[0-9.]*"
diskspacetotal2: !re_match "[0-9.]*"
speedlimit: !re_match "[0-9]*"
speedlimit_abs: !re_match "[0-9.]*" # Value may be empty if unset so !anyint won't work
have_warnings: !re_match "[0-9]*"
finishaction: !anything
quota: !re_search "[0-9][0-9.]*.?(\ [A-Z])?"
have_quota: !anybool
left_quota: !re_search "[0-9][0-9.]*.?(\ [A-Z])?"
cache_art: !re_search "[0-9]*"
cache_size: !re_search "[0-9][0-9.]*.?(\ [A-Z])?"
kbpersec: !re_match "[0-9.]*"
speed: !re_search "[0-9][0-9.]*.?(\ [A-Z])?"
mbleft: !re_match "[0-9.]*"
mb: !re_match "[0-9.]*"
sizeleft: !re_search "[0-9][0-9.]*.?(\ [A-Z]+)?"
size: !re_search "[0-9][0-9.]*.?(\ [A-Z]+)?"
noofslots_total: !anyint
status: "Paused"
timeleft: "0:00:00"
noofslots: !anyint
start: !anyint
limit: !anyint
finish: !anyint
slots: !anylist
index: !re_match "[0-9]+"
nzo_id: "!anystr"
unpackopts: !re_match "-?[0-9]+"
priority: "!anystr"
script: "!anystr"
filename: "!anystr"
labels: !anylist
password: "!anystr"
cat: "!anystr"
mbleft: !re_match "[0-9.]*"
mb: !re_match "[0-9.]*"
size: !re_search "[0-9][0-9.]*.?(\ [A-Z]+)?"
sizeleft: !re_search "[0-9][0-9.]*.?(\ [A-Z]+)?"
percentage: "[0-9.]+"
mbmissing: !re_search "[0-9][0-9.]*.?(\ [A-Z]+)?"
direct_unpack: !re_match "[0-9]+"
status: "Paused"
timeleft: "0:00:00"
avg_age: "!anystr"
time_added: !anyint
---
test_name: Check general queue format (xml output)
stages:
- name: queue format single entry
request:
url: "http://{SAB_HOST}:{SAB_PORT}/api"
method: GET
params:
mode: queue
apikey: "{SAB_APIKEY}"
output: xml
limit: 1
response:
status_code: 200
headers:
content-type: !re_match "text/xml"
content-type: !re_search "charset=(UTF|utf)-8"
cache-control: "no-store"
pragma: "no-cache"
access-control-allow-origin: "*"
content-length: !re_match "[0-9]+"
verify_response_with:
function: tavalidate:assert_xml
extra_kwargs:
strict: True
expected: |
<?xml version="1.0" encoding="UTF-8" ?>
<queue>
<version>!anystr</version>
<paused>!anybool</paused>
<pause_int>!anyint</pause_int>
<paused_all>!anybool</paused_all>
<diskspace1>!anyfloat</diskspace1>
<diskspace2>!anyfloat</diskspace2>
<diskspace1_norm>!anystr</diskspace1_norm>
<diskspace2_norm>!anystr</diskspace2_norm>
<diskspacetotal1>!anyfloat</diskspacetotal1>
<diskspacetotal2>!anyfloat</diskspacetotal2>
<speedlimit>!anyint</speedlimit>
<speedlimit_abs>!anystr</speedlimit_abs>
<have_warnings>!anyint</have_warnings>
<finishaction>!anything</finishaction>
<quota>!anystr</quota>
<have_quota>!anybool</have_quota>
<left_quota>!anystr</left_quota>
<cache_art>!anyint</cache_art>
<cache_size>!anystr</cache_size>
<kbpersec>!anyfloat</kbpersec>
<speed>!anystr</speed>
<mbleft>!anyfloat</mbleft>
<mb>!anyfloat</mb>
<sizeleft>!anystr</sizeleft>
<size>!anystr</size>
<noofslots_total>!anyint</noofslots_total>
<status>!anystr</status>
<timeleft>0:00:00</timeleft>
<noofslots>!anyint</noofslots>
<start>!anyint</start>
<limit>!anyint</limit>
<finish>!anyint</finish>
<slots>
<slot>
<index>!anyint</index>
<nzo_id>!anystr</nzo_id>
<unpackopts>!anyint</unpackopts>
<priority>!anystr</priority>
<script>!anystr</script>
<filename>!anystr</filename>
<labels>!anything</labels>
<password>!anystr</password>
<cat>!anystr</cat>
<mbleft>!anyfloat</mbleft>
<mb>!anyfloat</mb>
<size>!anystr</size>
<sizeleft>!anystr</sizeleft>
<percentage>!anyint</percentage>
<mbmissing>!anyfloat</mbmissing>
<direct_unpack>!anything</direct_unpack>
<status>!anystr</status>
<timeleft>0:00:00</timeleft>
<avg_age>!anystr</avg_age>
<time_added>!anyint</time_added>
</slot>
</slots>
</queue>