Holding a DatabaseLogPause on the dbQueue thread for its whole life kept the
errors of every queued write out of Logs too: Monitor_Status updates, Events,
Frames and Stats rows. Under lock contention most "Giving up on query" errors
were only in the log files.
Mark each queued row with whether the logger queued it, and pause database
logging only around those inserts into Logs and the backlog warning. The loop
stays closed: a line about a queued write queues one Logs insert, which logs
nothing to the database.
Test: an error from a queued write reaches Logs.
From review by @nabbi on #5208.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zmDbDo() kept its calling thread out of the database for the whole query,
which stopped its own "Success running sql query" line from queueing another
Logs insert. But the loop only starts where the Logs rows are inserted, on
the dbQueue thread, and pausing every zmDbDo() also kept the calling thread's
errors about its query out of Logs: "Can't run query" and the lock contention
retries were only in the log files.
Hold one Logger::DatabaseLogPause on the dbQueue thread for its whole life,
in zmDbQueue::process(), and none in zmDbDo(). Lines the queue thread logs,
the backlog warning included, still stay out of the database; every other
thread's query lines reach it, errors and, at a debug database level, the
success lines too.
Tests: the queue thread's inserts into Logs log nothing and the row count
settles, a query on another thread logs its success line at debug, and an
error from a failed query reaches Logs. The last two fail with the pause in
zmDbDo().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test: cover lines logged to the database while a query runs
Live-database tests ([notCI][dblive]) for how the logger and zmDbDo
interact:
- a line logged on one thread while another thread is inside zmDbDo
reaches the Logs table
- zmDbDo's own debug lines stay out of Logs when the database level is
debug, so inserting a Logs row cannot log another one without end
- a dbQueue backlog over 40 rows does not deadlock threads that log
The first fails on master: zmDbDo sets the process-wide database log
level to NOLOG for the whole query, so the line is dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: stop dropping other threads' database log lines during queries
zmDbDo lowered the logger's database level to NOLOG for as long as its
query ran, so that its own lines (and, at a debug database level, the line
about every insert into Logs) could not queue more inserts into Logs. The
level is shared by the whole process, so every line any other thread
logged in that window never reached the Logs table or the web log view,
while the log file and syslog still had it. The save and restore of that
shared level from several threads is also a data race: a thread can
restore a stale NOLOG and leave database logging off.
Replace the toggle with Logger::DatabaseLogPause, a thread_local count
that keeps only the current thread's lines out of the database. zmDbDo
and the queue's backlog warning use it; mDatabaseLevel is no longer
written outside of initialise().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: warn about the database queue backlog outside the queue's lock
zmDbQueue::process logged its "db queue size has grown larger" warning
while holding the queue mutex. logPrint holds the logger's mutex while it
pushes a Logs row, which takes the queue mutex. With more than 40 rows
waiting, a thread logging to the database and the queue thread could each
hold one of the two locks and wait for the other, stopping every thread
that logs.
Take the backlog size under the lock and warn after releasing it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
