mirror of
https://github.com/ZoneMinder/zoneminder.git
synced 2026-11-07 21:23:02 -05:00
Isaac Connor
022410c299
A DateTime lower bound was written as COALESCE(E.EndDateTime, '9999-12-31 23:59:59') >= T1 reading "an event that has not ended yet never ends". EndDateTime is NULL for an event that is still recording, but also for one zmc was killed part way through, and that stays NULL forever - so every abandoned event matched every window from then on. Wrapping the column in a function also meant no index could range over it: with the only MonitorId key being MonitorId alone, every montage review request read all of that monitor's events and filtered them in memory, however narrow the window. Length tells the two cases apart. It is flushed during recording, so a live event's effective end keeps advancing while an abandoned one's is frozen at whatever was recorded. Use the same CASE the SELECT list in ajax/events.php already uses, and bound it below by StartDateTime. The lower bound now emits three conjuncts: E.StartDateTime >= DATE_SUB(T1, INTERVAL 1 DAY) AND (E.EndDateTime IS NULL OR E.EndDateTime >= T1) AND <effective end> >= T1 The floor bounds the scan for a window in the past and drops events abandoned more than a day earlier - events do not outlive the nightly logrotate SIGHUP, which stops and restarts them, so a day is comfortably beyond any real event. The middle conjunct is implied by the third and exists only to give the optimiser a second indexable handle: it is narrow exactly when the floor is wide, so between them a window at either end of the retention period has something cheap to range over. Verified the optimiser picks correctly for both, unprompted. The third is the residual that gets the semantics right. Add the two composite keys those conjuncts need. Two range columns cannot both narrow one B-tree, and these are mirror images - EndDateTime >= T1 is open-ended upwards, StartDateTime <= T2 downwards: Events_MonitorId_StartDateTime_idx (MonitorId, StartDateTime) Events_EndDateTime_MonitorId_idx (EndDateTime, MonitorId) MonitorId leads the first because it is the equality; past a range column later columns can no longer narrow the scan, and (StartDateTime, MonitorId) measured 3.5x slower for the same rows. EndDateTime leads the second so it also covers zmaudit's hunt for events that were never closed, which has no monitor to scope it - 9 rows with a key against a 22,948 row table scan without. Both replacements are added before the keys they supersede are dropped: Events_MonitorId_idx (MonitorId) is now a leftmost prefix of the new key. Events_EndDateTime_DiskSpace (EndDateTime, DiskSpace) existed for the scan that hunted events with no DiskSpace set; DiskSpace is set when the event is finalised in C++, so nothing scans for it any more. Net index count on Events is unchanged. Measured on a 7,167 event monitor. Default one hour window: 7,055 rows examined / 25.4ms -> 173 rows / 4ms. Window scrubbed a week back, which persists in the zmFilter_StartDateTime cookie: 6,995 rows / 28ms -> 19 rows / 0.15ms. Adds migration db/zm_update-1.39.22.sql. Extends t/filter_sql.t; the PHP and Perl were checked to emit identical SQL, and the semantics checked against a table of seven event shapes - abandoned long ago, abandoned recently, overlapping, inside, before, still recording, and still recording for 17 hours.
77 lines
4.0 KiB
SQL
77 lines
4.0 KiB
SQL
--
|
|
-- Index Events for the "overlaps a time window" query.
|
|
--
|
|
-- Montage review, and any filter with DateTime terms, ask for events
|
|
-- overlapping a window: a monitor, an upper bound on StartDateTime, and a lower
|
|
-- bound on when the event ended. None of it was indexable - the only MonitorId
|
|
-- key was MonitorId alone - so every request read all of that monitor's events
|
|
-- and filtered them in memory, no matter how narrow the window.
|
|
--
|
|
-- Measured on a 7,167 event monitor. Default one hour window: 7,055 rows
|
|
-- examined / 25.4ms -> 173 rows / 4ms. Window scrubbed a week back: 6,995 rows
|
|
-- / 28ms -> 19 rows / 0.15ms.
|
|
--
|
|
-- Two range columns cannot both narrow one B-tree, and these two are mirror
|
|
-- images - EndDateTime >= T1 is open-ended upwards so it is cheap near now and
|
|
-- degenerate for an old window, StartDateTime <= T2 is the reverse - so both
|
|
-- keys are added and the optimiser picks per query.
|
|
--
|
|
-- Replaces two existing keys:
|
|
-- Events_MonitorId_idx (MonitorId) is a leftmost prefix of the new
|
|
-- (MonitorId, StartDateTime), so it is redundant.
|
|
-- Events_EndDateTime_DiskSpace (EndDateTime, DiskSpace) existed for the scan
|
|
-- that hunted events with no DiskSpace set. DiskSpace is now set when the
|
|
-- event is finalised in C++, so nothing scans for it. The EndDateTime half
|
|
-- is still needed - zmaudit looks for events that were never closed, with
|
|
-- no monitor to scope it - which is why EndDateTime leads the new key.
|
|
--
|
|
-- The replacements are added before the old keys are dropped so there is never
|
|
-- a point with neither. Idempotent, so re-running is a no-op.
|
|
--
|
|
|
|
-- (MonitorId, StartDateTime): MonitorId leads because it is the equality;
|
|
-- once a range column is reached later columns cannot narrow the scan.
|
|
SET @s = (SELECT IF(
|
|
(SELECT COUNT(*) FROM INFORMATION_SCHEMA.STATISTICS WHERE table_schema = DATABASE()
|
|
AND table_name = 'Events' AND index_name = 'Events_MonitorId_StartDateTime_idx') > 0,
|
|
"SELECT 'Events_MonitorId_StartDateTime_idx already exists on Events'",
|
|
"ALTER TABLE `Events` ADD INDEX `Events_MonitorId_StartDateTime_idx` (`MonitorId`,`StartDateTime`)"
|
|
));
|
|
PREPARE stmt FROM @s; EXECUTE stmt; DEALLOCATE PREPARE stmt;
|
|
|
|
-- (EndDateTime, MonitorId): EndDateTime leads so this also covers zmaudit's
|
|
-- EndDateTime IS NULL scan, which has no monitor to scope it.
|
|
SET @s = (SELECT IF(
|
|
(SELECT COUNT(*) FROM INFORMATION_SCHEMA.STATISTICS WHERE table_schema = DATABASE()
|
|
AND table_name = 'Events' AND index_name = 'Events_EndDateTime_MonitorId_idx') > 0,
|
|
"SELECT 'Events_EndDateTime_MonitorId_idx already exists on Events'",
|
|
"ALTER TABLE `Events` ADD INDEX `Events_EndDateTime_MonitorId_idx` (`EndDateTime`,`MonitorId`)"
|
|
));
|
|
PREPARE stmt FROM @s; EXECUTE stmt; DEALLOCATE PREPARE stmt;
|
|
|
|
-- Now superseded by the leftmost prefix of Events_MonitorId_StartDateTime_idx.
|
|
SET @s = (SELECT IF(
|
|
(SELECT COUNT(*) FROM INFORMATION_SCHEMA.STATISTICS WHERE table_schema = DATABASE()
|
|
AND table_name = 'Events' AND index_name = 'Events_MonitorId_idx') > 0,
|
|
"ALTER TABLE `Events` DROP INDEX `Events_MonitorId_idx`",
|
|
"SELECT 'Events_MonitorId_idx already removed from Events'"
|
|
));
|
|
PREPARE stmt FROM @s; EXECUTE stmt; DEALLOCATE PREPARE stmt;
|
|
|
|
-- The (EndDateTime, DiskSpace) key, under both names it has been created with.
|
|
SET @s = (SELECT IF(
|
|
(SELECT COUNT(*) FROM INFORMATION_SCHEMA.STATISTICS WHERE table_schema = DATABASE()
|
|
AND table_name = 'Events' AND index_name = 'Events_EndDateTime_DiskSpace') > 0,
|
|
"ALTER TABLE `Events` DROP INDEX `Events_EndDateTime_DiskSpace`",
|
|
"SELECT 'Events_EndDateTime_DiskSpace already removed from Events'"
|
|
));
|
|
PREPARE stmt FROM @s; EXECUTE stmt; DEALLOCATE PREPARE stmt;
|
|
|
|
SET @s = (SELECT IF(
|
|
(SELECT COUNT(*) FROM INFORMATION_SCHEMA.STATISTICS WHERE table_schema = DATABASE()
|
|
AND table_name = 'Events' AND index_name = 'Events_EndTime_DiskSpace') > 0,
|
|
"ALTER TABLE `Events` DROP INDEX `Events_EndTime_DiskSpace`",
|
|
"SELECT 'Events_EndTime_DiskSpace already removed from Events'"
|
|
));
|
|
PREPARE stmt FROM @s; EXECUTE stmt; DEALLOCATE PREPARE stmt;
|