Gating the measurement on AudioDetection made the graph useless for the job
it is most wanted for. AudioThreshold is a per-device number -- the floor on
one camera's mic is nothing like another's -- so it has to be measured before
it can be set, but nothing was measured until it was already set. Enabling
detection with a guessed threshold to find out what the real one should be is
backwards.
The level is now read for every monitor with decodable audio. AudioDetection
governs only whether crossing the threshold contributes a score, which is
what the setting is named for. shared_data->audio_alarm stays 0 when it is
off, so nothing downstream changes for a monitor that does not want audio
alarms.
Nothing here depends on Analysing either. The measurement is in
Monitor::Capture, which runs on whatever Analysing is set to, and frame rows
come from Event::AddFrame, which a continuously recording monitor reaches
through the RECORDING_ALWAYS path with motion detection off. So a monitor
that only records continuously still gets levels on its rows.
Since Monitor::Capture retries Open on every audio packet until it succeeds,
and that now happens for every monitor with audio rather than the handful with
detection on, AudioDetector remembers a codec it has already failed to find a
decoder for. Without it a stream ZoneMinder cannot decode logs a warning at
the audio packet rate for as long as the monitor runs. A reconnect bringing a
different codec is still tried.
The cost is one audio decode per monitor with audio, where before it was one
per monitor with detection enabled. That is small next to the video path, but
it is not nothing on a box with many cameras.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JpiSWBmtQkR5bcgpHWY4ME
zm_update-1.39.31.sql gave monitors audio detection, but the level only ever
existed in shared memory, so it was gone the moment the frame passed and there
was nothing for the event view to plot. Add Frames.AudioLevel next to Score, on
the same 0-100 dBFS-derived scale the threshold uses.
What is stored is the peak since the previous row, not the level at the instant
the row was written. Frames rows are written well below the capture rate --
only alarm, bulk and score-increasing frames get one -- so sampling at write
time would drop exactly the short loud noises worth seeing on a timeline.
AudioDetector accumulates the peak as it decodes and Event::AddFrame takes it
where the row is built, which clears it so each row covers its own interval.
The Event constructor takes and discards it once, otherwise an event's first
row reports the loudest moment since the previous event ended.
This needs no shared memory change: zma is now an offline re-analysis tool and
the live analysis runs in a thread of zmc, alongside the capture thread that
runs the decoder, so the peak can stay in the AudioDetector. SharedData keeps
its documented 888-byte layout and its fixed offsets.
The frames ajax returns the column, and Score with it. elements in
web/ajax/status.php is a whitelist that never listed Score, which is why the
event view's cue strip has been reading an undefined Score off every frame.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JpiSWBmtQkR5bcgpHWY4ME
scripts/ZoneMinder/lib/ZoneMinder/Control/ held both ONVIF.pm and onvif.pm, the
only pair of paths in the tree differing solely in case. On a case insensitive
filesystem git can materialise just one of them, so a macOS clone reports the
loser as modified forever and git add -A commits one module's contents over the
other.
The runtime consequence is worse than the checkout noise. Every seeded Controls
row uses Protocol='ONVIF', and zmcontrol.pl builds the module name from that
column. Where onvif.pm is the file that survived, require ZoneMinder::Control::ONVIF
still succeeds because the filename matches, but it defines the lowercase package,
so the bless lands in an empty ::ONVIF and the first method call dies.
ONVIF.pm has replaced onvif.pm since the unified module landed. Of the subs only
onvif.pm defines, all but horizontalPatrol and horizontalPatrolStop exist there
under underscore-prefixed names, and every ONVIF-protocol Controls row has
CanAutoScan=0, so nothing can reach those two.
Nothing ever moved existing installs onto the new protocol name, so add
zm_update-1.39.29.sql to do it. zm_update-1.35.23.sql sent Protocol='onvif' to
FoscamCGI, which was right in 2021 when onvif.pm held the Foscam CGI protocol and
was copied to FoscamCGI.pm, but onvif.pm was afterwards replaced with a real ONVIF
implementation and the seed row restored, so rows written since belong on ONVIF.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Alarm turns a light on when motion starts; nothing turned it back off. The
existing EventEnd is not the answer: the alarm ends when the monitor leaves the
alert state, while the event carries on recording for the rest of its section,
which on a continuous-recording monitor is up to ten minutes later.
Analyse() leaves the alarm condition by three paths and all three fire it: the
normal ALERT to IDLE, a signal change, and the trigger being turned off. The
abnormal two matter most - a light switched on by an alarm must not stay on
because the camera lost signal.
That needs the firing to be paired, so alarm_actions_fired is set when Alarm
fires and cleared by EndAlarmActions. It guards both directions:
- the three call sites can run back to back, so without it one alarm could
fire AlarmEnd several times
- signal loss and trigger off also run on monitors that never alarmed, and an
unpaired AlarmEnd would switch a light off for an alarm that never happened
- the ALERT to ALARM re-trigger inside one incident deliberately does not
re-fire Alarm, so a single AlarmEnd still has to balance it
The migration MODIFYs the enum and appends the value, so stored rows keep their
meaning; appending to an enum does not renumber what is already there.
Tested: 8 assertions over the pairing rule, run standalone because the real
class needs a database and a camera - the ordinary pair, repeated calls firing
once, an unpaired call firing nothing, two alarms pairing independently, and
the re-trigger case. The action test now also requires every trigger name to
round trip through the TriggerOn enum, so an action saved by the editor cannot
load with a trigger the C++ does not recognise and fire at the wrong moment.
Build clean at 1.39.33, Perl 15 files / 249 assertions, ESLint 0 problems,
tests/js 154 assertions, php -l clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTCzCbvGt8xKQRNCSA7o8
These cameras have a white light that ONVIF does not expose at all: imaging
offers only Brightness/ColorSaturation/Contrast/Sharpness, GetRelayOutputs is
empty with RelayOutputs="0", and there is no PTZ service and so no auxiliary
commands. The light lives behind a vendor JSON-RPC tunnel.
The vocabulary turns out to be Dahua's -- CoaxialControlIO for the light,
magicBox for reboot -- so this subclasses Dahua_RPC and replaces only open,
rpc_call and login. lightOn, lightOff, lightStatus and reboot are inherited
unchanged, and Dahua_RPC's lightStatus already parses the
params.status.WhiteLight these cameras return.
The transport is the whole of the work, and none of it is discoverable by
probing:
* The endpoint is /Onvif/device_service with a capital O. The lowercase
/onvif/device_service is the real ONVIF SOAP service; the capitalised
spelling is a separate vendor tunnel sharing the path. /RPC2, /RPC3,
/OutsideCmd and every /cgi-bin/* return 404 on this model, which is why
the API looked absent.
* Requests are <body><cmdType>T</cmdType><cmd>P</cmd></body>. Without
cmdType the camera drops the connection rather than returning an error.
* P is base64 of the JSON-RPC object, XOR-masked with a session key once one
exists. Masking is mandatory: an unmasked Request is refused even on a
freshly authenticated session, so the key exchange is not optional.
* That key needs a three step bootstrap -- log in, fetch the device RSA
public key over the unauthenticated OutsideCmd channel, then send an AES
key encrypted under it and decrypt the mask key from the reply. The salt
doubles as the AES key and must be 32 decimal digits; 16 is accepted by
the AES step and then rejected by getGeneralKey.
IO polarity matches Dahua_RPC: IO 1 is on, IO 2 is off. Worth stating because
the camera's own web UI is a toggle that does not reveal which is which, and
because a floodlight cannot be verified optically in daylight -- an RTSP
brightness comparison appeared to say the opposite and was exposure drift.
CoaxialControlIO.getStatus is the reliable witness.
Tested: 26 assertions over the pure transport helpers -- mask symmetry and
cycling, NUL bytes surviving the mask, the envelope, stripping the CRLF the
device pads its replies with, and the channel routing, which fails silently on
the wire if wrong. Verified live against both units: open negotiates a 32 byte
mask key and lightStatus reports Off -> On -> Off around lightOn/lightOff. Perl
suite 15 files / 249 assertions pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTCzCbvGt8xKQRNCSA7o8
ZoneMinder has carried audio for years without ever listening to it: the
packets go into the event and nothing reads them. AudioDetector decodes
them in the capture thread and reports a 0-100 level, which Analyse()
turns into score alongside motion, ONVIF and Amcrest.
The level is dBFS-derived, not a raw amplitude ratio. Linear RMS is
unusable as a setting: ordinary speech sits at 1-3% of full scale, so
every sound worth catching would be crammed into the bottom two points of
the range and no operator could tune it. Mapping -60..0 dBFS onto 0..100
puts speech around 43 instead.
Three columns on Monitors: AudioDetection to enable it, AudioThreshold
for the level to alarm at, and AudioAlarmScore for what it contributes.
AudioAlarmScore defaults to 9, matching what an ONVIF or Amcrest alarm
already adds. A threshold of 0 means off, so enabling detection without
choosing a threshold cannot alarm on silence -- a plain level >= threshold
test would alarm on every packet in that state.
The level and the alarm flag go in SharedData's two spare bytes, renamed
from reserved1/reserved2. Offsets and sizes are unchanged, so the
888-byte cross-process layout and the Memory.pm and Monitor.php offset
tables all still agree; the readers are renamed in the same commit so the
level is available to the web UI.
The decoder is opened lazily on the first audio packet rather than at
camera setup, so a monitor with detection off never carries one and a
stream that gains audio on reconnect still gets picked up. Scoring reads
the capture thread's most recent level rather than scoring per audio
packet, because the score belongs to a video frame and audio packets do
not arrive in step with them.
Tested: 242 assertions over the pure helpers - the RMS of both sample
formats, the dB scale's monotonicity and endpoints, full-scale clamping
of decoder overshoot, and the threshold-0 case. The repo's Catch2 harness
needs v3 and only v2 is installed here, so tests/zm_audio_detector.cpp
was compiled against a v2 shim to check it, and an identical set of
assertions was run standalone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTCzCbvGt8xKQRNCSA7o8
A camera that detects motion should be able to sound a speaker, and the
speaker is rarely the camera: it is a separate device with its own address,
credentials, stream and Controls entry. Model it as a monitor and let a
monitor's alarm drive actions on other monitors.
Add Monitors.DeviceClass enum('Camera','Speaker'). This is what the device
is, as distinct from Type, which selects the capture backend - an IP speaker
still captures over Ffmpeg like any other RTSP device, so Type could not
carry the distinction.
Add the MonitorActions table: MonitorId is the monitor that triggers,
TargetMonitorId the device acted on, and the two are frequently different.
TriggerOn covers EventStart, EventEnd, Alarm and Manual.
Which actions a device is offered is decided by its Controls row - CanLight,
CanIndicatorLight, CanAudioPlay - so a device can only be asked to do what it
has been measured to do. The editor filters on this and the save path
re-checks it, because the request is not to be trusted.
Execution goes straight to the target's zmcontrol socket rather than forking
zmcontrol.pl per action: the daemon already accepts a line of JSON there, and
it is the same path the control panel uses. ActionCommandName maps the DB
enum onto a method name as a whitelist, so nothing out of the database
reaches the control daemon uninspected. Actions are fire-and-forget - a
speaker that is offline is logged and skipped, never allowed to hold up event
handling.
Alarm actions fire only on the genuine entry into alarm, not on the
ALERT->ALARM re-entry, which would re-sound a speaker within one incident.
EventEnd runs on the calling thread before the event is handed to the closing
thread, which does not capture `this`.
Manual actions appear as buttons on the watch page, and are the reason the
control panel is now shown for a monitor that has actions but no control of
its own. Firing one sends only the action id; the command and target are
rebuilt server side, and Control rights are required on the target device and
not merely on the monitor the action hangs off.
Also add --file to zmcontrol.pl, without which audioPlay could not be driven
from the command line. The web path was unaffected as it bypasses GetOptions.
Tests: tests/zm_monitor_action.cpp covers the command whitelist, the message
format (including that file id 0 is a real id and that a stale AudioFile is
never passed to a command that takes none), trigger names, and that every
value of the ActionType enum maps to a command.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTCzCbvGt8xKQRNCSA7o8
(cherry picked from commit b561341e988af69c3a46db854da410309f7cfa82)
An IP speaker is controllable but has none of the capabilities the Controls
table models: nothing moves, focuses or lights up. It plays a sound file it
already holds, selected by a numeric id, and carries an output volume.
Add CanAudioPlay, MinAudioFile, MaxAudioFile and CanAudioVolume to Controls,
with migration zm_update-1.39.21.sql. CanAudioPlay renders one button per
sound id plus a stop button; MinAudioFile/MaxAudioFile bound that range;
CanAudioVolume renders the volume pair. The sound id travels in the command
name (audioPlay12), the way presets already do, because a control button has
no way to attach a separate parameter.
Add ZoneMinder::Control::IPSpeaker driving the device's own HTTP interface,
and a Controls entry for it. Playback goes over the vendor interface rather
than ONVIF because the ONVIF audio output service on this firmware only
describes the output and offers no way to start a stored file.
Measured against a device reporting ONVIF Manufacturer "IPSpeaker" and
firmware CS20-V3.3.45N:
- File ids fall in two windows, 10-14 built-in and 20-30 operator uploads.
Anything else is refused with result -2, and an id in a window with no file
uploaded with result -3, so valid_fileid refuses only the former: an empty
slot is a legitimate id the operator may fill later.
- config=audio.set replaces the whole audio section, so a partial write
reverts the microphone, codec list and echo-cancellation settings. Every
volume change reads the section and echoes it back with the new level.
audio.get also returns outmute, which audio.set rejects, so it is excluded
from the field list.
- The volume reads back and round-trips, but the device's RTSP audio is a
pre-volume tap of the playback signal: it is unchanged with outvolume at 0,
so it cannot confirm the loudspeaker's acoustic output. CanAudioVolume is
set on the strength of the setting persisting, and both the POD and the
seed row say the acoustic effect is unconfirmed.
The seed row covers the built-in window only, because a fresh device has
nothing uploaded and the firmware refuses an empty slot.
Adding columns to Controls extends the 43 positional INSERTs in
db/controls.sql, which carry no column list and so must match the column
count exactly.
Tests: scripts/ZoneMinder/t/ip_speaker.t, 32 assertions covering file id
windows, volume clamping and stepping, request building and the whole-section
write. All pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvTCzCbvGt8xKQRNCSA7o8
(cherry picked from commit d07e552a3c58373f7f0ccebf9aa28adb4d51baa3)
An audit of every table in zm_create.sql.in against the queries that read it
turned up seven indexes that are either never used or duplicate another index.
Logs.TimeKey duplicates Logs_TimeKey_idx, which zm_create.sql.in creates three
lines below it and zm_update-1.31.11.sql adds to upgraded installs. Both have
existed on every install since, and every log INSERT maintains both. Logs is
the second highest insert-rate table in the schema.
Stats.MonitorId and Stats.ZoneId are never read. Every query against Stats is
anchored on EventId: the per-frame zone stats view in skins functions.php and
ajax/stats.php, the ZoneId filter term in FilterTerm.php and Filter.pm, the
deletes in Event.php, Event.pm and zmaudit.pl, and the orphan scan. EventId_ZoneId
serves all of them. A Stats row is written per frame per zone when
ZM_RECORD_EVENT_STATS is on, which is the same insert-rate argument as the
Frames indexes in the previous commit.
EncoderTemplates.Encoder is the leftmost prefix of Encoder_Name, and the RoleId
indexes on Role_Groups_Permissions and Role_Monitors_Permissions are the
leftmost prefix of the UNIQUE index beside them. Those tables are small and
rarely written, so this is tidying rather than a saving.
Monitor_Status_UpdatedOn_idx is dropped by zm_update-1.37.76.sql but was left in
zm_create.sql.in, so fresh installs have carried it and upgraded installs have
not. Removed from zm_create.sql.in and repeated in the migration so the two
agree from here on.
Stats.MonitorId and Stats.ZoneId are the only two with no other index covering
their column, so their drop is guarded on there being no foreign key on the
column. 1.37.31 drops the Stats foreign keys only when they are named
Stats_ibfk_1..4; on an install where they survived under another name the drop
would fail with errno 150 and abort the upgrade, so it skips with a message
instead. Verified by building that case and confirming the migration completes.
Tested against MySQL 8.4: the schema built from master's zm_create.sql.in, with
1.39.27 and 1.39.28 applied, is index for index identical to a fresh install
from the updated zm_create.sql.in - 123 indexes across 55 tables. Both
migrations verified idempotent, and all 29 foreign keys survive.
Not changed: AI_Detections carries three single-column indexes and has no
consumer anywhere in the tree yet, so there is nothing to judge them against.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every query against Frames is anchored on EventId, and nearly all of them order
or range by FrameId within the event: the full-event reads in zm_eventstream.cpp
and zma.cpp, the prev/next bulk frame LIMIT 1 lookups in includes/Event.php and
ajax/status.php, zmfilter.pl's per-type scan, and the per-event aggregates in
Event.pm and zmaudit.pl.
A plain EventId index left all of those to a filesort. For the LIMIT 1 lookups
on the playback path that meant reading and sorting an entire event's frames to
return one row. Measured on a 200 event x 500 frame table, the prev-frame query
goes from "ref EventId_idx, rows 500, Using filesort" to "range
EventId_FrameId_idx, Using index condition, Backward index scan", the full-event
ORDER BY FrameId loses its filesort, and max(FrameId) WHERE EventId becomes
"Select tables optimized away".
Nothing filters or sorts on Type or TimeStamp without EventId, so neither index
was ever used for reading. Type is a three-value enum and cannot be selective in
any case. Both only cost insert time on the highest-insert-rate table in the
schema, plus space, plus work on every DELETE ... WHERE EventId.
The migration adds the composite index before dropping EventId_idx so that its
leftmost prefix covers EventId for any install still carrying the foreign key on
Frames.EventId that 1.35.11 added and 1.37.31 drops only when it is named
Frames_ibfk_1. Verified against a schema with that foreign key present: the drop
succeeds and the constraint survives. Verified idempotent by running it twice.
The index is left non-unique. FrameId is unique within an event, but a UNIQUE
constraint would turn a duplicate-id bug into frames dropped mid-recording
rather than a log line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Upstream's schema-drift fix landed db/zm_update-1.39.25.sql and bumped
version.txt to 1.39.25 while the trigger consolidation was using the same
number for db/zm_update-1.39.25.sql.in.
Git does not see this as a conflict -- the two files have different names in
the source tree -- but configure_file generates the .sql.in into
zm_update-1.39.25.sql, which is the same install target as the tracked .sql.
Confirmed by staging an install off the merge: the installed
zm_update-1.39.25.sql was upstream's column reconciliation and the trigger
consolidation was simply absent, with no error anywhere. An upgrade would
have applied one of the two and silently skipped the other.
Renumbered to 1.39.26, version.txt to match, and the two zmstats.pl.in
comments that name the migration updated. Upstream's 1.39.25 is untouched.
Verified by staging an install again: both files are now present and hold
what they should. The migration was re-run end to end at the new number
against a scratch database -- drops the eight cascade triggers, leaves four,
and the 29 trigger assertions pass afterwards.
ai_server carries the old 1.39.25 numbering and will need the same treatment
when master is next merged into it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Y6FieTwEXuLhhR4e2yiax
The only Foscam entries we ship are for the pan/tilt models, so a fixed
camera of the same generation - an FI9853EP, say - has nothing it can
be set to. That matters beyond the missing PTZ buttons: without a
ControlId a monitor has no protocol module at all, so cameratool.pl
cannot reach the camera to read or write its settings, including
pointing it at an NTP server.
Every movement flag is left at its default of 0 because the hardware
has no PTZ. CanReboot is 1 because rebootSystem is accepted, verified
on an FI9853EP running firmware 2.22.2.15.
The row goes in controls.sql for fresh installs and in the update for
1.39.24, which is the current unreleased version, for existing ones.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011BJjpSYbZRM8ucGW9HbgtR
The migration is produced by configure_file into the build directory, and
files that only exist there are not covered by the "*.sql" glob over the
source directory that installs the rest. Every other .sql.in migration has
a matching install(FILES) rule for exactly this reason; 1.39.25 was added
without one.
zmupdate.pl scans the installed directory and runs
zm_update-<version>.sql from there, so the effect was that the trigger
consolidation would never be applied on upgrade, silently -- there is no
error for a migration file that is not present, the version is just
skipped.
Verified by staging an install into a DESTDIR: all seven generated
migrations land with this rule, and 1.39.25 is absent without it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Y6FieTwEXuLhhR4e2yiax
Events_Hour, Events_Day, Events_Week and Events_Month each carried their
own update and delete trigger, and every one issued a separate UPDATE
against the same Event_Summaries row. A single Event delete therefore
locked that row five times: once from event_delete_trigger and once from
each bucket's cascade. That is the deadlock zmstats.pl hits when it bulk
deletes aged rows out of Events_Hour while zmc and zma are writing.
event_update_trigger and event_delete_trigger now modify the bucket tables
themselves and apply one consolidated UPDATE, using ROW_COUNT() after each
bucket statement to tell whether the event was still in that bucket so
aged-out events do not over-adjust the counters. Measured on a scratch
database, an Event delete goes from 5 Event_Summaries row updates to 1.
This also fixes a drift bug. The old event_update_trigger kept
ArchivedEventDiskSpace correct only in a branch that cannot be reached: it
sits under IF (NEW.Archived != OLD.Archived) and requires both to be
false. The branch that does run when an already-archived event grows
updated Events_Archived but not Event_Summaries, so the archived total
drifted for the life of the install. Reproduced on a scratch database:
after archiving a 250-byte event and growing it to 999,
ArchivedEventDiskSpace still read 100.
BEHAVIOUR CHANGE. A direct DELETE against a bucket table no longer adjusts
Event_Summaries at all, and that is how zmstats.pl prunes. zmstats.pl
already resyncs HourEvents/DayEvents/WeekEvents/MonthEvents and their disk
space columns from COUNT(*)/SUM(DiskSpace) on any pass where it pruned, so
those four pairs become eventually consistent within one
ZM_STATS_UPDATE_INTERVAL instead of exact at every instant. The Total and
Archived columns stay exact, because only the Events triggers touch them.
The zmstats.pl comments are updated to describe the new arrangement; its
code is unchanged.
Migration is db/zm_update-1.39.25.sql.in: drop the eight cascade triggers,
resync Event_Summaries from the events themselves so the new triggers start
from ground truth and the archived drift above is repaired, then source
db/triggers.sql. version.txt goes to 1.39.25. No schema change, so
zm_create.sql.in needs no edit -- it already sources triggers.sql.
Ported from the ai_server branch, where this migration sits at 1.39.7, a
number master passed long ago and an existing install would never run.
Repackaged above master's tip. The ai_server version also unwinds a
views-and-SWR experiment that only ever existed on that branch, which is
dropped here as a no-op on master, and carries an unrelated START_DELAY
change, which is not taken.
Tests: tests/perl/test_event_summaries_triggers.pl, 29 assertions against a
real server, skipped when no scratch database is configured. Verified they
fail on the current triggers, on exactly the two claims above: the archived
drift, and 5 row updates per delete. Also verified the migration repairs a
drifted install, is idempotent across a second run, and leaves a migrated
install with byte-identical triggers to a fresh one. Generated zmstats.pl
passes perl -Tc.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Y6FieTwEXuLhhR4e2yiax
Coords holds a space-separated list of "x,y" points, written with two
decimal places by pointsToCoords()/mapCoords(). Since 1.39.2 the Zones
values are percentages rather than pixels, which costs up to 13 bytes per
point ("100.00,100.00") against the 9 a pixel pair used ("1920,1080").
tinytext caps at 255 bytes, so a zone of roughly 19 points or more
overflows and is silently truncated, leaving a corrupt polygon. In pixel
form the same zone fit, so this only starts biting at the conversion.
Widen the column in 1.39.2 before its conversion runs, and again in a new
1.39.23 for installations that ran that conversion before the widening
existed - zmupdate.pl only applies files at or above the database version,
so those never revisit 1.39.2. ALTER ... MODIFY to the same type is a
no-op, so both are safe to re-run.
Maps stores coordinates in the same format and is widened to match. It
was added to zm_create.sql.in in 1.32 without a matching update script,
so installations older than that have never had the table; guard that
statement on the table existing rather than failing the whole migration.
Update zm_create.sql.in for both tables so a fresh install matches an
upgraded one, and bump version.txt so the new update script is reached.
The Coords pixel-to-percent conversion is already idempotent: the cursor
loop skips any zone whose Coords contain a '.', and CAST(... AS
DECIMAL(10,2)) always renders two decimals, so a converted zone can never
be mistaken for a pixel one.
The Area rescale that followed it had no such guard. It ran as a separate
statement over every zone with Area > 0, with nothing to distinguish a
zone converted by this run from one converted by an earlier one, so each
run divided Area by (Width * Height) / 10000 again. Re-running does not
repair it.
This is not hypothetical. zmupdate.pl applies every update file whose
version is >= the database version, so the file matching the current
version is re-run on each upgrade. A live database had a full-frame zone
on a 1920x1080 monitor holding Area 48, which is
ROUND(10000 * 10000.0 / 2073600) - the rescale applied to an already
rescaled value. Two sibling zones had reached 0. Their Coords were
intact throughout, confirming the conversion held while Area decayed.
Fold the rescale into the UPDATE inside the loop so it can only touch a
zone whose Coords were just converted. Zones skipped as already-percent
keep their Area.
Area is recoverable without a backup: web/includes/actions/zone.php
recomputes it with getPolyArea() from Coords whenever a zone is saved.
A DateTime lower bound was written as
COALESCE(E.EndDateTime, '9999-12-31 23:59:59') >= T1
reading "an event that has not ended yet never ends". EndDateTime is NULL for
an event that is still recording, but also for one zmc was killed part way
through, and that stays NULL forever - so every abandoned event matched every
window from then on. Wrapping the column in a function also meant no index
could range over it: with the only MonitorId key being MonitorId alone, every
montage review request read all of that monitor's events and filtered them in
memory, however narrow the window.
Length tells the two cases apart. It is flushed during recording, so a live
event's effective end keeps advancing while an abandoned one's is frozen at
whatever was recorded. Use the same CASE the SELECT list in ajax/events.php
already uses, and bound it below by StartDateTime.
The lower bound now emits three conjuncts:
E.StartDateTime >= DATE_SUB(T1, INTERVAL 1 DAY)
AND (E.EndDateTime IS NULL OR E.EndDateTime >= T1)
AND <effective end> >= T1
The floor bounds the scan for a window in the past and drops events abandoned
more than a day earlier - events do not outlive the nightly logrotate SIGHUP,
which stops and restarts them, so a day is comfortably beyond any real event.
The middle conjunct is implied by the third and exists only to give the
optimiser a second indexable handle: it is narrow exactly when the floor is
wide, so between them a window at either end of the retention period has
something cheap to range over. Verified the optimiser picks correctly for both,
unprompted. The third is the residual that gets the semantics right.
Add the two composite keys those conjuncts need. Two range columns cannot both
narrow one B-tree, and these are mirror images - EndDateTime >= T1 is
open-ended upwards, StartDateTime <= T2 downwards:
Events_MonitorId_StartDateTime_idx (MonitorId, StartDateTime)
Events_EndDateTime_MonitorId_idx (EndDateTime, MonitorId)
MonitorId leads the first because it is the equality; past a range column later
columns can no longer narrow the scan, and (StartDateTime, MonitorId) measured
3.5x slower for the same rows. EndDateTime leads the second so it also covers
zmaudit's hunt for events that were never closed, which has no monitor to scope
it - 9 rows with a key against a 22,948 row table scan without.
Both replacements are added before the keys they supersede are dropped:
Events_MonitorId_idx (MonitorId) is now a leftmost prefix of the new key.
Events_EndDateTime_DiskSpace (EndDateTime, DiskSpace) existed for the scan
that hunted events with no DiskSpace set; DiskSpace is set when the event is
finalised in C++, so nothing scans for it any more.
Net index count on Events is unchanged.
Measured on a 7,167 event monitor. Default one hour window: 7,055 rows
examined / 25.4ms -> 173 rows / 4ms. Window scrubbed a week back, which
persists in the zmFilter_StartDateTime cookie: 6,995 rows / 28ms -> 19 rows /
0.15ms.
Adds migration db/zm_update-1.39.22.sql. Extends t/filter_sql.t; the PHP and Perl were
checked to emit identical SQL, and the semantics checked against a table of
seven event shapes - abandoned long ago, abandoned recently, overlapping,
inside, before, still recording, and still recording for 17 hours.
Events_Lock holds advisory locks over events, so that two filters do not work
on the same event at the same time. Claiming an event is a single autocommitted
INSERT of one row, which means no InnoDB lock is held while the filter actually
works on the event.
No foreign key to Events on purpose: it would make every event delete take a
lock in this table, which is the coupling this exists to avoid. Rows left
behind for deleted events are harmless and expire.
ZM_FILTER_LOCK_TIMEOUT (default 3600) is how long a claim lasts. A filter that
is killed part way through an event cannot release anything, so claims have to
expire on their own. It needs to be longer than the slowest thing a filter does
to a single event, which is why it is configurable rather than fixed.
Adds migration db/zm_update-1.39.21.sql.
Three values in the entry added a few commits ago were wrong, all for the
same reason: I read them off http status codes, in the entry whose own
comment says http status codes cannot be trusted here. Re-derived with
amcrest_probe.pl, which decides every flag by comparing frames.
CanMoveDiag was 1 because LeftUp returns 200. It shifts the picture by
1.8 against a 6.0 threshold, so nothing moves. Now 0.
Pan and tilt speed were 1..8 from the Dahua documentation. Measured
displacement is 31.5, 36.2, 41.4 for speeds 1, 2, 4 and then flat through
8, 16, 32 and 64, so the usable range is 1..4 even though the firmware
accepts far higher.
NumPresets was 255, the firmware ceiling. That is the wrong thing to
measure: the classic skin draws one button per preset, and the camera
refuses GotoPreset for any index not yet stored, so 255 gives a wall of
buttons where all but the defined ones error. Now 25, matching the
sibling Dahua/Amcrest RPC entries.
CanMoveAbs stays 0 but for a better reason than before. PositionABS is
accepted, reaches distinct positions and reproduces a revisited
coordinate, so it passes the obvious tests. Its targets are 90 degrees
apart though, and the median step shifts the view 16.4 where a single one
second nudge shifts it 42.5 - the pan argument is being clamped to a
fraction of what was asked, so a moveMap click would not land where it
was aimed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LzHrvNtF13vyYBAJjktam6
The 49 control protocol definitions were sitting inline in
zm_create.sql.in, which makes adding a model-specific entry a scroll
through a wall of positional INSERTs. Move them to db/controls.sql and
pull them in with
source @PKGDATADIR@/db/controls.sql
which is how User_Preferences.sql, manufacturers.sql, models.sql,
triggers.sql, Object_Types.sql, AI_Models.sql and coco_dataset.sql are
already handled, and the existing install(FILES ${dbfileslist}) glob puts
the new file where that path resolves to. The 49 rows move across
byte-identical.
The new entry is the Amcrest IP5M-1190EW, verified against firmware
2.810.00AC004.0.R. The generic 'Amcrest HTTP API' entry advertises zoom
and continuous zoom and no presets, which is backwards for this model:
pan, tilt, the diagonals and continuous move all physically move it, arg2
really is the speed, and GotoPreset works, while zoom, focus and iris do
not. ZoomTele, ZoomWide, FocusNear and FocusFar answer OK and do
nothing; autoFocus, getFocusStatus, IrisLarge and AutoPanOn 400.
PositionABS answers OK and is inert, so CanMoveAbs and CanMoveMap stay 0.
The white light exists in Lighting_V2 but is not drivable, so CanLight
stays 0.
Worth recording for anyone rechecking this: ptz.cgi?action=getStatus on
that firmware always reports Postion=0/16.65 MoveStatus=Idle no matter
what the camera is doing, so status cannot tell you whether a move
worked. All of the above was confirmed by diffing snapshots.
zm_update-1.39.20.sql carries the same row to existing installs, guarded
with NOT EXISTS so it can be re-run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LzHrvNtF13vyYBAJjktam6
Nothing enforced uniqueness, but every reader already assumed one row per
name: User::Preferences() hashes the rows by Name, montagereview.php and
User_Preference::find_one() take the first match, and TagOrder::load()
reads a single Value. With duplicates present those pick an arbitrary
row, so a write could land on one row while reads returned another --
the tag order would appear to stop updating.
Replace the UserId-only index with UNIQUE (UserId, Name) and make Name
NOT NULL. The unique index still serves UserId-only lookups and the
UserId foreign key as a leftmost prefix, so the old index is redundant
and is dropped. Name has to be NOT NULL for the constraint to mean
anything, since a unique index permits any number of NULLs; a NULL-named
row is unreachable regardless, as a preference is only ever looked up by
name.
zm_update-1.39.19.sql discards NULL-named rows, makes the column NOT
NULL, collapses existing duplicates keeping the highest Id (the most
recently inserted), adds the unique index, then drops the old one. The
index is added before the old one is dropped so the foreign key is never
left without a usable index. Each schema change is guarded against
INFORMATION_SCHEMA so re-running is a no-op. db/User_Preferences.sql
gets the same shape for fresh installs.
With the constraint in place, TagOrder::recordUsage() replaces its
SELECT-then-INSERT-or-UPDATE with a single INSERT ... ON DUPLICATE KEY
UPDATE. That drops a query and closes the race where two concurrent tag
additions by the same user both see no row and both insert.
Verified on MariaDB 11.8.8 against a seeded table: duplicate rows
collapse to the expected survivors, NULL-named rows are removed, a
second run is a clean no-op, NULL/duplicate/bad-UserId inserts are
rejected (1048/1062/1452, so the foreign key survives losing its index),
and mysqldump of a fresh install matches a migrated database exactly.
Bump version.txt and the redhat spec Version to 1.39.19.
Server.php's ReadStats() query (SELECT ... WHERE ServerId=? ORDER BY
TimeStamp DESC LIMIT 1) had no supporting index, forcing a full table
scan on every page load that shows server stats. Add a composite
(ServerId, TimeStamp) index via zm_update-1.39.18.sql (idempotent,
checked against INFORMATION_SCHEMA.STATISTICS) and zm_create.sql.in
for fresh installs. Keep the existing TimeStamp-only index, since
zmstats.pl's prune DELETE filters by TimeStamp alone.
Also query ReadStats() with the server's actual Id() instead of
coercing Id() <= 1 to 0.
Bump version.txt and the redhat spec Version to 1.39.18.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q7jeoG8ptXRpXxif2JqkD7
Add the schema backing per-monitor object detection and the AI dataset/
model/class management UI:
- Monitors gains AnalysisImageOpacity and ObjectDetection,
ObjectDetectionModel, ObjectDetectionObjectThreshold,
ObjectDetectionNMSThreshold columns.
- New tables AI_Datasets, AI_Models, AI_Object_Classes,
AI_Detection_Settings and AI_Detections.
- Seed the COCO 2017 dataset (80 classes) and default per-class
detection settings via db/coco_dataset.sql.
Existing installs get zm_update-1.39.17.sql, which is idempotent and
adds the columns with their final VARCHAR(16) ObjectDetection shape
directly (no enum-churn intermediates). Fresh installs create the same
objects from zm_create.sql.in sourcing AI_Models.sql and
coco_dataset.sql. Both paths were verified to produce identical schema.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port the menu customization work from the ai_server branch onto master:
- Add/edit/delete custom navbar/sidebar menu entries via Options > Menu
- Per-entry Link column (new Menu_Items.Link) with ?view= fallback derived
the same way as built-in items; custom entries render via buildMenuItem
- Live icon preview (material/font awesome) with fixed-width preview cell
- Per-row delete icon; add/reset buttons with tooltips
- Surface DB errors when saving options config: capture the swallowed PDO
error via dbLastError() and show it in the page error banner instead of
redirecting past it
Menu_Items.Link is added to zm_create.sql.in and as an idempotent ALTER in
zm_update-1.39.16.sql.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Front Culdesac (a CMIP3CD42WI-28AISP ColorVu camera) exposes the same
colorVuWhiteLight supplement-light interface as the CMIP1342WE-28MDA but was
assigned a non-HikVision control, so it showed no Light button. Add a
model-specific Controls row (HikVision protocol, CanLight only; fixed camera
with no PTZ/focus/iris) in zm_create.sql.in and migration 1.39.16.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add lightOn/lightOff/lightStatus to the HikVision control module, driving the
camera's ISAPI supplement light (ISAPI/Image/channels/<n>/supplementLight).
"On" selects the white light where the model has one (colorVuWhiteLight), else
IR; "off" restores the camera's smart/auto default (eventIntelligence) so night
IR keeps working. The methods GET the current document and rewrite only
<supplementLightMode>, preserving the std-cgi namespace and the sibling
brightness fields the firmware rejects PUTs without. Mode selection is driven by
the model's advertised supplementLight/capabilities. lightStatus returns
{ WhiteLight => 'On'|'Off'|undef } in the shape the existing web toggle consumes,
so no web changes are needed (the CanLight UI is already generic).
Add a model-specific Controls row for the LTS CMIP1342WE-28MDA (a fixed ColorVu
camera: white light, reboot, no PTZ/focus/iris), in zm_create.sql.in and
migration zm_update-1.39.16.sql. Bump version to 1.39.16.
The pure mode-selection/XML helpers are covered by t/hikvision_light.t. Verified
live against a CMIP1342WE-28MDA: the GET-modify-PUT round-trip turns the white
light on and restores the prior mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The CanLight and CanIndicatorLight columns were added to the Controls table
(between MaxWhiteSpeed and HasPresets) without updating the legacy positional
INSERT ... VALUES (NULL, ...) seed rows. All 43 of those rows were left two
values short, so a fresh `mysql < zm_create.sql` load failed with ERROR 1136
(Column count doesn't match value count).
Splice CanLight=0,CanIndicatorLight=0 into each positional row at the matching
column position. None of these legacy models support either capability, so 0,0
is correct; existing installs are unaffected (the columns were added there by
the 1.39.13/1.39.14 ALTERs with the same default). Named-column rows already
specified the values and are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a control protocol module for cameras built on the HiSilicon
Hi3510 SoC which expose cgi-bin/hi3510/ptzctrl.cgi rather than the
older Foscam decoder_control.cgi interface. Contributed by Turgut
Kalfaoglu on the forums, tested on a Tenvis TH661.
Supports continuous pan/tilt with auto-stop, emulated diagonals,
presets 1-8, and horizontal/vertical patrol via presets 9/10.
Credential and host parsing uses the Control.pm guess_credentials
helper; the camera-tested wire format (usr/pwd query parameters)
is preserved.
Adds the Controls table row to zm_create.sql.in and an idempotent
zm_update-1.39.15.sql migration, and bumps version to 1.39.15.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a CanIndicatorLight capability and status-aware Indicator toggle button. The indicator LED on the ASH21-B, ADC2W and ASH42-B is controlled via the LightGlobal config (configManager get/setConfig); add indicatorLightOn/Off/Status to Dahua_RPC and a model-specific 'Amcrest ASH42-B RPC' Controls row, with the capability also enabled on the ASH21-B/ADC2W/generic rows. Migration zm_update-1.39.13.sql adds the column.
Add Dahua_RPC keepAlive (global.keepAlive) wired into a 30s zmcontrol idle tick, plus session-expiry re-login retry in set_config and the status queries, so the long-running control daemon does not silently fail after the ~60s session timeout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a CanLight control capability rendering a single status-aware Light toggle button. The ADC2W white light is driven via CoaxialControlIO.control (Type 1, numeric IO); the button queries live state and reflects it (amber when on).
To get device state to the browser, add an opt-in two-way response path to the control protocol: zmcontrol writes a JSON result back only when a request sets wants_response (fire-and-forget commands unchanged, SIGPIPE-safe); Monitor::sendControlCommandWithResponse and ajax/control.php return it.
Also adds get_config/set_config/probe to Dahua_RPC for characterising cameras, the CanLight column (migration zm_update-1.39.12.sql), edit-UI checkbox, and a config unit test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add ZoneMinder::Control::Dahua_RPC, a PTZ control module driving Amcrest Smart Home (ASH21/ASH42/ADC2W) and Dahua cameras over the JSON-RPC /RPC2 interface, since their cgi-bin API is disabled and ONVIF exposes no PTZ service. Two-stage MD5-challenge login with session reuse and self-healing re-login; continuous pan/tilt + diagonals, stop, presets, zoom, focus, reboot.
Adds a generic 'Dahua/Amcrest RPC' Controls row plus model-specific rows for the ASH21-B (pan/tilt only) and ADC2W (reboot only), the ASH21-B/ADC2W models, migration zm_update-1.39.11.sql, and a login-hash unit test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
zmupdate.pl reruns migrations where the file version is >= the current
DB version, so once the DB reaches 1.39.10 the script fires again. The
unconditional ALTER TABLE Logs ADD/DROP INDEX statements failed on the
second run with "Duplicate key name".
Guard both Logs index changes with INFORMATION_SCHEMA.STATISTICS checks
(same pattern used for the Sessions index), and DEALLOCATE PREPARE
after each block so prepared-statement names don't accumulate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous wording named a fictional Event::createNotification function;
the bucket+ES insert sequence in src/zm_event.cpp lives in the Event
constructor (line 46). Point future readers at the right symbol so they
can grep and verify the path.
Adds zm_event.cpp's create path (Events INSERT -> bucket INSERTs ->
Event_Summaries UPDATE; event_insert_trigger itself is commented out so
zmc does this directly), and lists zmaudit.pl alongside zmstats.pl, so
the comment enumerates every writer obligated to follow the canonical
order rather than just the trigger paths.
The session garbage collector ran DELETE FROM Sessions WHERE access < ?
against an unindexed column, forcing a full table scan and taking gap
locks across the access range. With REPLACE INTO Sessions happening on
every authenticated request, this is a deadlock hotspot.
- Add Sessions_access_idx on Sessions(access) in both fresh-install
schema (zm_create.sql.in) and a migration (zm_update-1.39.10.sql).
- Rewrite ZMSessionHandler::gc to a two-phase delete: SELECT up to 100
expired ids via the new index (consistent read, no locks), then
DELETE WHERE id IN (...) by primary key. InnoDB takes record locks
only on the matched rows, not gap locks on the access range.
- Bump version to 1.39.10 so zmupdate.pl picks up the new migration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The stock preset MinAlarm/MinFilter/MinBlob values were authored for
~320x240 cameras, where 5% of zone area was a 62x62 motion blob.
At 1080p, the same 5% requires a 322x322 blob to trigger — so even
the "Fast, high sensitivity" preset rarely fires on a person walking.
New scale (% of zone area): low=3, medium~0.5, high=0.1.
MaxPixelThreshold (per-pixel grayscale delta, 0-255) is unchanged.
Updates zm_create.sql.in for fresh installs and adds a migration
that rewrites Ids 1-7 on existing installs. Custom presets (Id > 7)
are left alone. Bumps version to 1.39.10.
