8 Commits

Author	SHA1	Message	Date
Tony C	342a7826fc	Pre-release tweaks (2026-06-01) (#395) ... * Added 'ADD NEW' buttons to top of all edit tab panels. * Phase 1 of reviewing and editing user-facing documentation. * Phase 2 of reviewing and editing user-facing documentation. * Added weather API simulators for four weather sources. * Improved weather side panel layout. * Tweaks to weather simulators: random forecasting values. * Fixed regression: attribute rendering missing management forms. * Fixed test timing issues. * Updated all screenshots. Document reference updates are WIP. * Finished adding new screenshots and document link updates. * Fixed javascript wheel handling bug. * Fixed initial SVG rendering viewbox bug. * Fixed edge cases around DB state using new ProvisioningState. * Suppressing tabs for integration config panels. * Improved performance of Sensor/Controller history queries. * Fixed flaky async sensor-response manager test (event loop + cross-connection). * Fixed lint issues (unused import, blank-line spacing).	2026-06-03 16:03:06 -05:00
Tony C	c752b41139	Add Immich integration (#381) + picker error-channel refactor (#383) ... * First version of the Immich integration. * Comment cleanup. * Fixes from code review recommendations.	2026-05-30 18:59:23 -05:00
Tony C	1a6a704875	Pre-release tweaks across simulator, attributes, weather, Frigate, naming (#376) ... * Simulator improvements: profile-load lifecycle, cache hygiene, naming Profile load lifecycle: - Drop _add_zm_server from the ZM seed builders. The ZmServer was being persisted by both the seed and the simulator's on-demand auto-create, doubling up under a hydration race. - Add ServiceSimulator.post_load_hook() — a deterministic lifecycle point invoked once per profile load after hydration. ZoneMinderSimulator overrides it to materialize the singleton ZM Server so freshly-created and freshly-switched profiles always show it. - Take the manager's data lock for the duration of _load_entities_for_simulator's clear+rehydrate, closing the race where a concurrent add_sim_entity could see an empty map mid-hydration and persist a duplicate singleton. The initialize() flow releases the lock before the cross-thread sync_to_async hop so the existing once-only _initialized guard still works without deadlocking. Cache hygiene: - Add NoStoreMiddleware (in both hi.middleware and hi.simulator.middleware so the apps stay independent) that sets Cache-Control: no-store on HTML and JSON responses lacking an explicit directive. Without this, browsers apply heuristic caching and may serve a stale page when the server is down — making controls appear functional while every AJAX call silently fails. Static assets and views with explicit Cache-Control are not touched. Naming: - ZM 'volume' profile cameras renamed 'Volume Camera NN' → 'ZM Camera NN'. - Frigate 'volume' profile cameras renamed 'Volume Camera NN' → 'Frigate Camera NN', with the wire-format camera_name tracking the rename so diagnostics stay coherent. * HA camera snapshot regression: strip substate suffix in state-id map Cameras stopped showing the polled snapshot in the live view after the motion_detection substate was added: cameras went through the multi-substate decomposition path and their Sensor rows now carry '~state' / '~motion_detection' integration_names instead of the bare 'camera.X'. The HassManager's entity_id->ha_state_id map was returning those suffixed names, but the latest-attrs cache is keyed by the bare HA state id, so the lookup missed and get_entity_video_snapshot returned None — the template fell through to the placeholder branch. Strip the suffix when building the map so the lookup hits. Multiple substate sensors for the same camera collapse to the same bare key by construction. Also added a code comment at the device-grouping short-name match documenting that pairing only spans the current sync pass: an operator who excludes 'camera' on one sync and re-enables it on a later sync gets a standalone motion-sensor orphan that the newly-arriving camera does not absorb. Rare power-user scenario; documented for the next reader rather than fixed with cross-pass absorption logic. * Fix two attribute restore regressions Restore-default icon (single attribute): _restoreTextarea updated the textarea and hidden field but left the read-mode div untouched. With the read-mode linkified rendering introduced more recently, the operator sees no visible change after Restore — the form's dirty state ticks over but the displayed value doesn't until they hit Save. _restoreTextarea now also overwrites the read-mode content div with the restored value as escaped plain text. Server-side linkification returns on the next save. Restore-from-history (config settings, multi-attribute page): SubsystemAttributeRestoreInlineView routes through <int:subsystem_id> while the initial ConfigSettingsView routes through a re_path \d+ capture (string). The template's tab-pane 'show active' comparison selected_subsystem_id == owner.id|stringformat:'s' succeeded for the initial render (string==string) and silently failed for the restore response (int==string is False in Django templates). The response replaced #attr-v2-content-subsystem correctly, but every tab-pane rendered without 'show active', so Bootstrap's .fade hid all of them. The action bar above the tabs stayed visible, looking exactly like 'only the UPDATE button re-rendered'. Normalize selected_subsystem_id to str in SubsystemAttributePageEditContext so callers can pass int or str and the comparison always succeeds. * Weather monitor: opt out of alarms, link banner to health modal The weather monitor's health status is now surfaced in the sidebar notice area, so it no longer needs to participate in the alarm/alert subsystem to be visible. Drop the alarm_ceiling override (which had opted WeatherMonitor in at AlarmLevel.INFO), the now-unused AlertMixin parent, and the corresponding imports. Inherited default behavior returns the monitor to 'publish health only, dispatch no alarms'. When the conditions-pane status banner shows (warning/error states), clicking it now opens the existing system_health_status modal for the weather monitor. The banner is an <a> with data-async='modal'; the HTML5 parser auto-closes the surrounding clickable-card <a> when it encounters the inner one, so the banner ends up as a DOM sibling and its click doesn't trigger the conditions-details modal. Antinode's delegated handler already calls preventDefault + stopPropagation, so no extra propagation control is needed. * Rename user-facing 'Trigger' vocabulary to 'Rule' / 'Rule Matches' The internal Event / EventDefinition models had been surfaced to operators as 'Trigger' / 'Trigger Rule' / 'Trigger History'. 'Trigger' collides with the term several integrations use (HA triggers, ZoneMinder events, Frigate events) and read poorly enough to want to change. Internal class names stay; only user-visible strings move: EventDefinition -> 'Rule' EventClause -> 'Rule Condition' EventHistory -> 'Rule Match' / 'Rule Matches' Trigger Type -> 'Match Type' (operator label) Trigger Window Secs -> 'Match Window Secs' (operator label) Trigger when ... -> 'Match when ...' (operator help) Config tab 'Triggers' -> 'Rules' Sidebar 'TRIGGER HISTORY' -> 'RULE MATCHES' Migration 0009 reflects the verbose_name / verbose_name_plural changes. Domain phrases like 'sensor triggers' and 'alarms trigger' (unrelated to EventDefinition) are kept; same with internal code identifiers, comments, and trigger_reason fields. * Frigate: expose Authorization Header field and document static-auth scenarios Flip AUTH_HEADER.is_editable to True so operators can set the Authorization header value themselves. The client already sends the value verbatim, so no client changes needed. Document the supported static-auth scenarios in docs/integrations/frigate.md with a table: no-auth (default), reverse proxy with HTTP Basic, reverse proxy with long-lived bearer, reverse proxy with anonymous backend, and Frigate's built-in JWT login (not supported in v1). The header field is the right answer for the first three; the last has a documented workaround (front Frigate with a proxy that handles auth).	2026-05-27 16:43:19 -05:00
Tony C	40f932e088	Paperless-ngx integration (ATTRIBUTE_REFERENCE capability) (#375) ... * ATTRIBUTE_REFERENCE capability + framework backend (#232 phase 1) Adds the framework seam for the new ATTRIBUTE_REFERENCE integration capability, parallel to the existing CONNECT and IMPORT capabilities. Integrations advertising this capability contribute a search-and-attach affordance to HI's attribute-edit UI: the operator searches the upstream corpus, multi-selects matching results, and HI creates one regular TEXT attribute per selection on the host Entity or Location. Pieces: - hi.integrations.enums.IntegrationCapability.ATTRIBUTE_REFERENCE joins CONNECT and IMPORT. - hi.integrations.referencer is the new package owning the per-capability abstract class (IntegrationAttributeReferencer), the picker-only result dataclass (AttributeReferenceResult), and the shared wire-field constants (WireField). - IntegrationGateway.get_attribute_referencer() returns None by default; integrations override to opt in. - Two POST endpoints under /api/integrations/referencer/: - search/<integration_id> dispatches a query to the integration's referencer and returns JSON results for the picker UI to render. - attach/<integration_id> takes a list of selections and an attribute-owning item (Entity or Location, keyed by ItemType) and creates one TEXT attribute per selection. Both views require edit-mode; the attach view rejects unsupported item types, malformed payloads, and stale references to integrations that no longer advertise the capability. No UI yet — that's phase 2. With a stub referencer, the search and attach endpoints are exercisable end-to-end and the test suite covers both happy paths and error branches (empty query, limit clamp/parse, upstream exception → 502, unknown integration → 404, unsupported item_type → 400, malformed selections → 400, title truncation to model max_length, etc.). * Phase 2: ATTRIBUTE_REFERENCE picker (HiModal + antinode) Adds the operator-facing picker that searches an enabled ATTRIBUTE_REFERENCE integration and attaches selected results as TEXT attributes on an Entity or Location. The picker is a single HiModalView with server-driven multi-select state; the picker URL takes no integration_id and the form discovers / selects integrations internally so the action-bar surface stays a single "Link" button regardless of how many referencers are configured. - Picker view (search re-render via antinode partial swap; attach via refresh_response). Multi-select state carried in the form across re-searches (selections_json + visible_url + result_url + remove_url). - Picker modal + body templates; integration selector shown only when more than one referencer is enabled. - has_attribute_referencers template tag; "Link" button wired into the Entity and Location edit content body action bars. - View tests cover GET, search re-render, attach, multi-select state transitions, and stale-integration rejection. * Phase 3: paperless-ngx simulator with parametric search responses Adds a paperless-ngx stub simulator that responds to any documents search with a synthetic, parametrically-shaped result list. Unlike the entity-shaped simulators, paperless has no SimEntity rows — it contributes only via ATTRIBUTE_REFERENCE, so its state is a small ephemeral settings dataclass on the singleton (same lifecycle as HomeBox's _api_version): result count, mime mix, thumbnails on/off, snippets on/off, and optional artificial latency. Operator tunes each knob from an extras-pane form that submits on change. The result generator seeds its RNG from the query string so repeated searches return the same set, and every title carries the query as a visible breadcrumb so the developer can verify the picker is wiring the search input through. /api/documents/<id>/thumb/ returns an inline SVG when thumbnails are on and 404s when off, exercising the picker's fallback-icon path; /documents/<id>/details/ stands in for the per-document view paperless source URLs link to. The shared services/pages/service.html now gates profile CRUD, the ADD ENTITY menu, and the entity list on sim_entity_definition_list being non-empty so ephemeral simulators don't render irrelevant controls. Connect simulators with zero current entities are unaffected — they still have definitions, so the empty-state alert fires as before. * Phase 4: paperless-ngx integration (ATTRIBUTE_REFERENCE) First integration to declare only the ATTRIBUTE_REFERENCE capability. Lets the operator search a paperless-ngx server from inside HI's picker modal and attach matching documents as TEXT-attribute links on existing Entity / Location records. No connector, no importer, no monitors, no manager singleton — the gateway returns a referencer and a single thumbnail proxy view bridges browser-embedded <img> tags to the upstream API. - PaperlessGateway, PaperlessAttributeReferencer, and a thin PaperlessClient over requests. Auth via Authorization: Token <value>. Trailing slash on the configured API URL is forgiving. - ThumbnailProxyView fetches upstream with the configured token and streams bytes back so the picker's embedded thumbnails work without exposing the token to the browser. Document source URLs point directly at paperless's web UI — operators authenticate with paperless's own session when clicking the saved link. - Snippet extraction is client-side (paperless returns full content per hit, no excerpt field): ~160-char window around the matched query with ellipses for clipped edges, leading-window fallback when the query does not match verbatim. - Wire-format strings centralized in pl_models.PaperlessApi per project convention. - Tests cover validation, client + factory (including disabled / missing-attribute paths), referencer translation + snippet extraction edge cases, gateway probe (200 / 401 / 5xx / connection error), and proxy view (unconfigured / upstream 404 / upstream auth failure / connection error / success). - Per-integration docs added (user-facing + developer-facing) and linked from the integrations landing page; the lead-in copy is updated to acknowledge the new "attachable references" shape alongside the existing CONNECT / IMPORT integrations. * Phase 5: Content Sources tab + CapabilityGateway abstraction Adds a Content Sources config tab for ATTRIBUTE_REFERENCE-capability integrations (paperless-ngx). Sibling to the renamed Connectors tab. Each tab now belongs to exactly one capability — the two surfaces are semantically different (live mirror vs. on-demand reference) and sharing a page was forcing capability awareness into framework chrome. Page model: per-integration sidebar nav, attribute form with a single ENABLE/UPDATE primary action that always runs schema + upstream-access validation, and a DISABLE button (when enabled) that rides the same form via an ``action=disable`` POST. ENABLE flips is_enabled to True only after the upstream probe succeeds (atomic "nothing changes on failure" semantics). DISABLE skips validation and just flips is_enabled to False. The framework re-renders the form area in place after each action, so the status badge and the primary-button label update without a full page reload. Architectural refactor: introduce ``CapabilityGateway`` as the shared base of ``IntegrationConnector`` / ``IntegrationImporter`` / ``IntegrationAttributeReferencer``. The three peers had nothing in common before, leaving cross-capability concerns (description text, action-bar template fragment) with no natural home. The new base carries ``capability`` (class attribute), ``get_description()``, and ``get_attribute_actions_template_name()`` — each capability owns its own defaults. The connector's existing sync-flow ``get_description (is_initial_connect)`` is renamed to ``get_sync_description`` so it does not collide with the general capability description. ``IntegrationAttributeItemEditContext`` now takes a ``CapabilityGateway`` instance instead of a capability enum and exposes it in the template context, so ``integrations/panes/integration_edit_content_body.html`` is now capability-agnostic: a single ``{% include %}`` of whatever fragment the active capability returns. The previous inline CONNECT-only health badge moves into ``connect_attribute_actions.html``. attr.js threads the submit ``event.submitter`` into ``FormData`` so multi-submit-button forms (e.g., the new ENABLE / DISABLE form) include the clicked button's ``name=value`` in the POST body. This matches native browser behavior. UI labels: the existing ``Integrations`` tab becomes ``Connectors`` (internal CONNECT routes / module names unchanged); the new tab is ``Content Sources`` (code naming uses ``reference`` to align with the capability and the surrounding ``referencer/`` directory). * Content Sources page: source description, label-stale fix, history/restore view capability cleanup - Per-source description below the integration title: "Find related content in <label> and link to it." Uses the integration label directly so future ATTRIBUTE_REFERENCE integrations get sensible copy automatically. - Rebuild the attribute edit context after the first-time ENABLE flips is_enabled to True; otherwise the async response re-rendered with the stale "ENABLE" button label and disabled-state action bar even though the integration was now enabled. - IntegrationAttributeHistoryInlineView / IntegrationAttributeRestoreInlineView pass capability_gateway=None instead of get_connector(). These ops target a specific attribute by id, so the capability filter inside the edit context is never consulted — the previous hardcoded CONNECT was a no-op that silently mismatched any non-CONNECT integration. * Apply attribute action-bar button styling to anchor-styled buttons The .attr-v2-action-buttons CSS targeted only <button> elements, so the new LINK anchor on entity / location edit pages rendered smaller and top-aligned instead of matching its <button> siblings (Add File, Add Info). Extend the selectors to cover a.btn inside the action bar and add explicit inline-flex centering so the icon and text line up with the other buttons. * Picker: JS-owned selection state, split search / attach endpoints The previous picker maintained selection state on the server, which meant ticking a result card did nothing visible until the next form submit — operators saw "Add 0 References" stuck disabled after their first check and assumed the picker was broken. Multi-selection now behaves as one transaction: JS owns the in-memory selection set, search calls are async within that state context, and the only POST that touches the server is the final Add submit. attr-picker.js (new, in js_hi_grid_header_content) owns: - The selection set (per-modal WeakMap keyed by the picker DOM). - Chip-row rendering (the one piece of HTML the JS generates). - The Add button's label and enabled state. - The async search request: POST to a new search endpoint that returns ONLY the result-cards partial; JS swaps it into the results container and re-applies checked state to any newly- visible result whose URL is already selected. - The Add submit: serialize the selection set into a hidden selections_json input, then route through antinode's public API (AN.hideModalIfNeeded + AN.post) so modal cleanup and the {refresh: true} response flow through the framework. Form deliberately omits data-async because antinode binds at <body> level and would otherwise race attr-picker's document-level submit handler. Server endpoints split: picker GET stays, plus new search and attach POST endpoints. Old multi-select bookkeeping (_compute_selections, visible_url[], remove_url, action=attach button) deleted along with the WireField class. Client/server constant sharing now follows the project's established DIVID pattern: ATTR_PICKER_* entries in hi.constants.DIVID (server) and Hi.ATTR_PICKER_* in main.js (client). Templates, view code, and JS all read from these — no more magic strings to drift between sides. Simulator fix: paperless's _generate_results used document_id = index + 1, which meant different queries produced overlapping ids (and thus URLs). The JS keys selections by source_url, so the first result of query A and the first result of query B were treated as the same document. Document id is now a hash of (query, index) so different queries produce distinct ids while repeating the same query stays stable — matching real paperless's "same document re-found in multiple searches" behavior. CSS: a.btn-styled buttons inside .attr-v2-action-buttons inherit the same min-height / padding / font-size / border-radius / focus / hover / icon-spacing rules as their <button> siblings, so the anchor-styled LINK button on Entity / Location edit pages lines up with Add File and Add Info. * Picker: footer-anchored chips + Add, sticky search, seeded initial query Layout reshape using Bootstrap's built-in ``modal-dialog-scrollable`` opt-in (only the picker modal opts in; other modals are unaffected): - Footer is now the commitment zone. It holds the selection chip row above a Cancel/Add row. The footer stays anchored regardless of how far the operator scrolls through results, so the accumulated selections and the Add button are always visible together. - Body holds only the search form and the results container. The search form sticks to the top of the body's scroll region (position: sticky) so refining the query stays one click away no matter how far down in the result list the operator is. - The picker-root class moved up to the modal-dialog so JS handlers in both body (result cards) and footer (chip-X buttons, attach form) can ``closest()`` to the same picker instance. CSS additions, scoped to ``modal-dialog-scrollable`` so the flex-shrink contract works correctly with this project's custom ``.modal-title`` (vs. Bootstrap's default ``.modal-header``): - ``.modal-dialog-scrollable .modal-title``, ``.modal-footer`` get ``flex-shrink: 0`` so the title and footer keep their natural height when the body grows. Without this the title squashed to near-zero height on tall result lists. - ``.hi-attr-picker .modal-body { padding-top: 0 }`` so the sticky search form sits flush against the title — otherwise the modal-body's default padding-top showed scrolling content through the gap above the sticky element. Initial-render seeded with owner's name: - The picker GET now runs the same ``_search_upstream`` the search endpoint uses, with ``owner.name`` as the query, and seeds the results container with the resulting ``picker_results.html`` partial. Operator opens the picker and sees relevant matches immediately — no retype, no extra click. Same template + same context variables (``query``, ``results``) as the search endpoint so initial-render and re-search-render are structurally identical. * Picker: source banner always rendered, wider modal, "Link Content" copy The picker UI now has the same shape for one source or many — the modal doesn't change structurally when a second ATTRIBUTE_REFERENCE integration is configured. - Source banner: replaces the previous conditional ``<select>`` in the multi-source case + hidden input in the single-source case. Always-rendered Bootstrap dropdown whose button face shows ``[logo] [name] [caret]``. With one source the caret is hidden and the button is disabled (the visual stays consistent without inviting a no-op click). On selection: attr-picker.js updates the banner face, sets the hidden ``integration_id`` input, and re-submits the search form — natural for "search the current query against a different source." - Modal width bumped to ``hi-modal-dialog-700`` so the title bar fits "Link Content" and the result cards have room to breathe. - Operator-facing copy aligned with the "Content Sources" tab: modal title is now constant "Link Content" (was conditional "Link References" / "Add from <name>"); attach button reads "Add N Link(s)"; the chip-row empty hint reads "No content selected yet."; the action-bar button on Entity / Location edit pages reads "Link Content". - DIVID + main.js gain matching ``ATTR_PICKER_SOURCE_*`` entries for the banner classes, the dropdown-item class, and the three data attributes the dropdown items carry (source id, logo URL, label) so the JS handler can read them without re-hardcoding any strings. - CSS keeps the banner on white so source logos meant for light backgrounds render correctly (the modal title bar uses a colored background and would clash). Single-source modifier class hides the caret and removes the disabled-button visual fade. * Align operator-facing copy with Connectors / Content Sources terminology When the config tab was renamed from "Integrations" to "Connectors" (and the new "Content Sources" tab was added for ATTRIBUTE_REFERENCE integrations), several operator-facing strings continued to use the older "Integration(s)" wording. Updated to match the page labels operators now see: - Connectors page sidebar button: "INTEGRATIONS" → "CONNECTORS" - Empty-state copy on the Connectors page: "no integrations currently configured" / "CONFIGURE INTEGRATIONS" - The "All Integrations" picker modal title → "All Connectors" - Disable-confirm modal: "Disable {label} Integration?" / "This integration has..." / "detached from this integration" / "If you re-configure this integration later..." → use "connector" consistently. - Pre-sync confirm modal: same "This integration has..." / "detached from this integration" → "connector". - Data Import card note: "Also available as an Integration." → "Also available as a Connector." - Data Import empty state: "No integrations support Data Import." → "No data importers currently defined." (the prior wording was technically accurate but ambiguous given the new tab labels; the new wording matches what the operator is looking at). - "Data Import vs. Integration" help modal → "Data Import vs. Connector"; body text refers to the contrast as Data Import vs. Connector and uses "external system" instead of "upstream source". Internal code identifiers (``IntegrationGateway``, ``integration_id``, ``ConfigPageType.INTEGRATIONS_CONNECT``, etc.) unchanged — they were never operator-visible. * Added new link icon and used for new attribute referrer integration. * Lift get_metadata onto CapabilityGateway base ``IntegrationConnector``, ``IntegrationImporter``, and ``IntegrationAttributeReferencer`` all returned the integration's ``IntegrationMetaData`` constant but disagreed on the method name — the connector used ``get_integration_metadata`` while the other two used ``get_metadata``. Same fact, three implementations, two names. Move the contract onto ``CapabilityGateway`` as a single abstract ``get_metadata()``, drop the duplicate declarations from the importer / referencer bases, and rename the connector's variant. - IntegrationConnector: ``get_integration_metadata`` → ``get_metadata`` (with the three internal call sites updated). - IntegrationImporter / IntegrationAttributeReferencer: declaration removed; inherited from CapabilityGateway. - HASS / ZoneMinder / Frigate / HomeBox connectors: method renamed to ``get_metadata``. - test_integration_connector.py / test_sync_check.py: fixture stubs renamed to match. The ``_get_integration_metadata`` private helper in ``integration_tags.py`` is unrelated (it takes an ``IntegrationDetailsModel``, not a gateway) and untouched. * Some integration modal cleanup * File name normalization for new integrations referencer. * Refresh integration-guidelines.md for current capability model The doc had drifted in several places: - Said "two capabilities exist today." Three do — ATTRIBUTE_REFERENCE is the third (paperless-ngx). Added the entry plus a note that ATTRIBUTE_REFERENCE-only integrations land on the sibling "Content Sources" tab rather than "Connectors." - Said the per-capability classes "don't share a base class — commonality is composed through shared helpers." They do now: ``CapabilityGateway`` (``hi/integrations/capability_gateway.py``) is the shared base, carrying ``capability``, ``get_metadata``, ``get_description``, and ``get_attribute_actions_template_name``. - Setup steps 3-5 referenced an obsolete ``IntegrationType`` enum, an ``activate / deactivate / manage`` method surface, and an ``integration_factory.py`` that doesn't exist. Rewrote those steps to describe declaring ``IntegrationMetaData`` with the right capability set, implementing the actual gateway methods (``get_metadata``, ``get_<capability>``, ``validate_configuration``, ``validate_access``, ``notify_settings_changed``), and the auto-discovery flow that replaces factory registration. - Gateway Implementation Patterns described the old activate/deactivate/manage contract and dict return shape — neither still exists. Replaced with a short summary of the current method contract. - Error Handling listed exceptions that don't exist (``ConnectionError``, ``AuthenticationError``, ``DataValidationError``) and missed the ones that do (``IntegrationDisabledError``, ``IntegrationAttributeError``, ``IntegrationConnectionError``). Corrected. - Key Base Classes & Modules had wrong dotted paths (``hi.integration.*`` → ``hi.integrations.*``, ``hi.utils.singleton`` → ``hi.apps.common.singleton``), referenced a non-existent ``IntegrationStatus`` enum and the obsolete factory module. Replaced with the current set, added ``CapabilityGateway`` and ``IntegrationManager``. - Example Integrations listed three; added Frigate and paperless, and noted that paperless has a slimmer file layout because ATTRIBUTE_REFERENCE-only integrations have no monitors / sync / converter / manager. - File layout section gained a short paragraph noting which role-files apply only to CONNECT/IMPORT-shaped integrations. * Accept single-label intranet hostnames in attribute URL linkification Django's URLValidator rejects hostnames without a TLD (e.g. http://cassandra:4100/...), so URLs pointing at LAN hosts saved as attribute values were not rendered as clickable links. Replace the strict validator with a permissive urlparse-based check (http(s) scheme + non-empty netloc) so intranet single-label hostnames are linkified just like IPs and FQDNs. * Refresh paperless-ngx docs for Content Sources tab and slim dev page User-facing changes follow the post-#232 UI: paperless is configured from the Content Sources tab (not Connectors), the action button on item/Location edit pages reads 'Link Content', and the picker's attach button reads 'Add N Links'. Integrations.md routes paperless to its own walkthrough rather than folding a non-Connectors flow into the shared steps. Dev page refocused on design rationale (capability shape, thumbnail proxy vs source-URL passthrough, no-manager, snippet strategy, single-deployment assumption). Dropped module/test enumerations and wire-format strings — those rot and a developer can read the directory or pl_models.py. * Connectors vocabulary sweep in capability-blocked modal and selector The capability-block modal still emitted 'GO TO INTEGRATIONS' / 'disable the … Integration' / 'configured as Integration with', contradicting the new three-tab operator vocabulary (Connectors / Content Sources / Data Import). The empty state in the integrations selector also still said 'No integrations found' under an 'All Connectors' title. Aligned both to 'Connector(s)' and updated the matching importer-view test assertion.	2026-05-27 12:22:47 -05:00
Tony C	ea991d691a	Implement HomeBox Importer; refactor integration framework (#360) ... * Phase 1 (#358): rename connect/ → connector/ and lift capability-agnostic files to umbrella root Mechanical refactor with no behavior change, setting up the layout for the new importer/ subpackage that arrives in Phase 2. The connect/ subpackage is renamed to connector/ for symmetry with the forthcoming importer/. Five files that were already capability- agnostic are lifted from connector/ up to the integrations/ umbrella root so neither capability subpackage will depend on the other: - placement_request.py - integration_attribute_edit_context.py - entity_operations.py - integration_data.py - integration_gateway.py The remaining connector/ contents (views, synchronizer, monitors, sync_check, sync_result, view_mixins, external_view_data, plus the Connect-mode controller/manage-view-pane stubs and user_data_detector) stay capability-specific. view_mixins.py stays in connector/ as a single file; a future refactor could split the capability-agnostic accessor methods up to umbrella, but the call-site benefit is small and bundling that with this rename would obscure the move. The defensive Disable-side data_source predicate originally listed in Phase 1 is deferred to Phase 7. The naive EXTERNAL filter would break HA/ZM/Frigate Disable (their entities are INTERNAL), so the guard needs to be conditional on the integration declaring both capabilities — designing that alongside Phase 7's symmetric block- modal capability-conditional detection. * Phase 2 (#358): Importer protocol skeleton Lands the abstract Importer base, transient models (CandidateItem, IntegrationDiscardResult), and IntegrationImportResult dataclass. Wires get_importer() into IntegrationGateway as the parallel of get_synchronizer(), defaulting to None. No live consumer yet — the HomeBox concrete implementation arrives in Phase 3. IntegrationImportResult is a parallel dataclass to IntegrationSyncResult, not a subclass. Import-specific shape: items_imported_count / items_skipped_count + imported_list + info/error lists + placement_input. No reconnect/detach/remove concepts (Import is add-only). * Phase 3 (#358): HomeBox concrete Importer + capability-aware entity factory Lands HomeBoxImporter as the first concrete consumer of the Phase 2 Importer protocol, plus the supporting infrastructure for it to share HomeBox's entity-creation code with the CONNECT-mode synchronizer. Capability-aware HbEntityFactory: - Moved from connector/ to shared/ (used by both Connect and Import now). - New ``capability`` parameter on create_models_for_hb_item routes to the correct flags: CONNECT keeps the read-only-mirror flags (can_user_delete=False, allow_internal_attributes=False, data_source=EXTERNAL); IMPORT sets HI-owned flags (deletable, editable, data_source=INTERNAL). - Fixes a latent post-#355 gap: data_source was never set at entity creation time. Phase 4 of #355 backfilled existing rows; new HomeBox-Connect entities created after #355 would have defaulted to INTERNAL until now. HbConverter payload defaults flipped to CUSTOM + is_editable=True. The Connect-mode copy-on-sync architecture (pre-#354) used these methods to produce PREDEFINED read-only attributes. Post-#354, the only live caller is the new Import code, which wants user-owned attributes. The methods are now unambiguously Import-flavored. HomeBoxImporter (services/homebox/importer/homebox_importer.py): - get_candidate_items: lightweight summary fetch, excludes archived. - run_import: per-entity transaction inside the existing integrations_sync exclusion lock; skip-if-already-imported by integration_name; per-item failures aggregate into error_list. - discard_imported_data: filters by data_source=INTERNAL so any coexisting Connect-mode entities are untouched. Dormant hb_attribute_sync.py (224 lines) deleted. Replaced by a ~15-line populate_attributes_for_imported_entity helper next to the existing HbImporter classmethods — Import is add-only so the sync-style dispatch / update / remove logic no longer applies. HbMetaData.capabilities now declares both CONNECT and IMPORT. HomeBoxGateway.get_importer() returns the importer instance. * Phase 4 (#358): Data Import config tab + Configure form (preview-side) Lands the user-visible Data Import tab parallel to Integrations, plus the configure-form workflow up through preview rendering. Confirm → run is Phase 5. New tab: ConfigPageType.DATA_IMPORT (url_name 'integrations_import_home', label 'Data Import'). URL prefix is integrations_import_* — matches the eventual rename of integrations_home → integrations_connect_home the framework will adopt. DataImportPageView: flat list, one row per IMPORT-capable integration. Per-row CONFIGURE button (always) and DISCARD button (only when imported entities exist for that integration_id with data_source=INTERNAL). Dual-capability rows (HomeBox today) carry a terse "Also available as an Integration" note linking to a bottom-of- page explanation contrasting Data Import vs Integration (Connect). ImporterConfigureView: credentials form (reuses IntegrationAttributeItemEditContext with capability=IMPORT, action button labeled IMPORT). POST validates configuration + access via the same validate_attributes_extra_helper Connect uses, then fetches candidates via Importer.get_candidate_items(), computes new vs. skipped counts by integration_name match against existing HI entities, and renders the preview modal. Preview modal: "Would import N new items. M existing items would be skipped." CONFIRM IMPORT and CANCEL actions. CONFIRM posts to the integrations_import_run URL — stub view returns 501 in this phase; Phase 5 wires the actual run. ImporterRunView and ImporterDiscardView land as 501-returning stubs so the URL reversals from this phase's templates resolve cleanly; Phase 5 and 6 fill in the implementations. view_mixins.py moved from connector/ up to umbrella root so both Connect and Import views can extend IntegrationViewMixin. The render_sync_result method on it is only called by Connect views; Import views never touch it. No import-time Connect coupling — the synchronizer.sync() call inside render_sync_result is runtime-only. * Phase 5(a) (#358): Confirm → Run → Result modal + placement reuse Wires the IMPORT workflow end-to-end. After the preview modal's CONFIRM IMPORT, the importer runs, the result modal renders with imported / skipped counts, and a "Place N new items" CTA hands off to the existing placement flow when entities were created. Importer base gains group_entities_for_placement (default ungrouped, mirrors IntegrationSynchronizer's same-named method). HomeBoxImporter stays with the default — HomeBox has no inherent grouping at v1 (Labels and Locations are metadata-on-entity). HomeBoxImporter._run_import_locked now tracks created entities and populates result.placement_input via group_entities_for_placement, mirroring HomeBoxSynchronizer's _sync_impl pattern. ImporterRunView.post: - Runs importer.run_import() inside a try/finally that invalidates the metadata + sensor-response caches afterward (mirrors Connect's render_sync_result). - Builds the placement URL via the existing PlacementUrlParams. - Reuses the existing integrations_placement URL — HomeBox is dual-capability so its synchronizer is available for the placement view's group_entities_for_placement call. import_result.html: counts hero (Imported / Skipped), per-item list, errors. "Place N new items" CTA on success; OK button on the no-imports / errors paths. Modeled on sync_result.html but trimmed for Import's narrower outcome shape. Manual verification: end-to-end Import works after the manager picks up newly-saved credentials. Phase 5(b) addresses the is_enabled / manager-reload race that surfaces this timing issue and the symmetric one in the Connect-enable flow. * Phase 5(b) (#358): decouple Connect-mode is_enabled from credential loading The HomeBoxManager (shared/) was checking the Connect-mode is_enabled flag inside _load_attributes and raising IntegrationDisabledError to nullify hb_client. That coupling produced two related bugs: 1. Import workflow: after the user disables HomeBox-Connect, HomeBoxImporter.get_candidate_items found hb_client=None and returned empty list — preview showed 0 items even though the upstream simulator had 25. 2. Connect-enable race: after IntegrationEnableView flipped is_enabled=True via enable_integration, the manager's post_save signal was deferred by ~0.1s (DelayedSignalProcessor). The synchronous synchronizer.sync() that immediately followed saw hb_client=None and failed with "integration is disabled" even though the user had just enabled it. Fix: * Strip the is_enabled gate out of HomeBoxManager._load_attributes (and the matching except IntegrationDisabledError branch from _reload_implementation). The Connect-mode enable lifecycle is not a shared/ concern — credentials load whenever they are structurally valid so the IMPORT capability works regardless of Connect-mode state. Connect-side guards (manage view, disable, pause, resume) check integration.is_enabled directly at the view layer, where the capability context is known. * IntegrationManager.enable_integration: synchronously call gateway.notify_settings_changed() after the save+refresh, before the monitor launch. Closes the delayed-signal race so the synchronizer immediately sees fresh credentials. * ImporterConfigureView.post: same synchronous notify after post_attribute_form succeeds, before invoking the importer. Credentials just changed; the manager needs to pick them up before run_import. HASS, ZM, Frigate managers are Connect-only and are left alone — their is_enabled gate is fine because they have no other capability that needs to bypass it. Manual verification: Import workflow shows correct candidate count with Connect disabled; Connect-enable runs sync successfully on the first click. * Phase 6 (#358): Discard flow Wires the DISCARD button on the Data Import page. GET renders a confirmation modal showing the count of imported items; POST runs the importer's discard_imported_data and redirects back to the Data Import page. Single-action confirm (no SAFE/ALL variants). Imported items ARE the user data — the Connect-side preserve-with-user-data semantic doesn't apply. Defensive scoping is in HomeBoxImporter.discard_imported_data (Phase 3): the deletion query filters by data_source=INTERNAL, so any coexisting Connect-mode (EXTERNAL) entities for the same integration_id stay untouched. Confirmed via test. Post-discard cache invalidations mirror the Connect-side Disable cleanup: IntegrationMetadataCache + SensorResponseManager. Sensor- response cache invalidation is symmetric with sync's post-write behavior even though imported items don't have sensors today — keeps the discard path defensive for future Importers that might attach sensors. * Clarify EntityDataSource semantics: every Connect-mode entity is EXTERNAL The Phase 4 (#355) migration used `allow_internal_attributes=False` as the proxy for EXTERNAL, which conflated two separate concerns: "data is sourced from upstream" and "user can edit HI-side attributes." Under the conflation, HA/ZM/Frigate-Connect entities were marked INTERNAL even though their state is sourced from upstream — only HomeBox-Connect (allow_internal_attributes=False) got EXTERNAL. The clarified semantics: data_source=EXTERNAL means data is actively being sourced from an external upstream system, which is true of EVERY Connect-mode integration regardless of whether HI permits user-edited attributes on top. data_source=INTERNAL means no live upstream link — Import-mode (HI-owned post-import) and native entities. | integration | mode | data_source | |-------------------|---------|-------------| | zm/frigate/ha | connect | EXTERNAL | | homebox | connect | EXTERNAL | | homebox | import | INTERNAL | | (native) | n/a | INTERNAL | Changes: * Migration 0021_entity_data_source.py: backfill broadens to mark every integration-attached entity EXTERNAL. Adjusted in place rather than as a fix-up migration since 0021 hasn't been released outside local environments and has no downstream migrations. * HassConverter, FrigateSynchronizer, ZmSynchronizer (both service and monitor entity creation): explicitly set data_source=EXTERNAL on new Connect-mode entities. Closes the post-#355 gap where the field wasn't being set at creation time for these integrations — the migration backfilled existing rows but new rows fell back to the INTERNAL default. * HomeBoxConnector.get_external_view_data: gates on data_source=EXTERNAL. Import-mode HomeBox entities are HI-owned after import, so the live HomeBox view isn't authoritative for them and would mislead the operator. * test_models.py: migration-backfill test renamed and assertions updated to reflect the broadened predicate (HA-like entity now becomes EXTERNAL too). * test_hb_connector.py: HomeBox entity fixture sets data_source=EXTERNAL so the connector's new gate doesn't suppress the connector's existing test surface. Required for Phase 7 (#358 block modal): the modal queries data_source to distinguish Connect-mode entities (EXTERNAL) from Import-mode entities (INTERNAL) for an integration. The old semantics would have miscategorized future dual-capability HA/ZM/ Frigate. * Phase 7 (#358): block modal for cross-capability mode switching When an operator starts an integration's initial-Configure flow for one capability while existing data from the other capability is still present, surface a block modal that routes them to remediate before proceeding. Connect ↔ Import switches are intentionally made harder — the user navigates explicitly, performs the cleanup themselves, then re-clicks CONFIGURE. No combined transition flow. CapabilityBlockViewMixin (in hi/integrations/view_mixins.py): - Single helper render_capability_block_if_conflict that takes only integration_data + capability_being_initiated. The mixin composes all user-facing copy (title, body clauses, navigation link) from per-capability config dicts so block modals read uniformly across both directions and any future capabilities. - Detects the OTHER capability automatically. Only fires when the integration declares both capabilities; single-capability integrations get None. - Counts entities of the OTHER capability's data_source for the integration; zero means no conflict. Connect side: IntegrationEnableView.get checks the block on the initial-Connect path (is_enabled=False). Re-Configure of an already-enabled integration always proceeds — that's not a new mode being initiated. Import side: ImporterConfigureView.get checks the block when no Import entities yet exist for the integration. Re-import (entities already present) always proceeds — that's the standard incremental- import flow. capability_blocked.html: generic block modal template parameterized by title + body clauses + link. Extends modals/action_2_with_cancel so the inherited CANCEL just dismisses. Single-capability integrations (HA/ZM/Frigate today) never trigger the block: the mixin's "other capability not in capabilities" guard short-circuits before any DB query. * Phase 8 (#358): documentation for the Data Import capability User-facing: * docs/DataImport.md (new) — landing page for the Data Import feature; importing/discarding walkthrough; lists supported integrations. Concept-level prose only; no hardcoded button labels so the doc doesn't drift if UI strings change. * docs/Integrations.md — Integrations vs. Data Import comparison pointing at the new landing page. * docs/integrations/homebox.md — added Data Import section describing the alternative mode and when to choose it. Developer-facing: * docs/dev/integrations/data-import.md (new) — framework orientation for the IMPORT capability. Lists where code lives (importer/, view_mixins.py, services/homebox/importer/), how a gateway opts in, and the key design decisions (per-entity transaction, skip-by-integration_name, shared lock, data_source as capability-state signal). Reference-style; points at the code rather than restating API. * docs/dev/integrations/integration-guidelines.md — Capability Model subsection refreshed: CONNECT and IMPORT now both have concrete pointers; the #355 "reserved for #358" placeholder is gone. * docs/dev/integrations/homebox.md — updated stale module paths to reflect the connector/importer/shared split landed in #355 Phase 3 + #358 Phase 1. Added a short Importer section pointing at the framework doc. * Integration framework cleanup: capability-aligned naming and template structure Rename the Connect/Import vocabulary uniformly across the integration framework now that HomeBox is the dual-capability archetype. Class renames: - IntegrationSynchronizer -> IntegrationConnector (+ Hass/Zm/Frigate/HomeBox subclasses) - Importer -> IntegrationImporter - IntegrationEnableView -> ConnectorConfigureView - HomeBoxConnector (the per-entity external-view resolver) -> HomeBoxExternalViewResolver Gateway method: - IntegrationGateway.get_synchronizer() -> get_connector() File renames (git mv, history preserved): - integration_synchronizer.py -> integration_connector.py - importer.py -> integration_importer.py - homebox/connector/hb_sync.py -> homebox_connector.py - homebox/connector/hb_connector.py -> hb_external_view_resolver.py - corresponding test files URL name renames: - integrations_home -> integrations_connect_home - integrations_enable -> integrations_connect_configure Enum renames (ConfigPageType): - INTEGRATIONS -> INTEGRATIONS_CONNECT - DATA_IMPORT -> INTEGRATIONS_IMPORT View mixin: - IntegrationViewMixin.get_integration_data(integration_id) -> (request, *args, **kwargs) Mirrors EntityViewMixin.get_entity; pulls integration_id from kwargs, raises BadRequest if missing and Http404 if not registered. ~20 call sites adopted the uniform pattern. Template restructure (src/hi/integrations/templates/integrations/): - import/ -> importer/ (avoids Python reserved word, matches sub-package convention) - importer/ now has modals/ subdir matching the pages/ convention - New connector/{modals,pages,panes}/ subtree for Connect-specific templates - Top-level modals/, panes/ now hold only genuinely shared cross-capability templates (capability_blocked, placement flow, attribute-form body) - Editor backup .html~ files removed Misc: - IntegrationImportResult folded into importer/transient_models.py (single-class file with no module-level documentation) * Remove dormant IntegrationManageViewPane seam; rename to ConnectorManageView The per-integration "manage view pane" extension point on IntegrationGateway was dead machinery: all four service subclasses returned templates containing only a {% comment %} block. The seam also pre-dated the capability model — its return type lives under connector/, but the hook was on the capability-agnostic gateway. Rather than relocate the hook to IntegrationConnector, delete the seam entirely (base class, 4 subclasses, 4 stub templates, gateway method, view context plumbing, template include). It can be reintroduced correctly on IntegrationConnector when an integration actually needs per-integration content in the manage page. Also rename the view to reflect that the Manage page is Connect- specific: - IntegrationManageView -> ConnectorManageView - integrations_manage URL name -> integrations_connect_manage * Push Connect-specific hooks from IntegrationGateway to IntegrationConnector IntegrationGateway accumulated eight Connect-mode hooks over time, all returning types that already lived under connector/. Now that the capability model has IntegrationConnector and IntegrationImporter as peer capability classes (cf. the recent IntegrationManageViewPane cleanup), these hooks belong on the connector — not on the capability-neutral gateway. Import-only integrations should not be forced to stub them out. Methods moved from IntegrationGateway to IntegrationConnector: - get_monitor() - get_controller() - get_health_status_provider() - get_entity_video_stream() - get_entity_video_snapshot() - get_sensor_response_video_stream() - get_sensor_response_event_snapshot_url() - get_external_view_data() Per-service implementations relocated from each *Gateway (integration.py) to the corresponding *Connector. Production call sites updated to chain through get_connector() with None-guards where the caller might encounter an Import-only or connector-less integration. The three Connect-side view sites trust the structural invariant. The do_control path raises a clear RuntimeError if no connector or controller is available. File renames to match the relocated class names: - frigate_sync.py -> frigate_connector.py (FrigateConnector) - hass_sync.py -> hass_connector.py (HassConnector) - zm_sync.py -> zm_connector.py (ZmConnector) - corresponding test files IntegrationGateway's surface is now its capability-neutral core: get_metadata, notify_settings_changed, validate_configuration, validate_access, get_connector, get_importer. * Fix re-Configure credentials race; remove redundant block-check pre-gates Two related issues in the Configure flows: 1. ConnectorConfigureView.post relied on enable_integration to fire notify_settings_changed(), but enable_integration only fires it on the disabled->enabled transition. Re-Configure of an already-enabled integration left the singleton manager holding stale credentials, and the subsequent sync ran against them. The post_save signal would eventually nudge via DelayedSignalProcessor, but the 0.1s delay races the immediate sync. Lift the synchronous notify out of enable_integration and into the view, right after post_attribute_form, so both initial-Connect and re-Configure get the nudge. The matching ImporterConfigureView.post already did this; the Connect side was the asymmetric outlier. 2. Both Configure views had a pre-gate that suppressed render_capability_block_if_conflict in certain re-entry paths (is_enabled=True for Connect, has_imported=True for Import). Under the mode-switch invariant the pre-gate is redundant — the block check itself short-circuits to None when there's no conflicting data. The pre-gate's only effect was to silently bypass the block check when the invariant was violated (mixed INTERNAL+EXTERNAL state), letting the user compound the corruption. Remove both pre-gates so the block fires in any mixed-state scenario and the user is told to remediate. Two tests deleted that asserted the now-removed bypass behavior: - test_get_import_not_blocked_when_already_imported - test_already_enabled_does_not_trigger_block The canonical block-fires-on-conflict and re-Configure-works paths remain covered by neighboring tests. * Extract CapabilityConfigureView shared base for both Configure modals ConnectorConfigureView and ImporterConfigureView shared a 5-step GET recipe (block check → ensure_all_attributes_exist → build edit context → render modal) and a 3-step POST recipe (post_attribute_form → error short-circuit → success handler). Pull the shared structure into a new framework-level CapabilityConfigureView in hi/integrations/views.py. Subclasses now declare four class constants (capability, button_label, template_name, error_title) and override handle_post_success() with the per-capability behavior. ConnectorConfigureView.handle_post_success: enable_integration, then sync or redirect-to-manage. ImporterConfigureView.handle_post_success: fetch candidates, compute new-vs-skipped, render preview. Each subclass owns the timing of notify_settings_changed() because the right moment is capability-specific: * Connect-side managers (Frigate/Hass/ZM) gate client (re)build on integration.is_enabled, so the notify must fire AFTER enable_integration — otherwise the manager reloads with is_enabled=False, nulls its client, and the immediately-following sync sees no client. * Import flows fire the notify before reading candidates. The unconditional call in both subclasses also covers the re-Configure race (already-enabled integration), where enable_integration early-returns without nudging and the singleton manager would otherwise keep stale credentials. * Flatten homebox/shared/ up one level The capability-layout convention is: inter-capability files live at the integration's top level (alongside integration.py / apps.py / enums.py), while capability-specific files live under connector/ or importer/. HomeBox's shared/ predated the convention; it's the last holdout. Move the six modules up: homebox/shared/hb_client.py -> homebox/hb_client.py homebox/shared/hb_client_factory.py -> homebox/hb_client_factory.py homebox/shared/hb_converter.py -> homebox/hb_converter.py homebox/shared/hb_entity_factory.py -> homebox/hb_entity_factory.py homebox/shared/hb_manager.py -> homebox/hb_manager.py homebox/shared/hb_models.py -> homebox/hb_models.py shared/__init__.py removed; shared/ directory gone. 21 absolute import-site rewrites plus 2 @patch strings updated. Docs and simulator docstring path references brought in line. * Data Import UI/UX refinements Tab order - Move Data Import to the right of Triggers in the Config tabs (matches natural read order: settings -> integrations -> trigger setup -> data import -> system info). Import preview modal (HomeBox confirm) - Title becomes "<Label> Import" when there's nothing to confirm (instead of the awkward "Import from HomeBox?" question). - Body now handles the no-new-items cases explicitly: "All N upstream items have already been imported." or "No items found to import." instead of "Would import 0 new items." - Single centered DONE button replaces the lone left-aligned CANCEL when the preview is informational only. - Add data-async="modal" to the CONFIRM IMPORT form so it submits through the modal pipeline like every other action on the page. - Show a contextual "Currently N items imported from <Label>." line whenever the integration has prior imports — signals re-import vs. first-time state. View filters the count by data_source=INTERNAL. Data Import page layout - Replace sparse rows with a responsive Bootstrap card grid (col-12 col-sm-6 col-md-4 col-lg-3), 4 columns on large screens. Logo centered above the integration name, action buttons stacked at the card bottom (CONFIGURE always; DISCARD only when has_imported). - New hi-integration-logo--card modifier (72px) for the card-prominent size. - Drop btn-control width:100% from the action buttons (the row layout no longer benefits from it; btn-block within the card stack is what fills card width now). - Remove the always-below-the-fold "Data Import vs. Integration" footer block. - Add DataImportInfoView (HiModalView) at integrations_import_info rendering a static info modal with the same explanation content. - On dual-capability rows, the entire "Also available as an Integration. <info-icon>" message is now a single clickable link that opens the info modal. Capability-blocked modal - Title was getting truncated ("Cannot configure <Label> as <Capability>" doesn't fit a narrow modal). Title is now just the integration label; the "Cannot configure as <Capability>." line moves into the body as an h5 sub-heading above the existing explanatory paragraph. View context drops the pre-composed title key and passes my_label so the template composes the sub-heading. * Tighten data_source scoping at mode-switch invariant seams The mode-switch invariant (a given integration's entities are all EXTERNAL for Connect, or all INTERNAL for Import, never mixed) was enforced at the Configure entry point but several downstream code paths filtered only on integration_id, leaving data-source-blind seams. The same code review also surfaced that the disconnect/ reconnect path mishandles data_source when transitioning between attached/detached states. POST block-check (CapabilityConfigureView.post) - GET ran the block check; POST didn't. A direct POST (cached form, replayed request) would bypass the invariant. POST now re-invokes render_capability_block_if_conflict before delegating to the capability-specific handle_post_success. disable_integration seed (IntegrationManager.disable_integration) - Seed query now filters by data_source_str=EXTERNAL. A Connect-side Disable can no longer silently delete imported INTERNAL data if the invariant ever leaks. Native and detached entities (integration_id NULL) were already outside the seed. HomeBoxConnector sync queries - _get_current_integration_keys (sync-check probe) and _get_existing_hb_entities (sync run) both filter by EXTERNAL. INTERNAL imports under the same integration_id no longer risk being adopted as Connect rows and routed through mass-remove or auto-reconnect. HomeBoxImporter preview/run symmetry - _run_import_locked's existing_integration_names set now filters by INTERNAL to match ImporterConfigureView.handle_post_success's preview predicate. A stale EXTERNAL row can no longer turn an advertised new candidate into a skip at run time. Disconnect/reconnect data_source transition - EntityIntegrationOperations.preserve_with_user_data now flips entity.data_source to INTERNAL on detach. A detached entity is no longer constrained by an upstream system — HI owns the editable representation. The reconnect path is already correct: all four service converters set data_source=EXTERNAL during the rebuild invoked by _rebuild_integration_components. Pre-migration detached rows are also correct: migration 0021's column default of 'internal' applied to rows with no integration_id. Test fixtures - Four entity-creation sites in test_integration_manager.py and test_homebox_connector.py now explicitly set data_source_str=EXTERNAL for integration-attached test entities, since the new EXTERNAL filters no longer match the implicit default. * Move IMPORT to previous_integration_id; introduce named entity-state queries The IMPORT path was overloading integration_id (the live-ownership column) to mean "where this entity was sourced from." That conflation forced data_source_str as a discriminator and made post-import code paths brittle: any query that scoped by integration_id and a data source filter could quietly diverge. Under the model in this commit, the entity's lifecycle state is encoded entirely by which integration_* column is populated: - integration_id set -> EXTERNAL (live Connect) - integration_id NULL, previous set -> imported OR detached (HI-owned, has provenance) - both NULL -> INTERNAL (native) IMPORT now writes to previous_integration_*. integration_id stays NULL for imported rows because no live integration owns them. Manager methods (EntityModelManager) give call sites a vocabulary matching the data-source semantics: - external_for(integration_id) active-Connect rows - imported_for(integration_id) import-side intent - detached_for(integration_id) reconnect-candidate intent - with_integration_provenance(id) any row with this integration's identity in either column imported_for / detached_for share an implementation today (the column shape is the same); each name documents intent at the call site so a future discriminator could differentiate without touching callers. Block check: only IMPORT initiation can collide (it would create new rows; an active-Connect row already exists for the upstream key). CONNECT initiation never needs blocking — sync's reconnect- then-create order adopts any pre-existing provenance entities into the live Connect session. CapabilityBlockViewMixin loses its symmetric dispatch and the _CAPABILITY_* dicts. Post-import placement bug: EntityPlacementService.query_unplaced filtered by integration_id, so imports (integration_id=NULL) were invisible to the placement modal — the user saw "No items found" after a successful import. Switched to with_integration_provenance so both EXTERNAL and imported rows are eligible. Other writes / queries swept to the new managers: - HbEntityFactory.create_models_for_hb_item IMPORT branch - HomeBoxConnector._get_existing_hb_entities / _get_current_integration_keys - IntegrationManager.disable_integration seed - EntityIntegrationOperations.find_reconnect_candidates / get_removal_entity_ids - DataImportPageView.has_imported per-row check - ImporterConfigureView preview's skip-detection - HomeBoxImporter._run_import_locked / discard_imported_data - ImporterDiscardView count No migration: IMPORT has never shipped in production; the data_source_str column stays for now and will be retired in the follow-up commit that introduces the derived data_source property. One test deleted (test_initial_connect_blocked_by_existing_import_data) because CONNECT-init no longer blocks under the unified model; the canonical IMPORT-init block direction is still covered by test_get_import_blocked_by_existing_connect_data. * Derive data_source from columns; drop EntityDataSource and data_source_str The data_source_str column was a stored discriminator that became fully redundant once IMPORT moved to previous_integration_id: an entity's state is now encoded entirely by which integration_* columns are populated. Removing the column eliminates a class of "stored flag drifted from column reality" bugs and gives the codebase one fewer thing to keep consistent. Model: - Drop the data_source_str field from Entity and the EntityDataSource enum. - Delete migration 0021_entity_data_source (the migration that introduced the column). Import has never shipped, so there is no production data dependency. Dev environments drop the column manually. - Add three derived properties on Entity that name the questions call sites actually ask: * is_external — actively attached to an integration * has_integration_provenance — carries record of an integration it came from (imported or detached) * is_imported / is_detached — synonyms today (same column shape); separate names document intent at the call site, parallel to the manager methods external_for / imported_for / detached_for. Sweep: - Remove every entity.data_source = ... write site. The four service converters and HbEntityFactory no longer need to set it — the columns ARE the state. preserve_with_user_data likewise. - Replace the one production read site in hb_external_view_resolver with `if not entity.is_external`. - Test fixtures: drop data_source_str=... kwargs from Entity.objects.create calls (the column is gone). Two legacy "integration_id + data_source_str=INTERNAL" mixes converted to provenance form (previous_integration_id + previous_integration_name). - Replace data_source assertions with property checks. UI rename: "Detached from X" -> "From X". The unified model treats imported and detached as the same state, so the badge text needs to read accurately for both origin stories. Two templates plus comment references. Sync-result reporting fixes (surfaced during manual testing of the above): - ZM _create_zm_entity now appends the service entity to created_list (was hidden in info_list). The user-facing "created" count and detail now match what got persisted. Placement input remains unaffected — _sync_impl builds it from monitor entities only. - ZM _create_monitor_entity and Frigate _create_camera_entity gate created_list.append on is_fresh_create. Auto-reconnect rebuilds no longer double-list the row in both created_list and reconnected_list (HomeBox/HASS did not have this bug; their _rebuild_integration_components calls the converter directly, bypassing the per-connector _create_entity). - Both result-modal placement CTAs now read "Place new items" without a count. The previous count came from created_list length, which includes reconnected entities and (for ZM) the service entity — but the placement modal correctly excludes those. Action prompt > misleading count.	2026-05-25 18:56:50 -05:00
Tony C	35c6f8532c	Integration Capabilities framework (#355) (#359) ... * Rename Connect vocabulary across the integrations framework Three coordinated rename groups (no behavior change): - test_connection -> validate_access on IntegrationGateway and all service overrides + call sites. The generalized verb covers both network probes (today) and access checks for future Import-capability integrations. - is_initial_import -> is_initial_connect across the synchronizer protocol, view layer, templates, and tests. Internal sync / synchronize vocabulary is preserved. - UI labels updated: IMPORT -> CONNECT (in live-discovery contexts), REFRESH -> UPDATE (button) / Update (inline link), Initial Import -> Initial Connect, Refresh Result -> Update Check Result. The no-items hero copy reads 'No items found.' with non-redundant sub-text. Prerequisite for the broader integration-capabilities work in #355. * Restructure HomeBox importer/ — Connect code to connector/, dormant preserved The reachable Connect-mode synchronizer code moves from services/homebox/importer/ to connector/: - HomeBoxSynchronizer relocates to connector/hb_sync.py. - New HbEntityFactory in connector/hb_entity_factory.py carries the two entity-create/update classmethods previously on HbImporter. The dormant attribute-population code stays in importer/ for #358: - HbImporter retains only its four attribute-side classmethods with a narrowed docstring noting their dormant status. - The synchronizer's removed attribute-orchestration methods extract as HbAttributeSync in importer/hb_attribute_sync.py with a docstring marking them preserved for the HomeBox Importer in #358. No behavior change. (#355) * Phase 3 (#355): split integrations app into umbrella + connect sub-package Relocate the Connect-capability machinery (gateways, synchronizers, sync-check, views, view mixins, entity operations, attribute edit context, sync-result, and related transient models) from hi/integrations/ into hi/integrations/connect/. Umbrella-level modules (models, forms, enums, manager, urls, exceptions, etc.) stay at hi/integrations/ as the framework root. This re-shapes the package along capability lines ahead of the Import capability being added by #358: the umbrella holds capability-agnostic framework code; each capability sub-package holds the per-capability machinery. No behavior changes. All call sites updated to the new module paths; all 3141 tests pass and lint is clean. * Tolerate unreadable sync-check cache entries The sync-check probe state is stored in Redis as a pickled SyncCheckResult dataclass. Any class rename, module move, or shape change can leave stale entries that fail to deserialize on read, which previously propagated ModuleNotFoundError / AttributeError / UnpicklingError straight out through the manage page. Wrap the cache.get in get_state with a broad except: log the failure, evict the bad key so the next request runs clean, and degrade to a cache miss. The next probe cycle (or a Refresh) writes fresh state. The cache is informational drift state, not load-bearing — a soft miss is the right failure mode. * Phase 4 (#355): add EntityDataSource enum + Entity.data_source field EntityDataSource is the entity-wide provenance signal (INTERNAL when HI owns the representation; EXTERNAL when an upstream system does). Surfaces via Entity.data_source — a typed accessor over the new data_source_str CharField, following the existing entity_type pattern. Migration 0021 backfills integration-attached entities that disallow internal attributes to EXTERNAL. Today this picks out only HomeBox- Connect entities; HA / ZM / Frigate / native entities stay INTERNAL. No UI impact yet. Phase 6 will consume data_source for cross-capability transition prompts and Import-discard scoping. * Phase 5 (#355): add IntegrationCapability enum + metadata declaration IntegrationCapability declares what an integration brings to HI: CONNECT (live mirror) or IMPORT (one-shot pull). IntegrationMetaData now carries a `capabilities` frozenset declaring which capability set the integration participates in; defaults to {CONNECT} since that is the existing behavior for all four current integrations. An empty set raises at construction time so a future integration cannot silently register with no capability. ALL_CAPABILITIES is provided as a frozenset convenience over all enum members for callers that want "available everywhere" semantics. No UI impact yet. Per-attribute capability filtering deferred until Phase 6 introduces the consumer. * Phase 6 (#355): make capability filtering explicit at integration list and attribute formset sites Extend IntegrationAttributeType to accept an optional 8th tuple element declaring which IntegrationCapabilities the attribute applies to (defaults to ALL_CAPABILITIES, so existing declarations are unchanged). Convert the previously-implicit "everything is Connect" assumption into explicit capability filters at four sites that all run in Connect-mode context today: * IntegrationSelectView (Enable Integrations picker modal) * IntegrationManageView (Config tab — enabled-integrations list and default-integration selector) * IntegrationAttributeItemEditContext (formset construction, used by both the Configure modal and the Config tab attribute pane) IntegrationManager.get_integration_data_list() and get_default_integration_data() gain an optional `capabilities` frozenset kwarg. IntegrationAttributeItemEditContext gains a required `capability` __init__ arg; its formset queryset is now filtered to attribute rows whose AttributeType includes that capability. The same class can be reused for the Import edit context in #358 by passing IntegrationCapability.IMPORT. Behavior unchanged today (all integrations are Connect, all attributes default to ALL_CAPABILITIES) but the filter rules are now testable with mixed-capability fixtures. The Importer protocol, workflow/views/templates, and cross-capability transition prompts originally listed in Phase 6 absorb into #358 (the HomeBox Importer) — they are easier to design with a concrete consumer driving them. * Phase 7 (#355): collapse Configure + Pre-Sync + Sync into one CONNECT click The initial-connect flow becomes a single workflow under IntegrationEnableView: the user fills the configure form once and clicks CONNECT, which validates access, saves attributes, enables the integration, and runs the synchronizer in one pass — landing directly on the sync-result modal. The previous Configure → Pre-Sync confirm → Sync handshake is gone from this path, along with the REVIEW CONFIG round-trip affordance. Extracted IntegrationViewMixin.render_sync_result so the synchronizer-invocation + cache-invalidation + sync-result rendering is shared between IntegrationEnableView (initial-connect) and IntegrationSyncView (update-check). Action button label flips from the conditional CONFIGURE/UPDATE to a fixed CONNECT. IntegrationPreSyncView and its template still serve the manage-page UPDATE / CONNECT-when-no-entities paths unchanged; the removal_summary policy choice (RETAIN MISSING / REMOVE MISSING) for user-data-bearing integrations is preserved end-to-end. Net behavior change: one fewer modal click on first-time setup. * Phase 8 (#355): align documentation with the new Connect/Update vocabulary Update the user-facing button-label references in the per-integration docs (Refresh / REFRESH → Update / UPDATE; Import action → Connect) and the developer-facing references where they were tied to user surfaces. Internal developer vocabulary stays as "sync" — that is the implementation-side term. Add a Capability Model subsection to integration-guidelines.md covering IntegrationCapability, the optional per-attribute capability restriction, and EntityDataSource. Points at the enum modules; keeps the explanation brief since the code is the authoritative source. Update the per-integration doc template's vocabulary instruction so new integrations start with the right terms. * Address review feedback: sharpen vocabulary and add IntegrationEnableView POST tests Four fixes from the pre-PR review pass: - Sharpen EntityDataSource enum descriptions to disambiguate from raw value provenance: "HI owns the editable representation" / "Upstream constrains HI-side edits" (the actual semantics codified by migration 0021's allow_internal_attributes-based backfill). - Replace the self-contradicting "first **Import**" / "post-import state" in docs/integrations/_template.md with the **Connect** vocabulary the same template instructs authors to use. - Reword frigate.md troubleshooting heading "Update imports zero cameras" → "Update finds zero cameras" to stop mixing the UPDATE button label with the "imports" verb. - Add two IntegrationEnableView.post() tests covering the Phase 7 collapse: the happy path verifies enable + synchronizer.sync() fire and the sync-result modal renders; the synchronizer-less path verifies enable still flips but no sync-result is returned.	2026-05-24 16:58:18 -05:00
Tony C	0eee15864c	Frigate NVR integration (v1) (#347) ... * Scaffold Frigate integration (issue #233) Lays down the directory + plumbing for hi.services.frigate and hi.simulator.services.frigate so both apps load under their respective Django settings, the FrigateGateway is auto-discovered by IntegrationManager, and the Frigate simulator module appears in the simulator's services-tab dispatch. All concrete methods stubbed (test_connection returns 'not yet implemented (scaffolding)', do_work records a healthy heartbeat without polling, etc.). Includes user-facing and developer-facing doc stubs structured around the v1 scope, and adds Frigate to the integrations landing page. * B1: Frigate simulator Camera entity definition Adds FrigateCameraSimEntityFields plus two SimState subclasses — FrigateCameraMotionState (MOVEMENT) and FrigateCameraObjectPresenceState (DISCRETE). Registers a single Camera SimEntityDefinition in FRIGATE_SIM_ENTITY_DEFINITION_LIST so the simulator's services-tab UI exposes 'ADD Frigate ENTITY' -> Camera. The object-presence choices carry raw Frigate-style labels (person / car / truck / dog / cat / package) plus a deliberately unmapped 'unicorn' label so the HI converter's 'other' bucket stays on the demo path. Canonical OBJECT_PRESENCE bucketing happens later on the HI side, not at the simulator boundary. * B2: Frigate simulator /api/config endpoint Adds FrigateSimCamera accessor wrapper (mirrors ZmSimMonitor), FrigateSimulator.get_sim_cameras(), and a Frigate-shape ConfigView at GET /services/frigate/api/config that returns the active profile's cameras keyed by camera_name. Real Frigate emits a much larger config document; HI's integration only needs the cameras map plus a stable JSON shape to recognize as Frigate. * Fix Frigate attribute lookup to use integration_key, not integration_attr_type The IntegrationAttribute model exposes integration_key (an IntegrationKey wrapping integration_id + integration_name), not a direct integration_attr_type field. The lookup pattern from HassManager._build_hass_attr_type_to_attribute_map is the canonical shape: build {attr.integration_key: attr} once, look up by constructing IntegrationKey(integration_id, str(attr_type)). * C1: FrigateClient.ping() + get_cameras() real implementations Both methods route through a private _get_config() helper that hits /api/config, validates status code + content-type, and parses JSON with boundary-level diagnostic errors (same shape as HassClient). ping() returns None on success; get_cameras() returns one dict per entry in the response's 'cameras' map carrying the camera name and raw per-camera config. 14 unit tests covering init / auth header / success / non-2xx / HTML body / malformed JSON / network error / empty cameras / multi-camera / config passthrough / missing or malformed 'cameras' field. * C2: Wire FrigateManager.test_connection to FrigateClient.ping Builds a temporary FrigateClient via FrigateClientFactory from the proposed attributes and exercises ping() against /api/config. ValueErrors from the client (configuration or response-shape failures) surface as failure messages verbatim; other exceptions get prefixed 'Connection error: '. Bounded by the provided timeout_secs so the Configure form fails interactively. * C3: Frigate sync creates camera entities with Movement + ObjectPresence sensors Adds the new EntityStateType.OBJECT_PRESENCE (canonical value range: NONE / PERSON / CAR / ANIMAL / PACKAGE / OTHER) plus the matching EntityStateValue members, type-default EntityStateRole, role-ordering entry (placed after MOVEMENT/PRESENCE), and a HiModelHelper.create_object_presence_sensor factory. FrigateManager grows a real _reload_implementation that builds the FrigateClient from current integration attributes, a frigate_client property gated on integration-enabled state, plus _to_integration_key and _frigate_integration_key helpers for stable key construction. FrigateSynchronizer._sync_impl mirrors the ZM pattern: fetch cameras from /api/config via the client, reconcile against existing HI camera entities by integration_key, create CAMERA entities for new upstream cameras (with Movement + ObjectPresence sensors), preserve the user-editable name on update, and remove entities whose upstream camera disappears. check_needs_sync wires the Issue #283 drift probe. 6 sync tests cover client-missing, create, idempotent re-sync, remove on upstream drop, user-name preservation, and multi-camera placement input. * Use camera friendly_name for HI entity display name Real Frigate cameras carry a friendly_name on each camera's config for display; the camera key itself is the snake_case technical identifier. Sync now prefers config.friendly_name when present and falls back to the camera key otherwise. Simulator's /api/config endpoint emits friendly_name from the camera sim_entity's display name field (the operator-set 'name' on the FrigateCameraSimEntityFields), keeping the simulator faithful to Frigate's response shape. * D1: Frigate simulator events synthesized from motion-state toggles Mirrors the ZM event-manager pattern: per-camera FrigateSimEventHistory ring buffers held by a FrigateSimEventManager singleton, keyed by camera_name. Events are pure transient FrigateSimEvent dataclasses (no DB, no CRUD UI) — the operator drives them by toggling the camera's existing Motion sim-state. FrigateSimulator.set_sim_state intercepts Motion-state changes and calls into the event manager. The event's label comes from the camera's current ObjectPresence value at the moment motion-ON fires (label fixed at first detection, matching real Frigate's behavior closely enough). ObjectPresence changes during an open event don't relabel it; operator toggles motion off-then-on to start a new event with a new label. Following the project convention of no simulator-internal unit tests — verification is via manual hands-on testing through the running simulator. A smoke run confirmed motion-ON opens a labeled event and motion-OFF closes it. * D2: Frigate simulator /api/events list + detail endpoints GET /api/events returns the active profile's events as a top-level JSON array (Frigate's convention), sorted newest-start_time-first. Supports the 'after' query parameter (epoch seconds) used by the HI polling cursor; other Frigate filters (before / cameras / labels / zones) are accepted and silently ignored — they're not yet on the HI integration's read path. GET /api/events/<id> returns a single event object; 404 for unknown ids so the HI client can distinguish 'event doesn't exist' from a broken Frigate. Verified end-to-end via the test client: motion-state toggles synthesize events that surface through both endpoints with correct ordering, filtering, and 404 behavior. * E1: FrigateClient.get_events() + get_event() real implementations Both methods route through a shared _get_json(path, params=None) helper that handles status-code, content-type, and JSON-parse validation in one place (the prior _get_config now delegates to it). get_events accepts the polling cursor 'after' and an optional limit as query params; get_event maps the URL onto /api/events/<id>. Non-2xx responses (including 404 from get_event for unknown ids) surface as ValueError with the status code in the message so monitor / test_connection paths can record meaningful diagnostics. 11 new unit tests covering get_events (empty / parsed list / 'after' + 'limit' query params / no-params / not-a-list response / non-2xx) and get_event (parsed dict / URL targeting / 404 / not-a-dict response). End-to-end smoke against the live simulator verifies both methods return data toggled in via the Motion sim-state. * E2: FrigateMonitor polls events and emits MOVEMENT sensor responses Mirrors the ZoneMinder monitor's order of operations exactly (those invariants took several debugging rounds to settle on the ZM side): 1. Fetch events with after=<polling-cursor>. 2. Two-phase collate: filter out fully-processed events, then partition the rest into open vs closed. 3. Per-camera aggregation: pick ONE canonical event per camera (earliest open -> ACTIVE; latest closed -> IDLE). Prevents multiple events on one camera from overwriting each other in the sensor-response map. 4. Emit one MOVEMENT SensorResponse per aggregated state with correlation_role (START/END) + correlation_id set. 5. Emit explicit IDLE SensorResponses for cameras that produced no events this cycle so their state doesn't go stale. 6. Cursor advance: hold back to the earliest open event's start when any open events remain; otherwise advance to the latest closed event's end; with NO events do NOT advance (would risk missing an event that started right after the poll). _fully_processed_event_ids TTLCache prevents closed events from being re-emitted on subsequent polls. Open events are deliberately NOT marked fully-processed — the cursor keeps them visible so we can detect the close transition. FrigateEvent.from_api_dict parses one /api/events JSON record into the typed wrapper. AggregatedCameraState (in frigate_models.py for parity with ZM's AggregatedMonitorState) carries the canonical event + all_events for cache bookkeeping. FrigateManager grows sync + async get_events / get_cameras shims used by the monitor. 21 unit tests cover aggregation across the open x closed cross-product, sensor-response correlation, the no-event IDLE case, cache bookkeeping, FrigateEvent.from_api_dict, and the four cursor-advance invariants (no events / all closed / has open / fully-processed re-skip). * E3: FrigateMonitor emits OBJECT_PRESENCE alongside MOVEMENT FrigateConverter.to_canonical_object_class gets the real lookup table: person -> OBJECT_PERSON; car/truck/bus/motorcycle/bicycle -> OBJECT_CAR; dog/cat/bird/horse/sheep/cow/bear/deer/raccoon/fox/ squirrel/rabbit -> OBJECT_ANIMAL; package -> OBJECT_PACKAGE; anything else -> OBJECT_OTHER. Lookup is case-insensitive. Empty / unknown labels deliberately bucket into OBJECT_OTHER (not OBJECT_NONE) so a custom-model class that hasn't been bucketed yet still surfaces in HI as 'something is here'. FrigateMonitor's per-camera sensor-response generation now emits TWO responses per aggregated state: the MOVEMENT response that already existed (ACTIVE/IDLE with correlation_role/id) plus a new OBJECT_PRESENCE response carrying the canonical class. IDLE camera states emit OBJECT_NONE — once an event closes, nothing is being detected. The idle-for-unseen-camera pass also pairs MOVEMENT IDLE with OBJECT_NONE so the object-presence state stays fresh even on quiet poll cycles. 7 converter tests + expanded monitor tests cover the table mapping, case-insensitive lookup, the unknown -> OTHER fallback, and per-cycle OBJECT_PRESENCE emission across ACTIVE / IDLE / no-events paths. * Frigate: collapse MOVEMENT + OBJECT_PRESENCE into a single per-camera state Frigate's data model couples motion to object detection — its events API never emits motion-without-class — so a separate MOVEMENT sensor on the HI side would always mirror OBJECT_PRESENCE. Removing it also eliminates the simulator's ordering trap where the operator had to toggle Motion before picking a label or the synthesized event landed with no object class. OBJECT_PRESENCE is now the single per-camera state across the integration. Any non-OBJECT_NONE value implies motion is currently happening and names the class; OBJECT_NONE means no current detection. Simulator: drop the Motion sim-state entirely; the ObjectPresence discrete control drives the event lifecycle directly. Transitioning to a new label closes the current event and opens a new one; transitioning to 'none' closes the current event. * Frigate: snapshot URLs (live + event) and gateway VideoSnapshot plumbing Real Frigate exposes still images at two paths the HI integration now consumes: - /api/<camera>/latest.jpg — the most recent decoded frame, used by the presentation layer as a live snapshot via the gateway's get_entity_video_snapshot method. - /api/events/<id>/snapshot.jpg — the frame captured at the time of detection, attached to the OBJECT_PRESENCE SensorResponse as source_image_url so the alert / history views can show what the camera saw. URL helpers on FrigateManager build both with a cache-bust timestamp so a re-rendered <img> tag with the same src refetches rather than showing the prior frame. Both helpers return None when the client isn't built yet, propagating "no snapshot capability" through the gateway. Simulator endpoints serve synthesized placeholder JPEGs stamped with the camera or event identifiers so artifacts viewed inside HI are obviously coming from the simulator. /api/events/<id>/snapshot.jpg is routed ahead of the existing /api/events/<id> JSON route so the trailing path segment dispatches correctly. MP4 clip endpoint deferred — v1 displays use snapshot-as-stream (via video_snapshot_stream_fps) rather than recorded-event playback. * Frigate: switch camera entities to snapshot-as-stream and clear manager health UNKNOWN Two bugs surfacing on the Live Feed page: (1) Frigate cameras were flagged with has_video_stream=True even though v1 has no native MJPEG/RTSP — without go2rtc, Frigate exposes only JPEG endpoints. The Live View template routed to the native- stream branch which fell through to "Live Stream Unavailable" because no gateway implementation existed. Flip the canonical sync shape to has_video_snapshot=True + video_snapshot_stream_fps=1.0 (matching the HASS camera pattern) so the pane routes through the snapshot-as-stream branch and consumes the snapshot URL plumbing that already exists. _update_entity self-heals existing rows by flipping the stale flag off and backfilling the missing fps. (2) The integration banner showed "Frigate: Warning" while the FrigateMonitor was reporting Healthy. The manager registers itself as an API health provider, but its api_health_status default (UNKNOWN) leaks into the aggregate — UNKNOWN counts as not-healthy under ALL_SOURCES_HEALTHY, dragging the aggregate to WARNING. Set the slot explicitly on every _reload_implementation: HEALTHY on a clean client build, DISABLED when the integration row is off or absent, UNAVAILABLE on a build failure. * Frigate: wrap manager API methods in api_call_context FrigateManager.get_cameras and get_events were making bare HTTP calls without feeding the manager's API-health slot. Result: total_calls stayed at 0, the slot stayed UNKNOWN, and the aggregate health stayed at WARNING. Wrap both methods in api_call_context so each successful poll increments the slot and moves it to HEALTHY — matching how ZoneMinderManager.get_zm_states/monitors/events handle the same concern. The reload-time update_api_health_status calls are retained so the slot also reflects a clean configuration even before the first poll lands. Comment cleanup pass on Phase F additions. * Frigate: seed simulator profiles (empty, baseline, baseline-changed, volume) Adds the standard four-profile catalog for the Frigate sub-app so the sync result modal can be exercised through created / removed / detached / reconnected transitions, plus a 10-camera stress profile. * OBJECT_PRESENCE styling: collapse onto Movement vocabulary, give priority over MOVEMENT OBJECT_PRESENCE is conceptually MOVEMENT with a larger value space. For visual purposes the two collapse onto the same status_value vocabulary (active / recent / past / idle), reusing existing CSS rules. Where a camera has both — uncommon but possible — OBJECT_PRESENCE wins because it carries finer-grained information. Changes: - EntityStateDisplayData gains _get_object_presence_status_style: same decay logic as MOVEMENT, discriminator is value != OBJECT_NONE (any detected class counts as active). - DEFAULT_ENTITY_STATE_ROLE_ORDER and ConsoleManager.STATUS_ENTITY_STATE_PRIORITY put OBJECT_PRESENCE ahead of MOVEMENT so it wins priority pickers. - EntityPairingManager.CREATE_BY_DEFAULT_MAP gets OBJECT_PRESENCE so a Frigate camera auto-creates an AREA delegate on first placement, matching MOVEMENT behavior. - Camera panel declares OBJECT_PRESENCE in optional_roles with an object_data alias. Templates prefer object_data over motion_data for the chip, and render OBJECT_PRESENCE as a standard state row (via the fallback row template) so the user can see the current class and click through to history. Tests cover all four OBJECT_PRESENCE decay states (active / recent / past / idle) plus the "other class is still active" branch. * Frigate: auto-create OBJECT_PRESENCE EventDefinition on sync (gated by Add Alarm Events) Brings Frigate to parity with ZoneMinder and Home Assistant on the default alarm-wiring concern. Sync now creates an OBJECT_PRESENCE EventDefinition per camera when the operator opts in via the new ADD_ALARM_EVENTS integration attribute. The default rule is the conservative EQ OBJECT_PERSON — person-only alarms — to sidestep the EventClauseOperator vocabulary's lack of NEQ/IN (tracked separately in #346). Operators wanting broader detection rules can author additional EventDefinitions in the UI. Changes: - FrigateAttributeType.ADD_ALARM_EVENTS: BOOLEAN attribute, default off. - FrigateManager._attribute_map: populated during reload from the current integration attributes; backs the new should_add_alarm_events property. - HiModelHelper.create_object_presence_event_definition: new helper, mirrors create_movement_event_definition with the conservative OBJECT_PERSON default value. - FrigateManager.OBJECT_PRESENCE_EVENT_PREFIX: integration-key prefix for the auto-created event definitions. - frigate_sync.py: creates the EventDefinition when should_add_alarm_events is true. Tests cover should_add_alarm_events (default off / true / false attribute values) and the sync's create / skip / idempotent-on- refresh paths. * Frigate: implement per-camera Detect on/off control end-to-end Outbound (HI -> Frigate): - FrigateClient.set_camera_detect issues POST /api/<camera>/detect/set with the Frigate-side state value ('ON' / 'OFF'). New _post helper. - FrigateConverter.hi_control_to_detect_state — explicit HI on/off -> Frigate ON/OFF mapping. Independent vocabularies; no string transforms. - FrigateManager.set_camera_detect wraps the client call in api_call_context. - FrigateController.do_control dispatches the detect-controller integration_name shape; surfaces translation and manager errors as IntegrationControlResult.error_list entries. - frigate_sync creates an on/off controller per camera paired with the OBJECT_PRESENCE sensor. Inbound (Frigate -> HI): - FrigateConverter.detect_enabled_to_hi_value reads /api/config's cameras.<name>.detect.enabled boolean and maps to HI's on/off. - FrigateMonitor emits a detect SensorResponse per camera each poll cycle so the panel display reflects the actual upstream state (closes the prior gap where the toggle would snap back for lack of confirming sensor data). Simulator: - New FrigateCameraDetectState sim-state per camera, defaulting to 'ON' (real Frigate's startup default). Independent of the object-presence event lifecycle so each signal exercises HI's wiring on its own. - /api/config exposes cameras.<name>.detect.enabled from the sim-state value. - POST /api/<camera>/detect/set writes the value into the sim-state via the simulator so the operator observes the round-trip in the simulator UI. * Rename sensor / event video-clip / video-snapshot fields to disambiguate from entity-level live-feed vocabulary The Sensor/SensorResponse/SensorHistory/EntityStateHistoryValue fields named has_video_stream / provides_video_stream / source_image_url predated the entity-level live-feed model (has_video_stream / has_video_snapshot / video_snapshot_stream_fps), and the name overlap made it hard to tell at a call site whether you were dealing with live observation (entity level) or historical event playback (sensor/event level). Rename to make the level explicit and reserve "event_" prefix for historical playback concepts: | Old | New | |----------------------------------------------|----------------------------------| | Sensor.provides_video_stream | Sensor.provides_event_video_clip | | (new) | Sensor.provides_event_video_snapshot | | SensorHistory.has_video_stream | SensorHistory.has_event_video_clip | | SensorHistory.source_image_url | SensorHistory.event_video_snapshot_url | | SensorResponse.has_video_stream | SensorResponse.has_event_video_clip | | SensorResponse.source_image_url | SensorResponse.event_video_snapshot_url | | EntityStateHistoryValue.has_video_stream | EntityStateHistoryValue.has_event_video_clip | | EntityStateHistoryValue.provides_video_stream | EntityStateHistoryValue.provides_event_video_clip | Entity-level has_video_stream is unchanged — it remains the live-feed capability flag, distinct from the per-event flags above. DB migration uses RenameField only (no column drops); the lone new column is the Sensor.provides_event_video_snapshot capability flag to mirror provides_event_video_clip on the snapshot side. * Frigate: event video clip playback (and HI render-layer MP4 support) HI's Video Browse rendered event recordings via <img> alone — fine for ZM (multipart MJPEG) but breaks Frigate's MP4 clip semantics. Add an explicit media-type axis on VideoStream and a parallel <video> render branch so MP4 event recordings play natively. VideoStreamType vocabulary: URL -> MJPEG (multipart, <img> renders) (new) MP4 (recording, <video> renders) OTHER OTHER Render layer: - entity_video_sensor_history.html branches on stream_type: <video> for MP4 (with controls + autoplay + muted + playsinline), <img> for MJPEG (existing path, unchanged). - video-timeline.js replay button: <video> branch seeks to 0 and plays; <img> branch keeps the cache-bust URL trick (ZM). - ZoneMinder gateway updated from VideoStreamType.URL to MJPEG to match the new explicit naming. Frigate side: - FrigateApi.EVENT_CLIP_PATH_TEMPLATE + FrigateManager.get_event_clip_url. - FrigateEvent.has_clip / has_snapshot parsed from /api/events payload; default True for backward compat with older Frigate. - FrigateMonitor propagates has_clip from the canonical event into the OBJECT_PRESENCE SensorResponse's has_event_video_clip. - FrigateGateway overrides get_sensor_response_video_stream to return a VideoStream(stream_type=MP4, source_url=<clip url>). - FrigateSync flags the OBJECT_PRESENCE sensor with both provides_event_video_clip and provides_event_video_snapshot. Simulator side: - New POST endpoint /api/events/<id>/clip.mp4 serves a pre-generated H.264 baseline placeholder (~14 KB) with labeled frame counter and ticking clock so the operator can verify the playback round-trip. Tests cover the wire-to-model has_clip parsing, the monitor's propagation, the gateway's MP4 stream return shape, the sync's sensor flags, and the controller's URL routing. * Replace SensorResponse event_video_snapshot_url with has_event_video_snapshot flag Bring event snapshot URLs in line with the clip URL pattern: gateways build the URL on demand from the event id rather than storing it. Robust to operator-side base URL changes — an integration host relocation no longer leaves historical SensorHistory rows with stale URLs pointing at the previous host. Symmetry now between the two affordances: Clip has_event_video_clip (flag) gateway.get_sensor_response_video_stream (URL builder) Snapshot has_event_video_snapshot (flag, new) gateway.get_sensor_response_event_snapshot_url (URL builder, new) Migration: 0013_replace_event_video_snapshot_url_with_flag adds the boolean, backfills True for every row that had a non-empty URL, then drops the URL column. Frigate side: - has_event_video_snapshot propagates from FrigateEvent.has_snapshot through the monitor to the SensorResponse. - FrigateGateway.get_sensor_response_event_snapshot_url builds the URL from the response's correlation_id. ZoneMinder side: - has_event_video_snapshot set True on START + END responses when detail_attrs carries an event_id. - ZoneMinderManager.get_event_snapshot_url returns the portal view=image URL fresh; gateway override builds per render. Render side: - Templates that displayed event_video_snapshot_url now use the new sensor_response_event_snapshot_url template tag. - alert.py's get_first_visual_content dict no longer carries the URL — templates resolve via the tag. * Drop misleading "video may be processing" caption from snapshot fallback The caption fired unconditionally on the snapshot-fallback branch, but the code has no actual signal distinguishing "clip is being processed" from "no clip exists for this event". For an event that will never have a clip (operator opted out, integration didn't capture one), the caption was misinformation. The surrounding heading + timestamp already communicate what the image is. * Camera modal: render MOVEMENT state row alongside chip The camera modal already rendered an OBJECT_PRESENCE state row below the stream so Frigate users could click through to event history. ZM-style MOVEMENT was driving the chip but had no row treatment, leaving the operator with no path to the history / video-browse view. Render both roles' rows when present. * Rename event-playback "recording" vocabulary to "clip" for consistency with model fields The model layer settled on "clip" (Sensor.provides_event_video_clip, SensorResponse.has_event_video_clip, etc.) but the CSS / JS / template layer was still named around "recording" from an earlier iteration. Future readers had to do an unnecessary translation step between the two halves of the same concept. View-layer renames: CSS .hi-video-recording -> .hi-video-clip .hi-video-recording-replay -> .hi-video-clip-replay .hi-video-recording--mp4 -> .hi-video-clip--mp4 HTML data-video-recording (attr) -> data-video-clip Tmpl video_recording_replay_button -> video_clip_replay_button JS videoRecordingReplayBuster() -> videoClipReplayBuster() img.dataset.videoRecordingSrc -> img.dataset.videoClipSrc Plus prose / docstring / aria-label / alt-text updates from "video recording" to "video clip" in the same view-layer files. Unchanged (different meanings of "recording"): HassApi.CAMERA_STATE_RECORDING — Home Assistant wire-value verbatim. TestFrigateManagerHealthRecording — recording-of-health concept. auto-view.js "interaction recording" — UI tracking concept. weather/daily_weather_tracker — weather-domain concept. * Sensor Response Details modal: add STATUS / HISTORY footer nav The modal is reached from the status and history modals, so a back-nav path was missing. Add STATUS / HISTORY / DONE footer buttons keyed on the underlying entity, mirroring the Entity Edit modal's footer shape. * Document correlation_id as sensor-scoped, not globally unique Make explicit that correlation_id pairs START/END readings within a single sensor's history but is not guaranteed unique across sensors or integrations. The integration's upstream event_id namespace is opaque to us; ZM and Frigate could theoretically produce overlapping strings. Every lookup MUST be scoped by sensor. * Integration config: seed and repair attribute order_id from enum definition order IntegrationAttribute rows had order_id=0 across the board, so the config page render relied on row-id tiebreak — definition-order preserved only by coincidence. Frigate's mid-life addition of ADD_ALARM_EVENTS exposed the brittleness when row creation ordering diverged from the operator-facing enum order. LabeledEnum auto-numbers members 1, 2, 3, ... in definition order, so attribute_type.value is exactly the order_id we want. Set it on new rows in _create_integration_attribute and self-repair existing rows in ensure_all_attributes_exist so any installed environment heals on next startup without a separate migration. * Frigate: pack event metadata into SensorResponse detail_attrs Mirror ZoneMinder's pattern of attaching event metadata to the SensorResponse so HI's event-detail UI surfaces it as label/value rows. Per FrigateDetailKeys: event id, start time, object class, score, sub-label, zones; plus duration on END responses (omitted on START since the event is still open). The pre-existing SNAPSHOT_URL / CLIP_URL keys in FrigateDetailKeys were never populated — URLs are generated by the gateway on demand. Drop them from the constants class so future readers don't expect them in detail_attrs. * Frigate: switch detect toggle to PUT /api/config/set (real Frigate's actual endpoint) The previous implementation called POST /api/<camera>/detect/set, which does not exist in real Frigate's HTTP API (checked against v0.17.1 source). Real Frigate exposes detect toggling only through the admin-only PUT /api/config/set runtime config update; the HTTP detect-toggle our simulator was honoring was a fabrication. Wire change: Old (fabricated): POST /api/<camera>/detect/set?state=ON New (real): PUT /api/config/set?cameras.<name>.detect.enabled=true The new endpoint requires admin role on real Frigate — the operator must supply an admin-scoped Authorization header in the existing FrigateAttributeType.AUTH_HEADER attribute. HI vocabulary translation: HI 'on' <-> Frigate 'true' (was 'ON') HI 'off' <-> Frigate 'false' (was 'OFF') Simulator endpoint replaced: CameraDetectSetView removed, ConfigSetView added at PUT /api/config/set. It recognizes the cameras.<name>.detect.enabled key in the query string and updates the camera's detect sim-state accordingly. Other config keys are accepted but no-op (real Frigate would persist them; the simulator has no config to mutate). * Frigate sim-state: correct the stale "wire representation" comment The previous comment claimed the sim-state's "ON"/"OFF" values WERE Frigate's wire vocabulary. After the detect-toggle switch to PUT /api/config/set, Frigate's wire vocabulary is "true"/"false". The sim-state values are now an internal-only representation that the simulator's ConfigSetView / ConfigView translate to/from Frigate's wire format. The values stay as-is (no data migration concern — sim-state is runtime, not persisted). * Frigate: drop the per-camera Detect on/off control surface from v1 Real Frigate has no transient detect-toggle endpoint reachable over HTTP — the only mechanism is PUT /api/config/set, which is a persistent config edit. Mapping a HI control surface onto a config edit creates surprising operator-facing behavior (the toggle permanently rewrites Frigate's YAML, with last-writer-wins between HI and Frigate's own UI). HI controls have generally meant transient state. Removing v1's Detect controller end-to-end: - FrigateClient.set_camera_detect, _put, _post all gone (none used elsewhere). - FrigateConverter.hi_control_to_detect_enabled and detect_enabled_to_hi_value mappings removed. - FrigateManager.set_camera_detect and DETECT_CONTROLLER_PREFIX removed. - FrigateController is back to a stub returning "no control mapping" for any incoming key. - frigate_sync no longer creates the on/off controller per camera. - FrigateMonitor no longer emits the detect SensorResponse on each poll cycle. - Simulator's FrigateCameraDetectState and ConfigSetView removed; /api/config no longer emits detect.enabled. Future revision can expose the transient detect toggle through Frigate's MQTT frigate/<cam>/detect/set topic, which is what Frigate's own UI button uses for the live update path. * Frigate: replace cursor-hold polling with per-id open-event tracking Frigate's /api/events?after=T filters strictly on start_time > T, so a cursor-hold approach (cursor pinned at an open event's start_time to keep it visible across polls) excludes that event from subsequent scans even after it closes — the END transition was never observed, the SensorHistory correlation pair was never completed, and the operator saw a bare OBJECT_NONE heartbeat instead. The replacement keeps the cursor strictly monotonic over each event's start_time, tracks open events by id in a dedicated set, and refreshes each via GET /api/events/<id> until closed (or force-closed on 404 or the MAX_OPEN_EVENT_AGE_SECS timeout). FrigateClient._get_json now raises Http404 directly so callers can distinguish a vanished event from a transport failure. Removes the AggregatedCameraState model and the per-camera open/closed aggregation that the old pipeline needed. Simulator's get_events_after switched to strict > to match real Frigate so behavior validated against the simulator matches production. Frigate user-facing docs gain CSP / event-clip-codec / authentication caveat sections; dev docs document the new polling model and Frigate's strict-> filter semantics. * Frigate monitor: post-review cleanups and rename for clarity Code-review follow-ups on the polling pipeline rewrite. Mostly refactor and naming polish; no behavior change. Refactor: - Phases return (response_map, touched_cameras) tuples instead of threading a mutable cameras_touched set as an out-parameter. - Extract _force_close() helper consolidating the three identical emit-response / mark-camera-touched / drop-from-tracking call sites in the refresh phase. - Drop trailing dead `continue` at end-of-loop bodies. - Cursor-regression guard kept (and its comment rewritten) as defensive protection against upstream contract violations. Rename for clarity: - OpenFrigateEvent → TrackedFrigateEvent (the wrapper holds HI tracking metadata, not just "openness"). - _open_events → _tracked_events; _refresh_open_events_phase → _refresh_tracked_events_phase; matching test class/method names. - Local variable `tracker` → `tracked_event` (stale leftover from the original OpenEventTracker dataclass name). - In the refresh phase, local `event` → `frigate_event` to avoid shadowing the wrapper's `.event` field at the assignment site. Tests: - _PipelineTestBase switched from TestCase to AsyncTaskFastTestCase so per-method asyncio.run() / `import asyncio` are replaced with self.run_async() — picks up the project's event-loop tracking. - Add coverage for cursor no-regression invariant, duplicate-id defensive guard, malformed phase-2 refresh payload, and camera-list fetch failure in the heartbeat phase. Dev doc updated for the renamed symbols.	2026-05-21 09:11:20 -05:00
Tony C	8314eced6e	Issue #280: Per-integration documentation structure and content (#291) ... Establishes a two-track per-integration doc structure (user-facing under docs/integrations/, developer-facing under docs/dev/integrations/) with templates that enforce a consistent set of sections for each integration. Populates the new structure for the three existing user-configured integrations (Home Assistant, ZoneMinder, HomeBox) and updates the integration-addition guideline to require both docs for any new integration. User-facing changes: - New per-integration pages with overview, prerequisites, credential acquisition, configuration values, setup walkthrough, troubleshooting, and known limitations. ZoneMinder's existing CORS and HTTPS/SSL troubleshooting content migrated out of the shared Integrations.md into its own page. - Restructured docs/Integrations.md as a short landing page that centralizes the conditional UI flow for enabling an integration (different button label and location depending on whether any are configured yet) so per-integration pages can link to it instead of restating it. - Conventions documented in the user-facing template: introduce Home Information (HI) on first mention; use 'item' for HI's representation, reserve 'entity' for upstream services that use that term; use 'Import' (first run) and 'Refresh' (subsequent), not 'sync', in user-facing copy. Developer-facing changes: - New per-integration dev docs for ZoneMinder and HomeBox; HA dev doc expanded from a single section to all six required sections. - Dev template explicitly directs writers to keep content high-level and refer to the code for details; existing dev docs follow that principle (key modules listed with one-line roles, no method signatures or field lists duplicated). - Weather integration doc gets a one-paragraph preface marking it as an internal-source exception that does not follow the per-integration template structure. Simulator documentation: - docs/dev/testing/test-simulator.md expanded to cover the simulator's purpose and architecture, the URL mappings for pointing each HI integration at the simulator, and the seed_sim_profiles command (with a profile summary table and a pointer to the command's docstring for the full operator workflow).	2026-05-05 23:02:07 -05:00