17 Commits

Author	SHA1	Message	Date
Tony C	c30c0a99ac	[Feature] External Reference framework — Linked Content tab (#392) ... * Renamed "Attribute Reference" to "External Reference" * New ExternalReference database models and integrations thumbnails. * New external reference linking sans the UI display. * Added thumbnail fallback path for paperless integration. * First version of full new workflow for external reference linking * Refactoring and fixes for new external references feature. * Many refactors and code cleanup after implementation phase of #388 * Tests for new external referencer views. * Simulator improvements for testing extrernal reference linking. * External reference feature: bug fixes and UI refinements. * Added hover for full thumbnail tooltip in reference picker modal. * Added file attribute card reordering and tweaked UI. * Refactoring of Entity/Locati0on edit modals for tabbed content. * Added session var for external reference picker default. * Comment cleanup for Issue #388 * Some fixes from pre-PR review of Issue #388.	2026-06-01 00:07:27 -05:00
Tony C	c752b41139	Add Immich integration (#381) + picker error-channel refactor (#383) ... * First version of the Immich integration. * Comment cleanup. * Fixes from code review recommendations.	2026-05-30 18:59:23 -05:00
Tony C	40f932e088	Paperless-ngx integration (ATTRIBUTE_REFERENCE capability) (#375) ... * ATTRIBUTE_REFERENCE capability + framework backend (#232 phase 1) Adds the framework seam for the new ATTRIBUTE_REFERENCE integration capability, parallel to the existing CONNECT and IMPORT capabilities. Integrations advertising this capability contribute a search-and-attach affordance to HI's attribute-edit UI: the operator searches the upstream corpus, multi-selects matching results, and HI creates one regular TEXT attribute per selection on the host Entity or Location. Pieces: - hi.integrations.enums.IntegrationCapability.ATTRIBUTE_REFERENCE joins CONNECT and IMPORT. - hi.integrations.referencer is the new package owning the per-capability abstract class (IntegrationAttributeReferencer), the picker-only result dataclass (AttributeReferenceResult), and the shared wire-field constants (WireField). - IntegrationGateway.get_attribute_referencer() returns None by default; integrations override to opt in. - Two POST endpoints under /api/integrations/referencer/: - search/<integration_id> dispatches a query to the integration's referencer and returns JSON results for the picker UI to render. - attach/<integration_id> takes a list of selections and an attribute-owning item (Entity or Location, keyed by ItemType) and creates one TEXT attribute per selection. Both views require edit-mode; the attach view rejects unsupported item types, malformed payloads, and stale references to integrations that no longer advertise the capability. No UI yet — that's phase 2. With a stub referencer, the search and attach endpoints are exercisable end-to-end and the test suite covers both happy paths and error branches (empty query, limit clamp/parse, upstream exception → 502, unknown integration → 404, unsupported item_type → 400, malformed selections → 400, title truncation to model max_length, etc.). * Phase 2: ATTRIBUTE_REFERENCE picker (HiModal + antinode) Adds the operator-facing picker that searches an enabled ATTRIBUTE_REFERENCE integration and attaches selected results as TEXT attributes on an Entity or Location. The picker is a single HiModalView with server-driven multi-select state; the picker URL takes no integration_id and the form discovers / selects integrations internally so the action-bar surface stays a single "Link" button regardless of how many referencers are configured. - Picker view (search re-render via antinode partial swap; attach via refresh_response). Multi-select state carried in the form across re-searches (selections_json + visible_url + result_url + remove_url). - Picker modal + body templates; integration selector shown only when more than one referencer is enabled. - has_attribute_referencers template tag; "Link" button wired into the Entity and Location edit content body action bars. - View tests cover GET, search re-render, attach, multi-select state transitions, and stale-integration rejection. * Phase 3: paperless-ngx simulator with parametric search responses Adds a paperless-ngx stub simulator that responds to any documents search with a synthetic, parametrically-shaped result list. Unlike the entity-shaped simulators, paperless has no SimEntity rows — it contributes only via ATTRIBUTE_REFERENCE, so its state is a small ephemeral settings dataclass on the singleton (same lifecycle as HomeBox's _api_version): result count, mime mix, thumbnails on/off, snippets on/off, and optional artificial latency. Operator tunes each knob from an extras-pane form that submits on change. The result generator seeds its RNG from the query string so repeated searches return the same set, and every title carries the query as a visible breadcrumb so the developer can verify the picker is wiring the search input through. /api/documents/<id>/thumb/ returns an inline SVG when thumbnails are on and 404s when off, exercising the picker's fallback-icon path; /documents/<id>/details/ stands in for the per-document view paperless source URLs link to. The shared services/pages/service.html now gates profile CRUD, the ADD ENTITY menu, and the entity list on sim_entity_definition_list being non-empty so ephemeral simulators don't render irrelevant controls. Connect simulators with zero current entities are unaffected — they still have definitions, so the empty-state alert fires as before. * Phase 4: paperless-ngx integration (ATTRIBUTE_REFERENCE) First integration to declare only the ATTRIBUTE_REFERENCE capability. Lets the operator search a paperless-ngx server from inside HI's picker modal and attach matching documents as TEXT-attribute links on existing Entity / Location records. No connector, no importer, no monitors, no manager singleton — the gateway returns a referencer and a single thumbnail proxy view bridges browser-embedded <img> tags to the upstream API. - PaperlessGateway, PaperlessAttributeReferencer, and a thin PaperlessClient over requests. Auth via Authorization: Token <value>. Trailing slash on the configured API URL is forgiving. - ThumbnailProxyView fetches upstream with the configured token and streams bytes back so the picker's embedded thumbnails work without exposing the token to the browser. Document source URLs point directly at paperless's web UI — operators authenticate with paperless's own session when clicking the saved link. - Snippet extraction is client-side (paperless returns full content per hit, no excerpt field): ~160-char window around the matched query with ellipses for clipped edges, leading-window fallback when the query does not match verbatim. - Wire-format strings centralized in pl_models.PaperlessApi per project convention. - Tests cover validation, client + factory (including disabled / missing-attribute paths), referencer translation + snippet extraction edge cases, gateway probe (200 / 401 / 5xx / connection error), and proxy view (unconfigured / upstream 404 / upstream auth failure / connection error / success). - Per-integration docs added (user-facing + developer-facing) and linked from the integrations landing page; the lead-in copy is updated to acknowledge the new "attachable references" shape alongside the existing CONNECT / IMPORT integrations. * Phase 5: Content Sources tab + CapabilityGateway abstraction Adds a Content Sources config tab for ATTRIBUTE_REFERENCE-capability integrations (paperless-ngx). Sibling to the renamed Connectors tab. Each tab now belongs to exactly one capability — the two surfaces are semantically different (live mirror vs. on-demand reference) and sharing a page was forcing capability awareness into framework chrome. Page model: per-integration sidebar nav, attribute form with a single ENABLE/UPDATE primary action that always runs schema + upstream-access validation, and a DISABLE button (when enabled) that rides the same form via an ``action=disable`` POST. ENABLE flips is_enabled to True only after the upstream probe succeeds (atomic "nothing changes on failure" semantics). DISABLE skips validation and just flips is_enabled to False. The framework re-renders the form area in place after each action, so the status badge and the primary-button label update without a full page reload. Architectural refactor: introduce ``CapabilityGateway`` as the shared base of ``IntegrationConnector`` / ``IntegrationImporter`` / ``IntegrationAttributeReferencer``. The three peers had nothing in common before, leaving cross-capability concerns (description text, action-bar template fragment) with no natural home. The new base carries ``capability`` (class attribute), ``get_description()``, and ``get_attribute_actions_template_name()`` — each capability owns its own defaults. The connector's existing sync-flow ``get_description (is_initial_connect)`` is renamed to ``get_sync_description`` so it does not collide with the general capability description. ``IntegrationAttributeItemEditContext`` now takes a ``CapabilityGateway`` instance instead of a capability enum and exposes it in the template context, so ``integrations/panes/integration_edit_content_body.html`` is now capability-agnostic: a single ``{% include %}`` of whatever fragment the active capability returns. The previous inline CONNECT-only health badge moves into ``connect_attribute_actions.html``. attr.js threads the submit ``event.submitter`` into ``FormData`` so multi-submit-button forms (e.g., the new ENABLE / DISABLE form) include the clicked button's ``name=value`` in the POST body. This matches native browser behavior. UI labels: the existing ``Integrations`` tab becomes ``Connectors`` (internal CONNECT routes / module names unchanged); the new tab is ``Content Sources`` (code naming uses ``reference`` to align with the capability and the surrounding ``referencer/`` directory). * Content Sources page: source description, label-stale fix, history/restore view capability cleanup - Per-source description below the integration title: "Find related content in <label> and link to it." Uses the integration label directly so future ATTRIBUTE_REFERENCE integrations get sensible copy automatically. - Rebuild the attribute edit context after the first-time ENABLE flips is_enabled to True; otherwise the async response re-rendered with the stale "ENABLE" button label and disabled-state action bar even though the integration was now enabled. - IntegrationAttributeHistoryInlineView / IntegrationAttributeRestoreInlineView pass capability_gateway=None instead of get_connector(). These ops target a specific attribute by id, so the capability filter inside the edit context is never consulted — the previous hardcoded CONNECT was a no-op that silently mismatched any non-CONNECT integration. * Apply attribute action-bar button styling to anchor-styled buttons The .attr-v2-action-buttons CSS targeted only <button> elements, so the new LINK anchor on entity / location edit pages rendered smaller and top-aligned instead of matching its <button> siblings (Add File, Add Info). Extend the selectors to cover a.btn inside the action bar and add explicit inline-flex centering so the icon and text line up with the other buttons. * Picker: JS-owned selection state, split search / attach endpoints The previous picker maintained selection state on the server, which meant ticking a result card did nothing visible until the next form submit — operators saw "Add 0 References" stuck disabled after their first check and assumed the picker was broken. Multi-selection now behaves as one transaction: JS owns the in-memory selection set, search calls are async within that state context, and the only POST that touches the server is the final Add submit. attr-picker.js (new, in js_hi_grid_header_content) owns: - The selection set (per-modal WeakMap keyed by the picker DOM). - Chip-row rendering (the one piece of HTML the JS generates). - The Add button's label and enabled state. - The async search request: POST to a new search endpoint that returns ONLY the result-cards partial; JS swaps it into the results container and re-applies checked state to any newly- visible result whose URL is already selected. - The Add submit: serialize the selection set into a hidden selections_json input, then route through antinode's public API (AN.hideModalIfNeeded + AN.post) so modal cleanup and the {refresh: true} response flow through the framework. Form deliberately omits data-async because antinode binds at <body> level and would otherwise race attr-picker's document-level submit handler. Server endpoints split: picker GET stays, plus new search and attach POST endpoints. Old multi-select bookkeeping (_compute_selections, visible_url[], remove_url, action=attach button) deleted along with the WireField class. Client/server constant sharing now follows the project's established DIVID pattern: ATTR_PICKER_* entries in hi.constants.DIVID (server) and Hi.ATTR_PICKER_* in main.js (client). Templates, view code, and JS all read from these — no more magic strings to drift between sides. Simulator fix: paperless's _generate_results used document_id = index + 1, which meant different queries produced overlapping ids (and thus URLs). The JS keys selections by source_url, so the first result of query A and the first result of query B were treated as the same document. Document id is now a hash of (query, index) so different queries produce distinct ids while repeating the same query stays stable — matching real paperless's "same document re-found in multiple searches" behavior. CSS: a.btn-styled buttons inside .attr-v2-action-buttons inherit the same min-height / padding / font-size / border-radius / focus / hover / icon-spacing rules as their <button> siblings, so the anchor-styled LINK button on Entity / Location edit pages lines up with Add File and Add Info. * Picker: footer-anchored chips + Add, sticky search, seeded initial query Layout reshape using Bootstrap's built-in ``modal-dialog-scrollable`` opt-in (only the picker modal opts in; other modals are unaffected): - Footer is now the commitment zone. It holds the selection chip row above a Cancel/Add row. The footer stays anchored regardless of how far the operator scrolls through results, so the accumulated selections and the Add button are always visible together. - Body holds only the search form and the results container. The search form sticks to the top of the body's scroll region (position: sticky) so refining the query stays one click away no matter how far down in the result list the operator is. - The picker-root class moved up to the modal-dialog so JS handlers in both body (result cards) and footer (chip-X buttons, attach form) can ``closest()`` to the same picker instance. CSS additions, scoped to ``modal-dialog-scrollable`` so the flex-shrink contract works correctly with this project's custom ``.modal-title`` (vs. Bootstrap's default ``.modal-header``): - ``.modal-dialog-scrollable .modal-title``, ``.modal-footer`` get ``flex-shrink: 0`` so the title and footer keep their natural height when the body grows. Without this the title squashed to near-zero height on tall result lists. - ``.hi-attr-picker .modal-body { padding-top: 0 }`` so the sticky search form sits flush against the title — otherwise the modal-body's default padding-top showed scrolling content through the gap above the sticky element. Initial-render seeded with owner's name: - The picker GET now runs the same ``_search_upstream`` the search endpoint uses, with ``owner.name`` as the query, and seeds the results container with the resulting ``picker_results.html`` partial. Operator opens the picker and sees relevant matches immediately — no retype, no extra click. Same template + same context variables (``query``, ``results``) as the search endpoint so initial-render and re-search-render are structurally identical. * Picker: source banner always rendered, wider modal, "Link Content" copy The picker UI now has the same shape for one source or many — the modal doesn't change structurally when a second ATTRIBUTE_REFERENCE integration is configured. - Source banner: replaces the previous conditional ``<select>`` in the multi-source case + hidden input in the single-source case. Always-rendered Bootstrap dropdown whose button face shows ``[logo] [name] [caret]``. With one source the caret is hidden and the button is disabled (the visual stays consistent without inviting a no-op click). On selection: attr-picker.js updates the banner face, sets the hidden ``integration_id`` input, and re-submits the search form — natural for "search the current query against a different source." - Modal width bumped to ``hi-modal-dialog-700`` so the title bar fits "Link Content" and the result cards have room to breathe. - Operator-facing copy aligned with the "Content Sources" tab: modal title is now constant "Link Content" (was conditional "Link References" / "Add from <name>"); attach button reads "Add N Link(s)"; the chip-row empty hint reads "No content selected yet."; the action-bar button on Entity / Location edit pages reads "Link Content". - DIVID + main.js gain matching ``ATTR_PICKER_SOURCE_*`` entries for the banner classes, the dropdown-item class, and the three data attributes the dropdown items carry (source id, logo URL, label) so the JS handler can read them without re-hardcoding any strings. - CSS keeps the banner on white so source logos meant for light backgrounds render correctly (the modal title bar uses a colored background and would clash). Single-source modifier class hides the caret and removes the disabled-button visual fade. * Align operator-facing copy with Connectors / Content Sources terminology When the config tab was renamed from "Integrations" to "Connectors" (and the new "Content Sources" tab was added for ATTRIBUTE_REFERENCE integrations), several operator-facing strings continued to use the older "Integration(s)" wording. Updated to match the page labels operators now see: - Connectors page sidebar button: "INTEGRATIONS" → "CONNECTORS" - Empty-state copy on the Connectors page: "no integrations currently configured" / "CONFIGURE INTEGRATIONS" - The "All Integrations" picker modal title → "All Connectors" - Disable-confirm modal: "Disable {label} Integration?" / "This integration has..." / "detached from this integration" / "If you re-configure this integration later..." → use "connector" consistently. - Pre-sync confirm modal: same "This integration has..." / "detached from this integration" → "connector". - Data Import card note: "Also available as an Integration." → "Also available as a Connector." - Data Import empty state: "No integrations support Data Import." → "No data importers currently defined." (the prior wording was technically accurate but ambiguous given the new tab labels; the new wording matches what the operator is looking at). - "Data Import vs. Integration" help modal → "Data Import vs. Connector"; body text refers to the contrast as Data Import vs. Connector and uses "external system" instead of "upstream source". Internal code identifiers (``IntegrationGateway``, ``integration_id``, ``ConfigPageType.INTEGRATIONS_CONNECT``, etc.) unchanged — they were never operator-visible. * Added new link icon and used for new attribute referrer integration. * Lift get_metadata onto CapabilityGateway base ``IntegrationConnector``, ``IntegrationImporter``, and ``IntegrationAttributeReferencer`` all returned the integration's ``IntegrationMetaData`` constant but disagreed on the method name — the connector used ``get_integration_metadata`` while the other two used ``get_metadata``. Same fact, three implementations, two names. Move the contract onto ``CapabilityGateway`` as a single abstract ``get_metadata()``, drop the duplicate declarations from the importer / referencer bases, and rename the connector's variant. - IntegrationConnector: ``get_integration_metadata`` → ``get_metadata`` (with the three internal call sites updated). - IntegrationImporter / IntegrationAttributeReferencer: declaration removed; inherited from CapabilityGateway. - HASS / ZoneMinder / Frigate / HomeBox connectors: method renamed to ``get_metadata``. - test_integration_connector.py / test_sync_check.py: fixture stubs renamed to match. The ``_get_integration_metadata`` private helper in ``integration_tags.py`` is unrelated (it takes an ``IntegrationDetailsModel``, not a gateway) and untouched. * Some integration modal cleanup * File name normalization for new integrations referencer. * Refresh integration-guidelines.md for current capability model The doc had drifted in several places: - Said "two capabilities exist today." Three do — ATTRIBUTE_REFERENCE is the third (paperless-ngx). Added the entry plus a note that ATTRIBUTE_REFERENCE-only integrations land on the sibling "Content Sources" tab rather than "Connectors." - Said the per-capability classes "don't share a base class — commonality is composed through shared helpers." They do now: ``CapabilityGateway`` (``hi/integrations/capability_gateway.py``) is the shared base, carrying ``capability``, ``get_metadata``, ``get_description``, and ``get_attribute_actions_template_name``. - Setup steps 3-5 referenced an obsolete ``IntegrationType`` enum, an ``activate / deactivate / manage`` method surface, and an ``integration_factory.py`` that doesn't exist. Rewrote those steps to describe declaring ``IntegrationMetaData`` with the right capability set, implementing the actual gateway methods (``get_metadata``, ``get_<capability>``, ``validate_configuration``, ``validate_access``, ``notify_settings_changed``), and the auto-discovery flow that replaces factory registration. - Gateway Implementation Patterns described the old activate/deactivate/manage contract and dict return shape — neither still exists. Replaced with a short summary of the current method contract. - Error Handling listed exceptions that don't exist (``ConnectionError``, ``AuthenticationError``, ``DataValidationError``) and missed the ones that do (``IntegrationDisabledError``, ``IntegrationAttributeError``, ``IntegrationConnectionError``). Corrected. - Key Base Classes & Modules had wrong dotted paths (``hi.integration.*`` → ``hi.integrations.*``, ``hi.utils.singleton`` → ``hi.apps.common.singleton``), referenced a non-existent ``IntegrationStatus`` enum and the obsolete factory module. Replaced with the current set, added ``CapabilityGateway`` and ``IntegrationManager``. - Example Integrations listed three; added Frigate and paperless, and noted that paperless has a slimmer file layout because ATTRIBUTE_REFERENCE-only integrations have no monitors / sync / converter / manager. - File layout section gained a short paragraph noting which role-files apply only to CONNECT/IMPORT-shaped integrations. * Accept single-label intranet hostnames in attribute URL linkification Django's URLValidator rejects hostnames without a TLD (e.g. http://cassandra:4100/...), so URLs pointing at LAN hosts saved as attribute values were not rendered as clickable links. Replace the strict validator with a permissive urlparse-based check (http(s) scheme + non-empty netloc) so intranet single-label hostnames are linkified just like IPs and FQDNs. * Refresh paperless-ngx docs for Content Sources tab and slim dev page User-facing changes follow the post-#232 UI: paperless is configured from the Content Sources tab (not Connectors), the action button on item/Location edit pages reads 'Link Content', and the picker's attach button reads 'Add N Links'. Integrations.md routes paperless to its own walkthrough rather than folding a non-Connectors flow into the shared steps. Dev page refocused on design rationale (capability shape, thumbnail proxy vs source-URL passthrough, no-manager, snippet strategy, single-deployment assumption). Dropped module/test enumerations and wire-format strings — those rot and a developer can read the directory or pl_models.py. * Connectors vocabulary sweep in capability-blocked modal and selector The capability-block modal still emitted 'GO TO INTEGRATIONS' / 'disable the … Integration' / 'configured as Integration with', contradicting the new three-tab operator vocabulary (Connectors / Content Sources / Data Import). The empty state in the integrations selector also still said 'No integrations found' under an 'All Connectors' title. Aligned both to 'Connector(s)' and updated the matching importer-view test assertion.	2026-05-27 12:22:47 -05:00
Tony C	ea991d691a	Implement HomeBox Importer; refactor integration framework (#360) ... * Phase 1 (#358): rename connect/ → connector/ and lift capability-agnostic files to umbrella root Mechanical refactor with no behavior change, setting up the layout for the new importer/ subpackage that arrives in Phase 2. The connect/ subpackage is renamed to connector/ for symmetry with the forthcoming importer/. Five files that were already capability- agnostic are lifted from connector/ up to the integrations/ umbrella root so neither capability subpackage will depend on the other: - placement_request.py - integration_attribute_edit_context.py - entity_operations.py - integration_data.py - integration_gateway.py The remaining connector/ contents (views, synchronizer, monitors, sync_check, sync_result, view_mixins, external_view_data, plus the Connect-mode controller/manage-view-pane stubs and user_data_detector) stay capability-specific. view_mixins.py stays in connector/ as a single file; a future refactor could split the capability-agnostic accessor methods up to umbrella, but the call-site benefit is small and bundling that with this rename would obscure the move. The defensive Disable-side data_source predicate originally listed in Phase 1 is deferred to Phase 7. The naive EXTERNAL filter would break HA/ZM/Frigate Disable (their entities are INTERNAL), so the guard needs to be conditional on the integration declaring both capabilities — designing that alongside Phase 7's symmetric block- modal capability-conditional detection. * Phase 2 (#358): Importer protocol skeleton Lands the abstract Importer base, transient models (CandidateItem, IntegrationDiscardResult), and IntegrationImportResult dataclass. Wires get_importer() into IntegrationGateway as the parallel of get_synchronizer(), defaulting to None. No live consumer yet — the HomeBox concrete implementation arrives in Phase 3. IntegrationImportResult is a parallel dataclass to IntegrationSyncResult, not a subclass. Import-specific shape: items_imported_count / items_skipped_count + imported_list + info/error lists + placement_input. No reconnect/detach/remove concepts (Import is add-only). * Phase 3 (#358): HomeBox concrete Importer + capability-aware entity factory Lands HomeBoxImporter as the first concrete consumer of the Phase 2 Importer protocol, plus the supporting infrastructure for it to share HomeBox's entity-creation code with the CONNECT-mode synchronizer. Capability-aware HbEntityFactory: - Moved from connector/ to shared/ (used by both Connect and Import now). - New ``capability`` parameter on create_models_for_hb_item routes to the correct flags: CONNECT keeps the read-only-mirror flags (can_user_delete=False, allow_internal_attributes=False, data_source=EXTERNAL); IMPORT sets HI-owned flags (deletable, editable, data_source=INTERNAL). - Fixes a latent post-#355 gap: data_source was never set at entity creation time. Phase 4 of #355 backfilled existing rows; new HomeBox-Connect entities created after #355 would have defaulted to INTERNAL until now. HbConverter payload defaults flipped to CUSTOM + is_editable=True. The Connect-mode copy-on-sync architecture (pre-#354) used these methods to produce PREDEFINED read-only attributes. Post-#354, the only live caller is the new Import code, which wants user-owned attributes. The methods are now unambiguously Import-flavored. HomeBoxImporter (services/homebox/importer/homebox_importer.py): - get_candidate_items: lightweight summary fetch, excludes archived. - run_import: per-entity transaction inside the existing integrations_sync exclusion lock; skip-if-already-imported by integration_name; per-item failures aggregate into error_list. - discard_imported_data: filters by data_source=INTERNAL so any coexisting Connect-mode entities are untouched. Dormant hb_attribute_sync.py (224 lines) deleted. Replaced by a ~15-line populate_attributes_for_imported_entity helper next to the existing HbImporter classmethods — Import is add-only so the sync-style dispatch / update / remove logic no longer applies. HbMetaData.capabilities now declares both CONNECT and IMPORT. HomeBoxGateway.get_importer() returns the importer instance. * Phase 4 (#358): Data Import config tab + Configure form (preview-side) Lands the user-visible Data Import tab parallel to Integrations, plus the configure-form workflow up through preview rendering. Confirm → run is Phase 5. New tab: ConfigPageType.DATA_IMPORT (url_name 'integrations_import_home', label 'Data Import'). URL prefix is integrations_import_* — matches the eventual rename of integrations_home → integrations_connect_home the framework will adopt. DataImportPageView: flat list, one row per IMPORT-capable integration. Per-row CONFIGURE button (always) and DISCARD button (only when imported entities exist for that integration_id with data_source=INTERNAL). Dual-capability rows (HomeBox today) carry a terse "Also available as an Integration" note linking to a bottom-of- page explanation contrasting Data Import vs Integration (Connect). ImporterConfigureView: credentials form (reuses IntegrationAttributeItemEditContext with capability=IMPORT, action button labeled IMPORT). POST validates configuration + access via the same validate_attributes_extra_helper Connect uses, then fetches candidates via Importer.get_candidate_items(), computes new vs. skipped counts by integration_name match against existing HI entities, and renders the preview modal. Preview modal: "Would import N new items. M existing items would be skipped." CONFIRM IMPORT and CANCEL actions. CONFIRM posts to the integrations_import_run URL — stub view returns 501 in this phase; Phase 5 wires the actual run. ImporterRunView and ImporterDiscardView land as 501-returning stubs so the URL reversals from this phase's templates resolve cleanly; Phase 5 and 6 fill in the implementations. view_mixins.py moved from connector/ up to umbrella root so both Connect and Import views can extend IntegrationViewMixin. The render_sync_result method on it is only called by Connect views; Import views never touch it. No import-time Connect coupling — the synchronizer.sync() call inside render_sync_result is runtime-only. * Phase 5(a) (#358): Confirm → Run → Result modal + placement reuse Wires the IMPORT workflow end-to-end. After the preview modal's CONFIRM IMPORT, the importer runs, the result modal renders with imported / skipped counts, and a "Place N new items" CTA hands off to the existing placement flow when entities were created. Importer base gains group_entities_for_placement (default ungrouped, mirrors IntegrationSynchronizer's same-named method). HomeBoxImporter stays with the default — HomeBox has no inherent grouping at v1 (Labels and Locations are metadata-on-entity). HomeBoxImporter._run_import_locked now tracks created entities and populates result.placement_input via group_entities_for_placement, mirroring HomeBoxSynchronizer's _sync_impl pattern. ImporterRunView.post: - Runs importer.run_import() inside a try/finally that invalidates the metadata + sensor-response caches afterward (mirrors Connect's render_sync_result). - Builds the placement URL via the existing PlacementUrlParams. - Reuses the existing integrations_placement URL — HomeBox is dual-capability so its synchronizer is available for the placement view's group_entities_for_placement call. import_result.html: counts hero (Imported / Skipped), per-item list, errors. "Place N new items" CTA on success; OK button on the no-imports / errors paths. Modeled on sync_result.html but trimmed for Import's narrower outcome shape. Manual verification: end-to-end Import works after the manager picks up newly-saved credentials. Phase 5(b) addresses the is_enabled / manager-reload race that surfaces this timing issue and the symmetric one in the Connect-enable flow. * Phase 5(b) (#358): decouple Connect-mode is_enabled from credential loading The HomeBoxManager (shared/) was checking the Connect-mode is_enabled flag inside _load_attributes and raising IntegrationDisabledError to nullify hb_client. That coupling produced two related bugs: 1. Import workflow: after the user disables HomeBox-Connect, HomeBoxImporter.get_candidate_items found hb_client=None and returned empty list — preview showed 0 items even though the upstream simulator had 25. 2. Connect-enable race: after IntegrationEnableView flipped is_enabled=True via enable_integration, the manager's post_save signal was deferred by ~0.1s (DelayedSignalProcessor). The synchronous synchronizer.sync() that immediately followed saw hb_client=None and failed with "integration is disabled" even though the user had just enabled it. Fix: * Strip the is_enabled gate out of HomeBoxManager._load_attributes (and the matching except IntegrationDisabledError branch from _reload_implementation). The Connect-mode enable lifecycle is not a shared/ concern — credentials load whenever they are structurally valid so the IMPORT capability works regardless of Connect-mode state. Connect-side guards (manage view, disable, pause, resume) check integration.is_enabled directly at the view layer, where the capability context is known. * IntegrationManager.enable_integration: synchronously call gateway.notify_settings_changed() after the save+refresh, before the monitor launch. Closes the delayed-signal race so the synchronizer immediately sees fresh credentials. * ImporterConfigureView.post: same synchronous notify after post_attribute_form succeeds, before invoking the importer. Credentials just changed; the manager needs to pick them up before run_import. HASS, ZM, Frigate managers are Connect-only and are left alone — their is_enabled gate is fine because they have no other capability that needs to bypass it. Manual verification: Import workflow shows correct candidate count with Connect disabled; Connect-enable runs sync successfully on the first click. * Phase 6 (#358): Discard flow Wires the DISCARD button on the Data Import page. GET renders a confirmation modal showing the count of imported items; POST runs the importer's discard_imported_data and redirects back to the Data Import page. Single-action confirm (no SAFE/ALL variants). Imported items ARE the user data — the Connect-side preserve-with-user-data semantic doesn't apply. Defensive scoping is in HomeBoxImporter.discard_imported_data (Phase 3): the deletion query filters by data_source=INTERNAL, so any coexisting Connect-mode (EXTERNAL) entities for the same integration_id stay untouched. Confirmed via test. Post-discard cache invalidations mirror the Connect-side Disable cleanup: IntegrationMetadataCache + SensorResponseManager. Sensor- response cache invalidation is symmetric with sync's post-write behavior even though imported items don't have sensors today — keeps the discard path defensive for future Importers that might attach sensors. * Clarify EntityDataSource semantics: every Connect-mode entity is EXTERNAL The Phase 4 (#355) migration used `allow_internal_attributes=False` as the proxy for EXTERNAL, which conflated two separate concerns: "data is sourced from upstream" and "user can edit HI-side attributes." Under the conflation, HA/ZM/Frigate-Connect entities were marked INTERNAL even though their state is sourced from upstream — only HomeBox-Connect (allow_internal_attributes=False) got EXTERNAL. The clarified semantics: data_source=EXTERNAL means data is actively being sourced from an external upstream system, which is true of EVERY Connect-mode integration regardless of whether HI permits user-edited attributes on top. data_source=INTERNAL means no live upstream link — Import-mode (HI-owned post-import) and native entities. | integration | mode | data_source | |-------------------|---------|-------------| | zm/frigate/ha | connect | EXTERNAL | | homebox | connect | EXTERNAL | | homebox | import | INTERNAL | | (native) | n/a | INTERNAL | Changes: * Migration 0021_entity_data_source.py: backfill broadens to mark every integration-attached entity EXTERNAL. Adjusted in place rather than as a fix-up migration since 0021 hasn't been released outside local environments and has no downstream migrations. * HassConverter, FrigateSynchronizer, ZmSynchronizer (both service and monitor entity creation): explicitly set data_source=EXTERNAL on new Connect-mode entities. Closes the post-#355 gap where the field wasn't being set at creation time for these integrations — the migration backfilled existing rows but new rows fell back to the INTERNAL default. * HomeBoxConnector.get_external_view_data: gates on data_source=EXTERNAL. Import-mode HomeBox entities are HI-owned after import, so the live HomeBox view isn't authoritative for them and would mislead the operator. * test_models.py: migration-backfill test renamed and assertions updated to reflect the broadened predicate (HA-like entity now becomes EXTERNAL too). * test_hb_connector.py: HomeBox entity fixture sets data_source=EXTERNAL so the connector's new gate doesn't suppress the connector's existing test surface. Required for Phase 7 (#358 block modal): the modal queries data_source to distinguish Connect-mode entities (EXTERNAL) from Import-mode entities (INTERNAL) for an integration. The old semantics would have miscategorized future dual-capability HA/ZM/ Frigate. * Phase 7 (#358): block modal for cross-capability mode switching When an operator starts an integration's initial-Configure flow for one capability while existing data from the other capability is still present, surface a block modal that routes them to remediate before proceeding. Connect ↔ Import switches are intentionally made harder — the user navigates explicitly, performs the cleanup themselves, then re-clicks CONFIGURE. No combined transition flow. CapabilityBlockViewMixin (in hi/integrations/view_mixins.py): - Single helper render_capability_block_if_conflict that takes only integration_data + capability_being_initiated. The mixin composes all user-facing copy (title, body clauses, navigation link) from per-capability config dicts so block modals read uniformly across both directions and any future capabilities. - Detects the OTHER capability automatically. Only fires when the integration declares both capabilities; single-capability integrations get None. - Counts entities of the OTHER capability's data_source for the integration; zero means no conflict. Connect side: IntegrationEnableView.get checks the block on the initial-Connect path (is_enabled=False). Re-Configure of an already-enabled integration always proceeds — that's not a new mode being initiated. Import side: ImporterConfigureView.get checks the block when no Import entities yet exist for the integration. Re-import (entities already present) always proceeds — that's the standard incremental- import flow. capability_blocked.html: generic block modal template parameterized by title + body clauses + link. Extends modals/action_2_with_cancel so the inherited CANCEL just dismisses. Single-capability integrations (HA/ZM/Frigate today) never trigger the block: the mixin's "other capability not in capabilities" guard short-circuits before any DB query. * Phase 8 (#358): documentation for the Data Import capability User-facing: * docs/DataImport.md (new) — landing page for the Data Import feature; importing/discarding walkthrough; lists supported integrations. Concept-level prose only; no hardcoded button labels so the doc doesn't drift if UI strings change. * docs/Integrations.md — Integrations vs. Data Import comparison pointing at the new landing page. * docs/integrations/homebox.md — added Data Import section describing the alternative mode and when to choose it. Developer-facing: * docs/dev/integrations/data-import.md (new) — framework orientation for the IMPORT capability. Lists where code lives (importer/, view_mixins.py, services/homebox/importer/), how a gateway opts in, and the key design decisions (per-entity transaction, skip-by-integration_name, shared lock, data_source as capability-state signal). Reference-style; points at the code rather than restating API. * docs/dev/integrations/integration-guidelines.md — Capability Model subsection refreshed: CONNECT and IMPORT now both have concrete pointers; the #355 "reserved for #358" placeholder is gone. * docs/dev/integrations/homebox.md — updated stale module paths to reflect the connector/importer/shared split landed in #355 Phase 3 + #358 Phase 1. Added a short Importer section pointing at the framework doc. * Integration framework cleanup: capability-aligned naming and template structure Rename the Connect/Import vocabulary uniformly across the integration framework now that HomeBox is the dual-capability archetype. Class renames: - IntegrationSynchronizer -> IntegrationConnector (+ Hass/Zm/Frigate/HomeBox subclasses) - Importer -> IntegrationImporter - IntegrationEnableView -> ConnectorConfigureView - HomeBoxConnector (the per-entity external-view resolver) -> HomeBoxExternalViewResolver Gateway method: - IntegrationGateway.get_synchronizer() -> get_connector() File renames (git mv, history preserved): - integration_synchronizer.py -> integration_connector.py - importer.py -> integration_importer.py - homebox/connector/hb_sync.py -> homebox_connector.py - homebox/connector/hb_connector.py -> hb_external_view_resolver.py - corresponding test files URL name renames: - integrations_home -> integrations_connect_home - integrations_enable -> integrations_connect_configure Enum renames (ConfigPageType): - INTEGRATIONS -> INTEGRATIONS_CONNECT - DATA_IMPORT -> INTEGRATIONS_IMPORT View mixin: - IntegrationViewMixin.get_integration_data(integration_id) -> (request, *args, **kwargs) Mirrors EntityViewMixin.get_entity; pulls integration_id from kwargs, raises BadRequest if missing and Http404 if not registered. ~20 call sites adopted the uniform pattern. Template restructure (src/hi/integrations/templates/integrations/): - import/ -> importer/ (avoids Python reserved word, matches sub-package convention) - importer/ now has modals/ subdir matching the pages/ convention - New connector/{modals,pages,panes}/ subtree for Connect-specific templates - Top-level modals/, panes/ now hold only genuinely shared cross-capability templates (capability_blocked, placement flow, attribute-form body) - Editor backup .html~ files removed Misc: - IntegrationImportResult folded into importer/transient_models.py (single-class file with no module-level documentation) * Remove dormant IntegrationManageViewPane seam; rename to ConnectorManageView The per-integration "manage view pane" extension point on IntegrationGateway was dead machinery: all four service subclasses returned templates containing only a {% comment %} block. The seam also pre-dated the capability model — its return type lives under connector/, but the hook was on the capability-agnostic gateway. Rather than relocate the hook to IntegrationConnector, delete the seam entirely (base class, 4 subclasses, 4 stub templates, gateway method, view context plumbing, template include). It can be reintroduced correctly on IntegrationConnector when an integration actually needs per-integration content in the manage page. Also rename the view to reflect that the Manage page is Connect- specific: - IntegrationManageView -> ConnectorManageView - integrations_manage URL name -> integrations_connect_manage * Push Connect-specific hooks from IntegrationGateway to IntegrationConnector IntegrationGateway accumulated eight Connect-mode hooks over time, all returning types that already lived under connector/. Now that the capability model has IntegrationConnector and IntegrationImporter as peer capability classes (cf. the recent IntegrationManageViewPane cleanup), these hooks belong on the connector — not on the capability-neutral gateway. Import-only integrations should not be forced to stub them out. Methods moved from IntegrationGateway to IntegrationConnector: - get_monitor() - get_controller() - get_health_status_provider() - get_entity_video_stream() - get_entity_video_snapshot() - get_sensor_response_video_stream() - get_sensor_response_event_snapshot_url() - get_external_view_data() Per-service implementations relocated from each *Gateway (integration.py) to the corresponding *Connector. Production call sites updated to chain through get_connector() with None-guards where the caller might encounter an Import-only or connector-less integration. The three Connect-side view sites trust the structural invariant. The do_control path raises a clear RuntimeError if no connector or controller is available. File renames to match the relocated class names: - frigate_sync.py -> frigate_connector.py (FrigateConnector) - hass_sync.py -> hass_connector.py (HassConnector) - zm_sync.py -> zm_connector.py (ZmConnector) - corresponding test files IntegrationGateway's surface is now its capability-neutral core: get_metadata, notify_settings_changed, validate_configuration, validate_access, get_connector, get_importer. * Fix re-Configure credentials race; remove redundant block-check pre-gates Two related issues in the Configure flows: 1. ConnectorConfigureView.post relied on enable_integration to fire notify_settings_changed(), but enable_integration only fires it on the disabled->enabled transition. Re-Configure of an already-enabled integration left the singleton manager holding stale credentials, and the subsequent sync ran against them. The post_save signal would eventually nudge via DelayedSignalProcessor, but the 0.1s delay races the immediate sync. Lift the synchronous notify out of enable_integration and into the view, right after post_attribute_form, so both initial-Connect and re-Configure get the nudge. The matching ImporterConfigureView.post already did this; the Connect side was the asymmetric outlier. 2. Both Configure views had a pre-gate that suppressed render_capability_block_if_conflict in certain re-entry paths (is_enabled=True for Connect, has_imported=True for Import). Under the mode-switch invariant the pre-gate is redundant — the block check itself short-circuits to None when there's no conflicting data. The pre-gate's only effect was to silently bypass the block check when the invariant was violated (mixed INTERNAL+EXTERNAL state), letting the user compound the corruption. Remove both pre-gates so the block fires in any mixed-state scenario and the user is told to remediate. Two tests deleted that asserted the now-removed bypass behavior: - test_get_import_not_blocked_when_already_imported - test_already_enabled_does_not_trigger_block The canonical block-fires-on-conflict and re-Configure-works paths remain covered by neighboring tests. * Extract CapabilityConfigureView shared base for both Configure modals ConnectorConfigureView and ImporterConfigureView shared a 5-step GET recipe (block check → ensure_all_attributes_exist → build edit context → render modal) and a 3-step POST recipe (post_attribute_form → error short-circuit → success handler). Pull the shared structure into a new framework-level CapabilityConfigureView in hi/integrations/views.py. Subclasses now declare four class constants (capability, button_label, template_name, error_title) and override handle_post_success() with the per-capability behavior. ConnectorConfigureView.handle_post_success: enable_integration, then sync or redirect-to-manage. ImporterConfigureView.handle_post_success: fetch candidates, compute new-vs-skipped, render preview. Each subclass owns the timing of notify_settings_changed() because the right moment is capability-specific: * Connect-side managers (Frigate/Hass/ZM) gate client (re)build on integration.is_enabled, so the notify must fire AFTER enable_integration — otherwise the manager reloads with is_enabled=False, nulls its client, and the immediately-following sync sees no client. * Import flows fire the notify before reading candidates. The unconditional call in both subclasses also covers the re-Configure race (already-enabled integration), where enable_integration early-returns without nudging and the singleton manager would otherwise keep stale credentials. * Flatten homebox/shared/ up one level The capability-layout convention is: inter-capability files live at the integration's top level (alongside integration.py / apps.py / enums.py), while capability-specific files live under connector/ or importer/. HomeBox's shared/ predated the convention; it's the last holdout. Move the six modules up: homebox/shared/hb_client.py -> homebox/hb_client.py homebox/shared/hb_client_factory.py -> homebox/hb_client_factory.py homebox/shared/hb_converter.py -> homebox/hb_converter.py homebox/shared/hb_entity_factory.py -> homebox/hb_entity_factory.py homebox/shared/hb_manager.py -> homebox/hb_manager.py homebox/shared/hb_models.py -> homebox/hb_models.py shared/__init__.py removed; shared/ directory gone. 21 absolute import-site rewrites plus 2 @patch strings updated. Docs and simulator docstring path references brought in line. * Data Import UI/UX refinements Tab order - Move Data Import to the right of Triggers in the Config tabs (matches natural read order: settings -> integrations -> trigger setup -> data import -> system info). Import preview modal (HomeBox confirm) - Title becomes "<Label> Import" when there's nothing to confirm (instead of the awkward "Import from HomeBox?" question). - Body now handles the no-new-items cases explicitly: "All N upstream items have already been imported." or "No items found to import." instead of "Would import 0 new items." - Single centered DONE button replaces the lone left-aligned CANCEL when the preview is informational only. - Add data-async="modal" to the CONFIRM IMPORT form so it submits through the modal pipeline like every other action on the page. - Show a contextual "Currently N items imported from <Label>." line whenever the integration has prior imports — signals re-import vs. first-time state. View filters the count by data_source=INTERNAL. Data Import page layout - Replace sparse rows with a responsive Bootstrap card grid (col-12 col-sm-6 col-md-4 col-lg-3), 4 columns on large screens. Logo centered above the integration name, action buttons stacked at the card bottom (CONFIGURE always; DISCARD only when has_imported). - New hi-integration-logo--card modifier (72px) for the card-prominent size. - Drop btn-control width:100% from the action buttons (the row layout no longer benefits from it; btn-block within the card stack is what fills card width now). - Remove the always-below-the-fold "Data Import vs. Integration" footer block. - Add DataImportInfoView (HiModalView) at integrations_import_info rendering a static info modal with the same explanation content. - On dual-capability rows, the entire "Also available as an Integration. <info-icon>" message is now a single clickable link that opens the info modal. Capability-blocked modal - Title was getting truncated ("Cannot configure <Label> as <Capability>" doesn't fit a narrow modal). Title is now just the integration label; the "Cannot configure as <Capability>." line moves into the body as an h5 sub-heading above the existing explanatory paragraph. View context drops the pre-composed title key and passes my_label so the template composes the sub-heading. * Tighten data_source scoping at mode-switch invariant seams The mode-switch invariant (a given integration's entities are all EXTERNAL for Connect, or all INTERNAL for Import, never mixed) was enforced at the Configure entry point but several downstream code paths filtered only on integration_id, leaving data-source-blind seams. The same code review also surfaced that the disconnect/ reconnect path mishandles data_source when transitioning between attached/detached states. POST block-check (CapabilityConfigureView.post) - GET ran the block check; POST didn't. A direct POST (cached form, replayed request) would bypass the invariant. POST now re-invokes render_capability_block_if_conflict before delegating to the capability-specific handle_post_success. disable_integration seed (IntegrationManager.disable_integration) - Seed query now filters by data_source_str=EXTERNAL. A Connect-side Disable can no longer silently delete imported INTERNAL data if the invariant ever leaks. Native and detached entities (integration_id NULL) were already outside the seed. HomeBoxConnector sync queries - _get_current_integration_keys (sync-check probe) and _get_existing_hb_entities (sync run) both filter by EXTERNAL. INTERNAL imports under the same integration_id no longer risk being adopted as Connect rows and routed through mass-remove or auto-reconnect. HomeBoxImporter preview/run symmetry - _run_import_locked's existing_integration_names set now filters by INTERNAL to match ImporterConfigureView.handle_post_success's preview predicate. A stale EXTERNAL row can no longer turn an advertised new candidate into a skip at run time. Disconnect/reconnect data_source transition - EntityIntegrationOperations.preserve_with_user_data now flips entity.data_source to INTERNAL on detach. A detached entity is no longer constrained by an upstream system — HI owns the editable representation. The reconnect path is already correct: all four service converters set data_source=EXTERNAL during the rebuild invoked by _rebuild_integration_components. Pre-migration detached rows are also correct: migration 0021's column default of 'internal' applied to rows with no integration_id. Test fixtures - Four entity-creation sites in test_integration_manager.py and test_homebox_connector.py now explicitly set data_source_str=EXTERNAL for integration-attached test entities, since the new EXTERNAL filters no longer match the implicit default. * Move IMPORT to previous_integration_id; introduce named entity-state queries The IMPORT path was overloading integration_id (the live-ownership column) to mean "where this entity was sourced from." That conflation forced data_source_str as a discriminator and made post-import code paths brittle: any query that scoped by integration_id and a data source filter could quietly diverge. Under the model in this commit, the entity's lifecycle state is encoded entirely by which integration_* column is populated: - integration_id set -> EXTERNAL (live Connect) - integration_id NULL, previous set -> imported OR detached (HI-owned, has provenance) - both NULL -> INTERNAL (native) IMPORT now writes to previous_integration_*. integration_id stays NULL for imported rows because no live integration owns them. Manager methods (EntityModelManager) give call sites a vocabulary matching the data-source semantics: - external_for(integration_id) active-Connect rows - imported_for(integration_id) import-side intent - detached_for(integration_id) reconnect-candidate intent - with_integration_provenance(id) any row with this integration's identity in either column imported_for / detached_for share an implementation today (the column shape is the same); each name documents intent at the call site so a future discriminator could differentiate without touching callers. Block check: only IMPORT initiation can collide (it would create new rows; an active-Connect row already exists for the upstream key). CONNECT initiation never needs blocking — sync's reconnect- then-create order adopts any pre-existing provenance entities into the live Connect session. CapabilityBlockViewMixin loses its symmetric dispatch and the _CAPABILITY_* dicts. Post-import placement bug: EntityPlacementService.query_unplaced filtered by integration_id, so imports (integration_id=NULL) were invisible to the placement modal — the user saw "No items found" after a successful import. Switched to with_integration_provenance so both EXTERNAL and imported rows are eligible. Other writes / queries swept to the new managers: - HbEntityFactory.create_models_for_hb_item IMPORT branch - HomeBoxConnector._get_existing_hb_entities / _get_current_integration_keys - IntegrationManager.disable_integration seed - EntityIntegrationOperations.find_reconnect_candidates / get_removal_entity_ids - DataImportPageView.has_imported per-row check - ImporterConfigureView preview's skip-detection - HomeBoxImporter._run_import_locked / discard_imported_data - ImporterDiscardView count No migration: IMPORT has never shipped in production; the data_source_str column stays for now and will be retired in the follow-up commit that introduces the derived data_source property. One test deleted (test_initial_connect_blocked_by_existing_import_data) because CONNECT-init no longer blocks under the unified model; the canonical IMPORT-init block direction is still covered by test_get_import_blocked_by_existing_connect_data. * Derive data_source from columns; drop EntityDataSource and data_source_str The data_source_str column was a stored discriminator that became fully redundant once IMPORT moved to previous_integration_id: an entity's state is now encoded entirely by which integration_* columns are populated. Removing the column eliminates a class of "stored flag drifted from column reality" bugs and gives the codebase one fewer thing to keep consistent. Model: - Drop the data_source_str field from Entity and the EntityDataSource enum. - Delete migration 0021_entity_data_source (the migration that introduced the column). Import has never shipped, so there is no production data dependency. Dev environments drop the column manually. - Add three derived properties on Entity that name the questions call sites actually ask: * is_external — actively attached to an integration * has_integration_provenance — carries record of an integration it came from (imported or detached) * is_imported / is_detached — synonyms today (same column shape); separate names document intent at the call site, parallel to the manager methods external_for / imported_for / detached_for. Sweep: - Remove every entity.data_source = ... write site. The four service converters and HbEntityFactory no longer need to set it — the columns ARE the state. preserve_with_user_data likewise. - Replace the one production read site in hb_external_view_resolver with `if not entity.is_external`. - Test fixtures: drop data_source_str=... kwargs from Entity.objects.create calls (the column is gone). Two legacy "integration_id + data_source_str=INTERNAL" mixes converted to provenance form (previous_integration_id + previous_integration_name). - Replace data_source assertions with property checks. UI rename: "Detached from X" -> "From X". The unified model treats imported and detached as the same state, so the badge text needs to read accurately for both origin stories. Two templates plus comment references. Sync-result reporting fixes (surfaced during manual testing of the above): - ZM _create_zm_entity now appends the service entity to created_list (was hidden in info_list). The user-facing "created" count and detail now match what got persisted. Placement input remains unaffected — _sync_impl builds it from monitor entities only. - ZM _create_monitor_entity and Frigate _create_camera_entity gate created_list.append on is_fresh_create. Auto-reconnect rebuilds no longer double-list the row in both created_list and reconnected_list (HomeBox/HASS did not have this bug; their _rebuild_integration_components calls the converter directly, bypassing the per-connector _create_entity). - Both result-modal placement CTAs now read "Place new items" without a count. The previous count came from created_list length, which includes reconnected entities and (for ZM) the service entity — but the placement modal correctly excludes those. Action prompt > misleading count.	2026-05-25 18:56:50 -05:00
Tony C	35c6f8532c	Integration Capabilities framework (#355) (#359) ... * Rename Connect vocabulary across the integrations framework Three coordinated rename groups (no behavior change): - test_connection -> validate_access on IntegrationGateway and all service overrides + call sites. The generalized verb covers both network probes (today) and access checks for future Import-capability integrations. - is_initial_import -> is_initial_connect across the synchronizer protocol, view layer, templates, and tests. Internal sync / synchronize vocabulary is preserved. - UI labels updated: IMPORT -> CONNECT (in live-discovery contexts), REFRESH -> UPDATE (button) / Update (inline link), Initial Import -> Initial Connect, Refresh Result -> Update Check Result. The no-items hero copy reads 'No items found.' with non-redundant sub-text. Prerequisite for the broader integration-capabilities work in #355. * Restructure HomeBox importer/ — Connect code to connector/, dormant preserved The reachable Connect-mode synchronizer code moves from services/homebox/importer/ to connector/: - HomeBoxSynchronizer relocates to connector/hb_sync.py. - New HbEntityFactory in connector/hb_entity_factory.py carries the two entity-create/update classmethods previously on HbImporter. The dormant attribute-population code stays in importer/ for #358: - HbImporter retains only its four attribute-side classmethods with a narrowed docstring noting their dormant status. - The synchronizer's removed attribute-orchestration methods extract as HbAttributeSync in importer/hb_attribute_sync.py with a docstring marking them preserved for the HomeBox Importer in #358. No behavior change. (#355) * Phase 3 (#355): split integrations app into umbrella + connect sub-package Relocate the Connect-capability machinery (gateways, synchronizers, sync-check, views, view mixins, entity operations, attribute edit context, sync-result, and related transient models) from hi/integrations/ into hi/integrations/connect/. Umbrella-level modules (models, forms, enums, manager, urls, exceptions, etc.) stay at hi/integrations/ as the framework root. This re-shapes the package along capability lines ahead of the Import capability being added by #358: the umbrella holds capability-agnostic framework code; each capability sub-package holds the per-capability machinery. No behavior changes. All call sites updated to the new module paths; all 3141 tests pass and lint is clean. * Tolerate unreadable sync-check cache entries The sync-check probe state is stored in Redis as a pickled SyncCheckResult dataclass. Any class rename, module move, or shape change can leave stale entries that fail to deserialize on read, which previously propagated ModuleNotFoundError / AttributeError / UnpicklingError straight out through the manage page. Wrap the cache.get in get_state with a broad except: log the failure, evict the bad key so the next request runs clean, and degrade to a cache miss. The next probe cycle (or a Refresh) writes fresh state. The cache is informational drift state, not load-bearing — a soft miss is the right failure mode. * Phase 4 (#355): add EntityDataSource enum + Entity.data_source field EntityDataSource is the entity-wide provenance signal (INTERNAL when HI owns the representation; EXTERNAL when an upstream system does). Surfaces via Entity.data_source — a typed accessor over the new data_source_str CharField, following the existing entity_type pattern. Migration 0021 backfills integration-attached entities that disallow internal attributes to EXTERNAL. Today this picks out only HomeBox- Connect entities; HA / ZM / Frigate / native entities stay INTERNAL. No UI impact yet. Phase 6 will consume data_source for cross-capability transition prompts and Import-discard scoping. * Phase 5 (#355): add IntegrationCapability enum + metadata declaration IntegrationCapability declares what an integration brings to HI: CONNECT (live mirror) or IMPORT (one-shot pull). IntegrationMetaData now carries a `capabilities` frozenset declaring which capability set the integration participates in; defaults to {CONNECT} since that is the existing behavior for all four current integrations. An empty set raises at construction time so a future integration cannot silently register with no capability. ALL_CAPABILITIES is provided as a frozenset convenience over all enum members for callers that want "available everywhere" semantics. No UI impact yet. Per-attribute capability filtering deferred until Phase 6 introduces the consumer. * Phase 6 (#355): make capability filtering explicit at integration list and attribute formset sites Extend IntegrationAttributeType to accept an optional 8th tuple element declaring which IntegrationCapabilities the attribute applies to (defaults to ALL_CAPABILITIES, so existing declarations are unchanged). Convert the previously-implicit "everything is Connect" assumption into explicit capability filters at four sites that all run in Connect-mode context today: * IntegrationSelectView (Enable Integrations picker modal) * IntegrationManageView (Config tab — enabled-integrations list and default-integration selector) * IntegrationAttributeItemEditContext (formset construction, used by both the Configure modal and the Config tab attribute pane) IntegrationManager.get_integration_data_list() and get_default_integration_data() gain an optional `capabilities` frozenset kwarg. IntegrationAttributeItemEditContext gains a required `capability` __init__ arg; its formset queryset is now filtered to attribute rows whose AttributeType includes that capability. The same class can be reused for the Import edit context in #358 by passing IntegrationCapability.IMPORT. Behavior unchanged today (all integrations are Connect, all attributes default to ALL_CAPABILITIES) but the filter rules are now testable with mixed-capability fixtures. The Importer protocol, workflow/views/templates, and cross-capability transition prompts originally listed in Phase 6 absorb into #358 (the HomeBox Importer) — they are easier to design with a concrete consumer driving them. * Phase 7 (#355): collapse Configure + Pre-Sync + Sync into one CONNECT click The initial-connect flow becomes a single workflow under IntegrationEnableView: the user fills the configure form once and clicks CONNECT, which validates access, saves attributes, enables the integration, and runs the synchronizer in one pass — landing directly on the sync-result modal. The previous Configure → Pre-Sync confirm → Sync handshake is gone from this path, along with the REVIEW CONFIG round-trip affordance. Extracted IntegrationViewMixin.render_sync_result so the synchronizer-invocation + cache-invalidation + sync-result rendering is shared between IntegrationEnableView (initial-connect) and IntegrationSyncView (update-check). Action button label flips from the conditional CONFIGURE/UPDATE to a fixed CONNECT. IntegrationPreSyncView and its template still serve the manage-page UPDATE / CONNECT-when-no-entities paths unchanged; the removal_summary policy choice (RETAIN MISSING / REMOVE MISSING) for user-data-bearing integrations is preserved end-to-end. Net behavior change: one fewer modal click on first-time setup. * Phase 8 (#355): align documentation with the new Connect/Update vocabulary Update the user-facing button-label references in the per-integration docs (Refresh / REFRESH → Update / UPDATE; Import action → Connect) and the developer-facing references where they were tied to user surfaces. Internal developer vocabulary stays as "sync" — that is the implementation-side term. Add a Capability Model subsection to integration-guidelines.md covering IntegrationCapability, the optional per-attribute capability restriction, and EntityDataSource. Points at the enum modules; keeps the explanation brief since the code is the authoritative source. Update the per-integration doc template's vocabulary instruction so new integrations start with the right terms. * Address review feedback: sharpen vocabulary and add IntegrationEnableView POST tests Four fixes from the pre-PR review pass: - Sharpen EntityDataSource enum descriptions to disambiguate from raw value provenance: "HI owns the editable representation" / "Upstream constrains HI-side edits" (the actual semantics codified by migration 0021's allow_internal_attributes-based backfill). - Replace the self-contradicting "first **Import**" / "post-import state" in docs/integrations/_template.md with the **Connect** vocabulary the same template instructs authors to use. - Reword frigate.md troubleshooting heading "Update imports zero cameras" → "Update finds zero cameras" to stop mixing the UPDATE button label with the "imports" verb. - Add two IntegrationEnableView.post() tests covering the Phase 7 collapse: the happy path verifies enable + synchronizer.sync() fire and the sync-result modal renders; the synchronizer-less path verifies enable still flips but no sync-result is returned.	2026-05-24 16:58:18 -05:00
Tony C	0eee15864c	Frigate NVR integration (v1) (#347) ... * Scaffold Frigate integration (issue #233) Lays down the directory + plumbing for hi.services.frigate and hi.simulator.services.frigate so both apps load under their respective Django settings, the FrigateGateway is auto-discovered by IntegrationManager, and the Frigate simulator module appears in the simulator's services-tab dispatch. All concrete methods stubbed (test_connection returns 'not yet implemented (scaffolding)', do_work records a healthy heartbeat without polling, etc.). Includes user-facing and developer-facing doc stubs structured around the v1 scope, and adds Frigate to the integrations landing page. * B1: Frigate simulator Camera entity definition Adds FrigateCameraSimEntityFields plus two SimState subclasses — FrigateCameraMotionState (MOVEMENT) and FrigateCameraObjectPresenceState (DISCRETE). Registers a single Camera SimEntityDefinition in FRIGATE_SIM_ENTITY_DEFINITION_LIST so the simulator's services-tab UI exposes 'ADD Frigate ENTITY' -> Camera. The object-presence choices carry raw Frigate-style labels (person / car / truck / dog / cat / package) plus a deliberately unmapped 'unicorn' label so the HI converter's 'other' bucket stays on the demo path. Canonical OBJECT_PRESENCE bucketing happens later on the HI side, not at the simulator boundary. * B2: Frigate simulator /api/config endpoint Adds FrigateSimCamera accessor wrapper (mirrors ZmSimMonitor), FrigateSimulator.get_sim_cameras(), and a Frigate-shape ConfigView at GET /services/frigate/api/config that returns the active profile's cameras keyed by camera_name. Real Frigate emits a much larger config document; HI's integration only needs the cameras map plus a stable JSON shape to recognize as Frigate. * Fix Frigate attribute lookup to use integration_key, not integration_attr_type The IntegrationAttribute model exposes integration_key (an IntegrationKey wrapping integration_id + integration_name), not a direct integration_attr_type field. The lookup pattern from HassManager._build_hass_attr_type_to_attribute_map is the canonical shape: build {attr.integration_key: attr} once, look up by constructing IntegrationKey(integration_id, str(attr_type)). * C1: FrigateClient.ping() + get_cameras() real implementations Both methods route through a private _get_config() helper that hits /api/config, validates status code + content-type, and parses JSON with boundary-level diagnostic errors (same shape as HassClient). ping() returns None on success; get_cameras() returns one dict per entry in the response's 'cameras' map carrying the camera name and raw per-camera config. 14 unit tests covering init / auth header / success / non-2xx / HTML body / malformed JSON / network error / empty cameras / multi-camera / config passthrough / missing or malformed 'cameras' field. * C2: Wire FrigateManager.test_connection to FrigateClient.ping Builds a temporary FrigateClient via FrigateClientFactory from the proposed attributes and exercises ping() against /api/config. ValueErrors from the client (configuration or response-shape failures) surface as failure messages verbatim; other exceptions get prefixed 'Connection error: '. Bounded by the provided timeout_secs so the Configure form fails interactively. * C3: Frigate sync creates camera entities with Movement + ObjectPresence sensors Adds the new EntityStateType.OBJECT_PRESENCE (canonical value range: NONE / PERSON / CAR / ANIMAL / PACKAGE / OTHER) plus the matching EntityStateValue members, type-default EntityStateRole, role-ordering entry (placed after MOVEMENT/PRESENCE), and a HiModelHelper.create_object_presence_sensor factory. FrigateManager grows a real _reload_implementation that builds the FrigateClient from current integration attributes, a frigate_client property gated on integration-enabled state, plus _to_integration_key and _frigate_integration_key helpers for stable key construction. FrigateSynchronizer._sync_impl mirrors the ZM pattern: fetch cameras from /api/config via the client, reconcile against existing HI camera entities by integration_key, create CAMERA entities for new upstream cameras (with Movement + ObjectPresence sensors), preserve the user-editable name on update, and remove entities whose upstream camera disappears. check_needs_sync wires the Issue #283 drift probe. 6 sync tests cover client-missing, create, idempotent re-sync, remove on upstream drop, user-name preservation, and multi-camera placement input. * Use camera friendly_name for HI entity display name Real Frigate cameras carry a friendly_name on each camera's config for display; the camera key itself is the snake_case technical identifier. Sync now prefers config.friendly_name when present and falls back to the camera key otherwise. Simulator's /api/config endpoint emits friendly_name from the camera sim_entity's display name field (the operator-set 'name' on the FrigateCameraSimEntityFields), keeping the simulator faithful to Frigate's response shape. * D1: Frigate simulator events synthesized from motion-state toggles Mirrors the ZM event-manager pattern: per-camera FrigateSimEventHistory ring buffers held by a FrigateSimEventManager singleton, keyed by camera_name. Events are pure transient FrigateSimEvent dataclasses (no DB, no CRUD UI) — the operator drives them by toggling the camera's existing Motion sim-state. FrigateSimulator.set_sim_state intercepts Motion-state changes and calls into the event manager. The event's label comes from the camera's current ObjectPresence value at the moment motion-ON fires (label fixed at first detection, matching real Frigate's behavior closely enough). ObjectPresence changes during an open event don't relabel it; operator toggles motion off-then-on to start a new event with a new label. Following the project convention of no simulator-internal unit tests — verification is via manual hands-on testing through the running simulator. A smoke run confirmed motion-ON opens a labeled event and motion-OFF closes it. * D2: Frigate simulator /api/events list + detail endpoints GET /api/events returns the active profile's events as a top-level JSON array (Frigate's convention), sorted newest-start_time-first. Supports the 'after' query parameter (epoch seconds) used by the HI polling cursor; other Frigate filters (before / cameras / labels / zones) are accepted and silently ignored — they're not yet on the HI integration's read path. GET /api/events/<id> returns a single event object; 404 for unknown ids so the HI client can distinguish 'event doesn't exist' from a broken Frigate. Verified end-to-end via the test client: motion-state toggles synthesize events that surface through both endpoints with correct ordering, filtering, and 404 behavior. * E1: FrigateClient.get_events() + get_event() real implementations Both methods route through a shared _get_json(path, params=None) helper that handles status-code, content-type, and JSON-parse validation in one place (the prior _get_config now delegates to it). get_events accepts the polling cursor 'after' and an optional limit as query params; get_event maps the URL onto /api/events/<id>. Non-2xx responses (including 404 from get_event for unknown ids) surface as ValueError with the status code in the message so monitor / test_connection paths can record meaningful diagnostics. 11 new unit tests covering get_events (empty / parsed list / 'after' + 'limit' query params / no-params / not-a-list response / non-2xx) and get_event (parsed dict / URL targeting / 404 / not-a-dict response). End-to-end smoke against the live simulator verifies both methods return data toggled in via the Motion sim-state. * E2: FrigateMonitor polls events and emits MOVEMENT sensor responses Mirrors the ZoneMinder monitor's order of operations exactly (those invariants took several debugging rounds to settle on the ZM side): 1. Fetch events with after=<polling-cursor>. 2. Two-phase collate: filter out fully-processed events, then partition the rest into open vs closed. 3. Per-camera aggregation: pick ONE canonical event per camera (earliest open -> ACTIVE; latest closed -> IDLE). Prevents multiple events on one camera from overwriting each other in the sensor-response map. 4. Emit one MOVEMENT SensorResponse per aggregated state with correlation_role (START/END) + correlation_id set. 5. Emit explicit IDLE SensorResponses for cameras that produced no events this cycle so their state doesn't go stale. 6. Cursor advance: hold back to the earliest open event's start when any open events remain; otherwise advance to the latest closed event's end; with NO events do NOT advance (would risk missing an event that started right after the poll). _fully_processed_event_ids TTLCache prevents closed events from being re-emitted on subsequent polls. Open events are deliberately NOT marked fully-processed — the cursor keeps them visible so we can detect the close transition. FrigateEvent.from_api_dict parses one /api/events JSON record into the typed wrapper. AggregatedCameraState (in frigate_models.py for parity with ZM's AggregatedMonitorState) carries the canonical event + all_events for cache bookkeeping. FrigateManager grows sync + async get_events / get_cameras shims used by the monitor. 21 unit tests cover aggregation across the open x closed cross-product, sensor-response correlation, the no-event IDLE case, cache bookkeeping, FrigateEvent.from_api_dict, and the four cursor-advance invariants (no events / all closed / has open / fully-processed re-skip). * E3: FrigateMonitor emits OBJECT_PRESENCE alongside MOVEMENT FrigateConverter.to_canonical_object_class gets the real lookup table: person -> OBJECT_PERSON; car/truck/bus/motorcycle/bicycle -> OBJECT_CAR; dog/cat/bird/horse/sheep/cow/bear/deer/raccoon/fox/ squirrel/rabbit -> OBJECT_ANIMAL; package -> OBJECT_PACKAGE; anything else -> OBJECT_OTHER. Lookup is case-insensitive. Empty / unknown labels deliberately bucket into OBJECT_OTHER (not OBJECT_NONE) so a custom-model class that hasn't been bucketed yet still surfaces in HI as 'something is here'. FrigateMonitor's per-camera sensor-response generation now emits TWO responses per aggregated state: the MOVEMENT response that already existed (ACTIVE/IDLE with correlation_role/id) plus a new OBJECT_PRESENCE response carrying the canonical class. IDLE camera states emit OBJECT_NONE — once an event closes, nothing is being detected. The idle-for-unseen-camera pass also pairs MOVEMENT IDLE with OBJECT_NONE so the object-presence state stays fresh even on quiet poll cycles. 7 converter tests + expanded monitor tests cover the table mapping, case-insensitive lookup, the unknown -> OTHER fallback, and per-cycle OBJECT_PRESENCE emission across ACTIVE / IDLE / no-events paths. * Frigate: collapse MOVEMENT + OBJECT_PRESENCE into a single per-camera state Frigate's data model couples motion to object detection — its events API never emits motion-without-class — so a separate MOVEMENT sensor on the HI side would always mirror OBJECT_PRESENCE. Removing it also eliminates the simulator's ordering trap where the operator had to toggle Motion before picking a label or the synthesized event landed with no object class. OBJECT_PRESENCE is now the single per-camera state across the integration. Any non-OBJECT_NONE value implies motion is currently happening and names the class; OBJECT_NONE means no current detection. Simulator: drop the Motion sim-state entirely; the ObjectPresence discrete control drives the event lifecycle directly. Transitioning to a new label closes the current event and opens a new one; transitioning to 'none' closes the current event. * Frigate: snapshot URLs (live + event) and gateway VideoSnapshot plumbing Real Frigate exposes still images at two paths the HI integration now consumes: - /api/<camera>/latest.jpg — the most recent decoded frame, used by the presentation layer as a live snapshot via the gateway's get_entity_video_snapshot method. - /api/events/<id>/snapshot.jpg — the frame captured at the time of detection, attached to the OBJECT_PRESENCE SensorResponse as source_image_url so the alert / history views can show what the camera saw. URL helpers on FrigateManager build both with a cache-bust timestamp so a re-rendered <img> tag with the same src refetches rather than showing the prior frame. Both helpers return None when the client isn't built yet, propagating "no snapshot capability" through the gateway. Simulator endpoints serve synthesized placeholder JPEGs stamped with the camera or event identifiers so artifacts viewed inside HI are obviously coming from the simulator. /api/events/<id>/snapshot.jpg is routed ahead of the existing /api/events/<id> JSON route so the trailing path segment dispatches correctly. MP4 clip endpoint deferred — v1 displays use snapshot-as-stream (via video_snapshot_stream_fps) rather than recorded-event playback. * Frigate: switch camera entities to snapshot-as-stream and clear manager health UNKNOWN Two bugs surfacing on the Live Feed page: (1) Frigate cameras were flagged with has_video_stream=True even though v1 has no native MJPEG/RTSP — without go2rtc, Frigate exposes only JPEG endpoints. The Live View template routed to the native- stream branch which fell through to "Live Stream Unavailable" because no gateway implementation existed. Flip the canonical sync shape to has_video_snapshot=True + video_snapshot_stream_fps=1.0 (matching the HASS camera pattern) so the pane routes through the snapshot-as-stream branch and consumes the snapshot URL plumbing that already exists. _update_entity self-heals existing rows by flipping the stale flag off and backfilling the missing fps. (2) The integration banner showed "Frigate: Warning" while the FrigateMonitor was reporting Healthy. The manager registers itself as an API health provider, but its api_health_status default (UNKNOWN) leaks into the aggregate — UNKNOWN counts as not-healthy under ALL_SOURCES_HEALTHY, dragging the aggregate to WARNING. Set the slot explicitly on every _reload_implementation: HEALTHY on a clean client build, DISABLED when the integration row is off or absent, UNAVAILABLE on a build failure. * Frigate: wrap manager API methods in api_call_context FrigateManager.get_cameras and get_events were making bare HTTP calls without feeding the manager's API-health slot. Result: total_calls stayed at 0, the slot stayed UNKNOWN, and the aggregate health stayed at WARNING. Wrap both methods in api_call_context so each successful poll increments the slot and moves it to HEALTHY — matching how ZoneMinderManager.get_zm_states/monitors/events handle the same concern. The reload-time update_api_health_status calls are retained so the slot also reflects a clean configuration even before the first poll lands. Comment cleanup pass on Phase F additions. * Frigate: seed simulator profiles (empty, baseline, baseline-changed, volume) Adds the standard four-profile catalog for the Frigate sub-app so the sync result modal can be exercised through created / removed / detached / reconnected transitions, plus a 10-camera stress profile. * OBJECT_PRESENCE styling: collapse onto Movement vocabulary, give priority over MOVEMENT OBJECT_PRESENCE is conceptually MOVEMENT with a larger value space. For visual purposes the two collapse onto the same status_value vocabulary (active / recent / past / idle), reusing existing CSS rules. Where a camera has both — uncommon but possible — OBJECT_PRESENCE wins because it carries finer-grained information. Changes: - EntityStateDisplayData gains _get_object_presence_status_style: same decay logic as MOVEMENT, discriminator is value != OBJECT_NONE (any detected class counts as active). - DEFAULT_ENTITY_STATE_ROLE_ORDER and ConsoleManager.STATUS_ENTITY_STATE_PRIORITY put OBJECT_PRESENCE ahead of MOVEMENT so it wins priority pickers. - EntityPairingManager.CREATE_BY_DEFAULT_MAP gets OBJECT_PRESENCE so a Frigate camera auto-creates an AREA delegate on first placement, matching MOVEMENT behavior. - Camera panel declares OBJECT_PRESENCE in optional_roles with an object_data alias. Templates prefer object_data over motion_data for the chip, and render OBJECT_PRESENCE as a standard state row (via the fallback row template) so the user can see the current class and click through to history. Tests cover all four OBJECT_PRESENCE decay states (active / recent / past / idle) plus the "other class is still active" branch. * Frigate: auto-create OBJECT_PRESENCE EventDefinition on sync (gated by Add Alarm Events) Brings Frigate to parity with ZoneMinder and Home Assistant on the default alarm-wiring concern. Sync now creates an OBJECT_PRESENCE EventDefinition per camera when the operator opts in via the new ADD_ALARM_EVENTS integration attribute. The default rule is the conservative EQ OBJECT_PERSON — person-only alarms — to sidestep the EventClauseOperator vocabulary's lack of NEQ/IN (tracked separately in #346). Operators wanting broader detection rules can author additional EventDefinitions in the UI. Changes: - FrigateAttributeType.ADD_ALARM_EVENTS: BOOLEAN attribute, default off. - FrigateManager._attribute_map: populated during reload from the current integration attributes; backs the new should_add_alarm_events property. - HiModelHelper.create_object_presence_event_definition: new helper, mirrors create_movement_event_definition with the conservative OBJECT_PERSON default value. - FrigateManager.OBJECT_PRESENCE_EVENT_PREFIX: integration-key prefix for the auto-created event definitions. - frigate_sync.py: creates the EventDefinition when should_add_alarm_events is true. Tests cover should_add_alarm_events (default off / true / false attribute values) and the sync's create / skip / idempotent-on- refresh paths. * Frigate: implement per-camera Detect on/off control end-to-end Outbound (HI -> Frigate): - FrigateClient.set_camera_detect issues POST /api/<camera>/detect/set with the Frigate-side state value ('ON' / 'OFF'). New _post helper. - FrigateConverter.hi_control_to_detect_state — explicit HI on/off -> Frigate ON/OFF mapping. Independent vocabularies; no string transforms. - FrigateManager.set_camera_detect wraps the client call in api_call_context. - FrigateController.do_control dispatches the detect-controller integration_name shape; surfaces translation and manager errors as IntegrationControlResult.error_list entries. - frigate_sync creates an on/off controller per camera paired with the OBJECT_PRESENCE sensor. Inbound (Frigate -> HI): - FrigateConverter.detect_enabled_to_hi_value reads /api/config's cameras.<name>.detect.enabled boolean and maps to HI's on/off. - FrigateMonitor emits a detect SensorResponse per camera each poll cycle so the panel display reflects the actual upstream state (closes the prior gap where the toggle would snap back for lack of confirming sensor data). Simulator: - New FrigateCameraDetectState sim-state per camera, defaulting to 'ON' (real Frigate's startup default). Independent of the object-presence event lifecycle so each signal exercises HI's wiring on its own. - /api/config exposes cameras.<name>.detect.enabled from the sim-state value. - POST /api/<camera>/detect/set writes the value into the sim-state via the simulator so the operator observes the round-trip in the simulator UI. * Rename sensor / event video-clip / video-snapshot fields to disambiguate from entity-level live-feed vocabulary The Sensor/SensorResponse/SensorHistory/EntityStateHistoryValue fields named has_video_stream / provides_video_stream / source_image_url predated the entity-level live-feed model (has_video_stream / has_video_snapshot / video_snapshot_stream_fps), and the name overlap made it hard to tell at a call site whether you were dealing with live observation (entity level) or historical event playback (sensor/event level). Rename to make the level explicit and reserve "event_" prefix for historical playback concepts: | Old | New | |----------------------------------------------|----------------------------------| | Sensor.provides_video_stream | Sensor.provides_event_video_clip | | (new) | Sensor.provides_event_video_snapshot | | SensorHistory.has_video_stream | SensorHistory.has_event_video_clip | | SensorHistory.source_image_url | SensorHistory.event_video_snapshot_url | | SensorResponse.has_video_stream | SensorResponse.has_event_video_clip | | SensorResponse.source_image_url | SensorResponse.event_video_snapshot_url | | EntityStateHistoryValue.has_video_stream | EntityStateHistoryValue.has_event_video_clip | | EntityStateHistoryValue.provides_video_stream | EntityStateHistoryValue.provides_event_video_clip | Entity-level has_video_stream is unchanged — it remains the live-feed capability flag, distinct from the per-event flags above. DB migration uses RenameField only (no column drops); the lone new column is the Sensor.provides_event_video_snapshot capability flag to mirror provides_event_video_clip on the snapshot side. * Frigate: event video clip playback (and HI render-layer MP4 support) HI's Video Browse rendered event recordings via <img> alone — fine for ZM (multipart MJPEG) but breaks Frigate's MP4 clip semantics. Add an explicit media-type axis on VideoStream and a parallel <video> render branch so MP4 event recordings play natively. VideoStreamType vocabulary: URL -> MJPEG (multipart, <img> renders) (new) MP4 (recording, <video> renders) OTHER OTHER Render layer: - entity_video_sensor_history.html branches on stream_type: <video> for MP4 (with controls + autoplay + muted + playsinline), <img> for MJPEG (existing path, unchanged). - video-timeline.js replay button: <video> branch seeks to 0 and plays; <img> branch keeps the cache-bust URL trick (ZM). - ZoneMinder gateway updated from VideoStreamType.URL to MJPEG to match the new explicit naming. Frigate side: - FrigateApi.EVENT_CLIP_PATH_TEMPLATE + FrigateManager.get_event_clip_url. - FrigateEvent.has_clip / has_snapshot parsed from /api/events payload; default True for backward compat with older Frigate. - FrigateMonitor propagates has_clip from the canonical event into the OBJECT_PRESENCE SensorResponse's has_event_video_clip. - FrigateGateway overrides get_sensor_response_video_stream to return a VideoStream(stream_type=MP4, source_url=<clip url>). - FrigateSync flags the OBJECT_PRESENCE sensor with both provides_event_video_clip and provides_event_video_snapshot. Simulator side: - New POST endpoint /api/events/<id>/clip.mp4 serves a pre-generated H.264 baseline placeholder (~14 KB) with labeled frame counter and ticking clock so the operator can verify the playback round-trip. Tests cover the wire-to-model has_clip parsing, the monitor's propagation, the gateway's MP4 stream return shape, the sync's sensor flags, and the controller's URL routing. * Replace SensorResponse event_video_snapshot_url with has_event_video_snapshot flag Bring event snapshot URLs in line with the clip URL pattern: gateways build the URL on demand from the event id rather than storing it. Robust to operator-side base URL changes — an integration host relocation no longer leaves historical SensorHistory rows with stale URLs pointing at the previous host. Symmetry now between the two affordances: Clip has_event_video_clip (flag) gateway.get_sensor_response_video_stream (URL builder) Snapshot has_event_video_snapshot (flag, new) gateway.get_sensor_response_event_snapshot_url (URL builder, new) Migration: 0013_replace_event_video_snapshot_url_with_flag adds the boolean, backfills True for every row that had a non-empty URL, then drops the URL column. Frigate side: - has_event_video_snapshot propagates from FrigateEvent.has_snapshot through the monitor to the SensorResponse. - FrigateGateway.get_sensor_response_event_snapshot_url builds the URL from the response's correlation_id. ZoneMinder side: - has_event_video_snapshot set True on START + END responses when detail_attrs carries an event_id. - ZoneMinderManager.get_event_snapshot_url returns the portal view=image URL fresh; gateway override builds per render. Render side: - Templates that displayed event_video_snapshot_url now use the new sensor_response_event_snapshot_url template tag. - alert.py's get_first_visual_content dict no longer carries the URL — templates resolve via the tag. * Drop misleading "video may be processing" caption from snapshot fallback The caption fired unconditionally on the snapshot-fallback branch, but the code has no actual signal distinguishing "clip is being processed" from "no clip exists for this event". For an event that will never have a clip (operator opted out, integration didn't capture one), the caption was misinformation. The surrounding heading + timestamp already communicate what the image is. * Camera modal: render MOVEMENT state row alongside chip The camera modal already rendered an OBJECT_PRESENCE state row below the stream so Frigate users could click through to event history. ZM-style MOVEMENT was driving the chip but had no row treatment, leaving the operator with no path to the history / video-browse view. Render both roles' rows when present. * Rename event-playback "recording" vocabulary to "clip" for consistency with model fields The model layer settled on "clip" (Sensor.provides_event_video_clip, SensorResponse.has_event_video_clip, etc.) but the CSS / JS / template layer was still named around "recording" from an earlier iteration. Future readers had to do an unnecessary translation step between the two halves of the same concept. View-layer renames: CSS .hi-video-recording -> .hi-video-clip .hi-video-recording-replay -> .hi-video-clip-replay .hi-video-recording--mp4 -> .hi-video-clip--mp4 HTML data-video-recording (attr) -> data-video-clip Tmpl video_recording_replay_button -> video_clip_replay_button JS videoRecordingReplayBuster() -> videoClipReplayBuster() img.dataset.videoRecordingSrc -> img.dataset.videoClipSrc Plus prose / docstring / aria-label / alt-text updates from "video recording" to "video clip" in the same view-layer files. Unchanged (different meanings of "recording"): HassApi.CAMERA_STATE_RECORDING — Home Assistant wire-value verbatim. TestFrigateManagerHealthRecording — recording-of-health concept. auto-view.js "interaction recording" — UI tracking concept. weather/daily_weather_tracker — weather-domain concept. * Sensor Response Details modal: add STATUS / HISTORY footer nav The modal is reached from the status and history modals, so a back-nav path was missing. Add STATUS / HISTORY / DONE footer buttons keyed on the underlying entity, mirroring the Entity Edit modal's footer shape. * Document correlation_id as sensor-scoped, not globally unique Make explicit that correlation_id pairs START/END readings within a single sensor's history but is not guaranteed unique across sensors or integrations. The integration's upstream event_id namespace is opaque to us; ZM and Frigate could theoretically produce overlapping strings. Every lookup MUST be scoped by sensor. * Integration config: seed and repair attribute order_id from enum definition order IntegrationAttribute rows had order_id=0 across the board, so the config page render relied on row-id tiebreak — definition-order preserved only by coincidence. Frigate's mid-life addition of ADD_ALARM_EVENTS exposed the brittleness when row creation ordering diverged from the operator-facing enum order. LabeledEnum auto-numbers members 1, 2, 3, ... in definition order, so attribute_type.value is exactly the order_id we want. Set it on new rows in _create_integration_attribute and self-repair existing rows in ensure_all_attributes_exist so any installed environment heals on next startup without a separate migration. * Frigate: pack event metadata into SensorResponse detail_attrs Mirror ZoneMinder's pattern of attaching event metadata to the SensorResponse so HI's event-detail UI surfaces it as label/value rows. Per FrigateDetailKeys: event id, start time, object class, score, sub-label, zones; plus duration on END responses (omitted on START since the event is still open). The pre-existing SNAPSHOT_URL / CLIP_URL keys in FrigateDetailKeys were never populated — URLs are generated by the gateway on demand. Drop them from the constants class so future readers don't expect them in detail_attrs. * Frigate: switch detect toggle to PUT /api/config/set (real Frigate's actual endpoint) The previous implementation called POST /api/<camera>/detect/set, which does not exist in real Frigate's HTTP API (checked against v0.17.1 source). Real Frigate exposes detect toggling only through the admin-only PUT /api/config/set runtime config update; the HTTP detect-toggle our simulator was honoring was a fabrication. Wire change: Old (fabricated): POST /api/<camera>/detect/set?state=ON New (real): PUT /api/config/set?cameras.<name>.detect.enabled=true The new endpoint requires admin role on real Frigate — the operator must supply an admin-scoped Authorization header in the existing FrigateAttributeType.AUTH_HEADER attribute. HI vocabulary translation: HI 'on' <-> Frigate 'true' (was 'ON') HI 'off' <-> Frigate 'false' (was 'OFF') Simulator endpoint replaced: CameraDetectSetView removed, ConfigSetView added at PUT /api/config/set. It recognizes the cameras.<name>.detect.enabled key in the query string and updates the camera's detect sim-state accordingly. Other config keys are accepted but no-op (real Frigate would persist them; the simulator has no config to mutate). * Frigate sim-state: correct the stale "wire representation" comment The previous comment claimed the sim-state's "ON"/"OFF" values WERE Frigate's wire vocabulary. After the detect-toggle switch to PUT /api/config/set, Frigate's wire vocabulary is "true"/"false". The sim-state values are now an internal-only representation that the simulator's ConfigSetView / ConfigView translate to/from Frigate's wire format. The values stay as-is (no data migration concern — sim-state is runtime, not persisted). * Frigate: drop the per-camera Detect on/off control surface from v1 Real Frigate has no transient detect-toggle endpoint reachable over HTTP — the only mechanism is PUT /api/config/set, which is a persistent config edit. Mapping a HI control surface onto a config edit creates surprising operator-facing behavior (the toggle permanently rewrites Frigate's YAML, with last-writer-wins between HI and Frigate's own UI). HI controls have generally meant transient state. Removing v1's Detect controller end-to-end: - FrigateClient.set_camera_detect, _put, _post all gone (none used elsewhere). - FrigateConverter.hi_control_to_detect_enabled and detect_enabled_to_hi_value mappings removed. - FrigateManager.set_camera_detect and DETECT_CONTROLLER_PREFIX removed. - FrigateController is back to a stub returning "no control mapping" for any incoming key. - frigate_sync no longer creates the on/off controller per camera. - FrigateMonitor no longer emits the detect SensorResponse on each poll cycle. - Simulator's FrigateCameraDetectState and ConfigSetView removed; /api/config no longer emits detect.enabled. Future revision can expose the transient detect toggle through Frigate's MQTT frigate/<cam>/detect/set topic, which is what Frigate's own UI button uses for the live update path. * Frigate: replace cursor-hold polling with per-id open-event tracking Frigate's /api/events?after=T filters strictly on start_time > T, so a cursor-hold approach (cursor pinned at an open event's start_time to keep it visible across polls) excludes that event from subsequent scans even after it closes — the END transition was never observed, the SensorHistory correlation pair was never completed, and the operator saw a bare OBJECT_NONE heartbeat instead. The replacement keeps the cursor strictly monotonic over each event's start_time, tracks open events by id in a dedicated set, and refreshes each via GET /api/events/<id> until closed (or force-closed on 404 or the MAX_OPEN_EVENT_AGE_SECS timeout). FrigateClient._get_json now raises Http404 directly so callers can distinguish a vanished event from a transport failure. Removes the AggregatedCameraState model and the per-camera open/closed aggregation that the old pipeline needed. Simulator's get_events_after switched to strict > to match real Frigate so behavior validated against the simulator matches production. Frigate user-facing docs gain CSP / event-clip-codec / authentication caveat sections; dev docs document the new polling model and Frigate's strict-> filter semantics. * Frigate monitor: post-review cleanups and rename for clarity Code-review follow-ups on the polling pipeline rewrite. Mostly refactor and naming polish; no behavior change. Refactor: - Phases return (response_map, touched_cameras) tuples instead of threading a mutable cameras_touched set as an out-parameter. - Extract _force_close() helper consolidating the three identical emit-response / mark-camera-touched / drop-from-tracking call sites in the refresh phase. - Drop trailing dead `continue` at end-of-loop bodies. - Cursor-regression guard kept (and its comment rewritten) as defensive protection against upstream contract violations. Rename for clarity: - OpenFrigateEvent → TrackedFrigateEvent (the wrapper holds HI tracking metadata, not just "openness"). - _open_events → _tracked_events; _refresh_open_events_phase → _refresh_tracked_events_phase; matching test class/method names. - Local variable `tracker` → `tracked_event` (stale leftover from the original OpenEventTracker dataclass name). - In the refresh phase, local `event` → `frigate_event` to avoid shadowing the wrapper's `.event` field at the assignment site. Tests: - _PipelineTestBase switched from TestCase to AsyncTaskFastTestCase so per-method asyncio.run() / `import asyncio` are replaced with self.run_async() — picks up the project's event-loop tracking. - Add coverage for cursor no-regression invariant, duplicate-id defensive guard, malformed phase-2 refresh payload, and camera-list fetch failure in the heartbeat phase. Dev doc updated for the renamed symbols.	2026-05-21 09:11:20 -05:00
Tony C	d2449ef0fc	Issue #319: EntityStateRole for semantic role per EntityState (#322) ... * Introduce EntityStateRole for semantic role per EntityState Adds a first-class semantic-role concept to EntityState so the data model can express what each state means within its enclosing entity. Resolves the multi-of-same-type case (a thermostat's four TEMPERATURE substates; a fan's two DISCRETE substates) and replaces the legacy EntityStateType-priority pre-filter that bucketed icon display by type rather than per-entity primary state. Schema and model - New EntityStateRole(LabeledEnum) in hi/apps/entity/enums.py with two tiers: type-default roles (one per EntityStateType, name- matched) and domain-prefixed refinements (THERMOSTAT_*, FAN_*, LIGHT_*, HVAC_*). - New EntityStateType.default_role() method. - New EntityState.role_str field + entity_state_role property / setter. Save() defaults role_str to the type's default when not set explicitly, so direct EntityState.objects.create() paths get a role automatically. - Migration 0016 backfills existing rows with their type-default role; behavior pre-refinement is unchanged. Factory and HA integration - HiModelHelper.create_sensor / create_controller gain an optional entity_state_role parameter; threaded through to EntityState creation when provided. - HassConverter._SubstateSpec renamed _StateSpec; new role field on the spec. Climate / fan / light substate spec lists assign domain-prefixed roles for the multi-of-same-type cases. Modal listing order - New EntityStateRoleOrdering class + module-level ENTITY_STATUS_VIEW_ORDERING instance in hi/apps/entity/entity_state_role_order.py with per-EntityType override maps. - EntityStatusData.to_template_context() emits state_status_data_list (sorted by role priority); the underlying field stays order- neutral for non-modal consumers. LocationView icon primary-state selection - ENTITY_PRIMARY_STATE_ORDERING (separate instance) selects the primary state for entity-level visual representation. - LocationViewData._get_latest_entity_state_status_data_map switched from timestamp-sort to role-priority sort. - StatusDisplayManager.get_entity_to_entity_state_status_data_list no longer pre-filters by EntityStateType priority; returns all states (including delegations). Shared _all_entity_states_including_delegations helper introduced. - LocationViewData._get_css_class_map now emits exactly one hi-entity-state-* class per entity (the primary state's), so per-state polling updates don't clobber the entity's status attribute via shared CSS classes. EntityStatusView template dispatch - entity_status.html outer wrapper dispatches via include_with_fallback to per-EntityType templates; entity_status_default.html carries the existing flat-list body. Placeholder thermostat / ceiling_fan templates establish the customization pattern. Decision-point markers - LocationViewType and OneClickControlService._find_controller carry comments noting the legacy entity_state_type_priority_list is retained for one-click, pending a parallel ENTITY_CONTROL_STATE_ORDERING follow-up. Docs - integration-guidelines.md gains "EntityStateType vs. EntityStateRole" section explaining the two axes and the integration's role-declaration responsibility. Closes #319 (with manual verification of thermostat / fan / icon status behavior). Audit of EntityStateType vs role conflation (MOVEMENT vs PRESENCE, POWER_LEVEL vs LIGHT_DIMMER) and one-click ordering consolidation are out of scope and tracked as follow-ups. * Migrate OneClickControlService to role-based selection; drop legacy type-priority filter Completes the role-mechanism conversion: one-click control no longer consults LocationViewType.entity_state_type_priority_list and instead walks ENTITY_CONTROL_STATE_ORDERING for the entity's EntityType. The legacy type-priority pre-filter on LocationViewType is removed entirely along with its dead consumers in StatusDisplayManager. OneClickControlService - execute_one_click_control and _find_controller no longer take a location_view_type kwarg. The view-level "only AUTOMATION invokes one-click" gate remains at LocationItemStatusView. - _find_controller walks ENTITY_CONTROL_STATE_ORDERING.order_for strictly: only states whose role is listed are eligible. Unlisted roles (e.g., a thermostat's THERMOSTAT_TARGET_TEMPERATURE) are not one-click targets even when controllable. - _is_toggle_eligible helper extracted to keep the picker honest (only states with toggle_values within ONE_CLICK_CHOICE_LIMIT qualify). ENTITY_CONTROL_STATE_ORDERING - New EntityStateRoleOrdering instance with a curated default list of universally safe-to-toggle roles: ON_OFF, OPEN_CLOSE, OPEN_CLOSE_POSITION, POWER_LEVEL, LIGHT_DIMMER. Binary roles come first so they win when both binary and continuous variants exist. - Per-EntityType overrides for LIGHT (LIGHT_ON_OFF + LIGHT_BRIGHTNESS), CEILING_FAN / EXHAUST_FAN (FAN_SPEED), GARAGE_DOOR_OPENER (OPEN_CLOSE). Switches / outlets / locks rely on the default ON_OFF fallback. Legacy removals - LocationViewType.entity_state_type_priority_list and its __init__ override are gone; the enum collapses to plain DEFAULT / SECURITY / AUTOMATION members carrying just label and description. - StatusDisplayManager.get_entity_state_list_for_status and get_entity_state_type_for_status removed (their last consumer was one-click; the icon path stopped using them earlier in #319). Tests - New test_one_click_control_service.py covers: default fallback for unrecognized EntityTypes; LIGHT override preferring LIGHT_ON_OFF; LIGHT_BRIGHTNESS fallback for fully-modeled color bulbs; speed-only fan via POWER_LEVEL default; cover with OPEN_CLOSE_POSITION via default; sensor-only and thermostat cases correctly NotSupported; toggle-value-limit guard. * Fix collection card sensor polling updates and apply role ordering to collection cards Audit of CollectionView rendering surfaced two issues that #319's work touched indirectly: Sensor values not updating from polling - Collection card sensor renders did not wrap the value template in the per-EntityState CSS class, so the entity_state_status.js dispatcher had no DOM target for `.hi-entity-state-{id}` polling updates. Controllers worked because controller_data.html carries the class itself; sensor-only states (thermometer, humidity, etc.) silently never updated. - entity_state_row.html now wraps the value include in a div with sensor_response.css_class, matching the modal's pattern in sensor_response_status_row.html. State ordering inconsistency between modal and collection cards - EntityStatusData.state_status_data_list promoted from a local in to_template_context() to a public property. The modal continues to consume it via the template context; collection card templates now access it directly on the dataclass instead of reading the unordered entity_state_status_data_list field. Single source of truth for "display order" across both surfaces. - entity_card_list.html and entity_card_grid.html updated to pass entity_status_data.state_status_data_list into the partial. * New urlib3 version with security patch * Rename "wire value" to "EntityState value" in canonical-value docstrings The term "wire value" is appropriate for integration-boundary code where values genuinely travel over a wire (HA REST/WS payloads). Internally, once an EntityState has stored its value, it is the canonical EntityState value, not a wire-format string. Updating docstrings, parameter names, and local variable names in EntityStateValue.to_display_label, EntityState.choices, value_label tag, and associated tests to match. * Promote delegations helper, dedup result, prefetch hot path, add review-cycle test Review-cycle follow-ups for the EntityStateRole work: - Rename ``StatusDisplayManager._all_entity_states_including_delegations`` to ``all_entity_states_including_delegations`` (public). It is called from OneClickControlService and reaching across module boundaries to a private helper is an antipattern. - Deduplicate the helper's return list to guard against entities that delegate one of their own states or that have two delegations resolving to the same EntityState. - Add a one-shot ``prefetch_related_objects`` in ``get_entity_to_entity_state_status_data_list`` covering the entity states, delegations, sensors, and controllers walked downstream. On a LocationView with 20 entities of ~3 states each this collapses from ~160 queries per render to ~6. Drop the now-defeated ``select_related('entity_state')`` inside the helper so prefetched callers actually use the cache; the FK fetch cost for non-prefetched callers is bounded. - Add comments documenting two intentional choices: EntityStateRole's label collisions between bare and domain-refined roles, and the intentional absence of sensor-only roles from ``DEFAULT_CONTROL_STATE_ROLE_ORDER``. - Add ``execute_one_click_control`` end-to-end tests pinning the full pipeline (find_controller → get_current_state_value → determine_control_value → ControllerManager.do_control) for both the "toggle from known state" and "no sensor history" branches.	2026-05-12 18:04:44 -05:00
Tony C	9e3502312e	Issue #316: Threshold-based EventClause for continuous-value alarms (#318) ... * Add EventClause operator dispatch and BATTERY_LEVEL low-battery alarm Implements phases 1-4 of #316: Phase 1 — Schema - New EventClauseOperator enum (EQ, LT, LTE, GT, GTE; default EQ). - New EventClause.value_operator_str field (default 'eq'); property + setter sourced from EventClauseOperator. - Migration 0008 with default 'eq' applies to every existing row; backward-compatible by construction. Phase 2 — Matching - EventManager._clause_matches() helper dispatches on the clause's operator. EQ is the historical string equality; LT / LTE / GT / GTE parse both sides as float() and silently no-op on parse failure so a transient malformed reading never raises into the matcher. - Single-line swap at the call site in _create_event_if_detected(). - Boundary tests (LT vs LTE off-by-one, GT vs GTE, non-numeric defensive case). Phase 3 — Factory plumbing - EventManager.create_simple_alarm_event_definition gains optional value_operator parameter (defaults to EQ); written to the clause. - HiModelHelper.create_battery_level_sensor mirrors the alarm-bearing factory pattern (smoke / moisture / CO / gas): with add_default_alarm=True it also wires the threshold alarm. - HiModelHelper.create_battery_level_event_definition with EventClauseOperator.LT, MAINTENANCE event type, INFO/INFO levels, 24-hour dedupe window, threshold default 20 percent. Phase 4 — HA wiring - HA converter's BATTERY_LEVEL branch now uses create_battery_level_sensor with add_default_alarm=add_alarm_events, so every imported `sensor + battery` device gains a low-battery alarm when the integration's add_alarm_events flag is on. Deferred to follow-up: Phase 5 (UI/form work for user-editable threshold clauses); Phase 6 (docs). Alert-grouping behavior (multiple alarms with the same event-type label sharing one Alert bucket) is by design for anti-fatigue; the deceptive Alert title when grouped is a separate UI concern. * Expose EventClause operator in the edit UI and document the pattern Phases 5-6 of #316. UI - EventClauseForm gains a required value_operator_str ChoiceField drawn from EventClauseOperator.choices (defaults to EQ). Form clean() rejects a non-numeric value when operator is non-EQ so users get immediate feedback instead of a clause that silently never fires. - Template adds an input-group for the operator dropdown between entity_state and value, matching the surrounding row pattern. - JS gains Hi.setEventClauseValueOperatorWidget: on operator change to LT / LTE / GT / GTE, swaps the value element to <input type="number" step="any"> preserving the entered value. EQ leaves the existing widget alone so the entity_state-driven choice swap stays authoritative for discrete-state clauses. - Admin's TabularInline picks up the new field automatically. - Existing event-view tests updated to include event-clause-0-value_operator_str='eq' in posted form data. Docs - Augment the existing "Default alarm wiring" section in integration-guidelines.md with one paragraph on the EventClauseOperator pattern for continuous-value thresholds and a pointer to create_battery_level_event_definition. Light touch on purpose; the existing section already covers the discrete pattern and the event-definition modal is slated for a separate UI overhaul. Closes #316. * Address review feedback: form validation test, debug log, template typo - Add focused test for EventClauseForm.clean(): non-numeric value with a numeric operator is rejected; numeric value is accepted. Closes the silent-failure path where a user could otherwise save a clause that never fires (matcher silently no-ops on non-numeric values under numeric operators). - Log threshold clause skips at DEBUG inside _clause_matches's except branch so a misconfigured clause is diagnosable. Kept at DEBUG so transient 'unknown' / 'unavailable' wire values don't spam at higher levels. - Fix pre-existing label for-attribute typo in event_clause_form.html (event_clause_form.entity.id_for_label → entity_state.id_for_label).	2026-05-11 22:45:43 -05:00
Tony C	6a2533f635	Issue #294: HA cleanup and alarm coverage expansion (#317) ... * Cleanup #1: non-Insteon switch / outlet / motion sim entities Switches, outlets, and motion sensors were the only device categories lacking non-Insteon variants in the HA simulator — adding them closes the gap left after #300 / #301. New sim entities: - HassMotionSensor (binary_sensor.x, device_class=motion) - HassSwitch (switch.x, no device_class) - HassOutlet (switch.x, device_class=outlet) Each is a single ON_OFF SimState mirroring the binary-sensor shape from #300; HA's switch / motion paths already handle them in the converter without changes. Zoo profile updates: - Six Insteon entries renamed with "Insteon" in the name so the new non-Insteon entries can take the clean labels ("Zoo Switch", "Zoo Motion", "Zoo Outlet"). - Standard hass profile entries unchanged. End-to-end verified: HI imports the new entities with EntityType.ON_OFF_SWITCH / ELECTRICAL_OUTLET / MOTION_SENSOR. * Cleanup #2: centralize HA wire-format strings in HassApi Adds ~36 HassApi constants for the HA wire vocabulary that was still inline in hass_converter.py and hass_service_composer.py: - Climate attributes: current_temperature, current_humidity, target_temp_low, target_temp_high, temperature (setpoint), hvac_mode/hvac_modes/hvac_action, fan_mode/fan_modes, temperature_unit, and the heat_cool dual-setpoint mode value. - Light attributes: brightness, brightness_pct, color_mode, color_temp_kelvin, hs_color, supported_color_modes. - The 10 color_mode wire values (unknown, onoff, brightness, color_temp, hs, rgb, rgbw, rgbww, white, xy). - Fan attributes: percentage, oscillating, direction, preset_mode/preset_modes, and the forward/reverse direction wire values. - Cover attributes: current_position, position (service param). - Media-player: volume_level (service param). A typo in any of these now surfaces at name resolution rather than silently mismatching at runtime; future HA API changes can be addressed in one place. Kept as literals: HI-internal format hints in parameter-shape dicts, HI-internal substate suffix names that diverge from HA attributes, and HI's flexible boolean / action input parsing. * Cleanup #1 follow-up: combo motion sensor + battery/illuminance mapping A real-world Z-Wave / Zigbee motion sensor exposes three entities sharing a device: motion (binary_sensor), battery (sensor.x with device_class=battery, numeric %), and illuminance (sensor.x with device_class=illuminance, lux). HA's converter previously mapped only the binary sensor; the numeric battery and illuminance sensors fell through to BLOB. Adds: - EntityStateType.BATTERY_LEVEL with label "Battery". - HassApi constants for illuminance device_class and id suffix. - Converter mappings: sensor.x + battery → BATTERY_LEVEL, sensor.x + illuminance → LIGHT_LEVEL (which already existed in HI but had no integration mapping). - Units captured verbatim from the wire ('%' for battery, 'lx' for illuminance, with sensible defaults). - HassComboMotionSensor sim entity with three states sharing a short_name so the converter collapses them into one HI Entity. - Zoo Smart Motion seed entry. DisplayValue.__str__ inserts a space between magnitude and unit symbol when the unit is alphabetic (120 lx, 85 kg) per NIST/SI convention; symbol-style units (°F, %) attach directly. * Cleanup #4 Stage A: HA-docs validation findings (Option A wins) Audited HI's HA binary_sensor and sensor coverage against the official device-class enumerations from HA's developer docs. Found 18 binary_sensor + 47 sensor device classes unmapped; implemented the highest-value subset (Option A from the stage discussion). Remaining gaps documented as meta-issue #315. Binary sensor additions (mapping table + value translation): - occupancy / presence → MOTION-like ACTIVE/IDLE (occupancy → EntityType.MOTION_SENSOR, presence → EntityType.PRESENCE_SENSOR). - opening → OPEN_CLOSE (added to OPEN_CLOSE_DEVICE_CLASS_SET). - moisture → ON_OFF (interim — water-leak alarms warrant the full SMOKE-style treatment in a separate change). Sensor additions (mapping table + EntityState creation with units captured from the wire): - power → ELECTRIC_USAGE - pressure → AIR_PRESSURE - wind_speed → WIND_SPEED These all reuse pre-existing HI EntityStateTypes — one-liner mapping additions plus a generic numeric-sensor creation branch in hass_converter that captures unit_of_measurement. New HassApi.MOTION_LIKE_DEVICE_CLASS_SET groups motion / occupancy / presence so _binary_sensor_value can branch off a single set rather than three parallel comparisons. New EntityStateType.PRESENCE creation branch — the type existed and StatusDisplayData handled it, but no converter path created entities with it before now. unknown / unavailable wire-value handling: HA emits these special states when an entity is offline or hasn't reported. hass_state_to_sensor_value_map now short-circuits on them so sensor_history doesn't accrue placeholder records that would later surface as "Unavailable" text in the polling refresh. HassStateValue gains UNKNOWN, UNAVAILABLE, and a NO_VALUE_STATES frozenset. End-to-end verified: 7 new device classes import with correct EntityType, state-type, and units; unknown/unavailable produce empty value maps. Meta-issue #315 inventories all remaining unmapped device classes with prioritization signals and per-pattern effort estimates. * Cleanup #4 Stage B: climate preset, color-temp Kelvin bounds Climate preset_mode substate. HA documents preset_modes (Eco / Away / Home / Sleep / Boost / Comfort / Activity) as a separate controllable axis from hvac_mode; HI's climate path previously ignored it. Now exposed as a DISCRETE controllable substate when the thermostat declares preset_modes, with set_preset_mode service dispatch on the HA side and a matching dispatcher branch on the simulator side. Light color-temperature device bounds. The COLOR_TEMPERATURE slider was hardcoded to 2000-6500K; real bulbs have narrower device-specific ranges (e.g., 2700-5000K for many warm-white LEDs). hass_converter now reads min_color_temp_kelvin / max_color_temp_kelvin from the live state with the broad fallback when not declared. HI substate suffix rename: preset → preset_mode. Eliminates the one substate suffix in the codebase that diverged from its HA attribute name (HA's attribute is preset_mode, fan's HI suffix was 'preset'). The five fan-test occurrences and the five production occurrences updated; HassApi.PRESET_MODE_ATTR now serves both as the HA attribute name and the HI substate suffix identifier. Existing imported HA fan entities with preset substates need re-import — the integration_key suffix changed from ~preset to ~preset_mode. Simulator additions: - HassThermostatFields.preset_modes default ['eco', 'away', 'home', 'sleep']. - HassThermostatPresetState mirrors HassThermostatFanModeState with sim_state_id='preset_mode' so the dispatcher tuple matches. - Thermostat SimEntityDefinition registers the new state. - service_dispatchers._thermostat adds a set_preset_mode branch so HI's outbound preset change reaches the simulator. Deferred items from Stage B added as a comment on meta-issue #315: target_humidity, swing modes, light effect/effect_list, brightness=1 rounding edge case, hs_color range verification. * Complete #294 Stage C HA validation: leak sensor support and unit display fixes - Add full leak-detector support (LEAK_SENSOR EntityType, MOISTURE EntityStateValues, SVG icon, alarm wiring, status decay) - Fix wind speed not converting under imperial display: compound-unit keys in IMPERIAL_TO_METRIC_UNITS were Quantities, not Units, and silently failed dict lookup; route through ureg.parse_units - Add console DISPLAY_UNIT_OVERRIDES so ELECTRIC_USAGE always displays in watts regardless of stored unit (avoids surprise W -> hp) - Make color temperature slider honor EntityState value_range_dict (template was hardcoding 2000-6500 instead of reading device bounds) - Expand HA simulator hass-zoo profile with non-Insteon switch/outlet/ motion variants, combo motion sensor, presence/opening/power-meter/ weather-station/occupancy-light/water-leak entities - Add display_unit properties on sim numeric states so simulator sliders label their units - Rename HI 'preset' substate suffix to 'preset_mode' for HA consistency - Centralize remaining HA wire-format strings in HassApi * Promote default-alarm wiring to HiModelHelper and document integration conventions Phase 1 of #294 item #5 promotion analysis: extract HA-side patterns that are home-automation-general up to the framework layer. - Add 'add_default_alarm: bool = False' to HiModelHelper's create_connectivity_sensor, create_open_close_sensor, create_movement_sensor, create_smoke_sensor, create_moisture_sensor. When True, the factory also creates the canonical default alarm event definition using the sensor's integration_key. - Collapse five HA converter blocks from sensor-call + conditional-alarm-call to a single sensor call. - Add "Code Conventions" section to integration-guidelines.md covering file layout / <prefix>_ naming, wire-constant centralization (HassApi as exemplar), and the default-alarm flag with guidance on the shared-key vs. separate-key cases. ZoneMinder's movement alarm is left untouched: it uses a separate integration_key (MOVEMENT_EVENT_PREFIX) from the sensor, so the explicit two-call pattern remains correct there. * Add PRESENCE, carbon monoxide, and combustible-gas alarm support Fills three alarm-coverage gaps surfaced by the audit in #294 item #5: - PRESENCE alarm: parallel to MOVEMENT (SECURITY, CRITICAL/INFO). Adds create_presence_sensor + create_presence_event_definition to HiModelHelper; HA converter's PRESENCE branch now uses the factory with add_default_alarm. - Carbon monoxide (binary_sensor + device_class=carbon_monoxide): full SMOKE-pattern replication. New EntityStateValue.CO_DETECTED/ CO_CLEAR, EntityStateType.CO, HiModelHelper factories, status decay, SvgStatusStyle entries, CSS rules, HA converter mapping + value translation + creation branch + entity-type inference. CO is life-safety: both security levels map to CRITICAL. - Combustible-gas (binary_sensor + device_class=gas): same shape as CO. New EntityType.GAS_DETECTOR with new SVG icon. Life-safety semantics. Simulator side: SimEntityType.CARBON_MONOXIDE_DETECTOR and GAS_DETECTOR (alphabetical position), sim entity classes, zoo profile entries (Zoo CO Detector, Zoo Gas Detector). Deferred to a separate effort (will track in #315): BATTERY_LEVEL threshold alarm — requires extending EventDefinition to support "value below threshold" triggers, which is a substantive change to the event subsystem better done in isolation.	2026-05-11 21:04:30 -05:00
Tony C	900f85394c	HA thermostat support + temperature unit handling (#299 / #308) (#309) ... * Issue #299 Phases 1+2: HA climate substate decomposition and thermostat simulator entity Climate is inherently multi-axis, so apply the substate-decomposition pattern from #295/#298. A climate entity that declares hvac_modes always decomposes into substates: current_temperature (sensor), hvac_mode (controllable from declared modes), hvac_action (sensor with HA's standard action choices), plus a setpoint substate set chosen from the supported modes — single target_temperature for any single-mode operation, low+high pair for heat_cool. Climate entities lacking hvac_modes fall through to the existing single-state TEMPERATURE mapping for backward compatibility. Per-domain composer methods _climate_substate_specs and _climate_substate_value join the existing _light/_fan ones. _climate_to_sensor_value_map handles the multi-axis decomposition; the dispatcher routes climate to it. A small _numeric_attr_as_str helper centralizes the recurring float-coercion-with-graceful-fallback pattern used across temperature attributes. Simulator side adds HassThermostatFields with operator-configurable hvac_modes list and temperature_unit, paired with six SimState classes (current_temperature, target_temperature, target_temp_low, target_temp_high, hvac_mode, hvac_action). The new HassApiComposer._thermostat collapses them into one HA climate.x entity, picking the setpoint shape (single temperature vs target_temp_low+high) based on the active hvac_mode — mirroring real HA's per-mode shape change. The HassServiceDispatcher._thermostat handler routes set_temperature (single or low/high) and set_hvac_mode to the matching SimStates. The hass-zoo profile gains two thermostats: Zoo Thermostat (full-feature heat/cool/heat_cool/off, °F) and Zoo Heater (heat-only, °C) — exercising both single-mode and multi-mode setpoint shapes plus both temperature units. * Issue #299 Phases 3+4: outbound dispatch for climate substates plus fan_mode and current_humidity axes Phase 3 — outbound dispatch _substate_service_call extended for climate suffixes: target_temperature routes to set_temperature with single-value payload; the target_temp_low / target_temp_high pair routes to set_temperature with the dual-value payload, using a companion-substate cache lookup that mirrors the hue/saturation pattern (so adjusting one bound carries the other along, with a safe ordering fallback when the partner value isn't yet cached); hvac_mode routes to set_hvac_mode. New HassServiceComposer methods for_temperature_range and for_hvac_mode encapsulate the wire shapes. New controller_temperature.html slider widget reads min/max from the EntityState's value_range_dict (set per-temperature_unit at Phase 1 substate creation time) so the slider bounds adapt to °F vs °C; unit-suffix display is a placeholder pending #308's full unit-handling work. Phase 4 — stretch axes Climate substate spec composer extended for fan_mode (DISCRETE controllable, choices from the per-thermostat fan_modes list when reported) and current_humidity (HUMIDITY sensor-only when reported). _climate_substate_value reads fan_mode and current_humidity attributes. New HassServiceComposer.for_fan_mode and HassApi.SET_FAN_MODE_SERVICE constant. The simulator's HassThermostatFields gains a fan_modes field (default fan-aware; Zoo Heater overrides to []) and two new SimStates: HassThermostatFanModeState (DISCRETE per-instance choices) and HassThermostatCurrentHumidityState (CONTINUOUS sensor-driven). The thermostat composer emits fan_mode / fan_modes / current_humidity attributes when the underlying SimStates supply them; the service dispatcher routes set_fan_mode to the matching SimState. * Issue #299 Phase 5: tests for HA climate converter and new HassServiceComposer methods - New test_hass_converter_climate.py: substate spec composition (no hvac_modes -> no decomposition; full vs heat-only vs heat_cool-only mode sets; per-unit value_range; choices from hvac_modes / fan_modes; fan_mode omitted when fan_modes absent); inbound translation for full-feature and single-mode thermostats including missing-attribute graceful drop and non-numeric temperature graceful empty; end-to-end import for both flavors verifying right EntityState count and per-substate types; outbound dispatch for target_temperature, hvac_mode, fan_mode, and the target_temp_low/high pair (companion-substate cache lookup with cached and uncached partner). - TestClimateAxisComposers in test_hass_service_composer: for_temperature_range / for_hvac_mode / for_fan_mode happy paths plus the ValueError-raising validation branches (low > high; empty mode strings). * Fix UPDATE button hidden on config settings page The action bar in edit_content_body.html gated UPDATE-button visibility (and its dirty/status message stack) on attr_item_context.can_add_custom_attributes. That flag did double duty for "page is editable" and "new attributes can be added" — and attr_item_context isn't part of the top-level template context in multi-edit (config settings) mode, so the gate silently evaluated false and the UPDATE button disappeared, blocking edits to user/system settings. Split the conflated concept and move both flags up to the page-level context so they work for single- and multi-edit surfaces alike: - allow_edits (new): drives UPDATE button + dirty/status visibility - can_add_custom_attributes: drives Add File / Add Info visibility only Entity overrides allow_edits to track its existing externally-managed flag (HomeBox-imported entities remain fully read-only). Subsystem contexts override can_add_custom_attributes to False (config exposes only system-defined attributes) while inheriting allow_edits=True. Also tightened the per-attribute Restore-from-history gate to require both page-level allow_edits AND the attribute's own is_editable, so predefined non-editable attributes correctly hide Restore even on an otherwise editable page. * Issue #308: HA temperature unit handling end-to-end Per-EntityState unit storage with conversion at all boundaries (HA wire ↔ HI canonical, HI canonical ↔ user display unit, UI input ↔ stored unit). HI's canonical temperature unit is declared once in the climate substate spec; downstream code consults EntityState.units / payload metadata so the canonical choice is data-driven, not code-distributed. Cross-cutting infrastructure: - IntegrationMetadataCache: process-wide, lazy-warmed cache of EntityState metadata (sync/async parallel APIs) so polling-loop unit lookups don't multiply DB queries inside the HA monitor's tight loop. Bulk warmup on first read; lazy fill for entities added later. Keyed by IntegrationKey; entry shape is a small dict so future metadata can be added without API churn. - IntegrationConverterHelper / ConsoleConverterHelper: symmetric to_/from_entity_state_value pairs for the integration and UI boundaries respectively. ConsoleConverterHelper returns a DisplayValue dataclass so magnitude-only consumers (slider numeric value) and combined-text consumers (status text, modal) share one helper output; templates use a new to_display filter that replaces the verbose as_quantity | format_magnitude chain. - ControlViewMixin.to_entity_state_value delegates to the helper. - StatusDisplayData.latest_display_value routes async UI poll refreshes through the same helper, fixing anomalies where canonical magnitudes leaked through unconverted (post-poll display reverting to raw °C when initial render correctly showed °F). HA simulator additions for end-to-end verification: - SimulatorRuntimeSettings: process-wide in-memory temperature unit override (Default / °F / °C) selectable from a header dropdown with antinode swap-in-place to preserve the active integration tab. Composer and dispatcher convert at the wire boundary; SimStates store in profile unit so the override is purely a wire-format toggle. - UnitTranslationHelper (HA-specific): SimTemperatureUnit → HA wire format mapping plus F↔C conversion at composer/dispatcher boundaries. - Thermostat seeds: friendly_name set on the emitted attributes so HI shows pretty entity names. Temperature SimState defaults and slider bounds are now unit-aware so °C-native profiles start at sensible Celsius values rather than 70°C-as-room-temp. Per-control unit suffix on the simulator's continuous slider; slider value displays use floatformat to avoid Pint float reprs leaking many decimals into the UI. UI: - continuous_slider_with_units.html (new): unit-aware slider routing through the helper. Slider step aligned with 1-decimal display precision so HTML range step coercion doesn't mismatch the template-rendered value on async polling refreshes. - sensor_response_value_temperature.html (new): per-EntityStateType render for TEMPERATURE. - controller_temperature.html updated to use the unit-aware slider. Phase 1's hi_temperature_unit field is removed from integration_payload — the cache provides EntityState.units to outbound dispatch directly, eliminating the redundancy. Trace-output column widths adjusted to better fit the longer substate-suffixed integration_name strings. Tests: 2788 passing. * Skip controller poll updates for actively-dragged sliders Polling-driven controller value updates could replace a slider's value mid-drag, yanking the thumb out from under the operator's fingers. Track active drag state via a WeakSet (mousedown / touchstart add; mouseup / touchend / change / blur remove); the polling-update path skips range inputs in that set. The slider's ``change`` event clears the flag on release and antinode's onchange-async handler submits the new value, so values still reconcile correctly — only the in-flight redraw is suppressed. * Tests for unit translation infrastructure (#299 / #308) High-value coverage for the unit-translation classes added with the thermostat / temperature unit work: - IntegrationMetadataCache: warmup loads units from Sensors, dedupes Sensor/Controller pairs sharing a key, lazy fill for entities created post-warmup, miss caching to avoid re-querying the DB, divergent Sensor/Controller pair resolution (Sensor wins), async variant returns same as sync, warm-cache async short-circuits the sync_to_async hop. - IntegrationConverterHelper.to_/from_entity_state_value: F→C and C→F conversion via cache lookup, passthrough when units match or are absent, round-trip preservation, sync/async variant equivalence. - ConsoleConverterHelper / DisplayValue: combined-text formatting via __str__, inbound conversion at the HTML/JS boundary, outbound rendering as DisplayValue, passthrough on no-units / non-numeric / None inputs, full round-trip via the to_/from pair. - UnitTranslationHelper (HA simulator boundary): emitted unit selection respects the runtime override, F↔C conversion at notable points (freezing, boiling, room temp), defensive passthrough on missing units / non-numeric / None, round-trip precision. Tests follow the codebase convention of disabling logging at module load (assertions are behavior-based, not log-based). Async paths use AsyncTaskTestCase so the event-loop lifecycle is managed and DB-locking deadlocks are avoided. * Document unit-conversion conventions and helper classes Frontend guidelines gain a "Unit-Bearing Values: Server ↔ UI Translation" section covering the canonical-storage convention, the ConsoleConverterHelper to_/from pair, the DisplayValue return shape, the to_display template filter, and the contract that initial render and polling refresh share the same helper. Integration guidelines gain a "Unit conversion at the integration boundary" section covering IntegrationMetadataCache and IntegrationConverterHelper.to_/from_entity_state_value, with a pointer to the HA converter as a worked example. Cross-links the parallel UI-boundary section in the frontend guidelines. * Review feedback: hoist canonical, Singleton cache, JS drag-protect Address review findings on the unit-translation infrastructure before PR: - Hoist ``CANONICAL_TEMPERATURE_UNIT`` out of ``HassConverter`` to ``hi/units.py`` so other integrations adopting the canonical-at- boundary pattern import the choice cross-cuttingly rather than duplicating or reaching into HA-specific code. The climate- specific slider bounds stay on ``HassConverter`` but are renamed ``_SETPOINT_MIN_CANONICAL`` / ``_SETPOINT_MAX_CANONICAL`` to make their narrower scope clear. - Convert ``IntegrationMetadataCache`` to use the project's ``Singleton`` base class for consistency with ``SimulatorRuntimeSettings`` and the rest of the codebase. Cache state moves to instance attrs; callers updated to ``IntegrationMetadataCache().method(...)`` form. - JS slider drag-protect: add Pointer Events (``pointerdown`` / ``pointerup`` / ``pointercancel``) for modern stylus / touch interactions, and move release-side handlers from ``body`` to ``document`` so off-element / off-window releases still clear the active-drag flag. - Rename the ``to_display`` template filter to ``as_display_value`` for clearer return-type signaling (filter outputs a ``DisplayValue``).	2026-05-10 12:49:12 -05:00
Tony C	a74b6a8b41	Issue #295: HA smart bulb support (brightness + color) (#303) ... * Add HASS smart bulb sim devices and continuous-slider infrastructure Phase 1 of #295. Three coupled additions to the HASS simulator that together let HI's existing dimmer infrastructure exercise on a non- Insteon shape and expose the gaps in HI's color handling. Continuous-state UI infrastructure: new ``sim_control_continuous.html`` widget renders an HTML range slider keyed off a SimState's ``min_value``/``max_value``/``value``. Reusable for upcoming thermostat/fan/cover work. Light variants: * ``HassSmartBulbState`` — brightness-only smart bulb. Single CONTINUOUS SimState in HA's 0-255 range; emits one HA ``light.x`` entity with ``brightness`` and ``color_mode='brightness'``. * ``HassColorSmartBulbBrightnessState`` plus three sibling SimStates for hue/saturation/color-temperature. Operator gets a slider per state in the simulator UI (HI-centric model); the four states collapse into ONE HA ``light.x`` entity at emit time via ``api_composers``. * ``HassInsteonDimmerLightLightState`` switched from ON_OFF to CONTINUOUS so it actually emits ``brightness``. Without this HI's ``_has_brightness_capability`` check failed and the Insteon dimmer imported as ON_OFF rather than LIGHT_DIMMER — making the existing ``controller_light_dimmer.html`` slider unreachable for HASS-imported dimmers. API composition: new ``api_composers.py`` introduces ``HassApiComposer`` with a per-device-type registry keyed off SimEntityFields class. Default composer keeps the existing one- state-per-HA-entity behavior so the motion detector / switches / sensors are unaffected. Color smart bulb's composer collapses its four SimStates into one HA entity, combining hue+saturation into the standard ``hs_color: [h, s]`` two-element list and emitting ``color_mode='hs'``. ``AllStatesView`` now iterates per-entity and dispatches through the composer. Seed: ``hass-zoo`` profile adds one Smart Bulb and one Color Smart Bulb so the two new shapes are immediately importable. * Add controller-value polling path so widgets reflect external state The existing polling mechanism set the bucketed status string as a DOM attribute and let CSS drive icon styling — works for the LocationView's visual representation, breaks for interactive widgets in the entity-status modal because a slider's thumb position, a checkbox's checked, and a select's selected option aren't driven by attribute updates. Symptom most visibly: the dimmer slider in the modal didn't follow external (sim-side) brightness changes, and the on/off toggle relied on a status-attribute-to-checked side-effect inlined in the polling JS. Introduce a parallel ``cssControllerValueMap`` in the polling response, keyed off the same EntityState class names as the existing status map but carrying widget-shaped values rather than bucketed status strings. ``StatusDisplayData.controller_data_value`` produces the value (default: raw latest sensor value; per-state- type reshape is the extension point). On the client side a new ``Hi.controllers.applyValueMap`` dispatches to the appropriate DOM property by element shape — slider ``.value``, checkbox ``.checked``, select ``.value`` — only updating when the value differs to avoid mid-drag interference. The legacy checkbox / select special-cases inside ``handleCssClassUpdates`` are removed now that the controller-value path covers those widgets explicitly. Three peripheral fixes the foundation revealed: * ``HassConverter._has_brightness_capability`` now recognizes a dimmer-capable light from its ``supported_color_modes`` declaration in addition to a currently-present ``brightness`` attribute. HA omits ``brightness`` when a light is off, which used to flip a known dimmer to the on/off conversion path each off→on cycle and surface ``"off"`` as the slider value (browser clamping to mid-range and the percent label displaying ``"off%"``). * Simulator's ``ServiceCallView`` previously hard-coded a ``(domain, service) → 'on'/'off'`` literal mapping that polluted CONTINUOUS state values (a brightness state would end up holding the literal string ``'on'``). Replaced with a per-device-type ``HassServiceDispatcher`` that translates HA payloads into the right SimState updates — turn_on with brightness/brightness_pct routes to the brightness state as a numeric, turn_on with hs_color routes to the hue and saturation states, etc. The default handler covers single- state devices; color smart bulb registers its own. * Brightness slider's percent label (``.brightness-value``) didn't track the slider value across drag, button click, or polling-driven updates. ``controllers.js`` now syncs the label from all three paths. The new module ``controllers.js`` is added to the ``js_before_content`` pipeline so it loads on every page that might render the dimmer (LocationView and the entity-status modal both included). * Issue #295: Color attribute support and HassController/HassConverter refactor - One HA state -> N HI EntityStates (brightness + hue/saturation/color_temp) on inbound; many HI values -> one HA service call on outbound - Move HI->HA conversion logic from HassController into HassConverter; controller becomes a thin orchestrator over hi_value_to_hass_service_call - Add IntegrationConverterMixin for cross-integration sibling-state lookup; SensorResponseManager gains a batch-shaped get_latest_sensor_response_map - Add hue, saturation, and color temperature controller templates with reference gradient tracks (rainbow / gray-to-saturated / warm-to-cool) - Generalize continuous-slider display sync via shared DIVID/Hi data-attribute constants - Align vocabulary: 'HA state' (bundle) vs 'HA substate' (each HI EntityState); hass_entity_id at the HA boundary; hi_control_value at the integration boundary * Issue #295: Code organization cleanup — extract HassServiceComposer, generalize slider widget - Split HassConverter outbound dispatch: pure HA-side composition moves to a new HassServiceComposer; HassConverter retains the HI<->HA bridge with explicit to_ha_* boundary helpers (to_ha_numeric_parameter_value, to_ha_on_off_intent) - Generalize substate-related names from color_sub_state -> substate throughout, so the framework reads as domain-agnostic for future one-to-many integrations beyond color lights - Unify continuous-slider templates behind one parameterized partial (continuous_slider.html); dimmer/hue/saturation/color_temperature collapse to single-line includes; brightness preset buttons generalized to opt-in min/max preset mechanism - Hoist remaining JS/template-coordinated strings into DIVID/Hi constants (CONTROLLER_SLIDER_*, CONTROLLER_PRESET_BTN_CLASS, DATA_VALUE_ATTR); controllers.js has no brightness-specific code - Re-sync loop now refreshes substate controller payloads and creates any newly-implied substate controllers on existing parents, symmetric with the existing shrink behavior - Replace top docstring of HassConverter with concise terminology (domain, device_class, entity, state, substate, device) and the three jobs the class performs - Vocabulary alignment: control_value -> hi_control_value at the IntegrationController boundary * Issue #295: Tests and docs for substate composition - Add test_hass_service_composer covering all 10 composer entry points (for_brightness, for_temperature, for_volume, for_position, for_numeric_parameter, for_numeric_best_effort, for_on_off_best_effort, for_payload_intent, for_color_temp, for_hs_color) - Add test_hass_converter_substate covering inbound substate decomposition for color lights, outbound substate dispatch (color_temp, hue/saturation with cached and missing partners), and the to_ha_* boundary helpers - Document one-to-many state composition in integration-guidelines.md (framework capability via IntegrationConverterMixin and the integration_key suffix convention) - Document HA state vs HA substate terminology and the HassConverter vs HassServiceComposer split in the HA integration's developer doc * Issue #295: Replace magic strings in HASS service-call composition - Replace raw domain/service literals throughout HassServiceComposer with HassApi constants (LIGHT_DOMAIN / TURN_ON_SERVICE etc.) so service-name typos surface as import errors rather than silent routing bugs - Introduce ControlIntent class for the four canonical operator intents (ON / OFF / OPEN / CLOSE) flowing across the HI->HA bridge; HassConverter.to_ha_on_off_intent now returns the named constants and HassServiceComposer dispatches on them - Align to_ha_on_off_intent with hass_entity_id_to_state_value_str by using EntityStateValue.ON/OFF/OPEN/CLOSED for the recognized HI input forms (alongside the existing 'true'/'false'/'1'/'0' shorthand) - Skip redundant DB write in re-sync substate-payload refresh when the payload is unchanged - Drop the _is_numeric_value predicate; _best_effort_service_call now does try/except around to_ha_numeric_parameter_value, so the parse work isn't duplicated. Rename _is_numeric_control to _payload_supports_numeric_control to reflect that it only checks payload flags now - Extract _substate_integration_key_for_suffix so the partner-key construction in _substate_service_call goes through the same IntegrationKey builder as the rest of the substate path * Issue #295: Review-driven cleanup — accuracy, efficiency, idempotency tests - Push payload-equality guard into update_integration_payload so it skips the DB save when nothing changed; both parent and substate re-sync paths benefit and the substate site no longer needs its own guard - Rename IntegrationConverterMixin to IntegrationConverterHelper (namespace-style classmethod helper, not a base for inheritance); internally caches a SensorResponseMixin-using instance so manager access stays on the project's coordination path. Brief docstring notes the proper Singleton-converter fix is deferred - Update _extract_substate_value docstring to accurately describe behavior: HI relays HA-reported color attribute values; we don't filter by color_mode (a value HA reports IS the bulb's last-known state regardless of which mode is currently authoritative) - Switch continuous_slider.html from default to default_if_none so a numeric 0 sensor reading isn't silently rewritten to the default - Add idempotency tests for substate controller creation: initial import produces three controllers; re-sync doesn't duplicate; re-sync creates newly-implied substate controllers when supported_color_modes grows on an already-imported parent - Add direct tests for SensorResponseManager.get_latest_sensor_response_map using fakeredis: empty-list short-circuit, uncached → None, cached returns latest, mixed batch preserves order - Update inlined mock impl in test_models to match the new update_integration_payload contract * Issue #295: Add COLOR_MODE substate, fix awaiting-reading display - New EntityStateType.COLOR_MODE (sensor-only, discrete) tracks HA's color_mode attribute on multi-mode lights. Adds 10 COLOR_MODE_* EntityStateValues covering HA's mode set, with COLOR_MODE_UNKNOWN catching null and 'unknown' from HA - Consolidate _HASS_SUBSTATE_SUFFIXES and _CONTROLLER_STATE_VALUE_RANGES into a single _SUBSTATE_SPECS registry keyed by EntityStateType, carrying suffix, is_controllable, and optional value_range - _create_substate_controllers becomes _create_substate_models; branches on is_controllable to create a Sensor or Controller per spec. Re-sync passes both existing-controllers and existing-sensors maps for symmetric idempotency - COLOR_MODE substate is created only for lights with more than one supported_color_mode — a single-mode bulb has a constant value and the substate adds no information - _extract_substate_value translates HA's color_mode strings to the corresponding HI EntityStateValue; missing-key still skips (last-known retained), null/'unknown' both map to UNKNOWN - Display layer: add has_sensor / has_controller flags to EntityStateStatusData (in-memory only, no polling-JSON impact); templates differentiate "awaiting sensor reading" (sensor defined, no cache yet) from "no sensors or controllers defined" (model-level setup error). Both messages now include the state name for context. Fixes a misleading alert that surfaced for any sensor-only EntityState pre-population - New tests cover COLOR_MODE value translation across the HA mode set, missing-key skip, sensor-only creation, and re-sync idempotency for the COLOR_MODE Sensor * Issue #295: Simulator color_mode SimState for color smart bulbs - Add HassColorSmartBulbColorModeState — DISCRETE SimState with HA's color-mode value set as choices. Operator-controllable via the simulator UI dropdown so testers can reach edge values (unknown, rgbww, etc.) directly without having to drive every mode through HI's controllers - Composer reads the SimState's value into the entity-level color_mode attribute, replacing the previous hard-coded 'hs' - Service dispatcher emits a color_mode update alongside the axis updates: hs_color writes set color_mode='hs', and color_temp_kelvin writes set color_mode='color_temp'. Mirrors HA's "active mode follows the most-recently-written attribute" behavior on the standard inbound flow * Issue #295: Promote brightness to peer substate for color bulbs For multi-substate lights (anything with chromatic or color_temp modes), brightness becomes a peer substate at the ~brightness integration_key alongside hue/saturation/color_temp/color_mode, rather than the bare-key "primary" with the others as siblings. Removes the asymmetric "one is special" model and makes UI labels ("{name} Brightness", "{name} Hue", etc.) symmetric across all controls. Single-state lights (brightness-only, on/off-only) keep the bare-key model unchanged. - LIGHT_DIMMER joins _SUBSTATE_SPECS with suffix='brightness' - _substate_types_for_hass_state includes LIGHT_DIMMER when any color substate is present - _create_hass_sensors_and_controllers skips parent EntityState creation for multi-substate lights; all states routed through _create_substate_models with suffixed keys - _dimmer_light_to_sensor_value_map skips bare-key brightness emit when LIGHT_DIMMER is in the substate set - _extract_substate_value handles LIGHT_DIMMER (delegates to _dimmer_brightness_value) - _substate_service_call handles 'brightness' substate via HassServiceComposer.for_numeric_best_effort with the parent entity_id as the HA call target - Re-sync existence check broadened: matches existing substates at suffixed keys, not just the bare key, so multi-substate lights don't fall through to "missing — adding" on every sync - Tests updated for the new key shape; idempotency tests now expect 4 substate controllers (brightness + hue + saturation + color_temp)	2026-05-08 22:04:14 -05:00
Tony C	8314eced6e	Issue #280: Per-integration documentation structure and content (#291) ... Establishes a two-track per-integration doc structure (user-facing under docs/integrations/, developer-facing under docs/dev/integrations/) with templates that enforce a consistent set of sections for each integration. Populates the new structure for the three existing user-configured integrations (Home Assistant, ZoneMinder, HomeBox) and updates the integration-addition guideline to require both docs for any new integration. User-facing changes: - New per-integration pages with overview, prerequisites, credential acquisition, configuration values, setup walkthrough, troubleshooting, and known limitations. ZoneMinder's existing CORS and HTTPS/SSL troubleshooting content migrated out of the shared Integrations.md into its own page. - Restructured docs/Integrations.md as a short landing page that centralizes the conditional UI flow for enabling an integration (different button label and location depending on whether any are configured yet) so per-integration pages can link to it instead of restating it. - Conventions documented in the user-facing template: introduce Home Information (HI) on first mention; use 'item' for HI's representation, reserve 'entity' for upstream services that use that term; use 'Import' (first run) and 'Refresh' (subsequent), not 'sync', in user-facing copy. Developer-facing changes: - New per-integration dev docs for ZoneMinder and HomeBox; HA dev doc expanded from a single section to all six required sections. - Dev template explicitly directs writers to keep content high-level and refer to the code for details; existing dev docs follow that principle (key modules listed with one-line roles, no method signatures or field lists duplicated). - Weather integration doc gets a one-paragraph preface marking it as an internal-source exception that does not follow the per-integration template structure. Simulator documentation: - docs/dev/testing/test-simulator.md expanded to cover the simulator's purpose and architecture, the URL mappings for pointing each HI integration at the simulator, and the seed_sim_profiles command (with a profile summary table and a pointer to the command's docstring for the full operator workflow).	2026-05-05 23:02:07 -05:00
Thiago Trautwein Santos	10444fc165	Feature/234 homebox integration (#249) ... * finished Items and Locations * finish maintenance, labels and start notifier * finish v1 of HomeBox Client API * sync now creates and updates entities and entities attributes * create sync for homebox item attachments * finish homebox integration * finish test implementation * Improve Homebox API client quality and pattern conformance Align the Homebox integration with patterns established by the existing Home Assistant (HASS) integration, and fix several minor issues found during code review of PR #249: - Assert required API options in HbClient.__init__ instead of silently logging a warning, matching the HASS client's defensive pattern - Add warning log when login response lacks a token to aid debugging - Translate Portuguese log message to English for consistency - Reference HbClient constants in factory instead of magic strings - Initialize all cache lists in HomeBoxManager.__init_singleton__ - Update factory test to work with constant references on mock class * Fix sync bugs and remove dead code in Homebox converter - Fix attachment order_id overlap: fields and attachments each enumerated from 0 independently, causing interleaved display since EntityAttribute orders by order_id. Attachments now start after the last field index. - Fix variable shadowing in _sync_helper_entity_attributes where the loop variable 'hb_attachment' was immediately reassigned by tuple unpacking on the next line. - Remove unused _create_entity_attributes_from_hb_fields method, which was superseded by the sync code using hb_item_to_attribute_field_list. * Fix entity edit dialog for externally managed entities HomeBox entities have non-editable attributes with disabled HTML form fields. Browsers omit disabled fields from POST data, which caused AttributeForm.clean() to see empty values and raise spurious validation errors when updating the EntityType. - Skip validation for non-editable attributes in AttributeForm.clean(), since save() already short-circuits without writing for these - Add suppress_add_new property to AttributeForm to control whether the "Add New" attribute card is shown, following the existing pattern of display-hint properties (suppress_history, show_as_editable, etc.) - Pass can_add_custom_attributes through form_kwargs in the entity formset so the form can suppress the add-new card for externally managed entities even after a bound form re-render * Add HomeBox integration to documentation Add HomeBox alongside Home Assistant and ZoneMinder across all user-facing and developer documentation: Integrations guide, Features, FAQ, Installation, architecture overview, integration guidelines, and service patterns. Also remove stale reference to non-existent hi.services.weather module from integration guidelines example list. --------- Co-authored-by: Anthony Cassandra <github@cassandra.org>	2026-03-28 09:53:46 -05:00
Tony C	06bddccc6a	Updated docs, claude agents and commands (#193) ... * Working on improving AI agent docs. * Update dev/docs and claude agent configs. * Updated claude commands.	2025-09-17 17:19:51 +00:00
Tony C	436821a415	Added duration to video browse timline [#165] (#187) ... * Added duration time to video browse timeline. * Fixed date/time display bug. Tweaks to video stream display points. * Added correlation role/id for better video browse semantics. * Fix weather module async/sync issues and enable parallel test execution - Convert weather tests to AsyncTaskTestCase for proper async handling - Add async version of weather alerts enabled check - Fix cache interference in parallel tests with unique test identifiers	2025-09-15 17:36:29 +00:00
Tony Cassandra	a2994ca20e	Added document about Home Assistant Integration.	2025-09-09 11:41:36 -05:00
Tony C	39c7266e77	Refactor developer documentation for role-based use (#126) ... * Refactored dev documentation extensivekly. * Style tweaks and link removals.	2025-08-26 23:17:26 +00:00