mirror of https://github.com/cassandra/home-information.git synced 2026-06-11 17:15:38 -04:00

Files

Tony C ea991d691a Implement HomeBox Importer; refactor integration framework (#360 )

* Phase 1 (#358): rename connect/ → connector/ and lift capability-agnostic files to umbrella root

Mechanical refactor with no behavior change, setting up the layout for
the new importer/ subpackage that arrives in Phase 2.

The connect/ subpackage is renamed to connector/ for symmetry with
the forthcoming importer/. Five files that were already capability-
agnostic are lifted from connector/ up to the integrations/ umbrella
root so neither capability subpackage will depend on the other:

- placement_request.py
- integration_attribute_edit_context.py
- entity_operations.py
- integration_data.py
- integration_gateway.py

The remaining connector/ contents (views, synchronizer, monitors,
sync_check, sync_result, view_mixins, external_view_data, plus the
Connect-mode controller/manage-view-pane stubs and user_data_detector)
stay capability-specific.

view_mixins.py stays in connector/ as a single file; a future
refactor could split the capability-agnostic accessor methods up to
umbrella, but the call-site benefit is small and bundling that with
this rename would obscure the move.

The defensive Disable-side data_source predicate originally listed
in Phase 1 is deferred to Phase 7. The naive EXTERNAL filter would
break HA/ZM/Frigate Disable (their entities are INTERNAL), so the
guard needs to be conditional on the integration declaring both
capabilities — designing that alongside Phase 7's symmetric block-
modal capability-conditional detection.

* Phase 2 (#358): Importer protocol skeleton

Lands the abstract Importer base, transient models (CandidateItem,
IntegrationDiscardResult), and IntegrationImportResult dataclass.
Wires get_importer() into IntegrationGateway as the parallel of
get_synchronizer(), defaulting to None.

No live consumer yet — the HomeBox concrete implementation arrives
in Phase 3.

IntegrationImportResult is a parallel dataclass to
IntegrationSyncResult, not a subclass. Import-specific shape:
items_imported_count / items_skipped_count + imported_list +
info/error lists + placement_input. No reconnect/detach/remove
concepts (Import is add-only).

* Phase 3 (#358): HomeBox concrete Importer + capability-aware entity factory

Lands HomeBoxImporter as the first concrete consumer of the Phase 2
Importer protocol, plus the supporting infrastructure for it to share
HomeBox's entity-creation code with the CONNECT-mode synchronizer.

Capability-aware HbEntityFactory:
- Moved from connector/ to shared/ (used by both Connect and
Import now).
- New ``capability`` parameter on create_models_for_hb_item routes
to the correct flags: CONNECT keeps the read-only-mirror flags
(can_user_delete=False, allow_internal_attributes=False,
data_source=EXTERNAL); IMPORT sets HI-owned flags (deletable,
editable, data_source=INTERNAL).
- Fixes a latent post-#355 gap: data_source was never set at entity
creation time. Phase 4 of #355 backfilled existing rows; new
HomeBox-Connect entities created after #355 would have defaulted
to INTERNAL until now.

HbConverter payload defaults flipped to CUSTOM + is_editable=True.
The Connect-mode copy-on-sync architecture (pre-#354) used these
methods to produce PREDEFINED read-only attributes. Post-#354, the
only live caller is the new Import code, which wants user-owned
attributes. The methods are now unambiguously Import-flavored.

HomeBoxImporter (services/homebox/importer/homebox_importer.py):
- get_candidate_items: lightweight summary fetch, excludes archived.
- run_import: per-entity transaction inside the existing
integrations_sync exclusion lock; skip-if-already-imported by
integration_name; per-item failures aggregate into error_list.
- discard_imported_data: filters by data_source=INTERNAL so any
coexisting Connect-mode entities are untouched.

Dormant hb_attribute_sync.py (224 lines) deleted. Replaced by a
~15-line populate_attributes_for_imported_entity helper next to the
existing HbImporter classmethods — Import is add-only so the
sync-style dispatch / update / remove logic no longer applies.

HbMetaData.capabilities now declares both CONNECT and IMPORT.
HomeBoxGateway.get_importer() returns the importer instance.

* Phase 4 (#358): Data Import config tab + Configure form (preview-side)

Lands the user-visible Data Import tab parallel to Integrations, plus
the configure-form workflow up through preview rendering. Confirm →
run is Phase 5.

New tab: ConfigPageType.DATA_IMPORT (url_name 'integrations_import_home',
label 'Data Import'). URL prefix is integrations_import_* — matches
the eventual rename of integrations_home → integrations_connect_home
the framework will adopt.

DataImportPageView: flat list, one row per IMPORT-capable integration.
Per-row CONFIGURE button (always) and DISCARD button (only when
imported entities exist for that integration_id with
data_source=INTERNAL). Dual-capability rows (HomeBox today) carry a
terse "Also available as an Integration" note linking to a bottom-of-
page explanation contrasting Data Import vs Integration (Connect).

ImporterConfigureView: credentials form (reuses
IntegrationAttributeItemEditContext with capability=IMPORT, action
button labeled IMPORT). POST validates configuration + access via
the same validate_attributes_extra_helper Connect uses, then fetches
candidates via Importer.get_candidate_items(), computes new vs.
skipped counts by integration_name match against existing HI
entities, and renders the preview modal.

Preview modal: "Would import N new items. M existing items would be
skipped." CONFIRM IMPORT and CANCEL actions. CONFIRM posts to the
integrations_import_run URL — stub view returns 501 in this phase;
Phase 5 wires the actual run.

ImporterRunView and ImporterDiscardView land as 501-returning stubs
so the URL reversals from this phase's templates resolve cleanly;
Phase 5 and 6 fill in the implementations.

view_mixins.py moved from connector/ up to umbrella root so both
Connect and Import views can extend IntegrationViewMixin. The
render_sync_result method on it is only called by Connect views;
Import views never touch it. No import-time Connect coupling — the
synchronizer.sync() call inside render_sync_result is runtime-only.

* Phase 5(a) (#358): Confirm → Run → Result modal + placement reuse

Wires the IMPORT workflow end-to-end. After the preview modal's
CONFIRM IMPORT, the importer runs, the result modal renders with
imported / skipped counts, and a "Place N new items" CTA hands off
to the existing placement flow when entities were created.

Importer base gains group_entities_for_placement (default ungrouped,
mirrors IntegrationSynchronizer's same-named method). HomeBoxImporter
stays with the default — HomeBox has no inherent grouping at v1
(Labels and Locations are metadata-on-entity).

HomeBoxImporter._run_import_locked now tracks created entities and
populates result.placement_input via group_entities_for_placement,
mirroring HomeBoxSynchronizer's _sync_impl pattern.

ImporterRunView.post:
- Runs importer.run_import() inside a try/finally that invalidates
the metadata + sensor-response caches afterward (mirrors
Connect's render_sync_result).
- Builds the placement URL via the existing PlacementUrlParams.
- Reuses the existing integrations_placement URL — HomeBox is
dual-capability so its synchronizer is available for the
placement view's group_entities_for_placement call.

import_result.html: counts hero (Imported / Skipped), per-item
list, errors. "Place N new items" CTA on success; OK button on
the no-imports / errors paths. Modeled on sync_result.html but
trimmed for Import's narrower outcome shape.

Manual verification: end-to-end Import works after the manager
picks up newly-saved credentials. Phase 5(b) addresses the
is_enabled / manager-reload race that surfaces this timing issue
and the symmetric one in the Connect-enable flow.

* Phase 5(b) (#358): decouple Connect-mode is_enabled from credential loading

The HomeBoxManager (shared/) was checking the Connect-mode is_enabled
flag inside _load_attributes and raising IntegrationDisabledError to
nullify hb_client. That coupling produced two related bugs:

1. Import workflow: after the user disables HomeBox-Connect,
HomeBoxImporter.get_candidate_items found hb_client=None and
returned empty list — preview showed 0 items even though the
upstream simulator had 25.

2. Connect-enable race: after IntegrationEnableView flipped
is_enabled=True via enable_integration, the manager's
post_save signal was deferred by ~0.1s
(DelayedSignalProcessor). The synchronous synchronizer.sync()
that immediately followed saw hb_client=None and failed with
"integration is disabled" even though the user had just
enabled it.

Fix:

* Strip the is_enabled gate out of HomeBoxManager._load_attributes
(and the matching except IntegrationDisabledError branch from
_reload_implementation). The Connect-mode enable lifecycle is
not a shared/ concern — credentials load whenever they are
structurally valid so the IMPORT capability works regardless
of Connect-mode state. Connect-side guards (manage view,
disable, pause, resume) check integration.is_enabled directly
at the view layer, where the capability context is known.

* IntegrationManager.enable_integration: synchronously call
gateway.notify_settings_changed() after the save+refresh,
before the monitor launch. Closes the delayed-signal race so
the synchronizer immediately sees fresh credentials.

* ImporterConfigureView.post: same synchronous notify after
post_attribute_form succeeds, before invoking the importer.
Credentials just changed; the manager needs to pick them up
before run_import.

HASS, ZM, Frigate managers are Connect-only and are left alone —
their is_enabled gate is fine because they have no other capability
that needs to bypass it.

Manual verification: Import workflow shows correct candidate count
with Connect disabled; Connect-enable runs sync successfully on the
first click.

* Phase 6 (#358): Discard flow

Wires the DISCARD button on the Data Import page. GET renders a
confirmation modal showing the count of imported items; POST runs
the importer's discard_imported_data and redirects back to the
Data Import page.

Single-action confirm (no SAFE/ALL variants). Imported items ARE
the user data — the Connect-side preserve-with-user-data semantic
doesn't apply.

Defensive scoping is in HomeBoxImporter.discard_imported_data
(Phase 3): the deletion query filters by data_source=INTERNAL, so
any coexisting Connect-mode (EXTERNAL) entities for the same
integration_id stay untouched. Confirmed via test.

Post-discard cache invalidations mirror the Connect-side Disable
cleanup: IntegrationMetadataCache + SensorResponseManager. Sensor-
response cache invalidation is symmetric with sync's post-write
behavior even though imported items don't have sensors today —
keeps the discard path defensive for future Importers that might
attach sensors.

* Clarify EntityDataSource semantics: every Connect-mode entity is EXTERNAL

The Phase 4 (#355) migration used `allow_internal_attributes=False`
as the proxy for EXTERNAL, which conflated two separate concerns:
"data is sourced from upstream" and "user can edit HI-side
attributes." Under the conflation, HA/ZM/Frigate-Connect entities
were marked INTERNAL even though their state is sourced from
upstream — only HomeBox-Connect (allow_internal_attributes=False)
got EXTERNAL.

The clarified semantics: data_source=EXTERNAL means data is
actively being sourced from an external upstream system, which is
true of EVERY Connect-mode integration regardless of whether HI
permits user-edited attributes on top. data_source=INTERNAL means
no live upstream link — Import-mode (HI-owned post-import) and
native entities.

| integration | mode | data_source |
|-------------------|---------|-------------|
| zm/frigate/ha | connect | EXTERNAL |
| homebox | connect | EXTERNAL |
| homebox | import | INTERNAL |
| (native) | n/a | INTERNAL |

Changes:

* Migration 0021_entity_data_source.py: backfill broadens to mark
every integration-attached entity EXTERNAL. Adjusted in place
rather than as a fix-up migration since 0021 hasn't been released
outside local environments and has no downstream migrations.

* HassConverter, FrigateSynchronizer, ZmSynchronizer (both service
and monitor entity creation): explicitly set
data_source=EXTERNAL on new Connect-mode entities. Closes the
post-#355 gap where the field wasn't being set at creation time
for these integrations — the migration backfilled existing rows
but new rows fell back to the INTERNAL default.

* HomeBoxConnector.get_external_view_data: gates on
data_source=EXTERNAL. Import-mode HomeBox entities are HI-owned
after import, so the live HomeBox view isn't authoritative for
them and would mislead the operator.

* test_models.py: migration-backfill test renamed and assertions
updated to reflect the broadened predicate (HA-like entity now
becomes EXTERNAL too).

* test_hb_connector.py: HomeBox entity fixture sets
data_source=EXTERNAL so the connector's new gate doesn't suppress
the connector's existing test surface.

Required for Phase 7 (#358 block modal): the modal queries
data_source to distinguish Connect-mode entities (EXTERNAL) from
Import-mode entities (INTERNAL) for an integration. The old
semantics would have miscategorized future dual-capability HA/ZM/
Frigate.

* Phase 7 (#358): block modal for cross-capability mode switching

When an operator starts an integration's initial-Configure flow for
one capability while existing data from the other capability is
still present, surface a block modal that routes them to remediate
before proceeding. Connect ↔ Import switches are intentionally made
harder — the user navigates explicitly, performs the cleanup
themselves, then re-clicks CONFIGURE. No combined transition flow.

CapabilityBlockViewMixin (in hi/integrations/view_mixins.py):
- Single helper render_capability_block_if_conflict that takes
only integration_data + capability_being_initiated. The mixin
composes all user-facing copy (title, body clauses, navigation
link) from per-capability config dicts so block modals read
uniformly across both directions and any future capabilities.
- Detects the OTHER capability automatically. Only fires when
the integration declares both capabilities; single-capability
integrations get None.
- Counts entities of the OTHER capability's data_source for the
integration; zero means no conflict.

Connect side: IntegrationEnableView.get checks the block on the
initial-Connect path (is_enabled=False). Re-Configure of an
already-enabled integration always proceeds — that's not a new
mode being initiated.

Import side: ImporterConfigureView.get checks the block when no
Import entities yet exist for the integration. Re-import (entities
already present) always proceeds — that's the standard incremental-
import flow.

capability_blocked.html: generic block modal template parameterized
by title + body clauses + link. Extends modals/action_2_with_cancel
so the inherited CANCEL just dismisses.

Single-capability integrations (HA/ZM/Frigate today) never trigger
the block: the mixin's "other capability not in capabilities" guard
short-circuits before any DB query.

* Phase 8 (#358): documentation for the Data Import capability

User-facing:
* docs/DataImport.md (new) — landing page for the Data Import
feature; importing/discarding walkthrough; lists supported
integrations. Concept-level prose only; no hardcoded button
labels so the doc doesn't drift if UI strings change.
* docs/Integrations.md — Integrations vs. Data Import comparison
pointing at the new landing page.
* docs/integrations/homebox.md — added Data Import section
describing the alternative mode and when to choose it.

Developer-facing:
* docs/dev/integrations/data-import.md (new) — framework
orientation for the IMPORT capability. Lists where code lives
(importer/, view_mixins.py, services/homebox/importer/), how a
gateway opts in, and the key design decisions (per-entity
transaction, skip-by-integration_name, shared lock,
data_source as capability-state signal). Reference-style;
points at the code rather than restating API.
* docs/dev/integrations/integration-guidelines.md — Capability
Model subsection refreshed: CONNECT and IMPORT now both have
concrete pointers; the #355 "reserved for #358" placeholder
is gone.
* docs/dev/integrations/homebox.md — updated stale module paths
to reflect the connector/importer/shared split landed in #355
Phase 3 + #358 Phase 1. Added a short Importer section
pointing at the framework doc.

* Integration framework cleanup: capability-aligned naming and template structure

Rename the Connect/Import vocabulary uniformly across the integration
framework now that HomeBox is the dual-capability archetype.

Class renames:
- IntegrationSynchronizer -> IntegrationConnector (+ Hass/Zm/Frigate/HomeBox subclasses)
- Importer -> IntegrationImporter
- IntegrationEnableView -> ConnectorConfigureView
- HomeBoxConnector (the per-entity external-view resolver) -> HomeBoxExternalViewResolver

Gateway method:
- IntegrationGateway.get_synchronizer() -> get_connector()

File renames (git mv, history preserved):
- integration_synchronizer.py -> integration_connector.py
- importer.py -> integration_importer.py
- homebox/connector/hb_sync.py -> homebox_connector.py
- homebox/connector/hb_connector.py -> hb_external_view_resolver.py
- corresponding test files

URL name renames:
- integrations_home -> integrations_connect_home
- integrations_enable -> integrations_connect_configure

Enum renames (ConfigPageType):
- INTEGRATIONS -> INTEGRATIONS_CONNECT
- DATA_IMPORT -> INTEGRATIONS_IMPORT

View mixin:
- IntegrationViewMixin.get_integration_data(integration_id) -> (request, *args, **kwargs)
Mirrors EntityViewMixin.get_entity; pulls integration_id from kwargs,
raises BadRequest if missing and Http404 if not registered. ~20 call
sites adopted the uniform pattern.

Template restructure (src/hi/integrations/templates/integrations/):
- import/ -> importer/ (avoids Python reserved word, matches sub-package convention)
- importer/ now has modals/ subdir matching the pages/ convention
- New connector/{modals,pages,panes}/ subtree for Connect-specific templates
- Top-level modals/, panes/ now hold only genuinely shared cross-capability
templates (capability_blocked, placement flow, attribute-form body)
- Editor backup .html~ files removed

Misc:
- IntegrationImportResult folded into importer/transient_models.py
(single-class file with no module-level documentation)

* Remove dormant IntegrationManageViewPane seam; rename to ConnectorManageView

The per-integration "manage view pane" extension point on
IntegrationGateway was dead machinery: all four service subclasses
returned templates containing only a {% comment %} block. The seam
also pre-dated the capability model — its return type lives under
connector/, but the hook was on the capability-agnostic gateway.

Rather than relocate the hook to IntegrationConnector, delete the
seam entirely (base class, 4 subclasses, 4 stub templates, gateway
method, view context plumbing, template include). It can be
reintroduced correctly on IntegrationConnector when an integration
actually needs per-integration content in the manage page.

Also rename the view to reflect that the Manage page is Connect-
specific:
- IntegrationManageView -> ConnectorManageView
- integrations_manage URL name -> integrations_connect_manage

* Push Connect-specific hooks from IntegrationGateway to IntegrationConnector

IntegrationGateway accumulated eight Connect-mode hooks over time, all
returning types that already lived under connector/. Now that the
capability model has IntegrationConnector and IntegrationImporter as
peer capability classes (cf. the recent IntegrationManageViewPane
cleanup), these hooks belong on the connector — not on the
capability-neutral gateway. Import-only integrations should not be
forced to stub them out.

Methods moved from IntegrationGateway to IntegrationConnector:
- get_monitor()
- get_controller()
- get_health_status_provider()
- get_entity_video_stream()
- get_entity_video_snapshot()
- get_sensor_response_video_stream()
- get_sensor_response_event_snapshot_url()
- get_external_view_data()

Per-service implementations relocated from each *Gateway (integration.py)
to the corresponding *Connector.

Production call sites updated to chain through get_connector() with
None-guards where the caller might encounter an Import-only or
connector-less integration. The three Connect-side view sites trust the
structural invariant. The do_control path raises a clear RuntimeError
if no connector or controller is available.

File renames to match the relocated class names:
- frigate_sync.py -> frigate_connector.py (FrigateConnector)
- hass_sync.py -> hass_connector.py (HassConnector)
- zm_sync.py -> zm_connector.py (ZmConnector)
- corresponding test files

IntegrationGateway's surface is now its capability-neutral core:
get_metadata, notify_settings_changed, validate_configuration,
validate_access, get_connector, get_importer.

* Fix re-Configure credentials race; remove redundant block-check pre-gates

Two related issues in the Configure flows:

1. ConnectorConfigureView.post relied on enable_integration to fire
notify_settings_changed(), but enable_integration only fires it on
the disabled->enabled transition. Re-Configure of an already-enabled
integration left the singleton manager holding stale credentials,
and the subsequent sync ran against them. The post_save signal would
eventually nudge via DelayedSignalProcessor, but the 0.1s delay
races the immediate sync. Lift the synchronous notify out of
enable_integration and into the view, right after post_attribute_form,
so both initial-Connect and re-Configure get the nudge.

The matching ImporterConfigureView.post already did this; the
Connect side was the asymmetric outlier.

2. Both Configure views had a pre-gate that suppressed
render_capability_block_if_conflict in certain re-entry paths
(is_enabled=True for Connect, has_imported=True for Import).
Under the mode-switch invariant the pre-gate is redundant — the
block check itself short-circuits to None when there's no
conflicting data. The pre-gate's only effect was to silently
bypass the block check when the invariant was violated (mixed
INTERNAL+EXTERNAL state), letting the user compound the
corruption. Remove both pre-gates so the block fires in any
mixed-state scenario and the user is told to remediate.

Two tests deleted that asserted the now-removed bypass behavior:
- test_get_import_not_blocked_when_already_imported
- test_already_enabled_does_not_trigger_block

The canonical block-fires-on-conflict and re-Configure-works paths
remain covered by neighboring tests.

* Extract CapabilityConfigureView shared base for both Configure modals

ConnectorConfigureView and ImporterConfigureView shared a 5-step GET
recipe (block check → ensure_all_attributes_exist → build edit context
→ render modal) and a 3-step POST recipe (post_attribute_form → error
short-circuit → success handler). Pull the shared structure into a
new framework-level CapabilityConfigureView in
hi/integrations/views.py. Subclasses now declare four class constants
(capability, button_label, template_name, error_title) and override
handle_post_success() with the per-capability behavior.

ConnectorConfigureView.handle_post_success: enable_integration, then
sync or redirect-to-manage.
ImporterConfigureView.handle_post_success: fetch candidates, compute
new-vs-skipped, render preview.

Each subclass owns the timing of notify_settings_changed() because
the right moment is capability-specific:
* Connect-side managers (Frigate/Hass/ZM) gate client (re)build on
integration.is_enabled, so the notify must fire AFTER
enable_integration — otherwise the manager reloads with
is_enabled=False, nulls its client, and the immediately-following
sync sees no client.
* Import flows fire the notify before reading candidates.

The unconditional call in both subclasses also covers the re-Configure
race (already-enabled integration), where enable_integration
early-returns without nudging and the singleton manager would
otherwise keep stale credentials.

* Flatten homebox/shared/ up one level

The capability-layout convention is: inter-capability files live at
the integration's top level (alongside integration.py / apps.py /
enums.py), while capability-specific files live under connector/ or
importer/. HomeBox's shared/ predated the convention; it's the last
holdout. Move the six modules up:

homebox/shared/hb_client.py -> homebox/hb_client.py
homebox/shared/hb_client_factory.py -> homebox/hb_client_factory.py
homebox/shared/hb_converter.py -> homebox/hb_converter.py
homebox/shared/hb_entity_factory.py -> homebox/hb_entity_factory.py
homebox/shared/hb_manager.py -> homebox/hb_manager.py
homebox/shared/hb_models.py -> homebox/hb_models.py

shared/__init__.py removed; shared/ directory gone. 21 absolute
import-site rewrites plus 2 @patch strings updated. Docs and simulator
docstring path references brought in line.

* Data Import UI/UX refinements

Tab order
- Move Data Import to the right of Triggers in the Config tabs (matches
natural read order: settings -> integrations -> trigger setup ->
data import -> system info).

Import preview modal (HomeBox confirm)
- Title becomes "<Label> Import" when there's nothing to confirm
(instead of the awkward "Import from HomeBox?" question).
- Body now handles the no-new-items cases explicitly: "All N upstream
items have already been imported." or "No items found to import."
instead of "Would import 0 new items."
- Single centered DONE button replaces the lone left-aligned CANCEL
when the preview is informational only.
- Add data-async="modal" to the CONFIRM IMPORT form so it submits
through the modal pipeline like every other action on the page.
- Show a contextual "Currently N items imported from <Label>." line
whenever the integration has prior imports — signals re-import
vs. first-time state. View filters the count by data_source=INTERNAL.

Data Import page layout
- Replace sparse rows with a responsive Bootstrap card grid
(col-12 col-sm-6 col-md-4 col-lg-3), 4 columns on large screens.
Logo centered above the integration name, action buttons stacked
at the card bottom (CONFIGURE always; DISCARD only when has_imported).
- New hi-integration-logo--card modifier (72px) for the card-prominent
size.
- Drop btn-control width:100% from the action buttons (the row layout
no longer benefits from it; btn-block within the card stack is what
fills card width now).
- Remove the always-below-the-fold "Data Import vs. Integration"
footer block.
- Add DataImportInfoView (HiModalView) at integrations_import_info
rendering a static info modal with the same explanation content.
- On dual-capability rows, the entire "Also available as an
Integration. <info-icon>" message is now a single clickable link
that opens the info modal.

Capability-blocked modal
- Title was getting truncated ("Cannot configure <Label> as
<Capability>" doesn't fit a narrow modal). Title is now just the
integration label; the "Cannot configure as <Capability>." line
moves into the body as an h5 sub-heading above the existing
explanatory paragraph. View context drops the pre-composed title
key and passes my_label so the template composes the sub-heading.

* Tighten data_source scoping at mode-switch invariant seams

The mode-switch invariant (a given integration's entities are all
EXTERNAL for Connect, or all INTERNAL for Import, never mixed) was
enforced at the Configure entry point but several downstream code
paths filtered only on integration_id, leaving data-source-blind
seams. The same code review also surfaced that the disconnect/
reconnect path mishandles data_source when transitioning between
attached/detached states.

POST block-check (CapabilityConfigureView.post)
- GET ran the block check; POST didn't. A direct POST (cached form,
replayed request) would bypass the invariant. POST now re-invokes
render_capability_block_if_conflict before delegating to the
capability-specific handle_post_success.

disable_integration seed (IntegrationManager.disable_integration)
- Seed query now filters by data_source_str=EXTERNAL. A Connect-side
Disable can no longer silently delete imported INTERNAL data if the
invariant ever leaks. Native and detached entities (integration_id
NULL) were already outside the seed.

HomeBoxConnector sync queries
- _get_current_integration_keys (sync-check probe) and
_get_existing_hb_entities (sync run) both filter by EXTERNAL.
INTERNAL imports under the same integration_id no longer risk
being adopted as Connect rows and routed through mass-remove or
auto-reconnect.

HomeBoxImporter preview/run symmetry
- _run_import_locked's existing_integration_names set now filters by
INTERNAL to match ImporterConfigureView.handle_post_success's
preview predicate. A stale EXTERNAL row can no longer turn an
advertised new candidate into a skip at run time.

Disconnect/reconnect data_source transition
- EntityIntegrationOperations.preserve_with_user_data now flips
entity.data_source to INTERNAL on detach. A detached entity is no
longer constrained by an upstream system — HI owns the editable
representation. The reconnect path is already correct: all four
service converters set data_source=EXTERNAL during the rebuild
invoked by _rebuild_integration_components. Pre-migration detached
rows are also correct: migration 0021's column default of
'internal' applied to rows with no integration_id.

Test fixtures
- Four entity-creation sites in test_integration_manager.py and
test_homebox_connector.py now explicitly set
data_source_str=EXTERNAL for integration-attached test entities,
since the new EXTERNAL filters no longer match the implicit
default.

* Move IMPORT to previous_integration_id; introduce named entity-state queries

The IMPORT path was overloading integration_id (the live-ownership
column) to mean "where this entity was sourced from." That conflation
forced data_source_str as a discriminator and made post-import code
paths brittle: any query that scoped by integration_id and a data
source filter could quietly diverge.

Under the model in this commit, the entity's lifecycle state is
encoded entirely by which integration_* column is populated:

- integration_id set -> EXTERNAL (live Connect)
- integration_id NULL, previous set -> imported OR detached
(HI-owned, has provenance)
- both NULL -> INTERNAL (native)

IMPORT now writes to previous_integration_*. integration_id stays
NULL for imported rows because no live integration owns them.

Manager methods (EntityModelManager) give call sites a vocabulary
matching the data-source semantics:

- external_for(integration_id) active-Connect rows
- imported_for(integration_id) import-side intent
- detached_for(integration_id) reconnect-candidate intent
- with_integration_provenance(id) any row with this integration's
identity in either column

imported_for / detached_for share an implementation today (the
column shape is the same); each name documents intent at the call
site so a future discriminator could differentiate without touching
callers.

Block check: only IMPORT initiation can collide (it would create
new rows; an active-Connect row already exists for the upstream
key). CONNECT initiation never needs blocking — sync's reconnect-
then-create order adopts any pre-existing provenance entities into
the live Connect session. CapabilityBlockViewMixin loses its
symmetric dispatch and the _CAPABILITY_* dicts.

Post-import placement bug: EntityPlacementService.query_unplaced
filtered by integration_id, so imports (integration_id=NULL) were
invisible to the placement modal — the user saw "No items found"
after a successful import. Switched to with_integration_provenance
so both EXTERNAL and imported rows are eligible.

Other writes / queries swept to the new managers:
- HbEntityFactory.create_models_for_hb_item IMPORT branch
- HomeBoxConnector._get_existing_hb_entities / _get_current_integration_keys
- IntegrationManager.disable_integration seed
- EntityIntegrationOperations.find_reconnect_candidates / get_removal_entity_ids
- DataImportPageView.has_imported per-row check
- ImporterConfigureView preview's skip-detection
- HomeBoxImporter._run_import_locked / discard_imported_data
- ImporterDiscardView count

No migration: IMPORT has never shipped in production; the
data_source_str column stays for now and will be retired in the
follow-up commit that introduces the derived data_source property.

One test deleted (test_initial_connect_blocked_by_existing_import_data)
because CONNECT-init no longer blocks under the unified model;
the canonical IMPORT-init block direction is still covered by
test_get_import_blocked_by_existing_connect_data.

* Derive data_source from columns; drop EntityDataSource and data_source_str

The data_source_str column was a stored discriminator that became
fully redundant once IMPORT moved to previous_integration_id: an
entity's state is now encoded entirely by which integration_*
columns are populated. Removing the column eliminates a class of
"stored flag drifted from column reality" bugs and gives the
codebase one fewer thing to keep consistent.

Model:
- Drop the data_source_str field from Entity and the
EntityDataSource enum.
- Delete migration 0021_entity_data_source (the migration that
introduced the column). Import has never shipped, so there is no
production data dependency. Dev environments drop the column
manually.
- Add three derived properties on Entity that name the questions
call sites actually ask:
* is_external — actively attached to an integration
* has_integration_provenance — carries record of an integration
it came from (imported or detached)
* is_imported / is_detached — synonyms today (same column shape);
separate names document intent at the call site, parallel to
the manager methods external_for / imported_for / detached_for.

Sweep:
- Remove every entity.data_source = ... write site. The four
service converters and HbEntityFactory no longer need to set it —
the columns ARE the state. preserve_with_user_data likewise.
- Replace the one production read site in hb_external_view_resolver
with `if not entity.is_external`.
- Test fixtures: drop data_source_str=... kwargs from
Entity.objects.create calls (the column is gone). Two legacy
"integration_id + data_source_str=INTERNAL" mixes converted to
provenance form (previous_integration_id + previous_integration_name).
- Replace data_source assertions with property checks.

UI rename: "Detached from X" -> "From X". The unified model treats
imported and detached as the same state, so the badge text needs to
read accurately for both origin stories. Two templates plus comment
references.

Sync-result reporting fixes (surfaced during manual testing of the
above):
- ZM _create_zm_entity now appends the service entity to created_list
(was hidden in info_list). The user-facing "created" count and
detail now match what got persisted. Placement input remains
unaffected — _sync_impl builds it from monitor entities only.
- ZM _create_monitor_entity and Frigate _create_camera_entity gate
created_list.append on is_fresh_create. Auto-reconnect rebuilds
no longer double-list the row in both created_list and
reconnected_list (HomeBox/HASS did not have this bug; their
_rebuild_integration_components calls the converter directly,
bypassing the per-connector _create_entity).
- Both result-modal placement CTAs now read "Place new items"
without a count. The previous count came from created_list length,
which includes reconnected entities and (for ZM) the service
entity — but the placement modal correctly excludes those. Action
prompt > misleading count.

2026-05-25 18:56:50 -05:00

6.6 KiB

Raw Permalink Blame History

Frigate

Overview

The Frigate integration follows the standard pattern in integration-guidelines.md: a FrigateGateway exposes the framework surface; a singleton FrigateManager owns shared client state and the active FrigateClient instance; the synchronizer imports each Frigate camera as a HI camera entity with a single object-presence sensor; FrigateMonitor polls in the background for event changes.

Frigate communicates over plain HTTP only — there is no MQTT path (HI doesn't have MQTT plumbing); the API surface is small enough that HTTP polling delivers a usable experience without sub-second latency.

User-facing setup lives in docs/integrations/frigate.md.

Key modules

src/hi/services/frigate/integration.py — FrigateGateway. Framework entry point.
src/hi/services/frigate/frigate_manager.py — FrigateManager. Singleton holding the active FrigateClient, integration attributes, and change-listener fan-out.
src/hi/services/frigate/frigate_client.py — FrigateClient. Encapsulated HTTP client wrapping the Frigate REST API.
src/hi/services/frigate/frigate_connector.py — FrigateConnector. Drives sync; per-camera entity creation in _create_camera_entity.
src/hi/services/frigate/frigate_converter.py — FrigateConverter. Wire-format ↔ HI model translation. Owns the canonical OBJECT_PRESENCE mapping (Frigate's raw object class → one of person / car / animal / package / other / none).
src/hi/services/frigate/monitors.py — FrigateMonitor. Periodic poll for camera events; emits SensorResponse updates for the object-presence sensor.
src/hi/services/frigate/frigate_controller.py — FrigateController. v1 stub: returns "no control mapping" for every input. Frigate's only HTTP-reachable operator-toggle (PUT /api/config/set on cameras.<name>.detect.enabled) is a config edit rather than transient state, so no control surface is exposed in v1.

API patterns

Frigate's REST API is the only command/query protocol; live snapshots are JPEG bytes from /api/<camera>/latest.jpg. The integration was validated only against installs with Frigate's authentication disabled. An optional verbatim Authorization header field is plumbed through but untested. No login flow, token refresh, or JWT handling exists.

Per-request timeouts and the polling cadence are defined in constants.py (FrigateTimeouts).

Event polling model

The polling pipeline is the load-bearing complexity of this integration. Frigate's /api/events?after=T filters strictly on start_time > T, which means once the polling cursor advances past an event's start_time, that event is invisible to cursor scans forever — even after it closes. A ZM-style cursor-hold approach (hold the cursor back at the open event's start_time) is incompatible: with strict > semantics, the held cursor excludes the very event being held for.

The monitor instead runs three phases per cycle:

Cursor scan (?after=cursor): for each event whose start_time is past the cursor, emit a START transition. If the event was already closed when first seen (lifetime shorter than the poll interval), also emit an END. Otherwise add the event to _tracked_events, keyed by id. Advance cursor to the latest start_time observed.
Per-id refresh (GET /api/events/<id>): for each id in _tracked_events, fetch its canonical state.
- Closed → emit END, drop from tracking.
- 404 (Frigate cleared the event) → force-close, drop.
- Aged past MAX_OPEN_EVENT_AGE_SECS → force-close, drop.
- Still open → refresh snapshot, keep tracking.
Heartbeat: emit OBJECT_NONE for cameras with no activity this cycle and no event currently in _tracked_events. Without this, a quiet camera's state goes stale.

The cursor never moves backward; the tracked-event set is the only state that can grow during a cycle. API budget per cycle is 1 + N calls where N is the count of currently-open events (typically 0 or 1 in normal home use).

FrigateMonitor._initialize seeds the cursor at datetimeproxy.now() on startup. Events open at the moment HI starts are not seeded — the v1 posture is "HI's detection window begins when HI starts."

Implementation notes

Object detection mapping. Frigate's raw object class set is model-dependent (default YOLO has ~80 classes; custom models can have arbitrary classes). HI maps these onto a canonical 6-value OBJECT_PRESENCE range — see FrigateConverter. Classes that don't map to a named bucket land in other. The integration is the only place this mapping lives; do not duplicate it elsewhere.
Single sensor per camera. Frigate couples motion to object detection — there is no motion-without-class signal on the events API — so OBJECT_PRESENCE subsumes the "is motion happening" signal and a separate MOVEMENT sensor would always mirror it.
MQTT is intentionally not supported. HI doesn't have an MQTT client and Frigate's HTTP API is sufficient for the use cases HI's spatial-display model requires.
Zones and sub-labels travel as event metadata. Frigate emits zone-enter / zone-leave events and rich sub-label data (face recognition, LPR); HI surfaces both as detail_attrs on the event without promoting them to typed states. Revisit if/when a use case needs rule-based branching on zones.
Force-close timeout. MAX_OPEN_EVENT_AGE_SECS (1 hour) caps how long an event may stay in _tracked_events before HI synthesizes an END row. Intended for orphaned events (Frigate restart, dropped detection). No equivalent to ZM's auto-close-on-no-update behavior — Frigate is consulted by id on every cycle, so a stuck event surfaces directly rather than via timeout heuristics.

Testing approach

Tests live in src/hi/services/frigate/tests/. The monitor's phase-by-phase invariants and the open→closed transition that motivated the rewrite are covered in test_frigate_monitor.py.

Manual end-to-end testing uses the simulator; Frigate simulator support lives at src/hi/simulator/services/frigate/. The simulator's get_events_after mirrors Frigate's strict > semantics so monitor behavior validated against the simulator matches real Frigate. For the operator workflow and profile descriptions, see docs/dev/testing/test-simulator.md.

References

Frigate HTTP API documentation
User-facing setup: docs/integrations/frigate.md
Tracking issue: https://github.com/cassandra/home-information/issues/233

6.6 KiB Raw Permalink Blame History