mirror of
https://github.com/cassandra/home-information.git
synced 2026-06-12 09:37:09 -04:00
ea991d691a
* Phase 1 (#358): rename connect/ → connector/ and lift capability-agnostic files to umbrella root Mechanical refactor with no behavior change, setting up the layout for the new importer/ subpackage that arrives in Phase 2. The connect/ subpackage is renamed to connector/ for symmetry with the forthcoming importer/. Five files that were already capability- agnostic are lifted from connector/ up to the integrations/ umbrella root so neither capability subpackage will depend on the other: - placement_request.py - integration_attribute_edit_context.py - entity_operations.py - integration_data.py - integration_gateway.py The remaining connector/ contents (views, synchronizer, monitors, sync_check, sync_result, view_mixins, external_view_data, plus the Connect-mode controller/manage-view-pane stubs and user_data_detector) stay capability-specific. view_mixins.py stays in connector/ as a single file; a future refactor could split the capability-agnostic accessor methods up to umbrella, but the call-site benefit is small and bundling that with this rename would obscure the move. The defensive Disable-side data_source predicate originally listed in Phase 1 is deferred to Phase 7. The naive EXTERNAL filter would break HA/ZM/Frigate Disable (their entities are INTERNAL), so the guard needs to be conditional on the integration declaring both capabilities — designing that alongside Phase 7's symmetric block- modal capability-conditional detection. * Phase 2 (#358): Importer protocol skeleton Lands the abstract Importer base, transient models (CandidateItem, IntegrationDiscardResult), and IntegrationImportResult dataclass. Wires get_importer() into IntegrationGateway as the parallel of get_synchronizer(), defaulting to None. No live consumer yet — the HomeBox concrete implementation arrives in Phase 3. IntegrationImportResult is a parallel dataclass to IntegrationSyncResult, not a subclass. Import-specific shape: items_imported_count / items_skipped_count + imported_list + info/error lists + placement_input. No reconnect/detach/remove concepts (Import is add-only). * Phase 3 (#358): HomeBox concrete Importer + capability-aware entity factory Lands HomeBoxImporter as the first concrete consumer of the Phase 2 Importer protocol, plus the supporting infrastructure for it to share HomeBox's entity-creation code with the CONNECT-mode synchronizer. Capability-aware HbEntityFactory: - Moved from connector/ to shared/ (used by both Connect and Import now). - New ``capability`` parameter on create_models_for_hb_item routes to the correct flags: CONNECT keeps the read-only-mirror flags (can_user_delete=False, allow_internal_attributes=False, data_source=EXTERNAL); IMPORT sets HI-owned flags (deletable, editable, data_source=INTERNAL). - Fixes a latent post-#355 gap: data_source was never set at entity creation time. Phase 4 of #355 backfilled existing rows; new HomeBox-Connect entities created after #355 would have defaulted to INTERNAL until now. HbConverter payload defaults flipped to CUSTOM + is_editable=True. The Connect-mode copy-on-sync architecture (pre-#354) used these methods to produce PREDEFINED read-only attributes. Post-#354, the only live caller is the new Import code, which wants user-owned attributes. The methods are now unambiguously Import-flavored. HomeBoxImporter (services/homebox/importer/homebox_importer.py): - get_candidate_items: lightweight summary fetch, excludes archived. - run_import: per-entity transaction inside the existing integrations_sync exclusion lock; skip-if-already-imported by integration_name; per-item failures aggregate into error_list. - discard_imported_data: filters by data_source=INTERNAL so any coexisting Connect-mode entities are untouched. Dormant hb_attribute_sync.py (224 lines) deleted. Replaced by a ~15-line populate_attributes_for_imported_entity helper next to the existing HbImporter classmethods — Import is add-only so the sync-style dispatch / update / remove logic no longer applies. HbMetaData.capabilities now declares both CONNECT and IMPORT. HomeBoxGateway.get_importer() returns the importer instance. * Phase 4 (#358): Data Import config tab + Configure form (preview-side) Lands the user-visible Data Import tab parallel to Integrations, plus the configure-form workflow up through preview rendering. Confirm → run is Phase 5. New tab: ConfigPageType.DATA_IMPORT (url_name 'integrations_import_home', label 'Data Import'). URL prefix is integrations_import_* — matches the eventual rename of integrations_home → integrations_connect_home the framework will adopt. DataImportPageView: flat list, one row per IMPORT-capable integration. Per-row CONFIGURE button (always) and DISCARD button (only when imported entities exist for that integration_id with data_source=INTERNAL). Dual-capability rows (HomeBox today) carry a terse "Also available as an Integration" note linking to a bottom-of- page explanation contrasting Data Import vs Integration (Connect). ImporterConfigureView: credentials form (reuses IntegrationAttributeItemEditContext with capability=IMPORT, action button labeled IMPORT). POST validates configuration + access via the same validate_attributes_extra_helper Connect uses, then fetches candidates via Importer.get_candidate_items(), computes new vs. skipped counts by integration_name match against existing HI entities, and renders the preview modal. Preview modal: "Would import N new items. M existing items would be skipped." CONFIRM IMPORT and CANCEL actions. CONFIRM posts to the integrations_import_run URL — stub view returns 501 in this phase; Phase 5 wires the actual run. ImporterRunView and ImporterDiscardView land as 501-returning stubs so the URL reversals from this phase's templates resolve cleanly; Phase 5 and 6 fill in the implementations. view_mixins.py moved from connector/ up to umbrella root so both Connect and Import views can extend IntegrationViewMixin. The render_sync_result method on it is only called by Connect views; Import views never touch it. No import-time Connect coupling — the synchronizer.sync() call inside render_sync_result is runtime-only. * Phase 5(a) (#358): Confirm → Run → Result modal + placement reuse Wires the IMPORT workflow end-to-end. After the preview modal's CONFIRM IMPORT, the importer runs, the result modal renders with imported / skipped counts, and a "Place N new items" CTA hands off to the existing placement flow when entities were created. Importer base gains group_entities_for_placement (default ungrouped, mirrors IntegrationSynchronizer's same-named method). HomeBoxImporter stays with the default — HomeBox has no inherent grouping at v1 (Labels and Locations are metadata-on-entity). HomeBoxImporter._run_import_locked now tracks created entities and populates result.placement_input via group_entities_for_placement, mirroring HomeBoxSynchronizer's _sync_impl pattern. ImporterRunView.post: - Runs importer.run_import() inside a try/finally that invalidates the metadata + sensor-response caches afterward (mirrors Connect's render_sync_result). - Builds the placement URL via the existing PlacementUrlParams. - Reuses the existing integrations_placement URL — HomeBox is dual-capability so its synchronizer is available for the placement view's group_entities_for_placement call. import_result.html: counts hero (Imported / Skipped), per-item list, errors. "Place N new items" CTA on success; OK button on the no-imports / errors paths. Modeled on sync_result.html but trimmed for Import's narrower outcome shape. Manual verification: end-to-end Import works after the manager picks up newly-saved credentials. Phase 5(b) addresses the is_enabled / manager-reload race that surfaces this timing issue and the symmetric one in the Connect-enable flow. * Phase 5(b) (#358): decouple Connect-mode is_enabled from credential loading The HomeBoxManager (shared/) was checking the Connect-mode is_enabled flag inside _load_attributes and raising IntegrationDisabledError to nullify hb_client. That coupling produced two related bugs: 1. Import workflow: after the user disables HomeBox-Connect, HomeBoxImporter.get_candidate_items found hb_client=None and returned empty list — preview showed 0 items even though the upstream simulator had 25. 2. Connect-enable race: after IntegrationEnableView flipped is_enabled=True via enable_integration, the manager's post_save signal was deferred by ~0.1s (DelayedSignalProcessor). The synchronous synchronizer.sync() that immediately followed saw hb_client=None and failed with "integration is disabled" even though the user had just enabled it. Fix: * Strip the is_enabled gate out of HomeBoxManager._load_attributes (and the matching except IntegrationDisabledError branch from _reload_implementation). The Connect-mode enable lifecycle is not a shared/ concern — credentials load whenever they are structurally valid so the IMPORT capability works regardless of Connect-mode state. Connect-side guards (manage view, disable, pause, resume) check integration.is_enabled directly at the view layer, where the capability context is known. * IntegrationManager.enable_integration: synchronously call gateway.notify_settings_changed() after the save+refresh, before the monitor launch. Closes the delayed-signal race so the synchronizer immediately sees fresh credentials. * ImporterConfigureView.post: same synchronous notify after post_attribute_form succeeds, before invoking the importer. Credentials just changed; the manager needs to pick them up before run_import. HASS, ZM, Frigate managers are Connect-only and are left alone — their is_enabled gate is fine because they have no other capability that needs to bypass it. Manual verification: Import workflow shows correct candidate count with Connect disabled; Connect-enable runs sync successfully on the first click. * Phase 6 (#358): Discard flow Wires the DISCARD button on the Data Import page. GET renders a confirmation modal showing the count of imported items; POST runs the importer's discard_imported_data and redirects back to the Data Import page. Single-action confirm (no SAFE/ALL variants). Imported items ARE the user data — the Connect-side preserve-with-user-data semantic doesn't apply. Defensive scoping is in HomeBoxImporter.discard_imported_data (Phase 3): the deletion query filters by data_source=INTERNAL, so any coexisting Connect-mode (EXTERNAL) entities for the same integration_id stay untouched. Confirmed via test. Post-discard cache invalidations mirror the Connect-side Disable cleanup: IntegrationMetadataCache + SensorResponseManager. Sensor- response cache invalidation is symmetric with sync's post-write behavior even though imported items don't have sensors today — keeps the discard path defensive for future Importers that might attach sensors. * Clarify EntityDataSource semantics: every Connect-mode entity is EXTERNAL The Phase 4 (#355) migration used `allow_internal_attributes=False` as the proxy for EXTERNAL, which conflated two separate concerns: "data is sourced from upstream" and "user can edit HI-side attributes." Under the conflation, HA/ZM/Frigate-Connect entities were marked INTERNAL even though their state is sourced from upstream — only HomeBox-Connect (allow_internal_attributes=False) got EXTERNAL. The clarified semantics: data_source=EXTERNAL means data is actively being sourced from an external upstream system, which is true of EVERY Connect-mode integration regardless of whether HI permits user-edited attributes on top. data_source=INTERNAL means no live upstream link — Import-mode (HI-owned post-import) and native entities. | integration | mode | data_source | |-------------------|---------|-------------| | zm/frigate/ha | connect | EXTERNAL | | homebox | connect | EXTERNAL | | homebox | import | INTERNAL | | (native) | n/a | INTERNAL | Changes: * Migration 0021_entity_data_source.py: backfill broadens to mark every integration-attached entity EXTERNAL. Adjusted in place rather than as a fix-up migration since 0021 hasn't been released outside local environments and has no downstream migrations. * HassConverter, FrigateSynchronizer, ZmSynchronizer (both service and monitor entity creation): explicitly set data_source=EXTERNAL on new Connect-mode entities. Closes the post-#355 gap where the field wasn't being set at creation time for these integrations — the migration backfilled existing rows but new rows fell back to the INTERNAL default. * HomeBoxConnector.get_external_view_data: gates on data_source=EXTERNAL. Import-mode HomeBox entities are HI-owned after import, so the live HomeBox view isn't authoritative for them and would mislead the operator. * test_models.py: migration-backfill test renamed and assertions updated to reflect the broadened predicate (HA-like entity now becomes EXTERNAL too). * test_hb_connector.py: HomeBox entity fixture sets data_source=EXTERNAL so the connector's new gate doesn't suppress the connector's existing test surface. Required for Phase 7 (#358 block modal): the modal queries data_source to distinguish Connect-mode entities (EXTERNAL) from Import-mode entities (INTERNAL) for an integration. The old semantics would have miscategorized future dual-capability HA/ZM/ Frigate. * Phase 7 (#358): block modal for cross-capability mode switching When an operator starts an integration's initial-Configure flow for one capability while existing data from the other capability is still present, surface a block modal that routes them to remediate before proceeding. Connect ↔ Import switches are intentionally made harder — the user navigates explicitly, performs the cleanup themselves, then re-clicks CONFIGURE. No combined transition flow. CapabilityBlockViewMixin (in hi/integrations/view_mixins.py): - Single helper render_capability_block_if_conflict that takes only integration_data + capability_being_initiated. The mixin composes all user-facing copy (title, body clauses, navigation link) from per-capability config dicts so block modals read uniformly across both directions and any future capabilities. - Detects the OTHER capability automatically. Only fires when the integration declares both capabilities; single-capability integrations get None. - Counts entities of the OTHER capability's data_source for the integration; zero means no conflict. Connect side: IntegrationEnableView.get checks the block on the initial-Connect path (is_enabled=False). Re-Configure of an already-enabled integration always proceeds — that's not a new mode being initiated. Import side: ImporterConfigureView.get checks the block when no Import entities yet exist for the integration. Re-import (entities already present) always proceeds — that's the standard incremental- import flow. capability_blocked.html: generic block modal template parameterized by title + body clauses + link. Extends modals/action_2_with_cancel so the inherited CANCEL just dismisses. Single-capability integrations (HA/ZM/Frigate today) never trigger the block: the mixin's "other capability not in capabilities" guard short-circuits before any DB query. * Phase 8 (#358): documentation for the Data Import capability User-facing: * docs/DataImport.md (new) — landing page for the Data Import feature; importing/discarding walkthrough; lists supported integrations. Concept-level prose only; no hardcoded button labels so the doc doesn't drift if UI strings change. * docs/Integrations.md — Integrations vs. Data Import comparison pointing at the new landing page. * docs/integrations/homebox.md — added Data Import section describing the alternative mode and when to choose it. Developer-facing: * docs/dev/integrations/data-import.md (new) — framework orientation for the IMPORT capability. Lists where code lives (importer/, view_mixins.py, services/homebox/importer/), how a gateway opts in, and the key design decisions (per-entity transaction, skip-by-integration_name, shared lock, data_source as capability-state signal). Reference-style; points at the code rather than restating API. * docs/dev/integrations/integration-guidelines.md — Capability Model subsection refreshed: CONNECT and IMPORT now both have concrete pointers; the #355 "reserved for #358" placeholder is gone. * docs/dev/integrations/homebox.md — updated stale module paths to reflect the connector/importer/shared split landed in #355 Phase 3 + #358 Phase 1. Added a short Importer section pointing at the framework doc. * Integration framework cleanup: capability-aligned naming and template structure Rename the Connect/Import vocabulary uniformly across the integration framework now that HomeBox is the dual-capability archetype. Class renames: - IntegrationSynchronizer -> IntegrationConnector (+ Hass/Zm/Frigate/HomeBox subclasses) - Importer -> IntegrationImporter - IntegrationEnableView -> ConnectorConfigureView - HomeBoxConnector (the per-entity external-view resolver) -> HomeBoxExternalViewResolver Gateway method: - IntegrationGateway.get_synchronizer() -> get_connector() File renames (git mv, history preserved): - integration_synchronizer.py -> integration_connector.py - importer.py -> integration_importer.py - homebox/connector/hb_sync.py -> homebox_connector.py - homebox/connector/hb_connector.py -> hb_external_view_resolver.py - corresponding test files URL name renames: - integrations_home -> integrations_connect_home - integrations_enable -> integrations_connect_configure Enum renames (ConfigPageType): - INTEGRATIONS -> INTEGRATIONS_CONNECT - DATA_IMPORT -> INTEGRATIONS_IMPORT View mixin: - IntegrationViewMixin.get_integration_data(integration_id) -> (request, *args, **kwargs) Mirrors EntityViewMixin.get_entity; pulls integration_id from kwargs, raises BadRequest if missing and Http404 if not registered. ~20 call sites adopted the uniform pattern. Template restructure (src/hi/integrations/templates/integrations/): - import/ -> importer/ (avoids Python reserved word, matches sub-package convention) - importer/ now has modals/ subdir matching the pages/ convention - New connector/{modals,pages,panes}/ subtree for Connect-specific templates - Top-level modals/, panes/ now hold only genuinely shared cross-capability templates (capability_blocked, placement flow, attribute-form body) - Editor backup .html~ files removed Misc: - IntegrationImportResult folded into importer/transient_models.py (single-class file with no module-level documentation) * Remove dormant IntegrationManageViewPane seam; rename to ConnectorManageView The per-integration "manage view pane" extension point on IntegrationGateway was dead machinery: all four service subclasses returned templates containing only a {% comment %} block. The seam also pre-dated the capability model — its return type lives under connector/, but the hook was on the capability-agnostic gateway. Rather than relocate the hook to IntegrationConnector, delete the seam entirely (base class, 4 subclasses, 4 stub templates, gateway method, view context plumbing, template include). It can be reintroduced correctly on IntegrationConnector when an integration actually needs per-integration content in the manage page. Also rename the view to reflect that the Manage page is Connect- specific: - IntegrationManageView -> ConnectorManageView - integrations_manage URL name -> integrations_connect_manage * Push Connect-specific hooks from IntegrationGateway to IntegrationConnector IntegrationGateway accumulated eight Connect-mode hooks over time, all returning types that already lived under connector/. Now that the capability model has IntegrationConnector and IntegrationImporter as peer capability classes (cf. the recent IntegrationManageViewPane cleanup), these hooks belong on the connector — not on the capability-neutral gateway. Import-only integrations should not be forced to stub them out. Methods moved from IntegrationGateway to IntegrationConnector: - get_monitor() - get_controller() - get_health_status_provider() - get_entity_video_stream() - get_entity_video_snapshot() - get_sensor_response_video_stream() - get_sensor_response_event_snapshot_url() - get_external_view_data() Per-service implementations relocated from each *Gateway (integration.py) to the corresponding *Connector. Production call sites updated to chain through get_connector() with None-guards where the caller might encounter an Import-only or connector-less integration. The three Connect-side view sites trust the structural invariant. The do_control path raises a clear RuntimeError if no connector or controller is available. File renames to match the relocated class names: - frigate_sync.py -> frigate_connector.py (FrigateConnector) - hass_sync.py -> hass_connector.py (HassConnector) - zm_sync.py -> zm_connector.py (ZmConnector) - corresponding test files IntegrationGateway's surface is now its capability-neutral core: get_metadata, notify_settings_changed, validate_configuration, validate_access, get_connector, get_importer. * Fix re-Configure credentials race; remove redundant block-check pre-gates Two related issues in the Configure flows: 1. ConnectorConfigureView.post relied on enable_integration to fire notify_settings_changed(), but enable_integration only fires it on the disabled->enabled transition. Re-Configure of an already-enabled integration left the singleton manager holding stale credentials, and the subsequent sync ran against them. The post_save signal would eventually nudge via DelayedSignalProcessor, but the 0.1s delay races the immediate sync. Lift the synchronous notify out of enable_integration and into the view, right after post_attribute_form, so both initial-Connect and re-Configure get the nudge. The matching ImporterConfigureView.post already did this; the Connect side was the asymmetric outlier. 2. Both Configure views had a pre-gate that suppressed render_capability_block_if_conflict in certain re-entry paths (is_enabled=True for Connect, has_imported=True for Import). Under the mode-switch invariant the pre-gate is redundant — the block check itself short-circuits to None when there's no conflicting data. The pre-gate's only effect was to silently bypass the block check when the invariant was violated (mixed INTERNAL+EXTERNAL state), letting the user compound the corruption. Remove both pre-gates so the block fires in any mixed-state scenario and the user is told to remediate. Two tests deleted that asserted the now-removed bypass behavior: - test_get_import_not_blocked_when_already_imported - test_already_enabled_does_not_trigger_block The canonical block-fires-on-conflict and re-Configure-works paths remain covered by neighboring tests. * Extract CapabilityConfigureView shared base for both Configure modals ConnectorConfigureView and ImporterConfigureView shared a 5-step GET recipe (block check → ensure_all_attributes_exist → build edit context → render modal) and a 3-step POST recipe (post_attribute_form → error short-circuit → success handler). Pull the shared structure into a new framework-level CapabilityConfigureView in hi/integrations/views.py. Subclasses now declare four class constants (capability, button_label, template_name, error_title) and override handle_post_success() with the per-capability behavior. ConnectorConfigureView.handle_post_success: enable_integration, then sync or redirect-to-manage. ImporterConfigureView.handle_post_success: fetch candidates, compute new-vs-skipped, render preview. Each subclass owns the timing of notify_settings_changed() because the right moment is capability-specific: * Connect-side managers (Frigate/Hass/ZM) gate client (re)build on integration.is_enabled, so the notify must fire AFTER enable_integration — otherwise the manager reloads with is_enabled=False, nulls its client, and the immediately-following sync sees no client. * Import flows fire the notify before reading candidates. The unconditional call in both subclasses also covers the re-Configure race (already-enabled integration), where enable_integration early-returns without nudging and the singleton manager would otherwise keep stale credentials. * Flatten homebox/shared/ up one level The capability-layout convention is: inter-capability files live at the integration's top level (alongside integration.py / apps.py / enums.py), while capability-specific files live under connector/ or importer/. HomeBox's shared/ predated the convention; it's the last holdout. Move the six modules up: homebox/shared/hb_client.py -> homebox/hb_client.py homebox/shared/hb_client_factory.py -> homebox/hb_client_factory.py homebox/shared/hb_converter.py -> homebox/hb_converter.py homebox/shared/hb_entity_factory.py -> homebox/hb_entity_factory.py homebox/shared/hb_manager.py -> homebox/hb_manager.py homebox/shared/hb_models.py -> homebox/hb_models.py shared/__init__.py removed; shared/ directory gone. 21 absolute import-site rewrites plus 2 @patch strings updated. Docs and simulator docstring path references brought in line. * Data Import UI/UX refinements Tab order - Move Data Import to the right of Triggers in the Config tabs (matches natural read order: settings -> integrations -> trigger setup -> data import -> system info). Import preview modal (HomeBox confirm) - Title becomes "<Label> Import" when there's nothing to confirm (instead of the awkward "Import from HomeBox?" question). - Body now handles the no-new-items cases explicitly: "All N upstream items have already been imported." or "No items found to import." instead of "Would import 0 new items." - Single centered DONE button replaces the lone left-aligned CANCEL when the preview is informational only. - Add data-async="modal" to the CONFIRM IMPORT form so it submits through the modal pipeline like every other action on the page. - Show a contextual "Currently N items imported from <Label>." line whenever the integration has prior imports — signals re-import vs. first-time state. View filters the count by data_source=INTERNAL. Data Import page layout - Replace sparse rows with a responsive Bootstrap card grid (col-12 col-sm-6 col-md-4 col-lg-3), 4 columns on large screens. Logo centered above the integration name, action buttons stacked at the card bottom (CONFIGURE always; DISCARD only when has_imported). - New hi-integration-logo--card modifier (72px) for the card-prominent size. - Drop btn-control width:100% from the action buttons (the row layout no longer benefits from it; btn-block within the card stack is what fills card width now). - Remove the always-below-the-fold "Data Import vs. Integration" footer block. - Add DataImportInfoView (HiModalView) at integrations_import_info rendering a static info modal with the same explanation content. - On dual-capability rows, the entire "Also available as an Integration. <info-icon>" message is now a single clickable link that opens the info modal. Capability-blocked modal - Title was getting truncated ("Cannot configure <Label> as <Capability>" doesn't fit a narrow modal). Title is now just the integration label; the "Cannot configure as <Capability>." line moves into the body as an h5 sub-heading above the existing explanatory paragraph. View context drops the pre-composed title key and passes my_label so the template composes the sub-heading. * Tighten data_source scoping at mode-switch invariant seams The mode-switch invariant (a given integration's entities are all EXTERNAL for Connect, or all INTERNAL for Import, never mixed) was enforced at the Configure entry point but several downstream code paths filtered only on integration_id, leaving data-source-blind seams. The same code review also surfaced that the disconnect/ reconnect path mishandles data_source when transitioning between attached/detached states. POST block-check (CapabilityConfigureView.post) - GET ran the block check; POST didn't. A direct POST (cached form, replayed request) would bypass the invariant. POST now re-invokes render_capability_block_if_conflict before delegating to the capability-specific handle_post_success. disable_integration seed (IntegrationManager.disable_integration) - Seed query now filters by data_source_str=EXTERNAL. A Connect-side Disable can no longer silently delete imported INTERNAL data if the invariant ever leaks. Native and detached entities (integration_id NULL) were already outside the seed. HomeBoxConnector sync queries - _get_current_integration_keys (sync-check probe) and _get_existing_hb_entities (sync run) both filter by EXTERNAL. INTERNAL imports under the same integration_id no longer risk being adopted as Connect rows and routed through mass-remove or auto-reconnect. HomeBoxImporter preview/run symmetry - _run_import_locked's existing_integration_names set now filters by INTERNAL to match ImporterConfigureView.handle_post_success's preview predicate. A stale EXTERNAL row can no longer turn an advertised new candidate into a skip at run time. Disconnect/reconnect data_source transition - EntityIntegrationOperations.preserve_with_user_data now flips entity.data_source to INTERNAL on detach. A detached entity is no longer constrained by an upstream system — HI owns the editable representation. The reconnect path is already correct: all four service converters set data_source=EXTERNAL during the rebuild invoked by _rebuild_integration_components. Pre-migration detached rows are also correct: migration 0021's column default of 'internal' applied to rows with no integration_id. Test fixtures - Four entity-creation sites in test_integration_manager.py and test_homebox_connector.py now explicitly set data_source_str=EXTERNAL for integration-attached test entities, since the new EXTERNAL filters no longer match the implicit default. * Move IMPORT to previous_integration_id; introduce named entity-state queries The IMPORT path was overloading integration_id (the live-ownership column) to mean "where this entity was sourced from." That conflation forced data_source_str as a discriminator and made post-import code paths brittle: any query that scoped by integration_id and a data source filter could quietly diverge. Under the model in this commit, the entity's lifecycle state is encoded entirely by which integration_* column is populated: - integration_id set -> EXTERNAL (live Connect) - integration_id NULL, previous set -> imported OR detached (HI-owned, has provenance) - both NULL -> INTERNAL (native) IMPORT now writes to previous_integration_*. integration_id stays NULL for imported rows because no live integration owns them. Manager methods (EntityModelManager) give call sites a vocabulary matching the data-source semantics: - external_for(integration_id) active-Connect rows - imported_for(integration_id) import-side intent - detached_for(integration_id) reconnect-candidate intent - with_integration_provenance(id) any row with this integration's identity in either column imported_for / detached_for share an implementation today (the column shape is the same); each name documents intent at the call site so a future discriminator could differentiate without touching callers. Block check: only IMPORT initiation can collide (it would create new rows; an active-Connect row already exists for the upstream key). CONNECT initiation never needs blocking — sync's reconnect- then-create order adopts any pre-existing provenance entities into the live Connect session. CapabilityBlockViewMixin loses its symmetric dispatch and the _CAPABILITY_* dicts. Post-import placement bug: EntityPlacementService.query_unplaced filtered by integration_id, so imports (integration_id=NULL) were invisible to the placement modal — the user saw "No items found" after a successful import. Switched to with_integration_provenance so both EXTERNAL and imported rows are eligible. Other writes / queries swept to the new managers: - HbEntityFactory.create_models_for_hb_item IMPORT branch - HomeBoxConnector._get_existing_hb_entities / _get_current_integration_keys - IntegrationManager.disable_integration seed - EntityIntegrationOperations.find_reconnect_candidates / get_removal_entity_ids - DataImportPageView.has_imported per-row check - ImporterConfigureView preview's skip-detection - HomeBoxImporter._run_import_locked / discard_imported_data - ImporterDiscardView count No migration: IMPORT has never shipped in production; the data_source_str column stays for now and will be retired in the follow-up commit that introduces the derived data_source property. One test deleted (test_initial_connect_blocked_by_existing_import_data) because CONNECT-init no longer blocks under the unified model; the canonical IMPORT-init block direction is still covered by test_get_import_blocked_by_existing_connect_data. * Derive data_source from columns; drop EntityDataSource and data_source_str The data_source_str column was a stored discriminator that became fully redundant once IMPORT moved to previous_integration_id: an entity's state is now encoded entirely by which integration_* columns are populated. Removing the column eliminates a class of "stored flag drifted from column reality" bugs and gives the codebase one fewer thing to keep consistent. Model: - Drop the data_source_str field from Entity and the EntityDataSource enum. - Delete migration 0021_entity_data_source (the migration that introduced the column). Import has never shipped, so there is no production data dependency. Dev environments drop the column manually. - Add three derived properties on Entity that name the questions call sites actually ask: * is_external — actively attached to an integration * has_integration_provenance — carries record of an integration it came from (imported or detached) * is_imported / is_detached — synonyms today (same column shape); separate names document intent at the call site, parallel to the manager methods external_for / imported_for / detached_for. Sweep: - Remove every entity.data_source = ... write site. The four service converters and HbEntityFactory no longer need to set it — the columns ARE the state. preserve_with_user_data likewise. - Replace the one production read site in hb_external_view_resolver with `if not entity.is_external`. - Test fixtures: drop data_source_str=... kwargs from Entity.objects.create calls (the column is gone). Two legacy "integration_id + data_source_str=INTERNAL" mixes converted to provenance form (previous_integration_id + previous_integration_name). - Replace data_source assertions with property checks. UI rename: "Detached from X" -> "From X". The unified model treats imported and detached as the same state, so the badge text needs to read accurately for both origin stories. Two templates plus comment references. Sync-result reporting fixes (surfaced during manual testing of the above): - ZM _create_zm_entity now appends the service entity to created_list (was hidden in info_list). The user-facing "created" count and detail now match what got persisted. Placement input remains unaffected — _sync_impl builds it from monitor entities only. - ZM _create_monitor_entity and Frigate _create_camera_entity gate created_list.append on is_fresh_create. Auto-reconnect rebuilds no longer double-list the row in both created_list and reconnected_list (HomeBox/HASS did not have this bug; their _rebuild_integration_components calls the converter directly, bypassing the per-connector _create_entity). - Both result-modal placement CTAs now read "Place new items" without a count. The previous count came from created_list length, which includes reconnected entities and (for ZM) the service entity — but the placement modal correctly excludes those. Action prompt > misleading count.
65 lines
2.9 KiB
Markdown
65 lines
2.9 KiB
Markdown
# Data Import (IMPORT capability)
|
|
|
|
The IMPORT capability copies upstream items into HI as locally-owned
|
|
entities. Once imported, the integration has no ongoing relationship
|
|
with the upstream; HI is the source of truth.
|
|
|
|
User-facing setup lives in
|
|
[`docs/DataImport.md`](../../DataImport.md). This page is the
|
|
developer orientation — read the linked modules for the
|
|
authoritative API.
|
|
|
|
## Where the code lives
|
|
|
|
- `src/hi/integrations/importer/` — abstract `IntegrationImporter` base,
|
|
transient models (`CandidateItem`, `IntegrationImportResult`,
|
|
`IntegrationDiscardResult`), framework-level views (Data Import
|
|
page, configure, preview, run, discard), templates.
|
|
- `src/hi/integrations/view_mixins.py` — `CapabilityBlockViewMixin`
|
|
for the cross-capability block-modal detection. Mixed into both
|
|
`ConnectorConfigureView` and `ImporterConfigureView`.
|
|
- `src/hi/services/homebox/importer/` — `HomeBoxImporter`, the first
|
|
concrete implementation. Reference example for new IMPORT-capable
|
|
integrations.
|
|
|
|
## How a gateway opts in
|
|
|
|
1. Add `IntegrationCapability.IMPORT` to `IntegrationMetaData.capabilities`.
|
|
2. Override `IntegrationGateway.get_importer()` to return a concrete
|
|
`IntegrationImporter` subclass.
|
|
|
|
The `IntegrationImporter` abstract sits parallel to `IntegrationConnector`,
|
|
not inheriting from it. Commonality between Connect and Import is
|
|
composed through shared helpers (`HbEntityFactory`, `HbConverter`,
|
|
`EntityIntegrationOperations`, `PlacementUrlParams`, etc.).
|
|
|
|
## Key design decisions
|
|
|
|
- **Per-entity transaction during `run_import`.** A single item's
|
|
failure does not abort the batch; errors are aggregated into
|
|
`IntegrationImportResult.error_list`. See HomeBoxImporter for
|
|
the pattern.
|
|
- **Skip-by-`integration_name`.** Imports are add-only. The framework
|
|
view computes new-vs-skipped against existing HI entities by
|
|
matching `integration_name`; `IntegrationImporter.get_candidate_items()`
|
|
itself returns the full upstream list.
|
|
- **Shared `integrations_sync` exclusion lock.** Connect-side sync
|
|
and Import-side run serialize against each other to prevent
|
|
upstream double-fetch races on a single integration.
|
|
- **State is encoded by the integration columns alone.** An entity
|
|
is `is_external` when `integration_id` is set (live Connect),
|
|
`has_integration_provenance` when `previous_integration_id` is
|
|
set (imported or detached), and neither when native. The
|
|
`EntityModelManager` exposes named helpers (`external_for`,
|
|
`imported_for`, `detached_for`, `with_integration_provenance`)
|
|
for query sites; the matching `Entity` properties are
|
|
`is_external`, `is_imported`, `is_detached`, and
|
|
`has_integration_provenance`.
|
|
|
|
## Reference
|
|
|
|
- HomeBox concrete: `src/hi/services/homebox/importer/`
|
|
- Tests: `src/hi/integrations/tests/test_importer*.py` and
|
|
`src/hi/services/homebox/tests/test_homebox_importer.py`.
|
|
- User-facing: [`docs/DataImport.md`](../../DataImport.md).
|