feat: Implement DevicesHistory feature with triggers and history tracking

- Added db_history.py to manage DevicesHistory table and triggers for INSERT and UPDATE operations. - Created device_history_instance.py for querying and grouping DevicesHistory records. - Developed change_history.php for displaying device change history with filtering and pagination. - Introduced skel_device_details_tab_history.php for skeleton loading state in device details tab. - Added unit tests in test_device_history.py to validate trigger functionality and history management. - Implemented filter population and pagination in the change history UI.
2026-07-29 00:48:13 -04:00 · 2026-07-04 23:34:52 +00:00
parent 351b6a7d28
commit 18ec0ce96c
30 changed files with 2578 additions and 42 deletions
--- a/.github/copilot-instructions.md
+++ b/.github/copilot-instructions.md
@@ -27,6 +27,7 @@ Procedural knowledge lives in `.github/skills/`. Load the appropriate skill when

 | Task | Skill |
 |------|-------|
+| Find or cross-reference any skill | `skills-overview` |
 | Run tests, check failures | `testing-workflow` |
 | Start/stop/restart services | `devcontainer-services` |
 | Wipe database, fresh start | `database-reset` |
@@ -42,6 +43,7 @@ Procedural knowledge lives in `.github/skills/`. Load the appropriate skill when
 | Settings and config | `settings-management` |
 | Find files and paths | `project-navigation` |
 | Coding standards | `code-standards` |
+| Devices table write paths, SQLite triggers, audit logging, `*Source` attribution | `database-patterns` |

 ## Execution Protocol

--- a/.github/skills/database-patterns/SKILL.md
+++ b/.github/skills/database-patterns/SKILL.md
@@ -0,0 +1,142 @@
+---
+name: netalertx-database-patterns
+description: NetAlertX database architecture patterns. Use this when designing features that write to the Devices table, implementing audit/history logging, or choosing between trigger-based vs Python-hook approaches.
+---
+
+# Database Patterns
+
+## Devices Table — Write-Path Inventory
+
+Before implementing any feature that reads or writes the `Devices` table, audit ALL write paths. The table is modified from many locations — missing one path is a correctness bug.
+
+**Known production write paths (as of 2026-07-04):**
+
+| File | Function | Fields written |
+|---|---|---|
+| `server/models/device_instance.py` | `setDeviceData()` | All user-editable fields |
+| `server/models/device_instance.py` | `updateField()` | Any single field (workflows) |
+| `server/models/device_instance.py` | `updateDeviceColumn()` | Any single column |
+| `server/models/device_instance.py` | `deleteDevices()` etc. | DELETE operations |
+| `server/scan/device_handling.py` | `update_devices_data_from_scan()` | Scan-derived fields |
+| `server/scan/device_handling.py` | `update_vendors_from_mac()` | `devVendor`, `devVendorSource` |
+| `server/scan/device_handling.py` | Name resolution block | `devName`, `devFQDN`, `*Source` |
+| `server/scan/device_handling.py` | `update_ipv4_ipv6()` | `devPrimaryIPv4`, `devPrimaryIPv6` |
+| `server/scan/device_handling.py` | `update_icons_and_types()` | `devIcon`, `devType` |
+| `server/scan/device_handling.py` | `update_presence_from_CurrentScan()` | `devPresentLastScan` |
+| `server/scan/device_handling.py` | `update_devLastConnection_from_CurrentScan()` | `devLastConnection` |
+| `server/scan/device_handling.py` | `update_devPresentLastScan_based_on_*()` | `devPresentLastScan` |
+| `server/db/authoritative_handler.py` | `enforce_source_on_user_update()` | `*Source` columns |
+| `server/db/authoritative_handler.py` | `lock_field()` / `unlock_field()` | `*Source` columns |
+| `server/models/notification_instance.py` | `clearPendingEmailFlag()` | `devLastNotification` |
+| `front/plugins/db_cleanup/script.py` | `cleanup_database()` | DELETE operations |
+
+**Key insight:** Most scan functions use `sql.executemany()` — there is no per-row Python state available. Python hooks before/after executemany require a pre-fetch+diff pattern that is expensive and error-prone.
+
+---
+
+## `*Source` Fields — Attribution System
+
+The `FIELD_SOURCE_MAP` in `server/db/authoritative_handler.py` defines 10 fields that carry write attribution via paired `*Source` columns:
+
+```python
+FIELD_SOURCE_MAP = {
+    "devMac":           "devMacSource",
+    "devName":          "devNameSource",
+    "devFQDN":          "devFQDNSource",
+    "devLastIP":        "devLastIPSource",
+    "devVendor":        "devVendorSource",
+    "devSSID":          "devSSIDSource",
+    "devParentMAC":     "devParentMACSource",
+    "devParentPort":    "devParentPortSource",
+    "devParentRelType": "devParentRelTypeSource",
+    "devVlan":          "devVlanSource",
+}
+```
+
+`*Source` values: `'USER'`, `'LOCKED'`, `'NEWDEV'`, or a plugin prefix (e.g., `'ARPSCAN'`, `'NSLOOKUP'`).
+
+These fields are updated **in the same transaction** as the primary field. A SQLite `AFTER UPDATE` trigger can read `NEW.devNameSource` to obtain correct attribution without any extra context-passing.
+
+**Attribution rules for features that need `changedBy`:**
+
+| Field category | Attribution |
+|---|---|
+| In `FIELD_SOURCE_MAP` | `COALESCE(NULLIF(NEW.<field>Source, ''), 'system')` |
+| User-only fields (`devGroup`, `devComments`, `devFavorite`, `devOwner`, `devLocation`, etc.) | `'user:api'` — only `setDeviceData()` writes these |
+| Auto-computed fields (`devIcon`, `devType`, `devPrimaryIPv4`, `devPrimaryIPv6`) | `'system'` |
+| `*Source` fields themselves | `'system'` |
+
+---
+
+## Cross-Cutting Concerns — Prefer SQLite Triggers Over Python Hooks
+
+When a feature needs to intercept **every write** to the `Devices` table (audit logging, computed columns, cascading logic), prefer a **SQLite `AFTER UPDATE` / `AFTER INSERT` trigger** over Python-layer hooks.
+
+**Why:**
+- Triggers catch all 14+ write paths automatically, including `executemany()` bulk updates
+- Zero modifications to existing write-path functions (DRY)
+- Self-healing: future write paths are automatically covered
+- Attribution is available via `NEW.*Source` fields (see above)
+
+**When Python hooks are still appropriate:**
+- The logic needs access to Python objects, settings, or services not available in SQL
+- The feature only fires from one or two known write paths
+- The logic is too complex to express in SQL (multi-table joins with app-layer business logic)
+
+### Trigger Performance Pattern
+
+```sql
+CREATE TRIGGER trg_example
+AFTER UPDATE ON Devices
+FOR EACH ROW
+-- Guard: short-circuit entire body when feature is disabled (zero cost)
+WHEN (SELECT CAST(setValue AS INTEGER) FROM Settings WHERE setKey = 'FEATURE_ENABLED') > 0
+BEGIN
+  -- Per-field conditional insert
+  INSERT INTO SomeTable (devGUID, column, oldVal, newVal, changedBy, ts)
+  SELECT NEW.devGUID, 'devName', OLD.devName, NEW.devName,
+         COALESCE(NULLIF(NEW.devNameSource, ''), 'system'),
+         datetime('now', 'utc')
+  WHERE OLD.devName IS NOT NEW.devName
+    AND instr(',' || (SELECT setValue FROM Settings WHERE setKey = 'TRACKED_FIELDS') || ',', ',devName,') > 0;
+  -- Repeat for each tracked field...
+END;
+```
+
+**Performance:** The Settings table is tiny (~100 rows) and stays in SQLite's page cache. Per-row Settings reads inside triggers are effectively in-memory lookups. The `WHEN` guard makes the disabled state zero-cost.
+
+---
+
+## Snapshot vs Event-Sourced Audit Logging
+
+When implementing change history, always use **event-sourced (per-field rows)** not **snapshots (full row copies)**.
+
+| | Event-sourced | Snapshot |
+|---|---|---|
+| Storage | Small — only changed fields | Large — all 40+ columns every mutation |
+| Filter by field | O(log n) via index | O(n) — must diff every adjacent pair |
+| Filter by source | O(log n) via index | Not possible without diffing |
+| `changedBy` attribution | Embedded at write time | Not available without extra context |
+| Retention calculation | Simple timestamp DELETE | Same, but much higher storage |
+
+At 1000 devices, 5-min scan interval, 14-day retention: snapshot storage ≈ 280 MB/day. Event-sourced storage for the same workload is typically <1 MB/day (most scans produce no tracked field changes).
+
+---
+
+## `DevicesHistory` Table — Reference Schema
+
+```sql
+CREATE TABLE IF NOT EXISTS DevicesHistory (
+    id           INTEGER PRIMARY KEY AUTOINCREMENT,
+    devGUID      TEXT NOT NULL,
+    timestamp    DATETIME DEFAULT CURRENT_TIMESTAMP,
+    changedBy    TEXT NOT NULL,
+    changedColumn TEXT NOT NULL,
+    oldValue     TEXT,
+    newValue     TEXT,
+    FOREIGN KEY (devGUID) REFERENCES Devices(devGUID) ON DELETE CASCADE
+);
+
+CREATE INDEX IF NOT EXISTS idx_devhist_guid_column ON DevicesHistory(devGUID, changedColumn);
+CREATE INDEX IF NOT EXISTS idx_devhist_timestamp   ON DevicesHistory(timestamp);
+```
--- a/.github/skills/mcp-activation/SKILL.md
+++ b/.github/skills/mcp-activation/SKILL.md
@@ -7,6 +7,11 @@ description: Enables live interaction with the NetAlertX runtime. This skill con

 This skill configures the environment to expose the Model Context Protocol (MCP) server to AI agents running inside the devcontainer.

+## Prerequisites
+
+1. **Devcontainer connected:** You must be working inside the NetAlertX devcontainer.
+2. **Backend running:** The Python backend must be running — it generates `app.conf` containing the API token. Use the `devcontainer-services` skill if it's not running.
+
 ## Usage

 This skill assumes you are already running within the NetAlertX devcontainer. 
--- a/.github/skills/settings-management/SKILL.md
+++ b/.github/skills/settings-management/SKILL.md
@@ -5,35 +5,175 @@ description: Manage NetAlertX configuration settings. Use this when asked to add

 # Settings Management

-## Reading Settings
+## Quick Reference

+**Read a setting (backend):**
 ```python
 from helper import get_setting_value
-
 value = get_setting_value('SETTING_NAME')
 ```
-
 Never hardcode ports, secrets, or configuration values. Always use `get_setting_value()`.

-## Adding Core Settings
-
-Use `ccd()` in `server/initialise.py`:
+**Read a setting (frontend):**
+```javascript
+getSetting("SETTING_NAME")
+```

+**Add a core setting** — use `ccd()` in `server/initialise.py`:
 ```python
 ccd('SETTING_NAME', 'default_value', 'description')
 ```

-## Adding Plugin Settings
-
-Define in plugin's `config.json` manifest under the settings section.
-
-## Config Files
+**Add a plugin setting** — define in the plugin's `config.json` under the `settings` key.

 | File | Purpose |
 |------|---------|
-| `/data/config/app.conf` | Runtime config (modified by app) |
+| `/data/config/app.conf` | Runtime config (source of truth, modified by app) |
 | `back/app.conf` | Default config (template) |

-## Environment Override
+Use `APP_CONF_OVERRIDE` for settings that must be set before startup.

-Use `APP_CONF_OVERRIDE` environment variable for settings that must be set before startup.
+---
+
+# Development Guide
+
+The goal is for every setting to be:
+
+- Declarative and self-documenting
+- Automatically rendered by the Settings UI
+- Fully localized
+- Backwards compatible
+- Easy to maintain
+
+---
+
+## 1. Follow existing patterns
+
+Before creating a new setting, search for similar ones. Reuse existing naming conventions, types, and validation patterns. Follow the style already used by similar plugins. Avoid introducing new setting types unless absolutely necessary.
+
+---
+
+## 2. Setting naming
+
+Setting keys use uppercase snake case: `UI_DEV_SECTIONS`, `SCAN_INTERVAL`, `MQTT_HOST`.
+Names should clearly describe the purpose.
+
+---
+
+## 3. Categories
+
+Place settings into the most appropriate category. Keep related settings together. Avoid creating new categories unless there is a clear need.
+
+---
+
+## 4. Choose the correct type
+
+Use the simplest type that correctly models the value: `string`, `integer`, `float`, `boolean`, `password`, `select`, `array`, `textarea`. Avoid encoding structured JSON inside string settings.
+
+---
+
+## 5. Validation
+
+Always define validation whenever appropriate (min/max values, regex, allowed options). Reject invalid configuration rather than silently accepting it.
+
+---
+
+## 6. Defaults
+
+Provide sensible defaults that work for a fresh installation. The `default_value` in `config.json` is the single source of truth — do not duplicate it in Python or JavaScript.
+
+---
+
+## 7. Names and descriptions
+
+Names should be concise. Descriptions should explain what the setting does, when to use it, and any important side effects — not implementation details.
+
+---
+
+## 8. Boolean settings
+
+Prefer verbs: Enable, Disable, Require, Allow, Ignore, Show, Hide. Examples: *Enable MQTT*, *Require NICs Online*, *Hide Offline Devices*.
+
+---
+
+## 9. Select settings
+
+Use when users must choose from predefined values. Each option must have a meaningful label — never require users to remember internal values.
+
+---
+
+## 10. Array settings
+
+Use only when multiple independent values are expected (e.g., ignored MACs, subnet lists, plugin lists).
+
+---
+
+## 11. Localization
+
+Every setting must have localized language strings. If not defined directly in `config.json`, they must exist in `en_us.json`.
+
+Example for `UI_DEV_SECTIONS`:
+```json
+"UI_DEV_SECTIONS_name": "Hide device sections",
+"UI_DEV_SECTIONS_description": "Select which UI elements to hide on the Devices page."
+```
+
+When using external language files, `config.json` must reference:
+```json
+"name": [{ "string": "_GLOBAL_LANG_FILES_" }]
+```
+
+---
+
+## 12. Source of truth
+
+`app.conf` is the source of truth. User-defined values always come from `app.conf`; defaults from `config.json` apply only when a setting is absent. Never read `app.conf` directly — always go through `get_setting_value()`.
+
+---
+
+## 13. Plugin-first settings
+
+Whenever possible, define new settings inside a plugin's `config.json`. Avoid adding hardcoded application settings unless there is a compelling architectural reason.
+
+---
+
+## 14. Runtime lifecycle
+
+```text
+config.json (defaults)
+    → app.conf (source of truth)
+    → Settings database table
+    → table_settings.json API
+    → Frontend getSetting() / Backend get_setting_value()
+```
+
+The database and API are runtime representations only, regenerated from `app.conf` at init.
+
+---
+
+## 15. Backwards compatibility
+
+Avoid renaming settings, changing data types, or changing semantics. If unavoidable, provide migration logic and preserve compatibility where possible.
+
+---
+
+## 16. Pull request checklist
+
+- [ ] Follows existing naming conventions (uppercase snake case)
+- [ ] Correct category used
+- [ ] Appropriate type selected, validation defined
+- [ ] Sensible default — defined only once (in `config.json`)
+- [ ] Name concise, description clear (no implementation details)
+- [ ] Localization strings added; `_GLOBAL_LANG_FILES_` used where appropriate
+- [ ] Plugin-first approach followed where applicable
+- [ ] Frontend renders automatically without custom JavaScript
+- [ ] Backend uses `get_setting_value()` — no direct `app.conf` reads
+- [ ] Backwards compatibility maintained
+- [ ] Documentation updated if required
+
+---
+
+## References
+
+- https://docs.netalertx.com/PLUGINS_DEV_SETTINGS/
+- https://docs.netalertx.com/SETTINGS_SYSTEM/
--- a/.github/skills/skills-overview/SKILL.md
+++ b/.github/skills/skills-overview/SKILL.md
@@ -0,0 +1,64 @@
+---
+name: skills-overview
+description: Index of all available skills across both GitHub Copilot (.github/skills/) and Gemini CLI (.gemini/skills/). Load this to find the right skill for a task, or to locate the counterpart skill in the other AI system.
+---
+
+# Skills Index — Cross-Reference
+
+Two AI assistants are configured for this project, each with their own skill directory:
+
+- **GitHub Copilot** → `.github/skills/`
+- **Gemini CLI** → `.gemini/skills/`
+
+Skills with the same purpose exist in both, sometimes under different names and with different depth. This index maps them so you can find the richer version when needed.
+
+---
+
+## Shared Skills (exist in both)
+
+| Topic | Copilot Skill | Gemini Skill | Notes |
+|-------|--------------|--------------|-------|
+| Testing | `testing-workflow` | `testing-workflow` | Copilot version covers `testFailure` tool and PYTHONPATH; Gemini version emphasises full suite preference and container detection |
+| Settings & config | `settings-management` | `settings` | Gemini version is more comprehensive (22-point guide + PR checklist); Copilot version covers `ccd()` and `get_setting_value()` usage |
+| MCP activation | `mcp-activation` | `mcp-activation` | Copilot version covers VS Code window reload; Gemini version covers Gemini CLI session restart |
+| Project navigation | `project-navigation` | `project-navigation` | Copilot version has full path tables and env vars; Gemini version is a brief reference |
+| Plugin dev | `plugin-run-development` | `plugin-development` | Copilot version is comprehensive (data contract, phases, formats); Gemini version is a brief checklist pointing to `docs/PLUGINS_DEV.md` |
+| Devcontainer | `devcontainer-services` + `devcontainer-setup` + `devcontainer-configs` | `devcontainer-management` | Copilot splits into 3 focused skills; Gemini combines into one (uses `docker exec`) |
+
+---
+
+## Copilot-Only Skills
+
+No Gemini equivalent yet:
+
+| Skill | Purpose |
+|-------|---------|
+| `api-development` | Creating REST API endpoints |
+| `authentication` | API tokens and 401/403 debugging |
+| `code-standards` | Coding conventions and style rules |
+| `database-patterns` | Device table write paths, SQLite triggers, audit logging, `*Source` attribution |
+| `database-reset` | Wipe and regenerate the database and config |
+| `docker-build` | Build Docker images for testing or production |
+| `docker-prune` | Clean unused Docker resources (destructive — requires confirmation) |
+| `sample-data` | Load synthetic device data into the devcontainer |
+
+---
+
+## Gemini-Only Skills
+
+No Copilot equivalent yet:
+
+| Skill | Purpose |
+|-------|---------|
+| `initiative-start` | Research methodology and structured approach for new tasks |
+
+---
+
+## Adding a New Skill
+
+When adding a skill, create it in **both** directories to keep both AI systems current:
+
+- `.github/skills/<name>/SKILL.md` — add an entry to the skills table in `.github/copilot-instructions.md`
+- `.gemini/skills/<name>/SKILL.md` — auto-discovered by Gemini CLI via YAML frontmatter
+
+Keep the body content identical between both files. Only the frontmatter `name`/`description` may differ slightly to match each system's discovery heuristics.