mirror of
https://github.com/twentyhq/twenty.git
synced 2026-06-11 17:37:18 -04:00
91f2f08995
## Why The five event-log streams (`workspaceEvent`, `pageview`, `objectEvent`, `usageEvent`, `applicationLog`) each wrote to ClickHouse through their own fire-and-forget writer (`AuditService`, `UsageEventWriterService`, and the `application-logs` driver), with the per-type knowledge (table names, normalization, access rules) spread across several modules. Three of them reimplemented the same ClickHouse insert, and the read side, the live stream, and the producers lived in different modules under two different names. This consolidates them into one `core-modules/event-logs/` subsystem (emit, write, live, read), with the per-type config in a single registry so adding an event type is roughly one file. The base Logs settings tab and free application logs shipped separately in #21180 (merged). This PR adds the unified backend, the registry, and the viewer's live mode and entitlement gating. ## Pipeline ```mermaid flowchart TB subgraph PROD["Producers"] A["auth, billing, impersonation,<br/>webhook, custom-domain"] U["usage listener"] F["logic-function executor (app logs)"] R["record CRUD (entity events)"] end EM["EventLogEmitterService<br/>createContext().insert* / dispatch()"] EQ(["entityEventsToDbQueue<br/>(existing, shared with timeline)"]) CIE["CreateEventLogFromInternalEvent"] SINK["WorkspaceEventSinkService.ingest()"] C1["ClickHouseEventSink"] C2["ConsoleEventSink"] LIVE["EventLogLiveService.publishWatched()<br/>(presence-gated)"] CH[("ClickHouse, 5 tables, async_insert")] CHAN(["WORKSPACE_EVENTS_CHANNEL"]) RS["EventLogsService (registry-driven read)"] LR["EventLogsLiveResolver"] UI["Settings > Logs"] A --> EM U --> EM F --> EM EM -->|direct| SINK R --> EQ --> CIE -->|ingest| SINK SINK --> C1 --> CH SINK --> C2 SINK --> LIVE -.->|if a viewer is watching| CHAN --> LR --> UI CH --> RS --> UI ``` ## What it does - Producers call `EventLogEmitterService.createContext().insert*()`, which builds a typed `WorkspaceEventEnvelope` and writes it through `WorkspaceEventSinkService` to the configured sinks (ClickHouse, Console) plus a presence-gated live fan-out. Record/CRUD events reach the same sink through the existing `entityEventsToDbQueue`. There is no dedicated queue; ClickHouse `async_insert` batches server-side. Writes are best-effort, as on main today. - `EVENT_LOG_TYPES[table]` is the per-type source of truth: the ClickHouse table, the required entitlement, the free-text filter column, and the row-to-GraphQL mapping. Read row shapes derive from the write rows. - Four modules along their dependency boundaries: `EventLogEmitterModule` (producer API), `EventLogIngestionModule` (sink layer), `EventLogLiveModule` (fan-out), and `EventLogsViewerModule` (the entitlement-gated GraphQL read, which is where billing/enterprise/permissions stay so producers stay light). - Logs viewer: per-table columns, filters (text, date, record), live mode, and an upgrade card that points to Billing on Cloud or the Admin Panel on self-hosted. Application logs are free on every plan; the other four require the `AUDIT_LOGS` entitlement (with a `NO_ENTITLEMENT` fallback to the upgrade card). - Renames `AuditService` to `EventLogEmitterService`, and the generic `Monitoring` event to a typed `Impersonation` event (`level` + `action`). - Removes `UsageEventWriterService`, the `application-logs` driver/module, and `AuditService`'s direct inserts. ## Durability Writes are best-effort, the same as main today (the old writers were fire-and-forget). A dedicated queue was tried mid-PR and removed: `async_insert` already batches server-side, so the queue only added durability, which isn't a requirement right now. The `EventSink` seam keeps a durable transport (e.g. a Redis-Streams buffer) easy to add later without touching producers. ## Out of scope S3 peer sink (seam only), Postgres or any second read path, `ReplicatedMergeTree`, ClickHouse table-schema changes, and the record-data `EVENT_STREAM_CHANNEL` (unchanged, separate concern). ## Testing Unit tests cover the registry definitions and row normalization, the entitlement gating, the envelope builders, and the producers. Integration tests cover the write paths (record create produces an `objectEvent`; the track mutation produces a `workspaceEvent`) and the read/query path across all five tables. Verified with typecheck, lint, a server boot, and GraphQL/SDK codegen.
103 lines
3.9 KiB
Plaintext
103 lines
3.9 KiB
Plaintext
# Use this for local setup
|
|
NODE_ENV=development
|
|
PG_DATABASE_URL=postgres://postgres:postgres@localhost:5432/default
|
|
REDIS_URL=redis://localhost:6379
|
|
APP_SECRET=replace_me_with_a_random_string
|
|
SIGN_IN_PREFILLED=true
|
|
IS_WORKSPACE_CREATION_LIMITED_TO_SERVER_ADMINS=false
|
|
|
|
FRONTEND_URL=http://localhost:3001
|
|
|
|
# ———————— Optional ————————
|
|
# PORT=3000
|
|
# ACCESS_TOKEN_EXPIRES_IN=30m
|
|
# LOGIN_TOKEN_EXPIRES_IN=15m
|
|
# REFRESH_TOKEN_EXPIRES_IN=90d
|
|
# FILE_TOKEN_EXPIRES_IN=1d
|
|
# MESSAGING_PROVIDER_GMAIL_ENABLED=false
|
|
# IS_IMAP_SMTP_CALDAV_ENABLED=true
|
|
# CALENDAR_PROVIDER_GOOGLE_ENABLED=false
|
|
# MESSAGING_PROVIDER_MICROSOFT_ENABLED=false
|
|
# CALENDAR_PROVIDER_MICROSOFT_ENABLED=false
|
|
# IS_BILLING_ENABLED=false
|
|
# BILLING_PLAN_REQUIRED_LINK=https://twenty.com/stripe-redirection
|
|
# AUTH_PASSWORD_ENABLED=false
|
|
# IS_MULTIWORKSPACE_ENABLED=false
|
|
# AUTH_MICROSOFT_ENABLED=false
|
|
# AUTH_MICROSOFT_CLIENT_ID=replace_me_with_azure_client_id
|
|
# AUTH_MICROSOFT_CLIENT_SECRET=replace_me_with_azure_client_secret
|
|
# AUTH_MICROSOFT_CALLBACK_URL=http://localhost:3000/auth/microsoft/redirect
|
|
# AUTH_MICROSOFT_APIS_CALLBACK_URL=http://localhost:3000/auth/microsoft-apis/get-access-token
|
|
# AUTH_GOOGLE_ENABLED=false
|
|
# AUTH_GOOGLE_CLIENT_ID=replace_me_with_google_client_id
|
|
# AUTH_GOOGLE_CLIENT_SECRET=replace_me_with_google_client_secret
|
|
# AUTH_GOOGLE_CALLBACK_URL=http://localhost:3000/auth/google/redirect
|
|
# AUTH_GOOGLE_APIS_CALLBACK_URL=http://localhost:3000/auth/google-apis/get-access-token
|
|
# CODE_INTERPRETER_TYPE=LOCAL
|
|
# LOGIC_FUNCTION_TYPE=LOCAL
|
|
# STORAGE_TYPE=local
|
|
# STORAGE_LOCAL_PATH=.local-storage
|
|
# SUPPORT_DRIVER=front
|
|
# SUPPORT_FRONT_HMAC_KEY=replace_me_with_front_chat_verification_secret
|
|
# SUPPORT_FRONT_CHAT_ID=replace_me_with_front_chat_id
|
|
# LOGGER_DRIVER=CONSOLE
|
|
# LOGGER_IS_BUFFER_ENABLED=true
|
|
# EXCEPTION_HANDLER_DRIVER=sentry
|
|
# METER_DRIVER=opentelemetry,console
|
|
# SENTRY_ENVIRONMENT=main
|
|
# SENTRY_DSN=https://xxx@xxx.ingest.sentry.io/xxx
|
|
# SENTRY_FRONT_DSN=https://xxx@xxx.ingest.sentry.io/xxx
|
|
# APPLICATION_LOG_DRIVER=CONSOLE
|
|
# LOG_LEVELS=error,warn
|
|
# SERVER_URL=http://localhost:3000
|
|
# WORKSPACE_INACTIVE_DAYS_BEFORE_NOTIFICATION=7
|
|
# WORKSPACE_INACTIVE_DAYS_BEFORE_SOFT_DELETION=14
|
|
# WORKSPACE_INACTIVE_DAYS_BEFORE_DELETION=21
|
|
# Email Server Settings, see this doc for more info: https://docs.twenty.com/start/self-hosting/#email
|
|
# IS_EMAIL_VERIFICATION_REQUIRED=false
|
|
# EMAIL_VERIFICATION_TOKEN_EXPIRES_IN=1h
|
|
# EMAIL_FROM_ADDRESS=contact@yourdomain.com
|
|
# EMAIL_FROM_NAME='John from YourDomain'
|
|
# EMAIL_DRIVER=LOGGER
|
|
# EMAIL_SMTP_HOST=
|
|
# EMAIL_SMTP_PORT=
|
|
# EMAIL_SMTP_USER=
|
|
# EMAIL_SMTP_PASSWORD=
|
|
# PASSWORD_RESET_TOKEN_EXPIRES_IN=5m
|
|
# CAPTCHA_DRIVER=
|
|
# CAPTCHA_SITE_KEY=
|
|
# CAPTCHA_SECRET_KEY=
|
|
# API_RATE_LIMITING_TTL=
|
|
# API_RATE_LIMITING_LIMIT=
|
|
# MUTATION_MAXIMUM_AFFECTED_RECORDS=100
|
|
# PG_SSL_ALLOW_SELF_SIGNED=true
|
|
# ENTERPRISE_KEY=replace_me_with_a_valid_enterprise_key
|
|
# SSL_KEY_PATH="./certs/your-cert.key"
|
|
# SSL_CERT_PATH="./certs/your-cert.crt"
|
|
# CLOUDFLARE_API_KEY=
|
|
# CLOUDFLARE_ZONE_ID=
|
|
# CLOUDFLARE_WEBHOOK_SECRET=
|
|
# IS_CONFIG_VARIABLES_IN_DB_ENABLED=false
|
|
# ANALYTICS_ENABLED=
|
|
# CLICKHOUSE_URL=http://default:clickhousePassword@localhost:8123/twenty
|
|
# HTTP_TOOL_SAFE_MODE_ENABLED=true
|
|
# ALLOW_REQUESTS_TO_TWENTY_ICONS=true
|
|
|
|
# ———————— AI ————————
|
|
# API keys for built-in providers (also editable from Admin Panel > Config Variables):
|
|
# OPENAI_API_KEY=
|
|
# ANTHROPIC_API_KEY=
|
|
# GOOGLE_API_KEY=
|
|
# XAI_API_KEY=
|
|
# GROQ_API_KEY=
|
|
# MISTRAL_API_KEY=
|
|
#
|
|
# Add custom providers (private gateway, extra regions, etc.):
|
|
# AI_PROVIDERS='{"my-gateway":{"type":"openai-compatible","baseUrl":"...","apiKey":"..."}}'
|
|
#
|
|
# Comma-separated model IDs (JSON array syntax also accepted, e.g. '["a","b"]'):
|
|
# AI_MODELS_DEFAULT_FAST=openai/gpt-5-mini,anthropic/claude-haiku-4-5-20251001
|
|
# AI_MODELS_DEFAULT_SMART=openai/gpt-5.2,anthropic/claude-sonnet-4-6
|
|
# AI_MODELS_DEFAULT_RECOMMENDED=openai/gpt-5.2,openai/gpt-4.1,anthropic/claude-sonnet-4-6
|
|
# AI_MODELS_DEFAULT_DISABLED=
|