## Why
The five event-log streams (`workspaceEvent`, `pageview`, `objectEvent`,
`usageEvent`, `applicationLog`) each wrote to ClickHouse through their
own fire-and-forget writer (`AuditService`, `UsageEventWriterService`,
and the `application-logs` driver), with the per-type knowledge (table
names, normalization, access rules) spread across several modules. Three
of them reimplemented the same ClickHouse insert, and the read side, the
live stream, and the producers lived in different modules under two
different names.
This consolidates them into one `core-modules/event-logs/` subsystem
(emit, write, live, read), with the per-type config in a single registry
so adding an event type is roughly one file.
The base Logs settings tab and free application logs shipped separately
in #21180 (merged). This PR adds the unified backend, the registry, and
the viewer's live mode and entitlement gating.
## Pipeline
```mermaid
flowchart TB
subgraph PROD["Producers"]
A["auth, billing, impersonation,<br/>webhook, custom-domain"]
U["usage listener"]
F["logic-function executor (app logs)"]
R["record CRUD (entity events)"]
end
EM["EventLogEmitterService<br/>createContext().insert* / dispatch()"]
EQ(["entityEventsToDbQueue<br/>(existing, shared with timeline)"])
CIE["CreateEventLogFromInternalEvent"]
SINK["WorkspaceEventSinkService.ingest()"]
C1["ClickHouseEventSink"]
C2["ConsoleEventSink"]
LIVE["EventLogLiveService.publishWatched()<br/>(presence-gated)"]
CH[("ClickHouse, 5 tables, async_insert")]
CHAN(["WORKSPACE_EVENTS_CHANNEL"])
RS["EventLogsService (registry-driven read)"]
LR["EventLogsLiveResolver"]
UI["Settings > Logs"]
A --> EM
U --> EM
F --> EM
EM -->|direct| SINK
R --> EQ --> CIE -->|ingest| SINK
SINK --> C1 --> CH
SINK --> C2
SINK --> LIVE -.->|if a viewer is watching| CHAN --> LR --> UI
CH --> RS --> UI
```
## What it does
- Producers call `EventLogEmitterService.createContext().insert*()`,
which builds a typed `WorkspaceEventEnvelope` and writes it through
`WorkspaceEventSinkService` to the configured sinks (ClickHouse,
Console) plus a presence-gated live fan-out. Record/CRUD events reach
the same sink through the existing `entityEventsToDbQueue`. There is no
dedicated queue; ClickHouse `async_insert` batches server-side. Writes
are best-effort, as on main today.
- `EVENT_LOG_TYPES[table]` is the per-type source of truth: the
ClickHouse table, the required entitlement, the free-text filter column,
and the row-to-GraphQL mapping. Read row shapes derive from the write
rows.
- Four modules along their dependency boundaries:
`EventLogEmitterModule` (producer API), `EventLogIngestionModule` (sink
layer), `EventLogLiveModule` (fan-out), and `EventLogsViewerModule` (the
entitlement-gated GraphQL read, which is where
billing/enterprise/permissions stay so producers stay light).
- Logs viewer: per-table columns, filters (text, date, record), live
mode, and an upgrade card that points to Billing on Cloud or the Admin
Panel on self-hosted. Application logs are free on every plan; the other
four require the `AUDIT_LOGS` entitlement (with a `NO_ENTITLEMENT`
fallback to the upgrade card).
- Renames `AuditService` to `EventLogEmitterService`, and the generic
`Monitoring` event to a typed `Impersonation` event (`level` +
`action`).
- Removes `UsageEventWriterService`, the `application-logs`
driver/module, and `AuditService`'s direct inserts.
## Durability
Writes are best-effort, the same as main today (the old writers were
fire-and-forget). A dedicated queue was tried mid-PR and removed:
`async_insert` already batches server-side, so the queue only added
durability, which isn't a requirement right now. The `EventSink` seam
keeps a durable transport (e.g. a Redis-Streams buffer) easy to add
later without touching producers.
## Out of scope
S3 peer sink (seam only), Postgres or any second read path,
`ReplicatedMergeTree`, ClickHouse table-schema changes, and the
record-data `EVENT_STREAM_CHANNEL` (unchanged, separate concern).
## Testing
Unit tests cover the registry definitions and row normalization, the
entitlement gating, the envelope builders, and the producers.
Integration tests cover the write paths (record create produces an
`objectEvent`; the track mutation produces a `workspaceEvent`) and the
read/query path across all five tables. Verified with typecheck, lint, a
server boot, and GraphQL/SDK codegen.
# Introduction
In this PR we're migrating both the `field` and `object` metadata to be
using the new `FlatEntityFromV2` that requires all the
`UniversalFlatEntityExtraProperties` to be spread at the flat entity
root.
This means that we have to update all of their flat declaration
This type swap allows to isole a specific entity migration into his own
type scope and avoid to have everything handled at once
```ts
/**
* Currently under migration but aims to replace FlatEntity afterwards
*/
export type FlatEntityFromV2<
TEntity,
TMetadataName extends AllMetadataName | undefined = undefined,
TInnerFlatEntity extends { __universal?: unknown } = FlatEntityFrom<
TEntity,
TMetadataName
>,
> = Omit<TInnerFlatEntity, '__universal'> & TInnerFlatEntity['__universal'];
```
## Impact
Both object and field:
- Create input transpilation utils
- from entity to flat tools
- mocks
## Note
Removed from the universal extra properties the jsonb properties that do
not contain a serialized
## Next
Next step is to incrementally make the builder and runner expect
`UniversalFlatEntity` for both of these metadata
This way we will be able to fully migrate an entity e2e typesafely
# Introduction
From `morphRelationCreationPayload` to `flatFieldMetadatas`
close https://github.com/twentyhq/core-team-issues/issues/1406
## Unit testing
I'm not a fan of covering things this way, where we could have some
strong integration testing tests in the first place
But I wanted to freeze the optimistic path computation, also covered the
exception because it was cheap, doesn't mean I won't cover them through
integration tests later anw
# Introduction
Moving validation directly in the builder that has the perfect
granularity to do it.
When importing we won't have to infer and dispatch on the operation
nature ( update delete create ) and validate accordingly
## Objects
Only migrated object validation for the moment even though create object
involves a validate flat field metadata creation call too
## TODO
- improve `otherFlatObjectMetadataMapsToValidate` naming too vague
## Next
- handle fields validation within fields actions build
- Unit test coverage validation issue on builder and validate
- integration test plugging with new feature flag
## Manual tested
- Update
- Delete
- Create TODO
# Introduction
Introduced `EachTesting` pattern for the builder unit tests.
As always any suggestions are more than welcomed !
Still need to:
- [x] implem basic tests for field
- [x] create `get-flat-index-field-metadata.mock.ts`
- [x] Implement basic tests for index and index-fields
- [ ] Implem standard edges cases tests TDD style
## Misc
- was https://github.com/twentyhq/twenty/pull/13132 closed due to mess
to rebase on main
No need to audit log workflow runs as it's already a form of audit log.
Add more audit log for other objects
Rename MessagingTelemetry to MessagingMonitoring
Merge Analytics and Audit in one (Audit)
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Fix all the broken CIs :p
This includes an ongoing effort to simplify test maintenance by having 1
unique source of truth about metadata and data mocks (that will later be
generated from a unique source of seeds: dev = demo = test)
Regressions:
- Unit line coverage: 60 > 55
- Storybook Pages branch coverage: 40 > 35
We will need to write tests to increase those coverage
- RelationFieldDisplay perf: 0.2ms to 0.22ms > We might have a
regression here
- Removed perf story about RawJSON > We will need to re-add it
### Summary
This PR introduces several integration tests, a mix of manually written
tests and those generated using the `generate-integration-tests` Python
script located in the `scripts` folder.
### Tests Added:
- **Authentication tests**: Validating login, registration, and token
handling.
- **FindMany queries**: Fetching multiple records for all existing
entities that do not require input arguments.
### How the Integration Tests Work:
- A `setupTest` function is called during the Jest test run. This
function initializes a test instance of the application and exposes it
on a dedicated port.
- Since tests are executed in isolated workers, they do not have direct
access to the in-memory app instance. Instead, the tests query the
application through the exposed port.
- A static accessToken is used, this one as a big expiration time so it
will never expire (365 years)
- The queries are executed, and the results are validated against
expected outcomes.
### Current State and Next Steps:
- These tests currently run using the existing development seed data. We
plan to introduce more comprehensive test data using `faker` to improve
coverage.
- At the moment, the only mutation tests implemented are for
authentication. Future updates should include broader mutation testing
for other entities.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Closes#5097
- Uses "nx affected" to detect what projects need to be checked in the
current PR (for now, `ci-front` and `ci-server` workflows only).
- Caches results of certain tasks (`lint`, `typecheck`, `test`,
`storybook:build`) when a PR pipeline runs. The next runs of the same
PR's pipeline will then be able to reuse the PR's task cache to execute
tasks faster.
- Caches Yarn's cache folder to install dependencies faster in CI jobs.
- Rewrites the node modules cache/install steps as a custom, reusable
Github action.
- Distributes `ci-front` jobs with a "matrix" strategy.
- Sets common tasks config at the root `nx.json`. For instance, to
activate the `typecheck` task in a project, add `typecheck: {}` to its
`project.json` and it'll use the default config set in `nx.json` for the
`typecheck` task. Options can be overridden in each individual
`project.json` if needed.
- Adds "scope" tags to some projects: `scope:frontend`, `scope:backend`,
`scope:shared`. An eslint rule ensures that `scope:frontend` only
depends on `scope:frontent` or `scope:shared` projects, same for
`scope:backend`. These tags are used by `nx affected` to filter projects
by scope and generates different task cache keys according to the
requested scope.
- Enables checks for twenty-emails in the `ci-server` workflow.
