* fix jmap.SetVacationResponse: was using JMAP create instead of update
with a patch; did so by using the modify template function as with
most other JMAP APIs
* better HTTP status code handling for the responses received from
Stalwart
* add support for not caching JMAP session retrieval failures in the
Groupware backend, by setting the error TTL to 0
* introduce a new toplevel package pkg/http for plain HTTP related
functions
* move these functions from pkg/jmap to pkg/http, including tests:
- PeekResponse
- DumpHttpRequest
- DumpHttpResponse
* also look into the OC_INSECURE environment variable to allow
accepting insecure TLS certificates
* rename GROUPWARE_ALLOW_INSECURE_TLS to GROUPWARE_TLS_INSECURE to
align with the nomenclature used by other backend services
* split off the Stalwart integration test boilerplate into a new
jmaptest package, in order to be able to use it from the groupware
package as well
* remove the JMAP Session patching in tests and, instead, pick random
free local ports for HTTP and IMAPS manually, and set those
explicitly in the Stalwart testcontainer, so that we can set the
PUBLIC_URL environment variable that Stalwart then uses to define the
URLs that are served as part of the JMAP Sessions (API URL, etc...)
* introduce configuration settings for TLS insecurity, tracing requests
and responses
* use more secure implementation when replacing JMAP placeholders in
URL templates, path escaping them to avoid injection
* add more efficient tracing of HTTP requests and responses between the
Groupware backend and the JMAP server, with the possibility of
specifying a maximum body size to trace, to avoid blowing up the logs
* use more efficient string building for HTTP Authorization headers
* change internal API to use streams (io.Reader) instead of reading
JSON responses fully into memory before parsing them
* implement supplier templating to allow addressbooks and contacts to
come from various sources ("suppliers", thus) as opposed to solely
from JMAP
* add creating, deleting, updating, getting changes
* add some search filter parameters, currently incomplete
* by adding a predicate that can analyze a filter and inform the
template methods whether the filter is applicable for that supplier
or not
* this fixed /groupware/accounts/.../addressbooks/.../contacts
* retrofit from using self-referencing generics parameters that were
introduced with Go 1.26, will re-enable when we upgrade to 1.26
* re-introduce a 'ptr' func since we don't have the new 'new' func yet
that comes with 1.26
* fix the inline dockerfile for building the stalwart:cli container,
cannot be based on scratch since we need a shell and wget, and it
should thus retain the alpine container as its base
* remove stdin_open and tty from the container attributes, not needed
* add instructions and container to create a key and certificate pair
for the built-in IDM LDAP as we need it to listen on LDAPS for
Stalwart, which has been disabled by default with
https://github.com/opencloud-eu/opencloud/pull/2880
* add environment variables to point to the certificates
* update services/groupware/DEVELOPERS.md with up-to-date instructions,
using Stalwart 0.16
* don't run the Stalwart container in recovery mode first, just run it
in "proper" mode, import the configuration and use it like that,
which only requires skipping the "destroy" steps
* add code documentation and logging
* replace the text templating of the Stalwart snapshot by doing the
same as the formatting script was doing, but directly in Go, removing
an additional step to perform when modifying a Stalwart
configuration to use in the integration tests
* add new configuration setting GROUPWARE_SEND_DURATIONS_RESPONSE
(defaults to false)
* keep track of lists of durations of backend calls
* when enabled, report them as response headers Durations (human
readable) and Durations-Nanos (as raw nanosecond values for machine
consumption)
* remove the following methods on the jmap.QueryCommand interface, as
they ended up not being needed:
- GetPosition()
- GetAnchor()
- GetAnchorOffset()
- GetLimit()
* refactor APIs in JMAP and Groupware in order to implement pagination
across multiple accountIds and multiple suppliers (currently
implemented using a mock supplier for contacts)
* requires go 1.26 due to use of self-reflecting generics type
constraints
* still missing: query criteria and sorting parameters
* still missing: multi-accountId support for emails
* errors are now all just 'error' in the APIs, instead of the
specialized implementations, and are interpreted dynamically where
necessary in order to transform them into HTTP responses
* remove position, anchor, anchorOffset as individual query parameters
as we now only support a 'next=...' token for subsequent pages
(except in emails for now), and use jmap.QueryParams instead; those
tokens have a header character for the format, followed by a JSON
encoded QueryParams map, all wrapped into base62 to make it clearer
that it is meant to be an opaque token, and not a parameter clients
should tinker with or construct themselves
* introduce QueryParamsSupplier as an interface to provide QueryParams
for various scenarios (single supplier, multiple supplier, ...) per
accountId
* implement multi-supplier template methods slist and squery
* in the JMAP API as well as in several places in the Groupware
framework, use a single jmap.Response[T] object to return the
payload, the language, the session state and the etag/state instead
of individual multi-valued return values
* the "position" attribute is always set to 0 when using an anchor for
pagination, but it does not accurately reflect the position and thus
we decided to remove the attribute from the resulting object for
clarity
* omit "canCalculateChanges" attribute when its value is "true", which
is going to be the case in 100% of calls against Stalwart
* JMAP query limit of 0 is synonymous with "no limit", but we actually
want to be able to perform queries without any results, for cases
where we only want to count the total number of objects, and also
because it makes more sense semantically
* introduce query parameter validation checks, in order to only allow
query parameters that are actually supported, which is going to be
useful during development of clients
* adds creating addressbooks, calendars, mailboxes
* adds deleting mailbox, event, identity
* adds modifying an email
* introduce template functions for the Groupware API in templates.go,
and use those in route function implementations whenever possible
* add capability checking for mail, quota, blobs
* adds Changes interface
* adds JmapResponse interface
* jmap: add UpdateContactCard and UpdateCalendarEvent funcs
* use JSON marshalling and unmarshalling into a map for toPatch()
implementations
* add updating ContactCards and CalendarEvents to tests
* add deleting ContactCards and CalendarEvents to tests
* make query response totals work when the value is 0
* tests: use CreateContactCard and CreateCalendarEvent funcs to create
objects in tests instead of using a different JMAP stack that works
with untyped maps
* make the JMAP internal API a bit more future-proof by passing
ObjectType objects instead of the JMAP namespaces
* remove the new attempt to contain operations even further using the
Factory objects
* move CalendarEvent operations to its own file, like everything else
* fix email tests
* ignore WS error when closing an already closed connection
* move ContactCard from jscontact to jmap, as it is actually a JMAP
specification item, but also was causing too many issues with
circular references from jscontact -> jmap
* introduce Foo, Idable, GetRequest, GetResponse, etc... types and
generics
* first attempt at a Foo factory type for Mailboxes, needs to be
expanded to further minimize repetition
* add more specialized template functions to avoid repetition
* introduce ChangesTemplate[T] for *Changes structs
* add Groupware APIs for creating and deleting addressbooks
* add Groupware APIs for creating and deleting calendars
* add JMAP APIs for creating and deleting addressbooks, calendars
* add JMAP APIs to retrieve Principals
* fix API tagging
* move addressbook JMAP APIs into its own file
* move addressbook Groupware APIs into its own file
* implement ContactCard retrieval endpoint for syncing
* re-implement that endpoint for Email too
* fix the Mailbox changes endpoint to actually return changes about
Mailboxes, and not about Emails
* when querying the diff of Mailboxes without any prior state, return
an error since the result is not what one would expect
* introduce the 'changes' API tag and group
* refactor the successful response functions to consistently return an
object type and object state whenever possible
* move the syncing endpoints under /accounts/*/changes/ for better
clarity, e.g. /changes/emails instead of /emails/mailbox/*/changes
- fix a bunch of minor issues and typos that were found using GoLand
and gosec
- add a gosec Makefile target for Groupware related files, in
services/groupware/Makefile
- enable checking JMAP session capabilities for events and contacts,
and only enable skipping that check for tasks until those are
implemented in Stalwart as well
- fix a CWE-190 (integer overflow or wraparound) found by gosec
- consistently use struct references for methods of Groupware and
Request, instead of mixing up references and copies
- always log errors when unable to register a Prometheus metric
* re-implement the auth-api service to authenticate Reva tokens
following the OIDC Userinfo endpoint specification
* pass the context where necessary and add an authenticator interface
to the JMAP HTTP driver, in order to select between master
authentication (which is used when GROUPWARE_JMAP_MASTER_USERNAME and
GROUPWARE_JMAP_MASTER_PASSWORD are both set) and OIDC token
forwarding through bearer auth
* add Stalwart directory configuration "idmoidc" which uses the
OpenCloud auth-api service API (/auth/) to validate the token it
received as bearer auth from the Groupware backend's JMAP client,
using it as an OIDC Userinfo endpoint
* implement optional additional shared secret to secure the Userinfo
service, as an additional path parameter
* add example generator infrastructure, with some examples for pkg/jmap
and pkg/groupware, with more needing to be done
* alter the apidoc Makefile to stop using go-swagger but, instead, use
the openapi.yml file that must be dropped into that directory using
groupware-apidocs (will improve the integration there later)
* add Makefile target to generate examples
* bump redocly from 2.4.0 to 2.14.5
* introduce Request.PathParam() and .PathParamDoc() to improve API
documentation, as well as future-proofing
* improve X-Request-ID and Trace-Id header handling in the middleware
by logging it safely when an error occurs in the middleware
* jmap/EmailCreate: add more attributes that were omitted: Headers,
InReplyTo, References, Sender
* add jmap GetEmailSubmissionStatus
* improve email integration tests by adding a thorough test for email
submission
* jmap integration tests: provision principals and domains using the
Stalwart Management API, switching from an in-memory to an internal
directory
* fix a bug in how email summaries are flattened across multiple
accounts, which was previous resulting in empty email objects
* allow negative offset in email pagination
* make all /emails endpoints return emails without bodies
* implement Request.AllAccountIds() to generalize the fetching (and
uniqifying) of all account IDs, which will allow us to implement
things such as "subscribed" accounts, or limiting the number of
accounts in one request
* add Account-Id response header
* add Object-Type response header
* upgrade Stalwart image for devtools/full to 0.14.1
* re-assert which features are implemented or not in 0.14.1
* refactor the integration tests yet again to make it clearer and
easier to see those "features-or-not"
* get rid of old tests that are now better covered by integration tests
* rewrite how we compare expected and actual objects in integration
tests, finally having found a way to ignore the @type attribute
properly instead of having to mutate all objects to remove it
* add endpoints for Mailboxes:
- PATCH mailboxes/{id}
- DELETE mailboxes/{id}
- POST mailboxes
* refactor the pkg/jmap and groupware framework to systematically
return a jmap.State out-of-band of the per-method payloads, since
they are almost always present in JMAP responses, which lead to the
artificial creation of a lot of composed struct types just to also
return the State; on the downside, it adds yet another return
parameter
* upgrade image version in devtools to 0.14.0
* fix idmldap configuration to use the cn attribute in order for that
to also work for groups (groups don't have a uid attribute in the IDM
built-in LDAP)
* group resources are now checked against LDAP, changed
demo-principals.yaml accordingly to refer to a group that exists in
LDAP as part of the demo data
* made several email related operations multi-account:
QueryEmailSnippets, QueryEmails, QueryEmailsWithSnippets
* add GetIdentitiesForAllAccounts
* add GetEmailsForAllAccounts
* jmap: add CreateIdentity, UpdateIdentity; groupware: add
GetIdentityById, AddIdentity, ModifyIdentity
* add temporary workaround until Calendars, Tasks, Contacts are
implemented in Stalwart when determining the default account for
those: use the mail one in the mean time
* changes from 0.13.4:
- JMAP: Protocol layer rewrite for zero-copy deserialization and
architectural improvements.
- IMAP: Unbounded memory allocation in request parser
(CVE-2025-61600)
- IMAP: Wrong permission checked for GETACL.
- JMAP: References to previous method fail when there are no results
(stalwartlabs#1507).
- JMAP: Enforce quota checks on Blob/copy.
- JMAP: Mailbox/get fails without accountId argument (stalwartlabs#1936).
- JMAP: Do not return invalidProperties when email update doesn't
contain changes (stalwartlabs#1139)
- iTIP: Include date properties in REPLY (stalwartlabs#2102).
- OIDC: Do not set username field if it is the same as the email field.
- Telemetry: Fix calculateMetrics housekeeper task (stalwartlabs#2155).
- Directory: Always use rsplit to extract the domain part from email
addresses.
* changes from 0.13.3:
- CLI: Health checks
- WebDAV: Assisted discovery v2
- iTIP: Do not send a REPLY when deleting an event that was not
accepted.
- iTIP: Include event details in REPLY messages (stalwart#2102).
- iTIP: Add organizer to iMIP replies if missing to deal with MS
Exchange 2010 bug.
- OIDC: Do not overwrite locally defined aliases (stalwart#2065).
- HTTP: Scan ban should only be triggered by HTTP parse errors.
- HTTP: Skip scanner fail2ban checks when the proxy client IP can't
be parsed (stalwart#2121).
- JMAP: Do not allow roles to be removed from system mailboxes
(stalwart#1977).
- JMAP WS: Fix panic when using invalid server url.
- SMTP: Do no send EHLO twice when STARTTLS is unavailable
(stalwart#2050).
- IMAP: Allow ENABLE UTF8 in IMAPrev1.
- IMAP: Include administer permission in ACL responses.
- IMAP: Add owner rights to ACL get responses.
- IMAP: Do not auto-train Bayes when moving messages from Junk to
Trash.
- IMAP/ManageSieve: Increase maximum quoted argument size
(stalwart#2039).
- CalDAV: Limit recurrence expansions in calendar reports
(CVE-2025-59045).
- WebDAV: Do not fix percent encoding on WebDAV FS (stalwart#2036).
* made a few changes in order to further simplify the setup for
developers of the Groupware backend
* add STALWART_DOMAIN to deployments/examples/opencloud_full/.env
* adapt the Stalwart configuration file to not set server.hostname and,
instead, pick it up from /etc/hostname, which is set by Docker
Compose as we can use default values for STALWART_DOMAIN there, in an
analogous fashion to the other containers in that project
* add config/keycloak/clients/groupware.json to avoid requiring manual
configuration of Keycloak via the admin web UI
* Stalwart container:
- listen for SMTPS on :1465
- remove the stalwart-logs volume, not needed (logs are going to
stdout)
* updated services/groupware/DEVELOPER.md:
- refer to a variable OCDIR to make instructions more copy-pasteable
- remove manual Keycloak configuration section as it is now obsolete,
replaced by provisioning a configuration file instead
Minor: be more Go idiomatic: just use a function to pick the attachment
from an Email's attachment list instead of using an interface with
multiple iplementation structs.
* fix(jmap): fix bug where CommandBlobUpload was used instead of
CommandBlobGet in GetBlob (now GetBlobMetadata)
* we currently don't need a variant of BlobGetCommand that also
retrieves the content of the blob, instead we only use it for
retrieving metadata about it
* the JMAP error handling was not working properly, fixed it and added
error definitions accordingly
* add operations to retrieve mailbox roles and mailboxes by role for
all accounts
There was really no reason to go with "Messages" as far as the
vocabulary of the Groupware API goes, since the objects those APIs serve
are "Emails", to stick with the wording of the JMAP specification.
* introduce a function 'mcid' to assemble method call IDs per account
instead of doing that inline in each function, in case the rules for
doing so change in the future
* move jmap.request() to jmap.Client.request() and pass the Session
and a Logger to introduce checking the number of methodCalls within a
request not exceeding the limit of the Session, as well as error
handling and logging there instead of in each caller
* a few bugfixes:
- add a few missing Send() calls in logs
- correct the response tag matching for
GetMailboxChangesForMultipleAccounts
- fix typo in Identity.ReplyTo json serialization rune
- fix response tag in pkg/jmap/testdata/mailboxes1.json after
changing them to be prefixed by the accountId
* add a launcher for running OpenCloud from within VSCode, but using
third-party services that are running within the docker compose
'full' example setup
* add URL to retrieve all the mailboxes for all the accounts of a user,
as a first use-case for an all-accounts operation, as
/accounts/all/mailboxes
* add URL to retrieve mailbox changes for all the mailboxes of all the
accounts of a user, as a first use-case for an all-accounts
operation, as /accounts/all/mailboxes/changes
* change the defaultAccountId from '*' to '_', as '*' rather indicates
"all" than "default", and we might want to use that for "all
accounts" operations in the future
* refactor(groupware): remove the accountId parameter from the logger()
function, as it is not used anyways, but also confusing for
operations that support multiple account ids
* refactor some pkg/jmap and groupware methods to make more sense from
an API point-of-view
* add path parameter documentation, but automate it by injecting their
definition into the OpenAPI YAML tree that is extracted from the
source code using go-swagger as it is too cumbersome, repetitive and
error-prine to document them in the source code; wrote a TypeScript
file apidoc-process.ts to do so
* add generating an offline HTML file for the OpenAPI documentation
using redocly, and injecting a favicon into the resulting HTML; wrote
a TypeScript file apidoc-postprocess-html.ts to do so
* move the logging of the username and session state away from pkg/jmap
and into services/groupware
* introduce more decoupling for the session cache, as well as moving
the implementation into groupware_session.go
* remove the baseurl from the JMAP client configuration, and pass it to
the session retrieval functions instead, as that is really the only
place where it is relevant, and we gain flexibility to discover that
session URL differently in the future without having to touch the
JMAP client
* move the default account identifier handling from the JMAP package to
the Groupware one, as it really has nothing to do with JMAP itself,
and is an opinionated feature of the Groupware REST API instead
* add an event listener interface for JMAP events to be more flexible
and universal, typically for metrics that are defined on the API
level that uses the JMAP client
* add errors for when default accounts cannot be determined
* split groupware_framework.go into groupware_framework.go,
groupware_request.go and groupware_response.go
* move the accountId logging into the Groupware level instead of JMAP
since it can also be relevant to other operations that might be
worthy of logging before the JMAP client is even invoked
* implement more metrics, in a more streamlined fashion
* use concurrent-map to store SSE streams instead of a regular map with
one big lock that will not scale when it grows, causing too much
contention on that one lock
* while testing error metrics, noticed a few bugs with error handling
when Stalwart is down: fixed
* implement correct Etag and If-None-Match handling, responding with
304 Not Modified if they match
* introduce SessionState and State string type aliases to ensure we are
using the correct fields for those, respectively
* extract the SessionState from the JMAP response bodies in the
groupware framework instead of having to do that in every single
groupware API
* use uint instead of int in some places to clarify that the values are
>= 0
* trace-log how long a Session was held in cache before being evicted
* add Trace-Id header handling: add to response when specified in
request, and implement a custom request logger to include it as a
field
* implement a more compact trace-logging of all the methods and URIs
that are served, to put them into a single log entry instead of
creating one log entry for every URI
* add a GET /accounts/{a}/boostrap URI that delivers the same as GET /
but also mailboxes for a given account, in case the UI remembers the
last used account identifier, to avoid an additional roundtrip
* streamline the use of simpleError()
* add logging of errors at the calling site
* add logging of evictions of Sessions from the cache
* change default Session cache TTL to 5min instead of 30sec
* add more documentation for properties
* fixes after a bit of trial-and-error with go-swagger
* fix email filter marshalling when there are no search criteria
* introduce an apidoc.yml that contains Swagger data and is merged when
generating the swagger.yml from sources
* ensure that all the jmap responses contain the SessionState
* implement missing errors that were marked as TODO
* moved common functions from pkg/jmap and pkg/services/groupware to
pkg/log and pkg/structs to commonalize them across both source trees
* implement error handling for SetError occurences
* Email: replace anonymous map[string]bool for mailbox rights with a
MailboxRights struct, as the keys are well-defined, which allows for
properly documenting them
* introduce ObjectType as an "enum"
* fix JSON marshalling and unmarshalling of EmailBodyStructure
* move the swagger documentation structs from groupware_api.go to
groupware_docs.go
* fix: change verb for /groupware/accounts/*/vacation from POST to PUT
* refactor the jmap package to split it into several files as the
jmap.api.go file was becoming too unwieldy
* refactor the Groupware handler function response to be a Response
object, to be more future-proof and avoid adding more and more
return parameters while handling "no content" response as well
* more godoc for the JMAP model
* add Email creation, updating, deleting (Email/set,
EmailSubmission/set)
* add endpoints
- POST /accounts/{accountid}/messages
- PATCH|PUT /accounts/{accountid}/messages/{messageid}
- DELETE /accounts/{accountid}/messages/{messageid}
* after having decided that the Groupware API should be a standalone
independent custom REST API that is using JMAP data models as much as
possible,
* removed Groupware APIs from the Graph service
* moved Groupware implementation to the Groupware service, and
refactored a few things accordingly
* refactored the models to be strongly typed with structs and mapstruct
to decompose the dynamic parts of the JMAP payloads
* externalized large JSON strings for tests into .json files under
testdata/
* added a couple of fantasy Graph groupware APIs to explore further
options
* added k6 scripts to test those graph/me/messages APIs, with a setup
program to set up users in LDAP, fill their IMAP inbox, activate them
in Stalwart, cleaning things up, etc...
* primitive implementation to demonstrate how it could work, still to
be considered WIP at best
* add new dependency: MicahParks/jwkset and MicahParks/keyfunc to
retrieve the JWK set from KeyCloak to verify the signature of the
JWTs sent as part of Bearer authentication in the /auth API
* (minor) opencloud/.../service.go: clean up a logging statement that
was introduced earlier to hunt down why the auth-api service was not
being started
Per review discussion: don't document accidental behavior. The contract is
simply "percent-encode each path segment, as MS Graph requires; encode ':'
as %3A" - OpenCloud allows ':' in names (OneDrive forbids it), so it's one
more character in the mandatory encode set, not a special case.
The parser is unchanged (split on ":/", decode once). This only rewrites the
docs (code comment, acceptance feature, PR description) to state the encode
contract, and drops the tests that relied on a raw, unencoded ':' in a file
name - keeping the "%3A" test that reflects the actual contract.
Review feedback: split the anchor/path and path/suffix on the structural
delimiter ":/" instead of a bare ":". Since the path and suffix always
start with "/", ":/" is the real delimiter, and a ":" *inside* a file or
directory name (which OpenCloud allows but MS Graph/OneDrive forbid) is
kept as part of the path instead of being mistaken for a separator.
A ":" sitting at a segment boundary (e.g. a name ending in ":") stays
ambiguous and must be percent-encoded as "%3A": the split works on the
literal ":/", so "%3A" is never a delimiter and decodes back to ":". This
is now documented in the code and the acceptance feature.
Tests: colon inside a name (with and without a suffix), the Stat path
carrying the colon, and the "%3A" boundary escape.
The colon-syntax parsing is plain string operations now, but a few
comments still referred to "the previous regex" / "captures". Update them
to match: parseColonPath extracts the itemID from the path (driveID comes
from the route param), and the shape checks stand on their own.
parseColonPath split the anchor asymmetrically (root trimmed the delimiter
colon as part of a literal "/root:" prefix, item cut on it), and the root
branch's comment talked about the driveID which isn't this function's
concern. Split once at the first colon into anchor + rest, then classify
the anchor (exactly "/root", or "/items/{id}" with a single-segment id).
Same behavior, easier to follow.
Also add explicit coverage for the rule that a suffix requires a second
colon: "/root:/Documents/children" (no second colon) is the path
"/Documents/children", not the path "/Documents" with a "/children"
suffix. Two tests pin it end-to-end - it must route to the bare item (not
/items/{id}/children) and Stat must receive the full path.
Review feedback: the two colon-syntax regexes were hard to read. Since the
middleware is scoped to /drives/{driveID}, the RoutePath it inspects is just
the sub-path (/root:/... or /items/{id}:/...), so a regex buys nothing.
Replace rootColonRe/itemColonRe (and the matchInto/extract helpers) with a
single parseColonPath that uses plain string operations
(HasPrefix/TrimPrefix/TrimSuffix/Cut), one commented step at a time.
Behavior is unchanged; the existing table tests (trailing colon, no suffix,
deep paths, multi-segment suffixes, item-anchored, encoded paths) still pass
and pin it.
Addresses review feedback: a sub-router middleware can re-route after all,
as long as it rewrites chi.RouteContext().RoutePath instead of r.URL.Path.
Once chi has descended into a sub-router its routeHTTP matches against
rctx.RoutePath and ignores r.URL.Path, which is why the earlier top-level
registration was thought to be required.
Move ResolveGraphPath off the top-level mux.Use and attach it to the
/drives/{driveID} sub-routers (v1.0 + v1beta1). It now reads driveID from
chi.URLParam and matches against RoutePath (the part below the drive), so
the regexes drop the version + drive prefix entirely.
RoutePath carries the percent-encoded wire form (Graph.ServeHTTP sets
RawPath), so the captured driveID/itemID/path are PathUnescape'd exactly
once - reproducing the decoded r.URL.Path a normal handler would see,
without the previous RawPath/EscapedPath workaround. r.URL.Path is now
left untouched; only chi's internal RoutePath is rewritten.
Tests are reworked to drive requests through a chi router mirroring the
production nesting (including the Graph.ServeHTTP RawPath behavior), so
chi's sub-router middleware ordering, RoutePath encoding and param
round-trip are all covered indirectly: a chi upgrade that changes any of
them fails these tests instead of silently breaking colon-path lookups.
Adds explicit coverage for percent-decoding (%20, %252F) and the `$`/`!`
sub-delimiter id round-trip.
Codacy flagged the colon-path middleware comment claiming both `$`
and `!` need percent-encoding for chi's tree match, while the
implementation only calls r.URL.RawPath = r.URL.EscapedPath() which
does not encode either character per the suggestion's reading.
In practice EscapedPath does encode `!` as `%21` (only `?` is the
hardcoded escape, `!` is escaped because Go's net/url treats it as
needing encoding outside specific contexts). It leaves `$` literal,
which chi handles fine. chi.URLParam returns the encoded segment
verbatim, and downstream OpenCloud handlers (parseIDParam,
GetDriveAndItemIDParam) already PathUnescape before parsing IDs, so
the round-trip works end-to-end. The acceptance tests on this
branch already exercise this with real `$`/`!`-containing IDs.
Add a focused unit test that mounts the middleware behind chi,
sends a colon URL, and asserts the actual contract:
- driveID (only `$`): chi.URLParam returns it literal
- itemID (with `!`): PathUnescape(chi.URLParam(...)) == original
Update the misleading comment so future readers (and reviewers) see
what the encoding actually does and which downstream contract it
relies on.
Cover the rewrite shapes the middleware handles end-to-end against a
real OpenCloud server: root-anchored, item-anchored, deep paths,
trailing colon, and the "/<path>:/<suffix>" sub-route form. Also
assert that NOT_FOUND and PERMISSION_DENIED both collapse to 404.
The /permissions sub-route is registered only at /v1beta1, and the
v1beta1 GetDriveItem handler is share-jail-only, so the v1beta1
mount of the middleware is exercised through the permissions
scenario, since there is no other v1beta1 endpoint that works for
regular personal-drive items.
Three more Copilot review nits on the colon-syntax middleware:
- CS3 Stat returning UNAUTHENTICATED now surfaces as HTTP 401 (was 500
via the default-case fallback). Distinct sentinel + handler branch.
- Item-anchored form (/drives/{driveID}/items/{itemID}:/...) now
validates that driveID's storage/space prefix matches the itemID's,
short-circuiting to 400 InvalidRequest on mismatch instead of doing
a CS3 Stat that would only fail at the handler layer.
- ParseID failures on the anchor id now surface as 400 (was 404). In
practice this branch is defensive — storagespace.ParseID is lenient
and only errors on empty input, which the regex already filters out
— but the right semantic for malformed client input is 400, not 404.
Tests: added two new cases (UNAUTHENTICATED → 401, drive/item id
mismatch → 400). The "malformed drive id" case Copilot suggested
isn't reachable in practice given ParseID's leniency, so it's not
covered.
Two follow-up Copilot review nits on the colon-syntax middleware:
- Don't blank r.URL.RawPath after the rewrite. Graph.ServeHTTP sets
RawPath = EscapedPath() as a workaround for chi's parameter-binding
quirks with `$`/`!` in IDs (see go-chi/chi#641). Clearing RawPath
for rewritten requests negates that workaround. Re-establish the
same invariant after the rewrite.
- Tests previously used log.NewLogger(), which mutates global zerolog
state. Switch to log.NopLogger() — order-independent, no global
side effects.
- Drop double-decoding of URL path components. r.URL.Path is already
decoded by net/http; calling url.PathUnescape again would let crafted
inputs like "%252F" become "/", changing path semantics. Path and
anchor-id strings now go straight to utils.MakeRelativePath /
storagespace.ParseID.
- Distinguish operational errors from "not found". Gateway selection
failure, RPC transport errors, and unexpected CS3 status codes now
surface as 500 (don't mask outages); only NOT_FOUND and PERMISSION_DENIED
collapse to 404 (no existence disclosure). Implemented via a sentinel
errPathNotFound + a 500 fall-through.
- Refactor the two regex-handling branches in rewriteColonPath to share a
resolution path via a small colonMatch struct + matchInto/extract helpers.
Removes the SonarCloud duplication finding without changing behavior.
- Update the docstring on ResolveGraphPath to reflect that the rewrite
preserves the requested API version (/{version}/...) rather than
hard-coding /v1beta1/...
- Test cleanup: register t.Cleanup to remove the per-subtest selector
entry from pool's global selectors map after the subtest, and update
the "unexpected status" case to expect 500 (matches the new error
semantics).
Adds a chi middleware that detects MS Graph colon-syntax URLs and rewrites
them to the canonical /items/{itemID}/... form before chi performs route
matching. Existing handlers, routes, and GetDriveAndItemIDParam stay
unchanged.
Two URL shapes are recognized at both /v1.0 and /v1beta1:
/drives/{driveID}/root:/<path>[:/<suffix>][:]
/drives/{driveID}/items/{itemID}:/<relativePath>[:/<suffix>][:]
Path resolution runs as the request user via CS3 Stat. Both NOT_FOUND and
PERMISSION_DENIED collapse to a 404 response so existence isn't disclosed
to unauthorized callers. URLs without colon syntax fast-path through with
a single substring check. The original URL is stashed in request context
under OriginalPathContextKey for downstream tracing/logging.
The middleware is registered as a top-level mux.Use so it runs before any
route matching: chi middleware on a sub-router runs after the prefix is
matched but cannot redirect to a different leaf route. Top-level
middleware lets URL rewriting actually re-route the request.
Tests cover regex matching across versions, all rewrite variants
(root/items anchored, with/without suffix, with/without trailing colon,
deep paths), NOT_FOUND -> 404, PERMISSION_DENIED -> 404 (security: no
existence disclosure), and original-URL preservation in request context.
#1877 reordered the api-test teardown to delete spaces before users, so the
state where a project space's manager has been deleted is no longer exercised.
This adds explicit coverage for it: the only manager of a project space is
deleted, and a space admin can still list and delete (disable + purge) the space.
Related: #1878
opencloud-eu/reva#655 changes the TUS create response for an invalid name
from 412 to 400. Update the invalid-name scenario to match.
Each example now carries its own expected status in an <http-status-code>
column. Names that fail ValidateName expect 400. The three folder/file rows
stay 412: filename() applies path.Base first, so "folder/file" becomes the
valid leaf "file" and never reaches the changed branch.
lowLevelCreationExtension.feature is unchanged; its missing-Upload-Length
case still returns 412.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a skill that let's agents bump web to a specified version, commit
the changes and create a PR for it. The user invokes the skill via e.g.
"bump web to v7.0.0".
Allow a "permission denied error" from reva to bubble up to the client.
Reva was fixed to return "permission denied" only when the space to be
delete can actually be listed by the user. Other wise it will return
"not found". See reva commit 1bf72cb76394671f373e87f15f23f978cf41ab08.
So when a user with the 'can manage' role tries to purge an already
disabled space it will now get "Forbidden" status instead of a "Not
found".
Also fixes the expected status codes in the tests.
This fixes error logs like
RR error when calling Createhome error="gateway: grpc failed with code CODE_INVALID_ARGUMENT" line=github.com/opencloud-eu/opencloud/services/proxy/pkg/middleware/create_home.go:87 service=proxy
e.g. during internal requests to the data provider.
Also adjusts the build image to use the opencloudeu/golang image. So
we're in sync with how the binaries are built in CI. It already contains
all the required dependencies.
GetRoleDefinition/s does now handle l10n correctly. Previsouly it just
returned the non-localized string. What made things worse was that
ListPermissions() mutated global list of available roles and replaced
some strings with translated values depending on the `accept-language`
header. Which resulted in GetRoleDefinition returning results in mixed
localization depending on who/what called ListPermissions before.
Fixes: #2800
Always including the parent directory into the build context was a bad
idea. This reverts the change and only add the opencloud tree to the
build context by default.
To build a docker image with workspaces enabled as new make target
"dev-docker-with-workspace" is introduced that adjust the docker build
call accordingly (by setting the right build context and the new SRCDIR
arg to point to the correct directory for the opencloud tree).
This fixes `opencloud init` to only print the admin password if it was
generated by `init`. No longer log is when is was set via env var or command
line arguments.
Additionally add a `--quiet` command line flag to disable all non error
output.
Fixes: #1022
This fixes the server-side translations for "activties" and e.g. the
default space description. We need to bump "leonelquinteros/gotext"
to latest master for that as even the latest release still contains
and issue that cause `go vet` to complain about non-constant
format strings.
Fixes: #2833, #2835
- allow it to open up to 4 PRs in parallel
- change the npm interval to daily
- set cooldown to 1 day to reflect the `minimumReleaseAge` default
of pnpm
escapeStringMap mutated its input map. The recipient loop in eventsNotifier.render reuses that map across iterations, so each recipient past the first got values with one extra HTML escape layer. Return a new map instead.
Fixes#2804
Signed-off-by: Michael Stingl <mail@michaelstingl.com>
The base logger created by log.Configure already attaches a "service"
field via its context. handleProbe added it a second time, producing
JSON log entries with two "service" keys. Remove the redundant
Str("service", name) and the now-unused name parameter.
In order to be able to run migrations, the "sharing" service now needs
the "service_account_id" and -"_secret" to be configured.
This is a "breaking/backwards incompatible" change.
activitylog was still using oklog run.Group{} for starting the service
this cause issue during shutdown. Seems this service was overlooked
when switching all the other services to use runner.
Fixes: #2746
The Libre Graph spec declares webUrl on driveItem but
cs3ResourceToDriveItem never set it. Build the /f/<resource-id>
permalink off the configured public base URL (same source drives.go
reads for drive.webUrl) and parse it once at service construction
so callers can use it as a plain field.
Set EnableRemoteLinkPicker: true in CheckFileInfo response so
Collabora Online exposes its "Smart Picker" UI for link
insertion. Clicking it sends a UI_PickLink postMessage to the WOPI
host, which is expected to reply with Action_InsertLink carrying the
URL of the selected file. Follows the same pattern as the existing
EnableInsertRemoteImage / EnableInsertRemoteFile flags.
Signed-off-by: Pedro Pinto Silva <pedro.silva@collabora.com>
Previously the Supervisor was shutdonw as soon as the main context was
canceled. Which cause the managed services to fail during shutdown (in
"trapShutdownCtx()") as the Supervisor was gone already.
We now pass a separate Context to Supervisor.ServeBackground() to avoid
this.
Fixes: #2282
The `id` property of the `permissions` on a space root does not
longer have that special `u:<userid>` format any. It now has the
same format as the permission id on "normal" driveItems.
This gets us rid of quite a bit of special casing for space permission.
Also provides us with "real" permission IDs instead of those faked
"u:<userid>" ones.
Instead of doing a lexical sort we sort the drive/space in a more
"natural" order so that e.g. "Space 2" is sorted before "Space 10".
Related: https://github.com/opencloud-eu/web/issues/2430
Bring the membership lookup in line with the existing repo convention
for set types (see services/thumbnails/pkg/thumbnail/mimetypes.go for
the same pattern). Storing struct{} values instead of bool makes the
set semantics explicit and rules out accidental false entries.
The FIXME pointed at #2632 (dotted keys in KQL property restrictions),
which is now merged. Use audio.artist — the originally intended target
field for this regression — so the test matches its name: a nested
string field that is not on the lowercase allowlist.
Copilot review pointed out that the comment claimed pre-lowercasing
makes non-analyzed query types (wildcard, fuzzy) match for every
allowlisted field. That is true for Name/Tags/Favorites, whose
lowercaseKeyword analyzer emits a single lowercased token, but the
Content analyzer also stems terms — so the guarantee doesn't hold
there. Drop the specific claim and keep the comment to the intent:
stay consistent with the field's analyzer.
The bleve compiler lowercased every query value (except Hidden)
before handing it to the engine. This matched the index tokens
for fields whose analyzer folds case — Name, Tags, Favorites,
Content — but silently broke matching for every other field,
whose default keyword analyzer preserves case. A query like
Title:"Some Title" parsed fine, lowercased to "some title", and
missed the indexed token "Some Title".
Replace the blanket lowercasing with an allowlist of the four
fields whose index mapping actually uses a lowercasing analyzer.
Every other field now passes through unchanged, which keeps
values like "deadmau5" or "Motörhead" intact instead of
normalising them to a case the tag writer didn't choose.
Address review feedback: now that the flag is read under its
registered name `force-rescan`, line the local variable up with the
operator-facing vocabulary. The proto field `ForceReindex` is left
untouched so the wire format stays the same.
The `opencloud search index` command registers the flag as
`--force-rescan` (see pflag registration below) but reads it via
`GetBool("force-reindex")`, so the value is always false — passing
`--force-rescan` had no effect and no force rescan was ever triggered.
Read the flag under its registered name.
Address review feedback: a straight int64 cast truncates toward zero,
so Tika values that produce results like 1234.999... millisecond would
land at 1234 ms instead of 1235 ms. Round before casting so durations
are as accurate as float64 allows.
Tika emits xmpDM:duration as seconds in floating-point form (for
example "154.57379150390625"), so strconv.ParseInt rejected every
value and the field was silently dropped — every indexed audio item
ended up without a duration.
Parse the value with strconv.ParseFloat and convert to milliseconds
ourselves. Adjust the existing extractor test to cover the fractional
case.
Four cases covering the distinct grammar paths touched by the
new Key rule: dotted key in a TextPropertyRestrictionNode,
multi-level key (stressing the Kleene part), dotted key in a
GroupNode (the k:Key? branch), and a dot on the value side
that must stay a literal character.
Introduce a dedicated Key rule (Char+ ("." Char+)*) so property
restriction keys can contain dots. Queries like
audio.artist:Motörhead or photo.cameraMake:Apple now parse.
Goal: keep the data structure aligned with the query namespace.
A driveItem exposes nested fields as dotted paths in its API
response (photo.cameraMake, location.latitude); users should be
able to query with the same path, without admin-configured slot
mappings or alias tables like Microsoft requires.
Both bleve and OpenSearch already treat dots as the nesting
separator for field paths and pass unknown keys through
unchanged — only the KQL grammar blocked dotted keys via
Char <- [A-Za-z]. The new Key rule lifts that restriction; no
compiler changes or field-alias maintenance is needed.
Values are unaffected — dots in values still parse as literal
characters.
That can be helpful when the search service configuration has changed,
e.g. by enabling TIKA. Previously files that had already been indexed
were not indexed again and thus were no part of the fulltext index.
Fixes#2285Fixes#2578
This is to reduce the number of "proxy->gateway->users->ldap" roundtrips
for the tenant id mapping.
The cache currently has a non-configurable ttl of 10 min.
Related: #2310
When the tenant id coming in via the OIDC claims doesn't match the
tenant id on the provisioned user, a mapping can be configured and
resolved via the reva TenantAPI service (now started as part of the
"users" service).
Closes: #2310
Add UserExtraInfo (avatar + mail) to the WOPI CheckFileInfo response for
authenticated, non-public-share users.
UserExtraInfo format (per Collabora SDK):
https://sdk.collaboraonline.com/docs/advanced_integration.html#userextrainfo
```json
{
"avatar": "http://url/to/user/avatar",
"mail": "user@server.com"
}
```
After this change, CheckFileInfo returns:
```json
{
"BaseFileName": "Pedro-filled-hazcom.docx",
"UserFriendlyName": "Admin",
"UserId": "346364...39323030",
"UserCanWrite": true,
"UserCanRename": true,
"IsAdminUser": true,
"EnableInsertRemoteImage": true,
"EnableInsertRemoteFile": true,
"EnableOwnerTermination": true,
"UserExtraInfo": {
"avatar": "https://host:9300/wopi/avatars/{userID}?access_token={wopiToken}",
"mail": "admin@example.org"
},
"PostMessageOrigin": "https://localhost:9200",
"message": "CheckFileInfo: success"
}
```
Avatars are served via a new /wopi/avatars/{userID} endpoint on the
collaboration service, authenticated by the WOPI token. The endpoint
calls the Graph service directly (bypassing the proxy) using the reva
access token via x-access-token header.
All tests pass:
go test ./services/collaboration/... ./services/graph/... ./services/proxy/...
Signed-off-by: Pedro Pinto Silva <pedro.silva@collabora.com>
* adr: Add thoughts about guest users
The approach to take has not been decide yet. This just reflects
the current research/concept work.
* adr(guests): incorporate review feedback
* Mark ADR as postponed for now
We'd like to go for a more light-weight approach as discussed in:
https://github.com/opencloud-eu/opencloud/issues/2513
When multi-tenancy is enable we now allow to specify an OIDC claim
against which the tenantid of the user resolved via CS3 apis is matched.
Partial: #2310
Set EnableInsertRemoteFile: true in CheckFileInfo response so Collabora
shows "Insert Multimedia" and "Compare Document" menu entries. This
triggers UI_InsertFile postMessages to the WOPI host, following the same
pattern as the existing EnableInsertRemoteImage flag.
Signed-off-by: Pedro Pinto Silva <pedro.silva@collabora.com>
Remove the ability to look up schools by externalId directly (from LDAP
filters, duplicate checks, and the EducationBackend interface). This
approach was somewhat unclean, we shouldn't add more an more attributes
as keys for direct lookup.
Instead, expose externalId filtering via the OData $filter query
parameter on GET /education/schools, following the same pattern as
for education users.
Related: #1598
This adds support of simple OData filters on the 'education/users'
endpoint. Filters of the type '$filter=<attr> eq <value>' are supported
now for the following educationUser properties:
"displayname", "mail", "userType", "primaryRole" and "externalId"
Closes: #1599
When the userinfo claims store in the usercache is found to be expired,
do not return an error but ignore the cached entry and force a
re-verification of the access token (either via parsing the JWT again or
via a UserInfo lookup).
This is required for setups with non-JWT access tokes where the expiry
date set in the cached claims does not reflect the actual token expiry,
but just the CacheTTL.
Fixes: #1493
This adds the variables 'OC_OIDC_CLIENT_ID' and
'OC_OIDC_CLIENT_SCOPES' as fallbacks for the platform specific settings.
For backwards compatibility with the "old" settings for the 'web'
service we also allow 'WEB_OIDC_CLIENT_ID' and 'WEB_OIDC_SCOPE' for the
"web" platform.
This works the previous commits so that clients can add an addtional
'platform' query parameter to the webfinger request that can be used
to query the oidc client id and list of scopes that the clients need
to use when connecting to the IDP.
This also removes the non-standard issuer relatation introduced in a
previous commit as we can just introduce new relations in the
http://openid.net name space.
For IDP like Authentik that create a separate issuer url per Client
(Application in Authentik's terms) it is suggested to just configure
as single Client and use that id for all platforms (i.e. setting
'WEBFINGER_ANDROID_OIDC_CLIENT_ID', 'WEBFINGER_DESKTOP_OIDC_CLIENT_ID',
'WEBFINGER_IOS_OIDC_CLIENT_ID' and 'WEBFINGER_WEB_OIDC_CLIENT_ID' to
same value.
Related: #2088
Related: https://github.com/opencloud-eu/desktop/issues/246
* fix: Show username in unprivileged search results
`onPremisesSamAccountName` is a mandatory attribute according to the
spec. There's no harm in returning it in the search results also for
unprivileged users.
Fixes: #144
* adapt tests
* adapt tests
* adapt tests
---------
Co-authored-by: Viktor Scharf <v.scharf@opencloud.eu>
be careful, the env:OC_EDITION, env:FRONTEND_EDITION, and conf:edition got removed as part of this commit, no deprecation because the flag is build time only!
description:Use when the user asks to bump, update, or upgrade the OpenCloud web assets/frontend to a specific version (e.g. "bump web to v7.0.0", "update web to v7.1.2"). Covers editing the version files, the single commit, and opening the PR against the right branch.
---
# Bumping Web
## Overview
Bumping web updates the pinned OpenCloud web frontend (assets + UI-test runner) to a tagged release of [opencloud-eu/web](https://github.com/opencloud-eu/web). It touches exactly two files, lands as one commit, and ships as a PR whose body is the web changelog for that version.
- **`gh` (GitHub CLI), authenticated** — every lookup and the PR creation go through `gh api` / `gh pr`. Verify with `gh auth status`; if it fails, ask the user to run `gh auth login` (suggest `! gh auth login` so it runs in-session). Do not proceed without it.
-`gh` needs the system keyring, so run all `gh` commands with the sandbox disabled.
-`git` and `base64` (decoding the changelog) — standard on macOS/Linux.
| `services/web/Makefile` | `WEB_ASSETS_VERSION` | the version tag, e.g. `v7.0.0` |
| `services/web/Makefile` | `WEB_ASSETS_BRANCH` | branch carrying the tag (`main` or `stable-X.Y`) |
| `.woodpecker.env` | `WEB_COMMITID` | full commit sha the tag points to |
| `.woodpecker.env` | `WEB_BRANCH` | same branch as `WEB_ASSETS_BRANCH` |
`WEB_ASSETS_BRANCH` and `WEB_BRANCH` are always the same value.
## Procedure
Let `VERSION` be the requested tag (always normalize to a leading `v`, e.g. `v7.0.0`).
### 1. Resolve the commit sha the tag points to
```bash
gh api repos/opencloud-eu/web/commits/$VERSION --jq '.sha'
```
This full sha is the new `WEB_COMMITID`.
### 2. Determine the branch (`main` vs `stable-X.Y`)
The branch is the line of development that carries the tag. Detect it:
```bash
gh api "repos/opencloud-eu/web/compare/main...$VERSION" --jq '.status'
```
-`identical` or `behind` → the tagged commit is reachable from `main` → use **`main`**.
-`ahead` or `diverged` → the tag lives on a release line → use **`stable-X.Y`** matching the version's major.minor (e.g. `v7.1.2` → `stable-7.1`).
Sanity-check that the stable branch actually exists:
```bash
gh api repos/opencloud-eu/web/branches --paginate --jq '.[].name'| grep -E 'main|stable'
```
Typically a freshly released minor (`v7.1.0`) still sits on `main`, while later patches on an older line (`v7.0.3` after `7.1` exists) sit on `stable-7.0`.
### 3. Fetch the changelog for the PR body
```bash
gh api "repos/opencloud-eu/web/contents/CHANGELOG.md?ref=$VERSION" --jq '.content'| base64 -d
```
Take **only the section for this version**. Match the template: start the PR body at the first content heading (e.g. `### 💥 Breaking changes` / `### 📈 Enhancement`) and drop the `# Changelog` title, the `## [x.y.z] - date` header, and the `### ❤️ Thanks to all contributors!` block. Stop before the next `## [...]` version header.
### 4. Apply the edits
Edit `services/web/Makefile` (`WEB_ASSETS_VERSION`, `WEB_ASSETS_BRANCH`) and `.woodpecker.env` (`WEB_COMMITID`, `WEB_BRANCH`).
### 5. Pick the target branch for the PR
- Major or minor release → target **`main`**.
- Patch release → may need to target a stable branch instead. If the user did not specify, **ask them** which branch to target before continuing.
### 6. Confirm before committing
Show the user the diff of both files and the target branch, and ask them to confirm. Do not commit until they approve.
### 7. Commit, push, open PR
- Create a branch (do not commit on `main`).
- One commit, conventional-commits format, **empty body**:
```
chore: bump web to v7.0.0
```
- PR title: `[full-ci] chore: bump web to v7.0.0` (the commit message prefixed with `[full-ci] `).
- PR base: the branch chosen in step 5.
- PR body: the trimmed changelog from step 3.
- Add the label `Type:Dependencies`.
```bash
gh pr create --base <target-branch> \
--title "[full-ci] chore: bump web to $VERSION" \
--label "Type:Dependencies" \
--body-file <changelog-file>
```
(`gh` commands need the sandbox disabled — they require the system keyring.)
## Quick reference
```bash
VERSION=v7.0.0
gh api repos/opencloud-eu/web/commits/$VERSION --jq '.sha' # WEB_COMMITID
gh api "repos/opencloud-eu/web/compare/main...$VERSION" --jq '.status' # main vs stable
gh api "repos/opencloud-eu/web/contents/CHANGELOG.md?ref=$VERSION" --jq '.content' | base64 -d # changelog
```
## Common mistakes
- Reading `WEB_COMMITID` or the changelog from web `main` instead of from the version tag (`?ref=$VERSION`). Always pin to the tag.
- Leaving `WEB_ASSETS_BRANCH`/`WEB_BRANCH` on `main` for a patch that belongs on a stable line.
- Forgetting `[full-ci] ` in the PR title or adding a commit body.
- Committing before the user confirms the diff and target branch.
* [ ] QA: [Smoke test](???) on Web Office (Collabora, Onlyoffice, Microsoft office)
* [ ] QA: Smoke test Hello extension
* [ ] QA: [Smoke test](???) ldap
* [ ] QA: Collecting errors found
* [ ] QA: check docs german translation
* [ ] QA: german translations desktop at 100%
* [ ] QA: exploratory testing
### After QA Phase
### Collected bugs
* [ ] Please place all bugs found here
* [ ] Brief company-wide heads up via mail @tbsbdr
* [ ] Create list of changed ENV vars and send to release-coordination@opencloud.eu
* [ ] Variable Name
* [ ] Introduced in version
* [ ] Default Value
* [ ] Description
* [ ] dependencies with user other components
* [ ] DEV: Create branch `release-x.x.x`
* [ ] DEV: bump OpenCloud version in necessary files
* [ ] DEV: `pkg/version/version.go`
* [ ] DEV: `sonar-project.properties`
* [ ] DEV: released deployment versions
* [ ] DEV: prepare changelog folder in `changelog/x.x.x_???`
* [ ] Release Notes + Breaking Changes @tbsbdr
* [ ] Migration + Breaking Changes Admin Doc @???
* [ ] DEV: Create final signed tag
* [ ] DEV: Check successful CI run on `vx.y.z` tag / BLOCKING for all further activity
* [ ] Merge release notes
### After QA Phase (IT related)
### Post-release communication
* [ ]DEV: Create a `docs-stable-x.y` branch based on the docs folder in the OpenCloud repo @micbar
* [ ] DEV/QA: Ping documentation in RC about the new release tag (for opencloud/helm chart version bump in docs)
* [ ]DEV/QA: Ping marketing to update all download links (download mirrors are updated at the full hour, wait with ping until download is actually available)
* [ ] DEV/QA: Ping @??? once the demo instances are running this release
* [ ]DEV: Merge back release branch
* [ ]DEV: Create stable-x.y branch in the OpenCloud repo from final tag
// user id of "admin", for user creation and admin role assignement
"OC_ADMIN_USER_ID":"some-admin-user-id-0000-000000000000",// FIXME currently must have the length of a UUID, see reva/pkg/storage/utils/decomposedfs/spaces.go:228
// admin user default password
"IDM_ADMIN_PASSWORD":"admin",
// system user
"OC_SYSTEM_USER_ID":"some-system-user-id-000-000000000000",// FIXME currently must have the length of a UUID, see reva/pkg/storage/utils/decomposedfs/spaces.go:228
- Prevent personal space creation for service- and lightweight users [[#2876](https://github.com/opencloud-eu/opencloud/pull/2876)]
- chore: bump reva to 2.46.1 [[#2869](https://github.com/opencloud-eu/opencloud/pull/2869)]
- fix: Send SSE events for SpaceCreated/-Disabled/-Deleted [[#2851](https://github.com/opencloud-eu/opencloud/pull/2851)]
- Only try to limit search to spaces if there's a space id to limit to [[#2834](https://github.com/opencloud-eu/opencloud/pull/2834)]
- fix(init): Only log admin password if it was generated [[#2839](https://github.com/opencloud-eu/opencloud/pull/2839)]
- fix: translations for activities and others [[#2836](https://github.com/opencloud-eu/opencloud/pull/2836)]
- fix-2824. run tests without remote.php [[#2826](https://github.com/opencloud-eu/opencloud/pull/2826)]
### 📈 Enhancement
- chore: bump web to v7.1.0 [[#2870](https://github.com/opencloud-eu/opencloud/pull/2870)]
### 📚 Documentation
- docs(adr): Remove erroneous mention of kanidm [[#2783](https://github.com/opencloud-eu/opencloud/pull/2783)]
### 📦️ Dependencies
- build(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.43.0 to 1.44.0 [[#2855](https://github.com/opencloud-eu/opencloud/pull/2855)]
- build(deps-dev): bump license-checker-rseidelsohn from 4.4.2 to 5.0.1 in /services/idp [[#2854](https://github.com/opencloud-eu/opencloud/pull/2854)]
- build(deps-dev): bump cldr from 7.9.0 to 8.0.0 in /services/idp [[#2853](https://github.com/opencloud-eu/opencloud/pull/2853)]
- build(deps): bump i18next from 26.1.0 to 26.3.0 in /services/idp [[#2849](https://github.com/opencloud-eu/opencloud/pull/2849)]
- build(deps-dev): bump sass-loader from 16.0.8 to 17.0.0 in /services/idp [[#2845](https://github.com/opencloud-eu/opencloud/pull/2845)]
- build(deps): bump google.golang.org/grpc from 1.80.0 to 1.81.1 [[#2848](https://github.com/opencloud-eu/opencloud/pull/2848)]
- build(deps): bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc from 0.68.0 to 0.69.0 [[#2847](https://github.com/opencloud-eu/opencloud/pull/2847)]
- build(deps-dev): bump typescript from 5.9.3 to 6.0.3 in /services/idp [[#2846](https://github.com/opencloud-eu/opencloud/pull/2846)]
- build(deps-dev): bump postcss-loader from 4.3.0 to 8.2.1 in /services/idp [[#2830](https://github.com/opencloud-eu/opencloud/pull/2830)]
- build(deps): bump github.com/riandyrn/otelchi from 0.12.2 to 0.12.3 [[#2814](https://github.com/opencloud-eu/opencloud/pull/2814)]
- build(deps-dev): bump workbox-webpack-plugin from 7.4.0 to 7.4.1 in /services/idp [[#2781](https://github.com/opencloud-eu/opencloud/pull/2781)]
- Persist space memberships in share manager [[#2760](https://github.com/opencloud-eu/opencloud/pull/2760)]
- [feature/guest-links] bump reva, add service user config to "sharing" service [[#2735](https://github.com/opencloud-eu/opencloud/pull/2735)]
### 🔒 Security
- fix: disallow thumbnails for tiff and jpeg2000 images [[#2758](https://github.com/opencloud-eu/opencloud/pull/2758)]
### 🐛 Bug Fixes
- fix(notifications): don't re-escape email vars for each recipient [[#2805](https://github.com/opencloud-eu/opencloud/pull/2805)]
- fix: remove unnecessary error log it the oidc access token verify method is set to none [[#2795](https://github.com/opencloud-eu/opencloud/pull/2795)]
- fix(debug): drop duplicate service field from probe fallback log [[#2786](https://github.com/opencloud-eu/opencloud/pull/2786)]
- No registry lookup in cli [[#2755](https://github.com/opencloud-eu/opencloud/pull/2755)]
- fix(webdav): register chi REPORT method in init to avoid race with settings [[#2712](https://github.com/opencloud-eu/opencloud/pull/2712)]
- fix: use runner to start activitylog service [[#2748](https://github.com/opencloud-eu/opencloud/pull/2748)]
- docs(search): fix force-rescan flag name in README [[#2747](https://github.com/opencloud-eu/opencloud/pull/2747)]
### ✅ Tests
- [full-ci] preview-tests. update fixtures for different processors [[#2767](https://github.com/opencloud-eu/opencloud/pull/2767)]
- test: modify exclude list and add coverage upload [[#2762](https://github.com/opencloud-eu/opencloud/pull/2762)]
- fix: cleaner debounce timer test [[#2743](https://github.com/opencloud-eu/opencloud/pull/2743)]
### 📚 Documentation
- Update README with LDAP certificate details [[#2759](https://github.com/opencloud-eu/opencloud/pull/2759)]
### 📈 Enhancement
- feat(graph): populate driveItem.webUrl per Libre Graph spec [[#2744](https://github.com/opencloud-eu/opencloud/pull/2744)]
### 📦️ Dependencies
- build(deps): bump github.com/go-jose/go-jose/v3 from 3.0.4 to 3.0.5 [[#2798](https://github.com/opencloud-eu/opencloud/pull/2798)]
- build(deps): bump golang.org/x/image from 0.38.0 to 0.40.0 [[#2740](https://github.com/opencloud-eu/opencloud/pull/2740)]
- build(deps): bump github.com/tidwall/gjson from 1.18.0 to 1.19.0 [[#2750](https://github.com/opencloud-eu/opencloud/pull/2750)]
- build(deps-dev): bump dotenv-expand from 12.0.3 to 13.0.0 in /services/idp [[#2710](https://github.com/opencloud-eu/opencloud/pull/2710)]
- build(deps): bump github.com/onsi/ginkgo/v2 from 2.28.1 to 2.28.3 [[#2739](https://github.com/opencloud-eu/opencloud/pull/2739)]
- enhancement: increase display size of graph flow diagram [[#2620](https://github.com/opencloud-eu/opencloud/pull/2620)]
### 📦️ Dependencies
- build(deps): bump go.opentelemetry.io/contrib/zpages from 0.67.0 to 0.68.0 [[#2666](https://github.com/opencloud-eu/opencloud/pull/2666)]
- build(deps): bump @types/node from 22.19.17 to 25.6.0 in /services/idp [[#2687](https://github.com/opencloud-eu/opencloud/pull/2687)]
- build(deps): bump go.opentelemetry.io/otel/exporters/stdout/stdouttrace from 1.42.0 to 1.43.0 [[#2601](https://github.com/opencloud-eu/opencloud/pull/2601)]
- build(deps): bump github.com/davidbyttow/govips/v2 from 2.17.0 to 2.18.0 [[#2656](https://github.com/opencloud-eu/opencloud/pull/2656)]
- build(deps): bump i18next from 25.10.10 to 26.0.4 in /services/idp [[#2609](https://github.com/opencloud-eu/opencloud/pull/2609)]
- build(deps): bump github.com/testcontainers/testcontainers-go/modules/opensearch from 0.41.0 to 0.42.0 [[#2645](https://github.com/opencloud-eu/opencloud/pull/2645)]
- build(deps): bump github.com/open-policy-agent/opa from 1.15.1 to 1.15.2 [[#2602](https://github.com/opencloud-eu/opencloud/pull/2602)]
- build(deps): bump github.com/nats-io/nats.go from 1.49.0 to 1.50.0 [[#2587](https://github.com/opencloud-eu/opencloud/pull/2587)]
- build(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.42.0 to 1.43.0 [[#2586](https://github.com/opencloud-eu/opencloud/pull/2586)]
- chore: bump reva to latest main [[#2584](https://github.com/opencloud-eu/opencloud/pull/2584)]
- build(deps): bump golang.org/x/image from 0.36.0 to 0.38.0 [[#2581](https://github.com/opencloud-eu/opencloud/pull/2581)]
- build(deps-dev): bump css-minimizer-webpack-plugin from 7.0.4 to 8.0.0 in /services/idp [[#2551](https://github.com/opencloud-eu/opencloud/pull/2551)]
- build(deps): bump github.com/go-ldap/ldap/v3 from 3.4.12 to 3.4.13 [[#2526](https://github.com/opencloud-eu/opencloud/pull/2526)]
- build(deps): bump github.com/open-policy-agent/opa from 1.14.1 to 1.15.0 [[#2535](https://github.com/opencloud-eu/opencloud/pull/2535)]
- build(deps): bump github.com/nats-io/nats-server/v2 from 2.12.5 to 2.12.6 [[#2525](https://github.com/opencloud-eu/opencloud/pull/2525)]
- build(deps-dev): bump postcss-preset-env from 10.1.3 to 11.2.0 in /services/idp [[#2392](https://github.com/opencloud-eu/opencloud/pull/2392)]
- build(deps): bump github.com/tus/tusd/v2 from 2.8.0 to 2.9.2 [[#2485](https://github.com/opencloud-eu/opencloud/pull/2485)]
- build(deps): bump google.golang.org/grpc from 1.79.2 to 1.79.3 [[#2519](https://github.com/opencloud-eu/opencloud/pull/2519)]
- build(deps): bump github.com/nats-io/nats-server/v2 from 2.12.4 to 2.12.5 [[#2499](https://github.com/opencloud-eu/opencloud/pull/2499)]
- build(deps): bump github.com/russellhaering/goxmldsig from 1.5.0 to 1.6.0 [[#2503](https://github.com/opencloud-eu/opencloud/pull/2503)]
- build(deps): bump golang.org/x/net from 0.51.0 to 0.52.0 [[#2472](https://github.com/opencloud-eu/opencloud/pull/2472)]
- build(deps): bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc from 0.65.0 to 0.67.0 [[#2473](https://github.com/opencloud-eu/opencloud/pull/2473)]
- build(deps): bump github.com/olekukonko/tablewriter from 1.1.3 to 1.1.4 [[#2468](https://github.com/opencloud-eu/opencloud/pull/2468)]
- build(deps): bump go.opentelemetry.io/contrib/zpages from 0.65.0 to 0.67.0 [[#2467](https://github.com/opencloud-eu/opencloud/pull/2467)]
- build(deps): bump github.com/testcontainers/testcontainers-go/modules/opensearch from 0.40.0 to 0.41.0 [[#2458](https://github.com/opencloud-eu/opencloud/pull/2458)]
- build(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.41.0 to 1.42.0 [[#2459](https://github.com/opencloud-eu/opencloud/pull/2459)]
- build(deps): bump github.com/testcontainers/testcontainers-go from 0.40.0 to 0.41.0 [[#2453](https://github.com/opencloud-eu/opencloud/pull/2453)]
- build(deps): bump golang.org/x/oauth2 from 0.35.0 to 0.36.0 [[#2452](https://github.com/opencloud-eu/opencloud/pull/2452)]
- build(deps): bump go.opentelemetry.io/otel/exporters/stdout/stdouttrace from 1.40.0 to 1.42.0 [[#2441](https://github.com/opencloud-eu/opencloud/pull/2441)]
- build(deps): bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp from 0.65.0 to 0.67.0 [[#2442](https://github.com/opencloud-eu/opencloud/pull/2442)]
- build(deps): bump github.com/open-policy-agent/opa from 1.13.2 to 1.14.0 [[#2427](https://github.com/opencloud-eu/opencloud/pull/2427)]
- build(deps): bump go.opentelemetry.io/otel from 1.40.0 to 1.41.0 [[#2425](https://github.com/opencloud-eu/opencloud/pull/2425)]
- build(deps): bump github.com/davidbyttow/govips/v2 from 2.16.0 to 2.17.0 [[#2420](https://github.com/opencloud-eu/opencloud/pull/2420)]
- build(deps): bump github.com/nats-io/nats.go from 1.48.0 to 1.49.0 [[#2390](https://github.com/opencloud-eu/opencloud/pull/2390)]
- build(deps): bump golang.org/x/net from 0.50.0 to 0.51.0 [[#2412](https://github.com/opencloud-eu/opencloud/pull/2412)]
- build(deps): bump github.com/kovidgoyal/imaging from 1.8.19 to 1.8.20 [[#2391](https://github.com/opencloud-eu/opencloud/pull/2391)]
- build(deps): bump github.com/grpc-ecosystem/grpc-gateway/v2 from 2.27.7 to 2.28.0 [[#2375](https://github.com/opencloud-eu/opencloud/pull/2375)]
- build(deps): bump github.com/open-policy-agent/opa from 1.13.1 to 1.13.2 [[#2374](https://github.com/opencloud-eu/opencloud/pull/2374)]
- build(deps): bump google.golang.org/grpc from 1.78.0 to 1.79.1 [[#2362](https://github.com/opencloud-eu/opencloud/pull/2362)]
- build(deps): bump github.com/onsi/ginkgo/v2 from 2.28.0 to 2.28.1 [[#2366](https://github.com/opencloud-eu/opencloud/pull/2366)]
- build(deps): bump go.opentelemetry.io/contrib/zpages from 0.64.0 to 0.65.0 [[#2363](https://github.com/opencloud-eu/opencloud/pull/2363)]
- build(deps): bump golang.org/x/net from 0.49.0 to 0.50.0 [[#2356](https://github.com/opencloud-eu/opencloud/pull/2356)]
- build(deps): bump github.com/go-resty/resty/v2 from 2.17.1 to 2.17.2 [[#2355](https://github.com/opencloud-eu/opencloud/pull/2355)]
- build(deps): bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp from 0.64.0 to 0.65.0 [[#2321](https://github.com/opencloud-eu/opencloud/pull/2321)]
- build(deps): bump github.com/open-policy-agent/opa from 1.12.3 to 1.13.1 [[#2350](https://github.com/opencloud-eu/opencloud/pull/2350)]
- fix: fix typo in variable description [[#2333](https://github.com/opencloud-eu/opencloud/pull/2333)]
- fix: include sessionID in sse logout event [[#2327](https://github.com/opencloud-eu/opencloud/pull/2327)]
- fix: fix typo in gateway service documentation [[#2332](https://github.com/opencloud-eu/opencloud/pull/2332)]
- Sanitize web config only once [[#2286](https://github.com/opencloud-eu/opencloud/pull/2286)]
### 📈 Enhancement
- external tenant id [[#2258](https://github.com/opencloud-eu/opencloud/pull/2258)]
### 📚 Documentation
- fix: make file urls [[#2304](https://github.com/opencloud-eu/opencloud/pull/2304)]
### 📦️ Dependencies
- build(deps): bump github.com/gabriel-vasile/mimetype from 1.4.12 to 1.4.13 [[#2316](https://github.com/opencloud-eu/opencloud/pull/2316)]
- build(deps): bump go.opentelemetry.io/otel/exporters/stdout/stdouttrace from 1.39.0 to 1.40.0 [[#2279](https://github.com/opencloud-eu/opencloud/pull/2279)]
- update reva after merge #514 [[#2309](https://github.com/opencloud-eu/opencloud/pull/2309)]
- build(deps): bump github.com/go-chi/chi/v5 from 5.2.4 to 5.2.5 [[#2278](https://github.com/opencloud-eu/opencloud/pull/2278)]
- [tests-only] test: wait post-processing to finish for MKCOL requests [[#2092](https://github.com/opencloud-eu/opencloud/pull/2092)]
- [tests-only] test: fix API tests [[#2087](https://github.com/opencloud-eu/opencloud/pull/2087)]
- [full-ci] use graph api in the enforcePasswordPublicLink.feature [[#2050](https://github.com/opencloud-eu/opencloud/pull/2050)]
- [full-ci][tests-only] test: check last email content with retries as emails can be delayed [[#2038](https://github.com/opencloud-eu/opencloud/pull/2038)]
- skip collaborativePosix tests in CI [[#2039](https://github.com/opencloud-eu/opencloud/pull/2039)]
- build(deps): bump github.com/olekukonko/tablewriter from 1.1.2 to 1.1.3 [[#2186](https://github.com/opencloud-eu/opencloud/pull/2186)]
- build(deps): bump github.com/grpc-ecosystem/grpc-gateway/v2 from 2.27.4 to 2.27.5 [[#2204](https://github.com/opencloud-eu/opencloud/pull/2204)]
- build(deps): bump github.com/go-resty/resty/v2 from 2.7.0 to 2.17.1 [[#2197](https://github.com/opencloud-eu/opencloud/pull/2197)]
- build(deps): bump github.com/open-policy-agent/opa from 1.11.1 to 1.12.3 [[#2166](https://github.com/opencloud-eu/opencloud/pull/2166)]
- build(deps): bump github.com/kovidgoyal/imaging from 1.8.18 to 1.8.19 [[#2167](https://github.com/opencloud-eu/opencloud/pull/2167)]
- build(deps): bump github.com/grpc-ecosystem/grpc-gateway/v2 from 2.27.3 to 2.27.4 [[#2164](https://github.com/opencloud-eu/opencloud/pull/2164)]
- build(deps): bump github.com/sirupsen/logrus from 1.9.4-0.20230606125235-dd1b4c2e81af to 1.9.4 [[#2163](https://github.com/opencloud-eu/opencloud/pull/2163)]
- build(deps): bump github.com/go-chi/chi/v5 from 5.2.3 to 5.2.4 [[#2162](https://github.com/opencloud-eu/opencloud/pull/2162)]
- build(deps): bump go.opentelemetry.io/contrib/zpages from 0.63.0 to 0.64.0 [[#2158](https://github.com/opencloud-eu/opencloud/pull/2158)]
- build(deps): bump github.com/blevesearch/bleve/v2 from 2.5.5 to 2.5.7 [[#2157](https://github.com/opencloud-eu/opencloud/pull/2157)]
- build(deps): bump go.opentelemetry.io/otel/exporters/stdout/stdouttrace from 1.38.0 to 1.39.0 [[#2154](https://github.com/opencloud-eu/opencloud/pull/2154)]
- build(deps): bump golang.org/x/image from 0.34.0 to 0.35.0 [[#2153](https://github.com/opencloud-eu/opencloud/pull/2153)]
- build(deps): bump github.com/nats-io/nats.go from 1.47.0 to 1.48.0 [[#2147](https://github.com/opencloud-eu/opencloud/pull/2147)]
- build(deps): bump github.com/onsi/ginkgo/v2 from 2.27.2 to 2.27.5 [[#2148](https://github.com/opencloud-eu/opencloud/pull/2148)]
- build(deps): bump github.com/olekukonko/tablewriter from 1.1.1 to 1.1.2 [[#2144](https://github.com/opencloud-eu/opencloud/pull/2144)]
- build(deps): bump github.com/spf13/cobra from 1.10.1 to 1.10.2 [[#2141](https://github.com/opencloud-eu/opencloud/pull/2141)]
- build(deps): bump golang.org/x/net from 0.48.0 to 0.49.0 [[#2140](https://github.com/opencloud-eu/opencloud/pull/2140)]
- build(deps): bump github.com/onsi/gomega from 1.38.2 to 1.39.0 [[#2133](https://github.com/opencloud-eu/opencloud/pull/2133)]
- build(deps): bump golang.org/x/crypto from 0.46.0 to 0.47.0 [[#2132](https://github.com/opencloud-eu/opencloud/pull/2132)]
- build(deps): bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp from 0.63.0 to 0.64.0 [[#2109](https://github.com/opencloud-eu/opencloud/pull/2109)]
- build(deps): bump github.com/kovidgoyal/imaging from 1.8.17 to 1.8.18 [[#2107](https://github.com/opencloud-eu/opencloud/pull/2107)]
- build(deps): bump google.golang.org/grpc from 1.77.0 to 1.78.0 [[#2106](https://github.com/opencloud-eu/opencloud/pull/2106)]
- build(deps): bump go.opentelemetry.io/otel/sdk from 1.38.0 to 1.39.0 [[#2069](https://github.com/opencloud-eu/opencloud/pull/2069)]
- build(deps): bump github.com/opensearch-project/opensearch-go/v4 from 4.5.0 to 4.6.0 [[#2068](https://github.com/opencloud-eu/opencloud/pull/2068)]
- build(deps): bump github.com/testcontainers/testcontainers-go/modules/opensearch from 0.39.0 to 0.40.0 [[#1967](https://github.com/opencloud-eu/opencloud/pull/1967)]
- build(deps): bump golang.org/x/net from 0.47.0 to 0.48.0 [[#2061](https://github.com/opencloud-eu/opencloud/pull/2061)]
- build(deps): bump github.com/open-policy-agent/opa from 1.10.1 to 1.11.0 [[#1930](https://github.com/opencloud-eu/opencloud/pull/1930)]
@@ -40,7 +40,7 @@ but it should be easily transferable to other (sub)projects.
> **Note:** Please don't file an issue to ask a question. You'll get faster results by using the resources below.
For general questions, please refer to [OpenCloud's FAQs](https://opencloud.eu/faq/) or check the [project page](https://github.com/opencloud-eu) for communication channels.
For general questions, please refer to [OpenCloud's FAQs](https://docs.opencloud.eu/docs/admin/resources/faq/) or check the [project page](https://github.com/opencloud-eu) for communication channels.
## What to know before getting started
@@ -55,7 +55,7 @@ The OpenCloud project follows the strict GitHub workflow of development as brief
### OpenCloud Company, Engineering Partners and Community
OpenCloud is largely created by developers who are employed by the [OpenCloud company](https://opencloud.eu), which is located in Germany.
It is providing support for OpenCloud for customers mainly in the EU. In addition, there are engineering partners who also work full-time on OpenCloud related code, for example, on the component [REVA](https://github.com/cs3org/reva/).
It is providing support for OpenCloud for customers mainly in the EU. In addition, there are engineering partners who also work full-time on OpenCloud related code, for example, on the component [REVA](https://github.com/opencloud-eu/reva/).
Because of that fact, the pace that the development is moving forward is sometimes high for people who are not willing and/or able to spend a comparable amount of time to contribute.
Even though this can be a challenge, it should not scare anybody away. Here is our clear commitment that we feel honored by everybody who is interested in our work and improves it, no matter how big the contribution might be.
@@ -113,7 +113,7 @@ Explain the problem and include additional details to help maintainers reproduce
Provide more context by answering these questions:
***Did the problem start happening recently** (e.g. after updating to a new version) or was this always a problem?
* If the problem started happening recently, **can you reproduce the problem in an older version?** What's the most recent version in which the problem doesn't happen? You can find more information about how to set up [test environments](https://docs.opencloud.eu/devel/testing) in the [developer documentation](https://docs.opencloud.eu/docs/dev/intro).
* If the problem started happening recently, **can you reproduce the problem in an older version?** What's the most recent version in which the problem doesn't happen? You can find more information about how to set up in the [Getting Started guide](https://docs.opencloud.eu/docs/admin/getting-started).
***Can you reliably reproduce the issue?** If not, provide details about how often the problem happens and under which conditions it normally happens.
Include details about your configuration and environment as asked for in the template.
@@ -146,15 +146,13 @@ Enhancement suggestions are tracked as [GitHub issues](https://guides.github.com
Unsure where to begin contributing to OpenCloud? You can start by looking through these `Needs-help` issues:
* The [Good first issue](https://github.com/opencloud-eu/opencloud/labels/Topic%3Agood-first-issue) label marks good items to start with.
*[Tests needed](https://github.com/opencloud-eu/opencloud/labels/Interaction%3ANeeds-tests) - issues which would benefit from a test.
* [Help wanted issues](https://github.com/opencloud-eu/opencloud/labels/Interaction%3ANeeds-help) - issues which should be a bit more involved.
* The [Good first issue](https://github.com/opencloud-eu/opencloud/labels/Type%3Agood-first-issue) label marks good items to start with.
*The [Feature Request](https://github.com/opencloud-eu/opencloud/issues?q=state%3Aopen%20label%3AType%3AFeature-Request) label lists features the community would like to see implemented.
It is fine to pick one of the lists following personal preference.
While not perfect, the number of comments is a reasonable proxy for the impact a given change will have.
To find out how to set up OpenCloud for local development, please refer to the [Developer Documentation](https://docs.opencloud.eu/docs/dev/web/getting-started).
It contains a lot of information that will come in handy when starting to work on the project.
To find out how to set up OpenCloud for local development, please refer to the [Developer Documentation](https://docs.opencloud.eu/docs/dev/web/getting-started) for the web side, and the general server [README](https://github.com/opencloud-eu/opencloud/blob/main/README.md) for backend setup. Both contain information that will come in handy when starting to work on the project.
### Pull Requests
@@ -221,7 +219,7 @@ To help you find issues and pull requests, each label can be used in search link
The labels are loosely grouped by their purpose, but it's not required that every issue has a label from every group or that an issue can't have more than one label from the same group.
The list here contains all the more general categories of issues which are followed by a colon and a specific value.
For example, severity 1 looks like `Severity:sev1-critical`.
For example, severity 1 looks like `Priority:p1-urgent`.
#### Platform
@@ -257,7 +255,7 @@ Categorizes the issue to also indicate the type of the issue.
#### Status
The status in the ticket life cycle. Keep an eye on that one, especially for the `Waiting-for-Feedback` tag which might indicate that the reporter is asked for feedback.
The status in the ticket life cycle. Keep an eye on that one, especially for the `Status:Needs-Review` tag which might indicate that the reporter is asked for feedback.
The mechanics are currently to mount a different configuration file depending on the environment, as we support two scenarios that are described in [`services/groupware/DEVELOPER.md`](../../../../../services/groupware/DEVELOPER.md):
* «production» setup, with OpenLDAP and Keycloak containers
* «homelab» setup, with the built-in IDM (LDAP) and IDP that run as part of the `opencloud` container
The Docker Compose setup (in [`stalwart.yml`](../../stalwart.yml)) mounts either [`idmldap.toml`](./idmldap.toml) or [`ldap.toml`](./ldap.toml) depending on how the variable `STALWART_AUTH_DIRECTORY` is set, which is either `idmldap` for the homelab setup, or `ldap` for the production setup.
This is thus all done automatically, but whenever changes are performed to Stalwart configuration files, they must be reflected across those two files, to keep them in sync, as the only entry that should differ is this one:
In a groupware environment, not every user will always use the OpenCloud UI to read their emails, some will resort to other [MUAs (Mail User Agents)](https://en.wikipedia.org/wiki/Email_client) that support a subset of features, use older protocols (IMAP, POP, SMTP, CalDAV, CardDAV) and lesser authentication methods (basic authentication). Those email clients will talk to Stalwart directly, as opposed to the OpenCloud UI which will make use of APIs of the OpenCloud Groupware service, since those protocols are provided by Stalwart and implementing them in OpenCloud would offer very little benefits, but definitely a lot of (almost completely) unnecessary effort.
Those protocols and operations that bypass the OpenCloud UI also need to be authenticated, this in and by Stalwart, and we need to find the best fitting approach that fulfills most or all of the following constraints:
### Single Provisioning
We want to avoid multiple provisioning of users, groups, passwords and other resources as much as possible.
While it is possible to have e.g. OpenCloud's user management also perform [Management API](https://stalw.art/docs/category/management-api/) calls, one still inevitably ends up in situations where users, user passwords, or other resources are not in sync, which becomes complex to debug and fix, and should thus be avoided if possible.
To do so, we should strive to have a single source of truth regarding users, their passwords, and similar resources and attributes such as groups, roles, application passwords, etc...
### Attack Detection
Coordinated attacks such as [denial of service](https://en.wikipedia.org/wiki/Denial-of-service_attack) attempts don't necessarily focus on a single protocol but are commonly multi-pronged, e.g. by brute forcing the [OIDC API](https://www.keycloak.org/docs/latest/authorization_services/index.html#token-endpoint), the OpenCloud Groupware API, IMAP and SMTP, \*DAV protocols, etc...
In order to detect those as well as to quickly react by blacklisting clients that are identified to attempt such attacks, it is useful to have a single authentication service for all the components of the system, all protocols, all clients (e.g. [PowerDNS Weakforced](https://github.com/PowerDNS/weakforced), [Nauthilus](https://nauthilus.org/), ...)
Furthermore, such services typically make use of [DNSBL/RBL services](https://en.wikipedia.org/wiki/Domain_Name_System_blocklist) that allow IP addresses of botnets to be blocked across many services of many providers as a shared defense mechanism.
As a bonus, a centralized authentication component can also provide metrics and observability capabilities across all those protocols.
### Custom Authentication Implementations
Some customers might want custom authentication implementations to integrate with their environment, in which case we would want those to be done once and in the technology stack we're all most familiar with (thus as a service in Go in the OpenCloud framework, and not e.g. a Lua script in Nauthilus, or a Rust plugin in Stalwart, etc...)
## Decision Drivers
TODO
*
## Considered Options
First off, here is a brief explanation of each of the scenarios that we potentially or absolutely need to support, which we will explore for each implementation option:
* MUAs with basic authentication
* these are external mail clients (Thunderbird, Apple Mail, ...) with which users authenticate using legacy protocols (IMAP, POP3, SMTP) and their primary username and password in clear text (encrypted through the mandatory use of TLS)
* MUAs with application password authentication
* these are external mail clients (Thunderbird, Apple Mail, ...) with which users authenticate using legacy protocols (IMAP, POP3, SMTP) and one of the application passwords that they created in the OpenCloud UI, which is a useful security mechanism as it reduces the attack surface when one such password is leaked or discovered
* MUAs with SASL bearer token authentication
* these are more modern external mail clients (Thunderbird) with which users authenticate using legacy protocols (IMAP, POP3, SMTP) but more secure OIDC token based authentication (SASL OAUTHBEARER or SASL XOAUTH2), which closely resembles the OIDC authentication used by the OpenCloud UI towards the OpenCloud backends
* JMAP clients with basic authentication
* modern mail clients (Thunderbird) that speak the JMAP protocol over HTTP and authenticate using their primary username and password in clear text (encrypted through the use of HTTPS)
* JMAP clients with bearer token authentication
* modern mail clients (Thunderbird) that speak the JMAP protocol over HTTP and authenticate using an OIDC token (JWT) obtained from an IDP (typically KeyCloak)
* OpenCloud Groupware with master authentication
* the OpenCloud UI client uses APIs from the OpenCloud Groupware backend (and authenticates using OIDC)
* the OpenCloud Groupware backend, in turn, performs JMAP operations with Stalwart, and authenticates using Stalwart's shared secret master authentication protocol
* OpenCloud Groupware with generated token authentication
* the OpenCloud UI client uses APIs from the OpenCloud Groupware backend (and authenticates using OIDC)
* the OpenCloud Groupware backend, in turn, performs JMAP operations with Stalwart, and authenticates against Stalwart using bearer authentication with JWTs that it generates itself
* in the future, that JWT might also be the JWT that the OpenCloud UI used to authenticate against the OpenCloud Groupware in the first place
### Stalwart with the LDAP Directory
```mermaid
flowchart LR
c(client)
s(Stalwart)
l(LDAP)
c -- IMAP/SMTP --> s
c -- JMAP --> s
s -- LDAP --> l
```
Clients authenticate directly against Stalwart, that is configured to use an LDAP authentication Directory.
An LDAP server (e.g. OpenLDAP) is needed as part of the infrastructure.
OpenCloud also has to make use of the same LDAP server.
* ✅ MUAs with basic authentication
* MUAs authenticate directly against Stalwart
* Stalwart's LDAP Directory plugin supports plain text authentication by looking up the userPassword attribute in the LDAP server
* ❌ MUAs with application password authentication
* MUAs authenticate directly against Stalwart
* Stalwart's LDAP Directory plugin does not support application password as it is hardwired to look up the password in the userPassword attribute in the LDAP server
* even if it did support looking up alternative passwords in LDAP, this would hardly be practical as the application passwords are currently created and stored in OpenCloud, which would need to be modified to store them in LDAP in the first place
* ❌ MUAs with SASL bearer token authentication
* MUAs authenticate directly against Stalwart
* Stalwart's LDAP Directory plugin does not support verifying OIDC tokens
* ✅ JMAP clients with basic authentication
* JMAP clients authenticate directly against Stalwart
* Stalwart's LDAP Directory plugin supports plain text authentication by looking up the userPassword attribute in the LDAP server
* ❌ JMAP clients with bearer token authentication
* JMAP clients authenticate directly against Stalwart
* Stalwart's LDAP Directory plugin does not support verifying OIDC tokens
* ✅ OpenCloud Groupware with master authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* Stalwart detects and supports clear text password master authentication regardless of the Directory that is being used, and verifies it against the shared secret password that is configured in the server
* ❌ OpenCloud Groupware with generated token authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* Stalwart's LDAP Directory plugin does not support verifying OIDC tokens
### Stalwart with the OIDC Directory
```mermaid
flowchart LR
c(client)
s(Stalwart)
o(IDP)
c -- IMAP/SMTP --> s
c -- JMAP --> s
s -- OIDC HTTP --> o
```
Clients authenticate directly against Stalwart, that is configured to use an OIDC authentication Directory.
An OIDC IDP (server) is needed as part of the infrastructure, e.g. KeyCloak.
Optionally, an LDAP server (e.g. OpenLDAP) might be used as well, and KeyCloak would look up users and their credentials in LDAP.
OpenCloud also has to make use of the same LDAP server, or would need to be modified to be capable of only making use of an OIDC IDP (which would include limitations that are yet to be resolved, e.g. the option of using KeyCloak Admin APIs to retreieve groups, group members, ...)
* ❌ MUAs with basic authentication
* MUAs authenticate directly against Stalwart
* Stalwart's OIDC Directory plugin does not support plain text authentication
* ❌ MUAs with application password authentication
* MUAs authenticate directly against Stalwart
* Stalwart's OIDC Directory plugin does not support application passwords
* ❓ MUAs with SASL bearer token authentication
* MUAs authenticate directly against Stalwart
* Stalwart's OIDC Directory plugin does not currently support external IDPs, but is expected to in future versions
* as of Stalwart 0.12, this would only work if Stalwart itself is used as the IDP when acquiring a token
* ❌ JMAP clients with basic authentication
* JMAP clients authenticate directly against Stalwart
* Stalwart's OIDC Directory plugin does not support plain text authentication
* ❓ JMAP clients with bearer token authentication
* JMAP clients authenticate directly against Stalwart
* Stalwart's OIDC Directory plugin does not currently support external IDPs, but is expected to in future versions
* as of Stalwart 0.12, this would only work if Stalwart itself is used as the IDP when acquiring a token
* ✅ OpenCloud Groupware with master authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* Stalwart detects and supports clear text password master authentication regardless of the Directory that is being used, and verifies it against the shared secret password that is configured in the server
* ❓ OpenCloud Groupware with generated token authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* Stalwart's OIDC Directory plugin does not currently support external IDPs, but is expected to in future versions
* as of Stalwart 0.12, this would only work if Stalwart itself is used as the IDP when acquiring a token, which is not the case with this approach as the tokens are generated by the Groupware backend itself
### Stalwart with the Internal Directory
```mermaid
flowchart LR
c(client)
s(Stalwart)
c -- IMAP/SMTP --> s
c -- JMAP --> s
```
Clients authenticate directly against Stalwart, that is configured to use an Internal authentication Directory.
Neither an OIDC IDP nor an LDAP server are needed as part of the infrastructure, as principal resources (users, groups) and their credentials exist in Stalwart's storage.
OpenCloud would not be capable of accessing those resources, which means that provisioning of groups, users, user passwords must be duplicated and kept in sync between Stalwart and OpenCloud.
* ✅ MUAs with basic authentication
* MUAs authenticate directly against Stalwart
* Stalwart's Internal Directory plugin supports plain text authentication
* users are able to create those themselves using the self-service web UI of Stalwart
* they are not shared with the OpenCloud application passwords though and would need to be provisioned into Stalwart when created in OpenCloud to provide a single UI
* ❌ MUAs with SASL bearer token authentication
* MUAs authenticate directly against Stalwart
* Stalwart's Internal Directory plugin does not support OIDC token authentication
* ✅ JMAP clients with basic authentication
* JMAP clients authenticate directly against Stalwart
* Stalwart's Internal Directory plugin supports plain text authentication
* ❌ JMAP clients with bearer token authentication
* JMAP clients authenticate directly against Stalwart
* Stalwart's Internal Directory plugin does not support OIDC token authentication
* ✅ OpenCloud Groupware with master authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* Stalwart detects and supports clear text password master authentication regardless of the Directory that is being used, and verifies it against the shared secret password that is configured in the server
* ❌ OpenCloud Groupware with generated token authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* Stalwart's Internal Directory plugin does not support OIDC token authentication
### Stalwart with the OpenCloud Authentication API
```mermaid
flowchart LR
c(client)
s(Stalwart)
o(OpenCloud)
l(LDAP)
c -- IMAP/SMTP --> s
c -- JMAP --> s
s -- REST --> o
o -- LDAP --> l
```
Clients authenticate directly against Stalwart, that is configured to use an "External" authentication Directory, that is yet to be developed. (warning)
Its protocol is currently not defined, but not particularly relevant at this time, as long as it supports accepting basic and bearer authentication in order to authenticate both username and password credentials as well as OIDC tokens.
That External Directory implementation forwards the basic or bearer credentials to an endpoint in the OpenCloud backend, that the responds with whether the authentication is successful or not, as well as with additional information that is needed for Stalwart (email address, display name, groups, roles, ...)
* ✅ MUAs with basic authentication
* MUAs authenticate directly against Stalwart
* Stalwart's External Directory supports plain text authentication by relaying the authentication operation to the OpenCloud backend, which can then authenticate users by username and password using an LDAP server
* note that this option requires having an LDAP server in the environment, including having it accessible by OpenCloud
* if that is not the case, then a viable option is also to support OIDC tokens and application passwords
* to clarify: this scenario is only about supporting authentication using the "primary" username and password
* ✅ MUAs with application password authentication
* MUAs authenticate directly against Stalwart
* Stalwart's External Directory supports application password authentication by relaying the authentication operation to the OpenCloud backend, which can then authenticate against its list of application passwords
* this is the ideal scenario for application passwords, since they are already supported by OpenCloud, and can be created and managed using the OpenCloud UI
* relaying the authentication operation to OpenCloud also prevents the need for duplicate provisioning of application passwords
* ✅ MUAs with SASL bearer token authentication
* MUAs authenticate directly against Stalwart
* Stalwart's External Directory supports OIDC token authentication by relaying the authentication operation to the OpenCloud backend, which can then either perform local token inspection and authentication by verifying the token's signature, or use the OIDC IDP's token introspection endpoint
* ✅ JMAP clients with basic authentication
* JMAP clients authenticate directly against Stalwart
* Stalwart's External Directory supports plain text authentication by relaying the authentication operation to the OpenCloud backend, which can then authenticate users by username and password using an LDAP server
* the same limitations/requirements as for the "MUAs with basic authentication" scenario apply here as well
* ✅ JMAP clients with bearer token authentication
* MUAs authenticate directly against Stalwart
* Stalwart's External Directory supports OIDC token authentication by relaying the authentication operation to the OpenCloud backend, which can then either perform local token inspection and authentication by verifying the token's signature, or use the OIDC IDP's token introspection endpoint
* ✅ OpenCloud Groupware with master authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* Stalwart detects and supports clear text password master authentication regardless of the Directory that is being used, and verifies it against the shared secret password that is configured in the server
* ✅ OpenCloud Groupware with generated token authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* in the worst case, the External Directory plugin in Stalwart would also perform a forwarding of the authentication operation to OpenCloud, which would obviously be able to verify a token it has created
* an optimization might be possible here, if the External Directory implementation permits for the configuration of specific issuers which should then be verifying against a JWK set directly, whereas the fallback behaviour would be to query the OpenCloud Authentication API
### Stalwart with Nauthilus and LDAP
```mermaid
flowchart LR
c(client)
s(Stalwart)
n(Nauthilus)
l(LDAP)
c -- IMAP/SMTP --> s
c -- JMAP --> s
s -- REST --> n
n -- LDAP --> l
```
In this scenario, we introduce the [Nauthilus authentication service](https://nauthilus.org/), which has its own API but also a KeyCloak integration plugin.
It supports various backends and can also be scripted for more complex combinations.
⚠️ It would require the implementation of a Stalwart Nauthilus Directory, **that is yet to be developed**.
We do not make use of any OpenCloud Authentication API but, instead, attempt to have everything go through Nauthilus instead, backed by an LDAP server that then contains the users, groups, and user passwords.
The upside of using Nauthilus is that it does brute force attack detection and can provide metrics across multiple protocols and clients in a centralized fashion.
* ✅ MUAs with basic authentication
* MUAs authenticate directly against Stalwart
* Stalwart's Nauthilus Directory supports plain text authentication by relaying the authentication operation to Nauthilus, e.g. using its JSON API
* Nauthilus provides a response that contains user attributes from LDAP (display name, email addresses, ...)
* ❓ MUAs with application password authentication
* Nauthilus has no support for application passwords in itself
* a Lua plugin could potentially be used in Nauthilus to detect whether the clear text password matches a regular expression for application passwords and, if that is the case, first attempt to verify it through an API call (that does not exist yet) to the OpenCloud backend, but that would definitely be more complex and less elegant than having a single API
* ❓ MUAs with SASL bearer token authentication
* it is currently unclear whether Nauthilus supports OIDC token authentication
* ✅ JMAP clients with basic authentication
* Stalwart's Nauthilus Directory supports plain text authentication by relaying the authentication operation to Nauthilus, e.g. using its JSON API
* Nauthilus provides a response that contains user attributes from LDAP (display name, email addresses, ...)
* ❓ JMAP clients with bearer token authentication
* it is currently unclear whether Nauthilus supports OIDC token authentication
* ✅ OpenCloud Groupware with master authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* Stalwart detects and supports clear text password master authentication regardless of the Directory that is being used, and verifies it against the shared secret password that is configured in the server
* ❓ OpenCloud Groupware with generated token authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* it is currently unclear whether Nauthilus supports OIDC token authentication
* an optimization might be possible here, if the Nauthilus Directory implementation permits for the configuration of specific issuers which should then be verifying against a JWK set directly, whereas the fallback behaviour would be to query the Nauthilus API, but that does sound like a stretch to fit into the concept
### Stalwart with Nauthilus and an OpenCloud Authentication API
```mermaid
flowchart LR
c(client)
j(client)
s(Stalwart)
n(Nauthilus)
o(OpenCloud)
l(LDAP)
k(Keycloak)
c -- IMAP/SMTP --> s
j -- JMAP --> s
s -- REST --> n
subgraph internal auth
n -- REST --> o
o -- LDAP --> l
o -- OIDC --> k
end
```
This option also makes use of the [Nauthilus authentication service](https://nauthilus.org/), but instead of it using LDAP to resolve users, we would either make use of its Lua scripting abilities to implement a backend that performs HTTP calls to an OpenCloud Authentication API, or implement an additional Nauthilus backend that uses the Nauthilus API to delegate to another instance, which would then be the OpenCloud Authentication API with support for the Nauthilus API.
⚠️ As with the previous option, it would require the implementation of a Stalwart Nauthilus Directory, **that is yet to be developed**.
Interestingly, if the OpenCloud Authentication API follows the Nauthilus API, this scenario can easily be degraded by dropping Nauthilus and, instead, having all services talk to the OpenCloud Authentication API directly.
* ✅ MUAs with basic authentication
* MUAs authenticate directly against Stalwart
* Stalwart's Nauthilus Directory supports plain text authentication by relaying the authentication operation to Nauthilus, e.g. using its JSON API
* Nauthilus provides a response that contains user attributes from LDAP (display name, email addresses, ...)
* ✅ MUAs with application password authentication
* Nauthilus would forward the authentication request to the OpenCloud Authentication API, which would support application passwords
* ❓ MUAs with SASL bearer token authentication
* it is currently unclear whether Nauthilus supports OIDC token authentication and whether it would be able to forward such requests to the OpenCloud Authentication API
* ✅ JMAP clients with basic authentication
* Stalwart's Nauthilus Directory supports plain text authentication by relaying the authentication operation to Nauthilus, e.g. using its JSON API
* Nauthilus then forwards that request to the OpenCloud Authentication API
* the OpenCloud Authentication API, and then Nauthilus, provides a response that contains user attributes from LDAP (display name, email addresses, ...) or claims from the JWT
* ❓ JMAP clients with bearer token authentication
* it is currently unclear whether Nauthilus supports OIDC token authentication and whether it would be able to forward such requests to the OpenCloud Authentication API
* ✅ OpenCloud Groupware with master authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* Stalwart detects and supports clear text password master authentication regardless of the Directory that is being used, and verifies it against the shared secret password that is configured in the server
* ❓ OpenCloud Groupware with generated token authentication
* the OpenCloud Groupware backend authenticates directly against Stalwart
* it is currently unclear whether Nauthilus supports OIDC token authentication and whether it would be able to forward such requests to the OpenCloud Authentication API
> [!IMPORTANT]
> We need to clarify whether the Nauthilus API allows for a JWT to be submitted for the authentication request, and not only username and password – not to secure the request in itself, but to forward an OIDC token based authentication attempt as part of the payload.
### Comparing Options
| | MUA basic | MUA app password | MUA sasl | JMAP clients with basic auth | JMAP clients with JWT auth | Groupware Middleware with master auth | Groupware Middleware with JWT auth |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Stalwart 0.12 with LDAP Directory | ✅ MUA → Stalwart | ❌ not supported with LDAP | ❌ not supported with LDAP | ✅ | ❌ | ✅ | ❌ |
| Stalwart 0.12 with Internal Directory | ✅ MUA → Stalwart, must be provisioned in Stalwart | ✅ MUA → Stalwart, must be provisioned in Stalwart | ❌ | ❌ | ❌ unless using Stalwart as IDP | ✅ | ❌ |
Should the Groupware backend be an independent microservice or be part of the OpenCloud single binary framework?
The OpenCloud backend is built on a framework that
* implements token based authentication between services
* allows for a "single binary" deployment mode that runs all services within that one binary
* integrates services such as a NATS event bus
This decision is about whether the Groupware backend service should be implemented within that framework or, instead, be implemented as a standalone backend service.
## Decision Drivers
* single binary deployment strategy is potentially important (TODO how important is it really? stakeholders:?)
## Considered Options
* have the Groupware Middleware as an independent microservice
* have the Groupware Middleware implemented within the existing OpenCloud framework
## Decision Outcome
TODO
### Consequences
TODO
### Confirmation
TODO
## Pros and Cons of the Options
### Independent Microservice
* (potentially) good: be free from technical decisions made for the existing OpenCloud stack, to avoid carrying potential technical baggage
* (potentially) good: make use of a framework that is more fitting for the tasks the Groupware backend needs to accomplish
* bad: re-implement framework components that already exist, with the need to maintain those in two separate codebases, or the added complexity of a shared library repository
* bad: not have the ability to include the Groupware backend in the single binary deployment
* neutral: a separate code repository and delivery for the Groupware backend, which might or might not be of advantage
* neutral: may be implemented on a completely different technology stack, including the programming language
### Part of the framework
* good: fit into the opinionated choices that were made for the OpenCloud framework so far
* good: many aspects are already implemented in the current framework and can be made use of, potentially enhanced for the needs of the Groupware backend
* good: the ability to include the Groupware backend in the single binary deployment
* neutral: be in the same code repository and part of the same delivery as other services in OpenCloud
* neutral: must be implemented in Go on top of the same technology stack
Which semantic and technical approach to take in order to provide strong integration of the various products and capabilities of OpenCloud, OpenTalk, and potentially other products as well?
## Decision Drivers
* a strong integration that allows users to access resources and relationships without having to switch views, which translates into a "mental switch" as well
* an innovative approach that differs from the traditional way groupware applications have been designed in the past
* TODO more decision drivers from PM
* a model that is open and generic enough to integrate many different types of resources and relationships
* a model that allows for independent and incremental upgrades to the resources and relationships that can be contributed by each service
## Considered Options
* resource linking
* application launchers
* TODO? can we come up with more ideas?
## Decision Outcome
TODO
### Consequences
TODO
### Confirmation
TODO
## Pros and Cons of the Options
### Resource Linking
This concept primarily resides on the idea of having resources, which have attributes, and relations between them, pretty much as [RDF (Resource Description Framework)](https://www.w3.org/RDF/) does, where the Groupware backend provides services to explore relations of a given resource.
* good: decoupling of UI, backends as well as other participants, as backends can gradually evolve the relationships and resources they understand and can contribute to over time, as well as for the UI that may just silently ignore resources it does not support yet or does not want to present to the user
* good: potential for an asynchronous architecture that would enable the UI to present some resources early without having to wait for those that require more processing time or are provided by services that happen to be under heavier load
* good: it should provide ammunition for a modern and original UI that is centered around resources and relationships rather than the usual visual paradigms
* bad: it might be a challenge to implement this approach in a performant way with rapid response times, as it could cause additional complexity and storage services (e.g. to denormalize reverse indexes, cache expensive resource graphs, etc...)
#### URNs
Each resource has a unique identifier, for which [URNs (Uniform Resource Names)](https://www.rfc-editor.org/rfc/rfc1737) seem the best representation.
Whenever the user puts a resource into focus in the OpenCloud Groupware UI (i.e. by selecting/clicking that resource, e.g. the sender of an email), it may send a request to the Groupware service API to inquire about related resources.
What those related resources are still stands to be determined, but examples could be along the lines of
* unread emails from the same sender
* emails exchanged with that sender in the last 7 days
* files recently shared with that user
* spaces or groups in common with that user
* OpenTalk meetings planned within the next 3 days
In order to decouple the Groupware service from which resources and relations are supported,
* whenever such an exploration request is received, the Groupware service forwards it to all known services, in a "fan-out" model
* each service can understand the focused resource, or not, but if it does it may return related resources that it is capable of providing using its data model (e.g. OpenTalk providing related meeting resources, OpenCloud Groupware providing related calendar events, contacts, mails, etc...)
* ideally, that happens in an asynchronous fashion, using e.g. [SSE (Server Side Events)](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) to push results to the OpenCloud UI to avoid having to wait for the slowest contributor, although that pushes the "reduce" part of this ["map-reduce" operation](https://en.wikipedia.org/wiki/MapReduce) to the client
This allows a decoupling of all the participants, enabling each service to add, remove or alter relationships that it is able to contribute for a given resource type.
Obviously, the UI needs to be able to understand resource types to know how to represent them, but if it silently ignores resource types that it does not know of, backends can evolve independently from the UI.
#### JSON-LD
[JSON-LD (JSON for Linking Data)](https://json-ld.org/) seems like a potent representation format for those relationships in a REST environment.
Have a UI that is comprised of multiple more-or-less separate applications, with an application launcher bar, with each application being an icon in itself in that launcher.
Similar to what e.g. Google does, or Open-Xchange App Suite.
* bad: does not make for an integrated application paradigm since users still have to context switch between those applications/views to perform tasks
To allow collaboration with external Users (Users that don't yet have an
account in the IDP, and might be external to the organization), it should
be possible to invite "Guest Users" into an OpenCloud instance.
## Requirements
- the audit trail of the external user accessing the resource needs to
be maintained, that means sharing via a password protected public link
is not sufficient as access to that one is tracked as if the creator
of the link accessed the resource
- external users need to be authenticated just like "normal" users, when
accessing the shared resource (including the possibility to use 2FA)
- the ability to invite external users is tied to a separate permission
(e.g. "can invite guest users")
- make it work with all (most) of the user-management configurations we support.
The built-in IDP (lico) does not need to be supported.
- avoid creating "Shadow IT" Infrastructure, e.g. we don't want to
create/maintain a separate IDP instance just for Guest User that would
allow bypassing corporate rules for Identity Management
- the process of inviting a guest must can be asynchrounous, i.e. the user
account of the guest user might not be created at the moment of
creating the share/sending the invitaion as the whole process crosses
multiple systems (OpenCloud, Identity Management System, Email) and might
even require manual steps.
- It should be possible to "convert" a guest user into a "normal" user without
the user loosing their shares.
- Guest user invitations should have an expiration date, after which they can
no longer be accepted.
### Privileges of guest users
- guest users can not share or invite other users to a space or create public
links. (primary focus of the feature is to provide a simple way to grant
external, authorized access. anything else like resharing would undermine
regular user accounts).
- guestusers can use the desktop and mobile client to access their shares or spaces
- all "normal" users are able to share with guest users, just as if they where "normal" users.
## Questions still to be answered
- what's the life cycle of a guest user?
- Who's responsible for deprovisioning?
- Do guest users expire after a certain time?
- Do we need to keep track of who invited whom and when? (not just in
the audit log?)
- What if the user already exists but used a different mail address in
his account (e.g. sub-addressing?).
## Obstacles
### UserIDs
- Every user in OpenCloud needs to have a userid assigned
- Sharing, as many other features, needs that userid for storing the
share (share service) and for assigning the grants on the shared
resources (storage provider)
- When an external IDP is used the generation of that userid is usually
not in control of OpenCloud (exception User-Autoprovisioning, or when
the Provisioning/Education API is used). In that case, the userid is
taken from a LDAP Attribute maintained in the external system
### Lots of identity management options
- OpenCloud provides many different ways to consume user-accounts. Guest
users are supposed to be working with all/most of them:
- External IDP, with external LDAP service
- External IDP, with manual provisioning via the
Education/Provisioning APIs (to a local OpenCloud specific LDAP
service) - e.g. in multi-tenant setups
- External IDP, with User-Autoprovisioning (also to a local OpenCloud
specific LDAP service)
- everything in-between and outside of the above
- Each of these options have different ways for user-provisioning and in
the way userids are generated and managed
### How do we keep track of invitations?
- Completely rely on external system?
- Track creation and acceptance of invitations somehow?
- Do invitation expire at some point?
## Possible solutions
### Re-vitalize the PoC implementation of the invitations service and finalize it (<https://github.com/opencloud-eu/opencloud/blob/main/services/invitations/README.md>)
- Implements parts of the MSGraph Invitation Specification
* [Stalwart](https://stalw.art/), contains not only mail but also collaborative features in an integrated package
* traditional IMAP/POP/SMTP stacks (e.g. [Dovecot](https://www.dovecot.org/) + [Postfix](https://www.postfix.org/))
## Decision Outcome
The decision was made to go with Stalwart, as it reduces the implementation effort on our end, allowing us for a much faster time-to-market with a significantly smaller team of developers.
### Consequences
We will most probably not need to develop much of a calendar or contact stack ourselves, as Stalwart is planning to implement those as part of the upcoming [JMAP](https://jmap.io/spec-core.html) specifications for [contacts](https://jmap.io/spec-contacts.html) and [calendars](https://jmap.io/spec-calendars.html).
The Groupware API will largely consist of a translation of high-level operations for the UI into JMAP operations sent to Stalwart.
#### Risks
On the flip side, there are a number of risks associated with that decision.
* Stalwart underdelivers on its promises
* calendaring provides insufficient features for our implementation (e.g. event series handling being too basic)
* not scaling for large deployments
* necessary adaptations (e.g. for authentication integration) are rejected upstream
* etc...
### Confirmation
TODO
## Pros and Cons of the Options
### Stalwart
* good: integrated package that contains IMAP/POP, SMTP, anti-spam, AI, encryption at rest and many other features in one
* good: modern stack
* good: capable of fault tolerance in large deployments through its use of [FoundationDB](https://www.foundationdb.org/)
* bad: relatively new project with few to no large scale productive deployments (yet)
* bad: significant human [SPoF](https://en.wikipedia.org/wiki/Single_point_of_failure)/[bus factor](https://en.wikipedia.org/wiki/Bus_factor) issue as the development team currently consists of one
* good: supports and drives the JMAP protocol ([JMAP Core](https://jmap.io/spec-core.html), [JMAP Mail](https://jmap.io/spec-mail.html), [JMAP Contacts](https://jmap.io/spec-contacts.html), [JMAP Calendars](https://jmap.io/spec-calendars.html), [JMAP Tasks](https://jmap.io/spec-tasks.html), ...),
* which provides more high-level operations that we don't need to implement ourselves,
* as well as a much cleaner specification that reduces efforts too,
* and additionally can be implemented with an efficient stateless HTTP I/O stack
* bad: no viable broad JMAP implementation alternatives in case Stalwart does not deliver ([Apache James](https://james.apache.org/) only seems to support a basic subset of JMAP)
* good: implements a lot of Groupware "business logic" on its own, reducing the implementation effort on our end,
* most notably by not having to deal with IMAP extensions and quirks,
* or the complexity of calendar events
### IMAP/SMTP
* good: there are a number of alternatives in case a specific implementation does not deliver
* good: the best implementation candidates are well-established, used in large amounts of productive deployments, supported by teams of developers
* bad: more complex stack composed of numerous components as opposed to an all-in-one implementation
* bad: the effort and complexity of having to deal with IMAP,
* its complexity due to its extensions and its many quirks,
* as well as a significantly less efficient I/O stack that requires stateful session handling
* bad: requires the complete implementation of contacts, calendars and tasks in our own stack, as none of those services are provided by IMAP/SMTP backends
We need a comprehensive HTTP API for the OpenCloud Web UI to provide access to the following (upcoming) modules and Groupware functionalities:
* Mail
* Contacts
* Calendar
* Tasks
* Chat
* Configuration
```mermaid
graph LR
subgraph clients
ui(OpenCloud UI)
muas(Other<br>MUAs)
end
subgraph Backend
subgraph OpenCloud
direction TB
groupware("OpenCloud<br>Groupware")
drive("OpenCloud<br>Drive")
end
stalwart(Stalwart)
end
subgraph Storage
drive_storage[(drive<br>storage)]
stalwart_metadata[(metadata<br>storage)]
stalwart_storage[(object<br>storage)]
end
ui x@==>|?|groupware
x@{ animate: true }
ui-->|Graph|drive
muas-->|IMAP,SMTP,*DAV|stalwart
groupware-->drive
groupware-->|JMAP|stalwart
drive-->drive_storage
stalwart-->stalwart_metadata
stalwart-->stalwart_storage
```
Additionally, the API must also be able to provide information about related resources and their relationships, as outlined in [the Resource Linking ADR](./0003-groupware-resource-linking.md).
For the OpenCloud Drive services, the communication between UI client and backend services is performed via the [LibreGraph API](https://github.com/opencloud-eu/libre-graph-api), which is based on [Microsoft Graph](https://developer.microsoft.com/en-us/graph). The goal of this ADR is **not** to question or change that decision, and the choice of an option is merely for the communication with the Groupware backend.
Communication between the OpenCloud Groupware and Stalwart will make use of the [JMAP (JSON Meta Application Protocol) protocol](https://jmap.io/spec-mail.html).
The API for the OpenCloud Web UI is **not** supposed to be an abstraction of that and thus may use JMAP data formats.
Other [MUAs (Mail User Agents)](https://en.wikipedia.org/wiki/Email_client) converse directly with Stalwart using [IMAP](https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol) or [POP3](https://en.wikipedia.org/wiki/Post_Office_Protocol), [SMTP](https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol), [CalDAV](https://en.wikipedia.org/wiki/CalDAV), [CardDAV](https://en.wikipedia.org/wiki/CardDAV), or JMAP itself.
This ADR concerns the decision regarding which API approach/process/technology/specification to use, not the details of the data model and such, which will need to be fleshed out following the requirements and priorities of the OpenCloud UI Client development, regardless of the selected approach.
## Decision Drivers
### UI Driven
The decision must be significantly driven by the OpenCloud UI Client developers, since they are the primary consumers of the API.
They will also be the sole consumers for a foreseeable while until the OpenCloud Groupware UI reaches a stable feature-complete milestone, which is the earliest point in time for the APIs to be considered stable as well and potentially be consumed by third parties.
Backend developers are stakeholders in that aspect as well though, as the choice of API approach has an impact on the complexity, costs and maintainability of the backend services as well.
### Economic Awareness
Reduction of complexity and implementation efforts, albeit not at all costs, and not only on the short run.
It is obviously of advantage when an option requires less implementation, or less complexity in its implementation.
### Efficiency
Regarding efficiency, the goal is to design an API that is tailored to providing responsiveness ([pagination](https://apisyouwonthate.com/blog/api-design-basics-pagination/), [SSEs (Server-Side Events)](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events), ...) and good network performance.
The latter is achieved by minimizing the number of roundtrips between the client and the servers, which, in turn, is typically achieved through the use of higher level APIs as opposed to a granular API that provides more flexibility but also, by its very nature, requires the combination of multiple request-response roundtrips over the wire.
### Third Party Consumption
We are assuming that the APIs are public APIs (not just technically) and may be consumed by SDKs and third parties.
Implications are that care must be put into providing an API that is stable, versioned, has a changelog, and potentially provided as a product with [LTS (Long-term Support)](https://en.wikipedia.org/wiki/Long-term_support) options.
This also hints at the necessity of a capability exchange/discovery protocol between clients and the Groupware backend, as we will have different versions of clients and servers in the wild, and they need to be able to understand each other. Crucially, if locally running clients are developed, they can go a long time without being updated.
## Considered Options
* [LibreGraph](#libregraph)
* [JMAP](#jmap)
* [custom REST API](#custom-rest-api) (albeit potentially based on standards, at least partially)
## Decision Outcome
The decision was made to go with the custom REST implementation option, mainly due to
* the use of LibreGraph providing little benefits
* if would provide us with a fleshed out API for groupware
* but we would not implement it fully
* and it is really an API for Outlook and Exchange, not a generic groupware standard
* furthermore, a significant blocker is that it does not provide for a way to support multiple accounts for a user
* the experience of implementing and using the LibreGraph API for the Drive components has made light of some challenges that we would not like to repeat
* using JMAP directly
* is a very interesting option in terms of standards, as it is an RFC,
* but we currently see that approach as too risky as per the potential complexity of parsing payloads of JMAP commands and their backreferences, plugging those across commands that must be forwarded as-is to Stalwart and others that need to be handled by the Groupware middleware itself, but also the potential need to reverse engineer the high-level meaning of chained low-level JMAP commands in order to implement enrichment, caches, reverse indexes, etc...
* however, it might be a better path forward in the future, especially if JMAP becomes a viable option for replacing the current use of LibreGraph as well
### Consequences
* we will need to design an API on our own, from scratch, albeit maximally making use of JMAP data structures
* that API will need to be maintained as a product, with documentation, versioning, LTS
## Pros and Cons of the Options
* [LibreGraph](#proscons-libregraph)
* [JMAP](#proscons-jmap)
* [Custom REST API](#proscons-custom)
### <a id="proscons-libregraph"/>LibreGraph
[LibreGraph](https://github.com/opencloud-eu/libre-graph-api) is an API specification that is heavily inspired by and based on [Microsoft Graph](https://developer.microsoft.com/en-us/graph), of which it is a partial implementation, but also with modifications where necessary.
Example:
```text
GET /v1.0/me/messages?$select=sender,subject&$count=50&$orderby=received
```
#### Good
* is already in use as the API for OpenCloud Drive operations, with a small stack to use it in the OpenCloud Web UI
* provides an API and data model that has already been thought out and used in production (albeit with only few different implementations)
#### Neutral
* does not have to follow the Microsoft Graph API, can be customized to our own needs, but in which case it becomes doubtful that there is any benefit in mimicking the Graph API in the first place if we diverge from it
* there is no compatibility benefit
* the only MUA that uses the Microsoft Graph API is Microsoft Outlook, and it is not a goal to support Microsoft Outlook as a MUA beyond standard IMAP/SMTP/CalDAV/CardDAV services (and that would be Microsoft Graph, not LibreGraph nor any customizations we would require)
* we will not implement all of the Microsoft Graph API
* we will not implement parts of the Microsoft Graph API as-is either, but will require to make modifications
* if there is a requirement for considering that API as a public API for third party integrators, then the API also needs to be documented, maintained, versioned, and kept stable as much as possible (this is neutral because it is a requirement that exists with every option)
#### Bad
* not an easy API to implement
* although we have libraries that take care of some of the more complex parts, such as parsing [OData](https://www.odata.org/) expressions
* really only easy to use when backed by a relational database and an object relational mapping framework using [ASP.NET](https://dotnet.microsoft.com/en-us/apps/aspnet) or [JPA](https://en.wikipedia.org/wiki/Jakarta_Persistence)/[Hibernate](https://hibernate.org/)
* its data model and peculiar interpretation of REST are really not [idomatic](https://en.wikipedia.org/wiki/HATEOAS) at all, and are clearly the result of reverse engineering the capabilities of Microsoft SQL Server and Exchange into a "standard" from the back, and then Microsoft Outlook's features and capabilities from the front
* not tailored to our needs
* we will most probably have a lot of cases in which we have to twist the Graph API to express what the UI needs
* will require using complex filters, which then require complex parsing in the backend in order to translate them into JMAP
* as opposed to directly using an expressive and maximally matching API in the first place
* we are likely to encounter use-cases that are not covered by the Graph API (especially due to our resource linking approach)
* does not support multiple accounts per user
* would require the addition of an account parameter, as a query parameter or as part of the path, which would make every URL in the API incompatible with Microsoft Graph
* more implementation effort than JMAP
* the JMAP RFCs already provides a data model, and we would end up converting between them all the time, with incompatibilities (Graph has attributes JMAP doesn't, and the other way around)
* possibly (probably?) more implementation effort than a custom REST API, due to its complexity
#### Decision Drivers
* UI Driven
* some members the OpenCloud Web Team strongly prefers not to use LibreGraph due to its complexity and to the fact that we would have to reftrofit operations into an existing API that was designed by a third party
* one upside is that there is already a client stack for performing LibreGraph operations, which could be reused to some degree for the Groupware APIs as well; it does not amount to all that much code though
* Economic Awareness
* more complexity and more effort as the other options due to the inherent complexity of the specification
* a data model is already specified in full, which might save us some time on that front
* although probably not really either since the actual data model we will work with on the backend is prescribed by JMAP, and we will only be looking to map attributes betsween JMAP and LibreGraph
* the data model is not necessarily thorougly documented either, which will leave room for interpretation, also due to incompatibilities between JMAP and Graph
* there will be attributes that are defined in JMAP and that we will receive from Stalwart that will not have a corresponding attribute in Graph (or be a list of values as opposed to a single value), and those will require to either lose some data by squashing it into the Graph data model, or extending the Graph data model which renders us incompatible with it
* Efficiency
* since the API is not tailored to our needs, we are much more likely to end up performing multiple roundtrips for single high level operations
* Third Party Consumption
* for some of the operations, we could point to the Microsoft Graph documentation, although that would not make for a great experience either, we would probably need to replicate it
* our deviations and extensions will have to be maintained just like the other options
* LibreGraph doesn't help with API stability either since
* we don't implement all of it, and need to document what we implement and what we don't,
* won't be compatible either due to modifications (additional parameters, unsupported parameters, different interpretations),
* and will just as equally need to evolve it as the other options, requiring the documentation of changes as well
* will be required to be maintained as a public API
* documentation
* LTS
* versioning
### <a id="proscons-jmap"/>JMAP
[JMAP (JSON Meta Application Protocol)](https://jmap.io/spec.html) is a set of specifications that are codified in RFCs:
To exemplify the JMAP protocol, the following code block is a JMAP request that
* fetches the 30 last received emails from a mailbox (folder)
* the threads of those emails
* email metadata of all of those threads, including a preview
<details open>
<summary>Click here to toggle the display of this example.</summary>
```json
[["Email/query",{
"accountId":"ue150411c",
"filter":{
"inMailbox":"fb666a55"
},
"sort":[{
"isAscending":false,
"property":"receivedAt"
}],
"collapseThreads":true,
"position":0,
"limit":30,
"calculateTotal":true
},"0"],
["Email/get",{
"accountId":"ue150411c",
"#ids":{
"resultOf":"0",
"name":"Email/query",
"path":"/ids"
},
"properties":[
"threadId"
]
},"1"],
["Thread/get",{
"accountId":"ue150411c",
"#ids":{
"resultOf":"1",
"name":"Email/get",
"path":"/list/*/threadId"
}
},"2"],
["Email/get",{
"accountId":"ue150411c",
"#ids":{
"resultOf":"2",
"name":"Thread/get",
"path":"/list/*/emailIds"
},
"properties":[
"threadId",
"mailboxIds",
"keywords",
"hasAttachment",
"from",
"subject",
"receivedAt",
"size",
"preview"
]
},"3"]]
```
</details>
#### Good
* flexible protocol that can easily be implemented by clients
* potentially does not require implementation efforts on the backend side
* would obviously support the full potential of JMAP and Stalwart
* we could potentially extend JMAP with our own data models and operations based on the [JMAP Core Protocol](https://jmap.io/spec-core.html), possibly even propose them as RFCs
* we can start with JMAP request objects that contain only a few or even only one JMAP methods (indicated by the [maxCallsInRequest capability](https://datatracker.ietf.org/doc/html/rfc8620#section-2)), allowing more calls as we need
* clients could implement the funtionality they need using multiple requests in the beginning, then we implement missing functionality on the server
* this would allow us to speed up requests that we need while at the same time giving clients the ability to make any necessary individual calls
* probably only a partially useful approach since chaining JMAP requests is necessary for even the most mundane operations, to avoid the inefficiency of multiple roundtrips
#### Neutral
* the [existing JMAP specifications](https://jmap.io/spec.html) will not cover 100% of the Web UI API needs (e.g. configuration settings[^config], [resource linking](./0003-groupware-resource-linking.md), ...), but that does not prevent us from implementing additional custom APIs, either as non-JMAP REST APIs, or as extensions of JMAP
* we would need to gauge whether JMAP communication
* should occur directly between the OpenCloud UI and Stalwart,
* or whether an OpenCloud Groupware service should be used as an intermediary and as an [anti-corruption layer](https://ddd-practitioners.com/home/glossary/bounded-context/bounded-context-relationship/anticorruption-layer/)
* if there is a requirement for considering that API as a public API for third party integrators, then the API also needs to be documented, maintained, versioned, and kept stable as much as possible (this is neutral because it is a requirement that exists with every option)
[^config]: although Stalwart will most likely have a [JMAP API for application configuration settings as well](https://matrix.to/#/!blIcSTIPwfKMtOEWcg:matrix.org/$CD9C6IZN28bbmN0Arb_Y-RapgsS4XqAqnDgf15yJahM?via=matrix.org&via=mozilla.org&via=chat.opencloud.eu)
> Message from [Mauro](https://github.com/mdecimus):
>
> Hi everyone, I'm curious what you think about standardizing a simple protocol/extension for users to easily manage certain account settings directly from their email clients. For instance, such a protocol could handle:
> * Spam reporting (though not strictly a setting)
> * Calendar-related preferences
> * Encryption-at-rest settings
> * Mail auto-expunge policies
> * ... and potentially more.
>
> My initial thought is to implement this as a JMAP extension rather than inventing another protocol similar to ManageSieve, which feels somewhat like a "Frankenstein" IMAP extension.
>
> Many mailbox providers already offer some or all of these settings through their web interfaces, but a standardized JMAP-based extension could let users adjust these directly within their preferred email clients or via APIs.
#### Bad
* potentially bad: most probably too flexible for its own good, as it makes it difficult to reverse-engineer the high-level meaning of a set of JMAP requests in order to capture its semantics, e.g. to implement caching or reverse indexes for performance
* since the OpenCloud Drive backends use the LibreGraph API, using a JMAP based API for Groupware bears the risk of having multiple APIs to do the same thing, which we need to be careful about, and avoid if possible
> [!NOTE]
> This seems like a mild "bad" item, but the risk risk here is significant: if it turns out that we need to capture the semantics of API requests to perform additional operations (e.g. caching or indexing for performance reasons, or to decorate the data from Stalwart with information from other services), then we would have to re-implement the whole API as JMAP is too complex to parse to extract semantics from.
#### Two Approaches
There are two approaches as to how to implement our protocol based on JMAP:
* either our clients must split JMAP operations and send some to Stalwart, and others to the Groupware backend (depending on which endpoint is in charge of which API)
* or our clients send all the JMAP operations to the Groupware backend, which is then in charge to relay JMAP commands that are to be handled by Stalwart to Stalwart
##### Directly to Stalwart
If the OpenCloud UI Client communicates directly with Stalwart (using JMAP), then
* good: we don't need to implement any sort of "bridge" in the OpenCloud Groupware service (although the implementation effort is likely to be low)
* good: we avoid an additional hop in the network, gaining on performance and potentially on throughput
* bad: it will have to perform additional API requests for data and features that are not provided by Stalwart with the OpenCloud Groupware service (e.g. [Resource Linking](./0003-groupware-resource-linking.md)) as well, which is likely to lead to an increase in the number of network roundtrips
* bad: would be unable to extend the protocol with OpenCloud Groupware specific models and data
* bad: would be unable to implement caching or similar performance improvements if necessary
* bad: prevents us from implementing infrastructure features that are not present in Stalwart and might never be in the way we would need them, e.g. sharding across multi-site redundancy
```mermaid
graph LR
subgraph clients
ui(OpenCloud UI)
muas(Other<br>MUAs)
end
subgraph Backend
subgraph OpenCloud
direction TB
groupware("OpenCloud<br>Groupware")
drive("OpenCloud<br>Drive")
end
stalwart(Stalwart)
end
subgraph Storage
drive_storage[(drive<br>storage)]
stalwart_metadata[(metadata<br>storage)]
stalwart_storage[(object<br>storage)]
end
ui x@==>|JMAP|stalwart
x@{ animate: true }
ui y@==>|JMAP or REST|groupware
y@{ animate: true }
ui-->|Graph|drive
muas-->|IMAP,SMTP,*DAV|stalwart
groupware-->drive
groupware-->|JMAP|stalwart
drive-->drive_storage
stalwart-->stalwart_metadata
stalwart-->stalwart_storage
```
##### Groupware intermediary
Alternatively, if the OpenCloud UI Client exclusively communicates with the OpenCloud Groupware service (using JMAP), then
* good: the OpenCloud Groupware service acts as a anti-corruption layer, which would allow us to
* implement caching and similar performance improvement measures if necessary (e.g. reverse indexing of costly data)
* implement infrastructure features that are not present in Stalwart and might never be in the way we would need them, e.g. sharding across multi-site redundancy
* extend the JMAP protocol
* good: it enables us to minimize network roundtrips between the OpenCloud UI Client and the OpenCloud Groupware backend as there is no need to perform additional requests elsewhere
* bad: we have an additional intermediary hop that "just" relays operations to Stalwart most of the time
* due to Go HTTP stack limitations (lack of zero-copy asynchronous I/O),
* that might incur a cost of "needlessly" copying data in memory
* as well as performing blocking I/O (at the very least since JMAP requests first need to be read in full by te OpenCloud Groupware before they then can be sent to Stalwart more or less as-is, and the same applies to the responses)
```mermaid
graph LR
subgraph clients
ui(OpenCloud UI)
muas(Other<br>MUAs)
end
subgraph Backend
subgraph OpenCloud
direction TB
groupware("OpenCloud<br>Groupware")
drive("OpenCloud<br>Drive")
end
stalwart(Stalwart)
end
subgraph Storage
drive_storage[(drive<br>storage)]
stalwart_metadata[(metadata<br>storage)]
stalwart_storage[(object<br>storage)]
end
ui y@==>|JMAP|groupware
y@{ animate: true }
ui-->|Graph|drive
muas-->|IMAP,SMTP,*DAV|stalwart
groupware-->drive
groupware-->|JMAP|stalwart
drive-->drive_storage
stalwart-->stalwart_metadata
stalwart-->stalwart_storage
```
#### Decision Drivers
* UI Driven
* the UI team did not express any particular preference for this option, but the JMAP protocol is simple to implement on any client
* Economic Awareness
* there would be less of a need to develop an API, but that doesn't put much into the balance
* developing a generic inbound JMAP command processing engine that is capable of resolving backreferences with requests that can be sent out to different backends (Stalwart, Drive, Groupware, OpenTalk, ...) seems risky in terms of complexity, also since Go doesn't have much of a [well-supported Reactive framework](https://github.com/ReactiveX/RxGo)
* Efficiency
* the ability of the JMAP protocol to chain multiple low-level commands provides for a very efficient way to compose higher-level operations without the need for multiple round-trips
* Third Party Consumption
* for some of the operations, we could point to the JMAP documentation and RFCs, although that would not make for a great experience either, we would probably need to replicate it
* our protocol extensions will have to be maintained just like the other options
* will be required to be maintained as a public API
* documentation
* LTS
* versioning
### <a id="proscons-custom"/>Custom REST API
A custom REST API would implement the resources and semantics as they are needed by the UI, and would be strongly if not fully UI-driven.
The data model should remain close or equal to JMAP's, to avoid data loss by converting back and forth.
We might look into existing specifications for formatting JSON payloads, such as [JSON:API](https://jsonapi.org/) or partial ones such as such as [JSON-LD](https://json-ld.org/) for relationships between resources, but this is currently outside of the scope of this ADR.
```mermaid
graph LR
subgraph clients
ui(OpenCloud UI)
muas(Other<br>MUAs)
end
subgraph Backend
subgraph OpenCloud
direction TB
groupware("OpenCloud<br>Groupware")
drive("OpenCloud<br>Drive")
end
stalwart(Stalwart)
end
subgraph Storage
drive_storage[(drive<br>storage)]
stalwart_metadata[(metadata<br>storage)]
stalwart_storage[(object<br>storage)]
end
ui y@==>|REST|groupware
y@{ animate: true }
ui-->|Graph|drive
muas-->|IMAP,SMTP,*DAV|stalwart
groupware-->drive
groupware-->|JMAP|stalwart
drive-->drive_storage
stalwart-->stalwart_metadata
stalwart-->stalwart_storage
```
Example:
```text
GET /groupware/startup/1/?mails=50
```
#### Good
* completely tailored to the needs of the OpenCloud UI
* a higher-level API allows for easily understanding the semantic of each operation, which enables the potential for keeping track of data in order to implement reverse indexes and caching, if necessary to achieve functional or performance goals, as opposed to using a lower-level API such as JMAP which is maximally flexible and difficult to reverse-engineer the meaning of the operation and data
* can also be tailored to the capabilities of JMAP without exposing all of its flexibility
* provides the potential for expanding upon what JMAP provides
* would support the full potential of JMAP and Stalwart since the API would be designed accordingly
* allows learning how to use and cache individual JMAP method call responses, allowing to make a better decision in the future if JMAP should be used by clients
#### Neutral
* if there is a requirement for considering that API as a public API for third party integrators, then the API also needs to be documented, maintained, versioned, and kept stable as much as possible (this is neutral because it is a requirement that exists with every option)
#### Bad
* only partially follows any standards (REST, JSON, JMAP for data models)
* requires designing the API from scratch, as opposed to using the Graph API which already prescribes one
* although it probably makes sense to re-use the data model of JMAP, which is prescribed in RFCs, also to avoid data loss and copying things around needlessly
* since the OpenCloud Drive backends use the LibreGraph API, using a custom REST API for Groupware bears the risk of having multiple APIs to do the same thing, which we need to be careful about, and avoid if possible
#### Decision Drivers
* UI Driven
* favoured solution for the OpenCloud Web UI team
* Economic Awareness
* designing a new custom API is not much effort since it is UI requirements driven
* maintaining a new custom API or JMAP extensions is not more effort either, since the exact same thing needs to be done with LibreGraph, as it will have numerous exceptions and will require documenting those, as well as which parts of the Microsoft Graph API are implemented and which aren't
* Efficiency
* the most efficient approach since it is tailored to what is actually needed for the OpenCloud UI, which will allow us to reduce the roundtrips to a minimum
* Third Party Consumption
* a custom API will be required to be maintained as a public API
User Preferences need to be configurable through the UI and persisted in a backend service in order to be reliably available and backed up.
Such configuration options have default values that need to be set on multiple levels:
* globally
* by tenant
* by sub-tenant
* by group of users
* by user
Some options might even be client-specific, e.g. differ between the OpenCloud Web UI on desktop and the OpenCloud Web UI on mobile.
Furthermore, some options might be enforced and may not be overridden on every level (e.g. only globally or by tenant, by not modifiable by users.)
Ideally, the configuration settings have an architecture that permits pluggable sources.
This level of necessary complexity has a few drawbacks, the primary one being that it can become difficult to find out why a user sees this or that behavior in their UI, and thus to trace down where a given configuration setting is made (globally, on tenant level, etc...). It is thus critical to include tooling that allows to debug them.
Usage:"Start a client that continuously makes web requests and prints stats. The options mimic curl, but URL must be at the end.",
Flags:[]cli.Flag{
// TODO with v3 'flag.Persistent: true' can be set to make the order of flags no longer relevant \o/
// flags mimicing curl
&cli.StringFlag{
Name:"request",
Aliases:[]string{"X"},
Value:"PROPFIND",
Usage:"Specifies a custom request method to use when communicating with the HTTP server.",
},
&cli.StringFlag{
Name:"user",
Aliases:[]string{"u"},
Value:"admin:admin",
Usage:"Specify the user name and password to use for server authentication.",
},
&cli.BoolFlag{
Name:"insecure",
Aliases:[]string{"k"},
Usage:"Skip the TLS verification step and proceed without checking.",
},
&cli.StringFlag{
Name:"data",
Aliases:[]string{"d"},
Usage:"Sends the specified data in a request to the HTTP server.",
// TODE support multiple data flags, support data-binary, data-raw
},
&cli.StringSliceFlag{
Name:"header",
Aliases:[]string{"H"},
Usage:"Extra header to include in information sent.",
},
&cli.StringFlag{
Name:"rate",
Usage:`Specify the maximum transfer frequency you allow a client to use - in number of transfer starts per time unit (sometimes called request rate).
The request rate is provided as "N/U" where N is an integer number and U is a time unit. Supported units are 's' (second), 'm' (minute), 'h' (hour) and 'd' /(day, as in a 24 hour unit). The default time unit, if no "/U" is provided, is number of transfers per hour.`,
},
/*
&cli.StringFlag{
Name: "oauth2-bearer",
Usage: "Specify the Bearer Token for OAUTH 2.0 server authentication.",
},
&cli.StringFlag{
Name: "user-agent",
Aliases: []string{"A"},
Value: "admin:admin",
Usage: "Specify the User-Agent string to send to the HTTP server.",
},
*/
// other flags
&cli.StringFlag{
Name:"bearer-token-command",
Usage:"Command to execute for a bearer token, e.g. 'oidc-token opencloud'. When set, disables basic auth.",
},
&cli.IntFlag{
Name:"every",
Usage:"Aggregate stats every time this amount of seconds has passed.",
benchClientCmd.Flags().StringP("request","X","PROPFIND","Specifies a custom request method to use when communicating with the HTTP server.")
benchClientCmd.Flags().StringP("user","u","admin:admin","Specify the user name and password to use for server authentication.")
benchClientCmd.Flags().BoolP("insecure","k",false,"Skip the TLS verification step and proceed without checking.")
benchClientCmd.Flags().StringP("data","d","","Sends the specified data in a POST request to the HTTP server, in the same way that a browser does when a user has filled in an HTML form and presses the submit button. If you start the data with the letter @, the rest should be a file name to read the data from, or - if you want to read the data from stdin. When -d, --data is told to read from a file like that, carriage returns and newlines are stripped out. If you do not want the @ character to have a special interpretation use --data-raw instead.")
benchClientCmd.Flags().StringP("data-raw","","","Sends the specified data in a request to the HTTP server.")
benchClientCmd.Flags().StringP("data-binary","","","This posts data exactly as specified with no extra processing whatsoever. If you start the data with the letter @, the rest should be a file name to read the data from, or - if you want to read the data from stdin.")
benchClientCmd.Flags().StringSliceP("headers","H",[]string{},"Extra header to include in information sent.")
benchClientCmd.Flags().String("rate","","Specify the maximum transfer frequency you allow a client to use - in number of transfer starts per time unit (sometimes called request rate). The request rate is provided as \"N/U\" where N is an integer number and U is a time unit. Supported units are 's' (second), 'm' (minute), 'h' (hour) and 'd' /(day, as in a 24 hour unit). The default time unit, if no \"/U\" is provided, is number of transfers per hour.")
// other flags
benchClientCmd.Flags().IntP("jobs","j",1,"Number of parallel clients to start. Defaults to 1.")
benchClientCmd.Flags().Int("every",0,"Aggregate stats every time this amount of seconds has passed.")
benchClientCmd.Flags().String("bearer-token-command","","Command to execute for a bearer token, e.g. 'oidc-token opencloud'. When set, disables basic auth.")
metaCmd.Flags().StringP("root","r","","Path to the root directory of the decomposedfs")
_=metaCmd.MarkFlagRequired("root")
metaCmd.Flags().StringP("node","n","","Path to or ID of the node to inspect")
_=metaCmd.MarkFlagRequired("node")
returnmetaCmd
}
funcdumpCmd(cfg*config.Config)*cli.Command{
return&cli.Command{
Name:"dump",
Usage:`print the metadata of the given node. String attributes will be enclosed in quotes. Binary attributes will be returned encoded as base64 with their value being prefixed with '0s'.`,
Action:func(c*cli.Context)error{
lu,backend:=getBackend(c)
path,err:=getNode(c,lu)
funcdumpCmd(_*config.Config)*cobra.Command{
return&cobra.Command{
Use:"dump",
Short:`print the metadata of the given node. String attributes will be enclosed in quotes. Binary attributes will be returned encoded as base64 with their value being prefixed with '0s'.`,
Usage:`print a specific attribute of the given node. String attributes will be enclosed in quotes. Binary attributes will be returned encoded as base64 with their value being prefixed with '0s'.`,
Flags:[]cli.Flag{
&cli.StringFlag{
Name:"attribute",
Aliases:[]string{"a"},
Usage:"attribute to inspect",
},
},
Action:func(c*cli.Context)error{
lu,backend:=getBackend(c)
path,err:=getNode(c,lu)
funcgetCmd(_*config.Config)*cobra.Command{
gCmd:=&cobra.Command{
Use:"get",
Short:`print a specific attribute of the given node. String attributes will be enclosed in quotes. Binary attributes will be returned encoded as base64 with their value being prefixed with '0s'.`,
gCmd.Flags().StringP("attribute","a","","attribute to inspect, can be a glob pattern (e.g. 'user.*' will match all attributes starting with 'user.').")
returngCmd
}
funcsetCmd(cfg*config.Config)*cli.Command{
return&cli.Command{
Name:"set",
Usage:`manipulate metadata of the given node. Binary attributes can be given hex encoded (prefix by '0x') or base64 encoded (prefix by '0s').`,
Flags:[]cli.Flag{
&cli.StringFlag{
Name:"attribute",
Required:true,
Aliases:[]string{"a"},
Usage:"attribute to inspect",
},
&cli.StringFlag{
Name:"value",
Required:true,
Aliases:[]string{"v"},
Usage:"value to set",
},
},
Action:func(c*cli.Context)error{
lu,backend:=getBackend(c)
n,err:=getNode(c,lu)
funcsetCmd(_*config.Config)*cobra.Command{
sCmd:=&cobra.Command{
Use:"set",
Short:`manipulate metadata of the given node. Binary attributes can be given hex encoded (prefix by '0x') or base64 encoded (prefix by '0s').`,
sCmd.Flags().StringP("attribute","a","","attribute to inspect, can be a glob pattern (e.g. 'user.*' will match all attributes starting with 'user.').")
_=sCmd.MarkFlagRequired("attribute")
sCmd.Flags().StringP("value","v","","value to set")
Loaded 100 of 5114 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Pascal Bleser
