Compare commits

..
Author SHA1 Message Date
Jörn Friedrich Dreyer 8d5912a5c4 refactor(thumbnails): remove the pull-based pipeline
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
2026-09-08 16:10:29 +02:00
Jörn Friedrich Dreyer b6f2adfffb refactor(webdav): drive the thumbnail workflow via the push-based generator
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
2026-09-08 16:10:29 +02:00
Jörn Friedrich Dreyer 9832ec25e5 feat(thumbnails): add imagor-compatible push-based thumbnail endpoint
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
2026-09-08 16:10:23 +02:00
174 changed files with 7843 additions and 6910 deletions

No files matched your search

-60
View File
@@ -1,65 +1,5 @@
# Changelog
## [8.0.0](https://github.com/opencloud-eu/opencloud/releases/tag/v8.0.0) - 2026-09-08
### ❤️ Thanks to all contributors! ❤️
@butonic, @dschmidt, @fredrikblau, @fschade, @maki5, @pbleser-oc, @v-scharf, @zerox80
### 💥 Breaking changes
- feat(search): check the index schema on startup and refuse breaking changes [[#3197](https://github.com/opencloud-eu/opencloud/pull/3197)]
- refactor: reflection-based search mapping + location geopoint [[#3345](https://github.com/opencloud-eu/opencloud/pull/3345)]
- fix(search): make openSearch and bleve behave the same [[#3408](https://github.com/opencloud-eu/opencloud/pull/3408)]
### 🐛 Bug Fixes
- fix(search): extract facets from the main tika document only [[#3484](https://github.com/opencloud-eu/opencloud/pull/3484)]
- fix(thumbnails): bound declared image dimensions before decoding [[#3457](https://github.com/opencloud-eu/opencloud/pull/3457)]
- test(search): re-search until the expected files are in the result [[#3488](https://github.com/opencloud-eu/opencloud/pull/3488)]
- fix(config): correct pending version annotations [[#3487](https://github.com/opencloud-eu/opencloud/pull/3487)]
- Activitylog event handler split [[#3241](https://github.com/opencloud-eu/opencloud/pull/3241)]
- test(search): wait for expected properties and documents [[#3486](https://github.com/opencloud-eu/opencloud/pull/3486)]
- fix: log jwt expired on debug level instead of error [[#3463](https://github.com/opencloud-eu/opencloud/pull/3463)]
- fix(proxy): restrict JWT signed urls to the allowed HTTP methods [[#3481](https://github.com/opencloud-eu/opencloud/pull/3481)]
- fix: notification handling for share removal and space membership expiry [[#3257](https://github.com/opencloud-eu/opencloud/pull/3257)]
- fix: posix cli commands [[#3348](https://github.com/opencloud-eu/opencloud/pull/3348)]
### 📈 Enhancement
- feat(graph): expand thumbnails on driveItems [[#3471](https://github.com/opencloud-eu/opencloud/pull/3471)]
- graph: expose lockInfo on driveItems [[#3444](https://github.com/opencloud-eu/opencloud/pull/3444)]
- graph: expose @libre.graph.shareTypes on driveItems [[#3438](https://github.com/opencloud-eu/opencloud/pull/3438)]
- feat(search): live photo facet [[#3202](https://github.com/opencloud-eu/opencloud/pull/3202)]
- feat: support $expand=children on the driveItem endpoint [[#3445](https://github.com/opencloud-eu/opencloud/pull/3445)]
- feat(search): motion photo facet [[#3200](https://github.com/opencloud-eu/opencloud/pull/3200)]
- feat(search): video facet [[#3201](https://github.com/opencloud-eu/opencloud/pull/3201)]
- graph: expose pendingOperations on driveItems [[#3437](https://github.com/opencloud-eu/opencloud/pull/3437)]
- feat(search): extract more data from tika 4 (if available) [[#3198](https://github.com/opencloud-eu/opencloud/pull/3198)]
- graph: expose following state, tags and allowed actions on driveItems [[#3113](https://github.com/opencloud-eu/opencloud/pull/3113)]
- feat(search): scope searches to a drive via the driveId field [[#3424](https://github.com/opencloud-eu/opencloud/pull/3424)]
- chore(policies): disable gRPC or event handlers by configuration + add metrics [[#3287](https://github.com/opencloud-eu/opencloud/pull/3287)]
- moved ShareCreated event consumer from frontend to shared service [[#3389](https://github.com/opencloud-eu/opencloud/pull/3389)]
### ✅ Tests
- test(search): fail the parity suite when the committed matrix is stale [[#3423](https://github.com/opencloud-eu/opencloud/pull/3423)]
- ci: run search API and e2e suites against OpenSearch [[#3379](https://github.com/opencloud-eu/opencloud/pull/3379)]
- test(search): engine parity suite for bleve and opensearch [[#3418](https://github.com/opencloud-eu/opencloud/pull/3418)]
### 📦️ Dependencies
- build(deps): bump go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp from 0.70.0 to 0.71.0 [[#3441](https://github.com/opencloud-eu/opencloud/pull/3441)]
- build(deps): bump go.opentelemetry.io/contrib/zpages from 0.70.0 to 0.71.0 [[#3442](https://github.com/opencloud-eu/opencloud/pull/3442)]
- build(deps): bump go.opentelemetry.io/otel from 1.45.0 to 1.46.0 [[#3426](https://github.com/opencloud-eu/opencloud/pull/3426)]
- build(deps): bump google.golang.org/grpc from 1.83.1 to 1.83.2 [[#3429](https://github.com/opencloud-eu/opencloud/pull/3429)]
- build(deps): bump github.com/blevesearch/bleve/v2 from 2.6.0 to 2.6.1 [[#3417](https://github.com/opencloud-eu/opencloud/pull/3417)]
- build(deps): bump github.com/sirupsen/logrus from 1.10.0 to 1.10.1 [[#3409](https://github.com/opencloud-eu/opencloud/pull/3409)]
- build(deps): bump github.com/opensearch-project/opensearch-go/v4 from 4.6.0 to 4.7.3 [[#3216](https://github.com/opencloud-eu/opencloud/pull/3216)]
- build(deps): bump github.com/KimMachineGun/automemlimit from 0.7.5 to 1.0.0 [[#3404](https://github.com/opencloud-eu/opencloud/pull/3404)]
- build(deps): bump github.com/nats-io/nats.go from 1.52.0 to 1.53.1 [[#3403](https://github.com/opencloud-eu/opencloud/pull/3403)]
- build(deps): bump github.com/open-policy-agent/opa from 1.19.0 to 1.19.1 [[#3402](https://github.com/opencloud-eu/opencloud/pull/3402)]
## [7.5.0](https://github.com/opencloud-eu/opencloud/releases/tag/v7.5.0) - 2026-08-25
### ❤️ Thanks to all contributors! ❤️
+1 -1
View File
@@ -95,7 +95,7 @@ require (
go-micro.dev/v4 v4.11.0
go.etcd.io/bbolt v1.5.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.70.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0
go.opentelemetry.io/contrib/zpages v0.71.0
go.opentelemetry.io/otel v1.46.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0
+2 -2
View File
@@ -1293,8 +1293,8 @@ go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.70.0 h1:oECp5f+hN7nkwjU/8BxQ/q23bGPb8FIrD839owX222E=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.70.0/go.mod h1:DqEFwLumhzMBDQv9PcWbyoDxHI/4lAk6CM4nJBH39sc=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0 h1:3g7B90UzBltIDKq1/5mrTGxTnOFDV0ICOhLoxiZ8jlg=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.71.0/go.mod h1:Ef8SuTh59BT7+ofpDxN9z+yOlc4t2GjLmKDgYNJL/NU=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 h1:LMuyCAyfalSjDyjdC65nK6N0zoTT63+E/u95X0JovZI=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0/go.mod h1:085m8qbm4hgc8rZWGDEa4vmyyo2c3nPxUslYUKUIU04=
go.opentelemetry.io/contrib/zpages v0.71.0 h1:OrQSEmdlfknIusEyGj/t30uBR7MnbmGcCRBt5ov8/rA=
go.opentelemetry.io/contrib/zpages v0.71.0/go.mod h1:C3IIpqkQbjswcSREqxQLyEapF0SI9bqwJpxpjwtzFrE=
go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc=
+1 -14
View File
@@ -68,7 +68,7 @@ func CreateConfig(insecure, forceOverwrite, diff bool, configPath, adminPassword
systemUserID, adminUserID, graphApplicationID, storageUsersMountID, serviceAccountID string
idmServicePassword, idpServicePassword, ocAdminServicePassword, revaServicePassword string
tokenManagerJwtSecret, collaborationWOPISecret, machineAuthAPIKey, systemUserAPIKey string
revaTransferSecret, thumbnailsTransferSecret, serviceAccountSecret, urlSigningSecret string
revaTransferSecret, serviceAccountSecret, urlSigningSecret string
adminPasswdwordGenerated bool
)
@@ -94,7 +94,6 @@ func CreateConfig(insecure, forceOverwrite, diff bool, configPath, adminPassword
machineAuthAPIKey = oldCfg.MachineAuthAPIKey
systemUserAPIKey = oldCfg.SystemUserAPIKey
revaTransferSecret = oldCfg.TransferSecret
thumbnailsTransferSecret = oldCfg.Thumbnails.Thumbnail.TransferSecret
serviceAccountSecret = oldCfg.Graph.ServiceAccount.ServiceAccountSecret
urlSigningSecret = oldCfg.URLSigningSecret
if urlSigningSecret == "" {
@@ -155,10 +154,6 @@ func CreateConfig(insecure, forceOverwrite, diff bool, configPath, adminPassword
if err != nil {
return fmt.Errorf("could not generate random secret for urlSigningSecret: %s", err)
}
thumbnailsTransferSecret, err = generators.GenerateRandomPassword(passwordLength)
if err != nil {
return fmt.Errorf("could not generate random password for thumbnailsTransferSecret: %s", err)
}
serviceAccountSecret, err = generators.GenerateRandomPassword(passwordLength)
if err != nil {
return fmt.Errorf("could not generate random secret for serviceAccountSecret: %s", err)
@@ -230,11 +225,6 @@ func CreateConfig(insecure, forceOverwrite, diff bool, configPath, adminPassword
},
ServiceAccount: serviceAccount,
},
Thumbnails: ThumbnailService{
Thumbnail: ThumbnailSettings{
TransferSecret: thumbnailsTransferSecret,
},
},
Gateway: Gateway{
StorageRegistry: StorageRegistry{
StorageUsersMountID: storageUsersMountID,
@@ -302,9 +292,6 @@ func CreateConfig(insecure, forceOverwrite, diff bool, configPath, adminPassword
},
ServiceAccount: serviceAccount,
}
cfg.Thumbnails.Thumbnail.WebdavAllowInsecure = true
cfg.Thumbnails.Thumbnail.Cs3AllowInsecure = true
}
yamlOutput, err := yaml.Marshal(cfg)
if err != nil {
-13
View File
@@ -33,7 +33,6 @@ type OpenCloudConfig struct {
Users UsersAndGroupsService `yaml:"users"`
Groups UsersAndGroupsService `yaml:"groups"`
Ocm OcmService `yaml:"ocm"`
Thumbnails ThumbnailService `yaml:"thumbnails"`
Search Search `yaml:"search"`
Audit Audit `yaml:"audit"`
Settings SettingsService `yaml:"settings"`
@@ -220,18 +219,6 @@ type StorageUsers struct {
ServiceAccount ServiceAccount `yaml:"service_account"`
}
// ThumbnailSettings is the configuration for the thumbnail settings
type ThumbnailSettings struct {
TransferSecret string `yaml:"transfer_secret"`
WebdavAllowInsecure bool `yaml:"webdav_allow_insecure"`
Cs3AllowInsecure bool `yaml:"cs3_allow_insecure"`
}
// ThumbnailService is the configuration for the thumbnail service
type ThumbnailService struct {
Thumbnail ThumbnailSettings
}
// TokenManager is the configuration for the token manager
type TokenManager struct {
JWTSecret string `yaml:"jwt_secret"`
@@ -1,326 +0,0 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.28.1
// protoc (unknown)
// source: opencloud/messages/thumbnails/v0/thumbnails.proto
package v0
import (
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
)
const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
// The file types to which the thumbnail can be encoded to.
type ThumbnailType int32
const (
ThumbnailType_PNG ThumbnailType = 0 // Represents PNG type
ThumbnailType_JPG ThumbnailType = 1 // Represents JPG type
ThumbnailType_GIF ThumbnailType = 2 // Represents GIF type
)
// Enum value maps for ThumbnailType.
var (
ThumbnailType_name = map[int32]string{
0: "PNG",
1: "JPG",
2: "GIF",
}
ThumbnailType_value = map[string]int32{
"PNG": 0,
"JPG": 1,
"GIF": 2,
}
)
func (x ThumbnailType) Enum() *ThumbnailType {
p := new(ThumbnailType)
*p = x
return p
}
func (x ThumbnailType) String() string {
return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
}
func (ThumbnailType) Descriptor() protoreflect.EnumDescriptor {
return file_opencloud_messages_thumbnails_v0_thumbnails_proto_enumTypes[0].Descriptor()
}
func (ThumbnailType) Type() protoreflect.EnumType {
return &file_opencloud_messages_thumbnails_v0_thumbnails_proto_enumTypes[0]
}
func (x ThumbnailType) Number() protoreflect.EnumNumber {
return protoreflect.EnumNumber(x)
}
// Deprecated: Use ThumbnailType.Descriptor instead.
func (ThumbnailType) EnumDescriptor() ([]byte, []int) {
return file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDescGZIP(), []int{0}
}
type WebdavSource struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
// REQUIRED.
Url string `protobuf:"bytes,1,opt,name=url,proto3" json:"url,omitempty"`
// REQUIRED.
IsPublicLink bool `protobuf:"varint,2,opt,name=is_public_link,json=isPublicLink,proto3" json:"is_public_link,omitempty"`
// OPTIONAL.
WebdavAuthorization string `protobuf:"bytes,3,opt,name=webdav_authorization,json=webdavAuthorization,proto3" json:"webdav_authorization,omitempty"`
// OPTIONAL.
RevaAuthorization string `protobuf:"bytes,4,opt,name=reva_authorization,json=revaAuthorization,proto3" json:"reva_authorization,omitempty"`
// OPTIONAL.
PublicLinkToken string `protobuf:"bytes,5,opt,name=public_link_token,json=publicLinkToken,proto3" json:"public_link_token,omitempty"`
}
func (x *WebdavSource) Reset() {
*x = WebdavSource{}
if protoimpl.UnsafeEnabled {
mi := &file_opencloud_messages_thumbnails_v0_thumbnails_proto_msgTypes[0]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *WebdavSource) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*WebdavSource) ProtoMessage() {}
func (x *WebdavSource) ProtoReflect() protoreflect.Message {
mi := &file_opencloud_messages_thumbnails_v0_thumbnails_proto_msgTypes[0]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use WebdavSource.ProtoReflect.Descriptor instead.
func (*WebdavSource) Descriptor() ([]byte, []int) {
return file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDescGZIP(), []int{0}
}
func (x *WebdavSource) GetUrl() string {
if x != nil {
return x.Url
}
return ""
}
func (x *WebdavSource) GetIsPublicLink() bool {
if x != nil {
return x.IsPublicLink
}
return false
}
func (x *WebdavSource) GetWebdavAuthorization() string {
if x != nil {
return x.WebdavAuthorization
}
return ""
}
func (x *WebdavSource) GetRevaAuthorization() string {
if x != nil {
return x.RevaAuthorization
}
return ""
}
func (x *WebdavSource) GetPublicLinkToken() string {
if x != nil {
return x.PublicLinkToken
}
return ""
}
type CS3Source struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Path string `protobuf:"bytes,1,opt,name=path,proto3" json:"path,omitempty"`
Authorization string `protobuf:"bytes,2,opt,name=authorization,proto3" json:"authorization,omitempty"`
}
func (x *CS3Source) Reset() {
*x = CS3Source{}
if protoimpl.UnsafeEnabled {
mi := &file_opencloud_messages_thumbnails_v0_thumbnails_proto_msgTypes[1]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *CS3Source) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*CS3Source) ProtoMessage() {}
func (x *CS3Source) ProtoReflect() protoreflect.Message {
mi := &file_opencloud_messages_thumbnails_v0_thumbnails_proto_msgTypes[1]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use CS3Source.ProtoReflect.Descriptor instead.
func (*CS3Source) Descriptor() ([]byte, []int) {
return file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDescGZIP(), []int{1}
}
func (x *CS3Source) GetPath() string {
if x != nil {
return x.Path
}
return ""
}
func (x *CS3Source) GetAuthorization() string {
if x != nil {
return x.Authorization
}
return ""
}
var File_opencloud_messages_thumbnails_v0_thumbnails_proto protoreflect.FileDescriptor
var file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDesc = []byte{
0x0a, 0x31, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x6d, 0x65, 0x73, 0x73,
0x61, 0x67, 0x65, 0x73, 0x2f, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x73, 0x2f,
0x76, 0x30, 0x2f, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x73, 0x2e, 0x70, 0x72,
0x6f, 0x74, 0x6f, 0x12, 0x20, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x6d,
0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69,
0x6c, 0x73, 0x2e, 0x76, 0x30, 0x22, 0xd4, 0x01, 0x0a, 0x0c, 0x57, 0x65, 0x62, 0x64, 0x61, 0x76,
0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x10, 0x0a, 0x03, 0x75, 0x72, 0x6c, 0x18, 0x01, 0x20,
0x01, 0x28, 0x09, 0x52, 0x03, 0x75, 0x72, 0x6c, 0x12, 0x24, 0x0a, 0x0e, 0x69, 0x73, 0x5f, 0x70,
0x75, 0x62, 0x6c, 0x69, 0x63, 0x5f, 0x6c, 0x69, 0x6e, 0x6b, 0x18, 0x02, 0x20, 0x01, 0x28, 0x08,
0x52, 0x0c, 0x69, 0x73, 0x50, 0x75, 0x62, 0x6c, 0x69, 0x63, 0x4c, 0x69, 0x6e, 0x6b, 0x12, 0x31,
0x0a, 0x14, 0x77, 0x65, 0x62, 0x64, 0x61, 0x76, 0x5f, 0x61, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69,
0x7a, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x13, 0x77, 0x65,
0x62, 0x64, 0x61, 0x76, 0x41, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69, 0x7a, 0x61, 0x74, 0x69, 0x6f,
0x6e, 0x12, 0x2d, 0x0a, 0x12, 0x72, 0x65, 0x76, 0x61, 0x5f, 0x61, 0x75, 0x74, 0x68, 0x6f, 0x72,
0x69, 0x7a, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x11, 0x72,
0x65, 0x76, 0x61, 0x41, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69, 0x7a, 0x61, 0x74, 0x69, 0x6f, 0x6e,
0x12, 0x2a, 0x0a, 0x11, 0x70, 0x75, 0x62, 0x6c, 0x69, 0x63, 0x5f, 0x6c, 0x69, 0x6e, 0x6b, 0x5f,
0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x18, 0x05, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0f, 0x70, 0x75, 0x62,
0x6c, 0x69, 0x63, 0x4c, 0x69, 0x6e, 0x6b, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x22, 0x45, 0x0a, 0x09,
0x43, 0x53, 0x33, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x70, 0x61, 0x74,
0x68, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x70, 0x61, 0x74, 0x68, 0x12, 0x24, 0x0a,
0x0d, 0x61, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69, 0x7a, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x02,
0x20, 0x01, 0x28, 0x09, 0x52, 0x0d, 0x61, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x69, 0x7a, 0x61, 0x74,
0x69, 0x6f, 0x6e, 0x2a, 0x2a, 0x0a, 0x0d, 0x54, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c,
0x54, 0x79, 0x70, 0x65, 0x12, 0x07, 0x0a, 0x03, 0x50, 0x4e, 0x47, 0x10, 0x00, 0x12, 0x07, 0x0a,
0x03, 0x4a, 0x50, 0x47, 0x10, 0x01, 0x12, 0x07, 0x0a, 0x03, 0x47, 0x49, 0x46, 0x10, 0x02, 0x42,
0x51, 0x5a, 0x4f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6f, 0x70,
0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2d, 0x65, 0x75, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63,
0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x67, 0x65, 0x6e, 0x2f, 0x67, 0x65,
0x6e, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x6d, 0x65, 0x73, 0x73,
0x61, 0x67, 0x65, 0x73, 0x2f, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x73, 0x2f,
0x76, 0x30, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
}
var (
file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDescOnce sync.Once
file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDescData = file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDesc
)
func file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDescGZIP() []byte {
file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDescOnce.Do(func() {
file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDescData = protoimpl.X.CompressGZIP(file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDescData)
})
return file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDescData
}
var file_opencloud_messages_thumbnails_v0_thumbnails_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
var file_opencloud_messages_thumbnails_v0_thumbnails_proto_msgTypes = make([]protoimpl.MessageInfo, 2)
var file_opencloud_messages_thumbnails_v0_thumbnails_proto_goTypes = []interface{}{
(ThumbnailType)(0), // 0: opencloud.messages.thumbnails.v0.ThumbnailType
(*WebdavSource)(nil), // 1: opencloud.messages.thumbnails.v0.WebdavSource
(*CS3Source)(nil), // 2: opencloud.messages.thumbnails.v0.CS3Source
}
var file_opencloud_messages_thumbnails_v0_thumbnails_proto_depIdxs = []int32{
0, // [0:0] is the sub-list for method output_type
0, // [0:0] is the sub-list for method input_type
0, // [0:0] is the sub-list for extension type_name
0, // [0:0] is the sub-list for extension extendee
0, // [0:0] is the sub-list for field type_name
}
func init() { file_opencloud_messages_thumbnails_v0_thumbnails_proto_init() }
func file_opencloud_messages_thumbnails_v0_thumbnails_proto_init() {
if File_opencloud_messages_thumbnails_v0_thumbnails_proto != nil {
return
}
if !protoimpl.UnsafeEnabled {
file_opencloud_messages_thumbnails_v0_thumbnails_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*WebdavSource); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_opencloud_messages_thumbnails_v0_thumbnails_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*CS3Source); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDesc,
NumEnums: 1,
NumMessages: 2,
NumExtensions: 0,
NumServices: 0,
},
GoTypes: file_opencloud_messages_thumbnails_v0_thumbnails_proto_goTypes,
DependencyIndexes: file_opencloud_messages_thumbnails_v0_thumbnails_proto_depIdxs,
EnumInfos: file_opencloud_messages_thumbnails_v0_thumbnails_proto_enumTypes,
MessageInfos: file_opencloud_messages_thumbnails_v0_thumbnails_proto_msgTypes,
}.Build()
File_opencloud_messages_thumbnails_v0_thumbnails_proto = out.File
file_opencloud_messages_thumbnails_v0_thumbnails_proto_rawDesc = nil
file_opencloud_messages_thumbnails_v0_thumbnails_proto_goTypes = nil
file_opencloud_messages_thumbnails_v0_thumbnails_proto_depIdxs = nil
}
@@ -1,15 +0,0 @@
// Code generated by protoc-gen-micro. DO NOT EDIT.
// source: opencloud/messages/thumbnails/v0/thumbnails.proto
package v0
import (
fmt "fmt"
proto "google.golang.org/protobuf/proto"
math "math"
)
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = fmt.Errorf
var _ = math.Inf
@@ -1,43 +0,0 @@
{
"swagger": "2.0",
"info": {
"title": "opencloud/messages/thumbnails/v0/thumbnails.proto",
"version": "version not set"
},
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"paths": {},
"definitions": {
"protobufAny": {
"type": "object",
"properties": {
"@type": {
"type": "string"
}
},
"additionalProperties": {}
},
"rpcStatus": {
"type": "object",
"properties": {
"code": {
"type": "integer",
"format": "int32"
},
"message": {
"type": "string"
},
"details": {
"type": "array",
"items": {
"$ref": "#/definitions/protobufAny"
}
}
}
}
}
}
@@ -1,123 +0,0 @@
// Code generated by mockery; DO NOT EDIT.
// github.com/vektra/mockery
// template: testify
package mocks
import (
"context"
"github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/thumbnails/v0"
mock "github.com/stretchr/testify/mock"
"go-micro.dev/v4/client"
)
// NewThumbnailService creates a new instance of ThumbnailService. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
func NewThumbnailService(t interface {
mock.TestingT
Cleanup(func())
}) *ThumbnailService {
mock := &ThumbnailService{}
mock.Mock.Test(t)
t.Cleanup(func() { mock.AssertExpectations(t) })
return mock
}
// ThumbnailService is an autogenerated mock type for the ThumbnailService type
type ThumbnailService struct {
mock.Mock
}
type ThumbnailService_Expecter struct {
mock *mock.Mock
}
func (_m *ThumbnailService) EXPECT() *ThumbnailService_Expecter {
return &ThumbnailService_Expecter{mock: &_m.Mock}
}
// GetThumbnail provides a mock function for the type ThumbnailService
func (_mock *ThumbnailService) GetThumbnail(ctx context.Context, in *v0.GetThumbnailRequest, opts ...client.CallOption) (*v0.GetThumbnailResponse, error) {
var tmpRet mock.Arguments
if len(opts) > 0 {
tmpRet = _mock.Called(ctx, in, opts)
} else {
tmpRet = _mock.Called(ctx, in)
}
ret := tmpRet
if len(ret) == 0 {
panic("no return value specified for GetThumbnail")
}
var r0 *v0.GetThumbnailResponse
var r1 error
if returnFunc, ok := ret.Get(0).(func(context.Context, *v0.GetThumbnailRequest, ...client.CallOption) (*v0.GetThumbnailResponse, error)); ok {
return returnFunc(ctx, in, opts...)
}
if returnFunc, ok := ret.Get(0).(func(context.Context, *v0.GetThumbnailRequest, ...client.CallOption) *v0.GetThumbnailResponse); ok {
r0 = returnFunc(ctx, in, opts...)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*v0.GetThumbnailResponse)
}
}
if returnFunc, ok := ret.Get(1).(func(context.Context, *v0.GetThumbnailRequest, ...client.CallOption) error); ok {
r1 = returnFunc(ctx, in, opts...)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// ThumbnailService_GetThumbnail_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'GetThumbnail'
type ThumbnailService_GetThumbnail_Call struct {
*mock.Call
}
// GetThumbnail is a helper method to define mock.On call
// - ctx context.Context
// - in *v0.GetThumbnailRequest
// - opts ...client.CallOption
func (_e *ThumbnailService_Expecter) GetThumbnail(ctx interface{}, in interface{}, opts ...interface{}) *ThumbnailService_GetThumbnail_Call {
return &ThumbnailService_GetThumbnail_Call{Call: _e.mock.On("GetThumbnail",
append([]interface{}{ctx, in}, opts...)...)}
}
func (_c *ThumbnailService_GetThumbnail_Call) Run(run func(ctx context.Context, in *v0.GetThumbnailRequest, opts ...client.CallOption)) *ThumbnailService_GetThumbnail_Call {
_c.Call.Run(func(args mock.Arguments) {
var arg0 context.Context
if args[0] != nil {
arg0 = args[0].(context.Context)
}
var arg1 *v0.GetThumbnailRequest
if args[1] != nil {
arg1 = args[1].(*v0.GetThumbnailRequest)
}
var arg2 []client.CallOption
var variadicArgs []client.CallOption
if len(args) > 2 {
variadicArgs = args[2].([]client.CallOption)
}
arg2 = variadicArgs
run(
arg0,
arg1,
arg2...,
)
})
return _c
}
func (_c *ThumbnailService_GetThumbnail_Call) Return(getThumbnailResponse *v0.GetThumbnailResponse, err error) *ThumbnailService_GetThumbnail_Call {
_c.Call.Return(getThumbnailResponse, err)
return _c
}
func (_c *ThumbnailService_GetThumbnail_Call) RunAndReturn(run func(ctx context.Context, in *v0.GetThumbnailRequest, opts ...client.CallOption) (*v0.GetThumbnailResponse, error)) *ThumbnailService_GetThumbnail_Call {
_c.Call.Return(run)
return _c
}
@@ -1,388 +0,0 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.28.1
// protoc (unknown)
// source: opencloud/services/thumbnails/v0/thumbnails.proto
package v0
import (
_ "github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2/options"
v0 "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/messages/thumbnails/v0"
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
)
const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
// A request to retrieve a thumbnail
type GetThumbnailRequest struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
// The path to the source image
Filepath string `protobuf:"bytes,1,opt,name=filepath,proto3" json:"filepath,omitempty"`
// The type to which the thumbnail should get encoded to.
ThumbnailType v0.ThumbnailType `protobuf:"varint,2,opt,name=thumbnail_type,json=thumbnailType,proto3,enum=opencloud.messages.thumbnails.v0.ThumbnailType" json:"thumbnail_type,omitempty"`
// The width of the thumbnail
Width int32 `protobuf:"varint,3,opt,name=width,proto3" json:"width,omitempty"`
// The height of the thumbnail
Height int32 `protobuf:"varint,4,opt,name=height,proto3" json:"height,omitempty"`
// Indicates which image processor to use
Processor string `protobuf:"bytes,5,opt,name=processor,proto3" json:"processor,omitempty"`
// Types that are assignable to Source:
//
// *GetThumbnailRequest_WebdavSource
// *GetThumbnailRequest_Cs3Source
Source isGetThumbnailRequest_Source `protobuf_oneof:"source"`
}
func (x *GetThumbnailRequest) Reset() {
*x = GetThumbnailRequest{}
if protoimpl.UnsafeEnabled {
mi := &file_opencloud_services_thumbnails_v0_thumbnails_proto_msgTypes[0]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *GetThumbnailRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*GetThumbnailRequest) ProtoMessage() {}
func (x *GetThumbnailRequest) ProtoReflect() protoreflect.Message {
mi := &file_opencloud_services_thumbnails_v0_thumbnails_proto_msgTypes[0]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use GetThumbnailRequest.ProtoReflect.Descriptor instead.
func (*GetThumbnailRequest) Descriptor() ([]byte, []int) {
return file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDescGZIP(), []int{0}
}
func (x *GetThumbnailRequest) GetFilepath() string {
if x != nil {
return x.Filepath
}
return ""
}
func (x *GetThumbnailRequest) GetThumbnailType() v0.ThumbnailType {
if x != nil {
return x.ThumbnailType
}
return v0.ThumbnailType(0)
}
func (x *GetThumbnailRequest) GetWidth() int32 {
if x != nil {
return x.Width
}
return 0
}
func (x *GetThumbnailRequest) GetHeight() int32 {
if x != nil {
return x.Height
}
return 0
}
func (x *GetThumbnailRequest) GetProcessor() string {
if x != nil {
return x.Processor
}
return ""
}
func (m *GetThumbnailRequest) GetSource() isGetThumbnailRequest_Source {
if m != nil {
return m.Source
}
return nil
}
func (x *GetThumbnailRequest) GetWebdavSource() *v0.WebdavSource {
if x, ok := x.GetSource().(*GetThumbnailRequest_WebdavSource); ok {
return x.WebdavSource
}
return nil
}
func (x *GetThumbnailRequest) GetCs3Source() *v0.CS3Source {
if x, ok := x.GetSource().(*GetThumbnailRequest_Cs3Source); ok {
return x.Cs3Source
}
return nil
}
type isGetThumbnailRequest_Source interface {
isGetThumbnailRequest_Source()
}
type GetThumbnailRequest_WebdavSource struct {
WebdavSource *v0.WebdavSource `protobuf:"bytes,6,opt,name=webdav_source,json=webdavSource,proto3,oneof"`
}
type GetThumbnailRequest_Cs3Source struct {
Cs3Source *v0.CS3Source `protobuf:"bytes,7,opt,name=cs3_source,json=cs3Source,proto3,oneof"`
}
func (*GetThumbnailRequest_WebdavSource) isGetThumbnailRequest_Source() {}
func (*GetThumbnailRequest_Cs3Source) isGetThumbnailRequest_Source() {}
// The service response
type GetThumbnailResponse struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
// The endpoint where the thumbnail can be downloaded.
DataEndpoint string `protobuf:"bytes,1,opt,name=data_endpoint,json=dataEndpoint,proto3" json:"data_endpoint,omitempty"`
// The transfer token to be able to download the thumbnail.
TransferToken string `protobuf:"bytes,2,opt,name=transfer_token,json=transferToken,proto3" json:"transfer_token,omitempty"`
// The mimetype of the thumbnail
Mimetype string `protobuf:"bytes,3,opt,name=mimetype,proto3" json:"mimetype,omitempty"`
}
func (x *GetThumbnailResponse) Reset() {
*x = GetThumbnailResponse{}
if protoimpl.UnsafeEnabled {
mi := &file_opencloud_services_thumbnails_v0_thumbnails_proto_msgTypes[1]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *GetThumbnailResponse) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*GetThumbnailResponse) ProtoMessage() {}
func (x *GetThumbnailResponse) ProtoReflect() protoreflect.Message {
mi := &file_opencloud_services_thumbnails_v0_thumbnails_proto_msgTypes[1]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use GetThumbnailResponse.ProtoReflect.Descriptor instead.
func (*GetThumbnailResponse) Descriptor() ([]byte, []int) {
return file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDescGZIP(), []int{1}
}
func (x *GetThumbnailResponse) GetDataEndpoint() string {
if x != nil {
return x.DataEndpoint
}
return ""
}
func (x *GetThumbnailResponse) GetTransferToken() string {
if x != nil {
return x.TransferToken
}
return ""
}
func (x *GetThumbnailResponse) GetMimetype() string {
if x != nil {
return x.Mimetype
}
return ""
}
var File_opencloud_services_thumbnails_v0_thumbnails_proto protoreflect.FileDescriptor
var file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDesc = []byte{
0x0a, 0x31, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x73, 0x65, 0x72, 0x76,
0x69, 0x63, 0x65, 0x73, 0x2f, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x73, 0x2f,
0x76, 0x30, 0x2f, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x73, 0x2e, 0x70, 0x72,
0x6f, 0x74, 0x6f, 0x12, 0x20, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x73,
0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2e, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69,
0x6c, 0x73, 0x2e, 0x76, 0x30, 0x1a, 0x31, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64,
0x2f, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2f, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e,
0x61, 0x69, 0x6c, 0x73, 0x2f, 0x76, 0x30, 0x2f, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69,
0x6c, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63,
0x2d, 0x67, 0x65, 0x6e, 0x2d, 0x6f, 0x70, 0x65, 0x6e, 0x61, 0x70, 0x69, 0x76, 0x32, 0x2f, 0x6f,
0x70, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x2f, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f,
0x6e, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x84, 0x03, 0x0a, 0x13, 0x47, 0x65, 0x74,
0x54, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74,
0x12, 0x1a, 0x0a, 0x08, 0x66, 0x69, 0x6c, 0x65, 0x70, 0x61, 0x74, 0x68, 0x18, 0x01, 0x20, 0x01,
0x28, 0x09, 0x52, 0x08, 0x66, 0x69, 0x6c, 0x65, 0x70, 0x61, 0x74, 0x68, 0x12, 0x56, 0x0a, 0x0e,
0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02,
0x20, 0x01, 0x28, 0x0e, 0x32, 0x2f, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64,
0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e,
0x61, 0x69, 0x6c, 0x73, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69,
0x6c, 0x54, 0x79, 0x70, 0x65, 0x52, 0x0d, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c,
0x54, 0x79, 0x70, 0x65, 0x12, 0x14, 0x0a, 0x05, 0x77, 0x69, 0x64, 0x74, 0x68, 0x18, 0x03, 0x20,
0x01, 0x28, 0x05, 0x52, 0x05, 0x77, 0x69, 0x64, 0x74, 0x68, 0x12, 0x16, 0x0a, 0x06, 0x68, 0x65,
0x69, 0x67, 0x68, 0x74, 0x18, 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, 0x06, 0x68, 0x65, 0x69, 0x67,
0x68, 0x74, 0x12, 0x1c, 0x0a, 0x09, 0x70, 0x72, 0x6f, 0x63, 0x65, 0x73, 0x73, 0x6f, 0x72, 0x18,
0x05, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x70, 0x72, 0x6f, 0x63, 0x65, 0x73, 0x73, 0x6f, 0x72,
0x12, 0x55, 0x0a, 0x0d, 0x77, 0x65, 0x62, 0x64, 0x61, 0x76, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63,
0x65, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2e, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c,
0x6f, 0x75, 0x64, 0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e, 0x74, 0x68, 0x75,
0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x73, 0x2e, 0x76, 0x30, 0x2e, 0x57, 0x65, 0x62, 0x64, 0x61,
0x76, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x48, 0x00, 0x52, 0x0c, 0x77, 0x65, 0x62, 0x64, 0x61,
0x76, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x4c, 0x0a, 0x0a, 0x63, 0x73, 0x33, 0x5f, 0x73,
0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x07, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2b, 0x2e, 0x6f, 0x70,
0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73,
0x2e, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x73, 0x2e, 0x76, 0x30, 0x2e, 0x43,
0x53, 0x33, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x48, 0x00, 0x52, 0x09, 0x63, 0x73, 0x33, 0x53,
0x6f, 0x75, 0x72, 0x63, 0x65, 0x42, 0x08, 0x0a, 0x06, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x22,
0x7e, 0x0a, 0x14, 0x47, 0x65, 0x74, 0x54, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x52,
0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x23, 0x0a, 0x0d, 0x64, 0x61, 0x74, 0x61, 0x5f,
0x65, 0x6e, 0x64, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0c,
0x64, 0x61, 0x74, 0x61, 0x45, 0x6e, 0x64, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0x12, 0x25, 0x0a, 0x0e,
0x74, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x65, 0x72, 0x5f, 0x74, 0x6f, 0x6b, 0x65, 0x6e, 0x18, 0x02,
0x20, 0x01, 0x28, 0x09, 0x52, 0x0d, 0x74, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x65, 0x72, 0x54, 0x6f,
0x6b, 0x65, 0x6e, 0x12, 0x1a, 0x0a, 0x08, 0x6d, 0x69, 0x6d, 0x65, 0x74, 0x79, 0x70, 0x65, 0x18,
0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x6d, 0x69, 0x6d, 0x65, 0x74, 0x79, 0x70, 0x65, 0x32,
0x91, 0x01, 0x0a, 0x10, 0x54, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x53, 0x65, 0x72,
0x76, 0x69, 0x63, 0x65, 0x12, 0x7d, 0x0a, 0x0c, 0x47, 0x65, 0x74, 0x54, 0x68, 0x75, 0x6d, 0x62,
0x6e, 0x61, 0x69, 0x6c, 0x12, 0x35, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64,
0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2e, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e,
0x61, 0x69, 0x6c, 0x73, 0x2e, 0x76, 0x30, 0x2e, 0x47, 0x65, 0x74, 0x54, 0x68, 0x75, 0x6d, 0x62,
0x6e, 0x61, 0x69, 0x6c, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x36, 0x2e, 0x6f, 0x70,
0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73,
0x2e, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x73, 0x2e, 0x76, 0x30, 0x2e, 0x47,
0x65, 0x74, 0x54, 0x68, 0x75, 0x6d, 0x62, 0x6e, 0x61, 0x69, 0x6c, 0x52, 0x65, 0x73, 0x70, 0x6f,
0x6e, 0x73, 0x65, 0x42, 0xff, 0x02, 0x5a, 0x4f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63,
0x6f, 0x6d, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2d, 0x65, 0x75, 0x2f,
0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x67,
0x65, 0x6e, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64,
0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2f, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e,
0x61, 0x69, 0x6c, 0x73, 0x2f, 0x76, 0x30, 0x92, 0x41, 0xaa, 0x02, 0x12, 0xbb, 0x01, 0x0a, 0x14,
0x4f, 0x70, 0x65, 0x6e, 0x43, 0x6c, 0x6f, 0x75, 0x64, 0x20, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e,
0x61, 0x69, 0x6c, 0x73, 0x22, 0x51, 0x0a, 0x0e, 0x4f, 0x70, 0x65, 0x6e, 0x43, 0x6c, 0x6f, 0x75,
0x64, 0x20, 0x47, 0x6d, 0x62, 0x48, 0x12, 0x29, 0x68, 0x74, 0x74, 0x70, 0x73, 0x3a, 0x2f, 0x2f,
0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63,
0x6c, 0x6f, 0x75, 0x64, 0x2d, 0x65, 0x75, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75,
0x64, 0x1a, 0x14, 0x73, 0x75, 0x70, 0x70, 0x6f, 0x72, 0x74, 0x40, 0x6f, 0x70, 0x65, 0x6e, 0x63,
0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x65, 0x75, 0x2a, 0x49, 0x0a, 0x0a, 0x41, 0x70, 0x61, 0x63, 0x68,
0x65, 0x2d, 0x32, 0x2e, 0x30, 0x12, 0x3b, 0x68, 0x74, 0x74, 0x70, 0x73, 0x3a, 0x2f, 0x2f, 0x67,
0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c,
0x6f, 0x75, 0x64, 0x2d, 0x65, 0x75, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64,
0x2f, 0x62, 0x6c, 0x6f, 0x62, 0x2f, 0x6d, 0x61, 0x69, 0x6e, 0x2f, 0x4c, 0x49, 0x43, 0x45, 0x4e,
0x53, 0x45, 0x32, 0x05, 0x31, 0x2e, 0x30, 0x2e, 0x30, 0x2a, 0x02, 0x01, 0x02, 0x32, 0x10, 0x61,
0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0x3a,
0x10, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f,
0x6e, 0x72, 0x42, 0x0a, 0x10, 0x44, 0x65, 0x76, 0x65, 0x6c, 0x6f, 0x70, 0x65, 0x72, 0x20, 0x4d,
0x61, 0x6e, 0x75, 0x61, 0x6c, 0x12, 0x2e, 0x68, 0x74, 0x74, 0x70, 0x73, 0x3a, 0x2f, 0x2f, 0x64,
0x6f, 0x63, 0x73, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x65, 0x75,
0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2f, 0x74, 0x68, 0x75, 0x6d, 0x62, 0x6e,
0x61, 0x69, 0x6c, 0x73, 0x2f, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
}
var (
file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDescOnce sync.Once
file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDescData = file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDesc
)
func file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDescGZIP() []byte {
file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDescOnce.Do(func() {
file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDescData = protoimpl.X.CompressGZIP(file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDescData)
})
return file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDescData
}
var file_opencloud_services_thumbnails_v0_thumbnails_proto_msgTypes = make([]protoimpl.MessageInfo, 2)
var file_opencloud_services_thumbnails_v0_thumbnails_proto_goTypes = []interface{}{
(*GetThumbnailRequest)(nil), // 0: opencloud.services.thumbnails.v0.GetThumbnailRequest
(*GetThumbnailResponse)(nil), // 1: opencloud.services.thumbnails.v0.GetThumbnailResponse
(v0.ThumbnailType)(0), // 2: opencloud.messages.thumbnails.v0.ThumbnailType
(*v0.WebdavSource)(nil), // 3: opencloud.messages.thumbnails.v0.WebdavSource
(*v0.CS3Source)(nil), // 4: opencloud.messages.thumbnails.v0.CS3Source
}
var file_opencloud_services_thumbnails_v0_thumbnails_proto_depIdxs = []int32{
2, // 0: opencloud.services.thumbnails.v0.GetThumbnailRequest.thumbnail_type:type_name -> opencloud.messages.thumbnails.v0.ThumbnailType
3, // 1: opencloud.services.thumbnails.v0.GetThumbnailRequest.webdav_source:type_name -> opencloud.messages.thumbnails.v0.WebdavSource
4, // 2: opencloud.services.thumbnails.v0.GetThumbnailRequest.cs3_source:type_name -> opencloud.messages.thumbnails.v0.CS3Source
0, // 3: opencloud.services.thumbnails.v0.ThumbnailService.GetThumbnail:input_type -> opencloud.services.thumbnails.v0.GetThumbnailRequest
1, // 4: opencloud.services.thumbnails.v0.ThumbnailService.GetThumbnail:output_type -> opencloud.services.thumbnails.v0.GetThumbnailResponse
4, // [4:5] is the sub-list for method output_type
3, // [3:4] is the sub-list for method input_type
3, // [3:3] is the sub-list for extension type_name
3, // [3:3] is the sub-list for extension extendee
0, // [0:3] is the sub-list for field type_name
}
func init() { file_opencloud_services_thumbnails_v0_thumbnails_proto_init() }
func file_opencloud_services_thumbnails_v0_thumbnails_proto_init() {
if File_opencloud_services_thumbnails_v0_thumbnails_proto != nil {
return
}
if !protoimpl.UnsafeEnabled {
file_opencloud_services_thumbnails_v0_thumbnails_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*GetThumbnailRequest); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_opencloud_services_thumbnails_v0_thumbnails_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*GetThumbnailResponse); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
}
file_opencloud_services_thumbnails_v0_thumbnails_proto_msgTypes[0].OneofWrappers = []interface{}{
(*GetThumbnailRequest_WebdavSource)(nil),
(*GetThumbnailRequest_Cs3Source)(nil),
}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDesc,
NumEnums: 0,
NumMessages: 2,
NumExtensions: 0,
NumServices: 1,
},
GoTypes: file_opencloud_services_thumbnails_v0_thumbnails_proto_goTypes,
DependencyIndexes: file_opencloud_services_thumbnails_v0_thumbnails_proto_depIdxs,
MessageInfos: file_opencloud_services_thumbnails_v0_thumbnails_proto_msgTypes,
}.Build()
File_opencloud_services_thumbnails_v0_thumbnails_proto = out.File
file_opencloud_services_thumbnails_v0_thumbnails_proto_rawDesc = nil
file_opencloud_services_thumbnails_v0_thumbnails_proto_goTypes = nil
file_opencloud_services_thumbnails_v0_thumbnails_proto_depIdxs = nil
}
@@ -1,91 +0,0 @@
// Code generated by protoc-gen-micro. DO NOT EDIT.
// source: opencloud/services/thumbnails/v0/thumbnails.proto
package v0
import (
fmt "fmt"
_ "github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2/options"
_ "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/messages/thumbnails/v0"
proto "google.golang.org/protobuf/proto"
math "math"
)
import (
context "context"
api "go-micro.dev/v4/api"
client "go-micro.dev/v4/client"
server "go-micro.dev/v4/server"
)
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = fmt.Errorf
var _ = math.Inf
// Reference imports to suppress errors if they are not otherwise used.
var _ api.Endpoint
var _ context.Context
var _ client.Option
var _ server.Option
// Api Endpoints for ThumbnailService service
func NewThumbnailServiceEndpoints() []*api.Endpoint {
return []*api.Endpoint{}
}
// Client API for ThumbnailService service
type ThumbnailService interface {
// Generates the thumbnail and returns it.
GetThumbnail(ctx context.Context, in *GetThumbnailRequest, opts ...client.CallOption) (*GetThumbnailResponse, error)
}
type thumbnailService struct {
c client.Client
name string
}
func NewThumbnailService(name string, c client.Client) ThumbnailService {
return &thumbnailService{
c: c,
name: name,
}
}
func (c *thumbnailService) GetThumbnail(ctx context.Context, in *GetThumbnailRequest, opts ...client.CallOption) (*GetThumbnailResponse, error) {
req := c.c.NewRequest(c.name, "ThumbnailService.GetThumbnail", in)
out := new(GetThumbnailResponse)
err := c.c.Call(ctx, req, out, opts...)
if err != nil {
return nil, err
}
return out, nil
}
// Server API for ThumbnailService service
type ThumbnailServiceHandler interface {
// Generates the thumbnail and returns it.
GetThumbnail(context.Context, *GetThumbnailRequest, *GetThumbnailResponse) error
}
func RegisterThumbnailServiceHandler(s server.Server, hdlr ThumbnailServiceHandler, opts ...server.HandlerOption) error {
type thumbnailService interface {
GetThumbnail(ctx context.Context, in *GetThumbnailRequest, out *GetThumbnailResponse) error
}
type ThumbnailService struct {
thumbnailService
}
h := &thumbnailServiceHandler{hdlr}
return s.Handle(s.NewHandler(&ThumbnailService{h}, opts...))
}
type thumbnailServiceHandler struct {
ThumbnailServiceHandler
}
func (h *thumbnailServiceHandler) GetThumbnail(ctx context.Context, in *GetThumbnailRequest, out *GetThumbnailResponse) error {
return h.ThumbnailServiceHandler.GetThumbnail(ctx, in, out)
}
@@ -1,129 +0,0 @@
{
"swagger": "2.0",
"info": {
"title": "OpenCloud thumbnails",
"version": "1.0.0",
"contact": {
"name": "OpenCloud GmbH",
"url": "https://github.com/opencloud-eu/opencloud",
"email": "support@opencloud.eu"
},
"license": {
"name": "Apache-2.0",
"url": "https://github.com/opencloud-eu/opencloud/blob/main/LICENSE"
}
},
"tags": [
{
"name": "ThumbnailService"
}
],
"schemes": [
"http",
"https"
],
"consumes": [
"application/json"
],
"produces": [
"application/json"
],
"paths": {},
"definitions": {
"protobufAny": {
"type": "object",
"properties": {
"@type": {
"type": "string"
}
},
"additionalProperties": {}
},
"rpcStatus": {
"type": "object",
"properties": {
"code": {
"type": "integer",
"format": "int32"
},
"message": {
"type": "string"
},
"details": {
"type": "array",
"items": {
"$ref": "#/definitions/protobufAny"
}
}
}
},
"v0CS3Source": {
"type": "object",
"properties": {
"path": {
"type": "string"
},
"authorization": {
"type": "string"
}
}
},
"v0GetThumbnailResponse": {
"type": "object",
"properties": {
"dataEndpoint": {
"type": "string",
"description": "The endpoint where the thumbnail can be downloaded."
},
"transferToken": {
"type": "string",
"description": "The transfer token to be able to download the thumbnail."
},
"mimetype": {
"type": "string",
"title": "The mimetype of the thumbnail"
}
},
"title": "The service response"
},
"v0ThumbnailType": {
"type": "string",
"enum": [
"PNG",
"JPG",
"GIF"
],
"default": "PNG",
"description": "The file types to which the thumbnail can be encoded to.\n\n - PNG: Represents PNG type\n - JPG: Represents JPG type\n - GIF: Represents GIF type"
},
"v0WebdavSource": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "REQUIRED."
},
"isPublicLink": {
"type": "boolean",
"description": "REQUIRED."
},
"webdavAuthorization": {
"type": "string",
"description": "OPTIONAL."
},
"revaAuthorization": {
"type": "string",
"description": "OPTIONAL."
},
"publicLinkToken": {
"type": "string",
"description": "OPTIONAL."
}
}
}
},
"externalDocs": {
"description": "Developer Manual",
"url": "https://docs.opencloud.eu/services/thumbnails/"
}
}
@@ -1,127 +0,0 @@
// Code generated by protoc-gen-go-grpc. DO NOT EDIT.
// versions:
// - protoc-gen-go-grpc v1.6.2
// - protoc (unknown)
// source: opencloud/services/thumbnails/v0/thumbnails.proto
package v0
import (
context "context"
grpc "google.golang.org/grpc"
codes "google.golang.org/grpc/codes"
status "google.golang.org/grpc/status"
)
// This is a compile-time assertion to ensure that this generated file
// is compatible with the grpc package it is being compiled against.
// Requires gRPC-Go v1.64.0 or later.
const _ = grpc.SupportPackageIsVersion9
const (
ThumbnailService_GetThumbnail_FullMethodName = "/opencloud.services.thumbnails.v0.ThumbnailService/GetThumbnail"
)
// ThumbnailServiceClient is the client API for ThumbnailService service.
//
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
//
// A Service for handling thumbnail generation
type ThumbnailServiceClient interface {
// Generates the thumbnail and returns it.
GetThumbnail(ctx context.Context, in *GetThumbnailRequest, opts ...grpc.CallOption) (*GetThumbnailResponse, error)
}
type thumbnailServiceClient struct {
cc grpc.ClientConnInterface
}
func NewThumbnailServiceClient(cc grpc.ClientConnInterface) ThumbnailServiceClient {
return &thumbnailServiceClient{cc}
}
func (c *thumbnailServiceClient) GetThumbnail(ctx context.Context, in *GetThumbnailRequest, opts ...grpc.CallOption) (*GetThumbnailResponse, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(GetThumbnailResponse)
err := c.cc.Invoke(ctx, ThumbnailService_GetThumbnail_FullMethodName, in, out, cOpts...)
if err != nil {
return nil, err
}
return out, nil
}
// ThumbnailServiceServer is the server API for ThumbnailService service.
// All implementations must embed UnimplementedThumbnailServiceServer
// for forward compatibility.
//
// A Service for handling thumbnail generation
type ThumbnailServiceServer interface {
// Generates the thumbnail and returns it.
GetThumbnail(context.Context, *GetThumbnailRequest) (*GetThumbnailResponse, error)
mustEmbedUnimplementedThumbnailServiceServer()
}
// UnimplementedThumbnailServiceServer must be embedded to have
// forward compatible implementations.
//
// NOTE: this should be embedded by value instead of pointer to avoid a nil
// pointer dereference when methods are called.
type UnimplementedThumbnailServiceServer struct{}
func (UnimplementedThumbnailServiceServer) GetThumbnail(context.Context, *GetThumbnailRequest) (*GetThumbnailResponse, error) {
return nil, status.Error(codes.Unimplemented, "method GetThumbnail not implemented")
}
func (UnimplementedThumbnailServiceServer) mustEmbedUnimplementedThumbnailServiceServer() {}
func (UnimplementedThumbnailServiceServer) testEmbeddedByValue() {}
// UnsafeThumbnailServiceServer may be embedded to opt out of forward compatibility for this service.
// Use of this interface is not recommended, as added methods to ThumbnailServiceServer will
// result in compilation errors.
type UnsafeThumbnailServiceServer interface {
mustEmbedUnimplementedThumbnailServiceServer()
}
func RegisterThumbnailServiceServer(s grpc.ServiceRegistrar, srv ThumbnailServiceServer) {
// If the following call panics, it indicates UnimplementedThumbnailServiceServer was
// embedded by pointer and is nil. This will cause panics if an
// unimplemented method is ever invoked, so we test this at initialization
// time to prevent it from happening at runtime later due to I/O.
if t, ok := srv.(interface{ testEmbeddedByValue() }); ok {
t.testEmbeddedByValue()
}
s.RegisterService(&ThumbnailService_ServiceDesc, srv)
}
func _ThumbnailService_GetThumbnail_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(GetThumbnailRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(ThumbnailServiceServer).GetThumbnail(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: ThumbnailService_GetThumbnail_FullMethodName,
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(ThumbnailServiceServer).GetThumbnail(ctx, req.(*GetThumbnailRequest))
}
return interceptor(ctx, in, info, handler)
}
// ThumbnailService_ServiceDesc is the grpc.ServiceDesc for ThumbnailService service.
// It's only intended for direct use with grpc.RegisterService,
// and not to be introspected or modified (even as a copy)
var ThumbnailService_ServiceDesc = grpc.ServiceDesc{
ServiceName: "opencloud.services.thumbnails.v0.ThumbnailService",
HandlerType: (*ThumbnailServiceServer)(nil),
Methods: []grpc.MethodDesc{
{
MethodName: "GetThumbnail",
Handler: _ThumbnailService_GetThumbnail_Handler,
},
},
Streams: []grpc.StreamDesc{},
Metadata: "opencloud/services/thumbnails/v0/thumbnails.proto",
}
@@ -1,30 +0,0 @@
syntax = "proto3";
package opencloud.messages.thumbnails.v0;
option go_package = "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/messages/thumbnails/v0";
message WebdavSource {
// REQUIRED.
string url = 1;
// REQUIRED.
bool is_public_link = 2;
// OPTIONAL.
string webdav_authorization = 3;
// OPTIONAL.
string reva_authorization = 4;
// OPTIONAL.
string public_link_token = 5;
}
message CS3Source {
string path = 1;
string authorization = 2;
}
// The file types to which the thumbnail can be encoded to.
enum ThumbnailType {
PNG = 0; // Represents PNG type
JPG = 1; // Represents JPG type
GIF = 2; // Represents GIF type
}
@@ -1,66 +0,0 @@
syntax = "proto3";
package opencloud.services.thumbnails.v0;
option go_package = "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/thumbnails/v0";
import "opencloud/messages/thumbnails/v0/thumbnails.proto";
import "protoc-gen-openapiv2/options/annotations.proto";
option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_swagger) = {
info: {
title: "OpenCloud thumbnails";
version: "1.0.0";
contact: {
name: "OpenCloud GmbH";
url: "https://github.com/opencloud-eu/opencloud";
email: "support@opencloud.eu";
};
license: {
name: "Apache-2.0";
url: "https://github.com/opencloud-eu/opencloud/blob/main/LICENSE";
};
};
schemes: HTTP;
schemes: HTTPS;
consumes: "application/json";
produces: "application/json";
external_docs: {
description: "Developer Manual";
url: "https://docs.opencloud.eu/services/thumbnails/";
};
};
// A Service for handling thumbnail generation
service ThumbnailService {
// Generates the thumbnail and returns it.
rpc GetThumbnail(GetThumbnailRequest) returns (GetThumbnailResponse);
}
// A request to retrieve a thumbnail
message GetThumbnailRequest {
// The path to the source image
string filepath = 1;
// The type to which the thumbnail should get encoded to.
opencloud.messages.thumbnails.v0.ThumbnailType thumbnail_type = 2;
// The width of the thumbnail
int32 width = 3;
// The height of the thumbnail
int32 height = 4;
// Indicates which image processor to use
string processor = 5;
oneof source {
opencloud.messages.thumbnails.v0.WebdavSource webdav_source = 6;
opencloud.messages.thumbnails.v0.CS3Source cs3_source = 7;
}
}
// The service response
message GetThumbnailResponse {
// The endpoint where the thumbnail can be downloaded.
string data_endpoint = 1;
// The transfer token to be able to download the thumbnail.
string transfer_token = 2;
// The mimetype of the thumbnail
string mimetype = 3;
}
+28 -130
View File
@@ -3,18 +3,17 @@ package command
import (
"context"
"fmt"
"strings"
"github.com/nats-io/nats.go"
"github.com/olekukonko/errors"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/pkg/runner"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/events/stream"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
"github.com/spf13/cobra"
"github.com/opencloud-eu/opencloud/pkg/config/configlog"
"github.com/opencloud-eu/opencloud/pkg/generators"
"github.com/opencloud-eu/opencloud/pkg/log"
natspkg "github.com/opencloud-eu/opencloud/pkg/nats"
"github.com/opencloud-eu/opencloud/pkg/registry"
"github.com/opencloud-eu/opencloud/pkg/runner"
ogrpc "github.com/opencloud-eu/opencloud/pkg/service/grpc"
"github.com/opencloud-eu/opencloud/pkg/tracing"
"github.com/opencloud-eu/opencloud/pkg/version"
@@ -25,12 +24,6 @@ import (
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/metrics"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/server/debug"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/server/http"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/activitylog"
svcEvents "github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/events"
svcHttp "github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/http"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/events/stream"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
)
var _registeredEvents = []events.Unmarshaller{
@@ -69,11 +62,19 @@ func Server(cfg *config.Config) *cobra.Command {
gr := runner.NewGroup()
ctx, cancel := context.WithCancel(cmd.Context())
defer cancel()
mtrcs := metrics.New()
mtrcs.BuildInfo.WithLabelValues(version.GetString()).Set(1)
defer cancel()
connName := generators.GenerateConnectionName(cfg.Service.Name, generators.NTypeBus)
evStream, err := stream.NatsFromConfig(connName, false, stream.NatsConfig(cfg.Events))
if err != nil {
logger.Error().Err(err).Msg("Failed to initialize event stream")
return err
}
tm, err := pool.StringToTLSMode(cfg.GRPCClientTLS.Mode)
if err != nil {
logger.Error().Err(err).Msg("Failed to parse tls mode")
@@ -98,101 +99,28 @@ func Server(cfg *config.Config) *cobra.Command {
return err
}
kv, err := ConnectNatsKV(cfg.Store)
if err != nil {
return err
}
activityLog, err := activitylog.New(kv,
activitylog.Logger(logger),
activitylog.MaxActivities(cfg.MaxActivities),
activitylog.WriteBufferDuration(cfg.WriteBufferDuration),
)
if err != nil {
logger.Error().Err(err).Msg("Failed to initialize activity log")
return err
}
hClient := ehsvc.NewEventHistoryService("eu.opencloud.api.eventhistory", grpcClient)
vClient := settingssvc.NewValueService("eu.opencloud.api.settings", grpcClient)
if !cfg.HTTP.Disabled {
hClient := ehsvc.NewEventHistoryService("eu.opencloud.api.eventhistory", grpcClient)
svc, err := svcHttp.New(
activityLog,
svcHttp.Logger(logger),
svcHttp.GatewaySelector(gatewaySelector),
svcHttp.RegisteredEvents(_registeredEvents),
//svcHttp.TraceProvider(tracerProvider),
svcHttp.HistoryClient(hClient),
)
if err != nil {
logger.Error().Err(err).Msg("handler init")
return err
}
// TODO svc = service.NewInstrument(svc, metrics)
// TODO svc = service.NewLogging(svc, logger) // this logs service specific data
// TODO svc = service.NewTracing(svc, traceProvider)
vClient := settingssvc.NewValueService("eu.opencloud.api.settings", grpcClient)
server, err := http.Server(
http.ValueClient(vClient),
{
svc, err := http.Server(
http.Logger(logger),
http.Context(ctx),
http.Config(cfg),
http.Service(svc),
http.Context(ctx), // NOTE: not passing this "option" leads to a panic in go-micro
http.TraceProvider(tracerProvider),
http.Stream(evStream),
http.GatewaySelector(gatewaySelector),
http.HistoryClient(hClient),
http.ValueClient(vClient),
http.RegisteredEvents(_registeredEvents),
)
if err != nil {
logger.Info().
Err(err).
Str("transport", "http").
Msg("Failed to initialize server")
if err != nil {
logger.Error().Err(err).Str("transport", "http").Msg("Failed to initialize server")
return err
}
gr.Add(runner.NewGoMicroHttpServerRunner(cfg.Service.Name+".http", server))
} else {
logger.Info().Msg("HTTP server disabled, not starting HTTP service")
}
if !cfg.Events.Disabled {
connName := generators.GenerateConnectionName(cfg.Service.Name, generators.NTypeBus)
evStream, err := stream.NatsFromConfig(connName, false, stream.NatsConfig{
Endpoint: cfg.Events.Endpoint,
Cluster: cfg.Events.Cluster,
EnableTLS: cfg.Events.EnableTLS,
TLSInsecure: cfg.Events.TLSInsecure,
TLSRootCACertificate: cfg.Events.TLSRootCACertificate,
AuthUsername: cfg.Events.AuthUsername,
AuthPassword: cfg.Events.AuthPassword,
})
if err != nil {
logger.Error().Err(err).Msg("Failed to initialize event stream")
return err
}
eventSvc, err := svcEvents.New(
activityLog,
evStream,
svcEvents.Context(ctx),
svcEvents.Logger(logger),
svcEvents.ServiceAccount(cfg.ServiceAccount),
svcEvents.GatewaySelector(gatewaySelector),
svcEvents.RegisteredEvents(_registeredEvents),
svcEvents.NumConsumers(cfg.NumConsumers),
)
if err != nil {
logger.Error().Err(err).Str("transport", "event").Msg("Failed to initialize server")
return err
}
gr.Add(runner.New(cfg.Service.Name+".svc", func() error {
return eventSvc.Run()
}, func() {
eventSvc.Close()
}))
} else {
logger.Info().Msg("event listening disabled, not starting event service")
gr.Add(runner.NewGoMicroHttpServerRunner(cfg.Service.Name+".http", svc))
}
{
@@ -221,33 +149,3 @@ func Server(cfg *config.Config) *cobra.Command {
},
}
}
func ConnectNatsKV(cfg config.Store) (nats.KeyValue, error) {
// Connect to NATS servers
secureOption := natspkg.Secure(cfg.EnableTLS, cfg.TLSInsecure, cfg.TLSRootCACertificate)
conn, err := nats.Connect(strings.Join(cfg.Nodes, ","), secureOption, nats.UserInfo(cfg.AuthUsername, cfg.AuthPassword))
if err != nil {
return nil, err
}
js, err := conn.JetStream()
if err != nil {
return nil, err
}
kv, err := js.KeyValue(cfg.Database)
if err != nil {
if !errors.Is(err, nats.ErrBucketNotFound) {
return nil, errors.Wrapf(err, "Failed to get bucket (%s)", cfg.Database)
}
kv, err = js.CreateKeyValue(&nats.KeyValueConfig{
Bucket: cfg.Database,
})
if err != nil {
return nil, errors.Wrapf(err, "Failed to create bucket (%s)", cfg.Database)
}
}
return kv, nil
}
@@ -35,12 +35,10 @@ type Config struct {
WriteBufferDuration time.Duration `yaml:"write_buffer_duration" env:"ACTIVITYLOG_WRITE_BUFFER_DURATION" desc:"The duration to wait before flushing the write buffer. This is used to reduce the number of writes to the store." introductionVersion:"4.0.0"`
MaxActivities int `yaml:"max_activities" env:"ACTIVITYLOG_MAX_ACTIVITIES" desc:"The maximum number of activities to keep in the store per resource. If the number of activities exceeds this value, the oldest activities will be removed." introductionVersion:"4.0.0"`
NumConsumers int `yaml:"num_consumers" env:"ACTIVITYLOG_NUM_CONSUMERS" desc:"The amount of concurrent event consumers to start. Event consumers are used for updating the list of activities. Multiple consumers increase parallelisation, but will also increase CPU and memory demands." introductionVersion:"%NEXT%"`
}
// Events combines the configuration options for the event bus.
type Events struct {
Disabled bool `yaml:"disabled" env:"ACTIVITYLOG_EVENTS_DISABLED" desc:"Disables listening for events. Set this to true if the service should only handle HTTP requests." introductionVersion:"%NEXT%"`
Endpoint string `yaml:"endpoint" env:"OC_EVENTS_ENDPOINT" desc:"The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture." introductionVersion:"1.0.0"`
Cluster string `yaml:"cluster" env:"OC_EVENTS_CLUSTER" desc:"The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system." introductionVersion:"1.0.0"`
TLSInsecure bool `yaml:"tls_insecure" env:"OC_INSECURE;OC_EVENTS_TLS_INSECURE" desc:"Whether to verify the server TLS certificates." introductionVersion:"1.0.0"`
@@ -79,7 +77,6 @@ type CORS struct {
// HTTP defines the available http configuration.
type HTTP struct {
Disabled bool `yaml:"disabled" env:"ACTIVITYLOG_HTTP_DISABLED" desc:"Disables the HTTP service. Set this to true if the service should only handle events." introductionVersion:"1.0.0"`
Addr string `yaml:"addr" env:"ACTIVITYLOG_HTTP_ADDR" desc:"The bind address of the HTTP service." introductionVersion:"1.0.0"`
Namespace string `yaml:"-"`
Root string `yaml:"root" env:"ACTIVITYLOG_HTTP_ROOT" desc:"Subdirectory that serves as the root for this HTTP service." introductionVersion:"1.0.0"`
@@ -52,9 +52,7 @@ func DefaultConfig() *config.Config {
},
},
WriteBufferDuration: 10 * time.Second,
// Nats runs into max payload exceeded errors at around 7k activities. Let's keep a buffer.
MaxActivities: 6000,
NumConsumers: 1,
MaxActivities: 6000,
}
}
+71 -23
View File
@@ -3,12 +3,18 @@ package http
import (
"context"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
"github.com/opencloud-eu/opencloud/pkg/log"
ehsvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/eventhistory/v0"
settingssvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/settings/v0"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/config"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/metrics"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
"github.com/spf13/pflag"
"go-micro.dev/v4/store"
"go.opentelemetry.io/otel/trace"
"go.opentelemetry.io/otel/trace/noop"
)
// Option defines a single option function.
@@ -16,15 +22,19 @@ type Option func(o *Options)
// Options defines the available options for this package.
type Options struct {
Name string
Namespace string
Logger log.Logger
Context context.Context
Config *config.Config
Flags []pflag.Flag
Service ActivityLogService
TraceProvider trace.TracerProvider
ValueClient settingssvc.ValueService
Logger log.Logger
Context context.Context
Config *config.Config
Metrics *metrics.Metrics
Flags []pflag.Flag
Namespace string
Store store.Store
Stream events.Stream
GatewaySelector pool.Selectable[gateway.GatewayAPIClient]
TraceProvider trace.TracerProvider
HistoryClient ehsvc.EventHistoryService
ValueClient settingssvc.ValueService
RegisteredEvents []events.Unmarshaller
}
// newOptions initializes the available default options.
@@ -59,10 +69,10 @@ func Config(val *config.Config) Option {
}
}
// Service provides a function to set the service option.
func Service(val ActivityLogService) Option {
// Metrics provides a function to set the metrics option.
func Metrics(val *metrics.Metrics) Option {
return func(o *Options) {
o.Service = val
o.Metrics = val
}
}
@@ -73,20 +83,58 @@ func Flags(flags ...pflag.Flag) Option {
}
}
// TraceProvider provides a function to configure the trace provider
func TraceProvider(traceProvider trace.TracerProvider) Option {
// Namespace provides a function to set the Namespace option.
func Namespace(val string) Option {
return func(o *Options) {
if traceProvider != nil {
o.TraceProvider = traceProvider
} else {
o.TraceProvider = noop.NewTracerProvider()
}
o.Namespace = val
}
}
// ValueClient adds a grpc client for the value service
func ValueClient(vs settingssvc.ValueService) Option {
// Store provides a function to configure the store
func Store(store store.Store) Option {
return func(o *Options) {
o.ValueClient = vs
o.Store = store
}
}
// Stream provides a function to configure the stream
func Stream(stream events.Stream) Option {
return func(o *Options) {
o.Stream = stream
}
}
// GatewaySelector provides a function to configure the gateway client selector
func GatewaySelector(gatewaySelector pool.Selectable[gateway.GatewayAPIClient]) Option {
return func(o *Options) {
o.GatewaySelector = gatewaySelector
}
}
// HistoryClient provides a function to configure the event history client
func HistoryClient(h ehsvc.EventHistoryService) Option {
return func(o *Options) {
o.HistoryClient = h
}
}
// RegisteredEvents provides a function to register events
func RegisteredEvents(evs []events.Unmarshaller) Option {
return func(o *Options) {
o.RegisteredEvents = evs
}
}
// TraceProvider provides a function to set the TracerProvider option
func TraceProvider(val trace.TracerProvider) Option {
return func(o *Options) {
o.TraceProvider = val
}
}
// ValueClient provides a function to set the ValueClient options
func ValueClient(val settingssvc.ValueService) Option {
return func(o *Options) {
o.ValueClient = val
}
}
+46 -103
View File
@@ -1,63 +1,49 @@
package http
import (
"context"
"embed"
"encoding/json"
"errors"
"net/http"
"fmt"
stdhttp "net/http"
"github.com/go-chi/chi/v5"
chimiddleware "github.com/go-chi/chi/v5/middleware"
libregraph "github.com/opencloud-eu/libre-graph-api-go"
"github.com/opencloud-eu/opencloud/pkg/account"
"github.com/opencloud-eu/opencloud/pkg/cors"
"github.com/opencloud-eu/opencloud/pkg/l10n"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/pkg/middleware"
ohttp "github.com/opencloud-eu/opencloud/pkg/service/http"
"github.com/opencloud-eu/opencloud/pkg/service/http"
"github.com/opencloud-eu/opencloud/pkg/tracing"
"github.com/opencloud-eu/opencloud/pkg/version"
settingssvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/settings/v0"
activityloghttp "github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/http"
revactx "github.com/opencloud-eu/reva/v2/pkg/ctx"
svc "github.com/opencloud-eu/opencloud/services/activitylog/pkg/service"
"github.com/riandyrn/otelchi"
"go-micro.dev/v4"
"google.golang.org/grpc/metadata"
)
var (
//go:embed l10n/locale
_localeFS embed.FS
// subfolder where the translation files are stored
_localeSubPath = "l10n/locale"
// domain of the activitylog service (transifex)
_domain = "activitylog"
)
// Service is the service interface
type Service any
// Server initializes the http service and server.
func Server(opts ...Option) (ohttp.Service, error) {
func Server(opts ...Option) (http.Service, error) {
options := newOptions(opts...)
service := options.Service
newService, err := ohttp.NewService(
ohttp.TLSConfig(options.Config.HTTP.TLS),
ohttp.Logger(options.Logger),
ohttp.Namespace(options.Config.HTTP.Namespace),
ohttp.Name(options.Config.Service.Name),
ohttp.Version(version.GetString()),
ohttp.Address(options.Config.HTTP.Addr),
ohttp.Context(options.Context),
ohttp.Flags(options.Flags...),
service, err := http.NewService(
http.TLSConfig(options.Config.HTTP.TLS),
http.Logger(options.Logger),
http.Namespace(options.Config.HTTP.Namespace),
http.Name(options.Config.Service.Name),
http.Version(version.GetString()),
http.Address(options.Config.HTTP.Addr),
http.Context(options.Context),
http.Flags(options.Flags...),
http.TraceProvider(options.TraceProvider),
)
if err != nil {
options.Logger.Error().
Err(err).
Msg("Error initializing http service")
return ohttp.Service{}, err
return http.Service{}, fmt.Errorf("could not initialize http service: %w", err)
}
middlewares := []func(http.Handler) http.Handler{
middlewares := []func(stdhttp.Handler) stdhttp.Handler{
chimiddleware.RequestID,
middleware.Version(
options.Config.Service.Name,
@@ -66,7 +52,6 @@ func Server(opts ...Option) (ohttp.Service, error) {
middleware.Logger(
options.Logger,
),
middleware.TraceContext,
middleware.ExtractAccountUUID(
account.Logger(options.Logger),
account.JWTSecret(options.Config.TokenManager.JWTSecret),
@@ -83,75 +68,33 @@ func Server(opts ...Option) (ohttp.Service, error) {
mux := chi.NewMux()
mux.Use(middlewares...)
t := l10n.NewTranslatorFromCommonConfig(options.Config.DefaultLanguage, _domain, options.Config.TranslationPath, _localeFS, _localeSubPath)
mux.Route(options.Config.HTTP.Root, func(r chi.Router) {
r.Get("/graph/v1beta1/extensions/org.libregraph/activities", GetItemActivitiesHandler(options.Logger, service, options.ValueClient, t))
})
mux.Use(
otelchi.Middleware(
"actitivylog",
otelchi.WithChiRoutes(mux),
otelchi.WithTracerProvider(options.TraceProvider),
otelchi.WithPropagators(tracing.GetPropagator()),
),
)
err = micro.RegisterHandler(newService.Server(), mux)
handle, err := svc.New(
svc.Logger(options.Logger),
svc.Stream(options.Stream),
svc.Mux(mux),
svc.Config(options.Config),
svc.GatewaySelector(options.GatewaySelector),
svc.TraceProvider(options.TraceProvider),
svc.HistoryClient(options.HistoryClient),
svc.ValueClient(options.ValueClient),
svc.RegisteredEvents(options.RegisteredEvents),
)
if err != nil {
options.Logger.Fatal().Err(err).Msg("failed to register the handler")
return http.Service{}, err
}
newService.Init()
return newService, nil
}
// Service defines the business logic implementations need to provide.
type ActivityLogService interface {
GetItemActivities(ctx context.Context, query, loc string, t l10n.Translator) ([]libregraph.Activity, error)
}
// GetActivitiesResponse is the response on GET activities requests
type GetActivitiesResponse struct {
Activities []libregraph.Activity `json:"value"`
}
func GetItemActivitiesHandler(log log.Logger, s ActivityLogService, vc settingssvc.ValueService, t l10n.Translator) func(w http.ResponseWriter, r *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
ctx = metadata.AppendToOutgoingContext(ctx, revactx.TokenHeader, r.Header.Get(revactx.TokenHeader))
activeUser, ok := revactx.ContextGetUser(ctx)
if !ok {
w.WriteHeader(http.StatusUnauthorized)
return
}
loc := l10n.MustGetUserLocale(ctx, activeUser.GetId().GetOpaqueId(), r.Header.Get(l10n.HeaderAcceptLanguage), vc)
activities, err := s.GetItemActivities(ctx, r.URL.Query().Get("kql"), loc, t)
if err != nil {
switch {
case errors.Is(err, activityloghttp.ErrBadRequest):
log.Debug().Str("query", r.URL.Query().Get("kql")).Err(err).Msg("error getting activities")
w.WriteHeader(http.StatusBadRequest)
return
case errors.Is(err, activityloghttp.ErrForbidden):
log.Debug().Err(err).Msg("error getting activities")
w.WriteHeader(http.StatusForbidden)
return
default:
log.Error().Err(err).Msg("error getting activities")
w.WriteHeader(http.StatusInternalServerError)
return
}
}
res := GetActivitiesResponse{
Activities: activities,
}
w.Header().Set("Content-Type", "application/json; odata.metadata=minimal")
w.Header().Set("OData-Version", "4.0")
if reqID := chimiddleware.GetReqID(ctx); reqID != "" {
w.Header().Set("request-id", reqID)
}
w.Header().Set("Cache-Control", "no-cache")
if err := json.NewEncoder(w).Encode(res); err != nil {
log.Error().Err(err).Msg("error encoding activities")
w.WriteHeader(http.StatusInternalServerError)
return
}
if err := micro.RegisterHandler(service.Server(), handle); err != nil {
return http.Service{}, err
}
return service, nil
}
@@ -1,366 +0,0 @@
package activitylog
import (
"context"
"encoding/base32"
"encoding/json"
"fmt"
"sort"
"strconv"
"strings"
"sync"
"time"
provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
"github.com/jellydator/ttlcache/v2"
"github.com/nats-io/nats.go"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/reva/v2/pkg/storagespace"
"github.com/vmihailenco/msgpack/v5"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/trace"
)
var tracer trace.Tracer
func init() {
tracer = otel.Tracer("github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/activitylog")
}
var (
_maxActivitiesDefault = 6000
_writeBufferDuration = 10 * time.Second
)
// Activitylog stores and retrieves activities for resources and their parents from a nats kv
type ActivityLog struct {
log log.Logger
// FIXME the lock does not protect agains concurrent resource activities on multiple instances
// known since https://github.com/owncloud/ocis/pull/9361#pullrequestreview-2135350157
// current ocis discussion in https://github.com/owncloud/ocis/issues/12475
lock sync.RWMutex
debouncer *Debouncer
parentIdCache *ttlcache.Cache
natskv nats.KeyValue
maxActivities int
}
type batchInfo struct {
key string
count int
timestamp time.Time
}
// New creates a new ActivitylogService
func New(kv nats.KeyValue, opts ...Option) (*ActivityLog, error) {
o := &Options{
MaxActivities: _maxActivitiesDefault,
WriteBufferDuration: _writeBufferDuration,
Logger: log.NopLogger(),
}
for _, opt := range opts {
opt(o)
}
cache := ttlcache.NewCache()
err := cache.SetTTL(30 * time.Second)
if err != nil {
return nil, err
}
s := &ActivityLog{
log: o.Logger,
lock: sync.RWMutex{},
parentIdCache: cache,
maxActivities: o.MaxActivities,
natskv: kv,
}
s.debouncer = NewDebouncer(o.WriteBufferDuration, s.StoreActivity)
// run migrations
err = s.runMigrations(context.Background(), kv)
if err != nil {
return nil, err
}
return s, nil
}
// RemoveResource removes the resource from the store
func (a *ActivityLog) RemoveResource(rid *provider.ResourceId) error {
if rid == nil {
return fmt.Errorf("resource id is required")
}
a.lock.Lock()
defer a.lock.Unlock()
err := a.natskv.Delete(storagespace.FormatResourceID(rid))
if err != nil {
return fmt.Errorf("could not delete resource %s: %w", rid.OpaqueId, err)
}
return nil
}
// RawActivity represents an activity as it is stored in the activitylog store
type RawActivity struct {
EventID string `json:"event_id"`
Depth int `json:"depth"`
Timestamp time.Time `json:"timestamp"`
}
func (a *ActivityLog) AddActivity(ctx context.Context, initRef *provider.Reference, parentId *provider.ResourceId, eventID string, timestamp time.Time, getResource func(context.Context, *provider.Reference) (*provider.ResourceInfo, error)) error {
var (
err error
depth int
ref = initRef
)
ctx, span := tracer.Start(ctx, "AddActivity")
defer span.End()
for {
var info *provider.ResourceInfo
id := ref.GetResourceId()
if ref.Path != "" {
// Path based reference, we need to resolve the resource id
ctx, span = tracer.Start(ctx, "AddActivity.getResource")
info, err = getResource(ctx, ref)
span.End()
if err != nil {
// TODO If the resource was deleted should we still log an activity in the parent?
return fmt.Errorf("could not get resource info for reference %v: %w", ref, err)
}
id = info.GetId()
}
if id == nil {
return fmt.Errorf("resource id is required")
}
key := storagespace.FormatResourceID(id)
a.debouncer.Debounce(key, RawActivity{
EventID: eventID,
Depth: depth,
Timestamp: timestamp,
})
if id.OpaqueId == id.SpaceId {
// we are at the root of the space, no need to go further
break
}
// check if parent id is cached
// parent id is cached in the format <storageid>$<spaceid>!<resourceid>
// if it is not cached, get the resource info and cache it
if parentId == nil {
if v, err := a.parentIdCache.Get(key); err != nil {
if info == nil {
ctx, span := tracer.Start(ctx, "AddActivity.getResource parent")
info, err = getResource(ctx, ref)
span.End()
if err != nil || info.GetParentId() == nil || info.GetParentId().GetOpaqueId() == "" {
return fmt.Errorf("could not get parent id: %w", err)
}
}
parentId = info.GetParentId()
a.parentIdCache.Set(key, parentId)
} else {
parentId = v.(*provider.ResourceId)
}
} else {
a.log.Debug().Msg("parent id is cached")
}
depth++
ref = &provider.Reference{ResourceId: parentId}
parentId = nil // reset parent id so it's not reused in the next iteration
}
return nil
}
func (a *ActivityLog) StoreActivity(resourceID string, activities []RawActivity) error {
a.lock.Lock()
defer a.lock.Unlock()
ctx, span := tracer.Start(context.Background(), "storeActivity")
defer span.End()
_, subspan := tracer.Start(ctx, "storeActivity.Marshal")
b, err := msgpack.Marshal(activities)
if err != nil {
return err
}
subspan.End()
_, subspan = tracer.Start(ctx, "storeActivity.natskv.Put")
key := natsKey(resourceID, len(activities))
_, err = a.natskv.Put(key, b)
if err != nil {
return err
}
subspan.End()
ctx, subspan = tracer.Start(ctx, "storeActivity.enforceMaxActivities")
a.enforceMaxActivities(ctx, resourceID)
subspan.End()
return nil
}
func (a *ActivityLog) enforceMaxActivities(ctx context.Context, resourceID string) {
if a.maxActivities <= 0 {
return
}
key := fmt.Sprintf("%s.>", base32.StdEncoding.EncodeToString([]byte(resourceID)))
_, subspan := tracer.Start(ctx, "enforceMaxActivities.watch")
watcher, err := a.natskv.Watch(key, nats.IgnoreDeletes())
if err != nil {
a.log.Error().Err(err).Str("resourceID", resourceID).Msg("could not watch")
return
}
defer watcher.Stop()
var keys []string
for update := range watcher.Updates() {
if update == nil {
break
}
var batchActivities []RawActivity
if err := msgpack.Unmarshal(update.Value(), &batchActivities); err != nil {
a.log.Debug().Err(err).Str("resourceID", resourceID).Msg("could not unmarshal messagepack, trying json")
}
keys = append(keys, update.Key())
}
subspan.End()
_, subspan = tracer.Start(ctx, "enforceMaxActivities.compile")
// Parse keys into batches
batches := make([]batchInfo, 0)
var activitiesCount int
for _, k := range keys {
parts := strings.SplitN(k, ".", 3)
if len(parts) < 3 {
a.log.Warn().Str("key", k).Msg("skipping key, not enough parts")
continue
}
c, err := strconv.Atoi(parts[1])
if err != nil {
a.log.Warn().Str("key", k).Msg("skipping key, can not parse count")
continue
}
// parse timestamp
nano, err := strconv.ParseInt(parts[2], 10, 64)
if err != nil {
a.log.Warn().Str("key", k).Msg("skipping key, can not parse timestamp")
continue
}
batches = append(batches, batchInfo{
key: k,
count: c,
timestamp: time.Unix(0, nano),
})
activitiesCount += c
}
// sort batches by timestamp
sort.Slice(batches, func(i, j int) bool {
return batches[i].timestamp.Before(batches[j].timestamp)
})
subspan.End()
_, subspan = tracer.Start(ctx, "enforceMaxActivities.delete")
// remove oldest keys until we are at max activities
for _, b := range batches {
if activitiesCount-b.count < a.maxActivities {
break
}
activitiesCount -= b.count
err = a.natskv.Delete(b.key)
if err != nil {
a.log.Error().Err(err).Str("key", b.key).Msg("could not delete key")
break
}
}
subspan.End()
}
func (a *ActivityLog) InvalidateCachedParentID(purgeId *provider.ResourceId) {
// The parent id cache is populated lazily and its entries expire, so a
// missing key is the expected case rather than an error.
if err := a.parentIdCache.Remove(storagespace.FormatResourceID(purgeId)); err != nil {
a.log.Debug().Interface("event", purgeId).Err(err).Msg("could not delete parent id cache")
}
}
func natsKey(resourceID string, activitiesCount int) string {
return fmt.Sprintf("%s.%d.%d",
base32.StdEncoding.EncodeToString([]byte(resourceID)),
activitiesCount,
time.Now().UnixNano())
}
func (a *ActivityLog) Activities(rid *provider.ResourceId) ([]RawActivity, error) {
a.lock.RLock()
defer a.lock.RUnlock()
return a.activities(rid)
}
func (a *ActivityLog) activities(rid *provider.ResourceId) ([]RawActivity, error) {
resourceID := storagespace.FormatResourceID(rid)
glob := fmt.Sprintf("%s.>", base32.StdEncoding.EncodeToString([]byte(resourceID)))
watcher, err := a.natskv.Watch(glob, nats.IgnoreDeletes())
if err != nil {
return nil, err
}
defer watcher.Stop()
var activities []RawActivity
for update := range watcher.Updates() {
if update == nil {
break
}
var batchActivities []RawActivity
if err := msgpack.Unmarshal(update.Value(), &batchActivities); err != nil {
a.log.Debug().Err(err).Str("resourceID", resourceID).Msg("could not unmarshal messagepack")
}
activities = append(activities, batchActivities...)
}
return activities, nil
}
// RemoveActivities removes the activities from the given resource
func (a *ActivityLog) RemoveActivities(rid *provider.ResourceId, toDelete map[string]struct{}) error {
a.lock.Lock()
defer a.lock.Unlock()
curActivities, err := a.activities(rid)
if err != nil {
return err
}
var acts []RawActivity
for _, a := range curActivities {
if _, ok := toDelete[a.EventID]; !ok {
acts = append(acts, a)
}
}
b, err := json.Marshal(acts)
if err != nil {
return err
}
_, err = a.natskv.Put(storagespace.FormatResourceID(rid), b)
return err
}
@@ -1,13 +0,0 @@
package activitylog_test
import (
"testing"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
func TestActivitylog(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Activitylog Suite")
}
@@ -1,76 +0,0 @@
package activitylog
import (
"sync"
"time"
)
// Debouncer is used to debounce writes to the activity log store.
type Debouncer struct {
after time.Duration
f func(id string, ra []RawActivity) error
pending sync.Map
inProgress sync.Map
mutex sync.Mutex
}
type queueItem struct {
activities []RawActivity
timer *time.Timer
}
// NewDebouncer returns a new Debouncer instance.
func NewDebouncer(d time.Duration, f func(id string, ra []RawActivity) error) *Debouncer {
return &Debouncer{
after: d,
f: f,
pending: sync.Map{},
inProgress: sync.Map{},
}
}
// Debounce restarts the debounce timer for the given space.
func (d *Debouncer) Debounce(id string, ra RawActivity) {
if d.after == 0 {
d.f(id, []RawActivity{ra})
return
}
d.mutex.Lock()
defer d.mutex.Unlock()
item := &queueItem{
activities: []RawActivity{ra},
}
if i, ok := d.pending.Load(id); ok {
// if the item is already in the queue, append the new activities
item, ok = i.(*queueItem)
if ok {
item.activities = append(item.activities, ra)
}
}
if item.timer == nil {
item.timer = time.AfterFunc(d.after, func() {
if _, ok := d.inProgress.Load(id); ok {
// Reschedule this run for when the previous run has finished
d.mutex.Lock()
if i, ok := d.pending.Load(id); ok {
i.(*queueItem).timer.Reset(d.after)
}
d.mutex.Unlock()
return
}
d.pending.Delete(id)
d.inProgress.Store(id, true)
defer d.inProgress.Delete(id)
d.f(id, item.activities)
})
}
d.pending.Store(id, item)
}
@@ -1,121 +0,0 @@
package activitylog_test
import (
"sync"
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/activitylog"
)
var _ = Describe("Debouncer", func() {
var (
mu sync.Mutex
callbacks []activitylog.RawActivity
newCallback func(id string, ra []activitylog.RawActivity) error
)
BeforeEach(func() {
mu.Lock()
callbacks = nil
mu.Unlock()
newCallback = func(id string, ra []activitylog.RawActivity) error {
mu.Lock()
defer mu.Unlock()
callbacks = append(callbacks, ra...)
return nil
}
})
Context("with zero duration", func() {
It("calls the callback immediately", func() {
d := activitylog.NewDebouncer(0, newCallback)
d.Debounce("space1", activitylog.RawActivity{EventID: "activity1"})
Expect(callbacks).To(HaveLen(1))
Expect(callbacks[0].EventID).To(Equal("activity1"))
})
It("calls the callback immediately for each event", func() {
d := activitylog.NewDebouncer(0, newCallback)
d.Debounce("space1", activitylog.RawActivity{EventID: "activity1"})
d.Debounce("space2", activitylog.RawActivity{EventID: "activity2"})
Expect(callbacks).To(HaveLen(2))
})
})
Context("with non-zero duration", func() {
It("batches activities with the same id", func() {
d := activitylog.NewDebouncer(10*time.Millisecond, newCallback)
d.Debounce("space1", activitylog.RawActivity{EventID: "activity1"})
d.Debounce("space1", activitylog.RawActivity{EventID: "activity2"})
d.Debounce("space1", activitylog.RawActivity{EventID: "activity3"})
Eventually(func() int {
mu.Lock()
defer mu.Unlock()
return len(callbacks)
}).Should(Equal(3))
})
It("handles different ids independently", func() {
d := activitylog.NewDebouncer(10*time.Millisecond, newCallback)
d.Debounce("space1", activitylog.RawActivity{EventID: "activity1"})
d.Debounce("space2", activitylog.RawActivity{EventID: "activity2"})
Eventually(func() int {
mu.Lock()
defer mu.Unlock()
return len(callbacks)
}).Should(Equal(2))
})
It("batches activities that arrive within the debounce window", func() {
d := activitylog.NewDebouncer(100*time.Millisecond, newCallback)
d.Debounce("space1", activitylog.RawActivity{EventID: "activity1"})
time.Sleep(20 * time.Millisecond)
d.Debounce("space1", activitylog.RawActivity{EventID: "activity2"})
time.Sleep(20 * time.Millisecond)
d.Debounce("space1", activitylog.RawActivity{EventID: "activity3"})
Eventually(func() int {
mu.Lock()
defer mu.Unlock()
return len(callbacks)
}).Should(Equal(3))
})
It("processes new batch after previous completes", func() {
d := activitylog.NewDebouncer(5*time.Millisecond, newCallback)
d.Debounce("space1", activitylog.RawActivity{EventID: "activity1"})
time.Sleep(20 * time.Millisecond) // let first batch complete
d.Debounce("space1", activitylog.RawActivity{EventID: "activity2"})
Eventually(func() int {
mu.Lock()
defer mu.Unlock()
return len(callbacks)
}).Should(Equal(2))
})
It("skips duplicate write when timer fires during in-progress callback", func() {
slowCallback := func(id string, ra []activitylog.RawActivity) error {
time.Sleep(50 * time.Millisecond) // simulate slow write
mu.Lock()
defer mu.Unlock()
callbacks = append(callbacks, ra...)
return nil
}
d := activitylog.NewDebouncer(10*time.Millisecond, slowCallback)
d.Debounce("space1", activitylog.RawActivity{EventID: "activity1"})
time.Sleep(20 * time.Millisecond) // timer fires while callback is running
Eventually(func() int {
mu.Lock()
defer mu.Unlock()
return len(callbacks)
}).Should(Equal(1))
})
})
})
@@ -1,35 +0,0 @@
package activitylog
import (
"time"
"github.com/opencloud-eu/opencloud/pkg/log"
)
// Option for the activitylog service
type Option func(*Options)
// Options for the activitylog service
type Options struct {
Logger log.Logger
MaxActivities int
WriteBufferDuration time.Duration
}
// Logger configures a logger for the activitylog service
func Logger(log log.Logger) Option {
return func(o *Options) {
o.Logger = log
}
}
func MaxActivities(max int) Option {
return func(o *Options) {
o.MaxActivities = max
}
}
func WriteBufferDuration(d time.Duration) Option {
return func(o *Options) {
o.WriteBufferDuration = d
}
}
@@ -1,302 +0,0 @@
package events
import (
"context"
"fmt"
"path/filepath"
"sync"
"sync/atomic"
"time"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/config"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/activitylog"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
"github.com/opencloud-eu/reva/v2/pkg/storagespace"
"github.com/opencloud-eu/reva/v2/pkg/utils"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/trace"
)
var tracer trace.Tracer
func init() {
tracer = otel.Tracer("github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/events")
}
var (
_numConsumersDefault = 1
)
// ActivitylogService logs events per resource
type ActivitylogService struct {
ctx context.Context
sa config.ServiceAccount
log log.Logger
stream events.Stream
gws pool.Selectable[gateway.GatewayAPIClient]
al *activitylog.ActivityLog
numConsumers int
events []events.Unmarshaller
stopCh chan struct{}
stopped *atomic.Bool
}
// New creates a new ActivitylogService
func New(al *activitylog.ActivityLog, stream events.Stream, opts ...Option) (*ActivitylogService, error) {
o := &Options{
NumConsumers: _numConsumersDefault,
}
for _, opt := range opts {
opt(o)
}
s := &ActivitylogService{
ctx: o.Context,
log: o.Logger,
sa: o.ServiceAccount,
stream: stream,
gws: o.GatewaySelector,
events: o.RegisteredEvents,
numConsumers: o.NumConsumers,
al: al,
stopCh: make(chan struct{}, 1),
stopped: new(atomic.Bool),
}
return s, nil
}
// Run to fulfil Runner interface
func (s *ActivitylogService) Run() error {
ch, err := events.Consume(s.stream, "activitylog", s.events...)
if err != nil {
return err
}
var wg sync.WaitGroup
ctx, cancel := context.WithCancel(s.ctx)
defer cancel()
s.log.Debug().Int("worker.count", s.numConsumers).
Str("messaging.consumer.group.name", "activitylog").
Str("messaging.system", "nats").
Str("messaging.operation.name", "receive").
Msg("starting event processing workers")
// start workers
for i := 0; i < s.numConsumers; i++ {
wg.Add(1)
go func(workerID int) {
defer wg.Done()
for {
select {
case <-ctx.Done():
return
case e, ok := <-ch:
if !ok {
return
}
if err := s.processEvent(e); err != nil {
s.log.Error().Err(err).
Int("worker", workerID).
Interface("event", e).
Msg("failed to process event")
}
}
}
}(i)
}
// wait for stop signal
<-s.stopCh
cancel() // signal workers to stop
wg.Wait()
return nil
}
// Close will make the service to stop processing, so the `Run`
// method can finish.
func (s *ActivitylogService) Close() {
if s.stopped.CompareAndSwap(false, true) {
close(s.stopCh)
}
}
func (s *ActivitylogService) processEvent(e events.Event) error {
ctx := e.GetTraceContext(s.ctx)
ctx, span := tracer.Start(ctx, "processEvent")
defer span.End()
s.log.Debug().Interface("event", e).Msg("updating activitylog")
switch ev := e.Event.(type) {
case events.UploadReady:
return s.AddActivity(ctx, ev.FileRef, ev.ParentID, e.ID, utils.TSToTime(ev.Timestamp))
case events.FileTouched:
return s.AddActivity(ctx, ev.Ref, ev.ParentID, e.ID, utils.TSToTime(ev.Timestamp))
// Disabled https://github.com/owncloud/ocis/issues/10293
//case events.FileDownloaded:
// we are only interested in public link downloads - so no need to store others.
//if ev.ImpersonatingUser.GetDisplayName() == "Public" {
// err = a.AddActivity(ev.Ref, e.ID, utils.TSToTime(ev.Timestamp))
//}
case events.ContainerCreated:
return s.AddActivity(ctx, ev.Ref, ev.ParentID, e.ID, utils.TSToTime(ev.Timestamp))
case events.ItemTrashed:
return s.AddActivityTrashed(ctx, ev.ID, ev.Ref, nil, e.ID, utils.TSToTime(ev.Timestamp))
case events.ItemPurged:
return s.al.RemoveResource(ev.ID)
case events.ItemMoved:
// remove the cached parent id for this resource
s.removeCachedParentID(ctx, ev.Ref)
return s.AddActivity(ctx, ev.Ref, nil, e.ID, utils.TSToTime(ev.Timestamp))
case events.ShareCreated:
return s.AddActivity(ctx, toRef(ev.ItemID), nil, e.ID, utils.TSToTime(ev.CTime))
case events.ShareUpdated:
if ev.Sharer != nil && ev.ItemID != nil && ev.Sharer.GetOpaqueId() != ev.ItemID.GetSpaceId() {
return s.AddActivity(ctx, toRef(ev.ItemID), nil, e.ID, utils.TSToTime(ev.MTime))
}
case events.ShareRemoved:
return s.AddActivity(ctx, toRef(ev.ItemID), nil, e.ID, ev.Timestamp)
case events.LinkCreated:
return s.AddActivity(ctx, toRef(ev.ItemID), nil, e.ID, utils.TSToTime(ev.CTime))
case events.LinkUpdated:
if ev.Sharer != nil && ev.ItemID != nil && ev.Sharer.GetOpaqueId() != ev.ItemID.GetSpaceId() {
return s.AddActivity(ctx, toRef(ev.ItemID), nil, e.ID, utils.TSToTime(ev.MTime))
}
case events.LinkRemoved:
return s.AddActivity(ctx, toRef(ev.ItemID), nil, e.ID, utils.TSToTime(ev.Timestamp))
case events.SpaceShared:
return s.AddSpaceActivity(ctx, ev.ID, e.ID, ev.Timestamp)
case events.SpaceUnshared:
return s.AddSpaceActivity(ctx, ev.ID, e.ID, ev.Timestamp)
}
return nil
}
// AddActivity adds the activity to the given resource and all its parents
func (a *ActivitylogService) AddActivity(ctx context.Context, initRef *provider.Reference, parentId *provider.ResourceId, eventID string, timestamp time.Time) error {
ctx, span := tracer.Start(ctx, "AddActivity")
defer span.End()
gwc, err := a.gws.Next()
if err != nil {
return fmt.Errorf("cant get gateway client: %w", err)
}
ctx, err = utils.GetServiceUserContextWithContext(ctx, gwc, a.sa.ServiceAccountID, a.sa.ServiceAccountSecret)
if err != nil {
return fmt.Errorf("cant get service user context: %w", err)
}
return a.al.AddActivity(ctx, initRef, parentId, eventID, timestamp, func(ctx context.Context, ref *provider.Reference) (*provider.ResourceInfo, error) {
return utils.GetResource(ctx, ref, gwc)
})
}
// AddActivityTrashed adds the activity to given trashed resource and all its former parents
func (a *ActivitylogService) AddActivityTrashed(ctx context.Context, resourceID *provider.ResourceId, reference *provider.Reference, parentId *provider.ResourceId, eventID string, timestamp time.Time) error {
ctx, span := tracer.Start(ctx, "AddActivityTrashed")
defer span.End()
gwc, err := a.gws.Next()
if err != nil {
return fmt.Errorf("cant get gateway client: %w", err)
}
ctx, err = utils.GetServiceUserContextWithContext(ctx, gwc, a.sa.ServiceAccountID, a.sa.ServiceAccountSecret)
if err != nil {
return fmt.Errorf("cant get service user context: %w", err)
}
// store activity on trashed item
if err := a.al.StoreActivity(storagespace.FormatResourceID(resourceID), []activitylog.RawActivity{
{
EventID: eventID,
Depth: 0,
Timestamp: timestamp,
},
}); err != nil {
return fmt.Errorf("could not store activity: %w", err)
}
// get previous parent
ref := &provider.Reference{
ResourceId: reference.GetResourceId(),
Path: filepath.Dir(reference.GetPath()),
}
return a.al.AddActivity(ctx, ref, parentId, eventID, timestamp, func(ctx context.Context, ref *provider.Reference) (*provider.ResourceInfo, error) {
return utils.GetResource(ctx, ref, gwc)
})
}
// AddSpaceActivity adds the activity to the given spaceroot
func (a *ActivitylogService) AddSpaceActivity(ctx context.Context, spaceID *provider.StorageSpaceId, eventID string, timestamp time.Time) error {
_, span := tracer.Start(ctx, "AddSpaceActivity")
defer span.End()
// spaceID is in format <providerid>$<spaceid>
// activitylog service uses format <providerid>$<spaceid>!<resourceid>
// lets do some converting, shall we?
rid, err := storagespace.ParseID(spaceID.GetOpaqueId())
if err != nil {
return fmt.Errorf("could not parse space id: %w", err)
}
rid.OpaqueId = rid.GetSpaceId()
err = a.al.StoreActivity(storagespace.FormatResourceID(&rid), []activitylog.RawActivity{
{
EventID: eventID,
Depth: 0,
Timestamp: timestamp,
},
})
if err != nil {
return fmt.Errorf("could not store activity: %w", err)
}
return nil
}
func toRef(r *provider.ResourceId) *provider.Reference {
return &provider.Reference{
ResourceId: r,
}
}
func (a *ActivitylogService) removeCachedParentID(ctx context.Context, ref *provider.Reference) {
var span trace.Span
ctx, span = tracer.Start(ctx, "removeCachedParentID")
defer span.End()
purgeId := ref.GetResourceId()
if ref.GetPath() != "" {
gwc, err := a.gws.Next()
if err != nil {
a.log.Error().Err(err).Msg("could not get gateway client")
return
}
ctx, err = utils.GetServiceUserContextWithContext(ctx, gwc, a.sa.ServiceAccountID, a.sa.ServiceAccountSecret)
if err != nil {
a.log.Error().Err(err).Msg("could not get service user context")
return
}
info, err := utils.GetResource(ctx, ref, gwc)
if err != nil {
a.log.Error().Err(err).Msg("could not get resource info")
return
}
purgeId = info.GetId()
}
a.al.InvalidateCachedParentID(purgeId)
}
@@ -1,123 +0,0 @@
package events_test
import (
"context"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/config"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/activitylog"
eventssvc "github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/events"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
)
var _ = Describe("ActivitylogService", func() {
Describe("New", func() {
var (
al *activitylog.ActivityLog
stream events.Stream
)
BeforeEach(func() {
al = &activitylog.ActivityLog{}
stream = nil
})
It("creates a service with minimal options", func() {
svc, err := eventssvc.New(al, stream)
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with context option", func() {
ctx := context.Background()
svc, err := eventssvc.New(al, stream, eventssvc.Context(ctx))
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with logger option", func() {
logger := log.NopLogger()
svc, err := eventssvc.New(al, stream, eventssvc.Logger(logger))
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with service account option", func() {
sa := config.ServiceAccount{
ServiceAccountID: "sa-id",
ServiceAccountSecret: "sa-secret",
}
svc, err := eventssvc.New(al, stream, eventssvc.ServiceAccount(sa))
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with registered events", func() {
evts := []events.Unmarshaller{&events.UploadReady{}, &events.FileTouched{}}
svc, err := eventssvc.New(al, stream, eventssvc.RegisteredEvents(evts))
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with gateway selector", func() {
var gs pool.Selectable[gateway.GatewayAPIClient]
svc, err := eventssvc.New(al, stream, eventssvc.GatewaySelector(gs))
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with num consumers option", func() {
svc, err := eventssvc.New(al, stream, eventssvc.NumConsumers(5))
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with all options", func() {
ctx := context.Background()
logger := log.NopLogger()
sa := config.ServiceAccount{
ServiceAccountID: "sa-id",
ServiceAccountSecret: "sa-secret",
}
evts := []events.Unmarshaller{&events.UploadReady{}, &events.ContainerCreated{}}
var gs pool.Selectable[gateway.GatewayAPIClient]
svc, err := eventssvc.New(
al,
stream,
eventssvc.Context(ctx),
eventssvc.Logger(logger),
eventssvc.ServiceAccount(sa),
eventssvc.RegisteredEvents(evts),
eventssvc.GatewaySelector(gs),
eventssvc.NumConsumers(3),
)
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
})
Describe("Close", func() {
It("can be called without panic on a new service", func() {
al := &activitylog.ActivityLog{}
svc, err := eventssvc.New(al, nil)
Expect(err).ToNot(HaveOccurred())
Expect(func() { svc.Close() }).ToNot(Panic())
})
It("can be called multiple times without panic", func() {
al := &activitylog.ActivityLog{}
svc, err := eventssvc.New(al, nil)
Expect(err).ToNot(HaveOccurred())
svc.Close()
svc.Close()
svc.Close()
})
})
})
@@ -1,97 +1,89 @@
package http
package service
import (
"context"
"embed"
"encoding/json"
"errors"
"net/http"
"path/filepath"
"reflect"
"slices"
"strconv"
"strings"
"time"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
"github.com/olekukonko/errors"
revactx "github.com/opencloud-eu/reva/v2/pkg/ctx"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/storagespace"
"github.com/opencloud-eu/reva/v2/pkg/utils"
"google.golang.org/grpc/metadata"
libregraph "github.com/opencloud-eu/libre-graph-api-go"
"github.com/opencloud-eu/opencloud/pkg/ast"
"github.com/opencloud-eu/opencloud/pkg/kql"
"github.com/opencloud-eu/opencloud/pkg/l10n"
"github.com/opencloud-eu/opencloud/pkg/log"
ehmsg "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/messages/eventhistory/v0"
ehsvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/eventhistory/v0"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/activitylog"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
"github.com/opencloud-eu/reva/v2/pkg/storagespace"
"github.com/opencloud-eu/reva/v2/pkg/utils"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/trace"
)
var tracer trace.Tracer
func init() {
tracer = otel.Tracer("github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/http")
}
// New returns a new instance of Service
func New(al *activitylog.ActivityLog, opts ...Option) (*svc, error) {
o := newOptions(opts...)
registeredEvents := make(map[string]events.Unmarshaller)
for _, e := range o.RegisteredEvents {
typ := reflect.TypeOf(e)
registeredEvents[typ.String()] = e
}
return &svc{
log: o.Logger,
evHistory: o.HistoryClient,
al: al,
registeredEvents: registeredEvents,
gws: o.GatewaySelector,
}, nil
}
type svc struct {
log log.Logger
evHistory ehsvc.EventHistoryService
gws pool.Selectable[gateway.GatewayAPIClient]
al *activitylog.ActivityLog
registeredEvents map[string]events.Unmarshaller
}
var (
ErrBadRequest = errors.New("bad request")
ErrForbidden = errors.New("forbidden")
//go:embed l10n/locale
_localeFS embed.FS
// subfolder where the translation files are stored
_localeSubPath = "l10n/locale"
// domain of the activitylog service (transifex)
_domain = "activitylog"
)
func (s *svc) GetItemActivities(ctx context.Context, query, loc string, t l10n.Translator) ([]libregraph.Activity, error) {
gwc, err := s.gws.Next()
if err != nil {
return nil, err
// ServeHTTP implements the http.Handler interface.
func (s *ActivitylogService) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.mux.ServeHTTP(w, r)
}
// HandleGetItemActivities handles the request to get the activities of an item.
func (s *ActivitylogService) HandleGetItemActivities(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
ctx = metadata.AppendToOutgoingContext(ctx, revactx.TokenHeader, r.Header.Get(revactx.TokenHeader))
activeUser, ok := revactx.ContextGetUser(ctx)
if !ok {
w.WriteHeader(http.StatusUnauthorized)
return
}
rid, limit, rawActivityAccepted, activityAccepted, sort, err := s.getFilters(query)
gwc, err := s.gws.Next()
if err != nil {
s.log.Info().Str("query", query).Err(err).Msg("error getting filters")
return nil, ErrBadRequest
w.WriteHeader(http.StatusInternalServerError)
return
}
rid, limit, rawActivityAccepted, activityAccepted, sort, err := s.getFilters(r.URL.Query().Get("kql"))
if err != nil {
s.log.Info().Str("query", r.URL.Query().Get("kql")).Err(err).Msg("error getting filters")
_, _ = w.Write([]byte(err.Error()))
w.WriteHeader(http.StatusBadRequest)
return
}
info, err := utils.GetResourceByID(ctx, rid, gwc)
if err != nil {
return nil, ErrForbidden
w.WriteHeader(http.StatusForbidden)
return
}
// you need ListGrants to see activities
if !info.GetPermissionSet().GetListGrants() {
return nil, ErrForbidden
w.WriteHeader(http.StatusForbidden)
return
}
raw, err := s.al.Activities(rid)
raw, err := s.Activities(rid)
if err != nil {
s.log.Error().Err(err).Msg("error getting activities")
return nil, err
w.WriteHeader(http.StatusInternalServerError)
return
}
ids := make([]string, 0, len(raw))
@@ -104,21 +96,21 @@ func (s *svc) GetItemActivities(ctx context.Context, query, loc string, t l10n.T
toDelete[a.EventID] = struct{}{}
}
evRes, err := s.evHistory.GetEvents(ctx, &ehsvc.GetEventsRequest{Ids: ids})
evRes, err := s.evHistory.GetEvents(r.Context(), &ehsvc.GetEventsRequest{Ids: ids})
if err != nil {
s.log.Error().Err(err).Msg("error getting events")
return nil, err
w.WriteHeader(http.StatusInternalServerError)
return
}
evs := evRes.GetEvents()
sort(evs)
// TODO cut the interface here?
activities := make([]libregraph.Activity, 0, len(evRes.GetEvents()))
resp := GetActivitiesResponse{Activities: make([]libregraph.Activity, 0, len(evRes.GetEvents()))}
for _, e := range evs {
delete(toDelete, e.GetId())
if limit > 0 && limit <= len(activities) {
if limit > 0 && limit <= len(resp.Activities) {
continue
}
@@ -132,6 +124,9 @@ func (s *svc) GetItemActivities(ctx context.Context, query, loc string, t l10n.T
vars map[string]any
)
loc := l10n.MustGetUserLocale(r.Context(), activeUser.GetId().GetOpaqueId(), r.Header.Get(l10n.HeaderAcceptLanguage), s.valService)
t := l10n.NewTranslatorFromCommonConfig(s.cfg.DefaultLanguage, _domain, s.cfg.TranslationPath, _localeFS, _localeSubPath)
switch ev := s.unwrapEvent(e).(type) {
case nil:
// error already logged in unwrapEvent
@@ -229,29 +224,35 @@ func (s *svc) GetItemActivities(ctx context.Context, query, loc string, t l10n.T
continue
}
activities = append(activities, NewActivity(t.Translate(message, loc), ts, e.GetId(), vars))
resp.Activities = append(resp.Activities, NewActivity(t.Translate(message, loc), ts, e.GetId(), vars))
}
// delete activities in separate go routine
if len(toDelete) > 0 {
go func() {
err := s.al.RemoveActivities(rid, toDelete)
err := s.RemoveActivities(rid, toDelete)
if err != nil {
s.log.Error().Err(err).Msg("error removing activities")
}
}()
}
return activities, nil
}
func toRef(r *provider.ResourceId) *provider.Reference {
return &provider.Reference{
ResourceId: r,
b, err := json.Marshal(resp)
if err != nil {
s.log.Error().Err(err).Msg("error marshalling activities")
w.WriteHeader(http.StatusInternalServerError)
return
}
if _, err := w.Write(b); err != nil {
s.log.Error().Err(err).Msg("error writing response")
w.WriteHeader(http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
}
func (s *svc) unwrapEvent(e *ehmsg.Event) any {
func (s *ActivitylogService) unwrapEvent(e *ehmsg.Event) any {
etype, ok := s.registeredEvents[e.GetType()]
if !ok {
s.log.Error().Str("eventid", e.GetId()).Str("eventtype", e.GetType()).Msg("event not registered")
@@ -267,13 +268,13 @@ func (s *svc) unwrapEvent(e *ehmsg.Event) any {
return einterface
}
func (s *svc) getFilters(query string) (*provider.ResourceId, int, func(activitylog.RawActivity) bool, func(*ehmsg.Event) bool, func([]*ehmsg.Event), error) {
func (s *ActivitylogService) getFilters(query string) (*provider.ResourceId, int, func(RawActivity) bool, func(*ehmsg.Event) bool, func([]*ehmsg.Event), error) {
qast, err := kql.Builder{}.Build(query)
if err != nil {
return nil, 0, nil, nil, nil, err
}
prefilters := make([]func(activitylog.RawActivity) bool, 0)
prefilters := make([]func(RawActivity) bool, 0)
postfilters := make([]func(*ehmsg.Event) bool, 0)
sortby := func(_ []*ehmsg.Event) {}
@@ -298,7 +299,7 @@ func (s *svc) getFilters(query string) (*provider.ResourceId, int, func(activity
break
}
prefilters = append(prefilters, func(a activitylog.RawActivity) bool {
prefilters = append(prefilters, func(a RawActivity) bool {
return a.Depth <= depth
})
case "limit":
@@ -321,11 +322,11 @@ func (s *svc) getFilters(query string) (*provider.ResourceId, int, func(activity
case *ast.DateTimeNode:
switch v.Operator.Value {
case "<", "<=":
prefilters = append(prefilters, func(a activitylog.RawActivity) bool {
prefilters = append(prefilters, func(a RawActivity) bool {
return a.Timestamp.Before(v.Value)
})
case ">", ">=":
prefilters = append(prefilters, func(a activitylog.RawActivity) bool {
prefilters = append(prefilters, func(a RawActivity) bool {
return a.Timestamp.After(v.Value)
})
}
@@ -344,7 +345,7 @@ func (s *svc) getFilters(query string) (*provider.ResourceId, int, func(activity
// space root requested - fix format
rid.OpaqueId = rid.GetSpaceId()
}
pref := func(a activitylog.RawActivity) bool {
pref := func(a RawActivity) bool {
for _, f := range prefilters {
if !f(a) {
return false
@@ -1,59 +0,0 @@
package http
import (
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
"github.com/opencloud-eu/opencloud/pkg/log"
ehsvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/eventhistory/v0"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
)
// Option defines a single option function.
type Option func(o *Options)
// Options defines the available options for this package.
type Options struct {
Logger log.Logger
RegisteredEvents []events.Unmarshaller
GatewaySelector pool.Selectable[gateway.GatewayAPIClient]
HistoryClient ehsvc.EventHistoryService
}
// newOptions initializes the available default options.
func newOptions(opts ...Option) Options {
opt := Options{}
for _, o := range opts {
o(&opt)
}
return opt
}
// Logger provides a function to set the logger option.
func Logger(val log.Logger) Option {
return func(o *Options) {
o.Logger = val
}
}
// RegisteredEvents registers the events the service should listen to
func RegisteredEvents(e []events.Unmarshaller) Option {
return func(o *Options) {
o.RegisteredEvents = e
}
}
// GatewaySelector adds a grpc client selector for the gateway service
func GatewaySelector(gatewaySelector pool.Selectable[gateway.GatewayAPIClient]) Option {
return func(o *Options) {
o.GatewaySelector = gatewaySelector
}
}
// HistoryClient adds a grpc client for the eventhistory service
func HistoryClient(hc ehsvc.EventHistoryService) Option {
return func(o *Options) {
o.HistoryClient = hc
}
}
@@ -1,150 +0,0 @@
package http_test
import (
"context"
"time"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
user "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1"
provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/opencloud-eu/opencloud/pkg/l10n"
httpsvc "github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/http"
)
var _ = Describe("Response", func() {
Describe("NewActivity", func() {
It("creates an activity with the given parameters", func() {
ts := time.Date(2024, 1, 15, 10, 30, 0, 0, time.UTC)
vars := map[string]any{
"user": "testuser",
"resource": "testfile.txt",
}
act := httpsvc.NewActivity("Test message", ts, "event-123", vars)
Expect(act.Id).To(Equal("event-123"))
Expect(act.Times.RecordedTime).To(Equal(ts))
Expect(act.Template.Message).To(Equal("Test message"))
Expect(act.Template.Variables).To(HaveKeyWithValue("user", "testuser"))
Expect(act.Template.Variables).To(HaveKeyWithValue("resource", "testfile.txt"))
})
It("handles empty variables map", func() {
act := httpsvc.NewActivity("", time.Time{}, "", map[string]any{})
Expect(act.Id).To(BeEmpty())
Expect(act.Times.RecordedTime).To(Equal(time.Time{}))
Expect(act.Template.Message).To(BeEmpty())
Expect(act.Template.Variables).To(BeEmpty())
})
})
Describe("WithOldResource", func() {
It("sets the oldResource variable from reference path", func() {
ref := &provider.Reference{
Path: "/old/path/oldname.txt",
}
vars := make(map[string]any)
opt := httpsvc.WithOldResource(ref)
err := opt(context.Background(), nil, vars)
Expect(err).ToNot(HaveOccurred())
Expect(vars).To(HaveKey("oldResource"))
})
})
Describe("WithUser", func() {
It("returns error when no user is provided", func() {
opt := httpsvc.WithUser(nil, nil, nil)
err := opt(context.Background(), nil, make(map[string]any))
Expect(err).To(HaveOccurred())
Expect(err.Error()).To(ContainSubstring("no user provided"))
})
It("uses impersonator when provided", func() {
impersonator := &user.User{
Id: &user.UserId{
OpaqueId: "imp-user-id",
},
DisplayName: "Impersonated User",
}
vars := make(map[string]any)
opt := httpsvc.WithUser(nil, nil, impersonator)
err := opt(context.Background(), nil, vars)
Expect(err).ToNot(HaveOccurred())
Expect(vars).To(HaveKey("user"))
})
It("uses executing user when no impersonator", func() {
execUser := &user.User{
Id: &user.UserId{
OpaqueId: "exec-user-id",
},
DisplayName: "Executing User",
}
vars := make(map[string]any)
opt := httpsvc.WithUser(nil, execUser, nil)
err := opt(context.Background(), nil, vars)
Expect(err).ToNot(HaveOccurred())
Expect(vars).To(HaveKey("user"))
})
})
Describe("WithVar", func() {
It("sets a simple key-value variable", func() {
vars := make(map[string]any)
opt := httpsvc.WithVar("token", "id123", "My Token")
err := opt(context.Background(), nil, vars)
Expect(err).ToNot(HaveOccurred())
Expect(vars).To(HaveKey("token"))
})
})
Describe("WithTranslation", func() {
It("sets translated field variable", func() {
var t l10n.Translator
vars := make(map[string]any)
opt := httpsvc.WithTranslation(&t, "en", "field", []string{"permission"})
err := opt(context.Background(), nil, vars)
Expect(err).ToNot(HaveOccurred())
Expect(vars).To(HaveKey("field"))
})
It("handles empty values slice", func() {
var t l10n.Translator
vars := make(map[string]any)
opt := httpsvc.WithTranslation(&t, "en", "field", []string{})
err := opt(context.Background(), nil, vars)
Expect(err).ToNot(HaveOccurred())
Expect(vars).To(HaveKey("field"))
})
})
Describe("ActivityOption type", func() {
It("allows composing multiple options", func() {
vars := make(map[string]any)
ctx := context.Background()
var gwc gateway.GatewayAPIClient
options := []httpsvc.ActivityOption{
httpsvc.WithVar("key1", "id1", "name1"),
httpsvc.WithVar("key2", "id2", "name2"),
}
for _, opt := range options {
err := opt(ctx, gwc, vars)
Expect(err).ToNot(HaveOccurred())
}
Expect(vars).To(HaveKey("key1"))
Expect(vars).To(HaveKey("key2"))
})
})
})
@@ -1,76 +0,0 @@
package http_test
import (
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/opencloud-eu/opencloud/pkg/log"
ehsvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/eventhistory/v0"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/activitylog"
httpsvc "github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/http"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
)
var _ = Describe("Service", func() {
Describe("New", func() {
var (
al *activitylog.ActivityLog
)
BeforeEach(func() {
al = &activitylog.ActivityLog{}
})
It("creates a service with default options", func() {
svc, err := httpsvc.New(al)
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with logger option", func() {
logger := log.NopLogger()
svc, err := httpsvc.New(al, httpsvc.Logger(logger))
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with registered events", func() {
evts := []events.Unmarshaller{&events.UploadReady{}}
svc, err := httpsvc.New(al, httpsvc.RegisteredEvents(evts))
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with gateway selector", func() {
var gs pool.Selectable[gateway.GatewayAPIClient]
svc, err := httpsvc.New(al, httpsvc.GatewaySelector(gs))
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with history client", func() {
var hc ehsvc.EventHistoryService
svc, err := httpsvc.New(al, httpsvc.HistoryClient(hc))
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
It("creates a service with all options", func() {
logger := log.NopLogger()
evts := []events.Unmarshaller{&events.UploadReady{}, &events.FileTouched{}}
var gs pool.Selectable[gateway.GatewayAPIClient]
var hc ehsvc.EventHistoryService
svc, err := httpsvc.New(
al,
httpsvc.Logger(logger),
httpsvc.RegisteredEvents(evts),
httpsvc.GatewaySelector(gs),
httpsvc.HistoryClient(hc),
)
Expect(err).ToNot(HaveOccurred())
Expect(svc).ToNot(BeNil())
})
})
})
@@ -1,4 +1,4 @@
package activitylog
package service
import (
"context"
@@ -15,7 +15,7 @@ const currentMigrationVersion = "1"
// RunMigrations checks the activitylog data version and runs migrations if necessary.
// It should be called during service startup, after the NATS KeyValue store is initialized.
func (a *ActivityLog) runMigrations(ctx context.Context, kv nats.KeyValue) error {
func (a *ActivitylogService) runMigrations(ctx context.Context, kv nats.KeyValue) error {
entry, err := kv.Get(activitylogVersionKey)
if err == nats.ErrKeyNotFound {
a.log.Info().Msg("activitylog version key not found. Running migration to V1...")
@@ -40,7 +40,7 @@ func (a *ActivityLog) runMigrations(ctx context.Context, kv nats.KeyValue) error
// For each such key, it creates a new key in the format "originalKey.count.timestamp"
// and stores the original list of strings (re-marshalled to messagepack) as its value.
// Finally, it sets the activitylog.version key to "1".
func (a *ActivityLog) migrateToV1(_ context.Context, kv nats.KeyValue) error {
func (a *ActivitylogService) migrateToV1(_ context.Context, kv nats.KeyValue) error {
lister, err := kv.ListKeys()
if err != nil {
return fmt.Errorf("migrateToV1: failed to list keys from NATS KV store: %w", err)
@@ -1,14 +1,17 @@
package events
package service
import (
"context"
"time"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
"github.com/go-chi/chi/v5"
"github.com/opencloud-eu/opencloud/pkg/log"
ehsvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/eventhistory/v0"
settingssvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/settings/v0"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/config"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
"go.opentelemetry.io/otel/trace"
)
// Option for the activitylog service
@@ -16,20 +19,17 @@ type Option func(*Options)
// Options for the activitylog service
type Options struct {
Context context.Context
Logger log.Logger
ServiceAccount config.ServiceAccount
Config *config.Config
TraceProvider trace.TracerProvider
Stream events.Stream
RegisteredEvents []events.Unmarshaller
GatewaySelector pool.Selectable[gateway.GatewayAPIClient]
Mux *chi.Mux
HistoryClient ehsvc.EventHistoryService
ValueClient settingssvc.ValueService
WriteBufferDuration time.Duration
NumConsumers int
}
func Context(ctx context.Context) Option {
return func(o *Options) {
o.Context = ctx
}
MaxActivities int
}
// Logger configures a logger for the activitylog service
@@ -39,10 +39,17 @@ func Logger(log log.Logger) Option {
}
}
// ServiceAccount configures a service account for the activitylog service
func ServiceAccount(sa config.ServiceAccount) Option {
// Config adds the config for the activitylog service
func Config(c *config.Config) Option {
return func(o *Options) {
o.ServiceAccount = sa
o.Config = c
}
}
// TraceProvider adds a tracer provider for the activitylog service
func TraceProvider(tp trace.TracerProvider) Option {
return func(o *Options) {
o.TraceProvider = tp
}
}
@@ -67,8 +74,23 @@ func GatewaySelector(gatewaySelector pool.Selectable[gateway.GatewayAPIClient])
}
}
func NumConsumers(num int) Option {
// Mux defines the muxer for the service
func Mux(m *chi.Mux) Option {
return func(o *Options) {
o.NumConsumers = num
o.Mux = m
}
}
// HistoryClient adds a grpc client for the eventhistory service
func HistoryClient(hc ehsvc.EventHistoryService) Option {
return func(o *Options) {
o.HistoryClient = hc
}
}
// ValueClient adds a grpc client for the value service
func ValueClient(vs settingssvc.ValueService) Option {
return func(o *Options) {
o.ValueClient = vs
}
}
@@ -1,4 +1,4 @@
package http
package service
import (
"context"
@@ -44,6 +44,11 @@ var (
StrDescription = l10n.Template("description")
)
// GetActivitiesResponse is the response on GET activities requests
type GetActivitiesResponse struct {
Activities []libregraph.Activity `json:"value"`
}
// Resource represents an item such as a file or folder
type Resource struct {
ID string `json:"id"`
@@ -306,7 +311,7 @@ func NewActivity(message string, ts time.Time, eventID string, vars map[string]a
}
// GetVars calls other service to gather the required data for the activity variables
func (s *svc) GetVars(ctx context.Context, opts ...ActivityOption) (map[string]any, error) {
func (s *ActivitylogService) GetVars(ctx context.Context, opts ...ActivityOption) (map[string]any, error) {
gwc, err := s.gws.Next()
if err != nil {
return nil, err
@@ -322,12 +327,6 @@ func (s *svc) GetVars(ctx context.Context, opts ...ActivityOption) (map[string]a
return vars, nil
}
func toSpace(r *provider.Reference) *provider.StorageSpaceId {
return &provider.StorageSpaceId{
OpaqueId: storagespace.FormatStorageID(r.GetResourceId().GetStorageId(), r.GetResourceId().GetSpaceId()),
}
}
func getFolderName(ctx context.Context, gwc gateway.GatewayAPIClient, ref *provider.Reference) string {
n := filepath.Base(filepath.Dir(ref.GetPath()))
if n == "." || n == "/" {
+676
View File
@@ -0,0 +1,676 @@
package service
import (
"context"
"crypto/tls"
"encoding/base32"
"encoding/json"
"fmt"
"path/filepath"
"reflect"
"sort"
"strconv"
"strings"
"sync"
"time"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
"github.com/go-chi/chi/v5"
"github.com/jellydator/ttlcache/v2"
"github.com/nats-io/nats.go"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
"github.com/opencloud-eu/reva/v2/pkg/storagespace"
"github.com/opencloud-eu/reva/v2/pkg/utils"
"github.com/pkg/errors"
"github.com/vmihailenco/msgpack/v5"
"go.opentelemetry.io/otel/trace"
"github.com/opencloud-eu/opencloud/pkg/log"
ehsvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/eventhistory/v0"
settingssvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/settings/v0"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/config"
)
// Nats runs into max payload exceeded errors at around 7k activities. Let's keep a buffer.
var _maxActivitiesDefault = 6000
// RawActivity represents an activity as it is stored in the activitylog store
type RawActivity struct {
EventID string `json:"event_id"`
Depth int `json:"depth"`
Timestamp time.Time `json:"timestamp"`
}
// ActivitylogService logs events per resource
type ActivitylogService struct {
cfg *config.Config
log log.Logger
events <-chan events.Event
gws pool.Selectable[gateway.GatewayAPIClient]
mux *chi.Mux
evHistory ehsvc.EventHistoryService
valService settingssvc.ValueService
lock sync.RWMutex
tp trace.TracerProvider
tracer trace.Tracer
debouncer *Debouncer
parentIdCache *ttlcache.Cache
natskv nats.KeyValue
maxActivities int
registeredEvents map[string]events.Unmarshaller
}
type Debouncer struct {
after time.Duration
f func(id string, ra []RawActivity) error
pending sync.Map
inProgress sync.Map
mutex sync.Mutex
}
type queueItem struct {
activities []RawActivity
timer *time.Timer
}
type batchInfo struct {
key string
count int
timestamp time.Time
}
// NewDebouncer returns a new Debouncer instance
func NewDebouncer(d time.Duration, f func(id string, ra []RawActivity) error) *Debouncer {
return &Debouncer{
after: d,
f: f,
pending: sync.Map{},
inProgress: sync.Map{},
}
}
// Debounce restarts the debounce timer for the given space
func (d *Debouncer) Debounce(id string, ra RawActivity) {
if d.after == 0 {
d.f(id, []RawActivity{ra})
return
}
d.mutex.Lock()
defer d.mutex.Unlock()
activities := []RawActivity{ra}
item := &queueItem{
activities: activities,
}
if i, ok := d.pending.Load(id); ok {
// if the item is already in the queue, append the new activities
item, ok = i.(*queueItem)
if ok {
item.activities = append(item.activities, ra)
}
}
if item.timer == nil {
item.timer = time.AfterFunc(d.after, func() {
if _, ok := d.inProgress.Load(id); ok {
// Reschedule this run for when the previous run has finished
d.mutex.Lock()
if i, ok := d.pending.Load(id); ok {
i.(*queueItem).timer.Reset(d.after)
}
d.mutex.Unlock()
return
}
d.pending.Delete(id)
d.inProgress.Store(id, true)
defer d.inProgress.Delete(id)
d.f(id, item.activities)
})
}
d.pending.Store(id, item)
}
// New creates a new ActivitylogService
func New(opts ...Option) (*ActivitylogService, error) {
o := &Options{
MaxActivities: _maxActivitiesDefault,
}
for _, opt := range opts {
opt(o)
}
if o.Stream == nil {
return nil, errors.New("stream is required")
}
ch, err := events.Consume(o.Stream, o.Config.Service.Name, o.RegisteredEvents...)
if err != nil {
return nil, err
}
cache := ttlcache.NewCache()
err = cache.SetTTL(30 * time.Second)
if err != nil {
return nil, err
}
// Connect to NATS servers
natsOptions := nats.Options{
Servers: o.Config.Store.Nodes,
}
if o.Config.Store.EnableTLS {
if o.Config.Store.TLSRootCACertificate != "" {
// when root ca is configured use it. an insecure flag is ignored.
nats.RootCAs(o.Config.Store.TLSRootCACertificate)(&natsOptions)
} else {
// enable tls and use insecure flag
nats.Secure(&tls.Config{MinVersion: tls.VersionTLS12, InsecureSkipVerify: o.Config.Store.TLSInsecure})(&natsOptions)
}
}
if o.Config.Store.AuthUsername != "" && o.Config.Store.AuthPassword != "" {
nats.UserInfo(o.Config.Store.AuthUsername, o.Config.Store.AuthPassword)(&natsOptions)
}
conn, err := natsOptions.Connect()
if err != nil {
return nil, err
}
js, err := conn.JetStream()
if err != nil {
return nil, err
}
kv, err := js.KeyValue(o.Config.Store.Database)
if err != nil {
if !errors.Is(err, nats.ErrBucketNotFound) {
return nil, errors.Wrapf(err, "Failed to get bucket (%s)", o.Config.Store.Database)
}
kv, err = js.CreateKeyValue(&nats.KeyValueConfig{
Bucket: o.Config.Store.Database,
})
if err != nil {
return nil, errors.Wrapf(err, "Failed to create bucket (%s)", o.Config.Store.Database)
}
}
if err != nil {
return nil, err
}
s := &ActivitylogService{
log: o.Logger,
cfg: o.Config,
events: ch,
gws: o.GatewaySelector,
mux: o.Mux,
evHistory: o.HistoryClient,
valService: o.ValueClient,
lock: sync.RWMutex{},
registeredEvents: make(map[string]events.Unmarshaller),
tp: o.TraceProvider,
tracer: o.TraceProvider.Tracer("github.com/opencloud-eu/opencloud/services/activitylog/pkg/service"),
parentIdCache: cache,
maxActivities: o.Config.MaxActivities,
natskv: kv,
}
s.debouncer = NewDebouncer(o.Config.WriteBufferDuration, s.storeActivity)
// run migrations
err = s.runMigrations(context.Background(), kv)
if err != nil {
return nil, err
}
s.mux.Get("/graph/v1beta1/extensions/org.libregraph/activities", s.HandleGetItemActivities)
for _, e := range o.RegisteredEvents {
typ := reflect.TypeOf(e)
s.registeredEvents[typ.String()] = e
}
go s.Run()
return s, nil
}
// Run runs the service
func (a *ActivitylogService) Run() {
for e := range a.events {
var err error
switch ev := e.Event.(type) {
case events.UploadReady:
err = a.AddActivity(ev.FileRef, ev.ParentID, e.ID, utils.TSToTime(ev.Timestamp))
case events.FileTouched:
err = a.AddActivity(ev.Ref, ev.ParentID, e.ID, utils.TSToTime(ev.Timestamp))
// Disabled https://github.com/owncloud/ocis/issues/10293
//case events.FileDownloaded:
// we are only interested in public link downloads - so no need to store others.
//if ev.ImpersonatingUser.GetDisplayName() == "Public" {
// err = a.AddActivity(ev.Ref, e.ID, utils.TSToTime(ev.Timestamp))
//}
case events.ContainerCreated:
err = a.AddActivity(ev.Ref, ev.ParentID, e.ID, utils.TSToTime(ev.Timestamp))
case events.ItemTrashed:
err = a.AddActivityTrashed(ev.ID, ev.Ref, nil, e.ID, utils.TSToTime(ev.Timestamp))
case events.ItemPurged:
err = a.RemoveResource(ev.ID)
case events.ItemMoved:
// remove the cached parent id for this resource
a.removeCachedParentID(ev.Ref)
err = a.AddActivity(ev.Ref, nil, e.ID, utils.TSToTime(ev.Timestamp))
case events.ShareCreated:
err = a.AddActivity(toRef(ev.ItemID), nil, e.ID, utils.TSToTime(ev.CTime))
case events.ShareUpdated:
if ev.Sharer != nil && ev.ItemID != nil && ev.Sharer.GetOpaqueId() != ev.ItemID.GetSpaceId() {
err = a.AddActivity(toRef(ev.ItemID), nil, e.ID, utils.TSToTime(ev.MTime))
}
case events.ShareRemoved:
err = a.AddActivity(toRef(ev.ItemID), nil, e.ID, ev.Timestamp)
case events.LinkCreated:
err = a.AddActivity(toRef(ev.ItemID), nil, e.ID, utils.TSToTime(ev.CTime))
case events.LinkUpdated:
if ev.Sharer != nil && ev.ItemID != nil && ev.Sharer.GetOpaqueId() != ev.ItemID.GetSpaceId() {
err = a.AddActivity(toRef(ev.ItemID), nil, e.ID, utils.TSToTime(ev.MTime))
}
case events.LinkRemoved:
err = a.AddActivity(toRef(ev.ItemID), nil, e.ID, utils.TSToTime(ev.Timestamp))
case events.SpaceShared:
err = a.AddSpaceActivity(ev.ID, e.ID, ev.Timestamp)
case events.SpaceUnshared:
err = a.AddSpaceActivity(ev.ID, e.ID, ev.Timestamp)
}
if err != nil {
a.log.Error().Err(err).Interface("event", e).Msg("could not process event")
}
}
}
// AddActivity adds the activity to the given resource and all its parents
func (a *ActivitylogService) AddActivity(initRef *provider.Reference, parentId *provider.ResourceId, eventID string, timestamp time.Time) error {
gwc, err := a.gws.Next()
if err != nil {
return fmt.Errorf("cant get gateway client: %w", err)
}
ctx, err := utils.GetServiceUserContext(a.cfg.ServiceAccount.ServiceAccountID, gwc, a.cfg.ServiceAccount.ServiceAccountSecret)
if err != nil {
return fmt.Errorf("cant get service user context: %w", err)
}
var span trace.Span
ctx, span = a.tracer.Start(ctx, "AddActivity")
defer span.End()
return a.addActivity(ctx, initRef, parentId, eventID, timestamp, func(ctx context.Context, ref *provider.Reference) (*provider.ResourceInfo, error) {
return utils.GetResource(ctx, ref, gwc)
})
}
// AddActivityTrashed adds the activity to given trashed resource and all its former parents
func (a *ActivitylogService) AddActivityTrashed(resourceID *provider.ResourceId, reference *provider.Reference, parentId *provider.ResourceId, eventID string, timestamp time.Time) error {
gwc, err := a.gws.Next()
if err != nil {
return fmt.Errorf("cant get gateway client: %w", err)
}
ctx, err := utils.GetServiceUserContext(a.cfg.ServiceAccount.ServiceAccountID, gwc, a.cfg.ServiceAccount.ServiceAccountSecret)
if err != nil {
return fmt.Errorf("cant get service user context: %w", err)
}
// store activity on trashed item
if err := a.storeActivity(storagespace.FormatResourceID(resourceID), []RawActivity{
{
EventID: eventID,
Depth: 0,
Timestamp: timestamp,
},
}); err != nil {
return fmt.Errorf("could not store activity: %w", err)
}
// get previous parent
ref := &provider.Reference{
ResourceId: reference.GetResourceId(),
Path: filepath.Dir(reference.GetPath()),
}
var span trace.Span
ctx, span = a.tracer.Start(ctx, "AddActivityTrashed")
defer span.End()
return a.addActivity(ctx, ref, parentId, eventID, timestamp, func(ctx context.Context, ref *provider.Reference) (*provider.ResourceInfo, error) {
return utils.GetResource(ctx, ref, gwc)
})
}
// AddSpaceActivity adds the activity to the given spaceroot
func (a *ActivitylogService) AddSpaceActivity(spaceID *provider.StorageSpaceId, eventID string, timestamp time.Time) error {
// spaceID is in format <providerid>$<spaceid>
// activitylog service uses format <providerid>$<spaceid>!<resourceid>
// lets do some converting, shall we?
rid, err := storagespace.ParseID(spaceID.GetOpaqueId())
if err != nil {
return fmt.Errorf("could not parse space id: %w", err)
}
rid.OpaqueId = rid.GetSpaceId()
return a.storeActivity(storagespace.FormatResourceID(&rid), []RawActivity{
{
EventID: eventID,
Depth: 0,
Timestamp: timestamp,
},
})
}
// Activities returns the activities for the given resource
func (a *ActivitylogService) Activities(rid *provider.ResourceId) ([]RawActivity, error) {
a.lock.RLock()
defer a.lock.RUnlock()
return a.activities(rid)
}
// RemoveActivities removes the activities from the given resource
func (a *ActivitylogService) RemoveActivities(rid *provider.ResourceId, toDelete map[string]struct{}) error {
a.lock.Lock()
defer a.lock.Unlock()
curActivities, err := a.activities(rid)
if err != nil {
return err
}
var acts []RawActivity
for _, a := range curActivities {
if _, ok := toDelete[a.EventID]; !ok {
acts = append(acts, a)
}
}
b, err := json.Marshal(acts)
if err != nil {
return err
}
_, err = a.natskv.Put(storagespace.FormatResourceID(rid), b)
return err
}
// RemoveResource removes the resource from the store
func (a *ActivitylogService) RemoveResource(rid *provider.ResourceId) error {
if rid == nil {
return fmt.Errorf("resource id is required")
}
a.lock.Lock()
defer a.lock.Unlock()
return a.natskv.Delete(storagespace.FormatResourceID(rid))
}
func (a *ActivitylogService) activities(rid *provider.ResourceId) ([]RawActivity, error) {
resourceID := storagespace.FormatResourceID(rid)
glob := fmt.Sprintf("%s.>", base32.StdEncoding.EncodeToString([]byte(resourceID)))
watcher, err := a.natskv.Watch(glob, nats.IgnoreDeletes())
if err != nil {
return nil, err
}
defer watcher.Stop()
var activities []RawActivity
for update := range watcher.Updates() {
if update == nil {
break
}
var batchActivities []RawActivity
if err := msgpack.Unmarshal(update.Value(), &batchActivities); err != nil {
a.log.Debug().Err(err).Str("resourceID", resourceID).Msg("could not unmarshal messagepack, trying json")
}
activities = append(activities, batchActivities...)
}
return activities, nil
}
// note: getResource is abstracted to allow unit testing, in general this will just be utils.GetResource
func (a *ActivitylogService) addActivity(ctx context.Context, initRef *provider.Reference, parentId *provider.ResourceId, eventID string, timestamp time.Time, getResource func(context.Context, *provider.Reference) (*provider.ResourceInfo, error)) error {
var (
err error
depth int
ref = initRef
)
ctx, span := a.tracer.Start(ctx, "addActivity")
defer span.End()
for {
var info *provider.ResourceInfo
id := ref.GetResourceId()
if ref.Path != "" {
// Path based reference, we need to resolve the resource id
ctx, span = a.tracer.Start(ctx, "addActivity.getResource")
info, err = getResource(ctx, ref)
span.End()
if err != nil {
return fmt.Errorf("could not get resource info: %w", err)
}
id = info.GetId()
}
if id == nil {
return fmt.Errorf("resource id is required")
}
key := storagespace.FormatResourceID(id)
a.debouncer.Debounce(key, RawActivity{
EventID: eventID,
Depth: depth,
Timestamp: timestamp,
})
if id.OpaqueId == id.SpaceId {
// we are at the root of the space, no need to go further
break
}
// check if parent id is cached
// parent id is cached in the format <storageid>$<spaceid>!<resourceid>
// if it is not cached, get the resource info and cache it
if parentId == nil {
if v, err := a.parentIdCache.Get(key); err != nil {
if info == nil {
ctx, span := a.tracer.Start(ctx, "addActivity.getResource parent")
info, err = getResource(ctx, ref)
span.End()
if err != nil || info.GetParentId() == nil || info.GetParentId().GetOpaqueId() == "" {
return fmt.Errorf("could not get parent id: %w", err)
}
}
parentId = info.GetParentId()
a.parentIdCache.Set(key, parentId)
} else {
parentId = v.(*provider.ResourceId)
}
} else {
a.log.Debug().Msg("parent id is cached")
}
depth++
ref = &provider.Reference{ResourceId: parentId}
parentId = nil // reset parent id so it's not reused in the next iteration
}
return nil
}
func (a *ActivitylogService) storeActivity(resourceID string, activities []RawActivity) error {
a.lock.Lock()
defer a.lock.Unlock()
ctx, span := a.tracer.Start(context.Background(), "storeActivity")
defer span.End()
_, subspan := a.tracer.Start(ctx, "storeActivity.Marshal")
b, err := msgpack.Marshal(activities)
if err != nil {
return err
}
subspan.End()
_, subspan = a.tracer.Start(ctx, "storeActivity.natskv.Put")
key := natsKey(resourceID, len(activities))
_, err = a.natskv.Put(key, b)
if err != nil {
return err
}
subspan.End()
ctx, subspan = a.tracer.Start(ctx, "storeActivity.enforceMaxActivities")
a.enforceMaxActivities(ctx, resourceID)
subspan.End()
return nil
}
func (a *ActivitylogService) enforceMaxActivities(ctx context.Context, resourceID string) {
if a.maxActivities <= 0 {
return
}
key := fmt.Sprintf("%s.>", base32.StdEncoding.EncodeToString([]byte(resourceID)))
_, subspan := a.tracer.Start(ctx, "enforceMaxActivities.watch")
watcher, err := a.natskv.Watch(key, nats.IgnoreDeletes())
if err != nil {
a.log.Error().Err(err).Str("resourceID", resourceID).Msg("could not watch")
return
}
defer watcher.Stop()
var keys []string
for update := range watcher.Updates() {
if update == nil {
break
}
var batchActivities []RawActivity
if err := msgpack.Unmarshal(update.Value(), &batchActivities); err != nil {
a.log.Debug().Err(err).Str("resourceID", resourceID).Msg("could not unmarshal messagepack, trying json")
}
keys = append(keys, update.Key())
}
subspan.End()
_, subspan = a.tracer.Start(ctx, "enforceMaxActivities.compile")
// Parse keys into batches
batches := make([]batchInfo, 0)
var activitiesCount int
for _, k := range keys {
parts := strings.SplitN(k, ".", 3)
if len(parts) < 3 {
a.log.Warn().Str("key", k).Msg("skipping key, not enough parts")
continue
}
c, err := strconv.Atoi(parts[1])
if err != nil {
a.log.Warn().Str("key", k).Msg("skipping key, can not parse count")
continue
}
// parse timestamp
nano, err := strconv.ParseInt(parts[2], 10, 64)
if err != nil {
a.log.Warn().Str("key", k).Msg("skipping key, can not parse timestamp")
continue
}
batches = append(batches, batchInfo{
key: k,
count: c,
timestamp: time.Unix(0, nano),
})
activitiesCount += c
}
// sort batches by timestamp
sort.Slice(batches, func(i, j int) bool {
return batches[i].timestamp.Before(batches[j].timestamp)
})
subspan.End()
_, subspan = a.tracer.Start(ctx, "enforceMaxActivities.delete")
// remove oldest keys until we are at max activities
for _, b := range batches {
if activitiesCount-b.count < a.maxActivities {
break
}
activitiesCount -= b.count
err = a.natskv.Delete(b.key)
if err != nil {
a.log.Error().Err(err).Str("key", b.key).Msg("could not delete key")
break
}
}
subspan.End()
}
func toRef(r *provider.ResourceId) *provider.Reference {
return &provider.Reference{
ResourceId: r,
}
}
func toSpace(r *provider.Reference) *provider.StorageSpaceId {
return &provider.StorageSpaceId{
OpaqueId: storagespace.FormatStorageID(r.GetResourceId().GetStorageId(), r.GetResourceId().GetSpaceId()),
}
}
func (a *ActivitylogService) removeCachedParentID(ref *provider.Reference) {
purgeId := ref.GetResourceId()
if ref.GetPath() != "" {
gwc, err := a.gws.Next()
if err != nil {
a.log.Error().Err(err).Msg("could not get gateway client")
return
}
ctx, err := utils.GetServiceUserContext(a.cfg.ServiceAccount.ServiceAccountID, gwc, a.cfg.ServiceAccount.ServiceAccountSecret)
if err != nil {
a.log.Error().Err(err).Msg("could not get service user context")
return
}
info, err := utils.GetResource(ctx, ref, gwc)
if err != nil {
a.log.Error().Err(err).Msg("could not get resource info")
return
}
purgeId = info.GetId()
}
// The parent id cache is populated lazily and its entries expire, so a
// missing key is the expected case rather than an error.
if err := a.parentIdCache.Remove(storagespace.FormatResourceID(purgeId)); err != nil {
a.log.Debug().Interface("event", ref).Err(err).Msg("could not delete parent id cache")
}
}
func natsKey(resourceID string, activitiesCount int) string {
return fmt.Sprintf("%s.%d.%d",
base32.StdEncoding.EncodeToString([]byte(resourceID)),
activitiesCount,
time.Now().UnixNano())
}
@@ -1,4 +1,4 @@
package events_test
package service_test
import (
"testing"
@@ -7,7 +7,7 @@ import (
. "github.com/onsi/gomega"
)
func TestEvents(t *testing.T) {
func TestService(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Events Suite")
RunSpecs(t, "Service Suite")
}
@@ -1,6 +1,7 @@
package activitylog_test
package service
import (
"bytes"
"context"
"net"
"os"
@@ -8,13 +9,17 @@ import (
"time"
provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
"github.com/go-chi/chi/v5"
"github.com/google/uuid"
nserver "github.com/nats-io/nats-server/v2/server"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/command"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/config"
"github.com/opencloud-eu/opencloud/services/activitylog/pkg/service/activitylog"
eventsmocks "github.com/opencloud-eu/reva/v2/pkg/events/mocks"
"github.com/opencloud-eu/reva/v2/pkg/storagespace"
"github.com/test-go/testify/mock"
"go.opentelemetry.io/otel/trace/noop"
)
var (
@@ -63,25 +68,31 @@ var _ = SynchronizedAfterSuite(func() {
var _ = Describe("ActivitylogService", func() {
var (
alog *activitylog.ActivityLog
alog *ActivitylogService
getResource func(_ context.Context, ref *provider.Reference) (*provider.ResourceInfo, error)
writebufferduration = 100 * time.Millisecond
)
JustBeforeEach(func() {
var err error
db := "activitylog-test-" + uuid.New().String()
kv, err := command.ConnectNatsKV(config.Store{
Nodes: []string{server.Addr().String()},
Database: db,
})
Expect(err).ToNot(HaveOccurred())
alog, err = activitylog.New(
kv,
activitylog.MaxActivities(4),
activitylog.WriteBufferDuration(writebufferduration),
stream := &eventsmocks.Stream{}
stream.EXPECT().Consume(mock.Anything, mock.Anything).Return(nil, nil)
alog, err = New(
Config(&config.Config{
Service: config.Service{
Name: "activitylog-test",
},
Store: config.Store{
Store: "nats-js-kv",
Nodes: []string{server.Addr().String()},
Database: "activitylog-test-" + uuid.New().String(),
},
MaxActivities: 4,
WriteBufferDuration: writebufferduration,
}),
Stream(stream),
TraceProvider(noop.NewTracerProvider()),
Mux(chi.NewMux()),
)
Expect(err).ToNot(HaveOccurred())
})
@@ -96,7 +107,7 @@ var _ = Describe("ActivitylogService", func() {
Name string
Tree map[string]*provider.ResourceInfo
Activities map[string]string
Expected map[string][]activitylog.RawActivity
Expected map[string][]RawActivity
}
testCases := []testCase{
@@ -110,7 +121,7 @@ var _ = Describe("ActivitylogService", func() {
Activities: map[string]string{
"activity": "base",
},
Expected: map[string][]activitylog.RawActivity{
Expected: map[string][]RawActivity{
"base": activitites("activity", 0),
"parent": activitites("activity", 1),
"spaceid": activitites("activity", 2),
@@ -127,7 +138,7 @@ var _ = Describe("ActivitylogService", func() {
"activity1": "base",
"activity2": "base",
},
Expected: map[string][]activitylog.RawActivity{
Expected: map[string][]RawActivity{
"base": activitites("activity1", 0, "activity2", 0),
"parent": activitites("activity1", 1, "activity2", 1),
"spaceid": activitites("activity1", 2, "activity2", 2),
@@ -144,7 +155,7 @@ var _ = Describe("ActivitylogService", func() {
}
for k, v := range tc.Activities {
err := alog.AddActivity(context.Background(), reference(v), nil, k, time.Time{}, getResource)
err := alog.addActivity(context.Background(), reference(v), nil, k, time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
}
})
@@ -183,9 +194,9 @@ var _ = Describe("ActivitylogService", func() {
It("debounces activities", func() {
err := alog.AddActivity(context.Background(), reference("base"), nil, "activity1", time.Time{}, getResource)
err := alog.addActivity(context.Background(), reference("base"), nil, "activity1", time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
err = alog.AddActivity(context.Background(), reference("base"), nil, "activity2", time.Time{}, getResource)
err = alog.addActivity(context.Background(), reference("base"), nil, "activity2", time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
Eventually(func(g Gomega) {
@@ -196,7 +207,7 @@ var _ = Describe("ActivitylogService", func() {
})
It("adheres to the MaxActivities setting", func() {
err := alog.AddActivity(context.Background(), reference("base"), nil, "activity1", time.Time{}, getResource)
err := alog.addActivity(context.Background(), reference("base"), nil, "activity1", time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
Eventually(func(g Gomega) {
activities, err := alog.Activities(resourceID("base"))
@@ -204,7 +215,7 @@ var _ = Describe("ActivitylogService", func() {
g.Expect(len(activities)).To(Equal(1))
}).Should(Succeed())
err = alog.AddActivity(context.Background(), reference("base"), nil, "activity2", time.Time{}, getResource)
err = alog.addActivity(context.Background(), reference("base"), nil, "activity2", time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
Eventually(func(g Gomega) {
activities, err := alog.Activities(resourceID("base"))
@@ -212,11 +223,11 @@ var _ = Describe("ActivitylogService", func() {
g.Expect(len(activities)).To(Equal(2))
}).Should(Succeed())
err = alog.AddActivity(context.Background(), reference("base"), nil, "activity3", time.Time{}, getResource)
err = alog.addActivity(context.Background(), reference("base"), nil, "activity3", time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
err = alog.AddActivity(context.Background(), reference("base"), nil, "activity4", time.Time{}, getResource)
err = alog.addActivity(context.Background(), reference("base"), nil, "activity4", time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
err = alog.AddActivity(context.Background(), reference("base"), nil, "activity5", time.Time{}, getResource)
err = alog.addActivity(context.Background(), reference("base"), nil, "activity5", time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
Eventually(func(g Gomega) {
@@ -233,9 +244,9 @@ var _ = Describe("ActivitylogService", func() {
return tree[ref.GetResourceId().GetOpaqueId()], nil
}
err := alog.AddActivity(context.Background(), reference("base"), nil, "activity1", time.Time{}, getResource)
err := alog.addActivity(context.Background(), reference("base"), nil, "activity1", time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
err = alog.AddActivity(context.Background(), reference("base"), nil, "activity2", time.Time{}, getResource)
err = alog.addActivity(context.Background(), reference("base"), nil, "activity2", time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
Eventually(func(g Gomega) {
@@ -244,9 +255,9 @@ var _ = Describe("ActivitylogService", func() {
g.Expect(activities).To(ConsistOf(activitites("activity1", 0, "activity2", 0)))
}).Should(Succeed())
err = alog.AddActivity(context.Background(), reference("base"), nil, "activity3", time.Time{}, getResource)
err = alog.addActivity(context.Background(), reference("base"), nil, "activity3", time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
err = alog.AddActivity(context.Background(), reference("base"), nil, "activity4", time.Time{}, getResource)
err = alog.addActivity(context.Background(), reference("base"), nil, "activity4", time.Time{}, getResource)
Expect(err).NotTo(HaveOccurred())
Eventually(func(g Gomega) {
@@ -257,11 +268,47 @@ var _ = Describe("ActivitylogService", func() {
})
})
})
Describe("removeCachedParentID", func() {
var logBuffer *bytes.Buffer
newLoggerAtLevel := func(level string) log.Logger {
logBuffer = &bytes.Buffer{}
return log.Logger{Logger: log.NewLogger(log.Level(level)).Output(logBuffer)}
}
It("does not log an error when the entry was never cached", func() {
alog.log = newLoggerAtLevel("error")
alog.removeCachedParentID(reference("never-cached"))
Expect(logBuffer.String()).To(BeEmpty())
})
It("logs a missing entry at debug level", func() {
alog.log = newLoggerAtLevel("debug")
alog.removeCachedParentID(reference("never-cached"))
Expect(logBuffer.String()).To(ContainSubstring("could not delete parent id cache"))
Expect(logBuffer.String()).To(ContainSubstring(`"level":"debug"`))
})
It("does not log at all when the entry was cached", func() {
alog.log = newLoggerAtLevel("debug")
ref := reference("cached")
Expect(alog.parentIdCache.Set(storagespace.FormatResourceID(ref.GetResourceId()), resourceID("parent"))).To(Succeed())
alog.removeCachedParentID(ref)
Expect(logBuffer.String()).To(BeEmpty())
})
})
})
func activitites(acts ...any) []activitylog.RawActivity {
var activities []activitylog.RawActivity
act := activitylog.RawActivity{}
func activitites(acts ...any) []RawActivity {
var activities []RawActivity
act := RawActivity{}
for _, a := range acts {
switch v := a.(type) {
case string:
+3 -3
View File
@@ -61,7 +61,7 @@ type Spaces struct {
}
type LDAPMetrics struct {
Disabled bool `yaml:"disabled" env:"GRAPH_LDAP_METRICS_DISABLE" desc:"Disables the metrics for outbound LDAP operations." introductionVersion:"%%NEXT%%"`
Disabled bool `yaml:"disabled" env:"GRAPH_LDAP_METRICS_DISABLE" desc:"Disables the metrics for outbound LDAP operations." introductionVersion:"%NEXT%"`
}
type LDAP struct {
@@ -121,7 +121,7 @@ type LDAPEducationConfig struct {
}
type IdentityMetrics struct {
Disabled bool `yaml:"disabled" env:"GRAPH_IDENTITY_BACKEND_METRICS_DISABLE" desc:"Disables the metrics for inbound identity backend operations." introductionVersion:"%%NEXT%%"`
Disabled bool `yaml:"disabled" env:"GRAPH_IDENTITY_BACKEND_METRICS_DISABLE" desc:"Disables the metrics for inbound identity backend operations." introductionVersion:"%NEXT%"`
}
type Identity struct {
@@ -141,7 +141,7 @@ type API struct {
// Events combines the configuration options for the event bus.
type Events struct {
DisabledConsumer bool `yaml:"disabled_consumer" env:"GRAPH_EVENTS_DISABLE_CONSUMER" desc:"Disables consuming events. Set this to true if the service should only handle HTTP requests." introductionVersion:"%%NEXT%%"`
DisabledConsumer bool `yaml:"disabled_consumer" env:"GRAPH_EVENTS_DISABLE_CONSUMER" desc:"Disables consuming events. Set this to true if the service should only handle HTTP requests." introductionVersion:"%NEXT%"`
Endpoint string `yaml:"endpoint" env:"OC_EVENTS_ENDPOINT;GRAPH_EVENTS_ENDPOINT" desc:"The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Set to a empty string to disable emitting events." introductionVersion:"1.0.0"`
Cluster string `yaml:"cluster" env:"OC_EVENTS_CLUSTER;GRAPH_EVENTS_CLUSTER" desc:"The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture." introductionVersion:"1.0.0"`
TLSInsecure bool `yaml:"tls_insecure" env:"OC_INSECURE;OC_EVENTS_TLS_INSECURE;GRAPH_EVENTS_TLS_INSECURE" desc:"Whether to verify the server TLS certificates." introductionVersion:"1.0.0"`
+2 -2
View File
@@ -3,12 +3,12 @@ package config
import "github.com/opencloud-eu/opencloud/pkg/shared"
type HTTPMetrics struct {
Disabled bool `yaml:"disabled" env:"GRAPH_HTTP_METRICS_DISABLE" desc:"Disables the metrics for the HTTP service." introductionVersion:"%%NEXT%%"`
Disabled bool `yaml:"disabled" env:"GRAPH_HTTP_METRICS_DISABLE" desc:"Disables the metrics for the HTTP service." introductionVersion:"%NEXT%"`
}
// HTTP defines the available http configuration.
type HTTP struct {
Disabled bool `yaml:"disabled" env:"GRAPH_HTTP_DISABLE" desc:"Disables the HTTP service. Set this to true if the service should only consume events." introductionVersion:"%%NEXT%%"`
Disabled bool `yaml:"disabled" env:"GRAPH_HTTP_DISABLE" desc:"Disables the HTTP service. Set this to true if the service should only consume events." introductionVersion:"%NEXT%"`
Addr string `yaml:"addr" env:"GRAPH_HTTP_ADDR" desc:"The bind address of the HTTP service." introductionVersion:"1.0.0"`
Namespace string `yaml:"-"`
Root string `yaml:"root" env:"GRAPH_HTTP_ROOT" desc:"Subdirectory that serves as the root for this HTTP service." introductionVersion:"1.0.0"`
+1 -1
View File
@@ -8,7 +8,7 @@ import (
libregraph "github.com/opencloud-eu/libre-graph-api-go"
"github.com/opencloud-eu/reva/v2/pkg/storagespace"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/thumbnail"
"github.com/opencloud-eu/opencloud/services/webdav/pkg/thumbnail"
)
const (
+2 -2
View File
@@ -29,7 +29,7 @@ type Service struct {
// GRPC defines the available grpc configuration.
type GRPC struct {
Disabled bool `yaml:"disabled" env:"POLICIES_GRPC_DISABLED" desc:"Disables listening for GRPC API calls. Set this to true if the service should only handle requests through events." introductionVersion:"%%NEXT%%"`
Disabled bool `yaml:"disabled" env:"POLICIES_GRPC_DISABLED" desc:"Disables listening for GRPC API calls. Set this to true if the service should only handle requests through events." introductionVersion:"%NEXT%"`
Addr string `yaml:"addr" env:"POLICIES_GRPC_ADDR" desc:"The bind address of the GRPC service." introductionVersion:"1.0.0"`
Namespace string `yaml:"-"`
TLS *shared.GRPCServiceTLS `yaml:"tls"`
@@ -50,7 +50,7 @@ type Postprocessing struct {
// Events combines the configuration options for the event bus.
type Events struct {
Disabled bool `yaml:"disabled" env:"POLICIES_EVENTS_DISABLED" desc:"Disables listening for events. Set this to true if the service should only handle GRPC requests." introductionVersion:"%%NEXT%%"`
Disabled bool `yaml:"disabled" env:"POLICIES_EVENTS_DISABLED" desc:"Disables listening for events. Set this to true if the service should only handle GRPC requests." introductionVersion:"%NEXT%"`
Endpoint string `yaml:"endpoint" env:"OC_EVENTS_ENDPOINT;POLICIES_EVENTS_ENDPOINT" desc:"The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture." introductionVersion:"1.0.0"`
Cluster string `yaml:"cluster" env:"OC_EVENTS_CLUSTER;POLICIES_EVENTS_CLUSTER" desc:"The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system." introductionVersion:"1.0.0"`
TLSInsecure bool `yaml:"tls_insecure" env:"OC_INSECURE;OC_EVENTS_TLS_INSECURE;POLICIES_EVENTS_TLS_INSECURE" desc:"Whether the server should skip the client certificate verification during the TLS handshake." introductionVersion:"1.0.0"`
+1 -19
View File
@@ -8,7 +8,6 @@ import (
"strings"
"time"
"github.com/golang-jwt/jwt/v5"
"github.com/pkg/errors"
"github.com/vmihailenco/msgpack/v5"
"go-micro.dev/v4/store"
@@ -187,13 +186,6 @@ func verifyExpiresAt(claims map[string]any, cmp time.Time) bool {
return cmp.Before(expiry)
}
// isExpectedTokenTimeError reports whether the error is caused by a token that
// failed its expiry (or not-before) validation. These are expected as part of
// the normal token renewal cycle and should not be logged at error level.
func isExpectedTokenTimeError(err error) bool {
return errors.Is(err, jwt.ErrTokenExpired) || errors.Is(err, jwt.ErrTokenNotValidYet)
}
func (m OIDCAuthenticator) shouldServe(req *http.Request) bool {
if m.OIDCIss == "" {
return false
@@ -220,17 +212,7 @@ func (m *OIDCAuthenticator) Authenticate(r *http.Request) (*http.Request, bool)
claims, newSession, err := m.getClaims(token, r)
if err != nil {
host, port, _ := net.SplitHostPort(r.RemoteAddr)
// Expired (or not yet valid) tokens are a normal part of the token
// renewal cycle: the client hits the proxy with a token that is just
// past its expiry, receives a 401 challenge and refreshes. This is not
// a failure of the proxy, so we log it at debug level to avoid noise.
logEvent := m.Logger.Debug()
if !isExpectedTokenTimeError(err) {
logEvent = m.Logger.Error()
}
logEvent.
m.Logger.Error().
Err(err).
Str("authenticator", "oidc").
Str("path", r.URL.Path).
@@ -1,7 +1,6 @@
package middleware
import (
"bytes"
"net/http"
"net/http/httptest"
"time"
@@ -12,7 +11,6 @@ import (
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/pkg/oidc"
oidcmocks "github.com/opencloud-eu/opencloud/pkg/oidc/mocks"
"github.com/rs/zerolog"
"github.com/stretchr/testify/mock"
"go-micro.dev/v4/store"
)
@@ -108,69 +106,4 @@ var _ = Describe("Authenticating requests", Label("OIDCAuthenticator"), func() {
Expect(req2).ToNot(BeNil())
})
})
When("the request contains an expired token", func() {
var buf bytes.Buffer
expiredAuthenticator := &OIDCAuthenticator{
OIDCIss: "http://idp.example.com",
Logger: log.Logger{zerolog.New(&buf).Level(zerolog.DebugLevel)},
oidcClient: &oidcmocks.OIDCClient{},
userInfoCache: store.NewMemoryStore(),
skipUserInfo: true,
}
BeforeEach(func() {
expiredAuthenticator.oidcClient.(*oidcmocks.OIDCClient).On("VerifyAccessToken", mock.Anything, mock.Anything).Return(
oidc.RegClaimsWithSID{}, jwt.MapClaims{}, jwt.ErrTokenExpired,
)
})
It("should reject the request", func() {
req := httptest.NewRequest(http.MethodGet, "http://example.com/example/path", http.NoBody)
req.Header.Set(_headerAuthorization, "Bearer jwt.token.sig")
req2, valid := expiredAuthenticator.Authenticate(req)
Expect(valid).To(Equal(false))
Expect(req2).To(BeNil())
})
It("should log the authentication failure at debug level", func() {
req := httptest.NewRequest(http.MethodGet, "http://example.com/example/path", http.NoBody)
req.Header.Set(_headerAuthorization, "Bearer jwt.token.sig")
expiredAuthenticator.Authenticate(req)
Expect(buf.String()).To(ContainSubstring("failed to authenticate the request"))
Expect(buf.String()).To(ContainSubstring(`"level":"debug"`))
Expect(buf.String()).ToNot(ContainSubstring(`"level":"error"`))
})
})
When("the request contains a token that cannot be verified", func() {
var buf bytes.Buffer
failingAuthenticator := &OIDCAuthenticator{
OIDCIss: "http://idp.example.com",
Logger: log.Logger{zerolog.New(&buf).Level(zerolog.DebugLevel)},
oidcClient: &oidcmocks.OIDCClient{},
userInfoCache: store.NewMemoryStore(),
skipUserInfo: true,
}
BeforeEach(func() {
failingAuthenticator.oidcClient.(*oidcmocks.OIDCClient).On("VerifyAccessToken", mock.Anything, mock.Anything).Return(
oidc.RegClaimsWithSID{}, jwt.MapClaims{}, jwt.ErrTokenMalformed,
)
})
It("should log the authentication failure at error level", func() {
req := httptest.NewRequest(http.MethodGet, "http://example.com/example/path", http.NoBody)
req.Header.Set(_headerAuthorization, "Bearer jwt.token.sig")
failingAuthenticator.Authenticate(req)
Expect(buf.String()).To(ContainSubstring("failed to authenticate the request"))
Expect(buf.String()).To(ContainSubstring(`"level":"error"`))
})
})
})
+31 -15
View File
@@ -82,9 +82,6 @@ func (t Tika) Extract(ctx context.Context, ri *provider.ResourceInfo) (Document,
if err != nil {
return doc, err
}
if len(metas) == 0 {
return doc, nil
}
for _, meta := range metas {
title, err := getFirstValue(meta, "dc:title")
@@ -101,21 +98,40 @@ func (t Tika) Extract(ctx context.Context, ri *provider.ResourceInfo) (Document,
} else if content, err := getFirstValue(meta, "X-TIKA:content"); err == nil {
doc.Content = strings.TrimSpace(fmt.Sprintf("%s %s", doc.Content, content))
}
// keep facets from earlier entries, an embedded resource's meta
// (e.g. cover art) must not reset them
if v := t.getLocation(meta); v != nil {
doc.Location = v
}
if v := t.getImage(meta); v != nil {
doc.Image = v
}
if v := t.getPhoto(meta); v != nil {
doc.Photo = v
}
if v := t.getAudio(meta); v != nil {
doc.Audio = v
}
if v := t.getLivePhoto(meta); v != nil {
doc.LivePhoto = v
}
}
// facets describe the file itself, not its embedded parts (cover art, clips)
m0 := metas[0]
doc.Location = t.getLocation(m0)
doc.Image = t.getImage(m0)
doc.Photo = t.getPhoto(m0)
doc.Audio = t.getAudio(m0)
doc.LivePhoto = t.getLivePhoto(m0)
doc.Video = t.getVideo(m0)
if len(metas) > 0 {
// the video facet says the file is a video, so it comes from the file
// itself: the clip tika extracts from a motion photo must not make its
// image look like one
doc.Video = t.getVideo(metas[0])
}
// a motion photo is the file's own xmp plus the video tika extracted from
// it; the xmp alone proves nothing, a share can strip the appended clip
if i := slices.IndexFunc(metas[1:], isVideo); i >= 0 {
doc.MotionPhoto = t.getMotionPhoto(m0, metas[i+1])
// a motion photo is the xmp on the file itself plus the video tika extracted
// from it. The xmp alone proves nothing: a share can keep it and strip the
// appended video.
if len(metas) > 0 {
if i := slices.IndexFunc(metas[1:], isVideo); i >= 0 {
doc.MotionPhoto = t.getMotionPhoto(metas[0], metas[i+1])
}
}
if langCode := t.detectLanguage(ctx, doc.Content); langCode != "" && t.CleanStopWords {
+2 -4
View File
@@ -170,9 +170,7 @@ var _ = Describe("Tika", func() {
Expect(doc.Content).To(Equal("body test stop words!!!"))
})
It("takes facets from the main document, not from an embedded resource", func() {
// metas[0] is the file (audio), metas[1] its embedded cover art. The
// cover must not give the track an image facet.
It("keeps the audio facet when an embedded resource follows", func() {
fullResponse = `[{"Content-Type": "audio/mpeg", "dc:title": "Sucker", "tk:content": "lyrics"}, {"Content-Type": "image/jpeg", "tiff:ImageWidth": "500"}]`
doc, err := tika.Extract(context.TODO(), &provider.ResourceInfo{
@@ -182,7 +180,7 @@ var _ = Describe("Tika", func() {
Expect(err).ToNot(HaveOccurred())
Expect(doc.Audio).ToNot(BeNil())
Expect(doc.Audio.Title).To(Equal(libregraph.PtrString("Sucker")))
Expect(doc.Image).To(BeNil())
Expect(doc.Image).ToNot(BeNil())
})
It("adds no audio facet to non-audio documents", func() {
-7
View File
@@ -9,10 +9,3 @@ include ../../.make/default.mk
include ../../.make/go.mk
include ../../.make/release.mk
include ../../.make/docs.mk
include ../../.make/protobuf.mk
.PHONY: go-generate
go-generate: protobuf
.PHONY: protobuf
protobuf: buf-generate
+31 -78
View File
@@ -1,99 +1,53 @@
# Thumbnails
The thumbnails service provides methods to generate thumbnails for various files and resolutions based on requests. It retrieves the sources at the location where the user files are stored and saves the thumbnails where system files are stored. Those locations have defaults but can be manually defined via environment variables.
The thumbnails service is a stateless image resizer. It exposes an imagor-compatible push endpoint that accepts an original image as a multipart upload and returns the resized thumbnail encoded in the requested format.
## File Locations Overview
> **Warning:** This service performs no authentication or authorization. By default it binds to `127.0.0.1:9186` so only local processes (the webdav service) can reach it. If you change `THUMBNAILS_HTTP_ADDR` to a non-loopback address, put it behind your reverse proxy's auth layer or an internal network boundary — anyone who can reach the endpoint can upload and retrieve arbitrary images through it.
The relevant environment variables defining file locations are:
## Push Endpoint
- (1) `OC_BASE_DATA_PATH`
- (2) `STORAGE_USERS_DECOMPOSED_ROOT`
- (3) `THUMBNAILS_FILESYSTEMSTORAGE_ROOT`
The webdav service (which owns the complete thumbnail workflow) POSTs the source file to this endpoint and receives the processed image back.
(1) ... Having a default set by the OpenCloud code, but if defined, used as base path for other services.
(2) ... Source files, defaults to (1) plus path component, but can be freely defined if required.
(3) ... Target files, defaults to (1) plus path component, but can be freely defined if required.
| Route | Description |
|---|---|
| `POST /unsafe/{width}x{height}` (optionally `/filters:format({format})`) | Fill the exact width x height (center crop, may upscale) — the default |
| `POST /unsafe/fit-in/{width}x{height}` (optionally `/filters:format({format})`) | Scale to fit within width x height, preserving aspect ratio and never upscaling (letterboxed) |
| `POST /unsafe/stretch/{width}x{height}` (optionally `/filters:format({format})`) | Resize to the exact width x height without preserving aspect ratio (distorts) |
For details and defaults for these environment variables see the OpenCloud admin documentation.
The filters segment is captured whole and parsed; the `format` and `no_upscale` filters are meaningful to this executor, other filters are ignored. The `format` filter is optional: when absent the input's own format is preserved (detected from the image header), matching imagor. Inputs we cannot re-encode (e.g. webp, tiff, bmp) fall back to JPEG, mirroring imagor's default for unsavable sources.
## Thumbnail Location
Like real imagor, the default fill resize **upscales** small sources to fill the box exactly (a 100x100 source requested at `320x320` returns a 320x320 image). Adding the `no_upscale()` filter caps the result at the source size instead (the same request returns 100x100). The `fit-in` route never upscales regardless of filters.
It may be beneficial to define the location of the thumbnails to be other than the default (with system files). This is due the fact that storing thumbnails can consume a lot of space over time which not necessarily needs to reside on the same partition or mount or expensive drives.
The webdav service selects the route based on the requested processor. The full mapping is:
## Thumbnail Source File Types
| Request | Operation |
|---|---|
| `processor=resize` | stretch (distort to the exact box) |
| `processor=fill` or `processor=thumbnail` | fill (center-crop to the exact box, may upscale) |
| `processor=fit` / `processor=fit-in` | fit-in (preserve aspect, fit in box, never upscale) |
| no processor, gif source | stretch (resize for gifs) |
| no processor, other sources | fill (default = thumbnail = fill) |
Thumbnails can be generated from the following source file types:
By default (no processor, non-gif) the fill form is used, which center-crops to the exact box and upscales small sources — this matches the legacy `thumbnail` processor behavior (e.g. a 200x100 image requested at 100x100 returns a square 100x100 image). Requesting `processor=fit` switches to the `fit-in` form, which preserves aspect ratio and never upscales, letterboxing a non-square source into the box (the same image returns 100x50).
- png
- jpg
- gif
- tiff
- bmp
- txt
### Legacy `a` parameter
The thumbnail service retrieves source files using the information provided by the backend. The Linux backend identifies source files usually based on the extension.
The webdav preview endpoint also accepts a legacy `a` flag: `a=1` (or absent) means "preserve aspect" (fit-in), `a=0` means "fill". An explicit `processor` always wins over `a`. When an explicit processor overrides a contradictory `a`, the thumbnail response includes the header `X-OpenCloud-Thumbnail-Aspect-Ignored` so developers can tell their client to send a consistent request.
If a file type was not properly assigned or the type identification failed, thumbnail generation will fail and an error will be logged.
The request body is a `multipart/form-data` upload with a single file field named `image`. Supported output formats are `jpg`, `png`, and `gif`.
## Thumbnail Target File Types
## Configuration
Thumbnails can either be generated as `png`, `jpg` or `gif` files. These types are hardcoded and no other types can be requested. A requestor, like another service or a client, can request one of the available types to be generated. If more than one type is required, each type must be requested individually.
| Environment variable | Description |
|---|---|
| `THUMBNAILS_HTTP_ADDR` | Bind address of the HTTP service (default `127.0.0.1:9186`) |
| `THUMBNAILS_LOG_LEVEL` | Log level (`panic`, `fatal`, `error`, `warn`, `info`, `debug`, `trace`) |
| `THUMBNAILS_MAX_CONCURRENT_REQUESTS` | Maximum number of thumbnail requests decoded and resized in parallel. Default is 0 (unlimited). Requests arriving while the limit is reached get HTTP 429 with a `Retry-After` header |
## Thumbnail Query String Parameters
## Using libvips for Image Processing
Clients can request thumbnail previews for files by adding `?preview=1` to the file URL. Requests for files with no thumbnail available respond with HTTP status `404`.
The following query parameters are supported:
| Parameter | Required | Default Value | Description |
|-----------|----------|------------------------------------------------------|---------------------------------------------------------------------------------|
| preview | YES | 1 | generates preview |
| x | YES | first x-value configured in `THUMBNAILS_RESOLUTIONS` | horizontal target size |
| y | YES | first y-value configured in `THUMBNAILS_RESOLUTIONS` | vertical target size |
| scalingup | NO | 0 | prevents up-scaling of small images |
| a | NO | 1 | aspect ratio |
| c | NO | Caching string | Clients should send the etag, so they get a fresh thumbnail after a file change |
| processor | NO | `resize` for gifs and `thumbnail` for all others | preferred thumbnail processor |
## Thumbnail Resolution
Various resolutions can be defined via `THUMBNAILS_RESOLUTIONS`. A requestor can request any arbitrary resolution and the thumbnail service will use the one closest to the requested resolution. If more than one resolution is required, each resolution must be requested individually.
Example:
Requested: 18x12\
Available: 30x20, 15x10, 9x6\
Returned: 15x10
## Thumbnail Processors
Normally, an image might get cropped when creating a preview, depending on the aspect ratio of the original image. This can have negative
impacts on previews as only a part of the image will be shown. When using an _optional_ processor in the request, cropping can be avoided by defining on how the preview image generation will be done. The following processors are available:
* `resize` resizes the image to the specified width and height and returns the transformed image. If one of width or height is 0, the image aspect ratio is preserved.
* `fit` scales down the image to fit the specified maximum width and height and returns the transformed image.
* `fill`: creates an image with the specified dimensions and fills it with the scaled source image. To achieve the correct aspect ratio without stretching, the source image will be cropped.
* `thumbnail` scales the image up or down, crops it to the specified width and height and returns the transformed image.
To apply one of those, a query parameter has to be added to the request, like `?processor=fit`. If no query parameter or processor is added, the default behaviour applies which is `resize` for gifs and `thumbnail` for all others.
## Deleting Thumbnails
As of now, there is no automated thumbnail deletion. This is especially true when a source file gets deleted or moved. This situation will be solved at a later stage. For the time being, if you run short on physical thumbnails space, you have to manually delete the thumbnail store to free space. Thumbnails will then be recreated on request.
## Memory Considerations
Since source files need to be loaded into memory when generating thumbnails, large source files could potentially crash this service if there is insufficient memory available. For bigger instances when using container orchestration deployment methods, this service can be dedicated to its own server(s) with more memory.
To have more control over memory (and CPU) consumption the maximum number of concurrent requests can be limited by setting the environment variable `THUMBNAILS_MAX_CONCURRENT_REQUESTS`. The default value is 0 which does not apply any restrictions to the number of concurrent requests. As soon as the number of concurrent requests is reached any further request will be responded with `429/Too Many Requests` and the client can retry at a later point in time.
## Thumbnails and SecureView
If a resource is shared using SecureView, the share reciever will get a 403 (forbidden) response when requesting a thumbnail. The requesting client needs to decide what to show and usually a placeholder thumbnail is used.
## Using libvips for Thumbnail Generation
To improve performance and to support a wider range of images formats, the thumbnails service is able to utilize the [libvips library](https://www.libvips.org/) for thumbnail generation. Support for libvips needs to be
enabled at buildtime and has a couple of implications:
To improve performance and to support a wider range of image formats, the thumbnails service is able to utilize the [libvips library](https://www.libvips.org/) for image processing. Support for libvips needs to be enabled at buildtime and has a couple of implications:
* With libvips support enabled, it is not possible to create a statically linked OpenCloud binary.
* Therefore, the libvips shared libraries need to be available at runtime in the same release that was used to build the OpenCloud binary.
@@ -120,4 +74,3 @@ go build -tags enable_vips -o opencloud -o bin/opencloud ./cmd/opencloud
When building a docker image using the Dockerfile in the top-level directory of OpenCloud, libvips support is enabled and the libvips shared libraries are included
in the resulting docker image.
-19
View File
@@ -8,14 +8,12 @@ import (
"github.com/opencloud-eu/opencloud/pkg/config/configlog"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/pkg/runner"
ogrpc "github.com/opencloud-eu/opencloud/pkg/service/grpc"
"github.com/opencloud-eu/opencloud/pkg/tracing"
"github.com/opencloud-eu/opencloud/pkg/version"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/config"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/config/parser"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/metrics"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/server/debug"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/server/grpc"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/server/http"
"github.com/spf13/cobra"
@@ -36,10 +34,6 @@ func Server(cfg *config.Config) *cobra.Command {
if err != nil {
return err
}
cfg.GrpcClient, err = ogrpc.NewClient(ogrpc.GetClientOptions(cfg.GRPCClientTLS)...)
if err != nil {
return err
}
var cancel context.CancelFunc
if cfg.Context == nil {
@@ -53,19 +47,6 @@ func Server(cfg *config.Config) *cobra.Command {
gr := runner.NewGroup()
service := grpc.NewService(
grpc.Logger(logger),
grpc.Context(ctx),
grpc.Config(cfg),
grpc.Name(cfg.Service.Name),
grpc.Namespace(cfg.GRPC.Namespace),
grpc.Address(cfg.GRPC.Addr),
grpc.Metrics(m),
grpc.TraceProvider(traceProvider),
grpc.MaxConcurrentRequests(cfg.GRPC.MaxConcurrentRequests),
)
gr.Add(runner.NewGoMicroGrpcServerRunner(cfg.Service.Name+".grpc", service))
server, err := debug.Server(
debug.Logger(logger),
debug.Config(cfg),
+1 -1
View File
@@ -24,7 +24,7 @@ func Version(cfg *config.Config) *cobra.Command {
fmt.Println("")
reg := registry.GetRegistry()
services, err := reg.GetService(cfg.GRPC.Namespace + "." + cfg.Service.Name)
services, err := reg.GetService(cfg.HTTP.Namespace + "." + cfg.Service.Name)
if err != nil {
fmt.Println(fmt.Errorf("could not get %s services from the registry: %v", cfg.Service.Name, err))
return err
+14 -23
View File
@@ -5,7 +5,6 @@ import (
"context"
"github.com/opencloud-eu/opencloud/pkg/shared"
"go-micro.dev/v4/client"
)
// Config combines all available configuration parts.
@@ -17,33 +16,25 @@ type Config struct {
LogLevel string `yaml:"loglevel" env:"OC_LOG_LEVEL;THUMBNAILS_LOG_LEVEL" desc:"The log level. Valid values are: 'panic', 'fatal', 'error', 'warn', 'info', 'debug', 'trace'." introductionVersion:"1.0.0"`
Debug Debug `yaml:"debug"`
GRPC GRPCConfig `yaml:"grpc"`
HTTP HTTP `yaml:"http"`
GRPCClientTLS *shared.GRPCClientTLS `yaml:"grpc_client_tls"`
GrpcClient client.Client `yaml:"-"`
HTTP HTTP `yaml:"http"`
Thumbnail Thumbnail `yaml:"thumbnail"`
Context context.Context `yaml:"-"`
}
// FileSystemStorage defines the available filesystem storage configuration.
type FileSystemStorage struct {
RootDirectory string `yaml:"root_directory" env:"THUMBNAILS_FILESYSTEMSTORAGE_ROOT" desc:"The directory where the filesystem storage will store the thumbnails. If not defined, the root directory derives from $OC_BASE_DATA_PATH/thumbnails." introductionVersion:"1.0.0"`
}
// Thumbnail defines the available thumbnail related configuration.
// Thumbnail holds the safety limits enforced by the push endpoint. The limits
// mirror the legacy thumbnail service so a request that was rejected before is
// still rejected, but the enforcement now lives at decode time in this service
// (which owns the image) rather than in webdav.
type Thumbnail struct {
Resolutions []string `yaml:"resolutions" env:"THUMBNAILS_RESOLUTIONS" desc:"The supported list of target resolutions in the format WidthxHeight like 32x32. You can define any resolution as required. See the Environment Variable Types description for more details." introductionVersion:"1.0.0"`
FileSystemStorage FileSystemStorage `yaml:"filesystem_storage"`
WebdavAllowInsecure bool `yaml:"webdav_allow_insecure" env:"OC_INSECURE;THUMBNAILS_WEBDAVSOURCE_INSECURE" desc:"Ignore untrusted SSL certificates when connecting to the webdav source." introductionVersion:"1.0.0"`
CS3AllowInsecure bool `yaml:"cs3_allow_insecure" env:"OC_INSECURE;THUMBNAILS_CS3SOURCE_INSECURE" desc:"Ignore untrusted SSL certificates when connecting to the CS3 source." introductionVersion:"1.0.0"`
RevaGateway string `yaml:"reva_gateway" env:"OC_REVA_GATEWAY" desc:"CS3 gateway used to look up user metadata" introductionVersion:"1.0.0"`
FontMapFile string `yaml:"font_map_file" env:"THUMBNAILS_TXT_FONTMAP_FILE" desc:"The path to a font file for txt thumbnails." introductionVersion:"1.0.0"`
TransferSecret string `yaml:"transfer_secret" env:"THUMBNAILS_TRANSFER_TOKEN" desc:"The secret to sign JWT to download the actual thumbnail file." introductionVersion:"1.0.0"`
DataEndpoint string `yaml:"data_endpoint" env:"THUMBNAILS_DATA_ENDPOINT" desc:"The HTTP endpoint where the actual thumbnail file can be downloaded." introductionVersion:"1.0.0"`
MaxInputWidth int `yaml:"max_input_width" env:"THUMBNAILS_MAX_INPUT_WIDTH" desc:"The maximum width of an input image which is being processed." introductionVersion:"1.0.0"`
MaxInputHeight int `yaml:"max_input_height" env:"THUMBNAILS_MAX_INPUT_HEIGHT" desc:"The maximum height of an input image which is being processed." introductionVersion:"1.0.0"`
MaxInputImageFileSize string `yaml:"max_input_image_file_size" env:"THUMBNAILS_MAX_INPUT_IMAGE_FILE_SIZE" desc:"The maximum file size of an input image which is being processed. Usable common abbreviations: [KB, KiB, MB, MiB, GB, GiB, TB, TiB, PB, PiB, EB, EiB], example: 2GB." introductionVersion:"1.0.0"`
// MaxInputWidth is the maximum width of an input image which is being processed.
MaxInputWidth int `yaml:"max_input_width" env:"THUMBNAILS_MAX_INPUT_WIDTH" desc:"The maximum width of an input image which is being processed." introductionVersion:"1.0.0"`
// MaxInputHeight is the maximum height of an input image which is being processed.
MaxInputHeight int `yaml:"max_input_height" env:"THUMBNAILS_MAX_INPUT_HEIGHT" desc:"The maximum height of an input image which is being processed." introductionVersion:"1.0.0"`
// MaxConcurrentRequests limits the number of thumbnail requests that are
// decoded and resized in parallel. 0 (the default) means unlimited, matching
// main's THUMBNAILS_MAX_CONCURRENT_REQUESTS. Requests arriving while the limit
// is reached get HTTP 429 Too Many Requests.
MaxConcurrentRequests int `yaml:"max_concurrent_requests" env:"THUMBNAILS_MAX_CONCURRENT_REQUESTS" desc:"Maximum number of concurrent thumbnail generation requests. Default is 0 which is unlimited." introductionVersion:"1.0.0"`
}
@@ -1,12 +1,6 @@
package defaults
import (
"path"
"strings"
"github.com/opencloud-eu/opencloud/pkg/config/defaults"
"github.com/opencloud-eu/opencloud/pkg/shared"
"github.com/opencloud-eu/opencloud/pkg/structs"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/config"
)
@@ -27,14 +21,8 @@ func DefaultConfig() *config.Config {
Pprof: false,
Zpages: false,
},
GRPC: config.GRPCConfig{
Addr: "127.0.0.1:9185",
Namespace: "eu.opencloud.api",
MaxConcurrentRequests: 0,
},
HTTP: config.HTTP{
Addr: "127.0.0.1:9186",
Root: "/thumbnails",
Namespace: "eu.opencloud.web",
CORS: config.CORS{
AllowedOrigins: []string{"*"},
@@ -47,17 +35,8 @@ func DefaultConfig() *config.Config {
Name: "thumbnails",
},
Thumbnail: config.Thumbnail{
Resolutions: []string{"16x16", "32x32", "64x64", "128x128", "500x280", "280x500", "1000x560", "560x1000", "512x2048", "1080x1920", "1920x1080", "2160x3840", "3840x2160", "4320x7680", "7680x4320"},
FileSystemStorage: config.FileSystemStorage{
RootDirectory: path.Join(defaults.BaseDataPath(), "thumbnails"),
},
WebdavAllowInsecure: false,
RevaGateway: shared.DefaultRevaConfig().Address,
CS3AllowInsecure: false,
DataEndpoint: "http://127.0.0.1:9186/thumbnails/data",
MaxInputWidth: 7680,
MaxInputHeight: 7680,
MaxInputImageFileSize: "50MB",
MaxInputWidth: 7680,
MaxInputHeight: 7680,
},
}
}
@@ -68,13 +47,6 @@ func EnsureDefaults(cfg *config.Config) {
cfg.LogLevel = "error"
}
if cfg.GRPCClientTLS == nil && cfg.Commons != nil {
cfg.GRPCClientTLS = structs.CopyOrZeroValue(cfg.Commons.GRPCClientTLS)
}
if cfg.GRPC.TLS == nil && cfg.Commons != nil {
cfg.GRPC.TLS = structs.CopyOrZeroValue(cfg.Commons.GRPCServiceTLS)
}
if cfg.Commons != nil {
cfg.HTTP.TLS = cfg.Commons.HTTPServiceTLS
}
@@ -82,8 +54,4 @@ func EnsureDefaults(cfg *config.Config) {
// Sanitize sanitized the configuration
func Sanitize(cfg *config.Config) {
// nothing to sanitize here atm
if len(cfg.Thumbnail.Resolutions) == 1 && strings.Contains(cfg.Thumbnail.Resolutions[0], ",") {
cfg.Thumbnail.Resolutions = strings.Split(cfg.Thumbnail.Resolutions[0], ",")
}
}
-11
View File
@@ -1,11 +0,0 @@
package config
import "github.com/opencloud-eu/opencloud/pkg/shared"
// GRPCConfig defines the available grpc configuration.
type GRPCConfig struct {
Addr string `yaml:"addr" env:"THUMBNAILS_GRPC_ADDR" desc:"The bind address of the GRPC service." introductionVersion:"1.0.0"`
Namespace string `yaml:"-"`
TLS *shared.GRPCServiceTLS `yaml:"tls"`
MaxConcurrentRequests int `yaml:"max_concurrent_requests" env:"THUMBNAILS_MAX_CONCURRENT_REQUESTS" desc:"Number of maximum concurrent thumbnail requests. Default is 0 which is unlimited." introductionVersion:"1.0.0"`
}
-1
View File
@@ -14,7 +14,6 @@ type CORS struct {
type HTTP struct {
Addr string `yaml:"addr" env:"THUMBNAILS_HTTP_ADDR" desc:"The bind address of the HTTP service." introductionVersion:"1.0.0"`
TLS shared.HTTPServiceTLS `yaml:"tls"`
Root string `yaml:"root" env:"THUMBNAILS_HTTP_ROOT" desc:"Subdirectory that serves as the root for this HTTP service." introductionVersion:"1.0.0"`
Namespace string `yaml:"-"`
CORS CORS `yaml:"cors"`
}
-22
View File
@@ -1,22 +0,0 @@
package errors
import "errors"
var (
// ErrImageTooLarge defines an error when an input image is too large
ErrImageTooLarge = errors.New("thumbnails: image is too large")
// ErrInvalidType represents the error when a type can't be encoded.
ErrInvalidType = errors.New("thumbnails: can't encode this type")
// ErrNoEncoderForType represents the error when an encoder couldn't be found for a type.
ErrNoEncoderForType = errors.New("thumbnails: no encoder for this type found")
// ErrNoGeneratorForType represents the error when a generator couldn't be found for a type.
ErrNoGeneratorForType = errors.New("thumbnails: no generator for this type found")
// ErrNoImageFromAudioFile defines an error when an image cannot be extracted from an audio file
ErrNoImageFromAudioFile = errors.New("thumbnails: could not extract image from audio file")
// ErrNoConverterForExtractedImageFromGgsFile defines an error when the extracted image from an ggs file could not be converted
ErrNoConverterForExtractedImageFromGgsFile = errors.New("thumbnails: could not find converter for image extracted from ggs file")
// ErrNoConverterForExtractedImageFromAudioFile defines an error when the extracted image from an audio file could not be converted
ErrNoConverterForExtractedImageFromAudioFile = errors.New("thumbnails: could not find converter for image extracted from audio file")
// ErrCS3AuthorizationMissing defines an error when the CS3 authorization is missing
ErrCS3AuthorizationMissing = errors.New("thumbnails: cs3source - authorization missing")
)
@@ -12,33 +12,12 @@ var (
// Metrics defines the available metrics of this service.
type Metrics struct {
Counter *prometheus.CounterVec
Latency *prometheus.SummaryVec
Duration *prometheus.HistogramVec
BuildInfo *prometheus.GaugeVec
}
// New initializes the available metrics.
func New() *Metrics {
m := &Metrics{
Counter: prometheus.NewCounterVec(prometheus.CounterOpts{
Namespace: Namespace,
Subsystem: Subsystem,
Name: "getthumbnail_total",
Help: "How many GetThumbnail requests processed",
}, []string{}),
Latency: prometheus.NewSummaryVec(prometheus.SummaryOpts{
Namespace: Namespace,
Subsystem: Subsystem,
Name: "getthumbnail_latency_microseconds",
Help: "GetThumbnail request latencies in microseconds",
}, []string{}),
Duration: prometheus.NewHistogramVec(prometheus.HistogramOpts{
Namespace: Namespace,
Subsystem: Subsystem,
Name: "getthumbnail_duration_seconds",
Help: "GetThumbnail method requests time in seconds",
}, []string{}),
BuildInfo: prometheus.NewGaugeVec(prometheus.GaugeOpts{
Namespace: Namespace,
Subsystem: Subsystem,
@@ -47,18 +26,6 @@ func New() *Metrics {
}, []string{"version"}),
}
_ = prometheus.Register(
m.Counter,
)
_ = prometheus.Register(
m.Latency,
)
_ = prometheus.Register(
m.Duration,
)
_ = prometheus.Register(
m.BuildInfo,
)
@@ -1,105 +0,0 @@
//go:build !enable_vips
package preprocessor
import (
"bytes"
"encoding/base64"
"encoding/json"
"image"
"image/jpeg"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
thumbnailerErrors "github.com/opencloud-eu/opencloud/services/thumbnails/pkg/errors"
)
// craftDimensionBomb encodes a tiny valid grayscale JPEG, then overwrites the
// SOF0 width/height so the header declares huge dimensions while the payload
// stays tiny.
func craftDimensionBomb(width, height uint16) []byte {
var buf bytes.Buffer
Expect(jpeg.Encode(&buf, image.NewGray(image.Rect(0, 0, 8, 8)), &jpeg.Options{Quality: 10})).To(Succeed())
b := buf.Bytes()
for i := 0; i+9 < len(b); i++ {
if b[i] == 0xff && b[i+1] == 0xc0 {
b[i+5], b[i+6] = byte(height>>8), byte(height)
b[i+7], b[i+8] = byte(width>>8), byte(width)
return b
}
}
Fail("no SOF0 marker in encoded jpeg")
return nil
}
var _ = Describe("ImageDecoder dimension guard", func() {
It("rejects a source whose declared dimensions exceed the limit before decoding", func() {
dec := ImageDecoder{limit: decodeLimit{maxWidth: 7680, maxHeight: 7680}}
_, err := dec.Convert(bytes.NewReader(craftDimensionBomb(20000, 20000)))
Expect(err).To(MatchError(thumbnailerErrors.ErrImageTooLarge))
})
It("decodes an image within the limit", func() {
var buf bytes.Buffer
Expect(jpeg.Encode(&buf, image.NewGray(image.Rect(0, 0, 800, 600)), nil)).To(Succeed())
dec := ImageDecoder{limit: decodeLimit{maxWidth: 7680, maxHeight: 7680}}
img, err := dec.Convert(bytes.NewReader(buf.Bytes()))
Expect(err).ToNot(HaveOccurred())
Expect(img).ToNot(BeNil())
})
It("applies no limit when the bounds are zero", func() {
dec := ImageDecoder{}
var buf bytes.Buffer
Expect(jpeg.Encode(&buf, image.NewGray(image.Rect(0, 0, 16, 16)), nil)).To(Succeed())
_, err := dec.Convert(bytes.NewReader(buf.Bytes()))
Expect(err).ToNot(HaveOccurred())
})
It("rejects an oversized gif before decoding", func() {
dec := GifDecoder{limit: decodeLimit{maxWidth: 7680, maxHeight: 7680}}
// LSD declares a 20000x20000 logical screen
g := []byte("GIF89a")
g = append(g, 0x20, 0x4e, 0x20, 0x4e, 0xf0, 0x00, 0x00) // 20000x20000, gct flag
g = append(g, bytes.Repeat([]byte{0}, 6)...) // minimal gct + terminator-ish
_, err := dec.Convert(bytes.NewReader(g))
Expect(err).To(MatchError(thumbnailerErrors.ErrImageTooLarge))
})
})
var _ = Describe("dimension limit propagation", func() {
limit := decodeLimit{maxWidth: 7680, maxHeight: 7680}
opts := map[string]any{"maxInputWidth": 7680, "maxInputHeight": 7680}
It("threads the limit into decoders that recurse into ForType", func() {
Expect(ForType("audio/mpeg", opts)).To(Equal(AudioDecoder{limit: limit}))
Expect(ForType("application/vnd.geogebra.pinboard", opts)).To(Equal(GgpDecoder{limit: limit}))
g, ok := ForType("application/vnd.geogebra.slides", opts).(GgsDecoder)
Expect(ok).To(BeTrue())
Expect(g.limit).To(Equal(limit))
})
It("rejects an oversized cover image embedded in a ggp file", func() {
bomb := craftDimensionBomb(20000, 20000)
payload := "data:image/jpeg;base64," + base64.StdEncoding.EncodeToString(bomb)
ggp := GGPStruct{}
ggp.Sections = append(ggp.Sections, struct {
Cards []struct {
Element struct {
Image struct{ Base64Image string }
}
}
}{Cards: []struct {
Element struct {
Image struct{ Base64Image string }
}
}{{Element: struct {
Image struct{ Base64Image string }
}{Image: struct{ Base64Image string }{Base64Image: payload}}}}})
raw, err := json.Marshal(ggp)
Expect(err).ToNot(HaveOccurred())
_, err = GgpDecoder{limit: limit}.Convert(bytes.NewReader(raw))
Expect(err).To(MatchError(thumbnailerErrors.ErrImageTooLarge))
})
})
@@ -1,31 +0,0 @@
//go:build enable_vips
package preprocessor
import (
"io"
"github.com/davidbyttow/govips/v2/vips"
thumbnailerErrors "github.com/opencloud-eu/opencloud/services/thumbnails/pkg/errors"
)
func init() {
vips.LoggingSettings(nil, vips.LogLevelError)
}
type ImageDecoder struct{ limit decodeLimit }
func (v ImageDecoder) Convert(r io.Reader) (interface{}, error) {
// NewImageFromReader is header-lazy, Width/Height read the header without
// materializing pixels: reject oversized sources before ThumbnailWithSize
img, err := vips.NewImageFromReader(r)
if err != nil {
return nil, err
}
if v.limit.exceeded(img.Width(), img.Height()) {
img.Close()
return nil, thumbnailerErrors.ErrImageTooLarge
}
return img, nil
}
@@ -16,8 +16,7 @@ func Server(opts ...Option) (*http.Server, error) {
checkHandler := handlers.NewCheckHandler(
handlers.NewCheckHandlerConfiguration().
WithLogger(options.Logger).
WithCheck("web reachability", checks.NewHTTPCheck(options.Config.HTTP.Addr)).
WithCheck("grpc reachability", checks.NewGRPCCheck(options.Config.GRPC.Addr)),
WithCheck("web reachability", checks.NewHTTPCheck(options.Config.HTTP.Addr)),
)
return debug.NewService(
@@ -1,100 +0,0 @@
package grpc
import (
"context"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/config"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/metrics"
"go.opentelemetry.io/otel/trace"
)
// Option defines a single option function.
type Option func(o *Options)
// Options defines the available options for this package.
type Options struct {
Name string
Address string
Logger log.Logger
Context context.Context
Config *config.Config
Metrics *metrics.Metrics
Namespace string
TraceProvider trace.TracerProvider
MaxConcurrentRequests int
}
// newOptions initializes the available default options.
func newOptions(opts ...Option) Options {
opt := Options{}
for _, o := range opts {
o(&opt)
}
return opt
}
// Logger provides a function to set the logger option.
func Logger(val log.Logger) Option {
return func(o *Options) {
o.Logger = val
}
}
// Name provides a name for the service.
func Name(val string) Option {
return func(o *Options) {
o.Name = val
}
}
// Address provides an address for the service.
func Address(val string) Option {
return func(o *Options) {
o.Address = val
}
}
// Context provides a function to set the context option.
func Context(val context.Context) Option {
return func(o *Options) {
o.Context = val
}
}
// Config provides a function to set the config option.
func Config(val *config.Config) Option {
return func(o *Options) {
o.Config = val
}
}
// Metrics provides a function to set the metrics option.
func Metrics(val *metrics.Metrics) Option {
return func(o *Options) {
o.Metrics = val
}
}
// Namespace provides a function to set the namespace option.
func Namespace(val string) Option {
return func(o *Options) {
o.Namespace = val
}
}
// TraceProvider provides a function to set the TraceProvider option
func TraceProvider(val trace.TracerProvider) Option {
return func(o *Options) {
o.TraceProvider = val
}
}
// MaxConcurrentRequests provides a function to set the MaxConcurrentRequests option.
func MaxConcurrentRequests(val int) Option {
return func(o *Options) {
o.MaxConcurrentRequests = val
}
}
@@ -1,92 +0,0 @@
package grpc
import (
"github.com/opencloud-eu/opencloud/pkg/registry"
"github.com/opencloud-eu/opencloud/pkg/service/grpc"
"github.com/opencloud-eu/opencloud/pkg/service/grpc/handler/ratelimiter"
"github.com/opencloud-eu/opencloud/pkg/version"
thumbnailssvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/thumbnails/v0"
svc "github.com/opencloud-eu/opencloud/services/thumbnails/pkg/service/grpc/v0"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/service/grpc/v0/decorators"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/thumbnail/imgsource"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/thumbnail/storage"
"github.com/opencloud-eu/reva/v2/pkg/bytesize"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
)
// NewService initializes the grpc service and server.
func NewService(opts ...Option) grpc.Service {
options := newOptions(opts...)
service, err := grpc.NewServiceWithClient(
options.Config.GrpcClient,
grpc.TLSEnabled(options.Config.GRPC.TLS.Enabled),
grpc.TLSCert(
options.Config.GRPC.TLS.Cert,
options.Config.GRPC.TLS.Key,
),
grpc.Logger(options.Logger),
grpc.Namespace(options.Namespace),
grpc.Name(options.Name),
grpc.Version(version.GetString()),
grpc.Address(options.Address),
grpc.Context(options.Context),
grpc.Version(version.GetString()),
grpc.TraceProvider(options.TraceProvider),
grpc.HandlerWrappers(ratelimiter.NewHandlerWrapper(options.MaxConcurrentRequests)),
)
if err != nil {
options.Logger.Fatal().Err(err).Msg("Error creating thumbnail service")
return grpc.Service{}
}
tconf := options.Config.Thumbnail
tm, err := pool.StringToTLSMode(options.Config.GRPCClientTLS.Mode)
if err != nil {
options.Logger.Error().Err(err).Msg("could not get gateway client tls mode")
return grpc.Service{}
}
gatewaySelector, err := pool.GatewaySelector(tconf.RevaGateway,
pool.WithTLSCACert(options.Config.GRPCClientTLS.CACert),
pool.WithTLSMode(tm),
pool.WithRegistry(registry.GetRegistry()),
pool.WithTracerProvider(options.TraceProvider),
)
if err != nil {
options.Logger.Error().Err(err).Msg("could not get gateway selector")
return grpc.Service{}
}
b, err := bytesize.Parse(tconf.MaxInputImageFileSize)
if err != nil {
options.Logger.Error().Err(err).Msg("could not parse MaxInputImageFileSize")
return grpc.Service{}
}
var thumbnail decorators.DecoratedService
{
thumbnail = svc.NewService(
svc.Config(options.Config),
svc.Logger(options.Logger),
svc.ThumbnailSource(imgsource.NewWebDavSource(tconf, b)),
svc.ThumbnailStorage(
storage.NewFileSystemStorage(
tconf.FileSystemStorage,
options.Logger,
),
),
svc.CS3Source(imgsource.NewCS3Source(tconf, gatewaySelector, b)),
svc.GatewaySelector(gatewaySelector),
)
thumbnail = decorators.NewInstrument(thumbnail, options.Metrics)
thumbnail = decorators.NewLogging(thumbnail, options.Logger)
thumbnail = decorators.NewTracing(thumbnail, options.TraceProvider)
}
_ = thumbnailssvc.RegisterThumbnailServiceHandler(
service.Server(),
thumbnail,
)
return service
}
@@ -9,7 +9,6 @@ import (
"github.com/opencloud-eu/opencloud/pkg/service/http"
"github.com/opencloud-eu/opencloud/pkg/version"
svc "github.com/opencloud-eu/opencloud/services/thumbnails/pkg/service/http/v0"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/thumbnail/storage"
"go-micro.dev/v4"
)
@@ -53,19 +52,8 @@ func Server(opts ...Option) (http.Service, error) {
),
opencloudmiddleware.Logger(options.Logger),
),
svc.ThumbnailStorage(
storage.NewFileSystemStorage(
options.Config.Thumbnail.FileSystemStorage,
options.Logger,
),
),
)
{
handle = svc.NewInstrument(handle, options.Metrics)
handle = svc.NewLogging(handle, options.Logger)
}
if err := micro.RegisterHandler(service.Server(), handle); err != nil {
return http.Service{}, err
}
@@ -1,63 +0,0 @@
package decorators
import (
"context"
thumbnailssvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/thumbnails/v0"
)
// DecoratedService is an interface acting as facade, holding all the interfaces that this
// thumbnails microservice is expecting to implement.
// For now, only the thumbnailssvc.ThumbnailServiceHandler is present,
// but a future configsvc.ConfigServiceHandler is expected to be added here
//
// This interface will also act as the base interface to implement
// a decorator pattern.
type DecoratedService interface {
thumbnailssvc.ThumbnailServiceHandler
}
// Decorator is the base type to implement the decorators. It will provide a basic implementation
// by delegating to the decoratedService
//
// Expected implementations will be like:
// ```
//
// type MyDecorator struct {
// Decorator
// myCustomOpts *opts
// additionalSrv *srv
// }
//
// func NewMyDecorator(next DecoratedService, customOpts *customOpts) DecoratedService {
// .....
// return MyDecorator{
// Decorator: Decorator{next: next},
// myCustomOpts: opts,
// additionalSrv: srv,
// }
// }
//
// ```
type Decorator struct {
next DecoratedService
}
// GetThumbnail is the base implementation for the thumbnailssvc.GetThumbnail.
// It will just delegate to the underlying decoratedService
//
// Your custom decorator is expected to overwrite this function,
// but it MUST call the underlying decoratedService at some point
// ```
//
// func (d MyDecorator) GetThumbnail(ctx context.Context, req *thumbnailssvc.GetThumbnailRequest, resp *thumbnailssvc.GetThumbnailResponse) error {
// doSomething()
// err := d.next.GetThumbnail(ctx, req, resp)
// doAnotherThing()
// return err
// }
//
// ```
func (deco Decorator) GetThumbnail(ctx context.Context, req *thumbnailssvc.GetThumbnailRequest, resp *thumbnailssvc.GetThumbnailResponse) error {
return deco.next.GetThumbnail(ctx, req, resp)
}
@@ -1,39 +0,0 @@
package decorators
import (
"context"
thumbnailssvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/thumbnails/v0"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/metrics"
"github.com/prometheus/client_golang/prometheus"
)
// NewInstrument returns a service that instruments metrics.
func NewInstrument(next DecoratedService, metrics *metrics.Metrics) DecoratedService {
return instrument{
Decorator: Decorator{next: next},
metrics: metrics,
}
}
type instrument struct {
Decorator
metrics *metrics.Metrics
}
// GetThumbnail implements the ThumbnailServiceHandler interface.
func (i instrument) GetThumbnail(ctx context.Context, req *thumbnailssvc.GetThumbnailRequest, rsp *thumbnailssvc.GetThumbnailResponse) error {
timer := prometheus.NewTimer(prometheus.ObserverFunc(func(v float64) {
us := v * 1000_000
i.metrics.Latency.WithLabelValues().Observe(us)
i.metrics.Duration.WithLabelValues().Observe(v)
}))
defer timer.ObserveDuration()
err := i.next.GetThumbnail(ctx, req, rsp)
if err != nil {
i.metrics.Counter.WithLabelValues().Inc()
}
return err
}
@@ -1,53 +0,0 @@
package decorators
import (
"context"
"net/http"
"time"
"github.com/opencloud-eu/opencloud/pkg/log"
thumbnailssvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/thumbnails/v0"
merrors "go-micro.dev/v4/errors"
)
// NewLogging returns a service that logs messages.
func NewLogging(next DecoratedService, logger log.Logger) DecoratedService {
return logging{
Decorator: Decorator{next: next},
logger: logger,
}
}
type logging struct {
Decorator
logger log.Logger
}
// GetThumbnail implements the ThumbnailServiceHandler interface.
func (l logging) GetThumbnail(ctx context.Context, req *thumbnailssvc.GetThumbnailRequest, rsp *thumbnailssvc.GetThumbnailResponse) error {
start := time.Now()
err := l.next.GetThumbnail(ctx, req, rsp)
logger := l.logger.With().
Str("method", "Thumbnails.GetThumbnail").
Dur("duration", time.Since(start)).
Logger()
if err != nil {
fromError := merrors.FromError(err)
switch fromError.GetCode() {
case http.StatusNotFound:
logger.Debug().
Str("error_detail", fromError.GetDetail()).
Msg("no thumbnail found")
default:
logger.Warn().
Err(err).
Msg("Failed to execute")
}
} else {
logger.Debug().
Msg("")
}
return err
}
@@ -1,46 +0,0 @@
package decorators
import (
"context"
"go.opentelemetry.io/otel/trace"
thumbnailssvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/thumbnails/v0"
"go.opentelemetry.io/otel/attribute"
)
// NewTracing returns a service that instruments traces.
func NewTracing(next DecoratedService, tp trace.TracerProvider) DecoratedService {
return tracing{
Decorator: Decorator{next: next},
tp: tp,
}
}
type tracing struct {
Decorator
tp trace.TracerProvider
}
// GetThumbnail implements the ThumbnailServiceHandler interface.
func (t tracing) GetThumbnail(ctx context.Context, req *thumbnailssvc.GetThumbnailRequest, rsp *thumbnailssvc.GetThumbnailResponse) error {
var span trace.Span
if t.tp != nil {
tracer := t.tp.Tracer("thumbnails")
spanOpts := []trace.SpanStartOption{
trace.WithSpanKind(trace.SpanKindServer),
}
ctx, span = tracer.Start(ctx, "Thumbnails.GetThumbnail", spanOpts...)
defer span.End()
span.SetAttributes(
attribute.KeyValue{Key: "filepath", Value: attribute.StringValue(req.GetFilepath())},
attribute.KeyValue{Key: "thumbnail_type", Value: attribute.StringValue(req.GetThumbnailType().String())},
attribute.KeyValue{Key: "width", Value: attribute.IntValue(int(req.GetWidth()))},
attribute.KeyValue{Key: "height", Value: attribute.IntValue(int(req.GetHeight()))},
)
}
return t.next.GetThumbnail(ctx, req, rsp)
}
@@ -1,79 +0,0 @@
package svc
import (
"net/http"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/config"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/thumbnail/imgsource"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/thumbnail/storage"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
)
// Option defines a single option function.
type Option func(o *Options)
// Options defines the available options for this package.
type Options struct {
Logger log.Logger
Config *config.Config
Middleware []func(http.Handler) http.Handler
ThumbnailStorage storage.Storage
ImageSource imgsource.Source
CS3Source imgsource.Source
GatewaySelector pool.Selectable[gateway.GatewayAPIClient]
}
// newOptions initializes the available default options.
func newOptions(opts ...Option) Options {
opt := Options{}
for _, o := range opts {
o(&opt)
}
return opt
}
// Logger provides a function to set the logger option.
func Logger(val log.Logger) Option {
return func(o *Options) {
o.Logger = val
}
}
// Config provides a function to set the config option.
func Config(val *config.Config) Option {
return func(o *Options) {
o.Config = val
}
}
// ThumbnailStorage provides a function to set the thumbnail storage option.
func ThumbnailStorage(val storage.Storage) Option {
return func(o *Options) {
o.ThumbnailStorage = val
}
}
// ThumbnailSource provides a function to set the image source option.
func ThumbnailSource(val imgsource.Source) Option {
return func(o *Options) {
o.ImageSource = val
}
}
// CS3Source provides a function to set the CS3Source option
func CS3Source(val imgsource.Source) Option {
return func(o *Options) {
o.CS3Source = val
}
}
// GatewaySelector adds a grpc client selector for the gateway service
func GatewaySelector(val pool.Selectable[gateway.GatewayAPIClient]) Option {
return func(o *Options) {
o.GatewaySelector = val
}
}
@@ -1,345 +0,0 @@
package svc
import (
"context"
"net/http"
"net/url"
"path"
"strings"
"time"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
rpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1"
provider "github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1"
"github.com/golang-jwt/jwt/v5"
"github.com/pkg/errors"
merrors "go-micro.dev/v4/errors"
"google.golang.org/grpc/metadata"
revactx "github.com/opencloud-eu/reva/v2/pkg/ctx"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
"github.com/opencloud-eu/reva/v2/pkg/storagespace"
"github.com/opencloud-eu/reva/v2/pkg/utils"
"github.com/opencloud-eu/opencloud/pkg/log"
thumbnailssvc "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/services/thumbnails/v0"
terrors "github.com/opencloud-eu/opencloud/services/thumbnails/pkg/errors"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/preprocessor"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/service/grpc/v0/decorators"
tjwt "github.com/opencloud-eu/opencloud/services/thumbnails/pkg/service/jwt"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/thumbnail"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/thumbnail/imgsource"
)
// NewService returns a service implementation for Service.
func NewService(opts ...Option) decorators.DecoratedService {
options := newOptions(opts...)
logger := options.Logger
resolutions, err := thumbnail.ParseResolutions(options.Config.Thumbnail.Resolutions)
if err != nil {
logger.Fatal().Err(err).Msg("resolutions not configured correctly")
}
svc := Thumbnail{
serviceID: options.Config.GRPC.Namespace + "." + options.Config.Service.Name,
manager: thumbnail.NewSimpleManager(
resolutions,
options.ThumbnailStorage,
logger,
options.Config.Thumbnail.MaxInputWidth,
options.Config.Thumbnail.MaxInputHeight,
),
webdavSource: options.ImageSource,
cs3Source: options.CS3Source,
logger: logger,
selector: options.GatewaySelector,
preprocessorOpts: PreprocessorOpts{
TxtFontFileMap: options.Config.Thumbnail.FontMapFile,
MaxInputWidth: options.Config.Thumbnail.MaxInputWidth,
MaxInputHeight: options.Config.Thumbnail.MaxInputHeight,
},
dataEndpoint: options.Config.Thumbnail.DataEndpoint,
transferSecret: options.Config.Thumbnail.TransferSecret,
}
return svc
}
// Thumbnail implements the GRPC handler.
type Thumbnail struct {
serviceID string
dataEndpoint string
transferSecret string
manager thumbnail.Manager
webdavSource imgsource.Source
cs3Source imgsource.Source
logger log.Logger
selector pool.Selectable[gateway.GatewayAPIClient]
preprocessorOpts PreprocessorOpts
}
// PreprocessorOpts holds the options for the preprocessor
type PreprocessorOpts struct {
TxtFontFileMap string
MaxInputWidth int
MaxInputHeight int
}
// GetThumbnail retrieves a thumbnail for an image
func (g Thumbnail) GetThumbnail(ctx context.Context, req *thumbnailssvc.GetThumbnailRequest, rsp *thumbnailssvc.GetThumbnailResponse) error {
var err error
var key string
switch {
case req.GetWebdavSource() != nil:
key, err = g.handleWebdavSource(ctx, req)
case req.GetCs3Source() != nil:
key, err = g.handleCS3Source(ctx, req)
default:
g.logger.Error().Msg("no image source provided")
return merrors.BadRequest(g.serviceID, "image source is missing")
}
if err != nil {
return err
}
claims := tjwt.ThumbnailClaims{
Key: key,
RegisteredClaims: jwt.RegisteredClaims{
IssuedAt: jwt.NewNumericDate(time.Now()),
ExpiresAt: jwt.NewNumericDate(time.Now().Add(1 * time.Minute)),
},
}
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
transferToken, err := token.SignedString([]byte(g.transferSecret))
if err != nil {
g.logger.Error().
Err(err).
Msg("GetThumbnail: failed to sign token")
return merrors.InternalServerError(g.serviceID, "couldn't finish request")
}
rsp.DataEndpoint = g.dataEndpoint
rsp.TransferToken = transferToken
return nil
}
func (g Thumbnail) checkThumbnail(req *thumbnailssvc.GetThumbnailRequest, sRes *provider.StatResponse) (string, thumbnail.Request, error) {
tr := thumbnail.Request{}
if !sRes.GetInfo().GetPermissionSet().GetInitiateFileDownload() {
return "", tr, merrors.Forbidden(g.serviceID, "no download permission")
}
tType := thumbnail.GetExtForMime(sRes.GetInfo().GetMimeType())
if tType == "" {
tType = req.GetThumbnailType().String()
}
tr, err := thumbnail.PrepareRequest(int(req.GetWidth()), int(req.GetHeight()), tType, sRes.GetInfo().GetChecksum().GetSum(), req.GetProcessor())
if err != nil {
return "", tr, merrors.BadRequest(g.serviceID, "%s", err.Error())
}
if key, exists := g.manager.CheckThumbnail(tr); exists {
return key, tr, nil
}
return "", tr, nil
}
func (g Thumbnail) handleCS3Source(ctx context.Context, req *thumbnailssvc.GetThumbnailRequest) (string, error) {
src := req.GetCs3Source()
sRes, err := g.stat(src.GetPath(), src.GetAuthorization())
if err != nil {
return "", err
}
key, tr, err := g.checkThumbnail(req, sRes)
switch {
case err != nil:
return "", err
case key != "":
// we have matching thumbnail already, use that
return key, nil
}
ctx = imgsource.ContextSetAuthorization(ctx, src.GetAuthorization())
r, err := g.cs3Source.Get(ctx, src.GetPath())
switch {
case errors.Is(err, terrors.ErrImageTooLarge):
return "", merrors.Forbidden(g.serviceID, "%s", err.Error())
case err != nil:
return "", merrors.InternalServerError(g.serviceID, "could not get image from source: %s", err.Error())
}
defer r.Close()
ppOpts := map[string]any{
"fontFileMap": g.preprocessorOpts.TxtFontFileMap,
"maxInputWidth": g.preprocessorOpts.MaxInputWidth,
"maxInputHeight": g.preprocessorOpts.MaxInputHeight,
}
pp := preprocessor.ForType(sRes.GetInfo().GetMimeType(), ppOpts)
img, err := pp.Convert(r)
if errors.Is(err, terrors.ErrImageTooLarge) {
return "", merrors.Forbidden(g.serviceID, "%s", err.Error())
}
if err != nil {
g.logger.Error().Err(err).Msg("failed to convert image")
}
if img == nil || err != nil {
return "", merrors.NotFound(g.serviceID, "could not get image")
}
key, err = g.manager.Generate(tr, img)
if errors.Is(err, terrors.ErrImageTooLarge) {
return "", merrors.Forbidden(g.serviceID, "%s", err.Error())
}
return key, err
}
func (g Thumbnail) handleWebdavSource(ctx context.Context, req *thumbnailssvc.GetThumbnailRequest) (string, error) {
src := req.GetWebdavSource()
imgURL, err := url.Parse(src.GetUrl())
if err != nil {
return "", errors.Wrap(err, "source url is invalid")
}
var auth, statPath string
if src.GetIsPublicLink() {
q := imgURL.Query()
var rsp *gateway.AuthenticateResponse
client, err := g.selector.Next()
if err != nil {
return "", merrors.InternalServerError(g.serviceID, "could not select next gateway client: %s", err.Error())
}
if q.Get("signature") != "" && q.Get("expiration") != "" {
// Handle pre-signed public links
sig := q.Get("signature")
exp := q.Get("expiration")
rsp, err = client.Authenticate(ctx, &gateway.AuthenticateRequest{
Type: "publicshares",
ClientId: src.GetPublicLinkToken(),
ClientSecret: strings.Join([]string{"signature", sig, exp}, "|"),
})
} else {
rsp, err = client.Authenticate(ctx, &gateway.AuthenticateRequest{
Type: "publicshares",
ClientId: src.GetPublicLinkToken(),
// We pass an empty password because we expect non pre-signed public links
// to not be password protected
ClientSecret: "password|",
})
}
if err != nil {
return "", merrors.InternalServerError(g.serviceID, "could not authenticate: %s", err.Error())
}
auth = rsp.GetToken()
statPath = path.Join("/public", src.GetPublicLinkToken(), req.GetFilepath())
} else {
auth = src.GetRevaAuthorization()
statPath = req.GetFilepath()
}
sRes, err := g.stat(statPath, auth)
if err != nil {
return "", err
}
key, tr, err := g.checkThumbnail(req, sRes)
switch {
case err != nil:
return "", err
case key != "":
// we have matching thumbnail already, use that
return key, nil
}
if src.GetWebdavAuthorization() != "" {
ctx = imgsource.ContextSetAuthorization(ctx, src.GetWebdavAuthorization())
}
// add signature and expiration to webdav url
signature, expiration := imgURL.Query().Get("signature"), imgURL.Query().Get("expiration")
params := url.Values{}
params.Add("signature", signature)
params.Add("expiration", expiration)
imgURL.RawQuery = params.Encode()
r, err := g.webdavSource.Get(ctx, imgURL.String())
switch {
case errors.Is(err, terrors.ErrImageTooLarge):
return "", merrors.Forbidden(g.serviceID, "%s", err.Error())
case err != nil:
return "", merrors.InternalServerError(g.serviceID, "could not get image from source: %s", err.Error())
}
defer r.Close()
ppOpts := map[string]any{
"fontFileMap": g.preprocessorOpts.TxtFontFileMap,
"maxInputWidth": g.preprocessorOpts.MaxInputWidth,
"maxInputHeight": g.preprocessorOpts.MaxInputHeight,
}
pp := preprocessor.ForType(sRes.GetInfo().GetMimeType(), ppOpts)
img, err := pp.Convert(r)
if errors.Is(err, terrors.ErrImageTooLarge) {
return "", merrors.Forbidden(g.serviceID, "%s", err.Error())
}
if img == nil || err != nil {
return "", merrors.NotFound(g.serviceID, "could not get image")
}
key, err = g.manager.Generate(tr, img)
if errors.Is(err, terrors.ErrImageTooLarge) {
return "", merrors.Forbidden(g.serviceID, "%s", err.Error())
}
return key, err
}
func (g Thumbnail) stat(path, auth string) (*provider.StatResponse, error) {
ctx := metadata.AppendToOutgoingContext(context.Background(), revactx.TokenHeader, auth)
ref, err := storagespace.ParseReference(path)
if err != nil {
// If the path is not a spaces reference try to handle it like a plain
// path reference.
ref = provider.Reference{
Path: path,
}
}
client, err := g.selector.Next()
if err != nil {
return nil, merrors.InternalServerError(g.serviceID, "could not select next gateway client: %s", err.Error())
}
req := &provider.StatRequest{Ref: &ref}
rsp, err := client.Stat(ctx, req)
if err != nil {
g.logger.Error().Err(err).Str("path", path).Msg("could not stat file")
return nil, merrors.InternalServerError(g.serviceID, "could not stat file: %s", err.Error())
}
if rsp.GetStatus().GetCode() != rpc.Code_CODE_OK {
switch rsp.GetStatus().GetCode() {
case rpc.Code_CODE_NOT_FOUND:
return nil, merrors.NotFound(g.serviceID, "could not stat file: %s", rsp.GetStatus().GetMessage())
default:
g.logger.Error().Str("status_message", rsp.GetStatus().GetMessage()).Str("path", path).Msg("could not stat file")
return nil, merrors.InternalServerError(g.serviceID, "could not stat file: %s", rsp.GetStatus().GetMessage())
}
}
if rsp.GetInfo().GetType() != provider.ResourceType_RESOURCE_TYPE_FILE {
return nil, merrors.BadRequest(g.serviceID, "Unsupported file type")
}
if utils.ReadPlainFromOpaque(rsp.GetInfo().GetOpaque(), "status") == "processing" {
return nil, &merrors.Error{
Id: g.serviceID,
Code: http.StatusTooEarly,
Detail: "File Processing",
Status: http.StatusText(http.StatusTooEarly),
}
}
if rsp.GetInfo().GetChecksum().GetSum() == "" {
g.logger.Error().Msg("resource info is missing checksum")
return nil, merrors.NotFound(g.serviceID, "resource info is missing a checksum")
}
if !thumbnail.IsMimeTypeSupported(rsp.GetInfo().GetMimeType()) {
return nil, merrors.NotFound(g.serviceID, "Unsupported file type")
}
return rsp, nil
}
@@ -1,30 +0,0 @@
package svc
import (
"net/http"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/metrics"
)
// NewInstrument returns a service that instruments metrics.
func NewInstrument(next Service, metrics *metrics.Metrics) Service {
return instrument{
next: next,
metrics: metrics,
}
}
type instrument struct {
next Service
metrics *metrics.Metrics
}
// ServeHTTP implements the Service interface.
func (i instrument) ServeHTTP(w http.ResponseWriter, r *http.Request) {
i.next.ServeHTTP(w, r)
}
// GetThumbnail implements the Service interface.
func (i instrument) GetThumbnail(w http.ResponseWriter, r *http.Request) {
i.next.GetThumbnail(w, r)
}
@@ -1,30 +0,0 @@
package svc
import (
"net/http"
"github.com/opencloud-eu/opencloud/pkg/log"
)
// NewLogging returns a service that logs messages.
func NewLogging(next Service, logger log.Logger) Service {
return logging{
next: next,
logger: logger,
}
}
type logging struct {
next Service
logger log.Logger
}
// ServeHTTP implements the Service interface.
func (l logging) ServeHTTP(w http.ResponseWriter, r *http.Request) {
l.next.ServeHTTP(w, r)
}
// GetThumbnail implements the Service interface.
func (l logging) GetThumbnail(w http.ResponseWriter, r *http.Request) {
l.next.GetThumbnail(w, r)
}
@@ -5,7 +5,6 @@ import (
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/config"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/thumbnail/storage"
"go.opentelemetry.io/otel/trace"
)
@@ -14,11 +13,10 @@ type Option func(o *Options)
// Options defines the available options for this package.
type Options struct {
Logger log.Logger
Config *config.Config
Middleware []func(http.Handler) http.Handler
ThumbnailStorage storage.Storage
TraceProvider trace.TracerProvider
Logger log.Logger
Config *config.Config
Middleware []func(http.Handler) http.Handler
TraceProvider trace.TracerProvider
}
// newOptions initializes the available default options.
@@ -52,10 +50,3 @@ func Middleware(val ...func(http.Handler) http.Handler) Option {
o.Middleware = val
}
}
// ThumbnailStorage provides a function to set the ThumbnailStorage option.
func ThumbnailStorage(storage storage.Storage) Option {
return func(o *Options) {
o.ThumbnailStorage = storage
}
}
@@ -0,0 +1,37 @@
package svc
import (
"image"
"image/color"
"image/draw"
)
// palettedAtOrigin re-pallettes img using Floyd-Steinberg dithering onto a new
// palette whose bounds are anchored at the origin. The source's own (possibly
// offset) bounds are honored so the pixel content lands in the right place, but
// the result always starts at (0,0). This is required for gif frames: after a
// resize the frame rectangle inherits the source frame's offset, and a frame
// whose rect extends past the gif logical screen makes gif.EncodeAll fail with
// "image block is out of bounds".
func palettedAtOrigin(img image.Image, p color.Palette) *image.Paletted {
b := img.Bounds()
pm := image.NewPaletted(image.Rect(0, 0, b.Dx(), b.Dy()), p)
draw.FloydSteinberg.Draw(pm, pm.Bounds(), shiftedImage(img, b.Min), image.Point{})
return pm
}
// shiftedImage wraps img so its bounds are shifted by off, letting a draw target
// anchored at the origin receive an image whose content is defined relative to a
// different origin.
type shiftedImageWrapper struct {
img image.Image
off image.Point
}
func (s shiftedImageWrapper) Bounds() image.Rectangle { return s.img.Bounds().Add(s.off) }
func (s shiftedImageWrapper) At(x, y int) color.Color { return s.img.At(x-s.off.X, y-s.off.Y) }
func (s shiftedImageWrapper) ColorModel() color.Model { return s.img.ColorModel() }
func shiftedImage(img image.Image, off image.Point) image.Image {
return shiftedImageWrapper{img: img, off: off}
}
@@ -0,0 +1,339 @@
package svc
import (
"bytes"
"fmt"
"image"
"image/gif"
"io"
"net/http"
"strconv"
"strings"
"github.com/go-chi/chi/v5"
)
// gifMagic is the leading signature of a GIF file (GIF87a or GIF89a). It guards
// against gif.DecodeAll panicking on non-GIF input.
var gifMagic = []byte("GIF8")
// Operations understood by the push endpoint. They map to imagor's resize/crop
// modes so a real imagor deployment can replace this service.
const (
OpFill = "fill"
OpFitIn = "fit-in"
OpStretch = "stretch"
)
// errInvalid mirrors imagor's ErrInvalid: a syntactically invalid request.
var errInvalid = fmt.Errorf("invalid")
// pushHandler handles POST requests for the imagor-like push-based thumbnail endpoint.
// Path patterns (operation and format are both optional; absent operation is the default
// center-crop fill, absent format keeps the input's own format):
// - POST /unsafe(/fit-in|/stretch)/{W}x{H}(/filters:{filters})
//
// The filters segment is captured whole and parsed by outputFormatFromFilters; only the
// format filter is meaningful to this executor, other filters are ignored. When no format
// filter is present the input's own format is preserved (detected via imaging), matching
// imagor. It returns only imagor's defined status codes: 400 for invalid requests or a
// file exceeding the max size, 422 for an image exceeding the max resolution.
func (s Thumbnails) pushHandler(w http.ResponseWriter, r *http.Request) {
ok, retryAfter := s.limiter.acquire()
if !ok {
writeTooManyRequests(w, retryAfter)
return
}
defer s.limiter.release()
width, err := parseDim(r, "width")
if err != nil {
writeInvalid(w, "invalid width")
return
}
height, err := parseDim(r, "height")
if err != nil {
writeInvalid(w, "invalid height")
return
}
operation := chi.URLParam(r, "operation")
if operation == "" {
operation = OpFill
}
switch operation {
case OpFill, OpFitIn, OpStretch:
default:
writeInvalid(w, fmt.Sprintf("unsupported operation: %s", operation))
return
}
if err := r.ParseMultipartForm(32 << 20); err != nil {
writeInvalid(w, "failed to parse multipart form")
return
}
file, _, err := r.FormFile("image")
if err != nil {
writeInvalid(w, "missing image field in form data")
return
}
defer file.Close()
imgData, err := io.ReadAll(file)
if err != nil {
writeInvalid(w, "failed to read image")
return
}
filtersSegment := chi.URLParam(r, "filters")
ext, hasFormat, err := outputFormatFromFilters(filtersSegment)
if err != nil {
writeInvalid(w, err.Error())
return
}
noUpscale := filterPresent(filtersSegment, "no_upscale")
if !hasFormat {
// No format filter: preserve the input's own format (imagor default),
// detected via imaging. Only runs on this path, so the common webdav
// request (which always sends a format) pays no decode cost.
ext = inputFormatViaImaging(imgData)
}
// imagor ErrMaxResolutionExceeded: the declared input exceeds the configured
// maximum width or height. Checked against the image header BEFORE any
// decoding so a dimension bomb (a tiny file declaring huge dimensions) is
// rejected without allocating the pixel buffer. webdav translates the
// resulting 422 into its legacy 403 response.
if s.maxWidth > 0 || s.maxHeight > 0 {
cfg, _, err := image.DecodeConfig(bytes.NewReader(imgData))
if err == nil {
if (s.maxWidth > 0 && cfg.Width > s.maxWidth) || (s.maxHeight > 0 && cfg.Height > s.maxHeight) {
writeMaxResolution(w)
return
}
}
}
// The box and operation are chosen by webdav (the sizing brain); this service
// is a dumb, imagor-like executor that just fits within the given box.
processed, err := processImage(bytes.NewReader(imgData), width, height, operation, noUpscale)
if err != nil {
writeInvalid(w, "failed to process image")
return
}
var (
buf bytes.Buffer
contentType string
encErr error
)
switch ext {
case "jpg":
contentType = "image/jpeg"
encErr = encodeJPEG(&buf, processed)
case "png":
contentType = "image/png"
encErr = encodePNG(&buf, processed)
default:
contentType = "image/gif"
g, ok := processed.(*gif.GIF)
if !ok {
writeInvalid(w, "processed image is not a gif")
return
}
encErr = gif.EncodeAll(&buf, g)
}
if encErr != nil {
w.WriteHeader(http.StatusInternalServerError)
fmt.Fprintln(w, "failed to encode image")
return
}
w.Header().Set("Content-Type", contentType)
w.WriteHeader(http.StatusOK)
buf.WriteTo(w)
}
func parseDim(r *http.Request, name string) (int, error) {
v := chi.URLParam(r, name)
n, err := strconv.Atoi(v)
if err != nil || n <= 0 {
return 0, fmt.Errorf("invalid %s: %s", name, v)
}
return n, nil
}
// writeInvalid responds with imagor's ErrInvalid status (400 Bad Request).
func writeInvalid(w http.ResponseWriter, msg string) {
w.WriteHeader(http.StatusBadRequest)
fmt.Fprintln(w, msg)
}
// writeMaxResolution responds with imagor's ErrMaxResolutionExceeded status
// (422 Unprocessable Entity).
func writeMaxResolution(w http.ResponseWriter) {
w.WriteHeader(http.StatusUnprocessableEntity)
fmt.Fprintln(w, "maximum resolution exceeded")
}
// concurrencyLimiter bounds the number of requests processed in parallel. It is
// nil when no limit is configured (THUMBNAILS_MAX_CONCURRENT_REQUESTS = 0).
type concurrencyLimiter struct {
tokens chan struct{}
}
func newConcurrencyLimiter(limit int) *concurrencyLimiter {
if limit <= 0 {
return nil
}
tokens := make(chan struct{}, limit)
for i := 0; i < limit; i++ {
tokens <- struct{}{}
}
return &concurrencyLimiter{tokens: tokens}
}
// acquire tries to take a slot without blocking. It reports whether the request
// may proceed and, when not, how many seconds to wait before retrying.
func (l *concurrencyLimiter) acquire() (bool, int) {
if l == nil {
return true, 0
}
select {
case <-l.tokens:
return true, 0
default:
return false, 1
}
}
func (l *concurrencyLimiter) release() {
if l == nil {
return
}
l.tokens <- struct{}{}
}
// writeTooManyRequests responds with 429 and a short Retry-After header.
func writeTooManyRequests(w http.ResponseWriter, retryAfter int) {
w.Header().Set("Retry-After", strconv.Itoa(retryAfter))
w.WriteHeader(http.StatusTooManyRequests)
fmt.Fprintln(w, "too many concurrent requests")
}
// mapFormatToExt maps imagor format names to the internal extension used by encoders.
func mapFormatToExt(format string) string {
switch strings.ToLower(format) {
case "jpeg":
return "jpg"
default:
return strings.ToLower(format)
}
}
// outputFormatFromFilters parses an imagor filters segment (e.g. "format(jpeg)") and
// reports the requested output extension. Only the format filter is meaningful to this
// executor; other filters are ignored because the operation already governs
// sizing/upscaling. It returns hasFormat=false when no format filter is present, in
// which case the caller falls back to inputFormatViaImaging. An explicit but
// unsupported format is an error.
func outputFormatFromFilters(segment string) (ext string, hasFormat bool, err error) {
for _, token := range strings.Split(segment, ":") {
name, args, ok := splitFilter(token)
if !ok || name != "format" {
continue
}
mapped := mapFormatToExt(args)
switch mapped {
case "jpg", "png", "gif":
return mapped, true, nil
default:
return "", false, fmt.Errorf("unsupported output format: %s", args)
}
}
return "", false, nil
}
// inputFormatViaImaging detects the uploaded image's format with the stdlib
// decoder registry (image.DecodeConfig reads only the header) and maps it to our
// output extension. jpeg/png/gif are kept; anything else (webp, tiff, bmp, or an
// undecodable body) falls back to jpeg, mirroring imagor's unsavable-source
// default.
func inputFormatViaImaging(imgData []byte) string {
_, format, err := image.DecodeConfig(bytes.NewReader(imgData))
if err != nil {
return "jpg"
}
switch format {
case "png":
return "png"
case "gif":
return "gif"
default: // jpeg, webp, tiff, bmp, ...
return "jpg"
}
}
// splitFilter splits a "name(args)" token into name and args. It returns ok=false for
// tokens that are not in parenthesized form.
func splitFilter(token string) (name, args string, ok bool) {
i := strings.IndexByte(token, '(')
if i < 0 || !strings.HasSuffix(token, ")") {
return "", "", false
}
return token[:i], token[i+1 : len(token)-1], true
}
// filterPresent reports whether a named filter occurs in an imagor filters
// segment (e.g. "no_upscale():format(jpeg)").
func filterPresent(segment, name string) bool {
for _, token := range strings.Split(segment, ":") {
if n, _, ok := splitFilter(token); ok && n == name {
return true
}
}
return false
}
// isGifReader reports whether the reader holds GIF data, based on the GIF magic
// bytes. It peeks at the first 4 bytes and rewinds the stream so it can be read
// again. This guard is required because gif.DecodeAll panics (nil dereference)
// when handed a non-GIF image instead of returning an error.
func isGifReader(r io.Reader) bool {
readSeeker, ok := r.(io.ReadSeeker)
if !ok {
return false
}
saved, err := readSeeker.Seek(0, io.SeekCurrent)
if err != nil {
return false
}
buf := make([]byte, 4)
if _, err := io.ReadFull(readSeeker, buf); err != nil {
return false
}
_, _ = readSeeker.Seek(saved, io.SeekStart)
return bytes.HasPrefix(buf, gifMagic)
}
// Backend-specific function set in init() by push_imaging.go or push_vips.go.
// For gif input it returns a *gif.GIF with every frame resized (preserving
// animation); for other inputs it returns a single image.Image. The operation
// selects the resize/crop mode (fill, fit-in, stretch). noUpscale mirrors
// imagor's no_upscale() filter: when set, fill and fit-in must not enlarge the
// source (stretch is exact and ignores it).
var processImage func(io.Reader, int, int, string, bool) (any, error)
// encodeJPEG and encodePNG write the processed image to w in the requested
// format. Set by init() in push_imaging.go / push_vips.go. The vips backend
// encodes via libvips (ExportJpeg/ExportPng) so its output matches the legacy
// pipeline (progressive JPEG, quality 80); the imaging backend uses the stdlib
// encoders. GIF encoding stays in the common handler because both backends
// return a *gif.GIF for animated input.
var (
encodeJPEG func(w io.Writer, processed any) error
encodePNG func(w io.Writer, processed any) error
)
@@ -0,0 +1,155 @@
//go:build !enable_vips
package svc
import (
"fmt"
"image"
"image/draw"
"image/gif"
"image/jpeg"
"image/png"
"io"
"github.com/kovidgoyal/imaging"
)
func init() {
processImage = processImageImaging
encodeJPEG = encodeJPEGImaging
encodePNG = encodePNGImaging
}
// processImageImaging resizes the input using the imaging backend. The operation
// selects the resize/crop mode: fill (center-crop to the box, upscaling by default
// like real imagor's default resize), fit-in (fit within the box without cropping,
// upscaling small sources by default like real imagor), or stretch (resize to the
// exact box). noUpscale caps the
// default fill at the source size, mirroring imagor's no_upscale() filter.
func processImageImaging(r io.Reader, width, height int, operation string, noUpscale bool) (any, error) {
if isGifReader(r) {
g, err := gif.DecodeAll(r)
if err == nil && len(g.Image) > 0 {
return resizeGIF(g, width, height, operation, noUpscale), nil
}
}
img, err := imaging.Decode(r, imaging.AutoOrientation(true))
if err != nil {
return nil, err
}
srcBounds := img.Bounds()
switch operation {
case OpStretch:
return imaging.Resize(img, width, height, imaging.Lanczos), nil
case OpFitIn:
// Like real imagor, fit-in fits within the box preserving aspect and
// upscales small sources by default; no_upscale() caps it at the source
// size. imaging.Fit never enlarges, so a small source is scaled up to the
// fitted dimensions explicitly.
if noUpscale && srcBounds.Dx() <= width && srcBounds.Dy() <= height {
return img, nil
}
if srcBounds.Dx() <= width && srcBounds.Dy() <= height {
srcAR := float64(srcBounds.Dx()) / float64(srcBounds.Dy())
boxAR := float64(width) / float64(height)
if srcAR > boxAR {
return imaging.Resize(img, width, int(float64(width)/srcAR), imaging.Lanczos), nil
}
return imaging.Resize(img, int(float64(height)*srcAR), height, imaging.Lanczos), nil
}
return imaging.Fit(img, width, height, imaging.Lanczos), nil
default: // OpFill
if noUpscale && srcBounds.Dx() <= width && srcBounds.Dy() <= height {
// imagor no_upscale(): never enlarge the source.
return img, nil
}
return imaging.Thumbnail(img, width, height, imaging.Lanczos), nil
}
}
// resizeGIF resizes every frame of an animated gif while preserving the
// animation. It composites each frame onto a running canvas honoring the gif
// disposal method, resizes with the requested processor, and re-pallettes the
// result using Floyd-Steinberg dithering. Code adapted from
// https://github.com/willnorris/gifresize. noUpscale caps the default fill at
// the source size, mirroring imagor's no_upscale() filter.
func resizeGIF(m *gif.GIF, width, height int, operation string, noUpscale bool) *gif.GIF {
srcX, srcY := m.Config.Width, m.Config.Height
b := image.Rect(0, 0, srcX, srcY)
tmp := image.NewRGBA(b)
for i, frame := range m.Image {
frameBounds := frame.Bounds()
prev := tmp
draw.Draw(tmp, frameBounds, frame, frameBounds.Min, draw.Over)
var processed image.Image
switch operation {
case OpStretch:
processed = imaging.Resize(tmp, width, height, imaging.Lanczos)
case OpFitIn:
if noUpscale && srcX <= width && srcY <= height {
processed = tmp
} else if srcX <= width && srcY <= height {
srcAR := float64(srcX) / float64(srcY)
boxAR := float64(width) / float64(height)
if srcAR > boxAR {
processed = imaging.Resize(tmp, width, int(float64(width)/srcAR), imaging.Lanczos)
} else {
processed = imaging.Resize(tmp, int(float64(height)*srcAR), height, imaging.Lanczos)
}
} else {
processed = imaging.Fit(tmp, width, height, imaging.Lanczos)
}
default: // OpFill
if noUpscale && srcX <= width && srcY <= height {
processed = tmp
} else {
processed = imaging.Fill(tmp, width, height, imaging.Center, imaging.Lanczos)
}
}
m.Image[i] = palettedAtOrigin(processed, frame.Palette)
switch m.Disposal[i] {
case gif.DisposalBackground:
tmp = image.NewRGBA(b)
case gif.DisposalPrevious:
tmp = prev
}
}
// stretch resizes every frame to exactly width x height, so the logical
// screen must always match the box even when that is larger than the source
// (webdav sends no_upscale(), which would otherwise skip the update and leave
// frames extending past the screen, making gif.EncodeAll fail with "image
// block is out of bounds"). fill/fit-in keep the source size when noUpscale
// caps them at the source, so they only grow the screen when actually resized.
if operation == OpStretch || !noUpscale || srcX > width || srcY > height {
m.Config.Width = width
m.Config.Height = height
}
return m
}
// encodeJPEGImaging encodes the processed image as JPEG using the stdlib encoder.
func encodeJPEGImaging(w io.Writer, processed any) error {
img, ok := processed.(image.Image)
if !ok {
return fmt.Errorf("cannot encode %T as jpeg", processed)
}
return jpeg.Encode(w, img, nil)
}
// encodePNGImaging encodes the processed image as PNG using the stdlib encoder.
func encodePNGImaging(w io.Writer, processed any) error {
img, ok := processed.(image.Image)
if !ok {
return fmt.Errorf("cannot encode %T as png", processed)
}
return png.Encode(w, img)
}
@@ -0,0 +1,996 @@
package svc
import (
"bytes"
"image"
"image/color"
"image/gif"
"image/jpeg"
"image/png"
"mime/multipart"
"net/http"
"net/http/httptest"
"testing"
"github.com/go-chi/chi/v5"
"golang.org/x/image/bmp"
)
func createTestImage(width, height int) []byte {
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
img.Set(x, y, color.RGBA{R: uint8(x % 256), G: uint8(y % 256), B: 128, A: 255})
}
}
var buf bytes.Buffer
if err := png.Encode(&buf, img); err != nil {
panic(err)
}
return buf.Bytes()
}
func createMultipartBody(imgBytes []byte) (*bytes.Buffer, string) {
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
part, err := writer.CreateFormFile("image", "test.png")
if err != nil {
panic(err)
}
if _, err := part.Write(imgBytes); err != nil {
panic(err)
}
if err := writer.Close(); err != nil {
panic(err)
}
return body, writer.FormDataContentType()
}
func newTestMux() *chi.Mux {
return newTestMuxWithLimits(0, 0)
}
// newTestMuxWithLimits builds the route table with explicit max width/height so
// tests can exercise the ErrMaxResolutionExceeded (422) path. A limit of 0
// disables the corresponding bound.
func newTestMuxWithLimits(maxWidth, maxHeight int) *chi.Mux {
mux := chi.NewRouter()
h := Thumbnails{maxWidth: maxWidth, maxHeight: maxHeight}.pushHandler
mux.Post("/unsafe/{operation}/{width}x{height}/filters:{filters}/", h)
mux.Post("/unsafe/{operation}/{width}x{height}/filters:{filters}", h)
mux.Post("/unsafe/{width}x{height}/filters:{filters}/", h)
mux.Post("/unsafe/{width}x{height}/filters:{filters}", h)
return mux
}
// createTestGIF returns an animated gif with the given number of frames, each
// frame a solid color so that animation preservation can be verified.
func createTestGIF(width, height, frames int) []byte {
pal := color.Palette{color.RGBA{255, 0, 0, 255}, color.RGBA{0, 255, 0, 255}, color.RGBA{0, 0, 255, 255}}
g := &gif.GIF{}
for i := 0; i < frames; i++ {
frame := image.NewPaletted(image.Rect(0, 0, width, height), pal)
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
frame.SetColorIndex(x, y, uint8(i%len(pal)))
}
}
g.Image = append(g.Image, frame)
g.Delay = append(g.Delay, 10)
}
var buf bytes.Buffer
if err := gif.EncodeAll(&buf, g); err != nil {
panic(err)
}
return buf.Bytes()
}
func TestPushEndpoint_FitIn_JPEG(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(200, 200)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/100x100/filters:format(jpeg)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
if ct := rec.Header().Get("Content-Type"); ct != "image/jpeg" {
t.Errorf("expected Content-Type image/jpeg, got %s", ct)
}
if rec.Body.Len() == 0 {
t.Error("expected non-empty response body")
}
}
func TestPushEndpoint_FitIn_PNG(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(200, 200)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/100x100/filters:format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
if ct := rec.Header().Get("Content-Type"); ct != "image/png" {
t.Errorf("expected Content-Type image/png, got %s", ct)
}
if rec.Body.Len() == 0 {
t.Error("expected non-empty response body")
}
}
func TestPushEndpoint_FitIn_GIF(t *testing.T) {
mux := newTestMux()
imgBytes := createTestGIF(200, 200, 3)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/100x100/filters:format(gif)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
if ct := rec.Header().Get("Content-Type"); ct != "image/gif" {
t.Errorf("expected Content-Type image/gif, got %s", ct)
}
if rec.Body.Len() == 0 {
t.Error("expected non-empty response body")
}
out, err := gif.DecodeAll(rec.Body)
if err != nil {
t.Fatalf("failed to decode response gif: %v", err)
}
if len(out.Image) != 3 {
t.Errorf("expected 3 frames preserved, got %d", len(out.Image))
}
}
func TestPushEndpoint_FitIn_GIFPreservesAnimation(t *testing.T) {
mux := newTestMux()
imgBytes := createTestGIF(200, 100, 4)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/50x50/filters:format(gif)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
out, err := gif.DecodeAll(rec.Body)
if err != nil {
t.Fatalf("failed to decode response gif: %v", err)
}
if len(out.Image) != 4 {
t.Errorf("expected 4 frames preserved, got %d", len(out.Image))
}
// fit-in of a 200x100 source into a 50x50 box downscales (no upscaling needed),
// so every frame must stay within the box.
for i, frame := range out.Image {
b := frame.Bounds()
if b.Dx() > 50 || b.Dy() > 50 {
t.Errorf("frame %d exceeds requested dimensions: %dx%d", i, b.Dx(), b.Dy())
}
}
}
func TestPushEndpoint_Fill_GIF(t *testing.T) {
mux := newTestMux()
imgBytes := createTestGIF(200, 100, 3)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/64x64/filters:format(gif)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
out, err := gif.DecodeAll(rec.Body)
if err != nil {
t.Fatalf("failed to decode response gif: %v", err)
}
if len(out.Image) != 3 {
t.Errorf("expected 3 frames preserved, got %d", len(out.Image))
}
for i, frame := range out.Image {
b := frame.Bounds()
if b.Dx() != 64 || b.Dy() != 64 {
t.Errorf("fill should produce exact dimensions, frame %d is %dx%d", i, b.Dx(), b.Dy())
}
}
}
// TestPushEndpoint_Stretch_GIFUpscaleNoUpscaleFilter pins the fix for stretching a
// gif to a box larger than the source while the no_upscale() filter is set (which
// is exactly what webdav sends). The resized frames are larger than the original
// logical screen, so without updating m.Config the frame rects extend past the
// screen and gif.EncodeAll fails with "image block is out of bounds" (surfaced as
// HTTP 500). The endpoint must return 200 with every frame exactly the requested
// size.
func TestPushEndpoint_Stretch_GIFUpscaleNoUpscaleFilter(t *testing.T) {
mux := newTestMux()
imgBytes := createTestGIF(100, 80, 3)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/stretch/200x160/filters:no_upscale():format(gif)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
out, err := gif.DecodeAll(rec.Body)
if err != nil {
t.Fatalf("failed to decode response gif: %v", err)
}
if len(out.Image) != 3 {
t.Fatalf("expected 3 frames preserved, got %d", len(out.Image))
}
for i, frame := range out.Image {
b := frame.Bounds()
if b.Dx() != 200 || b.Dy() != 160 {
t.Errorf("stretch should produce exactly 200x160, frame %d is %dx%d", i, b.Dx(), b.Dy())
}
}
}
func TestPushEndpoint_FitIn_AspectRatio(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(400, 200)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/100x100/filters:format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
img, err := png.Decode(rec.Body)
if err != nil {
t.Fatalf("failed to decode response image: %v", err)
}
bounds := img.Bounds()
if bounds.Dx() > 100 || bounds.Dy() > 100 {
t.Errorf("fit-in should not exceed requested dimensions, got %dx%d", bounds.Dx(), bounds.Dy())
}
}
// TestPushEndpoint_FitIn_UpscalesByDefault documents that, like real imagor,
// fit-in upscales a small source to fit the box when no_upscale() is absent. A
// 50x50 source requested at 200x200 must come back as exactly 200x200.
func TestPushEndpoint_FitIn_UpscalesByDefault(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(50, 50)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/200x200/filters:format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
img, err := png.Decode(rec.Body)
if err != nil {
t.Fatalf("failed to decode response image: %v", err)
}
bounds := img.Bounds()
if bounds.Dx() != 200 || bounds.Dy() != 200 {
t.Errorf("fit-in should upscale 50x50 to fit a 200x200 box as 200x200, got %dx%d", bounds.Dx(), bounds.Dy())
}
}
// TestPushEndpoint_FitIn_NoUpscaleFilter pins imagor's no_upscale() filter on
// fit-in: with it, a small source must not be enlarged. A 50x50 source requested
// at 200x200 comes back as 50x50 instead of 200x200.
func TestPushEndpoint_FitIn_NoUpscaleFilter(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(50, 50)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/200x200/filters:no_upscale():format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
img, err := png.Decode(rec.Body)
if err != nil {
t.Fatalf("failed to decode response image: %v", err)
}
bounds := img.Bounds()
if bounds.Dx() > 50 || bounds.Dy() > 50 {
t.Errorf("fit-in with no_upscale should not enlarge the image, got %dx%d", bounds.Dx(), bounds.Dy())
}
}
// TestPushEndpoint_Default_SquareFill pins the default processor's outcome:
// center-crop to fill the requested box exactly (the legacy "thumbnail"
// processor behavior). A 200x100 source requested at 100x100 must come back as
// exactly 100x100 — center-cropped, NOT letterboxed to 100x50. This is the route
// the webdav service uses by default (no fit-in segment).
func TestPushEndpoint_Default_SquareFill(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(200, 100)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/100x100/filters:format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
img, err := png.Decode(rec.Body)
if err != nil {
t.Fatalf("failed to decode response image: %v", err)
}
bounds := img.Bounds()
if bounds.Dx() != 100 || bounds.Dy() != 100 {
t.Errorf("default should center-crop 200x100 into a 100x100 box as 100x100, got %dx%d", bounds.Dx(), bounds.Dy())
}
}
// TestPushEndpoint_Default_Upscales documents that the default (fill) processor
// upscales small sources to fill the box exactly, matching the legacy
// "thumbnail" processor. A 50x50 source requested at 200x200 must come back as
// exactly 200x200.
func TestPushEndpoint_Default_Upscales(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(50, 50)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/200x200/filters:format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
img, err := png.Decode(rec.Body)
if err != nil {
t.Fatalf("failed to decode response image: %v", err)
}
bounds := img.Bounds()
if bounds.Dx() != 200 || bounds.Dy() != 200 {
t.Errorf("default should upscale 50x50 to fill a 200x200 box as 200x200, got %dx%d", bounds.Dx(), bounds.Dy())
}
}
// TestPushEndpoint_Default_NoUpscaleFilter pins imagor's no_upscale() filter:
// with it, the default resize must not enlarge a small source. A 50x50 source
// requested at 200x200 comes back as 50x50 instead of 200x200.
func TestPushEndpoint_Default_NoUpscaleFilter(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(50, 50)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/200x200/filters:no_upscale():format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
img, err := png.Decode(rec.Body)
if err != nil {
t.Fatalf("failed to decode response image: %v", err)
}
bounds := img.Bounds()
if bounds.Dx() != 50 || bounds.Dy() != 50 {
t.Errorf("no_upscale should keep the source at 50x50, got %dx%d", bounds.Dx(), bounds.Dy())
}
}
// TestPushEndpoint_FitIn_LetterboxOutcome pins the opt-in "fit" processor's
// outcome: aspect-preserving fit within the box. A 200x100 source requested at
// 100x100 must come back as exactly 100x50 — centered/letterboxed in the box,
// NOT center-cropped to 100x100.
func TestPushEndpoint_FitIn_LetterboxOutcome(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(200, 100)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/100x100/filters:format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
img, err := png.Decode(rec.Body)
if err != nil {
t.Fatalf("failed to decode response image: %v", err)
}
bounds := img.Bounds()
if bounds.Dx() != 100 || bounds.Dy() != 50 {
t.Errorf("fit-in should letterbox 200x100 into a 100x100 box as 100x50, got %dx%d", bounds.Dx(), bounds.Dy())
}
}
func TestPushEndpoint_Fill_JPEG(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(200, 200)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/100x100/filters:format(jpeg)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
if ct := rec.Header().Get("Content-Type"); ct != "image/jpeg" {
t.Errorf("expected Content-Type image/jpeg, got %s", ct)
}
if rec.Body.Len() == 0 {
t.Error("expected non-empty response body")
}
}
func TestPushEndpoint_Fill_ExactDimensions(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(400, 200)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/100x100/filters:format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
img, err := png.Decode(rec.Body)
if err != nil {
t.Fatalf("failed to decode response image: %v", err)
}
bounds := img.Bounds()
if bounds.Dx() != 100 || bounds.Dy() != 100 {
t.Errorf("fill should produce exact dimensions, got %dx%d", bounds.Dx(), bounds.Dy())
}
}
func TestPushEndpoint_BadDimensions(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(100, 100)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/abcxdef/filters:format(jpeg)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusBadRequest {
t.Errorf("expected status 400, got %d", rec.Code)
}
}
func TestPushEndpoint_ZeroDimensions(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(100, 100)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/0x0/filters:format(jpeg)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusBadRequest {
t.Errorf("expected status 400 for zero dimensions, got %d", rec.Code)
}
}
func TestPushEndpoint_UnsupportedFormat(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(100, 100)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/50x50/filters:format(webp)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusBadRequest {
t.Errorf("expected status 400 for unsupported format, got %d", rec.Code)
}
}
func TestPushEndpoint_InvalidImageData(t *testing.T) {
mux := newTestMux()
body, contentType := createMultipartBody([]byte("not a valid image"))
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/50x50/filters:format(jpeg)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusBadRequest {
t.Errorf("expected status 400 for invalid image data, got %d: %s", rec.Code, rec.Body.String())
}
}
func TestPushEndpoint_MissingImageField(t *testing.T) {
mux := newTestMux()
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
_ = writer.WriteField("other_field", "some value")
if err := writer.Close(); err != nil {
panic(err)
}
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/50x50/filters:format(jpeg)/", body)
req.Header.Set("Content-Type", writer.FormDataContentType())
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusBadRequest {
t.Errorf("expected status 400 for missing image field, got %d: %s", rec.Code, rec.Body.String())
}
}
func TestPushEndpoint_NotMultipart(t *testing.T) {
mux := newTestMux()
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/50x50/filters:format(jpeg)/", bytes.NewReader([]byte("plain text")))
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusBadRequest {
t.Errorf("expected status 400 for non-multipart request, got %d: %s", rec.Code, rec.Body.String())
}
}
func TestPushEndpoint_NoTrailingSlash(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(200, 200)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/100x100/filters:format(png)", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200 without trailing slash, got %d: %s", rec.Code, rec.Body.String())
}
if ct := rec.Header().Get("Content-Type"); ct != "image/png" {
t.Errorf("expected Content-Type image/png, got %s", ct)
}
}
// TestPushEndpoint_Stretch pins the stretch operation: resize to the exact box
// without preserving aspect ratio. A 200x100 source requested at 64x32 must come
// back as exactly 64x32 (distorted), not letterboxed or cropped.
func TestPushEndpoint_Stretch(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(200, 100)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/stretch/64x32/filters:format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
img, err := png.Decode(rec.Body)
if err != nil {
t.Fatalf("failed to decode response image: %v", err)
}
bounds := img.Bounds()
if bounds.Dx() != 64 || bounds.Dy() != 32 {
t.Errorf("stretch should produce exactly 64x32, got %dx%d", bounds.Dx(), bounds.Dy())
}
}
// TestPushEndpoint_MaxResolutionExceeded pins the ErrMaxResolutionExceeded path:
// a decoded input exceeding the configured max width/height returns 422.
func TestPushEndpoint_MaxResolutionExceeded(t *testing.T) {
mux := newTestMuxWithLimits(100, 100)
imgBytes := createTestImage(500, 500)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/64x64/filters:format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusUnprocessableEntity {
t.Errorf("expected status 422 for oversized input, got %d: %s", rec.Code, rec.Body.String())
}
}
// TestPushEndpoint_MaxResolutionExceeded_OneAxis pins the per-axis check: a source
// that exceeds only the max width (but is well under the max height) must still be
// rejected with 422. This catches an inverted "both axes" condition.
func TestPushEndpoint_MaxResolutionExceeded_OneAxis(t *testing.T) {
mux := newTestMuxWithLimits(100, 100)
imgBytes := createTestImage(500, 10)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/64x64/filters:format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusUnprocessableEntity {
t.Errorf("expected status 422 for width-oversized input, got %d: %s", rec.Code, rec.Body.String())
}
}
// TestPushEndpoint_WithinMaxResolution confirms an input within the configured
// limits still succeeds (the 422 check must not over-trigger).
func TestPushEndpoint_WithinMaxResolution(t *testing.T) {
mux := newTestMuxWithLimits(100, 100)
imgBytes := createTestImage(50, 50)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/32x32/filters:format(png)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Errorf("expected status 200 for in-bounds input, got %d: %s", rec.Code, rec.Body.String())
}
}
// TestConcurrencyLimiter pins the THUMBNAILS_MAX_CONCURRENT_REQUESTS behavior:
// up to N requests may proceed in parallel, the (N+1)th is rejected non-blocking
// with a 429. A limit of 0 disables the limiter entirely.
func TestConcurrencyLimiter(t *testing.T) {
t.Run("limit 2 rejects the third concurrent request", func(t *testing.T) {
l := newConcurrencyLimiter(2)
if ok, _ := l.acquire(); !ok {
t.Fatal("first acquire should succeed")
}
if ok, _ := l.acquire(); !ok {
t.Fatal("second acquire should succeed")
}
ok, retryAfter := l.acquire()
if ok {
t.Fatal("third acquire should fail while the limit is reached")
}
if retryAfter <= 0 {
t.Errorf("expected a positive Retry-After hint, got %d", retryAfter)
}
l.release()
if ok, _ := l.acquire(); !ok {
t.Fatal("acquire after release should succeed")
}
})
t.Run("zero limit is unlimited", func(t *testing.T) {
var l *concurrencyLimiter
for i := 0; i < 100; i++ {
if ok, _ := l.acquire(); !ok {
t.Fatalf("acquire %d should succeed without a limiter", i)
}
}
l.release()
})
t.Run("429 is returned when the handler is saturated", func(t *testing.T) {
s := Thumbnails{limiter: newConcurrencyLimiter(1)}
if ok, _ := s.limiter.acquire(); !ok {
t.Fatal("could not saturate the limiter")
}
defer s.limiter.release()
req := httptest.NewRequest(http.MethodPost, "/unsafe/64x64/filters:format(png)/", nil)
rec := httptest.NewRecorder()
s.pushHandler(rec, req)
if rec.Code != http.StatusTooManyRequests {
t.Errorf("expected 429 from a saturated limiter, got %d", rec.Code)
}
if rec.Header().Get("Retry-After") == "" {
t.Error("expected a Retry-After header on 429")
}
})
}
func createTestJPEG(width, height int) []byte {
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
img.Set(x, y, color.RGBA{R: uint8(x % 256), G: uint8(y % 256), B: 128, A: 255})
}
}
var buf bytes.Buffer
if err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 80}); err != nil {
panic(err)
}
return buf.Bytes()
}
func createTestBMP(width, height int) []byte {
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := 0; y < height; y++ {
for x := 0; x < width; x++ {
img.Set(x, y, color.RGBA{R: uint8(x % 256), G: uint8(y % 256), B: 128, A: 255})
}
}
var buf bytes.Buffer
if err := bmp.Encode(&buf, img); err != nil {
panic(err)
}
return buf.Bytes()
}
// TestPushEndpoint_ToleratesExtraFilters proves the generic filters parser ignores
// filters other than format (a real imagor client may send no_upscale or others).
func TestPushEndpoint_ToleratesExtraFilters(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(200, 200)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/100x100/filters:no_upscale():format(jpeg)/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
if ct := rec.Header().Get("Content-Type"); ct != "image/jpeg" {
t.Errorf("expected Content-Type image/jpeg, got %s", ct)
}
}
// TestPushEndpoint_NoFormat_FallbackGIF pins the 1c pass-through fallback: a filters
// segment with no format filter (here just no_upscale) keeps a gif input as gif.
func TestPushEndpoint_NoFormat_FallbackGIF(t *testing.T) {
mux := newTestMux()
imgBytes := createTestGIF(200, 200, 2)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/100x100/filters:no_upscale()/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
if ct := rec.Header().Get("Content-Type"); ct != "image/gif" {
t.Errorf("expected Content-Type image/gif, got %s", ct)
}
}
// TestPushEndpoint_NoFormat_FallbackPNG pins the 1c pass-through fallback: a filters
// segment with no format filter keeps a png input as png.
func TestPushEndpoint_NoFormat_FallbackPNG(t *testing.T) {
mux := newTestMux()
imgBytes := createTestImage(200, 200)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/100x100/filters:no_upscale()/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
if ct := rec.Header().Get("Content-Type"); ct != "image/png" {
t.Errorf("expected Content-Type image/png, got %s", ct)
}
}
// TestPushEndpoint_NoFormat_FallbackJPEG pins the 1c pass-through fallback: a filters
// segment with no format filter keeps a jpeg input as jpeg.
func TestPushEndpoint_NoFormat_FallbackJPEG(t *testing.T) {
mux := newTestMux()
imgBytes := createTestJPEG(200, 200)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/100x100/filters:no_upscale()/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
if ct := rec.Header().Get("Content-Type"); ct != "image/jpeg" {
t.Errorf("expected Content-Type image/jpeg, got %s", ct)
}
}
// TestPushEndpoint_NoFormat_UnsupportedInputFallsBackToJPEG pins imagor's unsavable
// source behavior: an input we cannot re-encode (here a BMP) with no format filter is
// returned as JPEG, matching imagor's default for formats outside its savable set.
func TestPushEndpoint_NoFormat_UnsupportedInputFallsBackToJPEG(t *testing.T) {
mux := newTestMux()
imgBytes := createTestBMP(200, 200)
body, contentType := createMultipartBody(imgBytes)
req := httptest.NewRequest(http.MethodPost, "/unsafe/fit-in/100x100/filters:no_upscale()/", body)
req.Header.Set("Content-Type", contentType)
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("expected status 200, got %d: %s", rec.Code, rec.Body.String())
}
if ct := rec.Header().Get("Content-Type"); ct != "image/jpeg" {
t.Errorf("expected Content-Type image/jpeg, got %s", ct)
}
}
func TestInputFormatViaImaging(t *testing.T) {
tests := []struct {
name string
data []byte
want string
}{
{"jpeg", createTestJPEG(16, 16), "jpg"},
{"png", createTestImage(16, 16), "png"},
{"gif", createTestGIF(16, 16, 2), "gif"},
{"bmp falls back to jpeg", createTestBMP(16, 16), "jpg"},
{"undecodable falls back to jpeg", []byte("not an image at all"), "jpg"},
{"empty falls back to jpeg", []byte{}, "jpg"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := inputFormatViaImaging(tt.data); got != tt.want {
t.Errorf("inputFormatViaImaging() = %q, want %q", got, tt.want)
}
})
}
}
func TestSplitFilter(t *testing.T) {
tests := []struct {
name string
token string
wantName string
wantArgs string
wantOK bool
}{
{"format", "format(jpeg)", "format", "jpeg", true},
{"no args name", "no_upscale()", "no_upscale", "", true},
{"no parens", "plain", "", "", false},
{"open only", "format(jpeg", "", "", false},
{"close only", "formatjpeg)", "", "", false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
name, args, ok := splitFilter(tt.token)
if ok != tt.wantOK || name != tt.wantName || args != tt.wantArgs {
t.Errorf("splitFilter(%q) = (%q,%q,%v), want (%q,%q,%v)",
tt.token, name, args, ok, tt.wantName, tt.wantArgs, tt.wantOK)
}
})
}
}
func TestOutputFormatFromFilters(t *testing.T) {
tests := []struct {
name string
segment string
wantExt string
wantHas bool
wantErr bool
}{
{"format jpeg", "format(jpeg)", "jpg", true, false},
{"extra filters ignored", "no_upscale():format(png)", "png", true, false},
{"unsupported format", "format(webp)", "", false, true},
{"no format filter", "no_upscale()", "", false, false},
{"empty segment", "", "", false, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, has, err := outputFormatFromFilters(tt.segment)
if (err != nil) != tt.wantErr {
t.Fatalf("outputFormatFromFilters() error = %v, wantErr %v", err, tt.wantErr)
}
if got != tt.wantExt || has != tt.wantHas {
t.Errorf("outputFormatFromFilters() = (%q,%v), want (%q,%v)", got, has, tt.wantExt, tt.wantHas)
}
})
}
}
@@ -0,0 +1,179 @@
//go:build enable_vips
package svc
import (
"fmt"
"image"
"image/draw"
"image/gif"
"io"
"github.com/davidbyttow/govips/v2/vips"
"github.com/kovidgoyal/imaging"
)
func init() {
processImage = processImageVips
encodeJPEG = encodeJPEGVips
encodePNG = encodePNGVips
}
// processImageVips resizes the input using libvips. The operation selects the
// resize/crop mode: fill (center-crop to the box, upscaling by default like real
// imagor's default resize), fit-in (fit within the box without cropping, upscaling
// small sources by default like real imagor), or stretch (resize to the exact box).
// noUpscale caps the upscaling at the source size, mirroring imagor's no_upscale()
// filter.
func processImageVips(r io.Reader, width, height int, operation string, noUpscale bool) (any, error) {
if isGifReader(r) {
g, err := gif.DecodeAll(r)
if err == nil && len(g.Image) > 0 {
return resizeGIFVips(g, width, height, operation, noUpscale), nil
}
}
imgData, err := io.ReadAll(r)
if err != nil {
return nil, err
}
m, err := vips.NewImageFromBuffer(imgData)
if err != nil {
return nil, err
}
// Note: no defer m.Close() here. The ImageRef is returned to the caller and
// encoded by encodeJPEGVips/encodePNGVips, which own its lifetime and close it
// after exporting. Closing it here would free the underlying C image before the
// export runs (use-after-free).
switch operation {
case OpStretch:
// Resize to the exact box without preserving aspect ratio.
hScale := float64(width) / float64(m.Width())
vScale := float64(height) / float64(m.Height())
if err := m.ResizeWithVScale(hScale, vScale, vips.KernelLanczos3); err != nil {
return nil, err
}
case OpFitIn:
// Like real imagor, fit-in upscales small sources to fit the box by
// default (SizeBoth); no_upscale() caps it at the source size (SizeDown).
size := vips.SizeBoth
if noUpscale {
size = vips.SizeDown
}
if err := m.ThumbnailWithSize(width, height, vips.InterestingNone, size); err != nil {
return nil, err
}
default: // OpFill
// Center-crop to fill the box exactly. SizeBoth matches real imagor's
// default resize (which upscales small sources); no_upscale() caps it
// at the source size via SizeDown.
size := vips.SizeBoth
if noUpscale {
size = vips.SizeDown
}
if err := m.ThumbnailWithSize(width, height, vips.InterestingAttention, size); err != nil {
return nil, err
}
}
if err := m.RemoveMetadata(); err != nil {
return nil, err
}
// Keep the image in libvips form so the encode hooks can export it via
// ExportJpeg/ExportPng (matching the legacy pipeline's progressive JPEG and
// quality-80 output) instead of a lossy PNG round-trip + stdlib re-encode.
return m, nil
}
// resizeGIFVips resizes every frame of an animated gif while preserving the
// animation, compositing each frame onto a running canvas honoring the gif
// disposal method and re-palletting with Floyd-Steinberg dithering. noUpscale
// caps the default fill at the source size, mirroring imagor's no_upscale()
// filter.
func resizeGIFVips(m *gif.GIF, width, height int, operation string, noUpscale bool) *gif.GIF {
srcX, srcY := m.Config.Width, m.Config.Height
b := image.Rect(0, 0, srcX, srcY)
tmp := image.NewRGBA(b)
for i, frame := range m.Image {
frameBounds := frame.Bounds()
prev := tmp
draw.Draw(tmp, frameBounds, frame, frameBounds.Min, draw.Over)
var processed image.Image
switch operation {
case OpStretch:
processed = imaging.Resize(tmp, width, height, imaging.Lanczos)
case OpFitIn:
if noUpscale && srcX <= width && srcY <= height {
processed = tmp
} else {
processed = imaging.Fit(tmp, width, height, imaging.Lanczos)
}
default: // OpFill
if noUpscale && srcX <= width && srcY <= height {
processed = tmp
} else {
processed = imaging.Fill(tmp, width, height, imaging.Center, imaging.Lanczos)
}
}
m.Image[i] = palettedAtOrigin(processed, frame.Palette)
switch m.Disposal[i] {
case gif.DisposalBackground:
tmp = image.NewRGBA(b)
case gif.DisposalPrevious:
tmp = prev
}
}
// stretch resizes every frame to exactly width x height, so the logical
// screen must always match the box even when that is larger than the source
// (webdav sends no_upscale(), which would otherwise skip the update and leave
// frames extending past the screen, making gif.EncodeAll fail with "image
// block is out of bounds"). fill/fit-in keep the source size when noUpscale
// caps them at the source, so they only grow the screen when actually resized.
if operation == OpStretch || !noUpscale || srcX > width || srcY > height {
m.Config.Width = width
m.Config.Height = height
}
return m
}
// encodeJPEGVips exports the processed libvips image as JPEG using libvips's
// default export parameters (progressive/interlaced, quality 80), matching the
// legacy pipeline's output byte-for-byte where the resize operation is unchanged.
func encodeJPEGVips(w io.Writer, processed any) error {
m, ok := processed.(*vips.ImageRef)
if !ok {
return fmt.Errorf("cannot encode %T as jpeg", processed)
}
defer m.Close()
buf, _, err := m.ExportJpeg(vips.NewJpegExportParams())
if err != nil {
return err
}
_, err = w.Write(buf)
return err
}
// encodePNGVips exports the processed libvips image as PNG via libvips.
func encodePNGVips(w io.Writer, processed any) error {
m, ok := processed.(*vips.ImageRef)
if !ok {
return fmt.Errorf("cannot encode %T as png", processed)
}
defer m.Close()
buf, _, err := m.ExportPng(vips.NewPngExportParams())
if err != nil {
return err
}
_, err = w.Write(buf)
return err
}
@@ -1,32 +1,18 @@
package svc
import (
"context"
"fmt"
"net/http"
"strconv"
"github.com/go-chi/chi/v5"
"github.com/golang-jwt/jwt/v5"
"github.com/riandyrn/otelchi"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/pkg/tracing"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/config"
tjwt "github.com/opencloud-eu/opencloud/services/thumbnails/pkg/service/jwt"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/thumbnail"
)
type contextKey string
const (
keyContextKey contextKey = "key"
)
// Service defines the service handlers.
type Service interface {
ServeHTTP(w http.ResponseWriter, r *http.Request)
GetThumbnail(w http.ResponseWriter, r *http.Request)
}
// NewService returns a service implementation for Service.
@@ -45,28 +31,25 @@ func NewService(opts ...Option) Service {
),
)
logger := options.Logger
resolutions, err := thumbnail.ParseResolutions(options.Config.Thumbnail.Resolutions)
if err != nil {
logger.Fatal().Err(err).Msg("resolutions not configured correctly")
}
limits := options.Config.Thumbnail
svc := Thumbnails{
config: options.Config,
mux: m,
logger: options.Logger,
manager: thumbnail.NewSimpleManager(
resolutions,
options.ThumbnailStorage,
logger,
options.Config.Thumbnail.MaxInputWidth,
options.Config.Thumbnail.MaxInputHeight,
),
logger: options.Logger,
mux: m,
maxWidth: limits.MaxInputWidth,
maxHeight: limits.MaxInputHeight,
limiter: newConcurrencyLimiter(limits.MaxConcurrentRequests),
}
m.Route(options.Config.HTTP.Root, func(r chi.Router) {
r.Use(svc.TransferTokenValidator)
r.Get("/data", svc.GetThumbnail)
})
// Push-based thumbnail generation endpoint (imagor-compatible). The optional
// operation segment selects the resize/crop mode; its absence is the default
// center-crop fill. The filters segment is captured whole and parsed by the
// handler so any set/order of imagor filters is accepted.
handler := svc.pushHandler
m.Post("/unsafe/{operation}/{width}x{height}/filters:{filters}/", handler)
m.Post("/unsafe/{operation}/{width}x{height}/filters:{filters}", handler)
m.Post("/unsafe/{width}x{height}/filters:{filters}/", handler)
m.Post("/unsafe/{width}x{height}/filters:{filters}", handler)
_ = chi.Walk(m, func(method string, route string, _ http.Handler, middlewares ...func(http.Handler) http.Handler) error {
options.Logger.Debug().Str("method", method).Str("route", route).Int("middlewares", len(middlewares)).Msg("serving endpoint")
@@ -78,73 +61,14 @@ func NewService(opts ...Option) Service {
// Thumbnails implements the business logic for Service.
type Thumbnails struct {
config *config.Config
logger log.Logger
mux *chi.Mux
manager thumbnail.Manager
logger log.Logger
mux *chi.Mux
maxWidth int
maxHeight int
limiter *concurrencyLimiter
}
// ServeHTTP implements the Service interface.
func (s Thumbnails) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.mux.ServeHTTP(w, r)
}
// GetThumbnail implements the Service interface.
func (s Thumbnails) GetThumbnail(w http.ResponseWriter, r *http.Request) {
logger := s.logger.SubloggerWithRequestID(r.Context())
key := r.Context().Value(keyContextKey).(string)
thumbnailBytes, err := s.manager.GetThumbnail(key)
if err != nil {
logger.Debug().
Err(err).
Str("key", key).
Msg("could not get the thumbnail")
w.WriteHeader(http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
w.Header().Set("Content-Length", strconv.Itoa(len(thumbnailBytes)))
if _, err = w.Write(thumbnailBytes); err != nil {
logger.Error().
Err(err).
Str("key", key).
Msg("could not write the thumbnail response")
}
}
// TransferTokenValidator validates a transfer token
func (s Thumbnails) TransferTokenValidator(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
tokenString := r.Header.Get("Transfer-Token")
if tokenString == "" {
next.ServeHTTP(w, r)
return
}
logger := s.logger.SubloggerWithRequestID(r.Context())
token, err := jwt.ParseWithClaims(tokenString, &tjwt.ThumbnailClaims{}, func(token *jwt.Token) (any, error) {
if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
return nil, fmt.Errorf("unexpected signing method: %v", token.Header["alg"])
}
return []byte(s.config.Thumbnail.TransferSecret), nil
})
if err != nil {
logger.Debug().
Err(err).
Str("transfer-token", tokenString).
Msg("failed to parse transfer token")
w.WriteHeader(http.StatusUnauthorized)
return
}
if claims, ok := token.Claims.(*tjwt.ThumbnailClaims); ok && token.Valid {
ctx := context.WithValue(r.Context(), keyContextKey, claims.Key)
next.ServeHTTP(w, r.WithContext(ctx))
return
}
logger.Debug().Msg("invalid transfer token")
w.WriteHeader(http.StatusUnauthorized)
})
}
@@ -1,9 +0,0 @@
package jwt
import "github.com/golang-jwt/jwt/v5"
// ThumbnailClaims defines the claims for thumb-nailing
type ThumbnailClaims struct {
jwt.RegisteredClaims
Key string `json:"key"`
}
@@ -1,81 +0,0 @@
package thumbnail
import (
"image/gif"
"io"
"strings"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/errors"
)
const (
typePng = "png"
typeJpg = "jpg"
typeJpeg = "jpeg"
typeGif = "gif"
typeGgs = "ggs"
typeGgp = "ggp"
typeWebp = "webp"
)
// Encoder encodes the thumbnail to a specific format.
type Encoder interface {
// Encode encodes the image to a format.
Encode(w io.Writer, img any) error
// Types returns the formats suffixes.
Types() []string
// MimeType returns the mimetype used by the encoder.
MimeType() string
}
// GifEncoder encodes to gif
type GifEncoder struct{}
// Encode encodes the image to a gif format
func (e GifEncoder) Encode(w io.Writer, img any) error {
g, ok := img.(*gif.GIF)
if !ok {
return errors.ErrInvalidType
}
return gif.EncodeAll(w, g)
}
// Types returns the supported types of the GifEncoder
func (e GifEncoder) Types() []string {
return []string{typeGif}
}
// MimeType returns the mimetype used by the encoder.
func (e GifEncoder) MimeType() string {
return "image/gif"
}
// EncoderForType returns the encoder for a given file type
// or nil if the type is not supported.
func EncoderForType(fileType string) (Encoder, error) {
switch strings.ToLower(fileType) {
case typePng, typeGgs, typeGgp:
return PngEncoder{}, nil
case typeJpg, typeJpeg, typeWebp:
return JpegEncoder{}, nil
case typeGif:
return GifEncoder{}, nil
default:
return nil, errors.ErrNoEncoderForType
}
}
// GetExtForMime return the supported extension by mime
func GetExtForMime(fileType string) string {
ext := strings.TrimPrefix(strings.TrimSpace(strings.ToLower(fileType)), "image/")
switch ext {
case typeJpg, typeJpeg, typePng, typeGif, typeWebp:
return ext
case "application/vnd.geogebra.slides":
return typeGgs
case "application/vnd.geogebra.pinboard":
return typeGgp
default:
return ""
}
}
@@ -1,56 +0,0 @@
//go:build !enable_vips
package thumbnail
import (
"image"
"image/jpeg"
"image/png"
"io"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/errors"
)
// PngEncoder encodes to png
type PngEncoder struct{}
// Encode encodes to png format
func (e PngEncoder) Encode(w io.Writer, img any) error {
m, ok := img.(image.Image)
if !ok {
return errors.ErrInvalidType
}
return png.Encode(w, m)
}
// Types returns the png suffix
func (e PngEncoder) Types() []string {
return []string{typePng}
}
// MimeType returns the mimetype for png files.
func (e PngEncoder) MimeType() string {
return "image/png"
}
// JpegEncoder encodes to jpg
type JpegEncoder struct{}
// Encode encodes to jpg
func (e JpegEncoder) Encode(w io.Writer, img any) error {
m, ok := img.(image.Image)
if !ok {
return errors.ErrInvalidType
}
return jpeg.Encode(w, m, nil)
}
// Types returns the jpg suffixes.
func (e JpegEncoder) Types() []string {
return []string{typeJpeg, typeJpg}
}
// MimeType returns the mimetype for jpg files.
func (e JpegEncoder) MimeType() string {
return "image/jpeg"
}
@@ -1,22 +0,0 @@
package thumbnail
import "testing"
func TestEncoderForType(t *testing.T) {
table := map[string]Encoder{
"jpg": JpegEncoder{},
"JPG": JpegEncoder{},
"jpeg": JpegEncoder{},
"JPEG": JpegEncoder{},
"png": PngEncoder{},
"PNG": PngEncoder{},
"invalid": nil,
}
for k, v := range table {
e, _ := EncoderForType(k)
if e != v {
t.Fail()
}
}
}
@@ -1,66 +0,0 @@
//go:build enable_vips
package thumbnail
import (
"io"
"github.com/davidbyttow/govips/v2/vips"
"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/errors"
)
// PngEncoder encodes to png
type PngEncoder struct{}
// Encode encodes to png format
func (e PngEncoder) Encode(w io.Writer, img interface{}) error {
m, ok := img.(*vips.ImageRef)
if !ok {
return errors.ErrInvalidType
}
buf, _, err := m.ExportPng(vips.NewPngExportParams())
if err != nil {
return err
}
_, err = w.Write(buf)
return err
}
// Types returns the png suffix
func (e PngEncoder) Types() []string {
return []string{typePng}
}
// MimeType returns the mimetype for png files.
func (e PngEncoder) MimeType() string {
return "image/png"
}
// JpegEncoder encodes to jpg
type JpegEncoder struct{}
// Encode encodes to jpg
func (e JpegEncoder) Encode(w io.Writer, img interface{}) error {
m, ok := img.(*vips.ImageRef)
if !ok {
return errors.ErrInvalidType
}
buf, _, err := m.ExportJpeg(vips.NewJpegExportParams())
if err != nil {
return err
}
_, err = w.Write(buf)
return err
}
// Types returns the jpg suffixes.
func (e JpegEncoder) Types() []string {
return []string{typeJpeg, typeJpg}
}
// MimeType returns the mimetype for jpg files.
func (e JpegEncoder) MimeType() string {
return "image/jpeg"
}
Loaded 100 of 174 files, more files were not shown because too many files have changed in this diff. Show more