mirror of
https://github.com/opencloud-eu/opencloud.git
synced 2026-07-29 08:48:49 -04:00
Merge pull request #3328 from owncloud/docs-adr-search
[docs-only] ADR: Search query language, Indexing and API
This commit is contained in:
Michael Barz
60
docs/ocis/adr/0018-file-search-api.md
Normal file
60
docs/ocis/adr/0018-file-search-api.md
Normal file
@@ -0,0 +1,60 @@
|
||||
---
|
||||
title: "18. File Search API"
|
||||
date: 2022-03-18T09:00:00+01:00
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs/ocis/adr
|
||||
geekdocFilePath: 0018-file-search-api.md
|
||||
---
|
||||
|
||||
* Status: proposed
|
||||
* Deciders: @butonic, @micbar, @dragotin, @C0rby
|
||||
* Date: 2022-03-18
|
||||
|
||||
## Context and Problem Statement
|
||||
|
||||
The ability to find files based on certain search terms is a key requirement for a system that provides the ability to store unstructured data on a large scale.
|
||||
|
||||
## Decision Drivers
|
||||
|
||||
* Have a simple yet powerful, scalable and performant way of finding files in oCIS
|
||||
* Be able to construct intelligent searches based on metadata
|
||||
* Allow the user to filter the search queries based on metadata
|
||||
|
||||
## Considered Options
|
||||
|
||||
* [Libre Graph API](#libre-graph-api)
|
||||
* [WebDAV API](#webdav-api)
|
||||
|
||||
## Decision Outcome
|
||||
|
||||
Chosen option: [WebDAV API](#webdav-api) because the current WebUI is compatible with that API. We may use the GraphAPI later in a second iteration.
|
||||
|
||||
### Positive Consequences
|
||||
|
||||
* The existing Clients can continue to use the well-known API
|
||||
* There are existing API tests which cover the basic behavior
|
||||
|
||||
### Negative consequences
|
||||
|
||||
* We have no server side result filtering capabilities
|
||||
|
||||
## Pros and Cons of the Options
|
||||
|
||||
### Libre Graph API
|
||||
|
||||
* Good, because we try to switch most of our HTTP requests to Libre Graph
|
||||
* Good, because the Graph API supports scopes, sorting and query language
|
||||
* Good, because it supports server side result filtering
|
||||
* Bad, because there are currently no clients which support that
|
||||
|
||||
### WebDAV API
|
||||
|
||||
* Good, because WebDAV is a well-known and widely adopted Standard
|
||||
* Good, because existing Clients continue to work without extra efforts
|
||||
* Bad, because the syntax is limited
|
||||
* Bad, because we cannot do server side result filtering
|
||||
|
||||
## Links
|
||||
|
||||
* [Search Indexing](0019-file-search-index.md)
|
||||
* [Search Query Language](0020-file-search-query-language.md)
|
||||
79
docs/ocis/adr/0019-file-search-index.md
Normal file
79
docs/ocis/adr/0019-file-search-index.md
Normal file
@@ -0,0 +1,79 @@
|
||||
---
|
||||
title: "19. File Search Index"
|
||||
date: 2022-03-18T09:00:00+01:00
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs/ocis/adr
|
||||
geekdocFilePath: 0019-file-search-index.md
|
||||
---
|
||||
|
||||
* Status: proposed
|
||||
* Deciders: @butonic, @micbar, @dragotin, @C0rby
|
||||
* Date: 2022-03-18
|
||||
|
||||
## Context and Problem Statement
|
||||
|
||||
The ability to find files based on certain search terms is a key requirement for a system that provides the ability to store unstructured data on a large scale.
|
||||
|
||||
More sophisticated search capabilities are expected and can be implemented, especially based on metadata.
|
||||
|
||||
To trigger the indexing of a file, the search service listens to create, update and delete events on the internal event bus of oCIS.
|
||||
|
||||
The events need to contain a valid reference that defines the file space and file id of the file in question. The event only must be sent when the file operation (update, creation, removal) is finished.
|
||||
|
||||
Sharing adds more complexity because the index also needs to react to create, delete and modify shares events. Sharing should not duplicate the indexed data, especially within spaces or group shares.
|
||||
|
||||
## Decision Drivers
|
||||
|
||||
* Have a simple yet powerful, scalable and performant way of finding files in oCIS
|
||||
* Be able to construct intelligent searches based on metadata
|
||||
* Allow the user to filter the search queries based on metadata
|
||||
* Basic File Search needs to be implemented out of the box without external dependencies
|
||||
* The Search Indexing Service should be replacable with more sophisticated technologies like Elasticsearch
|
||||
* Make use of the spaces architecture to shard search indexes by space
|
||||
* The Search Indexing Service needs to deal with multiple users accessing the same resources due to shares
|
||||
* The Search Service should be compatible with different search indexing technologies
|
||||
|
||||
## Considered Options
|
||||
|
||||
* [Bleve Search](#bleve-search)
|
||||
* [Elastic Search](#elastic-search)
|
||||
|
||||
## Decision Outcome
|
||||
|
||||
Chosen option: Bleve Search, because we can fulfill the MVP and include it into the single binary.
|
||||
|
||||
### Positive Consequences
|
||||
|
||||
* Basic File Search works out of the box
|
||||
* We do not need heavy external dependencies which need to be deployed alongside
|
||||
|
||||
### Negative consequences
|
||||
|
||||
* We need to be aware of the scaling limits
|
||||
* We need to find a way to work with shares and spaces
|
||||
* It has a limited query language
|
||||
|
||||
## Pros and Cons of the Options
|
||||
|
||||
### Bleve Search
|
||||
|
||||
* Good, because it is written in GoLang and can be bundled into the single oCIS binary
|
||||
* Good, because it is a lightweight but powerful solution which could fulfill a lot of use cases
|
||||
* Bad, because we do not know exactly how we can represent shares in the index without duplicating data
|
||||
* Bad, because it is a single process
|
||||
* Bad, because the query language is limited
|
||||
|
||||
### Elastic Search
|
||||
|
||||
* Good, because it has become an industry standard
|
||||
* Good, because it supports a rich query language
|
||||
* Good, because it has built in cluster support and scales well
|
||||
* Good, because it has a permission system and supports multiple users and groups to access the same resource
|
||||
* Bad, because it is a heavy setup and needs extra effort and knowledge
|
||||
|
||||
## Links
|
||||
|
||||
* [Search API](0018-file-search-api.md)
|
||||
* [Search Query Language](0020-file-search-query-language.md)
|
||||
* [Bleve Search on GitHub](https://github.com/blevesearch/bleve)
|
||||
* [ElasticSearch](https://www.elastic.co/elastic-stack/)
|
||||
95
docs/ocis/adr/0020-file-search-query-language.md
Normal file
95
docs/ocis/adr/0020-file-search-query-language.md
Normal file
@@ -0,0 +1,95 @@
|
||||
---
|
||||
title: "20. File Search Query Language"
|
||||
date: 2022-03-18T09:00:00+01:00
|
||||
geekdocRepo: https://github.com/owncloud/ocis
|
||||
geekdocEditPath: edit/master/docs/ocis/adr
|
||||
geekdocFilePath: 0018-file-search-query-language.md
|
||||
---
|
||||
|
||||
* Status: proposed
|
||||
* Deciders: @butonic, @micbar, @dragotin, @C0rby
|
||||
* Date: 2022-03-18
|
||||
|
||||
## Context and Problem Statement
|
||||
|
||||
From the users perspective, the interface to search is just a single form field where the user enters one or more search terms. The minimum expectation is that the search returns file names and links to files that
|
||||
|
||||
* have a file name that contains at least one of the search terms
|
||||
* contain at least one of the search terms in the file contents
|
||||
* have meta data that is equal or contains one of the search terms
|
||||
|
||||
## Decision Drivers
|
||||
|
||||
* The standard user should not be bothered by a query syntax
|
||||
* The power user should also be able to narrow his search with an efficient and flexible syntax
|
||||
* We need to consider different backend technologies which we need to access through an abstraction layer
|
||||
* Using different indexing systems should lead to a slightly different feature set whitout changing the syntax completely
|
||||
|
||||
## Considered Options
|
||||
|
||||
* [Simple Query](#simplified-query)
|
||||
* [Lucene Query Language](#lucene-query-language)
|
||||
* [Solr Query Language](#solr-query-language)
|
||||
* [Elasticsearch Query Language](#elasticsearch-query-language)
|
||||
|
||||
## Decision Outcome
|
||||
|
||||
Chosen option: [Simple Query](#simplified-query), because it is a suitable MVP.
|
||||
|
||||
### Positive Consequences
|
||||
|
||||
* We can do a quick implementation
|
||||
|
||||
### Negative consequences
|
||||
|
||||
* We cannot specify terms or other search criteria
|
||||
* We will need to find a good point in time to use a query language before we start working around it
|
||||
|
||||
## Pros and Cons of the Options
|
||||
|
||||
### Simplified Query
|
||||
|
||||
Implement a very simple search approach: Return all files which contain at least one of the keywords in their name, path, alias or selected metadata.
|
||||
|
||||
* Good, because that covers 80% of the users needs
|
||||
* Good, because it is very straightforward
|
||||
* Good, because it is a suitable solution for GA
|
||||
* Bad, because it is below the industry standard
|
||||
* Bad, because it only provides one search query
|
||||
|
||||
### Lucene Query Language
|
||||
|
||||
The Lucene Query Parser syntax supports advanced queries like term, phrase, wildcard, fuzzy search, proximity search, regular expressions, boosting, boolean operators and grouping. It is a well known query syntax used by the Apache Lucene Project. Popular Platforms like Wikipedia are using Lucene or Solr, which is the successor of Lucene
|
||||
|
||||
* Good, because it is a well documented and powerful syntax
|
||||
* Good, because it is very close to the Elasticsearch and the Solr syntax which enhances compatibility
|
||||
* Bad, because there is no powerful and well tested query parser for golang available
|
||||
* Bad, because it adds complexity and fulfilling all the different query usecases can be an "uphill battle"
|
||||
|
||||
### Solr Query Language
|
||||
|
||||
Solr is highly reliable, scalable and fault tolerant, providing distributed indexing, replication and load-balanced querying, automated failover and recovery, centralized configuration and more. Solr powers the search and navigation features of many of the world's largest internet sites.
|
||||
|
||||
* Good, because it is a well documented and powerful syntax
|
||||
* Good, because it is very close to the Elasticsearch and the Lucene syntax which enhances compatibility
|
||||
* Good, because it has a strong community with large resources and knowledge
|
||||
* Bad, because it adds complexity and fulfilling all the different query usecases can be an "uphill battle"
|
||||
|
||||
### Elasticsearch Query Language
|
||||
|
||||
Elasticsearch provides a full Query DSL (Domain Specific Language) based on JSON to define queries. Think of the Query DSL as an AST (Abstract Syntax Tree) of queries, consisting of two types of clauses. It is able to combine multiple query types into compound queries. It is also a successor of Solr.
|
||||
|
||||
* Good, because it is a well documented and powerful syntax
|
||||
* Good, because it is very close to the Elasticsearch and the Solr syntax which enhances compatibility
|
||||
* Good, because there is a stable and well tested go client which brings a query builder
|
||||
* Good, because it could be used as the query language which supports different search backends by just implementing what is needed for our usecase
|
||||
* Bad, because it adds complexity and fulfilling all the different query usecases can be an "uphill battle"
|
||||
|
||||
## Links
|
||||
|
||||
* [Search API](0018-file-search-api.md)
|
||||
* [Search Indexing](0019-file-search-index.md)
|
||||
* [Apache Lucene](https://lucene.apache.org/)
|
||||
* [Apache Solr](https://solr.apache.org/)
|
||||
* [Elastic Search](https://solr.apache.org/)
|
||||
* [Elastic Search for go](https://github.com/elastic/go-elasticsearch)
|
||||
Reference in New Issue
Block a user