From 09da9305de8ca8b75df21b280dfd084c8ae77ad2 Mon Sep 17 00:00:00 2001 From: "A.Unger" Date: Tue, 24 Aug 2021 13:54:05 +0200 Subject: [PATCH 1/5] add profiling docs --- docs/ocis/development/profiling.md | 121 +++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 docs/ocis/development/profiling.md diff --git a/docs/ocis/development/profiling.md b/docs/ocis/development/profiling.md new file mode 100644 index 0000000000..c43b506313 --- /dev/null +++ b/docs/ocis/development/profiling.md @@ -0,0 +1,121 @@ +--- +title: "Profiling" +date: 2021-08-24T12:32:20+01:00 +weight: 56 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/ocis/development +geekdocFilePath: profiling.md +--- + +{{< toc >}} + +The only way to enable the profiler currently is to explicitly select which areas to collect samples for. In order to do this, the following steps have to be followed. + +## 1. Clone Reva + +`git clone github.com/cs3org/reva` + +## 2. Patch reva with the area that you want sampled. + +For the purposes of this docs let's use the WebDAV `PROPFIND` path. + +```diff +diff --git a/internal/http/services/owncloud/ocdav/propfind.go b/internal/http/services/owncloud/ocdav/propfind.go +index 0e9c99be..f271572f 100644 +--- a/internal/http/services/owncloud/ocdav/propfind.go ++++ b/internal/http/services/owncloud/ocdav/propfind.go +@@ -32,6 +32,8 @@ import ( + "strings" + "time" + ++ _ "net/http/pprof" ++ + userv1beta1 "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1" + rpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1" + link "github.com/cs3org/go-cs3apis/cs3/sharing/link/v1beta1" +@@ -311,6 +313,12 @@ func requiresExplicitFetching(n *xml.Name) bool { + return true + } + ++func init() { ++ go func() { ++ http.ListenAndServe(":1234", nil) ++ }() ++} ++ + // from https://github.com/golang/net/blob/e514e69ffb8bc3c76a71ae40de0118d794855992/webdav/xml.go#L178-L205 + func readPropfind(r io.Reader) (pf propfindXML, status int, err error) { + c := countingReader{r: r} +``` + +The previous patch will: + +1. import `net/http/pprof`, which will register debug handlers in `DefaultServeMux`. +2. define a `init()` function that starts an http server with the previously registered handlers. + +With everything running one should have access to http://localhost:1234/debug/pprof/ + +## 3. Replace ocis go.mod to local reva and build a new binary + +```diff +diff --git a/go.mod b/go.mod +index 131d14d7b..9668c38e4 100644 +--- a/go.mod ++++ b/go.mod +@@ -78,6 +78,7 @@ require ( + + replace ( + github.com/crewjam/saml => github.com/crewjam/saml v0.4.5 ++ github.com/cs3org/reva => path/to/your/reva + go.etcd.io/etcd/api/v3 => go.etcd.io/etcd/api/v3 v3.0.0-20210204162551-dae29bb719dd + go.etcd.io/etcd/pkg/v3 => go.etcd.io/etcd/pkg/v3 v3.0.0-20210204162551-dae29bb719dd + ) +``` + +Make sure to replace `github.com/cs3org/reva => path/to/your/reva` with the correct location of your reva. + +## 4. Build a new ocis binary + +From owncloud/ocis root: + +```console +$ cd ocis +$ make clean build +``` + +## 5. Start oCIS server + +From owncloud/ocis root: + +```console +$ OCIS_LOG_PRETTY=true OCIS_LOG_COLOR=true ocis/bin/ocis server +``` + +## 6. Run `pprof` + +### Install pprof + +If `pprof` is not installed make sure to get it; one way of installing it is using the Go tools: + +```console +$ go get -u github.com/google/pprof +``` + +### Collecting samples + +Collect 30 seconds of samples: + +```console +$ prof -web http://:1234/debug/pprof/profile\?seconds\=30 +``` + +Once the collection is done a browser tab will open with the result `svg`, looking similar to this: + +![img](https://i.imgur.com/vo0EbcX.jpg) + +For references on how to interpret this graph, [continue reading here](https://github.com/google/pprof/blob/master/doc/README.md#interpreting-the-callgraph). + +## References + +- https://medium.com/swlh/go-profile-your-code-like-a-master-1505be38fdba +- https://dave.cheney.net/ From 147a8f8004082908d8ea7150efcbdc8d1f750f4c Mon Sep 17 00:00:00 2001 From: Alex Unger <6905948+refs@users.noreply.github.com> Date: Tue, 24 Aug 2021 15:48:54 +0200 Subject: [PATCH 2/5] Update docs/ocis/development/profiling.md Co-authored-by: Pascal Wengerter --- docs/ocis/development/profiling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ocis/development/profiling.md b/docs/ocis/development/profiling.md index c43b506313..51ddf89db8 100644 --- a/docs/ocis/development/profiling.md +++ b/docs/ocis/development/profiling.md @@ -55,7 +55,7 @@ The previous patch will: With everything running one should have access to http://localhost:1234/debug/pprof/ -## 3. Replace ocis go.mod to local reva and build a new binary +## 3. Replace reva in oCIS go.mod with local version and build a new binary ```diff diff --git a/go.mod b/go.mod From 06a4a8ef0e55528b596ce4980a0f6c767757118c Mon Sep 17 00:00:00 2001 From: Alex Unger <6905948+refs@users.noreply.github.com> Date: Tue, 24 Aug 2021 15:50:06 +0200 Subject: [PATCH 3/5] Apply suggestions from code review Co-authored-by: Pascal Wengerter --- docs/ocis/development/profiling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ocis/development/profiling.md b/docs/ocis/development/profiling.md index 51ddf89db8..be7091390d 100644 --- a/docs/ocis/development/profiling.md +++ b/docs/ocis/development/profiling.md @@ -106,7 +106,7 @@ $ go get -u github.com/google/pprof Collect 30 seconds of samples: ```console -$ prof -web http://:1234/debug/pprof/profile\?seconds\=30 +$ pprof -web http://:1234/debug/pprof/profile\?seconds\=30 ``` Once the collection is done a browser tab will open with the result `svg`, looking similar to this: From f5797866c5c1c5f278bd4edfb03052312efc4998 Mon Sep 17 00:00:00 2001 From: "A.Unger" Date: Tue, 24 Aug 2021 15:57:17 +0200 Subject: [PATCH 4/5] docs: apply review --- docs/ocis/development/profiling.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/ocis/development/profiling.md b/docs/ocis/development/profiling.md index be7091390d..22807c00a5 100644 --- a/docs/ocis/development/profiling.md +++ b/docs/ocis/development/profiling.md @@ -9,6 +9,15 @@ geekdocFilePath: profiling.md {{< toc >}} +# 0. Prerequisites + +- Go development kit of a [supported version](https://golang.org/doc/devel/release.html#policy). + Follow [these instructions](http://golang.org/doc/code.html) to install the + go tool and set up GOPATH. + +- Graphviz: http://www.graphviz.org/ + Optional, used to generate graphic visualizations of profiles + The only way to enable the profiler currently is to explicitly select which areas to collect samples for. In order to do this, the following steps have to be followed. ## 1. Clone Reva @@ -118,4 +127,4 @@ For references on how to interpret this graph, [continue reading here](https://g ## References - https://medium.com/swlh/go-profile-your-code-like-a-master-1505be38fdba -- https://dave.cheney.net/ +- https://dave.cheney.net/2013/07/07/introducing-profile-super-simple-profiling-for-go-programs From 3c22e886e073b814b469c359f0a209ff762d1608 Mon Sep 17 00:00:00 2001 From: "A.Unger" Date: Thu, 9 Dec 2021 10:14:27 +0100 Subject: [PATCH 5/5] added more background information to the steps --- docs/ocis/development/profiling.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/ocis/development/profiling.md b/docs/ocis/development/profiling.md index 22807c00a5..9afb747cda 100644 --- a/docs/ocis/development/profiling.md +++ b/docs/ocis/development/profiling.md @@ -14,19 +14,19 @@ geekdocFilePath: profiling.md - Go development kit of a [supported version](https://golang.org/doc/devel/release.html#policy). Follow [these instructions](http://golang.org/doc/code.html) to install the go tool and set up GOPATH. - -- Graphviz: http://www.graphviz.org/ - Optional, used to generate graphic visualizations of profiles +- Graphviz: http://www.graphviz.org/. Used to generate graphic visualizations of profiles, which this example setup does. The only way to enable the profiler currently is to explicitly select which areas to collect samples for. In order to do this, the following steps have to be followed. ## 1. Clone Reva +Reva is the reference implementation of the CS3 APIs that we use for our daily business between oCIS and its storages. It is in charge of accessing the storage, as well as managing shares. Because of this fact, the examples will modify code in this dependency. You can think of Reva as the framework we use in order to interface with different storage providers. + `git clone github.com/cs3org/reva` ## 2. Patch reva with the area that you want sampled. -For the purposes of this docs let's use the WebDAV `PROPFIND` path. +For the purposes of these docs let's use the WebDAV `PROPFIND` path. This patch is needed in order to have the WebDAV process reporting profiling traces to the `pprof`. ```diff diff --git a/internal/http/services/owncloud/ocdav/propfind.go b/internal/http/services/owncloud/ocdav/propfind.go @@ -60,12 +60,14 @@ index 0e9c99be..f271572f 100644 The previous patch will: 1. import `net/http/pprof`, which will register debug handlers in `DefaultServeMux`. -2. define a `init()` function that starts an http server with the previously registered handlers. +2. define a `init()` function that starts an HTTP server with the previously registered handlers. With everything running one should have access to http://localhost:1234/debug/pprof/ ## 3. Replace reva in oCIS go.mod with local version and build a new binary +In Go, the `go.mod` file controls the dependencies of your module. Because we patched an external library, Go provides with a mechanism to overwrite an existing dependency with one on your local machine, which we previously installed. + ```diff diff --git a/go.mod b/go.mod index 131d14d7b..9668c38e4 100644 @@ -85,6 +87,8 @@ Make sure to replace `github.com/cs3org/reva => path/to/your/reva` with the corr ## 4. Build a new ocis binary +Using the new dependency with the pprof patch. + From owncloud/ocis root: ```console @@ -97,11 +101,13 @@ $ make clean build From owncloud/ocis root: ```console -$ OCIS_LOG_PRETTY=true OCIS_LOG_COLOR=true ocis/bin/ocis server +$ ocis/bin/ocis server ``` ## 6. Run `pprof` +[Pprof](https://github.com/google/pprof) is a tool developed at Google. It is a tool for visualization and analysis of profiling data. It will take the reported profiled data from our server, and represent it in a meaningful manner. + ### Install pprof If `pprof` is not installed make sure to get it; one way of installing it is using the Go tools: @@ -124,6 +130,10 @@ Once the collection is done a browser tab will open with the result `svg`, looki For references on how to interpret this graph, [continue reading here](https://github.com/google/pprof/blob/master/doc/README.md#interpreting-the-callgraph). +## Room for improvement + +Because these docs are intended to be read by developers they are quite technical in content. Requiring the user to alter the code. This is done so that we do not include, or assume, third party dependencies such as Graphviz in our binary, making it heavier. Having said this, the profiler is only meant to be used in development + ## References - https://medium.com/swlh/go-profile-your-code-like-a-master-1505be38fdba