Commit Graph

25 Commits

Author SHA1 Message Date
Matthieu MOREL
675e958877 chore(ci): bump golangci-lint to v2.1.2 (#4500)
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2025-04-15 22:49:13 -07:00
Julio López
1f9f9a1846 chore(general): use non-formatting log variants when there is no formatting (#3931)
Use non-formatting logging functions for message without formatting.
For example, `log.Info("message")` instead of `log.Infof("message")`

Configure linter for printf-like functions
2024-06-18 23:13:17 -07:00
Jarek Kowalski
fcb8197f3f chore(ci): upgraded linter to 1.59.0 (#3883) 2024-05-29 20:31:57 -07:00
Jarek Kowalski
09415e0c7d chore(ci): upgraded to go 1.22 (#3746)
Upgrades go to 1.22 and switches to new-style for loops

---------

Co-authored-by: Julio López <1953782+julio-lopez@users.noreply.github.com>
2024-04-08 09:52:47 -07:00
Jarek Kowalski
a8e4d50600 build(deps): upgraded linter to v1.55.2, fixed warnings (#3611)
* build(deps): upgraded linter to v1.55.2, fixed warnings

* removed unsafe hacks with better equivalents

* test fixes
2024-02-02 23:34:34 -08:00
Jarek Kowalski
80423cf5f6 feat(server): fixed server logging file & console (#3262)
Previously some logs from a running server were only kept in memory
(including storage activity logs) which was confusing to many folks.

This changes the behavior so that logs are sent to their regular
(console/file) file locations in addition to the UI tasks.

Old behavior can be restored by adding `--no-persistent-logs` to
server.
2023-09-02 18:10:18 -07:00
Jarek Kowalski
cbc66f936d chore(ci): upgraded linter to 1.53.3 (#3079)
* chore(ci): upgraded linter to 1.53.3

This flagged a bunch of unused parameters, so the PR is larger than
usual, but 99% mechanical.

* separate lint CI task

* run Lint in separate CI
2023-06-18 13:26:01 -07:00
Jarek Kowalski
f69424961f chore(ci): upgrade golang to 1.19.2 and linter to 1.50.1 (#2526)
Lack of generics support is blocking various dependency upgrades,
so this unblocks that.

Temporarily disabled `checklocks` linter until it is fixed upstream.
2022-10-28 11:02:47 -07:00
Jarek Kowalski
51dcaa985d chore(ci): upgraded linter to 1.48.0 (#2294)
Mechanically fixed all issues, added `lint-fix` make target.
2022-08-09 06:07:54 +00:00
Jarek Kowalski
34da3a2415 feat(general): implemented custom log encoder for ZAP (#2116)
Turns out standard ConsoleEncoder is not optimized at all and we're
emitting up to 8 log entries per file. This change avoids massive
amount of allocations and brings in some latency reduction.

Backing up 100k files in a flat directory:

duration: current:7.8 baseline:7.9 change:-1.0 %
repo_size: current:1019954521.4 baseline:1019976318.9 change:-0.0 %
num_files: current:58.0 baseline:58.0 change:0%
avg_heap_objects: current:6682141.0 baseline:7508631.3 change:-11.0 %
avg_heap_bytes: current:845404672.0 baseline:867325413.9 change:-2.5 %
avg_ram: current:156.5 baseline:159.1 change:-1.7 %
max_ram: current:278.3 baseline:287.2 change:-3.1 %
avg_cpu: current:153.8 baseline:156.4 change:-1.7 %
max_cpu: current:298.4 baseline:297.1 change:+0.4 %

Backing up Linux 5.18.4:

duration: current:3.6 baseline:4.2 change:-14.2 %
repo_size: current:1081624213.7 baseline:1081635886.8 change:-0.0 %
num_files: current:60.0 baseline:60.0 change:0%
avg_heap_objects: current:5180192.3 baseline:5831270.7 change:-11.2 %
avg_heap_bytes: current:783468754.2 baseline:804350042.1 change:-2.6 %
avg_ram: current:239.0 baseline:240.6 change:-0.6 %
max_ram: current:384.8 baseline:368.0 change:+4.6 %
avg_cpu: current:259.8 baseline:230.8 change:+12.6 %
max_cpu: current:321.6 baseline:303.9 change:+5.9 %
2022-07-02 13:55:01 +00:00
Jarek Kowalski
70e24106ee refactor(general): unified logging.Logger with *zap.SugaredLogger (#2090)
- removed a bunch of hacks and should improve the logging
performance by avoiding interfaces and data translation. This will
allow using of de-sugared loggers in performance-critical
logging situations.

- this will also allow using features of ZAP more directly without
having to reimplement them.

- moved logging.Printf() to testlogging

- refactored `uitask` to store logs in a structural format and
present them as JSON only in the UI

- renamed printf_logger.go to printf.go so that fewer columns are used
in the logs
2022-06-26 05:11:52 +00:00
Jarek Kowalski
a2fdbdd717 chore(repository): reduced memory allocations when logging content.ID (#2024)
This fixes regression introduced in #1960.

Tested by backing up Linux 5.14.8 source on M1 Mac (average of 15 runs):

Before: duration=6.7s avg_heap_objects=7411657 avg_heap_bytes=871794888
After: duration=5.6s (17% faster) avg_heap_objects=5947800 (20% less) avg_heap_bytes=795762120 (9% less)
2022-06-09 06:23:45 +00:00
Jarek Kowalski
e67f84e0ba chore(general): updated linter to 1.44.0 (#1681) 2022-01-25 21:21:13 -08:00
Jarek Kowalski
32ed220a6c build(lint): enabled gochecknoglobals and tagged existing globals (#1664) 2022-01-15 12:54:56 -08:00
Jarek Kowalski
89edfbf257 maintenance: send logs to content log as well (#1496) 2021-11-06 23:08:00 -07:00
Jarek Kowalski
dcff6c285d Added support for logging policies (#1472)
* policy: introduced OptionalBool - refactoring

* policy: added logging policy

* testing: added support for symlinks and modtime to mockfs

* logging: exposed NullLogger instance

* upload: emit debug logs according to logging policies

* cli: logging policy support
2021-11-06 10:06:05 -07:00
Jarek Kowalski
a0cfa2556f introduced structural debug logging and optional JSON output (#1475)
* logging: added Logger.Debugw(message, key1, value1, ..., keyN, valueN)

This is based on ZAP and allows structural logs to be emitted.

* cli: added --json-log-console and --json-log-file flags

* logging: updated storage logging wrapper to use structural logging

* pr feedback
2021-11-03 21:57:37 -07:00
Jarek Kowalski
4a47bc3210 logging: switched from go-logging to zap (#1376)
This is much more efficient in terms of memory allocations
and speeds up backup due to less GC pressure.

Fixes #1345
2021-10-12 22:52:24 -07:00
Jarek Kowalski
8b760b66a8 logging: added memoization of Logger instances per context (#1369) 2021-10-09 05:02:18 -07:00
Jarek Kowalski
d84c884321 Added content manager internal logging (#1116)
* logging: added logger wrappers for Broadcast and Prefix

* nit: moved max hash size to a named constant

* content: added internal logger

* content: replaced context-based logging with explicit Loggers

This will capture the logger.Logger associated with the context when
the repository is opened and will reuse it for all logs instead of
creating new logger for each log message.

The new logger will also write logs to the internal logger in addition
to writing to a log file/console.

* cli: allow decrypting all blobs whose names start with _

* maintenance: added logs cleanup

* cli: commands to view logs

* cli: log selected command on each write session
2021-06-05 08:48:43 -07:00
Jarek Kowalski
fcd507a56d Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)
* cli: fixed remaining testability indirections for output and logging

* cli: added cli.RunSubcommand() which is used in testing to execute a subcommand in the same process

* tests: refactored most e2e tests to invoke kopia subcommands in-process

* Makefile: enable code coverage for cli/ and internal/

* testing: pass 'testing' tag to unit tests which uses much faster (insecure) password hashing scheme

* Makefile: push coverage from PRs again

* tests: disable buffer management to reduce memory usage on ARM

* cli: fixed misaligned atomic field on ARMHF

also temporarily fixed statup-time benign race condition when setting
default on the timeZone variable, which is the last global variable.
2021-05-11 22:26:28 -07:00
Jarek Kowalski
f4347886b8 logging: simplified log levels (#954)
Removed Warning, Notify and Fatal:

* `Warning` => `Error` or `Info`
* `Notify` => `Info`
* `Fatal` was never used.

Note that --log-level=warning is still supported for backwards
compatibility, but it is the same as --log-level=error.

Co-authored-by: Julio López <julio+gh@kasten.io>
2021-04-09 07:27:35 -07:00
Jarek Kowalski
9d7cf71a37 Logging flags (#674)
* logging: cleaned up stderr logging

- do not show module
- do not show timestamps by default (enable with --console-timestamps)

* logging: replaced most printStderr() with log.Info

* cli: additional logging cleanup
2020-10-10 10:48:37 -07:00
Jarek Kowalski
9a6dea898b Linter upgrade to v1.30.0 (#526)
* fixed godot linter errors
* reformatted source with gofumpt
* disabled some linters
* fixed nolintlint warnings
* fixed gci warnings
* lint: fixed 'nestif' warnings
* lint: fixed 'exhaustive' warnings
* lint: fixed 'gocritic' warnings
* lint: fixed 'noctx' warnings
* lint: fixed 'wsl' warnings
* lint: fixed 'goerr113' warnings
* lint: fixed 'gosec' warnings
* lint: upgraded linter to 1.30.0
* lint: more 'exhaustive' warnings

Co-authored-by: Nick <nick@kasten.io>
2020-08-12 19:28:53 -07:00
Jarek Kowalski
c8fcae93aa logging: refactored logging
This is mostly mechanical and changes how loggers are instantiated.

Logger is now associated with a context, passed around all methods,
(most methods had ctx, but had to add it in a few missing places).

By default Kopia does not produce any logs, but it can be overridden,
either locally for a nested context, by calling

ctx = logging.WithLogger(ctx, newLoggerFunc)

To override logs globally, call logging.SetDefaultLogger(newLoggerFunc)

This refactoring allowed removing dependency from Kopia repo
and go-logging library (the CLI still uses it, though).

It is now also possible to have all test methods emit logs using
t.Logf() so that they show up in failure reports, which should make
debugging of test failures suck less.
2020-02-25 17:24:44 -08:00