mirror of https://github.com/containers/podman.git synced 2026-03-20 23:53:25 -04:00

Files

Ed Santiago 589248d2f3 Implement review feedback

- document a recommended convention for fail-fast tests

- document the requirement for jq. (And, add a fail-fast
  test for its presence; remove the duplicated checks
  in subtests)

- add further sanity checks to 'help' test. Add missing
  documentation. Remove a no-longer-needed workaround for
  usage-message bug fixed in #2486

- add a documented TEMPLATE

- and, since we're at 1.1, enable 'Remote API' check in
  version test

- better diagnostics in setup/teardown; add vim filetype hint;
  better formatting of actual-vs-expect errors

- new pod-top, logs, build tests

- improve error messages

- add $IMAGE alias for ridiculous $PODMAN_TEST_IMAGE_FQN

- final cleanup, in prep for merge

Signed-off-by: Ed Santiago <santiago@redhat.com>

2019-03-07 14:09:00 -07:00

2.9 KiB

Raw Blame History

Quick overview of podman system tests. The idea is to use BATS, but with a framework for making it easy to add new tests and to debug failures.

Quick Start

Look at 030-run.bats for a simple but packed example. This introduces the basic set of helper functions:

setup (implicit) - resets container storage so there's one and only one (standard) image, and no running containers.
parse_table - you can define tables of inputs and expected results, then read those in a while loop. This makes it easy to add new tests. Because bash is not a programming language, the caller of parse_table sometimes needs to massage the returned values; 015-run.bats offers examples of how to deal with the more typical such issues.
run_podman - runs command defined in $PODMAN (default: 'podman' but could also be './bin/podman' or 'podman-remote'), with a timeout. Checks its exit status.
is - compare actual vs expected output. Emits a useful diagnostic on failure.
die - output a properly-formatted message to stderr, and fail test
skip_if_rootless - if rootless, skip this test with a helpful message.
random_string - returns a pseudorandom alphanumeric string

Test files are of the form NNN-name.bats where NNN is a three-digit number. Please preserve this convention, it simplifies viewing the directory and understanding test order. In particular, 00x tests should be reserved for a first-pass fail-fast subset of tests:

bats test/system/00*.bats || exit 1
bats test/system

...the goal being to provide quick feedback on catastrophic failures without having to wait for the entire test suite.

Analyzing test failures

The top priority for this scheme is to make it easy to diagnose what went wrong. To that end, podman_run always logs all invoked commands, their output and exit codes. In a normal run you will never see this, but BATS will display it on failure. The goal here is to give you everything you need to diagnose without having to rerun tests.

The is comparison function is designed to emit useful diagnostics, in particular, the actual and expected strings. Please do not use the horrible BATS standard of [ x = y ]; that's nearly useless for tracking down failures.

If the above are not enough to help you track down a failure:

Debugging tests

Some functions have dprint statements. To see the output of these, set PODMAN_TEST_DEBUG="funcname" where funcname is the name of the function or perhaps just a substring.

Requirements

The jq tool is needed for parsing JSON output.

Further Details

TBD. For now, look in helpers.bash; each helper function has (what are intended to be) helpful header comments. For even more examples, see and/or run helpers.t; that's a regression test and provides a thorough set of examples of how the helpers work.

2.9 KiB Raw Blame History