Commit Graph

30 Commits

Author SHA1 Message Date
Nícolas F. R. A. Prado
2dd0b5a8fc selftests: ktap_helpers: Add a helper to finish the test
Similar to the C counterpart, keep track of the number of test cases in
the test plan and add a helper function to be called at the end of the
test to print the results and exit with the corresponding exit code.

Signed-off-by: Nícolas F. R. A. Prado <nfraprado@collabora.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2024-02-20 15:53:20 -07:00
Nícolas F. R. A. Prado
d63fde98b8 selftests: ktap_helpers: Add a helper to abort the test
Similar to the C counterpart, add a helper function to abort the
remainder of the test.

Signed-off-by: Nícolas F. R. A. Prado <nfraprado@collabora.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2024-02-20 15:53:15 -07:00
Nícolas F. R. A. Prado
d90b7c705c selftests: ktap_helpers: Add helper to pass/fail test based on exit code
Similar to the C counterpart, add a helper function that runs a command
and passes or fails the test based on the result.

Signed-off-by: Nícolas F. R. A. Prado <nfraprado@collabora.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2024-02-20 15:53:08 -07:00
Nícolas F. R. A. Prado
6934eea269 selftests: ktap_helpers: Add helper to print diagnostic messages
Similar to the C counterpart, add a helper to print a diagnostic
message.

Signed-off-by: Nícolas F. R. A. Prado <nfraprado@collabora.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2024-02-20 15:53:02 -07:00
Laura Nao
7c079e909b selftests: Move KTAP bash helpers to selftests common folder
Move bash helpers for outputting in KTAP format to the common selftests
folder. This allows kselftests other than the dt one to source the file
and make use of the helper functions.
Define pass, fail and skip codes in the same file too.

Signed-off-by: Laura Nao <laura.nao@collabora.com>
Reviewed-by: Nícolas F. R. A. Prado <nfraprado@collabora.com>
Tested-by: Nícolas F. R. A. Prado <nfraprado@collabora.com>
Acked-by: Rob Herring <robh@kernel.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2024-02-20 15:52:55 -07:00
Hangbin Liu
9d0b4ad82d kselftest/runner.sh: add netns support
Add a variable RUN_IN_NETNS if the user wants to run all the selected tests
in namespace in parallel. With this, we can save a lot of testing time.

Note that some tests may not fit to run in namespace, e.g.
net/drop_monitor_tests.sh, as the dwdump needs to be run in init ns.

I also added another parameter -p to make all the logs reported separately
instead of mixing them in the stdout or output.log.

Nit: the NUM in run_one is not used, rename it to test_num.

Acked-by: David Ahern <dsahern@kernel.org>
Signed-off-by: Hangbin Liu <liuhangbin@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2023-12-23 00:26:32 +00:00
Linus Torvalds
a747acc0b7 linux-kselftest-next-6.6-rc2
This kselftest fixes update for Linux 6.6-rc2 consists of fixes
 
 -- kselftest runner script to propagate SIGTERM to runner child
    to avoid kselftest hang.
 -- to install symlinks required for test execution to avoid test
    failures.
 -- kselftest dependency checker script argument parsing.
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEPZKym/RZuOCGeA/kCwJExA0NQxwFAmT/StAACgkQCwJExA0N
 QxxeZRAA1wBztlE9JF9cBcjXICXXlCBIGL3tfPMJJrkv5KezZPegzdOwXQfEfHyp
 2bDtzpamrnDUNtV5xbCu+NVWhlLvyNHb9Irr/wcOLLopsLxtvnodVyrPbOs2Spsd
 dV6szfMEP7+sbjlSjPD/96OPRLNRcPEFiVr/bjjzMbpAO0AjDjjONKSeN6zce4K/
 7hsh4EZjb7mUnKLDi+ZF2+HTKEBiyejpOC2zvEoL5nba7voVxy/bNYarbpAMas7u
 XkYSrxPMQM5moA6MYs+As0IVwlEDk+4XGgYN9Z5eTgai5nz1q/8mNxGvlK2pATI+
 vWo1yRAdzDLuBSpesXjocAZWMxR3BD7BhbF3IAWKVkotbwiB0zZGy0umta9qF/e8
 izT298QA9YiVz+Um/sCirpZGdegtaMKUcFN+AyGbIPzd/xmDLWoUlIy1qlg3dPW1
 ZtiEMgNqqvJgi3bh/kjLUPUoMdswkM3Zuhyn9sBq+Z215XoyJqVt2vW0NrlTuTf8
 3dunqIDxcyhaGvnCDBDvH80TArg91eERtRrWe9aISYH5Y4IZl7kXRtFlCGi8c48M
 hP0kYpenbo/rnU8GoR+Yuo2aNma7YBYg3ZJwGJAE55ZgG4euNYXWv4NsVYyzDRTF
 dtwl5GCggYGtCRDQXNEmxAIuEZgCH4PIAt742CgkrVxEI+lrWlQ=
 =8/pm
 -----END PGP SIGNATURE-----

Merge tag 'linux-kselftest-next-6.6-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest

Pull kselftest fixes from Shuah Khan:

 - kselftest runner script to propagate SIGTERM to runner child
   to avoid kselftest hang

 - install symlinks required for test execution to avoid test
   failures

 - kselftest dependency checker script argument parsing

* tag 'linux-kselftest-next-6.6-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest:
  selftests: Keep symlinks, when possible
  selftests: fix dependency checker script
  kselftest/runner.sh: Propagate SIGTERM to runner child
  selftests/ftrace: Correctly enable event in instance-event.tc
2023-09-12 09:10:36 -07:00
Björn Töpel
9616cb34b0 kselftest/runner.sh: Propagate SIGTERM to runner child
Timeouts in kselftest are done using the "timeout" command with the
"--foreground" option. Without the "foreground" option, it is not
possible for a user to cancel the runner using SIGINT, because the
signal is not propagated to timeout which is running in a different
process group. The "forground" options places the timeout in the same
process group as its parent, but only sends the SIGTERM (on timeout)
signal to the forked process. Unfortunately, this does not play nice
with all kselftests, e.g. "net:fcnal-test.sh", where the child
processes will linger because timeout does not send SIGTERM to the
group.

Some users have noted these hangs [1].

Fix this by nesting the timeout with an additional timeout without the
foreground option.

Link: https://lore.kernel.org/all/7650b2eb-0aee-a2b0-2e64-c9bc63210f67@alu.unizg.hr/ # [1]
Fixes: 651e0d8814 ("kselftest/runner: allow to properly deliver signals to tests")
Signed-off-by: Björn Töpel <bjorn@rivosinc.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2023-09-08 10:06:11 -06:00
Ryan Roberts
58e2847ad2 selftests: line buffer test program's stdout
Patch series "selftests/mm fixes for arm64", v3.

Given my on-going work on large anon folios and contpte mappings, I
decided it would be a good idea to start running mm selftests to help
guard against regressions.  However, it soon became clear that I
couldn't get the suite to run cleanly on arm64 with a vanilla v6.5-rc1
kernel (perhaps I'm just doing it wrong??), so got stuck in a rabbit
hole trying to debug and fix all the issues.  Some were down to
misconfigurations, but I also found a number of issues with the tests
and even a couple of issues with the kernel.


This patch (of 8):

The selftests runner pipes the test program's stdout to tap_prefix.  The
presence of the pipe means that the test program sets its stdout to be
fully buffered (as aposed to line buffered when directly connected to the
terminal).  The block buffering means that there is often content in the
buffer at fork() time, which causes the output to end up duplicated.  This
was causing problems for mm:cow where test results were duplicated 20-30x.

Solve this by using `stdbuf`, when available to force the test program to
use line buffered mode.  This means previously printf'ed results are
flushed out of the program before any fork().

Additionally, explicitly set line buffer mode in ksft_print_header(),
which means that all test programs that use the ksft framework will
benefit even if stdbuf is not present on the system.

[ryan.roberts@arm.com: add setvbuf() to set buffering mode]
  Link: https://lkml.kernel.org/r/20230726070655.2713530-1-ryan.roberts@arm.com
Link: https://lkml.kernel.org/r/20230724082522.1202616-1-ryan.roberts@arm.com
Link: https://lkml.kernel.org/r/20230724082522.1202616-2-ryan.roberts@arm.com
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>
Reviewed-by: Mark Brown <broonie@kernel.org>
Cc: David Hildenbrand <david@redhat.com>
Cc: Florent Revest <revest@chromium.org>
Cc: Jérôme Glisse <jglisse@redhat.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-08-18 10:12:42 -07:00
Luis Chamberlain
f6a01213e3 selftests: allow runners to override the timeout
The default timeout for selftests tests is 45 seconds. Although
we already have 13 settings for tests of about 96 sefltests which
use a timeout greater than this, we want to try to avoid encouraging
more tests to forcing a higher test timeout as selftests strives to
run all tests quickly. Selftests also uses the timeout as a non-fatal
error. Only tests runners which have control over a system would know
if to treat a timeout as fatal or not.

To help with all this:

  o Enhance documentation to avoid future increases of insane timeouts
  o Add the option to allow overriding the default timeout with test
    runners with a command line option

Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2023-06-12 16:39:11 -06:00
Cristian Marussi
e005ff01bf selftests/kselftest/runner.sh: Pass optional command parameters in environment
Some testcases allow for optional commandline parameters but as of now
there is now way to provide such arguments to the runner script.

Add support to retrieve such optional command parameters fron environment
variables named so as to include the all-uppercase test executable name,
sanitized substituting any non-acceptable varname characters with "_",
following the pattern:

	KSELFTEST_<UPPERCASE_SANITIZED_TEST_NAME>_ARGS="options"

Optional command parameters support is not available if 'tr' is not
installed on the test system.

Cc: Kees Cook <keescook@chromium.org>
Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2022-02-23 17:14:36 -07:00
SeongJae Park
303f8e2d02 selftests/kselftest/runner/run_one(): allow running non-executable files
When running a test program, 'run_one()' checks if the program has the
execution permission and fails if it doesn't.  However, it's easy to
mistakenly lose the permissions, as some common tools like 'diff' don't
support the permission change well[1].  Compared to that, making mistakes
in the test program's path would only rare, as those are explicitly listed
in 'TEST_PROGS'.  Therefore, it might make more sense to resolve the
situation on our own and run the program.

For this reason, this commit makes the test program runner function still
print the warning message but to try parsing the interpreter of the
program and to explicitly run it with the interpreter, in this case.

[1] https://lore.kernel.org/mm-commits/YRJisBs9AunccCD4@kroah.com/

Link: https://lkml.kernel.org/r/20210810164534.25902-1-sj38.park@gmail.com
Signed-off-by: SeongJae Park <sjpark@amazon.de>
Suggested-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Shuah Khan <shuah@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2021-11-09 10:02:53 -08:00
Finn Behrens
c25ce589dc tweewide: Fix most Shebang lines
Change every shebang which does not need an argument to use /usr/bin/env.
This is needed as not every distro has everything under /usr/bin,
sometimes not even bash.

Signed-off-by: Finn Behrens <me@kloenk.de>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
2020-12-08 23:30:04 +09:00
Linus Torvalds
dd27111e32 Driver core changes for 5.9-rc1
Here is the "big" set of changes to the driver core, and some drivers
 using the changes, for 5.9-rc1.
 
 "Biggest" thing in here is the device link exposure in sysfs, to help
 to tame the madness that is SoC device tree representations and driver
 interactions with it.
 
 Other stuff in here that is interesting is:
 	- device probe log helper so that drivers can report problems in
 	  a unified way easier.
 	- devres functions added
 	- DEVICE_ATTR_ADMIN_* macro added to make it harder to write
 	  incorrect sysfs file permissions
 	- documentation cleanups
 	- ability for debugfs to be present in the kernel, yet not
 	  exposed to userspace.  Needed for systems that want it
 	  enabled, but do not trust users, so they can still use some
 	  kernel functions that were otherwise disabled.
 	- other minor fixes and cleanups
 
 The patches outside of drivers/base/ all have acks from the respective
 subsystem maintainers to go through this tree instead of theirs.
 
 All of these have been in linux-next with no reported issues.
 
 Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
 -----BEGIN PGP SIGNATURE-----
 
 iG0EABECAC0WIQT0tgzFv3jCIUoxPcsxR9QN2y37KQUCXylhOQ8cZ3JlZ0Brcm9h
 aC5jb20ACgkQMUfUDdst+ylGdACeKqxm8IIDZycj0QjLUlPiEwVIROgAnjpf5jAB
 mb4jMvgEGsB6/FwxypPG
 =RUss
 -----END PGP SIGNATURE-----

Merge tag 'driver-core-5.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core

Pull driver core updates from Greg KH:
 "Here is the "big" set of changes to the driver core, and some drivers
  using the changes, for 5.9-rc1.

  "Biggest" thing in here is the device link exposure in sysfs, to help
  to tame the madness that is SoC device tree representations and driver
  interactions with it.

  Other stuff in here that is interesting is:

   - device probe log helper so that drivers can report problems in a
     unified way easier.

   - devres functions added

   - DEVICE_ATTR_ADMIN_* macro added to make it harder to write
     incorrect sysfs file permissions

   - documentation cleanups

   - ability for debugfs to be present in the kernel, yet not exposed to
     userspace. Needed for systems that want it enabled, but do not
     trust users, so they can still use some kernel functions that were
     otherwise disabled.

   - other minor fixes and cleanups

  The patches outside of drivers/base/ all have acks from the respective
  subsystem maintainers to go through this tree instead of theirs.

  All of these have been in linux-next with no reported issues"

* tag 'driver-core-5.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core: (39 commits)
  drm/bridge: lvds-codec: simplify error handling
  drm/bridge/sii8620: fix resource acquisition error handling
  driver core: add deferring probe reason to devices_deferred property
  driver core: add device probe log helper
  driver core: Avoid binding drivers to dead devices
  Revert "test_firmware: Test platform fw loading on non-EFI systems"
  firmware_loader: EFI firmware loader must handle pre-allocated buffer
  selftest/firmware: Add selftest timeout in settings
  test_firmware: Test platform fw loading on non-EFI systems
  driver core: Change delimiter in devlink device's name to "--"
  debugfs: Add access restriction option
  tracefs: Remove unnecessary debug_fs checks.
  driver core: Fix probe_count imbalance in really_probe()
  kobject: remove unused KOBJ_MAX action
  driver core: Fix sleeping in invalid context during device link deletion
  driver core: Add waiting_for_supplier sysfs file for devices
  driver core: Add state_synced sysfs file for devices that support it
  driver core: Expose device link details in sysfs
  driver core: Drop mention of obsolete bus rwsem from kernel-doc
  debugfs: file: Remove unnecessary cast in kfree()
  ...
2020-08-05 11:52:17 -07:00
Kees Cook
4505e2154e selftest/firmware: Add selftest timeout in settings
The firmware tests would always time out for me. Add a correct timeout,
including details on how the value was reached. Additionally allow the
test harness to skip comments in settings files and report how long a
given timeout was.

Reviewed-by: SeongJae Park <sjpark@amazon.de>
Acked-by: Scott Branden <scott.branden@broadcom.com>
Signed-off-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20200724213640.389191-3-keescook@chromium.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2020-07-25 12:05:20 +02:00
Paolo Bonzini
b85d387c9b kselftest: fix TAP output for skipped tests
According to the TAP specification, a skipped test must be marked as "ok"
and annotated with the SKIP directive, for example

   ok 23 # skip Insufficient flogiston pressure.
   (https://testanything.org/tap-specification.html)

Fix the kselftest infrastructure to match this.

For ksft_exit_skip, it is preferrable to emit a dummy plan line that
indicates the whole test was skipped, but this is not always possible
because of ksft_exit_skip being used as a "shortcut" by the tests.
In that case, print the test counts and a normal "ok" line.  The format
is now the same independent of whether msg is NULL or not (but it is
never NULL in any caller right now).

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-07-06 15:17:59 -06:00
Andrea Righi
651e0d8814 kselftest/runner: allow to properly deliver signals to tests
While running seccomp_bpf, kill_after_ptrace() gets stuck if we run it
via /usr/bin/timeout (that is the default), until the timeout expires.

This is because /usr/bin/timeout is preventing to properly deliver
signals to ptrace'd children (SIGSYS in this case).

This problem can be easily reproduced by running:

 $ sudo make TARGETS=seccomp kselftest
 ...

 # [ RUN      ] TRACE_syscall.skip_a#
 not ok 1 selftests: seccomp: seccomp_bpf # TIMEOUT

The test is hanging at this point until the timeout expires and then it
reports the timeout error.

Prevent this problem by passing --foreground to /usr/bin/timeout,
allowing to properly deliver signals to children processes.

Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-04-14 09:49:13 -06:00
Matthieu Baerts
ac87813d43 selftests: settings: tests can be in subsubdirs
Commit 852c8cbf34 ("selftests/kselftest/runner.sh: Add 45 second
timeout per test") adds support for a new per-test-directory "settings"
file. But this only works for tests not in a sub-subdirectories, e.g.

 - tools/testing/selftests/rtc (rtc) is OK,
 - tools/testing/selftests/net/mptcp (net/mptcp) is not.

We have to increase the timeout for net/mptcp tests which are not
upstreamed yet but this fix is valid for other tests if they need to add
a "settings" file, see the full list with:

  tools/testing/selftests/*/*/**/Makefile

Note that this patch changes the text header message printed at the end
of the execution but this text is modified only for the tests that are
in sub-subdirectories, e.g.

  ok 1 selftests: net/mptcp: mptcp_connect.sh

Before we had:

  ok 1 selftests: mptcp: mptcp_connect.sh

But showing the full target name is probably better, just in case a
subsubdir has the same name as another one in another subdirectory.

Fixes: 852c8cbf34 (selftests/kselftest/runner.sh: Add 45 second timeout per test)
Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2020-01-27 10:49:52 -07:00
SeongJae Park
4eac734486 kselftest: Support old perl versions
On an old perl such as v5.10.1, `kselftest/prefix.pl` gives below error
message:

    Can't locate object method "autoflush" via package "IO::Handle" at kselftest/prefix.pl line 10.

This commit fixes the error by explicitly specifying the use of the
`IO::Handle` package.

Signed-off-by: SeongJae Park <sjpark@amazon.de>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-12-11 10:31:16 -07:00
SeongJae Park
d187801d1a kselftest/runner: Print new line in print of timeout log
If a timeout failure occurs, kselftest kills the test process and prints
the timeout log.  If the test process has killed while printing a log
that ends with new line, the timeout log can be printed in middle of the
test process output so that it can be seems like a comment, as below:

    # test_process_log	not ok 3 selftests: timers: nsleep-lat # TIMEOUT

This commit avoids such problem by printing one more line before the
TIMEOUT failure log.

Signed-off-by: SeongJae Park <sjpark@amazon.de>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-12-11 10:30:24 -07:00
Michael Ellerman
3a019636e6 selftests: Fix dangling documentation references to kselftest_module.sh
Commit c78fd76f2b ("selftests: Move kselftest_module.sh into
kselftest/") moved kselftest_module.sh but missed updating a few
references to the path in documentation.

Fixes: c78fd76f2b ("selftests: Move kselftest_module.sh into kselftest/")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-12-11 10:27:01 -07:00
Kees Cook
c78fd76f2b selftests: Move kselftest_module.sh into kselftest/
The kselftest_module.sh file was not being installed by the Makefile
"install" target, rendering the lib/*.sh tests nonfunction. This fixes
that and takes the opportunity to move it into the kselftest/ subdirectory
which is where the kselftest infrastructure bits are collecting.

Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>
Link: https://lore.kernel.org/lkml/CA+G9fYsfJpXQvOvHdjtg8z4a89dSStOQZOKa9zMjjQgWKng1aw@mail.gmail.com
Fixes: d346052770 ("kselftest: Add test runner creation script")
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-11-07 14:46:08 -07:00
Kees Cook
852c8cbf34 selftests/kselftest/runner.sh: Add 45 second timeout per test
Commit a745f7af3c ("selftests/harness: Add 30 second timeout per
test") solves the problem of kselftest_harness.h-using binary tests
possibly hanging forever. However, scripts and other binaries can still
hang forever. This adds a global timeout to each test script run.

To make this configurable (e.g. as needed in the "rtc" test case),
include a new per-test-directory "settings" file (similar to "config")
that can contain kselftest-specific settings. The first recognized field
is "timeout".

Additionally, this splits the reporting for timeouts into a specific
"TIMEOUT" not-ok (and adds exit code reporting in the remaining case).

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-10-01 13:33:28 -06:00
Kees Cook
e8108866ca selftests: Remove forced unbuffering for test running
As it turns out, the "stdbuf" command will actually force all
subprocesses into unbuffered output, and some implementations of "echo"
turn into single-character writes, which utterly wrecks writes to /sys
and /proc files.

Instead, drop the "stdbuf" usage, and for any tests that want explicit
flushing between newlines, they'll have to add "fflush(stdout);" as
needed.

Reported-by: Takashi Iwai <tiwai@suse.de>
Fixes: 5c069b6ded ("selftests: Move test output to diagnostic lines")
Signed-off-by: Kees Cook <keescook@chromium.org>
Tested-by: Takashi Iwai <tiwai@suse.de>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-05-21 09:24:30 -06:00
Kees Cook
f41c322f17 selftests: Remove KSFT_TAP_LEVEL
Since sub-testing can now be detected by indentation level, this removes
KSFT_TAP_LEVEL so that subtests report their TAP header for later parsing.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-04-25 13:15:26 -06:00
Kees Cook
5c069b6ded selftests: Move test output to diagnostic lines
This changes the selftest output so that each test's output is prefixed
with "# " as a TAP "diagnostic line".

This creates a bit of a kernel-specific TAP dialect where the diagnostics
precede the results. The TAP spec isn't entirely clear about this, though,
so I think it's the correct solution so as to keep interactive runs making
sense. If the output _followed_ the result line in the spec-suggested
YAML form, each test would dump all of its output at once instead of as
it went, making debugging harder.

This does, however, solve the recursive TAP output problem, as sub-tests
will simply be prefixed by "# ". Parsing sub-tests becomes a simple
problem of just removing the first two characters of a given top-level
test's diagnostic output, and parsing the results.

Note that the shell construct needed to both get an exit code from
the first command in a pipe and still filter the pipe (to add the "# "
prefix) uses a POSIX solution rather than the bash "pipefail" option
which is not supported by dash.

Since some test environments may have a very minimal set of utilities
available, the new prefixing code will fall back to doing line-at-a-time
prefixing if perl and/or stdbuf are not available.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-04-25 13:15:19 -06:00
Kees Cook
fd63b2eae5 selftests: Distinguish between missing and non-executable
If a test was missing (e.g. wrong architecture, etc), the test runner
would incorrectly claim the test was non-executable. This adds an
existence check to report correctly.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-04-25 13:15:10 -06:00
Kees Cook
b0df366bbd selftests: Add plan line and fix result line syntax
The TAP version 13 spec requires a "plan" line, which has been missing.
Since we always know how many tests we're going to run, emit the count on
the plan line. This also fixes the result lines to remove the "1.." prefix
which is against spec, and to mark skips with the correct "# SKIP" suffix.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-04-25 13:14:45 -06:00
Kees Cook
bf66078235 selftests: Extract logic for multiple test runs
This moves the logic for running multiple tests into a single "run_many"
function of runner.sh. Both "run_tests" and "emit_tests" are modified to
use it. Summary handling is now controlled by the "per_test_logging"
shell flag.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-04-25 13:14:38 -06:00
Kees Cook
42d46e57ec selftests: Extract single-test shell logic from lib.mk
In order to improve the reusability of the kselftest test running logic,
this extracts the single-test logic from lib.mk into kselftest/runner.sh
which lib.mk can call directly. No changes in output.

As part of the change, this moves the "summary" Makefile logic around
to set a new "logfile" output. This will be used again in the future
"emit_tests" target as well.

Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2019-04-25 13:14:13 -06:00