Previously, using "%m" in a ksft_* format string can result in strange
output because the errno value wasn't saved before calling other libc
functions. The solution is to simply save and restore the errno before
we format the user-supplied format string.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
The test plan for TAP needs to be declared immediately after the header.
This adds the test plan API to kselftest.h and updates all callers to
declare their expected test counts.
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
This changes the selftest output so that each test's output is prefixed
with "# " as a TAP "diagnostic line".
This creates a bit of a kernel-specific TAP dialect where the diagnostics
precede the results. The TAP spec isn't entirely clear about this, though,
so I think it's the correct solution so as to keep interactive runs making
sense. If the output _followed_ the result line in the spec-suggested
YAML form, each test would dump all of its output at once instead of as
it went, making debugging harder.
This does, however, solve the recursive TAP output problem, as sub-tests
will simply be prefixed by "# ". Parsing sub-tests becomes a simple
problem of just removing the first two characters of a given top-level
test's diagnostic output, and parsing the results.
Note that the shell construct needed to both get an exit code from
the first command in a pipe and still filter the pipe (to add the "# "
prefix) uses a POSIX solution rather than the bash "pipefail" option
which is not supported by dash.
Since some test environments may have a very minimal set of utilities
available, the new prefixing code will fall back to doing line-at-a-time
prefixing if perl and/or stdbuf are not available.
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
The TAP version 13 spec requires a "plan" line, which has been missing.
Since we always know how many tests we're going to run, emit the count on
the plan line. This also fixes the result lines to remove the "1.." prefix
which is against spec, and to mark skips with the correct "# SKIP" suffix.
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
While playing around with a way to skip the seccomp get_metadata test, I
noticed that this header uses printf() without defining it, leading to,
../kselftest.h: In function ‘ksft_print_header’:
../kselftest.h:61:3: warning: implicit declaration of function ‘printf’ [-Wimplicit-function-declaration]
printf("TAP version 13\n");
^~~~~~
../kselftest.h:61:3: warning: incompatible implicit declaration of built-in function ‘printf’
../kselftest.h:61:3: note: include ‘<stdio.h>’ or provide a declaration of ‘printf’
if user code doesn't also use printf.
Signed-off-by: Tycho Andersen <tycho@tycho.ws>
Acked-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Shuah Khan <shuah@kernel.org>
Commit 3c07aaef65 ("selftests: kselftest: change KSFT_SKIP=4 instead of
KSFT_PASS") reverted commit 11867a77eb ("selftests: kselftest framework:
change skip exit code to 0") but missed removing the comment which that
commit added, so do that now.
Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com>
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
KSFT_SKIP points to KSFT_PASS resulting in reporting skipped tests as
Passed, when test programs exit with KSFT_SKIP or call ksft_exit_skip().
If tests are skipped because of unmet dependencies and/or unsupported
configuration, reporting them as passed leads to too many false positives.
Fix it to return a skip code of 4 to clearly differentiate the skipped
tests.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Introduce environment variable KSFT_TAP_LEVEL to avoid printing
nested TAP headers for each test. lib.mk run_tests target prints
TAP header before invoking the test program or test script. Tests
need a way to suppress TAP headers if it is already printed out.
This new environment variable adds a way for ksft_print_header()
print TAP header only when KSFT_TAP_LEVEL isn't set.
lib.mk run_tests and test program should print TAP header and set
KSFT_TAP_LEVEL to avoid a second TAP header to be printed.
selftests Makefile should export KSFT_TAP_LEVEL and add TAP Header
echo to the run_kselftest.sh script from emit_tests target handling.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
When a test is skipped, instead of using a special exit code of 4, treat
it as pass condition and use exit code of 0. It makes sense to treat skip
as pass since the test couldn't be run as opposed to a failed test.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Some tests track errors in addition to test failures. Add ksft_error
counter, ksft_get_error_cnt(), and ksft_test_result_error() API to
get the counter value and print error message.
Update ksft_print_cnts(), and ksft_test_num() to include error counter.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Some tests print final pass/fail message based on fail count. Add
ksft_get_*_cnt() API to kselftest framework to return counts.
Update ksft_print_cnts() to print the test results summary message with
individual pass, fail, ... counters.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Add a generic information output function: ksft_print_msg()
Signed-off-by: Paul Elder <paul.elder@pitt.edu>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Make the ksft_* output functions variadic to allow string formatting
directly in these functions.
Signed-off-by: Paul Elder <paul.elder@pitt.edu>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Make ksft_exit_skip() input an optional message string as the reason
for skipping all the tests and outputs it prior to exiting.
Signed-off-by: Paul Elder <paul.elder@pitt.edu>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Add TAP13 conformat output functions to kselftest.h.
Also add exit functions that output TAP13 exiting text, as well as
functions to keep track of testing progress.
Signed-off-by: Paul Elder <paul.elder@pitt.edu>
Signed-off-by: Alice Ferrazzi <alice.ferrazzi@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Define the exit codes with KSFT_PASS and similar so tests can use these
directly if they choose. Also enable harnesses and other tooling to use
the defines instead of hardcoding the return codes.
Cc: Shuah Khan <shuahkh@osg.samsung.com>
Cc: linux-api@vger.kernel.org
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Add kselftest framework for tests to use. This is a light
weight framework provides a set of interfaces to report test
results. Tests can use these interfaces to report pass, and
fail cases as well as when failure is due to configuration
problems such as missing modules, or when a test that is should
fail, fails as expected, and a test that should fail, passes.
The framework uses POSIX standard return codes for reporting
results to address the needs of users that want to run the kernel
selftests from their user-space test suites and want to know why a
test failed. In addition, the framework includes interfaces to use
to report test statistics on number of tests passed and failed.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
