For those adding per check-in or nightly tests this is good stuff to read/understand. A
change was just merged that adds the following capability:
New tests that run some sort of CLI tool or existing tests that are running a CLI tool
(most everything we do) can all benefit from this as most of the time our test scripts
simply count on the CLI "not exploding" or returning with an exist code of 0/1
to determine if the test passed or not. With this new utility we can capture the expected
output from a CLI, fix it up so that it will work on any platform, and then confirm any
level of detail that we want after the tool runs. Please read through this and yell if
there are any questions either here or on IRC...
: (check out the match
source for more info, you can find it in /test/apps/match)
The 'match' util was borrowed from the PMDK folks after first
adding the "ignore" feature upstream to make it easier to use
in SPDK. It works like this:
When the developer checks in a test they create and check in
the output of the test with two different file extensions:
.ignore: should include a string per line for output lines
that we want to totally ignore typically because they're
platform specific so the output could be different from
machine to machine. In this case I'm ignoring all output
lines with 'DPDK' or 'EAL' or '...' in them. The first
few are obvious, the last is because the test tool will
print a varying number of these as progress indicators.
.match: this is a copy of the output that the developer
'fixes' up by replacing platform specific output strings
with replicable tokens as described in the 'match' help.
This is where you'd want to match an entire line minus
something like a CPU count or free block count or
something. The 'ignore' feature was added simply so we
wouldn't have to edit every single line of an output
file that had DPDK or EAL in it.
Then you modify the test script to save the output and
simply run the match util providing the name of the
match file and if it fails to match the actual output
with the saved output that's been token'ized the script
The obvious advantage here is that now we can confirm all
of the output from a test executable is as we expect.