[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Enabling/disabling tests by command line parameters


On Wednesday 27 January 2016 21:46:29 Pavel Kretov wrote:
> Hi all,

Hi,

> As far as I know from the reference manual and code, cmocka doesn't
> come with a build-in command line runner. What do you think about
> implementing one?

Jakub and I already wanted to do that but time didn't permit it yet. FOSDEM is 
ahead and we worked on getting pam_wrapper out before FOSDEM so we can do a 
presentation.

> It would be nice to be able to run only specific test, or prevent some
> tests from running. It may be valuable for, say, debugging or when
> some tests require additional hardware or software resources to be
> actually run (for example, testing OpenCL code is useless without
> having CL-capable hardware).

We discussed it the way that we create a function named e.g.

cmocka_set_test_filter()

In the internal test structure we add a new struct member called enabled. 
Based an what is set in the filter we enable or disable the tests to run.

> 
> Basically, I propose something like the following syntax:
> 
>   $ ./test                        # run all registered tests/groups;
>   $ ./test snafu tarfu            # run only 'snafu' and 'tarfu'; if
>                                   # any of them is a group, run all
> 				  # tests it contains;
>   $ ./test --without snafu tarfu  # run the rest of tests.
> 
> Specifying both whitelist and blacklist doesn't make much sense,
> unless you should run all test from a group without some specific
> tests:
> 
>   $ ./test snafu tarfu --without test_foobar_is_fubar
> 
> The exact switch name may vary: '--without', '--except', '--exclude',
> or their variants with a single dash. There may be even more command
> lines, for example, for choosing report output format.

We will not do command line argument parsing as this is different on each 
platform (Linux, BSD, Windows, embedded platforms) (popt, argp, ...).

You, the author of the test implements that parsind and just pass the right 
argument to the cmocka filter function.

> It seems to be easy to implement, and I'm going to do it for one
> of my projects. But isn't it better to have one "standardized" way for
> running specific tests built-in?
> 
> This runner may or may not rely on getopt for arguments parsing, it
> may even reside in a separate code and header files from the rest of
> library.
> 
> P.S.:
> BTW, the issue tracker have a similar feature request open [1].
> The idea is nice, but the implementation may require having an
> additional dependency to a regular expression or wildcard matching
> library.

Writing a simple pattern matching function which supports * and ? is not hard 
to do (~50 loc).


Cheers,


	-- andreas

-- 
Andreas Schneider                   GPG-ID: CC014E3D
www.cryptomilk.org                asn@xxxxxxxxxxxxxx

Follow-Ups:
Re: Enabling/disabling tests by command line parametersPavel Kretov <firegurafiku@xxxxxxxxx>
References:
Enabling/disabling tests by command line parametersPavel Kretov <firegurafiku@xxxxxxxxx>