› Code Snippets › Shell helper: running KDE unit tests (ctests) the easy way Syndicate content

Thu, 03/26/2009 - 03:09

Unit tests are in my eyes a very important part of programming. KDE uses them, KDevelop does - the PHP plugin I help writing does as well. cmake comes with a ctest program which does quite well to give you a quick glance on which test suite you just broke with your new fance feature :)

But I am very dissatisfied with it. Right now I usually do the following

  1. # lets assume I'm in the source directory
  2. cb && ctest
  3. # look for failed test suites
  4. cd $failed_test_suite_path
  5. ./$failed_test_suite.shell | less
  6. # search for FAIL
  7. cs
  8. cd $to_whereever_I_was_before

That’s pretty much for just running a test. Especially all that cding and lessing became very tedious. Tedious is good, because I eventually fix it:

introducing kdetest

I wrote a bash function (with autocompletion!!!) called kdetest. Calling it without any parameter will run all test suites and gives a nice report of failed functions at the end. Here’s an example (run via cs php && kdetest).

  1. kdetest
  2. # ... lots of test output
  3.  
  4.  --- ALL PASSED TESTS ---
  5. ...
  6. PASS   : Php::TestCompletion::implementMethods()
  7. PASS   : Php::TestCompletion::inArray()
  8. PASS   : Php::TestCompletion::cleanupTestCase()
  9.  
  10. 143 passed tests in total
  11.  
  12.  --- ALL FAILED TESTS ---
  13. FAIL!  : Php::TestCompletion::newExtends() Compared values are not the same
  14. FAIL!  : Php::TestCompletion::updateExtends() '! forbiddenIdentifiers.contains(item->declaration()->identifier().toString())' returned FALSE. ()
  15. FAIL!  : Php::TestCompletion::updateExtends() '! forbiddenIdentifiers.contains(item->declaration()->identifier().toString())' returned FALSE. ()
  16. FAIL!  : Php::TestCompletion::updateExtends() Compared values are not the same
  17. FAIL!  : Php::TestCompletion::newImplements() Compared values are not the same
  18. FAIL!  : Php::TestCompletion::updateImplements() Compared values are not the same
  19.  
  20. 6 failed tests in total
usage
  • kdetest, i.e. without any arguments runs all tests in this directory and below
  • kdetest path/to/test.shell ... runs that test suite only, ... can by any argument the test suite accepts.
autocompletion

kdetest comes with full support for autocompletion of tests and functions, for example:

  1. milian@odin:~/projects/kde4/php$ kdetest TABTAB
  2. completion/tests/completiontest.shell     duchain/tests/expressionparsertest.shell  parser/test/lexertest.shell
  3. duchain/tests/duchaintest.shell           duchain/tests/usestest.shell
  4. milian@odin:~/projects/kde4/php$ kdetest duchain/tests/usestest.shell TABTAB
  5. classAndConstWithSameName          classSelf                          interfaceExtendsMultiple           staticMemberFunctionCall
  6. classAndFunctionWithSameName       constAndVariableWithSameName       memberFunctionCall                 staticMemberVariable
  7. classConstant                      constant                           memberFunctionInString             variable
  8. classExtends                       constantInClassMember              memberVariable                     variableTwoDeclarations
  9. classImplements                    functionAndClassWithSameName       memberVarInString                  variableTwoDeclarationsInFunction
  10. classImplementsMultiple            functionCall                       newObject                          varInString
  11. classParent                        interfaceExtends                   objectWithClassName
the code

You can find the code below, or you can obtain the most up-to-date version on github. Just head over to my shell-helpers repo and peek into the bash_setup_kde4_programming file.

  1. # run a given unit-test or all via ctest
  2. function kdetest
  3. {
  4.   local tests test args old_pwd tmpfile;
  5.  
  6.   old_pwd=$(pwd)
  7.   cb
  8.  
  9.   tests=$(LANG=en_US.UTF-8 ctest -N -V | grep "Test command:" | cut -c $(echo "Test command: $(pwd)/" | wc -c)-)
  10.  
  11.   if [[ "$tests" == "" ]]; then
  12.     echo "this directory does not contain any unit tests!"
  13.     echo
  14.     cd "$old_pwd"
  15.     return 1
  16.   fi
  17.  
  18.   tmpfile=/tmp/testoutput_$$
  19.  
  20.   if [[ "$1" != "" ]]; then
  21.     test=$1
  22.     shift 1
  23.     args=$@
  24.  
  25.     if [ ! -f "$test" ] || ! in_array "$test" $tests ; then
  26.       echo "could not find unittest '$test'. available are:"
  27.       echo $tests
  28.       echo
  29.       cd "$old_pwd"
  30.       return 1
  31.     fi
  32.  
  33.     ./$test -maxwarnings 0 $args | tee -a "$tmpfile"
  34.     echo
  35.   else
  36.     # run all tests
  37.     for test in $tests; do
  38.       ./$test -maxwarnings 0 | tee -a "$tmpfile"
  39.     done
  40.   fi
  41.  
  42.   echo
  43.   echo " --- ALL PASSED TESTS --- "
  44.   grep --color=never "^PASS " "$tmpfile"
  45.   echo 
  46.   echo $(grep -c "^PASS " "$tmpfile")" passed tests in total"
  47.  
  48.   echo
  49.   echo " --- ALL FAILED TESTS --- "
  50.   grep --color=never "^FAIL!" "$tmpfile"
  51.   echo
  52.   echo $(grep -c "^FAIL!" "$tmpfile")" failed tests in total"
  53.  
  54.   rm "$tmpfile"
  55.  
  56.   cd "$old_pwd"
  57. }
  58.  
  59. # completion for kdetest
  60. function _kdetest
  61. {
  62.   local tests;
  63.  
  64.   old_pwd=$(pwd)
  65.  
  66.   cb
  67.   tests=$(LANG=en_US.UTF-8 ctest -N -V | grep "Test command:" | cut -c $(echo "Test command: $(pwd)/" | wc -c)-)
  68.  
  69.   COMPREPLY=()
  70.   cur="${COMP_WORDS[COMP_CWORD]}"
  71.   prev="${COMP_WORDS[COMP_CWORD-1]}"
  72.  
  73.   if [[ "$prev" == "kdetest" ]]; then
  74.     # completion of tests
  75.     COMPREPLY=( $(compgen -W "${tests}" -- ${cur}) )
  76.   elif in_array "$prev" $tests; then
  77.     # completion of available functions
  78.     COMPREPLY=( $(compgen -W "$(./$prev -functions 2>/dev/null | cut -f 1 -d \( )" -- ${cur}) )
  79.   fi
  80.  
  81.   cd "$old_pwd"
  82. }
  83.  
  84. complete -F _kdetest kdetest
  85.  
  86. # see http://ftp.hu.freebsd.org/pub/linux/distributions/frugalware/frugalware-testing/t/functions.sh
  87. function in_array
  88. {
  89.   local i
  90.   needle=$1
  91.   shift 1
  92.   # array() undefined
  93.   [ -z "$1" ] && return 1
  94.   for i in $*
  95.   do
  96.     [ "$i" == "$needle" ] && return 0
  97.   done
  98.   return 1
  99. }

Comments

cool - sound Thu, 03/26/2009 - 08:12 — Niko (not verified)

cool - sound useful! (especially the tab-completion :D)

You know that ctest puts the Thu, 03/26/2009 - 11:42 — Alexander Neundorf (not verified)

You know that ctest puts the output from all tests into <builddir>/Testing/Temporary/LastTest.log ?

I.e. you don’t have to rerun the test to see the output.

If you run ctest with “-V” (verbose) you will also get more output while the tests are running.

Is the shell function you wrote in any way KDE-specific ? If not, why not renaming it so other see it doesn’t have any KDE dependencies and maybe put it in the cmake wiki ?

Or could ctest be enhanced so it does what you want ?

Alex

Nope, I did not know that Thu, 03/26/2009 - 12:05 — Milian Wolff

Nope, I did not know that LastTest.log existed. I might be able to leverage that in my script. Yet the log alone won’t help me much since manually doing less <builddir>/Testing/Temporary/LastTest.log everytime ctest reports a failed testsuite is still a royal PITA :)

As far as I remember ctest -V still does not output theQDEBUG lines for example - often essential to understand what’s going wrong. Or something else was not satisfying, don’t remember - but I’ve checked that flag.

And kdetest is KDE-specific in the way that it depends on cb, i.e. on a build environment setup for KDE as described on techbase. You could of course build non-KDE programs in that environment.

Improving ctest would be the best, but I doubt I could have been able to get to a point near what kdetest does in around an hour.

Post new comment

The content of this field is kept private and will not be shown publicly.
  • You can use Markdown syntax to format and style the text. Also see Markdown Extra for tables, footnotes, and more.
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>, <pre>. The supported tag styles are: <foo>, [foo]. PHP source code can also be enclosed in <?php ... ?> or <% ... %>.
  • Web page addresses and e-mail addresses turn into links automatically.

More information about formatting options