MW
Welcome Blog Code Snippets Contact

Shell helper: running KDE unit tests (ctests) the easy way

Unit tests are in my eyes a very important part of programming. KDE uses them, KDevelop does - the PHP plugin I help writing does as well. cmake comes with a ctest program which does quite well to give you a quick glance on which test suite you just broke with your new fance feature :)

But I am very dissatisfied with it. Right now I usually do the following

    # lets assume I'm in the source directory
    cb && ctest
    # look for failed test suites
    cd $failed_test_suite_path
    ./$failed_test_suite.shell | less
    # search for FAIL
    cs
    cd $to_whereever_I_was_before

That’s pretty much for just running a test. Especially all that cding and lessing became very tedious. Tedious is good, because I eventually fix it:

introducing kdetest

I wrote a bash function (with autocompletion!!!) called kdetest. Calling it without any parameter will run all test suites and gives a nice report of failed functions at the end. Here’s an example (run via cs php && kdetest).

    kdetest
    # ... lots of test output
     
     --- ALL PASSED TESTS ---
    ...
    PASS   : Php::TestCompletion::implementMethods()
    PASS   : Php::TestCompletion::inArray()
    PASS   : Php::TestCompletion::cleanupTestCase()
     
    143 passed tests in total
     
     --- ALL FAILED TESTS ---
    FAIL!  : Php::TestCompletion::newExtends() Compared values are not the same
    FAIL!  : Php::TestCompletion::updateExtends() '! forbiddenIdentifiers.contains(item->declaration()->identifier().toString())' returned FALSE. ()
    FAIL!  : Php::TestCompletion::updateExtends() '! forbiddenIdentifiers.contains(item->declaration()->identifier().toString())' returned FALSE. ()
    FAIL!  : Php::TestCompletion::updateExtends() Compared values are not the same
    FAIL!  : Php::TestCompletion::newImplements() Compared values are not the same
    FAIL!  : Php::TestCompletion::updateImplements() Compared values are not the same
     
    6 failed tests in total

usage

autocompletion

kdetest comes with full support for autocompletion of tests and functions, for example:

    milian@odin:~/projects/kde4/php$ kdetest TABTAB
    completion/tests/completiontest.shell     duchain/tests/expressionparsertest.shell  parser/test/lexertest.shell
    duchain/tests/duchaintest.shell           duchain/tests/usestest.shell
    milian@odin:~/projects/kde4/php$ kdetest duchain/tests/usestest.shell TABTAB
    classAndConstWithSameName          classSelf                          interfaceExtendsMultiple           staticMemberFunctionCall
    classAndFunctionWithSameName       constAndVariableWithSameName       memberFunctionCall                 staticMemberVariable
    classConstant                      constant                           memberFunctionInString             variable
    classExtends                       constantInClassMember              memberVariable                     variableTwoDeclarations
    classImplements                    functionAndClassWithSameName       memberVarInString                  variableTwoDeclarationsInFunction
    classImplementsMultiple            functionCall                       newObject                          varInString
    classParent                        interfaceExtends                   objectWithClassName

the code

You can find the code below, or you can obtain the most up-to-date version on github. Just head over to my shell-helpers repo and peek into the bash_setup_kde4_programming file.

    # run a given unit-test or all via ctest
    function kdetest
    {
      local tests test args old_pwd tmpfile;
     
      old_pwd=$(pwd)
      cb
     
      tests=$(LANG=en_US.UTF-8 ctest -N -V | grep "Test command:" | cut -c $(echo "Test command: $(pwd)/" | wc -c)-)
     
      if [[ "$tests" == "" ]]; then
        echo "this directory does not contain any unit tests!"
        echo
        cd "$old_pwd"
        return 1
      fi
     
      tmpfile=/tmp/testoutput_$$
     
      if [[ "$1" != "" ]]; then
        test=$1
        shift 1
        args=$@
     
        if [ ! -f "$test" ] || ! in_array "$test" $tests ; then
          echo "could not find unittest '$test'. available are:"
          echo $tests
          echo
          cd "$old_pwd"
          return 1
        fi
     
        ./$test -maxwarnings 0 $args | tee -a "$tmpfile"
        echo
      else
        # run all tests
        for test in $tests; do
          ./$test -maxwarnings 0 | tee -a "$tmpfile"
        done
      fi
     
      echo
      echo " --- ALL PASSED TESTS --- "
      grep --color=never "^PASS " "$tmpfile"
      echo 
      echo $(grep -c "^PASS " "$tmpfile")" passed tests in total"
     
      echo
      echo " --- ALL FAILED TESTS --- "
      grep --color=never "^FAIL!" "$tmpfile"
      echo
      echo $(grep -c "^FAIL!" "$tmpfile")" failed tests in total"
     
      rm "$tmpfile"
     
      cd "$old_pwd"
    }
     
    # completion for kdetest
    function _kdetest
    {
      local tests;
     
      old_pwd=$(pwd)
     
      cb
      tests=$(LANG=en_US.UTF-8 ctest -N -V | grep "Test command:" | cut -c $(echo "Test command: $(pwd)/" | wc -c)-)
     
      COMPREPLY=()
      cur="${COMP_WORDS[COMP_CWORD]}"
      prev="${COMP_WORDS[COMP_CWORD-1]}"
     
      if [[ "$prev" == "kdetest" ]]; then
        # completion of tests
        COMPREPLY=( $(compgen -W "${tests}" -- ${cur}) )
      elif in_array "$prev" $tests; then
        # completion of available functions
        COMPREPLY=( $(compgen -W "$(./$prev -functions 2>/dev/null | cut -f 1 -d \( )" -- ${cur}) )
      fi
     
      cd "$old_pwd"
    }
     
    complete -F _kdetest kdetest
     
    # see http://ftp.hu.freebsd.org/pub/linux/distributions/frugalware/frugalware-testing/t/functions.sh
    function in_array
    {
      local i
      needle=$1
      shift 1
      # array() undefined
      [ -z "$1" ] && return 1
      for i in $*
      do
        [ "$i" == "$needle" ] && return 0
      done
      return 1
    }

Comments

Want to comment? Send me an email!

Comment by Alexander Neundorf (not verified) (2009-03-26 11:42:00)

You know that ctest puts the output from all tests into /Testing/Temporary/LastTest.log ?

I.e. you don’t have to rerun the test to see the output.

If you run ctest with “-V” (verbose) you will also get more output while the tests are running.

Is the shell function you wrote in any way KDE-specific ? If not, why not renaming it so other see it doesn’t have any KDE dependencies and maybe put it in the cmake wiki ?

Or could ctest be enhanced so it does what you want ?

Alex

Comment by Milian Wolff (2009-03-26 12:05:00)

Nope, I did not know that LastTest.log existed. I might be able to leverage that in my script. Yet the log alone won’t help me much since manually doing less <builddir>/Testing/Temporary/LastTest.log everytime ctest reports a failed testsuite is still a royal PITA :)

As far as I remember ctest -V still does not output theQDEBUG lines for example - often essential to understand what’s going wrong. Or something else was not satisfying, don’t remember - but I’ve checked that flag.

And kdetest is KDE-specific in the way that it depends on cb, i.e. on a build environment setup for KDE as described on techbase. You could of course build non-KDE programs in that environment.

Improving ctest would be the best, but I doubt I could have been able to get to a point near what kdetest does in around an hour.

Comment by Niko (not verified) (2009-03-26 08:12:00)

cool - sound useful! (especially the tab-completion :D)

Published on March 26, 2009.

Tags: