Code Snippets Syndicate content

  • » A git hook to prevent pushes with untracked source files

    Fri, 08/08/2014 - 13:40

    Hey all,

    do you know this: You work on something locally in git, ensure everything compiles and the tests pass, then commit and hit git push.What could possibly go wrong at that point, eh? Well, far too often I forgot to git add some new source file. Best-case I’ll notice this directly, worst-case I’ll see my CI complaining. But, like yesterday in kdev-clang, I might be afk at that point and someone else will have to revert my change and I’ll have to fix it up the day after, polluting the git history while at it…

    Thanks to some simple shell scripting and the powerful git hook architecture, it is pretty simple to protect oneself against such issues:

    1. #!/bin/sh
    2.  
    3. #
    4. # A hook script to verify that a push is not done with untracked source file
    5. #
    6. # To use it, either symlink this script to $your-git-clone/.git/hooks/pre-push
    7. # or include it in your existing pre-push script.
    8. #
    9.  
    10. # Perl-style regular expression which limits the files we interpret as source files.
    11. # The default pattern here excludes CMakeLists.txt files and any .h/.cpp/.cmake files.
    12. # Extend/adapt this to your needs. Alternatively, set the pattern in your repo via:
    13. # git config hooks.prepush.sourcepattern "$your-pattern"
    14. pattern=$(git config --get hooks.prepush.sourcepattern)
    15. if [ -z "$pattern" ]; then
    16. pattern="(?:(?:^|/)CMakeLists\.txt|\.h|\.cpp|\.cmake)$"
    17. fi
    18.  
    19. files=$(git status -u --porcelain --no-column | sed "s/^?? //" | grep -P "$pattern")
    20. if [ -z "$files" ]; then
    21. exit 0
    22. fi
    23.  
    24. echo
    25. echo "ERROR: Preventing push with untracked source files:"
    26. echo
    27. echo "$files" | sed "s/^/ /"
    28. echo
    29. echo "Either include these files in your commits, add them to .gitignore"
    30. echo "or stash them with git stash -u."
    31. echo
    32. exit 1

    Note: The last version of the above code can be found on GitHub: pre-push-check-untracked

    When you then try to push with some untracked source files, i.e. files matched by the regex pattern which can be configured via git config hooks.prepush.sourcepattern, you’ll see output like this:

    1. # to show the current status, note the untracked bar file which is not included as a source file
    2. $ git status
    3. On branch master
    4. Your branch is up-to-date with 'origin/master'.
    5.  
    6. Untracked files:
    7. (use "git add <file>..." to include in what will be committed)
    8.  
    9. CMakeLists.txt
    10. bar
    11. foo/
    12.  
    13. nothing added to commit but untracked files present (use "git add" to track)
    14.  
    15. # now try to push something
    16.  
    17. $ git push
    18.  
    19. ERROR: Preventing push with untracked source files:
    20.  
    21. CMakeLists.txt
    22. foo/asdf.h
    23.  
    24. Either include these files in your commits, add them to .gitignore
    25. or stash them with git stash -u.
    26.  
    27. error: failed to push some refs to '/tmp/repo'

    Happy hacking!

  • » Maxwell distribution in C++11

    Tue, 07/29/2014 - 11:51

    Hey all,

    I recently needed to generate values following a Maxwell distribution. Wikipedia gives the hint that this is a Gamma distribution, which in C++11 is easily useable. Thus, thanks to <random>, we can setup a Maxwell distribution in a few lines of code:

    1. #include <random>
    2. #include <chrono>
    3. #include <iostream>
    4.  
    5. using namespace std;
    6.  
    7. int main()
    8. {
    9. unsigned seed = chrono::system_clock::now().time_since_epoch().count();
    10. default_random_engine generator(seed);
    11.  
    12. // Boltzmann factor times temperature
    13. const double k_T = 0.1;
    14.  
    15. // setup the Maxwell distribution, i.e. gamma distribution with alpha = 3/2
    16. gamma_distribution<double> maxwell(3./2., k_T);
    17.  
    18. // generate Maxwell-distributed values
    19. for (int i = 0; i < 10000; ++i) {
    20. cout << maxwell(generator) << endl;
    21. }
    22.  
    23. return 0;
    24. }

    Pretty neat, I have to say!

  • » Git commit message highlighting in nano

    Sat, 11/24/2012 - 16:07

    For those who use nano as their CLI editor of choice: Here’s a syntax highlighting file for Git commit messages which also supports the special KDE commit hook keywords.

    1. ## syntax highlighting for git commit messages of KDE projects
    2. syntax "patch" ".git/COMMIT_EDITMSG$"
    3.  
    4. # overlong lines
    5. color brightred "^.{70,}.+$"
    6.  
    7. # KDE commit hook keywords, see: http://community.kde.org/Sysadmin/GitKdeOrgManual#Commit_hook_keywords
    8. color yellow "^(FEATURE|BUG|CCBUG|FIXED-IN|CCMAIL|REVIEW|GUI|DIGEST):.*$"
    9. color yellow "(SVN_SILENT|GIT_SILENT|SVN_MERGE)"
    10.  
    11. # comment
    12. color blue "^#.*$"
    13.  
    14. # special comment lines
    15. color green "^# Changes to be committed:"
    16. color red "^# Changes not staged for commit:"
    17. color brightblue "^# Untracked files:"
    18. color brightblue "^# On branch .+$"
    19. color brightblue "^# Your branch is ahead of .+$"
    20.  
    21. # diff files
    22. # meh - cannot match against \t ... should be: ^#\t.*$
    23. color cyan "^#[^ a-zA-Z0-9][^ ].*$"

    Just put this file as gitcommit.nanorc or similar somewhere on your disk and load it from your .nanorc or even /etc/nanorc like this:

    1. include "/path/to/gitcommit.nanorc"
    Screenshot of highlighted git commit message
    Screenshot of highlighted git commit message
  • » Tracking Memory Consumption Using Pmap

    Fri, 03/16/2012 - 14:55

    Massif is a really nifty tool which is very powerful, especially paired with my visualizer. The caveat of course is that it slows down the application considerably, I’ve seen anything up to a factor of 100… I see no alternative to Massif when it comes to investigating where your memory problems come from. But if you just want to see whether you have a problem at all, tracking the total memory consumption should suffice.

    A few days ago, I came across pmap on Stack Overflow, which makes it easy to track the RSS memory consumption of an application using the -x switch. Of course I had to write some bash magic to automate this process and visualize the data using Gnuplot! Behold:

    memory consumption of PhantomJS
    memory consumption of a PhantomJS script over ~30min
    usage

    It’s simple, really: track_memory.sh $(pidof myapp).

    The default timeout is ~1s between snapshots, you can pass a different timeout as second parameter. Bash’s sleep can also take float numbers such as 0.1 to get more snapshots for fast-running apps.

    You can also run show_memory.sh mem.log.$(pidof myapp) while you are still tracking the memory. The gnuplot window that appears allows you to update the data intermittently, to zoom in and to create images such as the above.

    Note: This kind of memory usage tracking costs nearly nothing, your application continues to work at full speed. Also be aware that this just shows the RSS memory consumption. Massif will always give you better, more detailed and accurate results. Still, I think this should already give you an idea on how your application behaves. If the graph goes up and up, you probably got a memory leak! Then it’s time to run Memcheck and/or Massif to find the issue and fix it!

    track_memory.sh

    You can find the most-recent version on GitHub: https://github.com/milianw/shell-helpers/blob/master/track_memory.sh

    1. #!/bin/bash
    2.  
    3. #
    4. # track memory of given application, identified by PID,
    5. # using pmap -x, to show RSS and Dirty memory usage.
    6. #
    7. # visualization can later on be done with the
    8. # show_memory.sh script.
    9. #
    10.  
    11. pid=$1
    12. sleep=$2;
    13.  
    14. if [[ "$sleep" == "" ]]; then
    15. sleep=1
    16. fi
    17.  
    18. if [[ "$(ps -p $pid | grep $pid)" == "" ]]; then
    19. echo "cannot find program with pid $pid"
    20. echo "track_memory.sh PID [SLEEP_TIMEOUT]"
    21. echo
    22. echo "example: track_memory.sh \$(pidof someapp) 0.1"
    23. exit
    24. fi
    25.  
    26. logfile=mem.log.$pid
    27.  
    28. echo "# $(ps -o command= -p $pid)" > $logfile
    29. echo "# $sleep" >> $logfile
    30.  
    31. cat $logfile
    32.  
    33. while [[ "$(ps -p $pid | grep $pid)" != "" ]]; do
    34. echo "snapshot " $pid
    35. pmap -x $pid | tail -n1 >> $logfile
    36. echo "$sleep"
    37. sleep $sleep;
    38. done
    39.  
    40. echo "done tracking, visualizing"
    41. $(dirname $0)/show_memory.sh "$logfile"
    show_memory.sh

    You can find the most-recent version on GitHub: https://github.com/milianw/shell-helpers/blob/master/show_memory.sh

    1. #!/bin/bash
    2.  
    3. #
    4. # visualize memory consumption over time
    5. # as recorded by pmap / track_memory.sh
    6. # script
    7. #
    8.  
    9. logfile=$1
    10.  
    11. if [ ! -f "$logfile" ]; then
    12. echo "cannot find memory logfile: $1"
    13. echo
    14. echo "usage: show_memory.sh LOGFILE"
    15. echo
    16. echo "example: show_memory.sh mem.log.12345"
    17. exit
    18. fi
    19.  
    20. title=$(head -n1 "$logfile")
    21. timeout=$(head -n2 "$logfile" | tail -n1)
    22.  
    23. title=${title/\# /}
    24. timeout=${timeout/\# /}
    25.  
    26. # total:
    27. # '$logfile' using 3 w lines title 'Kbytes', \
    28.  
    29. gnuplot -p -e "
    30. set title '$title';
    31. set xlabel 'snapshot ~${timeout}s';
    32. set ylabel 'memory consumption in kB';
    33. set key bottom right;
    34. plot \
    35. '$logfile' using 4 w lines title 'RSS' lt 1, \
    36. '$logfile' using 4 smooth bezier w lines title 'RSS (smooth)' lt 7, \
    37. '$logfile' using 5 w lines title 'Dirty' lt 2, \
    38. '$logfile' using 5 smooth bezier w lines title 'Dirty (smooth)' lt 3;
    39. ";
    Future

    The above is nice, but I’m wondering on whether one should not add this kind of utility to ksysguard: It already allows you to track the total memory consumption of your system, yet I did not find a way to just track a single application and visualize it’s memory consumption.

  • » Qt: Tint alphatransparent PNG

    Wed, 09/21/2011 - 18:07

    Let’s assume you want to display the logo of your company in your Qt app. Most probably that logo has just single color with an alpha channel.But: Having the color hard coded in the image is not nice, there are users (like me!) out there, who use a custom (dark!) color scheme. Meaning: If your logo is black/dark and assumes a bright background and you just embed it blindly in your app, I probably won’t see it since the background will be dark in my case.

    Here is a solution for the simple case of a mono-colored PNG with an alpha channel which I came up with:

    1. QLabel* label = new QLabel;
    2. // load your image
    3. QImage img(QString("..."));
    4. // morph it into a grayscale image
    5. img = img.alphaChannel();
    6. // the new color we want the logo to have
    7. QColor foreground = label->palette().foreground().color();
    8. // now replace the colors in the image
    9. for(int i = 0; i < img.colorCount(); ++i) {
    10. foreground.setAlpha(qGray(img.color(i)));
    11. img.setColor(i, foreground.rgba());
    12. }
    13. // display the new logo
    14. label->setPixmap(QPixmap::fromImage(img));
    15. label->show();

    This seems to work just fine for me. YMMV.

  • » Shell helper: running KDE unit tests (ctests) the easy way

    Thu, 03/26/2009 - 03:09

    Unit tests are in my eyes a very important part of programming. KDE uses them, KDevelop does - the PHP plugin I help writing does as well. cmake comes with a ctest program which does quite well to give you a quick glance on which test suite you just broke with your new fance feature :)

    But I am very dissatisfied with it. Right now I usually do the following

    1. # lets assume I'm in the source directory
    2. cb && ctest
    3. # look for failed test suites
    4. cd $failed_test_suite_path
    5. ./$failed_test_suite.shell | less
    6. # search for FAIL
    7. cs
    8. cd $to_whereever_I_was_before

    That’s pretty much for just running a test. Especially all that cding and lessing became very tedious. Tedious is good, because I eventually fix it:

    introducing kdetest

    I wrote a bash function (with autocompletion!!!) called kdetest. Calling it without any parameter will run all test suites and gives a nice report of failed functions at the end. Here’s an example (run via cs php && kdetest).

    1. kdetest
    2. # ... lots of test output
    3.  
    4. --- ALL PASSED TESTS ---
    5. ...
    6. PASS : Php::TestCompletion::implementMethods()
    7. PASS : Php::TestCompletion::inArray()
    8. PASS : Php::TestCompletion::cleanupTestCase()
    9.  
    10. 143 passed tests in total
    11.  
    12. --- ALL FAILED TESTS ---
    13. FAIL! : Php::TestCompletion::newExtends() Compared values are not the same
    14. FAIL! : Php::TestCompletion::updateExtends() '! forbiddenIdentifiers.contains(item->declaration()->identifier().toString())' returned FALSE. ()
    15. FAIL! : Php::TestCompletion::updateExtends() '! forbiddenIdentifiers.contains(item->declaration()->identifier().toString())' returned FALSE. ()
    16. FAIL! : Php::TestCompletion::updateExtends() Compared values are not the same
    17. FAIL! : Php::TestCompletion::newImplements() Compared values are not the same
    18. FAIL! : Php::TestCompletion::updateImplements() Compared values are not the same
    19.  
    20. 6 failed tests in total
    usage
    • kdetest, i.e. without any arguments runs all tests in this directory and below
    • kdetest path/to/test.shell ... runs that test suite only, ... can by any argument the test suite accepts.
    autocompletion

    kdetest comes with full support for autocompletion of tests and functions, for example:

    1. milian@odin:~/projects/kde4/php$ kdetest TABTAB
    2. completion/tests/completiontest.shell duchain/tests/expressionparsertest.shell parser/test/lexertest.shell
    3. duchain/tests/duchaintest.shell duchain/tests/usestest.shell
    4. milian@odin:~/projects/kde4/php$ kdetest duchain/tests/usestest.shell TABTAB
    5. classAndConstWithSameName classSelf interfaceExtendsMultiple staticMemberFunctionCall
    6. classAndFunctionWithSameName constAndVariableWithSameName memberFunctionCall staticMemberVariable
    7. classConstant constant memberFunctionInString variable
    8. classExtends constantInClassMember memberVariable variableTwoDeclarations
    9. classImplements functionAndClassWithSameName memberVarInString variableTwoDeclarationsInFunction
    10. classImplementsMultiple functionCall newObject varInString
    11. classParent interfaceExtends objectWithClassName
    the code

    You can find the code below, or you can obtain the most up-to-date version on github. Just head over to my shell-helpers repo and peek into the bash_setup_kde4_programming file.

  • » Attack of the shell helpers

    Mon, 03/02/2009 - 02:46

    Everyone who uses the command line regularly has a bunch of (at least for him) useful helper scripts. I now took the liberty to put mine on github for general consumption.

    You can find them at: http://github.com/milianw/shell-helpers/tree/master

    Some of these might be more useful than others, you decide :) Personally, I can’t live without the following:

    apupgrade
    a shortcut to update your Debian system with one command - no questions asked
    openurl
    opens a given URL in an already opened browser instance or starts a new browser session. Not only one browser is checked. I use it because firefox is slow to start and konqueror is blazingly fast to start. But when firefox is already open I want to use that.
    xerr
    shortcut for fast error checking in your Xorg log
    clipboard
    makes KDE4 Klipper contents available on the CLI (read and write access!)
    debug

    shortcut to start a GDB session: debug APP APP_ARGS is all you have to do. Its basically the same as doing:

    1. $ gdb APP
    2. > run APP_ARGS
  • » Take 2: Download script for springerlink.com Ebooks

    Tue, 02/24/2009 - 22:58

    NOTE: This script is apparently against the licensing contract between universities and Springer, see: http://www.bib.hm.edu/aktuelles/news/newsdetail_9984.de.html

    NOTE 2: I do not maintain this script anymore. Please look for an alternative.

    Seems like quite some people are interested in my bash script for downloading ebooks from http://springerlink.com.

    That script has some quirks, the greatest of all that it was written in bash which makes it kind of hard to implement new features. And one which was requested was support for books which span multiple pages on SpringerLink.

    So here I present springer_download.py - a Python rewrite which should handle all the old links and some more. This is the very first program I’ve written in Python. And since it has to run on the Zedat servers it’s limited to Python 2.4.x without any fancy shmancy additions (a pity, since I’d love to use urlgrabber or pycurl).

    the script

    You can find the sources on GitHub: http://milianw.github.com/springer_download/

    I plan to put all my future code snippets in public repositories on GitHub. That way you can easily track changes and stay up to date. GitHub also has a nice “download” feature which you can use to get the current version. You can find my profile and my repositories at http://github.com/milianw

    Note: This script is intended to be run under Linux or other *nix’es which fulfill the requirements (Python 2.4.x, iconv and pdftk). Windows is not supported.

    TODO
    • introduce multithreading for faster / simultaneous downloads
    • add speed to progressbar
    • use progressbar in source-downloader
    • use one git-repo per project (makes links work properly)
  • » Download script for springerlink.com Ebooks

    Sat, 11/08/2008 - 17:28

    After a long period of silence I present you the following bash script for downloading books from http://springerlink.com. This is not a way to circumvent their login mechanisms, you will need proper rights to download books. But many students in Germany get free access to those ebooks via their universities. I for example study at the FU Berlin and put the script in my Zedat home folder and start the download process via SSH from home. Afterwards I download the tarball to my home system.

    Read on for the script.

  • » Access klipper clipboard on CLI under KDE4

    Wed, 08/13/2008 - 23:12

    NOTE: find most recent version on github: https://github.com/milianw/shell-helpers/blob/master/clipboard

    Here’s a little script you can save in your path and do things like

    1. # paste current clipboard into file
    2. clipboard > "some_file"
    3. # copy some file into clipboard
    4. cat "some_file" | clipboard

    Actually I find it rather useful so I thought I should share it.

    1. #!/bin/bash
    2.  
    3. # Access your KDE 4 klipper on the command line
    4. # usage:
    5. # ./clipboard
    6. # will output current contents of klipper
    7. # echo "foobar" | ./clipboard
    8. # will put "foobar" into your clipboard/klipper
    9.  
    10. # check for stdin
    11. if ! tty -s && stdin=$(</dev/stdin) && [[ "$stdin" ]]; then
    12. # get the rest of stdin
    13. stdin=$stdin$'\n'$(cat)
    14. # oh, nice - user input! we set that as current
    15. # clipboard content
    16. qdbus org.kde.klipper /klipper setClipboardContents "$stdin"
    17. exit
    18. fi
    19.  
    20. # if we reach this point no user input was given and we
    21. # print out the current contents of the clipboard
    22. qdbus org.kde.klipper /klipper getClipboardContents

    As usually, save the file (attached below) in your $PATH and make it executable.

    PS: Thanks to Martin Vidner for his article on D-BUS btw. - it gave me the proper dbus commands. PPS: Thanks the the various comments below!