› Code Snippets 
» Tracking Memory Consumption Using Pmap
Fri, 03/16/2012 - 14:55
Massif is a really nifty tool which is very powerful, especially paired with my visualizer. The caveat of course is that it slows down the application considerably, I’ve seen anything up to a factor of 100… I see no alternative to Massif when it comes to investigating where your memory problems come from. But if you just want to see whether you have a problem at all, tracking the total memory consumption should suffice.
A few days ago, I came across pmap on Stack Overflow, which makes it easy to track the RSS memory consumption of an application using the
-xswitch. Of course I had to write some bash magic to automate this process and visualize the data using Gnuplot! Behold:
memory consumption of a PhantomJS script over ~30minusage
It’s simple, really:
track_memory.sh $(pidof myapp).The default timeout is ~1s between snapshots, you can pass a different timeout as second parameter. Bash’s
sleepcan also take float numbers such as0.1to get more snapshots for fast-running apps.You can also run
show_memory.sh mem.log.$(pidof myapp)while you are still tracking the memory. The gnuplot window that appears allows you to update the data intermittently, to zoom in and to create images such as the above.Note: This kind of memory usage tracking costs nearly nothing, your application continues to work at full speed. Also be aware that this just shows the RSS memory consumption. Massif will always give you better, more detailed and accurate results. Still, I think this should already give you an idea on how your application behaves. If the graph goes up and up, you probably got a memory leak! Then it’s time to run Memcheck and/or Massif to find the issue and fix it!
track_memory.sh
You can find the most-recent version on GitHub: https://github.com/milianw/shell-helpers/blob/master/track_memory.sh
- #!/bin/bash
- #
- # track memory of given application, identified by PID,
- # using pmap -x, to show RSS and Dirty memory usage.
- #
- # visualization can later on be done with the
- # show_memory.sh script.
- #
- pid=$1
- sleep=$2;
- if [[ "$sleep" == "" ]]; then
- sleep=1
- fi
- if [[ "$(ps -p $pid | grep $pid)" == "" ]]; then
- echo "cannot find program with pid $pid"
- echo "track_memory.sh PID [SLEEP_TIMEOUT]"
- echo
- echo "example: track_memory.sh \$(pidof someapp) 0.1"
- exit
- fi
- logfile=mem.log.$pid
- echo "# $(ps -o command= -p $pid)" > $logfile
- echo "# $sleep" >> $logfile
- cat $logfile
- while [[ "$(ps -p $pid | grep $pid)" != "" ]]; do
- echo "snapshot " $pid
- pmap -x $pid | tail -n1 >> $logfile
- echo "$sleep"
- sleep $sleep;
- done
- echo "done tracking, visualizing"
- $(dirname $0)/show_memory.sh "$logfile"
show_memory.sh
You can find the most-recent version on GitHub: https://github.com/milianw/shell-helpers/blob/master/show_memory.sh
- #!/bin/bash
- #
- # visualize memory consumption over time
- # as recorded by pmap / track_memory.sh
- # script
- #
- logfile=$1
- if [ ! -f "$logfile" ]; then
- echo "cannot find memory logfile: $1"
- echo
- echo "usage: show_memory.sh LOGFILE"
- echo
- echo "example: show_memory.sh mem.log.12345"
- exit
- fi
- title=$(head -n1 "$logfile")
- timeout=$(head -n2 "$logfile" | tail -n1)
- title=${title/\# /}
- timeout=${timeout/\# /}
- # total:
- # '$logfile' using 3 w lines title 'Kbytes', \
- gnuplot -p -e "
- set title '$title';
- set xlabel 'snapshot ~${timeout}s';
- set ylabel 'memory consumption in kB';
- set key bottom right;
- plot \
- '$logfile' using 4 w lines title 'RSS' lt 1, \
- '$logfile' using 4 smooth bezier w lines title 'RSS (smooth)' lt 7, \
- '$logfile' using 5 w lines title 'Dirty' lt 2, \
- '$logfile' using 5 smooth bezier w lines title 'Dirty (smooth)' lt 3;
- ";
Future
The above is nice, but I’m wondering on whether one should not add this kind of utility to
ksysguard: It already allows you to track the total memory consumption of your system, yet I did not find a way to just track a single application and visualize it’s memory consumption.» Qt: Tint alphatransparent PNG
Wed, 09/21/2011 - 18:07
Let’s assume you want to display the logo of your company in your Qt app. Most probably that logo has just single color with an alpha channel.But: Having the color hard coded in the image is not nice, there are users (like me!) out there, who use a custom (dark!) color scheme. Meaning: If your logo is black/dark and assumes a bright background and you just embed it blindly in your app, I probably won’t see it since the background will be dark in my case.
Here is a solution for the simple case of a mono-colored PNG with an alpha channel which I came up with:
- // load your image
- // morph it into a grayscale image
- img = img.alphaChannel();
- // the new color we want the logo to have
- // now replace the colors in the image
- for(int i = 0; i < img.colorCount(); ++i) {
- foreground.setAlpha(qGray(img.color(i)));
- img.setColor(i, foreground.rgba());
- }
- // display the new logo
- label->show();
This seems to work just fine for me. YMMV.
» Shell helper: running KDE unit tests (ctests) the easy way
Thu, 03/26/2009 - 03:09
Unit tests are in my eyes a very important part of programming. KDE uses them, KDevelop does - the PHP plugin I help writing does as well.
cmakecomes with actestprogram which does quite well to give you a quick glance on which test suite you just broke with your new fance feature :)But I am very dissatisfied with it. Right now I usually do the following
- # lets assume I'm in the source directory
- cb && ctest
- # look for failed test suites
- cd $failed_test_suite_path
- ./$failed_test_suite.shell | less
- # search for FAIL
- cs
- cd $to_whereever_I_was_before
That’s pretty much for just running a test. Especially all that
cding andlessing became very tedious. Tedious is good, because I eventually fix it:introducing
kdetestI wrote a bash function (with autocompletion!!!) called
kdetest. Calling it without any parameter will run all test suites and gives a nice report of failed functions at the end. Here’s an example (run viacs php && kdetest).- kdetest
- # ... lots of test output
- --- ALL PASSED TESTS ---
- ...
- PASS : Php::TestCompletion::implementMethods()
- PASS : Php::TestCompletion::inArray()
- PASS : Php::TestCompletion::cleanupTestCase()
- 143 passed tests in total
- --- ALL FAILED TESTS ---
- FAIL! : Php::TestCompletion::newExtends() Compared values are not the same
- FAIL! : Php::TestCompletion::updateExtends() '! forbiddenIdentifiers.contains(item->declaration()->identifier().toString())' returned FALSE. ()
- FAIL! : Php::TestCompletion::updateExtends() '! forbiddenIdentifiers.contains(item->declaration()->identifier().toString())' returned FALSE. ()
- FAIL! : Php::TestCompletion::updateExtends() Compared values are not the same
- FAIL! : Php::TestCompletion::newImplements() Compared values are not the same
- FAIL! : Php::TestCompletion::updateImplements() Compared values are not the same
- 6 failed tests in total
usage
kdetest, i.e. without any arguments runs all tests in this directory and belowkdetest path/to/test.shell ...runs that test suite only,...can by any argument the test suite accepts.
autocompletion
kdetestcomes with full support for autocompletion of tests and functions, for example:- milian@odin:~/projects/kde4/php$ kdetest TABTAB
- completion/tests/completiontest.shell duchain/tests/expressionparsertest.shell parser/test/lexertest.shell
- duchain/tests/duchaintest.shell duchain/tests/usestest.shell
- milian@odin:~/projects/kde4/php$ kdetest duchain/tests/usestest.shell TABTAB
- classAndConstWithSameName classSelf interfaceExtendsMultiple staticMemberFunctionCall
- classAndFunctionWithSameName constAndVariableWithSameName memberFunctionCall staticMemberVariable
- classConstant constant memberFunctionInString variable
- classExtends constantInClassMember memberVariable variableTwoDeclarations
- classImplements functionAndClassWithSameName memberVarInString variableTwoDeclarationsInFunction
- classImplementsMultiple functionCall newObject varInString
- classParent interfaceExtends objectWithClassName
the code
You can find the code below, or you can obtain the most up-to-date version on github. Just head over to my shell-helpers repo and peek into the
bash_setup_kde4_programmingfile.» Attack of the shell helpers
Mon, 03/02/2009 - 02:46
Everyone who uses the command line regularly has a bunch of (at least for him) useful helper scripts. I now took the liberty to put mine on github for general consumption.
You can find them at: http://github.com/milianw/shell-helpers/tree/master
Some of these might be more useful than others, you decide :) Personally, I can’t live without the following:
apupgrade- a shortcut to update your Debian system with one command - no questions asked
openurl- opens a given URL in an already opened browser instance or starts a new browser session. Not only one browser is checked. I use it because firefox is slow to start and konqueror is blazingly fast to start. But when firefox is already open I want to use that.
xerr- shortcut for fast error checking in your Xorg log
clipboard- makes KDE4 Klipper contents available on the CLI (read and write access!)
debug-
shortcut to start a GDB session:
debug APP APP_ARGSis all you have to do. Its basically the same as doing:- $ gdb APP
- > run APP_ARGS
» Take 2: Download script for springerlink.com Ebooks
Tue, 02/24/2009 - 22:58
Seems like quite some people are interested in my bash script for downloading ebooks from http://springerlink.com.
That script has some quirks, the greatest of all that it was written in bash which makes it kind of hard to implement new features. And one which was requested was support for books which span multiple pages on SpringerLink.
So here I present
springer_download.py- a Python rewrite which should handle all the old links and some more. This is the very first program I’ve written in Python. And since it has to run on the Zedat servers it’s limited to Python 2.4.x without any fancy shmancy additions (a pity, since I’d love to use urlgrabber or pycurl).the script
You can find the sources on GitHub: http://milianw.github.com/springer_download/
I plan to put all my future code snippets in public repositories on GitHub. That way you can easily track changes and stay up to date. GitHub also has a nice “download” feature which you can use to get the current version. You can find my profile and my repositories at http://github.com/milianw
Note: This script is intended to be run under Linux or other *nix’es which fulfill the requirements (Python 2.4.x, iconv and pdftk). Windows is not supported.
TODO
- introduce multithreading for faster / simultaneous downloads
- add speed to progressbar
- use progressbar in source-downloader
- use one git-repo per project (makes links work properly)
» Download script for springerlink.com Ebooks
Sat, 11/08/2008 - 17:28
After a long period of silence I present you the following bash script for downloading books from http://springerlink.com. This is not a way to circumvent their login mechanisms, you will need proper rights to download books. But many students in Germany get free access to those ebooks via their universities. I for example study at the FU Berlin and put the script in my Zedat home folder and start the download process via SSH from home. Afterwards I download the tarball to my home system.
Read on for the script.
» Access klipper clipboard on CLI under KDE4
Wed, 08/13/2008 - 23:12
NOTE: find most recent version on github: https://github.com/milianw/shell-helpers/blob/master/clipboard
Here’s a little script you can save in your path and do things like
- # paste current clipboard into file
- clipboard > "some_file"
- # copy some file into clipboard
- cat "some_file" | clipboard
Actually I find it rather useful so I thought I should share it.
- #!/bin/bash
- # Access your KDE 4 klipper on the command line
- # usage:
- # ./clipboard
- # will output current contents of klipper
- # echo "foobar" | ./clipboard
- # will put "foobar" into your clipboard/klipper
- # check for stdin
- if ! tty -s && stdin=$(</dev/stdin) && [[ "$stdin" ]]; then
- # get the rest of stdin
- stdin=$stdin$'\n'$(cat)
- # oh, nice - user input! we set that as current
- # clipboard content
- qdbus org.kde.klipper /klipper setClipboardContents "$stdin"
- exit
- fi
- # if we reach this point no user input was given and we
- # print out the current contents of the clipboard
- qdbus org.kde.klipper /klipper getClipboardContents
As usually, save the file (attached below) in your
$PATHand make it executable.PS: Thanks to Martin Vidner for his article on D-BUS btw. - it gave me the proper dbus commands. PPS: Thanks the the various comments below!
» profile.class.php
Tue, 06/24/2008 - 18:29
Every now and then I want to profile a given part of PHP code. For example I want to quickly check wether my current changeset to GeSHi works faster or is horribly slower. For a big change I’ll stick to Xdebug and KCachegrind. But for a quick overview? Overkill in my eyes.
Say hello to
profile.class.php, a simple timer class for PHP5 which you can use to get an overview about where how much time is spent. This is in no way a scientific method nor should you take the results of a single run as a basis for you decisions.I’ve set an emphasize on an easy API so you don’t have to pollute your code with arbitrary hoops and whistles.
UPDATE: You can find the current updated source in the SVN repo of GeSHi.
» mp3dump take two - better audio quality
Sat, 03/29/2008 - 16:29
So just yesterday I’ve published a bash script which rips the audio stream of Flash Videos (
*.flv) to mp3 format. It’s nice, fast and imo all-purpose. But I didn’t like the audio quality. Thus below you can find a second version of the script which is usingmplayerandlameinstead offfmpeg. Usage and behaviour should be pretty much the same. Only the audio quality should be better on cost of a bit more computing.Since it relies on the fact that the input
*.flvvideo already uses MP3 encoding for its audio stream this might not work for every flash file! Youtube works just fine though. You can find the script after the break, it’s also attached below. For usage / installation / more information read the old article.If you wonder why I reencode the audiodump with lame: The dumped mp3 file is considered by most (all?) audio players to be ~10x the length. A five minute video gets a 45 minute audio dumpfile. It plays fine but seeking is a mess. Reencoding fixes this. If you know an alternative which does not require reencoding or is generally faster - please drop a comment!
» Patching Kirocker Music Display for volume control via mouse scrolling
Fri, 03/28/2008 - 16:32
Kirocker is such a great application, it’s a pity the developer Sébastien Laoût has given up on it and is using Windows now!
Usually I only use the kicker applet, the full screen window is nice for parties though. But I didn’t like the way the applet handles scroll events: It seeks (if possible). Since I rarely seek when listening to music I’d rather have the behaviour of the Amarok tray icon: volume control!
I had a look inside the sources and found this part in
src/coverdisplay.cpp:- {
- if (areControlsShown()) {
- if (event->orientation() == Qt::Vertical) {
- PlayerInformation *infos = PlayerInformation::instance();
- if (infos->canSeek()) {
- int deltaSeconds = 10 * (event->delta() > 0 ? 1 : -1);
- m_infos->seekRelative(deltaSeconds);
- }
- } else {
- if (event->delta() > 0)
- AmarokApi::volumeUp();
- else
- AmarokApi::volumeDown();
- }
- }
- }
As you can see it already supports volume control via scrolling! The thing is I’ve disable four-way scrolling for my MX1000. Thus I’ve simply swapped the
ifstatements and now I’m really happy with Kirocker. Here’s the updated snippet:- {
- if (areControlsShown()) {
- if (event->orientation() == Qt::Vertical) {
- if (event->delta() > 0)
- AmarokApi::volumeUp();
- else
- AmarokApi::volumeDown();
- } else {
- PlayerInformation *infos = PlayerInformation::instance();
- if (infos->canSeek()) {
- int deltaSeconds = 10 * (event->delta() > 0 ? 1 : -1);
- m_infos->seekRelative(deltaSeconds);
- }
- }
- }
- }
Replace the old code, compile Kirocker, install it. Then restart kicker (
killall kicker; sleep 1; kicker;) and Kirocker should accept scroll events in the way the Amarok tray icon does!