Milian Wolff

Massif is a really nifty tool which is very powerful, especially paired with my visualizer. The caveat of course is that it slows down the application considerably, I’ve seen anything up to a factor of 100… I see no alternative to Massif when it comes to investigating where your memory problems come from. But if you just want to see whether you have a problem at all, tracking the total memory consumption should suffice.

A few days ago, I came across pmap on Stack Overflow, which makes it easy to track the RSS memory consumption of an application using the -x switch. Of course I had to write some bash magic to automate this process and visualize the data using Gnuplot! Behold:

memory consumption of a PhantomJS script over ~30min

usage

It’s simple, really: track_memory.sh $(pidof myapp).

The default timeout is ~1s between snapshots, you can pass a different timeout as second parameter. Bash’s sleep can also take float numbers such as 0.1 to get more snapshots for fast-running apps.

You can also run show_memory.sh mem.log.$(pidof myapp) while you are still tracking the memory. The gnuplot window that appears allows you to update the data intermittently, to zoom in and to create images such as the above.

Note: This kind of memory usage tracking costs nearly nothing, your application continues to work at full speed. Also be aware that this just shows the RSS memory consumption. Massif will always give you better, more detailed and accurate results. Still, I think this should already give you an idea on how your application behaves. If the graph goes up and up, you probably got a memory leak! Then it’s time to run Memcheck and/or Massif to find the issue and fix it!

track_memory.sh

You can find the most-recent version on GitHub: https://github.com/milianw/shell-helpers/blob/master/track_memory.sh

#!/bin/bash
 
#
# track memory of given application, identified by PID,
# using pmap -x, to show RSS and Dirty memory usage.
#
# visualization can later on be done with the
# show_memory.sh script.
#
 
pid=$1
sleep=$2;
 
if [[ "$sleep" == "" ]]; then
  sleep=1
fi
 
if [[ "$(ps -p $pid | grep $pid)" == "" ]]; then
  echo "cannot find program with pid $pid"
  echo "track_memory.sh PID [SLEEP_TIMEOUT]"
  echo
  echo "example: track_memory.sh \$(pidof someapp) 0.1"
  exit
fi
 
logfile=mem.log.$pid
 
echo "# $(ps -o command= -p $pid)" > $logfile
echo "# $sleep" >> $logfile
 
cat $logfile
 
while [[ "$(ps -p $pid | grep $pid)" != "" ]]; do
  echo "snapshot " $pid
  pmap -x $pid | tail -n1 >> $logfile
  echo "$sleep"
  sleep $sleep;
done
 
echo "done tracking, visualizing"
$(dirname $0)/show_memory.sh "$logfile"

show_memory.sh

You can find the most-recent version on GitHub: https://github.com/milianw/shell-helpers/blob/master/show_memory.sh

#!/bin/bash
 
#
# visualize memory consumption over time
# as recorded by pmap / track_memory.sh
# script
#
 
logfile=$1
 
if [ ! -f "$logfile" ]; then
  echo "cannot find memory logfile: $1"
  echo
  echo "usage: show_memory.sh LOGFILE"
  echo
  echo "example: show_memory.sh mem.log.12345"
  exit
fi
 
title=$(head -n1 "$logfile")
timeout=$(head -n2 "$logfile" | tail -n1)
 
title=${title/\# /}
timeout=${timeout/\# /}
 
# total:
#  '$logfile' using 3 w lines title 'Kbytes', \
 
gnuplot -p -e "
set title '$title';
set xlabel 'snapshot ~${timeout}s';
set ylabel 'memory consumption in kB';
set key bottom right;
plot \
  '$logfile' using 4 w lines title 'RSS' lt 1, \
  '$logfile' using 4 smooth bezier w lines title 'RSS (smooth)' lt 7, \
  '$logfile' using 5 w lines title 'Dirty' lt 2, \
  '$logfile' using 5 smooth bezier w lines title 'Dirty (smooth)' lt 3;
";

Future

The above is nice, but I’m wondering on whether one should not add this kind of utility to ksysguard: It already allows you to track the total memory consumption of your system, yet I did not find a way to just track a single application and visualize it’s memory consumption.

Unit tests are in my eyes a very important part of programming. KDE uses them, KDevelop does - the PHP plugin I help writing does as well. cmake comes with a ctest program which does quite well to give you a quick glance on which test suite you just broke with your new fance feature :)

But I am very dissatisfied with it. Right now I usually do the following

# lets assume I'm in the source directory
cb && ctest
# look for failed test suites
cd $failed_test_suite_path
./$failed_test_suite.shell | less
# search for FAIL
cs
cd $to_whereever_I_was_before

That’s pretty much for just running a test. Especially all that cding and lessing became very tedious. Tedious is good, because I eventually fix it:

introducing kdetest

I wrote a bash function (with autocompletion!!!) called kdetest. Calling it without any parameter will run all test suites and gives a nice report of failed functions at the end. Here’s an example (run via cs php && kdetest).

kdetest
# ... lots of test output
 
 --- ALL PASSED TESTS ---
...
PASS   : Php::TestCompletion::implementMethods()
PASS   : Php::TestCompletion::inArray()
PASS   : Php::TestCompletion::cleanupTestCase()
 
143 passed tests in total
 
 --- ALL FAILED TESTS ---
FAIL!  : Php::TestCompletion::newExtends() Compared values are not the same
FAIL!  : Php::TestCompletion::updateExtends() '! forbiddenIdentifiers.contains(item->declaration()->identifier().toString())' returned FALSE. ()
FAIL!  : Php::TestCompletion::updateExtends() '! forbiddenIdentifiers.contains(item->declaration()->identifier().toString())' returned FALSE. ()
FAIL!  : Php::TestCompletion::updateExtends() Compared values are not the same
FAIL!  : Php::TestCompletion::newImplements() Compared values are not the same
FAIL!  : Php::TestCompletion::updateImplements() Compared values are not the same
 
6 failed tests in total

usage

• kdetest, i.e. without any arguments runs all tests in this directory and below
• kdetest path/to/test.shell ... runs that test suite only, ... can by any argument the test suite accepts.

autocompletion

kdetest comes with full support for autocompletion of tests and functions, for example:

milian@odin:~/projects/kde4/php$ kdetest TABTAB
completion/tests/completiontest.shell     duchain/tests/expressionparsertest.shell  parser/test/lexertest.shell
duchain/tests/duchaintest.shell           duchain/tests/usestest.shell
milian@odin:~/projects/kde4/php$ kdetest duchain/tests/usestest.shell TABTAB
classAndConstWithSameName          classSelf                          interfaceExtendsMultiple           staticMemberFunctionCall
classAndFunctionWithSameName       constAndVariableWithSameName       memberFunctionCall                 staticMemberVariable
classConstant                      constant                           memberFunctionInString             variable
classExtends                       constantInClassMember              memberVariable                     variableTwoDeclarations
classImplements                    functionAndClassWithSameName       memberVarInString                  variableTwoDeclarationsInFunction
classImplementsMultiple            functionCall                       newObject                          varInString
classParent                        interfaceExtends                   objectWithClassName

the code

You can find the code below, or you can obtain the most up-to-date version on github. Just head over to my shell-helpers repo and peek into the bash_setup_kde4_programming file.

Everyone who uses the command line regularly has a bunch of (at least for him) useful helper scripts. I now took the liberty to put mine on github for general consumption.

You can find them at: http://github.com/milianw/shell-helpers/tree/master

Some of these might be more useful than others, you decide :) Personally, I can’t live without the following:

apupgrade

a shortcut to update your Debian system with one command - no questions asked

openurl

opens a given URL in an already opened browser instance or starts a new browser session. Not only one browser is checked. I use it because firefox is slow to start and konqueror is blazingly fast to start. But when firefox is already open I want to use that.

xerr

shortcut for fast error checking in your Xorg log

clipboard

makes KDE4 Klipper contents available on the CLI (read and write access!)

debug

shortcut to start a GDB session: debug APP APP_ARGS is all you have to do. Its basically the same as doing:

$ gdb APP
> run APP_ARGS

After a long period of silence I present you the following bash script for downloading books from http://springerlink.com. This is not a way to circumvent their login mechanisms, you will need proper rights to download books. But many students in Germany get free access to those ebooks via their universities. I for example study at the FU Berlin and put the script in my Zedat home folder and start the download process via SSH from home. Afterwards I download the tarball to my home system.

Read on for the script.

NOTE: find most recent version on github: https://github.com/milianw/shell-helpers/blob/master/clipboard

Here’s a little script you can save in your path and do things like

# paste current clipboard into file
clipboard > "some_file"
# copy some file into clipboard
cat "some_file" | clipboard

Actually I find it rather useful so I thought I should share it.

#!/bin/bash
 
# Access your KDE 4 klipper on the command line
# usage:
#  ./clipboard
#    will output current contents of klipper
#  echo "foobar" | ./clipboard
#    will put "foobar" into your clipboard/klipper
 
# check for stdin
if ! tty -s && stdin=$(</dev/stdin) && [[ "$stdin" ]]; then
  # get the rest of stdin
  stdin=$stdin$'\n'$(cat)
  # oh, nice - user input! we set that as current
  # clipboard content
  qdbus org.kde.klipper /klipper setClipboardContents "$stdin"
  exit
fi
 
# if we reach this point no user input was given and we
# print out the current contents of the clipboard
qdbus org.kde.klipper /klipper getClipboardContents

As usually, save the file (attached below) in your $PATH and make it executable.

PS: Thanks to Martin Vidner for his article on D-BUS btw. - it gave me the proper dbus commands. PPS: Thanks the the various comments below!

I’ve had a major annoyance today: The plot generated by gnuplot looked good inside the wxt terminal but I simply couldn’t get a proper fullsized DIN A4 postscript exported. This is how I’ve done it now:

1. Inside gnuplot:

   set size ratio 0.71 # this is the ratio of a DIN A4 page (21/29.7)
   set terminal postscript enhanced landscape "Arial" 9 # you can change landscape to portrait and the fontname and -size
   set output 'yourfilename.ps' # this is your export file
   replot # or put your custom plot command here

2. In a shell:

   ps2ps -sPAGESIZE=a4 yourfilename.ps new_dina4_file.ps

3. Now you can simply print `new_dina4_file.ps` from within KGhostView for example. Have fun!

So just yesterday I’ve published a bash script which rips the audio stream of Flash Videos (*.flv) to mp3 format. It’s nice, fast and imo all-purpose. But I didn’t like the audio quality. Thus below you can find a second version of the script which is using mplayer and lame instead of ffmpeg. Usage and behaviour should be pretty much the same. Only the audio quality should be better on cost of a bit more computing.

Since it relies on the fact that the input *.flv video already uses MP3 encoding for its audio stream this might not work for every flash file! Youtube works just fine though. You can find the script after the break, it’s also attached below. For usage / installation / more information read the old article.

If you wonder why I reencode the audiodump with lame: The dumped mp3 file is considered by most (all?) audio players to be ~10x the length. A five minute video gets a 45 minute audio dumpfile. It plays fine but seeking is a mess. Reencoding fixes this. If you know an alternative which does not require reencoding or is generally faster - please drop a comment!

UPDATE: Also check out mp3dump part 2

On some undefined occasion you might want to dump the audio stream of a flash file. This is how you do it:

1. Get the *.flv, e.g. visit youtube with Konqueror and check /tmp for a file like FlashV5SQLt - this is your FLV.
2. Install some dependencies: ffmpeg is required, mp3info and ecasound are optional but strongly recommended.
3. Download the script below (it’s attached!) and save it inside your $PATH, remember how you’ve called it!

4. I’ve saved the script inside ~/.bin which is inside my $PATH so all I have to do now is running it:

   mp3dump FlashV5SQLt "Foo Bar" "All Your Base Are Belong To Us"

   This would rip the audio stream of FlashV5SQLt to a file named Foo Bar - All Your Base Are Belong To Us.mp3 - with emulated stereo sound and basic MP3 tags (artist and title). Coolio!

For those of you, who use the original NVIDIA display driver and have to reinstall it with every kernel update: Here is a little bashscript for your convenience.

#!/bin/bash
sudo /etc/init.d/kdm stop
sudo chvt 1
sudo sh ~/Downloads/NVIDIA-Linux-x86-*.run --ui='none' -aNq
sudo /etc/init.d/kdm start
sudo chvt 7

Save it e.g. as ~/.bin/nv_reinst.sh, make it executable (chmod +x ~/.bin/nv_reinst.sh). Then put your NVIDIA driver (only the newest version please) into ~/Downloads/ (or change the path above). Now run your script.

Attention: Save all your work, since kdm will be stopped! You’ll lose all unsaved work!

I was recently annoyed by how URLs were started in KDE: I used an antiquated firefox command to run an URL in a new tab (firefox-remote "OpenURL(%u, new-window)") which does the job quite well. But: If you have no running firefox instance this command will simply do nothing. Right - nothing! You’ll have to start firefox manually and click once again on the link (in konversation, kopete or wherever).

But the newer firefox versions (I don’t know when this feature was added) support the firefox -newtab %u command. This is all you need if you only use firefox. Configure KDE via kcontrol to use this command as a standard webbrowser.

This was still not enough for me though, as I regularly switch to konqueror or opera, because firefox is sometimes to slow for a quick browse. This is what I came up with: