kde Syndicate content

» KDevelop 4.3.1 Released with improved GCC 4.7 compliance

Thu, 04/19/2012 - 10:58

Hey all!

KDevelop 4.3.1 is out! Go read the announcement and update.

Many thanks to all contributors, you rock :)

» KDevelop 4.3.0 - Go Get It!

Tue, 03/20/2012 - 13:21

Finally I managed to get my job as the release dude done: http://kdevelop.org/kdevelop/kdevelop-430-final-released-basic-c11-support

Thanks to all the developers who sent in patches! The same goes to our loyal users for their continued support and bug reports :)

It’s really fun to work on KDevelop and - I’ve said it many times before - I’m really looking forward to our next releases! Even now our code in the master branches has some neat commits that make the eventual 4.4 release something to look forward to!

» Improving Massif-Visualizer For Large Data Files

Fri, 03/16/2012 - 15:42

As I just wrote in another article, Massif is an invaluable tool. The [Visualizer](https://projects.kde.org/massif-visualizer] I wrote is well appreciated and widely used as far as I can see.

A few days ago though, I did a very long (~16h) Massif run on an application, which resulted in a 204MB massif.out data file. This proved to be a very good stress test for my visualizer, which triggered me to spent some time on optimizing it. The results are pretty nice I thing, so look forward to Massif-Visualizer 0.4:

Reduced Memory Consumption

Yeah, meta eh? Just how I like it! I’ve used Massif to improve the memory consumption of Massif-Visualizer, and analyzed the data in the Visualizer of course… :)

Initial Version
initial memory consumption of the visualizer
fig. 1: initial memory consumption of the visualizer

The initial version of my visualizer took about ~470MB of memory to load the 204MB data file above. 80% of that was required for QString allocations in the callgraph of each detailed snapshot, i.e. the function signatures and location. See fig. 1 for the details.

QString to QByteArray
memory consumption of the visualizer using QByteArray instead of QString
fig. 2: `QByteArray` instead of `QString`: 50% less memory

Thomas McGuire gave me the tip of using QByteArray instead, since the Massif callgraph data is just ASCII data. We can convert the data to QString where required, essentially saving us 50% of the memory consumption. You can see that applied in fig. 2. It was simple to code and already reduced the memory consumption considerably.

Implicit Sharing
memory consumption of the visualizer leveraging implicit sharing
fig. 3: leveraging implicit sharing

I committed the above, thinking this was it. But thanks to the awesome people in the KDE community, this time André Wöbbeking, I was thankfully shown wrong: He commented on my commit, arguing that I should try out to leverage the implicit sharing of Qt containers, such as QByteArray. After all, the strings we have here are function signatures and file locations, which are repeated quite often. Especially when you have recursion in your call tree, or the same functions are encountered again and again in Massif snapshots, you can potentially safe a lot of memory by leveraging implicit sharing.

Personally, I’m suprised to see just how much this gains in this case! See fig 3., where the string allocations are nearly gone completely from the Massif log! Now only the tree node allocations, and the containers saving them, are visible in the memory log - something I do not plan to reduce further.

If you are interested in how this was implemented, take a look at commit 4be5dad13fb.

Final Notes

I think this shows quite nicely how to improve the memory consumption of an application. If you want to verify my results, I’ve uploaded the massif log files. Remember that you can open compressed files seamlessly in Massif-Visualizer. The massif.out.data.bz2 file contains the test-data of the 16h Massif run.

You should probably use the latest Massif-Visualizer code though, since I’ve also optimized the performance of it considerably compared to the last released version 0.3. Furthermore, data files are now loaded in the background, showing a nice progress bar while doing that. If you open the big data file in 0.3 you’ll notice why I decided to optimize the visualizer :)

An interesting thing to note btw. is that the callgrind data format compresses files and function signatures, yielding much smaller data files and reducing the KCacheGrind’s memory consumption, esp. since it will automagically leverage the implicit sharing of Qt’s string classes.

Now it is probably time to stop slacking and start work-work again :) I do have quite a few ideas more for the next Massif-Visualizer though, especially an export functionality for the graphs is high on my TODO list!

» Tracking Memory Consumption Using Pmap

Fri, 03/16/2012 - 14:55

Massif is a really nifty tool which is very powerful, especially paired with my visualizer. The caveat of course is that it slows down the application considerably, I’ve seen anything up to a factor of 100… I see no alternative to Massif when it comes to investigating where your memory problems come from. But if you just want to see whether you have a problem at all, tracking the total memory consumption should suffice.

A few days ago, I came across pmap on Stack Overflow, which makes it easy to track the RSS memory consumption of an application using the -x switch. Of course I had to write some bash magic to automate this process and visualize the data using Gnuplot! Behold:

memory consumption of PhantomJS
memory consumption of a PhantomJS script over ~30min
usage

It’s simple, really: track_memory.sh $(pidof myapp).

The default timeout is ~1s between snapshots, you can pass a different timeout as second parameter. Bash’s sleep can also take float numbers such as 0.1 to get more snapshots for fast-running apps.

You can also run show_memory.sh mem.log.$(pidof myapp) while you are still tracking the memory. The gnuplot window that appears allows you to update the data intermittently, to zoom in and to create images such as the above.

Note: This kind of memory usage tracking costs nearly nothing, your application continues to work at full speed. Also be aware that this just shows the RSS memory consumption. Massif will always give you better, more detailed and accurate results. Still, I think this should already give you an idea on how your application behaves. If the graph goes up and up, you probably got a memory leak! Then it’s time to run Memcheck and/or Massif to find the issue and fix it!

track_memory.sh

You can find the most-recent version on GitHub: https://github.com/milianw/shell-helpers/blob/master/track_memory.sh

  1. #!/bin/bash
  2.  
  3. #
  4. # track memory of given application, identified by PID,
  5. # using pmap -x, to show RSS and Dirty memory usage.
  6. #
  7. # visualization can later on be done with the
  8. # show_memory.sh script.
  9. #
  10.  
  11. pid=$1
  12. sleep=$2;
  13.  
  14. if [[ "$sleep" == "" ]]; then
  15. sleep=1
  16. fi
  17.  
  18. if [[ "$(ps -p $pid | grep $pid)" == "" ]]; then
  19. echo "cannot find program with pid $pid"
  20. echo "track_memory.sh PID [SLEEP_TIMEOUT]"
  21. echo
  22. echo "example: track_memory.sh \$(pidof someapp) 0.1"
  23. exit
  24. fi
  25.  
  26. logfile=mem.log.$pid
  27.  
  28. echo "# $(ps -o command= -p $pid)" > $logfile
  29. echo "# $sleep" >> $logfile
  30.  
  31. cat $logfile
  32.  
  33. while [[ "$(ps -p $pid | grep $pid)" != "" ]]; do
  34. echo "snapshot " $pid
  35. pmap -x $pid | tail -n1 >> $logfile
  36. echo "$sleep"
  37. sleep $sleep;
  38. done
  39.  
  40. echo "done tracking, visualizing"
  41. $(dirname $0)/show_memory.sh "$logfile"
show_memory.sh

You can find the most-recent version on GitHub: https://github.com/milianw/shell-helpers/blob/master/show_memory.sh

  1. #!/bin/bash
  2.  
  3. #
  4. # visualize memory consumption over time
  5. # as recorded by pmap / track_memory.sh
  6. # script
  7. #
  8.  
  9. logfile=$1
  10.  
  11. if [ ! -f "$logfile" ]; then
  12. echo "cannot find memory logfile: $1"
  13. echo
  14. echo "usage: show_memory.sh LOGFILE"
  15. echo
  16. echo "example: show_memory.sh mem.log.12345"
  17. exit
  18. fi
  19.  
  20. title=$(head -n1 "$logfile")
  21. timeout=$(head -n2 "$logfile" | tail -n1)
  22.  
  23. title=${title/\# /}
  24. timeout=${timeout/\# /}
  25.  
  26. # total:
  27. # '$logfile' using 3 w lines title 'Kbytes', \
  28.  
  29. gnuplot -p -e "
  30. set title '$title';
  31. set xlabel 'snapshot ~${timeout}s';
  32. set ylabel 'memory consumption in kB';
  33. set key bottom right;
  34. plot \
  35. '$logfile' using 4 w lines title 'RSS' lt 1, \
  36. '$logfile' using 4 smooth bezier w lines title 'RSS (smooth)' lt 7, \
  37. '$logfile' using 5 w lines title 'Dirty' lt 2, \
  38. '$logfile' using 5 smooth bezier w lines title 'Dirty (smooth)' lt 3;
  39. ";
Future

The above is nice, but I’m wondering on whether one should not add this kind of utility to ksysguard: It already allows you to track the total memory consumption of your system, yet I did not find a way to just track a single application and visualize it’s memory consumption.

» KDevelop Forum & Screenshots

Mon, 02/27/2012 - 17:44

Hey all,

following the recent blog post on getting a forum for KDE software I decided to get one setup for KDevelop. Minutes after, it was all done, we now have a KDevelop Forum. Feel free to use it for discussions and user support around KDevelop and related applications. Development discussions etc. will still happen on our mailing list though.

To get the forum rolling, I’ve had the idea to use it for some crowdsourcing of KDevelop screenshots, see also the post on the KDevelop website: http://kdevelop.org/community/new-forum-screenshots .

» KDevelop 4.3.0 RC 1 Released

Sat, 02/25/2012 - 18:45

Hey all!

Please help us test KDevelop 4.3 RC1! Grab it while it’s hot: http://kdevelop.org/43/kdevelop-430-rc-1-released

This release comes with some more bug fixes and better support for some C++11 language features.

Cheers!

» KDevelop 4.3.0 Beta 2 Released

Tue, 02/14/2012 - 15:10

Hey all!

I’ve just posted the news on the KDevelop website: KDevelop 4.3.0 Beta 2 is released!

Please test it and report feedback as usual. I think it’s safe to assume that we will release 4.3.0 final in about 2-4 weeks from now.

Considering that my university semester is nearing its end, I will finally have more royal hacking time again! I’ll continue to squash bugs and improve the performance of KDevelop of course :) Most definitely I’ll try to further improve the C++11 support. But maybe I finally have some time again to work on “something bigger”, like helping Miha Čančula in writing a kick-ass unit-test integration for KDevelop (see unittest branches). Then I plan to finally release some more of our “playground” plugins, most notably CSS language support and QMake project management…

Stay tuned for a bright KDevelop future :]

PS: I’ll step up as a mentor for a KDevelop GSOC this year, yet I’m still wondering about a proper topic… Ideas?

» Massif Visualizer 0.3 released

Sun, 11/20/2011 - 19:17

Hey all!

I’m happy to announce the release of Massif-Visualizer 0.3. You can download the sources here:

http://download.kde.org/download.php?url=stable/massif-visualizer/0.3/src

Highlights of this release:

  • translations into 18 different languages
  • basic support for hiding of functions via context menu
  • basic support for custom allocators
  • configurable precision of memory consumption display
  • various optimizations, bug fixes and other improvements. take a look at the changelog for more information
Future Development

It took me much too long to get this release out and hope to do better in the future. Current git master already contains some new patches - try it out! I especially like the improved display of the callgraph which now aggregates the tails of the callgraph tree, i.e. the end of the backtrace which mostly starts main() etc.

If you have ideas for the future, or have any issues with Massif-Visualizer, report a bug on https://bugs.kde.org/ .

Packaging

I don’t want to waste my time on packaging and instead concentrate on development. So here again a public plead to distributors: Please include Massif-Visualizer in future releases and let users download it seamlessly through your package manager. Starting with this release, Massif-Visualizer releases will follow the usual KDE procedure and the source tarballs will be hosted on the KDE mirrors. Hope this makes it simpler for packagers!

Ubuntu users can use Aurélien Gâteau’s PPA: https://launchpad.net/~agateau/+archive/ppa . ArchLinux users you can get it via AUR, and Gentoo also has an overlay for it it seems. I also think that OpenSUSE has a package for it. Please add infos for $your-distro in the comments below.

Cheers

» Qt: Tint alphatransparent PNG

Wed, 09/21/2011 - 18:07

Let’s assume you want to display the logo of your company in your Qt app. Most probably that logo has just single color with an alpha channel.But: Having the color hard coded in the image is not nice, there are users (like me!) out there, who use a custom (dark!) color scheme. Meaning: If your logo is black/dark and assumes a bright background and you just embed it blindly in your app, I probably won’t see it since the background will be dark in my case.

Here is a solution for the simple case of a mono-colored PNG with an alpha channel which I came up with:

  1. QLabel* label = new QLabel;
  2. // load your image
  3. QImage img(QString("..."));
  4. // morph it into a grayscale image
  5. img = img.alphaChannel();
  6. // the new color we want the logo to have
  7. QColor foreground = label->palette().foreground().color();
  8. // now replace the colors in the image
  9. for(int i = 0; i < img.colorCount(); ++i) {
  10. foreground.setAlpha(qGray(img.color(i)));
  11. img.setColor(i, foreground.rgba());
  12. }
  13. // display the new logo
  14. label->setPixmap(QPixmap::fromImage(img));
  15. label->show();

This seems to work just fine for me. YMMV.

» VTune and KDE

Fri, 09/09/2011 - 21:09

Hey all,

been some time since I blogged last time. My TODO list is ever increasing and I took my day job at KDAB up again. Among others, I attended a marketing talk by Edmund Preiss. He actually made that marketing talk interesting, not least by his huge knowledge in the business, thanks to ~20 years of working for Intel. Probably the most important info I got out of it is this:

VTune is available free-of-charge under a non-commercial license

Yes, you heard right. Take these links:

  • Intel’s non-commercial offering

    note this entry from the FAQ:

    What does noncommercial mean?
    Non-commercial means that you are not getting compensated in any form for the products and/or services you develop using these Intel® Software Products.

  • Register for free license

  • Register for Download Access

    you’ll need the serial number that gets send to you via email after registering for the license

  • install VTune and profile the hell out of KDE/FOSS software and improve it all!

speeding up KDevelop

Personally I did the latter for KDevelop the last two days, and the results are astonishing. I just tested the results from today and an unscientific time kdevelop -s lotsofprojects - wait until parsing finished - stop showed roughly 50% decrease in time, from ~12min to ~6min. Yes, a whopping 50% - try it out for yourself and see how big the gain is. Don’t forget to whipe the DUChain cache though (i.e. via setting the environment variable CLEAR_DUCHAIN_DIR=1).

Why VTune rocks

I’m a huge fan of the Valgrind toolsuite, but it is simply too slow for profiling some things. Like opening ten medium to big sized projects in KDevelop and taking a look at the parsing speed. This can easily take a few minutes, but in Valgrind it would take ages. With VTune on the other hand, thanks to it’s sampling based approach, I don’t really notice the performance delay.

Then you might have heard of the new perf profiling utility in the Linux kernel. It is also sampling based, but sadly requires special compile options on 64 Bit (-fno-omit-frame-pointers), and the UI is horrible, I haven’t found anything worthwhile with it so far…

VTune on the other hand has an incredible GUI, which makes profiling a joy. You can look at call stacks top-down or bottom-up, visualize locks and waits, easily find hotspots, … I’m blasted. Especially the utilities to look at multi threaded performance (of e.g. KDevelop) kills every single other performance tool I have ever tested. Oh and did I mention that you can attach to an app at runtime, analyze some thing, and detach again?

Seriously, Intel: You just found a new fan boy in me. Thanks for giving this tool away for free for us “I hack on this tool in my spare time, yet still want it to perform nicely” people :) And kudos to the VTune developers - I’m blown away by it!

I really hope more people in the KDE community will try out VTune and try to improve the performance of our apps, I bet there is lots of potential!

Pitfalls

There are some negative aspects to VTune though: First of all it’s UI is sometimes freezing. I wonder if the developers should not maybe spent some time on analyzing the tool itself ;-)

The biggest gripe though is that VTune does not work everywhere. I tried to run it on my Arch box, but sadly Linux 3.0 is not supported by VTune yet. It worked like a charm on two Ubuntu boxes with some 2.6.X kernel though.

This also means that I have no idea if, and how, VTune works on non-Intel CPUs. I think some of it works nicely. I did not install any of the Kernel modules for examples, which would be required for hardcore lowlevel CPU profiling. I think the same feature set I praised so much above, should hence be available on e.g. AMD CPUs. But well, this is left to be tested.

So, I’m now drinking a well deserved beer and look positively into the future of a fast KDevelop/KDE :)

bye