massif-visualizer Syndicate content

» Improving Massif-Visualizer For Large Data Files

Fri, 03/16/2012 - 15:42

As I just wrote in another article, Massif is an invaluable tool. The [Visualizer](https://projects.kde.org/massif-visualizer] I wrote is well appreciated and widely used as far as I can see.

A few days ago though, I did a very long (~16h) Massif run on an application, which resulted in a 204MB massif.out data file. This proved to be a very good stress test for my visualizer, which triggered me to spent some time on optimizing it. The results are pretty nice I thing, so look forward to Massif-Visualizer 0.4:

Reduced Memory Consumption

Yeah, meta eh? Just how I like it! I’ve used Massif to improve the memory consumption of Massif-Visualizer, and analyzed the data in the Visualizer of course… :)

Initial Version
initial memory consumption of the visualizer
fig. 1: initial memory consumption of the visualizer

The initial version of my visualizer took about ~470MB of memory to load the 204MB data file above. 80% of that was required for QString allocations in the callgraph of each detailed snapshot, i.e. the function signatures and location. See fig. 1 for the details.

QString to QByteArray
memory consumption of the visualizer using QByteArray instead of QString
fig. 2: `QByteArray` instead of `QString`: 50% less memory

Thomas McGuire gave me the tip of using QByteArray instead, since the Massif callgraph data is just ASCII data. We can convert the data to QString where required, essentially saving us 50% of the memory consumption. You can see that applied in fig. 2. It was simple to code and already reduced the memory consumption considerably.

Implicit Sharing
memory consumption of the visualizer leveraging implicit sharing
fig. 3: leveraging implicit sharing

I committed the above, thinking this was it. But thanks to the awesome people in the KDE community, this time André Wöbbeking, I was thankfully shown wrong: He commented on my commit, arguing that I should try out to leverage the implicit sharing of Qt containers, such as QByteArray. After all, the strings we have here are function signatures and file locations, which are repeated quite often. Especially when you have recursion in your call tree, or the same functions are encountered again and again in Massif snapshots, you can potentially safe a lot of memory by leveraging implicit sharing.

Personally, I’m suprised to see just how much this gains in this case! See fig 3., where the string allocations are nearly gone completely from the Massif log! Now only the tree node allocations, and the containers saving them, are visible in the memory log - something I do not plan to reduce further.

If you are interested in how this was implemented, take a look at commit 4be5dad13fb.

Final Notes

I think this shows quite nicely how to improve the memory consumption of an application. If you want to verify my results, I’ve uploaded the massif log files. Remember that you can open compressed files seamlessly in Massif-Visualizer. The massif.out.data.bz2 file contains the test-data of the 16h Massif run.

You should probably use the latest Massif-Visualizer code though, since I’ve also optimized the performance of it considerably compared to the last released version 0.3. Furthermore, data files are now loaded in the background, showing a nice progress bar while doing that. If you open the big data file in 0.3 you’ll notice why I decided to optimize the visualizer :)

An interesting thing to note btw. is that the callgrind data format compresses files and function signatures, yielding much smaller data files and reducing the KCacheGrind’s memory consumption, esp. since it will automagically leverage the implicit sharing of Qt’s string classes.

Now it is probably time to stop slacking and start work-work again :) I do have quite a few ideas more for the next Massif-Visualizer though, especially an export functionality for the graphs is high on my TODO list!

» Massif Visualizer 0.3 released

Sun, 11/20/2011 - 19:17

Hey all!

I’m happy to announce the release of Massif-Visualizer 0.3. You can download the sources here:

http://download.kde.org/download.php?url=stable/massif-visualizer/0.3/src

Highlights of this release:

  • translations into 18 different languages
  • basic support for hiding of functions via context menu
  • basic support for custom allocators
  • configurable precision of memory consumption display
  • various optimizations, bug fixes and other improvements. take a look at the changelog for more information
Future Development

It took me much too long to get this release out and hope to do better in the future. Current git master already contains some new patches - try it out! I especially like the improved display of the callgraph which now aggregates the tails of the callgraph tree, i.e. the end of the backtrace which mostly starts main() etc.

If you have ideas for the future, or have any issues with Massif-Visualizer, report a bug on https://bugs.kde.org/ .

Packaging

I don’t want to waste my time on packaging and instead concentrate on development. So here again a public plead to distributors: Please include Massif-Visualizer in future releases and let users download it seamlessly through your package manager. Starting with this release, Massif-Visualizer releases will follow the usual KDE procedure and the source tarballs will be hosted on the KDE mirrors. Hope this makes it simpler for packagers!

Ubuntu users can use Aurélien Gâteau’s PPA: https://launchpad.net/~agateau/+archive/ppa . ArchLinux users you can get it via AUR, and Gentoo also has an overlay for it it seems. I also think that OpenSUSE has a package for it. Please add infos for $your-distro in the comments below.

Cheers

» Massif Visualizer 0.2 released

Sun, 11/07/2010 - 00:16

Hey all!

I’m happy to release Massif Visualizer v0.2. This is mainly a “fix the build-system” release, no new features have been added.

You can download it here: http://kde-apps.org/content/show.php?content=122409

Mac Support

Thanks to the reports by Chris Jones it’s now possible to build and use Massif Visualizer on Max OS X, see e.g.:

http://www.hep.phy.cam.ac.uk/~jonesc/massif-visualizer-OSX-1.png
http://www.hep.phy.cam.ac.uk/~jonesc/massif-visualizer-OSX-2.png

He has also submitted the portsfile for inclusion in Macports: https://trac.macports.org/ticket/27168

KGraphViewer now optional

I’ve made the KGraphViewer dependency optional, if anyone does not want it (even though this removes like 50% of the tools features).

KDE Infrastructure

I’ve also prepared the steps for moving Massif-Visualizer into KDE Extragear and asked kde-devel for review. I already use the KDE infrastructure now:

Website:
https://projects.kde.org/projects/playground/sdk/massif-visualizer
Git:
git clone git://git.kde.org/massif-visualizer
Bug tracker:
https://bugs.kde.org/enter_bug.cgi?product=massif-visualizer&format=guided
Mailing List:
https://mail.kde.org/mailman/listinfo/massif-visualizer
massif-visualizer@kde.org

This also means that I’ll shortly get translations by the awesome KDE-i18n-Team, so stay tuned for a 0.3 including translations!

Open Suse Buildservice

I’ve also spent quite some time today battling with OBS and can provide at least packages for Fedora, Mandriva and Open Suse now. I’m still waiting for help on the remaining issues and once they are resolved I’ll add the remaining packages.

Changelog
  1. ChangeLog for massif-visualizer v0.2
  2. ====================================
  3.  
  4. * Milian Wolff: set version to 0.2
  5. * Milian Wolff: fix conditional
  6. * Milian Wolff: make kgraphviewer dependency optional
  7. * Milian Wolff: fix FindKGraphViewer.cmake
  8. * Milian Wolff: fix .po name
  9. * Milian Wolff: remove some esoteric cli option for XGETTEXT that does not make
  10. any sense according to Albert
  11. * Milian Wolff: fix: install libs to make sure they can get loaded on OSX e.g.
  12. * Milian Wolff: fix compile warning about init order, improve style by having
  13. just one init per line
  14. * Milian Wolff: add export macros everywhere, make visualizer helper use the
  15. Massif namespace as well
  16. * Milian Wolff: add Messages.sh

» Massif Visualizer 0.1 released

Tue, 11/02/2010 - 19:06

Good news everyone!

Since Gaël finally came around to release KGraphViewer 2.1, I can go ahead and do the same for Massif Visualizer!

Download Massif Visualizer 0.1

This is the first release and I would be very happy if more users gave me their feedback. I intend to move to git.kde.org soon in order to leverage the KDE infrastructure (mostly translations, bug tracker, releases)… This also means: There are no translations yet! I also intend to update my OBS repository to provide packages for the first release.

Stay tuned for updates.

» Massif Visualizer - now with user interaction

Sat, 03/13/2010 - 16:55

visualization of massif data #2

Just a quick status update: Massif Visualizer now reacts on user input. Meaning: You can click on the graph and the corresponding item in the treeview gets selected and vice versa. It’s a bit buggy since KDChart is not reliable on what it reports, but it works quite well already.

Furthermore the colors should be better now, peaks are labeled (better readable on bright color schemes, I’m afraid to say…), legend is shown, …

Now lets see how I can make the treeview more useful!