Thu, 10/25/2012 - 13:37
Finally I take some time to blog again. I’m currently in Vienna for the joint KDevelop/Kate sprint together with lots of other hackers. Many thanks to Joseph for planning and partially financing this sprint! And of course as usual many thanks to the KDE e.V. and all the donors for bringing in the rest of the money required to pull something like this off!
Anyhow, considering that the sprint is running since Tuesday, I need to catch up quite a bit… Actually, I have to start even before that since I committed something quite noteworthy in KDevelop and KMail last week.
Reducing Memory Consumption
Shared Data References
I attended the recent Akonadi sprint that took place at the KDAB office in Berlin (where I work btw.). I heard that Alex Fiestas would come and show us his memory problems in KMail, which sooner or later was eating multiple GBs of memory for him. That sounded like a fun task to improve, fixing performance issues is what I love to do :) So I investigated it with Valgrind/Massif and my pmap script. After quite some time I came up with a patch to fix the memory increase, which is waiting for Stephen Kelly to review. It should be merged into master very soon™.
Now some technical background: What was the issue here? Why wasn’t it found earlier? Usually developers run e.g. KMail through Valgrind with the leak checker and fix all issues. The same was done lots of times in KMail, there where no problems reported. Why then is the memory still increasing over time? The issue is, that this is technically not a “leak”, i.e. when you close the application all memory is properly released. Instead, there was a logical error that resulted in KMail’s ETM (basically the item model for mails and stuff) push shared data items into a QHash without ever deleting them. If you close the app though, the QHash is cleared automatically and all shared data is properly freed, hence Valgrind won’t report any leaks.
How does one find such an issue then, though? Actually, this is really hard… I ran KMail through Valgrind and looked at the top allocation, which relates to the shared data item. But Massif will only show you where the data is being allocated, not where the shared data items are still referenced and thus prevent a proper deallocation. What now? GDB! Yes, the best way I found was adding a breakpoint in the copy constructor of the shared data class and looking at the surrounding code to see whether it behaves properly… Does anyone have a more efficient way to debug such issues? I could imagine that this can potentially take ages to figure out… In KMail at least I could find the problematic place quite fast.
Now while this apparently fixes the ever increasing memory consumption of KMail somewhat, I thought we could do some more improvements. Take a look at https://git.reviewboard.kde.org/r/106836/ e.g. This patch is similar to what I also did for Massif Visualizer once. By creating a central cache we can leverage Qt’s implicit sharing for common strings (in this case email e.g. adresses, domains, …). This way, if you load a folder containing e.g. a mailing list, you will have the main email address (like
email@example.com) only once in memory. Before, that address would be loaded into memory once for every email in the folder…
Now the above was an interesting detour into a project that I don’t usually contribute to. Since I use KMail all the time though, it is just fair to give back and help out the few KDEPIM people a bit.
Back to my favorite pet project: KDevelop :) In the spirit of the memory fixes above, I took another look at the memory consumption of KDevelop. Turns out, we had a similar issue where we did not reuse implicit sharing properly. This resulted in quite some useless allocations blowing up the memory consumption (in this case, KUrl’s of every file in the projects loaded for a session). The fix is already in master. Not only should that decrease the memory consumption considerably for kdevelop sessions with many files in it. No, it actually should save quite a few instructions and thus be much faster as well. Enjoy!
So… quite a long blog post again - sorry for that :) Expect some more KDevelop news the next days - we have lots of interesting stuff happening here at the sprint! Cheers and many thanks to Joseph and the KDE e.V. again!
Fri, 03/16/2012 - 15:42
As I just wrote in another article, Massif is an invaluable tool. The [Visualizer](https://projects.kde.org/massif-visualizer] I wrote is well appreciated and widely used as far as I can see.
A few days ago though, I did a very long (~16h) Massif run on an application, which resulted in a 204MB
massif.out data file. This proved to be a very good stress test for my visualizer, which triggered me to spent some time on optimizing it. The results are pretty nice I thing, so look forward to Massif-Visualizer 0.4:
Reduced Memory Consumption
Yeah, meta eh? Just how I like it! I’ve used Massif to improve the memory consumption of Massif-Visualizer, and analyzed the data in the Visualizer of course… :)
The initial version of my visualizer took about ~470MB of memory to load the 204MB data file above. 80% of that was required for
QString allocations in the callgraph of each detailed snapshot, i.e. the function signatures and location. See fig. 1 for the details.
QString to QByteArray
Thomas McGuire gave me the tip of using
QByteArray instead, since the Massif callgraph data is just ASCII data. We can convert the data to QString where required, essentially saving us 50% of the memory consumption. You can see that applied in fig. 2. It was simple to code and already reduced the memory consumption considerably.
I committed the above, thinking this was it. But thanks to the awesome people in the KDE community, this time André Wöbbeking, I was thankfully shown wrong: He commented on my commit, arguing that I should try out to leverage the implicit sharing of Qt containers, such as QByteArray. After all, the strings we have here are function signatures and file locations, which are repeated quite often. Especially when you have recursion in your call tree, or the same functions are encountered again and again in Massif snapshots, you can potentially safe a lot of memory by leveraging implicit sharing.
Personally, I’m suprised to see just how much this gains in this case! See fig 3., where the string allocations are nearly gone completely from the Massif log! Now only the tree node allocations, and the containers saving them, are visible in the memory log - something I do not plan to reduce further.
If you are interested in how this was implemented, take a look at commit 4be5dad13fb.
I think this shows quite nicely how to improve the memory consumption of an application. If you want to verify my results, I’ve uploaded the massif log files. Remember that you can open compressed files seamlessly in Massif-Visualizer. The
massif.out.data.bz2 file contains the test-data of the 16h Massif run.
You should probably use the latest Massif-Visualizer code though, since I’ve also optimized the performance of it considerably compared to the last released version 0.3. Furthermore, data files are now loaded in the background, showing a nice progress bar while doing that. If you open the big data file in 0.3 you’ll notice why I decided to optimize the visualizer :)
An interesting thing to note btw. is that the callgrind data format compresses files and function signatures, yielding much smaller data files and reducing the KCacheGrind’s memory consumption, esp. since it will automagically leverage the implicit sharing of Qt’s string classes.
Now it is probably time to stop slacking and start work-work again :) I do have quite a few ideas more for the next Massif-Visualizer though, especially an export functionality for the graphs is high on my TODO list!
Sun, 11/20/2011 - 19:17
I’m happy to announce the release of Massif-Visualizer 0.3. You can download the sources here:
Highlights of this release:
- translations into 18 different languages
- basic support for hiding of functions via context menu
- basic support for custom allocators
- configurable precision of memory consumption display
- various optimizations, bug fixes and other improvements. take a look at the changelog for more information
It took me much too long to get this release out and hope to do better in the future. Current git master already contains some new patches - try it out! I especially like the improved display of the callgraph which now aggregates the tails of the callgraph tree, i.e. the end of the backtrace which mostly starts
If you have ideas for the future, or have any issues with Massif-Visualizer, report a bug on https://bugs.kde.org/ .
I don’t want to waste my time on packaging and instead concentrate on development. So here again a public plead to distributors: Please include Massif-Visualizer in future releases and let users download it seamlessly through your package manager. Starting with this release, Massif-Visualizer releases will follow the usual KDE procedure and the source tarballs will be hosted on the KDE mirrors. Hope this makes it simpler for packagers!
Ubuntu users can use Aurélien Gâteau’s PPA: https://launchpad.net/~agateau/+archive/ppa . ArchLinux users you can get it via AUR, and Gentoo also has an overlay for it it seems. I also think that OpenSUSE has a package for it. Please add infos for
$your-distro in the comments below.
Sun, 11/07/2010 - 00:16
I’m happy to release Massif Visualizer v0.2. This is mainly a “fix the build-system” release, no new features have been added.
You can download it here: http://kde-apps.org/content/show.php?content=122409
Thanks to the reports by Chris Jones it’s now possible to build and use Massif Visualizer on Max OS X, see e.g.:
He has also submitted the portsfile for inclusion in Macports: https://trac.macports.org/ticket/27168
KGraphViewer now optional
I’ve made the KGraphViewer dependency optional, if anyone does not want it (even though this removes like 50% of the tools features).
I’ve also prepared the steps for moving Massif-Visualizer into KDE Extragear and asked kde-devel for review. I already use the KDE infrastructure now:
git clone git://git.kde.org/massif-visualizer
- Bug tracker:
- Mailing List:
This also means that I’ll shortly get translations by the awesome KDE-i18n-Team, so stay tuned for a 0.3 including translations!
Open Suse Buildservice
I’ve also spent quite some time today battling with OBS and can provide at least packages for Fedora, Mandriva and Open Suse now. I’m still waiting for help on the remaining issues and once they are resolved I’ll add the remaining packages.
- ChangeLog for massif-visualizer v0.2
- * Milian Wolff: set version to 0.2
- * Milian Wolff: fix conditional
- * Milian Wolff: make kgraphviewer dependency optional
- * Milian Wolff: fix FindKGraphViewer.cmake
- * Milian Wolff: fix .po name
- * Milian Wolff: remove some esoteric cli option for XGETTEXT that does not make
- any sense according to Albert
- * Milian Wolff: fix: install libs to make sure they can get loaded on OSX e.g.
- * Milian Wolff: fix compile warning about init order, improve style by having
- just one init per line
- * Milian Wolff: add export macros everywhere, make visualizer helper use the
- Massif namespace as well
- * Milian Wolff: add Messages.sh
Tue, 11/02/2010 - 19:06
Good news everyone!
This is the first release and I would be very happy if more users gave me their feedback. I intend to move to git.kde.org soon in order to leverage the KDE infrastructure (mostly translations, bug tracker, releases)… This also means: There are no translations yet! I also intend to update my OBS repository to provide packages for the first release.
Stay tuned for updates.
Mon, 03/29/2010 - 12:47
In my opinion, the massif visualizer is ready for testing. I bet there are still a few rough edges, but the most important features are in. So if you are going to do any memory profiling these days, please take a look at my tool and give me feedback. I’d be especially interested in whether the massif visualizer helps in the work flow to analyze massif data files.
My personal work flow so far is the following:
- generate massif log, one way or the other (unit tests preferred since they give you reproducible test cases)
- open log in massif-visualizer, look at overall consumption chart
- how does the memory consumption evolve? is there a memleak?
- are there designated peaks which could be reduced?
- are there any (significant) contributions to the memory consumption, that needlessly stay over the whole application life?
- to find the actual culprits in code and/or to grasp the composition of a memory peak, use the detailed snapshot analysis
I wonder how I could improve the tool to also help with verifying that a fix helps, e.g. by either overlaying two charts or by only showing the difference. The problem here is of course that it would only work with reproducible test cases and that it needs interpolation since the snapshots are taken at random points in time. Still, it would be nice to open two massif logs and seeing the impact on memory consumption of a patch visually.
Note: To anyone interested: I generate the callgraphs by converting the massif snapshot trees into a graphviz DOT file. That one I than visualize with KGraphViewer KPart. Since KGraphViewer was in no shape to visualize the huge amount of data in my use case, I had to optimize it greatly and push in some more features to make it better suitable for thirdparty users. To integrate it better, I had to write a public interface, which also means that you need KGraphViewer installed from source to be able to compile Massif Visualizer (I’ll make it optional later on). Hence get the sources from here (packages by your distributor won’t be enough): http://websvn.kde.org/trunk/extragear/graphics/kgraphviewer/
Sat, 03/13/2010 - 16:55
Just a quick status update: Massif Visualizer now reacts on user input. Meaning: You can click on the graph and the corresponding item in the treeview gets selected and vice versa. It’s a bit buggy since KDChart is not reliable on what it reports, but it works quite well already.
Furthermore the colors should be better now, peaks are labeled (better readable on bright color schemes, I’m afraid to say…), legend is shown, …
Now lets see how I can make the treeview more useful!