Kate/KDevelop Sprint Vienna 2012 Take 1
Hello everyone!
Finally I take some time to blog again. I’m currently in Vienna for the joint KDevelop/Kate sprint together with lots of other hackers. Many thanks to Joseph for planning and partially financing this sprint! And of course as usual many thanks to the KDE e.V. and all the donors for bringing in the rest of the money required to pull something like this off!
Anyhow, considering that the sprint is running since Tuesday, I need to catch up quite a bit… Actually, I have to start even before that since I committed something quite noteworthy in KDevelop and KMail last week.
Reducing Memory Consumption
KMail
Shared Data References
I attended the recent Akonadi sprint that took place at the KDAB office in Berlin (where I work btw.). I heard that Alex Fiestas would come and show us his memory problems in KMail, which sooner or later was eating multiple GBs of memory for him. That sounded like a fun task to improve, fixing performance issues is what I love to do :) So I investigated it with Valgrind/Massif and my pmap script. After quite some time I came up with a patch to fix the memory increase, which is waiting for Stephen Kelly to review. It should be merged into master very soon™.
Now some technical background: What was the issue here? Why wasn’t it found earlier? Usually developers run e.g. KMail through Valgrind with the leak checker and fix all issues. The same was done lots of times in KMail, there where no problems reported. Why then is the memory still increasing over time? The issue is, that this is technically not a “leak”, i.e. when you close the application all memory is properly released. Instead, there was a logical error that resulted in KMail’s ETM (basically the item model for mails and stuff) push shared data items into a QHash without ever deleting them. If you close the app though, the QHash is cleared automatically and all shared data is properly freed, hence Valgrind won’t report any leaks.
How does one find such an issue then, though? Actually, this is really hard… I ran KMail through Valgrind and looked at the top allocation, which relates to the shared data item. But Massif will only show you where the data is being allocated, not where the shared data items are still referenced and thus prevent a proper deallocation. What now? GDB! Yes, the best way I found was adding a breakpoint in the copy constructor of the shared data class and looking at the surrounding code to see whether it behaves properly… Does anyone have a more efficient way to debug such issues? I could imagine that this can potentially take ages to figure out… In KMail at least I could find the problematic place quite fast.
Implicit Sharing
Now while this apparently fixes the ever increasing memory consumption of KMail somewhat, I thought we could do some more improvements. Take a look at https://git.reviewboard.kde.org/r/106836/ e.g. This patch is similar to what I also did for Massif Visualizer once. By creating a central cache we can leverage Qt’s implicit sharing for common strings (in this case email e.g. adresses, domains, …). This way, if you load a folder containing e.g. a mailing list, you will have the main email address (like list@domain.org) only once in memory. Before, that address would be loaded into memory once for every email in the folder…
Now the above was an interesting detour into a project that I don’t usually contribute to. Since I use KMail all the time though, it is just fair to give back and help out the few KDEPIM people a bit.
KDevelop
Back to my favorite pet project: KDevelop :) In the spirit of the memory fixes above, I took another look at the memory consumption of KDevelop. Turns out, we had a similar issue where we did not reuse implicit sharing properly. This resulted in quite some useless allocations blowing up the memory consumption (in this case, KUrl’s of every file in the projects loaded for a session). The fix is already in master. Not only should that decrease the memory consumption considerably for kdevelop sessions with many files in it. No, it actually should save quite a few instructions and thus be much faster as well. Enjoy!
So… quite a long blog post again - sorry for that :) Expect some more KDevelop news the next days - we have lots of interesting stuff happening here at the sprint! Cheers and many thanks to Joseph and the KDE e.V. again!
| Attachment | Size |
|---|---|
| kmail-shareditems-leak.png | 299.81 KB |
Comments
Want to comment? Send me an email!
Comment by Robert (not verified) (2012-10-26 12:04:00)
It’s been a while since I did Qt coding, but from the linked patch I can’t see how any of those strings ever get freed. Surely the QSet keeps a reference to them (keeping them alive) forever.
No?
Comment by Juan (not verified) (2012-10-26 13:10:00)
I see the same issue. You are using String sharing to reduce memory consumption, but unfortunately, not in a clever way, so you introduce a memory leak with this patch. You are pushing strings to a set, but you never delete it when are not needed any more, so you has a memory leak. The only way to retrieve this memory is restarting Kmail, but this may be not an acceptable solution for someone who works all the day with kmail. I think that the correct solution is some like the way that the amarok team resolved this issue. In fact, is straightforward to delete a string that is not needed anymore by any other structure (hint: always use a ref count like system for shared resources).
Comment by Milian Wolff (2012-10-28 12:57:00)
Both of you are right in that this is also “leaking”, but we are aware of that. What you should take into account though is that even though we “leak” here, this is a great improvement (see analysis and graphs). That is also why we agreed to merge this even if it is leaking technically.
I agree though that a proper refcounted implementation would be cool to have in future. Sadly though, a clean solution for that is not yet easily done. Take into account that QStrings are implicitly shared (i.e. refcounted), but you have no way to get notified that a refcount dropped to 1 (i.e. the instance that lives in our cache) and then act upon that and remove it from the hash as well. The only solution possible right now is adding a second custom refcounter on top that notifies the cache when appropriate which is ugly from an technical POV. Still, we’ll probably implement something like that in the future.
Comment by Franz (not verified) (2012-10-26 10:23:00)
Great! Those commits reduced opening kdelibs from >50minutes (!already loaded, so no really new import!) to ~5 minutes. But that’s still more then qtcreator (30 secs to 1 min). I watched top + iotop during kdevelop loading project “kdelibs”. Though the number of CPUs is set to 2 (the default for duchain) top shows me a usage of 40-60% - qtcreator goes up to >100%. Setting used cores up to 4 (or even 8) (I have an i7 2600K) seems to reduce perfomance even more. But what’s even worse is iotop. kdevelop seams to only read files from time to time, with a ridiculous speed of <100kB/s. creator produces a quite continuous read load of >700kB (goes up to 2MB, which kdevelop never ever reaches!)
/dev/sda: Timing cached reads: 27866 MB in 2.00 seconds = 13948.70 MB/sec Timing buffered disk reads: 390 MB in 3.01 seconds = 129.48 MB/sec
This is while many apps (inc. 3 instances of kdevelop and 2 of qt-creator) are running.
Is there a way to get this optimized? I love kdevelop because it finds usage of SIGNALS/SLOTS all over the source files, which creator does not. But having this extreme startup delay turned me over to creator :(
And a short suggestion: I am currently writing a template based header-only lib. There is absolutely no need for a buildsystem, but projects depend on cmake (qmake). It would be nice to have such a build-system-less project/template :)
Comment by Milian Wolff (2012-10-28 13:08:00)
Wow that performance increase sounds… fishy. While I do think it will be faster somewhat, an increase of an order of magnitude should not happen here. I’d be interested in what version of kdevelop you compare against, maybe there are more improvements since you tried it the last time?
Yes, the multi core performance of KDevelop is still abysmal. The reason is mostly the freakish amount of locks and locking which lead to a huge lock contention. Sadly, to fix this will be a huge task and take quite some time. I have some ideas but no idea when I get to finally implement them.
Regarding project manager: We have the generic manager and the custom build system, which should help you out. Try them!
Comment by Franz (not verified) (2012-10-28 15:43:00)
calligra was the other project that drove me crazy ;) When I knew I would find the time to look into it I started kdevelop before breakfast and let kdevelop run the whole day :D I started the calligra-session some weeks ago and it took the usual time to load - I can’t say exactly when that was, but when I tried it now after the commit also calligra loaded really fast :) (I am following git quite some time, as I need kdev-qmake - that’s only available through git and needed git at the time I installed it).
I will try the custom build system - if I can find it :( It is not listed here in the project template view (project -> new from template), and I can find nothing in the menus. When it is on the startpage - will try as soon as kdevelop compiles again:
/var/tmp/paludis/dev-util-kdevelop-9999/work/kdevelop-9999/languages/cpp/cppduchain/templateresolver.h:56:20: sorry, unimplemented: non-static data member initializers /var/tmp/paludis/dev-util- kdevelop-9999/work/kdevelop-9999/languages/cpp/cppduchain/templateresolver.h:56:20: error: ISO C++ forbids in-class initialization of non-const static member ‘valid’
As I have updated kdevplatform which has made some incompatible changes I can’t start kdevelop ATM (crash on startup).
Comment by Milian Wolff (2012-10-30 21:46:00)
a) kdevelop and kdevplatform master compile just fine for me, so you do something wrong. b) neither it is crashing for me on startup, so again - you probably do something wrong.
for both things though, more information is required, and double check you don’t mix binary incompatible libraries/plugins (i.e. see the HowTo Compile wiki page and it’s find command).
Regarding custom build system: A template would be cool, true, but you can just open/import an existing project or folder for now and select the project manager there. Its shipped with kdevelop master btw.
Comment by Franz (not verified) (2012-10-31 06:42:00)
commit 6ca2eb653242015dff7dc808b714c3e68f6b8fe6 already fixed the compilation issue (gcc >= 4.7 would be needed for non- static data member initializers) And after recompiling kdevelop starts up fine, again.
OK, that’s what I already thought was the desired way. But I need at least a Makefile, then. So I could also create a minimum CMakeLists.txt.
Comment by Ian Monroe (not verified) (2012-10-25 22:05:00)
In Amarok 1.4 we had a cache just like that, using implicit sharing, since obviously in a music player you have a lot of repeat strings for albums and artists.
When we refactored the code in Amarok 2 one side effect was that the cache wasn’t needed, since now tracks of the same artist share a common Artist data structure. So that’s another solution to the same problem.
Comment by isemenov (not verified) (2012-10-25 21:49:00)
Happy devs living in Europe, I envy the increased mobility granted by the EU integration. Hope to join a sprint sooner or later, too, an experience that would be both very interesting and novel to me.
The work you are doing is of real interest to me, too, since this kind of programming (debugging/profiling/bugfixing) is not covered by any course or textbook, and one can learn to do that only by practicing a lot. I also find KDE software to be very difficult to debug or profile, partly due to the qt signals/slots and delayed event processing, partly to the multi-thread/process architecture, party to the weird ways of starting programs (knotify4 restarts on auto, to start it in gdb or valgrind, you need to mod the .desktop file in a certain way) or kdeinit modules or dbus autostart (brrrr), large parts of code being plugins, not executables, and so on. So I admire any kind of optimization/profiling or complex debugging performed on KDE software. Keep up the good work!
The first RB link seems to be off-topic, it is a kdelibs RR, not kmail or akonadi :)
Comment by Milian Wolff (2012-10-25 22:05:00)
Ups, fixed the RB link…
Comment by Mark (not verified) (2012-10-25 15:38:00)
Oh, could you look into the memory usage of Akregator and Konqueror too? Both use a lot of memory. It seems like memory of closed tabs is never release. There is an long outstanding bug on this, but devs have ruled that it is “very hard to fix”. https://bugs.kde.org/show_bug.cgi?id=196051