Thu, 10/25/2012 - 13:37
Finally I take some time to blog again. I’m currently in Vienna for the joint KDevelop/Kate sprint together with lots of other hackers. Many thanks to Joseph for planning and partially financing this sprint! And of course as usual many thanks to the KDE e.V. and all the donors for bringing in the rest of the money required to pull something like this off!
Anyhow, considering that the sprint is running since Tuesday, I need to catch up quite a bit… Actually, I have to start even before that since I committed something quite noteworthy in KDevelop and KMail last week.
Reducing Memory Consumption
Shared Data References
I attended the recent Akonadi sprint that took place at the KDAB office in Berlin (where I work btw.). I heard that Alex Fiestas would come and show us his memory problems in KMail, which sooner or later was eating multiple GBs of memory for him. That sounded like a fun task to improve, fixing performance issues is what I love to do :) So I investigated it with Valgrind/Massif and my pmap script. After quite some time I came up with a patch to fix the memory increase, which is waiting for Stephen Kelly to review. It should be merged into master very soon™.
Now some technical background: What was the issue here? Why wasn’t it found earlier? Usually developers run e.g. KMail through Valgrind with the leak checker and fix all issues. The same was done lots of times in KMail, there where no problems reported. Why then is the memory still increasing over time? The issue is, that this is technically not a “leak”, i.e. when you close the application all memory is properly released. Instead, there was a logical error that resulted in KMail’s ETM (basically the item model for mails and stuff) push shared data items into a QHash without ever deleting them. If you close the app though, the QHash is cleared automatically and all shared data is properly freed, hence Valgrind won’t report any leaks.
How does one find such an issue then, though? Actually, this is really hard… I ran KMail through Valgrind and looked at the top allocation, which relates to the shared data item. But Massif will only show you where the data is being allocated, not where the shared data items are still referenced and thus prevent a proper deallocation. What now? GDB! Yes, the best way I found was adding a breakpoint in the copy constructor of the shared data class and looking at the surrounding code to see whether it behaves properly… Does anyone have a more efficient way to debug such issues? I could imagine that this can potentially take ages to figure out… In KMail at least I could find the problematic place quite fast.
Now while this apparently fixes the ever increasing memory consumption of KMail somewhat, I thought we could do some more improvements. Take a look at https://git.reviewboard.kde.org/r/106836/ e.g. This patch is similar to what I also did for Massif Visualizer once. By creating a central cache we can leverage Qt’s implicit sharing for common strings (in this case email e.g. adresses, domains, …). This way, if you load a folder containing e.g. a mailing list, you will have the main email address (like
email@example.com) only once in memory. Before, that address would be loaded into memory once for every email in the folder…
Now the above was an interesting detour into a project that I don’t usually contribute to. Since I use KMail all the time though, it is just fair to give back and help out the few KDEPIM people a bit.
Back to my favorite pet project: KDevelop :) In the spirit of the memory fixes above, I took another look at the memory consumption of KDevelop. Turns out, we had a similar issue where we did not reuse implicit sharing properly. This resulted in quite some useless allocations blowing up the memory consumption (in this case, KUrl’s of every file in the projects loaded for a session). The fix is already in master. Not only should that decrease the memory consumption considerably for kdevelop sessions with many files in it. No, it actually should save quite a few instructions and thus be much faster as well. Enjoy!
So… quite a long blog post again - sorry for that :) Expect some more KDevelop news the next days - we have lots of interesting stuff happening here at the sprint! Cheers and many thanks to Joseph and the KDE e.V. again!