Tue, 01/21/2014 - 19:24
Hey all! Greetings from the joint Kate/KDevelop sprint at the Blue Systems office in Barcelona!
I only arrived yesterday but already I have great news for you: After months of work I finally merged the
sharedurls branches into master for KDevelop/KDevplatform etc. pp. There I worked on a optimization in our handling of file paths.
The status quo up until know was the following: When importing a folder as a project in KDevelop, we filled a model with every file and folder in the project (recursively). For every item we also stored its path as a
KUrl to the potentially remote location.
QUrl are awesome when you have to work with paths and urls, but as soon as you store potentially thousands of them at the same time it becomes quite inefficient. Assume e.g. you open
/foo/bar/blub/ which contains
/foo/bar/blub/bla.h. When you use
QUrl to store these paths, you cannot share any memory between the two, as internally basically a
QString is used. Thus, when you import deep folder trees or folders with many files, you’ll waste a lot of memory for common sub-paths. Furthermore, due to the amount of allocations required, reading the tree is pretty slow.
So in the
sharedurls branch I worked on a internal replacement for
KUrl in KDevelop: It’s called
KDevelop::Path and is a glorified
QVector<QString> with convenience API to simulate a
KUrl and simplify porting. Every entry in the vector contains a path segment. It leverages
QStrings implicit sharing to minimize the memory overhead. Furthermore, when you parse a tree structure recursively, all you do is copying vectors and appending strings to them - which is rather cheap as a
QString is a small handle structure.
So all in all this should greatly improve the performance of opening projects in KDevelop. Especially for large sessions containing thousands of files (eg.: Qt 4, multiple Qt 5 modules, LibreOffice, Kernel, WebKit, …) the new code is much faster and consumes less memory. I’ve seen time savings in the order of multiple seconds in total as well as memory consumption going down in the order of 100MB.
While this sounds like a fairy-tale, I have to admit that it was/is a lot of work: By using a custom class, you have to convert to
QString quite often when interacting with existing API. This of course is costly and potentially marginalizes or even pessimises the potential performance gains. Hence one needs to pay special attention and port code such that it minimizes these conversions. As such I can only recommend anyone doing something like that when you have similar extreme usecase. For a normal file browser or web browser I doubt the you’ll gain much if anything.
So please compile the current master branches and take a look for yourself. My tests and benchmarks look all good, yet I might have overlooked something. If you spot any regressions, please shout!
Now that this is mostly done and polished, I’ll continue working on Clang integration in KDevelop. Stay tuned for the next blog entry about that topic :) And already a huge thank you to Aleix Pol for organizing this sprint, to Blue Systems for having us, and to the KDE e.V. for sponsoring the trip and accommodation!