MW
Welcome Blog Code Snippets Contact

Kate/KDevelop Sprint 2014: Let There Be Clang

Time flies… The extremely productive hack sprint at the friendly Blue Systems office in Barcelona is over for more than a week. I haven’t had time to blog about the last days yet which I hereby make good for!

I spent a lot of time at the sprint polishing the KDevelop Clang plugin. A up-to-now semi-secretly developed language plugin based on clang which will replace our current C++ language support in KDevelop in the long-run.


Code completion ofstd::string in a std::vector.


Showing a clang-generated warning inside the editor.


Nested clang diagnostics with browsing support.

KDevelop’s current C++ plugin

But a bit of history first: KDevelop’s current C++ language plugin can pull off what it does by essentially implementing (parts of) the C++ standard. We have our own C++ parser, we have our own implementation of the C++ template mechanism, we have our own share of “compiler bugs”, …. It should not surprise anyone that this is a maintenance nightmare. And indeed, with C++11 and the wealth of changes it came with our language simply fails in many areas. The current codebase is big (~55k sloc) and adding support for complicated features such as variadic templates or constexpr is extremely hard and fragile. So instead of doing that, my colleague Olivier JG started looking into using clang of LLVM fame instead. It promises an API for third-party tools to steer a C/C++/Objective-C compiler to their needs. Its the magic bullet which we waited for, and which did not exist back when KDevelop 4.0 was started initially.

clang

So Olivier started hacking on kdev-clang. In his initial prototyping he experimented with the C++ API to clang but eventually found that the clang-c API suffices for our needs. The big advantage is that the latter comes with an API/ABI guarantee which is important for us as a consumer. After some failed attempts in getting clang to parse medium- sized projects at reasonable speeds, he eventually found the magic flags and options. So after the hard work was done, I started contributing by playing the plumber who massages the output of clang-c into the existing KDevelop framework.

status quo

Currently, the kdev-clang plugin consists of ca. 2000 lines of code. Add a bit more than 600 lines of tests and you are still a magnitude below our current C++ plugin. Sure, if you count the lines of code in clang/LLVM this won’t stand anymore, but from a maintenance point of view this is a blessing. We can actually concentrate on making our IDE shine instead of playing catchup with the compiler.

The plugin already supports:

What’s currently missing:

Performance

I always feared that using clang instead of our hand-written parser would lead to abysmal performance. I’ve heard bad stories about XCode 4.0 which was the first IDE to use clang internally. And the Qt-Creator guys also said they don’t want to use clang for their code model as its performance was not good enough. Thankfully, all of this turned out to be false alarm: On one hand Apple invested a lot into clang and made it much faster, also for the usage in XCode apparently. Secondly, Qt-Creator’s blazingly fast status-quo comes at the cost of only supporting a (relatively small) subset of C++. So yes, clang is slower at parsing a big project than doing the same in QtCreator. But more importantly: clang is faster than the old KDevelop C++ support! The reason is that it scales far batter with multiple cores than our old language plugin. So yes, it uses more instructions to analyse a code sample. At the same time it gives us much more than we had in terms of language support and diagnostics. But since it can do so in parallel, it finishes parsing quicker!

A bit of technical background: In our parsesession.cpp we invoke clang_parseTranslationUnit(). We configure it to use a CXTranslationUnit_PrecompiledPreamble and cache the resulting translation unit for future clang_reparseTranslationUnit() when the document is currently open. Combined, this allows for extremely fast updates when the user types something in his editor. In addition, we can use our internal cache to minimize the amount of times we must invoke clang. And I plan to investigate how to generate precompiled headers, which one could then do for third-party libraries such as the STL, Qt or the KDE frameworks. This would speed up clang even more, both inside KDevelop and while parsing! Even better, eventually we’ll get C++ modules and all of this will become extremely fast and easy to handle!

TAKE MY CODE

So the above is like a fairy tale come true from my pov! If you are thinking “shut up and tell me how to use this awesome stuff” despite of the shortcomings I listed above, then the following is just what you want.

But, first a big warning : The old C++ plugin and kdev-clang don’t like each other, in the sense of leading to runtime asserts when both are loaded into the same KDevelop session. See below for how to circumvent this.

AGAIN: Don’t run KDevelop with both, kdev-clang and kdevcppsupport!

Otherwise: Happy hacking and make sure to keep an eye on kdev-clang! I’m sure its going to progress nicely and I’m eagerly awaiting the day where I can close all C++-related bug reports and delete over 55 thousand lines of code :)

Thanks

Many thanks to Olivier for spearheading this plugin! And thanks already to the first contributors and early-adapters. Finally, once more many thanks to Aleix for organizing this awesome KDevelop/Kate sprint in Barcelona. Looking forward to the next time! Thanks also to his employer Blue Systems for providing the hack space. And as usual my gratitude to the KDE e.V. for sponsoring this event and to all the donors.

If you like Kate and/or KDevelop, please consider Joining the Game to ensure sprits such as this one can continue to take place in the future!

Comments

Want to comment? Send me an email!

Comment by Anonymous (2014-08-16 09:35:00)

I stumbled across this while KDev-CLang is probably being reworked, and couldn’t wait to try it out. Pity there seems to be no Debian/Ubuntu packaging stuff available (and that I didn’t manage whip up a kdevelop-daily ppa following the online instructions, but that’s a different issue :-/ ). So checkinstall to the rescue and we’ll see if and how much better this plugin works on my ultimate performance test platform … an AMD C60 netbook ;)

Comment by Anonymous (2014-08-16 15:52:00)

Well … couldn’t wait to try it out, but with kdevplatform,kdevelop and kdev-clang git/master/20140816 I couldn’t stop waiting for the background parser. Orders of magnitude slower than the standard C++ plugin. Will check again in a month or two…

Comment by Wolf (not verified) (2014-03-02 19:46:00)

Hi Milian,

thanks for pointing out Chandler Carruth’s talk. In my KDevelop, I am now using Meta-F as shortcut to format my source (using clang-format as custom formatter, the first C++ formatter that I really like!).

I had to program in Java for several month and I quite liked the Eclipse save actions (i.e. automatically format the source and add final to variables). This was helpful because everybody in my team was automatically formatting the code the same way - no diffs (and code reviews) caused by formatting differences. What do you think about KDevelop save actions (for example, run clang-format)?

Anyway, thanks to all KDevelop contributors!

Comment by Sergey Shambir (not verified) (2014-02-05 21:06:00)

About autocompletion: both function implementation and virtual method completions need patches to clang itself; you can find my unfinished patch here: http://llvm-reviews.chandlerc.com/D622 It would be nice if someone will finish it and will add new context to differ function implementation and static function call completions (currently there are no way to differ them).

Also you can read:

http://qt-project.org/wiki/wip-clang-todo-items

http://qt-project.org/wiki/clang-bugs-at-wip-clang-branch

http://qt-project.org/wiki/clang-hacking-notes-for-wip-clang-branch

Comment by Milian Wolff (2014-02-06 12:22:00)

Great, thanks for these links and the input Sergey!

Comment by Sergey Shambir (not verified) (2014-02-05 18:25:00)

Also take a look at QtCreator clang plugin source code: it has nice converter from clang-c completion result to QtCreator’s internal completion proposals.

Comment by Martin (not verified) (2014-02-05 13:10:00)

Great work. Thanks. Is it currently possible to configure additional paths where the parser looks for includes? .kdev_include_paths seems to be ignored.

Comment by Milian Wolff (2014-02-05 13:25:00)

https://git.reviewboard.kde.org/r/115479/

:)

Comment by Martin (not verified) (2014-02-05 14:38:00)

Well, that was… quick. :-) Looking forward for the full integration!

Comment by Alex (not verified) (2014-02-05 04:23:00)

Fantastic, I have been waiting for years for this to happen. If you remember I also played a little bit with KDevelop/CLang and even posted a small WIP video: http://www.youtube.com/watch?v=FbZ_0r4eFfo . Came to the same conclusion as you guys wrt. clang-c and the C++ API.

When we discussed CLang on the mailing list at that time, it seemed like most people prefered to stay with the old parser for the foreseeable future, but I’m glad to see that Olivier has been more persistent than I have been. ;) This is a major turning point in KDevelop’s history.

The great thing about CLang, besides removing the burden of code parsing from the KDev team and allow it to focus on more interesting areas, is that is also supports C without assuming it is C++. For kernel development this was a major drawback of the old parser and the reason why I maintain my own KDevelop branch on Github (I attempted to merge some patches to enable this in KDevelop, but they didn’t take). I suppose that C/C++ differenciation is not yet implemented in kdev-clang, but it should be easy to do so and maybe I will give it another try. Then might very well try to improve kdev-kernel now as we should end up with perfect kernel code parsing.

Thanks guys! m(__)m

Comment by Milian Wolff (2014-02-05 14:21:00)

Someone is actually working on a patch for mime-type based compiler flags for clang. So once thats in, we should get proper C support for most cases. It might still need some help and patches from our project managers (CMake,…) or some config flag though, to figure out whether headers should be parsed as C++ or C though.

Anyhow, I’m also super excited of this! I’d love to see more people using KDevelop for Kernel development or other C projects!

Comment by Milian Wolff (2014-02-05 17:32:00)

https://git.reviewboard.kde.org/r/115303/

Comment by OlivierJG (not verified) (2014-02-05 08:07:00)

To make C support usable should just be a matter of changing the args passed to Clang (currently we hardcode -xc++). Changing the arg based on the source type is something we need to do anyhow as clang wants to differentiate between c++-source and c++-header. Obj-C (for anyone watching who cares), would require more DUChain-level support.
Personally speaking I think it’d be very cool to have full KDevelop support for kernel development, I hope you find the time!

Comment by Micket (not verified) (2014-02-04 22:08:00)

This is great news. I had been hoping for this. Template code completion would be a very nice addition. I hope to find some time to help improve this plugin. Keep up the good work

Comment by Roberto Raggi (not verified) (2014-02-04 19:25:00)

Cool stuff!

about the Qt Creator C++ front end.

Well, it is fast because we designed its “distributed snapshot model” to make it support our use-cases (Qt, >1MLOC). Unfortunately, we were always busy with other “equally important components” so we called the C++ front end “good enough” :-)

These days I’m having fun writing an alternative C++ front end (and eventually a compiler) for the new draft of C++. I’m not really using C++ these days so this “little compiler experiment” is just so I can learn C++1y.

The code is WIP, I started to work on it “for real” (aka a few hours a week) on github, if you are interested go to https://github.com/robertoraggi/cplusplus the magic front end is generated using https://github.com/robertoraggi/cplusplus/blob/master/tools/pgen.cpp from https://github.com/robertoraggi/cplusplus/blob/master/src/Parser.pgen

As I said it’s WIP, don’t expect too much from the current code but as soon I’m done with pgen and a couple of new tools that I’m writing it will be awesome :-)

ciao Robe

Comment by Milian Wolff (2014-02-05 14:18:00)

Hey Roberto, long time no see!

Glad you like this. And I have to admit I laughed when I saw that wrote yet another C++ parser, do you keep count? :D

Anyhow, this sounds like a very interesting project! But do you also plan to work on semantic analysis? Or only parsing? As far as I’m concerned, even with your old parser in KDevelop it was often quite straight forward to implement new features. What was troublesome was actually interpreting the resulting AST according to the standard.

Anyhow, looking forward to what will evolve of your project! Bye

Comment by Shantanu Tushar (not verified) (2014-02-04 18:31:00)

Just tested, works as advertised. Can’t wait for this to be complete!

Comment by Laszlo Papp (not verified) (2014-02-04 16:18:00)

Sounds productive, Milian. :)

Yep, llvm sounds like a good move. Hopefully, there will be stable C++ APIs in the future.

Comment by Ian Monroe (not verified) (2014-02-04 10:11:00)

I was really happy that QtCreator was going to start clang support, since their C++ understanding is so basic. OTOH from the POV of a user clang support for KDevelop is quite unneeded. :)

That Clang is faster is a plus for sure, along with more verbose error messages. Also just compiling my code with a third compiler couldn’t hurt at finding issues. But will this enable new refactoring features? Or perhaps some interesting static analysis beyond just whether it compiles or not.

Comment by Milian Wolff (2014-02-04 12:48:00)

Hehe I’m happy that you like KDevelop’s current status so much Ian :) But really, there are many cases where it simply fails. Just try code completion on std::vector!

But you are certainly right. Once we have clang in place and can get rid of the old codebase we are open for much more interesting things. Better refactoring and tooling is certainly something I want to play around with. And clang is the optimal foundation for that! Just take a look at this e.g.: http://channel9.msdn.com/Events/GoingNative/2013/The-Care-and-Feeding-of…

So, stay tuned for a bright future :)

Comment by scroogie (not verified) (2014-02-04 08:48:00)

This is a nice article, thanks for writing it. I’ve always wondered when things like this would pop up, since first reading about the modular library architecture of LLVM/clang. Its great to hear that my favorite IDE will benefit from this. :) But out of curiosity, for a project I once had where I would have had to do a lot of repetitive refactoring I intended to write a small tool using LibTooling, but in the end turned away to stupid string replacement scripts because the documentation, of libtooling specifically, seemed a bit lacking (thats an understatement). How was the experience for you?

Comment by Milian Wolff (2014-02-04 12:45:00)

We are using clang-c which is well documented. I have not yet played with the C++ APIs such as LibTooling.

Published on February 04, 2014.

Tags: