Wed, 04/28/2010 - 19:19
Yay I got a GSOC slot :)
So I hope I don’t have to introduce myself anymore to you guys. Instead I’ll show you what I’ve planned to do over the summer:
Motivation for Proposal / Goal:
Back in KDE 3 times, Quanta+ was one of the reasons for me to use KDE. In my eyes it was the IDE for web development out there, and I loved to use it. Sadly it’s bitrotting nowadays without a finished KDE 4 port. That, combined with the fact that more and more distributions drop all KDE 3 packages, makes the need for a port more urgent than ever.
Thankfully, KDevelop 4 is nearing it’s first release and the KDevplatform is mature enough nowadays. This means that during summer I shall finish the port of Quanta+ to KDevplatform and supply it with all the plugins required for a proper webdevelopment IDE. My goal is it to provide a proper IDE for PHP webdevelopment. In more detail:
- make Quanta+ 4 compile
- remove obsolete plugins or code parts in Quanta+
- port required plugins to KDevplatform structure
- polish PHP plugin, including XDebug support
- polish Script Execute plugin
- polish CSS plugin
- get a first working version of a XHTML/XML plugin, if time allows even with HTML (SGML) support
- support autocompletion
- support inline validation
- polish the UI/Workflow for Webdevelopment
- hide KDevelop/C++ specific actions
- add templates for common PHP frameworks
Put these all together with the existing features in KDevplatform we can reuse, we’ll end up with a hopefully useable IDE for webdevelopment. Hence my final goal is it to release a first Beta version of Quanta+ for KDE4.
- getting rough first shell of Quanta+ 4 up and running, removing old cruft, cleaning up old code and porting required things
~ 3 weeks
- polish existing plugins (PHP, XDebug, Execute Script, CSS, Upload)
~ 2 weeks
- create XHTML/XML plugin
~ 3 weeks
- polish UI/workflow
~ 2 weeks
- bug hunting etc., ending in a first beta release of Quanta+ for KDE 4:
~ 2 weeks
Lets see whether it works out as planned. But I think this commit shows you that I’m on the right track:
Sun, 04/25/2010 - 18:15
I’m now abusing the fact that my blog is aggregated on the planet to bring this diamond of a documentary some more coverage it deserves so greatly. I’m speaking about Rip: A remix Manifesto. Go and watch it. Now!
I bet every single FOSS user, developer, advocate thrives in watching it. I’m totally blown away and hope that as many people as possible watch it.
And gosh - open source cinema, how cool is that :)
Mon, 04/05/2010 - 03:12
Ok, you should know by now that I love profiling and making things faster. Yet there’s always a “but”. For me it’s blocking syscalls, or anything that makes the app “slow” for the user but doesn’t show up in Callgrind as the Instruction Fetch cost doesn’t go up.
The usual suspect is of course locks (which we have quite a lot in KDevelop) or QProcesses with
waitForFinished() or similar… You won’t see them in any Callgrind profile. Does anyone know a way to achieve that? Something that makes Callgrind increase the Ir cost for blocking func calls depending on the time it blocks? Or some other tool that would show me these?
And if you are interested: I was still able to find the cause for slow parsing of Custom Make Manager projects (Qt, Linux Kernel, …) in KDevelop: The cache in the IncludePathResolver never hit, since a
operator== was improperly implemented ;-) I really wonder how we could have missed that for so long! I’ve also added some more changes that should make it much faster to parse projects that rely on the IncludePathResolver. I was personally now able to parse 10.000 files of the Linux Kernel in about 9.5 minutes. This is roughly a third of the Kernel, so I’d get to a total of approx 30min. Compare that to the 2.5h for 5% that one of our users reported ;-)
Wed, 03/31/2010 - 01:24
I just need to get this out quickly:
We were aware that KDevelop’s CMake support was slow. Too slow actually. It was profiled months ago and after a quick look that turned up QRegExp, it was discarded in fear of having to rewrite the whole parser properly, without using QRegExp. Which btw. is still a good idea of course.
But well, today I felt like I should do some more tinkering. I mean I managed to optimize KDevelop’s Cpp support recently (parsing Boost’s huge generated template headers, like e.g.
vector200.hpp is now 30% faster). I managed to make KGraphViewer usable for huge callgraphs I produce in Massif Visualizer. So how hard could it be to make KDevelop’s CMake at least /a bit/ faster, he?
Yeah well an hour later and two commits later, I managed to find and fix two bottlenecks. Both where related to QRegExp. Neither was the actual parser, instead it was the part that evaluated CMake files, esp. the
STRING(...) function. So even if we’d used a proper parser generator, this would still been slow.
The first one was the typical “don’t reinvent the wheel” kinda commit which already made the CMake support two times faster for projects that used
FindQt4.cmake, i.e. any Qt or KDE project. Not bad, right? Well, while I fixed that I saw that KDevelop tried to do some Regular expression replacement on the output of
qmake --help, this could not been right, could it? With help of Andreas and Aleix we found the bug in the parser and that made the CMake support 10 times faster.
So yeah, CMake projects using Qt or KDE should now get opened a whopping 20 times faster in KDevelop :)
I really love KCacheGrind and Valgrind’s callgrind - again it proved to be the most awesome tool one can imagine! If you are interested in the callgrind files:
Note: with KCacheGrind from trunk you can open these compressed files transparently :)
Mon, 03/29/2010 - 12:47
In my opinion, the massif visualizer is ready for testing. I bet there are still a few rough edges, but the most important features are in. So if you are going to do any memory profiling these days, please take a look at my tool and give me feedback. I’d be especially interested in whether the massif visualizer helps in the work flow to analyze massif data files.
My personal work flow so far is the following:
- generate massif log, one way or the other (unit tests preferred since they give you reproducible test cases)
- open log in massif-visualizer, look at overall consumption chart
- how does the memory consumption evolve? is there a memleak?
- are there designated peaks which could be reduced?
- are there any (significant) contributions to the memory consumption, that needlessly stay over the whole application life?
- to find the actual culprits in code and/or to grasp the composition of a memory peak, use the detailed snapshot analysis
I wonder how I could improve the tool to also help with verifying that a fix helps, e.g. by either overlaying two charts or by only showing the difference. The problem here is of course that it would only work with reproducible test cases and that it needs interpolation since the snapshots are taken at random points in time. Still, it would be nice to open two massif logs and seeing the impact on memory consumption of a patch visually.
Note: To anyone interested: I generate the callgraphs by converting the massif snapshot trees into a graphviz DOT file. That one I than visualize with KGraphViewer KPart. Since KGraphViewer was in no shape to visualize the huge amount of data in my use case, I had to optimize it greatly and push in some more features to make it better suitable for thirdparty users. To integrate it better, I had to write a public interface, which also means that you need KGraphViewer installed from source to be able to compile Massif Visualizer (I’ll make it optional later on). Hence get the sources from here (packages by your distributor won’t be enough): http://websvn.kde.org/trunk/extragear/graphics/kgraphviewer/
Sat, 03/13/2010 - 16:55
Just a quick status update: Massif Visualizer now reacts on user input. Meaning: You can click on the graph and the corresponding item in the treeview gets selected and vice versa. It’s a bit buggy since KDChart is not reliable on what it reports, but it works quite well already.
Furthermore the colors should be better now, peaks are labeled (better readable on bright color schemes, I’m afraid to say…), legend is shown, …
Now lets see how I can make the treeview more useful!
Thu, 03/11/2010 - 23:43
I just committed an (imo) insanely useful feature for KCacheGrind: Transparent loading of compressed Callgrind files. Finally one does not have to keep those Callgrind files around uncompressed, hogging up lots of space. And what is even more important: It’s much easier to share these files now, as you can send or upload them as
.gz or better yet
.bz2 and open them directly. KDE architecture just rocks :) So in KDE 4.5 the best profiling visualizer just got better :D
In related news: I’m spending my time as intern at KDAB currently by creating an application to visualize Massif. If you are interested, check the sources out on gitorious: http://gitorious.org/massif-visualizer
It’s still pretty limited in what it offers, yet is probably already more useful than the plain ASCII graph that
This is very WIP but the visuals are somewhat working now. I plan to make the whole graph react on user input, i.e. zoomable, click to show details about snapshots, show information about the heap items that make up the stacked part of the diagram, …
Also very high on my wish list is some kind of interaction with the KCacheGrind libraries, to reuse it’s nice features like callgraphs, cost maps, etc. pp. you name it :) All these features that make KCacheGrind such an insanely useful application.
Oh and remember: Never do performance optimizations without checking the facts first ;-)
Fri, 02/26/2010 - 16:45
steveire is experimenting with QML so I couldn’t stop but notice that there is no highlighting for it in Kate. Well, there was none ;-) Now you get pretty colors, rejoice!
.js as well. Enjoy!
Wed, 02/17/2010 - 19:43
Woha, quite a few days flew by without me blogging about anything. Thankfully the others started to write so I don’t have to repeat it all ;-) Instead I’ll concentrate on stuff I did or learned.
GHNS for Snippets
Well, first I think an excuse is in oder: There is a GHNS button for Kate Snippets in 4.4.0 but it’s broken, neither me nor Joseph had time to acutally use and fix it… But anyways, I fixed it now for 4.4.1. For 4.5 we’ll also have an Upload Dialog.
I also added both now to KDevelop, you can now upload and download snippets from it. I added a few dump examples but will probably improve it steadily.
On Saturday and Sunday I started to profile Kate highlighting for a large MySQL dump and managed to greatly improve the speed. Actually the funny thing is that I could improve RegExp based highlighting (you still should try to prevent using it, it will always be slower than simple char/string based highlighting). And the knowledge for this optimization I had from my time as an active contributor for GeSHi. I feel like it was ages ago, he funny :)
So if anybody has a big file that takes ages to load (but only if you use KDE 4.5 trunk or higher), tell me and give me the file. I might find some more ways to optimize different languages.
Other than that I also managed to fix a few more bugs in Kate and KDevelop and had a good time with the other guys here in Berlin. Yesterday I showed some others how I spent quite a few nights: Partying in Berlin is nice :)
Oh and Bernhard showed me a little gem of his: Kate Standalone which you can use to build only KatePart from kdelibs trunk. This actually works very well (up until some new API from KDE trunk is used).
Sun, 02/14/2010 - 01:58
So, first day of the Kate/KDevelop hacksprint.
We just talked and hacked at the rented flat,got to know each other and had a fun time. Everybody made it more or less in time, even last minute attendee Adymo from Ukraine, nice! Hacking-wise the productivity wasn’t that high, esp. for me, but a few patches got committed here and there.
Right now I’m working on a little speedup for Kate, esp. for big MySQL files - lets see how it turns out. Cullmann showed me a few things I could do so maybe it works out, lets see.
Over the next week I plan to push in user configurable include paths for the PHP plugin and do some more Snippets & Scripting work in Kate, lets see how it turns out. I’ll go home now, kinda sucks that I don’t stay with the others here at the flat but have to take a 1h ride into the city… Berlin is definitely too big :D