Stop Watches Anyone? (Or "About Continuous Profiling")

One of the most interesting talks at the latest Community One in San Francisco, NetBeans track, was the speech by NetBeans lead architect Yarda Tulach about the performance improvements in the latest NetBeans IDE 6.1. Yarda also talked about the impact of performance tuning on the development process and—summing up—one of the main points is that you should do continuous profiling. In other words, you could be happy with the performance of a certain feature, then you start fixing bugs, adding new functions, etc... and you don't realize that the performance is slowly degrading; and at a certain point has degraded so much that you need some major refactoring. It's more advisable to keep it under control continuously and immediately be aware if something is getting slower.

I've actually experienced this during blueMarine development in the hard way, something that now I can't afford any longer as I get closer to a real release. My beta testers have been pretty clear about their performance expectations of some features and one of the real, "political" disadvantages of Java is that the persisting myth "Java is slow", while false, can give you additional problems if you do something wrong in this area. Just be inefficient once in your code, and somebody will soon poo-poo your desktop application since it's "Java crap" (that is, even if the error is yours, and you can actually fix it, you risk a perpetual refusal of your application because of the technology it is made with).

So I've started doing some periodical, manual measurements of performance in the most important areas of blueMarine. So far I'm running some special load tests, getting the numbers out of it and putting them into an OpenOffice spreadsheet where I can track the progress, see trends and eventually make comparisons between figures etc... (the very first example has been talked about on my blog a couple of months ago).

Now, in perspective, I see the usual problem of "continuous" tasks: it's taking longer and longer, and the more performance tests I add, the worse. So this must be automated in some way, and possibly integrated with Hudson.

Of the many problems about continuous profiling, the very first is how to collect data. Now, NetBeans has got an excellent profiler (apart from a few bugs that are causing me troubles in my specific RCP scenario), but in my opinion this is a different thing:

1. The Java profiler is a specific development tool, while I want to be possible that some profiling data can be collected at any time with the production code, even by some (beta) tester. While it's true that the Java 6 profiler is much easier to set up, it's still a JDK tool that would scare a casual user; what I want is a specific UI option pane where the user can click on a checkbox and possibly select the functional area to profile.
2. The Java profiler is not "semantic aware". What I mean is that it can tell me how many times and how long a certain method—say loadImage()—has been executed, while I need e.g. to segment the collected data in function of the parameters (for instance, the image type and so on). Please read the example below for a better clarification of the point.
3. The Java profiler collects too many data for a "continuous profiling" approach, and this of course also impacts on the performance. Of course, with the Java profiler it's perfectly possible to fine tune the profiling points, but again this is a task for a programmer, not for a (beta) user.

Of the previous points, I'd say that the stronger is the number #2, considering that improvements in the Java Profiler may weaken the #1 and #3 as time goes on.