thoughts.on.code

Matt Payne posted a link to a Sun Book called Java Platform Performance. Thanks Matt!

I have been thinking a lot about performance lately. More so than I have in about 5 years, actually. First it was the presentation I did for my consulting company, and now I will be giving some expanded presentations in the next few months that will require a bit more thought from me, as well as some performance related work on my current project.
So, when I saw Matt’s blog post, it was like “WooHoo!!! Yet more reading!!! Yay!!!” (thats sarcasm, btw), but I need to, so I started reading. I’m not too far into it yet, (I really hate HTML-Books, but that can wait for another post but didn’t need to go far before finding something interesting.

As they should, the authors start out by asking the simplist performance question to ask, and one of the morst difficult to answer. Ok, they are all difficult, but this one seems to be even trickier:

What it performance?

The reason this is so tricky is because of perspective. In the right crowds you can talk about Transactions-per-second, or millisecond costs (or or or). In others you can say “yes, its fast”. And I’ll just ignore marketing-based performance completely for the sake of not ranting.

This falls inline with what the authors say:

Before we can discuss how to improve performance, it’s necessary to define what performance is. This isn’t as simple as it sounds-people often mean very different things when they talk about performance. There are several aspects of performance, each of which contribute to the overall performance of an application.
Computers fascinate people with their ability to carry out tasks with blinding speed. When a computer, for whatever reason, doesn’t perform a task quickly, users are disappointed. Developers often use the terms “speed” and “performance” interchangeably. However, to understand the different types of problems that can be encountered, all of the different aspects of performance must be considered:

• Computational performance
• AM footprint
• Startup time
• Scalability
• Perceived performance

These factors lay the foundation for a better understanding of the performance landscape. Some aspects of performance are primarily applicable to client-side systems, some to server-side systems, and some to both. Understanding how each factor can contribute to the performance characteristics of a system will help you analyze the performance of your own applications.

Fair enough, but I’m not satisfied.
Is there a general answer to this question?

Yes, I think we can come up with one. Lets first look at what characteristics make up Performance. “Whoa there, isn’t that the same thing as the first question?” Well, yes. But maybe breaking down the whole into smaller parts will allow us to find insight into the bigger question.

So, in my mind, Performance is made up of the following components or characteristics:
Measureable: there is no point in trying to make something faster, if you have no idea how fast it is now. You must be able to measure the output for meaningful data.

Has a Goal: there needs to be a focal point in your hunt for “Performance”. If you do not have something to look for, even something general, you will be looking at a big pile of measurements, with no idea where to start.

Limited: as with having the focused goal, you need to limit your search. It is far too easy to get caught up in what-if? games, or perhaps a little more common, have such a big a pile of measurement data to look at that you get lost trying to find your way to the restroom. Not a good thing.
Is this all of them? I’m not sure yet. So, at this point our answer is:

A focused look at a measurable aspect of a system* within a limited scope.
* system in this case could mean just about anything.
I used to have a job where I analyzed the performance of systems. Fortunately I worked with some people much smarter than me, and I learned a lot from them. One of the things I learned, came from our FAQ.

It depends.

It always depends. There are too many varables for anyone to keep straight everything involved in Performance., which is what the authors of the book are really trying to tell you. Lets make up an example not so far from the truth. Lets say you want to measure the general performance of your system, just to get a baseline. Where and when do you do this?

Unfortunately that is not an easy question to answer (see, told ya they are all hard!). Do you run the test on your production server? Probably not. Do you have an exact duplicate of your production environment (including configuration) that you could use? Very unlikely. So, you now have to take your measurements in some environment other than production and somehow extrapolate the findings from that into your production environment. So, how do you do that? Well, again a tricky answer. Do you trust the vendors information? Oh hell no (holding… back… rant…). If those numbers actually were correct, then somehow, somewhere, a performance engineer had a very impressive coup. So what do you do there? It depends. Ha!

And, so if you do by some miracle of executive foresight get the chance to do your testing in a production or near production environment, then what? You need to know when to test.

“Premature optimization is the root of all evil.”
- Hoare and Knuth [wikipedia]

Do you start testing right away, or after you are finished with development? Well, if you are an agilist, its an easy answer. The sooner you start performance testing, the better.

“But wait! Why would I want to do that right away, since you just quoted Hoare and Knuth (of all people!) on premature optimization being the root of all evil?!?!”

Who said anything about optimization? I said testing. BIG difference. What I would recommend is to find out what your performance targets are (that will have to be another post - sorry), and build into your Continuous Integration cycle a performance test, scheduled for once a day or once a week. You do have a continuous integration environment, don’t you?

In this type of environment, as long as you are under your performance targets, you are in good shape. The moment you go over your targets, then you need to do some serious tuning of the system.

I will follow up on this essay at some point in the (hopefully near) future with more about finding those pesky performance targets and some more on performance in general.