What are the main skills to be a performance consultant?

Question

If I want to become a performance consultant for Java applications and systems, what are the main skills I need?

• long experience with Java itself
• using a profiler (or stack shots)
• database knowledge (to avoid/detect common performance mistakes: indexes, etc.)
• Caching library
• Java concurrency

Do you agree on the importance of these? What else would you add?

---

UPDATED from answers:

Additional Skills:

• Garbage collection, and tuning
• Efficient Java code
• Design (high-level vision)
• UI technologies (JavaScript, DOM, CSS, Swing, SWT)
• Networking understanding (also used for Ajax)
• Algorithmics, Big O
• Hardware understand for scaling

Mindset (offered by several answers, although I didn't ask for it):

• Analytical (really measure)
• Pressure-resistant
• People/political skills

Answer 1

I think the most important skill is an analytical mind able to follow the evidence and not the "gut feel" and really break down and measure the issues.

After that, it would take some creative ways to understand measuring. When someone hires a performance consultant they are often stuck, which most likely means they have real users doing real things, which means Java won't be the only thing in the stack (as you noted databases, but there may be JavaScript and networking issues and who knows what else as well).

In addition to all of that, in terms of the laundry list, the big thing missing is garbage collection. Understanding how that works and how to tweak that is critical. Many cases will involve problems with stop-the-world garbage collection just stopping an important process for too long.

You really need to have an extensive toolkit of potential solutions, as each client will need different things. Some will need an object pool for their immutable objects, some will need to introduce immutable objects to reduce synchronization, some will need to introduce mutable objects to prevent excessive object creation, etc.. Performance is really a case-by-case thing, and you need to have a range of experience and knowledge to pull from to help with each case.

Answer 2

I think hardware is also important to understand (at least it's important in designing high performing databases). When do you need to throw more hardware at the issue and when do you need to fix the code and when do you need to do both.

As a consultant on these sorts of things you are going to need serious people/political skills. You will be interviewing and collecting data from the people who already tried to fix the performance problem and couldn't. Many of them will be unhappy a consultant was called in. You'll have to deal with their resistance. IF you are going to do this as a business of your own, you will also need good accounting knowledge (and a tax consultant) and sales skills. YOu will need speaking skills to presnt your service to potential customers.

Make sure not only to measure but to document what was tried and the difference in time. Keep your own records of such things in a database and pretty soon, you will have a way to see the most likely performance tunings to try based on hard data over many clients. Developing a knowledgebase program will help you immensely as time goes on.

I'd also invest in a set of books on database performance tuning (At least one for each of the major database backends as tuning is very database specific) and database design. I think you will be able to trace many, many performance problems to bad database design and lack of knowledge of how to write good SQL code that will perform well. Database performance tuning is way more extensive than knowing what indexes to create.

Answer 3

I think that depending upon the particular client and problem at hand, you'll need to have excellent skills in algorithmic analysis and optimization. Is your problem at the math-algorithmic level, or is it at the Java-algorithmic level?

You'll also need to be very familiar with the implementations of JVM you'll be working with. Knowing the language and API isn't good enough: you'll have to know in intimate detail which language constructs to use in certain situations, and this can vary depending upon the specific JVM you are using. A profiler won't always reveal JVM-specific performance issues. As others have noted, the same JVM can behave differently on disparate harware.

Answer 4

I would add:

• Be sceptical, even about the profiler results : use your brain, first.
• Don't believe it until you see it.
• Don't blame before you measure.

Answer 5

Being able to speed up Java code. You'll most likely be called in to look at slow code, and you'll need to find and fix the slow portions. Can't speed it up, and you'll have angry clients, speed it up and everyone is happy. This could put you under some pressure to perform, if you like that kind of thing.

You could be called in to advise on designs. As a result, you'll need to know a lot of best practices, and have good design skills.

Answer 6

Also, there is a subfield of this that deals with GUI performance enhancements (many high-speed applications such as finance are written in things like Swing). That brings in a whole bunch of additional skills.

Answer 7

A strong understanding of algorithms and Big O notation. If you spend 10 hours optimizing an O(n) algorithm when a O(logn) exists you are wasting your time. I would add BigO to your list.

Also, a good understanding of the relationship between memory and CPU. Often you can trade one for the other (i.e. caching).

Answer 8

What others said, plus this. It goes along with this and this and this.

Added: Understanding Big-O, JVMs, cacheing, behavior of DBs is all important. However, when it comes to identifying the problem, you can often find it easily and quickly by a very simple procedure not requiring any special tools. It involves taking several random stackshots and looking at each one carefully, not summarizing.

This refutes much common wisdom about profiling, as you can easily prove to yourself. The links above explain it at length.

Answer 9

Some general skill requirements can be found in The Art of Application Performance Testing. Skills I've found useful are:

• Statistics - performance tests never return the exact same timings twice. You need to find out what the margin of error is in order to see if changes are statistically significant.
• business analyst skills - you need to see what the end users are doing, what parts they feel are slow, and then build up test cases around those events. At the end of your performance improvements, the users should be able to notice the difference, and your test results should quantify this.
• technology generalist - Performance problems can occur anywhere. You need to understand every bit of technology in the stack to be able to isolate the problem. Performance problems can be caused by latency, networks, database, and code. You may find developers have 4 GB of RAM on their boxes, and the data entry people who use the code have a 512 MB machine.

The advantage of being a consultant is you can take an end-to-end view. Developers probably know areas of their code which run slow, but it may not be significant to the total processing time. Yes, that ArrayList#contains call runs in O(n) time, but may only take a few ms on modern hardware.