Getting a dose of Java performance wisdom: Interview with Kirk Pepperdine
Looking to fine tune your Java-based system? Then you need to hear the wise words of Kirk Pepperdine, who is hereby dubbed the all-round Java mechanic and performance specialist.
Kirk, for years you’ve been a pioneer of Java performance tuning; I’d like to talk about your “secret recipes” for speeding up Java-based systems. But first of all, could you please give us a sense of the different domains you see in the market and which ones are most performance oriented?
I need to correct something here. The real pioneer is Jack Shirazi. He wrote the first real book on Java Performance Tuning that went beyond tips that are only as good as the current release. I did have a hand in that book but it was Jack that put it together. To answer the second question first, I think that just about all domains are facing users that expect faster and faster systems. And, systems are expected to hold onto and process larger and larger quantities of data. A more general question could be; do I offer a better performance profile to my customers than my competitor does? Or, if there is a trading opportunity, is my system tuned well enough that I’m always going to win the race to it? With that question in mind, I’d say there are very few domains where performance isn’t important.
Now for the secret recipes. First the big secret, there are no secrets. Performance is all about understanding the hardware and working with it to get the best out of it. It was that way when I was working with Cray Supercomputers to today working with everyday ordinary commodity hardware. It’s been said that hardware wants to go fast and all you need to do is stay out of it’s way. Once you know how it works you’ll have a better understanding of how to stay out of the way.
Performance is obviously a complex thing – on the one hand there are many tuning aspects, all the nuts and bolts at the core level of Java; on the other hand there’s the end-to-end view on systems (from data storage to the end user’s interface). Can you give us a hint as to where one should start in general?
There are two types of performance tuning, top down and surprisingly, bottom up. The biggest gains always come from top down tuning. That is, look at the system from the highest point of view, find the biggest consumers of your latency budget and target your efforts there. Listen to the hardware, it will tell you how you’re getting in the way. This way you should always know where and why.
SEE ALSO: 10 things you didn’t know about Java
For bottom up, you want to start at the hardware level and let it tell you how you are getting in the way. This is where we might look at MSR or Machine Specific Registers. These are special diagnostic registers inside the CPU that can give you important information such as cache hit/miss ratios, instruction retirement rates and so on. Again, no secret recipes, just tell me where I’m in the way so I can get out of the way.
In the age of the Cloud, isn’t it much easier to just add more instances of a system than to tune it internally? In other words, to which degree do more resources (previously CPUs, now instances in Cloud times) help and in which area can only Swiss precision help on the core software level?
Certainly the huge win for the clouds is that it helps ease the administration burden and that makes it easier to scale out. However, if you’ve not built your system to scale out you won’t be able to use the extra hardware. Plus, you can only put so much on a single piece of hardware before you’ve overtaxed the hardware. Virtualization doesn’t buy you more hardware. It means that you have to know that your application will get along with any other application running on the same physical hardware.
I’ve seen mission critical applications in a VM on the same hardware as an Oracle DB. That simply can’t work no matter how you slice it. I’ve also seen boxes where one resource or another, typically the network card, is completely over-subscribed to. In addition, wasting a cloud instance is like wasting any other resource, you’re not going to see as big an ROI. So yes, clouds are very useful.
That said, they are not a preverbal silver bullet. You still need to be careful with how you provision hardware. Certainly, in its current incantation, you have to be very careful if you have low latency requirements. In fact for ultra-low latency bear metal is still a necessity.
And how about distributed systems, or let’s say Microservices (which tend to shift complexity from inside a monolith to the network)?
Gosh, its fun how this technology shifts about. I just recently responded to a tweet published by Eberhard Wolff where he gave a definition of micro-services. According to that definition we were building systems with micro-services long before Java was even thought of.
In ’92 I was working on a system that I’d describe as being based on Micro-services. It was a well built system even by today’s standards. All of the workflows through the micro-services could be specified with a DSL, all of the micro-services could be migrated around the compute cluster. Very flexible, very, very resilient and it performed very quickly where needed. The only downside was the network and that will be the downside today.
“Gosh, it’s fun how this technology shifts about”
If your workload isn’t large enough to offset the network distribution costs then network latency will be the dominate cost. And that is a huge cost. Some might say a cost that would cause you not to distribute your micro-services.
As you consult many different development teams all over Europe and the world – would you say that there’s a good level of performance wisdom in the community?
There are several different questions in here. Let me answer the overt one first. I would say that in general the level of performance wisdom for each team was as high as it needs to be most of the time for each team. Teams run into difficulty is when they fail to recognize that they’ve run into a problem that is bigger than they can manage or they wait too long to look for help. I can give you an example. Just a short while ago I got a call from a company that has a very strong, very performance focused group of developers. They had a problem where the application suddenly experienced a 5x increase in response times. The team spent several months for the cause before the finally reached out for help. The diagnosis took a few hours and it involved looking at part of the JVM that very few even know exist let alone know how it functions. That is my answer to the overt question. The covert question is more about the failing of the educational system to provide more training in diagnostic skills.
No post secondary institute that I know of offers a course in performance tuning. They are very good at giving people facts and recipes for building systems but there is a huge hole in the curriculum on how to perform a diagnosis. There is practically no tie into the importance of scientific methods, statistical analysis and performance benchmarking at the undergrad level. Undergrad courses at Uni are pretty much focused on pre-mature optimizations like merge sort is faster than a bubble sort, and performance estimations provided by Omega notation that assume static compilations in static run times. There are some Universities that are looking to add this to the curriculum, but I’ve not seen anything appear to date. So, this leaves people with the task of tuning but without proper tools to take on the task. What do they do? They get on with it, they muddle through and more often than not the job gets done. That said, they would have been a whole lot more efficient or effective had they’d had some formalized process to help them.
As a parting shot, that is what I believe I offer, a start at a methodology that formalizes the performance tuning process. I do not claim the methodology is complete nor would I claim it’s the only way to proceed. However, it’s a methodology that has helped many people bring structure and predicability to the muddle that is performance troubleshooting.
Kirk Pepperdine will speak at the Java Code Camp in Berlin on March 26, 2015