Banks “pay 33% to 50% more” in developer salaries
Finance may be a “dry subject”, says HFT programmer Peter Lawrey, but it’s never bad to work for an employer that has money. Stack Overflow’s most active Java commenter explains what it’s like to work in high frequency trading and banking.
How did you end up programming in high frequency trading. Is it your range of skills that got you there, or a clear interest?
An interest in low latency coding, a background in working for finance, breadth of knowledge of hardware and software, and some luck.
I had programmed nearly every day since I was 14 until about a year after I graduated. After that I spent seven years as a Unix Administrator and some time in Technical Pre Sales for Sun Microsystems and Project Management, always looking for a new challenge. However, I always loved developing software and 16 years ago I went back into software development and at the time, fell into Java development.
Again looking for new challenges, I moved to the UK for more opportunities and worked in finance for a number of years before realising what I really found interesting was low latency trading. This combined solving tricky problems over very short time intervals, with instant feedback in term of the difference you made to the business. Often you can see the difference you are making to the groups profit and loss the same day you make a change.
I was also lucky enough to be doing this at a time when using Java for low latency applications was emerging.
Do you ever encounter resistance from clients that are scared of innovation in finance technology?
In HFT, innovation is critical and this is well understood. They make money from competitive advantage and they are more interested in opportunity risk. How much more money could I have made if I had acted sooner?
“In HFT, innovation is critical”
Investment Banks tend to be much slower to move and much more risk adverse. They are more reluctant to be the first to try something. Once an innovation take hold and a few banks are using it, others often want to not be left behind. The area they are most interested in is saving money. Banks are very process orientated and over time many of these processes have been automated in IT systems.
IT costs banks a lot of money and efficiency gains are important to them. Efficiency improvement helps them offer better services, for less money.
What would you say are the main differences between programming for finance tech and other enterprise technologies?
Finance is a fairly dry subject. Being risk adverse, the last thing you want is too much excitement. There is less emphasis on look and feel and more emphasis on functionality. On the plus side, one thing banks do have is money, and they can pay 33% to 50% more in terms of salary for the same job and you get to work on more expensive systems.
“It is next to impossible to find a female Java developer
with ten years fintech experience.”
IMHO, one of the challenges which face banks is how to improve diversity to ensure they have the best talent available. Banks have a lot of difficulty attracting and retaining women in IT, much more so than other areas of IT. It is next to impossible to find a female Java developer with ten years fintech experience. If you are a woman with ten years fintech and Java experience, I would be very happy to hear from you.
What kind of problems are you needing to reproduce frequently in order to get the tech right? And what are the advantages to determinism in finance-related programming?
The key problem I try to solve is having a deterministic system, both in terms of behaviour and latencies. In finance, a single business unit can turn over billions of dollars in a day. Even the smallest error can cost a lot of money and a big error can mean the end of your business in minutes.
You want a system which behaves in a well understood manner, and if there is an error, you want to a complete history of exactly why or why not the system performed an action. You need a very cheap way to record everything, every input and output, with detailed timestamps to see delays in micro-seconds.
The advantage of such a model is that you can have systems which are much more responsive; for example around 100 micro-second latencies 90% of the time, external to your network, is not unusual in Java. The systems are designed to handle very high loads, e.g. peaks of 25 million transactions per second. Also they can turn over incomprehensible amounts of money which can be amazing to watch when it works.
Nowhere does latency matter more than in financial IT systems. In your experience, what are the most common causes of high latency?
IMHO, the most common causes of latency are poor measurement. You can’t fix what you can’t see. A common mistake is to only look at throughput or it’s inverse, average latency. This can hide some appalling latencies.
“The most common causes of latency are poor measurement.”
Only after having profiled the latency distribution with the aim of looking for problems, do you find that relatively rare events like waiting for network services, GCs, or page faults are introducing delays.
Are there any features from the latest 8u40 release of Java you’re excited about, or potential Java 9 features you look forward to?
The main feature of Java 8 which I love is Escape Analysis. It was in previous version of Java but didn’t do very much. In Java 8, when there is enough inlining, the JVM can work out that an object doesn’t need to be created on the heap, but instead sits on the stack (or in many cases optimised away completely). This reduces work for the CPU but produces less garbage or even no garbage.
Java 8 update 40 in particular has improved flight recorder functionality. I suggest you try it for your application. It doesn’t replace a commercial profiler IMHO, but it is very useful complimentary tool. I find that it’s not as easy to use as a profiler, but the information that is there is more accurate for profiling.
You can hear Peter speak about low latency at the JAX Finance conference in London, April 28-29.