days
0
-10
-2
hours
-1
-3
minutes
-4
-1
seconds
-3
-7
search
Ram Lakshmanan writes...

Automating – OutOfMemoryError troubleshooting

Ram Lakshmanan
jvm
© Shutterstock / Tasty_Cat

The OutOfMemoryError can be a real pain to troubleshoot, and it’s still done manually for the most part, even in this day and age. Ram Lakshmanan teaches you how to automate troubleshooting and identifying the root cause of this error.

Troubleshooting OutOfMemoryError or any memory-related problem is done in manually even in 2019. However, even troubleshooting and identifying the root cause of OutOfMemoryError can be automated by following the 3 steps below:

  1. Capture heap dump
  2. Restart application
  3. Problem diagnosis

Let’s discuss these steps in detail.

Capture heap dump

Heap dump is basically snapshot of memory. It contains details about objects that are present in memory, actual data that is present within those objects, references originating from those objects, etc. Heap dump is vital to troubleshooting.

In order to diagnose OutOfMemoryError or any memory related problem, one would have to capture heap dump at the exact moment or few moments before the application starts to experience OutOfMemoryError. It’s hard to capture the heap dump at the right moment manually because we don’t know when an OutOfMemoryError is going to be thrown. However, capturing heap dumps can be automated by passing the following JVM arguments:

-XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath={HEAP-DUMP-FILE-PATH}

Example:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/crashes/my-heap-dump.hprof

In -XX:HeapDumpPath you need to specify the filepath where the heap dump should be stored.

When you pass these two JVM arguments, heap dumps will be automatically captured and written to specified file path when OutOfMemoryError is thrown.

Restart application

Most of the time, OutOfMemoryError doesn’t crash the application. However, it’s better to restart the application once OutOfMemoryError is thrown because it can potentially leave the application in an unstable state. Requests served from an unstable application instance can lead to erroneous results.

You can automate this restart process as well. Write a restart-myapp.sh script, which will shutdown and restart your application gracefully. Now specify this script’s path in the -XX:OnOutOfMemoryError JVM argument.

Example:

-XX:OnOutOfMemoryError=/scripts/restart-myapp.sh

When you pass this argument, JVM will invoke the /scripts/restart-myapp.sh script whenever OutOfMemoryError is thrown. Thus, your application will be automatically restarted right after it experiences OutOfMemoryError.

SEE ALSO: How much memory is my application wasting?

Problem diagnosis

Now we have captured the heap dump (which is needed to troubleshoot the problem) and restarted the application (to reduce the impact of the outage). The next step is troubleshooting. This can be a little tricky 😊, but can be achieved using the right tools. You can use tools like Eclipse MAT or HeapHero to analyze heap dumps. These tools generate good memory analysis reports, highlighting the objects that are causing memory leaks. However, most organizations do this step manually.

Even this step can be automated by invoking HeapHero REST API. This API analyzes heap dump and returns an excellent analysis report. You may invoke this API right after the restart-myapp.sh script. Thus, you will be able to automate OutOfMemoryError troubleshooting end to end.

Happy troubleshooting!

Author
Ram Lakshmanan
Every single day, millions & millions of people in North America—bank, travel, and commerce—use the applications that Ram Lakshmanan has architected. Ram is an acclaimed speaker in major conferences on scalability, availability, and performance topics. Recently, he has founded a startup, which specializes in troubleshooting performance problems.

Leave a Reply

Be the First to Comment!

avatar
400
  Subscribe  
Notify of