days
-1
-1
hours
-1
-9
minutes
0
-8
seconds
0
-5
search
Understanding duplicate strings

UseStringDeduplication – pros and cons

Ram Lakshmanan
duplicate strings
© Shutterstock /Thammanoon Khamchalee

What are duplicate strings? 25% of Java application memory is filled up with strings, and 13.5% of those are duplicate strings. In this article, Ram Lakshmanan discusses why there are so many duplicate strings, what the common patterns are, and what to do about it.

Let me start this article with an interesting statistic (based on the research conducted by the JDK development team):

+ 25% of Java applications memory is filled up with strings.
+ 13.5% are duplicate strings in Java applications.
+ Average String length is 45 characters.

Yes, you are right 13.5% of memory is wasted due to duplicate strings 😊. 13.5% is the average amount of duplicate strings present in java application. To figure out how much memory your application is wasting because of duplicate strings, you may use tools like HeapHero, which can report how much memory is wasted because of duplicate strings and other inefficient programming practices.

What are duplicate strings?

First, let’s understand what does duplicate string mean. Look at the below code snippet:

String string1 = new String("Hello World");
String string2 = new String("Hello World");

In the above code there are two string objects string1 and string2 they have the same contents i.e. “Hello World”, but they are stored in two different objects. When you do string1.equals(string2), it will return ‘true’, but ‘string1 == string2’ will return ‘false’. This is what we call duplicate strings.

Why are there so many duplicate strings?

There are several reasons why applications end up having a lot of duplicate strings. In this section lets review two most common patterns:

SEE ALSO: Aion4j: two new tools for building smart contracts with Java

Developers create new string objects for every request, instead of referencing/reusing ‘public static final string literal’. The example below can be optimally written using string literal pattern:

public static final String HELLO_WORLD = "Hello World";
String string1 = HELLO_WORLD;
String string2 = HELLO_WORLD;

Suppose you are building banking/e-commerce applications. You are storing currency (i.e. ‘USD’, ‘EUR’, ‘INR’, ….) for every transaction record in the database. Say now customer login to your application and he is viewing his transaction history page. Now your application will end up reading all transactions pertaining to this customer from the database. Suppose this customer lives in the USA (then most if not all his transactions would be in USD). Since every transaction record has currency, your application will end up creating a ‘USD’ string object for every transaction record read from the database. If this customer has thousands of transactions, you will end up creating thousands of duplicate ‘USD’ string objects in memory that too just for this one single customer.

Similarly, your application could be reading multiple columns (customer name, address, state, country, account number, Ids,…..) from databases multiple times. There could be duplicates among them. Your application reads and writes XML/JSON with external applications, it manipulates a lot of strings. All these operations can/will create duplicate strings.

This problem has been long recognized by the JDK team since its origin (the mid-1990s), thus have come up with multiple solutions so far. The latest addition to this solution list is ‘-XX:+UseStringDeduplication’

-XX:+UseStringDeduplication

Least effort attempt to eliminate duplicate strings is to pass ‘-XX:+UseStringDeduplication’ JVM argument. When you pass this JVM argument during application startup, JVM will try to eliminate duplicate strings as part of garbage collection process. During garbage collection process, JVM inspects all the objects in memory, thus as part of that process, it tries to identify duplicate strings among them and tries to eliminate it.

Does that mean if you just pass ‘-XX:+UseStringDeduplication’ JVM argument will you be able to save 13.5% of memory immediately? :-) Sounds very easy, right? We wish it is that easy. But there are some catches to this ‘-XX:+UseStringDeduplication’ solution. Let’s discuss them:

1. Works only with G1 GC algorithm

There are several garbage collection algorithms (Serial, Parallel, CMS, G1,…). ‘-XX:+UseStringDeduplication’ works only if you are using the G1 GC algorithm. So, if you are using some other GC algorithm, you need to switch to G1 GC algorithm to use ‘-XX:+UseStringDeduplication’.

2. Works only on long lived objects

‘-XX:+UseStringDeduplication’ eliminates duplicate strings that live for a longer period of time. They don’t eliminate duplicate strings among short-lived string objects. If objects are short-lived, they are going to die down soon then what is the point spending resources to eliminate duplicate strings among them. Here is a real-life case study conducted on a major Java web application that didn’t show any memory relief when ‘-XX:+UseStringDeduplication’ was used. However ‘-XX:+UseStringDeduplication’ can be of value, if your application has a lot of caches (since cache objects typically tend to be long-lived objects).

3. -XX:StringDeduplicationAgeThreshold

By default, strings become eligible for deduplication if they have survived 3 GC runs. It can be changed by passing this ‘-XX:StringDeduplicationAgeThreshold’. Example:

-XX:StringDeduplicationAgeThreshold=6

4. Impact on GC pause times

Since String Deduplication is performed during garbage collection, it has the potential to impact GC pause time. However, assumption is that a high enough deduplication success rate will balance out most or all of this impact, because deduplication can reduce the amount of work needed in other phases of a GC pause (like reduced number of objects to evacuate) as well as reduce the GC frequency (due to reduced pressure on the heap).

SEE ALSO: The fight for performance – Is reactive programming the right approach?

To analyze GC pause time impact, you may consider using tools like GCeasy.

5. Only underlying char[] is replaced

The java.lang.String class has two fields:

private final char[] value
private int hash

‘-XX:+UseStringDeduplication’ doesn’t eliminate duplicate string object itself. It only replaces the underlying char[]. Deduplicating a String object is conceptually just a re-assignment of the value field, i.e., aString.value = anotherString.value.

Each string object takes at least 24 bytes (the exact size of a string object depends on the JVM configuration, but 24 bytes is a minimum). Thus, this feature saves less memory if there are a lot of short duplicate strings.

6. Java 8 update 20

‘-XX:+UseStringDeduplication’ feature is supported only from Java 8 update 20. Thus, if you are running on any older versions of Java, you will not be able to use this feature.

7. -XX:+PrintStringDeduplicationStatistics

If you would like to see String deduplication statistics, such as how much time it took to run, how much duplicate strings were evacuated, how much savings you gained, you may pass ‘-XX:+PrintStringDeduplicationStatistics’ JVM argument. In the error console statistics will be printed.

Conclusion

If your application is using G1 GC and running on a version above Java 8 update 20, you may consider enabling ‘-XX:+UseStringDeduplication’. You might get fruitful results especially if there are a lot of duplicate strings among long-lived objects. However, do thorough testing before enabling this argument in your production environment.

Author
duplicate strings

Ram Lakshmanan

Every single day, millions & millions of people in North America—bank, travel, and commerce—use the applications that Ram Lakshmanan has architected. Ram is an acclaimed speaker in major conferences on scalability, availability, and performance topics. Recently, he has founded a startup, which specializes in troubleshooting performance problems.


Leave a Reply

1 Comment on "UseStringDeduplication – pros and cons"

avatar
400
  Subscribe  
Notify of
Keith W. Campbell
Guest

OpenJ9 uses a different tactic to reduce string duplication: if the application calls string1.equals(string2), and the answer is true, the redundant char array is eliminated with no need for special command line arguments and almost no application overhead.

[1] https://github.com/eclipse/openj9/blob/master/jcl/src/java.base/share/classes/java/lang/String.java#L1618