Analyse Java Memory Dump with Eclipse
This is the third and final part of the series of articles on Java heap and thread dump tools. This article covers the Eclipse Memory Analysis Tool and the Thread Dump Analysis Tool from IBM.
The default command line tools that come with Oracle JDK are useful to a limited extent but have a steep learning curve. Another problem with the default command line tools is that these are not portable across different JVMs—IBM and Oracle. In situations where the dump needs to be taken from an IBM JVM, the Oracle tools are not very helpful.
Eclipse Memory Analysis Tool
Eclipse Memory Analysis Tool (MAT) is a free tool that can be used to acquire and analyse memory dumps from both IBM and Oracle JVMs. As discussed earlier, Oracle JVM dumps are in the HPROF format and IBM dumps usually are in PHD format. The default installation of MAT can only acquire and analyse HPROF dumps. MAT can be downloaded from: www.eclipse.org/mat. Support for IBM dumps needs to be installed separately. Instructions to install the plugin to enable PHD format support are available at: http://www.ibm.com/developerworks/java/jdk/tools/iema/. Once this plugin is installed, MAT can be used to acquire and analyse both HPROF and PHD dumps.
Acquiring heap dump
MAT can obtain heap dumps of a locally running JVM process. To acquire a heap dump, go to File Acquire→Heap Dump. This option lists the various heap dump providers— Sun and IBM. It also gives the options to pass while dumping memory. Details on the options to pass can be obtained from the MAT Help menu. In case the dump cannot be acquired from MAT for any reason, the methods described in the first article in the series can be used to obtain the dump. The dumps obtained using command line utilities can be opened
in MAT. The techniques listed in that first article are especially useful in case of IBM JVMs, where the dump provider may not be able to detect a locally running JVM. MAT can also be used to analyse heap dumps that are automatically thrown when the JVM exits due to an Out Of Memory Error.
Analysing dumps
Choose File→Open Heap Dump to open a heap dump file. This brings up a dialogue box that gives options to analyse the heap dump. The commonly used option is Leak Suspects Report.
Choosing the leak suspects brings up the Overview page that gives the overall heap information such as the heap size and the number of classes. It also reports the biggest objects in terms of memory consumption. These objects are usually a good place to start the analysis of any memory issues.
The left pane of MAT, by default, shows the object inspector view. This has two sections and is constantly available in most of the views associated with viewing objects. The top part of this view gives the object meta-data such as: Memory address of the object Class and package names Parent object Memory address of the class object and class loader object GC roots of the object The bottom part gives object values such as: Static members of the object Instance variables Class hierarchy Object value. This is not applicable for most objects, except objects such as char arrays. In order to see the biggest objects, the ‘dominator tree’ view is very useful. Dominator tree is a term used in graph theory to represent a special type of parent node in a tree. A parent node ‘x’ is a dominating node of child node ‘y’ if, from the root of the tree, all traversal paths to child node ‘y’ go through the parent node ‘x’. The same concept applies to heap analysis, since objects present on the heap can be represented as a graph due to references from one object to another. The objects represented as a dominator tree view give information about the ‘containing’ objects of a particular object or the ‘child’ objects of a particular object. This helps in identifying the biggest objects that consume memory from a GC root.
In order to understand the concept of the dominator tree better, consider the following code segment:
In the above code, the ArrayList object ‘list’ is the dominator object of all the MyBigObject instances created. So, in MAT, all the instances of MyBigObject can be viewed by expanding the ‘list’ object. The retained size of ArrayList will include the retained size of all instances of MyBigObject. On the other hand, consider the following code segment:
In the above code, ‘list’ is no longer the dominating object for all the instances of MyBigObject. This is because these instances are now accessible through another path ‘map’ as well. Hence, in the dominator tree, the retained size of ‘list’ does not include the retained sizes of instances of MyBigObject. It only contains the memory occupied by the references to these instances. This is a very important
aspect to consider while debugging memory issues. If only one collection object is expected to contain (hold references to) instances of a particular object, but in the dominator view, if that collection object’s retained size does not include the retained size of these instances, then it can be concluded that another object also is holding a reference to those objects. The other object that is holding these references could be preventing a garbage collection of these objects, thus resulting in a memory leak.
The dominator tree can be opened by clicking on the Dominator Tree link in the bottom half of the Overview page.
The objects present on the heap, traced from their GC root, are listed in the dominator tree. In the dump being considered, the largest retained heap size originates from the main thread. This thread has a reference to the instance of com.perf.memory. MemoryHogger. This object, in turn, has a reference to an instance of java.util.ArrayList that is backed by an object array. This array is the object that consumes maximum heap space. Drilling down into this object array reveals that it holds a large number of objects of type MyBigObject. This object contains another object of type TestObject and so on. The shallow and the retained size of each object is shown in the dominator tree view.
While the dominator tree view gives a graph view of objects by their retained size from the root, the histogram view lists the objects based on the instance count, shallow size and retained size. By default, neither the dominator tree view nor the histogram view differentiate between classes loaded by the JVM or classes loaded by the bootstrap class loader. Due to this, it will be very difficult to find out the objects created by the applications that are consuming memory vs the objects created by JVM itself. MAT provides a feature to group the objects by class loader. This feature is very useful in analysing the objects created by the program. Though this feature is available in all the major views, it is less useful in the dominator tree view because the dominating objects of the application program could have been created by the system class loader. For instance, in the above example, the GC root of the object consuming maximum memory – the ‘list’ object – is the main thread of type java.lang.Thread. Since this thread class is loaded by the system class loader, the dominator tree view shows the ‘list’ object in the system class loader group.
The grouping feature is much more helpful in the histogram view, where the objects loaded by the application can be viewed readily by looking at the application class loader’s objects.
This feature of grouping by class loader is more useful when analysing memory dumps from an application server such as Websphere. In most typical configurations, the class loader isolation policy for Web and enterprise applications is set and, hence, each application gets its own class loader. In this situation, even though code in different applications could be in different packages, classes for supporting functions such as logging could be common across all applications. In this kind of situation, it is very important to find out to which application a particular object belongs to. Grouping by class loader comes in very handy in troubleshooting memory issues in these conditions.
An interesting observation from the histogram in Figure 6 is that the objects do not show their parent or child references. To get those references, right click an object and choose Merge Shortest Paths to GC. This action shows which GC root is holding a reference (direct or via another object) to this object. This is essentially the same as going back to the dominator tree view, but for the selected object alone.
A couple of utilities are available in MAT to show the memory wastage in the application. The Java collections utilities show the wastage in collection objects and the hash collisions
in Map objects. This is useful in finding out if searches can be optimised by choosing a better hash code algorithm for the keys or by setting a different load factor for these maps.
Prior to Java 1.7, substring() implementation in java. lang.String returned a string object that was still backed by the original char array, but with a different offset and starting pointer. This was done to make substring() method faster. But this resulted in a memory leak because even though an original large string was eligible for garbage collection, its underlying huge char array could not be garbage collected even if there was one small sub-string object created. This wastage can be viewed in the ‘Waste in char arrays’ feature.
In addition to the tools described above, MAT also has an OQL console, where specific queries can be entered and executed. The OQL console can be launched by clicking the OQL button. The exact syntax of the OQL commands can be obtained by hitting F1 in the OQL console.
IBM Thread Analyser
IBM Thread and Monitor Dump Analyser for Java can be downloaded from the IBM developer works site: https:// www.ibm.com/developerworks/community/groups/service/ html/communityview?communityUuid=2245aa39fa5c4475b89114c205f7333c
This is in the form of an executable jar file. It can be used to analyse thread dumps created by IBM JVMs. As mentioned in the introduction, thread and heap dumps for an IBM JVM can be triggered by sending a Control+Break signal to the JVM. Once the dump is generated, the core file, usually with the name javacore.<timestamp>.txt can be opened in the IBM Thread Analyser. Like JVisual VM’s thread analyser, the IBM Thread Analyser also detects any deadlocks in the running application and reports them when the dump is loaded.
When a dump file is loaded, the Java system properties and environment variables are listed along with the thread details. In addition, a heap usage summary at the point where the dump was taken and information on previous garbage collection cycles is given. Most of the useful features are available in the ‘Analysis’ menu option.
The basic analysis that can be done is Thread Status Analysis. This gives a graphical view of the threads, based on their states. The details of those threads are displayed in the bottom panel. For each thread, the name, state, native thread ID, the Java method and stack depth are displayed. Individual threads from this view can be selected and details about them are displayed on the right hand side. The Java stack traces of a thread, along with the monitor that it owns and the monitor it is waiting on, are displayed. This is useful in identifying the piece of code that has caused a deadlock (if any) and the monitor that is needed to break a deadlock.
The tool also distinguishes between threads that are waiting on a monitor and threads that are waiting on a condition. A thread that is waiting to acquire a lock is in BLOCKED or waiting on lock condition. A thread that is waiting to be notified, usually via notify() or notifyAll(), of some condition so that execution can proceed will be in WAITING or wait on condition state. In general, threads that are in BLOCKED state are a cause of concern as they indicate a lock condition. Threads in WAITING state may or may not be a cause of concern based on which monitor’s condition they are waiting on. There can be overlaps in thread states. For instance, a blocked thread could also have been deadlocked. In this case, the status is displayed as Deadlock/Blocked. The percentage of threads in each state, displayed in Thread Status Analysis, could exceed 100 per cent because threads in dual conditions, as described above, are reported under both states.
The Method Analysis view organises the threads based on their status and the Java method they are in. No Java stack can be reported in case of the native code.
A very useful feature in analysing the performance of applications is the ability to compare two thread dumps. By comparing the thread dumps taken at different points in
time, the state changes each thread has undergone can be understood and, possibly, the reason behind a deadlocked or a poorly responding thread can be understood better. To compare thread dumps, load multiple thread dumps, choose the dumps to compare, right click and choose Compare Threads. Similarly, the monitors owned by threads in different dumps can be visualised by choosing Compare Monitors.
The native memory analysis gives information on the various memory areas, threads, JIT compiler details and class libraries in a single view. This is useful to analyse the overall memory profile of the JVM, i.e., analyse the memory utilisation of each of these memory areas. The Monitor Detail view is the most useful tool in quickly identifying the monitor dependencies among the threads that have led to a deadlock. The cyclic dependency amongst threads and the monitors that caused these dependencies can be visualised by drilling down the threads. Clicking on a thread gives information on the monitor owned by the thread and the monitor it is waiting on. It can be seen that this monitor is being held by another thread.
A variety of free tools is available to profile and analyse Java applications in order to find memory issues and performance bottlenecks. Most of the tools provide the basic information required to perform the analysis. Advanced features such as Object Query Language (OQL) can be used by experts to perform more detailed analysis. The choice of tool depends on the familiarity with the tool and the kind of JVM being monitored (IBM or Oracle). Expertise in the JVM memory and threading concepts is more important for analysing performance issues than for choosing a tool.