Discover the Power of Java Heap and Thread Dump Tools
In the earlier article published in the December 2013 issue of OSFY, the author covered the basics like dump types, dump formats, modes of acquiring dumps, the terminology associated with dumps, etc. With this grounding the reader is well equipped to delv
In this article, we explore the usage options and features available with tools from Oracle. JDK is shipped with various command line tools and a visual tool jvisualvm that is shipped with JDK 1.6, update 7. In this part, we examine jvisualvm since it is most commonly used and provides much better user interface than command line tools. Command line tools need to be used in situations where remote connection to production machines is not possible and those tools are covered in a subsequent part.
Oracle JVisualVM
JVisualVM is a memory and thread analysis tool that is shipped along with the JDK itself. This tool can be considered a combination of all command line tools. It does not work with IBM JVMs but only with Oracle JVMs. This tool is shipped by default from Java version 1.6 update 7 and is available in the JDK bin directory. JVisualVM also can be downloaded from http://visualvm.java.net. Various plug-ins are available for use with this tool. One of the most useful plug-ins is the Visual Garbage Collection (VisualGC) plug-in that can be used to visually observe the different generations on the heap and monitor their sizes, in real time. JVisualVM can access and display information about processes running locally and Java processes running on remote hosts. If JVisualVM does not start, you can try the following options. 1) Run it from the command line and set an explicit JDK:
C:\visualvm_135\bin\visualvm --jdkhome c:\jdk1.7.0_07 2) After starting, if there is an error message that states: ‘Local Java Applications cannot be monitored’, the reason could be a case mismatch between the login user name and the user name in the default application directory. To solve this, go to the system temp directory such as C:\Users\<user-name>\AppData\Local\Temp\ or C:\Temp and delete the directory hsperfdata
Overview
Troubleshooting start-up issues
Once JVisualVM is open, it automatically monitors the local machine and lists any newly started Java processes. Information about the Java process can be obtained by opening that process.
On the main landing page of local applications, JVisualVM lists the host name, operating system and the memory information— including the physical and swap size.
Monitor
The Monitor tab gives an overview about the CPU, memory, threads and the classes. The CPU tab gives information on the CPU utilisation by this Java process and the percentage of time spent in garbage collection.
The Classes tab gives information on the classes loaded and unloaded. In JVMs where class sharing is enabled, the Classes tab shows information on the shared classes. This tab also shows the number of classes loaded and the number of classes unloaded, both for JVM specific and shared classes.
The Threads tab gives information on the number of live threads and daemon threads.
Live threads show the number of threads currently started. The Live peak shows the highest number of simultaneous live threads that were running in the JVM since it was started. Total started gives the total number of threads started so far in the JVM. So, while live and daemon threads give the current information, Live peak and Total started take into account the historical (accumulated) information as well.
The memory section gives information on the memory utilisation in heap and the PermGen areas. The information consists of the current size, maximum size and the amount used in the corresponding memory area.
The heap usage graph can be used to quickly identify a memory leak. This graph typically resembles a ‘sawtooth’. The memory utilisation increases to a threshold level and falls back to a base level after a garbage collection cycle. In programs that have memory leaks, the troughs in the graph after a GC cycle show an increasing trend, i.e., a line drawn connecting the troughs in this graph will have an increasing slope.
PermGen usage also can be viewed from the PermGen tab. In most programs, this graph would be horizontal, indicating a constant usage of PermGen. Prior to Java 7, a runtime constant pool was allocated from the PermGen and, hence, if the number of constants created in the program was huge, there was a risk of running out of PermGen space. Nevertheless, it is helpful to observe both the heap and PermGen spaces on an Oracle JVM while troubleshooting memory issues.
The two buttons on top of the Memory tab can be used to force a garbage collection or take a heap dump. The Heap Dump button can be used to manually trigger a heap dump that can be loaded at a later point of time, for examination.
Threads
The Threads tab gives the complete details of all the threads —both live and the completed threads. Timeline view shows a timeline of the threads that can be scrolled to view the state changes that a thread underwent ever since it was started, right up to the time it terminated. Each state of the thread is indicated by a different colour.
Threads that have a significant monitor time are a cause for concern, as they indicate bottlenecks in the code. A monitor is used to synchronise multiple threads and if the monitor time of threads is more, it indicates that they individually ran without any issue, but were constrained by an underlying resource or a data structure that was shared among all of them.
The tabular view gives the same information but aggregated over a period of time. In the tabular view, there is no way to correlate the running state of a Thread A against, say, the wait state of Thread B. It just lists the amount of time spent in each state and the total time since the thread was started. Further detailed information on the percentage of time spent in each state, along with the exact details on when the state transitions happened, can be obtained from the Details tab. In this tab, general information on the amount of time spent can be obtained in the General tab. In the Details tab, the exact moments in time when the state transitions occurred can be obtained.
In addition to the above information, a thread dump can be taken by clicking the Thread Dump button for offline analysis of the threads.
An important feature of thread dump analysis tools, including jvisualvm, is the ability to detect deadlocks automatically. As explained in the section on jstack, these tools can identify the monitors held by threads and analyse if there are deadlocks in the program. When jvisualvm is used to profile a running application, it detects the deadlock when it occurs. The thread dump taken from jvisualvm is the same as the dump generated by jstack.
Monitoring
Monitoring is the most commonly used feature in identifying performance (response time) bottlenecks and in spotting memory issues with applications. Both CPU and memory usage can be monitored. In case of the CPU, the amount of time spent in each method is reported and, hence, the method that is taking the maximum amount of time can be easily seen. In the case of the memory, the memory consumed by different objects in the program is reported, which shows the top memory consumers.
A couple of options are available for monitoring. The first one is Sampler, which as its name indicates, just samples the application periodically and gathers CPU and memory information about the running process. The second option available is Profiler, which modifies the byte code of the running application to report the CPU and memory statistics more accurately. For instance, if Sampler runs with a frequency of three seconds, it just reports the information gathered at the three-second snapshot times. On the other hand, a profiler modifies the byte code so that classes and method report the instrumentation information to an agent. Usually these agents implement interfaces in java.lang.instrument interface. This kind of monitoring has an additional overhead on the running program, but the results in terms of the exact number of bytes used by an object, or the percentage of CPU time spent in a method, are much more accurate than a sampler. If quick information is needed without much overhead to the running application or a lot of accuracy, a sampler can be used. If more accuracy is needed, use a profiler. Using a sampler is common in situations where we may need to quickly identify a bottleneck module, class or a huge memory consuming object. A profiler could be used to accurately identify the exact problem within that module or class, or in situations where the bottleneck cannot be identified using a plain sampler.
In JVisualVM, the CPU or memory can be monitored in either Sampling mode or in the Profiling mode using the CPU or Memory buttons.
Sampler
The sampler settings can be viewed by clicking the Settings check box. This shows the options available on CPU and memory sampling. A filter on packages can be specified to either include or exclude specific packages from sampling. In addition to this filter, options are also available to adjust the rate of sampling the application and the rate of displaying the results.
In the case of the memory, only the sampling rate can be adjusted. The filter on the CPU is very useful in application server environments as well. Application server related classes such as com.ibm.* or com.oracle.* can be filtered out and profiling can be performed only on the application code.
A CPU sampler gives the amount of time spent in various methods. The three values given out are as follows.
Self-time [%]: This is the percentage of the overall time spent in a particular method.
Self-time: The amount of time spent in a method. This does not include the time spent in the methods invoked from this method. So, this does not accumulate the time spent in subsequent methods. To drill down into a particular method and find out the time spent in subsequent method calls, a snapshot can be used.
Self-time (CPU): This is the estimated real CPU time spent in executing the method. Like the self-time mentioned above, this does not include the time spent in subsequently invoked methods.
The thread CPU time tab gives the same information mentioned above on a per-thread basis instead of a permethod basis.
Snapshot
A snapshot of the running application’s CPU utilisation can be obtained by clicking the Snapshot button. In the snapshot view, a drill down of the time spent in each method can be obtained.
As can be seen from the screen shot, the full call tree of the main() method can be drilled down to identify the times spent in each sub-method. In the above example, it can be seen that intern() in String is the method that takes a significant amount of time. In addition to the call tree, the method hotspots can be viewed by clicking the Hot Spots tab. Instead of a method view, a class or package level view also can be obtained. During identification of bottlenecks, the method view is most useful.
Similar to CPU sampling, memory sampling also can be performed to identify the largest memory consumers by bytes occupied or the number of instances. Memory allocation in each memory area – heap or PermGen - can be monitored. A per thread allocation of memory can also be viewed. Like CPU sampling, memory occupied by specific classes or packages can be viewed by setting an appropriate filter.
Snapshots can be taken at various intervals. However, unlike CPU sampling, a drill down cannot be performed on memory snapshots. Snapshots taken at multiple instances of time can be compared to view the differences in the memory allocation of those objects. As in the case of viewing memory samples, the snapshot difference can also be filtered by a specific package or class. This feature is very useful when a problematic class is identified and specifics about the amount of space taken by that object are needed.
Profiler
Profiler’s user interface is very similar to the Sampler user interface. It can be invoked by clicking the Profiler tab and choosing CPU or Memory. Once this choice is made, a confirmation that byte code modification for instrumentation and profiling is taking place can be seen on the Java command line. Depending on the code base, CPU utilisation on the machine showing profiling results could take a significant amount of time since the JVM needs to instrument the classes, reload the classes in JVM and start the profiling.
Visual GC plug-in
A very useful plug-in to visually examine the different memory areas within the heap region is the VisualGC plug-in. This can be by downloaded navigating to Tools -- > Options -- > Available Plugins and choosing Visual GC.
Visual GC shows the same information as verbose GC, but in a more user-friendly way. In addition to the different memory areas, it also shows information on the Just-In-Time (JIT) compilation. The different memory areas can be tuned based on information from Visual GC. For instance, if Eden space is getting filled up frequently, but the Old Generation is underutilised, the size of Eden can be adjusted using the NewRatio parameter on the JVM command line.