Detailed explanation of JAVA thread problem diagnosis tool Thread Dump

tutorials

Thread Dump is a very useful tool for diagnosing problems with Java applications. Every Java virtual machine has the ability to generate a thread-dump of the state of all threads at a given point in time. Although the thread dump printed by each Java virtual machine varies slightly, most of them provide a snapshot of the currently active thread and a stack trace of all Java threads in the JVM. The stack information usually contains the full class name and the methods executed and, if possible, the source code line number.

Thread Dump Features

  • Can be used under a variety of operating systems.
  • Can be used under various Java application servers.
  • Can be used in a production environment without affecting system performance.
  • the ability to locate problems directly to lines of application code.

Thread Dump Crawl

Generally when the server hangs, crashes or has low performance, you need to capture the server’s thread stack (Thread Dump) for subsequent analysis. In practice, a single dump is often not enough information to confirm the problem. In order to reflect the dynamic change of thread state, we need to do thread dump several times in succession, each time with an interval of 10-20s. It is recommended to generate dump information at least three times, and only if each dump points to the same problem, we can determine the typicality of the problem.

  • OS command to get ThreadDump
ps -ef | grep java
kill -3 <pid>

Caution.

Be careful, one wrong step can kill the server process. kill -9 command will kill the process.

  • The JVM comes with tools to get the thread stack
jps or ps -ef | grep java (get PID)
jstack [-l ] <pid> | tee -a jstack.log (get ThreadDump)

Thread Dump analysis

Thread Dump information

  • Header information: time, JVM information
2011-11-02 19:05:06  
Full thread dump Java HotSpot(TM) Server VM (16.3-b01 mixed mode):
  • Thread INFO message block.
1. "Timer-0" daemon prio=10 tid=0xac190c00 nid=0xaef in Object.wait() [0xae77d000] 
# Thread name: Timer-0; thread type: daemon; priority: 10, default is 5.
# JVM thread id: tid=0xac190c00, the unique identifier of the thread inside the JVM (obtained via java.lang.Thread.getId(), usually implemented in a self-incrementing way).
# corresponding to the system thread id (NativeThread ID): nid=0xaef, corresponding to the thread pid viewed by the top command, but one is in decimal and one is in hexadecimal. (With the command: top -H -p pid, you can see all the thread information of the process)
# Thread status: in Object.wait().
# Start stack address: [0xae77d000], the memory address of the object, through the JVM memory viewer tool, to be able to see where the thread is waiting on an object.
State: TIMED_WAITING (on object monitor)
3. at java.lang.Object.wait(Native Method)
4. -waiting on <0xb3885f60> (a java.util.TaskQueue) # continue waiting 
5. at java.util.TimerThread.mainLoop(Timer.java:509)
6. -locked <0xb3885f60> (a java.util.TaskQueue) # already locked
7. at java.util.TimerThread.run(Timer.java:462)
Java thread statck trace: This is the information from lines 2-7 above. By far the most important piece of data, the Java stack trace provides most of the information to pinpoint the root cause of the problem.
  • Java thread statck trace details.

Stack information should be interpreted in reverse: the program executes first on line 7, then on line 6, and so on.

- locked <0xb3885f60> (a java.util.)
- waiting on <0xb3885f60> (a java.util.ArrayList)

That means the object is locked first, locking the object 0xb3885f60, and then releasing the object lock and entering the waiting state. Why does this happen? Take a look at the following java code example to understand.

synchronized(obj) { 
 ......... 
 obj.wait(); 
 ......... 
}

As above, the thread is executed by first obtaining the Monitor of the object (corresponding to locked <0xb3885f60>) using synchronized. When obj.wait () is executed, the thread relinquishes ownership of the Monitor and enters the “wait set” queue (corresponding to waiting on <0xb3885f60>).

The state of the thread at the code level is further indicated in the first line of information on the stack, e.g.

java.lang.Thread.State: TIMED_WAITING (parking)

Explain as follows.

|blocked|
> This thread is blocked until the lock gets released.
|blocked (on thin lock)|
> This is the same state asblocked, but the lock in question is a thin lock.
|waiting|
> wait() on an object. The thread will remain there until some otherthread sends a notification to that object.
|sleeping|
Sleeping| > This thread calledjava.lang.
|parked|
> This thread calledjava.util.concurrent.locks.LockSupport.park().
|suspended|
> The thread's execution wassuspended by java.lang.Thread.suspend() or a JVMTI agent call.

Thread state analysis

The state of a thread is a very important thing, so the thread dump shows this state. By analyzing this state, you can derive the running status of the thread and thus find possible problems. The state of a thread is defined in the Thread.State enumeration type.

public enum State 
{ 
 /** 
        * Thread state for a thread which has not yet started. 
        */ 
       NEW, 
 /** 
        * Thread state for a runnable thread.  A thread in the runnable 
        * state is executing in the Java virtual machine but it may 
        * be waiting for other resources from the operating system 
        * such as processor. 
        */ 
       RUNNABLE, 
 /** 
        * Thread state for a thread blocked waiting for a monitor lock. 
        * A thread in the blocked state is waiting for a monitor lock 
        * to enter a synchronized block/method or  
        * reenter a synchronized block/method after calling 
        * {@link Object#wait() Object.wait}. 
        */ 
       BLOCKED, 
 /** 
        * Thread state for a waiting thread. 
        * A thread is in the waiting state due to calling one of the  
        * following methods: 
        * <ul> 
        *   <li>{@link Object#wait() Object.wait} with no timeout</li> 
        *   <li>{@link #join() Thread.join} with no timeout</li> 
        *   <li>{@link LockSupport#park() LockSupport.park}</li> 
        * </ul> 
        *  
        * <p>A thread in the waiting state is waiting for another thread to 
        * perform a particular action.   
        * 
        * For example, a thread that has called <tt>Object.wait()</tt> 
        * on an object is waiting for another thread to call  
        * <tt>Object.notify()</tt> or <tt>Object.notifyAll()</tt> on  
        * that object. A thread that has called <tt>Thread.join()</tt>  
        * is waiting for a specified thread to terminate. 
        */ 
       WAITING, 
 /** 
        * Thread state for a waiting thread with a specified waiting time. 
        * A thread is in the timed waiting state due to calling one of  
        * the following methods with a specified positive waiting time: 
        * <ul> 
        *   <li>{@link #sleep Thread.sleep}</li> 
        *   <li>{@link Object#wait(long) Object.wait} with timeout</li> 
        *   <li>{@link #join(long) Thread.join} with timeout</li> 
        *   <li>{@link LockSupport#parkNanos LockSupport.parkNanos}</li>  
        *   <li>{@link LockSupport#parkUntil LockSupport.parkUntil}</li> 
        * </ul> 
        */ 
       TIMED_WAITING, 
 /** 
        * Thread state for a terminated thread. 
        * The thread has completed execution. 
        */ 
       TERMINATED; 
}
  • NEW.

Each thread has a corresponding Thread object in heap memory. thread t = new Thread (); When a Thread object has just been created in heap memory, the thread is in the NEW state before the t.start () method has been called. In this state, the thread is no different from a normal java object, it is just an object in heap memory.

  • RUNNABLE.

This state means that the thread has all the conditions to run, is in the run queue ready to be scheduled by the OS, or is running. This state is normal, but if the thread stays in this state for a long time, it is not normal. This means that the thread has been running for a long time (performance problem), or the thread has not been given the chance to execute (thread starvation problem).

  • BLOCKED.

A thread is waiting for access to a java object’s monitor (also called a built-in lock), i.e., it is waiting to enter a method or block protected by synchronized. synchronized is used to ensure atomicity, and at most one thread can enter the critical area at any given time, and other threads can only wait in line.

  • WAITING.

The state of the thread that is waiting for a certain event to occur, and only if a specific condition is met will it get the chance to execute. And it is usually another thread that generates this particular event. That is, if a specific event does not occur, then the thread in that state is waiting and cannot get the opportunity to execute. For example.

Thread A calls the obj.wait () method of the obj object, and if no thread calls obj.notify or obj.notifyAll, then there is no way for thread A to resume running; if thread A calls LockSupport.park () and no other thread calls LockSupport.unpark (A), then there is no way for A to resume running. then there is no way for A to resume running. TIMED_WAITING.

Many thread-related classes in J.U.C provide both time-limited and untimed versions of the API. timed_waiting means that the thread has called the time-limited version of the API and is waiting for time to pass. When the wait time has passed, the thread can resume running as well. If a thread enters the WAITING state, a specific event must occur for it to resume; a thread in TIMED_WAITING will resume if a specific event occurs or if the time elapses.

  • TERMINATED.

A thread stays in this state when it finishes executing, when it returns normally from executing the run method, or when it throws a runtime exception and ends. At this point the thread is left with only the Thread object, which is useless.

Key state analysis

  • Wait on condition: The thread is either sleeping or waiting to be notified by another thread.

This state means that it is either waiting for another condition to occur to wake itself up, or simply that it is calling sleep (n).

At this point the thread state is roughly as follows.

java.lang.Thread.State: WAITING (parking): waiting for that condition to occur.
java.lang.Thread.State: TIMED_WAITING (parking or sleeping): timed, that condition does not come, will also wake itself up at regular intervals.
  • Waiting for Monitor Entry and in Object.wait (): The thread is waiting to get the lock for an object (some other thread may be holding the lock). This happens if two or more threads try to execute synchronized code. Note that the lock is always for an object and not for individual methods.

In multi-threaded JAVA programs, synchronization between threads, we should talk about Monitor.Monitor is the main means used in Java to achieve mutual exclusion and collaboration between threads, it can be thought of as a lock on an object or class. Every object has one and only one Monitor. The following diagram depicts the relationship between threads and Monitors, as well as the state transition diagram for threads.

thread

As shown above, each Monitor can only be owned by one thread at a given time, that thread is the “ActiveThread” and all other threads are “Waiting Threads”, waiting in two queues The other threads are “Waiting Threads” and are waiting in two queues, “Entry Set” and “Wait Set” respectively. The status of the waiting thread in “Entry Set” is “Waiting for monitor entry”, while the status of the waiting thread in “Wait Set” is “Waiting for monitor entry”. Waiting thread state is “in Object.wait ()”.

Let’s look at the threads inside the “Entry Set”. We call the section of code protected by synchronized the critical zone. When a thread requests to enter the critical zone, it enters the “Entry Set” queue. The corresponding code looks like.

synchronized(obj) {
 .........
}

There are two possibilities at this point.

  • The monitor is not owned by another thread and there is no other waiting thread in the Entry Set. This thread becomes the owner of the Monitor of the corresponding class or object and executes the code in the critical area.
  • The monitor is owned by another thread, and this thread is waiting in the Entry Set queue.

In the first case, the thread will be in the state of “Runnable”, while in the second case, the thread DUMP will show that it is “waiting for monitor entry”. As follows.

"Thread-0" prio=10 tid=0x08222eb0 nid=0x9 waiting for monitor entry [0xf927b000..0xf927bdb8] 
at testthread.WaitThread.run(WaitThread.java:39) 
- waiting to lock <0xef63bf08> (a java.lang.Object) 
- locked <0xef63beb8> (a java.util.ArrayList) 
at java.lang.Thread.run(Thread.java:595)

Critical areas are set up to ensure the atomicity and integrity of code execution within them. But because the critical zone only allows threads to pass serially at any given time, this is the opposite of the original intent of our multi-threaded program. If synchronized is used extensively, or improperly, in a multi-threaded program, it can cause a large number of threads to wait at the entrance to the critical zone, causing a significant degradation in system performance. If this is found in a threaded DUMP, the source code should be reviewed and the program improved.

Look again at the threads inside the “Wait Set”. **When a thread gets a Monitor and enters the critical zone, if it finds that the conditions for the thread to continue are not met, it calls the wait () method of the object (usually the synchronized object), abandons the Monitor, and enters the “Wait Set” queue. Only when another thread calls notify () or ** notifyAll () on the object does the thread in the “Wait Set” queue get a chance to compete, but only one thread gets the object’s Monitor and returns to the running state. The thread in the “Wait Set” is represented in DUMP as: in Object.wait (). As follows.

"Thread-1" prio=10 tid=0x08223250 nid=0xa in Object.wait() [0xef47a000..0xef47aa38] 
 at java.lang.Object.wait(Native Method) 
 - waiting on <0xef63beb8> (a java.util.ArrayList) 
 at java.lang.Object.wait(Object.java:474) 
 at testthread.MyWaitThread.run(MyWaitThread.java:40) 
 - locked <0xef63beb8> (a java.util.ArrayList) 
 at java.lang.Thread.run(Thread.java:595) 
In summary, generally when the CPU is very busy, then focus on the runnable thread, and when the CPU is idle, then focus on the thread waiting for monitor entry.
  • Lock for JDK 5.0

As mentioned above, if the synchronized and monitor mechanisms are not used properly, it may cause performance problems for multi-threaded programs. In JDK 5.0, the Lock mechanism was introduced to give developers more flexibility to develop high-performance concurrent multithreaded programs, replacing the previous synchronized and monitor mechanisms in the JDK. However,** it is important to note that because the Lock class is just an ordinary class, the JVM has no way of knowing the occupancy of the Lock object, so the thread DUMP will not contain information about Lock**, and problems such as deadlocks are not as easy to identify as with synchronized programming.