Java generates a lot of zombie process problems
Recently I encountered a problem, a Java process using Runtime().exec() to execute script files, creating a large number of zombie processes, while the Java process is running in the container.
At that time, I saw this situation on the Host machine, and I could see that there were a large number of zombie processes.
Locate the zombie process by ps aux | grep Z
.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2518 0.0 0.0 0 0 ? Z Jan23 0:00 [docker] <defunct>
root 2521 0.0 0.0 0 0 ? Z Jan23 0:00 [docker] <defunct>
root 2533 0.0 0.0 0 0 ? Z 00:19 0:00 [docker] <defunct>
root 2535 0.0 0.0 0 0 ? Z 00:19 0:00 [docker] <defunct>
root 2539 0.0 0.0 0 0 ? Z Jan23 0:00 [docker] <defunct>
root 2541 0.0 0.0 0 0 ? Z Jan23 0:00 [docker] <defunct>
root 2546 0.0 0.0 0 0 ? Z Jan23 0:00 [docker] <defunct>
...
Then pstree -apscl <pid>
found that it was created by some Java process at
systemd,1 maybe-ubiquity
`-containerd-shim,6387 -namespace moby -id 4a720871cfb7fdbce7a33eb89cb723acec09bdc61f73a2de8ea81e20973f57d1 -address/run/
`-java,6505...
|-(docker,324)
|-(docker,325)
|-(docker,326)
|-(docker,354)
|-(docker,362)
|-(docker,364)
|-(docker,369)
|-(docker,375)
|-(docker,384)
|-(docker,386)
|-(docker,393)
|-(docker,401)
Reading the code then reveals that the Java program executes the shell script in two ways.
Process p = Runtime.getRuntime().exec(new String["<cmd>"]);
Process p = Runtime.getRuntime().exec(new String["sh", "-c", "<cmd>"]);
Roughly speaking Zombie process appears on child processes: when a child process finishes execution, but its parent does not read its exit status via the wait system call, it is not cleared in the process table, so it becomes a zombie process.
The Process objects in the code are destroyedForcibly().waitFor() in finally, which means that it ends normally.
Later, by elimination, we found that the problem lies in this execution of sh -c , and a minimally reproducible version of DockerZombie.java is provided here.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class DockerZombie {
public static void main(String[] args) throws IOException, InterruptedException {
ExecutorService executorService = Executors.newFixedThreadPool(10);
String[] cmd;
if ("sh".equals(args[0])) {
cmd = new String[] {"sh", "-c", "ls"};
} else if ("/bin/sh".equals(args[0])) {
cmd = new String[] {"/bin/sh", "-c", "ls"};
} else if ("/bin/bash".equals(args[0])) {
cmd = new String[] {"/bin/bash", "-c", "ls"};
} else {
cmd = new String[] {"ls"};
}
while (true) {
executorService.submit(() -> {
try {
runCmd(cmd);
} catch (InterruptedException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
});
}
}
private static void runCmd(String[] args) throws InterruptedException, IOException {
BufferedReader stdout = null;
Process process = null;
try {
process = Runtime.getRuntime().exec(args);
stdout = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line = stdout.readLine();
if (line == null) {
return;
}
} catch (IOException e) {
e.printStackTrace();
} finally {
process.destroyForcibly().waitFor();
stdout.close();
}
}
}
The above code provides 4 ways to execute scripts, and the following table shows the corresponding docker commands.
script execution method | docker command |
---|---|
sh -c <cmd> |
docker run --rm docker-zombie:latest sh |
/bin/sh -c <cmd> |
docker run --rm docker-zombie:lest /bin/sh |
/bin/bash -c <cmd> |
docker run --rm docker-zombie:lest /bin/bash |
<cmd> |
docker run --rm docker-zombie:lest foo |
If the base image is using openjdk:8
, openjdk:8-slim
then.
- the first two definitely generate zombie processes and the number spikes
- The last two will also be generated, but in small numbers
Corresponding to Dockerfile
.
FROM openjdk:8
RUN mkdir /java-app
WORKDIR /java-app
COPY DockerZombie.java .
RUN javac DockerZombie.java
ENTRYPOINT ["java", "DockerZombie"]
If the base image is using openjdk:8-alpine then.
then all will generate zombie processes, but the number is small and stable at about 10 Corresponding Dockerfile.
FROM openjdk:8-alpine
RUN apk add bash
RUN mkdir /java-app
WORKDIR /java-app
COPY DockerZombie.java .
RUN javac DockerZombie.java
ENTRYPOINT ["java", "DockerZombie"]
How to cure this problem, we will provide solutions in subsequent articles.