Recently I encountered a problem, a Java process using Runtime().exec() to execute script files, creating a large number of zombie processes, while the Java process is running in the container.

At that time, I saw this situation on the Host machine, and I could see that there were a large number of zombie processes.

java process

Locate the zombie process by ps aux | grep Z.

1
2
3
4
5
6
7
8
9

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      2518  0.0  0.0      0     0 ?        Z    Jan23   0:00 [docker] <defunct>
root      2521  0.0  0.0      0     0 ?        Z    Jan23   0:00 [docker] <defunct>
root      2533  0.0  0.0      0     0 ?        Z    00:19   0:00 [docker] <defunct>
root      2535  0.0  0.0      0     0 ?        Z    00:19   0:00 [docker] <defunct>
root      2539  0.0  0.0      0     0 ?        Z    Jan23   0:00 [docker] <defunct>
root      2541  0.0  0.0      0     0 ?        Z    Jan23   0:00 [docker] <defunct>
root      2546  0.0  0.0      0     0 ?        Z    Jan23   0:00 [docker] <defunct>
...

Then pstree -apscl <pid> found that it was created by some Java process at

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

systemd,1 maybe-ubiquity
  `-containerd-shim,6387 -namespace moby -id 4a720871cfb7fdbce7a33eb89cb723acec09bdc61f73a2de8ea81e20973f57d1 -address/run/
      `-java,6505...
          |-(docker,324)
          |-(docker,325)
          |-(docker,326)
          |-(docker,354)
          |-(docker,362)
          |-(docker,364)
          |-(docker,369)
          |-(docker,375)
          |-(docker,384)
          |-(docker,386)
          |-(docker,393)
          |-(docker,401)

Reading the code then reveals that the Java program executes the shell script in two ways.

1
2

Process p = Runtime.getRuntime().exec(new String["<cmd>"]);
Process p = Runtime.getRuntime().exec(new String["sh", "-c", "<cmd>"]);

Roughly speaking Zombie process appears on child processes: when a child process finishes execution, but its parent does not read its exit status via the wait system call, it is not cleared in the process table, so it becomes a zombie process.

The Process objects in the code are destroyedForcibly().waitFor() in finally, which means that it ends normally.

Later, by elimination, we found that the problem lies in this execution of sh -c , and a minimally reproducible version of DockerZombie.java is provided here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class DockerZombie {

  public static void main(String[] args) throws IOException, InterruptedException {

    ExecutorService executorService = Executors.newFixedThreadPool(10);

    String[] cmd;

    if ("sh".equals(args[0])) {
      cmd = new String[] {"sh", "-c", "ls"};
    } else if ("/bin/sh".equals(args[0])) {
      cmd = new String[] {"/bin/sh", "-c", "ls"};
    } else if ("/bin/bash".equals(args[0])) {
      cmd = new String[] {"/bin/bash", "-c", "ls"};
    } else {
      cmd = new String[] {"ls"};
    }

    while (true) {
      executorService.submit(() -> {
        try {
          runCmd(cmd);
        } catch (InterruptedException e) {
          e.printStackTrace();
        } catch (IOException e) {
          e.printStackTrace();
        }
      });
    }
  }

  private static void runCmd(String[] args) throws InterruptedException, IOException {
    BufferedReader stdout = null;
    Process process = null;
    try {
      process = Runtime.getRuntime().exec(args);
      stdout = new BufferedReader(new InputStreamReader(process.getInputStream()));
      String line = stdout.readLine();
      if (line == null) {
        return;
      }
    } catch (IOException e) {
      e.printStackTrace();
    } finally {
      process.destroyForcibly().waitFor();
      stdout.close();
    }
  }
}

The above code provides 4 ways to execute scripts, and the following table shows the corresponding docker commands.

script execution method docker command
sh -c <cmd> docker run --rm docker-zombie:latest sh
/bin/sh -c <cmd> docker run --rm docker-zombie:lest /bin/sh
/bin/bash -c <cmd> docker run --rm docker-zombie:lest /bin/bash
<cmd> docker run --rm docker-zombie:lest foo

If the base image is using openjdk:8, openjdk:8-slim then.

  • the first two definitely generate zombie processes and the number spikes
  • The last two will also be generated, but in small numbers

Corresponding to Dockerfile.

1
2
3
4
5
6

FROM openjdk:8
RUN mkdir /java-app
WORKDIR /java-app
COPY DockerZombie.java .
RUN javac DockerZombie.java
ENTRYPOINT ["java", "DockerZombie"]

If the base image is using openjdk:8-alpine then.

then all will generate zombie processes, but the number is small and stable at about 10 Corresponding Dockerfile.

1
2
3
4
5
6
7

FROM openjdk:8-alpine
RUN apk add bash
RUN mkdir /java-app
WORKDIR /java-app
COPY DockerZombie.java .
RUN javac DockerZombie.java
ENTRYPOINT ["java", "DockerZombie"]

How to cure this problem, we will provide solutions in subsequent articles.