Making Java Agent work better on Dragonwell

tutorials

Background

Java Agent technology can dynamically modify Java application behavior without reworking the code.

It is because of these features that many middleware teams, cloud vendor teams, and open source products, started using Java Agent technology to provide some basic capabilities, such as Apache Skywalking, OpenTelemetry all provide Java Agent.

Earlier, middleware teams provided capabilities through SDKs (e.g., observable, microservice governance capabilities, etc.); however, each time middleware teams added new features and fixed defects, they needed each business party to update the SDK version and re-release it.

As the company’s architecture becomes more and more complex, and as cloud vendors start to provide middleware capabilities, this way of pushing SDK users to update one by one becomes more and more troublesome.

With Java Agent, the business students only need to write business code; the middleware capability is dynamically injected into Java Agent by setting environment variables, and the update of Java Agent only requires restarting the application.

Problem

Let’s take a microservice demo as an example. First deploy the demo in a Kubernetes cluster, then use the Java Agent via JAVA_TOOL_OPTIONS to.

1 2 $ echo $JAVA_TOOL_OPTIONS -javaagent:/home/admin/.opt/ArmsAgent/arms-bootstrap-1.7.0-SNAPSHOT.jar …

When we log into the container, we can see the injected Java Agent:.

java-agent-in-openjdk.png

But this results in all the JVMs in the container, mounting the Java Agent.

For example, to execute a java -version that also mounts the Java Agent.

java -version

And also jstack will load Java Agent :

jstack

  1. java/jps/jstack/jcmd and other problem troubleshooting tools that come with the JDK will go ahead and load the Java Agent from the environment variable.
  2. because Java Agent is loaded at the beginning of the JVM, Java Agent will first take 6-7s to load the agent logic.
  3. but as a JDK tool, there is actually no business logic and does not require microservice governance capabilities. It can be loaded without Java Agent.

We imagine this scenario: the online application has a problem, the operations and maintenance students to catch the scene, go up and want to jstack to pull the stacktrace information. The result is that the java agent has to be loaded first, which not only wastes CPU and memory, but also makes it easier to miss the scene of the problem troubleshooting.

But, while trying to inject Java Agent without intrusion through environment variables, and not injecting it within certain processes. Seems insurmountable?

Fix

First of all, it is the JVM that determines whether to inject Java Agent or not. We just need to modify the JVM’s logic to determine this. The Dragonwell team has a lot of experience in this area.

Second, let’s look at the behavior of the JVM. The existing open source behavior is as follows.

  • JDK related commands, all load Java Agent from JAVA_TOOL_OPTIONS
  • After OpenJDK9, JDK_JAVA_OPTIONS was introduced, this environment variable will only be used by java commands. jps/jstack and other commands will not be loaded.
  • Some JDK vendors, will have their own extended environment variables, for example, IBM will read IBM_JAVA_OPTIONS, after the open source OpenJ9 started to use OPENJ9_JAVA_OPTIONS, and Oracle/OpenJDK use _JAVA_OPTIONS.

Originally JDK_JAVA_OPTIONS can well meet the needs, but as a “you send any you send, I use Java8” business development students, stability first, so Java 8 must be supported.

After discussing with Dragonwell, the following changes were made.

  • DRAGONWELL_JAVA_OPTIONS, similar to IBM_JAVA_OPTIONS, to set certain Java parameters only for Dragonwell’s Java parameters.
  • DRAGONWELL_JAVA_TOOL_OPTIONS_JDK_ONLY, similar to JDK_JAVA_OPTIONS. If DRAGONWELL_JAVA_TOOL_OPTIONS_JDK_ONLY = true, then JAVA_TOOL_OPTIONS will only be loaded by the java command. jps/jstack will not load the JDK_JAVA_OPTIONS environment variable and will not load the Java Agent.

With the above modification, it is possible to load the Java Agent only for business Java processes. At the same time, it does not affect jps/jstack and other JDK’s own operation and maintenance commands.

Of course, Dragonwell as an open source project, the whole process of discussion is done on GitHub Issue, welcome to watch, eat and spit: https://github.com/alibaba/dragonwell8/issues/330#issuecomment-1138083844

The final result

Let’s use the latest Dragonwell version, run the business application, and simulate the O&M scenario:.

Dragonwell

As you can see, having JAVA_TOOL_OPTIONS ensures that the business process loads the Java Agent, and also ensures that java -version and other JDK operations tools don’t load the Java Agent anymore.

Summary

In the cloud-native domain, Java Agent serves as a non-intrusive, low-overhead runtime enhancement capability, and business students can access observable, microservice governance and other middleware capabilities by modifying one line of code differently. Let the business focus on business value.