- Plugin Development Guide
- References
The core pivot of a Javaagent is the ability to enhance specific methods to implement enhanced business, such as Tracing and Metric business. Therefore, EaseAgent needs an easy-to-understand and use, efficient, and reliable plugin framework that allows users to easily enhance specific methods to achieve business requirements.
To make the plugin framework easy to understand and use, we have abstracted the plugin into the three elements, Points, Interceptor, and AgentPlugin.
- Points is used to define where to enhance.
- Interceptor is used to define what to do at enhanced Points.
- AgentPlugin makes the plugin configurable and configuration can be updated dynamically at runtime.
Efficient and reliable, the architecture design needs to address two situations.
- Multiple plugins perform multiple enhancements to a method. We let multiple plugin interceptors on the same method form an interceptors Chain. Then, let each method point only be enhanced by a simple piece of bytecode, and allocate a unique Id (unique Index) for the enhanced method, which is used as an array index to get the corresponding Interceptors Chains.
- Plugins can be independent or collaborative. In an Interceptors Chain, Interceptors can be scheduled by priorities and a mechanism for exchanging data between interceptors is provided.
Moreover, Interceptors are enhanced to achieve business requirements, we provide a set of APIs for the most common Tracing and Metric services so that the enhancement plugin can complete the Tracing and Metric collection requirements quickly with the help of API. The Report component is responsible for formatting the data and uploading it to the backend server. It can also be customized and extended to meet the needs of different data formats and network architectures.
This document describes how to develop plugins for Easeagent, and it will be divided into the following sections to introduce plugin development.
- Plugin structure and examples, the plugin contains three core components, which are the Points, Interceptor, and AgentPlugin definition.
- Tracing API, which helps users complete the transaction tracing task.
- Metric API, helps users to complete metrics data collection.
- Logging API
- Configuration API
- Debug FAQ
All plugin modules are located in the plugins folder under the top-level directory of the Easeagent project and a plugin module can contain several plugins, eg. a "Tracking Plugin" and a "Metric Plugin".
As mentioned before, we abstract the plugin into the "three elements" corresponding to three interfaces, Points, Interceptor and AgentPlugin. The development of a plugin is an implementation of these three interfaces, which complete the definition of where to enhance, what to do at the enhancement point and the configuration of the plugin respectively.
Points implementation specifies methods that will be enhanced and if a dynamic private field with access methods for that field is added to instances of matched classes.
When there is only one MethodMatcher in the return set of getMethodMather(), the qualifier value defaults to 'default', and there is no need to explicitly assign a value.
When there are multiple methods in a matched class that require enhancement with different interceptors, a qualifier needs to be assigned to each MethodMatcher as the keyword used by different interceptors to bind.
Decoupled from ByteBuddy, the EaseAgent "ClassMatcher" and "MethodMatcher" were designed by learning the ByteBuddy's DSL, instead of using the ByteBuddy interface directly
The DSL of ClassMatcher and MethodMatcher is described in Matcher DSL
public interface Points {
/**
* return the defined class matcher matching a class or a group of classes
* eg.
* ClassMatcher.builder()
* .hadInterface(A)
* .isPublic()
* .isAbstract()
* .build()
* .or()
* .hasSuperClass(B)
* .isPublic()
* .build())
*/
IClassMatcher getClassMatcher();
/**
* return the defined method matcher
* eg.
* MethodMatcher.builder().named("execute")
* .isPublic()
* .argNum(2)
* .arg(1, "java.lang.String")
* .build().toSet()
* or
* MethodMatcher.multiBuilder()
* .match(MethodMatcher.builder().named("<init>")
* .argsLength(3)
* .arg(0, "org.apache.kafka.clients.consumer.ConsumerConfig")
* .qualifier("constructor")
* .build())
* .match(MethodMatcher.builder().named("poll")
* .argsLength(1)
* .arg(0, "java.time.Duration")
* .qualifier("poll")
* .build())
* .build();
*/
Set<IMethodMatcher> getMethodMatcher();
/**
* When returning true, the transformer will add an Object field and an accessor
* The dynamically added member can be accessed by AgentDynamicFieldAccessor:
*
* AgentDynamicFieldAccessor.setDynamicFieldValue(instance, value)
* value = AgentDynamicFieldAccessor.getDynamicFieldValue(instance)
*/
default boolean isAddDynamicField() {
return false;
}
}Interceptor is the core of implementing specific enhancements.
Interceptor interface has a name method getType() and a initialization method init.
- The name will be used as
typein combination with thedomainandnamespaceof the binding plugin to get the plugin configuration which will be automatically injected into theContext. The description of plugin configuration will be given in AgentPlugin session. - The
initmethod is invoked during transform, allowing developers to initialize static resources of an interceptor, and also allowing them to load third party classes which can't load by runtime classloader.
The before and after methods of the interceptor are invoked when the method being enhanced enters and returns, respectively.
Both before and after methods have parameters MethodInfo and Context.
MethodInfocontains all method information, including class name, method name, parameters, return value and exception information.Contextcontains the Interceptor configuration that is automatically injected and updated and other interfaces that supporttracing, for details, please refer to the Tracing API section.
public interface Interceptor extends Ordered {
/**
* @param methodInfo instrumented method info
* @param context Interceptor can pass data, method `after` of interceptor can receive context data
*/
void before(MethodInfo methodInfo, Context context);
/**
* @param methodInfo instrumented method info
* @param context Interceptor can pass data, method `after` of interceptor can receive context data
*/
default void after(MethodInfo methodInfo, Context context) {
};
/**
* Interceptor can get interceptor config thought Config API :
* EaseAgent.configFactory.getConfig
* Config API requires 3 params: domain, nameSpace, name
* domain and namespace are defined by the plugin, the third param, the name is defined here
*
* @return name, eg. tracing, metric, etc.
*/
default String getType() {
return Order.TRACING.getName();
}
/**
* Initialization method for the interceptor,
* This method will be called and only be called once for every method which is injected by this interceptor,
* which means this method may be called several times, when there are several methods matched
*
* @param config interceptor configuration
* @param className injected method's class name
* @param methodName injected method name
* @param methodDescriptor injected method descriptor
*/
default void init(Config config, String className, String methodName, String methodDescriptor) {
}
}The Interceptor interface also includes the Order interface that defines the order of the interceptors.
When several Interceptors are injected into a single enhancement point, the order of execution of the Interceptors is determined by the Order of the Interceptor and the Order of the Plugin the interceptor belongs to.
Effective Order = Interceptor Order << 8 + Plugin Order
The before method of the interceptor will be invoked in descending order of the Effective Order, that is, the smaller order will have higher priority.
The after method will be invoked in the opposite order as the before method was invoked.
Within a plugin, there may be multiple interceptors, and multiple enhancement points, so which enhancement point is a particular interceptor used for?
This can be specified through the @AdviceTo annotation, which is applied to the Interceptor's implementation to specify the enhancement point binding with the Interceptor.
/**
* use to annotate Interceptor implementation,
* to link Interceptor to Points and AgentPlugin
*/
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Repeatable(AdvicesTo.class)
public @interface AdviceTo {
Class<? extends Points> value();
Class<? extends AgentPlugin> plugin() default AgentPlugin.class;
String qualifier() default "default";
}The @AdviceTo annotation associates an Interceptor to the enhanced Points implementation specified by value(), and more specifically to the method matcher in that Points with the name specified by qualifier() which defaults to "default".
@AdviceTo(value = DoFilterPoints.class, plugin = SimplePlugin.class)
public class ResponseHeaderInterceptor implements Interceptor {
}The @AdviceTo annotation also binds the Interceptor to a specific plugin via plugin(), which gives the Interceptor the ability to fetch dynamically updated configurations of the plugin. The Interceptor's plugin configuration is accessible via the Context's getConfig() method, the details will be described in the following AgentPlugin section.
Plugin definition defines what domain and namespace of this plugin by implementing the AgentPlugin interface.
public interface AgentPlugin extends Ordered {
/**
* define the plugin name, avoiding conflicts with others
* it will be used as a namespace when getting the configuration.
*/
String getNamespace();
/**
* define the plugin domain,
* it will be used to get configuration when loaded:
*/
String getDomain();
}The domain and namespace of a plugin determine the configuration prefix for each interceptor bound to the plugin by @AdviceTo annotation.
The format of the plugin configuration is defined as follows.
plugin.[domain].[namespace].[type].[key] = [value]
Take the tracing switch of httpclient as an example.
plugin.observability.httpclient.tracing.enabled=true
domain : observability
namespace : httpclient
type : tracing
key : enabled
value : true
[domain] and [namespace] are defined by AgentPlugin interface implementations.
The type is provided by the Interceptor interface implementation's getType() method, and this method needs to return a String value like 'tracing', 'metric', and 'redirect' which are already defined by Easeagent, or any other user-defined keyword.
This prefix plugin.[domain].[namespace].[type] is used to maintained configuration for this Interceptor, and in this Interceptor developer can get its configuration by the getConfig() method of the Context param.
The AgentPlugin interface also includes the Order interface that defines the order of the plugins, which is related to plugin orchestration. Plugin orchestration will be described in the Plugin Orchestration section.
Only three interfaces need to be implemented to complete the development of a simple plugin. Although plugin development varies in complexity depending on the business, for illustrating the plugin mechanism, the simplest Simple plugin with only three interface implementations is the most appropriate.
Suppose this plugin is to help complete the Tracing function, the name of the current microservice needs to be added to the Response header. So we can define the domain of the plugin as "observability", and then we will give the simple plugin a namespace "simple". The full source code is available here.
This plugin contain only three interface implementations: AgentPlugin, Points and Interceptor, corresponding to the classes SimplePlugin, DoFilterPoints and ResponseHeaderInterceptor respectively.
# Full code structure
▾ src/main/java/com/megaease/easeagent/plugin/simple/
▾ points/
DoFilterPoints.java
▾ interceptor/
ResponseHeaderInterceptor.java
SimplePlugin.java
pom.xml
public class DoFilterPoints implements Points {
@Override
public IClassMatcher getClassMatcher() {
return ClassMatcher.builder()
.hasInterface("javax.servlet.Filter")
.or()
.hasSuperClass("javax.servlet.http.HttpServlet")
.build();
}
@Override
public Set<IMethodMatcher> getMethodMatcher() {
return MethodMatcher.builder().named("doFilter")
.isPublic()
.argsLength(3)
.arg(0, "javax.servlet.ServletRequest")
.arg(1, "javax.servlet.ServletResponse")
.returnType("void")
.or()
.named("service")
.arg(0, "javax.servlet.ServletRequest")
.arg(1, "javax.servlet.ServletResponse")
.build().toSet();
}
}This ResponseHeaderInterceptor is bound to the enhancement point defined above via the @AdviceTo annotation, and does not need to be explicitly assigned a qualifier value when qualifier is the default value.
@AdviceTo(value = DoFilterPoints.class, plugin = SimplePlugin.class)
public class ResponseHeaderInterceptor implements Interceptor {
@Override
public void before(MethodInfo methodInfo, Context context) {
HttpServletResponse httpServletResponse = (HttpServletResponse) methodInfo.getArgs()[1];
String serviceName = EaseAgent.getConfig(ConfigConst.SERVICE_NAME);
httpServletResponse.setHeader("easeagent-srv-name", serviceName);
}
......
}public class SimplePlugin implements AgentPlugin {
@Override
public String getNamespace() {
return "simple";
}
// ConfigConst.OBSERVABILITY;
@Override
public String getDomain() {
return "observability";
}
}- Compile As mentioned above, the source code is available here.
$ git clone [email protected]:megaease/easeagent-test-demo.git
$ cd easeagent-test-demo/simple-plugin
$ mvn clean package
$
- Install Plugin
This simple plugin is compiled independently of easeagent, so the compiled output plugin jar package
simple-plugin-1.0.0.jarneed to be copied to the plugins directory which are at the same level directory as easeagent.jar (create if not existing), to allow easeagent to detect it.
$ export EASE_AGENT_PATH=[Replace with agent path]
$ mkdir $EASE_AGENT_PATH/plugins
$ cp target/simple-plugin-1.0.0.jar $EASE_AGENT_PATH/plugins
- Run
Taking the
spring-webmodule under ease-test-demo as a test demo project, run the demo application with the EaseAgent.
$ export EASE_AGENT_PATH=[Replace with agent path]
$ cd ../
$ mvn clean package -Dmaven.test.skip
$ java -javaagent:${EASE_AGENT_PATH}/easeagent-dep.jar -Deaseagent.config.path=agent.properties -jar spring-gateway/employee/target/employee-0.0.1.jar
- Test Execute the following test and the header information added can be seen in the HTTP Response.
easeagent-srv-name: demo-springweb
#
# curl -v http://127.0.0.1:18081/employee/message
* Trying 127.0.0.1:18081...
* Connected to 127.0.0.1 (127.0.0.1) port 18081 (#0)
> GET /employee/message HTTP/1.1
> Host: 127.0.0.1:18081
> User-Agent: curl/7.77.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< easeagent-srv-name: demo-springweb
< easeagent-duration: 55
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 34
< Date: Thu, 17 Mar 2022 03:46:40 GMT
<
* Connection #0 to host 127.0.0.1 left intact
Gateway Called in employee Service%
In addition to the easeagent-srv-name response header, we also see another response header easeagent-duration indicating the response time of the request, which is achieved by adding another enhancement point ResponseProcessPoints and ResponseDurationInterceptor. if interested, please visit the source code for details.
https://github.com/megaease/easeagent-test-demo/tree/master/simple-plugin
When the plugin is integrated into the plugins subdirectory in the easeagent project source tree, it will be compiled into the easeagent-dep.jar package.
In this simple plugin project, the com.megaease.easeagent:plugin-api dependency is wrapped in the local maven repository which locates in the simple-plugin/lib directory, user can also download the Easeagent source tree then install the plugin-api module.
$ git clone https://github.com/megaease/easeagent.git
$ cd easeagent/plugin-api
$ mvn clean install
We have described the EaseAgent architecture, plugin concepts and design details above, and now we are familiar with three core elements of a plugin through the minimalist Simple plugin example.
However, to develop a practical business plugin, it is necessary to have an understanding of the business API. Next, we outline how to use the Tracing API to complete a Tracing plugin development in combination with the actual Tracing plugin HttpServletPlugin.
Note: Image From Jaeger., Retrieved March 08, 2022, from Architecture., https://www.jaegertracing.io/docs/1.31/architecture
Before we look at the concrete implementation, let's briefly introduce the core concepts in Tracing, Trace and Span.
As shown above, a Trace represents a complete transaction, containing multiple Spans, A-E; a Span represents an independent service sub-unit of a complete transaction, such as a database request, a method call or an external request; Trace is a logical concept only and is represented by the traceId in the Span, and there are multiple Spans with the same traceId forming a directed acyclic graph.
A more concrete illustration of the Span concept and interface comes from a concrete sample of data in OpenZipkin format.
In the above data, id is the unique id of the current Span; traceId represents the unique id of the trace of the current transaction request, often directly using the id of the first Span, like Span-A's id in the above figure; parentId represents the id of the parent Span, like B's parentId in the above figure is the id of Span-A; for more specific details can be found in Openzipkin's documentation.
The key point in the development of the Tracing plugin is the generation and reporting of Span. So what interface does EaseAgent use to provide the generation and reporting of Spans?
Among the three elements of a plugin, the Interceptor interface implementation is the core of a plugin development, where the before/after methods carry the Context parameter, and it is through the Context interface that developers can make Tracing API calls to complete the collection and reporting of Span.
As shown above, the nextSpan() interface is used to create a Span, while the servReceive() and clientRequest() interfaces are wrappers of the nextSpan() interface, corresponding to the creation of a Span for a request received by the server and the creation of a Span for an external request, and initialize the fields within the Span.
The next step is to look at the specific HttpServlet Tracing plugin implementation.
We skip the corresponding implementations of the plugin Points and AgentPlugin and look directly at Tracing's Interceptor implementation DoFilterTracingInterceptor.
@Override
public void doBefore(MethodInfo methodInfo, Context context) {
HttpServletRequest httpServletRequest = methodInfo.getArgs()[0];
...
HttpRequest httpRequest = new HttpServerRequest(httpServletRequest);
requestContext = context.serverReceive(httpRequest);
httpServletRequest.setAttribute(PROGRESS_CONTEXT, requestContext);
HttpUtils.handleReceive(requestContext.span(), httpRequest);
}As you can see from the above code snippet, the context.serverReceive() method is called in before to create and initialize the Span, and the HttpUtils::handleReceive() method is used to make HTTP request-related injections to the fields in the Span.
@Override
public void doAfter(MethodInfo methodInfo, Context context) {
...
try {
Span span = requestContext.span();
if (!httpServletRequest.isAsyncStarted()) {
...
HttpUtils.finish(span, response);
} else if (methodInfo.getThrowable() != null) {
span.error(methodInfo.getThrowable());
span.finish();
} else {
// async
...
}
} finally {
requestContext.scope().close();
}
}In the after method, the HttpUtils::finish() or span.finish() method is called to complete the Span collection and reporting. This is an overview of the Trace API interface calls in the actual Tracing plugin, for more specific details you can check the source code of this file in EaseAgent:
https://github.com/megaease/easeagent/blob/master/plugins/httpservlet/src/main/java/com/megaease/easeagent/plugin/httpservlet/interceptor/DoFilterTraceInterceptor.java
The Context section provides a comprehensive description of the Context.
If you need to print logs in the plugin to the EaseAgent log output, you can use the Slf4j interface directly.
Regarding configuration, we have a set of rules to follow. For detailed rules, please see: Plugin Configuration
When you want to get your own configuration file in the plugin, you only need to get it from the Context. The framework itself will automatically maintain configuration changes and modifications.
class InterceptorImpl implements Interceptor {
@Override
public void before(MethodInfo methodInfo, Context context) {
Config config = context.getConfig();
// You don’t need to verify enabled here, because enabled is a reserved attribute. If it is false, the Interceptor will not be run.
// boolean enabled = config.enabled();
Integer outputSize = config.getInt("output.size"); //it will be get config key: plugin.[domain].[namespace].[function].output.size = [value]
}
@Override
public void after(MethodInfo methodInfo, Context context) {
Config config = context.getConfig();
}
}When you want to get the configuration outside of the plugin, we provide tools to get it.
This tool will automatically maintain configuration updates and modifications, and the configuration obtained each time will be the latest configuration.
It is a singleton registration factory, which also means that the singleton acquisition is locked, so it is hoped that the user can acquire it as little as possible.
for example: acquire it once during initialization, and then put it in a static variable.
The registered key is domain, namespace, id.
import com.megaease.easeagent.plugin.api.config.AutoRefreshPluginConfigRegistry;
import com.megaease.easeagent.plugin.api.config.AutoRefreshPluginConfigImpl;
class Demo{
AutoRefreshConfigImpl config = AutoRefreshRegistry.getOrCreate("observability", "httpclient", "metric");
}When you need to customize Config, implement the com.megaease.easeagent.plugin.api.config.AutoRefreshPluginConfig interface, and then register
The registered key is domain, namespace, id and the type of Class.
public class ServiceNameConfig implements AutoRefreshConfig {
private volatile String propagateHead = "X-Mesh-RPC-Service";
public String getPropagateHead() {
return propagateHead;
}
@Override
public void onChange(Config oldConfig, Config newConfig) {
String propagateHead = newConfig.getString("propagate.head");
if (StringUtils.isEmpty(propagateHead) || StringUtils.isEmpty(propagateHead.trim())) {
return;
}
this.propagateHead = propagateHead.trim();
}
}
public class ServiceNameInterceptor implements Interceptor {
protected static ServiceNameConfig config = null;
@Override
public void init(Config pConfig, String className, String methodName, String methodDescriptor) {
config = AutoRefreshRegistry.getOrCreate(pConfig.domain(), pConfig.namespace(), pConfig.id(),
new AutoRefreshConfigSupplier<ServiceNameConfig>() {
@Override
public ServiceNameConfig newInstance() {
return new ServiceNameConfig();
}
});
}
}The above covers all aspects of plugin development, the following are common debugging issues during plugin development.
Debugging environment configuration, briefly described below:
- Download the EaseAgent source code to the localhost and add it to the workspace of the IDE, build and the output located in the source code directory build/target/easeagent-dep.jar, and then create the directory
build/target/plugins； - Add the plugin project (e.g. simple-plugin) to the same workspace and copy the compiled and packaged JAR file to the
build/target/pluginsdirectory created in the previous step so that the plugin can be loaded. - Add the source code of the application (e.g. spring-gateway/employee) to the workspace and configure the JVM options in the Debug menu to start the application with easeagent-dep.jar for debugging later.
eg.:
The
-javaagent:/path-to-easeagent/build/target/easeagent-dep.jar -Deaseagent.config.path=/my-own-if-changed-or-add/agent.properties -Deaseagent.log.conf=/my-own-if-changed/easeagent-log4j2.xml -Dnet.bytebuddy.dump=/path-to-dump/pathabove need to be replaced with the user's actual environment path. - Set breakpoints, launch debug session.
-
How can I determine whether target classes and methods are enhanced? The following debug options were set in step 3 of the previous section of environment configuration:
-Dnet.bytebuddy.dump=/path-to-dump/The class files of all the enhanced classes will be printed in this directory. Decompile the class files (IDEA can pull them in and open them directly) to see if the corresponding method has the enhanced bytecode to call the EaseAgent method.
-
If the check confirms that the target method is not enhanced, how do I debug it?
There are three key checkpoints: ClassMatchers, MethodMatchers and all other issues.- All classes that are matched will run into the ForAdviceTransformer::transform(...) method, where conditional breakpoints can be added, then checking ClassMatchers if the breakpoint is not interrupted.
- All methods matched will run into the AdviceRegistry::check(...) method, where conditional breakpoints can be added, then checking MethodMatchers if the breakpoint is not interrupted.
- Set a breakpoint by going back through the breakpoint stack in step 1 and navigating to the ByteBuddy source code where throw exception when enhance fail, check the cause of the exception.
All enhancement failures can be resolved by analyzing the three breakpoints above.
- Why the Interceptor is not called when the class method has been enhanced?
All enhanced methods run into the following two methods.
Breakpoints can be set at the entrances and exits to further trace the execution logic. Commonly, and most likely, the plugin's corresponding configuration has enabled=false or is not configured.
com.megaease.easeagent.core.plugin.Dispatcher::enter com.megaease.easeagent.core.plugin.Dispatcher::exit
plugin.[domain].[namespace].[type].enabled=true
To verify the impact of EageAgent on performance/latency, the Profiler tool can be used to determine the performance/latency impact by sampling the CPU and focusing on the percentage of Megaease related function executions within the stack.
There are profiler tools such as Async-profiler, Arthas and VisualVm.
Finally, have fun using and extending EaseAgent, and feel free to raise Issues or join the Slack community to discuss.
- Github Issues
- Join Slack Workspace for function requirement, issues, and discussion.
- MegaEase on Twitter
- Chen, H., Zhao, K., & Zhou, J. (2022). EaseAgent 2.0 Architecture Design and Code Implementation. megaease.com. Retrieved March 9, 2022, from https://www.youtube.com/watch?v=u6Aoa2roGuA
- Data model. Data Model · OpenZipkin. (n.d.). Retrieved March 08, 2022, from https://zipkin.io/pages/data_model.html
- Architecture. Jaeger. (n.d.). Retrieved March 08, 2022, from https://www.jaegertracing.io/docs/1.31/architecture
- Oaks, S. (2020). Java performance 2nd Edition. O'Reilly.