Skip to content

Commit

Permalink
Provide built-in GraalVM Reachability Metadata
Browse files Browse the repository at this point in the history
  • Loading branch information
linghengqian committed Sep 19, 2023
1 parent de93e07 commit 471e302
Show file tree
Hide file tree
Showing 24 changed files with 1,168 additions and 44 deletions.
45 changes: 45 additions & 0 deletions .github/workflows/graalvm-native-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

name: NativeTest by GraalVM CE with Maven

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:
build:
strategy:
matrix:
java: [ '17.0.8' ]
os: [ 'ubuntu-latest' ]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- name: Set up GraalVM CE ${{ matrix.java }}
uses: graalvm/setup-graalvm@v1
with:
java-version: ${{ matrix.java }}
distribution: 'graalvm-community'
github-token: ${{ secrets.GITHUB_TOKEN }}
cache: 'maven'
- name: Build NativeTest with Maven
continue-on-error: true
run: |
./mvnw -PnativeTestInElasticJob -T1C -B -e clean test
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
+++
title = "GraalVM Native Image 支持"
weight = 6
chapter = true
+++

## 对 GraalVM Native Image 的支持

ElasticJob Lite 提供了构建 GraalVM Native Image 所需要的 GraalVM Reachability Metadata。在 `pom.xml` 引入 `elasticjob-lite-core` 后无需额外处理。

```xml
<project>
<dependencies>
<dependency>
<groupId>org.apache.shardingsphere.elasticjob</groupId>
<artifactId>elasticjob-lite-core</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</project>
```

## 内部的 GraalVM Reachability Metadata 条目

`org.apache.shardingsphere.elasticjob:elasticjob-infra-reachability-metadata` 托管了如下 Maven 库的 GraalVM Reachability
Metadata,一旦 https://github.com/oracle/graalvm-reachability-metadata 对如下库的 GraalVM Reachability Metadata 进行了
Release 流程, 下列第三方库的相关 JSON 文件将在 ElasticJob 一侧被删除。

- `org.apache.zookeeper:zookeeper:3.9.0`
- `org.apache.curator:curator-client:5.5.0`
- `org.apache.curator:curator-framework:5.5.0`
- `org.apache.curator:curator-recipes:5.5.0`
- `org.apache.shardingsphere.elasticjob:elasticjob-lite-core:${project.version}`

用户可以通过在自有项目的资源根目录或测试资源文件夹,通过新建对应库信息的 `/META-INF/native-image/${project.groupId}/${project.artifactId}/${project.version}`
文件夹来覆写被传入项目的 GraalVM Reachability Metadata 。

用户依然可以自己编写 GraalVM Reachability Metadata 的相关文件来为缺少 GraalVM Reachability Metadata 的类库提供 GraalVM Native Image 支持。

## 已知限制

1. 用户无法在 GraalVM Native Image 使用 elasticJobType 为 SCRIPT 的 Job。参考 https://github.com/oracle/graal/issues/7390

2. 用户无法在 GraalVM Native Image 下使用与 `org.apache.shardingsphere.elasticjob.tracing.api.TracingConfiguration` 相关的 Tracing 功能。

3. ElasticJob 的 Spring Boot Starter 尚未在 GraalVM Native Image 下可用。

## 贡献 GraalVM Reachability Metadata

此节文本针对贡献者。假设贡献者位于新的 Ubuntu 22.04.3 实例下,可通过 `SDKMAN!` 初始化 `GraalVM CE` 环境。

```bash
sdk install java 17.0.8-graalce
sdk use java 17.0.8-graalce
sudo apt-get install build-essential libz-dev zlib1g-dev -y
```

对于与 ElasticJob 无关的 GraalVM Reachability Metadata,相关的 issue 和 PR 应当首先创建在 https://github.com/oracle/graalvm-reachability-metadata
该存储库使 GraalVM Native Image 的用户能够共享和重用 Java 生态系统中的库和框架的元数据。

ElasticJob 维护了一组单元测试子集,此子集避开对 Mockito 相关的
类的使用,以服务于项目 CI 对 Native Image 的测试。参考 https://github.com/mockito/mockito/issues/2862

- `org.apache.shardingsphere.elasticjob.lite.fixture.reachability.metadata.**`

在 ElasticJob 一侧,存在 Maven Profile 为 `generateMetadata`,用户可在 ElasticJob 项目根目录执行如下命令来采集 GraalVM Reachability Metadata。
请手动删除无任何具体条目的 JSON 文件。

```bash
./mvnw -PgenerateMetadata -DskipNativeTests -e -T1C clean test native:metadata-copy
```

在 ElasticJob 一侧,存在 Maven Profile 为 `nativeTestInElasticJob`,用户可在 ElasticJob 项目根目录执行如下命令来执行特定于 GraalVM
Native Build Tools 的 nativeTest,以验证 GraalVM Native Image 中的单元测试覆盖率。

```bash
./mvnw -PnativeTestInElasticJob -T1C -e clean test
```

对于特定于单元测试的 GraalVM Reachability Metadata,请放置在 `${project.basedir}/src/test/resources/META-INF/native-image/${project.artifactId}-test-metadata/`
文件夹下,`${project.basedir}``${project.artifactId}` 为对应单元测试涉及的子模块。如有需要,请使用 `org.junit.jupiter.api.condition.DisabledInNativeImage`
注解或 `org.graalvm.nativeimage.imagecode` 的 System Property 屏蔽部分单元测试在 GraalVM Native Image 下运行。
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
+++
title = "GraalVM Native Image Support"
weight = 6
chapter = true
+++

## Support for GraalVM Native Image

ElasticJob Lite provides the GraalVM Reachability Metadata needed to build GraalVM Native Images. No additional
processing is required after the introduction of `elasticjob-lite-core` in `pom.xml`.

```xml
<project>
<dependencies>
<dependency>
<groupId>org.apache.shardingsphere.elasticjob</groupId>
<artifactId>elasticjob-lite-core</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</project>
```

## Internal GraalVM Reachability Metadata entry

`org.apache.shardingsphere.elasticjob:elasticjob-infra-reachability-metadata` hosts GraalVM Reachability Metadata for
the following Maven library, once https://github.com/oracle/graalvm-reachability-metadata is done Release process with
the GraalVM Reachability Metadata for the following libraries, the relevant JSON files for the following third-party
libraries will be deleted on the ElasticJob side.

- `org.apache.zookeeper:zookeeper:3.9.0`
- `org.apache.curator:curator-client:5.5.0`
- `org.apache.curator:curator-framework:5.5.0`
- `org.apache.curator:curator-recipes:5.5.0`
- `org.apache.shardingsphere.elasticjob:elasticjob-lite-core:${project.version}`

Users can create a new corresponding library folder called `/META-INF/native-image/${project.groupId}/${project.artifactId}/${project.version}`
in the resource root directory of their own project or test resource folder to override the GraalVM Reachability
Metadata of the incoming project.

Users can still write their own GraalVM Reachability Metadata files to provide GraalVM Native Image support for
libraries that lack GraalVM Reachability Metadata.

## Known limitations

1. Users cannot use jobs with `elasticJobType` as `SCRIPT` in GraalVM Native Image. Refer to https://github.com/oracle/graal/issues/7390 .

2. Users cannot use the tracing function related to `org.apache.shardingsphere.elasticjob.tracing.api.TracingConfiguration`
under GraalVM Native Image.

3. ElasticJob's Spring Boot Starter is not yet available under GraalVM Native Image.

## Contribute GraalVM Reachability Metadata

This section text is for contributors. Assuming the contributor is under a new Ubuntu 22.04.3 instance, the `GraalVM CE`
environment can be initialized via `SDKMAN!`.

```bash
sdk install java 17.0.8-graalce
sdk use java 17.0.8-graalce
sudo apt-get install build-essential libz-dev zlib1g-dev -y
```

For GraalVM Reachability Metadata that is not related to ElasticJob, issues and PRs should first be created in the https://github.com/oracle/graalvm-reachability-metadata .
The repository enables users of GraalVM Native Image to share and reuse metadata for libraries and frameworks in the
Java ecosystem.

ElasticJob maintains a subset of unit tests that eschews the use of Mockito-related classes to serve the project CI's
testing of Native Images. Refer to https://github.com/mockito/mockito/issues/2862 .

- `org.apache.shardingsphere.elasticjob.lite.fixture.reachability.metadata.**`

On the ElasticJob side, there is a Maven Profile as `generateMetadata`, and users can execute the following command in
the root directory of the ElasticJob project to capture GraalVM Reachability Metadata. Manually delete the JSON file
without any specific entries.

```bash
./mvnw -PgenerateMetadata -DskipNativeTests -e -T1C clean test native:metadata-copy
```

On the ElasticJob side, there is a Maven Profile as `nativeTestInElasticJob`, and users can execute the following
command at the root of the ElasticJob project to execute GraalVM-specific nativeTest of Native Build Tools to verify
unit test coverage in GraalVM Native Image.

```bash
./mvnw -PnativeTestInElasticJob -T1C -e clean test
```

For GraalVM Reachability Metadata specific to unit tests, be placed in `${project.basedir}/src/test/resources/META-INF/native-image/${project.artifactId}-test-metadata/`
folder, `${project.basedir}` and `${project.artifactId}` are submodules involved in the corresponding unit test. If
needed, use `org.junit.jupiter.api.condition.DisabledInNativeImage` annotations or `org.graalvm.nativeimage.imagecode`
System Property masked unit tests to run under GraalVM Native Image.
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,12 @@
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>


<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
Expand Down
30 changes: 30 additions & 0 deletions elasticjob-infra/elasticjob-infra-reachability-metadata/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.shardingsphere.elasticjob</groupId>
<artifactId>elasticjob-infra</artifactId>
<version>3.1.0-SNAPSHOT</version>
</parent>
<artifactId>elasticjob-infra-reachability-metadata</artifactId>
<name>${project.artifactId}</name>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
[
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"java.io.FilePermission"
},
{
"condition":{"typeReachable":"org.apache.curator.retry.RetryForever"},
"name":"java.lang.Object",
"methods":[{"name":"toString","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"java.lang.RuntimePermission"
},
{
"condition":{"typeReachable":"org.apache.curator.retry.RetryForever"},
"name":"java.lang.StackWalker",
"methods":[{"name":"getInstance","parameterTypes":["java.util.Set","int"] }, {"name":"walk","parameterTypes":["java.util.function.Function"] }]
},
{
"condition":{"typeReachable":"org.apache.curator.retry.RetryForever"},
"name":"java.lang.StackWalker$Option"
},
{
"condition":{"typeReachable":"org.apache.curator.retry.RetryForever"},
"name":"java.lang.StackWalker$StackFrame",
"methods":[{"name":"getClassName","parameterTypes":[] }, {"name":"getFileName","parameterTypes":[] }, {"name":"getLineNumber","parameterTypes":[] }, {"name":"getMethodName","parameterTypes":[] }]
},
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"java.net.NetPermission"
},
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"java.net.SocketPermission"
},
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"java.net.URLPermission",
"methods":[{"name":"<init>","parameterTypes":["java.lang.String","java.lang.String"] }]
},
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"java.security.AllPermission"
},
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"java.security.SecureRandomParameters"
},
{
"condition":{"typeReachable":"org.apache.curator.test.QuorumConfigBuilder"},
"name":"java.security.SecureRandomParameters"
},
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"java.security.SecurityPermission"
},
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"java.util.PropertyPermission"
},
{
"condition":{"typeReachable":"org.apache.curator.test.ZooKeeperServerEmbeddedAdapter"},
"name":"java.util.concurrent.ForkJoinTask",
"fields":[{"name":"aux"}, {"name":"status"}]
},
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"javax.smartcardio.CardPermission"
},
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"sun.security.provider.NativePRNG",
"methods":[{"name":"<init>","parameterTypes":[] }, {"name":"<init>","parameterTypes":["java.security.SecureRandomParameters"] }]
},
{
"condition":{"typeReachable":"org.apache.curator.test.QuorumConfigBuilder"},
"name":"sun.security.provider.NativePRNG",
"methods":[{"name":"<init>","parameterTypes":[] }, {"name":"<init>","parameterTypes":["java.security.SecureRandomParameters"] }]
},
{
"condition":{"typeReachable":"org.apache.curator.test.DirectoryUtils"},
"name":"sun.security.provider.SHA",
"methods":[{"name":"<init>","parameterTypes":[] }]
}
]
Loading

0 comments on commit 471e302

Please sign in to comment.