Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FrameworkBenchmarks performance test #3746

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add FrameworkBenchmarks performance test #3746

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions perf/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ subdirectory and given a meaningful name. Once the reorganization of this direc
//├── dacapo
//├── renaissance
//├── liberty
//├── FrameworkBenchmarks

```
Each subdirectory requires a build.xml file describing where to pull the benchmark suite from, and how to build and run it. Each subdirectory also requires a playlist.xml file which describes 1 or more benchmarks and what commands to run in order to execute a particular benchmark run.
Expand All @@ -49,4 +50,7 @@ Liberty benchmarks from https://github.com/OpenLiberty - including liberty-dt7-s
#### renaissance
Renaissance benchmarks from https://github.com/renaissance-benchmarks/renaissance - including renaissance-akka-uct, renaissance-als, renaissance-chi-square, renaissance-db-shootout, renaissance-dec-tree, renaissance-finagle-chirper, renaissance-finagle-http, renaissance-fj-kmeans, renaissance-future-genetic, renaissance-gauss-mix, renaissance-log-regression, renaissance-mnemonics, renaissance-movie-lens, renaissance-naive-bayes, renaissance-par-mnemonics, renaissance-philosophers and renaissance-scala-kmeans

#### FrameworkBenchmarks
FrameworkBenchmarks from https://github.com/TechEmpower/FrameworkBenchmarks - including a wide field of web application frameworks for JAVA.

Additional benchmarks are being reviewed for addition and if you wish to include more, please comment in the open performance benchmarks [issue 1112](https://github.com/adoptium/aqa-tests/issues/1112).
26 changes: 26 additions & 0 deletions perf/framework-benchmarks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# FrameworkBenchmarks

[FrameworkBenchmarks](https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Java) provides representative performance measures across a wide field of web application frameworks. With much help from the community, coverage is quite broad and we are happy to broaden it further with contributions.

Here we focus on the wide range of Java frameworks for web applications scenarios.

FrameworkBenchmarks covers common workloads as plaintext processing, json content parsing, popular database manipulations
for different frameworks.

Notice the different from external tests:
```aidl
external tests runs framework inner unit tests to ensure frameworks behaving correctly.
perf framework tests stresses common worloads to ensure frameworks are performing well under test VMs.
```

We are enabling the following list of frameworks to avoid regression:

| Framework | Workloads |
|-----------|----------------------------------------|
| netty | plaintext,json |
| quarkus | plaintext,json,query,fortune,update,db |
| spark | plaintext,json,query,fortune,update,db |
| spring | plaintext,json,query,fortune,update,db |
| spring-jpa | plaintext,json,query,fortune,update,db |
| servlet | plaintext,json |
| servlet3 | plaintext,json |
39 changes: 39 additions & 0 deletions perf/framework-benchmarks/build.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
<?xml version="1.0"?>
<project name="FrameworkBenchmarks-Test" default="build" basedir=".">
<taskdef resource="net/sf/antcontrib/antlib.xml" />
<description>
Build FrameworkBenchmarks-Test Docker image
</description>

<!-- set properties for this build -->
<property name="TEST" value="framework-benchmarks" />
<property name="DEST" value="${BUILD_ROOT}/perf/${TEST}" />
<property name="src" location="." />

<condition property="git-prefix" value="git" else="https">
<isset property="isZOS"/>
</condition>

<target name="init">
<mkdir dir="${DEST}"/>
</target>

<target name="getFrameworkBenchmarks" depends="init" description="Clone the distribution">
<echo message="Cloning FrameworkBenchmarks"/>
<var name="git_command" value="clone --depth 1 -b master ${git-prefix}://github.com/TechEmpower/FrameworkBenchmarks.git ${DEST}"/>
<echo message="git ${git_command}" />
<exec executable="git" failonerror="false" dir=".">
<arg line="${git_command}" />
</exec>
</target>

<target name="dist" depends="getFrameworkBenchmarks" description="generate the distribution">
<copy todir="${DEST}">
<fileset dir="${src}" includes="*.xml, *.mk"/>
</copy>
</target>

<target name="build">
<antcall target="dist" inheritall="true" />
</target>
</project>
135 changes: 135 additions & 0 deletions perf/framework-benchmarks/playlist.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
<?xml version='1.0' encoding='UTF-8'?>
<!--
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-->
<playlist xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../TKG/playlist.xsd">
<test>
<testCaseName>FrameworkBenchmarks-netty</testCaseName>
<command>
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh netty netty.dockerfile ${TEST_RESROOT}; \
$(TEST_STATUS)
</command>
<levels>
<level>special</level>
</levels>
<groups>
<group>perf</group>
</groups>
<versions>
<version>11+</version>
</versions>
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
</test>
<test>
<testCaseName>FrameworkBenchmarks-quarkus</testCaseName>
<command>
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh quarkus quarkus.dockerfile ${TEST_RESROOT}; \
$(TEST_STATUS)
</command>
<levels>
<level>special</level>
</levels>
<groups>
<group>perf</group>
</groups>
<versions>
<version>11+</version>
</versions>
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
</test>
<test>
<testCaseName>FrameworkBenchmarks-spring</testCaseName>
<command>
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh spring spring.dockerfile ${TEST_RESROOT}; \
$(TEST_STATUS)
</command>
<levels>
<level>special</level>
</levels>
<groups>
<group>perf</group>
</groups>
<versions>
<version>11+</version>
</versions>
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
</test>
<test>
<testCaseName>FrameworkBenchmarks-spring-jpa</testCaseName>
<command>
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh spring spring-jpa.dockerfile ${TEST_RESROOT}; \
$(TEST_STATUS)
</command>
<levels>
<level>special</level>
</levels>
<groups>
<group>perf</group>
</groups>
<versions>
<version>11+</version>
</versions>
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
</test>
<test>
<testCaseName>FrameworkBenchmarks-spark</testCaseName>
<command>
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh spark spark.dockerfile ${TEST_RESROOT}; \
$(TEST_STATUS)
</command>
<levels>
<level>special</level>
</levels>
<groups>
<group>perf</group>
</groups>
<versions>
<version>11+</version>
</versions>
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
</test>
<test>
<testCaseName>FrameworkBenchmarks-servlet3</testCaseName>
<command>
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh servlet3 servlet3.dockerfile ${TEST_RESROOT}; \
$(TEST_STATUS)
</command>
<levels>
<level>special</level>
</levels>
<groups>
<group>perf</group>
</groups>
<versions>
<version>11+</version>
</versions>
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
</test>
<test>
<testCaseName>FrameworkBenchmarks-servlet</testCaseName>
<command>
${TEST_RESROOT}/../../../aqa-tests/perf/tfb-runner.sh servlet servlet.dockerfile ${TEST_RESROOT}; \
$(TEST_STATUS)
</command>
<levels>
<level>special</level>
</levels>
<groups>
<group>perf</group>
</groups>
<versions>
<version>11+</version>
</versions>
<platformRequirements>os.linux,arch.x86,bits.64</platformRequirements>
</test>
</playlist>
22 changes: 22 additions & 0 deletions perf/tfb-runner.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
test=$1
dockerfile=$2
TEST_RESROOT=$3

cd ${TEST_RESROOT}
testcase=` echo ${dockerfile} | cut -d'.' -f 1`
dockerfile=${TEST_RESROOT}/frameworks/Java/${test}/${dockerfile}

if [ ! -f ${dockerfile} ]; then
echo "Warning: ${dockerfile} not found"
exit 1
fi

cp -r ${TEST_JDK_HOME} ${TEST_RESROOT}/frameworks/Java/${test}/jdk
awk '/CMD.*/ && !x {print "COPY jdk /opt/java/openjdk"; x=1} 1' ${dockerfile} > tmpt && mv tmpt ${dockerfile}
awk '/CMD.*/ && !x {print "ENV JAVA_HOME=/opt/java/openjdk"; x=1} 1' ${dockerfile} > tmpt && mv tmpt ${dockerfile}
awk '/CMD.*/ && !x {print "ENV PATH=/opt/java/openjdk/bin:$PATH"; x=1} 1' ${dockerfile} > tmpt && mv tmpt ${dockerfile}
cat ${dockerfile}
./tfb --test $testcase

# get removable results
docker run --entrypoint /bin/bash --rm --network tfb -v /var/run/docker.sock:/var/run/docker.sock -v ${TEST_RESROOT}:/FrameworkBenchmarks techempower/tfb -c "chmod -R 777 /FrameworkBenchmarks/results"