Skip to content

Commit

Permalink
#152 Migrate to Exasol 8 (#154)
Browse files Browse the repository at this point in the history
  • Loading branch information
kaklakariada authored Oct 20, 2023
1 parent 9a02e20 commit 148f1dc
Show file tree
Hide file tree
Showing 24 changed files with 329 additions and 251 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/ci-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@ on:

jobs:
build:
runs-on: ubuntu-20.04 # UDFs fail with "VM error: Internal error: VM crashed" on ubuntu-latest
runs-on: ubuntu-latest
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-build
cancel-in-progress: true
env:
EXASOL_VERSION: 7.1.23
EXASOL_VERSION: "8.23.0"
steps:
- name: Check out code
uses: actions/checkout@v4
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ on:

jobs:
build:
runs-on: ubuntu-20.04 # UDFs fail with "VM error: Internal error: VM crashed" on ubuntu-latest
runs-on: ubuntu-latest
steps:
- name: Checkout the repository
uses: actions/checkout@v4
Expand Down
2 changes: 1 addition & 1 deletion .project-keeper.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ sources:
path: registry-upload/package.json
- type: npm
path: pkg/parameterValidator/package.json
version: 0.5.3
version: 0.5.4
linkReplacements:
- "Unknown|https://github.com/DATA-DOG/go-sqlmock/blob/master/LICENSE"
- "https://github.com/swagger-api/swagger-core/modules/swagger-annotations|https://github.com/swagger-api/swagger-core/tree/master/modules/swagger-annotations"
Expand Down
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,14 @@ This API is documented via Swagger. In order to view it, checkout this repo, run
* [Changelog](doc/changes/changelog.md)
* [Software Design (online, main branch)](https://exasol.github.io/extension-manager/design.html)
* [Software Design (local)](doc/design.md)
* [Developer Guide](doc/developer_guide.md)
* [Dependencies](dependencies.md)

### Developer Guides

* [Guide for embedding Extension Manager in another application](doc/embedding_extension_manager.md)
* [Guide for developing an extension for Extension Manager](doc/extension_developer_guide.md)
* [Guide for developing Extension Manager](doc/developer_guide.md)

## Related Projects

* [extension-manager](https://github.com/exasol/extension-manager): Extension manager backend written in Go (this repo)
Expand Down
2 changes: 1 addition & 1 deletion dependencies.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions doc/changes/changelog.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

33 changes: 33 additions & 0 deletions doc/changes/changes_0.5.4.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Extension Manager 0.5.4, released 2023-10-20

Code name: Migrate to Exasol 8

## Summary

Starting with this release, Extension Manager requires Exasol 8. Also integration tests for extension definitions must run with Exasol 8.

To skip tests on non v8 versions, you can use the following new method:

```java
import com.exasol.extensionmanager.itest;
// ...
ExasolVersionCheck.assumeExasolVersion8(exasolTestSetup)
```

## Features

* #152: Migrated tests to Exasol 8

## Dependency Updates

### Extension-manager

#### Compile Dependency Updates

* Updated `github.com/exasol/exasol-test-setup-abstraction-server/go-client:v0.3.3` to `v0.3.4`

### Extension Integration Tests Library

#### Compile Dependency Updates

* Updated `com.exasol:extension-manager-client-java:0.5.3` to `0.5.4`
52 changes: 3 additions & 49 deletions doc/developer_guide.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Developer Guide

This guide describes how to develop, test and build the Extension Manager.
This guide describes how to develop, test and build Extension Manager (EM).

If you want to embed EM into another application, see the [embedding guide](./embedding_extension_manager.md). If you want to develop an extension for EM, see the [extension developer guide](./extension_developer_guide.md).

## Building

Expand Down Expand Up @@ -196,51 +198,3 @@ poetry run mypy .
cd pkg/extensionController/bfs/udf
poetry run pylint *.py
```

## Embedding Extension Manager in Other Go Programs

### Embedding the REST API

You can embed the Extension Manager's REST API in other programs that use the [Nightapes/go-rest](https://github.com/Nightapes/go-rest) library:

```go
// Create an instance of `openapi.API`
api := openapi.NewOpenAPI()
// Create a new configuration
config := extensionController.ExtensionManagerConfig{
ExtensionRegistryURL: "https://example.com/registry.json",
BucketFSBasePath: "/buckets/bfsdefault/default/",
ExtensionSchema: "EXA_EXTENSIONS",
}
// Add endpoints
err := restAPI.AddPublicEndpoints(api, config)
if err != nil {
return err
}
// Start the server
```

### Embedding the Extension Controller

If you want to directly use the extension controller in your application you can use the following code as an example:

```go
// Create a new configuration
config := extensionController.ExtensionManagerConfig{
ExtensionRegistryURL: "https://example.com/registry.json",
BucketFSBasePath: "/buckets/bfsdefault/default/",
ExtensionSchema: "EXA_EXTENSIONS",
}
// Create controller and handle configuration validation error
ctrl, err := extensionController.CreateWithValidatedConfig(config)
if err != nil {
return err
}
// Create database connection (required as an argument for all controller methods)
var db *sql.DB = createDBConnection()
// Call controller method and process result. Use a custom context if available.
extensions, err := ctrl.GetAllExtensions(context.Background(), db)
// ...
```
49 changes: 49 additions & 0 deletions doc/embedding_extension_manager.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Embedding Extension Manager in Another Application

This page describes how to integrate Extension Manager functionality into another Go application.

## Embedding the REST API

You can embed the Extension Manager's REST API in another Go application that uses the [Nightapes/go-rest](https://github.com/Nightapes/go-rest) library:

```go
// Create an instance of `openapi.API`
api := openapi.NewOpenAPI()
// Create a new configuration
config := extensionController.ExtensionManagerConfig{
ExtensionRegistryURL: "https://example.com/registry.json",
BucketFSBasePath: "/buckets/bfsdefault/default/",
ExtensionSchema: "EXA_EXTENSIONS",
}
// Add endpoints
err := restAPI.AddPublicEndpoints(api, config)
if err != nil {
return err
}
// Start the server
```

## Embedding the Extension Controller

If you want to directly use the extension controller in your Go application you can use the following code as an example:

```go
// Create a new configuration
config := extensionController.ExtensionManagerConfig{
ExtensionRegistryURL: "https://example.com/registry.json",
BucketFSBasePath: "/buckets/bfsdefault/default/",
ExtensionSchema: "EXA_EXTENSIONS",
}
// Create controller and handle configuration validation error
ctrl, err := extensionController.CreateWithValidatedConfig(config)
if err != nil {
return err
}

// Create database connection (required as an argument for all controller methods)
var db *sql.DB = createDBConnection()

// Call controller method and process result. Use a custom context if available.
extensions, err := ctrl.GetAllExtensions(context.Background(), db)
// ...
```
17 changes: 15 additions & 2 deletions doc/extension_developer_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,21 @@ See [`ExampleIT.java`](../extension-manager-integration-test-java/src/test/java/

Depending on the requirements of your extension you might also need to upload the adapter JAR or a JDBC driver to BucketFS in `@BeforeAll`.

#### Features of class `ExtensionManagerSetup`
#### Preconditions for Using EITFJ

EM only works with Exasol DB version 8 or later and does not support 7.1. `ExtensionManagerSetup.create()` verifies the correct DB version by executing a query against the given Exasol test setup.

When you test your project also with Exasol version 7.1, you can use the following code to skip the extension integration tests for version 7.1:

```java
import com.exasol.extensionmanager.itest;
// ...
exasolTestSetup = ...
ExasolVersionCheck.assumeExasolVersion8(exasolTestSetup);
setup = ExtensionManagerSetup.create(exasolTestSetup, /* ... */);
```

#### Features of Class `ExtensionManagerSetup`

Class `ExtensionManagerSetup` offers the following useful features:

Expand All @@ -82,4 +96,3 @@ EITFJ works without additional configuration. During development you can however
* `buildExtension`: Set this to `false` in order to skip building the extension definition before the tests. Use this to speedup tests when the extension definition is not modified.
* `buildExtensionManager`: Set this to `false` to skip building/installing the extension manager binary. Use this to speedup tests when extension manager is not modified.
* `extensionManagerVersion`: Version of EM to use during tests. By default EITFJ uses the same version as the version defined in `pom.xml` for `extension-manager-integration-test-java`. Changing this is not recommended.

Original file line number Diff line number Diff line change
Expand Up @@ -76,16 +76,16 @@ public void assertNoConnections() {
*/
public void assertVirtualSchema(final Matcher<ResultSet> matcher) {
assertResult(
"SELECT SCHEMA_NAME, SCHEMA_OWNER, ADAPTER_SCRIPT, ADAPTER_NOTES FROM SYS.EXA_ALL_VIRTUAL_SCHEMAS ORDER BY SCHEMA_NAME ASC",
"SELECT SCHEMA_NAME, SCHEMA_OWNER, ADAPTER_SCRIPT_SCHEMA, ADAPTER_SCRIPT_NAME, ADAPTER_NOTES FROM SYS.EXA_ALL_VIRTUAL_SCHEMAS ORDER BY SCHEMA_NAME ASC",
matcher);
}

/**
* Verify that the {@code SYS.EXA_ALL_VIRTUAL_SCHEMAS} table is empty.
*/
public void assertNoVirtualSchema() {
assertVirtualSchema(
ResultSetStructureMatcher.table(VARCHAR_TYPE, VARCHAR_TYPE, VARCHAR_TYPE, VARCHAR_TYPE).matches());
assertVirtualSchema(ResultSetStructureMatcher
.table(VARCHAR_TYPE, VARCHAR_TYPE, VARCHAR_TYPE, VARCHAR_TYPE, VARCHAR_TYPE).matches());
}

private void assertResult(final String sql, final Matcher<ResultSet> matcher) {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
package com.exasol.extensionmanager.itest;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.junit.jupiter.api.Assumptions.assumeTrue;

import java.sql.*;

import com.exasol.exasoltestsetup.ExasolTestSetup;

/**
* This class contains static methods for checking the Exasol DB version.
*/
public class ExasolVersionCheck {
private ExasolVersionCheck() {
// Not instantiable
}

/**
* Ensures that the given test setup is connected to a Exasol DB in version 8.
* <p>
* This executes a SQL query using the given test setup to determine the major version of the database.
*
* @param testSetup test setup to check
* @throws AssertionError if the major version number is not {@code 8}
*/
public static void assertExasolVersion8(final ExasolTestSetup testSetup) {
final String version = getExasolMajorVersion(testSetup);
assertEquals("8", version, "Exasol version");
}

/**
* Assumes that the given test setup is connected to a Exasol DB in version 8. *
* <p>
* This executes a SQL query using the given test setup to determine the major version of the database.
*
* @param testSetup test setup to check
* @throws org.opentest4j.TestAbortedException if the major version number is not {@code 8}
*/
public static void assumeExasolVersion8(final ExasolTestSetup testSetup) {
final String version = getExasolMajorVersion(testSetup);
assumeTrue("8".equals(version), "Expected Exasol version 8 but got '" + version + "'");
}

static String getExasolMajorVersion(final ExasolTestSetup testSetup) {
try (Statement stmt = testSetup.createConnection().createStatement()) {
final ResultSet result = stmt
.executeQuery("SELECT PARAM_VALUE FROM SYS.EXA_METADATA WHERE PARAM_NAME='databaseMajorVersion'");
assertTrue(result.next(), "no result");
return result.getString(1);
} catch (final SQLException exception) {
throw new IllegalStateException("Failed to query Exasol version: " + exception.getMessage(), exception);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,18 @@ private ExtensionManagerSetup(final ExtensionManagerProcess extensionManager, fi
* Prepare and create a new instance of {@link ExtensionManagerSetup}. Usually you call this in a
* {@link org.junit.jupiter.api.BeforeAll} method. Make sure to close this by calling {@link #close()} in an
* {@link org.junit.jupiter.api.AfterAll} method.
* <p>
* This method verifies, that the given {@link ExasolTestSetup} is connected to an Exasol database in version 8,
* else it will throw an exception.
*
* @param exasolTestSetup exasol test setup to use for the tests
* @param exasolTestSetup Exasol test setup connected to an Exasol v8 database to use for the tests
* @param extensionBuilder builder for building the extension under test
* @return a new instance
* @throws AssertionError if the Exasol database version is wrong.
*/
public static ExtensionManagerSetup create(final ExasolTestSetup exasolTestSetup,
final ExtensionBuilder extensionBuilder) {
ExasolVersionCheck.assertExasolVersion8(exasolTestSetup);
final Path extensionFolder = createTempDir();
final ExtensionTestConfig config = ExtensionTestConfig.read();
final ExtensionManagerProcess extensionManager = startExtensionManager(extensionBuilder, extensionFolder,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,42 +12,58 @@
import com.exasol.exasoltestsetup.ExasolTestSetup;
import com.exasol.exasoltestsetup.ExasolTestSetupFactory;
import com.exasol.extensionmanager.client.model.ExtensionsResponseExtension;
import com.exasol.extensionmanager.itest.ExasolVersionCheck;
import com.exasol.extensionmanager.itest.ExtensionManagerSetup;
import com.exasol.extensionmanager.itest.builder.ExtensionBuilder;

/**
* This integration test illustrates the usage of {@link ExtensionManagerSetup}. See the
* This integration test illustrates the usage of {@link ExtensionManagerSetup} for testing extensions for Extension
* Manager (EM). See the
* <a href="https://github.com/exasol/extension-manager/blob/main/doc/extension_developer_guide.md">Extension Developer
* Guide</a> for details.
*/
// [doc -> dsn~eitfj-start-extension-manager~1]
// [doc -> dsn~eitfj-access-extension-manager-rest-interface~1]
class ExampleIT {
/** Relative path to the directory containing the extension definition sources */
private static final Path EXTENSION_SOURCE_DIR = Paths.get("testing-extension").toAbsolutePath();
/** File name of the built JavaScript file (= extension ID) */
private static final String EXTENSION_ID = "testing-extension.js";

private static ExasolTestSetup exasolTestSetup;
private static ExtensionManagerSetup setup;

@BeforeAll
static void setup() {
// Overwrite default Exasol version
System.setProperty("com.exasol.dockerdb.image", "8.23.0");

exasolTestSetup = new ExasolTestSetupFactory(Path.of("cloud-setup")).getTestSetup();

// Skip test in case this is an older Exasol version
ExasolVersionCheck.assumeExasolVersion8(exasolTestSetup);

// Create EM setup
setup = ExtensionManagerSetup.create(exasolTestSetup, ExtensionBuilder.createDefaultNpmBuilder(
EXTENSION_SOURCE_DIR, EXTENSION_SOURCE_DIR.resolve("dist").resolve(EXTENSION_ID)));
}

@AfterAll
static void teardown() throws Exception {
// Close EM setup and Exasol DB after running all tests
setup.close();
exasolTestSetup.close();
}

@AfterEach
void cleanup() {
// Cleanup resources after each test
setup.cleanup();
}

@Test
void listExtensions() {
// Use the EM client to list available extensions and verify the result
final List<ExtensionsResponseExtension> extensions = setup.client().getExtensions();
assertThat(extensions, hasSize(1));
}
Expand Down
Loading

0 comments on commit 148f1dc

Please sign in to comment.