#101 Upgrade extensions (#127)

Co-authored-by: Christoph Kuhnke <[email protected]>

.gitattributes

@@ -1,5 +1,6 @@
 *.sh text eol=lf
 
+.github/workflows/broken_links_checker.yml          linguist-generated=true
 .github/workflows/project-keeper.sh                 linguist-generated=true
 .github/workflows/project-keeper-verify.yml         linguist-generated=true
 dependencies.md                                     linguist-generated=true
@@ -15,3 +16,6 @@ extension-manager-integration-test-java/pk_generated_parent.pom               li
 extension-manager-integration-test-java/versionsMavenPluginRules.xml          linguist-generated=true
 extension-manager-integration-test-java/.settings/org.eclipse.jdt.core.prefs  linguist-generated=true
 extension-manager-integration-test-java/.settings/org.eclipse.jdt.ui.prefs    linguist-generated=true
+
+registry/package-lock.json                                                    linguist-generated=true
+extension-manager-integration-test-java/testing-extension/package-lock.json   linguist-generated=true

.project-keeper.yml

@@ -26,11 +26,9 @@ sources:
     path: pkg/parameterValidator/package.json
 version: 0.5.0
 linkReplacements:
-  - "https://github.com/exasol/exasol-test-setup-abstraction-server/blob/go-client/v0.2.3/go-client/LICENSE|https://github.com/exasol/exasol-test-setup-abstraction-server/blob/main/LICENSE"
   - "Unknown|https://github.com/DATA-DOG/go-sqlmock/blob/master/LICENSE"
   - "https://github.com/swagger-api/swagger-core/modules/swagger-annotations|https://github.com/swagger-api/swagger-core/tree/master/modules/swagger-annotations"
   - "jquery.org/license|https://jquery.org/license/"
   - "https://github.com/swagger-api/swagger-codegen/modules/swagger-codegen-maven-plugin|https://github.com/swagger-api/swagger-codegen/tree/master/modules/swagger-codegen-maven-plugin"
-  - "https://netty.io/netty-handler/|https://netty.io"
 excludes:
-  - "E-PK-CORE-18: Outdated content: '.github/workflows/broken_links_checker.yml'"
+  - "E-PK-CORE-18: Outdated content: '.github/workflows/project-keeper-verify.yml'"

cmd/main.go

@@ -1,10 +1,12 @@
 package main
 
 import (
+	"bytes"
 	"flag"
 	"fmt"
 	"os"
 	"path"
+	"strings"
 
 	log "github.com/sirupsen/logrus"
 
@@ -19,6 +21,7 @@ func main() {
 	var openAPIOutputPath = flag.String("openAPIOutputPath", "", "Generate the OpenAPI spec at the given path instead of starting the server")
 	flag.Parse()
 	log.SetLevel(log.DebugLevel)
+	log.SetFormatter(&simpleFormatter{})
 	if openAPIOutputPath != nil && *openAPIOutputPath != "" {
 		err := generateOpenAPISpec(*openAPIOutputPath)
 		if err != nil {
@@ -76,3 +79,15 @@ func generateOpenAPIJson() ([]byte, error) {
 	}
 	return json, err
 }
+
+type simpleFormatter struct {
+}
+
+func (f *simpleFormatter) Format(entry *log.Entry) ([]byte, error) {
+	b := &bytes.Buffer{}
+	b.WriteString(strings.ToUpper(entry.Level.String()))
+	b.WriteByte(' ')
+	b.WriteString(entry.Message)
+	b.WriteByte('\n')
+	return b.Bytes(), nil
+}

doc/changes/changes_0.5.0.md

@@ -1,15 +1,21 @@
-# Extension Manager 0.5.0, released 2023-??-??
+# Extension Manager 0.5.0, released 2023-08-16
 
-Code name:
+Code name: Upgrade Extensions
 
 ## Summary
 
-This release allows configuring the BucketFS base path where EM expects extension files to be located. EM searches this path recursively, so files are also found in subdirectories.
+This release supports upgrading installed extensions to their latest version. Extensions must implement the latest `extension-manager-interface` version 0.3.0 to support this.
 
-This release improves error handling when using extensions not implementing all functions required by EM. EM now returns a helpful error message instead of failing with a `nil`-pointer error.
+This release also allows configuring the BucketFS base path where EM expects extension files to be located. EM searches this path recursively, so files are also found in subdirectories.
+
+This release also improves error handling when using extensions not implementing all functions required by EM. EM now returns a helpful error message instead of failing with a `nil`-pointer error.
 
 A common scenario for an extension not implementing a required function is when the extension had been built using an older version of EM's extension interface.
 
+## Features
+
+* #101 Added support for upgrading installed extensions
+
 ## Bugfixes
 
 * #105: Ensured that EM can load and use compatible extensions
@@ -41,11 +47,13 @@ A common scenario for an extension not implementing a required function is when
 #### Test Dependency Updates
 
 * Updated `nl.jqno.equalsverifier:equalsverifier:3.14.3` to `3.15`
+* Updated `org.junit.jupiter:junit-jupiter-api:5.9.3` to `5.10.0`
+* Added `org.junit.jupiter:junit-jupiter-params:5.10.0`
 
 #### Plugin Dependency Updates
 
 * Updated `com.exasol:error-code-crawler-maven-plugin:1.2.3` to `1.3.0`
-* Updated `com.exasol:project-keeper-maven-plugin:2.9.7` to `2.9.9`
+* Updated `com.exasol:project-keeper-maven-plugin:2.9.7` to `2.9.10`
 * Updated `org.apache.maven.plugins:maven-gpg-plugin:3.0.1` to `3.1.0`
 * Updated `org.apache.maven.plugins:maven-surefire-plugin:3.0.0` to `3.1.2`
 * Updated `org.basepom.maven:duplicate-finder-maven-plugin:1.5.1` to `2.0.1`
@@ -61,15 +69,17 @@ A common scenario for an extension not implementing a required function is when
 * Updated `com.exasol:exasol-test-setup-abstraction-java:2.0.1` to `2.0.2`
 * Updated `com.exasol:extension-manager-client-java:0.4.0` to `0.5.0`
 * Removed `io.netty:netty-handler:4.1.94.Final`
+* Updated `org.junit.jupiter:junit-jupiter-api:5.9.3` to `5.10.0`
 
 #### Test Dependency Updates
 
 * Updated `com.exasol:udf-debugging-java:0.6.8` to `0.6.10`
+* Updated `org.junit.jupiter:junit-jupiter-params:5.9.3` to `5.10.0`
 
 #### Plugin Dependency Updates
 
 * Updated `com.exasol:error-code-crawler-maven-plugin:1.2.3` to `1.3.0`
-* Updated `com.exasol:project-keeper-maven-plugin:2.9.7` to `2.9.9`
+* Updated `com.exasol:project-keeper-maven-plugin:2.9.7` to `2.9.10`
 * Updated `org.apache.maven.plugins:maven-failsafe-plugin:3.0.0` to `3.1.2`
 * Updated `org.apache.maven.plugins:maven-gpg-plugin:3.0.1` to `3.1.0`
 * Updated `org.apache.maven.plugins:maven-surefire-plugin:3.0.0` to `3.1.2`
@@ -102,4 +112,4 @@ A common scenario for an extension not implementing a required function is when
 #### Development Dependency Updates
 
 * Updated `typescript:5.0.3` to `5.1.6`
-* Updated `esbuild:0.17.15` to `0.18.13`
+* Updated `esbuild:0.17.15` to `0.18.16`

doc/design.md

@@ -170,19 +170,6 @@ As described in [`dsn~extension-components~1`](#components-of-an-extension) an e
 
 Needs: impl, utest, itest
 
-#### Resolving Files in BucketFS
-`dsn~resolving-files-in-bucketfs~1`
-
-EM resolves filenames to absolute paths in BucketFS.
-
-Rationale:
-
-Extensions require files in BucketFS. The exact location in BucketFS of the files is not known when creating an extension, only the file name (e.g. `document-files-virtual-schema-dist-7.3.3-s3-2.6.2.jar`). In order to create adapter scripts, the extension must know the absolute path of the file in BucketFS (e.g. `/buckets/bfsdefault/default/document-files-virtual-schema-dist-7.3.3-s3-2.6.2.jar`). So EM must allow the extension to resolve file names to an absolute path in BucketFS.
-
-Needs: impl, utest, itest
-
-#### Resolving Artifacts in BucketFS
-
 ### Extension Definitions
 `dsn~extension-definition~1`
 
@@ -340,7 +327,7 @@ Rationale: Parameters can change over time, see [Updates](#updates).
 
 Covers:
 * [`req~define-configuration-parameters~1`](system_requirements.md#parameter-types)
-* [`req~update-extension~1`](system_requirements.md#uninstalling-extensions)
+* [`req~upgrade-extension~1`](system_requirements.md#uninstalling-extensions)
 
 Needs: impl, utest
 
@@ -617,16 +604,76 @@ Covers:
 
 Needs: impl, utest, itest
 
-#### Updates
-`dsn~update-extension~1`
-Status: draft
+#### Upgrades
+`dsn~upgrade-extension~1`
+
+EM can upgrade an installed extensions and its instances to the latest version.
+
+Covers:
+* [`req~upgrade-extension~1`](system_requirements.md#uninstalling-extensions)
+* [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
+
+Needs: impl, utest, itest
+
+### Extension Context
 
-EM can update an installed extensions and its instances to the latest version.
+The extension context allows extension definitions to interact with the extension manager and the database (e.g. by executing queries).
+
+#### Extension Context SQL Client
+`dsn~extension-context-sql-client~1`
+
+The SQL client in the extension context allows the extension definition to execute statements and run queries against the database and process query results.
+
+Rationale:
+
+* This allows the extension to create necessary database objects like `SCRIPT`s, `CONNECTION`s and `VIRTUAL SCHEMA`s.
+* The extension can also run arbitrary queries in order to read required information about their installations (e.g. scripts) and instances (e.g. virtual schemas).
+* While EM also provides [access to metadata](#extension-context-metadata) via the context, this information may not be sufficient. Executing arbitrary queries ensures maximum flexibility for extensions.
 
 Covers:
-* [`req~update-extension~1`](system_requirements.md#uninstalling-extensions)
 * [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
 
+Needs: impl, utest, itest
+
+#### Extension Context BucketFS
+`dsn~extension-context-bucketfs~1`
+
+The BucketFS client in the extension context allows the extension definition to interact with BucketFS.
+
+Covers:
+* [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
+
+Needs: impl, utest, itest
+
+##### Resolving Files in BucketFS
+`dsn~resolving-files-in-bucketfs~1`
+
+EM resolves filenames to absolute paths in BucketFS.
+
+Rationale:
+
+Extensions require files in BucketFS. The exact location in BucketFS of the files is not known when creating an extension, only the file name (e.g. `document-files-virtual-schema-dist-7.3.3-s3-2.6.2.jar`). In order to create adapter scripts, the extension must know the absolute path of the file in BucketFS (e.g. `/buckets/bfsdefault/default/document-files-virtual-schema-dist-7.3.3-s3-2.6.2.jar`). So EM must allow the extension to resolve file names to an absolute path in BucketFS.
+
+Needs: impl, utest, itest
+
+#### Extension Context Metadata
+`dsn~extension-context-metadata~1`
+
+The Metadata client in the extension context allows the extension definition to read Metadata (e.g. tables `SYS.EXA_ALL_SCRIPTS` or `SYS.EXA_ALL_VIRTUAL_SCHEMAS`) from the database.
+
+Rationale:
+
+* This information is necessary for the extension to find its installations (e.g. scripts) and instances (e.g. virtual schemas).
+* As an alternative extensions could also use the [SQL client](#extension-context-sql-client) included in the context, but this would require duplicating SQL queries across many extensions.
+* The schema for metadata table `SYS.EXA_ALL_VIRTUAL_SCHEMAS` has changed between Exasol v7 and v8. So the code for reading this table requires distinction between the two versions.
+  * Moving this code to the extensions would cause even more code duplication that is hard to maintain.
+  * **Note:** EM will only need to work with Exasol v8, so support for v7 is actually not necessary. However the Docker container for v8 currently does not support running Python and Java UDFs. Until this is fixed we still need v7 for integration testing.
+
+Covers:
+* [`req~install-extension-database-objects~1`](system_requirements.md#update-extension)
+
+Needs: impl, utest, itest
+
 ## Cross-cutting Concerns
 
 ## Design Decisions
@@ -707,3 +754,15 @@ TypeScript adds a bit of complexity at build time for transpiling TypeScript to
 ## Quality Scenarios
 
 ## Risks
+
+### Multiple Versions of the Same Extension Definition
+
+When multiple versions of the same extension definition (i.e. JavaScript files) are present in the extension registry, EM will query all of them when listing installations. For each version EM will find the same installations (e.g. adapter script). In result EM might return multiple installations where only one is present.
+
+#### Mitigation
+
+A possible mitigation could be that EM filters the registry and for each extension definition keeps only the latest version.
+
+#### Decision
+
+We accept the risk for now and ensure that the registry only contains a single version of each extension definition.

doc/system_requirements.md

@@ -65,8 +65,8 @@ EM allows the DBA to configure an extension, e.g. in order to set up a Virtual S
 
 Needs: req
 
-#### Updating an Extension
-`feat~update-extension~1`
+#### Upgrading an Extension
+`feat~upgrade-extension~1`
 
 EM allows DBA to install a new version of an extension that was already installed in an older version.
 
@@ -160,20 +160,22 @@ Covers:
 
 Needs: dsn
 
-#### Update extension
-`req~update-extension~1`
+#### Upgrade Extension
+`req~upgrade-extension~1`
 
-When updating an extension that is already installed in an older version, EM checks if any parameter definition has changed. If there were breaking changes, EM cannot perform the update automatically and aborts the installation with an appropriate error message.
+When upgrading an extension that is already installed in an older version, EM runs code provided by the extension.
 
-(( how to define "breaking" changes?
-any change in parameters is a breaking change
-user can uninstall the old version and install the new one
-))
+Rationale:
 
-Rationale: The only option would be to add update scripts that define how to convert the parameters from one version to another. However, that is currently out of scope.
+* Upgrading to a new version requires actions specific for a concrete extension. Thus a general implementation in EM is not feasible.
+* Extensions usually upgrade themselves by updating referenced files in `ADAPTER SCRIPT`s and `SET SCRIPT`s. This will automatically upgrade all instances (e.g. virtual schemas) using these scripts.
+* If necessary the extension's upgrade code may list and modify individual instances of the extension, e.g. when parameters of a virtual schema change between versions and can be converted automatically.
+  * The extension's upgrade code must fail with a meaningful error message if automatic modification is not possible.
+  * EM currently does not allow reading parameter values from a `CONNECTION`. This means that upgrades requiring modifying parameters are not possible at the moment.
+  * If necessary, we can implement this using a [Lua UDF](https://docs.exasol.com/db/latest/database_concepts/udf_scripts/lua.htm#AccessingConnectionDefinitions) and provide a context method to the extension's code.
 
 Covers:
-* [`feat~update-extension~1`](#uninstall-an-extension)
+* [`feat~upgrade-extension~1`](#upgrading-an-extension)
 
 Needs: dsn