From 0de6500f25e812f130bf9a192e1d70ca0ed3e34e Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 15 Nov 2022 14:45:42 +0100 Subject: [PATCH 1/5] Jobs and services: Added `create_parameters` property in responses to define the parameters that can be submitted suring creation #276 --- CHANGELOG.md | 1 + openapi.yaml | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d922c7c..33975166 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) +- `GET /jobs` and `GET /services`: Added `create_parameters` property in responses to define the parameters that can be submitted when creating jobs or services. [#276](https://github.com/Open-EO/openeo-api/issues/276) - `GET /jobs` and `GET /jobs/{job_id}`: Added a links property that can for example link to results and logs. [#453](https://github.com/Open-EO/openeo-api/issues/453) - `GET /jobs/{job_id}/results`: - Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) diff --git a/openapi.yaml b/openapi.yaml index 7c7ac79d..28c5b17c 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2580,6 +2580,8 @@ paths: $ref: '#/components/schemas/service' links: $ref: '#/components/schemas/links_pagination' + create_parameters: + $ref: '#/components/schemas/processing_create_parameters' 4XX: $ref: '#/components/responses/client_error_auth' 5XX: @@ -2824,6 +2826,8 @@ paths: $ref: '#/components/schemas/batch_job' links: $ref: '#/components/schemas/links_pagination' + create_parameters: + $ref: '#/components/schemas/processing_create_parameters' 4XX: $ref: '#/components/responses/client_error_auth' 5XX: @@ -5687,6 +5691,22 @@ components: "foo" but cannot be 1. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.10.2). allOf: - $ref: '#/components/schemas/json_schema' + processing_create_parameters: + title: Creation Parameters + description: |- + List of additional custom parameters that a back-end offers during the creation + of batch jobs (`POST /jobs`) and secondary web services (`POST /services`) respectively. + type: array + items: + $ref: '#/components/schemas/parameter' + example: + - name: memory + description: Maximum amount of memory that will be allocated for processing, in gigabytes. + optional: true + default: 32 + schema: + type: integer + minimum: 1 error: title: General Error description: >- From e3265f775e7c0ce13649ca1293ec827e3e754e27 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 16 Nov 2022 11:40:20 +0100 Subject: [PATCH 2/5] Add processing options to a separate endpoint #276 --- CHANGELOG.md | 2 +- openapi.yaml | 39 +++++++++++++++++++++++++++++++++++---- 2 files changed, 36 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 33975166..19d9a009 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) -- `GET /jobs` and `GET /services`: Added `create_parameters` property in responses to define the parameters that can be submitted when creating jobs or services. [#276](https://github.com/Open-EO/openeo-api/issues/276) +- Added `GET /processing_options` to define the parameters that can be submitted when creating jobs or services. [#276](https://github.com/Open-EO/openeo-api/issues/276) - `GET /jobs` and `GET /jobs/{job_id}`: Added a links property that can for example link to results and logs. [#453](https://github.com/Open-EO/openeo-api/issues/453) - `GET /jobs/{job_id}/results`: - Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) diff --git a/openapi.yaml b/openapi.yaml index e2779d58..acf43aa6 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2386,6 +2386,41 @@ paths: $ref: '#/components/responses/client_error_auth' 5XX: $ref: '#/components/responses/server_error' + /processing_options: + get: + summary: Additional processing options + operationId: list-processing-options + description: |- + Lists additional custom processing options that a back-end offers for the + different processing modes (synchronous processing, batch jobs, secondary web services). + + The options specified here can be added to the corresponding `POST` requests + at the top-level of the object sent in the payload. + tags: + - Capabilities + - Data Processing + - Batch Jobs + - Secondary Services + security: + - {} + - Bearer: [] + responses: + '200': + description: >- + An object with a list of parameters per processing mode. + content: + application/json: + schema: + description: Processing parameters per processing mode. + type: object + properties: + create_job_parameters: + $ref: '#/components/schemas/processing_create_parameters' + create_service_parameters: + $ref: '#/components/schemas/processing_create_parameters' + create_synchronous_parameters: + $ref: '#/components/schemas/processing_create_parameters' + /service_types: get: summary: Supported secondary web service protocols @@ -2582,8 +2617,6 @@ paths: $ref: '#/components/schemas/service' links: $ref: '#/components/schemas/links_pagination' - create_parameters: - $ref: '#/components/schemas/processing_create_parameters' 4XX: $ref: '#/components/responses/client_error_auth' 5XX: @@ -2830,8 +2863,6 @@ paths: $ref: '#/components/schemas/batch_job' links: $ref: '#/components/schemas/links_pagination' - create_parameters: - $ref: '#/components/schemas/processing_create_parameters' 4XX: $ref: '#/components/responses/client_error_auth' 5XX: From 514b35b739bfb3e4b5953a4df6b870e75dd75320 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 28 Nov 2022 15:41:34 +0100 Subject: [PATCH 3/5] Options should be optional --- openapi.yaml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/openapi.yaml b/openapi.yaml index 5b8865a1..a23d9d2d 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2478,7 +2478,9 @@ paths: different processing modes (synchronous processing, batch jobs, secondary web services). The options specified here can be added to the corresponding `POST` requests - at the top-level of the object sent in the payload. + at the top-level of the object that is sent as the payload. + All options SHOULD explicitly be made optional with reasonable defaults + as otherwise the interoperability between the implementations decreases. tags: - Capabilities - Data Processing From 830f9c2fcaa9a6d75ccb190cccba630d4a294260 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 18 Dec 2024 16:40:35 +0100 Subject: [PATCH 4/5] Move to an extension --- CHANGELOG.md | 2 +- README.md | 1 + .../processing-parameters/.spectral.yml | 12 +++ extensions/processing-parameters/README.md | 11 +++ extensions/processing-parameters/openapi.yaml | 98 +++++++++++++++++++ extensions/processing-parameters/package.json | 24 +++++ openapi.yaml | 53 ---------- 7 files changed, 147 insertions(+), 54 deletions(-) create mode 100644 extensions/processing-parameters/.spectral.yml create mode 100644 extensions/processing-parameters/README.md create mode 100644 extensions/processing-parameters/openapi.yaml create mode 100644 extensions/processing-parameters/package.json diff --git a/CHANGELOG.md b/CHANGELOG.md index db2cb135..7e1204f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **New extensions:** - [Remote Process Definition Extension](./extensions/remote-process-definition/README.md) + - [Processing Parameters Extension](./extensions/processing-parameters/README.md) - Added `version` property to `GET /processes` [#517](https://github.com/Open-EO/openeo-api/issues/517) ### Fixed @@ -33,7 +34,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /file_formats`: Add `pointcloud` to the `gis_data_types`. [#475](https://github.com/Open-EO/openeo-api/issues/475) - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) -- Added `GET /processing_options` to define the parameters that can be submitted when creating jobs or services. [#276](https://github.com/Open-EO/openeo-api/issues/276) - `GET /jobs` and `GET /jobs/{job_id}`: Added a links property that can for example link to results and logs. [#453](https://github.com/Open-EO/openeo-api/issues/453) - `GET /jobs/{job_id}/results`: - Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) diff --git a/README.md b/README.md index 9917cdfa..f677fee1 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ See also the [changelog](CHANGELOG.md) and the [milestones](https://github.com/O | ------------------------------------------------------------------ | ------- | ------------ | ----------- | | [Commercial Data](extensions/commercial-data/) | 0.1.0 | experimental | Provides an interface for discovering, ordering and using commercial data. | | [Federation](extensions/federation/) | 0.1.0 | experimental | Covers federation aspects, i.e. where multiple back-ends are exposed as a single API. | +| [Processing Parameters](extensions/processing-parameters/) | 0.1.0 | experimental | Explore and handle additional processing parameters that a back-end can offer for the processing modes (sync. processing, batch jobs, web services). | | [Remote Process Definition](extensions/remote-process-definition/) | 0.1.0 | experimental | Load user-defined processes that are hosted externally through the process namespace into process graphs. | ## Repository diff --git a/extensions/processing-parameters/.spectral.yml b/extensions/processing-parameters/.spectral.yml new file mode 100644 index 00000000..ad10553b --- /dev/null +++ b/extensions/processing-parameters/.spectral.yml @@ -0,0 +1,12 @@ +extends: "spectral:oas" +rules: + contact-properties: true + tag-description: true + oas3-parameter-description: true + oas3-unused-component: true + operation-id-kebab-case: + given: "$..operationId" + then: + function: pattern + functionOptions: + match: ^[a-z][a-z0-9\-]*$ \ No newline at end of file diff --git a/extensions/processing-parameters/README.md b/extensions/processing-parameters/README.md new file mode 100644 index 00000000..68acc6af --- /dev/null +++ b/extensions/processing-parameters/README.md @@ -0,0 +1,11 @@ +# Processing Parameters Extension + +The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing options that a back-end can offer for the three processing modes (synchronous processing, batch jobs, secondary web services). + +- Version: **0.1.0** +- Stability: **experimental** +- [OpenAPI document](openapi.yaml) +- Conformance class: `https://api.openeo.org/extensions/processing-parameters/0.1.0` + +**Note:** This document only documents the additions to the specification. +Extensions can not change or break existing behavior of the openEO API. diff --git a/extensions/processing-parameters/openapi.yaml b/extensions/processing-parameters/openapi.yaml new file mode 100644 index 00000000..b19b6c7a --- /dev/null +++ b/extensions/processing-parameters/openapi.yaml @@ -0,0 +1,98 @@ +openapi: 3.0.2 +info: + title: openEO API - Processing Parameters Extension + version: 0.1.0 + description: |- + The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing options that a back-end can offer for the three processing modes (synchronous processing, batch jobs, secondary web services). + contact: + name: openEO Consortium + url: 'https://openeo.org' + email: openeo.psc@uni-muenster.de + license: + name: Apache 2.0 + url: 'http://www.apache.org/licenses/LICENSE-2.0.html' +externalDocs: + url: https://github.com/Open-EO/openeo-api/blob/draft/extensions/workspaces/README.md +tags: + - name: Capabilities + description: General information about the API implementation and other supported capabilities provided by the back-end. + - name: Data Processing + description: Organizes and manages data processing on the back-end, either as synchronous on-demand computation or batch jobs. + - name: Batch Jobs + description: Management of batch processing tasks (jobs) and their results. + - name: Secondary Services + description: On-demand access to data using other web service protocols. +servers: + - url: 'https://openeo.example/api/{version}' + description: >- + The URL of the API MAY freely be chosen by the back-end providers. The + path, including API versioning, is a *recommendation* only. Nevertheless, + all servers MUST support HTTPS as the authentication methods are not + secure with HTTP only! + variables: + version: + default: v1 + description: >- + API versioning is RECOMMENDED. As the openEO API is following + [SemVer](https://semver.org/) only the MAJOR part of the stable + version numbers (i.e. versions >= 1.0.0) SHOULD be used for API + versioning in the URL. The reason is that backward-incompatible + changes are usually introduced by major changes. Therefore, the + version number in the URL MUST not be used by the clients to detect + the version number of the API. Use the version number returned from + `GET /` instead. +paths: + /processing_options: + get: + summary: Additional processing options + operationId: list-processing-options + description: |- + Lists additional custom processing options ("additional back-end specific properties" in the API specification) + that a back-end offers for the different processing modes (synchronous processing, batch jobs, secondary web services). + The options specified here can be added to the corresponding `POST` requests at the top-level of the object that is sent as the payload. + All options SHOULD explicitly be made optional with reasonable defaults as otherwise the interoperability between the implementations decreases. + tags: + - Capabilities + - Data Processing + - Batch Jobs + - Secondary Services + security: + - {} + - Bearer: [] + responses: + '200': + description: >- + An object with a list of parameters per processing mode. + content: + application/json: + schema: + description: Processing parameters per processing mode. + type: object + properties: + create_job_parameters: + $ref: '#/components/schemas/processing_create_parameters' + create_service_parameters: + $ref: '#/components/schemas/processing_create_parameters' + create_synchronous_parameters: + $ref: '#/components/schemas/processing_create_parameters' +components: + schemas: + processing_create_parameters: + title: Creation Parameters + description: |- + List of additional custom parameters that a back-end offers during the creation + of batch jobs (`POST /jobs`) and secondary web services (`POST /services`) respectively. + type: array + items: + $ref: '../../openapi.yaml#/components/schemas/parameter' + example: + - name: memory + description: Maximum amount of memory that will be allocated for processing, in gigabytes. + optional: true + default: 32 + schema: + type: integer + minimum: 1 + securitySchemes: + Bearer: + $ref: '../../openapi.yaml#/components/securitySchemes/Bearer' diff --git a/extensions/processing-parameters/package.json b/extensions/processing-parameters/package.json new file mode 100644 index 00000000..841560a5 --- /dev/null +++ b/extensions/processing-parameters/package.json @@ -0,0 +1,24 @@ +{ + "name": "@openeo/api-extension-processing-parameters", + "version": "0.1.0", + "author": "openEO Consortium", + "license": "Apache-2.0", + "description": "The openEO API specification.", + "homepage": "https://openeo.org", + "bugs": { + "url": "https://github.com/Open-EO/openeo-api/issues" + }, + "repository": { + "type": "git", + "url": "git+https://github.com/Open-EO/openeo-api.git" + }, + "devDependencies": { + "@stoplight/spectral-cli": "^6.6.0", + "redoc-cli": "^0.13.21" + }, + "scripts": { + "start": "redoc-cli serve openapi.yaml --watch --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true", + "build": "redoc-cli bundle openapi.yaml -o redoc.html --title \"openEO API - Processing Parameters Extension\" --cdn --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true", + "test": "spectral lint openapi.yaml" + } +} diff --git a/openapi.yaml b/openapi.yaml index a23d9d2d..52313981 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2469,43 +2469,6 @@ paths: $ref: '#/components/responses/client_error_auth' 5XX: $ref: '#/components/responses/server_error' - /processing_options: - get: - summary: Additional processing options - operationId: list-processing-options - description: |- - Lists additional custom processing options that a back-end offers for the - different processing modes (synchronous processing, batch jobs, secondary web services). - - The options specified here can be added to the corresponding `POST` requests - at the top-level of the object that is sent as the payload. - All options SHOULD explicitly be made optional with reasonable defaults - as otherwise the interoperability between the implementations decreases. - tags: - - Capabilities - - Data Processing - - Batch Jobs - - Secondary Services - security: - - {} - - Bearer: [] - responses: - '200': - description: >- - An object with a list of parameters per processing mode. - content: - application/json: - schema: - description: Processing parameters per processing mode. - type: object - properties: - create_job_parameters: - $ref: '#/components/schemas/processing_create_parameters' - create_service_parameters: - $ref: '#/components/schemas/processing_create_parameters' - create_synchronous_parameters: - $ref: '#/components/schemas/processing_create_parameters' - /service_types: get: summary: Supported secondary web service protocols @@ -5883,22 +5846,6 @@ components: "foo" but cannot be 1. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.10.2). allOf: - $ref: '#/components/schemas/json_schema' - processing_create_parameters: - title: Creation Parameters - description: |- - List of additional custom parameters that a back-end offers during the creation - of batch jobs (`POST /jobs`) and secondary web services (`POST /services`) respectively. - type: array - items: - $ref: '#/components/schemas/parameter' - example: - - name: memory - description: Maximum amount of memory that will be allocated for processing, in gigabytes. - optional: true - default: 32 - schema: - type: integer - minimum: 1 error: title: General Error description: >- From da2eefee834f7e56165331c65440e6e90f2bac29 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 18 Dec 2024 18:17:38 +0100 Subject: [PATCH 5/5] Add UDP extension --- extensions/processing-parameters/README.md | 45 +++++++++++++++++++ extensions/processing-parameters/openapi.yaml | 8 ++-- 2 files changed, 50 insertions(+), 3 deletions(-) diff --git a/extensions/processing-parameters/README.md b/extensions/processing-parameters/README.md index 68acc6af..ca18becd 100644 --- a/extensions/processing-parameters/README.md +++ b/extensions/processing-parameters/README.md @@ -9,3 +9,48 @@ The Processing Parameters Extension to the openEO API provides an interface to e **Note:** This document only documents the additions to the specification. Extensions can not change or break existing behavior of the openEO API. + +This extension adds a new endpoint (`GET /processing_options`, see [OpenAPI document](openapi.yaml)) +to discover the additional processing options that a back-end offers. + +Additionally, this extension allows to provide specific default values for user-defined processes (UDPs, see below). + +In both cases, the parameters and its values are provided separately for each processing mode. + +## Processing Parameters in UDPs + +UDPs can provide default values for specific processing parameters. + +The values for each parameter are provided separately for each processing mode. +The following properties are added to the top-level of a UDP for the respective processing modes: + +- `default_synchronous_parameters` for synchronous processing +- `default_job_parameters` for batch jobs +- `default_service_parameters` for secondary web services + +The schema for each of these properties is: + +```yaml +type: object +additionalProperties: + description: Any type +``` + +The keys of the object are the respecive parameter names. +The values of the object are the (default) values for the parameters. +Schematic restrictions are not defined for the object, but the schemas for the parameters as defined in `GET /processing_options` apply to the given values. +These values provide the defaults unless a user overrides them in the actual data processing request (e.g. `POST /jobs`, see below). + +## Resolving parameters + +Due to the variety of places where processing parameters can be provided, the following +list defines how the parameters must be resolved. The prioritization is as follows: + +1. If present, use the parameter specified in the processing request directly (e.g. in `POST /jobs` as a top-level property) +2. If present, use the default parameter specified in the UDP +3. Otherwise, use the default value for the parameter as specified in `GET /processing_options` + +"Present" means that the property is present in the JSON representation regardless of the value given, i.e. +properties are present if an empty string, an empty array, an empty object, `false`, `0`, or `null` are provided. + +Unrecognized/unknown parameters that are provided through UDPs must be ignored by backends. diff --git a/extensions/processing-parameters/openapi.yaml b/extensions/processing-parameters/openapi.yaml index b19b6c7a..450166ea 100644 --- a/extensions/processing-parameters/openapi.yaml +++ b/extensions/processing-parameters/openapi.yaml @@ -2,8 +2,10 @@ openapi: 3.0.2 info: title: openEO API - Processing Parameters Extension version: 0.1.0 - description: |- - The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing options that a back-end can offer for the three processing modes (synchronous processing, batch jobs, secondary web services). + description: >- + The Processing Parameters Extension to the openEO API provides an interface to explore and handle additional processing options that a back-end can offer for the three processing modes (synchronous processing, batch jobs, and secondary web services). + The openEO API specification allows back-ends to accept "additional back-end specific properties" for each of the processing modes. + As the openEO API specification does not define the name and schemas of these properties, this extension provides a standardized way to define and document these properties. contact: name: openEO Consortium url: 'https://openeo.org' @@ -47,7 +49,7 @@ paths: summary: Additional processing options operationId: list-processing-options description: |- - Lists additional custom processing options ("additional back-end specific properties" in the API specification) + Lists additional custom processing options that a back-end offers for the different processing modes (synchronous processing, batch jobs, secondary web services). The options specified here can be added to the corresponding `POST` requests at the top-level of the object that is sent as the payload. All options SHOULD explicitly be made optional with reasonable defaults as otherwise the interoperability between the implementations decreases.