From 14050fc6a7cfa5d9a9059a27442b58b5fc17126c Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 18 May 2021 11:59:36 +0200 Subject: [PATCH 001/103] Fixed errors.json #394, #395 --- CHANGELOG.md | 4 ++++ errors.json | 10 +--------- 2 files changed, 5 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7750b7a8..de4a5a5c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## Unreleased / Draft +### Fixed + +- Fixed inconsistencies in errors.json: removed `ProcessGraphIdDoesntMatch`, clarified `ProcessGraphMissing`. [#394](https://github.com/Open-EO/openeo-api/issues/394) [#395](https://github.com/Open-EO/openeo-api/issues/395) + ## [1.1.0] - 2021-05-17 ### Added diff --git a/errors.json b/errors.json index c3b9e2f7..4896af22 100644 --- a/errors.json +++ b/errors.json @@ -192,7 +192,7 @@ ] }, "ProcessGraphMissing": { - "description": "The parameter `process` doesn't contain a valid process.", + "description": "The request doesn't contain a valid process. For jobs, services, and sync. processing it is contained in the parameter `process`.", "message": "Invalid process specified.", "http": 400, "tags": [ @@ -202,14 +202,6 @@ "Data Processing" ] }, - "ProcessGraphIdDoesntMatch": { - "description": "If a user-defined process is stored and the ID in the request path and the JSON document are not equal.", - "message": "The ids in the path and in the document don't match.", - "http": 400, - "tags": [ - "User-Defined Processes" - ] - }, "PredefinedProcessExists": { "description": "If a user wants to store a user-defined process with the id of a pre-defined process.", "message": "A pre-defined process with the given identifier exists.", From 6ce62823a195cb9ae1d9e1f1bb6e3ccdc18acb98 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 18 May 2021 12:14:33 +0200 Subject: [PATCH 002/103] Fixed the default value for the version number in the API url #393 --- CHANGELOG.md | 3 ++- openapi.yaml | 25 ++++++++++++++++--------- 2 files changed, 18 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index de4a5a5c..1bc687ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- Fixed inconsistencies in errors.json: removed `ProcessGraphIdDoesntMatch`, clarified `ProcessGraphMissing`. [#394](https://github.com/Open-EO/openeo-api/issues/394) [#395](https://github.com/Open-EO/openeo-api/issues/395) +- Fixed inconsistencies in errors.json: removed `ProcessGraphIdDoesntMatch`, clarified `ProcessGraphMissing`. [#394](https://github.com/Open-EO/openeo-api/issues/394), [#395](https://github.com/Open-EO/openeo-api/issues/395) +- Fixed the default value for the version number in the API url (`v1.0` -> `v1`) and improved the description for API versioning. [#393](https://github.com/Open-EO/openeo-api/issues/393) ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index 45dcf7a2..9356c87c 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -529,16 +529,23 @@ servers: secure with HTTP only! variables: version: - default: v1.0 - description: >- + default: v1 + description: |- API versioning is RECOMMENDED. As the openEO API is following - [SemVer](https://semver.org/) only the MAJOR part of the stable - version numbers (i.e. versions >= 1.0.0) SHOULD be used for API - versioning in the URL. The reason is that backward-incompatible - changes are usually introduced by major changes. Therefore, the - version number in the URL MUST not be used by the clients to detect - the version number of the API. Use the version number returned from - `GET /` instead. + [SemVer](https://semver.org/) only the **major** part of the version + numbers SHOULD be used for API versioning in the URL. To make clear + that it is a version number, it is RECOMMENDED to add the prefix `v`. + + Example: API version `1.2.3` is recommended to use `v1`. + + The reason to only consider the major part is that backward-incompatible + changes are introduced by major changes only. All changes from minor + and patch releases can usually be integrated without breakages and thus + a change in the URL is not really needed. + + The version number in the URL MUST not be used by the clients to detect + the version number of the API. Use the version number returned in the + property `api_version` from `GET /` instead. paths: /: get: From 72f0b0549a135cfa5375d97488e15812071814a0 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Fri, 4 Jun 2021 13:44:00 +0200 Subject: [PATCH 003/103] =?UTF-8?q?Fixed=20the=20Collection=20example=20to?= =?UTF-8?q?=20use=20`gsd`=C2=A0instead=20of=20`eo:gsd`.=20#399?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 3 ++- openapi.yaml | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1bc687ec..e9e9e5c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Fixed inconsistencies in errors.json: removed `ProcessGraphIdDoesntMatch`, clarified `ProcessGraphMissing`. [#394](https://github.com/Open-EO/openeo-api/issues/394), [#395](https://github.com/Open-EO/openeo-api/issues/395) - Fixed the default value for the version number in the API url (`v1.0` -> `v1`) and improved the description for API versioning. [#393](https://github.com/Open-EO/openeo-api/issues/393) +- Fixed the Collection example to use `gsd` instead of `eo:gsd`. [#399](https://github.com/Open-EO/openeo-api/issues/399) ## [1.1.0] - 2021-05-17 @@ -39,7 +40,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Clarified that job results require the property `datetime` and allow for additional properties. [#362](https://github.com/Open-EO/openeo-api/issues/362) - Clarified that billing plans, service names and file formats must be accepted case-insensitive. [#371](https://github.com/Open-EO/openeo-api/issues/371) - Clarified that the first provider listed at `GET /credentials/oidc` is the default provider for OpenID Connect. -- Clarified that `GET /jobs/{job_id}/results` should always return valid signed URLs and the endpoint can be used to renew the signed URLs. [#379](https://github.com/Open-EO/openeo-api/issues/379) +- Clarified that `GET /jobs/{job_id}/results` should always return valid signed URLs and the endpoint can be used to renew the signed URLs. [#379](https://github.com/Open-EO/openeo-api/issues/379) - Fixed casing of potential endpoints `GET /collections/{collection_id}/items` and `GET /collections/{collection_id}/items/{feature_id}`. - Clarified allowed characters in the `path` for uploaded user files. diff --git a/openapi.yaml b/openapi.yaml index 9356c87c..d6612f79 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1425,7 +1425,7 @@ paths: 'sat:orbit_state': - ascending - descending - 'eo:gsd': + 'gsd': - 10 - 20 - 60 From ac1351e03365566f953614d82e47dc36a49163c9 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Fri, 4 Jun 2021 16:17:38 +0200 Subject: [PATCH 004/103] Update stac-spec to 1.0.0 --- CHANGELOG.md | 6 +++++- openapi.yaml | 16 ++++++++-------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e9e9e5c2..9ec34eef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,11 +6,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## Unreleased / Draft +### Changed + +- Updated STAC specification examples and references to v1.0.0. + ### Fixed - Fixed inconsistencies in errors.json: removed `ProcessGraphIdDoesntMatch`, clarified `ProcessGraphMissing`. [#394](https://github.com/Open-EO/openeo-api/issues/394), [#395](https://github.com/Open-EO/openeo-api/issues/395) - Fixed the default value for the version number in the API url (`v1.0` -> `v1`) and improved the description for API versioning. [#393](https://github.com/Open-EO/openeo-api/issues/393) -- Fixed the Collection example to use `gsd` instead of `eo:gsd`. [#399](https://github.com/Open-EO/openeo-api/issues/399) +- Fixed the Collection example to use `gsd` instead of `eo:gsd`. [#399](https://github.com/Open-EO/openeo-api/issues/399) ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index d6612f79..a8dfd56b 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1194,7 +1194,7 @@ paths: $ref: '#/components/schemas/links_pagination' example: collections: - - stac_version: 1.0.0-rc.2 + - stac_version: 1.0.0 type: Collection id: Sentinel-2A title: Sentinel-2A MSI L1C @@ -1237,7 +1237,7 @@ paths: - rel: license href: >- https://scihub.copernicus.eu/twiki/pub/SciHubWebPortal/TermsConditions/Sentinel_Data_Terms_and_Conditions.pdf - - stac_version: 1.0.0-rc.2 + - stac_version: 1.0.0 type: Collection id: MOD09Q1 title: >- @@ -1319,7 +1319,7 @@ paths: allOf: - $ref: '#/components/schemas/collection' example: - stac_version: 1.0.0-rc.2 + stac_version: 1.0.0 stac_extensions: - https://stac-extensions.github.io/datacube/v1.0.0/schema.json type: Collection @@ -3011,9 +3011,9 @@ paths: job has stopped due to an error. The response includes additional metadata. It is a valid - [STAC Item](https://github.com/radiantearth/stac-spec/tree/v1.0.0-rc.2/item-spec) + [STAC Item](https://github.com/radiantearth/stac-spec/tree/v1.0.0/item-spec) (if it has spatial and temporal references included) or a valid - [STAC Collection](https://github.com/radiantearth/stac-spec/tree/v1.0.0-rc.2/collection-spec) + [STAC Collection](https://github.com/radiantearth/stac-spec/tree/v1.0.0/collection-spec) (supported since openEO API version 1.1.0). The assets to download are in both cases available in the property `assets` and have the same structure. All additional metadata is not strictly required @@ -3083,7 +3083,7 @@ paths: allOf: - $ref: '#/components/schemas/collection' example: - stac_version: 1.0.0-rc.2 + stac_version: 1.0.0 id: a3cca2b2aa1e3b5b title: NDVI based on Sentinel 2 description: Deriving minimum NDVI measurements over pixel time series of Sentinel 2 @@ -4285,9 +4285,9 @@ components: Potential fields for the summaries can be found here: - * **[STAC Common Metadata](https://github.com/radiantearth/stac-spec/blob/v1.0.0-rc.2/item-spec/common-metadata.md)**: + * **[STAC Common Metadata](https://github.com/radiantearth/stac-spec/blob/v1.0.0/item-spec/common-metadata.md)**: A list of commonly used fields throughout all domains - * **[Content Extensions](https://github.com/radiantearth/stac-spec/blob/v1.0.0-rc.2/extensions/README.md#list-of-content-extensions)**: + * **[Content Extensions](https://github.com/radiantearth/stac-spec/blob/v1.0.0/extensions/README.md#list-of-content-extensions)**: Domain-specific fields for domains such as EO, SAR and point clouds. * **Custom Properties**: It is generally allowed to add custom fields. From d8e58f5935954fe5370c272d980c2f887347b031 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Fri, 4 Jun 2021 16:35:31 +0200 Subject: [PATCH 005/103] `GET /jobs/{job_id}/results`: Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. #397 --- CHANGELOG.md | 4 ++++ openapi.yaml | 27 ++++++++++++++++++++++++--- 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ec34eef..d99aaff8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## Unreleased / Draft +### Added + +- `GET /jobs/{job_id}/results`: Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) + ### Changed - Updated STAC specification examples and references to v1.0.0. diff --git a/openapi.yaml b/openapi.yaml index a8dfd56b..0fc339a8 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3048,6 +3048,11 @@ paths: but regenerated with new expiration time. Signed URLs that expired MAY return the openEO error `ResultLinkExpired`. + It is **strongly recommended** to add a link with relation type `canonical` + to the STAC Item or STAC Collection, which points to this STAC document + using a signed URL. This way the STAC metadata can be used by non-openEO + clients without additional authentication steps. + If processing has not finished yet requests to this endpoint MUST be rejected with openEO error `JobNotFinished`. tags: @@ -3139,11 +3144,14 @@ paths: roles: - metadata links: - - rel: item + - rel: canonical type: application/json + href: https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/collection.json + - rel: item + type: application/geo+json href: https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/item_1.json - rel: item - type: application/json + type: application/geo+json href: https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/item_2.json application/geo+json: schema: @@ -3783,7 +3791,7 @@ components: $ref: '#/components/schemas/stac_assets' links: type: array - description: >- + description: |- Links related to this batch job result, e.g. a link to an invoice, additional log files or external documentation. @@ -3792,10 +3800,19 @@ components: Clients MUST NOT download the data referenced in the links by default. + It is **strongly recommended** to add a link with relation type + `canonical`, which points to this STAC document using a signed URL. + This way the STAC metadata can be used by non-openEO clients + without additional authentication steps. + For relation types see the lists of [common relation types in openEO](#section/API-Principles/Web-Linking). items: $ref: '#/components/schemas/link' + example: + - rel: canonical + type: application/geo+json + href: https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/item.json file_format: x-additionalPropertiesName: File Format Name title: File Format @@ -4233,6 +4250,10 @@ components: catalog service such as OGC CSW, a human-readable HTML version or a metadata document following another standard such as ISO 19115 or DCAT. + 6. `canonical` (primarily for batch job results): A signed URL + which points to this STAC document. This way the STAC metadata can + be used by non-openEO clients without additional authentication steps. + For additional relation types see also the lists of [common relation types in openEO](#section/API-Principles/Web-Linking) and the STAC specification for Collections. From 5a7a320b9bb347cfd14933ae1f84af6b6eff778b Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 29 Jun 2021 12:22:49 +0200 Subject: [PATCH 006/103] Clarified error.json, added new error codes --- CHANGELOG.md | 2 +- errors.json | 26 ++++++++++++++++++++++++-- 2 files changed, 25 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ec34eef..1455dc96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,7 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- Fixed inconsistencies in errors.json: removed `ProcessGraphIdDoesntMatch`, clarified `ProcessGraphMissing`. [#394](https://github.com/Open-EO/openeo-api/issues/394), [#395](https://github.com/Open-EO/openeo-api/issues/395) +- Fixed inconsistencies in errors.json: removed `ProcessGraphIdDoesntMatch`, clarified `ProcessGraphMissing`, added `ProcessInvalid` and `ProcessGraphInvalid`. [#394](https://github.com/Open-EO/openeo-api/issues/394), [#395](https://github.com/Open-EO/openeo-api/issues/395), [#401](https://github.com/Open-EO/openeo-api/issues/401) - Fixed the default value for the version number in the API url (`v1.0` -> `v1`) and improved the description for API versioning. [#393](https://github.com/Open-EO/openeo-api/issues/393) - Fixed the Collection example to use `gsd` instead of `eo:gsd`. [#399](https://github.com/Open-EO/openeo-api/issues/399) diff --git a/errors.json b/errors.json index 4896af22..276d95a5 100644 --- a/errors.json +++ b/errors.json @@ -191,8 +191,8 @@ "User-Defined Processes" ] }, - "ProcessGraphMissing": { - "description": "The request doesn't contain a valid process. For jobs, services, and sync. processing it is contained in the parameter `process`.", + "ProcessInvalid": { + "description": "The process given is invalid, which ususlly means that the process metadata is invalid.", "message": "Invalid process specified.", "http": 400, "tags": [ @@ -202,6 +202,28 @@ "Data Processing" ] }, + "ProcessGraphMissing": { + "description": "The process doesn't contain a process graph. For jobs, services, and sync. processing the parameter `process` must contain a `process_graph`.", + "message": "Invalid process specified. It doesn't contain a process graph.", + "http": 400, + "tags": [ + "User-Defined Processes", + "Batch Jobs", + "Secondary Services", + "Data Processing" + ] + }, + "ProcessGraphInvalid": { + "description": "The process doesn't contain a valid process graph, which means it doesn't comply to the general structure / schema.", + "message": "Invalid process graph specified.", + "http": 400, + "tags": [ + "User-Defined Processes", + "Batch Jobs", + "Secondary Services", + "Data Processing" + ] + }, "PredefinedProcessExists": { "description": "If a user wants to store a user-defined process with the id of a pre-defined process.", "message": "A pre-defined process with the given identifier exists.", From af13c634de9e2c5e9dff34add52584a00fa605f8 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 8 Jul 2021 17:48:02 +0200 Subject: [PATCH 007/103] "stac_version" example has value "string" #406 --- openapi.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/openapi.yaml b/openapi.yaml index 0fc339a8..d260655d 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -4337,6 +4337,7 @@ components: which MAY not be equal to the [STAC API version](#section/STAC). Supports versions 0.9.x and 1.x.x. pattern: '^(0\.9.\d+|1\.\d+.\d+)' + example: 1.0.0 stac_extensions: type: array description: >- From 07117cc96e5e3bcde76a8afd0252263581bd7eec Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 26 Jul 2021 14:30:03 +0200 Subject: [PATCH 008/103] STAC data cube version number update in examples --- openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi.yaml b/openapi.yaml index d260655d..c89208cc 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1321,7 +1321,7 @@ paths: example: stac_version: 1.0.0 stac_extensions: - - https://stac-extensions.github.io/datacube/v1.0.0/schema.json + - https://stac-extensions.github.io/datacube/v2.0.0/schema.json type: Collection id: Sentinel-2 title: Sentinel-2 MSI L2A From b5a73fcd5b184d3f02d6c9a4b2aa2ebd45055e44 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 27 Jul 2021 17:09:24 +0200 Subject: [PATCH 009/103] Clarify user_id, add relation types for registration, password recovery and other user account related tasks #404 --- CHANGELOG.md | 7 +++++++ openapi.yaml | 56 +++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 49 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a00c455..9aa9a266 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - `GET /jobs/{job_id}/results`: Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) +- `GET /`: New Relation types: [#404](https://github.com/Open-EO/openeo-api/issues/404) + - `create-form` to link to the registration page + - `recovery-form` to link to the credentials recovery page. +- `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) +- Recommendation to add media types and titles to links for a better user experience. ### Changed @@ -19,6 +24,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Fixed inconsistencies in errors.json: removed `ProcessGraphIdDoesntMatch`, clarified `ProcessGraphMissing`, added `ProcessInvalid` and `ProcessGraphInvalid`. [#394](https://github.com/Open-EO/openeo-api/issues/394), [#395](https://github.com/Open-EO/openeo-api/issues/395), [#401](https://github.com/Open-EO/openeo-api/issues/401) - Fixed the default value for the version number in the API url (`v1.0` -> `v1`) and improved the description for API versioning. [#393](https://github.com/Open-EO/openeo-api/issues/393) - Fixed the Collection example to use `gsd` instead of `eo:gsd`. [#399](https://github.com/Open-EO/openeo-api/issues/399) +- Clarify use of `user_id`. [#404](https://github.com/Open-EO/openeo-api/issues/404) +- Clarify that the relation type `version-history` should include `/.well-known/openeo` in the URL. ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index c89208cc..d3f0e4ec 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -46,6 +46,8 @@ info: 3. `about`: A resource that is related or further explains the resource, e.g. a user guide. + Generally, it is RECOMMENDED to add descriptive titles (propertty `title`) and media type information (propertty `type`) for a better user experience. + ## Error Handling The success of requests MUST be indicated using [HTTP status codes](https://www.rfc-editor.org/rfc/rfc7231.html#section-6) according to [RFC 7231](https://www.rfc-editor.org/rfc/rfc7231.html). @@ -472,7 +474,7 @@ tags: Therefore, the API leaves some aspects open that have to be handled by the back-ends separately, including - * credential recovery, e.g. retrieving a forgotten password + * credentials recovery, e.g. retrieving a forgotten password * user data management, e.g. changing the users payment details or email address * payments, i.e. topping up credits for pre-paid services or paying for post-paid services * other accounting related tasks, e.g. creating invoices, @@ -793,8 +795,8 @@ paths: following `rel` (relation) types: 1. `version-history`: A link back to the Well-Known URL - (see `/.well-known/openeo`) to allow clients to work on - the most recent version. + (including `/.well-known/openeo`, see the corresponding endpoint for details) + to allow clients to work on the most recent version. 2. `terms-of-service`: A link to the terms of service. If a back-end provides a link to the terms of service, the @@ -824,6 +826,12 @@ paths: 6. `data`: A link to the collections (see `/collections`). Required if full OGC API compatibility is desired. + 7. `create-form`: A link to a user registration page. + + 8. `recovery-form`: A link to a page where a user can + recover a user account (e.g. to reset the password or send + a reminder about the username to the user's email account). + For additional relation types see also the lists of [common relation types in openEO](#section/API-Principles/Web-Linking). type: array @@ -842,6 +850,14 @@ paths: rel: privacy-policy type: text/html title: Privacy Policy + - href: 'https://www.cool-cloud-corp.com/register' + rel: create-form + type: text/html + title: User Registration + - href: 'https://www.cool-cloud-corp.com/lost-password' + rel: recovery-form + type: text/html + title: Reset Password - href: 'http://www.cool-cloud-corp.com/.well-known/openeo' rel: version-history type: application/json @@ -3437,7 +3453,13 @@ paths: - user_id properties: user_id: - $ref: '#/components/schemas/user_id' + type: string + description: >- + A unique user identifier specific to the back-end, which could either be chosen by a user or is automatically generated by the back-end during the registration process at the back-end. + + It is meant to be used as an identifier in URIs (e.g. for sharing purposes), which is primarily used in machine-to-machine communication. Preferrably use the human-readable property `name` to display the user's name in user interfaces instead of the user identifier. + pattern: '^[\w\-\.~]+$' + example: john_doe name: type: string description: >- @@ -3490,6 +3512,14 @@ paths: 2. `edit-form`: Points to a page where the user can edit his user profile. + 3. `alternate`: Any other representation of these (and potentially additional) + user information, e.g. the (public) user profile page. + It is RECOMMENDED to add descriptive titles for a better user experience. + + 4. `related`: Any other user-specific links to be shown in clients, + e.g. to user-specific settings, invoices, etc. It is RECOMMENDED to + add descriptive titles for a better user experience. + For additional relation types see also the lists of [common relation types in openEO](#section/API-Principles/Web-Linking). type: array @@ -3508,6 +3538,10 @@ paths: rel: alternate type: text/vcard title: vCard of John Doe + - href: 'https://example.openeo.org/john_doe/invoices' + rel: related + type: text/html + title: Invoices 4XX: $ref: '#/components/responses/client_error_auth' 5XX: @@ -3549,7 +3583,8 @@ components: the following `rel` (relation) types: 1. `about`: A resource that further explains the runtime, - e.g. a user guide or the documentation. + e.g. a user guide or the documentation. It is RECOMMENDED to + add descriptive titles for a better user experience. For additional relation types see also the lists of [common relation types in openEO](#section/API-Principles/Web-Linking). @@ -3614,7 +3649,8 @@ components: the following `rel` (relation) types: 1. `about`: A resource that further explains the library, - e.g. a user guide or the documentation. + e.g. a user guide or the documentation. It is RECOMMENDED to + add descriptive titles for a better user experience. 2. `latest-version`: If a library has been marked as deprecated, a link SHOULD point to either a new library replacing the deprecated @@ -5314,14 +5350,6 @@ components: Date and time of the last status change, formatted as a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. example: '2017-01-01T09:36:18Z' - user_id: - type: string - description: >- - Unique identifier of the user. MUST match the specified pattern. - This is usually a randomly generated internal identifier from the provider - not meant for displaying purposes. - pattern: '^[\w\-\.~]+$' - example: john_doe description: type: string format: commonmark From 9d958225c7e33fbc5942808c951ce8b5a99b6643 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 27 Jul 2021 17:34:14 +0200 Subject: [PATCH 010/103] Allow the relation type `canonical` to be used generally for resources without requiring Bearer authentication. #405 --- CHANGELOG.md | 1 + openapi.yaml | 12 +++++++++--- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9aa9a266..235d35f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - Recommendation to add media types and titles to links for a better user experience. +- Allow the relation type `canonical` to be used generally for (shared) resources (e.g. UDPs or batch jobs) without requiring Bearer authentication. [#405](https://github.com/Open-EO/openeo-api/issues/405) ### Changed diff --git a/openapi.yaml b/openapi.yaml index d3f0e4ec..0a006747 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -4286,9 +4286,15 @@ components: catalog service such as OGC CSW, a human-readable HTML version or a metadata document following another standard such as ISO 19115 or DCAT. - 6. `canonical` (primarily for batch job results): A signed URL - which points to this STAC document. This way the STAC metadata can - be used by non-openEO clients without additional authentication steps. + 6. `canonical`: This relation type points to a (permanent) URL for resources which + usually require authentication but can can be accessed without (Bearer) authentication. + This way the the exposed resources can be used by non-openEO clients without additional + authentication steps. + For example, a shared user-defined process or batch job results could be + exposed via a canonical link. If a URL should be publicly available to everyone, + it can simply a user-specific URL, e.g. `https://example.com/processes/john_doe/ndvi`. + For resources that should only be accessible to a certain group of user, + a signed URL could be given, e.g. `https://example.com/processes/81zjh1tc2pt52gbx/ndvi`. For additional relation types see also the lists of [common relation types in openEO](#section/API-Principles/Web-Linking) From 66b7483fc56145b1eb7a456b8dfcff760fd895f3 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 17 Aug 2021 15:01:23 +0200 Subject: [PATCH 011/103] pre-defined => predefined, backend => back-end --- CHANGELOG.md | 4 ++-- errors.json | 6 +++--- openapi.yaml | 42 +++++++++++++++++++++--------------------- 3 files changed, 26 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 235d35f2..e258cfff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -105,9 +105,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Additional dimensions in `cube:dimensions` can only be of type `other`. - The extents `interval` and `bbox` can have multiple entries. - Allow all STAC versions that are compatible to STAC 0.9.0. -- Process graph nodes have an additional field `namespace` to distinguish pre-defined and user-defined processes. The default behavior has not changed. [#305](https://github.com/Open-EO/openeo-api/issues/305) +- Process graph nodes have an additional field `namespace` to distinguish predefined and user-defined processes. The default behavior has not changed. [#305](https://github.com/Open-EO/openeo-api/issues/305) - Added `format: commonmark` to all properties supporting CommonMark formatting. -- `errors.json`: The pre-defined error messages have been reworked. [#272](https://github.com/Open-EO/openeo-api/issues/272), [#273](https://github.com/Open-EO/openeo-api/issues/273) +- `errors.json`: The predefined error messages have been reworked. [#272](https://github.com/Open-EO/openeo-api/issues/272), [#273](https://github.com/Open-EO/openeo-api/issues/273) - Added `FolderOperationUnsupported`, `UnsupportedApiVersion`, `PermissionsInsufficient`, `ProcessGraphIdDoesntMatch` and `PredefinedProcessExists`. - Added variable `reason` to error `FilePathInvalid` and `type` to `FileTypeInvalid` and`ServiceUnsupported`. - Replaced the following error messages. The variables in the messages may have changed, too. diff --git a/errors.json b/errors.json index 276d95a5..fa4ad5bf 100644 --- a/errors.json +++ b/errors.json @@ -225,8 +225,8 @@ ] }, "PredefinedProcessExists": { - "description": "If a user wants to store a user-defined process with the id of a pre-defined process.", - "message": "A pre-defined process with the given identifier exists.", + "description": "If a user wants to store a user-defined process with the id of a predefined process.", + "message": "A predefined process with the given identifier exists.", "http": 400, "tags": [ "User-Defined Processes" @@ -241,7 +241,7 @@ ] }, "ProcessUnsupported": { - "description": "A process (pre-defined or user-defined) with the specified identifier is not available. To be used when validating or executing process graphs.", + "description": "A process (predefined or user-defined) with the specified identifier is not available. To be used when validating or executing process graphs.", "message": "Process with identifier '{process}' is not available in namespace '{namespace}'.", "http": 400, "tags": [ diff --git a/openapi.yaml b/openapi.yaml index 0a006747..53c8921c 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -216,12 +216,12 @@ info: A **process** is an operation that performs a specific task on a set of parameters and returns a result. An example is computing a statistical operation, such as mean or median, on selected EO data. A process is similar to a function or method in programming languages. In openEO, processes are used to build a chain of processes ([process graph](#section/Processes/Process-Graphs)), which can be applied to EO data to derive your own findings from the data. - A **pre-defined process** is a process provided by the *back-end*. There is a set of predefined processes by openEO to improve interoperability between back-ends. - Back-ends SHOULD follow these specifications whenever possible. Not all processes need to be implemented by all back-ends. See the **[process reference](https://processes.openeo.org)** for pre-defined processes. + A **predefined process** is a process provided by the *back-end*. There is a set of predefined processes by openEO to improve interoperability between back-ends. + Back-ends SHOULD follow these specifications whenever possible. Not all processes need to be implemented by all back-ends. See the **[process reference](https://processes.openeo.org)** for predefined processes. A **user-defined process** is a process defined by the *user*. It can directly be part of another process graph or be stored as custom process on a back-end. Internally it is a *process graph* with optional additional metadata. - A **process graph** chains specific process calls from the set of pre-defined and user-defined processes together. A process graph itself can be stored as a (user-defined) process again. Similarly to scripts in the context of programming, process graphs organize and automate the execution of one or more processes that could alternatively be executed individually. In a process graph, processes need to be specific, i.e. concrete values or "placeholders" for input parameters need to be specified. These values can be scalars, arrays, objects, references to parameters or previous computations or other process graphs. + A **process graph** chains specific process calls from the set of predefined and user-defined processes together. A process graph itself can be stored as a (user-defined) process again. Similarly to scripts in the context of programming, process graphs organize and automate the execution of one or more processes that could alternatively be executed individually. In a process graph, processes need to be specific, i.e. concrete values or "placeholders" for input parameters need to be specified. These values can be scalars, arrays, objects, references to parameters or previous computations or other process graphs. ## Defining Processes @@ -229,7 +229,7 @@ info: **Back-end providers** MUST follow the schema for predefined processes as in [`GET /processes`](#operation/list-processes) to define new processes. This includes: - * Choosing a intuitive process id, consisting of only letters (a-z), numbers and underscores. It MUST be unique across the pre-defined processes. + * Choosing a intuitive process id, consisting of only letters (a-z), numbers and underscores. It MUST be unique across the predefined processes. * Defining the parameters and their exact (JSON) schemes. * Specifying the return value of a process also with a (JSON) schema. * Providing examples or compliance tests. @@ -241,7 +241,7 @@ info: * Defining the algorithm as a process graph. * Optionally, specifying the additional metadata for processes. - If new process are potentially useful for other back-ends the openEO consortium is happily accepting [pull requests](https://github.com/Open-EO/openeo-processes/pulls) to include them in the list of pre-defined processes. + If new process are potentially useful for other back-ends the openEO consortium is happily accepting [pull requests](https://github.com/Open-EO/openeo-processes/pulls) to include them in the list of predefined processes. ### Schemas @@ -302,7 +302,7 @@ info: One of the nodes in a map of processes (the final one) MUST have the `result` flag set to `true`, all the other nodes can omit it as the default value is `false`. Having such a node is important as multiple end nodes are possible, but in most use cases it is important to exactly specify the return value to be used by other processes. Each child process graph must also specify a result node similar to the "main" process graph. - `process_id` MUST be a valid process ID in the `namespace` given. Clients SHOULD warn the user if a user-defined process is added with the same identifier as one of the pre-defined process. + `process_id` MUST be a valid process ID in the `namespace` given. Clients SHOULD warn the user if a user-defined process is added with the same identifier as one of the predefined process. ### Arguments @@ -349,7 +349,7 @@ info: consists of a single process `absolute`, but it can be arbitrary complex in general. A `` argument MUST at least consist of an object with a key `process_graph`. - Optionally, it can also be described with the same additional properties available for pre-defined processes such as an id, parameters, return values etc. + Optionally, it can also be described with the same additional properties available for predefined processes such as an id, parameters, return values etc. When embedded in a process graph, these additional properties of a user-defined process are usually not used, except for validation purposes. ``` @@ -1545,8 +1545,8 @@ paths: processes: type: array items: - title: Pre-Defined Process - description: A pre-defined process made available by the back-end. + title: Predefined Process + description: A predefined process made available by the back-end. type: object required: - id @@ -1853,13 +1853,13 @@ paths: List of default OpenID Connect clients that can be used by an openEO client for OpenID Connect based authentication. - A default OpenID Connect client is managed by the backend implementer. + A default OpenID Connect client is managed by the back-end implementer. It MUST be configured to be usable without a client secret, which limits its applicability to OpenID Connect grant types like "Authorization Code Grant with PKCE" and "Device Authorization Grant with PKCE" A default OpenID Connect client is provided without availability guarantees. - The backend implementer CAN revoke, reset or update it any time. + The back-end implementer CAN revoke, reset or update it any time. As such, openEO clients SHOULD NOT store or cache default OpenID Connect client information for long term usage. A default OpenID Connect client is intended to simplify authentication for novice users. @@ -2281,7 +2281,7 @@ paths: be changed for stored user-defined processes. The id MUST be unique for the authenticated user, including all - pre-defined processes by the back-end. + predefined processes by the back-end. Partially updating user-defined processes is not supported. @@ -3563,7 +3563,7 @@ components: description: |- The type of the UDF runtime. - Pre-defined types are: + Predefined types are: * `language` for Programming Languages and * `docker` for Docker Containers. @@ -5071,16 +5071,16 @@ components: description: |- The namespace the `process_id` is valid for. - The following options are pre-defined by the openEO API, but additional + The following options are predefined by the openEO API, but additional namespaces may be introduced by back-ends or in a future version of the API. - * `null` (default): Checks both user-defined and pre-defined processes, + * `null` (default): Checks both user-defined and predefined processes, but prefers user-defined processes if both are available. - This allows users to add missing pre-defined processes for portability, + This allows users to add missing predefined processes for portability, e.g. common processes from [processes.openeo.org](https://processes.openeo.org) that have a process graph included. It is RECOMMENDED to log the namespace selected by the back-end for debugging purposes. - * `backend`: Uses exclusively the pre-defined processes listed at `GET /processes`. + * `backend`: Uses exclusively the predefined processes listed at `GET /processes`. * `user`: Uses exclusively the user-defined processes listed at `GET /process_graphs`. If multiple processes with the same identifier exist, Clients SHOULD @@ -5089,10 +5089,10 @@ components: type: string description: |- The identifier for the process. It MUST be unique across its namespace - (e.g. pre-defined processes or user-defined processes). + (e.g. predefined processes or user-defined processes). Clients SHOULD warn the user if a user-defined process is added with the - same identifier as one of the pre-defined process. + same identifier as one of the predefined process. pattern: '^\w+$' example: ndvi process_summary: @@ -6118,9 +6118,9 @@ components: pagination MUST return all resources. If the response is paginated, the `links` array MUST be used to communicate the - links for browsing the pagination with pre-defined `rel` types. See the `links` array schema + links for browsing the pagination with predefined `rel` types. See the `links` array schema for supported `rel` types. - Backend implementations can, unless specified otherwise, use all kind of pagination techniques, + Back-end implementations can, unless specified otherwise, use all kind of pagination techniques, depending on what is supported best by their infrastructure: page-based, offset-based, token-based or something else. The clients SHOULD use whatever is specified in the links with the corresponding `rel` types. From 36dfcb13104f60dd14d5b2641323326fe7eb8ea0 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 18 Aug 2021 12:05:03 +0200 Subject: [PATCH 012/103] Move canonical references to the right place... --- openapi.yaml | 24 +++++++++--------------- 1 file changed, 9 insertions(+), 15 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 53c8921c..c5ce3bef 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -46,6 +46,12 @@ info: 3. `about`: A resource that is related or further explains the resource, e.g. a user guide. + 4. `canonical`: This relation type usually points to a publicly accessible and more long-lived URL for a resource that otherwise often requires (Bearer) authentication with a short-lived token. + This way the the exposed resources can be used by non-openEO clients without additional authentication steps. + For example, a shared user-defined process or batch job results could be exposed via a canonical link. + If a URL should be publicly available to everyone, it can simply a user-specific URL, e.g. `https://example.com/processes/john_doe/ndvi`. + For resources that should only be accessible to a certain group of user, a signed URL could be given, e.g. `https://example.com/processes/81zjh1tc2pt52gbx/ndvi`. + Generally, it is RECOMMENDED to add descriptive titles (propertty `title`) and media type information (propertty `type`) for a better user experience. ## Error Handling @@ -3065,9 +3071,7 @@ paths: Signed URLs that expired MAY return the openEO error `ResultLinkExpired`. It is **strongly recommended** to add a link with relation type `canonical` - to the STAC Item or STAC Collection, which points to this STAC document - using a signed URL. This way the STAC metadata can be used by non-openEO - clients without additional authentication steps. + to the STAC Item or STAC Collection (see the `links` property for details). If processing has not finished yet requests to this endpoint MUST be rejected with openEO error `JobNotFinished`. @@ -3838,8 +3842,8 @@ components: It is **strongly recommended** to add a link with relation type `canonical`, which points to this STAC document using a signed URL. - This way the STAC metadata can be used by non-openEO clients - without additional authentication steps. + This way the STAC metadata can be used by non-openEO clients + without additional authentication steps. For relation types see the lists of [common relation types in openEO](#section/API-Principles/Web-Linking). @@ -4286,16 +4290,6 @@ components: catalog service such as OGC CSW, a human-readable HTML version or a metadata document following another standard such as ISO 19115 or DCAT. - 6. `canonical`: This relation type points to a (permanent) URL for resources which - usually require authentication but can can be accessed without (Bearer) authentication. - This way the the exposed resources can be used by non-openEO clients without additional - authentication steps. - For example, a shared user-defined process or batch job results could be - exposed via a canonical link. If a URL should be publicly available to everyone, - it can simply a user-specific URL, e.g. `https://example.com/processes/john_doe/ndvi`. - For resources that should only be accessible to a certain group of user, - a signed URL could be given, e.g. `https://example.com/processes/81zjh1tc2pt52gbx/ndvi`. - For additional relation types see also the lists of [common relation types in openEO](#section/API-Principles/Web-Linking) and the STAC specification for Collections. From 2e6d17e462d7b31ba8b7494203b2440a88ccd7c8 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 19 Aug 2021 13:20:29 +0200 Subject: [PATCH 013/103] Recommendation for UDF runtime names. #409 --- CHANGELOG.md | 1 + openapi.yaml | 27 +++++++++++++++++---------- 2 files changed, 18 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e258cfff..b6786ac5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - Recommendation to add media types and titles to links for a better user experience. - Allow the relation type `canonical` to be used generally for (shared) resources (e.g. UDPs or batch jobs) without requiring Bearer authentication. [#405](https://github.com/Open-EO/openeo-api/issues/405) +- Recommendation for UDF runtime names. [#409](https://github.com/Open-EO/openeo-api/issues/409) ### Changed diff --git a/openapi.yaml b/openapi.yaml index c5ce3bef..8318e6c8 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1675,19 +1675,24 @@ paths: schema: title: UDF Runtimes type: object - description: >- - Map of available runtime environments. Each runtime - environment has a unique name, which is used as the property - key. The name is used in processes to select the runtime - environment for UDFs, so the names should be stable and - meaningful (see example). Runtime environments can be either a - programming language environment or Docker-based. + description: |- + Map of available runtime environments. Runtime environments + can be either a programming language environment or Docker-based. + + Each runtime environment has a unique name, which is used as + the property key. The name is used in processes to select the + runtime environment for UDFs, so the names should be stable and + meaningful. + It is RECOMMENDED to use the following naming and casing: + * For programming langauge environments use the names as provided + in in the [Scriptol List of Programming Languages](https://www.scriptol.com/programming/list-programming-languages.php). + * For docker images use the docker image identifier excluding the registry path. additionalProperties: x-additionalPropertiesName: UDF Runtime name allOf: - $ref: '#/components/schemas/udf_runtime' example: - PHP7: + PHP: title: PHP v7.x description: Just an example how to reference a docker image. experimental: true @@ -3675,8 +3680,10 @@ components: docker: type: string description: >- - Identifier of a Docker image on Docker Hub or a - private repository, i.e. the docker image name. + Identifier of a Docker image on Docker Hub or a private repository, + which can run openEO UDFs. + This is what is used in the `docker pull` command excluding the tag, + e.g. `php`, `mcr.microsoft.com/dotnet/runtime` or `myregistry.local:5000/testing/test-image`. default: type: string description: The default tag. MUST be one of the values in the `tags` array. From 5a6dfd281cb5b706f50ee7ad96b3bf2d881dce4f Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 25 Aug 2021 13:38:12 +0200 Subject: [PATCH 014/103] =?UTF-8?q?OIDC:=20Allow=20auth=20code=20flow=20an?= =?UTF-8?q?d=20device=20flow=C2=A0(both=20without=20PKCE)=20as=20grants=20?= =?UTF-8?q?for=20`default=5Fclients`.=20#410?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 1 + openapi.yaml | 8 +++++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b6786ac5..1ee25c8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `create-form` to link to the registration page - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) +- `GET /credentials/oids`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) - Recommendation to add media types and titles to links for a better user experience. - Allow the relation type `canonical` to be used generally for (shared) resources (e.g. UDPs or batch jobs) without requiring Bearer authentication. [#405](https://github.com/Open-EO/openeo-api/issues/405) - Recommendation for UDF runtime names. [#409](https://github.com/Open-EO/openeo-api/issues/409) diff --git a/openapi.yaml b/openapi.yaml index 8318e6c8..287800e4 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1897,8 +1897,8 @@ paths: Allowed values: - `implicit`: Implicit Grant as specified in [RFC 6749, sec. 1.3.2](https://www.rfc-editor.org/rfc/rfc6749.html#section-1.3.2) - - `authorization_code+pkce`: Authorization Code Grant as specified in [RFC 6749, sec. 1.3.1](https://www.rfc-editor.org/rfc/rfc6749.html#section-1.3.1), with PKCE extension. - - `urn:ietf:params:oauth:grant-type:device_code+pkce`: Device Authorization Grant (aka Device Code Flow) as specified in [RFC 8628](https://www.rfc-editor.org/rfc/rfc8628.html), with PKCE extension. Note that the combination of this grant with the PKCE extension is *not standardized* yet. + - `authorization_code` / `authorization_code+pkce`: Authorization Code Grant as specified in [RFC 6749, sec. 1.3.1](https://www.rfc-editor.org/rfc/rfc6749.html#section-1.3.1), with or without PKCE extension. + - `urn:ietf:params:oauth:grant-type:device_code` / `urn:ietf:params:oauth:grant-type:device_code+pkce`: Device Authorization Grant (aka Device Code Flow) as specified in [RFC 8628](https://www.rfc-editor.org/rfc/rfc8628.html), with or without PKCE extension. Note that the combination of this grant with the PKCE extension is *not standardized* yet. - `refresh_token`: Refresh Token as specified in [RFC 6749, sec. 1.5](https://www.rfc-editor.org/rfc/rfc6749.html#section-1.5) minItems: 1 uniqueItems: true @@ -1906,7 +1906,9 @@ paths: type: string enum: - 'implicit' + - 'authorization_code' - 'authorization_code+pkce' + - 'urn:ietf:params:oauth:grant-type:device_code' - 'urn:ietf:params:oauth:grant-type:device_code+pkce' - 'refresh_token' redirect_urls: @@ -1914,7 +1916,7 @@ paths: description: |- List of redirect URLs that are whitelisted by the OpenID Connect client. Redirect URLs MUST be provided when the OpenID Connect client supports - the `implicit` or `authorization_code+pkce` authorization flows. + the Implicit Grant or the Authorization Code Grant (with or without PKCE extension). uniqueItems: true items: type: string From 05a2827a4fa1e019738365f0ff5036e6cbbf5daa Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 19 Oct 2021 16:40:07 +0200 Subject: [PATCH 015/103] Add first draft for federation extension --- extensions/federation/README.md | 151 ++++++++++++++++++++++++++++++++ 1 file changed, 151 insertions(+) create mode 100644 extensions/federation/README.md diff --git a/extensions/federation/README.md b/extensions/federation/README.md new file mode 100644 index 00000000..6950c781 --- /dev/null +++ b/extensions/federation/README.md @@ -0,0 +1,151 @@ +# Federation Extension + +The openEO API is a specification for interoperable cloud-based processing of large Earth observation datasets. + +This is an extension for federation aspects, i.e. where multiple back-ends are exposed as a single API. + +Version: **0.1.0** +Stability: **experimental** + +**Note:** This document only documents the additions to the specification. +Extensions can not change or break existing behavior of the openEO API. + +## Backend details + +A new required field `federated:backends` is added to `GET /`: + +```yaml +schema: + type: object + required: + - 'federation:backends' + properties: + 'federation:backends': + description: >- + Lists all back-ends that are part of this federation with details. + They keys of the object are the unique identifiers for the back-ends that are returned in sub-sequent requests (see below). + type: object + minProperties: 2 + additionalProperties: + type: object + required: + - url + properties: + url: + type: string + format: uri + description: >- + URL to the API of the back-end, without the `/.well-known/openeo` + suffix. + title: + type: string + description: Name of the back-end. + description: + type: string + description: A description of the back-end and its specifics. + status: + type: string + enum: + - online + - offline + description: Current status of the back-ends. + default: online +``` + +Example: + +```json +{ + "api_version": "1.1.0", + "federation:backends": { + "vito": { + "title": "VITO", + "url": "https://openeo.vito.be" + }, + "eodc": { + "title": "EODC", + "url": "https://openeo.eodc.eu" + }, + "wwu": { + "title": "WWU Münster", + "url": "https://openeo.wwu.de", + "status": "offline", + "description": "Experimental integration of the WWU HPC" + } + }, + ... +} +``` + +## Lists of resources + +Clients will assume that all lists of resources (e.g. collections, processes, jobs, ...) are the combined from all back-ends listed in `GET /`. Federated APIs can expose if any of the back-ends is not available and thus is not part of the response. + +```yaml +schema: + type: object + properties: + 'federation:missing': + description: >- + Lists all back-ends that were not considered in the response (e.g. because they were not accessible). + If not given or empty, all back-ends were considered for creating the response. + Back-ends that were listed as offline in the capabilities still need to be listed here. + type: array + items: + type: string + description: The IDs of the back-ends that are not present in the response. +``` + +Example: + +```json +{ + "federation:missing": ["wwu"], + "collections": [...], + "links": [...] +} +``` + +Please note that this does not apply to `GET /udf_runtimes` and `GET /service_types`, which can only expose the availibility per UDF runtime / service type. + +## Resources supported only by a subset of back-ends + +Every discoverable resource that is defined as an object and allows to contain additional properties, can list the backends that support or host the exposed resource/functionality (e.g. collections, processes, process parameters, file formats, file format parameters, services, jobs, ...) + +```yaml +schema: + type: object + properties: + 'federation:hosted': + description: >- + Lists all back-ends that support or host the resource. + If not given, all back-ends support the resource. + type: array + minItems: 1 + items: + type: string + description: The IDs of the back-ends that are not present in the response. +``` + +Example: + +```json +{ + "process_id": "example", + "federation:hosted": ["vito", "eodc"], + "parameters": [ + { + "name": "parameter1", + "description": "A parameter supported by both back-ends...", + "schema": {} + }, + { + "name": "proprietary_parameter", + "description": "A parameter supported only by EODC.", + "federation:hosted": ["eodc"], + "schema": {} + } + ] + ... +} +``` \ No newline at end of file From 1ba501bf2aa4df6ff94555f8256a07b4d536e9d7 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 19 Oct 2021 18:30:45 +0200 Subject: [PATCH 016/103] Clarify Collection placement and changed naming. --- extensions/federation/README.md | 45 +++++++++++++++++++++++++-------- 1 file changed, 34 insertions(+), 11 deletions(-) diff --git a/extensions/federation/README.md b/extensions/federation/README.md index 6950c781..0a7c8af4 100644 --- a/extensions/federation/README.md +++ b/extensions/federation/README.md @@ -4,23 +4,25 @@ The openEO API is a specification for interoperable cloud-based processing of la This is an extension for federation aspects, i.e. where multiple back-ends are exposed as a single API. -Version: **0.1.0** -Stability: **experimental** +- Version: **0.1.0** +- Stability: **experimental** **Note:** This document only documents the additions to the specification. Extensions can not change or break existing behavior of the openEO API. ## Backend details -A new required field `federated:backends` is added to `GET /`: +A new required field `federation` is added to `GET /` to enable federation. + +### OpenAPI fragment ```yaml schema: type: object required: - - 'federation:backends' + - 'federation' properties: - 'federation:backends': + 'federation': description: >- Lists all back-ends that are part of this federation with details. They keys of the object are the unique identifiers for the back-ends that are returned in sub-sequent requests (see below). @@ -52,12 +54,12 @@ schema: default: online ``` -Example: +### Example ```json { "api_version": "1.1.0", - "federation:backends": { + "federation": { "vito": { "title": "VITO", "url": "https://openeo.vito.be" @@ -81,6 +83,8 @@ Example: Clients will assume that all lists of resources (e.g. collections, processes, jobs, ...) are the combined from all back-ends listed in `GET /`. Federated APIs can expose if any of the back-ends is not available and thus is not part of the response. +### OpenAPI fragment + ```yaml schema: type: object @@ -96,7 +100,7 @@ schema: description: The IDs of the back-ends that are not present in the response. ``` -Example: +### Example ```json { @@ -116,7 +120,7 @@ Every discoverable resource that is defined as an object and allows to contain a schema: type: object properties: - 'federation:hosted': + 'federation:backends': description: >- Lists all back-ends that support or host the resource. If not given, all back-ends support the resource. @@ -127,12 +131,16 @@ schema: description: The IDs of the back-ends that are not present in the response. ``` -Example: +**Note:** In Collections this should be placed inside `summaries` if users should be able to filter on back-ends in `load_collection`. + +### Examples + +#### Process ```json { "process_id": "example", - "federation:hosted": ["vito", "eodc"], + "federation:backends": ["vito", "eodc"], "parameters": [ { "name": "parameter1", @@ -148,4 +156,19 @@ Example: ] ... } +``` + +#### Collection + +```json +{ + "stac_version": "1.0.0", + "id": "example", + "description": "...", + "summaries": { + "federation:backends": ["vito", "eodc"], + ... + }, + ... +} ``` \ No newline at end of file From 592e2662c23843745e2e8c28269b19685cf13cd1 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Fri, 22 Oct 2021 16:48:38 +0200 Subject: [PATCH 017/103] Relax requirement to use HTTP 501 #415 (#418) --- CHANGELOG.md | 1 + openapi.yaml | 6 +++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ee25c8c..654a3601 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed - Updated STAC specification examples and references to v1.0.0. +- Relaxed requirement that unsupported endpoints must return HTTP status code 501. Instead also HTTP status code 404 can be used (and is regularly used in practice). [#415](https://github.com/Open-EO/openeo-api/issues/415) ### Fixed diff --git a/openapi.yaml b/openapi.yaml index 287800e4..e32e9913 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -112,11 +112,15 @@ info: - **404 Not Found**: The resource specified by the path does not exist, i.e. one of the resources belonging to the specified identifiers are not available at the back-end. - *Note:* Unsupported endpoints MUST use HTTP status code 501. + *Note:* Unsupported endpoints MAY also return HTTP status code 501. - **500 Internal Server Error**: The error has its origin on server side and no other status code in the 500 range is suitable. + - **501 Not Implemented**: + The requested endpoint is specified by the openEO API, but is not implemented (yet) by the back-end. + *Note:* Unsupported endpoints MAY also return HTTP status code 404. + If a HTTP status code in the 400 range is returned, the client SHOULD NOT repeat the request without modifications. For HTTP status code in the 500 range, the client MAY repeat the same request later. From 9c57dfa1e2c6f55a0c88a0066717d89f2e5864d2 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 28 Oct 2021 15:08:04 +0200 Subject: [PATCH 018/103] Clarify currency description #421 --- openapi.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index e32e9913..c800ca6c 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -728,10 +728,10 @@ paths: description: >- The currency the back-end is billing in. The currency MUST be either a valid currency code as defined in - ISO-4217 or a proprietary currency, e.g. tiles or - back-end specific credits. If set to the default value - `null`, budget and costs are not supported by the - back-end and users can't be charged. + ISO-4217 or a back-end specific unit that is used for + billing such as credits, tiles or CPU hours. + If set to the default value `null`, budget and costs are + not supported by the back-end and users can't be charged. type: string nullable: true default: null From 59b4bab5f1e5f30ca9b938c378740759e83343c4 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 28 Oct 2021 18:19:16 +0200 Subject: [PATCH 019/103] `GET /`: Removed the superfluous default value for `currency`. #423 --- CHANGELOG.md | 1 + openapi.yaml | 1 - 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 654a3601..672fa250 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Fixed the Collection example to use `gsd` instead of `eo:gsd`. [#399](https://github.com/Open-EO/openeo-api/issues/399) - Clarify use of `user_id`. [#404](https://github.com/Open-EO/openeo-api/issues/404) - Clarify that the relation type `version-history` should include `/.well-known/openeo` in the URL. +- `GET /`: Removed the superfluous default value for `currency`. [#423](https://github.com/Open-EO/openeo-api/issues/423) ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index c800ca6c..5c780730 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -734,7 +734,6 @@ paths: not supported by the back-end and users can't be charged. type: string nullable: true - default: null example: USD default_plan: type: string From 42e963bed6450e59fc7c6e1d4b4d7f7bc66934e6 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 28 Oct 2021 18:49:54 +0200 Subject: [PATCH 020/103] Improvements from code-review --- extensions/federation/README.md | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/extensions/federation/README.md b/extensions/federation/README.md index 0a7c8af4..355b38e6 100644 --- a/extensions/federation/README.md +++ b/extensions/federation/README.md @@ -24,7 +24,7 @@ schema: properties: 'federation': description: >- - Lists all back-ends that are part of this federation with details. + Lists all back-ends that are part of this federation with details. They keys of the object are the unique identifiers for the back-ends that are returned in sub-sequent requests (see below). type: object minProperties: 2 @@ -37,8 +37,9 @@ schema: type: string format: uri description: >- - URL to the API of the back-end, without the `/.well-known/openeo` - suffix. + URL to the versioned API endpoint of the back-end, + so a URL that is available through well-known discovery on the back-end. + example: https://openeo.provider.org/api/v1/ title: type: string description: Name of the back-end. @@ -52,6 +53,16 @@ schema: - offline description: Current status of the back-ends. default: online + last_status_check: + type: string + format: date-time + description: The time at which the status of the back-end was checked last. + last_successful_check: + type: string + format: date-time + description: >- + If the `status` is `offline`: The time at which the back-end was checked and available the last time. + Otherwise, this is equal to the property `last_status_check`. ``` ### Example @@ -91,13 +102,13 @@ schema: properties: 'federation:missing': description: >- - Lists all back-ends that were not considered in the response (e.g. because they were not accessible). + Lists all back-ends that were not considered in the response (e.g. because they were not accessible). If not given or empty, all back-ends were considered for creating the response. Back-ends that were listed as offline in the capabilities still need to be listed here. type: array items: type: string - description: The IDs of the back-ends that are not present in the response. + description: The ID of a back-end. ``` ### Example @@ -110,7 +121,8 @@ schema: } ``` -Please note that this does not apply to `GET /udf_runtimes` and `GET /service_types`, which can only expose the availibility per UDF runtime / service type. +Please note that this does not apply to `GET /udf_runtimes` and `GET /service_types`. These endpoints are not extensible and can only expose the availibility per UDF runtime / service type. +This is due to the fact that these two endpoints return an object with a key-value-pair per UDF runtime / service type and adding a key-value-pair for federation purposes would be interpreted as a new UDF runtime or service type by clients. ## Resources supported only by a subset of back-ends @@ -128,7 +140,7 @@ schema: minItems: 1 items: type: string - description: The IDs of the back-ends that are not present in the response. + description: The ID of a back-end. ``` **Note:** In Collections this should be placed inside `summaries` if users should be able to filter on back-ends in `load_collection`. From ce41e2d37e99c8ef19ba4eac3b6ef0086adff3d9 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 28 Oct 2021 19:00:14 +0200 Subject: [PATCH 021/103] Add federation:description --- extensions/federation/README.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/extensions/federation/README.md b/extensions/federation/README.md index 355b38e6..20c0a033 100644 --- a/extensions/federation/README.md +++ b/extensions/federation/README.md @@ -141,6 +141,16 @@ schema: items: type: string description: The ID of a back-end. + 'federation:description': + description: |- + Allows back-ends to provide backend-specific details about this resource in a human-readable way. + + [CommonMark 0.29](http://commonmark.org/) syntax MAY be used for rich text representation. + type: array + minItems: 1 + items: + type: string + description: The ID of a back-end. ``` **Note:** In Collections this should be placed inside `summaries` if users should be able to filter on back-ends in `load_collection`. @@ -162,7 +172,7 @@ schema: { "name": "proprietary_parameter", "description": "A parameter supported only by EODC.", - "federation:hosted": ["eodc"], + "federation:backends": ["eodc"], "schema": {} } ] @@ -177,6 +187,10 @@ schema: "stac_version": "1.0.0", "id": "example", "description": "...", + "federation:description": { + "eodc": "The collection contains data for Austria only.", + "vito": "The collection contains data for Belgium only.", + }, "summaries": { "federation:backends": ["vito", "eodc"], ... From 85d558ee5af0c6a8a6aab0cb4c18f0deb21ed84c Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Fri, 29 Oct 2021 16:02:14 +0200 Subject: [PATCH 022/103] `GET /`: Removed the superfluous default value for `currency`. #423 (part two) --- openapi.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 5c780730..82158ec0 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -730,8 +730,8 @@ paths: MUST be either a valid currency code as defined in ISO-4217 or a back-end specific unit that is used for billing such as credits, tiles or CPU hours. - If set to the default value `null`, budget and costs are - not supported by the back-end and users can't be charged. + If set to `null`, budget and costs are not supported + by the back-end and users can't be charged. type: string nullable: true example: USD From e0f1c83702b8e0d01fe7cb1ac08435d07007e912 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 2 Nov 2021 13:40:17 +0100 Subject: [PATCH 023/103] Clarify: Send token to discovery endpoints #416 (#417) Co-authored-by: Stefaan Lippens --- CHANGELOG.md | 1 + openapi.yaml | 9 +++++++++ 2 files changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 672fa250..08636dcd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Fixed the Collection example to use `gsd` instead of `eo:gsd`. [#399](https://github.com/Open-EO/openeo-api/issues/399) - Clarify use of `user_id`. [#404](https://github.com/Open-EO/openeo-api/issues/404) - Clarify that the relation type `version-history` should include `/.well-known/openeo` in the URL. +- Clarify that clients should (re-)request capabilities and discovery endpoints with token if available and supported. [#416](https://github.com/Open-EO/openeo-api/issues/416) - `GET /`: Removed the superfluous default value for `currency`. [#423](https://github.com/Open-EO/openeo-api/issues/423) ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index 82158ec0..3cc8de81 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -142,6 +142,9 @@ info: + **Note:** Although it is possible to request several public endpoints for capabilities and discovery that don't require authorization, it is RECOMMENDED that clients (re-)request the public endpoints that support Bearer authentication with the Bearer token once available to also retrieve any private data that is made available specifically for the authenticated user. + This may require that clients clear any cached data they retrieved from public endpoints before. + # Cross-Origin Resource Sharing (CORS) > Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources [...] on a web page to be requested from another domain outside the domain from which the first resource was served. [...] @@ -1192,6 +1195,10 @@ paths: features / extensions and [STAC extensions](https://stac-extensions.github.io) can be implemented in addition to what is documented here. + + Note: Although it is possible to request public collections without + authorization, it is RECOMMENDED that clients (re-)request the collections + with the Bearer token once available to also retrieve any private collections. tags: - EO Data Discovery security: @@ -1319,6 +1326,8 @@ paths: features / extensions and [STAC extensions](https://stac-extensions.github.io) can be implemented in addition to what is documented here. + + Note: Providing the Bearer token is REQUIRED for private collections. tags: - EO Data Discovery security: From 034329d6b3abf93ab40959bc3d2e4dfe28c27530 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 17 Nov 2021 10:44:15 +0100 Subject: [PATCH 024/103] Make the web service parameter example a little less cumbersome from a user perspective --- openapi.yaml | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 3cc8de81..daa33337 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2455,22 +2455,28 @@ paths: schema: type: string default: roads - - name: spatial_extent_west - description: The lower left corner for coordinate axis 1 of the extent currently shown to the consumer. + - name: spatial_extent + description: A bounding box in WGS84. schema: - type: number - - name: spatial_extent_south - description: The lower left corner for coordinate axis 2 of the extent currently shown to the consumer. - schema: - type: number - - name: spatial_extent_east - description: The upper right corner for coordinate axis 1 of the extent currently shown to the consumer. - schema: - type: number - - name: spatial_extent_north - description: The upper right corner for coordinate axis 2 of the extent currently shown to the consumer. - schema: - type: number + type: object + required: + - west + - south + - east + - north + properties: + west: + description: West (lower left corner, coordinate axis 1). + type: number + south: + description: South (lower left corner, coordinate axis 2). + type: number + east: + description: East (upper right corner, coordinate axis 1). + type: number + north: + description: North (upper right corner, coordinate axis 2). + type: number links: - href: 'https://www.opengeospatial.org/standards/wms' rel: about From c0855ce51324afccb1b0b0221b053414d5a4ebf9 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 24 Nov 2021 14:32:59 +0100 Subject: [PATCH 025/103] Upgrade CI --- .github/workflows/docs.yml | 5 ++--- .github/workflows/tests.yml | 5 ++--- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index a5bd1e48..63e6176d 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -9,13 +9,12 @@ on: jobs: deploy: runs-on: ubuntu-latest - strategy: - matrix: - node-version: [14.x] steps: - name: Inject env variables uses: rlespinasse/github-slug-action@v3.x - uses: actions/setup-node@v1 + with: + node-version: '16' - uses: actions/checkout@v2 - run: | npm install diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index b61f951c..0f896959 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -3,11 +3,10 @@ on: [push, pull_request] jobs: deploy: runs-on: ubuntu-latest - strategy: - matrix: - node-version: [14.x] steps: - uses: actions/setup-node@v1 + with: + node-version: '16' - uses: actions/checkout@v2 - name: Run tests run: | From 3582c3112ac58ba4d5be49d896cde440bafc5925 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 29 Nov 2021 18:42:25 +0100 Subject: [PATCH 026/103] Fix typo in CHANGELOG --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 08636dcd..c3c1aa51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `create-form` to link to the registration page - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) -- `GET /credentials/oids`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) +- `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) - Recommendation to add media types and titles to links for a better user experience. - Allow the relation type `canonical` to be used generally for (shared) resources (e.g. UDPs or batch jobs) without requiring Bearer authentication. [#405](https://github.com/Open-EO/openeo-api/issues/405) - Recommendation for UDF runtime names. [#409](https://github.com/Open-EO/openeo-api/issues/409) From f232fec14c3f0a056196d6fa4dd174fced026d8c Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 8 Dec 2021 16:38:52 +0100 Subject: [PATCH 027/103] Logs: Add data processing parameter for minimum severity #329 (#431) --- CHANGELOG.md | 5 ++- openapi.yaml | 98 ++++++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 83 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c3c1aa51..87a99c27 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,12 +8,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- `GET /jobs/{job_id}/results`: Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) - `GET /`: New Relation types: [#404](https://github.com/Open-EO/openeo-api/issues/404) - `create-form` to link to the registration page - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) +- `GET /jobs/{job_id}/results`: Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) +- `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs` and `POST /result`: Added `log_level` property in responses to reflect the minimum log level that has been chosen by the user. [#329](https://github.com/Open-EO/openeo-api/issues/329) +- Added property `log_level` to secondary web service, batch job and synchronous processing endpoints to indicate the minimum severity level that should be stored for logs. [#329](https://github.com/Open-EO/openeo-api/issues/329) - Recommendation to add media types and titles to links for a better user experience. - Allow the relation type `canonical` to be used generally for (shared) resources (e.g. UDPs or batch jobs) without requiring Bearer authentication. [#405](https://github.com/Open-EO/openeo-api/issues/405) - Recommendation for UDF runtime names. [#409](https://github.com/Open-EO/openeo-api/issues/409) @@ -32,6 +34,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Clarify that the relation type `version-history` should include `/.well-known/openeo` in the URL. - Clarify that clients should (re-)request capabilities and discovery endpoints with token if available and supported. [#416](https://github.com/Open-EO/openeo-api/issues/416) - `GET /`: Removed the superfluous default value for `currency`. [#423](https://github.com/Open-EO/openeo-api/issues/423) +- Reflect that the `debug` process has been renamed to `inspect`. ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index daa33337..4121cb11 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2176,6 +2176,8 @@ paths: $ref: '#/components/schemas/budget' plan: $ref: '#/components/schemas/billing_plan_null_default' + log_level: + $ref: '#/components/schemas/min_log_level_default' /process_graphs: get: summary: List all user-defined processes @@ -2631,6 +2633,8 @@ paths: $ref: '#/components/schemas/billing_plan_null_default' budget: $ref: '#/components/schemas/budget' + log_level: + $ref: '#/components/schemas/min_log_level_default' description: The base data for the secondary web service to create '/services/{service_id}': parameters: @@ -2638,11 +2642,10 @@ paths: patch: summary: Modify a service operationId: update-service - description: >- + description: |- Modifies an existing secondary web service at the back-end, but maintain the identifier. Changes can be grouped in a single request. - User have to create a new service to change the service type. tags: - Secondary Services @@ -2677,6 +2680,8 @@ paths: $ref: '#/components/schemas/billing_plan_null' budget: $ref: '#/components/schemas/budget_update' + log_level: + $ref: '#/components/schemas/min_log_level_update' description: The data to change for the specified secondary web service. get: summary: Full metadata for a service @@ -2731,7 +2736,7 @@ paths: Back-ends can log any information that may be relevant for a user. Users can log information during data processing using respective - processes such as `debug`. + processes such as `inspect`. If requested consecutively while the secondary service is enabled, it is RECOMMENDED that clients use the offset parameter to get only the entries @@ -2851,6 +2856,8 @@ paths: $ref: '#/components/schemas/billing_plan_null_default' budget: $ref: '#/components/schemas/budget' + log_level: + $ref: '#/components/schemas/min_log_level_default' description: 'Specifies the job details, e.g. the user-defined process and billing details.' '/jobs/{job_id}': parameters: @@ -2896,6 +2903,8 @@ paths: $ref: '#/components/schemas/billing_plan_null' budget: $ref: '#/components/schemas/budget_update' + log_level: + $ref: '#/components/schemas/min_log_level_update' description: Specifies the job details to update. get: summary: Full metadata for a batch job @@ -3023,7 +3032,7 @@ paths: Back-ends can log any information that may be relevant for a user at any stage (status) of the batch job. Users can log information during data processing using respective - processes such as `debug`. + processes such as `inspect`. If requested consecutively, it is RECOMMENDED that clients use the offset parameter to get only the entries they have not received yet. @@ -5356,6 +5365,8 @@ components: logs (e.g. `GET /jobs/{job_id}/logs`) with the same schema. allOf: - $ref: '#/components/schemas/usage' + log_level: + $ref: '#/components/schemas/min_log_level_default' job_id: type: string description: >- @@ -5485,6 +5496,8 @@ components: same schema. allOf: - $ref: '#/components/schemas/usage' + log_level: + $ref: '#/components/schemas/min_log_level_default' service_type: description: >- Definition of the service type to access result data. All available @@ -5596,7 +5609,7 @@ components: type: string description: >- The code is either one of the standardized error codes or a custom code, - for example specified by a user in the `debug` process. + for example specified by a user in the `inspect` process. example: SampleError log_links: description: |- @@ -5611,6 +5624,49 @@ components: example: - href: 'https://example.openeo.org/docs/errors/SampleError' rel: about + log_level: + description: |- + The severity level of the log entry. + + The order of the levels is as follows (from low to high severity): `debug`, `info`, `warning`, `error`. + + The level `error` usually corresponds with critical issues that usually terminate the data processing. + type: string + enum: + - error + - warning + - info + - debug + example: error + min_log_level_default: + description: |- + The minimum severity level for log entries that the back-end stores for the processing request. + + The order of the levels is as follows (from low to high severity): `debug`, `info`, `warning`, `error`. + That means if `warning` is set, the back-end will only store log entries with the level `warning` and `error`. + + The default minimum log level is `info`. Users need to specifically set this property to `debug` to get *all* log entries. + It is RECOMMENDED that users set the level at least to "warning" in production workflows. + type: string + enum: + - error + - warning + - info + - debug + default: info + example: warning + min_log_level_update: + description: |- + Updates the minimum severity level for log entries that the back-end stores for the processing requests. + + The back-end doesn't need to update existing log entries. + type: string + enum: + - error + - warning + - info + - debug + example: warning data_type_schema: title: Data Types description: Either a single data type or a list of data types. @@ -5917,19 +5973,7 @@ components: code: $ref: '#/components/schemas/log_code' level: - description: |- - The severity level of the log entry. - - The order of the levels is as follows (from high to low severity): `error`, `warning`, `info`, `debug`. - - The level `error` usually stops processing the data. - type: string - enum: - - error - - warning - - info - - debug - example: error + $ref: '#/components/schemas/log_level' message: type: string description: A message explaining the log entry. @@ -5948,8 +5992,9 @@ components: Data of any type. It is the back-ends task to decide how to best present passed data to a user. - For example, a raster-cube passed to the `debug` SHOULD return the + For example, a raster-cube passed to the `inspect` SHOULD return the metadata similar to the collection metadata, including `cube:dimensions`. + There are implementation guidelines available for the `inspect` process. nullable: true path: description: |- @@ -6059,6 +6104,21 @@ components: - logs - links properties: + log_level: + description: |- + The minimum severity level for log entries that the back-end returns. + This property MUST reflect the `log_level` that has been specified by the user + for the processing request. + + The order of the levels is as follows (from low to high severity): `debug`, `info`, `warning`, `error`. + That means if `warning` is set, the logs will only contain entries with the level `warning` and `error`. + type: string + enum: + - error + - warning + - info + - debug + default: debug logs: type: array items: From 18d638f63033db842ff01f431ed44a3844802a1a Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 8 Dec 2021 16:44:13 +0100 Subject: [PATCH 028/103] Explicitly specify metadata filters for collections #396 (#434) --- CHANGELOG.md | 1 + openapi.yaml | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 87a99c27..1c511f66 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `create-form` to link to the registration page - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) +- `GET /collections` and `GET /collections/{collection_id}` (and implicitly `GET /jobs/{job_id}/results`): Added a field that can indicate which properties can be filtered on when loading data from a collection. [#396](https://github.com/Open-EO/openeo-api/issues/396) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) - `GET /jobs/{job_id}/results`: Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) - `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs` and `POST /result`: Added `log_level` property in responses to reflect the minimum log level that has been chosen by the user. [#329](https://github.com/Open-EO/openeo-api/issues/329) diff --git a/openapi.yaml b/openapi.yaml index 4121cb11..5d2710bb 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1445,6 +1445,10 @@ paths: - B10 - B11 - B12 + 'openeo:property_filters': + - platform + - eo:cloud_cover + - sat:orbit_state summaries: 'constellation': - Sentinel-2 @@ -4355,6 +4359,21 @@ components: x-additionalPropertiesName: Dimension Name allOf: - $ref: '#/components/schemas/dimension' + 'openeo:property_filters': + description: |- + Gives a list of properties you can filter on when loading data from the collection. + Returning an empty array means that no property-based filters are available. + + This is a guide for users to get an overview of available properties to filter on. + More metadata about the properties can be found in `summaries`. + + If this property is not present, it means that the collections does + not provide any collection-specific details about available property-based filters. + It's then up to other means of documentation to inform the user, + e.g. there could be documentation about property filters that work across collections. + type: array + items: + type: string summaries: title: STAC Summaries (Collection Properties) description: |- From 7d7292d172eace415b26869bdcfea8897ecc87d9 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 9 Dec 2021 19:22:30 +0100 Subject: [PATCH 029/103] Support partial/incomplete batch job results #430 (#433) --- CHANGELOG.md | 7 +++++-- openapi.yaml | 48 ++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 45 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1c511f66..c8837792 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,7 +14,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - `GET /collections` and `GET /collections/{collection_id}` (and implicitly `GET /jobs/{job_id}/results`): Added a field that can indicate which properties can be filtered on when loading data from a collection. [#396](https://github.com/Open-EO/openeo-api/issues/396) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) -- `GET /jobs/{job_id}/results`: Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) +- `GET /jobs/{job_id}/results`: + - Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) + - Added metadata field `openeo:status` to indicate the job status (and whether the result is complete or not). + - Added parameter `partial` to allow retrieving incomplete results, which must also add the new property `openeo:status` to the metadata. [#430](https://github.com/Open-EO/openeo-api/issues/430) - `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs` and `POST /result`: Added `log_level` property in responses to reflect the minimum log level that has been chosen by the user. [#329](https://github.com/Open-EO/openeo-api/issues/329) - Added property `log_level` to secondary web service, batch job and synchronous processing endpoints to indicate the minimum severity level that should be stored for logs. [#329](https://github.com/Open-EO/openeo-api/issues/329) - Recommendation to add media types and titles to links for a better user experience. @@ -303,7 +306,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /jobs/{job_id}/estimate` can return the estimated required storage capacity. [#122](https://github.com/Open-EO/openeo-api/issues/122) - `GET /jobs/{job_id}` has two new properties: - `progress` indicates the batch job progress when running. [#82](https://github.com/Open-EO/openeo-api/issues/82) - - `error` states the error message when a job errored out. + - `error` states the error message when a job stopped due to an error. `GET /jobs/{job_id}/result` mirrors this error message in a response with HTTP status code 424. [#165](https://github.com/Open-EO/openeo-api/issues/165) - `GET /.well-known/openeo` allows clients to choose between versions. [#148](https://github.com/Open-EO/openeo-api/issues/148) - `GET /` (Capabilities): diff --git a/openapi.yaml b/openapi.yaml index 5d2710bb..14acd8d0 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3063,12 +3063,12 @@ paths: parameters: - $ref: '#/components/parameters/job_id' get: - summary: Download results for a completed batch job + summary: Download results for a batch job operationId: list-results description: |- - Lists signed URLs pointing to the processed files after the batch job + Lists signed URLs pointing to the processed files, usually after the batch job has finished. Back-ends may also point to intermediate results after the - job has stopped due to an error. + job has stopped due to an error or if the `partial` parameter has been set. The response includes additional metadata. It is a valid [STAC Item](https://github.com/radiantearth/stac-spec/tree/v1.0.0/item-spec) @@ -3111,13 +3111,26 @@ paths: It is **strongly recommended** to add a link with relation type `canonical` to the STAC Item or STAC Collection (see the `links` property for details). - If processing has not finished yet requests to this endpoint MUST be - rejected with openEO error `JobNotFinished`. + If processing has not finished yet and the `partial` parameter is not set to `true` + requests to this endpoint MUST be rejected with openEO error `JobNotFinished`. tags: - Data Processing - Batch Jobs security: - Bearer: [] + parameters: + - name: partial + description: >- + If set to `true`, the results endpoint returns incomplete results while + still running. + + Enabling this parameter requires to indicate the status of the batch job + in the STAC metadata by setting the `openeo:status`. + in: query + allowEmptyValue: true + schema: + type: boolean + default: false responses: '200': description: >- @@ -3131,7 +3144,8 @@ paths: to `0` if the requester is the owner of the job and still has free downloads included in his processing charges estimated by `GET /jobs/{job_id}/estimate`. - If a requester other than the owner is requesting the data of a shared job this header indicates the costs for the requester. + If a requester other than the owner is requesting the data of a + shared job this header indicates the costs for the requester. schema: $ref: '#/components/schemas/money' content: @@ -3143,6 +3157,9 @@ paths: type: object required: - assets + properties: + 'openeo:status': + $ref: '#/components/schemas/result_status' allOf: - $ref: '#/components/schemas/collection' example: @@ -3726,6 +3743,19 @@ components: minItems: 1 items: type: string + result_status: + type: string + enum: + - running + - canceled + - finished + - error + description: |- + The status of a batch job. + + This field is REQUIRED if the `partial` parameter is given. + This field is strongly RECOMMENDED if the job has stopped due to an error. + default: finished batch_job_result: title: Batch Job Results Response as STAC Item type: object @@ -3867,6 +3897,8 @@ components: Time until which the assets are accessible, in UTC. Formatted as a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. example: '2020-11-01T00:00:00Z' + 'openeo:status': + $ref: '#/components/schemas/result_status' assets: $ref: '#/components/schemas/stac_assets' links: @@ -5378,8 +5410,8 @@ components: Metrics about the resource usage of the batch job. Back-ends are not expected to update the metrics while processing data, - so the metrics can only be available after the job has been finished - or has errored. + so the metrics can only be available after the job has finished + or has stopped due to an error. For usage metrics during processing, metrics can better be added to the logs (e.g. `GET /jobs/{job_id}/logs`) with the same schema. allOf: From 4a7ed2636418fb4f20db8de0a4a542e8280d1129 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 15 Dec 2021 17:08:01 +0100 Subject: [PATCH 030/103] STAC: reference_system in cube:dimensions oudated #435 --- CHANGELOG.md | 1 + openapi.yaml | 12 +++++++----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c8837792..c07fcaf4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed - Updated STAC specification examples and references to v1.0.0. +- `cube:dimensions`: `reference_system` is allowed to be PROJJSON, too. - Relaxed requirement that unsupported endpoints must return HTTP status code 501. Instead also HTTP status code 404 can be used (and is regularly used in practice). [#415](https://github.com/Open-EO/openeo-api/issues/415) ### Fixed diff --git a/openapi.yaml b/openapi.yaml index 14acd8d0..d40b7ac1 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -4724,13 +4724,15 @@ components: $ref: '#/components/schemas/collection_dimension_step' reference_system: description: >- - The spatial reference system for the data, specified as [EPSG code](http://www.epsg-registry.org/), [WKT2 (ISO 19162) string](http://docs.opengeospatial.org/is/18-010r7/18-010r7.html) or [PROJ definition (deprecated)](https://proj.org/usage/quickstart.html). Defaults to EPSG code 4326. + The spatial reference system for the data, specified as [EPSG code](http://www.epsg-registry.org/), [WKT2 (ISO 19162) string](http://docs.opengeospatial.org/is/18-010r7/18-010r7.html), [PROJJSON object](https://proj.org/specifications/projjson.html) or [PROJ definition (deprecated)](https://proj.org/usage/quickstart.html). Defaults to EPSG code 4326. + default: 4326 oneOf: - - type: string - description: WKT2 or PROJ definition - type: number - default: 4326 - description: EPSG code + title: EPSG code + - type: string + title: WKT2 or PROJ definition (deprecated) + - type: object + title: PROJJSON discriminator: propertyName: axis mapping: From a46fe242d9f7df07664bfcdb6f9613f34a3645bb Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 15 Dec 2021 17:09:32 +0100 Subject: [PATCH 031/103] Clarify (default) billing plans (#428) * Clarify billing (default) plans #425 #426 * Updates after code review --- CHANGELOG.md | 1 + errors.json | 10 +++++++++ openapi.yaml | 62 ++++++++++++++++++++++++++++------------------------ 3 files changed, 44 insertions(+), 29 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c07fcaf4..12eaf52c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Clarify use of `user_id`. [#404](https://github.com/Open-EO/openeo-api/issues/404) - Clarify that the relation type `version-history` should include `/.well-known/openeo` in the URL. - Clarify that clients should (re-)request capabilities and discovery endpoints with token if available and supported. [#416](https://github.com/Open-EO/openeo-api/issues/416) +- Clarify the fields `plan` (for processing requests) and `billing_plan` (in `GET /` and `GET /me`). [#425](https://github.com/Open-EO/openeo-api/issues/425) [#426](https://github.com/Open-EO/openeo-api/issues/426) - `GET /`: Removed the superfluous default value for `currency`. [#423](https://github.com/Open-EO/openeo-api/issues/423) - Reflect that the `debug` process has been renamed to `inspect`. diff --git a/errors.json b/errors.json index fa4ad5bf..1e2f3239 100644 --- a/errors.json +++ b/errors.json @@ -331,6 +331,16 @@ "Data Processing" ] }, + "BillingPlanMissing": { + "description": "No billing plan has been specified by the user and the billing plan can't be determined unambiguously.", + "message": "A billing plan must be specified.", + "http": 400, + "tags": [ + "Batch Jobs", + "Secondary Services", + "Data Processing" + ] + }, "AuthenticationRequired": { "description": "The client did not provide any authentication details for a resource requiring authentication or the provided authentication details are not correct.", "message": "Unauthorized.", diff --git a/openapi.yaml b/openapi.yaml index d40b7ac1..c4eb00b7 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -740,10 +740,20 @@ paths: example: USD default_plan: type: string - description: >- - Name of the default plan to use when the user doesn't - specify a plan or has no default plan has been assigned - for the user. + description: |- + Name of the plan the back-end uses for billing in case + 1. the user has not subscribed to a specific plan + (see `default_plan` in `GET /me`) and + 2. also did not provide a specific plan with the + processing request. + + If a free plan is available at the back-end, it is + probably most useful to provide this as the back-end + wide default plan and override it with paid plans through + the user-specific default plan in `GET /me`. + Otherwise, users that have not set up payment yet MAY + receive an error for each processing requests where + they did not provide a free plan specifically. example: free plans: description: Array of plans @@ -3526,14 +3536,7 @@ paths: the user's real name or a nickname. default_plan: type: string - nullable: true - description: |- - Name of the plan the user has subscribed to. - - Overrides the default plan of the back-end, but back-ends - MAY also allow overriding this plan for each individual - processing request (e.g. job or service) with the - corresponding `plan` property. + description: Name of the single plan the user is currently subscribed to if any. example: free storage: title: User Storage @@ -4115,7 +4118,7 @@ components: description: |- The billing plan to process and charge the job or service with. - Billing plans MUST be accepted in a *case insensitive* manner. + Billing plans MUST be handled in a *case insensitive* manner. The plans can be retrieved from `GET /`, but the value returned here may not be in the list of plans any longer. @@ -4126,14 +4129,11 @@ components: The billing plan to process and charge the job or service with. Billing plans MUST be accepted in a *case insensitive* manner. - If `null` is specified by the client explicitly, - the server MUST store the default plan and persist it regardless of any - changes to the default billing plan in the future. If given, the default - plan specific to the user has higher priority than the default plan of - the back-end. - - The list of plans and the default plan for the back-end can be retrieved - from `GET /`. The default plan for the user can be retrieved from `GET /me`. + Back-ends MUST resolve the billing plan in the following way if billing is supported: + + * If a value is given and it is not `null`: Persist the `plan` that has been provided in the request. + * Otherwise, don't change the billing plan. + Billing plans not on the list of available plans MUST be rejected with openEO error `BillingPlanInvalid`. example: free @@ -4144,14 +4144,18 @@ components: The billing plan to process and charge the job or service with. Billing plans MUST be accepted in a *case insensitive* manner. - If `null` is given implicitly (through the default value) or explicitly by the client, - the server MUST store the default plan and persist it regardless of any - changes to the default billing plan in the future. If given, the default - plan specific to the user has higher priority than the default plan of - the back-end. - - The list of plans and the default plan for the back-end can be retrieved - from `GET /`. The default plan for the user can be retrieved from `GET /me`. + Back-ends MUST resolve the billing plan in the following way: + + * If a non-`null` value is given: Persist the `plan` that has been provided in the request. + * Otherwise: + 1. Persist the `default_plan` exposed through `GET /me`, if available. + 2. Persist the `default_plan` exposed through `GET /`, if available. + 3. If a single plan is exposed by the back-end, persist it. + 4. Otherwise, the back-end MUST throw a `BillingPlanMissing` error. + + The resolved plan MUST be persisted permanently, regardless of any + changes to the exposed billing plans in `GET /` in the future. + Billing plans not on the list of available plans MUST be rejected with openEO error `BillingPlanInvalid`. example: free From 220e8a38d05116946a67f54599c3eafb17cd7d97 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 11 Jan 2022 16:59:11 +0100 Subject: [PATCH 032/103] Clarify that clients may add additional scopes https://github.com/Open-EO/openeo-aggregator/issues/26 --- CHANGELOG.md | 3 ++- openapi.yaml | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 12eaf52c..0ae00be4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - `GET /collections` and `GET /collections/{collection_id}` (and implicitly `GET /jobs/{job_id}/results`): Added a field that can indicate which properties can be filtered on when loading data from a collection. [#396](https://github.com/Open-EO/openeo-api/issues/396) -- `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) +- `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) - `GET /jobs/{job_id}/results`: - Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) - Added metadata field `openeo:status` to indicate the job status (and whether the result is complete or not). @@ -41,6 +41,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Clarify the fields `plan` (for processing requests) and `billing_plan` (in `GET /` and `GET /me`). [#425](https://github.com/Open-EO/openeo-api/issues/425) [#426](https://github.com/Open-EO/openeo-api/issues/426) - `GET /`: Removed the superfluous default value for `currency`. [#423](https://github.com/Open-EO/openeo-api/issues/423) - Reflect that the `debug` process has been renamed to `inspect`. +- `GET /credentials/oidc`: Clarify that clients may add additional scopes ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index c4eb00b7..8ff6128d 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1863,7 +1863,9 @@ paths: scopes: type: array description: >- - A list of OpenID Connect scopes that the client MUST include when requesting authorization. + A list of OpenID Connect scopes that the client MUST at least include when requesting authorization. + Clients MAY add additional scopes such as the `offline_access` scope to retrieve a refresh token. + If scopes are specified, the list MUST at least contain the `openid` scope. items: type: string From 3b6e5962392fab56589906a210c311dd69f37c99 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 13 Jan 2022 13:29:38 +0100 Subject: [PATCH 033/103] Add development notes --- README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/README.md b/README.md index e8fcb2cb..9b016707 100644 --- a/README.md +++ b/README.md @@ -33,3 +33,15 @@ This repository contains a set of files formally describing the openEO API, each * [openapi.yaml](openapi.yaml) provides the [OpenAPI](https://www.openapis.org/) 3.0 definition of the openEO API. See the table above for human-readable versions of the OpenAPI definition. * [errors.json](errors.json) is a list of potential global error codes and messages, excluding specific exceptions separately available for each process. * The [assets](assets/) folder contains some useful additional files such as examples or schemas. All of these are non-binding additions. The source of truth are the top-level specification files. + +# Development + +The `draft` branch is the latest version and is the one to create Pull Requests against. + +For development some tools can be used: + +1. Install [node and npm](https://nodejs.org) - should run with any recent version +2. Run `npm install` in this folder to install the dependencies +3. Run the linter for the OpenAPI file with `npm test`. This will lint the files and check against some best-practices. It uses `spectral` in the background. +4. To show the files nicely formatted in a web browser, run `npm start`. It starts a server and opens the API specification rendered with ReDoc in a web browser. +5. To create a static HTML page (e.g. for hosting it on GitHub Pages), you can run `npm run build` and it will create a `redoc.html` in this folder. From a886c8f2e190ccd39e5eb14b79a10d75c95426d7 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Fri, 4 Feb 2022 11:08:10 +0100 Subject: [PATCH 034/103] Clarify scalar --- openapi.yaml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 8ff6128d..06cc4aca 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2538,8 +2538,8 @@ paths: description: |- Lists all secondary web services submitted by a user. - It is **strongly RECOMMENDED** to keep the response size small by - omitting all optional non-scalar values from objects in`services` + It is **strongly RECOMMENDED** to keep the response size small by omitting + all optional non-scalar values (i.e. arrays and objects) from objects in `services` (i.e. the `process`, `configuration` and `attributes` properties). To get the full metadata for a secondary web service clients MUST request `GET /services/{service_id}`. @@ -2783,7 +2783,8 @@ paths: Lists all batch jobs submitted by a user. It is **strongly RECOMMENDED** to keep the response size small by - omitting all optional non-scalar values from objects in `jobs` (i.e. the `process` property). + omitting all optional non-scalar values (i.e. arrays and objects) from + objects in `jobs` (i.e. the `process` property). To get the full metadata for a job clients MUST request `GET /jobs/{job_id}`. tags: - Data Processing From 5a048b52f8476a582230e4e376bc9f7790a1fff0 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 7 Feb 2022 16:36:32 +0100 Subject: [PATCH 035/103] Remvoe federation:description --- extensions/federation/README.md | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/extensions/federation/README.md b/extensions/federation/README.md index 20c0a033..af476868 100644 --- a/extensions/federation/README.md +++ b/extensions/federation/README.md @@ -141,16 +141,6 @@ schema: items: type: string description: The ID of a back-end. - 'federation:description': - description: |- - Allows back-ends to provide backend-specific details about this resource in a human-readable way. - - [CommonMark 0.29](http://commonmark.org/) syntax MAY be used for rich text representation. - type: array - minItems: 1 - items: - type: string - description: The ID of a back-end. ``` **Note:** In Collections this should be placed inside `summaries` if users should be able to filter on back-ends in `load_collection`. @@ -187,10 +177,6 @@ schema: "stac_version": "1.0.0", "id": "example", "description": "...", - "federation:description": { - "eodc": "The collection contains data for Austria only.", - "vito": "The collection contains data for Belgium only.", - }, "summaries": { "federation:backends": ["vito", "eodc"], ... From 91d24b8a6675a702bf3a258655c7e317d998b1fb Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 8 Feb 2022 14:12:30 +0100 Subject: [PATCH 036/103] Improvements for federation extension --- CHANGELOG.md | 3 ++- extensions/federation/README.md | 33 +++++++++++++++++++++++++++------ 2 files changed, 29 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ee25c8c..0b6f80a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,12 +8,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- New experimental [Federation Extension](./extensions/federation/README.md) - `GET /jobs/{job_id}/results`: Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) - `GET /`: New Relation types: [#404](https://github.com/Open-EO/openeo-api/issues/404) - `create-form` to link to the registration page - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) -- `GET /credentials/oids`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) +- `GET /credentials/oids`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) - Recommendation to add media types and titles to links for a better user experience. - Allow the relation type `canonical` to be used generally for (shared) resources (e.g. UDPs or batch jobs) without requiring Bearer authentication. [#405](https://github.com/Open-EO/openeo-api/issues/405) - Recommendation for UDF runtime names. [#409](https://github.com/Open-EO/openeo-api/issues/409) diff --git a/extensions/federation/README.md b/extensions/federation/README.md index af476868..a04b9709 100644 --- a/extensions/federation/README.md +++ b/extensions/federation/README.md @@ -92,7 +92,28 @@ schema: ## Lists of resources -Clients will assume that all lists of resources (e.g. collections, processes, jobs, ...) are the combined from all back-ends listed in `GET /`. Federated APIs can expose if any of the back-ends is not available and thus is not part of the response. +Clients will assume that all lists of resources are the a combination of all back-ends listed in `GET /`. +Federated APIs can expose if any of the back-ends is not available and thus is not part of the response. + +Applies to: + +- `GET /collections` +- `GET /processes` +- `GET /file_formats` +- `GET /process_graphs` +- `GET /jobs` +- `GET /jobs/{job_id}` +- `GET /jobs/{job_id}/logs` +- `GET /jobs/{job_id}/results` +- `GET /services` +- `GET /services/{service_id}/logs` + +The following two endpoints are not extensible and as such the additional field can't be added: + +- `GET /udf_runtimes` +- `GET /service_types` + +*This is due to the fact that these two endpoints return an object with a key-value-pair per UDF runtime / service type and adding a key-value-pair for federation purposes would be interpreted as a new UDF runtime or service type by clients.* ### OpenAPI fragment @@ -121,12 +142,10 @@ schema: } ``` -Please note that this does not apply to `GET /udf_runtimes` and `GET /service_types`. These endpoints are not extensible and can only expose the availibility per UDF runtime / service type. -This is due to the fact that these two endpoints return an object with a key-value-pair per UDF runtime / service type and adding a key-value-pair for federation purposes would be interpreted as a new UDF runtime or service type by clients. - ## Resources supported only by a subset of back-ends -Every discoverable resource that is defined as an object and allows to contain additional properties, can list the backends that support or host the exposed resource/functionality (e.g. collections, processes, process parameters, file formats, file format parameters, services, jobs, ...) +Every discoverable resource that is defined as an object and allows to contain additional properties, can list the backends that support or host the exposed resource/functionality. +This can also be embeded deeply into a hierarchical structure, e.g. for process or file format parameters. ```yaml schema: @@ -143,7 +162,9 @@ schema: description: The ID of a back-end. ``` -**Note:** In Collections this should be placed inside `summaries` if users should be able to filter on back-ends in `load_collection`. +**Note:** In Collections this should generally be provided on the top-level of the object. +If users should be able to filter on back-ends in `load_collection`, please provide `federation:backends` as an element in `openeo:property_filters`. +Then, `federation:backends` as defined in the schema above should also be provided in the `summaries`. ### Examples From 818fb9fca631cd09648f949754553ce23fa07153 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 8 Feb 2022 16:21:34 +0100 Subject: [PATCH 037/103] Updates --- extensions/federation/README.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/extensions/federation/README.md b/extensions/federation/README.md index a04b9709..5dcdf32e 100644 --- a/extensions/federation/README.md +++ b/extensions/federation/README.md @@ -103,18 +103,15 @@ Applies to: - `GET /process_graphs` - `GET /jobs` - `GET /jobs/{job_id}` -- `GET /jobs/{job_id}/logs` - `GET /jobs/{job_id}/results` - `GET /services` -- `GET /services/{service_id}/logs` -The following two endpoints are not extensible and as such the additional field can't be added: +The following endpoints define the resources (UDF runtimes / service types) at the top level of their response as key-value pairs. +Consequently, they are not extensible with additional properties for federation purposes. - `GET /udf_runtimes` - `GET /service_types` -*This is due to the fact that these two endpoints return an object with a key-value-pair per UDF runtime / service type and adding a key-value-pair for federation purposes would be interpreted as a new UDF runtime or service type by clients.* - ### OpenAPI fragment ```yaml From 8c24b1fb584fe5880af110e51cc17f02a81a3810 Mon Sep 17 00:00:00 2001 From: Stefaan Lippens Date: Thu, 17 Feb 2022 13:35:31 +0100 Subject: [PATCH 038/103] Update broken links to file format GDAL codes (#439) --- openapi.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 06cc4aca..8c2237ca 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1007,8 +1007,8 @@ paths: **Note**: Format names and parameters MUST be fully aligned with the GDAL codes if available, see [GDAL Raster - Formats](http://www.gdal.org/formats_list.html) and [OGR Vector - Formats](http://www.gdal.org/ogr_formats.html). It is OPTIONAL to + Formats](https://gdal.org/drivers/raster/index.html) and [OGR Vector + Formats](https://gdal.org/drivers/vector/index.html). It is OPTIONAL to support all output format parameters supported by GDAL. Some file formats not available through GDAL may be defined centrally for openEO. Custom file formats or parameters MAY be defined. From 0d6ba69b3577d4df49612aa93583a9555f5a13aa Mon Sep 17 00:00:00 2001 From: Stefaan Lippens Date: Fri, 18 Feb 2022 12:16:12 +0100 Subject: [PATCH 039/103] Update broken links to some GDAL drivers (#440) --- openapi.yaml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 8c2237ca..5cd425c2 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1100,7 +1100,7 @@ paths: maximum: 100 default: 75 links: - - href: 'https://www.gdal.org/frmt_gtiff.html' + - href: 'https://gdal.org/drivers/raster/gtiff.html' rel: about title: GDAL on the GeoTiff file format and storage options GPKG: @@ -1121,10 +1121,10 @@ paths: - '1.2' default: auto links: - - href: 'https://www.gdal.org/drv_geopackage_raster.html' + - href: 'https://gdal.org/drivers/raster/gpkg.html' rel: about title: GDAL on GeoPackage for raster data - - href: 'https://www.gdal.org/drv_geopackage.html' + - href: 'https://gdal.org/drivers/vector/gpkg.html' rel: about title: GDAL on GeoPackage for vector data input: @@ -1142,10 +1142,10 @@ paths: table, this option is not necessary. Otherwise, it is required. links: - - href: 'https://www.gdal.org/drv_geopackage_raster.html' + - href: 'https://gdal.org/drivers/raster/gpkg.html' rel: about title: GDAL on GeoPackage for raster data - - href: 'https://www.gdal.org/drv_geopackage.html' + - href: 'https://gdal.org/drivers/vector/gpkg.html' rel: about title: GDAL on GeoPackage for vector data 4XX: From 2e407f5a5de9c1d684ff9572b0ad1614c4cea7c1 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 1 Mar 2022 15:53:22 +0100 Subject: [PATCH 040/103] Clarify data cube link in stac_extensions --- openapi.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 5cd425c2..93dcae56 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -4391,8 +4391,8 @@ components: * `t` for the dimension of type `temporal` * `bands` for dimensions of type `bands` - This property REQUIRES to add `datacube` to the list - of `stac_extensions`. + This property REQUIRES to add the data cube extension to the list + of `stac_extensions`: `https://stac-extensions.github.io/datacube/v2.1.0/schema.json`. type: object additionalProperties: x-additionalPropertiesName: Dimension Name From 8467193c6a73c58017874e0625bb88fe6c080dfe Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 4 Apr 2022 16:58:35 +0200 Subject: [PATCH 041/103] Clarify case-insensitivity of HTTP headers --- openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi.yaml b/openapi.yaml index 93dcae56..56d6b3e6 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -15,7 +15,7 @@ info: Unless otherwise stated the API works **case sensitive**. - All names SHOULD be written in snake case, i.e. words are separated with one underscore character (`_`) and no spaces, with all letters lower-cased. Example: `hello_world`. This applies particularly to endpoints and JSON property names. HTTP header fields follow their respective casing conventions, e.g. `Content-Type` or `OpenEO-Costs`, despite being case-insensitive according to [RFC 7230](https://www.rfc-editor.org/rfc/rfc7230.html#section-3.2). + All names SHOULD be written in snake case, i.e. words are separated with one underscore character (`_`) and no spaces, with all letters lower-cased. Example: `hello_world`. This applies particularly to endpoints and JSON property names. HTTP header fields are generally case-insensitive according to [RFC 7230](https://www.rfc-editor.org/rfc/rfc7230.html#section-3.2) and in the specification we follow their respective casing conventions, e.g. `Content-Type` or `OpenEO-Costs`, for better readability and consistency. ## HTTP / REST From 53a9c2da3925fe11af0d0b8a7cff55237ea94be0 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 19 Apr 2022 15:33:08 +0200 Subject: [PATCH 042/103] Fix typo for Collection Assets #446 --- openapi.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 56d6b3e6..1f848c37 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3094,10 +3094,10 @@ paths: STAC Collections can either (1) add all assets as collection-level assets or (2) link to STAC Catalogs and STAC Items with signed URLs, which will provide a full - STAC catalog structure a client has to go through. While (2) is overall the better + STAC catalog structure a client has to go through. Option 2 is overall the better architectural choice and allows a fine-grained description of the processed data, - (1) is not compliant with previous versions of the openEO API. - **To maintain backward compatibility, it is REQUIRED for (2) to still copy + but it is not compliant with previous versions of the openEO API. + **To maintain backward compatibility, it is REQUIRED to still copy all assets in the STAC catalog structure into the collection-level assets.** This requirement is planned to be removed in openEO API version 2.0.0. A client can enforce that the server returns a GeoJSON through content negotiation From f55f8d2265985ae0e5ce060f580f8555327fe0e2 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Tue, 3 May 2022 16:40:44 +0200 Subject: [PATCH 043/103] Add documentation for GET /commercial_data/collections/{collection_id} endpoint: information about a commercial data collection. --- openapi.yaml | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 148 insertions(+) diff --git a/openapi.yaml b/openapi.yaml index 45dcf7a2..585aa333 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3497,6 +3497,154 @@ paths: $ref: '#/components/responses/client_error_auth' 5XX: $ref: '#/components/responses/server_error' + '/commercial_data/collections/{collection_id}': + get: + summary: Information about searching and ordering commercial data collection. + operationId: commercial-data-collection-info + description: |- + Lists information about pricing and available parameters for searching and ordering products for a commercial data collection specified by the identifier `collection_id`. + STAC-compliant general information about the collection must be available at `GET collections/{collection_id}`. + tags: + - Commercial data + security: + - {} + - Bearer: [] + responses: + '200': + description: Information about commercial data collection. + content: + application/json: + schema: + title: Commercial Collection Info + type: object + required: + - id + - pricing + - search_parameters + - order_parameters + properties: + id: + type: string + description: A unique identifier for the collection, which MUST match the specified pattern. + pattern: '^[\w\-\.~\/]+$' + provider: + type: string + description: Provider of the data. + pricing: + type: object + description: Information about pricing for searching and/or ordering data. + required: + - searching + - ordering + properties: + searching: + type: object + description: Pricing for searching available products in the collection. Must be explicitly set to `null` if searching is free of charge. + nullable: true + properties: + description: + type: string + description: Human-readable description of pricing. + links: + type: array + description: |- + Additional information about the pricing options. + ordering: + type: object + description: General information about pricing for ordering products. + properties: + description: + type: string + description: Human-readable description of pricing. + links: + type: array + description: |- + Additional information about the pricing options. + search_parameters: + type: object + description: Available parameters that can be used for searching products using `POST /commercial_data/search`. + additionalProperties: + x-additionalPropertiesName: Search Parameter + title: Search Parameter + type: object + required: + - required + - description + properties: + description: + type: string + description: Description of the parameter. + required: + type: boolean + description: Whether or not the parameter must be explicitly set when searching data. + links: + type: array + description: |- + Additional information about the parameter. + order_parameters: + type: object + description: Available parameters that can be used when ordering products using `POST /commercial_data/orders`. + additionalProperties: + x-additionalPropertiesName: Search Parameter + title: Search Parameter + type: object + required: + - required + - description + properties: + description: + type: string + description: Description of the parameter. + required: + type: boolean + description: Whether or not the parameter must be explicitly set when ordering data. + links: + type: array + description: |- + Additional information about the parameter. + example: + id: PLEIADES + provider: AIRBUS + pricing: + searching: null + ordering: + description: Minimum area per order is 0.25 km2. The price is calculated based on a 6-months sliding window. + links: + - title: Airbus Pleiades pricing + rel: related + href: https://www.sentinel-hub.com/pricing/ + search_parameters: + bounds: + description: The request bounding box. Specify with a coordinate pair on two (opposite) vertices of the bounding box rectangle. Coordinates need to be in easting,northing or longitude, latitude, in EPSG:4326. + required: true + links: + - title: Bounds specification + - rel: related + - href: https://api.docs.com/tpdi/airbus#bounds + time_range: + description: A time interval to filter data by acquisition date. It is defined by a start and end date and time, e.g. 2019-01-31T14:00:00+01:00. A date representation without time (e.g. 2019-01-31) will not work. Both the start and end of a time interval are inclusive and can be equal. + required: false + processing_level: + description: 'Enum: SENSOR,ALBUM. When searching, you will receive results from the full catalog as well as the Living Library, which holds images that have cloud cover under 30% and Incidence angle under 40°. If you want to search only Living Library results, you will need to filter using processingLevel. This value could be equal to SENSOR (images which meet Living Library criteria) and ALBUM (images that do not meeting Living Library criteria in terms of incidence angle and cloud cover).' + required: false + max_snow_coverage: + description: 'The maximum allowable snow coverage in percent. Default: 100.' + required: false + max_incidence_angle: + description: 'he maximum allowable incidence angle in degrees. Default: 90.' + required: false + order_parameters: + bounds: + description: The request bounding box. Specify with a coordinate pair on two (opposite) vertices of the bounding box rectangle. Coordinates need to be in easting,northing or longitude, latitude, in EPSG:4326. + required: true + links: + - title: Bounds specification + - rel: related + - href: https://api.docs.com/tpdi/airbus#bounds + 4XX: + $ref: '#/components/responses/client_error_auth' + 5XX: + $ref: '#/components/responses/server_error' components: schemas: udf_runtime: From 2c21fad212499d711920d599f9850721f802e0e1 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Tue, 3 May 2022 18:07:02 +0200 Subject: [PATCH 044/103] Add documentation for GET /commercial_data/search endpoint: searching available products. --- openapi.yaml | 255 ++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 254 insertions(+), 1 deletion(-) diff --git a/openapi.yaml b/openapi.yaml index 585aa333..c3bf0c87 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3505,7 +3505,7 @@ paths: Lists information about pricing and available parameters for searching and ordering products for a commercial data collection specified by the identifier `collection_id`. STAC-compliant general information about the collection must be available at `GET collections/{collection_id}`. tags: - - Commercial data + - Commercial Data security: - {} - Bearer: [] @@ -3645,6 +3645,259 @@ paths: $ref: '#/components/responses/client_error_auth' 5XX: $ref: '#/components/responses/server_error' + /commercial_data/search: + post: + summary: Search for available products in a commercial data collection. + operationId: commercial-data-search + description: |- + Search for products in a commercial data collection. Available filtering parameters are listed at `GET /commercial_data/collections/{collection_id}`. + tags: + - Commercial Data + security: + - Bearer: [] + parameters: + - $ref: '#/components/parameters/pagination_limit' + responses: + '200': + description: Lists of available products and related links. + content: + application/json: + schema: + title: Available Products + type: object + required: + - products + - links + properties: + products: + type: array + description: An array of available products as STAC Items. + title: Item + items: + type: object + required: + - stac_version + - id + - type + - geometry + - properties + - assets + - links + properties: + stac_version: + $ref: '#/components/schemas/stac_version' + stac_extensions: + $ref: '#/components/schemas/stac_extensions' + id: + type: string + description: Unique identifier of a product. + type: + type: string + description: >- + The GeoJSON type that applies to this metadata document, + which MUST always be a "Feature" according to the STAC specification. + + This type does **not** describe the spatial data type of the assets. + enum: + - Feature + bbox: + description: |- + Potential *spatial extent* covered by the data. + + The bounding box is provided as four or six numbers. Six numbers are + specified, if the coordinate reference system includes a vertical axis + (height or depth). The order of the elements in the array: + + - West (lower left corner, coordinate axis 1) + - South (lower left corner, coordinate axis 2) + - Base (optional, lower left corner, coordinate axis 3) + - East (upper right corner, coordinate axis 1) + - North (upper right corner, coordinate axis 2) + - Height (optional, upper right corner, coordinate axis 3) + + The coordinate reference system of the values is WGS84 + longitude/latitude. + + Specifying the `bbox` is strongly RECOMMENDED for STAC compliance, + but can be omitted if the result is unlocated and the `geometry` + is set to `null`. + type: array + oneOf: + - title: 4 elements + minItems: 4 + maxItems: 4 + - title: 6 elements + minItems: 6 + maxItems: 6 + items: + type: number + example: + - -180 + - -90 + - 180 + - 90 + geometry: + description: |- + Defines the full footprint of the asset represented by this item as + GeoJSON Geometry. + + Results without a known location can set this value to `null`. + nullable: true + allOf: + - $ref: '#/components/schemas/GeoJsonGeometry' + example: + type: Polygon + coordinates: + - - - -180 + - -90 + - - 180 + - -90 + - - 180 + - 90 + - - -180 + - 90 + - - -180 + - -90 + properties: + type: object + title: Item Properties + description: >- + MAY contain additional properties other than the required property `datetime`, + e.g. custom properties or properties from the STAC specification or STAC extensions. + required: + - datetime + additionalProperties: true + properties: + datetime: + title: Date and Time + description: >- + The searchable date/time of the data, in UTC. Formatted as a + [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + + + If this field is set to `null` (usually for larger time ranges), + it is STRONGLY RECOMMENDED to specify both `start_datetime` and + `end_datetime` for STAC compliance. + type: string + format: date-time + nullable: true + start_datetime: + type: string + format: date-time + description: >- + For time series: The first or start date and time for the data, + in UTC. Formatted as a [RFC + 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + end_datetime: + type: string + format: date-time + description: >- + For time series: The last or end date and time for the data, in + UTC. Formatted as a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) + date-time. + title: + $ref: '#/components/schemas/eo_title' + description: + $ref: '#/components/schemas/eo_description' + license: + $ref: '#/components/schemas/stac_license' + providers: + $ref: '#/components/schemas/stac_providers' + created: + $ref: '#/components/schemas/created' + updated: + $ref: '#/components/schemas/updated' + expires: + type: string + format: date-time + description: >- + Time until which the assets are accessible, in UTC. Formatted as + a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + example: '2020-11-01T00:00:00Z' + assets: + $ref: '#/components/schemas/stac_assets' + links: + type: array + description: Links related to this product + items: + $ref: '#/components/schemas/link' + links: + $ref: '#/components/schemas/links_pagination' + example: + products: + - type: Feature + stac_version: 0.9.0 + id: c8a1f88d-89cf-4933-9118-45e9c1a5df20 + geometry: + type: Polygon + coordinates: + - + - + - 12.36555287044679 + - 41.94403289260048 + - + - 12.36571746774068 + - 41.86399361096971 + - + - 12.60746759743069 + - 41.86372776276345 + - + - 12.60758647471871 + - 41.94379931812686 + - + - 12.36555287044679 + - 41.94403289260048 + properties: + constellation: PHR + acquisitionDate: "2022-03-21T10:11:15.055Z" + azimuthAngle: 179.9852862071639 + cloudCover: 0 + geometryCentroid: + lat: 41.903935647240964 + lon: 12.486569672582828 + processingLevel: SENSOR + sensorType: OPTICAL + spectralRange: VISIBLE + links: [] + 4XX: + $ref: '#/components/responses/client_error_auth' + 5XX: + $ref: '#/components/responses/server_error' + requestBody: + required: true + description: The base data for the secondary web service to create + content: + application/json: + schema: + title: Search Commercial Products Request + type: object + required: + - collection_id + - parameters + properties: + collection_id: + type: string + description: Unique identifier of the collection. + parameters: + type: object + description: Key-value pairs of available `search_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. + additionalProperties: + x-additionalPropertiesName: Search Parameter Name + AnyValue: + description: Value of the search parameter to be used in the request. + example: + collection_id: "PLEIADES" + parameters: + bounds: + - 3 + - 15 + - 4 + - 16 + time_range: + - "01-01-2022" + - "01-02-2022" + processing_level: SENSOR + max_snow_coverage: 0 components: schemas: udf_runtime: From 33ad759c948fe2501a0edadf75a6696402145768 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Wed, 4 May 2022 10:35:15 +0200 Subject: [PATCH 045/103] Add documentation for POST /commercial_data/orders endpoint: order products. --- openapi.yaml | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 83 insertions(+), 1 deletion(-) diff --git a/openapi.yaml b/openapi.yaml index c3bf0c87..8773c2b5 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3865,7 +3865,7 @@ paths: $ref: '#/components/responses/server_error' requestBody: required: true - description: The base data for the secondary web service to create + description: Data with desired filtering information to search for. content: application/json: schema: @@ -3898,6 +3898,88 @@ paths: - "01-02-2022" processing_level: SENSOR max_snow_coverage: 0 + /commercial_data/orders: + post: + summary: Create an order. + operationId: commercial-data-order + description: |- + Create an order for products of a commercial data collection. The order is NOT yet executed and awaits confirmation. + tags: + - Commercial Data + security: + - Bearer: [] + responses: + '200': + description: Returns unique id and cost of the created order. + content: + application/json: + schema: + title: Order Information + type: object + required: + - id + - status + - costs + properties: + id: + type: string + description: Unique identifier of the order, which MUST match the specified pattern. + pattern: '^[\w\-\.~\/]+$' + status: + type: string + enum: + - NOT_CONFIRMED + costs: + $ref: '#/components/schemas/money' + example: + id: 40264b5-c3ae-46f4-a907-0f612d763d97 + status: NOT_CONFIRMED + costs: 42 + 4XX: + $ref: '#/components/responses/client_error_auth' + 5XX: + $ref: '#/components/responses/server_error' + requestBody: + required: true + description: The base data to create the order + content: + application/json: + schema: + title: Create Order Request + type: object + required: + - collection_id + - products + - parameters + properties: + collection_id: + type: string + description: Unique identifier of the collection. + products: + type: array + description: Array of IDs of products to order. + items: + type: string + description: Unique identifier of the order, which MUST match the specified pattern. + pattern: '^[\w\-\.~\/]+$' + parameters: + type: object + description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. + additionalProperties: + x-additionalPropertiesName: Order Parameter Name + AnyValue: + description: Value of the order parameter to be used in the request. + example: + collection_id: "PLEIADES" + products: + - c8a1f88d-89cf-4933-9118-45e9c1a5df20 + - bc298be8-9381-4ea4-a169-cb9fceda0540 + parameters: + bounds: + - 3 + - 15 + - 4 + - 16 components: schemas: udf_runtime: From 7be50b0fc257a48d248cc84cef27371271ae9f11 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Wed, 4 May 2022 11:32:48 +0200 Subject: [PATCH 046/103] Add documentation for POST /commercial_data/orders/{order_id} endpoint: confirm an order. --- openapi.yaml | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/openapi.yaml b/openapi.yaml index 8773c2b5..99a928e0 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3980,6 +3980,47 @@ paths: - 15 - 4 - 16 + '/commercial_data/orders/{order_id}': + post: + summary: Confirm an order. + operationId: commercial-data-order-confirm + description: |- + Confirm a created commercial data order. If order status was `NOT_CONFIRMED` execution of the order begins. Otherwise this endpoint has no effect. + tags: + - Commercial Data + security: + - Bearer: [] + parameters: + - $ref: '#/components/parameters/pagination_limit' + responses: + '200': + description: Returns unique id and the status of the order. + content: + application/json: + schema: + title: Order Confirmation + type: object + required: + - id + - status + properties: + id: + type: string + description: Unique identifier of the order, which MUST match the specified pattern. + pattern: '^[\w\-\.~\/]+$' + status: + type: string + enum: + - RUNNING + - FAILED + - DONE + example: + id: 40264b5-c3ae-46f4-a907-0f612d763d97 + status: RUNNING + 4XX: + $ref: '#/components/responses/client_error_auth' + 5XX: + $ref: '#/components/responses/server_error' components: schemas: udf_runtime: From 27a806cc44ddb4e8bc1551c37b995464f6569046 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Wed, 4 May 2022 11:33:18 +0200 Subject: [PATCH 047/103] Add documentation for GET /commercial_data/orders endpoint: List all orders. --- openapi.yaml | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) diff --git a/openapi.yaml b/openapi.yaml index 99a928e0..3085ac1b 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3899,6 +3899,85 @@ paths: processing_level: SENSOR max_snow_coverage: 0 /commercial_data/orders: + get: + summary: List all orders. + operationId: commercial-data-orders + description: |- + Lists all orders created by the user. + tags: + - Commercial Data + security: + - Bearer: [] + parameters: + - $ref: '#/components/parameters/pagination_limit' + responses: + '200': + description: Array of order information. + content: + application/json: + schema: + title: All Orders + type: object + required: + - orders + - links + properties: + orders: + type: array + items: + type: object + title: Order + description: Information about an order + required: + - id + - status + - created + - collection_id + - byoc_collection_id + - costs + properties: + id: + type: string + description: Unique identifier of the order, which MUST match the specified pattern. + pattern: '^[\w\-\.~\/]+$' + status: + type: string + enum: + - NOT_CONFIRMED + - RUNNING + - FAILED + - DONE + created: + $ref: '#/components/schemas/created' + collection_id: + type: string + description: Unique identifier of the collection. + byoc_collection_id: + type: string + description: Unique identifier required to access the purchased data. + costs: + $ref: '#/components/schemas/money' + links: + $ref: '#/components/schemas/links_pagination' + example: + orders: + - id: 40264b5-c3ae-46f4-a907-0f612d763d97 + status: DONE + created: '2017-01-01T09:32:12Z' + collection_id: PLEAIDES + byoc_collection_id: 50b230c3-e1b5-431c-a6f7-f779684072bb + costs: 42 + - id: 27ce4888-776e-449d-ae9b-52fa86977554 + status: RUNNING + created: '2022-04-01T15:03:56Z' + collection_id: SPOT + byoc_collection_id: 1efa3541-ca84-4f80-a5d1-ea57cdea36c1 + costs: 10 + links: [] + 4XX: + $ref: '#/components/responses/client_error_auth' + 5XX: + $ref: '#/components/responses/server_error' post: summary: Create an order. operationId: commercial-data-order From 83224f5de19d97299864a687eb946ceb2d0483ab Mon Sep 17 00:00:00 2001 From: jjezersek Date: Wed, 4 May 2022 11:45:27 +0200 Subject: [PATCH 048/103] Add documentation for GET /commercial_data/orders/{order_id} endpoint: get detailed information about an order. --- openapi.yaml | 257 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 257 insertions(+) diff --git a/openapi.yaml b/openapi.yaml index 3085ac1b..46f68237 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -4060,6 +4060,263 @@ paths: - 4 - 16 '/commercial_data/orders/{order_id}': + get: + summary: Get detailed information about an order. + operationId: commercial-data-order-info + description: |- + Detailed information about an order. + tags: + - Commercial Data + security: + - Bearer: [] + responses: + '200': + description: Order information. + content: + application/json: + schema: + title: Order Information + type: object + required: + - id + - status + - created + - collection_id + - products + - parameters + - byoc_collection_id + - costs + properties: + id: + type: string + description: Unique identifier of the order, which MUST match the specified pattern. + pattern: '^[\w\-\.~\/]+$' + status: + type: string + enum: + - NOT_CONFIRMED + - RUNNING + - FAILED + - DONE + created: + $ref: '#/components/schemas/created' + products: + type: array + description: An array of available products as STAC Items. + items: + type: object + required: + - stac_version + - id + - type + - geometry + - properties + - assets + - links + properties: + stac_version: + $ref: '#/components/schemas/stac_version' + stac_extensions: + $ref: '#/components/schemas/stac_extensions' + id: + type: string + description: Unique identifier of a product. + type: + type: string + description: >- + The GeoJSON type that applies to this metadata document, + which MUST always be a "Feature" according to the STAC specification. + + This type does **not** describe the spatial data type of the assets. + enum: + - Feature + bbox: + description: |- + Potential *spatial extent* covered by the data. + + The bounding box is provided as four or six numbers. Six numbers are + specified, if the coordinate reference system includes a vertical axis + (height or depth). The order of the elements in the array: + + - West (lower left corner, coordinate axis 1) + - South (lower left corner, coordinate axis 2) + - Base (optional, lower left corner, coordinate axis 3) + - East (upper right corner, coordinate axis 1) + - North (upper right corner, coordinate axis 2) + - Height (optional, upper right corner, coordinate axis 3) + + The coordinate reference system of the values is WGS84 + longitude/latitude. + + Specifying the `bbox` is strongly RECOMMENDED for STAC compliance, + but can be omitted if the result is unlocated and the `geometry` + is set to `null`. + type: array + oneOf: + - title: 4 elements + minItems: 4 + maxItems: 4 + - title: 6 elements + minItems: 6 + maxItems: 6 + items: + type: number + example: + - -180 + - -90 + - 180 + - 90 + geometry: + description: |- + Defines the full footprint of the asset represented by this item as + GeoJSON Geometry. + + Results without a known location can set this value to `null`. + nullable: true + allOf: + - $ref: '#/components/schemas/GeoJsonGeometry' + example: + type: Polygon + coordinates: + - - - -180 + - -90 + - - 180 + - -90 + - - 180 + - 90 + - - -180 + - 90 + - - -180 + - -90 + properties: + type: object + title: Item Properties + description: >- + MAY contain additional properties other than the required property `datetime`, + e.g. custom properties or properties from the STAC specification or STAC extensions. + required: + - datetime + additionalProperties: true + properties: + datetime: + title: Date and Time + description: >- + The searchable date/time of the data, in UTC. Formatted as a + [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + + + If this field is set to `null` (usually for larger time ranges), + it is STRONGLY RECOMMENDED to specify both `start_datetime` and + `end_datetime` for STAC compliance. + type: string + format: date-time + nullable: true + start_datetime: + type: string + format: date-time + description: >- + For time series: The first or start date and time for the data, + in UTC. Formatted as a [RFC + 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + end_datetime: + type: string + format: date-time + description: >- + For time series: The last or end date and time for the data, in + UTC. Formatted as a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) + date-time. + title: + $ref: '#/components/schemas/eo_title' + description: + $ref: '#/components/schemas/eo_description' + license: + $ref: '#/components/schemas/stac_license' + providers: + $ref: '#/components/schemas/stac_providers' + created: + $ref: '#/components/schemas/created' + updated: + $ref: '#/components/schemas/updated' + expires: + type: string + format: date-time + description: >- + Time until which the assets are accessible, in UTC. Formatted as + a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + example: '2020-11-01T00:00:00Z' + assets: + $ref: '#/components/schemas/stac_assets' + links: + type: array + description: Links related to this product + items: + $ref: '#/components/schemas/link' + parameters: + type: object + description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. + additionalProperties: + x-additionalPropertiesName: Order Parameter Name + AnyValue: + description: Value of the order parameter to be used in the request. + collection_id: + type: string + description: Unique identifier of the collection. + byoc_collection_id: + type: string + description: Unique identifier required to access the purchased data. + costs: + $ref: '#/components/schemas/money' + example: + id: 40264b5-c3ae-46f4-a907-0f612d763d97 + status: DONE + created: '2017-01-01T09:32:12Z' + products: + - type: Feature + stac_version: 0.9.0 + id: c8a1f88d-89cf-4933-9118-45e9c1a5df20 + geometry: + type: Polygon + coordinates: + - + - + - 12.36555287044679 + - 41.94403289260048 + - + - 12.36571746774068 + - 41.86399361096971 + - + - 12.60746759743069 + - 41.86372776276345 + - + - 12.60758647471871 + - 41.94379931812686 + - + - 12.36555287044679 + - 41.94403289260048 + properties: + constellation: PHR + acquisitionDate: "2022-03-21T10:11:15.055Z" + azimuthAngle: 179.9852862071639 + cloudCover: 0 + geometryCentroid: + lat: 41.903935647240964 + lon: 12.486569672582828 + processingLevel: SENSOR + sensorType: OPTICAL + spectralRange: VISIBLE + parameters: + bounds: + - 3 + - 15 + - 4 + - 16 + collection_id: PLEAIDES + byoc_collection_id: 50b230c3-e1b5-431c-a6f7-f779684072bb + costs: 42 + 4XX: + $ref: '#/components/responses/client_error_auth' + 5XX: + $ref: '#/components/responses/server_error' post: summary: Confirm an order. operationId: commercial-data-order-confirm From 697ca665c138d2fa6ea6e25d79aecec7d12bf9d6 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Wed, 4 May 2022 12:12:16 +0200 Subject: [PATCH 049/103] Cleanup, move STAC item to components. --- openapi.yaml | 457 ++++++++++++++++++--------------------------------- 1 file changed, 157 insertions(+), 300 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 46f68237..98edf383 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3499,7 +3499,7 @@ paths: $ref: '#/components/responses/server_error' '/commercial_data/collections/{collection_id}': get: - summary: Information about searching and ordering commercial data collection. + summary: Information about searching and ordering a commercial data collection operationId: commercial-data-collection-info description: |- Lists information about pricing and available parameters for searching and ordering products for a commercial data collection specified by the identifier `collection_id`. @@ -3647,7 +3647,7 @@ paths: $ref: '#/components/responses/server_error' /commercial_data/search: post: - summary: Search for available products in a commercial data collection. + summary: Search for available products in a commercial data collection operationId: commercial-data-search description: |- Search for products in a commercial data collection. Available filtering parameters are listed at `GET /commercial_data/collections/{collection_id}`. @@ -3674,153 +3674,7 @@ paths: description: An array of available products as STAC Items. title: Item items: - type: object - required: - - stac_version - - id - - type - - geometry - - properties - - assets - - links - properties: - stac_version: - $ref: '#/components/schemas/stac_version' - stac_extensions: - $ref: '#/components/schemas/stac_extensions' - id: - type: string - description: Unique identifier of a product. - type: - type: string - description: >- - The GeoJSON type that applies to this metadata document, - which MUST always be a "Feature" according to the STAC specification. - - This type does **not** describe the spatial data type of the assets. - enum: - - Feature - bbox: - description: |- - Potential *spatial extent* covered by the data. - - The bounding box is provided as four or six numbers. Six numbers are - specified, if the coordinate reference system includes a vertical axis - (height or depth). The order of the elements in the array: - - - West (lower left corner, coordinate axis 1) - - South (lower left corner, coordinate axis 2) - - Base (optional, lower left corner, coordinate axis 3) - - East (upper right corner, coordinate axis 1) - - North (upper right corner, coordinate axis 2) - - Height (optional, upper right corner, coordinate axis 3) - - The coordinate reference system of the values is WGS84 - longitude/latitude. - - Specifying the `bbox` is strongly RECOMMENDED for STAC compliance, - but can be omitted if the result is unlocated and the `geometry` - is set to `null`. - type: array - oneOf: - - title: 4 elements - minItems: 4 - maxItems: 4 - - title: 6 elements - minItems: 6 - maxItems: 6 - items: - type: number - example: - - -180 - - -90 - - 180 - - 90 - geometry: - description: |- - Defines the full footprint of the asset represented by this item as - GeoJSON Geometry. - - Results without a known location can set this value to `null`. - nullable: true - allOf: - - $ref: '#/components/schemas/GeoJsonGeometry' - example: - type: Polygon - coordinates: - - - - -180 - - -90 - - - 180 - - -90 - - - 180 - - 90 - - - -180 - - 90 - - - -180 - - -90 - properties: - type: object - title: Item Properties - description: >- - MAY contain additional properties other than the required property `datetime`, - e.g. custom properties or properties from the STAC specification or STAC extensions. - required: - - datetime - additionalProperties: true - properties: - datetime: - title: Date and Time - description: >- - The searchable date/time of the data, in UTC. Formatted as a - [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - - - If this field is set to `null` (usually for larger time ranges), - it is STRONGLY RECOMMENDED to specify both `start_datetime` and - `end_datetime` for STAC compliance. - type: string - format: date-time - nullable: true - start_datetime: - type: string - format: date-time - description: >- - For time series: The first or start date and time for the data, - in UTC. Formatted as a [RFC - 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - end_datetime: - type: string - format: date-time - description: >- - For time series: The last or end date and time for the data, in - UTC. Formatted as a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) - date-time. - title: - $ref: '#/components/schemas/eo_title' - description: - $ref: '#/components/schemas/eo_description' - license: - $ref: '#/components/schemas/stac_license' - providers: - $ref: '#/components/schemas/stac_providers' - created: - $ref: '#/components/schemas/created' - updated: - $ref: '#/components/schemas/updated' - expires: - type: string - format: date-time - description: >- - Time until which the assets are accessible, in UTC. Formatted as - a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - example: '2020-11-01T00:00:00Z' - assets: - $ref: '#/components/schemas/stac_assets' - links: - type: array - description: Links related to this product - items: - $ref: '#/components/schemas/link' + $ref: '#/components/schemas/product' links: $ref: '#/components/schemas/links_pagination' example: @@ -3900,7 +3754,7 @@ paths: max_snow_coverage: 0 /commercial_data/orders: get: - summary: List all orders. + summary: List all orders operationId: commercial-data-orders description: |- Lists all orders created by the user. @@ -3979,7 +3833,7 @@ paths: 5XX: $ref: '#/components/responses/server_error' post: - summary: Create an order. + summary: Create an order operationId: commercial-data-order description: |- Create an order for products of a commercial data collection. The order is NOT yet executed and awaits confirmation. @@ -4061,7 +3915,7 @@ paths: - 16 '/commercial_data/orders/{order_id}': get: - summary: Get detailed information about an order. + summary: Get detailed information about an order operationId: commercial-data-order-info description: |- Detailed information about an order. @@ -4104,153 +3958,7 @@ paths: type: array description: An array of available products as STAC Items. items: - type: object - required: - - stac_version - - id - - type - - geometry - - properties - - assets - - links - properties: - stac_version: - $ref: '#/components/schemas/stac_version' - stac_extensions: - $ref: '#/components/schemas/stac_extensions' - id: - type: string - description: Unique identifier of a product. - type: - type: string - description: >- - The GeoJSON type that applies to this metadata document, - which MUST always be a "Feature" according to the STAC specification. - - This type does **not** describe the spatial data type of the assets. - enum: - - Feature - bbox: - description: |- - Potential *spatial extent* covered by the data. - - The bounding box is provided as four or six numbers. Six numbers are - specified, if the coordinate reference system includes a vertical axis - (height or depth). The order of the elements in the array: - - - West (lower left corner, coordinate axis 1) - - South (lower left corner, coordinate axis 2) - - Base (optional, lower left corner, coordinate axis 3) - - East (upper right corner, coordinate axis 1) - - North (upper right corner, coordinate axis 2) - - Height (optional, upper right corner, coordinate axis 3) - - The coordinate reference system of the values is WGS84 - longitude/latitude. - - Specifying the `bbox` is strongly RECOMMENDED for STAC compliance, - but can be omitted if the result is unlocated and the `geometry` - is set to `null`. - type: array - oneOf: - - title: 4 elements - minItems: 4 - maxItems: 4 - - title: 6 elements - minItems: 6 - maxItems: 6 - items: - type: number - example: - - -180 - - -90 - - 180 - - 90 - geometry: - description: |- - Defines the full footprint of the asset represented by this item as - GeoJSON Geometry. - - Results without a known location can set this value to `null`. - nullable: true - allOf: - - $ref: '#/components/schemas/GeoJsonGeometry' - example: - type: Polygon - coordinates: - - - - -180 - - -90 - - - 180 - - -90 - - - 180 - - 90 - - - -180 - - 90 - - - -180 - - -90 - properties: - type: object - title: Item Properties - description: >- - MAY contain additional properties other than the required property `datetime`, - e.g. custom properties or properties from the STAC specification or STAC extensions. - required: - - datetime - additionalProperties: true - properties: - datetime: - title: Date and Time - description: >- - The searchable date/time of the data, in UTC. Formatted as a - [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - - - If this field is set to `null` (usually for larger time ranges), - it is STRONGLY RECOMMENDED to specify both `start_datetime` and - `end_datetime` for STAC compliance. - type: string - format: date-time - nullable: true - start_datetime: - type: string - format: date-time - description: >- - For time series: The first or start date and time for the data, - in UTC. Formatted as a [RFC - 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - end_datetime: - type: string - format: date-time - description: >- - For time series: The last or end date and time for the data, in - UTC. Formatted as a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) - date-time. - title: - $ref: '#/components/schemas/eo_title' - description: - $ref: '#/components/schemas/eo_description' - license: - $ref: '#/components/schemas/stac_license' - providers: - $ref: '#/components/schemas/stac_providers' - created: - $ref: '#/components/schemas/created' - updated: - $ref: '#/components/schemas/updated' - expires: - type: string - format: date-time - description: >- - Time until which the assets are accessible, in UTC. Formatted as - a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - example: '2020-11-01T00:00:00Z' - assets: - $ref: '#/components/schemas/stac_assets' - links: - type: array - description: Links related to this product - items: - $ref: '#/components/schemas/link' + $ref: '#/components/schemas/product' parameters: type: object description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. @@ -4318,7 +4026,7 @@ paths: 5XX: $ref: '#/components/responses/server_error' post: - summary: Confirm an order. + summary: Confirm an order operationId: commercial-data-order-confirm description: |- Confirm a created commercial data order. If order status was `NOT_CONFIRMED` execution of the order begins. Otherwise this endpoint has no effect. @@ -4359,6 +4067,155 @@ paths: $ref: '#/components/responses/server_error' components: schemas: + product: + type: object + title: Item + required: + - stac_version + - id + - type + - geometry + - properties + - assets + - links + properties: + stac_version: + $ref: '#/components/schemas/stac_version' + stac_extensions: + $ref: '#/components/schemas/stac_extensions' + id: + type: string + description: Unique identifier of a product. + type: + type: string + description: >- + The GeoJSON type that applies to this metadata document, + which MUST always be a "Feature" according to the STAC specification. + + This type does **not** describe the spatial data type of the assets. + enum: + - Feature + bbox: + description: |- + Potential *spatial extent* covered by the data. + + The bounding box is provided as four or six numbers. Six numbers are + specified, if the coordinate reference system includes a vertical axis + (height or depth). The order of the elements in the array: + + - West (lower left corner, coordinate axis 1) + - South (lower left corner, coordinate axis 2) + - Base (optional, lower left corner, coordinate axis 3) + - East (upper right corner, coordinate axis 1) + - North (upper right corner, coordinate axis 2) + - Height (optional, upper right corner, coordinate axis 3) + + The coordinate reference system of the values is WGS84 + longitude/latitude. + + Specifying the `bbox` is strongly RECOMMENDED for STAC compliance, + but can be omitted if the result is unlocated and the `geometry` + is set to `null`. + type: array + oneOf: + - title: 4 elements + minItems: 4 + maxItems: 4 + - title: 6 elements + minItems: 6 + maxItems: 6 + items: + type: number + example: + - -180 + - -90 + - 180 + - 90 + geometry: + description: |- + Defines the full footprint of the asset represented by this item as + GeoJSON Geometry. + + Results without a known location can set this value to `null`. + nullable: true + allOf: + - $ref: '#/components/schemas/GeoJsonGeometry' + example: + type: Polygon + coordinates: + - - - -180 + - -90 + - - 180 + - -90 + - - 180 + - 90 + - - -180 + - 90 + - - -180 + - -90 + properties: + type: object + title: Item Properties + description: >- + MAY contain additional properties other than the required property `datetime`, + e.g. custom properties or properties from the STAC specification or STAC extensions. + required: + - datetime + additionalProperties: true + properties: + datetime: + title: Date and Time + description: >- + The searchable date/time of the data, in UTC. Formatted as a + [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + + + If this field is set to `null` (usually for larger time ranges), + it is STRONGLY RECOMMENDED to specify both `start_datetime` and + `end_datetime` for STAC compliance. + type: string + format: date-time + nullable: true + start_datetime: + type: string + format: date-time + description: >- + For time series: The first or start date and time for the data, + in UTC. Formatted as a [RFC + 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + end_datetime: + type: string + format: date-time + description: >- + For time series: The last or end date and time for the data, in + UTC. Formatted as a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) + date-time. + title: + $ref: '#/components/schemas/eo_title' + description: + $ref: '#/components/schemas/eo_description' + license: + $ref: '#/components/schemas/stac_license' + providers: + $ref: '#/components/schemas/stac_providers' + created: + $ref: '#/components/schemas/created' + updated: + $ref: '#/components/schemas/updated' + expires: + type: string + format: date-time + description: >- + Time until which the assets are accessible, in UTC. Formatted as + a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + example: '2020-11-01T00:00:00Z' + assets: + $ref: '#/components/schemas/stac_assets' + links: + type: array + description: Links related to this product + items: + $ref: '#/components/schemas/link' udf_runtime: type: object required: From ccd3b854debdbb7d506ce8e97a28a77d83a29135 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Wed, 4 May 2022 14:10:23 +0200 Subject: [PATCH 050/103] Add a brief description of commercial data. --- openapi.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/openapi.yaml b/openapi.yaml index 98edf383..8bfc273a 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -520,6 +520,9 @@ tags: description: On-demand access to data using other web service protocols. - name: File Storage description: Management of user-uploaded assets and processed data. + - name: Commercial Data + description: |- + These endpoints provide utilies for purchasing commercial data. Commercial data is EO data that is not accessible free of charge. Processing costs come atop of data costs. servers: - url: 'https://localhost/api/{version}' description: >- From 2dfc979c1bd1ddc2525bc9b2eeae396ae25fa4e9 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Wed, 4 May 2022 16:24:01 +0200 Subject: [PATCH 051/103] Fix issues with request parameters not being defined and missing required fields in examples. --- openapi.yaml | 58 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 36 insertions(+), 22 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 8bfc273a..0aeae529 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3512,6 +3512,13 @@ paths: security: - {} - Bearer: [] + parameters: + - name: collection_id + in: path + description: Collection identifier + required: true + schema: + $ref: '#/components/schemas/collection_id' responses: '200': description: Information about commercial data collection. @@ -3715,6 +3722,8 @@ paths: processingLevel: SENSOR sensorType: OPTICAL spectralRange: VISIBLE + assets: {} + links: [] links: [] 4XX: $ref: '#/components/responses/client_error_auth' @@ -3740,7 +3749,6 @@ paths: description: Key-value pairs of available `search_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. additionalProperties: x-additionalPropertiesName: Search Parameter Name - AnyValue: description: Value of the search parameter to be used in the request. example: collection_id: "PLEIADES" @@ -3794,9 +3802,7 @@ paths: - costs properties: id: - type: string - description: Unique identifier of the order, which MUST match the specified pattern. - pattern: '^[\w\-\.~\/]+$' + $ref: '#/components/schemas/order_id' status: type: string enum: @@ -3858,9 +3864,7 @@ paths: - costs properties: id: - type: string - description: Unique identifier of the order, which MUST match the specified pattern. - pattern: '^[\w\-\.~\/]+$' + $ref: '#/components/schemas/order_id' status: type: string enum: @@ -3895,15 +3899,12 @@ paths: type: array description: Array of IDs of products to order. items: - type: string - description: Unique identifier of the order, which MUST match the specified pattern. - pattern: '^[\w\-\.~\/]+$' + $ref: '#/components/schemas/order_id' parameters: type: object description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. additionalProperties: x-additionalPropertiesName: Order Parameter Name - AnyValue: description: Value of the order parameter to be used in the request. example: collection_id: "PLEIADES" @@ -3926,6 +3927,13 @@ paths: - Commercial Data security: - Bearer: [] + parameters: + - name: order_id + in: path + description: Order identifier + required: true + schema: + $ref: '#/components/schemas/order_id' responses: '200': description: Order information. @@ -3945,9 +3953,7 @@ paths: - costs properties: id: - type: string - description: Unique identifier of the order, which MUST match the specified pattern. - pattern: '^[\w\-\.~\/]+$' + $ref: '#/components/schemas/order_id' status: type: string enum: @@ -3967,7 +3973,6 @@ paths: description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. additionalProperties: x-additionalPropertiesName: Order Parameter Name - AnyValue: description: Value of the order parameter to be used in the request. collection_id: type: string @@ -4015,6 +4020,8 @@ paths: processingLevel: SENSOR sensorType: OPTICAL spectralRange: VISIBLE + assets: {} + links: [] parameters: bounds: - 3 @@ -4039,6 +4046,12 @@ paths: - Bearer: [] parameters: - $ref: '#/components/parameters/pagination_limit' + - name: order_id + in: path + description: Order identifier + required: true + schema: + $ref: '#/components/schemas/order_id' responses: '200': description: Returns unique id and the status of the order. @@ -4052,9 +4065,7 @@ paths: - status properties: id: - type: string - description: Unique identifier of the order, which MUST match the specified pattern. - pattern: '^[\w\-\.~\/]+$' + $ref: '#/components/schemas/collection_id' status: type: string enum: @@ -4070,6 +4081,12 @@ paths: $ref: '#/components/responses/server_error' components: schemas: + order_id: + type: string + description: >- + Unique identifier of the order, which MUST match the specified pattern.. + pattern: '^[\w\-\.~]+$' + example: 40264b5-c3ae-46f4-a907-0f612d763d97 product: type: object title: Item @@ -4160,10 +4177,7 @@ components: type: object title: Item Properties description: >- - MAY contain additional properties other than the required property `datetime`, - e.g. custom properties or properties from the STAC specification or STAC extensions. - required: - - datetime + MAY contain custom properties or properties from the STAC specification or STAC extensions. additionalProperties: true properties: datetime: From e618e101392a9ba5fce5662fe70431d6969e8dcf Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 2 Jun 2022 12:28:59 +0200 Subject: [PATCH 052/103] Small clarification for the endpoints in GET / --- openapi.yaml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 1f848c37..bb2458f6 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -653,11 +653,12 @@ paths: type: array description: >- Lists all supported endpoints. Supported are all - endpoints, which are implemented, return a 2XX or 3XX HTTP - status code and are fully compatible to the API - specification. + endpoints, which are implemented, return usually + a 2XX or 3XX HTTP status code and are fully compatible + to the API specification. An entry for this endpoint (path `/` with method `GET`) SHOULD NOT be listed. + Each path MUST only be listed once in the array. items: title: Endpoint type: object From 2d9b4e596c09954a2d1335cad69716facbf89eb4 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Mon, 20 Jun 2022 14:04:19 +0200 Subject: [PATCH 053/103] Change provider to use STAC Provider field. --- openapi.yaml | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 0aeae529..18467c9c 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3538,8 +3538,36 @@ paths: description: A unique identifier for the collection, which MUST match the specified pattern. pattern: '^[\w\-\.~\/]+$' provider: - type: string - description: Provider of the data. + type: object + title: Commercial data provider + required: + - name + properties: + name: + description: The id of the commercial data provider. + type: string + description: + description: >- + Multi-line description to add further provider information such as + processing details, hosting details or basic contact information. + + + CommonMark 0.29 syntax MAY be used for rich text representation. + type: string + roles: + description: >- + Role of the provider. Must be `producer`. + type: array + items: + type: string + enum: + - producer + url: + description: >- + Homepage on which the provider describes the dataset and publishes + contact information. + type: string + format: uri pricing: type: object description: Information about pricing for searching and/or ordering data. @@ -3614,7 +3642,10 @@ paths: Additional information about the parameter. example: id: PLEIADES - provider: AIRBUS + provider: + name: AIRBUS + roles: + - producer pricing: searching: null ordering: From 8c37ad4cac6b9d1ce908e9d4b76df3ac6f430a62 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Mon, 20 Jun 2022 16:05:33 +0200 Subject: [PATCH 054/103] Align Commercial data order with STAC extension 'order'. --- openapi.yaml | 108 ++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 77 insertions(+), 31 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 18467c9c..9faa040e 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3825,24 +3825,47 @@ paths: title: Order description: Information about an order required: - - id - - status - - created + - order:id + - order:status + - order:date - collection_id - byoc_collection_id - costs properties: - id: + order:id: $ref: '#/components/schemas/order_id' - status: + order:status: type: string + description: |- + The status of the order. + * `orderable`: The item or asset is orderable via the provider scenario. + + * `ordered`: The item or asset is ordered and the provider is preparing to make it available. + + * `shipping`: The item or asset order are being processed by the provider to provide the user with the asset(s). + + * `delivered`: The provider has delivered the order and asset(s) are available. + + * `unable_to_deliver`: The provider is not able to deliver the order. + + * `canceled`: The order has been canceled. enum: - - NOT_CONFIRMED - - RUNNING - - FAILED - - DONE - created: - $ref: '#/components/schemas/created' + - orderable + - ordered + - shipping + - delivered + - unable_to_deliver + - canceled + order:date: + type: string + format: date-time + description: The order time. Formatted as a + [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + order:expiration_date: + type: string + format: date-time + description: The validity time of the order. Formatted as a + [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. collection_id: type: string description: Unique identifier of the collection. @@ -3855,15 +3878,15 @@ paths: $ref: '#/components/schemas/links_pagination' example: orders: - - id: 40264b5-c3ae-46f4-a907-0f612d763d97 - status: DONE - created: '2017-01-01T09:32:12Z' + - order:id: 40264b5-c3ae-46f4-a907-0f612d763d97 + order:status: delivered + order:date: '2017-01-01T09:32:12Z' collection_id: PLEAIDES byoc_collection_id: 50b230c3-e1b5-431c-a6f7-f779684072bb costs: 42 - - id: 27ce4888-776e-449d-ae9b-52fa86977554 - status: RUNNING - created: '2022-04-01T15:03:56Z' + - order:id: 27ce4888-776e-449d-ae9b-52fa86977554 + order:status: shippping + order:date: '2022-04-01T15:03:56Z' collection_id: SPOT byoc_collection_id: 1efa3541-ca84-4f80-a5d1-ea57cdea36c1 costs: 10 @@ -3974,26 +3997,49 @@ paths: title: Order Information type: object required: - - id - - status - - created + - order:id + - order:status + - order:date - collection_id - products - parameters - byoc_collection_id - costs properties: - id: + order:id: $ref: '#/components/schemas/order_id' - status: + order:status: type: string + description: |- + The status of the order. + * `orderable`: The item or asset is orderable via the provider scenario. + + * `ordered`: The item or asset is ordered and the provider is preparing to make it available. + + * `shipping`: The item or asset order are being processed by the provider to provide the user with the asset(s). + + * `delivered`: The provider has delivered the order and asset(s) are available. + + * `unable_to_deliver`: The provider is not able to deliver the order. + + * `canceled`: The order has been canceled. enum: - - NOT_CONFIRMED - - RUNNING - - FAILED - - DONE - created: - $ref: '#/components/schemas/created' + - orderable + - ordered + - shipping + - delivered + - unable_to_deliver + - canceled + order:date: + type: string + format: date-time + description: The order time. Formatted as a + [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + order:expiration_date: + type: string + format: date-time + description: The validity time of the order. Formatted as a + [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. products: type: array description: An array of available products as STAC Items. @@ -4014,9 +4060,9 @@ paths: costs: $ref: '#/components/schemas/money' example: - id: 40264b5-c3ae-46f4-a907-0f612d763d97 - status: DONE - created: '2017-01-01T09:32:12Z' + order:id: 40264b5-c3ae-46f4-a907-0f612d763d97 + order:status: delivered + order:date: '2017-01-01T09:32:12Z' products: - type: Feature stac_version: 0.9.0 From 8d7f36407c5a5bb476db424dafba37c24c765f11 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Tue, 21 Jun 2022 15:43:24 +0200 Subject: [PATCH 055/103] Align searching commercial data products with STAC /search. --- openapi.yaml | 206 +++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 190 insertions(+), 16 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 9faa040e..3c708f91 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3687,7 +3687,7 @@ paths: 5XX: $ref: '#/components/responses/server_error' /commercial_data/search: - post: + get: summary: Search for available products in a commercial data collection operationId: commercial-data-search description: |- @@ -3707,10 +3707,15 @@ paths: title: Available Products type: object required: - - products + - type + - features - links properties: - products: + type: + type: string + enum: + - FeatureCollection + features: type: array description: An array of available products as STAC Items. title: Item @@ -3719,7 +3724,8 @@ paths: links: $ref: '#/components/schemas/links_pagination' example: - products: + type: FeatureCollection + features: - type: Feature stac_version: 0.9.0 id: c8a1f88d-89cf-4933-9118-45e9c1a5df20 @@ -3768,19 +3774,26 @@ paths: schema: title: Search Commercial Products Request type: object - required: - - collection_id - - parameters properties: - collection_id: - type: string - description: Unique identifier of the collection. - parameters: - type: object - description: Key-value pairs of available `search_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. - additionalProperties: - x-additionalPropertiesName: Search Parameter Name - description: Value of the search parameter to be used in the request. + bbox: + $ref: '#/components/schemas/bbox' + datetime: + $ref: '#/components/schemas/datetime' + intersects: + $ref: '#/components/schemas/intersects' + limit: + $ref: '#/components/schemas/limit' + collectionsArray: + $ref: '#/components/schemas/collectionsArray' + filter: + description: |- + **Extension:** Filter + A CQL2 filter expression for filtering items. + + The names and types of the properties or keys that may be used to construct a filter expression for a collection must accessible at `GET commercial_data/{collectionId}/queryables`. + oneOf: + - $ref: '#/components/schemas/filter-cql2-json' + - $ref: '#/components/schemas/filter-cql2-text' example: collection_id: "PLEIADES" parameters: @@ -4158,6 +4171,167 @@ paths: $ref: '#/components/responses/server_error' components: schemas: + intersects: + type: object + description: Only returns items that intersect with the provided polygon. + properties: + intersects: + $ref: https://geojson.org/schema/Geometry.json + filter-cql2-text: + description: | + A CQL2 filter expression in the 'cql2-text' encoding. + type: string + filter-cql2-json: + $ref: 'https://raw.githubusercontent.com/opengeospatial/ogcapi-features/master/cql2/standard/schema/cql2.yml#/components/schemas/booleanExpression' + bbox: + name: bbox + in: query + description: >- + Only features that have a geometry that intersects the bounding box are + selected. + + The bounding box is provided as four or six numbers, depending on + + whether the coordinate reference system includes a vertical axis (height + + or depth): + + + * Lower left corner, coordinate axis 1 + + * Lower left corner, coordinate axis 2 + + * Minimum value, coordinate axis 3 (optional) + + * Upper right corner, coordinate axis 1 + + * Upper right corner, coordinate axis 2 + + * Maximum value, coordinate axis 3 (optional) + + + The coordinate reference system of the values is WGS 84 + + longitude/latitude (http://www.opengis.net/def/crs/OGC/1.3/CRS84). + + + For WGS 84 longitude/latitude the values are in most cases the sequence + + of minimum longitude, minimum latitude, maximum longitude and maximum + + latitude. However, in cases where the box spans the antimeridian the + + first value (west-most box edge) is larger than the third value + + (east-most box edge). + + + If the vertical axis is included, the third and the sixth number are + + the bottom and the top of the 3-dimensional bounding box. + + + If a feature has multiple spatial geometry properties, it is the + + decision of the server whether only a single spatial geometry property + + is used to determine the extent or all relevant geometries. + + + Example: The bounding box of the New Zealand Exclusive Economic Zone in + + WGS 84 (from 160.6°E to 170°W and from 55.95°S to 25.89°S) would be + + represented in JSON as `[160.6, -55.95, -170, -25.89]` and in a query as + + `bbox=160.6,-55.95,-170,-25.89`. + required: false + schema: + type: array + minItems: 4 + maxItems: 6 + items: + type: number + style: form + explode: false + collectionId: + name: collectionId + in: path + description: local identifier of a collection + required: true + schema: + type: string + datetime: + name: datetime + in: query + description: >- + Either a date-time or an interval, open or closed. Date and time + expressions + + adhere to RFC 3339. Open intervals are expressed using double-dots. + + + Examples: + + + * A date-time: "2018-02-12T23:20:50Z" + + * A closed interval: "2018-02-12T00:00:00Z/2018-03-18T12:31:12Z" + + * Open intervals: "2018-02-12T00:00:00Z/.." or "../2018-03-18T12:31:12Z" + + + Only features that have a temporal property that intersects the value of + + `datetime` are selected. + + + If a feature has multiple temporal properties, it is the decision of the + + server whether only a single temporal property is used to determine + + the extent or all relevant temporal properties. + required: false + schema: + type: string + style: form + explode: false + limit: + name: limit + in: query + description: >- + The optional limit parameter limits the number of items that are + presented in the response document. + + + Only items are counted that are on the first level of the collection in + the response document. + + Nested objects contained within the explicitly requested items shall not + be counted. + + + Minimum = 1. Maximum = 100. Default = 10. + required: false + schema: + type: integer + minimum: 1 + maximum: 100 + default: 10 + style: form + explode: false + collectionsArray: + name: collections + description: | + Array with the id of the collection to search products for. + required: false + schema: + type: array + items: + type: string + explode: false + minItems: 1 + maxItems: 1 order_id: type: string description: >- From 5bd050fd0528b140e1d130b0bb227789b958fa0e Mon Sep 17 00:00:00 2001 From: jjezersek Date: Tue, 21 Jun 2022 19:15:25 +0200 Subject: [PATCH 056/103] Add /commercial_data/{collection_id}/queryables endpoint. Fix request payload example in commercial_data/search. --- openapi.yaml | 159 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 115 insertions(+), 44 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index 3c708f91..b2b3adfa 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3500,12 +3500,100 @@ paths: $ref: '#/components/responses/client_error_auth' 5XX: $ref: '#/components/responses/server_error' + '/commercial_data/{collection_id}/queryables': + get: + summary: Queryables for searching data products of a commercial data collection. + operationId: commercial-data-queryables + description: The names and types of the properties or keys that may be used to construct a filter expression for a commercial data collection at `GET /commercial_data/search`. + tags: + - Commercial Data + security: + - {} + - Bearer: [] + responses: + '200': + description: Returns the JSON Schema document with queryables for a commercial data collection. + content: + application/json: + schema: + title: Order Information + type: object + required: + - $schema + - $id + - type + - properties + properties: + $schema: + type: string + enum: + - http://json-schema.org/draft-07/schema# + - https://json-schema.org/draft/2019-09/schema + $id: + type: string + description: The URI of the resource without query parameters. + type: + type: string + enum: + - object + title: + type: string + description: Human-readable title. + description: + type: string + description: Human-readable description. + properties: + type: object + description: Object with available properties that can be evaluated in the CQL filter expression. + additionalProperties: + x-additionalPropertiesName: Property Name + description: JSON schema of the property. + type: object + + example: + $schema: http://json-schema.org/draft-07/schema# + $id: http://example.com/commercial_data/PLEIADES/queryables + type: object + title: Queryables of Airbus Pleiades collection. + description: Available properties for CQL filtering of products. + properties: + processing_level: + title: Processing level + description: When searching, you will receive results from the full catalog as well as the Living Library, which holds images that have cloud cover under 30% and Incidence angle under 40°. If you want to search only Living Library results, you will need to filter using processingLevel. This value could be equal to SENSOR (images which meet Living Library criteria) and ALBUM (images that do not meeting Living Library criteria in terms of incidence angle and cloud cover).' + type: string + enum: + - SENSOR + - ALBUM + max_snow_coverage: + title: Maximum snow coverage + description: The maximum allowable snow coverage in percent. + type: number + minimum: 0 + maximum: 100 + default: 100 + max_incidence_angle: + title: Maximum incidence angle + description: The maximum allowable incidence angle in degrees. + minimum: 0 + maximum: 90 + default: 90 + 4XX: + $ref: '#/components/responses/client_error_auth' + 5XX: + $ref: '#/components/responses/server_error' + parameters: + - name: collection_id + in: path + description: Collection identifier + required: true + schema: + $ref: '#/components/schemas/collection_id' '/commercial_data/collections/{collection_id}': get: - summary: Information about searching and ordering a commercial data collection + summary: Information ordering a commercial data collection operationId: commercial-data-collection-info description: |- - Lists information about pricing and available parameters for searching and ordering products for a commercial data collection specified by the identifier `collection_id`. + Lists information about pricing and available parameters for ordering products for a commercial data collection specified by the identifier `collection_id`. STAC-compliant general information about the collection must be available at `GET collections/{collection_id}`. tags: - Commercial Data @@ -3530,7 +3618,6 @@ paths: required: - id - pricing - - search_parameters - order_parameters properties: id: @@ -3598,27 +3685,6 @@ paths: type: array description: |- Additional information about the pricing options. - search_parameters: - type: object - description: Available parameters that can be used for searching products using `POST /commercial_data/search`. - additionalProperties: - x-additionalPropertiesName: Search Parameter - title: Search Parameter - type: object - required: - - required - - description - properties: - description: - type: string - description: Description of the parameter. - required: - type: boolean - description: Whether or not the parameter must be explicitly set when searching data. - links: - type: array - description: |- - Additional information about the parameter. order_parameters: type: object description: Available parameters that can be used when ordering products using `POST /commercial_data/orders`. @@ -3654,14 +3720,6 @@ paths: - title: Airbus Pleiades pricing rel: related href: https://www.sentinel-hub.com/pricing/ - search_parameters: - bounds: - description: The request bounding box. Specify with a coordinate pair on two (opposite) vertices of the bounding box rectangle. Coordinates need to be in easting,northing or longitude, latitude, in EPSG:4326. - required: true - links: - - title: Bounds specification - - rel: related - - href: https://api.docs.com/tpdi/airbus#bounds time_range: description: A time interval to filter data by acquisition date. It is defined by a start and end date and time, e.g. 2019-01-31T14:00:00+01:00. A date representation without time (e.g. 2019-01-31) will not work. Both the start and end of a time interval are inclusive and can be equal. required: false @@ -3795,18 +3853,31 @@ paths: - $ref: '#/components/schemas/filter-cql2-json' - $ref: '#/components/schemas/filter-cql2-text' example: - collection_id: "PLEIADES" - parameters: - bounds: - - 3 - - 15 - - 4 - - 16 - time_range: - - "01-01-2022" - - "01-02-2022" - processing_level: SENSOR - max_snow_coverage: 0 + bbox: + - 3 + - 15 + - 4 + - 16 + datetime: + - "01-01-2022" + - "01-02-2022" + collectionsArray: + - PLEIADES + filter: + op: and + args: + - + op: "=" + args: + - + property: processing_level + - SENSOR + - + op: "=" + args: + - + property: max_snow_coverage + - 0 /commercial_data/orders: get: summary: List all orders From a8424275399c85bdd0d990e9c5ed78218ed6fed2 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Wed, 22 Jun 2022 11:30:41 +0200 Subject: [PATCH 057/103] Fix linting issues. --- openapi.yaml | 205 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 159 insertions(+), 46 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index b2b3adfa..a77c42a1 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3502,7 +3502,7 @@ paths: $ref: '#/components/responses/server_error' '/commercial_data/{collection_id}/queryables': get: - summary: Queryables for searching data products of a commercial data collection. + summary: Queryables for searching data products of a commercial data collection operationId: commercial-data-queryables description: The names and types of the properties or keys that may be used to construct a filter expression for a commercial data collection at `GET /commercial_data/search`. tags: @@ -3969,7 +3969,7 @@ paths: byoc_collection_id: 50b230c3-e1b5-431c-a6f7-f779684072bb costs: 42 - order:id: 27ce4888-776e-449d-ae9b-52fa86977554 - order:status: shippping + order:status: shipping order:date: '2022-04-01T15:03:56Z' collection_id: SPOT byoc_collection_id: 1efa3541-ca84-4f80-a5d1-ea57cdea36c1 @@ -4243,11 +4243,150 @@ paths: components: schemas: intersects: - type: object description: Only returns items that intersect with the provided polygon. - properties: - intersects: - $ref: https://geojson.org/schema/Geometry.json + oneOf: + - title: GeoJSON Point + type: object + required: + - type + - coordinates + properties: + type: + type: string + enum: + - Point + coordinates: + type: array + minItems: 2 + items: + type: number + bbox: + type: array + minItems: 4 + items: + type: number + - title: GeoJSON LineString + type: object + required: + - type + - coordinates + properties: + type: + type: string + enum: + - LineString + coordinates: + type: array + minItems: 2 + items: + type: array + minItems: 2 + items: + type: number + bbox: + type: array + minItems: 4 + items: + type: number + - title: GeoJSON Polygon + type: object + required: + - type + - coordinates + properties: + type: + type: string + enum: + - Polygon + coordinates: + type: array + items: + type: array + minItems: 4 + items: + type: array + minItems: 2 + items: + type: number + bbox: + type: array + minItems: 4 + items: + type: number + - title: GeoJSON MultiPoint + type: object + required: + - type + - coordinates + properties: + type: + type: string + enum: + - MultiPoint + coordinates: + type: array + items: + type: array + minItems: 2 + items: + type: number + bbox: + type: array + minItems: 4 + items: + type: number + - title: GeoJSON MultiLineString + type: object + required: + - type + - coordinates + properties: + type: + type: string + enum: + - MultiLineString + coordinates: + type: array + items: + type: array + minItems: 2 + items: + type: array + minItems: 2 + items: + type: number + bbox: + type: array + minItems: 4 + items: + type: number + - title: GeoJSON MultiPolygon + type: object + required: + - type + - coordinates + properties: + type: + type: string + enum: + - MultiPolygon + coordinates: + type: array + items: + type: array + items: + type: array + minItems: 4 + items: + type: array + minItems: 2 + items: + type: number + bbox: + type: array + minItems: 4 + items: + type: number filter-cql2-text: description: | A CQL2 filter expression in the 'cql2-text' encoding. @@ -4255,8 +4394,6 @@ components: filter-cql2-json: $ref: 'https://raw.githubusercontent.com/opengeospatial/ogcapi-features/master/cql2/standard/schema/cql2.yml#/components/schemas/booleanExpression' bbox: - name: bbox - in: query description: >- Only features that have a geometry that intersects the bounding box are selected. @@ -4316,25 +4453,15 @@ components: represented in JSON as `[160.6, -55.95, -170, -25.89]` and in a query as `bbox=160.6,-55.95,-170,-25.89`. - required: false - schema: - type: array - minItems: 4 - maxItems: 6 - items: - type: number - style: form - explode: false + type: array + minItems: 4 + maxItems: 6 + items: + type: number collectionId: - name: collectionId - in: path description: local identifier of a collection - required: true - schema: - type: string + type: string datetime: - name: datetime - in: query description: >- Either a date-time or an interval, open or closed. Date and time expressions @@ -4362,14 +4489,8 @@ components: server whether only a single temporal property is used to determine the extent or all relevant temporal properties. - required: false - schema: - type: string - style: form - explode: false + type: string limit: - name: limit - in: query description: >- The optional limit parameter limits the number of items that are presented in the response document. @@ -4383,24 +4504,16 @@ components: Minimum = 1. Maximum = 100. Default = 10. - required: false - schema: - type: integer - minimum: 1 - maximum: 100 - default: 10 - style: form - explode: false + type: integer + minimum: 1 + maximum: 100 + default: 10 collectionsArray: - name: collections description: | Array with the id of the collection to search products for. - required: false - schema: - type: array - items: - type: string - explode: false + type: array + items: + type: string minItems: 1 maxItems: 1 order_id: From 4cec5e67d304e1167e3ffe205a6d4963ba3208be Mon Sep 17 00:00:00 2001 From: jjezersek Date: Tue, 19 Jul 2022 13:48:26 +0200 Subject: [PATCH 058/103] Revert commits f55f8d2265985ae0e5ce060f580f8555327fe0e2, 2c21fad212499d711920d599f9850721f802e0e1, 33ad759c948fe2501a0edadf75a6696402145768, 7be50b0fc257a48d248cc84cef27371271ae9f11, 27a806cc44ddb4e8bc1551c37b995464f6569046, 83224f5de19d97299864a687eb946ceb2d0483ab, 697ca665c138d2fa6ea6e25d79aecec7d12bf9d6, ccd3b854debdbb7d506ce8e97a28a77d83a29135, 2dfc979c1bd1ddc2525bc9b2eeae396ae25fa4e9, 2d9b4e596c09954a2d1335cad69716facbf89eb4, 8c37ad4cac6b9d1ce908e9d4b76df3ac6f430a62, 8d7f36407c5a5bb476db424dafba37c24c765f11, 5bd050fd0528b140e1d130b0bb227789b958fa0e, a8424275399c85bdd0d990e9c5ed78218ed6fed2 --- openapi.yaml | 1169 -------------------------------------------------- 1 file changed, 1169 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index a77c42a1..45dcf7a2 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -520,9 +520,6 @@ tags: description: On-demand access to data using other web service protocols. - name: File Storage description: Management of user-uploaded assets and processed data. - - name: Commercial Data - description: |- - These endpoints provide utilies for purchasing commercial data. Commercial data is EO data that is not accessible free of charge. Processing costs come atop of data costs. servers: - url: 'https://localhost/api/{version}' description: >- @@ -3500,1174 +3497,8 @@ paths: $ref: '#/components/responses/client_error_auth' 5XX: $ref: '#/components/responses/server_error' - '/commercial_data/{collection_id}/queryables': - get: - summary: Queryables for searching data products of a commercial data collection - operationId: commercial-data-queryables - description: The names and types of the properties or keys that may be used to construct a filter expression for a commercial data collection at `GET /commercial_data/search`. - tags: - - Commercial Data - security: - - {} - - Bearer: [] - responses: - '200': - description: Returns the JSON Schema document with queryables for a commercial data collection. - content: - application/json: - schema: - title: Order Information - type: object - required: - - $schema - - $id - - type - - properties - properties: - $schema: - type: string - enum: - - http://json-schema.org/draft-07/schema# - - https://json-schema.org/draft/2019-09/schema - $id: - type: string - description: The URI of the resource without query parameters. - type: - type: string - enum: - - object - title: - type: string - description: Human-readable title. - description: - type: string - description: Human-readable description. - properties: - type: object - description: Object with available properties that can be evaluated in the CQL filter expression. - additionalProperties: - x-additionalPropertiesName: Property Name - description: JSON schema of the property. - type: object - - example: - $schema: http://json-schema.org/draft-07/schema# - $id: http://example.com/commercial_data/PLEIADES/queryables - type: object - title: Queryables of Airbus Pleiades collection. - description: Available properties for CQL filtering of products. - properties: - processing_level: - title: Processing level - description: When searching, you will receive results from the full catalog as well as the Living Library, which holds images that have cloud cover under 30% and Incidence angle under 40°. If you want to search only Living Library results, you will need to filter using processingLevel. This value could be equal to SENSOR (images which meet Living Library criteria) and ALBUM (images that do not meeting Living Library criteria in terms of incidence angle and cloud cover).' - type: string - enum: - - SENSOR - - ALBUM - max_snow_coverage: - title: Maximum snow coverage - description: The maximum allowable snow coverage in percent. - type: number - minimum: 0 - maximum: 100 - default: 100 - max_incidence_angle: - title: Maximum incidence angle - description: The maximum allowable incidence angle in degrees. - minimum: 0 - maximum: 90 - default: 90 - 4XX: - $ref: '#/components/responses/client_error_auth' - 5XX: - $ref: '#/components/responses/server_error' - parameters: - - name: collection_id - in: path - description: Collection identifier - required: true - schema: - $ref: '#/components/schemas/collection_id' - '/commercial_data/collections/{collection_id}': - get: - summary: Information ordering a commercial data collection - operationId: commercial-data-collection-info - description: |- - Lists information about pricing and available parameters for ordering products for a commercial data collection specified by the identifier `collection_id`. - STAC-compliant general information about the collection must be available at `GET collections/{collection_id}`. - tags: - - Commercial Data - security: - - {} - - Bearer: [] - parameters: - - name: collection_id - in: path - description: Collection identifier - required: true - schema: - $ref: '#/components/schemas/collection_id' - responses: - '200': - description: Information about commercial data collection. - content: - application/json: - schema: - title: Commercial Collection Info - type: object - required: - - id - - pricing - - order_parameters - properties: - id: - type: string - description: A unique identifier for the collection, which MUST match the specified pattern. - pattern: '^[\w\-\.~\/]+$' - provider: - type: object - title: Commercial data provider - required: - - name - properties: - name: - description: The id of the commercial data provider. - type: string - description: - description: >- - Multi-line description to add further provider information such as - processing details, hosting details or basic contact information. - - - CommonMark 0.29 syntax MAY be used for rich text representation. - type: string - roles: - description: >- - Role of the provider. Must be `producer`. - type: array - items: - type: string - enum: - - producer - url: - description: >- - Homepage on which the provider describes the dataset and publishes - contact information. - type: string - format: uri - pricing: - type: object - description: Information about pricing for searching and/or ordering data. - required: - - searching - - ordering - properties: - searching: - type: object - description: Pricing for searching available products in the collection. Must be explicitly set to `null` if searching is free of charge. - nullable: true - properties: - description: - type: string - description: Human-readable description of pricing. - links: - type: array - description: |- - Additional information about the pricing options. - ordering: - type: object - description: General information about pricing for ordering products. - properties: - description: - type: string - description: Human-readable description of pricing. - links: - type: array - description: |- - Additional information about the pricing options. - order_parameters: - type: object - description: Available parameters that can be used when ordering products using `POST /commercial_data/orders`. - additionalProperties: - x-additionalPropertiesName: Search Parameter - title: Search Parameter - type: object - required: - - required - - description - properties: - description: - type: string - description: Description of the parameter. - required: - type: boolean - description: Whether or not the parameter must be explicitly set when ordering data. - links: - type: array - description: |- - Additional information about the parameter. - example: - id: PLEIADES - provider: - name: AIRBUS - roles: - - producer - pricing: - searching: null - ordering: - description: Minimum area per order is 0.25 km2. The price is calculated based on a 6-months sliding window. - links: - - title: Airbus Pleiades pricing - rel: related - href: https://www.sentinel-hub.com/pricing/ - time_range: - description: A time interval to filter data by acquisition date. It is defined by a start and end date and time, e.g. 2019-01-31T14:00:00+01:00. A date representation without time (e.g. 2019-01-31) will not work. Both the start and end of a time interval are inclusive and can be equal. - required: false - processing_level: - description: 'Enum: SENSOR,ALBUM. When searching, you will receive results from the full catalog as well as the Living Library, which holds images that have cloud cover under 30% and Incidence angle under 40°. If you want to search only Living Library results, you will need to filter using processingLevel. This value could be equal to SENSOR (images which meet Living Library criteria) and ALBUM (images that do not meeting Living Library criteria in terms of incidence angle and cloud cover).' - required: false - max_snow_coverage: - description: 'The maximum allowable snow coverage in percent. Default: 100.' - required: false - max_incidence_angle: - description: 'he maximum allowable incidence angle in degrees. Default: 90.' - required: false - order_parameters: - bounds: - description: The request bounding box. Specify with a coordinate pair on two (opposite) vertices of the bounding box rectangle. Coordinates need to be in easting,northing or longitude, latitude, in EPSG:4326. - required: true - links: - - title: Bounds specification - - rel: related - - href: https://api.docs.com/tpdi/airbus#bounds - 4XX: - $ref: '#/components/responses/client_error_auth' - 5XX: - $ref: '#/components/responses/server_error' - /commercial_data/search: - get: - summary: Search for available products in a commercial data collection - operationId: commercial-data-search - description: |- - Search for products in a commercial data collection. Available filtering parameters are listed at `GET /commercial_data/collections/{collection_id}`. - tags: - - Commercial Data - security: - - Bearer: [] - parameters: - - $ref: '#/components/parameters/pagination_limit' - responses: - '200': - description: Lists of available products and related links. - content: - application/json: - schema: - title: Available Products - type: object - required: - - type - - features - - links - properties: - type: - type: string - enum: - - FeatureCollection - features: - type: array - description: An array of available products as STAC Items. - title: Item - items: - $ref: '#/components/schemas/product' - links: - $ref: '#/components/schemas/links_pagination' - example: - type: FeatureCollection - features: - - type: Feature - stac_version: 0.9.0 - id: c8a1f88d-89cf-4933-9118-45e9c1a5df20 - geometry: - type: Polygon - coordinates: - - - - - - 12.36555287044679 - - 41.94403289260048 - - - - 12.36571746774068 - - 41.86399361096971 - - - - 12.60746759743069 - - 41.86372776276345 - - - - 12.60758647471871 - - 41.94379931812686 - - - - 12.36555287044679 - - 41.94403289260048 - properties: - constellation: PHR - acquisitionDate: "2022-03-21T10:11:15.055Z" - azimuthAngle: 179.9852862071639 - cloudCover: 0 - geometryCentroid: - lat: 41.903935647240964 - lon: 12.486569672582828 - processingLevel: SENSOR - sensorType: OPTICAL - spectralRange: VISIBLE - assets: {} - links: [] - links: [] - 4XX: - $ref: '#/components/responses/client_error_auth' - 5XX: - $ref: '#/components/responses/server_error' - requestBody: - required: true - description: Data with desired filtering information to search for. - content: - application/json: - schema: - title: Search Commercial Products Request - type: object - properties: - bbox: - $ref: '#/components/schemas/bbox' - datetime: - $ref: '#/components/schemas/datetime' - intersects: - $ref: '#/components/schemas/intersects' - limit: - $ref: '#/components/schemas/limit' - collectionsArray: - $ref: '#/components/schemas/collectionsArray' - filter: - description: |- - **Extension:** Filter - A CQL2 filter expression for filtering items. - - The names and types of the properties or keys that may be used to construct a filter expression for a collection must accessible at `GET commercial_data/{collectionId}/queryables`. - oneOf: - - $ref: '#/components/schemas/filter-cql2-json' - - $ref: '#/components/schemas/filter-cql2-text' - example: - bbox: - - 3 - - 15 - - 4 - - 16 - datetime: - - "01-01-2022" - - "01-02-2022" - collectionsArray: - - PLEIADES - filter: - op: and - args: - - - op: "=" - args: - - - property: processing_level - - SENSOR - - - op: "=" - args: - - - property: max_snow_coverage - - 0 - /commercial_data/orders: - get: - summary: List all orders - operationId: commercial-data-orders - description: |- - Lists all orders created by the user. - tags: - - Commercial Data - security: - - Bearer: [] - parameters: - - $ref: '#/components/parameters/pagination_limit' - responses: - '200': - description: Array of order information. - content: - application/json: - schema: - title: All Orders - type: object - required: - - orders - - links - properties: - orders: - type: array - items: - type: object - title: Order - description: Information about an order - required: - - order:id - - order:status - - order:date - - collection_id - - byoc_collection_id - - costs - properties: - order:id: - $ref: '#/components/schemas/order_id' - order:status: - type: string - description: |- - The status of the order. - * `orderable`: The item or asset is orderable via the provider scenario. - - * `ordered`: The item or asset is ordered and the provider is preparing to make it available. - - * `shipping`: The item or asset order are being processed by the provider to provide the user with the asset(s). - - * `delivered`: The provider has delivered the order and asset(s) are available. - - * `unable_to_deliver`: The provider is not able to deliver the order. - - * `canceled`: The order has been canceled. - enum: - - orderable - - ordered - - shipping - - delivered - - unable_to_deliver - - canceled - order:date: - type: string - format: date-time - description: The order time. Formatted as a - [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - order:expiration_date: - type: string - format: date-time - description: The validity time of the order. Formatted as a - [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - collection_id: - type: string - description: Unique identifier of the collection. - byoc_collection_id: - type: string - description: Unique identifier required to access the purchased data. - costs: - $ref: '#/components/schemas/money' - links: - $ref: '#/components/schemas/links_pagination' - example: - orders: - - order:id: 40264b5-c3ae-46f4-a907-0f612d763d97 - order:status: delivered - order:date: '2017-01-01T09:32:12Z' - collection_id: PLEAIDES - byoc_collection_id: 50b230c3-e1b5-431c-a6f7-f779684072bb - costs: 42 - - order:id: 27ce4888-776e-449d-ae9b-52fa86977554 - order:status: shipping - order:date: '2022-04-01T15:03:56Z' - collection_id: SPOT - byoc_collection_id: 1efa3541-ca84-4f80-a5d1-ea57cdea36c1 - costs: 10 - links: [] - 4XX: - $ref: '#/components/responses/client_error_auth' - 5XX: - $ref: '#/components/responses/server_error' - post: - summary: Create an order - operationId: commercial-data-order - description: |- - Create an order for products of a commercial data collection. The order is NOT yet executed and awaits confirmation. - tags: - - Commercial Data - security: - - Bearer: [] - responses: - '200': - description: Returns unique id and cost of the created order. - content: - application/json: - schema: - title: Order Information - type: object - required: - - id - - status - - costs - properties: - id: - $ref: '#/components/schemas/order_id' - status: - type: string - enum: - - NOT_CONFIRMED - costs: - $ref: '#/components/schemas/money' - example: - id: 40264b5-c3ae-46f4-a907-0f612d763d97 - status: NOT_CONFIRMED - costs: 42 - 4XX: - $ref: '#/components/responses/client_error_auth' - 5XX: - $ref: '#/components/responses/server_error' - requestBody: - required: true - description: The base data to create the order - content: - application/json: - schema: - title: Create Order Request - type: object - required: - - collection_id - - products - - parameters - properties: - collection_id: - type: string - description: Unique identifier of the collection. - products: - type: array - description: Array of IDs of products to order. - items: - $ref: '#/components/schemas/order_id' - parameters: - type: object - description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. - additionalProperties: - x-additionalPropertiesName: Order Parameter Name - description: Value of the order parameter to be used in the request. - example: - collection_id: "PLEIADES" - products: - - c8a1f88d-89cf-4933-9118-45e9c1a5df20 - - bc298be8-9381-4ea4-a169-cb9fceda0540 - parameters: - bounds: - - 3 - - 15 - - 4 - - 16 - '/commercial_data/orders/{order_id}': - get: - summary: Get detailed information about an order - operationId: commercial-data-order-info - description: |- - Detailed information about an order. - tags: - - Commercial Data - security: - - Bearer: [] - parameters: - - name: order_id - in: path - description: Order identifier - required: true - schema: - $ref: '#/components/schemas/order_id' - responses: - '200': - description: Order information. - content: - application/json: - schema: - title: Order Information - type: object - required: - - order:id - - order:status - - order:date - - collection_id - - products - - parameters - - byoc_collection_id - - costs - properties: - order:id: - $ref: '#/components/schemas/order_id' - order:status: - type: string - description: |- - The status of the order. - * `orderable`: The item or asset is orderable via the provider scenario. - - * `ordered`: The item or asset is ordered and the provider is preparing to make it available. - - * `shipping`: The item or asset order are being processed by the provider to provide the user with the asset(s). - - * `delivered`: The provider has delivered the order and asset(s) are available. - - * `unable_to_deliver`: The provider is not able to deliver the order. - - * `canceled`: The order has been canceled. - enum: - - orderable - - ordered - - shipping - - delivered - - unable_to_deliver - - canceled - order:date: - type: string - format: date-time - description: The order time. Formatted as a - [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - order:expiration_date: - type: string - format: date-time - description: The validity time of the order. Formatted as a - [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - products: - type: array - description: An array of available products as STAC Items. - items: - $ref: '#/components/schemas/product' - parameters: - type: object - description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. - additionalProperties: - x-additionalPropertiesName: Order Parameter Name - description: Value of the order parameter to be used in the request. - collection_id: - type: string - description: Unique identifier of the collection. - byoc_collection_id: - type: string - description: Unique identifier required to access the purchased data. - costs: - $ref: '#/components/schemas/money' - example: - order:id: 40264b5-c3ae-46f4-a907-0f612d763d97 - order:status: delivered - order:date: '2017-01-01T09:32:12Z' - products: - - type: Feature - stac_version: 0.9.0 - id: c8a1f88d-89cf-4933-9118-45e9c1a5df20 - geometry: - type: Polygon - coordinates: - - - - - - 12.36555287044679 - - 41.94403289260048 - - - - 12.36571746774068 - - 41.86399361096971 - - - - 12.60746759743069 - - 41.86372776276345 - - - - 12.60758647471871 - - 41.94379931812686 - - - - 12.36555287044679 - - 41.94403289260048 - properties: - constellation: PHR - acquisitionDate: "2022-03-21T10:11:15.055Z" - azimuthAngle: 179.9852862071639 - cloudCover: 0 - geometryCentroid: - lat: 41.903935647240964 - lon: 12.486569672582828 - processingLevel: SENSOR - sensorType: OPTICAL - spectralRange: VISIBLE - assets: {} - links: [] - parameters: - bounds: - - 3 - - 15 - - 4 - - 16 - collection_id: PLEAIDES - byoc_collection_id: 50b230c3-e1b5-431c-a6f7-f779684072bb - costs: 42 - 4XX: - $ref: '#/components/responses/client_error_auth' - 5XX: - $ref: '#/components/responses/server_error' - post: - summary: Confirm an order - operationId: commercial-data-order-confirm - description: |- - Confirm a created commercial data order. If order status was `NOT_CONFIRMED` execution of the order begins. Otherwise this endpoint has no effect. - tags: - - Commercial Data - security: - - Bearer: [] - parameters: - - $ref: '#/components/parameters/pagination_limit' - - name: order_id - in: path - description: Order identifier - required: true - schema: - $ref: '#/components/schemas/order_id' - responses: - '200': - description: Returns unique id and the status of the order. - content: - application/json: - schema: - title: Order Confirmation - type: object - required: - - id - - status - properties: - id: - $ref: '#/components/schemas/collection_id' - status: - type: string - enum: - - RUNNING - - FAILED - - DONE - example: - id: 40264b5-c3ae-46f4-a907-0f612d763d97 - status: RUNNING - 4XX: - $ref: '#/components/responses/client_error_auth' - 5XX: - $ref: '#/components/responses/server_error' components: schemas: - intersects: - description: Only returns items that intersect with the provided polygon. - oneOf: - - title: GeoJSON Point - type: object - required: - - type - - coordinates - properties: - type: - type: string - enum: - - Point - coordinates: - type: array - minItems: 2 - items: - type: number - bbox: - type: array - minItems: 4 - items: - type: number - - title: GeoJSON LineString - type: object - required: - - type - - coordinates - properties: - type: - type: string - enum: - - LineString - coordinates: - type: array - minItems: 2 - items: - type: array - minItems: 2 - items: - type: number - bbox: - type: array - minItems: 4 - items: - type: number - - title: GeoJSON Polygon - type: object - required: - - type - - coordinates - properties: - type: - type: string - enum: - - Polygon - coordinates: - type: array - items: - type: array - minItems: 4 - items: - type: array - minItems: 2 - items: - type: number - bbox: - type: array - minItems: 4 - items: - type: number - - title: GeoJSON MultiPoint - type: object - required: - - type - - coordinates - properties: - type: - type: string - enum: - - MultiPoint - coordinates: - type: array - items: - type: array - minItems: 2 - items: - type: number - bbox: - type: array - minItems: 4 - items: - type: number - - title: GeoJSON MultiLineString - type: object - required: - - type - - coordinates - properties: - type: - type: string - enum: - - MultiLineString - coordinates: - type: array - items: - type: array - minItems: 2 - items: - type: array - minItems: 2 - items: - type: number - bbox: - type: array - minItems: 4 - items: - type: number - - title: GeoJSON MultiPolygon - type: object - required: - - type - - coordinates - properties: - type: - type: string - enum: - - MultiPolygon - coordinates: - type: array - items: - type: array - items: - type: array - minItems: 4 - items: - type: array - minItems: 2 - items: - type: number - bbox: - type: array - minItems: 4 - items: - type: number - filter-cql2-text: - description: | - A CQL2 filter expression in the 'cql2-text' encoding. - type: string - filter-cql2-json: - $ref: 'https://raw.githubusercontent.com/opengeospatial/ogcapi-features/master/cql2/standard/schema/cql2.yml#/components/schemas/booleanExpression' - bbox: - description: >- - Only features that have a geometry that intersects the bounding box are - selected. - - The bounding box is provided as four or six numbers, depending on - - whether the coordinate reference system includes a vertical axis (height - - or depth): - - - * Lower left corner, coordinate axis 1 - - * Lower left corner, coordinate axis 2 - - * Minimum value, coordinate axis 3 (optional) - - * Upper right corner, coordinate axis 1 - - * Upper right corner, coordinate axis 2 - - * Maximum value, coordinate axis 3 (optional) - - - The coordinate reference system of the values is WGS 84 - - longitude/latitude (http://www.opengis.net/def/crs/OGC/1.3/CRS84). - - - For WGS 84 longitude/latitude the values are in most cases the sequence - - of minimum longitude, minimum latitude, maximum longitude and maximum - - latitude. However, in cases where the box spans the antimeridian the - - first value (west-most box edge) is larger than the third value - - (east-most box edge). - - - If the vertical axis is included, the third and the sixth number are - - the bottom and the top of the 3-dimensional bounding box. - - - If a feature has multiple spatial geometry properties, it is the - - decision of the server whether only a single spatial geometry property - - is used to determine the extent or all relevant geometries. - - - Example: The bounding box of the New Zealand Exclusive Economic Zone in - - WGS 84 (from 160.6°E to 170°W and from 55.95°S to 25.89°S) would be - - represented in JSON as `[160.6, -55.95, -170, -25.89]` and in a query as - - `bbox=160.6,-55.95,-170,-25.89`. - type: array - minItems: 4 - maxItems: 6 - items: - type: number - collectionId: - description: local identifier of a collection - type: string - datetime: - description: >- - Either a date-time or an interval, open or closed. Date and time - expressions - - adhere to RFC 3339. Open intervals are expressed using double-dots. - - - Examples: - - - * A date-time: "2018-02-12T23:20:50Z" - - * A closed interval: "2018-02-12T00:00:00Z/2018-03-18T12:31:12Z" - - * Open intervals: "2018-02-12T00:00:00Z/.." or "../2018-03-18T12:31:12Z" - - - Only features that have a temporal property that intersects the value of - - `datetime` are selected. - - - If a feature has multiple temporal properties, it is the decision of the - - server whether only a single temporal property is used to determine - - the extent or all relevant temporal properties. - type: string - limit: - description: >- - The optional limit parameter limits the number of items that are - presented in the response document. - - - Only items are counted that are on the first level of the collection in - the response document. - - Nested objects contained within the explicitly requested items shall not - be counted. - - - Minimum = 1. Maximum = 100. Default = 10. - type: integer - minimum: 1 - maximum: 100 - default: 10 - collectionsArray: - description: | - Array with the id of the collection to search products for. - type: array - items: - type: string - minItems: 1 - maxItems: 1 - order_id: - type: string - description: >- - Unique identifier of the order, which MUST match the specified pattern.. - pattern: '^[\w\-\.~]+$' - example: 40264b5-c3ae-46f4-a907-0f612d763d97 - product: - type: object - title: Item - required: - - stac_version - - id - - type - - geometry - - properties - - assets - - links - properties: - stac_version: - $ref: '#/components/schemas/stac_version' - stac_extensions: - $ref: '#/components/schemas/stac_extensions' - id: - type: string - description: Unique identifier of a product. - type: - type: string - description: >- - The GeoJSON type that applies to this metadata document, - which MUST always be a "Feature" according to the STAC specification. - - This type does **not** describe the spatial data type of the assets. - enum: - - Feature - bbox: - description: |- - Potential *spatial extent* covered by the data. - - The bounding box is provided as four or six numbers. Six numbers are - specified, if the coordinate reference system includes a vertical axis - (height or depth). The order of the elements in the array: - - - West (lower left corner, coordinate axis 1) - - South (lower left corner, coordinate axis 2) - - Base (optional, lower left corner, coordinate axis 3) - - East (upper right corner, coordinate axis 1) - - North (upper right corner, coordinate axis 2) - - Height (optional, upper right corner, coordinate axis 3) - - The coordinate reference system of the values is WGS84 - longitude/latitude. - - Specifying the `bbox` is strongly RECOMMENDED for STAC compliance, - but can be omitted if the result is unlocated and the `geometry` - is set to `null`. - type: array - oneOf: - - title: 4 elements - minItems: 4 - maxItems: 4 - - title: 6 elements - minItems: 6 - maxItems: 6 - items: - type: number - example: - - -180 - - -90 - - 180 - - 90 - geometry: - description: |- - Defines the full footprint of the asset represented by this item as - GeoJSON Geometry. - - Results without a known location can set this value to `null`. - nullable: true - allOf: - - $ref: '#/components/schemas/GeoJsonGeometry' - example: - type: Polygon - coordinates: - - - - -180 - - -90 - - - 180 - - -90 - - - 180 - - 90 - - - -180 - - 90 - - - -180 - - -90 - properties: - type: object - title: Item Properties - description: >- - MAY contain custom properties or properties from the STAC specification or STAC extensions. - additionalProperties: true - properties: - datetime: - title: Date and Time - description: >- - The searchable date/time of the data, in UTC. Formatted as a - [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - - - If this field is set to `null` (usually for larger time ranges), - it is STRONGLY RECOMMENDED to specify both `start_datetime` and - `end_datetime` for STAC compliance. - type: string - format: date-time - nullable: true - start_datetime: - type: string - format: date-time - description: >- - For time series: The first or start date and time for the data, - in UTC. Formatted as a [RFC - 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - end_datetime: - type: string - format: date-time - description: >- - For time series: The last or end date and time for the data, in - UTC. Formatted as a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) - date-time. - title: - $ref: '#/components/schemas/eo_title' - description: - $ref: '#/components/schemas/eo_description' - license: - $ref: '#/components/schemas/stac_license' - providers: - $ref: '#/components/schemas/stac_providers' - created: - $ref: '#/components/schemas/created' - updated: - $ref: '#/components/schemas/updated' - expires: - type: string - format: date-time - description: >- - Time until which the assets are accessible, in UTC. Formatted as - a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - example: '2020-11-01T00:00:00Z' - assets: - $ref: '#/components/schemas/stac_assets' - links: - type: array - description: Links related to this product - items: - $ref: '#/components/schemas/link' udf_runtime: type: object required: From 863fc784d6b1eab7a30b163fd6d6a39485cfceb3 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 19 Jul 2022 16:56:22 +0200 Subject: [PATCH 059/103] Clarified uniqueness constraints for identifiers. #454 --- CHANGELOG.md | 1 + openapi.yaml | 19 ++++++++++--------- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 71ca994c..ca5f4f1b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,6 +43,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /`: Removed the superfluous default value for `currency`. [#423](https://github.com/Open-EO/openeo-api/issues/423) - Reflect that the `debug` process has been renamed to `inspect`. - `GET /credentials/oidc`: Clarify that clients may add additional scopes +- Clarified uniqueness constraints for identifiers. [#454](https://github.com/Open-EO/openeo-api/issues/454) ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index bb2458f6..1de79573 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -250,7 +250,7 @@ info: **Users** MUST follow the schema for user-defined processes as in [`GET /process_graphs/{process_graph_id}`](#operation/describe-custom-process) to define new processes. This includes: - * Choosing a intuitive name as process id, consisting of only letters (a-z), numbers and underscores. It MUST be unique across the user-defined processes. + * Choosing a intuitive name as process id, consisting of only letters (a-z), numbers and underscores. It MUST be unique per user across the user-defined processes. * Defining the algorithm as a process graph. * Optionally, specifying the additional metadata for processes. @@ -1839,7 +1839,7 @@ paths: id: type: string description: >- - A **unique** identifier for the OpenID Connect Provider to + A per-backend **unique** identifier for the OpenID Connect Provider to be as prefix for the Bearer token. pattern: '[\d\w]{1,20}' issuer: @@ -2282,7 +2282,7 @@ paths: parameters: - name: process_graph_id in: path - description: Unique identifier for a user-defined process. + description: Per-user unique identifier for a user-defined process. required: true schema: $ref: '#/components/schemas/process_id' @@ -4379,7 +4379,8 @@ components: 'cube:dimensions': title: STAC Collection Cube Dimensions description: |- - Uniquely named dimensions of the data cube. + The named default dimensions of the data cube. + Names must be unique per collection. The keys of the object are the dimension names. For interoperability, it is RECOMMENDED to use the @@ -5431,7 +5432,7 @@ components: job_id: type: string description: >- - Unique identifier of the batch job, generated by the + Per-backend unique identifier of the batch job, generated by the back-end during creation. MUST match the specified pattern. pattern: '^[\w\-\.~]+$' example: a3cca2b2aa1e3b5b @@ -5591,8 +5592,8 @@ components: service_id: type: string description: >- - Unique identifier of the secondary web service, generated by the - back-end during creation. MUST match the specified pattern. + A per-backend unique identifier of the secondary web service, generated + by the back-end during creation. MUST match the specified pattern. pattern: '^[\w\-\.~]+$' example: wms-a3cca9 resource_parameter: @@ -6283,14 +6284,14 @@ components: service_id: name: service_id in: path - description: Unique secondary web service identifier. + description: Identifier of the secondary web service. required: true schema: $ref: '#/components/schemas/service_id' job_id: name: job_id in: path - description: Unique job identifier. + description: Identifier of the batch job. required: true schema: $ref: '#/components/schemas/job_id' From fe4a563957e49fc2b2b2e76503f5bb24116ba7b5 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 19 Jul 2022 18:23:46 +0200 Subject: [PATCH 060/103] Clarified uniqueness constraints for PG identifiers. #449 --- CHANGELOG.md | 2 +- openapi.yaml | 11 +++++------ 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ca5f4f1b..d987a1ce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,7 +43,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /`: Removed the superfluous default value for `currency`. [#423](https://github.com/Open-EO/openeo-api/issues/423) - Reflect that the `debug` process has been renamed to `inspect`. - `GET /credentials/oidc`: Clarify that clients may add additional scopes -- Clarified uniqueness constraints for identifiers. [#454](https://github.com/Open-EO/openeo-api/issues/454) +- Clarified uniqueness constraints for identifiers. [#449](https://github.com/Open-EO/openeo-api/issues/449) [#454](https://github.com/Open-EO/openeo-api/issues/454) ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index 1de79573..a03835b9 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2320,12 +2320,11 @@ paths: operationId: store-custom-process description: |- Stores a provided user-defined process with process graph that can be - reused in other processes. If a process with the specified - `process_graph_id` exists, the process is fully replaced. The id can't - be changed for stored user-defined processes. - - The id MUST be unique for the authenticated user, including all - predefined processes by the back-end. + reused in other processes. + + If a process with the specified `process_graph_id` exists, the process + is fully replaced. The id can't be changed for existing user-defined + processes. The id MUST be unique across its namespace. Partially updating user-defined processes is not supported. From 4aa598049cc0f98347c5febc04e3f59fddd089c8 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Tue, 19 Jul 2022 18:42:19 +0200 Subject: [PATCH 061/103] Add README for Commercial Data extension. --- extension/commercial_data/README.md | 416 ++++++++++++++++++++++++++++ 1 file changed, 416 insertions(+) create mode 100644 extension/commercial_data/README.md diff --git a/extension/commercial_data/README.md b/extension/commercial_data/README.md new file mode 100644 index 00000000..9758cc7e --- /dev/null +++ b/extension/commercial_data/README.md @@ -0,0 +1,416 @@ +# Commercial Data Extension + +The Commercial Data API extension provides an interface for discovering, ordering and using commercial data in the openEO API. + +- Version: **0.1.0** +- Stability: **experimental** + +**Note:** This document only documents the additions to the specification. +Extensions can not change or break existing behavior of the openEO API. + +## Overview of the workflow + +All the available datasets provided by a backend are listed on the `/collections` endpoint. The collections are normally freely accessible. This extensions adds capabilities for providing collections that are not free of charge and require purchasing data products that can thereupon be used in processing. Commercial data collections usually allow purchasing small subsets of the data (products), for example a single observation of an area. + +Therefore, the client must have an ability to search the available products that match their desired criteria and inspect their metadata to decide whether the products should be purchased. + +The client can then create an order for the desired products. Because of the financial cost of purchasing the data, a separate endpoint for confirming the execution of that specific order should be implemented. + +When the order is completed, the user should be able to access the data using the `collection-id` at the temporal and spatial location of the purchased products. + +### Collection discovery + +A backend should add general information about a commercial data collection to the `/collections` and `/collections/{collection-id}` endpoints, the same as with freely available collections. Only the metadata about the entire dataset needs to be provided, not about the specific data products that a user has already purchased. + +Commercial data collections are distinguished from freely available collections by including `"order:status": "orderable"` as specified in [STAC Order specification](https://github.com/stac-extensions/order). + +Commercial data collections can include `order_parameters` field, if ordering supports additional parameters that specify how the products should be delivered. + +Commercial data collections must also include a link to human-readable pricing information for searching and ordering the products. If searching the products is free it should be set to `null`. + +#### Example + +```json +{ + "stac_version": "1.0.0", + "stac_extensions": [ + "https://stac-extensions.github.io/datacube/v1.0.0/schema.json", + "https://stac-extensions.github.io/order/v1.0.0/schema.json" + ], + "type": "Collection", + "id": "PLEIADES", + "title": "Airbus Pleiades", + "description": "Pleiades is a satellite constellation providing very high-resolution optical imagery and is owned by Airbus. Pléiades is composed of two twin satellites orbiting the Earth 180° apart. The satellites deliver 0.5 m optical imagery and offer a daily revisit capability to any point on the globe.", + "license": "proprietary", + "providers": [...], + "extent": {...}, + "links": [...], + "cube:dimensions": {...}, + "summaries": {...}, + "assets": {...}, + "order:status": "orderable", + "order_parameters": [ + { + "name": "sample_type", + "description": "sample type of the output raster.", + "schema": { + "type": "string" + }, + "enum": [ + "UINT16", + "UINT8" + ] + }, + ], + "pricing": { + "searching": null, + "ordering": { + "description": "Minimum area per order is 0.25 km2. The price is calculated based on a 6-months sliding window.", + "links": [ + { + "title": "Airbus Pleiades pricing", + "rel": "related", + "href": "https://www.sentinel-hub.com/pricing/#TPD_pricing" + } + ] + } + } +} +``` + +### Filtering parameters discovery + +Searching for products can support refining the search by filtering with general or collection-specific attributes. Backends should implement a top level `/queryables` endpoint for attributes available for all collections, and collection-specific attributes should be provided at `/collections/{collection-id}/queryables` according to [OGC Queryables specification](https://portal.ogc.org/files/96288#filter-queryables). + +#### Example + +Example response from `GET collections/PLEIADES/queryables`: +```json +{ + "$schema":"http://json-schema.org/draft-07/schema#", + "$id":"http://example.com/commercial_data/PLEIADES/queryables", + "type":"object", + "title":"Queryables of Airbus Pleiades collection.", + "description":"Available properties for CQL filtering of products.", + "properties":{ + "processing_level":{ + "title":"Processing level", + "description":"When searching, you will receive results from the full catalog as well as the Living Library, which holds images that have cloud cover under 30% and Incidence angle under 40°. If you want to search only Living Library results, you will need to filter using processingLevel. This value could be equal to SENSOR (images which meet Living Library criteria) and ALBUM (images that do not meeting Living Library criteria in terms of incidence angle and cloud cover).'", + "type":"string", + "enum":[ + "SENSOR", + "ALBUM" + ] + }, + "max_snow_coverage":{ + "title":"Maximum snow coverage", + "description":"The maximum allowable snow coverage in percent.", + "type":"number", + "minimum":0, + "maximum":100, + "default":100 + }, + "max_incidence_angle":{ + "title":"Maximum incidence angle", + "description":"The maximum allowable incidence angle in degrees.", + "minimum":0, + "maximum":90, + "default":90 + } + } +} +``` + +### Searching available products + +Backends should implement top-level `GET /search` endpoint as specified in [STAC Item Search API specification](https://github.com/radiantearth/stac-api-spec/tree/v1.0.0-rc.1/item-search). This should include the [Filter Extension](https://github.com/radiantearth/stac-api-spec/tree/v1.0.0-rc.1/fragments/filter), which enables filtering the available products by attributes specified in `/queryables` and `collections/{collection-id}/queryables`. +The endpoint should return a list of `STAC Item`s that match the criteria. + +#### Example + +Example request payload to `GET /search` for `PLEIADES` products from "Living Library" with no snow coverage. + +```json +{ + "bbox": [ + 3, + 15, + 4, + 16 + ], + "datetime": [ + "01-01-2022", + "01-02-2022" + ], + "collectionsArray": [ + "PLEIADES" + ], + "filter": { + "op": "and", + "args": [ + { + "op": "=", + "args": [ + { + "property": "processing_level" + }, + "SENSOR" + ] + }, + { + "op": "=", + "args": [ + { + "property": "max_snow_coverage" + }, + 0 + ] + } + ] + } +} +``` + +### Ordering products + +Backends should implement the following endpoints: + +- `GET /orders`: Get a list of all created orders +- `POST /orders`: Create an order +- `GET /orders/{order-id}`: Get full metadata of a specific order +- `POST /orders/{order-id}`: Confirm a created order + +#### `GET /orders` + +Lists all created orders, regardless of the status or the collection. The items should follow the [STAC Order Extension](https://github.com/stac-extensions/order). + +```yaml +schema: + title: All Orders + type: object + required: + - orders + - links + properties: + orders: + type: array + items: + type: object + title: Order + description: Information about an order + required: + - order:id + - order:status + - order:date + - collection_id + - costs + properties: + order:id: + type: string + description: >- + Unique identifier of the order, which MUST match the specified pattern.. + pattern: '^[\w\-\.~]+$' + order:status: + type: string + description: |- + The status of the order. + * `orderable`: The item or asset is orderable via the provider scenario. + * `ordered`: The item or asset is ordered and the provider is preparing to make it available. + * `shipping`: The item or asset order are being processed by the provider to provide the user with the asset(s). + * `delivered`: The provider has delivered the order and asset(s) are available. + * `unable_to_deliver`: The provider is not able to deliver the order. + * `canceled`: The order has been canceled. + enum: + - orderable + - ordered + - shipping + - delivered + - unable_to_deliver + - canceled + order:date: + type: string + format: date-time + description: The order time. Formatted as a + [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + order:expiration_date: + type: string + format: date-time + description: The validity time of the order. Formatted as a + [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + collection_id: + type: string + description: Unique identifier of the collection. + costs: + $ref: 'https://raw.githubusercontent.com/Open-EO/openeo-api/1.0.0/openapi.yaml#/components/schemas/money' + links: + $ref: 'https://raw.githubusercontent.com/Open-EO/openeo-api/1.0.0/openapi.yaml#/components/schemas/links_pagination' +``` + +#### `POST /orders` + +Create an order for selected products. Order can contain some additional parameters that specify how the products should be delivered. For example, depending on the collection it might be possible to set the projection, resampling method, bit depth etc of the delivered data. + +Backends should expose the available ordering parameters in `/collections/{collection-id}` in the `order_parameters` field, following the `process_parameters` schema of [`GET /service_types`](https://docs.openeo.cloud/api/#tag/Secondary-Services/operation/list-service-types). + +The request should return the `order-id`, current status and the costs of the order. + +```yaml +schema: + title: Create Order Request + type: object + required: + - collection_id + - products + - parameters + properties: + collection_id: + type: string + description: Unique identifier of the collection. + products: + type: array + description: Array of IDs of products to order. + items: + $ref: 'https://raw.githubusercontent.com/Open-EO/openeo-api/1.0.0/openapi.yaml#/components/schemas/order_id' + parameters: + type: object + description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. + additionalProperties: + x-additionalPropertiesName: Order Parameter Name + description: Value of the order parameter to be used in the request. +``` + +#### `GET /orders/{order-id}` + +Get full metadata of the order. The item should follow the [STAC Order Extension](https://github.com/stac-extensions/order), but extended with the spatial and temporal extent information and other metadata about the products. + +```json +{ + "order:id": "40264b5-c3ae-46f4-a907-0f612d763d97", + "order:status": "delivered", + "order:date": "2017-01-01T09:32:12Z", + "products": [ + { + "type": "Feature", + "stac_version": "0.9.0", + "id": "c8a1f88d-89cf-4933-9118-45e9c1a5df20", + "geometry": { + "type": "Polygon", + "coordinates": [ + [ + [ + 12.36555287044679, + 41.94403289260048 + ] + ], + [ + 12.36571746774068, + 41.86399361096971 + ], + [ + 12.60746759743069, + 41.86372776276345 + ], + [ + 12.60758647471871, + 41.94379931812686 + ], + [ + 12.36555287044679, + 41.94403289260048 + ] + ] + }, + "properties": { + "constellation": "PHR", + "acquisitionDate": "2022-03-21T10:11:15.055Z", + "azimuthAngle": 179.9852862071639, + "cloudCover": 0, + "geometryCentroid": { + "lat": 41.903935647240964, + "lon": 12.486569672582828 + }, + "processingLevel": "SENSOR", + "sensorType": "OPTICAL", + "spectralRange": "VISIBLE" + }, + "assets": {}, + "links": [] + } + ], + "parameters": { + "bounds": [ + 3, + 15, + 4, + 16 + ] + }, + "collection_id": "PLEAIDES", + "costs": 42 +} +``` + +#### `POST /orders/{order-id}` + +When an order is created, the data isn't yet ordered from the commercial data provider. The client should explicitly confirm the order whereupon the order is executed, the costs are deducted from the client's account and the data is ingested in the associated collection. The `order:status` changes to `ordered`. + +If the user doesn't have sufficient funds, the endpoint should return an error and `order:status` should not change. + +### Payment + +Payment should be done in the currency used by the backend, listed at `GET /` under `billing`. When an order is created, the backend should return the full final cost of the order. + +### Example usage with Python client + +Python client could be extented to support commercial data. Here an example of a possible workflow is provided. + +```python +>>> import openeo +>>> connection = openeo.connect("openeo.sentinel-hub.com") +>>> connection.list_collection_ids(status="orderable") +["PLEIADES", "SPOT", ...] +``` + +We can fetch queryables and hold them in a class that validates user's parameters against the schema and constructs the payload. +```python +>>> pleiades_queryables = connection.get_queryables(collection_id="PLEIADES") +>>> pleiades_queryables.list() +[{'name': 'processing_level', 'type': 'string', 'enum': ['SENSOR', 'ALBUM']}, {'name': 'max_snow_coverage', 'type': 'number', 'minimum': 0, 'maximum': 100, 'default': 100}, {'name': 'max_incidence_angle', 'type': 'number', 'minimum': 0, 'maximum': 90, 'default': 90}] +``` + +Setting the values. +```python +>>> pleiades_queryables.set('processing_level', 'SENSOR') +>>> pleiades_queryables.set('max_snow_coverage', 0) +>>> pleiades_queryables.generate_cql_filter() +{'op': 'and', 'args': [{'op': '=', 'args': [{'property': 'processing_level'}, 'SENSOR']}, {'op': '=', 'args': [{'property': 'max_snow_coverage'}, 0]}]} +``` + +Searching the products: +```python +>>> connection.search_items(collection_id="PLEIADES", bbox=(1,2,3,4), filter=pleiades_queryables.generate_cql_filter()) +{'type': 'FeatureCollection', 'features': [{'id': 'c8a1f88d-89cf-4933-9118-45e9c1a5df20','type': 'Feature', 'stac_version': '0.9.0', 'geometry': {'type': 'Polygon', 'coordinates': [[[12.36555287044679, 41.94403289260048]], [12.36571746774068, 41.86399361096971], [12.60746759743069, 41.86372776276345], [12.60758647471871, 41.94379931812686], [12.36555287044679, 41.94403289260048]]}, 'properties': {'constellation': 'PHR', 'acquisitionDate': '2022-03-21T10:11:15.055Z', 'azimuthAngle': 179.9852862071639, 'cloudCover': 0, 'geometryCentroid': {'lat': 41.903935647240964, 'lon': 12.486569672582828}, 'processingLevel': 'SENSOR', 'sensorType': 'OPTICAL', 'spectralRange': 'VISIBLE'}, 'assets': {}, 'links': []}, ...], 'links': []} +``` + +Create and confirm and order: +```python +>>> connection.create_order(collection_id="PLEIADES", products=["c8a1f88d-89cf-4933-9118-45e9c1a5df20"]) +{'id': '40264b5-c3ae-46f4-a907-0f612d763d97', 'status': 'NOT_CONFIRMED', 'costs': 42} +>>> connection.confirm_order('40264b5-c3ae-46f4-a907-0f612d763d97') +{'id': '40264b5-c3ae-46f4-a907-0f612d763d97', 'status': 'RUNNING'} +``` + +When the order has finished, you can process the data as with a normal collection. + +```python +>>> order = connection.get_order(id="40264b5-c3ae-46f4-a907-0f612d763d97") +>>> order["order:status"] +'delivered' +>>> first_product = order["products"][0] +>>> pleiades_cube = connection.load_collection( + "PLEIADES", + spatial_extent=first_product["geometry"], + temporal_extent = [first_product["acquisitionDate"], None], + bands=["B1"], +) +``` \ No newline at end of file From 13b0542680d073855d374d79b4de493340670621 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Thu, 21 Jul 2022 13:52:05 +0200 Subject: [PATCH 062/103] * Make sure stac_version is listed as 1.0.0 * change order-id to order_id * change collection-id to collection_id * move links to additional pricing information to Collection links * fix invalid param collectionArray to collections * add extra links to STAC specs for filter * clarify POST orders/order_id endpoint if called multiple times * grammar and other fixes --- extension/commercial_data/README.md | 90 ++++++++++++----------------- 1 file changed, 36 insertions(+), 54 deletions(-) diff --git a/extension/commercial_data/README.md b/extension/commercial_data/README.md index 9758cc7e..7823841b 100644 --- a/extension/commercial_data/README.md +++ b/extension/commercial_data/README.md @@ -10,23 +10,23 @@ Extensions can not change or break existing behavior of the openEO API. ## Overview of the workflow -All the available datasets provided by a backend are listed on the `/collections` endpoint. The collections are normally freely accessible. This extensions adds capabilities for providing collections that are not free of charge and require purchasing data products that can thereupon be used in processing. Commercial data collections usually allow purchasing small subsets of the data (products), for example a single observation of an area. +All the available datasets provided by a backend are listed on the `GET /collections` endpoint. The collections are normally freely accessible. This extension adds capabilities for providing collections that are not free of charge and require purchasing data products that can thereupon be used in processing. Commercial data collections usually allow purchasing small subsets of the data (products), for example, a single observation of an area. Therefore, the client must have an ability to search the available products that match their desired criteria and inspect their metadata to decide whether the products should be purchased. The client can then create an order for the desired products. Because of the financial cost of purchasing the data, a separate endpoint for confirming the execution of that specific order should be implemented. -When the order is completed, the user should be able to access the data using the `collection-id` at the temporal and spatial location of the purchased products. +When the order is completed, the user should be able to access the data using the `collection_id` at the temporal and spatial location of the purchased products. ### Collection discovery -A backend should add general information about a commercial data collection to the `/collections` and `/collections/{collection-id}` endpoints, the same as with freely available collections. Only the metadata about the entire dataset needs to be provided, not about the specific data products that a user has already purchased. +A backend should add general information about a commercial data collection to the `/collections` and `/collections/{collection_id}` endpoints, the same as with freely available collections. Only the metadata about the entire dataset needs to be provided, not about the specific data products that a user has already purchased. -Commercial data collections are distinguished from freely available collections by including `"order:status": "orderable"` as specified in [STAC Order specification](https://github.com/stac-extensions/order). +Commercial data collections are distinguished from freely available collections by including `"order:status": "orderable"` as specified in the [STAC Order specification](https://github.com/stac-extensions/order). -Commercial data collections can include `order_parameters` field, if ordering supports additional parameters that specify how the products should be delivered. +Commercial data collections can include an `order_parameters` field if ordering supports additional parameters that specify how the products should be delivered. -Commercial data collections must also include a link to human-readable pricing information for searching and ordering the products. If searching the products is free it should be set to `null`. +Commercial data collections must also include human-readable pricing information for searching and ordering the products. If searching the products is free it should be set to `null`. If needed, references to additional information about pricing can be added to `links`. #### Example @@ -44,7 +44,6 @@ Commercial data collections must also include a link to human-readable pricing i "license": "proprietary", "providers": [...], "extent": {...}, - "links": [...], "cube:dimensions": {...}, "summaries": {...}, "assets": {...}, @@ -66,25 +65,26 @@ Commercial data collections must also include a link to human-readable pricing i "searching": null, "ordering": { "description": "Minimum area per order is 0.25 km2. The price is calculated based on a 6-months sliding window.", - "links": [ - { - "title": "Airbus Pleiades pricing", - "rel": "related", - "href": "https://www.sentinel-hub.com/pricing/#TPD_pricing" - } - ] } - } + }, + "links": [ + { + "title": "Airbus Pleiades pricing", + "rel": "related", + "href": "https://www.sentinel-hub.com/pricing/#tpd_pricing" + }, + ... + ] } ``` ### Filtering parameters discovery -Searching for products can support refining the search by filtering with general or collection-specific attributes. Backends should implement a top level `/queryables` endpoint for attributes available for all collections, and collection-specific attributes should be provided at `/collections/{collection-id}/queryables` according to [OGC Queryables specification](https://portal.ogc.org/files/96288#filter-queryables). +Searching for products can support refining the search by filtering with general or collection-specific attributes. Backends should implement a top level `/queryables` endpoint for attributes available for all collections, and collection-specific attributes should be provided at `/collections/{collection_id}/queryables` according to [OGC Queryables specification](https://portal.ogc.org/files/96288#filter-queryables) and [STAC Filter extension](https://github.com/radiantearth/stac-api-spec/tree/v1.0.0-rc.1/fragments/filter). #### Example -Example response from `GET collections/PLEIADES/queryables`: +Example response from `GET /collections/PLEIADES/queryables`: ```json { "$schema":"http://json-schema.org/draft-07/schema#", @@ -95,7 +95,7 @@ Example response from `GET collections/PLEIADES/queryables`: "properties":{ "processing_level":{ "title":"Processing level", - "description":"When searching, you will receive results from the full catalog as well as the Living Library, which holds images that have cloud cover under 30% and Incidence angle under 40°. If you want to search only Living Library results, you will need to filter using processingLevel. This value could be equal to SENSOR (images which meet Living Library criteria) and ALBUM (images that do not meeting Living Library criteria in terms of incidence angle and cloud cover).'", + "description":"Limit search to only Living Library images with 'SENSOR' or access all images with 'ALBUM'.", "type":"string", "enum":[ "SENSOR", @@ -123,8 +123,8 @@ Example response from `GET collections/PLEIADES/queryables`: ### Searching available products -Backends should implement top-level `GET /search` endpoint as specified in [STAC Item Search API specification](https://github.com/radiantearth/stac-api-spec/tree/v1.0.0-rc.1/item-search). This should include the [Filter Extension](https://github.com/radiantearth/stac-api-spec/tree/v1.0.0-rc.1/fragments/filter), which enables filtering the available products by attributes specified in `/queryables` and `collections/{collection-id}/queryables`. -The endpoint should return a list of `STAC Item`s that match the criteria. +Backends should implement the top-level `GET /search` endpoint as specified in the [STAC Item Search API specification](https://github.com/radiantearth/stac-api-spec/tree/v1.0.0-rc.1/item-search). This should include the [Filter Extension](https://github.com/radiantearth/stac-api-spec/tree/v1.0.0-rc.1/fragments/filter), which enables filtering the available products by attributes specified in `GET /queryables` and `GET /collections/{collection-id}/queryables`. +The endpoint returns a list of STAC Items that match the criteria. #### Example @@ -132,39 +132,19 @@ Example request payload to `GET /search` for `PLEIADES` products from "Living Li ```json { - "bbox": [ - 3, - 15, - 4, - 16 - ], - "datetime": [ - "01-01-2022", - "01-02-2022" - ], - "collectionsArray": [ - "PLEIADES" - ], + "bbox": [3, 15, 4, 16], + "datetime": ["01-01-2022", "01-02-2022"], + "collections": ["PLEIADES"], "filter": { "op": "and", "args": [ { "op": "=", - "args": [ - { - "property": "processing_level" - }, - "SENSOR" - ] + "args": [{"property": "processing_level"}, "SENSOR"] }, { "op": "=", - "args": [ - { - "property": "max_snow_coverage" - }, - 0 - ] + "args": [{"property": "max_snow_coverage"}, 0] } ] } @@ -177,8 +157,8 @@ Backends should implement the following endpoints: - `GET /orders`: Get a list of all created orders - `POST /orders`: Create an order -- `GET /orders/{order-id}`: Get full metadata of a specific order -- `POST /orders/{order-id}`: Confirm a created order +- `GET /orders/{order_id}`: Get full metadata of a specific order +- `POST /orders/{order_id}`: Confirm a created order #### `GET /orders` @@ -250,9 +230,9 @@ schema: Create an order for selected products. Order can contain some additional parameters that specify how the products should be delivered. For example, depending on the collection it might be possible to set the projection, resampling method, bit depth etc of the delivered data. -Backends should expose the available ordering parameters in `/collections/{collection-id}` in the `order_parameters` field, following the `process_parameters` schema of [`GET /service_types`](https://docs.openeo.cloud/api/#tag/Secondary-Services/operation/list-service-types). +Backends should expose the available ordering parameters in `/collections/{collection_id}` in the `order_parameters` field, following the `process_parameters` schema of [`GET /service_types`](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Secondary-Services/operation/list-service-types). -The request should return the `order-id`, current status and the costs of the order. +The request should return the `order_id`, current status and the costs of the order. ```yaml schema: @@ -279,7 +259,7 @@ schema: description: Value of the order parameter to be used in the request. ``` -#### `GET /orders/{order-id}` +#### `GET /orders/{order_id}` Get full metadata of the order. The item should follow the [STAC Order Extension](https://github.com/stac-extensions/order), but extended with the spatial and temporal extent information and other metadata about the products. @@ -291,7 +271,7 @@ Get full metadata of the order. The item should follow the [STAC Order Extension "products": [ { "type": "Feature", - "stac_version": "0.9.0", + "stac_version": "1.0.0", "id": "c8a1f88d-89cf-4933-9118-45e9c1a5df20", "geometry": { "type": "Polygon", @@ -350,12 +330,14 @@ Get full metadata of the order. The item should follow the [STAC Order Extension } ``` -#### `POST /orders/{order-id}` +#### `POST /orders/{order_id}` -When an order is created, the data isn't yet ordered from the commercial data provider. The client should explicitly confirm the order whereupon the order is executed, the costs are deducted from the client's account and the data is ingested in the associated collection. The `order:status` changes to `ordered`. +When an order is created, the data isn't yet ordered from the commercial data provider. The client should explicitly confirm the order whereupon the order is executed, the costs are deducted from the client's account and the data is ingested in the associated collection. The `order:status` changes from `orderable` to `ordered`. If the user doesn't have sufficient funds, the endpoint should return an error and `order:status` should not change. +This endpoint only has an effect if `order:status` is `orderable` + ### Payment Payment should be done in the currency used by the backend, listed at `GET /` under `billing`. When an order is created, the backend should return the full final cost of the order. @@ -389,7 +371,7 @@ Setting the values. Searching the products: ```python >>> connection.search_items(collection_id="PLEIADES", bbox=(1,2,3,4), filter=pleiades_queryables.generate_cql_filter()) -{'type': 'FeatureCollection', 'features': [{'id': 'c8a1f88d-89cf-4933-9118-45e9c1a5df20','type': 'Feature', 'stac_version': '0.9.0', 'geometry': {'type': 'Polygon', 'coordinates': [[[12.36555287044679, 41.94403289260048]], [12.36571746774068, 41.86399361096971], [12.60746759743069, 41.86372776276345], [12.60758647471871, 41.94379931812686], [12.36555287044679, 41.94403289260048]]}, 'properties': {'constellation': 'PHR', 'acquisitionDate': '2022-03-21T10:11:15.055Z', 'azimuthAngle': 179.9852862071639, 'cloudCover': 0, 'geometryCentroid': {'lat': 41.903935647240964, 'lon': 12.486569672582828}, 'processingLevel': 'SENSOR', 'sensorType': 'OPTICAL', 'spectralRange': 'VISIBLE'}, 'assets': {}, 'links': []}, ...], 'links': []} +{'type': 'FeatureCollection', 'features': [{'id': 'c8a1f88d-89cf-4933-9118-45e9c1a5df20','type': 'Feature', 'stac_version': '1.0.0', 'geometry': {'type': 'Polygon', 'coordinates': [[[12.36555287044679, 41.94403289260048]], [12.36571746774068, 41.86399361096971], [12.60746759743069, 41.86372776276345], [12.60758647471871, 41.94379931812686], [12.36555287044679, 41.94403289260048]]}, 'properties': {'constellation': 'PHR', 'acquisitionDate': '2022-03-21T10:11:15.055Z', 'azimuthAngle': 179.9852862071639, 'cloudCover': 0, 'geometryCentroid': {'lat': 41.903935647240964, 'lon': 12.486569672582828}, 'processingLevel': 'SENSOR', 'sensorType': 'OPTICAL', 'spectralRange': 'VISIBLE'}, 'assets': {}, 'links': []}, ...], 'links': []} ``` Create and confirm and order: From fab174c05eaa5a7dc95b73320fbd7f44b2487429 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Fri, 29 Jul 2022 12:39:29 +0200 Subject: [PATCH 063/103] Make example results more STAC-compliant. Fix schema. Use relation type `pricing_info`. --- extension/commercial_data/README.md | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/extension/commercial_data/README.md b/extension/commercial_data/README.md index 7823841b..af4d356e 100644 --- a/extension/commercial_data/README.md +++ b/extension/commercial_data/README.md @@ -26,7 +26,7 @@ Commercial data collections are distinguished from freely available collections Commercial data collections can include an `order_parameters` field if ordering supports additional parameters that specify how the products should be delivered. -Commercial data collections must also include human-readable pricing information for searching and ordering the products. If searching the products is free it should be set to `null`. If needed, references to additional information about pricing can be added to `links`. +Commercial data collections must also include human-readable pricing information for searching and ordering the products. If searching the products is free it should be set to `null`. If needed, references to additional information about pricing can be added to `links` with the relation type `pricing_info`. #### Example @@ -53,12 +53,12 @@ Commercial data collections must also include human-readable pricing information "name": "sample_type", "description": "sample type of the output raster.", "schema": { - "type": "string" + "type": "string", + "enum": [ + "UINT16", + "UINT8" + ] }, - "enum": [ - "UINT16", - "UINT8" - ] }, ], "pricing": { @@ -70,7 +70,7 @@ Commercial data collections must also include human-readable pricing information "links": [ { "title": "Airbus Pleiades pricing", - "rel": "related", + "rel": "pricing_info", "href": "https://www.sentinel-hub.com/pricing/#tpd_pricing" }, ... @@ -272,6 +272,7 @@ Get full metadata of the order. The item should follow the [STAC Order Extension { "type": "Feature", "stac_version": "1.0.0", + "stac_extensions": ["proj", "eo", "view", "processing"], "id": "c8a1f88d-89cf-4933-9118-45e9c1a5df20", "geometry": { "type": "Polygon", @@ -302,14 +303,14 @@ Get full metadata of the order. The item should follow the [STAC Order Extension }, "properties": { "constellation": "PHR", - "acquisitionDate": "2022-03-21T10:11:15.055Z", - "azimuthAngle": 179.9852862071639, - "cloudCover": 0, - "geometryCentroid": { + "datetime": "2022-03-21T10:11:15.055Z", + "view:azimuth": 179.9852862071639, + "eo:cloud_cover": 0, + "proj:centroid": { "lat": 41.903935647240964, "lon": 12.486569672582828 }, - "processingLevel": "SENSOR", + "processing:level": "SENSOR", "sensorType": "OPTICAL", "spectralRange": "VISIBLE" }, @@ -371,7 +372,7 @@ Setting the values. Searching the products: ```python >>> connection.search_items(collection_id="PLEIADES", bbox=(1,2,3,4), filter=pleiades_queryables.generate_cql_filter()) -{'type': 'FeatureCollection', 'features': [{'id': 'c8a1f88d-89cf-4933-9118-45e9c1a5df20','type': 'Feature', 'stac_version': '1.0.0', 'geometry': {'type': 'Polygon', 'coordinates': [[[12.36555287044679, 41.94403289260048]], [12.36571746774068, 41.86399361096971], [12.60746759743069, 41.86372776276345], [12.60758647471871, 41.94379931812686], [12.36555287044679, 41.94403289260048]]}, 'properties': {'constellation': 'PHR', 'acquisitionDate': '2022-03-21T10:11:15.055Z', 'azimuthAngle': 179.9852862071639, 'cloudCover': 0, 'geometryCentroid': {'lat': 41.903935647240964, 'lon': 12.486569672582828}, 'processingLevel': 'SENSOR', 'sensorType': 'OPTICAL', 'spectralRange': 'VISIBLE'}, 'assets': {}, 'links': []}, ...], 'links': []} +{'type': 'FeatureCollection', 'features': [{'id': 'c8a1f88d-89cf-4933-9118-45e9c1a5df20','type': 'Feature', 'stac_version': '1.0.0', "stac_extensions": ["proj", "eo", "view", "processing"], 'geometry': {'type': 'Polygon', 'coordinates': [[[12.36555287044679, 41.94403289260048]], [12.36571746774068, 41.86399361096971], [12.60746759743069, 41.86372776276345], [12.60758647471871, 41.94379931812686], [12.36555287044679, 41.94403289260048]]}, 'properties': {'constellation': 'PHR', 'datetime': '2022-03-21T10:11:15.055Z', 'view:azimuth': 179.9852862071639, 'eo:cloud_cover': 0, 'proj:centroid': {'lat': 41.903935647240964, 'lon': 12.486569672582828}, 'processing:level': 'SENSOR', 'sensorType': 'OPTICAL', 'spectralRange': 'VISIBLE'}, 'assets': {}, 'links': []}, ...], 'links': []} ``` Create and confirm and order: @@ -392,7 +393,7 @@ When the order has finished, you can process the data as with a normal collectio >>> pleiades_cube = connection.load_collection( "PLEIADES", spatial_extent=first_product["geometry"], - temporal_extent = [first_product["acquisitionDate"], None], + temporal_extent = [first_product["datetime"], None], bands=["B1"], ) ``` \ No newline at end of file From b3516aa19b01e6229339b208627319fcd1921945 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Mon, 1 Aug 2022 16:45:00 +0200 Subject: [PATCH 064/103] Use correct identifier for STAC extension schemas, clarify products should be aligned with STAC as much as possible. --- extension/commercial_data/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/extension/commercial_data/README.md b/extension/commercial_data/README.md index af4d356e..e2ca100c 100644 --- a/extension/commercial_data/README.md +++ b/extension/commercial_data/README.md @@ -124,7 +124,7 @@ Example response from `GET /collections/PLEIADES/queryables`: ### Searching available products Backends should implement the top-level `GET /search` endpoint as specified in the [STAC Item Search API specification](https://github.com/radiantearth/stac-api-spec/tree/v1.0.0-rc.1/item-search). This should include the [Filter Extension](https://github.com/radiantearth/stac-api-spec/tree/v1.0.0-rc.1/fragments/filter), which enables filtering the available products by attributes specified in `GET /queryables` and `GET /collections/{collection-id}/queryables`. -The endpoint returns a list of STAC Items that match the criteria. +The endpoint returns a list of [STAC Items](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md) that match the criteria. The products should be aligned with the STAC specification, utilising the existing [STAC extensions](https://github.com/radiantearth/stac-spec/blob/v1.0.0/extensions/README.md) as much as possible, and trying avoiding custom attributes, if a generally accepted definition does not exist. #### Example @@ -272,7 +272,7 @@ Get full metadata of the order. The item should follow the [STAC Order Extension { "type": "Feature", "stac_version": "1.0.0", - "stac_extensions": ["proj", "eo", "view", "processing"], + "stac_extensions": ["https://stac-extensions.github.io/projection/v1.0.0/schema.json", "https://stac-extensions.github.io/eo/v1.0.0/schema.json", "https://stac-extensions.github.io/view/v1.0.0/schema.json", "https://stac-extensions.github.io/processing/v1.1.0/schema.json"], "id": "c8a1f88d-89cf-4933-9118-45e9c1a5df20", "geometry": { "type": "Polygon", @@ -372,7 +372,7 @@ Setting the values. Searching the products: ```python >>> connection.search_items(collection_id="PLEIADES", bbox=(1,2,3,4), filter=pleiades_queryables.generate_cql_filter()) -{'type': 'FeatureCollection', 'features': [{'id': 'c8a1f88d-89cf-4933-9118-45e9c1a5df20','type': 'Feature', 'stac_version': '1.0.0', "stac_extensions": ["proj", "eo", "view", "processing"], 'geometry': {'type': 'Polygon', 'coordinates': [[[12.36555287044679, 41.94403289260048]], [12.36571746774068, 41.86399361096971], [12.60746759743069, 41.86372776276345], [12.60758647471871, 41.94379931812686], [12.36555287044679, 41.94403289260048]]}, 'properties': {'constellation': 'PHR', 'datetime': '2022-03-21T10:11:15.055Z', 'view:azimuth': 179.9852862071639, 'eo:cloud_cover': 0, 'proj:centroid': {'lat': 41.903935647240964, 'lon': 12.486569672582828}, 'processing:level': 'SENSOR', 'sensorType': 'OPTICAL', 'spectralRange': 'VISIBLE'}, 'assets': {}, 'links': []}, ...], 'links': []} +{'type': 'FeatureCollection', 'features': [{'id': 'c8a1f88d-89cf-4933-9118-45e9c1a5df20','type': 'Feature', 'stac_version': '1.0.0', "stac_extensions": ["https://stac-extensions.github.io/projection/v1.0.0/schema.json", "https://stac-extensions.github.io/eo/v1.0.0/schema.json", "https://stac-extensions.github.io/view/v1.0.0/schema.json", "https://stac-extensions.github.io/processing/v1.1.0/schema.json"], 'geometry': {'type': 'Polygon', 'coordinates': [[[12.36555287044679, 41.94403289260048]], [12.36571746774068, 41.86399361096971], [12.60746759743069, 41.86372776276345], [12.60758647471871, 41.94379931812686], [12.36555287044679, 41.94403289260048]]}, 'properties': {'constellation': 'PHR', 'datetime': '2022-03-21T10:11:15.055Z', 'view:azimuth': 179.9852862071639, 'eo:cloud_cover': 0, 'proj:centroid': {'lat': 41.903935647240964, 'lon': 12.486569672582828}, 'processing:level': 'SENSOR', 'sensorType': 'OPTICAL', 'spectralRange': 'VISIBLE'}, 'assets': {}, 'links': []}, ...], 'links': []} ``` Create and confirm and order: From 7724b184f09fe9a0b285625c0f1dc28e685b2a7e Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 1 Aug 2022 17:58:21 +0200 Subject: [PATCH 065/103] Several improvements for estimates, budget and costs #443 (#458) --- CHANGELOG.md | 4 ++++ errors.json | 8 ++++++++ openapi.yaml | 30 +++++++++++++++++++++++------- 3 files changed, 35 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d987a1ce..b73f3b8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Updated STAC specification examples and references to v1.0.0. - `cube:dimensions`: `reference_system` is allowed to be PROJJSON, too. - Relaxed requirement that unsupported endpoints must return HTTP status code 501. Instead also HTTP status code 404 can be used (and is regularly used in practice). [#415](https://github.com/Open-EO/openeo-api/issues/415) +- Minimum value for `costs` and `budget` is 0. +- `GET /jobs/{job_id}/estimate`: If a batch job can't be estimated reliably, a `EstimateComplexity` error should be returned. [#443](https://github.com/Open-EO/openeo-api/issues/443) ### Fixed @@ -43,6 +45,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /`: Removed the superfluous default value for `currency`. [#423](https://github.com/Open-EO/openeo-api/issues/423) - Reflect that the `debug` process has been renamed to `inspect`. - `GET /credentials/oidc`: Clarify that clients may add additional scopes +- `GET /me`: Clarify the behavior of the field `budget`. +- `GET /jobs/{job_id}/estimate`: Don't require that the costs are the upper limit. Services may specify the costs more freely depending on their terms of service. - Clarified uniqueness constraints for identifiers. [#449](https://github.com/Open-EO/openeo-api/issues/449) [#454](https://github.com/Open-EO/openeo-api/issues/454) ## [1.1.0] - 2021-05-17 diff --git a/errors.json b/errors.json index 1e2f3239..7288ba6b 100644 --- a/errors.json +++ b/errors.json @@ -66,6 +66,14 @@ "Data Processing" ] }, + "EstimateComplexity": { + "description": "The process is too complex to calculate an estimate, e.g. due to a UDF or other processes that are complex to estimate costs reliably.", + "message": "The process is too complex to calculate an estimate.", + "http": 500, + "tags": [ + "Batch Jobs" + ] + }, "NoDataForUpdate": { "description": "For PATCH requests: No valid data specified at all.", "message": "No data specified to be updated.", diff --git a/openapi.yaml b/openapi.yaml index a03835b9..db57b0bc 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2973,12 +2973,20 @@ paths: description: >- Calculates an estimate for a batch job. Back-ends can decide to either calculate the duration, the costs, the size or a combination of - them. This MUST be the upper limit of the incurring costs. Clients can - be charged less than specified, but never more. Back-end providers MAY - specify an expiry time for the estimate. Starting to process data - afterwards MAY be charged at a higher cost. Costs MAY NOT include - downloading costs. This can be indicated with the `downloads_included` - flag. + them. + + Back-end providers MAY specify an expiry time for the estimate. + Starting to process data afterwards MAY be charged at a higher cost. + Costs do often not include download costs. Whether download costs are included or not can be indicated explicitly with the + `downloads_included` flag. + + The estimate SHOULD be the upper limit of the costs, but back-end are + free to use the field according to their terms of service. + + For some batch jobs it is not (easily) possible to estimate the costs + reliably, e.g. if a UDF or ML model is part of the process. + In this case, the server SHOULD return a `EstimateComplexity` error + with HTTP status code 500. tags: - Data Processing - Batch Jobs @@ -3565,7 +3573,12 @@ paths: to the user. example: 1073741824 budget: - $ref: '#/components/schemas/budget' + type: number + nullable: true + description: |- + The remaining budget a user has available. + The value MUST be specified in the currency of the back-end. + The value SHOULD be set to `null` if no explicit limit applies. links: description: |- Links related to the user profile, e.g. where payments @@ -4095,11 +4108,13 @@ components: currency the back-end is working with. The currency can be retrieved by calling `GET /`. If no currency is set, this field MUST be `null`. type: number + minimum: 0 example: 12.98 nullable: true default: null budget: type: number + minimum: 0 nullable: true description: |- Maximum amount of costs the request is allowed to produce. @@ -4110,6 +4125,7 @@ components: default: null budget_update: type: number + minimum: 0 nullable: true description: |- Maximum amount of costs the request is allowed to produce. From 24956b34cc87cde89d67427502b0e3d7b0d340da Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 2 Aug 2022 12:31:55 +0200 Subject: [PATCH 066/103] Clarify the message field in logs (#457) * Clarified the formatting of the `message` property. #455 --- CHANGELOG.md | 5 +++-- openapi.yaml | 9 ++++++++- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b73f3b8c..b680ebb7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,12 +42,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Clarify that the relation type `version-history` should include `/.well-known/openeo` in the URL. - Clarify that clients should (re-)request capabilities and discovery endpoints with token if available and supported. [#416](https://github.com/Open-EO/openeo-api/issues/416) - Clarify the fields `plan` (for processing requests) and `billing_plan` (in `GET /` and `GET /me`). [#425](https://github.com/Open-EO/openeo-api/issues/425) [#426](https://github.com/Open-EO/openeo-api/issues/426) -- `GET /`: Removed the superfluous default value for `currency`. [#423](https://github.com/Open-EO/openeo-api/issues/423) - Reflect that the `debug` process has been renamed to `inspect`. +- Clarified uniqueness constraints for identifiers. [#449](https://github.com/Open-EO/openeo-api/issues/449) [#454](https://github.com/Open-EO/openeo-api/issues/454) +- `GET /`: Removed the superfluous default value for `currency`. [#423](https://github.com/Open-EO/openeo-api/issues/423) - `GET /credentials/oidc`: Clarify that clients may add additional scopes - `GET /me`: Clarify the behavior of the field `budget`. +- `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs` and `POST /result`: Clarified the formatting of the `message` property. [#455](https://github.com/Open-EO/openeo-api/issues/455) - `GET /jobs/{job_id}/estimate`: Don't require that the costs are the upper limit. Services may specify the costs more freely depending on their terms of service. -- Clarified uniqueness constraints for identifiers. [#449](https://github.com/Open-EO/openeo-api/issues/449) [#454](https://github.com/Open-EO/openeo-api/issues/454) ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index db57b0bc..ebad1bb1 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -6053,7 +6053,14 @@ components: $ref: '#/components/schemas/log_level' message: type: string - description: A message explaining the log entry. + description: >- + A concise message explaining the log entry. + + Messages do *not* explicitly support [CommonMark 0.29](http://commonmark.org/) + syntax as other descriptive fields in the openEO API do, + but the messages MAY contain line breaks or indentation. + + It is NOT RECOMMENDED to add stacktraces to the `message`. example: >- Can't load the UDF file from the URL `https://example.com/invalid/file.txt`. Server responded with error 404. From 3c153984ba25ccff737689a7afc53e872aeba7ea Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 2 Aug 2022 15:48:08 +0200 Subject: [PATCH 067/103] Update redoc-cli --- package.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/package.json b/package.json index 6019eb94..a256cbc8 100644 --- a/package.json +++ b/package.json @@ -14,11 +14,11 @@ }, "devDependencies": { "@stoplight/spectral": "^5.9.1", - "redoc-cli": "0.10.1" + "redoc-cli": "^0.13.17" }, "scripts": { "start": "redoc-cli serve openapi.yaml --watch --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true", - "build": "redoc-cli bundle openapi.yaml -o redoc.html --title \"openEO API\" --cdn --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true", + "build": "redoc-cli build openapi.yaml -o redoc.html --title \"openEO API\" --cdn --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true", "test": "spectral lint openapi.yaml" } } From fe93a606a74336c3dd9e3dcc6a1b090b7e0df9a7 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Thu, 4 Aug 2022 16:12:42 +0200 Subject: [PATCH 068/103] * Change response for POST /orders to be more similar to other openEO creation endpoints * Add optional DELETE /orders/{order_id} * Add and to order information and clarify how the data can be accessed * Remove inlined product metadata and clarify it's optional to link it. * Add optional endpoint for product metadata and clarify the relation type to be used linking the associated orders. * Update Python API example to align with changes, and simplify/fix part of the workflow. --- extension/commercial_data/README.md | 117 ++++++++++------------------ 1 file changed, 41 insertions(+), 76 deletions(-) diff --git a/extension/commercial_data/README.md b/extension/commercial_data/README.md index e2ca100c..55340117 100644 --- a/extension/commercial_data/README.md +++ b/extension/commercial_data/README.md @@ -14,9 +14,9 @@ All the available datasets provided by a backend are listed on the `GET /collect Therefore, the client must have an ability to search the available products that match their desired criteria and inspect their metadata to decide whether the products should be purchased. -The client can then create an order for the desired products. Because of the financial cost of purchasing the data, a separate endpoint for confirming the execution of that specific order should be implemented. +The client can then create an order for the desired products from a orderable collection. Because of the financial cost of purchasing the data, a separate endpoint for confirming the execution of that specific order should be implemented. -When the order is completed, the user should be able to access the data using the `collection_id` at the temporal and spatial location of the purchased products. +When the order is completed, the data is ingested in a collection and its ID is available at `/orders/{order_id}` as `target_collection_id`. The user should be able to access the data using the `target_collection_id` at the temporal and spatial location of the purchased products. ### Collection discovery @@ -160,6 +160,9 @@ Backends should implement the following endpoints: - `GET /orders/{order_id}`: Get full metadata of a specific order - `POST /orders/{order_id}`: Confirm a created order +Optionally, they can also implement: +- `DELETE /orders/{order_id}`: Delete an order + #### `GET /orders` Lists all created orders, regardless of the status or the collection. The items should follow the [STAC Order Extension](https://github.com/stac-extensions/order). @@ -232,7 +235,7 @@ Create an order for selected products. Order can contain some additional paramet Backends should expose the available ordering parameters in `/collections/{collection_id}` in the `order_parameters` field, following the `process_parameters` schema of [`GET /service_types`](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Secondary-Services/operation/list-service-types). -The request should return the `order_id`, current status and the costs of the order. +Returns `Location` and `OpenEO-Identifier` header with the link to the detailed information about the order. ```yaml schema: @@ -261,72 +264,23 @@ schema: #### `GET /orders/{order_id}` -Get full metadata of the order. The item should follow the [STAC Order Extension](https://github.com/stac-extensions/order), but extended with the spatial and temporal extent information and other metadata about the products. +Get full metadata of the order. The item should follow the [STAC Order Extension](https://github.com/stac-extensions/order), but extended with the source collection id, target collection id, list of ordered product, order parameters and optionally links to products metadata. + +`source_collection_id` is the id of the orderable collection the products were ordered from. `target_collection_id` is the id of the collection to be used in `load_collection` to load into the DataCube and contains the ordered products. `source_collection_id` and `target_collection_id` can be the same or different, it's up to the backend if the products are ingested into a new collection or the existing orderable collection. + +Additionally, if the backend implements order-specific collections, it can provide access to the union of all purchased products when using the ID of the original orderable collection. + +Backends can optionally link to the spatial and temporal extent information and other metadata about the products, preferrably by implementing `/collections/{collection_id}/items/{item_id}` as described [here](#product-metadata). ```json { "order:id": "40264b5-c3ae-46f4-a907-0f612d763d97", "order:status": "delivered", "order:date": "2017-01-01T09:32:12Z", - "products": [ - { - "type": "Feature", - "stac_version": "1.0.0", - "stac_extensions": ["https://stac-extensions.github.io/projection/v1.0.0/schema.json", "https://stac-extensions.github.io/eo/v1.0.0/schema.json", "https://stac-extensions.github.io/view/v1.0.0/schema.json", "https://stac-extensions.github.io/processing/v1.1.0/schema.json"], - "id": "c8a1f88d-89cf-4933-9118-45e9c1a5df20", - "geometry": { - "type": "Polygon", - "coordinates": [ - [ - [ - 12.36555287044679, - 41.94403289260048 - ] - ], - [ - 12.36571746774068, - 41.86399361096971 - ], - [ - 12.60746759743069, - 41.86372776276345 - ], - [ - 12.60758647471871, - 41.94379931812686 - ], - [ - 12.36555287044679, - 41.94403289260048 - ] - ] - }, - "properties": { - "constellation": "PHR", - "datetime": "2022-03-21T10:11:15.055Z", - "view:azimuth": 179.9852862071639, - "eo:cloud_cover": 0, - "proj:centroid": { - "lat": 41.903935647240964, - "lon": 12.486569672582828 - }, - "processing:level": "SENSOR", - "sensorType": "OPTICAL", - "spectralRange": "VISIBLE" - }, - "assets": {}, - "links": [] - } - ], - "parameters": { - "bounds": [ - 3, - 15, - 4, - 16 - ] - }, - "collection_id": "PLEAIDES", + "products": [ "c8a1f88d-89cf-4933-9118-45e9c1a5df20"], + "parameters": { "resamplingKernel": "NN" }, + "source_collection_id": "PLEAIDES", + "target_collection_id": "PLEAIDES", "costs": 42 } ``` @@ -337,7 +291,20 @@ When an order is created, the data isn't yet ordered from the commercial data pr If the user doesn't have sufficient funds, the endpoint should return an error and `order:status` should not change. -This endpoint only has an effect if `order:status` is `orderable` +This endpoint only has an effect if `order:status` is `orderable`. + + +#### `DELETE /orders/{order_id}` + +Optional, removes the order entry and corresponding ordered data. + +### Product metadata + +The backend can provide full product metadata as STAC Items following [STAC API Features specification](https://github.com/radiantearth/stac-api-spec/tree/main/ogcapi-features). This requires implementing two additional endpoints, `/collections/{collection_id}/items` and `/collections/{collection_id}/items/{item_id}`. + +`/collections/{collection_id}/items/{item_id}` may return an error if the data has not been ingested yet. + +Items should contain links to the respective orders that made them available using relation type `parent_order`. ### Payment @@ -377,23 +344,21 @@ Searching the products: Create and confirm and order: ```python ->>> connection.create_order(collection_id="PLEIADES", products=["c8a1f88d-89cf-4933-9118-45e9c1a5df20"]) -{'id': '40264b5-c3ae-46f4-a907-0f612d763d97', 'status': 'NOT_CONFIRMED', 'costs': 42} ->>> connection.confirm_order('40264b5-c3ae-46f4-a907-0f612d763d97') -{'id': '40264b5-c3ae-46f4-a907-0f612d763d97', 'status': 'RUNNING'} +>>> order = connection.create_order(collection_id="PLEIADES", products=["c8a1f88d-89cf-4933-9118-45e9c1a5df20"]) +>>> order.costs +42 +>>> order.status +'orderable' +>>> order.confirm_order() +>>> order.status +'ordered' ``` When the order has finished, you can process the data as with a normal collection. ```python >>> order = connection.get_order(id="40264b5-c3ae-46f4-a907-0f612d763d97") ->>> order["order:status"] +>>> order.status 'delivered' ->>> first_product = order["products"][0] ->>> pleiades_cube = connection.load_collection( - "PLEIADES", - spatial_extent=first_product["geometry"], - temporal_extent = [first_product["datetime"], None], - bands=["B1"], -) +>>> pleiades_cube = connection.load_collection(order.target_collection_id) ``` \ No newline at end of file From f21193f85750b01f9a9e2a697c0661d03cc45d8f Mon Sep 17 00:00:00 2001 From: jjezersek Date: Thu, 4 Aug 2022 18:43:00 +0200 Subject: [PATCH 069/103] Fix Order schema. Change order id reference to something more important. Clarify the relation type to be used when linking to ordered products metadata. --- extension/commercial_data/README.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/extension/commercial_data/README.md b/extension/commercial_data/README.md index 55340117..9eb56e6f 100644 --- a/extension/commercial_data/README.md +++ b/extension/commercial_data/README.md @@ -185,7 +185,8 @@ schema: - order:id - order:status - order:date - - collection_id + - source_collection_id + - target_collection_id - costs properties: order:id: @@ -220,9 +221,12 @@ schema: format: date-time description: The validity time of the order. Formatted as a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - collection_id: + source_collection_id: type: string - description: Unique identifier of the collection. + description: Unique identifier of the source collection. + target_collection_id: + type: string + description: Unique identifier of the target collection. Can be the same as `source_collection_id`. costs: $ref: 'https://raw.githubusercontent.com/Open-EO/openeo-api/1.0.0/openapi.yaml#/components/schemas/money' links: @@ -253,7 +257,7 @@ schema: type: array description: Array of IDs of products to order. items: - $ref: 'https://raw.githubusercontent.com/Open-EO/openeo-api/1.0.0/openapi.yaml#/components/schemas/order_id' + $ref: '#/components/schemas/order_id' parameters: type: object description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. @@ -270,7 +274,7 @@ Get full metadata of the order. The item should follow the [STAC Order Extension Additionally, if the backend implements order-specific collections, it can provide access to the union of all purchased products when using the ID of the original orderable collection. -Backends can optionally link to the spatial and temporal extent information and other metadata about the products, preferrably by implementing `/collections/{collection_id}/items/{item_id}` as described [here](#product-metadata). +Backends can optionally link to the spatial and temporal extent information and other metadata about the products, preferrably by implementing `/collections/{collection_id}/items/{item_id}` as described [here](#product-metadata), using relation type `ordered_product_metadata`. ```json { From 5254f231979b9c626fe9dd9654a75acc4c72b290 Mon Sep 17 00:00:00 2001 From: jjezersek Date: Thu, 4 Aug 2022 18:47:30 +0200 Subject: [PATCH 070/103] Add OpenAPI document for the /orders and /orders/{order_id} endpoints. --- extension/commercial_data/openapi.yaml | 328 +++++++++++++++++++++++++ 1 file changed, 328 insertions(+) create mode 100644 extension/commercial_data/openapi.yaml diff --git a/extension/commercial_data/openapi.yaml b/extension/commercial_data/openapi.yaml new file mode 100644 index 00000000..4fea12a9 --- /dev/null +++ b/extension/commercial_data/openapi.yaml @@ -0,0 +1,328 @@ +openapi: 3.0.2 +info: + title: openEO Commercial data API extension + version: 0.1.0 + description: + $ref: README.md + contact: + name: openEO Consortium + url: 'https://openeo.org' + email: openeo.psc@uni-muenster.de + license: + name: Apache 2.0 + url: 'http://www.apache.org/licenses/LICENSE-2.0.html' +tags: + - name: Orders + description: Management of commercial data orders +servers: + - url: 'https://localhost/api/{version}' + description: >- + The URL of the API MAY freely be chosen by the back-end providers. The + path, including API versioning, is a *recommendation* only. Nevertheless, + all servers MUST support HTTPS as the authentication methods are not + secure with HTTP only! + variables: + version: + default: v1.0 + description: >- + API versioning is RECOMMENDED. As the openEO API is following + [SemVer](https://semver.org/) only the MAJOR part of the stable + version numbers (i.e. versions >= 1.0.0) SHOULD be used for API + versioning in the URL. The reason is that backward-incompatible + changes are usually introduced by major changes. Therefore, the + version number in the URL MUST not be used by the clients to detect + the version number of the API. Use the version number returned from + `GET /` instead. +paths: + /orders: + get: + summary: List all orders created by the user + operationId: list-orders + description: |- + Lists all created orders, regardless of the status or the collection. + tags: + - Orders + security: + - Bearer: [] + parameters: + - $ref: '../../openapi.yaml#/components/parameters/pagination_limit' + responses: + '200': + description: List of orders and related links. + content: + application/json: + schema: + title: All Orders + type: object + required: + - orders + - links + properties: + orders: + type: array + items: + $ref: '#/components/schemas/order' + links: + $ref: '../../openapi.yaml#/components/schemas/links_pagination' + 4XX: + $ref: '../../openapi.yaml#/components/responses/client_error_auth' + 5XX: + $ref: '../../openapi.yaml#/components/responses/server_error' + post: + summary: Create order request + operationId: create-order + description: |- + Create an order for selected products. Order can contain some additional parameters that specify how the products should be delivered. For example, depending on the collection it might be possible to set the projection, resampling method, bit depth etc of the delivered data. + + Backends should expose the available ordering parameters in `/collections/{collection_id}` in the `order_parameters` field, following the `process_parameters` schema of [`GET /service_types`](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Secondary-Services/operation/list-service-types). + + Returns `Location` and `OpenEO-Identifier` header with the link to the detailed information about the order. + tags: + - Orders + security: + - Bearer: [] + requestBody: + required: true + content: + application/json: + schema: + type: object + required: + - collection_id + - products + - parameters + properties: + collection_id: + type: string + description: Unique identifier of the collection. + products: + type: array + description: Array of IDs of products to order. + items: + $ref: '#/components/schemas/order_id' + parameters: + type: object + description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. + additionalProperties: + x-additionalPropertiesName: Order Parameter Name + description: Value of the order parameter to be used in the request. + example: + resamplingKernel: 'NN' + responses: + '201': + description: The order has been created successfully. + headers: + Location: + required: true + schema: + description: |- + Absolute URL to the newly created order. + + The URL points to the metadata endpoint + `GET /orders/{order_id}` with the `{order_id}` being the + unique identifier (ID) of the created order. + format: uri + type: string + example: 'https://example.openeo.org/api/orders/40264b5-c3ae-46f4-a907-0f612d763d97' + OpenEO-Identifier: + required: true + schema: + $ref: '#/components/schemas/order_id' + 4XX: + $ref: '../../openapi.yaml#/components/responses/client_error_auth' + 5XX: + $ref: '../../openapi.yaml#/components/responses/server_error' + '/orders/{order_id}': + parameters: + - $ref: '#/components/parameters/order_id' + get: + summary: Full metadata for an order + operationId: describe-order + description: |- + Get full metadata of the order. The item should follow the [STAC Order Extension](https://github.com/stac-extensions/order), but extended with additional information. Backends can optionally link to the spatial and temporal extent information and other metadata about the products. + tags: + - Orders + security: + - Bearer: [] + responses: + '200': + description: Full order information. + content: + application/json: + schema: + allOf: + - $ref: '#/components/schemas/order' + - type: object + required: + - products + properties: + products: + type: array + description: List of IDs of ordered products. + items: + $ref: '#/components/schemas/product_id' + products_metadata: + type: array + description: Array of links to metadata for ordered products listed in `products`. + items: + title: Link + description: >- + A link to the metadata for the ordered product. + type: object + required: + - href + - rel + properties: + rel: + type: string + enum: ["ordered_product_metadata"] + href: + type: string + description: The value MUST be a valid URL, preferably pointing to `/collections/{collection_id}/items/{product_id}`. + format: uri + example: 'https://example.openeo.org/collections/PLEIADES/items/c8a1f88d-89cf-4933-9118-45e9c1a5df20' + type: + type: string + description: >- + The value MUST be a string that hints at the format used to + represent data at the provided URI, preferably a media (MIME) type. + example: text/html + title: + type: string + description: Used as a human-readable label for a link. + example: openEO + + 4XX: + $ref: '../../openapi.yaml#/components/responses/client_error_auth' + 5XX: + $ref: '../../openapi.yaml#/components/responses/server_error' + post: + summary: Confirm an order + operationId: confirm-order + description: |- + Confirm the order to execute the purchase and ingest into the target collection. The `order:status` changes from `orderable` to `ordered`. + + If the user doesn't have sufficient funds, the endpoint should return an error and `order:status` should not change. + + This endpoint only has an effect if `order:status` is `orderable`. + tags: + - Orders + security: + - Bearer: [] + responses: + '202': + description: The order has been confirmed successfully. + 4XX: + $ref: '../../openapi.yaml#/components/responses/client_error_auth' + 5XX: + $ref: '../../openapi.yaml#/components/responses/server_error' + delete: + summary: Delete an order + operationId: delete-order + description: |- + This endpoint is optional. Upon the request, order entry and corresponding ordered data should be removed. + tags: + - Orders + security: + - Bearer: [] + responses: + '204': + description: The order and data has been successfully deleted. + 4XX: + $ref: '../../openapi.yaml#/components/responses/client_error_auth' + 5XX: + $ref: '../../openapi.yaml#/components/responses/server_error' + +components: + schemas: + order_id: + type: string + description: >- + Unique identifier of the order, which MUST match the specified pattern.. + pattern: '^[\w\-\.~]+$' + example: 40264b5-c3ae-46f4-a907-0f612d763d97 + + product_id: + type: string + description: >- + Unique identifier of the product. Must be unique within a collection. + example: c8a1f88d-89cf-4933-9118-45e9c1a5df20 + + order: + type: object + title: Order + description: Information about an order + required: + - order:id + - order:status + - order:date + - source_collection_id + - target_collection_id + - costs + properties: + order:id: + $ref: '#/components/schemas/order_id' + order:status: + type: string + description: |- + The status of the order. + * `orderable`: The item or asset is orderable via the provider scenario. + * `ordered`: The item or asset is ordered and the provider is preparing to make it available. + * `shipping`: The item or asset order are being processed by the provider to provide the user with the asset(s). + * `delivered`: The provider has delivered the order and asset(s) are available. + * `unable_to_deliver`: The provider is not able to deliver the order. + * `canceled`: The order has been canceled. + enum: + - orderable + - ordered + - shipping + - delivered + - unable_to_deliver + - canceled + order:date: + type: string + format: date-time + description: The order time. Formatted as a + [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + example: '2017-01-01T09:32:12Z' + order:expiration_date: + type: string + format: date-time + description: The validity time of the order. Formatted as a + [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. + example: '2999-01-01T00:00:00Z' + source_collection_id: + type: string + description: Unique identifier of the source collection. + example: 'PLEIADES' + target_collection_id: + type: string + description: Unique identifier of the target collection. Can be the same as `source_collection_id`. + example: 'PLEIADES' + costs: + $ref: '../../openapi.yaml#/components/schemas/money' + + parameters: + order_id: + name: order_id + in: path + description: Unique identifier of an order. + required: true + schema: + $ref: '#/components/schemas/order_id' + + securitySchemes: + Bearer: + type: http + scheme: bearer + bearerFormat: >- + The Bearer Token MUST consist of the authentication method, a provider + ID (if available) and the token itself. All separated by a forward slash + `/`. Examples (replace `TOKEN` with the actual access token): (1) Basic + authentication (no provider ID available): `basic//TOKEN` (2) OpenID + Connect (provider ID is `ms`): `oidc/ms/TOKEN`. For OpenID Connect, the + provider ID corresponds to the value specified for `id` for each + provider in `GET /credentials/oidc`. + Basic: + type: http + scheme: basic From ea237fdb2a0380653a747dea8587ba8c5154dcb5 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 15 Aug 2022 12:59:54 +0200 Subject: [PATCH 071/103] Update redoc-cli --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index a256cbc8..368dd975 100644 --- a/package.json +++ b/package.json @@ -14,7 +14,7 @@ }, "devDependencies": { "@stoplight/spectral": "^5.9.1", - "redoc-cli": "^0.13.17" + "redoc-cli": "^0.13.18" }, "scripts": { "start": "redoc-cli serve openapi.yaml --watch --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true", From ca30724f6206dbf020211f6c331734f8851b8e98 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 15 Aug 2022 19:34:29 +0200 Subject: [PATCH 072/103] Align with openEO and STAC, improve docs etc --- extension/commercial_data/README.md | 140 +------------------- extension/commercial_data/openapi.yaml | 173 ++++++++++++++----------- extension/commercial_data/package.json | 24 ++++ 3 files changed, 126 insertions(+), 211 deletions(-) create mode 100644 extension/commercial_data/package.json diff --git a/extension/commercial_data/README.md b/extension/commercial_data/README.md index 9eb56e6f..415c0ba0 100644 --- a/extension/commercial_data/README.md +++ b/extension/commercial_data/README.md @@ -4,6 +4,7 @@ The Commercial Data API extension provides an interface for discovering, orderin - Version: **0.1.0** - Stability: **experimental** +- [OpenAPI document](openapi.yaml) **Note:** This document only documents the additions to the specification. Extensions can not change or break existing behavior of the openEO API. @@ -163,144 +164,7 @@ Backends should implement the following endpoints: Optionally, they can also implement: - `DELETE /orders/{order_id}`: Delete an order -#### `GET /orders` - -Lists all created orders, regardless of the status or the collection. The items should follow the [STAC Order Extension](https://github.com/stac-extensions/order). - -```yaml -schema: - title: All Orders - type: object - required: - - orders - - links - properties: - orders: - type: array - items: - type: object - title: Order - description: Information about an order - required: - - order:id - - order:status - - order:date - - source_collection_id - - target_collection_id - - costs - properties: - order:id: - type: string - description: >- - Unique identifier of the order, which MUST match the specified pattern.. - pattern: '^[\w\-\.~]+$' - order:status: - type: string - description: |- - The status of the order. - * `orderable`: The item or asset is orderable via the provider scenario. - * `ordered`: The item or asset is ordered and the provider is preparing to make it available. - * `shipping`: The item or asset order are being processed by the provider to provide the user with the asset(s). - * `delivered`: The provider has delivered the order and asset(s) are available. - * `unable_to_deliver`: The provider is not able to deliver the order. - * `canceled`: The order has been canceled. - enum: - - orderable - - ordered - - shipping - - delivered - - unable_to_deliver - - canceled - order:date: - type: string - format: date-time - description: The order time. Formatted as a - [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - order:expiration_date: - type: string - format: date-time - description: The validity time of the order. Formatted as a - [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. - source_collection_id: - type: string - description: Unique identifier of the source collection. - target_collection_id: - type: string - description: Unique identifier of the target collection. Can be the same as `source_collection_id`. - costs: - $ref: 'https://raw.githubusercontent.com/Open-EO/openeo-api/1.0.0/openapi.yaml#/components/schemas/money' - links: - $ref: 'https://raw.githubusercontent.com/Open-EO/openeo-api/1.0.0/openapi.yaml#/components/schemas/links_pagination' -``` - -#### `POST /orders` - -Create an order for selected products. Order can contain some additional parameters that specify how the products should be delivered. For example, depending on the collection it might be possible to set the projection, resampling method, bit depth etc of the delivered data. - -Backends should expose the available ordering parameters in `/collections/{collection_id}` in the `order_parameters` field, following the `process_parameters` schema of [`GET /service_types`](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Secondary-Services/operation/list-service-types). - -Returns `Location` and `OpenEO-Identifier` header with the link to the detailed information about the order. - -```yaml -schema: - title: Create Order Request - type: object - required: - - collection_id - - products - - parameters - properties: - collection_id: - type: string - description: Unique identifier of the collection. - products: - type: array - description: Array of IDs of products to order. - items: - $ref: '#/components/schemas/order_id' - parameters: - type: object - description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. - additionalProperties: - x-additionalPropertiesName: Order Parameter Name - description: Value of the order parameter to be used in the request. -``` - -#### `GET /orders/{order_id}` - -Get full metadata of the order. The item should follow the [STAC Order Extension](https://github.com/stac-extensions/order), but extended with the source collection id, target collection id, list of ordered product, order parameters and optionally links to products metadata. - -`source_collection_id` is the id of the orderable collection the products were ordered from. `target_collection_id` is the id of the collection to be used in `load_collection` to load into the DataCube and contains the ordered products. `source_collection_id` and `target_collection_id` can be the same or different, it's up to the backend if the products are ingested into a new collection or the existing orderable collection. - -Additionally, if the backend implements order-specific collections, it can provide access to the union of all purchased products when using the ID of the original orderable collection. - -Backends can optionally link to the spatial and temporal extent information and other metadata about the products, preferrably by implementing `/collections/{collection_id}/items/{item_id}` as described [here](#product-metadata), using relation type `ordered_product_metadata`. - -```json -{ - "order:id": "40264b5-c3ae-46f4-a907-0f612d763d97", - "order:status": "delivered", - "order:date": "2017-01-01T09:32:12Z", - "products": [ "c8a1f88d-89cf-4933-9118-45e9c1a5df20"], - "parameters": { "resamplingKernel": "NN" }, - "source_collection_id": "PLEAIDES", - "target_collection_id": "PLEAIDES", - "costs": 42 -} -``` - -#### `POST /orders/{order_id}` - -When an order is created, the data isn't yet ordered from the commercial data provider. The client should explicitly confirm the order whereupon the order is executed, the costs are deducted from the client's account and the data is ingested in the associated collection. The `order:status` changes from `orderable` to `ordered`. - -If the user doesn't have sufficient funds, the endpoint should return an error and `order:status` should not change. - -This endpoint only has an effect if `order:status` is `orderable`. - - -#### `DELETE /orders/{order_id}` - -Optional, removes the order entry and corresponding ordered data. +See the [OpenAPI document](openapi.yaml) for details. ### Product metadata diff --git a/extension/commercial_data/openapi.yaml b/extension/commercial_data/openapi.yaml index 4fea12a9..b89384ee 100644 --- a/extension/commercial_data/openapi.yaml +++ b/extension/commercial_data/openapi.yaml @@ -2,8 +2,9 @@ openapi: 3.0.2 info: title: openEO Commercial data API extension version: 0.1.0 - description: - $ref: README.md + description: >- + The Commercial Data API extension provides an interface for discovering, + ordering and using commercial data in the openEO API. contact: name: openEO Consortium url: 'https://openeo.org' @@ -11,6 +12,8 @@ info: license: name: Apache 2.0 url: 'http://www.apache.org/licenses/LICENSE-2.0.html' +externalDocs: + url: https://github.com/Open-EO/openeo-api/blob/draft/extensions/federation/README.md tags: - name: Orders description: Management of commercial data orders @@ -36,10 +39,15 @@ servers: paths: /orders: get: - summary: List all orders created by the user + summary: List all commercial data orders operationId: list-orders description: |- - Lists all created orders, regardless of the status or the collection. + Lists all created orders submitted by a user, regardless of the status + or the collection. + + It is **strongly RECOMMENDED** to keep the response size small by + omitting all optional non-scalar values from objects in `orders` (e.g. `parameters`). + To get the full metadata for an order clients MUST request `GET /orders/{order_id}`. tags: - Orders security: @@ -52,7 +60,7 @@ paths: content: application/json: schema: - title: All Orders + title: Orders type: object required: - orders @@ -69,12 +77,17 @@ paths: 5XX: $ref: '../../openapi.yaml#/components/responses/server_error' post: - summary: Create order request + summary: Create a new order operationId: create-order description: |- - Create an order for selected products. Order can contain some additional parameters that specify how the products should be delivered. For example, depending on the collection it might be possible to set the projection, resampling method, bit depth etc of the delivered data. + Create an order for selected items (products). + + When an order is created through through this endpoint, the data isn't yet ordered from the commercial data provider. The client MUST explicitly confirm the order through `POST /orders/{order_id}` whereupon the order is executed. + + The order can contain additional parameters that specify how the items should be delivered. + For example, it might be possible to set the projection, resampling method, bit depth etc. of the delivered data. - Backends should expose the available ordering parameters in `/collections/{collection_id}` in the `order_parameters` field, following the `process_parameters` schema of [`GET /service_types`](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Secondary-Services/operation/list-service-types). + Backends SHOULD expose the available ordering parameters in `GET /collections/{collection_id}` in the `order_parameters` field, following the `process_parameters` schema of [`GET /service_types`](https://openeo.org/documentation/1.0/developers/api/reference.html#tag/Secondary-Services/operation/list-service-types). Returns `Location` and `OpenEO-Identifier` header with the link to the detailed information about the order. tags: @@ -88,26 +101,20 @@ paths: schema: type: object required: - - collection_id - - products + - source_collection_id + - items - parameters properties: - collection_id: + source_collection_id: type: string description: Unique identifier of the collection. - products: + items: type: array - description: Array of IDs of products to order. + description: Array of items IDs to order. items: $ref: '#/components/schemas/order_id' parameters: - type: object - description: Key-value pairs of available `order_parameters` as listed at `GET /commercial_data/collections/{collection_id}` for filtering available products. - additionalProperties: - x-additionalPropertiesName: Order Parameter Name - description: Value of the order parameter to be used in the request. - example: - resamplingKernel: 'NN' + $ref: '#/components/schemas/order_parameters' responses: '201': description: The order has been created successfully. @@ -139,7 +146,9 @@ paths: summary: Full metadata for an order operationId: describe-order description: |- - Get full metadata of the order. The item should follow the [STAC Order Extension](https://github.com/stac-extensions/order), but extended with additional information. Backends can optionally link to the spatial and temporal extent information and other metadata about the products. + Get full metadata of the order. + + Backends can optionally link to the spatial and temporal extent information and other metadata about the items using the relation type `item`, preferrably by implementing and linking to `GET /collections/{collection_id}/items/{item_id}`. tags: - Orders security: @@ -154,44 +163,7 @@ paths: - $ref: '#/components/schemas/order' - type: object required: - - products - properties: - products: - type: array - description: List of IDs of ordered products. - items: - $ref: '#/components/schemas/product_id' - products_metadata: - type: array - description: Array of links to metadata for ordered products listed in `products`. - items: - title: Link - description: >- - A link to the metadata for the ordered product. - type: object - required: - - href - - rel - properties: - rel: - type: string - enum: ["ordered_product_metadata"] - href: - type: string - description: The value MUST be a valid URL, preferably pointing to `/collections/{collection_id}/items/{product_id}`. - format: uri - example: 'https://example.openeo.org/collections/PLEIADES/items/c8a1f88d-89cf-4933-9118-45e9c1a5df20' - type: - type: string - description: >- - The value MUST be a string that hints at the format used to - represent data at the provided URI, preferably a media (MIME) type. - example: text/html - title: - type: string - description: Used as a human-readable label for a link. - example: openEO - + - items 4XX: $ref: '../../openapi.yaml#/components/responses/client_error_auth' 5XX: @@ -200,11 +172,17 @@ paths: summary: Confirm an order operationId: confirm-order description: |- - Confirm the order to execute the purchase and ingest into the target collection. The `order:status` changes from `orderable` to `ordered`. + Confirm the order, execute the purchase and ingest the data into a target collection. - If the user doesn't have sufficient funds, the endpoint should return an error and `order:status` should not change. + When an order is created through `POST /order`, the data isn't yet ordered from the commercial data provider. The client MUST explicitly confirm the order whereupon the order is executed, the costs are deducted from the client's account and the data is ingested in the target collection. + + The target collection is the collection through which the ordered data is made available. The source and the target collection can be the same or different, it's up to the backend if the ordered data id ingested into a new target collection or the existing source collection. Additionally, if the backend creates new target collections, it can provide access to the union of all purchased products through the source collection. This endpoint only has an effect if `order:status` is `orderable`. + + If the user doesn't have sufficient funds, the endpoint should return an error and `order:status` should not change. + + Otherwise, the `order:status` changes from `orderable` to `ordered` until a new order status (e.g. `delivered` or `unable_to_deliver`) becomes available. tags: - Orders security: @@ -220,7 +198,10 @@ paths: summary: Delete an order operationId: delete-order description: |- - This endpoint is optional. Upon the request, order entry and corresponding ordered data should be removed. + Removes the order and all ordered items from the back-ends. + + If an order is still in progress, it cancels the order. + Costs MAY still occur. tags: - Orders security: @@ -232,36 +213,54 @@ paths: $ref: '../../openapi.yaml#/components/responses/client_error_auth' 5XX: $ref: '../../openapi.yaml#/components/responses/server_error' - components: schemas: order_id: type: string description: >- - Unique identifier of the order, which MUST match the specified pattern.. + Unique identifier of the order at the back-end, which MUST match the specified pattern. pattern: '^[\w\-\.~]+$' + example: fb6799ab + stac_order_id: + type: string + description: >- + Identifier of the order at the data provider. example: 40264b5-c3ae-46f4-a907-0f612d763d97 - - product_id: + item_id: type: string description: >- - Unique identifier of the product. Must be unique within a collection. + Unique identifier of the item. Must be unique within a collection. example: c8a1f88d-89cf-4933-9118-45e9c1a5df20 - + order_parameters: + type: object + description: >- + Key-value pairs of available `order_parameters` as listed at + `GET /collections/{collection_id}` for filtering available items. + additionalProperties: + x-additionalPropertiesName: Order Parameter Name + description: Value of the order parameter for the request. + example: + resamplingKernel: 'NN' order: type: object title: Order - description: Information about an order + description: >- + Information about an order. + + The properties with the `order:` prefix follow the fields defined in the + [STAC Order Extension](https://github.com/stac-extensions/order). required: - - order:id + - id - order:status - order:date - source_collection_id - target_collection_id - costs properties: - order:id: + id: $ref: '#/components/schemas/order_id' + order:id: + $ref: '#/components/schemas/stac_order_id' order:status: type: string description: |- @@ -294,14 +293,43 @@ components: source_collection_id: type: string description: Unique identifier of the source collection. + pattern: '^[\w\-\.~\/]+$' example: 'PLEIADES' target_collection_id: type: string - description: Unique identifier of the target collection. Can be the same as `source_collection_id`. + description: >- + Unique identifier of the target collection. + Can be the same as `source_collection_id`. + pattern: '^[\w\-\.~\/]+$' example: 'PLEIADES' costs: $ref: '../../openapi.yaml#/components/schemas/money' + parameters: + $ref: '#/components/schemas/order_parameters' + items: + type: array + description: List of ordered item IDs. + items: + $ref: '#/components/schemas/item_id' + links: + type: array + description: |- + Links related to this order, e.g. to metadata for ordered items + listed in `items`. + + It is highly RECOMMENDED to provide links with the + following `rel` (relation) types: + + 1. `item`: A link back to a valid STAC Item that has been ordered. + Could be a link to `GET /collections/{collection_id}/items/{item_id}`. + 2. `order`: A link pointing to a dcoument describing the ordering + terms and conditions of the data provider. + + For additional relation types see also the lists of + [common relation types in openEO](#section/API-Principles/Web-Linking). + items: + $ref: '../../openapi.yaml#/components/schemas/link' parameters: order_id: name: order_id @@ -310,7 +338,6 @@ components: required: true schema: $ref: '#/components/schemas/order_id' - securitySchemes: Bearer: type: http diff --git a/extension/commercial_data/package.json b/extension/commercial_data/package.json new file mode 100644 index 00000000..0fd067ed --- /dev/null +++ b/extension/commercial_data/package.json @@ -0,0 +1,24 @@ +{ + "name": "@openeo/api-extension-commercial-data", + "version": "0.1.0", + "author": "openEO Consortium", + "license": "Apache-2.0", + "description": "The openEO API specification.", + "homepage": "https://openeo.org", + "bugs": { + "url": "https://github.com/Open-EO/openeo-api/issues" + }, + "repository": { + "type": "git", + "url": "git+https://github.com/Open-EO/openeo-api.git" + }, + "devDependencies": { + "@stoplight/spectral": "^5.9.1", + "redoc-cli": "^0.13.18" + }, + "scripts": { + "start": "redoc-cli serve openapi.yaml --watch --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true", + "build": "redoc-cli bundle openapi.yaml -o redoc.html --title \"openEO API\" --cdn --options.expandResponses \"200,201,202,203,204\" --options.pathInMiddlePanel true", + "test": "spectral lint openapi.yaml" + } +} From fb7bc3f2de518b8f8f71859ab9de73182998ed69 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 15 Aug 2022 19:41:03 +0200 Subject: [PATCH 073/103] Rename folder --- extension/{commercial_data => commercial-data}/README.md | 0 extension/{commercial_data => commercial-data}/openapi.yaml | 2 +- extension/{commercial_data => commercial-data}/package.json | 0 3 files changed, 1 insertion(+), 1 deletion(-) rename extension/{commercial_data => commercial-data}/README.md (100%) rename extension/{commercial_data => commercial-data}/openapi.yaml (99%) rename extension/{commercial_data => commercial-data}/package.json (100%) diff --git a/extension/commercial_data/README.md b/extension/commercial-data/README.md similarity index 100% rename from extension/commercial_data/README.md rename to extension/commercial-data/README.md diff --git a/extension/commercial_data/openapi.yaml b/extension/commercial-data/openapi.yaml similarity index 99% rename from extension/commercial_data/openapi.yaml rename to extension/commercial-data/openapi.yaml index b89384ee..059f1987 100644 --- a/extension/commercial_data/openapi.yaml +++ b/extension/commercial-data/openapi.yaml @@ -1,6 +1,6 @@ openapi: 3.0.2 info: - title: openEO Commercial data API extension + title: openEO Commercial Data API extension version: 0.1.0 description: >- The Commercial Data API extension provides an interface for discovering, diff --git a/extension/commercial_data/package.json b/extension/commercial-data/package.json similarity index 100% rename from extension/commercial_data/package.json rename to extension/commercial-data/package.json From 2cf38d8514eb62e72311ce62cd51da6a3c91b7a9 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 15 Aug 2022 19:42:52 +0200 Subject: [PATCH 074/103] Move to correct folder --- {extension => extensions}/commercial-data/README.md | 0 {extension => extensions}/commercial-data/openapi.yaml | 0 {extension => extensions}/commercial-data/package.json | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename {extension => extensions}/commercial-data/README.md (100%) rename {extension => extensions}/commercial-data/openapi.yaml (100%) rename {extension => extensions}/commercial-data/package.json (100%) diff --git a/extension/commercial-data/README.md b/extensions/commercial-data/README.md similarity index 100% rename from extension/commercial-data/README.md rename to extensions/commercial-data/README.md diff --git a/extension/commercial-data/openapi.yaml b/extensions/commercial-data/openapi.yaml similarity index 100% rename from extension/commercial-data/openapi.yaml rename to extensions/commercial-data/openapi.yaml diff --git a/extension/commercial-data/package.json b/extensions/commercial-data/package.json similarity index 100% rename from extension/commercial-data/package.json rename to extensions/commercial-data/package.json From 636f6327f014a757eb6448a23056174b85b08e56 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 15 Aug 2022 19:54:45 +0200 Subject: [PATCH 075/103] Add to README and CHANGELOG --- CHANGELOG.md | 4 +++- README.md | 8 ++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b680ebb7..d67e5a78 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- New experimental [Federation Extension](./extensions/federation/README.md) +- **New extensions:** + - [Commercial Data Extension](./extensions/commercial-data/README.md) + - [Federation Extension](./extensions/federation/README.md) - `GET /`: New Relation types: [#404](https://github.com/Open-EO/openeo-api/issues/404) - `create-form` to link to the registration page - `recovery-form` to link to the credentials recovery page. diff --git a/README.md b/README.md index 9b016707..a1cc5883 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,13 @@ The [master branch](https://github.com/Open-EO/openeo-api/tree/master) is the 's See also the [changelog](CHANGELOG.md) and the [milestones](https://github.com/Open-EO/openeo-api/milestones) for a rough roadmap based on GitHub issues. +## Extensions + +| Name | Version | Stability | Description | +|-----------------|---------|--------------|---------------------------------------------------------------------------------------| +| [Commercial Data](extensions/commercial-data/) | 0.1.0 | experimental | Provides an interface for discovering, ordering and using commercial data. | +| [Federation](extensions/federation/) | 0.1.0 | experimental | Covers federation aspects, i.e. where multiple back-ends are exposed as a single API. | + ## Repository This repository contains a set of files formally describing the openEO API, each with a human-readable and easily browseable version: @@ -33,6 +40,7 @@ This repository contains a set of files formally describing the openEO API, each * [openapi.yaml](openapi.yaml) provides the [OpenAPI](https://www.openapis.org/) 3.0 definition of the openEO API. See the table above for human-readable versions of the OpenAPI definition. * [errors.json](errors.json) is a list of potential global error codes and messages, excluding specific exceptions separately available for each process. * The [assets](assets/) folder contains some useful additional files such as examples or schemas. All of these are non-binding additions. The source of truth are the top-level specification files. +* The [extensions](extensions/) folder contains extensions to the openEO API. # Development From 086bd3784910949f5ec320d08484ce082307095a Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 15 Aug 2022 19:59:56 +0200 Subject: [PATCH 076/103] Clarification --- extensions/commercial-data/openapi.yaml | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/extensions/commercial-data/openapi.yaml b/extensions/commercial-data/openapi.yaml index 059f1987..c0abf9e8 100644 --- a/extensions/commercial-data/openapi.yaml +++ b/extensions/commercial-data/openapi.yaml @@ -221,11 +221,6 @@ components: Unique identifier of the order at the back-end, which MUST match the specified pattern. pattern: '^[\w\-\.~]+$' example: fb6799ab - stac_order_id: - type: string - description: >- - Identifier of the order at the data provider. - example: 40264b5-c3ae-46f4-a907-0f612d763d97 item_id: type: string description: >- @@ -260,7 +255,10 @@ components: id: $ref: '#/components/schemas/order_id' order:id: - $ref: '#/components/schemas/stac_order_id' + type: string + description: >- + Identifier of the order at the data provider. + example: 40264b5-c3ae-46f4-a907-0f612d763d97 order:status: type: string description: |- @@ -334,7 +332,10 @@ components: order_id: name: order_id in: path - description: Unique identifier of an order. + description: >- + Unique identifier of an order. + + This corresponds to the `id` property of an order. required: true schema: $ref: '#/components/schemas/order_id' From 4ce58e45c35fcca879f34af81c549cc49dc0722c Mon Sep 17 00:00:00 2001 From: jjezersek Date: Thu, 18 Aug 2022 15:09:09 +0200 Subject: [PATCH 077/103] Update order:status values to match released version, fix links to STAC Order to released version. --- extensions/commercial-data/README.md | 4 ++-- extensions/commercial-data/openapi.yaml | 18 ++++++++++-------- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/extensions/commercial-data/README.md b/extensions/commercial-data/README.md index 415c0ba0..f90a399d 100644 --- a/extensions/commercial-data/README.md +++ b/extensions/commercial-data/README.md @@ -23,7 +23,7 @@ When the order is completed, the data is ingested in a collection and its ID is A backend should add general information about a commercial data collection to the `/collections` and `/collections/{collection_id}` endpoints, the same as with freely available collections. Only the metadata about the entire dataset needs to be provided, not about the specific data products that a user has already purchased. -Commercial data collections are distinguished from freely available collections by including `"order:status": "orderable"` as specified in the [STAC Order specification](https://github.com/stac-extensions/order). +Commercial data collections are distinguished from freely available collections by including `"order:status": "orderable"` as specified in the [STAC Order specification](https://github.com/stac-extensions/order/tree/v1.0.0). Commercial data collections can include an `order_parameters` field if ordering supports additional parameters that specify how the products should be delivered. @@ -227,6 +227,6 @@ When the order has finished, you can process the data as with a normal collectio ```python >>> order = connection.get_order(id="40264b5-c3ae-46f4-a907-0f612d763d97") >>> order.status -'delivered' +'succeeded' >>> pleiades_cube = connection.load_collection(order.target_collection_id) ``` \ No newline at end of file diff --git a/extensions/commercial-data/openapi.yaml b/extensions/commercial-data/openapi.yaml index c0abf9e8..0bac3678 100644 --- a/extensions/commercial-data/openapi.yaml +++ b/extensions/commercial-data/openapi.yaml @@ -182,7 +182,7 @@ paths: If the user doesn't have sufficient funds, the endpoint should return an error and `order:status` should not change. - Otherwise, the `order:status` changes from `orderable` to `ordered` until a new order status (e.g. `delivered` or `unable_to_deliver`) becomes available. + Otherwise, the `order:status` changes from `orderable` to `ordered` until a new order status (e.g. `succeeded` or `failed`) becomes available. tags: - Orders security: @@ -243,7 +243,7 @@ components: Information about an order. The properties with the `order:` prefix follow the fields defined in the - [STAC Order Extension](https://github.com/stac-extensions/order). + [STAC Order Extension](https://github.com/stac-extensions/order/tree/v1.0.0). required: - id - order:status @@ -265,16 +265,18 @@ components: The status of the order. * `orderable`: The item or asset is orderable via the provider scenario. * `ordered`: The item or asset is ordered and the provider is preparing to make it available. - * `shipping`: The item or asset order are being processed by the provider to provide the user with the asset(s). - * `delivered`: The provider has delivered the order and asset(s) are available. - * `unable_to_deliver`: The provider is not able to deliver the order. - * `canceled`: The order has been canceled. + * `pending`: The item or asset is ordered but wait for an activation before being able for shipping. + * `shipping` The item or asset order are being processed by the provider to provide you with the asset(s). + * `succeeded`: The provider has delivered your order and asset(s) are available. + * `failed`: The provider is not able to deliver the order. + * `canceled` The order has been canceled. enum: - orderable - ordered + - pending - shipping - - delivered - - unable_to_deliver + - succeeded + - failed - canceled order:date: type: string From 8a91f69b8e4abdb16bf1a602c1013cdbf58b691c Mon Sep 17 00:00:00 2001 From: jjezersek Date: Mon, 29 Aug 2022 11:18:49 +0200 Subject: [PATCH 078/103] Change 'STAC Order specification' to 'STAC Order extension' for consistency. --- extensions/commercial-data/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/extensions/commercial-data/README.md b/extensions/commercial-data/README.md index f90a399d..b4dc50ed 100644 --- a/extensions/commercial-data/README.md +++ b/extensions/commercial-data/README.md @@ -23,7 +23,7 @@ When the order is completed, the data is ingested in a collection and its ID is A backend should add general information about a commercial data collection to the `/collections` and `/collections/{collection_id}` endpoints, the same as with freely available collections. Only the metadata about the entire dataset needs to be provided, not about the specific data products that a user has already purchased. -Commercial data collections are distinguished from freely available collections by including `"order:status": "orderable"` as specified in the [STAC Order specification](https://github.com/stac-extensions/order/tree/v1.0.0). +Commercial data collections are distinguished from freely available collections by including `"order:status": "orderable"` as specified in the [STAC Order extension](https://github.com/stac-extensions/order/tree/v1.0.0). Commercial data collections can include an `order_parameters` field if ordering supports additional parameters that specify how the products should be delivered. From 6e79b8f047e947b5c53332ddd642896c282bc301 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 2 Nov 2022 15:23:15 +0100 Subject: [PATCH 079/103] Clarify the user-defined processes #464 --- openapi.yaml | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index ebad1bb1..4dbb2174 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -232,7 +232,7 @@ info: A **predefined process** is a process provided by the *back-end*. There is a set of predefined processes by openEO to improve interoperability between back-ends. Back-ends SHOULD follow these specifications whenever possible. Not all processes need to be implemented by all back-ends. See the **[process reference](https://processes.openeo.org)** for predefined processes. - A **user-defined process** is a process defined by the *user*. It can directly be part of another process graph or be stored as custom process on a back-end. Internally it is a *process graph* with optional additional metadata. + A **user-defined process** is a process defined by the *user*. It can directly be part of another process graph or be stored as custom process on a back-end. Internally, it is a *process graph* with a variety of additional metadata. A **process graph** chains specific process calls from the set of predefined and user-defined processes together. A process graph itself can be stored as a (user-defined) process again. Similarly to scripts in the context of programming, process graphs organize and automate the execution of one or more processes that could alternatively be executed individually. In a process graph, processes need to be specific, i.e. concrete values or "placeholders" for input parameters need to be specified. These values can be scalars, arrays, objects, references to parameters or previous computations or other process graphs. @@ -351,15 +351,17 @@ info: This tells the back-end that the process expects the result (i.e. the return value) from another process node to be passed as argument. The `` is strictly scoped and can only reference nodes from within the same process graph, not child or parent process graphs. - ### User-defined process + ### Child processes - A user-defined process in a process graph is a child process graph, to be evaluated as part of another process. + Some processes can run child processes, which is similar to the concept that other programming languages call + [callbacks](https://en.wikipedia.org/wiki/Callback_(computer_programming)) or lambda functions. + Each child process is simply a user-defined process again and can in theory be arbritarily complex. - **Example**: You want to calculate the absolute value of each pixel in a data cube. - This can be achieved in openEO by executing the `apply` process and pass it - a user-defined process as the "operator" to apply to each pixel. - In this simple example, the "child" process graph defining the user-defined process - consists of a single process `absolute`, but it can be arbitrary complex in general. + A very simple example would be to calculate the absolute value of each pixel in a data cube. + This can be achieved in openEO by using the `apply` process which gets the `absolute` process passed as child process. + In this example, the "child" processes consists of a single process `absolute`, but it can also be a more complex computation such as an NDVI or a prediciton based on a machine learning model. + + **Example**: A `` argument MUST at least consist of an object with a key `process_graph`. Optionally, it can also be described with the same additional properties available for predefined processes such as an id, parameters, return values etc. @@ -374,7 +376,7 @@ info: ### Accessing process parameters - A "parent" process that works with a user-defined process can make so called *process graph parameters* + A "parent" process that works with a child process can make so called *process graph parameters* available to the "child" logic. Processes in the "child" process graph can access these parameters by passing a `ParameterReference` object as argument. It is an object with key `from_parameter` specifying the name of the process graph parameter: From 932c7366861cb5ce88741b1a7c98a550cd46fa06 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 2 Nov 2022 16:06:53 +0100 Subject: [PATCH 080/103] Upgrade to Spectral v6 (#432) --- .spectral.yml | 20 ++++---------------- CHANGELOG.md | 1 + openapi.yaml | 16 +++++++++++----- package.json | 2 +- 4 files changed, 17 insertions(+), 22 deletions(-) diff --git a/.spectral.yml b/.spectral.yml index 0f3ff244..cfe29b52 100644 --- a/.spectral.yml +++ b/.spectral.yml @@ -1,27 +1,15 @@ extends: "spectral:oas" -# don't error on circular schemas, these are currently not supported by spectral -# https://github.com/stoplightio/spectral/issues/894 +# Work around broken rules: https://github.com/stoplightio/spectral/issues/1972 except: - "openapi.yaml#/components/schemas/process_graph/example": + "openapi.yaml#/paths/~1udf_runtimes/get/responses/200/content/application~1json/example": - oas3-valid-schema-example - "openapi.yaml#/paths/~1processes/get/responses/200/content/application~1json/example": - - oas3-valid-oas-content-example - "openapi.yaml#/paths/~1process_graphs/get/responses/200/content/application~1json/example": - - oas3-valid-oas-content-example - "openapi.yaml#/paths/~1service_types/get/responses/200/content/application~1json/example": - - oas3-valid-oas-content-example - "openapi.yaml#/paths/~1file_formats/get/responses/200/content/application~1json/example": - - oas3-valid-oas-content-example rules: - # Ported from Spectral 4.0 contact-properties: true tag-description: true oas3-parameter-description: true - oas3-unused-components-schema: false # Broken: https://github.com/stoplightio/spectral/issues/1271 + oas3-unused-component: false # Broken: https://github.com/stoplightio/spectral/issues/1271 operation-summary-formatted: description: Operation `summary` should start with upper case and not end with a dot. - recommended: true - type: style given: '$.paths.*[?( @property === ''get'' || @property === ''put'' || @property === ''post'' || @property === ''delete'' || @property === ''options'' || @property === ''head'' || @property === ''patch'' || @property === ''trace'' )]' then: field: summary @@ -31,7 +19,7 @@ rules: tags: - operation operation-id-kebab-case: - given: "$" + given: "$..operationId" then: function: pattern functionOptions: diff --git a/CHANGELOG.md b/CHANGELOG.md index d67e5a78..b4588ed3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -51,6 +51,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /me`: Clarify the behavior of the field `budget`. - `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs` and `POST /result`: Clarified the formatting of the `message` property. [#455](https://github.com/Open-EO/openeo-api/issues/455) - `GET /jobs/{job_id}/estimate`: Don't require that the costs are the upper limit. Services may specify the costs more freely depending on their terms of service. +- Several appearances of `nullable` were clarified according to the lint report by Spectral ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index 4dbb2174..62052134 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1090,7 +1090,7 @@ paths: compress: type: string description: Set the compression to use. - default: none + default: NONE enum: - JPEG - LZW @@ -5112,7 +5112,7 @@ components: arguments: $ref: '#/components/schemas/process_arguments' returns: - nullable: true + description: The return value which can by of any data type. links: type: array description: >- @@ -5144,12 +5144,16 @@ components: - id properties: summary: + type: string nullable: true description: + type: string nullable: true parameters: + type: array nullable: true returns: + type: object nullable: true allOf: - $ref: '#/components/schemas/process' @@ -5161,14 +5165,19 @@ components: - process_graph properties: id: + type: string nullable: true summary: + type: string nullable: true description: + type: string nullable: true parameters: + type: array nullable: true returns: + type: object nullable: true allOf: - $ref: '#/components/schemas/process' @@ -5343,7 +5352,6 @@ components: description: >- The default value for this parameter. Required parameters SHOULD NOT specify a default value. Optional parameters SHOULD always specify a default value. - nullable: true parameter: title: Parameter type: object @@ -5649,7 +5657,6 @@ components: value MUST conform to the defined type for the parameter defined at the same level. For example, if type is string, then default can be "foo" but cannot be 1. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.10.2). - nullable: true allOf: - $ref: '#/components/schemas/json_schema' error: @@ -6081,7 +6088,6 @@ components: For example, a raster-cube passed to the `inspect` SHOULD return the metadata similar to the collection metadata, including `cube:dimensions`. There are implementation guidelines available for the `inspect` process. - nullable: true path: description: |- Describes where the log entry originates from. diff --git a/package.json b/package.json index 368dd975..4cc5beaf 100644 --- a/package.json +++ b/package.json @@ -13,7 +13,7 @@ "url": "git+https://github.com/Open-EO/openeo-api.git" }, "devDependencies": { - "@stoplight/spectral": "^5.9.1", + "@stoplight/spectral-cli": "^6.6.0", "redoc-cli": "^0.13.18" }, "scripts": { From 3034cdf76d96ed878676b61698936d41dd7445bc Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 2 Nov 2022 17:28:39 +0100 Subject: [PATCH 081/103] `GET /jobs` and `GET /jobs/{job_id}`: Added a links property that can for example link to results and logs. #453 --- CHANGELOG.md | 1 + openapi.yaml | 40 ++++++++++++++++++++++++++++++++++++---- 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b4588ed3..8db55f4c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - `GET /collections` and `GET /collections/{collection_id}` (and implicitly `GET /jobs/{job_id}/results`): Added a field that can indicate which properties can be filtered on when loading data from a collection. [#396](https://github.com/Open-EO/openeo-api/issues/396) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) +- `GET /jobs` and `GET /jobs/{job_id}`: Added a links property that can for example link to results and logs. [#453](https://github.com/Open-EO/openeo-api/issues/453) - `GET /jobs/{job_id}/results`: - Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) - Added metadata field `openeo:status` to indicate the job status (and whether the result is complete or not). diff --git a/openapi.yaml b/openapi.yaml index 62052134..8ca66a10 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -643,7 +643,7 @@ paths: [CommonMark 0.29](http://commonmark.org/) syntax MAY be used for rich text representation. example: |- - This service is provided to you by [Cool EO Cloud Corp.](http://cool-eo-cloud-corp.com). It implements the full openEO API and allows to process a range of 999 EO data sets, including + This service is provided to you by [Cool EO Cloud Corp.](https://cool-eo-cloud-corp.com). It implements the full openEO API and allows to process a range of 999 EO data sets, including * Sentinel 1/2/3 and 5 * Landsat 7/8 @@ -797,19 +797,19 @@ paths: URL to a web page with more details about the plan. format: uri - example: 'http://cool-cloud-corp.com/plans/free-plan' + example: 'https://cool-cloud-corp.com/plans/free-plan' example: - name: free description: >- Free plan. Calculates one tile per second and a maximum amount of 100 tiles per hour. - url: 'http://cool-cloud-corp.com/plans/free-plan' + url: 'https://cool-cloud-corp.com/plans/free-plan' paid: false - name: premium description: >- Premium plan. Calculates unlimited tiles and each calculated tile costs 0.003 USD. - url: 'http://cool-cloud-corp.com/plans/premium-plan' + url: 'https://cool-cloud-corp.com/plans/premium-plan' paid: true links: description: |- @@ -5454,6 +5454,38 @@ components: - $ref: '#/components/schemas/usage' log_level: $ref: '#/components/schemas/min_log_level_default' + links: + type: array + description: |- + Links related to this batch job, e.g. a links to + invoices, log files or results. + + It is RECOMMENDED to provide links with the following `rel` (relation) types: + + 1. `monitor`: If logs are available, a link to the [logs endpoint](#tag/Batch-Jobs/operation/debug-job). + 2. `result`: If batch job results are available, a link to the [results endpoint](#tag/Batch-Jobs/operation/list-results). + + The relation types `monitor` and `result` may occur for various batch job states: + + 1. `created`: When the batch job was executed before and has been reset to `created` after an + [update](#tag/Batch-Jobs/operation/update-job) there could still be results and logs available + until they get discarded by [queueing the batch job again](#tag/Batch-Jobs/operation/start-job). + 2. `finished`: The full log and results are expected to be available. + 3. `error` / `canceled`: Partial results and logs may be available. + + For more relation types see the lists of + [common relation types in openEO](#section/API-Principles/Web-Linking). + items: + $ref: '#/components/schemas/link' + example: + - rel: result + type: application/json + title: Batch Job Results + href: https://www.cool-cloud-corp.com/api/v1.0/jobs/123/logs + - rel: result + type: application/json + title: Batch Job Logs + href: https://www.cool-cloud-corp.com/api/v1.0/jobs/123/logs job_id: type: string description: >- From be6e9628a594f57dd4a2fc69c5ff7a575f8681be Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 2 Nov 2022 18:40:28 +0100 Subject: [PATCH 082/103] Remove openeo:property_filters again #396 --- CHANGELOG.md | 1 - extensions/federation/README.md | 2 -- openapi.yaml | 23 ++--------------------- 3 files changed, 2 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8db55f4c..eb45fea0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `create-form` to link to the registration page - `recovery-form` to link to the credentials recovery page. - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) -- `GET /collections` and `GET /collections/{collection_id}` (and implicitly `GET /jobs/{job_id}/results`): Added a field that can indicate which properties can be filtered on when loading data from a collection. [#396](https://github.com/Open-EO/openeo-api/issues/396) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) - `GET /jobs` and `GET /jobs/{job_id}`: Added a links property that can for example link to results and logs. [#453](https://github.com/Open-EO/openeo-api/issues/453) - `GET /jobs/{job_id}/results`: diff --git a/extensions/federation/README.md b/extensions/federation/README.md index 5dcdf32e..515fecf3 100644 --- a/extensions/federation/README.md +++ b/extensions/federation/README.md @@ -160,8 +160,6 @@ schema: ``` **Note:** In Collections this should generally be provided on the top-level of the object. -If users should be able to filter on back-ends in `load_collection`, please provide `federation:backends` as an element in `openeo:property_filters`. -Then, `federation:backends` as defined in the schema above should also be provided in the `summaries`. ### Examples diff --git a/openapi.yaml b/openapi.yaml index 8ca66a10..df143b06 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -500,7 +500,7 @@ tags: ## STAC - For data discovery of Earth Observation Collections at the back-ends, openEO strives for compatibility with the specifications [SpatioTemporal Asset Catalog (STAC)](https://stacspec.org/) and [OGC API - Features - Part 1: Core](http://docs.opengeospatial.org/is/17-069r3/17-069r3.html) as far as possible. Implementing the data discovery endpoints of openEO should also produce valid STAC API 0.9.0 and OGC API - Features 1.0 responses, including ([partial](#provide-data-for-download)) compatibility with their APIs. + For data discovery of Earth Observation Collections at the back-ends, openEO strives for compatibility with the specifications [SpatioTemporal Asset Catalog (STAC)](https://stacspec.org/) and [OGC API - Features - Part 1: Core](http://docs.opengeospatial.org/is/17-069r3/17-069r3.html) as far as possible. Implementing the data discovery endpoints of openEO also produced valid STAC API 1.0 and OGC API - Features 1.0 responses, including ([partial](#provide-data-for-download)) compatibility with their APIs. The data discovery endpoints `GET /collections` and `GET /collections/{collection_id}` are compatible with OGC API - Features and STAC. Both specifications define additional endpoints that need to be implemented to be fully compatible. The additional endpoints can easily be integrated into an openEO API implementation. A rough list of actions for compatibility is available below, but please refer to their specifications to find out the full details. @@ -1333,7 +1333,7 @@ paths: Lists **all** information about a specific collection specified by the identifier `collection_id`. - This endpoint is compatible with [STAC API 0.9.0](https://stacspec.org) and + This endpoint is compatible with [STAC API](https://stacspec.org) and [OGC API - Features](http://docs.opengeospatial.org/is/17-069r3/17-069r3.html). [STAC API](https://github.com/radiantearth/stac-spec/tree/v0.9.0/api-spec) features / extensions and @@ -1458,10 +1458,6 @@ paths: - B10 - B11 - B12 - 'openeo:property_filters': - - platform - - eo:cloud_cover - - sat:orbit_state summaries: 'constellation': - Sentinel-2 @@ -4417,21 +4413,6 @@ components: x-additionalPropertiesName: Dimension Name allOf: - $ref: '#/components/schemas/dimension' - 'openeo:property_filters': - description: |- - Gives a list of properties you can filter on when loading data from the collection. - Returning an empty array means that no property-based filters are available. - - This is a guide for users to get an overview of available properties to filter on. - More metadata about the properties can be found in `summaries`. - - If this property is not present, it means that the collections does - not provide any collection-specific details about available property-based filters. - It's then up to other means of documentation to inform the user, - e.g. there could be documentation about property filters that work across collections. - type: array - items: - type: string summaries: title: STAC Summaries (Collection Properties) description: |- From 1790f3e6a7189e9f93d687331f0b110ce335b588 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 3 Nov 2022 12:30:44 +0100 Subject: [PATCH 083/103] Clarify that logs should be sorted by time. --- openapi.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/openapi.yaml b/openapi.yaml index df143b06..74765076 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -6225,6 +6225,7 @@ components: - debug default: debug logs: + description: A chronological list of logs. type: array items: $ref: '#/components/schemas/log_entry' From 77a0cb3e4f624012bf2dfebda12e6d73fd548eef Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 7 Nov 2022 12:38:20 +0100 Subject: [PATCH 084/103] Clarify that UTF-8 is the default charset #462 (#466) --- CHANGELOG.md | 2 ++ openapi.yaml | 11 +++++++++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index eb45fea0..af043a0e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- Explicitly mention the use of HTTP content negotiation +- Clarify that the default charset is UTF-8 [#462](https://github.com/Open-EO/openeo-api/issues/462) - Fixed inconsistencies in errors.json: removed `ProcessGraphIdDoesntMatch`, clarified `ProcessGraphMissing`, added `ProcessInvalid` and `ProcessGraphInvalid`. [#394](https://github.com/Open-EO/openeo-api/issues/394), [#395](https://github.com/Open-EO/openeo-api/issues/395), [#401](https://github.com/Open-EO/openeo-api/issues/401) - Fixed the default value for the version number in the API url (`v1.0` -> `v1`) and improved the description for API versioning. [#393](https://github.com/Open-EO/openeo-api/issues/393) - Fixed the Collection example to use `gsd` instead of `eo:gsd`. [#399](https://github.com/Open-EO/openeo-api/issues/399) diff --git a/openapi.yaml b/openapi.yaml index 74765076..99b9a2e9 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -27,9 +27,16 @@ info: Naming of endpoints follow the REST principles. Therefore, endpoints are centered around resources. Resource identifiers MUST be named with a noun in plural form except for single actions that can not be modelled with the regular HTTP verbs. Single actions MUST be single endpoints with a single HTTP verb (POST is RECOMMENDED) and no other endpoints beneath it. - ## JSON + The openEO API makes use of [HTTP Content Negotiation](https://www.rfc-editor.org/rfc/rfc9110.html#name-content-negotiation), + including, but not limited to, the request headers `Accept`, `Accept-Charset` and `Accept-Language`. - The API uses JSON for request and response bodies whenever feasible. Services use JSON as the default encoding. Other encodings can be requested using [Content Negotiation](https://www.w3.org/Protocols/rfc2616/rfc2616-sec12.html). Clients and servers MUST NOT rely on the order in which properties appears in JSON. Collections usually don't include nested JSON objects if those information can be requested from the individual resources. + ### JSON + + The API uses JSON for request and response bodies whenever feasible. Services use JSON as the default encoding. Other encodings can be requested using HTTP Content Negotiation ([`Accept` header](https://www.rfc-editor.org/rfc/rfc9110.html#name-accept)). Clients and servers MUST NOT rely on the order in which properties appear in JSON. To keep the response size small, lists of resources (e.g. the list of batch jobs) usually should not include nested JSON objects, if this information can be requested from the individual resource endpoints (e.g. the metadata for a single batch job). + + ### Charset + + Services use [UTF-8](https://en.wikipedia.org/wiki/UTF-8) as the default charset if not negotiated otherwise with HTTP Content Negotiation ([`Accept-Charset` header](https://www.rfc-editor.org/rfc/rfc9110.html#name-accept-charset)). ## Web Linking From d018e4b2a5b81775a77919186e2a46f4730b0230 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 10 Nov 2022 15:04:44 +0100 Subject: [PATCH 085/103] Clarify how the well-known document works #460 (#467) Co-authored-by: Stefaan Lippens --- CHANGELOG.md | 1 + openapi.yaml | 30 +++++++++++++++++++----------- 2 files changed, 20 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index af043a0e..1d922c7c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -54,6 +54,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs` and `POST /result`: Clarified the formatting of the `message` property. [#455](https://github.com/Open-EO/openeo-api/issues/455) - `GET /jobs/{job_id}/estimate`: Don't require that the costs are the upper limit. Services may specify the costs more freely depending on their terms of service. - Several appearances of `nullable` were clarified according to the lint report by Spectral +- Clarify how the well-known document works [#460](https://github.com/Open-EO/openeo-api/issues/460) ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index 99b9a2e9..7c7ac79d 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -928,17 +928,25 @@ paths: The Well-Known URI is the entry point for clients and users, so make sure it is permanent and easy to use and remember. Clients MUST NOT - require the well-known path (`./well-known/openeo`) in the URL that is - specified by a user to connect to the back-end. A client MUST request - `https://example.com/.well-known/openeo` if a user tries to connect to - `https://example.com`. If the request to the well-known URI fails, the - client SHOULD try to request the capabilities at `/` from - `https://example.com`. - - **This URI MUST NOT be versioned as the other endpoints.** If your API - is available at `https://example.com/api/v1.0`, the Well-Known URI - SHOULD be located at `https://example.com/.well-known/openeo` and the - URI users connect to SHOULD be `https://example.com`. + require the well-known path (`/.well-known/openeo`) in the URL that is + specified by a user to connect to the back-end. + + For clients, the usual behavior SHOULD follow these steps: + 1. The user provides a URI, which may consist of a scheme (protocol), + an authority (host, port) and a path. + 2. The client parses the URI and appends `/.well-knwon/openeo` to the + path. Make sure to correctly handle leading/trailing slashes. + 3. Send a request to the new URI. + A. On success: Detect the most suitable API instance/version (see above) + and read the [capabilites](#tag/Capabilities/operation/capabilities) + from there. + B. On failure: Directly try to read the capabilities from the original URI + given by the user. + + **This URI MUST NOT be versioned as the other endpoints.** + If your API is available at `https://example.com/openeo/api/v1.0`, and + you instruct your API users to use `https://example.com/openeo` as connection URI, + the Well-Known URI SHOULD be located at `https://example.com/openeo/.well-known/openeo`. Clients MAY get additional information (e.g. title or description) about a back-end from the most recent version that has the `production` flag From 150bef441a9705cd7f0cea49dcb2cf315760f616 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 15 Nov 2022 18:53:11 +0100 Subject: [PATCH 086/103] Clearly describe that you can add additional properties to requests --- openapi.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/openapi.yaml b/openapi.yaml index 7c7ac79d..c352a18b 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2208,6 +2208,8 @@ paths: $ref: '#/components/schemas/billing_plan_null_default' log_level: $ref: '#/components/schemas/min_log_level_default' + additionalProperties: + description: You can add additional back-end specific properties here. /process_graphs: get: summary: List all user-defined processes @@ -2664,6 +2666,8 @@ paths: $ref: '#/components/schemas/budget' log_level: $ref: '#/components/schemas/min_log_level_default' + additionalProperties: + description: You can add additional back-end specific properties here. description: The base data for the secondary web service to create '/services/{service_id}': parameters: @@ -2888,6 +2892,8 @@ paths: $ref: '#/components/schemas/budget' log_level: $ref: '#/components/schemas/min_log_level_default' + additionalProperties: + description: You can add additional back-end specific properties here. description: 'Specifies the job details, e.g. the user-defined process and billing details.' '/jobs/{job_id}': parameters: From 65d3f58445649c58ab05b8b609a46596331cbc21 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 16 Nov 2022 17:04:46 +0100 Subject: [PATCH 087/103] Conformance classes / OGC API and STAC API alignment (#468) * Align with OGC APIs regarding /conformance, add conformance classes for openEO #186 --- CHANGELOG.md | 1 + extensions/commercial-data/README.md | 1 + extensions/federation/README.md | 1 + openapi.yaml | 27 +++++++++++++++++++-------- 4 files changed, 22 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d922c7c..4c5eaa8c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Relaxed requirement that unsupported endpoints must return HTTP status code 501. Instead also HTTP status code 404 can be used (and is regularly used in practice). [#415](https://github.com/Open-EO/openeo-api/issues/415) - Minimum value for `costs` and `budget` is 0. - `GET /jobs/{job_id}/estimate`: If a batch job can't be estimated reliably, a `EstimateComplexity` error should be returned. [#443](https://github.com/Open-EO/openeo-api/issues/443) +- The `/conformance` endpoint is now generally used for OGC APIs, STAC API and openEO. All openEO and all extensions got individual conformance classes. [#186](https://github.com/Open-EO/openeo-api/issues/186) ### Fixed diff --git a/extensions/commercial-data/README.md b/extensions/commercial-data/README.md index b4dc50ed..bf32e3f7 100644 --- a/extensions/commercial-data/README.md +++ b/extensions/commercial-data/README.md @@ -5,6 +5,7 @@ The Commercial Data API extension provides an interface for discovering, orderin - Version: **0.1.0** - Stability: **experimental** - [OpenAPI document](openapi.yaml) +- Conformance class: `https://api.openeo.org/extensions/commercial-data/0.1.0` **Note:** This document only documents the additions to the specification. Extensions can not change or break existing behavior of the openEO API. diff --git a/extensions/federation/README.md b/extensions/federation/README.md index 515fecf3..f3d32ea8 100644 --- a/extensions/federation/README.md +++ b/extensions/federation/README.md @@ -6,6 +6,7 @@ This is an extension for federation aspects, i.e. where multiple back-ends are e - Version: **0.1.0** - Stability: **experimental** +- Conformance class: `https://api.openeo.org/extensions/federation/0.1.0` **Note:** This document only documents the additions to the specification. Extensions can not change or break existing behavior of the openEO API. diff --git a/openapi.yaml b/openapi.yaml index c352a18b..1847ef3a 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1172,14 +1172,22 @@ paths: $ref: '#/components/responses/server_error' /conformance: get: - summary: OGC specifications this API conforms to + summary: Conformance classes this API implements operationId: conformance - description: >- - Lists all conformance classes specified in OGC standards that the - server conforms to. - - This endpoint is only required if full compliance with OGC API standards is desired. - Therefore, openEO back-ends MAY implement it for compatibility with OGC API clients and openEO clients don't need to request it. + description: |- + Lists all conformance classes specified in various standards that the + implementation conforms to. Conformance classes are commonly used in + all OGC APIs and the STAC API specification. openEO adds relatively + broadly defined conformance classes, especially for the extensions. + Otherwise, the implemented functionality can usually be retrieved from + the [capabilties](#tag/Capabilities/operation/capabilities) in openEO. + + The general openEO conformance class is `https://api.openeo.org/1.1.0`. + See the individual extensions for their conformance classes. + + More details: + - [STAC API](https://github.com/radiantearth/stac-api-spec), especially "STAC API - Collections and Features Specification" + - [OGC APIs](https://ogcapi.ogc.org/) tags: - Capabilities responses: @@ -1201,7 +1209,10 @@ paths: example: http://www.opengis.net/spec/ogcapi-features-1/1.0/conf/core example: conformsTo: - - 'http://www.opengis.net/spec/ogcapi-features-1/1.0/conf/core' + - https://api.openeo.org/1.1.0 + - https://api.openeo.org/extensions/commercial-data/0.1.0 + - https://api.openeo.org/extensions/federation/0.1.0 + - https://api.stacspec.org/v1.0.0-rc.2/collections 5XX: $ref: '#/components/responses/server_error' /collections: From 8a112b7fa44056abbc30f484634d63a519e23352 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 17 Nov 2022 12:42:01 +0100 Subject: [PATCH 088/103] Update collection summaries to 1.0.0 #469 --- CHANGELOG.md | 2 +- openapi.yaml | 59 ++++++++++++++++++++++++++++++++++------------------ 2 files changed, 40 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c5eaa8c..83c1da04 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,7 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- Updated STAC specification examples and references to v1.0.0. +- Updated STAC specification examples and references to v1.0.0, please see the [STAC changelog](https://github.com/radiantearth/stac-spec/blob/master/CHANGELOG.md) for all changes between 0.9 and 1.0. - `cube:dimensions`: `reference_system` is allowed to be PROJJSON, too. - Relaxed requirement that unsupported endpoints must return HTTP status code 501. Instead also HTTP status code 404 can be used (and is regularly used in practice). [#415](https://github.com/Open-EO/openeo-api/issues/415) - Minimum value for `costs` and `budget` is 0. diff --git a/openapi.yaml b/openapi.yaml index 1847ef3a..bc10ad7e 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -4452,9 +4452,9 @@ components: SAR, Satellite or Scientific) or even custom extensions. Summaries are either a unique set of all available - values *or* statistics. Statistics by default only - specify the range (minimum and maximum values), but - can optionally be accompanied by additional + values, statistics *or* a JSON Schema. Statistics only + specify the range (minimum and maximum values) by default, + but can optionally be accompanied by additional statistical values. The range can specify the potential range of values, but it is recommended to be as precise as possible. The set of values MUST contain @@ -4482,6 +4482,7 @@ components: items: description: A value of any type. - $ref: '#/components/schemas/collection_summary_stats' + - $ref: '#/components/schemas/json_schema' assets: description: |- Dictionary of asset objects for data that can be downloaded, @@ -4640,7 +4641,7 @@ components: - metadata collection_summary_stats: type: object - title: Statistics + title: Statistics / Range description: >- By default, only ranges with a minimum and a maximum value can be specified. Ranges can be @@ -4652,14 +4653,16 @@ components: to add other derived statistical values to the object, for example `mean` or `stddev`. required: - - min - - max + - minimum + - maximum properties: - min: + minimum: + description: The minimum value (inclusive). anyOf: - type: string - type: number - max: + maximum: + description: The maximum value (inclusive). anyOf: - type: string - type: number @@ -5703,7 +5706,7 @@ components: the same level. For example, if type is string, then default can be "foo" but cannot be 1. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.10.2). allOf: - - $ref: '#/components/schemas/json_schema' + - $ref: '#/components/schemas/process_json_schema' error: title: General Error description: >- @@ -5802,14 +5805,14 @@ components: title: Data Types description: Either a single data type or a list of data types. oneOf: - - $ref: '#/components/schemas/json_schema' + - $ref: '#/components/schemas/process_json_schema' - title: Multiple data types description: A list of data types this parameter supports, specified as JSON Schemas. type: array minItems: 1 uniqueItems: true items: - $ref: '#/components/schemas/json_schema' + $ref: '#/components/schemas/process_json_schema' parameter_schema: title: Parameter Data Types description: Either a single data type or a list of data types. @@ -5848,10 +5851,10 @@ components: schema: $ref: '#/components/schemas/data_type_schema' allOf: - - $ref: '#/components/schemas/json_schema' + - $ref: '#/components/schemas/process_json_schema' allOf: - - $ref: '#/components/schemas/json_schema' - json_schema: + - $ref: '#/components/schemas/process_json_schema' + process_json_schema: type: object title: Single data type description: |- @@ -5869,6 +5872,23 @@ components: JSON Schemas SHOULD always be dereferenced (i.e. all `$refs` should be resolved). This allows clients to consume the schemas much better. Clients are not expected to support dereferencing `$refs`. + Note: The specified schema is only a common subset of JSON Schema. Additional keywords MAY be used. + allOf: + - $ref: '#/components/schemas/json_schema' + - properties: + subtype: + type: string + description: The allowed sub data type for a value. See the chapter on [subtypes](#section/Processes/Defining-Processes) for more information. + deprecated: + $ref: '#/components/schemas/deprecated' + json_schema: + type: object + title: JSON Schema + description: |- + A JSON Schema compliant to [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html). + JSON Schemas SHOULD always be dereferenced (i.e. all `$refs` should be resolved). This allows clients to consume the schemas much better. + Clients are not expected to support dereferencing `$refs`. + Note: The specified schema is only a common subset of JSON Schema. Additional keywords MAY be used. properties: type: @@ -5883,9 +5903,6 @@ components: uniqueItems: true items: $ref: '#/components/schemas/json_schema_type' - subtype: - type: string - description: The allowed sub data type for a value. See the chapter on [subtypes](#section/Processes/Defining-Processes) for more information. pattern: type: "string" format: "regex" @@ -5917,9 +5934,11 @@ components: items: $ref: '#/components/schemas/json_schema' - $ref: '#/components/schemas/json_schema' - deprecated: - $ref: '#/components/schemas/deprecated' - additionalProperties: true + additionalProperties: + description: >- + You can add any other property supported by JSON Schema, either + [draft-07](https://json-schema.org/draft-07/json-schema-validation.html) + or any later version. json_schema_type: type: string enum: From ddccbfde62b37be6801496d18ddf34e00d1acd64 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Fri, 9 Dec 2022 15:09:47 +0100 Subject: [PATCH 089/103] Add "pointcloud" to the gis_data_types #475 --- CHANGELOG.md | 1 + openapi.yaml | 1 + 2 files changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 83c1da04..ce5da842 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /`: New Relation types: [#404](https://github.com/Open-EO/openeo-api/issues/404) - `create-form` to link to the registration page - `recovery-form` to link to the credentials recovery page. +- `GET /file_formats`: Add `pointcloud` to the `gis_data_types`. [#475](https://github.com/Open-EO/openeo-api/issues/475) - `GET /me`: New Relation types `alternate` and `related` for user-specific external pages. [#404](https://github.com/Open-EO/openeo-api/issues/404) - `GET /credentials/oidc`: Allow `authorization_code` and `urn:ietf:params:oauth:grant-type:device_code` (both without PKCE) as grants for `default_clients`. [#410](https://github.com/Open-EO/openeo-api/issues/410) - `GET /jobs` and `GET /jobs/{job_id}`: Added a links property that can for example link to results and logs. [#453](https://github.com/Open-EO/openeo-api/issues/453) diff --git a/openapi.yaml b/openapi.yaml index bc10ad7e..519f3810 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -4001,6 +4001,7 @@ components: - raster - vector - table + - pointcloud - other deprecated: $ref: '#/components/schemas/deprecated' From 926fb558f285b512431986b54495ec346b077ef7 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 4 Jan 2023 12:42:27 +0100 Subject: [PATCH 090/103] Clarify that `enabled` is required by removing the default value. #473 (#477) --- CHANGELOG.md | 1 + openapi.yaml | 12 ++++-------- 2 files changed, 5 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ce5da842..ffcef205 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -55,6 +55,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /me`: Clarify the behavior of the field `budget`. - `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs` and `POST /result`: Clarified the formatting of the `message` property. [#455](https://github.com/Open-EO/openeo-api/issues/455) - `GET /jobs/{job_id}/estimate`: Don't require that the costs are the upper limit. Services may specify the costs more freely depending on their terms of service. +- `GET /services` and `GET /services/{service_id}`: Clarify that `enabled` is required by removing the default value. [#473](https://github.com/Open-EO/openeo-api/issues/473) - Several appearances of `nullable` were clarified according to the lint report by Spectral - Clarify how the well-known document works [#460](https://github.com/Open-EO/openeo-api/issues/460) diff --git a/openapi.yaml b/openapi.yaml index 519f3810..6d4037a4 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2668,7 +2668,9 @@ paths: type: $ref: '#/components/schemas/service_type' enabled: - $ref: '#/components/schemas/service_enabled_default' + allOf: + - $ref: '#/components/schemas/service_enabled' + - default: true configuration: $ref: '#/components/schemas/service_configuration' plan: @@ -5598,7 +5600,7 @@ components: type: $ref: '#/components/schemas/service_type' enabled: - $ref: '#/components/schemas/service_enabled_default' + $ref: '#/components/schemas/service_enabled' process: $ref: '#/components/schemas/process_graph_with_metadata' configuration: @@ -5652,12 +5654,6 @@ components: any other service dependant configuration. example: version: 1.3.0 - service_enabled_default: - type: boolean - description: >- - Describes whether a secondary web service is responding to requests - (true) or not (false). Disabled services don't produce any costs. - default: true service_enabled: type: boolean description: >- From 749bfc724c1c4c43205737288c40f30c7f94f50d Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Mon, 30 Jan 2023 16:18:01 +0100 Subject: [PATCH 091/103] Vector data cubes (#441) --- CHANGELOG.md | 3 + openapi.yaml | 426 +++++++++++++++++++++++++++++++++------------------ 2 files changed, 277 insertions(+), 152 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ffcef205..b648b378 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Recommendation to add media types and titles to links for a better user experience. - Allow the relation type `canonical` to be used generally for (shared) resources (e.g. UDPs or batch jobs) without requiring Bearer authentication. [#405](https://github.com/Open-EO/openeo-api/issues/405) - Recommendation for UDF runtime names. [#409](https://github.com/Open-EO/openeo-api/issues/409) +- Processes: Added `dimensions` schema for subtype `datacube` +- Collections: Added `vector` dimension type to `cube:dimensions` ### Changed @@ -50,6 +52,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Clarify the fields `plan` (for processing requests) and `billing_plan` (in `GET /` and `GET /me`). [#425](https://github.com/Open-EO/openeo-api/issues/425) [#426](https://github.com/Open-EO/openeo-api/issues/426) - Reflect that the `debug` process has been renamed to `inspect`. - Clarified uniqueness constraints for identifiers. [#449](https://github.com/Open-EO/openeo-api/issues/449) [#454](https://github.com/Open-EO/openeo-api/issues/454) +- Clarified schematically the applicability of JSON Schema extensions (`parameters`, `returns`, `dimensions`) and their relation to the subtypes - `GET /`: Removed the superfluous default value for `currency`. [#423](https://github.com/Open-EO/openeo-api/issues/423) - `GET /credentials/oidc`: Clarify that clients may add additional scopes - `GET /me`: Clarify the behavior of the field `budget`. diff --git a/openapi.yaml b/openapi.yaml index 6d4037a4..f8bad098 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -267,10 +267,11 @@ info: Each process parameter and the return values of a process define a schema that the value MUST comply to. The schemas are based on [JSON Schema draft-07](http://json-schema.org/). - Three custom keywords have been defined: + Multiple custom keywords have been defined: * `subtype` for more fine-grained data-types than JSON Schema supports. - * `parameters` to specify the parameters of a process graph (object with subtype `process-graph`). - * `returns` to describe the return value of a process graph (object with subtype `process-graph`). + * `dimensions` to further define the dimension types required if the `subtype` is `datacube`. + * `parameters` to specify the parameters of a process graph if the `subtype` is `process-graph`. + * `returns` to describe the return value of a process graph if the `subtype` is `process-graph`. ### Subtypes @@ -280,7 +281,7 @@ info: and standardizes a number of openEO related data types that extend the native data types, for example: `bounding-box` (object with at least `west`, `south`, `east` and `north` properties), `date-time` (string representation of date and time following RFC 3339), - `raster-cube` (the type of data cubes), etc. + `datacube` (a datacube with dimensions), etc. The subtypes should be re-used in process schema definitions whenever suitable. If a general data type such as `string` or `number` is used in a schema, all subtypes with the same parent data type can be passed, too. @@ -1394,7 +1395,7 @@ paths: example: stac_version: 1.0.0 stac_extensions: - - https://stac-extensions.github.io/datacube/v2.0.0/schema.json + - https://stac-extensions.github.io/datacube/v2.2.0/schema.json type: Collection id: Sentinel-2 title: Sentinel-2 MSI L2A @@ -1627,7 +1628,7 @@ paths: description: A data cube. schema: type: object - subtype: raster-cube + subtype: datacube - name: process description: 'A unary process to be applied on each value, may consist of multiple sub-processes.' schema: @@ -1642,7 +1643,7 @@ paths: description: 'A data cube with the newly computed values. The resolution, cardinality and the number of dimensions are the same as for the original data cube.' schema: type: object - subtype: raster-cube + subtype: datacube - id: multiply summary: Multiplication of two numbers description: |- @@ -2471,7 +2472,7 @@ paths: description: List of parameters made available to user-defined processes. type: array items: - $ref: '#/components/schemas/parameter' + $ref: '#/components/schemas/process_parameter' links: description: |- Links related to this service type, e.g. more @@ -3809,6 +3810,12 @@ components: default: finished batch_job_result: title: Batch Job Results Response as STAC Item + description: + The STAC specification should be the main guidance for implementing this. + + Specifying the `bbox` is strongly RECOMMENDED for STAC compliance, + but can be omitted if the result is unlocated and the `geometry` + is set to `null`. type: object required: - stac_version @@ -3835,41 +3842,7 @@ components: enum: - Feature bbox: - description: |- - Potential *spatial extent* covered by the data. - - The bounding box is provided as four or six numbers. Six numbers are - specified, if the coordinate reference system includes a vertical axis - (height or depth). The order of the elements in the array: - - - West (lower left corner, coordinate axis 1) - - South (lower left corner, coordinate axis 2) - - Base (optional, lower left corner, coordinate axis 3) - - East (upper right corner, coordinate axis 1) - - North (upper right corner, coordinate axis 2) - - Height (optional, upper right corner, coordinate axis 3) - - The coordinate reference system of the values is WGS84 - longitude/latitude. - - Specifying the `bbox` is strongly RECOMMENDED for STAC compliance, - but can be omitted if the result is unlocated and the `geometry` - is set to `null`. - type: array - oneOf: - - title: 4 elements - minItems: 4 - maxItems: 4 - - title: 6 elements - minItems: 6 - maxItems: 6 - items: - type: number - example: - - -180 - - -90 - - 180 - - 90 + $ref: '#/components/schemas/bbox' geometry: description: |- Defines the full footprint of the asset represented by this item as @@ -4316,46 +4289,7 @@ components: type: array minItems: 1 items: - description: |- - Each bounding box is provided as four or six numbers, - depending on whether the coordinate reference system - includes a vertical axis (height or depth): - - * West (lower left corner, coordinate axis 1) - * South (lower left corner, coordinate axis 2) - * Base (optional, minimum value, coordinate axis 3) - * East (upper right corner, coordinate axis 1) - * North (upper right corner, coordinate axis 2) - * Height (optional, maximum value, coordinate axis 3) - - The coordinate reference system of the values is WGS 84 - longitude/latitude (http://www.opengis.net/def/crs/OGC/1.3/CRS84). - - For WGS 84 longitude/latitude the values are in most cases - the sequence of minimum longitude, minimum latitude, maximum - longitude and maximum latitude. - - However, in cases where the box spans the antimeridian the - first value (west-most box edge) is larger than the third value - (east-most box edge). - - If the vertical axis is included, the third and the sixth - number are the bottom and the top of the 3-dimensional bounding box. - type: array - oneOf: - - title: 4 elements - minItems: 4 - maxItems: 4 - - title: 6 elements - minItems: 6 - maxItems: 6 - items: - type: number - example: - - -180 - - -90 - - 180 - - 90 + $ref: '#/components/schemas/bbox' temporal: title: Collection Temporal Extent description: >- @@ -4440,9 +4374,10 @@ components: * `z` for the dimension of type `spatial` with the axis set to `z` * `t` for the dimension of type `temporal` * `bands` for dimensions of type `bands` + * `geometries` for dimensions of type `geometries` - This property REQUIRES to add the data cube extension to the list - of `stac_extensions`: `https://stac-extensions.github.io/datacube/v2.1.0/schema.json`. + This property REQUIRES to add a version of the data cube extension to the list + of `stac_extensions`, e.g. `https://stac-extensions.github.io/datacube/v2.2.0/schema.json`. type: object additionalProperties: x-additionalPropertiesName: Dimension Name @@ -4669,6 +4604,47 @@ components: anyOf: - type: string - type: number + bbox: + description: |- + Each bounding box is provided as four or six numbers, + depending on whether the coordinate reference system + includes a vertical axis (height or depth): + + * West (lower left corner, coordinate axis 1) + * South (lower left corner, coordinate axis 2) + * Base (optional, minimum value, coordinate axis 3) + * East (upper right corner, coordinate axis 1) + * North (upper right corner, coordinate axis 2) + * Height (optional, maximum value, coordinate axis 3) + + The coordinate reference system of the values is WGS 84 + longitude/latitude (http://www.opengis.net/def/crs/OGC/1.3/CRS84). + + For WGS 84 longitude/latitude the values are in most cases + the sequence of minimum longitude, minimum latitude, maximum + longitude and maximum latitude. + + However, in cases where the box spans the antimeridian the + first value (west-most box edge) is larger than the third value + (east-most box edge). + + If the vertical axis is included, the third and the sixth + number are the bottom and the top of the 3-dimensional bounding box. + type: array + oneOf: + - title: 4 elements + minItems: 4 + maxItems: 4 + - title: 6 elements + minItems: 6 + maxItems: 6 + items: + type: number + example: + - -180 + - -90 + - 180 + - 90 collection_id: type: string description: >- @@ -4678,17 +4654,19 @@ components: example: Sentinel-2A dimension: title: Dimension + description: A dimension, each object represents a distinct dimension with the key being the dimension name. type: object required: - type properties: type: - description: 'Type of the dimension.' + description: Type of the dimension. type: string enum: - spatial - temporal - bands + - geometries - other description: $ref: '#/components/schemas/description' @@ -4698,12 +4676,21 @@ components: spatial: '#/components/schemas/dimension_spatial' temporal: '#/components/schemas/dimension_temporal' bands: '#/components/schemas/dimension_bands' + geometries: '#/components/schemas/dimension_geometries' other: '#/components/schemas/dimension_other' dimension_other: allOf: - $ref: '#/components/schemas/dimension' - title: Additional Dimension type: object +# This is not supported by ReDoc yet: +# oneOf: +# - title: Additional Dimension with Extent +# required: +# - extent +# - title: Additional Dimension with Values +# required: +# - values properties: extent: $ref: '#/components/schemas/collection_dimension_extent_open' @@ -4714,8 +4701,40 @@ components: unit: $ref: '#/components/schemas/collection_dimension_unit' reference_system: - description: The reference system for the data. + description: The reference system for the dimension. type: string + dimension_geometries: + allOf: + - $ref: '#/components/schemas/dimension' + - title: Geometries Dimension + type: object + required: + - bbox + properties: + axes: + description: Axes of the vector dimension as an ordered set of `x`, `y` and `z`. Defaults to `x` and `y`. + default: + - 'x' + - 'y' + type: array + uniqueItems: true + items: + $ref: '#/components/schemas/dimension_axis_xyz' + bbox: + $ref: '#/components/schemas/bbox' + values: + description: Optionally, a representation of the vectors. This can be a list of WKT string or other free-form identifiers. + type: array + items: + type: string + geometry_types: + description: A set of all geometry types included in this dimension. If not present, mixed geometry types must be assumed. + type: array + uniqueItems: true + items: + $ref: '#/components/schemas/geometry_type' + reference_system: + $ref: '#/components/schemas/collection_dimension_srs' dimension_bands: allOf: - $ref: '#/components/schemas/dimension' @@ -4737,18 +4756,13 @@ components: allOf: - $ref: '#/components/schemas/dimension' - title: Spatial Dimension - description: A spatial dimension in one of the horizontal (x or y) or vertical (z) directions. + description: A spatial (raster) dimension in one of the horizontal (x or y) or vertical (z) directions. type: object required: - axis properties: axis: - description: 'Axis of the spatial dimension (`x`, `y` or `z`).' - type: string - enum: - - 'x' - - 'y' - - 'z' + $ref: '#/components/schemas/dimension_axis_xyz' extent: description: >- Extent (lower and upper bounds) of the @@ -4768,22 +4782,21 @@ components: step: $ref: '#/components/schemas/collection_dimension_step' reference_system: - description: >- - The spatial reference system for the data, specified as [EPSG code](http://www.epsg-registry.org/), [WKT2 (ISO 19162) string](http://docs.opengeospatial.org/is/18-010r7/18-010r7.html), [PROJJSON object](https://proj.org/specifications/projjson.html) or [PROJ definition (deprecated)](https://proj.org/usage/quickstart.html). Defaults to EPSG code 4326. - default: 4326 - oneOf: - - type: number - title: EPSG code - - type: string - title: WKT2 or PROJ definition (deprecated) - - type: object - title: PROJJSON + $ref: '#/components/schemas/collection_dimension_srs' discriminator: propertyName: axis mapping: x: '#/components/schemas/dimension_spatial_horizontal' y: '#/components/schemas/dimension_spatial_horizontal' z: '#/components/schemas/dimension_spatial_vertical' + dimension_axis_xyz: + title: Axis + description: Axis of a geometry or dimension (`x`, `y` or `z`) + type: string + enum: + - 'x' + - 'y' + - 'z' dimension_spatial_horizontal: allOf: - $ref: '#/components/schemas/dimension_spatial' @@ -4851,6 +4864,18 @@ components: spaced steps. type: string nullable: true + collection_dimension_srs: + title: Spatial reference system + description: >- + The spatial reference system for the data, specified as [EPSG code](http://www.epsg-registry.org/), [WKT2 (ISO 19162) string](http://docs.opengeospatial.org/is/18-010r7/18-010r7.html), [PROJJSON object](https://proj.org/specifications/projjson.html) or [PROJ definition (deprecated)](https://proj.org/usage/quickstart.html). Defaults to EPSG code 4326. + default: 4326 + oneOf: + - type: number + title: EPSG code + - type: string + title: WKT2 or PROJ definition (deprecated) + - type: object + title: PROJJSON collection_dimension_extent_open: description: >- If the dimension consists of @@ -5251,7 +5276,7 @@ components: description: $ref: '#/components/schemas/process_description' schema: - $ref: '#/components/schemas/data_type_schema' + $ref: '#/components/schemas/process_schema' experimental: type: boolean description: >- @@ -5388,7 +5413,7 @@ components: - schema properties: schema: - $ref: '#/components/schemas/parameter_schema' + $ref: '#/components/schemas/process_schema' allOf: - $ref: '#/components/schemas/base_parameter' batch_job: @@ -5810,22 +5835,57 @@ components: uniqueItems: true items: $ref: '#/components/schemas/process_json_schema' - parameter_schema: - title: Parameter Data Types - description: Either a single data type or a list of data types. + process_schema: + title: Process Data types + description: Either a single data type or a list of data types for process parameter or process return values. oneOf: - - $ref: '#/components/schemas/parameter_json_schema' + - $ref: '#/components/schemas/process_json_schema' - title: Multiple data types description: A list of data types supported, specified as JSON Schemas. type: array minItems: 1 uniqueItems: true items: - $ref: '#/components/schemas/parameter_json_schema' - parameter_json_schema: + $ref: '#/components/schemas/process_json_schema' + process_json_schema: + type: object title: Single Data Type + description: |- + Specifies a data type supported by a parameter or return value. + + The data types are specified according to the [JSON Schema draft-07](http://json-schema.org/) specification. + See the chapter ['Schemas' in 'Defining Processes'](#section/Processes/Defining-Processes) for more information. + + JSON Schemas SHOULD NOT contain `default`, `anyOf`, `oneOf`, `allOf` or `not` at the top-level of the schema. + Instead specify each data type in a separate array element. + + The following more complex JSON Schema keywords SHOULD NOT be used: + `if`, `then`, `else`, `readOnly`, `writeOnly`, `dependencies`, `minProperties`, `maxProperties`, `patternProperties`. + + JSON Schemas SHOULD always be dereferenced (i.e. all `$refs` should be resolved). This allows clients to consume the schemas much better. + Clients are not expected to support dereferencing `$refs`. + + Note: The specified schema is only a common subset of JSON Schema. Additional keywords MAY be used. + properties: + subtype: + type: string + description: The allowed sub data type for a value. See the chapter on [subtypes](#section/Processes/Defining-Processes) for more information. + deprecated: + $ref: '#/components/schemas/deprecated' + allOf: + - $ref: '#/components/schemas/json_schema' + oneOf: + - title: Generic + - $ref: '#/components/schemas/process_graph_json_schema' + - $ref: '#/components/schemas/datacube_json_schema' + process_graph_json_schema: + title: Process Graph type: object properties: + subtype: + type: string + enum: + - process-graph parameters: type: array title: Process Graph Parameters @@ -5839,7 +5899,8 @@ components: returns: type: object title: Process Graph Return Value - description: Description of the data that is returned by the child process graph. + description: |- + Description of the data that is returned by the child process graph. required: - schema properties: @@ -5849,35 +5910,57 @@ components: $ref: '#/components/schemas/data_type_schema' allOf: - $ref: '#/components/schemas/process_json_schema' - allOf: - - $ref: '#/components/schemas/process_json_schema' - process_json_schema: - type: object - title: Single data type - description: |- - Specifies a data type supported by a parameter or return value. - - The data types are specified according to the [JSON Schema draft-07](http://json-schema.org/) specification. - See the chapter ['Schemas' in 'Defining Processes'](#section/Processes/Defining-Processes) for more information. - - JSON Schemas SHOULD NOT contain `default`, `anyOf`, `oneOf`, `allOf` or `not` at the top-level of the schema. - Instead specify each data type in a separate array element. - - The following more complex JSON Schema keywords SHOULD NOT be used: - `if`, `then`, `else`, `readOnly`, `writeOnly`, `dependencies`, `minProperties`, `maxProperties`, `patternProperties`. - - JSON Schemas SHOULD always be dereferenced (i.e. all `$refs` should be resolved). This allows clients to consume the schemas much better. - Clients are not expected to support dereferencing `$refs`. - - Note: The specified schema is only a common subset of JSON Schema. Additional keywords MAY be used. - allOf: - - $ref: '#/components/schemas/json_schema' - - properties: - subtype: - type: string - description: The allowed sub data type for a value. See the chapter on [subtypes](#section/Processes/Defining-Processes) for more information. - deprecated: - $ref: '#/components/schemas/deprecated' + datacube_json_schema: + title: Datacube + properties: + subtype: + type: string + enum: + - datacube + dimensions: + title: Datacube constraints + description: |- + Allows to specify requirements the data cube has to fulfill. + Right now, it only allows to specify the dimension types and + adds for specific dimension types: + * axes for `spatial` dimensions in raster datacubes + * geometry types for `geometries` dimensions in vector datacubes + type: array + items: + type: object + required: + - type + oneOf: + - title: Spatial (raster) + properties: + type: + type: string + enum: + - spatial + axis: + type: array + minItems: 1 + items: + $ref: '#/components/schemas/dimension_axis_xyz' + - title: Spatial (vector) + properties: + type: + type: string + enum: + - geometries + geometry_type: + type: array + minItems: 1 + items: + $ref: '#/components/schemas/geometry_type' + - title: Other + properties: + type: + type: string + enum: + - bands + - temporal + - other json_schema: type: object title: JSON Schema @@ -5976,6 +6059,17 @@ components: Date and time the file has lastly been modified, formatted as a [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) date-time. example: '2018-01-03T10:55:29Z' + geometry_type: + title: Geometry type + type: string + enum: + - Point + - MultiPoint + - LineString + - MultiLineString + - Polygon + - MultiPolygon + - GeometryCollection GeoJsonPoint3D: type: array description: Point in 3D space @@ -5996,22 +6090,50 @@ components: - Point coordinates: $ref: '#/components/schemas/GeoJsonPoint3D' - GeoJsonGeometry: - title: GeoJSON Geometry + GeoJsonFeatureCollection: type: object required: - type + - features properties: type: type: string enum: - - Point - - LineString - - Polygon - - MultiPoint - - MultiLineString - - MultiPolygon - - GeometryCollection + - FeatureCollection + features: + type: array + items: + $ref: '#/components/schemas/GeoJsonFeature' + GeoJsonFeature: + type: object + required: + - type + - geometry + - properties + properties: + type: + type: string + enum: + - Feature + geometry: + $ref: '#/components/schemas/GeoJsonGeometry' + properties: + type: object + nullable: true + GeoJson: + title: GeoJSON + oneOf: + - $ref: '#/components/schemas/GeoJsonFeatureCollection' + - $ref: '#/components/schemas/GeoJsonFeature' + - $ref: '#/components/schemas/GeoJsonGeometry' + GeoJsonGeometry: + title: GeoJSON Geometry + type: object + required: + - type + properties: + type: + $ref: '#/components/schemas/geometry_type' discriminator: propertyName: type mapping: @@ -6146,7 +6268,7 @@ components: Data of any type. It is the back-ends task to decide how to best present passed data to a user. - For example, a raster-cube passed to the `inspect` SHOULD return the + For example, a datacube passed to the `inspect` SHOULD return the metadata similar to the collection metadata, including `cube:dimensions`. There are implementation guidelines available for the `inspect` process. path: From 6da01956eadc12f5ab5389036722585e0e9916e5 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Fri, 10 Mar 2023 17:12:59 +0100 Subject: [PATCH 092/103] Update name of the geometries dimension #479 --- CHANGELOG.md | 2 +- openapi.yaml | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b648b378..a96b415e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,7 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Allow the relation type `canonical` to be used generally for (shared) resources (e.g. UDPs or batch jobs) without requiring Bearer authentication. [#405](https://github.com/Open-EO/openeo-api/issues/405) - Recommendation for UDF runtime names. [#409](https://github.com/Open-EO/openeo-api/issues/409) - Processes: Added `dimensions` schema for subtype `datacube` -- Collections: Added `vector` dimension type to `cube:dimensions` +- Collections: Added `geometry` dimension type to `cube:dimensions` ### Changed diff --git a/openapi.yaml b/openapi.yaml index f8bad098..5d06b8f2 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -4374,7 +4374,7 @@ components: * `z` for the dimension of type `spatial` with the axis set to `z` * `t` for the dimension of type `temporal` * `bands` for dimensions of type `bands` - * `geometries` for dimensions of type `geometries` + * `geometry` for dimensions of type `geometry` This property REQUIRES to add a version of the data cube extension to the list of `stac_extensions`, e.g. `https://stac-extensions.github.io/datacube/v2.2.0/schema.json`. @@ -4666,7 +4666,7 @@ components: - spatial - temporal - bands - - geometries + - geometry - other description: $ref: '#/components/schemas/description' @@ -4676,7 +4676,7 @@ components: spatial: '#/components/schemas/dimension_spatial' temporal: '#/components/schemas/dimension_temporal' bands: '#/components/schemas/dimension_bands' - geometries: '#/components/schemas/dimension_geometries' + geometry: '#/components/schemas/dimension_geometry' other: '#/components/schemas/dimension_other' dimension_other: allOf: @@ -4703,10 +4703,10 @@ components: reference_system: description: The reference system for the dimension. type: string - dimension_geometries: + dimension_geometry: allOf: - $ref: '#/components/schemas/dimension' - - title: Geometries Dimension + - title: Geometry Dimension type: object required: - bbox @@ -5924,7 +5924,7 @@ components: Right now, it only allows to specify the dimension types and adds for specific dimension types: * axes for `spatial` dimensions in raster datacubes - * geometry types for `geometries` dimensions in vector datacubes + * geometry types for `geometry` dimensions in vector datacubes type: array items: type: object @@ -5947,7 +5947,7 @@ components: type: type: string enum: - - geometries + - geometry geometry_type: type: array minItems: 1 From d72b34762f1e8cae68f23ef78cb697167f3a5612 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 30 Mar 2023 11:44:03 +0200 Subject: [PATCH 093/103] Update CI --- .github/workflows/docs.yml | 6 +++--- .github/workflows/tests.yml | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 63e6176d..6480d290 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -12,10 +12,10 @@ jobs: steps: - name: Inject env variables uses: rlespinasse/github-slug-action@v3.x - - uses: actions/setup-node@v1 + - uses: actions/setup-node@v3 with: - node-version: '16' - - uses: actions/checkout@v2 + node-version: 'lts/*' + - uses: actions/checkout@v3 - run: | npm install npm run build diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 0f896959..64b0febd 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -4,10 +4,10 @@ jobs: deploy: runs-on: ubuntu-latest steps: - - uses: actions/setup-node@v1 + - uses: actions/setup-node@v3 with: - node-version: '16' - - uses: actions/checkout@v2 + node-version: 'lts/*' + - uses: actions/checkout@v3 - name: Run tests run: | npm install From b7398b3129e0a1d379dab684a0f23528af5c7548 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Thu, 30 Mar 2023 11:44:37 +0200 Subject: [PATCH 094/103] Fix batch job result summary #482 --- openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi.yaml b/openapi.yaml index 5d06b8f2..1eb124ca 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -3117,7 +3117,7 @@ paths: parameters: - $ref: '#/components/parameters/job_id' get: - summary: Download results for a batch job + summary: List batch job results operationId: list-results description: |- Lists signed URLs pointing to the processed files, usually after the batch job From c3f8512ca2c7f3a20c49b0b0540e3f255ee86216 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 4 Apr 2023 15:21:24 +0200 Subject: [PATCH 095/103] Fixed broken STAC (internal) link #486 --- openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi.yaml b/openapi.yaml index 1eb124ca..afe83d34 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -4435,7 +4435,7 @@ components: type: string description: >- The [version of the STAC specification](https://github.com/radiantearth/stac-spec/releases), - which MAY not be equal to the [STAC API version](#section/STAC). + which MAY not be equal to the [STAC API version](#tag/EO-Data-Discovery/STAC). Supports versions 0.9.x and 1.x.x. pattern: '^(0\.9.\d+|1\.\d+.\d+)' example: 1.0.0 From 47ae3226a5cf2b17aed8f76b7f6c149647e6687f Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 11 Apr 2023 13:07:45 +0200 Subject: [PATCH 096/103] Explicitly specify metadata filters for collections through queryables #396 (#487) --- CHANGELOG.md | 2 + openapi.yaml | 130 ++++++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 110 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a96b415e..f507139b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Recommendation for UDF runtime names. [#409](https://github.com/Open-EO/openeo-api/issues/409) - Processes: Added `dimensions` schema for subtype `datacube` - Collections: Added `geometry` dimension type to `cube:dimensions` +- New endpoint for metadata filters (queryables): `/collections/{collection_id}/queryables`. Also adds a new rel type to the collection links. [#396](https://github.com/Open-EO/openeo-api/issues/396) ### Changed @@ -61,6 +62,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /services` and `GET /services/{service_id}`: Clarify that `enabled` is required by removing the default value. [#473](https://github.com/Open-EO/openeo-api/issues/473) - Several appearances of `nullable` were clarified according to the lint report by Spectral - Clarify how the well-known document works [#460](https://github.com/Open-EO/openeo-api/issues/460) +- Clarify handling of JSON Schema versions ## [1.1.0] - 2021-05-17 diff --git a/openapi.yaml b/openapi.yaml index afe83d34..34ef8f05 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1374,12 +1374,7 @@ paths: - {} - Bearer: [] parameters: - - name: collection_id - in: path - description: Collection identifier - required: true - schema: - $ref: '#/components/schemas/collection_id' + - $ref: '#/components/parameters/collection_id' responses: '200': description: JSON object with the full collection metadata. @@ -1436,6 +1431,9 @@ paths: - rel: license href: https://scihub.copernicus.eu/twiki/pub/SciHubWebPortal/TermsConditions/Sentinel_Data_Terms_and_Conditions.pdf type: application/pdf + - rel: http://www.opengis.net/def/rel/ogc/1.0/queryables + href: https://example.openeo.org/api/collections/Sentinel-2A/queryables + type: application/schema+json - rel: about href: https://earth.esa.int/web/sentinel/user-guides/sentinel-2-msi/product-types/level-1c type: text/html @@ -1574,6 +1572,67 @@ paths: $ref: '#/components/responses/client_error_auth' 5XX: $ref: '#/components/responses/server_error' + '/collections/{collection_id}/queryables': + get: + summary: Metadata filters for a specific dataset + operationId: list-collection-queryables + description: |- + Lists **all** supported metadata filters (also called "queryables") for + a specific collection. + + This endpoint is compatible with endpoint defined in the STAC API extension + [`filter`](https://github.com/stac-api-extensions/filter#queryables) and + [OGC API - Features - Part 3: Filtering](https://github.com/opengeospatial/ogcapi-features/tree/master/extensions/filtering). + For a precise definition please follow those specifications. + + This endpoints provides a JSON Schema for each queryable that openEO + users can use in multiple scenarios: + 1. For loading data from the collection, e.g. in the process `load_collection`. + 2. For filtering items using CQL2 on the `/collections/{collection_id}/items` endpoint + (if [STAC API - Features is implemented in addition to the openEO API](#tag/EO-Data-Discovery/STAC)). + + Note: Providing the Bearer token is REQUIRED for private collections. + tags: + - EO Data Discovery + security: + - {} + - Bearer: [] + parameters: + - $ref: '#/components/parameters/collection_id' + responses: + '200': + description: |- + A JSON Schema defining the queryables. + + It is RECOMMENDED to dereference all "$refs". + content: + application/schema+json: + schema: + $ref: '#/components/schemas/json_schema' + example: + $schema: https://json-schema.org/draft/2019-09/schema + $id: https://example.com/api/v1.0/collections/Sentinel-2A/queryables + type: object + title: Sentinel 2A + properties: + 'eo:cloud_cover': + title: Cloud Cover + type: number + minimum: 0 + maximum: 100 + platform: + title: Platform + description: The satellite platform. + type: string + enum: + - sentinel-2a + - sentinel-2b + additionalProperties: false + + 4XX: + $ref: '#/components/responses/client_error_auth' + 5XX: + $ref: '#/components/responses/server_error' /processes: get: summary: Supported predefined processes @@ -4327,7 +4386,7 @@ components: - '2011-11-11T12:22:11Z' - null links: - description: >- + description: |- Links related to this collection. Could reference to licensing information, other meta data formats with additional information or a preview image. @@ -4352,6 +4411,10 @@ components: catalog service such as OGC CSW, a human-readable HTML version or a metadata document following another standard such as ISO 19115 or DCAT. + 6. `http://www.opengis.net/def/rel/ogc/1.0/queryables`: URL to the + queryables endpoint at `/collections/{collection_id}/queryables`. + For JSON Schema documents, the `type` field must be set to `application/schema+json`. + For additional relation types see also the lists of [common relation types in openEO](#section/API-Principles/Web-Linking) and the STAC specification for Collections. @@ -5159,7 +5222,7 @@ components: description: The return value which can by of any data type. links: type: array - description: >- + description: |- Links related to this process, e.g. additional external documentation. It is RECOMMENDED to provide links with the following `rel` (relation) types: @@ -5965,15 +6028,32 @@ components: type: object title: JSON Schema description: |- - A JSON Schema compliant to [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html). - JSON Schemas SHOULD always be dereferenced (i.e. all `$refs` should be resolved). This allows clients to consume the schemas much better. + A JSON Schema compliant to [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html) or later. + + JSON Schemas SHOULD always be dereferenced (i.e. all `$refs` should be resolved). + This allows clients to consume the schemas much better. Clients are not expected to support dereferencing `$refs`. - Note: The specified schema is only a common subset of JSON Schema. Additional keywords MAY be used. + Note: The specified schema in the OpenAPI document is only a common subset of JSON Schema. + Additional keywords from the JSON Schema specification MAY be used. properties: + $schema: + description: |- + The JSON Schema version. If not given in the context of openEO, + defaults to `draft-07`. + + You may need to add the default value for `$schema` property explicitly to the JSON Schema + object before passing it to a JSON Schema validator. + type: string + format: uri + default: http://json-schema.org/draft-07/schema# + $id: + description: ID of your JSON Schema. + type: string + format: uri type: description: |- - The allowed basic data type(s) for a value according to [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.6.1.1). + The allowed basic data type(s) for a value. If this property is not present, all data types are allowed. oneOf: @@ -5986,28 +6066,28 @@ components: pattern: type: "string" format: "regex" - description: The regular expression a string value must match against. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.6.3.3). + description: The regular expression a string value must match against. enum: type: array items: {} - description: An exclusive list of allowed values. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.6.1.2). + description: An exclusive list of allowed values. minimum: type: number - description: The minimum value (inclusive) allowed for a numerical value. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.6.2.4). + description: The minimum value (inclusive) allowed for a numerical value. maximum: type: number - description: The maximum value (inclusive) allowed for a numerical value. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.6.2.2). + description: The maximum value (inclusive) allowed for a numerical value. minItems: type: number minimum: 0 default: 0 - description: The minimum number of items required in an array. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.6.4.4). + description: The minimum number of items required in an array. maxItems: type: number minimum: 0 - description: The maximum number of items required in an array. See [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.6.4.3). + description: The maximum number of items required in an array. items: - description: Specifies schemas for the items in an array according to [JSON Schema draft-07](https://json-schema.org/draft-07/json-schema-validation.html#rfc.section.6.4.1). + description: Specifies schemas for the items in an array. anyOf: - type: array minItems: 1 @@ -6016,9 +6096,8 @@ components: - $ref: '#/components/schemas/json_schema' additionalProperties: description: >- - You can add any other property supported by JSON Schema, either - [draft-07](https://json-schema.org/draft-07/json-schema-validation.html) - or any later version. + You can add any other property supported by the JSON Schema version that is given through the property `$schema`, + so either [draft-07](https://json-schema.org/draft-07/json-schema-validation.html) or any later version. json_schema_type: type: string enum: @@ -6509,6 +6588,13 @@ components: required: true schema: $ref: '#/components/schemas/job_id' + collection_id: + name: collection_id + in: path + description: Collection identifier + required: true + schema: + $ref: '#/components/schemas/collection_id' examples: evi_user_defined_process: description: A user-defined process that computes the EVI. From 380a756700c32408ab1428a59e85a191fe89a5e8 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 18 Apr 2023 15:20:27 +0200 Subject: [PATCH 097/103] Add minimum log_level parameter for GET /../logs #485 (#488) --- CHANGELOG.md | 3 ++- openapi.yaml | 32 ++++++++++++++++++++++++++++---- 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f507139b..b36fd047 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,8 +22,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Recommendation to add a link with relation type `canonical` which points to a signed URL with the same content as the response. [#397](https://github.com/Open-EO/openeo-api/issues/397) - Added metadata field `openeo:status` to indicate the job status (and whether the result is complete or not). - Added parameter `partial` to allow retrieving incomplete results, which must also add the new property `openeo:status` to the metadata. [#430](https://github.com/Open-EO/openeo-api/issues/430) -- `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs` and `POST /result`: Added `log_level` property in responses to reflect the minimum log level that has been chosen by the user. [#329](https://github.com/Open-EO/openeo-api/issues/329) +- `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs`: Added `level` parameter to requests to set the minimum log level returned by the response. [#485](https://github.com/Open-EO/openeo-api/issues/485) - Added property `log_level` to secondary web service, batch job and synchronous processing endpoints to indicate the minimum severity level that should be stored for logs. [#329](https://github.com/Open-EO/openeo-api/issues/329) +- `GET /jobs/{job_id}/logs`, `GET /services/{service_id}/logs` and `POST /result`: Added `level` property in responses to reflect the minimum log level that may appear in the response. [#329](https://github.com/Open-EO/openeo-api/issues/329) - Recommendation to add media types and titles to links for a better user experience. - Allow the relation type `canonical` to be used generally for (shared) resources (e.g. UDPs or batch jobs) without requiring Bearer authentication. [#405](https://github.com/Open-EO/openeo-api/issues/405) - Recommendation for UDF runtime names. [#409](https://github.com/Open-EO/openeo-api/issues/409) diff --git a/openapi.yaml b/openapi.yaml index 34ef8f05..d0efd735 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2857,6 +2857,7 @@ paths: parameters: - $ref: '#/components/parameters/service_id' - $ref: '#/components/parameters/log_offset' + - $ref: '#/components/parameters/log_level' - $ref: '#/components/parameters/pagination_limit' responses: '200': @@ -3164,6 +3165,7 @@ paths: parameters: - $ref: '#/components/parameters/job_id' - $ref: '#/components/parameters/log_offset' + - $ref: '#/components/parameters/log_level' - $ref: '#/components/parameters/pagination_limit' responses: '200': @@ -6458,11 +6460,13 @@ components: - logs - links properties: - log_level: + level: description: |- The minimum severity level for log entries that the back-end returns. - This property MUST reflect the `log_level` that has been specified by the user - for the processing request. + This property MUST reflect the effective lowest `level` that may appear in the document, + which is (if implemented) the highest level of: + 1. the `log_level` specified by the user for the processing request. + 2. the `level` specified by the user for the log request. The order of the levels is as follows (from low to high severity): `debug`, `info`, `warning`, `error`. That means if `warning` is set, the logs will only contain entries with the level `warning` and `error`. @@ -6571,9 +6575,29 @@ components: description: The last identifier (property `id` of a log entry) the client has received. If provided, the back-ends only sends the entries that occurred after the specified identifier. If not provided or empty, start with the first entry. in: query allowEmptyValue: true - example: "log1234" + example: log1234 schema: type: string + log_level: + name: level + description: |- + The minimum severity level for log entries that the back-end returns. + + The order of the levels is as follows (from low to high severity): `debug`, `info`, `warning`, `error`. + That means if `warning` is set, the back-end will only return log entries with the level `warning` and `error`. + + The default minimum log level is `debug`, which returns all log levels. + in: query + allowEmptyValue: true + example: error + schema: + type: string + enum: + - error + - warning + - info + - debug + default: info service_id: name: service_id in: path From 23703078b8d8f73d02bef73d2bba2c05a0c5c1d4 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Tue, 18 Apr 2023 15:57:56 +0200 Subject: [PATCH 098/103] Better example (openeo) domains #489 and minor clean-up --- extensions/commercial-data/README.md | 2 +- extensions/commercial-data/openapi.yaml | 6 +- openapi.yaml | 167 ++++++++++++------------ 3 files changed, 86 insertions(+), 89 deletions(-) diff --git a/extensions/commercial-data/README.md b/extensions/commercial-data/README.md index bf32e3f7..3fe7c177 100644 --- a/extensions/commercial-data/README.md +++ b/extensions/commercial-data/README.md @@ -90,7 +90,7 @@ Example response from `GET /collections/PLEIADES/queryables`: ```json { "$schema":"http://json-schema.org/draft-07/schema#", - "$id":"http://example.com/commercial_data/PLEIADES/queryables", + "$id":"http://openeo.example/collections/PLEIADES/queryables", "type":"object", "title":"Queryables of Airbus Pleiades collection.", "description":"Available properties for CQL filtering of products.", diff --git a/extensions/commercial-data/openapi.yaml b/extensions/commercial-data/openapi.yaml index 0bac3678..c2c2d1a2 100644 --- a/extensions/commercial-data/openapi.yaml +++ b/extensions/commercial-data/openapi.yaml @@ -18,7 +18,7 @@ tags: - name: Orders description: Management of commercial data orders servers: - - url: 'https://localhost/api/{version}' + - url: 'https://openeo.example/api/{version}' description: >- The URL of the API MAY freely be chosen by the back-end providers. The path, including API versioning, is a *recommendation* only. Nevertheless, @@ -26,7 +26,7 @@ servers: secure with HTTP only! variables: version: - default: v1.0 + default: v1 description: >- API versioning is RECOMMENDED. As the openEO API is following [SemVer](https://semver.org/) only the MAJOR part of the stable @@ -130,7 +130,7 @@ paths: unique identifier (ID) of the created order. format: uri type: string - example: 'https://example.openeo.org/api/orders/40264b5-c3ae-46f4-a907-0f612d763d97' + example: 'https://openeo.example/api/v1/orders/40264b5-c3ae-46f4-a907-0f612d763d97' OpenEO-Identifier: required: true schema: diff --git a/openapi.yaml b/openapi.yaml index d0efd735..30b5b287 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -56,8 +56,8 @@ info: 4. `canonical`: This relation type usually points to a publicly accessible and more long-lived URL for a resource that otherwise often requires (Bearer) authentication with a short-lived token. This way the the exposed resources can be used by non-openEO clients without additional authentication steps. For example, a shared user-defined process or batch job results could be exposed via a canonical link. - If a URL should be publicly available to everyone, it can simply a user-specific URL, e.g. `https://example.com/processes/john_doe/ndvi`. - For resources that should only be accessible to a certain group of user, a signed URL could be given, e.g. `https://example.com/processes/81zjh1tc2pt52gbx/ndvi`. + If a URL should be publicly available to everyone, it can simply a user-specific URL, e.g. `https://openeo.example/processes/john_doe/ndvi`. + For resources that should only be accessible to a certain group of user, a signed URL could be given, e.g. `https://openeo.example/processes/81zjh1tc2pt52gbx/ndvi`. Generally, it is RECOMMENDED to add descriptive titles (propertty `title`) and media type information (propertty `type`) for a better user experience. @@ -80,7 +80,7 @@ info: "id": "936DA01F-9ABD-4D9D-80C7-02AF85C822A8", "code": "SampleError", "message": "A sample error message.", - "url": "https://example.openeo.org/docs/errors/SampleError" + "url": "https://openeo.example/docs/errors/SampleError" } ``` @@ -180,8 +180,8 @@ info: ```http POST /api/v1/jobs HTTP/1.1 - Host: openeo.cloudprovider.com - Origin: https://client.org:8080 + Host: openeo.example + Origin: https://company.example:8080 Authorization: Bearer basic//ZXhhbXBsZTpleGFtcGxl ``` @@ -192,7 +192,7 @@ info: Access-Control-Allow-Origin: * Access-Control-Expose-Headers: Location, OpenEO-Identifier, OpenEO-Costs, Link Content-Type: application/json - Location: https://openeo.cloudprovider.com/api/v1/jobs/abc123 + Location: https://openeo.example/api/v1/jobs/abc123 OpenEO-Identifier: abc123 ``` @@ -213,8 +213,8 @@ info: ```http OPTIONS /api/v1/jobs HTTP/1.1 - Host: openeo.cloudprovider.com - Origin: https://client.org:8080 + Host: openeo.example + Origin: https://company.example:8080 Access-Control-Request-Method: POST Access-Control-Request-Headers: Authorization, Content-Type ``` @@ -546,7 +546,7 @@ tags: - name: File Storage description: Management of user-uploaded assets and processed data. servers: - - url: 'https://localhost/api/{version}' + - url: 'https://openeo.example/api/{version}' description: >- The URL of the API MAY freely be chosen by the back-end providers. The path, including API versioning, is a *recommendation* only. Nevertheless, @@ -640,7 +640,7 @@ paths: title: type: string description: The name of the service. - example: Cool EO Cloud + example: Example Cloud Corp. description: type: string format: commonmark @@ -651,12 +651,12 @@ paths: [CommonMark 0.29](http://commonmark.org/) syntax MAY be used for rich text representation. example: |- - This service is provided to you by [Cool EO Cloud Corp.](https://cool-eo-cloud-corp.com). It implements the full openEO API and allows to process a range of 999 EO data sets, including + This service is provided to you by [Example Cloud Corp.](https://cloud.example). It implements the full openEO API and allows to process a range of 999 EO data sets, including * Sentinel 1/2/3 and 5 * Landsat 7/8 - A free plan is available to test the service. For further information please contact our customer service at [support@cool-eo-cloud-corp.com](mailto:support@cool-eo-cloud-corp.com). + A free plan is available to test the service. For further information please contact our customer service at [support@cloud.example](mailto:support@cloud.example). production: $ref: '#/components/schemas/production' endpoints: @@ -805,19 +805,19 @@ paths: URL to a web page with more details about the plan. format: uri - example: 'https://cool-cloud-corp.com/plans/free-plan' + example: 'https://cloud.example/plans/free-plan' example: - name: free description: >- Free plan. Calculates one tile per second and a maximum amount of 100 tiles per hour. - url: 'https://cool-cloud-corp.com/plans/free-plan' + url: 'https://cloud.example/plans/free-plan' paid: false - name: premium description: >- Premium plan. Calculates unlimited tiles and each calculated tile costs 0.003 USD. - url: 'https://cool-cloud-corp.com/plans/premium-plan' + url: 'https://cloud.example/plans/premium-plan' paid: true links: description: |- @@ -871,35 +871,35 @@ paths: items: $ref: '#/components/schemas/link' example: - - href: 'http://www.cool-cloud-corp.com' + - href: 'https://cloud.example' rel: about type: text/html title: Homepage of the service provider - - href: 'https://www.cool-cloud-corp.com/tos' + - href: 'https://cloud.example/tos' rel: terms-of-service type: text/html title: Terms of Service - - href: 'https://www.cool-cloud-corp.com/privacy' + - href: 'https://cloud.example/privacy' rel: privacy-policy type: text/html title: Privacy Policy - - href: 'https://www.cool-cloud-corp.com/register' + - href: 'https://cloud.example/register' rel: create-form type: text/html title: User Registration - - href: 'https://www.cool-cloud-corp.com/lost-password' + - href: 'https://cloud.example/lost-password' rel: recovery-form type: text/html title: Reset Password - - href: 'http://www.cool-cloud-corp.com/.well-known/openeo' + - href: 'https://cloud.example/.well-known/openeo' rel: version-history type: application/json title: List of supported openEO versions - - href: 'http://www.cool-cloud-corp.com/api/v1.0/conformance' + - href: 'https://cloud.example/api/v1/conformance' rel: conformance type: application/json title: OGC Conformance Classes - - href: 'http://www.cool-cloud-corp.com/api/v1.0/collections' + - href: 'https://cloud.example/api/v1/collections' rel: data type: application/json title: List of Datasets @@ -945,9 +945,11 @@ paths: given by the user. **This URI MUST NOT be versioned as the other endpoints.** - If your API is available at `https://example.com/openeo/api/v1.0`, and - you instruct your API users to use `https://example.com/openeo` as connection URI, - the Well-Known URI SHOULD be located at `https://example.com/openeo/.well-known/openeo`. + If your API is available at `https://openeo.example/api/v1`, and + you instruct your API users to use `https://openeo.example` as connection URI, + the Well-Known URI SHOULD be located at `https://openeo.example/.well-known/openeo`. + The Well-Known URI is usually directly located at the top-level, but it is not a + requirement. For example, `https://openeo.example/eo/.well-known/openeo` is also allowed. Clients MAY get additional information (e.g. title or description) about a back-end from the most recent version that has the `production` flag @@ -957,7 +959,7 @@ paths: security: - {} servers: - - url: 'https://localhost' + - url: 'https://openeo.example' description: >- The Well-Known URI SHOULD be available directly at `https://{{domain}}/.well-known/openeo` in contrast to the other @@ -989,7 +991,7 @@ paths: type: string format: uri description: '*Absolute* URLs to the service.' - example: 'https://example.com/api/v1.0' + example: 'https://openeo.example/api/v1' production: $ref: '#/components/schemas/production' api_version: @@ -999,13 +1001,13 @@ paths: back-end implements. example: versions: - - url: 'https://example.openeo.org/api/v0.5' + - url: 'https://openeo.example/api/v0' api_version: 0.5.1 - - url: 'https://example.openeo.org/api/v1.0' - api_version: 1.0.0 - - url: 'https://example.openeo.org/api/v1.1' + - url: 'https://openeo.example/api/v1' + api_version: 1.2.0 + - url: 'https://dev.openeo.example/api/v2' production: false - api_version: 1.1.0-beta + api_version: 2.0.0-beta 4XX: $ref: '#/components/responses/client_error' 5XX: @@ -1343,10 +1345,10 @@ paths: - null links: - rel: license - href: 'https://example.openeo.org/api/collections/MOD09Q1/license' + href: 'https://openeo.example/api/v1/collections/MOD09Q1/license' links: - rel: alternate - href: 'https://example.openeo.org/csw' + href: 'https://openeo.example/csw' title: openEO catalog (OGC Catalogue Services 3.0) 4XX: $ref: '#/components/responses/client_error_auth' @@ -1432,18 +1434,18 @@ paths: href: https://scihub.copernicus.eu/twiki/pub/SciHubWebPortal/TermsConditions/Sentinel_Data_Terms_and_Conditions.pdf type: application/pdf - rel: http://www.opengis.net/def/rel/ogc/1.0/queryables - href: https://example.openeo.org/api/collections/Sentinel-2A/queryables + href: https://openeo.example/api/v1/collections/Sentinel-2A/queryables type: application/schema+json - rel: about href: https://earth.esa.int/web/sentinel/user-guides/sentinel-2-msi/product-types/level-1c type: text/html title: ESA Sentinel-2 MSI Level-1C User Guide - rel: example - href: 'https://example.openeo.org/api/collections/Sentinel-2/examples/true-color.json' + href: 'https://openeo.example/api/v1/collections/Sentinel-2/examples/true-color.json' type: application/json title: Example Process for True-Color Visualization - rel: example - href: 'https://example.openeo.org/api/collections/Sentinel-2/examples/ndvi.json' + href: 'https://openeo.example/api/v1/collections/Sentinel-2/examples/ndvi.json' type: application/json title: Example Process for NDVI Calculation and Visualization 'cube:dimensions': @@ -1556,13 +1558,13 @@ paths: max: 32660 assets: thumbnail: - href: 'https://example.openeo.org/api/collections/Sentinel-2/thumbnail.png' + href: 'https://openeo.example/api/v1/collections/Sentinel-2/thumbnail.png' type: image/png title: Preview roles: - thumbnail inspire: - href: 'https://example.openeo.org/api/collections/Sentinel-2/inspire.xml' + href: 'https://openeo.example/api/v1/collections/Sentinel-2/inspire.xml' type: application/xml title: INSPIRE metadata description: INSPIRE compliant XML metadata @@ -1611,7 +1613,7 @@ paths: $ref: '#/components/schemas/json_schema' example: $schema: https://json-schema.org/draft/2019-09/schema - $id: https://example.com/api/v1.0/collections/Sentinel-2A/queryables + $id: https://openeo.example/api/v1/collections/Sentinel-2A/queryables type: object title: Sentinel 2A properties: @@ -1757,7 +1759,7 @@ paths: title: IEEE Standard 754-2019 for Floating-Point Arithmetic links: - rel: alternate - href: 'https://provider.com/processes' + href: 'https://openeo.example/processes' type: text/html title: HTML version of the processes 4XX: @@ -2255,7 +2257,7 @@ paths: schema: type: string pattern: ^<[^>]+>;\s?rel="monitor" - example: ; rel="monitor" + example: ; rel="monitor" 4XX: $ref: '#/components/responses/client_error_auth' 5XX: @@ -2699,7 +2701,7 @@ paths: metadata endpoint in the property `url`. format: uri type: string - example: 'https://example.openeo.org/api/services/123' + example: 'https://openeo.example/api/v1/services/123' OpenEO-Identifier: required: true schema: @@ -2935,7 +2937,7 @@ paths: unique identifier (ID) of the created batch job. format: uri type: string - example: 'https://example.openeo.org/api/jobs/123' + example: 'https://openeo.example/api/v1/jobs/123' OpenEO-Identifier: required: true schema: @@ -3284,13 +3286,13 @@ paths: description: Deriving minimum NDVI measurements over pixel time series of Sentinel 2 license: Apache-2.0 providers: - - name: Cool EO Cloud Corp + - name: Example Cloud Corp. description: No further processing applied. roles: - producer - licensor - host - url: http://cool-eo-cloud-corp.com + url: https://cloud.example extent: temporal: interval: @@ -3304,30 +3306,30 @@ paths: - 90 assets: preview.png: - href: 'https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/preview.png' + href: 'https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/preview.png' type: image/png title: Thumbnail roles: - thumbnail process.json: - href: 'https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/process.json' + href: 'https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/process.json' type: application/json title: Original Process roles: - process - reproduction 1.tif: - href: 'https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/1.tif' + href: 'https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/1.tif' type: image/tiff; application=geotiff roles: - data 2.tif: - href: 'https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/2.tif' + href: 'https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/2.tif' type: image/tiff; application=geotiff roles: - data inspire.xml: - href: 'https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/inspire.xml' + href: 'https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/inspire.xml' type: application/xml title: INSPIRE metadata description: INSPIRE compliant XML metadata @@ -3336,13 +3338,13 @@ paths: links: - rel: canonical type: application/json - href: https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/collection.json + href: https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/collection.json - rel: item type: application/geo+json - href: https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/item_1.json + href: https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/item_1.json - rel: item type: application/geo+json - href: https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/item_2.json + href: https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/item_2.json application/geo+json: schema: $ref: '#/components/schemas/batch_job_result' @@ -3698,19 +3700,19 @@ paths: items: $ref: '#/components/schemas/link' example: - - href: 'https://example.openeo.org/john_doe/payment/' + - href: 'https://openeo.example/john_doe/payment/' rel: payment - - href: 'https://example.openeo.org/john_doe/edit/' + - href: 'https://openeo.example/john_doe/edit/' rel: edit-form - - href: 'https://example.openeo.org/john_doe/' + - href: 'https://openeo.example/john_doe/' rel: alternate type: text/html title: User profile - - href: 'https://example.openeo.org/john_doe.vcf' + - href: 'https://openeo.example/john_doe.vcf' rel: alternate type: text/vcard title: vCard of John Doe - - href: 'https://example.openeo.org/john_doe/invoices' + - href: 'https://openeo.example/john_doe/invoices' rel: related type: text/html title: Invoices @@ -4009,7 +4011,7 @@ components: example: - rel: canonical type: application/geo+json - href: https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/item.json + href: https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/item.json file_format: x-additionalPropertiesName: File Format Name title: File Format @@ -4113,7 +4115,7 @@ components: type: string description: The value MUST be a valid URL. format: uri - example: 'https://example.openeo.org' + example: 'https://openeo.example' type: type: string description: >- @@ -4516,7 +4518,7 @@ components: - title: Reference to a JSON Schema type: string format: uri - example: 'https://www.example.com/stac/custom-extemsion/v1.0/schema.json' + example: 'https://openeo.example/stac/custom-extemsion/v1.0.0/schema.json' - title: Reference to a core extension (STAC < 1.0.0-rc.1 only) type: string example: datacube @@ -4550,7 +4552,7 @@ components: name: description: The name of the organization or the individual. type: string - example: Cool EO Cloud Corp + example: Example Cloud Corp. description: description: >- Multi-line description to add further provider information such as @@ -4562,24 +4564,19 @@ components: type: string example: No further processing applied. roles: - description: >- + description: |- Roles of the provider. - The provider's role(s) can be one or more of the following elements: - - * licensor: The organization that is licensing the dataset under + * `licensor`: The organization that is licensing the dataset under the license specified in the collection's license field. - - * producer: The producer of the data is the provider that + * `producer`: The producer of the data is the provider that initially captured and processed the source data, e.g. ESA for Sentinel-2 data. - - * processor: A processor is any provider who processed data to a + * `processor`: A processor is any provider who processed data to a derived product. - - * host: The host is the actual provider offering the data on their + * `host`: The host is the actual provider offering the data on their storage. There SHOULD be no more than one host, specified as last element of the list. type: array @@ -4600,7 +4597,7 @@ components: contact information. type: string format: uri - example: http://cool-eo-cloud-corp.com + example: https://cloud.example stac_assets: type: object title: Assets @@ -4611,32 +4608,32 @@ components: $ref: '#/components/schemas/asset' example: preview.png: - href: 'https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/preview.png' + href: 'https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/preview.png' type: image/png title: Thumbnail roles: - thumbnail process.json: - href: 'https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/process.json' + href: 'https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/process.json' type: application/json title: Original Process roles: - process - reproduction 1.tif: - href: 'https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/1.tif' + href: 'https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/1.tif' type: image/tiff; application=geotiff title: Band 1 roles: - data 2.tif: - href: 'https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/2.tif' + href: 'https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/2.tif' type: image/tiff; application=geotiff title: Band 2 roles: - data inspire.xml: - href: 'https://example.openeo.org/api/download/583fba8b2ce583fba8b2ce/inspire.xml' + href: 'https://openeo.example/api/v1/download/583fba8b2ce583fba8b2ce/inspire.xml' type: application/xml title: INSPIRE metadata description: INSPIRE compliant XML metadata @@ -5590,11 +5587,11 @@ components: - rel: result type: application/json title: Batch Job Results - href: https://www.cool-cloud-corp.com/api/v1.0/jobs/123/logs + href: https://openeo.example/api/v1/jobs/123/logs - rel: result type: application/json title: Batch Job Logs - href: https://www.cool-cloud-corp.com/api/v1.0/jobs/123/logs + href: https://openeo.example/api/v1/jobs/123/logs job_id: type: string description: >- @@ -5686,7 +5683,7 @@ components: description: >- URL at which the secondary web service is accessible. Doesn't necessarily need to be located within the API. - example: 'https://example.openeo.org/wms/wms-a3cca9' + example: 'https://openeo.example/wms/wms-a3cca9' type: $ref: '#/components/schemas/service_type' enabled: @@ -5843,7 +5840,7 @@ components: items: $ref: '#/components/schemas/link' example: - - href: 'https://example.openeo.org/docs/errors/SampleError' + - href: 'https://openeo.example/docs/errors/SampleError' rel: about log_level: description: |- @@ -6335,7 +6332,7 @@ components: It is NOT RECOMMENDED to add stacktraces to the `message`. example: >- - Can't load the UDF file from the URL `https://example.com/invalid/file.txt`. + Can't load the UDF file from the URL `https://openeo.example/invalid/file.txt`. Server responded with error 404. time: type: string From 9fcebd104c9854ee302a2a4ee3873eb2e20fc1c4 Mon Sep 17 00:00:00 2001 From: Matthias Mohr <8262166+m-mohr@users.noreply.github.com> Date: Tue, 2 May 2023 16:04:19 +0200 Subject: [PATCH 099/103] Add experimental and deprecated to back-ends in federation extension (#491) --- extensions/federation/README.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/extensions/federation/README.md b/extensions/federation/README.md index f3d32ea8..3c8aa4a7 100644 --- a/extensions/federation/README.md +++ b/extensions/federation/README.md @@ -64,6 +64,18 @@ schema: description: >- If the `status` is `offline`: The time at which the back-end was checked and available the last time. Otherwise, this is equal to the property `last_status_check`. + experimental: + type: boolean + description: >- + Declares the back-end to be experimental, which means that + it is likely to change or may produce unpredictable behaviour. + default: false + deprecated: + type: boolean + description: |- + Declares the back-end to be deprecated with the potential + to be removed in any of the next versions. + default: false ``` ### Example @@ -84,7 +96,8 @@ schema: "title": "WWU Münster", "url": "https://openeo.wwu.de", "status": "offline", - "description": "Experimental integration of the WWU HPC" + "description": "Experimental integration of the WWU HPC", + "experimental": true } }, ... From b2dd2fa94b7a4f0a53ea5b0f9e7f384f46f4986e Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 3 May 2023 15:40:25 +0200 Subject: [PATCH 100/103] Update conformance classes #186 --- CHANGELOG.md | 2 +- README.md | 8 ++++---- openapi.yaml | 38 ++++++++++++++++++++++++++------------ 3 files changed, 31 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b36fd047..7b2c2a22 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,7 +39,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Relaxed requirement that unsupported endpoints must return HTTP status code 501. Instead also HTTP status code 404 can be used (and is regularly used in practice). [#415](https://github.com/Open-EO/openeo-api/issues/415) - Minimum value for `costs` and `budget` is 0. - `GET /jobs/{job_id}/estimate`: If a batch job can't be estimated reliably, a `EstimateComplexity` error should be returned. [#443](https://github.com/Open-EO/openeo-api/issues/443) -- The `/conformance` endpoint is now generally used for OGC APIs, STAC API and openEO. All openEO and all extensions got individual conformance classes. [#186](https://github.com/Open-EO/openeo-api/issues/186) +- The `/conformance` endpoint is now generally used for OGC APIs, STAC API and openEO. `conformsTo` is also exposed in `GET /` for STAC APIs. The openEO API and all extensions got individual conformance classes. [#186](https://github.com/Open-EO/openeo-api/issues/186) ### Fixed diff --git a/README.md b/README.md index a1cc5883..44fd9ce1 100644 --- a/README.md +++ b/README.md @@ -28,10 +28,10 @@ See also the [changelog](CHANGELOG.md) and the [milestones](https://github.com/O ## Extensions -| Name | Version | Stability | Description | -|-----------------|---------|--------------|---------------------------------------------------------------------------------------| -| [Commercial Data](extensions/commercial-data/) | 0.1.0 | experimental | Provides an interface for discovering, ordering and using commercial data. | -| [Federation](extensions/federation/) | 0.1.0 | experimental | Covers federation aspects, i.e. where multiple back-ends are exposed as a single API. | +| Name | Version | Stability | Description | +| ---------------------------------------------- | ------- | ------------ | ----------- | +| [Commercial Data](extensions/commercial-data/) | 0.1.0 | experimental | Provides an interface for discovering, ordering and using commercial data. | +| [Federation](extensions/federation/) | 0.1.0 | experimental | Covers federation aspects, i.e. where multiple back-ends are exposed as a single API. | ## Repository diff --git a/openapi.yaml b/openapi.yaml index 30b5b287..c25186b6 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -5,6 +5,8 @@ info: description: |- The openEO API specification for interoperable cloud-based processing of large Earth observation datasets. + **Conformance class:** `https://api.openeo.org/1.1.0` + # API Principles ## Language @@ -657,6 +659,8 @@ paths: * Landsat 7/8 A free plan is available to test the service. For further information please contact our customer service at [support@cloud.example](mailto:support@cloud.example). + conformsTo: + $ref: '#/components/schemas/conformsTo' production: $ref: '#/components/schemas/production' endpoints: @@ -1186,7 +1190,10 @@ paths: the [capabilties](#tag/Capabilities/operation/capabilities) in openEO. The general openEO conformance class is `https://api.openeo.org/1.1.0`. - See the individual extensions for their conformance classes. + See the individual openEO API extensions for their conformance classes. + + The conformance classes listed at this endpoint and listed in the + corresponding `conformsTo` property in `GET /` MUST be equal. More details: - [STAC API](https://github.com/radiantearth/stac-api-spec), especially "STAC API - Collections and Features Specification" @@ -1205,17 +1212,7 @@ paths: - conformsTo properties: conformsTo: - type: array - items: - type: string - format: uri - example: http://www.opengis.net/spec/ogcapi-features-1/1.0/conf/core - example: - conformsTo: - - https://api.openeo.org/1.1.0 - - https://api.openeo.org/extensions/commercial-data/0.1.0 - - https://api.openeo.org/extensions/federation/0.1.0 - - https://api.stacspec.org/v1.0.0-rc.2/collections + $ref: '#/components/schemas/conformsTo' 5XX: $ref: '#/components/responses/server_error' /collections: @@ -3722,6 +3719,23 @@ paths: $ref: '#/components/responses/server_error' components: schemas: + conformsTo: + description: |- + Lists all conformance classes specified in various standards that the + implementation conforms to. Conformance classes are commonly used in + all OGC APIs and the STAC API specification. + + The general openEO conformance class is `https://api.openeo.org/1.1.0`. + See the individual openEO API extensions for their conformance classes. + type: array + items: + type: string + format: uri + example: + - https://api.openeo.org/1.1.0 + - https://api.openeo.org/extensions/commercial-data/0.1.0 + - https://api.openeo.org/extensions/federation/0.1.0 + - https://api.stacspec.org/v1.0.0/collections udf_runtime: type: object required: From 34ae614a036cd930f2bb6ea856ffa9dba1389184 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 3 May 2023 15:57:18 +0200 Subject: [PATCH 101/103] Update STAC references #469 --- openapi.yaml | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/openapi.yaml b/openapi.yaml index c25186b6..90378bb7 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -515,7 +515,7 @@ tags: The data discovery endpoints `GET /collections` and `GET /collections/{collection_id}` are compatible with OGC API - Features and STAC. Both specifications define additional endpoints that need to be implemented to be fully compatible. The additional endpoints can easily be integrated into an openEO API implementation. A rough list of actions for compatibility is available below, but please refer to their specifications to find out the full details. **Important:** [STAC specification](https://github.com/radiantearth/stac-spec) and [STAC API](https://github.com/radiantearth/stac-api-spec) are different specifications and have different version numbers after version 0.9.0. - The openEO API only implements [STAC API version 0.9.0](https://github.com/radiantearth/stac-spec/blob/v0.9.0/api-spec/README.md), which allows to serve all STAC specification versions in the range of 0.9.x and 1.x.x (see the `stac_version` property). + The openEO API implements STAC API versions >= 0.9.0 (preferrably [STAC API version 1.0.0](https://github.com/radiantearth/stac-api-spec/blob/v1.0.0/README.md), but [0.9.0](https://github.com/radiantearth/stac-spec/blob/v0.9.0/api-spec/README.md) is allowed for backward compatibility), which allows to serve all STAC specification versions in the range of 0.9.x and 1.x.x (see the `stac_version` property). ### Content Extensions @@ -534,6 +534,10 @@ tags: ### Provide data for download If you'd like to provide your data for download in addition to offering the cloud processing service, you can implement the full STAC API. Therefore you can implement the endpoints `GET /collections/{collection_id}/items` and `GET /collections/{collection_id}/items/{feature_id}` to support retrieval of individual items. To benefit from the STAC ecosystem and allow searching for items you can also implement `POST /search` and `GET /search`. Further information can be found in the [STAC API repository](https://github.com/radiantearth/stac-spec/tree/v0.9.0/api-spec). + + ### API Extensions + + STAC API has several [extensions](https://stac-api-extensions.github.io) that can be implemented on top of the openEO API to enrich the API functionality, e.g. for searching. - name: Process Discovery description: |- These endpoints allow to list the predefined processes that are available at the back-end. To list user-defined processes see '[User-Defined Processes](#tag/User-Defined-Processes)'. @@ -1196,7 +1200,7 @@ paths: corresponding `conformsTo` property in `GET /` MUST be equal. More details: - - [STAC API](https://github.com/radiantearth/stac-api-spec), especially "STAC API - Collections and Features Specification" + - [STAC API](https://github.com/radiantearth/stac-api-spec), especially the conformance class "STAC API - Collections" - [OGC APIs](https://ogcapi.ogc.org/) tags: - Capabilities @@ -1228,10 +1232,9 @@ paths: To get the full metadata for a collection clients MUST request `GET /collections/{collection_id}`. - This endpoint is compatible with [STAC API 0.9.0](https://stacspec.org) and - [OGC API - Features](http://docs.opengeospatial.org/is/17-069r3/17-069r3.html). - [STAC API](https://github.com/radiantearth/stac-spec/tree/v0.9.0/api-spec) - features / extensions and + This endpoint is compatible with [STAC API 0.9.0 and later](https://stacspec.org) and + [OGC API - Features 1.0](http://docs.opengeospatial.org/is/17-069r3/17-069r3.html). + [STAC API extensions](https://stac-api-extensions.github.io) and [STAC extensions](https://stac-extensions.github.io) can be implemented in addition to what is documented here. @@ -1359,10 +1362,9 @@ paths: Lists **all** information about a specific collection specified by the identifier `collection_id`. - This endpoint is compatible with [STAC API](https://stacspec.org) and - [OGC API - Features](http://docs.opengeospatial.org/is/17-069r3/17-069r3.html). - [STAC API](https://github.com/radiantearth/stac-spec/tree/v0.9.0/api-spec) - features / extensions and + This endpoint is compatible with [STAC API 0.9.0 and later](https://stacspec.org) and + [OGC API - Features 1.0](http://docs.opengeospatial.org/is/17-069r3/17-069r3.html). + [STAC API extensions](https://stac-api-extensions.github.io) and [STAC extensions](https://stac-extensions.github.io) can be implemented in addition to what is documented here. @@ -3727,6 +3729,8 @@ components: The general openEO conformance class is `https://api.openeo.org/1.1.0`. See the individual openEO API extensions for their conformance classes. + + This property is REQUIRED for STAC API versions 1.0.0-beta.1 and later. type: array items: type: string @@ -4309,7 +4313,7 @@ components: Version of the collection. This property REQUIRES to add `version` (STAC < 1.0.0-rc.1) or - `https://stac-extensions.github.io/version/v1.0.0/schema.json` (STAC >= 1.0.0-rc.1) + `https://stac-extensions.github.io/version/v1.2.0/schema.json` (STAC >= 1.0.0-rc.1) to the list of `stac_extensions`. deprecated: type: boolean @@ -4323,7 +4327,7 @@ components: links and MUST refer to the collection that can be used instead. This property REQUIRES to add `version` (STAC < 1.0.0-rc.1) or - `https://stac-extensions.github.io/version/v1.0.0/schema.json` (STAC >= 1.0.0-rc.1) + `https://stac-extensions.github.io/version/v1.2.0/schema.json` (STAC >= 1.0.0-rc.1) to the list of `stac_extensions`. license: $ref: '#/components/schemas/stac_license' From f47c072ca41315b32ed4910797aef167e965ff57 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 3 May 2023 17:37:14 +0200 Subject: [PATCH 102/103] Clarified ambiguous batch job status changes (#444) * Clarified ambiguous batch job status changes. * Update openapi.yaml Co-authored-by: Stefaan Lippens * Allow back-ends to not restart. --------- Co-authored-by: Stefaan Lippens --- CHANGELOG.md | 1 + openapi.yaml | 31 ++++++++++++++++++------------- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b2c2a22..23093c42 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,6 +52,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Clarify that the relation type `version-history` should include `/.well-known/openeo` in the URL. - Clarify that clients should (re-)request capabilities and discovery endpoints with token if available and supported. [#416](https://github.com/Open-EO/openeo-api/issues/416) - Clarify the fields `plan` (for processing requests) and `billing_plan` (in `GET /` and `GET /me`). [#425](https://github.com/Open-EO/openeo-api/issues/425) [#426](https://github.com/Open-EO/openeo-api/issues/426) +- Clarified ambiguous batch job status changes. - Reflect that the `debug` process has been renamed to `inspect`. - Clarified uniqueness constraints for identifiers. [#449](https://github.com/Open-EO/openeo-api/issues/449) [#454](https://github.com/Open-EO/openeo-api/issues/454) - Clarified schematically the applicability of JSON Schema extensions (`parameters`, `returns`, `dimensions`) and their relation to the subtypes diff --git a/openapi.yaml b/openapi.yaml index 90378bb7..030d9791 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -2976,13 +2976,14 @@ paths: patch: summary: Modify a batch job operationId: update-job - description: >- + description: |- Modifies an existing job at the back-end, but maintains the identifier. Changes can be grouped in a single request. + The job status does not change. - Jobs can only be modified when the job is not queued or running. - Otherwise requests to this endpoint MUST be rejected with openEO error + Jobs can only be modified when the job is not queued and not running. + Otherwise, requests to this endpoint MUST be rejected with openEO error `JobLocked`. tags: - Data Processing @@ -3371,12 +3372,12 @@ paths: The result will be stored in the format specified in the process. To specify the format use a process such as `save_result`. - This endpoint has no effect if the job status is already `queued` or - `running`. In particular, it doesn't restart a running job. Processing - MUST be canceled before to restart it. - The job status is set to `queued`, if processing doesn't start - instantly. + instantly. The same applies if the job status is `canceled`, `finished`, + or `error`, which restarts the job and discards previous results if the + back-end doesn't reject the request with an error. + Clients SHOULD warn users and ask for confirmation if results may get + discarded. * Once the processing starts the status is set to `running`. * Once the data is available to download the status is set to @@ -3384,6 +3385,10 @@ paths: * Whenever an error occurs during processing, the status MUST be set to `error`. + This endpoint has no effect if the job status is already `queued` or + `running`. In particular, it doesn't restart a running job. To restart + a queued or running job, processing MUST have been canceled before. + Back-ends SHOULD reject queueing jobs with openEO error `PaymentRequired`, if the back-end is able to detect that the budget is too low to fully process the request. Alternatively, back-ends MAY provide partial results @@ -3405,20 +3410,20 @@ paths: delete: summary: Cancel processing a batch job operationId: stop-job - description: >- + description: |- Cancels all related computations for this job at the back-end. It will stop generating additional costs for processing. - A subset of processed results may be available for downloading depending - on the state of the job as it was canceled. Finished results MUST NOT be - deleted until the job is deleted or job processing is started again. + on the state of the job at the time it was canceled. + Results MUST NOT be deleted until the job processing is started again or + the job is completely deleted through a request to + `DELETE /jobs/{job_id}`. This endpoint only has an effect if the job status is `queued` or `running`. - The job status is set to `canceled` if the status was `running` beforehand and partial or preliminary results are available to be downloaded. Otherwise the status is set to `created`. From a98c588273eb3a51154e138a56ad413c3e0f4640 Mon Sep 17 00:00:00 2001 From: Matthias Mohr Date: Wed, 3 May 2023 17:51:09 +0200 Subject: [PATCH 103/103] Update version number to v1.2.0 --- CHANGELOG.md | 3 +++ README.md | 5 +++-- extensions/federation/README.md | 2 +- openapi.yaml | 12 ++++++------ package.json | 2 +- 5 files changed, 14 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b2c2a22..366fa0df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## Unreleased / Draft +## [1.2.0] - 2021-05-31 + ### Added - **New extensions:** @@ -428,6 +430,7 @@ Initial version. [Unreleased]: +[1.2.0]: [1.1.0]: [1.0.1]: [1.0.0]: diff --git a/README.md b/README.md index 44fd9ce1..aef37924 100644 --- a/README.md +++ b/README.md @@ -6,12 +6,13 @@ openEO develops an open API to connect R, Python and JavaScript clients to big E ## Versions / Branches -The [master branch](https://github.com/Open-EO/openeo-api/tree/master) is the 'stable' version of the openEO API specification. It is currently version **1.1.0** of the specification. The [draft branch](https://github.com/Open-EO/openeo-api/tree/draft) is where active development takes place. +The [master branch](https://github.com/Open-EO/openeo-api/tree/master) is the 'stable' version of the openEO API specification. It is currently version **1.2.0** of the specification. The [draft branch](https://github.com/Open-EO/openeo-api/tree/draft) is where active development takes place. | Version / Branch | Status | Description | | --------------------------------------------------------- | ----------- | ----------- | | [draft](https://api.openeo.org/draft) | planned | *Unstable* - Next version. | -| [**1.1.0**](https://api.openeo.org) | **current** | Clarifications, STAC updates, return value for child processes, more details for logs and jobs, default clients for OIDC. [Changelog](CHANGELOG.md#110---2021-06-15). | +| [**1.2.0**](https://api.openeo.org) | **current** | Clarifications, new extensions, vector data cubes, STAC (API) updates, more link relation types, improved batch job results and logs. [Changelog](CHANGELOG.md#120---2023-05-31). | +| [1.1.0](https://api.openeo.org/1.1.0) | legacy | Clarifications, STAC updates, return value for child processes, more details for logs and jobs, default clients for OIDC. [Changelog](CHANGELOG.md#110---2021-06-15). | | [1.0.1](https://api.openeo.org/1.0.1) | legacy | Clarifications, bugfixes and CORS improvements. [Changelog](CHANGELOG.md#101---2020-12-07). | | [1.0.0](https://api.openeo.org/1.0.0) | legacy | First stable version of openEO. [Changelog](CHANGELOG.md#100---2020-07-17). | | [1.0.0-rc.2](https://api.openeo.org/1.0.0-rc.2) | legacy | Introduced user-defined processes. [Changelog](CHANGELOG.md#100-rc2---2020-02-20). | diff --git a/extensions/federation/README.md b/extensions/federation/README.md index 3c8aa4a7..16e0d2ae 100644 --- a/extensions/federation/README.md +++ b/extensions/federation/README.md @@ -82,7 +82,7 @@ schema: ```json { - "api_version": "1.1.0", + "api_version": "1.2.0", "federation": { "vito": { "title": "VITO", diff --git a/openapi.yaml b/openapi.yaml index 90378bb7..a69c1bff 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -1,11 +1,11 @@ openapi: 3.0.2 info: title: openEO API - version: 1.1.0 + version: 1.2.0 description: |- The openEO API specification for interoperable cloud-based processing of large Earth observation datasets. - **Conformance class:** `https://api.openeo.org/1.1.0` + **Conformance class:** `https://api.openeo.org/1.2.0` # API Principles @@ -618,7 +618,7 @@ paths: Version number of the openEO specification this back-end implements. enum: - - 1.1.0 + - 1.2.0 backend_version: type: string description: >- @@ -1193,7 +1193,7 @@ paths: Otherwise, the implemented functionality can usually be retrieved from the [capabilties](#tag/Capabilities/operation/capabilities) in openEO. - The general openEO conformance class is `https://api.openeo.org/1.1.0`. + The general openEO conformance class is `https://api.openeo.org/1.2.0`. See the individual openEO API extensions for their conformance classes. The conformance classes listed at this endpoint and listed in the @@ -3727,7 +3727,7 @@ components: implementation conforms to. Conformance classes are commonly used in all OGC APIs and the STAC API specification. - The general openEO conformance class is `https://api.openeo.org/1.1.0`. + The general openEO conformance class is `https://api.openeo.org/1.2.0`. See the individual openEO API extensions for their conformance classes. This property is REQUIRED for STAC API versions 1.0.0-beta.1 and later. @@ -3736,7 +3736,7 @@ components: type: string format: uri example: - - https://api.openeo.org/1.1.0 + - https://api.openeo.org/1.2.0 - https://api.openeo.org/extensions/commercial-data/0.1.0 - https://api.openeo.org/extensions/federation/0.1.0 - https://api.stacspec.org/v1.0.0/collections diff --git a/package.json b/package.json index 4cc5beaf..8a19865b 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@openeo/api", - "version": "1.1.0", + "version": "1.2.0", "author": "openEO Consortium", "license": "Apache-2.0", "description": "The openEO API specification.",