load_result: Align with load_collection

[m-mohr]

load_result (and load_uploaded_files) can only load a full result, but not filter extents like in load_collection. Should load_result be aligned (i.e. add spatial, temporal extents and metadata filters?)

[soxofaan]

We have already processes like filter_temporal, filter_bbox, filter_bands to cover that, no?

load_result can only load a full result,

abstractly speaking yes, but a backend can work in a lazy-loading approach and limit the actual loading within the constraints provided by subsequent filter_ processes. Lazy loading is of course less straightforward to implement than handling direct load_result arguments.

The same question can be raise about load_uploaded_files, FYI

[m-mohr]

We have already processes like filter_temporal, filter_bbox, filter_bands to cover that, no?

Yes, but the argumentation for adding those parameters to load_collection was that you have to load all data first just to discard a lot of data afterwards. Thus we added them and I think this argumentation is still valid, although I'm aware that VITO optimizes filter_ operations more than other back-ends.

The same question can be raise about load_uploaded_files, FYI

Indeed, added above.

[m-mohr]

Related issue: Open-EO/openeo-api#376

[m-mohr]

With the recent discussions around #241 and read/get_vector in the Python driver, I'm wondering whether we should aim for something generic that doesn't care about the actual place the data is stored and can load cubes from different sources. I'm not sure we actually need to have specific functions for all of them. It would avoid the issue in #241 a bit at least.

Something like load_collection, but instead of specifying a collection ID, you specify either a URL, file on the back-end user workspace or a result (is basically also just a URL where you check whether it's accessible by the current user).

import(location, ?spatial_extent, ?temporal_extent, ?bands, ?filter) -> raster/vector cube

[soxofaan]

that sounds like an interesting solution

but then the "collection_id" (or "location" in your snippet) probably has to become a more complex argument: a string that's a traditional collection_id or some kind of URL. Or optionally also an object that allows setting additional load options (e.g. filename globs, file type whitelists or blacklists, ...)

The next problem is then probably that you need a new "capabilities endpoint" where a backend can declare which kind of "locations" are supported

[m-mohr]

I'm not sure that is clear, but I'd leave load_collection untouched and not allow (internal) collection IDs in import.

The location could be defined as follows in schema:

{
	"name": "location",
	"description": "...",
	"schema": [
		{
			"title": "Multiple files on server-side user workspace",
			"type": "array",
			"subtype": "file-paths",
			"items": {
				"type": "string",
				"subtype": "file-path",
				"pattern": "^[^\r\n\\:'\"]+$"
			}
		},
		{
			"title": "Single file on server-side user workspace (we may want to remove this for simplicity)",
			"type": "string",
			"subtype": "file-path",
			"pattern": "^[^\r\n\\:'\"]+$"
		},
		{
			"title": "Remote files (Absolute URL)",
			"type": "string",
			"subtype": "uri",
			"pattern": "^(http|https|s3)://"
		},
		{
			"title": "Batch Job ID",
			"type": "string",
			"subtype": "job-id",
			"description": "A batch job id, either one of the jobs a user has stored or a publicly available job.",
			"pattern": "^[\\w\\-\\.~]+$"
		}
	]
}

Yes, we have the issue that we overload some data types a bit in a number of processes.

If we want to have very specific things like globs, then this idea doesn't work very well and I'd say we need to stick with individual processes.

[m-mohr]

@aljacob @lforesta @sophieherrmann I heard this (load_result) being discussed today as being required by a use-case. I didn't have that on my list as being required for UC3 or 6 in openEO Platform. Could you please clarify?

[m-mohr]

I've got a PR up, although this is just load_result now. Reasoning:

1. For load_uploaded_files I'm now thinking we don't need additional parameters as I'd expect that someone really only uploads what is needed for a use case and then doesn't need additional filtering in the process. If that rare use case is still required, use filter functions.
2. As such, I've simply added spatial_extent, temporal_extent and bands to the load_result function and allow loading by URLs now. These additions are depending a lot on the underlying data structure, e.g. I'd think you could remotely filter by spatial_extent easily on a COG, but for bands and temporal_extent you need proper metadata. For other file formats this may not work at all. Also, loading by URL is not available on all back-ends (only introduced in API v1.1.0). I've excluded property filtering for now as this would require an API to be present: STAC API for batch job results? openeo-api#398 (or User-generated Collections openeo-api#376)

See PR #292.