Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Phase out read_vector usage #104

Closed
soxofaan opened this issue Dec 17, 2019 · 6 comments
Closed

Phase out read_vector usage #104

soxofaan opened this issue Dec 17, 2019 · 6 comments

Comments

@soxofaan
Copy link
Member

soxofaan commented Dec 17, 2019

when passing a string as polygon argument to ImageCollectionClient.mask:

if isinstance(polygon, str):
new_collection = self.graph_add_process('read_vector', args={
'filename': polygon
})
mask = {
'from_node': new_collection.node_id
}

or in _get_geometry_argument:

# TODO #104: `read_vector` is non-standard process.

this implementation assumes process read_vector which is currently a VITO specific process and I'm not aware of anything alike in the official process collection.

Background: At VITO we currently use this to work with very large polygon files (> 100k polygons) stored at backend side, which we don't want to pass with the openEO request for obvious reasons.

In the client we should avoid hardcoding non-official processes of course

Solution:

  • check which processes are supported by backend: read_vector, load_url, load_geojson
  • check if the string is an http url, a path to geojson that exists locally, or a path that does not exist locally
  • depending on the combination of available processes and the type of string, do something that's more sensible than current implementation
  • Some care should be taken to avoid passing a huge geojson object with the request (e.g. throw an exception when above a threshold)
  • Still allow the possibility to inject a read_vector based argument to support the VITO use cases

cc @jdries

@jdries
Copy link
Collaborator

jdries commented Dec 17, 2019

A fix for this is being proposed in this pull request:
Open-EO/openeo-processes#106

@soxofaan
Copy link
Member Author

ah nice, I somehow missed that thread

@soxofaan
Copy link
Member Author

however, load_uploaded_files is meant for user uploaded files, which is not exactly what we currently do in the VITO use cases that depend on read_vector.

A closer proposal is probably the import_nfs process from Open-EO/openeo-processes#105

@jdries jdries changed the title polygon file handling in ImageCollectionClient.mask Phase out read_vector usage Mar 25, 2024
@soxofaan
Copy link
Member Author

way forward:

@soxofaan
Copy link
Member Author

Another use case that has to be updated: this suggestion was made in a user support channel to do aggregate_spatial with a (large) geometry from a URL:

datacube = datacube.aggregate_spatial(
    geometries="https://example.com/path/to/geometries.json",
    reducer="mean",
)

will currently produce a process graph using read_vector, so that's not future proof at the moment

(cc @EmileSonneveld)

soxofaan added a commit that referenced this issue Nov 27, 2024
soxofaan added a commit that referenced this issue Nov 27, 2024
when providing URL to aggregate_spatial, mask_polygon, ...
soxofaan added a commit that referenced this issue Nov 27, 2024
when providing URL to aggregate_spatial, mask_polygon, ...
soxofaan added a commit that referenced this issue Nov 27, 2024
soxofaan added a commit that referenced this issue Nov 27, 2024
instead, support reading GeoJSON from local path
soxofaan added a commit that referenced this issue Nov 27, 2024
soxofaan added a commit that referenced this issue Nov 27, 2024
instead, support reading GeoJSON from local path
@soxofaan
Copy link
Member Author

done:

  • remove read_vector usage from default geometry handling (but documented how to reconstruct for workflows that still need it)
  • add support for passing a geometry URL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants