Python/fix/raster tests #452
Conversation
… into python/fix/gdal-enabler bringing inline
…ix/gdal-enabler merge
…ncountered along the way
Codecov Report
@@ Coverage Diff @@
## main #452 +/- ##
==========================================
- Coverage 92.14% 92.07% -0.08%
==========================================
Files 243 243
Lines 6671 6675 +4
Branches 234 233 -1
==========================================
- Hits 6147 6146 -1
- Misses 524 529 +5
|
| @@ -11,7 +11,7 @@ runs: | |||
| shell: bash | |||
| run: | | |||
| cd python | |||
| pip install build wheel pyspark==${{ matrix.spark }} | |||
| pip install build wheel pyspark==${{ matrix.spark }} numpy==${{ matrix.numpy }} | |||
There was a problem hiding this comment.
When we move up to Ubuntu 22.04 / GDAL 3.6.x we can pull this back out.
| - name: Add packaged GDAL dependencies | ||
| shell: bash | ||
| run : | | ||
| sudo apt-get update && sudo apt-get install -y unixodbc libcurl3-gnutls libsnappy-dev libopenjp2-7 |
There was a problem hiding this comment.
@sllynn this is so much better than before!
| @@ -21,8 +21,8 @@ GDAL_RESOURCE_DIR=$(find $DATABRICKS_ROOT_VIRTUALENV_ENV -name "databricks-mosai | |||
|
|
|||
| # -- untar files to root | |||
| # - from databricks-mosaic-gdal install dir | |||
| tar -xf $GDAL_RESOURCE_DIR/resources/gdal-3.4.3-filetree.tar.xz -C / | |||
| tar -xf $GDAL_RESOURCE_DIR/resources/gdal-3.4.3-filetree.tar.xz --skip-old-files -C / | |||
There was a problem hiding this comment.
@sllynn why is --skip-old-files needed here?
There was a problem hiding this comment.
Running the script successfully is now part of the python test suite. The untar process is not idempotent and untar will raise an error if the target file(s) already exists. That's fine when running the script for the first time (on cluster start or in GH actions) but for local development you need to be able to re-run this multiple times without it hitting an error.
| | div = np.sum(stacked_array > 0, axis=0) | ||
| | div = np.where(div==0, 1, div) | ||
| | np.divide(pixel_sum, div, out=out_ar, casting='unsafe') | ||
| | np.clip(out_ar, stacked_array.min(), stacked_array.max(), out=out_ar) |
There was a problem hiding this comment.
@sllynn this was clever!!!
There was a problem hiding this comment.
Took forever to get it working using trial and error. Reading the docs helped a bit, mind you 🙄 Might be worth us building a little function that will allow us to pass in a script and inspect the internal state while it's executed?
adding tests for raster functions to python bindings