Release/v1.2.0: SAR Support, Multi-Resolution Tiles, and Feature Util…

…ities (#32)

This release extends OversightML Imagery Toolkit support to include additional image formats and processing utilities. Key changes include:

* Support for SAR imagery including the SICD, and SIDD sensor models and ability to create georeferenced tiles. Also includes basic preprocessing utilities for converting complex data into 8-bit grayscales for display.
* RSM sensor models are enabled by default for datasets created using load_gdal_dataset().
* Addition of an optional output_size parameter for the GDALTileFactory that can be used to produce reduced resolution tiles.
* Addition of a new aws.osml.features module that contains utilities for working with GeoJSON features derived from imagery. These utilities include spatial indices for creating vector tiles and utilities for working with properties of features that have both geospatil and pixel coordinates.


---------

Pull Requests Merged:
* feat: add SICD sensor model with planar projection (#10)
* feat: add optional dynamic pixel range operations to tile factory (#9)
* feat: add HAE and DEM projections to SICD sensor model (#12)
* feat: Fix documentation commands for github actions (#13)
* docs: Updating CONTRIBUTING.md (#11)
* fix: correct calculation of INCA R/Rdot projection (#14)
* feat: update DRA to work with SICDs (#15)
* feat: enable RSM sensor model by default (#16)
* feat: gdal sensor model now supports non-wgs84 crs (#17)
* docs: update readme for SAR features (#18)
* feat: SAR examples and display processing (#21)
* feat: add support for sidd imagery (#22)
* feat: Update GitHub actions (#23)
* build: Update Github Actions
* fix: errors constructing RSMSectionedPolynomialSensorModel from TREs
* feat: adds features module with 2D spatial index and basic property utils
* feat: adds generic DEM tile set and geodetic coordinate formatting
* fix: lint error in generic_dem_tile_set.py
* feat: add ability to create scaled tiles and tiles from SIDD
* docs: add examples to package level documentation
* build: update library version to v1.2.0

---------

Co-authored-by: Ranbir Aulakh <[email protected]>
Co-authored-by: drduhe <[email protected]>

.github/workflows/documentation.yaml → .github/workflows/documentation-publish.yml

@@ -1,12 +1,17 @@
-name: Docs
-on: [push, pull_request, workflow_dispatch]
+name: Generate and Publish Documentation for OSML Imagery Toolkit
+
+on:
+  workflow_call:
+
 permissions:
     contents: write
+
 jobs:
   docs:
+    if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - uses: actions/setup-python@v3
       - name: Install dependencies
         run: |
@@ -19,7 +24,6 @@ jobs:
           tox -e docs
       - name: Deploy
         uses: peaceiris/[email protected]
-        if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
         with:
           publish_branch: gh-pages
           github_token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/osml-imagery-toolkit-build.yml

@@ -0,0 +1,10 @@
+name: "OSML Imagery Toolkit Build Workflow"
+
+on:
+  pull_request:
+    branches: ["main", "dev"]
+
+jobs:
+  Build_Validate_Tox:
+    uses: ./.github/workflows/python-tox.yml
+    secrets: inherit

.github/workflows/osml-imagery-toolkit-publish.yml

@@ -0,0 +1,18 @@
+name: "OSML Imagery Toolkit Build and Publish Workflow"
+
+on:
+  push:
+    branches: ["main"]
+
+jobs:
+  Build_Validate_Tox:
+    uses: ./.github/workflows/python-tox.yml
+    secrets: inherit
+  Publish_Python:
+    needs: [Build_Validate_Tox]
+    uses: ./.github/workflows/python-publish.yml
+    secrets: inherit
+  Publish_Documentation:
+    needs: [Publish_Python]
+    uses: ./.github/workflows/documentation-publish.yml
+    secrets: inherit

.github/workflows/python-publish.yml

@@ -6,9 +6,10 @@
 # separate terms of service, privacy policy, and support
 # documentation.
 
-name: Upload Python Package
+name: Publish Python Package
 
 on:
+  workflow_call:
   release:
     types: [published]
 
@@ -17,11 +18,10 @@ permissions:
 
 jobs:
   deploy:
-
+    if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
     runs-on: ubuntu-latest
-
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
     - name: Set up Python
       uses: actions/setup-python@v3
       with:
@@ -33,7 +33,7 @@ jobs:
     - name: Build package
       run: python -m build
     - name: Publish package
-      uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
+      uses: pypa/gh-action-pypi-publish@v1.8.10
       with:
         user: __token__
         password: ${{ secrets.PYPI_API_TOKEN }}

.github/workflows/python-package.yml → .github/workflows/python-tox.yml

@@ -1,27 +1,21 @@
 # This workflow will install Python dependencies, run tests and lint with a single version of Python
 # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
 
-name: Tox Build/Validation
+name: Build/Validation with Tox
 
 on:
-  push:
-    branches: [ "main" ]
-  pull_request:
-    branches: [ "main" ]
+  workflow_call:
 
 permissions:
   contents: read
 
 jobs:
   build:
-
     runs-on: ubuntu-latest
-
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
       with:
-          lfs: 'true'
-          ssh-key: ${{ secrets.git_ssh_key }}
+        lfs: 'true'
     - name: Set up Python 3.10
       uses: actions/setup-python@v3
       with:

.pre-commit-config.yaml

@@ -24,7 +24,7 @@ repos:
     rev: 6.0.0
     hooks:
       - id: flake8
-        args: ["--ignore=E203,W503,W605", "--max-line-length=125"]
+        args: ["--ignore=E203,W503,W605", "--max-line-length=160", "--extend-exclude=src/aws/osml/formats"]
         types: [file, python]
 
   - repo: https://github.com/pre-commit/mirrors-autopep8

README.md

@@ -2,14 +2,24 @@
 
 The OversightML Imagery Toolkit is a Python package that contains image processing and photogrammetry routines commonly
 used during the analysis of imagery collected by satellites and unmanned aerial vehicles (UAVs). It builds upon GDAL
-by providing additional support for images compliant with the National Imagery Transmission Format (NITF) and Sensor
-Independent Complex Data (SICD) standards.
+by providing additional support for images compliant with the National Imagery Transmission Format (NITF), Sensor
+Independent Complex Data (SICD), and Sensor Independent Derived Data (SIDD) standards.
 
+This library contains four core packages under the `aws.osml` namespace:
+* **photogrammetry**: convert locations between the image (x, y) and geodetic (lon, lat, elev) coordinate systems
+* **gdal**: utilities to work with datasets loaded by GDAL
+* **image_processing**: common image manipulation routines
+* **features**: common geospatial feature manipulation routines
+
+## Documentation
+
+* **APIs**: You can find API documentation for the OSML Imagery Toolkit hosted on our [GitHub project page](https://aws-solutions-library-samples.github.io/osml-imagery-toolkit/).
+If you are working from the source code running `tox -e docs` will trigger the Sphinx documentation build.
+* **Example Notebooks**: Example notebooks for some operations are in the `examples` directory
 ## Installation
 
-The intent is to vend / distribute this software through a Python Package Index.
-If your environment has a distribution,
-you should be able to install it using pip:
+This software is available through a Python Package Index.
+If your environment has a distribution, you should be able to install it using pip:
 ```shell
 pip install osml-imagery-toolkit[gdal]
 ```
@@ -23,97 +33,6 @@ Note that GDAL is listed as an extra dependency for this package. This is done t
 don't want to use GDAL or those that have their own custom installation steps for that library. Future versions of
 this package will include image IO backbones that have fewer dependencies.
 
-
-## Documentation
-
-You can find documentation for this library in the `./doc` directory. Sphinx is used to construct a searchable HTML
-version of the API documents.
-
-```shell
-tox -e docs
-```
-
-## Example Usage
-
-This library contains four core packages under the `aws.osml` namespace.
-* photogrammetry: convert locations between the image (x, y) and geodetic (lon, lat, elev) coordinate systems
-* gdal: help load and manage datasets loaded by GDAL
-* image_processing: common image manipulation routines
-* formats: utilities for handling format specific information; normally not accessed directly
-
-```python
-from aws.osml.gdal import GDALImageFormats, GDALCompressionOptions, load_gdal_dataset
-from aws.osml.image_processing import GDALTileFactory
-from aws.osml.photogrammetry import ImageCoordinate, GeodeticWorldCoordinate, SensorModel
-```
-
-### Tiling with Updated Image Metadata
-
-Many applications break large remote sensing images into smaller chips or tiles for distributed processing or
-dissemination. GDAL's Translate function provides basic capabilities, but it does not correctly update geospatial
-metadata to reflect the new image extent. These utilities provide those functions so tile consumers can correctly
-interpret the pixel information they have been provided. For NITF imagery that includes the addition of a new ICHIPB
-TRE. With SICD the XML ImageData elements are adjusted to identify the sub-image bounds.
-
-```python
-# Load the image and create a sensor model
-ds, sensor_model = load_gdal_dataset("./imagery/sample.nitf")
-tile_factory = GDALTileFactory(ds,
-                               sensor_model,
-                               GDALImageFormats.NITF,
-                               GDALCompressionOptions.NONE
-                               )
-
-# Bounds are [left_x, top_y, width, height]
-nitf_encoded_tile_bytes = tile_factory.create_encoded_tile([0, 0, 1024, 1024])
-```
-
-### Tiling for Display
-
-Some images, for example 11-bit panchromatic images or SAR imagery with floating point complex data, can not be
-displayed directly without remapping the pixels into an 8-bit per pixel grayscale or RGB color model. The TileFactory
-supports creation of tiles suitable for human review by setting both the output_type and range_adjustment options.
-
-```python
-viz_tile_factory = GDALTileFactory(ds,
-                                   sensor_model,
-                                   GDALImageFormats.PNG,
-                                   GDALCompressionOptions.NONE,
-                                   output_type=gdalconst.GDT_Byte,
-                                   range_adjustment=RangeAdjustmentType.DRA)
-
-viz_tile = viz_tile_factory.create_encoded_tile([0, 0, 1024, 1024])
-```
-
-### More Precise Sensor Models
-
-OversightML provides implementations of the Replacement Sensor Model (RSM), Rational Polynomial
-Camera (RPC), and Sensor Independent Complex Data (SICD) sensor models to assist in geo positioning.
-When loading a dataset, the toolkit will construct the most accurate sensor model
-from the available image metadata. That sensor model can be used in conjunction with an optional
-elevation model to convert between image and geodetic coordinates.
-
-```python
-ds, sensor_model = load_gdal_dataset("./imagery/sample.nitf")
-elevation_model = DigitalElevationModel(
-    SRTMTileSet(version="1arc_v3"),
-    GDALDigitalElevationModelTileFactory("./local-SRTM-tiles"))
-
-# Note the order of ImageCoordinate is (x, y)
-geodetic_location_of_ul_corner = sensor_model.image_to_world(
-    ImageCoordinate([0, 0]),
-    elevation_model=elevation_model)
-
-lon_degrees = -77.404453
-lat_degrees = 38.954831
-meters_above_ellipsoid = 100.0
-
-image_location = sensor_model.world_to_image(
-    GeodeticWorldCoordinate([radians(lon_degrees),
-                             radians(lat_degrees),
-                             meters_above_ellipsoid]))
-```
-
 ## Contributing
 
 This project welcomes contributions and suggestions. If you would like to submit a pull request, see our

doc/conf.py

@@ -46,7 +46,7 @@ def setup(app):
 # -- Project information -----------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
 
-project = "OversightML Imagery Core"
+project = "OversightML Imagery Toolkit"
 copyright = "{}, Amazon.com".format(datetime.datetime.now().year)
 author = "Amazon Web Services"
 

doc/index.rst

@@ -1,18 +1,19 @@
 
-aws-osml-imagery-core
+osml-imagery-toolkit
 =====================
 
-The OversightML Imagery Core is a Python library that contains image processing and photogrammetry routines commonly
+The OversightML Imagery Toolkit is a Python package that contains image processing and photogrammetry routines commonly
 used during the analysis of imagery collected by satellites and unmanned aerial vehicles (UAVs). It builds upon GDAL
-by providing additional support for images compliant with the Sensor Independent Complex Data (SICD) and National
-Imagery Transmission Format (NITF) standards.
+by providing additional support for images compliant with the National Imagery Transmission Format (NITF), Sensor
+Independent Complex Data (SICD), and Sensor Independent Derived Data (SIDD) standards.
 
 .. toctree::
    :maxdepth: 1
 
    _apidoc/aws.osml.photogrammetry
    _apidoc/aws.osml.gdal
    _apidoc/aws.osml.image_processing
+   _apidoc/aws.osml.features
 
 
 Indices and tables