Release v0.9.0

.github/workflows/test-and-build.yml

@@ -17,7 +17,7 @@ jobs:
     with:
       local_package_name: hyp3_srg
       python_versions: >-
-        ["3.10", "3.11", "3.12"]
+        ["3.10", "3.11", "3.12", "3.13"]
 
   call-version-info-workflow:
     # Docs: https://github.com/ASFHyP3/actions
@@ -28,11 +28,12 @@ jobs:
   call-docker-ghcr-workflow:
     needs: call-version-info-workflow
     # Docs: https://github.com/ASFHyP3/actions
-    uses: ASFHyP3/actions/.github/workflows/reusable-docker-ghcr.yml@v0.11.2
+    uses: ASFHyP3/actions/.github/workflows/reusable-docker-ghcr.yml@v0.12.0
     with:
       version_tag: ${{ needs.call-version-info-workflow.outputs.version_tag }}
       release_branch: main
       develop_branch: develop
       user: tools-bot
+      file: Dockerfile.gpu
     secrets:
       USER_TOKEN: ${{ secrets.GITHUB_TOKEN }}

CHANGELOG.md

@@ -6,6 +6,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
 and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.0]
+
+### Added
+* A new `--use-gslc-prefix` option has been added to the `back_projection` and `time_series` workflows:
+  * This option causes `back_projection` to upload the GSLC outputs to a `GSLC_granules/` subprefix located within the S3 bucket and prefix given by the `--bucket` and `--bucket-prefix` options.
+  * This option causes `time_series` to download the GSLC inputs from the `GSLC_granules/` subprefix located within the bucket and prefix given by the `--bucket` and `--bucket-prefix` options.
+
+### Changed
+* Releases and test deployments now trigger a Docker build for the GPU container, rather than the CPU container.
+
+### Fixed
+* Fixed the parsing for the `--bounds` option for `time_series`.
+
 ## [0.8.0]
 
 ### Added

README.md

@@ -77,21 +77,17 @@ aws ssm get-parameters --names /aws/service/ecs/optimized-ami/amazon-linux-2/gpu
 ### GPU Docker Container
 Once you have a compute environment set up as described above, you can build the GPU version of the container by running:
 ```bash
-docker build --build-arg="GPU_ARCH={YOUR_ARCH}" -t ghcr.io/asfhyp3/hyp3-srg:{RELEASE}.gpu -f Dockerfile.gpu .
+docker build --build-arg="GPU_ARCH={YOUR_ARCH}" -t srg -f Dockerfile.gpu .
 ```
 
 You can get the value of `COMPUTE_CAPABILITY_VERSION` by running `nvidia-smi` on the instance to obtain GPU type, then cross-reference this information with NVIDIA's [GPU type compute capability list](https://developer.nvidia.com/cuda-gpus). For a g6.2xlarge instance, this would be:
 ```bash
-docker --build-arg="GPU_ARCH=89" -t ghcr.io/asfhyp3/hyp3-srg:{RELEASE}.gpu -f Dockerfile.gpu .
+docker --build-arg="GPU_ARCH=89" -t srg -f Dockerfile.gpu .
 ```
 The compute capability version will always be the same for a given instance type, so you will only need to look this up once per instance type.
 The default value for this argument is `89` - the correct value for g6.2xlarge instances.
 **THE COMPUTE CAPABILITY VERSION MUST MATCH ON BOTH THE BUILDING AND RUNNING MACHINE!**
 
-The value of `RELEASE` can be obtained from the git tags.
-
-You can push a manual container to HyP3-SRG's container repository by following [this guide](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#pushing-container-images).
-
 ### EC2 Setup
 > [!CAUTION]
 > Running the docker container on an Amazon Linux 2023 Deep Learning AMI runs, but will result in all zero outputs. Work is ongoing to determine what is causing this issue. For now, we recommend using option 2.3.
@@ -104,5 +100,5 @@ When running on an EC2 instance, the following setup is recommended:
     3. Use the latest AWS ECS-optimized GPU AMI (`aws ssm get-parameters --names /aws/service/ecs/optimized-ami/amazon-linux-2/gpu/recommended --region us-west-2`)
 3. Build the GPU docker container with the correct compute capability version (see section above). To determine this value, run `nvidia-smi` on the instance to obtain GPU type, then cross-referencke this information with NVIDIA's [GPU type compute capability list](https://developer.nvidia.com/cuda-gpus). For a g6.2xlarge instance, this would be:
 ```bash
-docker --build-arg="GPU_ARCH=89" -t ghcr.io/asfhyp3/hyp3-srg:{RELEASE}.gpu -f Dockerfile.gpu .
+docker --build-arg="GPU_ARCH=89" -t srg -f Dockerfile.gpu .
 ```

environment.yml

@@ -19,4 +19,4 @@ dependencies:
   - asf_search
   # For running
   - hyp3lib>=3,<4
-  - s1_orbits
+  - s1_orbits

pyproject.toml

@@ -18,6 +18,8 @@ classifiers=[
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
 ]
 dependencies = [
     "hyp3lib>=3,<4",

src/hyp3_srg/back_projection.py

@@ -92,6 +92,7 @@ def back_project(
     earthdata_password: str = None,
     bucket: str = None,
     bucket_prefix: str = '',
+    use_gslc_prefix: bool = False,
     work_dir: Optional[Path] = None,
     gpu: bool = False,
 ):
@@ -104,9 +105,15 @@ def back_project(
         earthdata_password: Password for NASA's EarthData service
         bucket: AWS S3 bucket for uploading the final product(s)
         bucket_prefix: Add a bucket prefix to the product(s)
+        use_gslc_prefix: Upload GSLCs to a subprefix
         work_dir: Working directory for processing
         gpu: Use the GPU-based version of the workflow
     """
+    if use_gslc_prefix:
+        if not (bucket and bucket_prefix):
+            raise ValueError('bucket and bucket_prefix must be given if use_gslc_prefix is True')
+        bucket_prefix += '/GSLC_granules'
+
     utils.set_creds('EARTHDATA', earthdata_username, earthdata_password)
     if work_dir is None:
         work_dir = Path.cwd()
@@ -149,6 +156,14 @@ def main():
     parser.add_argument('--earthdata-password', default=None, help="Password for NASA's EarthData")
     parser.add_argument('--bucket', help='AWS S3 bucket HyP3 for upload the final product(s)')
     parser.add_argument('--bucket-prefix', default='', help='Add a bucket prefix to product(s)')
+    parser.add_argument(
+        '--use-gslc-prefix',
+        action='store_true',
+        help=(
+            'Upload GSLC granules to a subprefix located within the bucket and prefix given by the'
+            ' --bucket and --bucket-prefix options'
+        )
+    )
     parser.add_argument('--gpu', default=False, action='store_true', help='Use the GPU-based version of the workflow.')
     parser.add_argument(
         '--bounds',
@@ -159,11 +174,14 @@ def main():
     )
     parser.add_argument('granules', type=str.split, nargs='+', help='Level-0 S1 granule(s) to back-project.')
     args = parser.parse_args()
+
     args.granules = [item for sublist in args.granules for item in sublist]
+
     if args.bounds is not None:
         args.bounds = [float(item) for sublist in args.bounds for item in sublist]
         if len(args.bounds) != 4:
             parser.error('Bounds must have exactly 4 values: [min lon, min lat, max lon, max lat] in EPSG:4326.')
+
     back_project(**args.__dict__)
 
 

src/hyp3_srg/dem.py

@@ -32,7 +32,7 @@ def download_dem_for_srg(bounds: list[float], work_dir: Optional[Path]):
 
     Args:
         bounds: The bounds of the extent of the desired DEM - [min_lon, min_lat, max_lon, max_lat].
-        work_dir: The directory to save create the DEM in
+        work_dir: The directory to save the DEM in
 
     Returns:
         The path to the downloaded DEM

src/hyp3_srg/time_series.py

@@ -11,49 +11,29 @@
 from shutil import copyfile
 from typing import Iterable, Optional
 
-from boto3 import client
 from hyp3lib.aws import upload_file_to_s3
 from hyp3lib.fetch import download_file as download_from_http
 
 from hyp3_srg import dem, utils
 
 
-S3 = client('s3')
 log = logging.getLogger(__name__)
 
 
-def get_s3_args(uri: str, dest_dir: Optional[Path] = None) -> None:
-    """Retrieve the arguments for downloading from an S3 bucket
+def get_gslc_uris_from_s3(bucket: str, prefix: str = '') -> list[str]:
+    """Retrieve granule (zip files) uris from the given s3 bucket and prefix.
 
     Args:
-        uri: URI of the file to download
-        dest_dir: the directory to place the downloaded file in
+        bucket: the s3 bucket name
+        prefix: the path after the bucket and before the file
 
     Returns:
-        bucket: the s3 bucket to download from
-        key: the path to the file following the s3 bucket
-        out_path: the destination path of the file to download
+        uris: a list of uris to the zip files
     """
-    if dest_dir is None:
-        dest_dir = Path.cwd()
-
-    simple_s3_uri = Path(uri.replace('s3://', ''))
-    bucket = simple_s3_uri.parts[0]
-    key = '/'.join(simple_s3_uri.parts[1:])
-    out_path = dest_dir / simple_s3_uri.parts[-1]
-    return bucket, key, out_path
-
-
-def download_from_s3(uri: str, dest_dir: Optional[Path] = None) -> None:
-    """Download a file from an S3 bucket
-
-    Args:
-        uri: URI of the file to download
-        dest_dir: the directory to place the downloaded file in
-    """
-    bucket, key, out_path = get_s3_args(uri, dest_dir)
-    S3.download_file(bucket, key, out_path)
-    return out_path
+    res = utils.s3_list_objects(bucket, prefix)
+    keys = [item['Key'] for item in res['Contents']]
+    uris = ['/'.join(['s3://' + bucket, key]) for key in keys]
+    return uris
 
 
 def load_products(uris: Iterable[str], overwrite: bool = False):
@@ -74,7 +54,7 @@ def load_products(uris: Iterable[str], overwrite: bool = False):
         if product_exists and not overwrite:
             pass
         elif uri.startswith('s3'):
-            download_from_s3(uri, dest_dir=work_dir)
+            utils.download_from_s3(uri, dest_dir=work_dir)
         elif uri.startswith('http'):
             download_from_http(uri, directory=work_dir)
         elif len(Path(uri).parts) > 1:
@@ -88,26 +68,26 @@ def load_products(uris: Iterable[str], overwrite: bool = False):
     return granule_names
 
 
-def get_size_from_dem(dem_file: str) -> tuple[int]:
+def get_size_from_dem(dem_path: str) -> tuple[int, int]:
     """Get the length and width from a .rsc DEM file
 
     Args:
-        dem_file: path to the .rsc dem file.
+        dem_path: path to the .rsc dem file.
 
     Returns:
         dem_width, dem_length: tuple containing the dem width and dem length
     """
-    with open(dem_file) as dem:
-        width_line = dem.readline()
+    with open(dem_path) as dem_file:
+        width_line = dem_file.readline()
         dem_width = width_line.split()[1]
-        length_line = dem.readline()
+        length_line = dem_file.readline()
         dem_length = length_line.split()[1]
 
     return int(dem_width), int(dem_length)
 
 
 def generate_wrapped_interferograms(
-    looks: tuple[int], baselines: tuple[int], dem_shape: tuple[int], work_dir: Path
+    looks: tuple[int, int], baselines: tuple[int, int], dem_shape: tuple[int, int], work_dir: Path
 ) -> None:
     """Generates wrapped interferograms from GSLCs
 
@@ -127,7 +107,7 @@ def generate_wrapped_interferograms(
     utils.call_stanford_module('sentinel/ps_sbas_igrams.py', args=sbas_args, work_dir=work_dir)
 
 
-def unwrap_interferograms(dem_shape: tuple[int], unw_shape: tuple[int], work_dir: Path) -> None:
+def unwrap_interferograms(dem_shape: tuple[int, int], unw_shape: tuple[int, int], work_dir: Path) -> None:
     """Unwraps wrapped interferograms in parallel
 
     Args:
@@ -144,7 +124,7 @@ def unwrap_interferograms(dem_shape: tuple[int], unw_shape: tuple[int], work_dir
 
 
 def compute_sbas_velocity_solution(
-    threshold: float, do_tropo_correction: bool, unw_shape: tuple[int], work_dir: Path
+    threshold: float, do_tropo_correction: bool, unw_shape: tuple[int, int], work_dir: Path
 ) -> None:
     """Computes the sbas velocity solution from the unwrapped interferograms
 
@@ -167,11 +147,9 @@ def compute_sbas_velocity_solution(
         tropo_correct_args = ['unwlist', unw_width, unw_length]
         utils.call_stanford_module('int/tropocorrect.py', args=tropo_correct_args, work_dir=work_dir)
 
-    num_unw_files = 0
     with open(work_dir / 'unwlist', 'r') as unw_list:
         num_unw_files = len(unw_list.readlines())
 
-    num_slcs = 0
     with open(work_dir / 'geolist', 'r') as slc_list:
         num_slcs = len(slc_list.readlines())
 
@@ -180,8 +158,8 @@ def compute_sbas_velocity_solution(
 
 
 def create_time_series(
-    looks: tuple[int] = (10, 10),
-    baselines: tuple[int] = (1000, 1000),
+    looks: tuple[int, int] = (10, 10),
+    baselines: tuple[int, int] = (90, 1000),
     threshold: float = 0.5,
     do_tropo_correction: bool = True,
     work_dir: Path | None = None,
@@ -198,7 +176,7 @@ def create_time_series(
     dem_shape = get_size_from_dem('elevation.dem.rsc')
     generate_wrapped_interferograms(looks=looks, baselines=baselines, dem_shape=dem_shape, work_dir=work_dir)
 
-    unw_shape = get_size_from_dem(work_dir / 'dem.rsc')
+    unw_shape = get_size_from_dem(str(work_dir / 'dem.rsc'))
     unwrap_interferograms(dem_shape=dem_shape, unw_shape=unw_shape, work_dir=work_dir)
 
     compute_sbas_velocity_solution(
@@ -213,7 +191,7 @@ def create_time_series_product_name(
     """Create a product name for the given granules.
 
     Args:
-        granules: list of the granule names
+        granule_names: list of the granule names
         bounds: bounding box that was used to generate the GSLCs
 
     Returns:
@@ -290,13 +268,14 @@ def package_time_series(
         'velocity',
     ]
     [shutil.copy(sbas_dir / f, product_path / f) for f in to_keep]
-    shutil.make_archive(product_path, 'zip', product_path)
+    shutil.make_archive(str(product_path), 'zip', product_path)
     return zip_path
 
 
 def time_series(
     granules: Iterable[str],
     bounds: list[float],
+    use_gslc_prefix: bool,
     bucket: str = None,
     bucket_prefix: str = '',
     work_dir: Optional[Path] = None,
@@ -306,6 +285,7 @@ def time_series(
     Args:
         granules: List of Sentinel-1 GSLCs
         bounds: bounding box that was used to generate the GSLCs
+        use_gslc_prefix: Whether to download input granules from S3
         bucket: AWS S3 bucket for uploading the final product(s)
         bucket_prefix: Add a bucket prefix to the product(s)
         work_dir: Working directory for processing
@@ -316,6 +296,16 @@ def time_series(
     if not sbas_dir.exists():
         mkdir(sbas_dir)
 
+    if not (granules or use_gslc_prefix):
+        raise ValueError('use_gslc_prefix must be True if granules not provided')
+
+    if use_gslc_prefix:
+        if granules:
+            raise ValueError('granules must not be provided if use_gslc_prefix is True')
+        if not (bucket and bucket_prefix):
+            raise ValueError('bucket and bucket_prefix must be given if use_gslc_prefix is True')
+        granules = get_gslc_uris_from_s3(bucket, f'{bucket_prefix}/GSLC_granules')
+
     granule_names = load_products(granules)
     dem_path = dem.download_dem_for_srg(bounds, work_dir)
 
@@ -340,14 +330,33 @@ def main():
         S1A_IW_RAW__0SDV_20231229T134404_20231229T134436_051870_064437_5F38.geo
     """
     parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.ArgumentDefaultsHelpFormatter)
-    parser.add_argument(
-        '--bounds', default=None, type=float, nargs=4, help='Bounding box that was used to generate the GSLCs'
-    )
     parser.add_argument('--bucket', help='AWS S3 bucket HyP3 for upload the final product(s)')
     parser.add_argument('--bucket-prefix', default='', help='Add a bucket prefix to product(s)')
-    parser.add_argument('granules', type=str.split, nargs='+', help='GSLC granules.')
+    parser.add_argument(
+        '--bounds',
+        default=None,
+        type=str.split,
+        nargs='+',
+        help='DEM extent bbox in EPSG:4326: [min_lon, min_lat, max_lon, max_lat].'
+    )
+    parser.add_argument(
+        '--use-gslc-prefix',
+        action='store_true',
+        help=(
+            'Download GSLC input granules from a subprefix located within the bucket and prefix given by the'
+            ' --bucket and --bucket-prefix options'
+        )
+    )
+    parser.add_argument('granules', type=str.split, nargs='*', default='', help='GSLC granules.')
     args = parser.parse_args()
+
     args.granules = [item for sublist in args.granules for item in sublist]
+
+    if args.bounds is not None:
+        args.bounds = [float(item) for sublist in args.bounds for item in sublist]
+        if len(args.bounds) != 4:
+            parser.error('Bounds must have exactly 4 values: [min lon, min lat, max lon, max lat] in EPSG:4326.')
+
     time_series(**args.__dict__)