TypeError: A valid dtype is required. raised by dem_stitcher

[asjohnston-asf]

https://hyp3-api.asf.alaska.edu/jobs/5b3666ae-807f-48c6-8ceb-48d878f496fd failed with:

/opt/conda/envs/hyp3-isce2/lib/python3.11/site-packages/hyp3_isce2/__main__.py:40: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
  eps = entry_points()['hyp3']
2024-06-12 16:23:51,422 - hyp3_isce2.insar_tops_burst - INFO - Begin ISCE2 TopsApp run
2024-06-12 16:23:55,312 - hyp3_isce2.burst - INFO - Download attempt #1 for https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20191114T030111_20191114T030138_029901_036948_5BDC/IW1/VV/5.xml
2024-06-12 16:23:55,314 - hyp3_isce2.burst - INFO - Download attempt #1 for https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20191220T030110_20191220T030137_030426_037B6A_0B70/IW1/VV/5.xml
2024-06-12 16:23:55,316 - hyp3_isce2.burst - INFO - Download attempt #1 for https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20191114T030111_20191114T030138_029901_036948_5BDC/IW1/VV/5.tiff
2024-06-12 16:23:55,317 - hyp3_isce2.burst - INFO - Download attempt #1 for https://sentinel1-burst.asf.alaska.edu/S1A_IW_SLC__1SDV_20191220T030110_20191220T030137_030426_037B6A_0B70/IW1/VV/5.tiff
2024-06-12 16:23:59,500 - hyp3_isce2.burst - INFO - SAFEs created!
Input XML files:  ['/home/conda/S1A_IW_SLC__1SDV_20191114T030111_20191114T030138_029901_036948_5BDC.SAFE/annotation/s1a-iw1-slc-vv-20191114t030113-20191114t030138-029901-036948-004.xml']
Input TIFF files:  ['/home/conda/S1A_IW_SLC__1SDV_20191114T030111_20191114T030138_029901_036948_5BDC.SAFE/measurement/s1a-iw1-slc-vv-20191114t030113-20191114t030138-029901-036948-004.tiff']
Manifest files:  ['/home/conda/S1A_IW_SLC__1SDV_20191114T030111_20191114T030138_029901_036948_5BDC.SAFE/manifest.safe']
Setting IPF version to :  003.10
Extracting orbit from annotation XML file
Input XML files:  ['/home/conda/S1A_IW_SLC__1SDV_20191220T030110_20191220T030137_030426_037B6A_0B70.SAFE/annotation/s1a-iw1-slc-vv-20191220t030112-20191220t030137-030426-037b6a-004.xml']
Input TIFF files:  ['/home/conda/S1A_IW_SLC__1SDV_20191220T030110_20191220T030137_030426_037B6A_0B70.SAFE/measurement/s1a-iw1-slc-vv-20191220t030112-20191220t030137-030426-037b6a-004.tiff']
Manifest files:  ['/home/conda/S1A_IW_SLC__1SDV_20191220T030110_20191220T030137_030426_037B6A_0B70.SAFE/manifest.safe']
Setting IPF version to :  003.10
Extracting orbit from annotation XML file
2024-06-12 16:23:59,903 - hyp3_isce2.insar_tops_burst - INFO - InSAR ROI: (47.10616805417106, 37.85863091739162, 47.11616805417106, 37.86863091739163)
2024-06-12 16:23:59,903 - hyp3_isce2.insar_tops_burst - INFO - DEM ROI: (45.930848466765575, 37.863630917391625, 47.11116805417106, 38.21522409353816)

Opening glo_30 Datasets:   0%|          | 0/4 [00:00<?, ?it/s]
Opening glo_30 Datasets:  25%|██▌       | 1/4 [00:04<00:12,  4.13s/it]
Opening glo_30 Datasets: 100%|██████████| 4/4 [00:04<00:00,  1.03s/it]

Reading tile metadata:   0%|          | 0/4 [00:00<?, ?it/s]
Reading tile metadata: 100%|██████████| 4/4 [00:00<00:00, 4212.21it/s]

Reading tile imagery:   0%|          | 0/4 [00:00<?, ?it/s]
Reading tile imagery:  25%|██▌       | 1/4 [00:02<00:08,  2.89s/it]
Reading tile imagery:  50%|█████     | 2/4 [00:06<00:06,  3.03s/it]
Reading tile imagery: 100%|██████████| 4/4 [00:06<00:00,  1.50s/it]

Opening glo_90_missing Datasets:   0%|          | 0/2 [00:00<?, ?it/s]
Opening glo_90_missing Datasets:  50%|█████     | 1/2 [00:02<00:02,  2.38s/it]
Opening glo_90_missing Datasets: 100%|██████████| 2/2 [00:02<00:00,  1.19s/it]

Reading tile metadata:   0%|          | 0/2 [00:00<?, ?it/s]
Reading tile metadata: 100%|██████████| 2/2 [00:00<00:00, 4488.29it/s]

Reading tile imagery:   0%|          | 0/2 [00:00<?, ?it/s]
Reading tile imagery:  50%|█████     | 1/2 [00:06<00:06,  6.34s/it]
Reading tile imagery: 100%|██████████| 2/2 [00:06<00:00,  3.17s/it]
Traceback (most recent call last):
  File "rasterio/_io.pyx", line 1392, in rasterio._io.DatasetWriterBase.__init__
AssertionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/conda/envs/hyp3-isce2/lib/python3.11/site-packages/hyp3_isce2/__main__.py", line 51, in <module>
    main()
  File "/opt/conda/envs/hyp3-isce2/lib/python3.11/site-packages/hyp3_isce2/__main__.py", line 47, in main
    sys.exit(process_entry_point.load()())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hyp3-isce2/lib/python3.11/site-packages/hyp3_isce2/insar_tops_burst.py", line 530, in main
    insar_tops_burst(
  File "/opt/conda/envs/hyp3-isce2/lib/python3.11/site-packages/hyp3_isce2/insar_tops_burst.py", line 116, in insar_tops_burst
    dem_path = download_dem_for_isce2(dem_roi, dem_name='glo_30', dem_dir=dem_dir, buffer=0, resample_20m=False)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hyp3-isce2/lib/python3.11/site-packages/hyp3_isce2/dem.py", line 130, in download_dem_for_isce2
    with rasterio.open(dem_path, 'w', **dem_profile) as ds:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hyp3-isce2/lib/python3.11/site-packages/rasterio/env.py", line 451, in wrapper
    return f(*args, **kwds)
           ^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/hyp3-isce2/lib/python3.11/site-packages/rasterio/__init__.py", line 314, in open
    dataset = writer(
              ^^^^^^^
  File "rasterio/_io.pyx", line 1395, in rasterio._io.DatasetWriterBase.__init__
TypeError: A valid dtype is required.

I notice these bursts are in the neighborhood of Armenia/Azerbaijan, but an alternate job https://hyp3-api.asf.alaska.edu/jobs/5161a4db-ab75-4dfb-a25f-3d9dee77a8d4 in the area succeeded without issue, so it's not clear to me if that's a red herring.

[asjohnston-asf]

I over 90 examples of this error in the last 24 hours, all for jobs for burst id 079_168387_IW1