Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add cnm i/o lambdas to build #414

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat: add cnm i/o lambdas to build #414

wants to merge 1 commit into from

Conversation

aliziel
Copy link
Contributor

@aliziel aliziel commented Oct 30, 2024 •
edited
Loading

What I am changing

Adds PO.DAAC-maintained lambdas to build.

CNMtoCMA (https://github.com/NASA-IMPACT/csda-project/issues/863)
CNMResponse (https://github.com/NASA-IMPACT/csda-project/issues/864)

Colocated with other ingest workflow lambdas (in cumulus/main.tf), but absolutely open to relocating while I'm still getting a feel for conventions in this repo !

NOTE: These two lambdas are what I'll be demoing on Friday (11/1), in case that helps with review, and I'm happy to tag up for a more technical walkthrough as well.

How I did it

  • Add PO.DAAC zips to build
  • Define lambdas

How you can test it

These functions expect Cumulus tasks as input, so the below JSON can be used for testing:

CNMtoCMA input 
{
  "cumulus_meta": {
    "cumulus_version": "18.2.0",
    "execution_name": "90c621d2-b73e-49ec-bff1-dc70d5b2c531",
    "message_source": "sfn",
    "queueExecutionLimits": {
      "https://sqs.us-west-2.amazonaws.com/772632177894/cumulus-ali-backgroundJobQueue": 200.0,
      "https://sqs.us-west-2.amazonaws.com/772632177894/cumulus-ali-backgroundProcessing": 5.0
    },
    "state_machine": "arn:aws:states:us-west-2:772632177894:stateMachine:cumulus-ali-CNMExampleWorkflow",
    "system_bucket": "csda-cumulus-ali-internal-7894",
    "workflow_start_time": 1730066233898.0,
    "queueUrl": "arn:aws:sqs:us-west-2:772632177894:cumulus-ali-startSF"
  },
  "meta": {
    "buckets": {
      "dashboard": {
        "name": "csda-cumulus-ali-dashboard-7894",
        "type": "dashboard"
      },
      "internal": {
        "name": "csda-cumulus-ali-internal-7894",
        "type": "internal"
      },
      "orca_default": {
        "name": "csda-cumulus-cba-uat-orca-archive",
        "type": "orca"
      },
      "orca_reports": {
        "name": "csda-cumulus-cba-uat-orca-reports",
        "type": "orca"
      },
      "private": {
        "name": "csda-cumulus-ali-private-7894",
        "type": "private"
      },
      "protected": {
        "name": "csda-cumulus-ali-protected-7894",
        "type": "protected"
      },
      "provider": {
        "name": "csda-cumulus-ali-provider-7894",
        "type": "provider"
      },
      "public": {
        "name": "csda-cumulus-ali-public-7894",
        "type": "public"
      }
    },
    "cmr": {
      "clientId": "csdap-cumulus-ali",
      "cmrEnvironment": "UAT",
      "cmrLimit": 100.0,
      "cmrPageSize": 50.0,
      "oauthProvider": "launchpad",
      "passwordSecretName": "",
      "provider": "CSDA",
      "username": ""
    },
    "collection": {
      "createdAt": 1729634300936.0,
      "updatedAt": 1729637773530.0,
      "name": "GE01_Pan_L1B",
      "version": "1",
      "url_path": "{cmrMetadata.CollectionReference.ShortName}___{cmrMetadata.CollectionReference.Version}/{dateFormat(cmrMetadata.TemporalExtent.SingleDateTime, YYYY)}/{dateFormat(cmrMetadata.TemporalExtent.SingleDateTime, DDDD)}/{cmrMetadata.GranuleUR}",
      "duplicateHandling": "skip",
      "granuleId": ".*",
      "granuleIdExtraction": "^(GE01_.+-P1BS-.+_P\\\\d{3}).+$",
      "files": [
        {
          "type": "browse",
          "regex": ".*-thumb[.]jpg$",
          "bucket": "public",
          "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001-thumb.jpg"
        },
        {
          "type": "data",
          "regex": ".*-BROWSE[.]jpg$",
          "bucket": "protected",
          "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001-BROWSE.jpg"
        },
        {
          "type": "metadata",
          "regex": ".*-cmr[.]json$",
          "bucket": "protected",
          "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001-cmr.json"
        },
        {
          "type": "data",
          "regex": ".*[.]tar$",
          "bucket": "protected",
          "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001.tar"
        },
        {
          "type": "data",
          "regex": ".*[.]ntf$",
          "bucket": "protected",
          "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001.ntf"
        },
        {
          "type": "data",
          "regex": ".*[.]xml$",
          "bucket": "protected",
          "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001.xml"
        }
      ],
      "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001-BROWSE.jpg",
      "ignoreFilesConfigForDiscovery": false,
      "meta": {
        "preferredQueueBatchSize": 1.0
      }
    },
    "distribution_endpoint": "https://ir1mh0lwu9.execute-api.us-west-2.amazonaws.com/dev/",
    "launchpad": {
      "api": "https://api.launchpad.nasa.gov/icam/api/sm/v1",
      "certificate": "launchpad.pfx",
      "passphraseSecretName": "cumulus-ali-message-template-launchpad-passphrase20241020004234462700000003"
    },
    "provider": {},
    "stack": "cumulus-ali",
    "template": "s3://csda-cumulus-ali-internal-7894/cumulus-ali/workflow_template.json",
    "workflow_name": "CNMExampleWorkflow",
    "workflow_tasks": {
      "0": {
        "name": "cumulus-ali-CNMToCMA",
        "version": "$LATEST",
        "arn": "arn:aws:lambda:us-west-2:772632177894:function:cumulus-ali-CNMToCMA"
      }
    },
    "triggerRule": "GE01_Pan_L1B___1_CNMTest",
    "snsSourceArn": "arn:aws:sns:us-west-2:772632177894:cumulus-ali-cnm-inbound"
  },
  "payload": {
    "version": "1.5.1",
    "provider": "maxar",
    "deliveryTime": "2024-10-12T16:50:23.458100",
    "collection": "GE01_Pan_L1B",
    "identifier": "c5c828ac328c97b5d3d1036d08898b30-12",
    "product": {
      "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001",
      "dataVersion": "1",
      "files": [
        {
          "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001-BROWSE.jpg",
          "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001-BROWSE.jpg",
          "size": 92168.0,
          "checksum": "66891025235e76269668f6ea574192c5c03296621f03ccf2f976e2ea5ac8559f",
          "checksumType": "SHA-256",
          "type": "browse"
        },
        {
          "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.tar",
          "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.tar",
          "size": 1648640.0,
          "checksum": "99496734affcd2c51a37a6c3351ae5c9617d5cfc1bb362a06b0edbe5c38a94f8",
          "checksumType": "SHA-256",
          "type": "ancillary"
        },
        {
          "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.ntf",
          "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.ntf",
          "size": 218914893.0,
          "checksum": "9f5505a51fc0b54d50e26703fd1ebc561dfa6797c671244ecc0dcc46ebe3edab",
          "checksumType": "SHA-256",
          "type": "data"
        },
        {
          "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.xml",
          "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.xml",
          "size": 1661126.0,
          "checksum": "824483c06b651802150686ddf4ba96ca4ea8b305c2487ffe7d14137badd59bf5",
          "checksumType": "SHA-256",
          "type": "metadata"
        }
      ]
    }
  },
  "task_config": {
    "collection": "{$.meta.collection}",
    "cumulus_message": {
      "outputs": [
        {
          "source": "{$.cnm}",
          "destination": "{$.meta.granule.queryFields.cnm}"
        },
        {
          "source": "{$.cnm}",
          "destination": "{$.meta.cnm}"
        },
        {
          "source": "{$.output}",
          "destination": "{$.payload}"
        }
      ]
    }
  }
}
CNMResponse input 
{
  "cma": {
    "task_config": {
      "OriginalCNM": "{$.meta.cnm}",
      "distribution_endpoint": "{$.meta.distribution_endpoint}",
      "response-endpoint": "{$.meta.cnmResponseStream}",
      "region": "us-west-2",
      "type": "{$.meta.cnmResponseMethod}",
      "WorkflowException": "{$.exception}",
      "cumulus_message": {
        "outputs": [
          {
            "source": "{$.cnm}",
            "destination": "{$.meta.granule.queryFields.cnm}"
          },
          {
            "source": "{$.cnm}",
            "destination": "{$.meta.cnmResponse}"
          },
          {
            "source": "{$.input.input}",
            "destination": "{$.payload}"
          }
        ]
      }
    },
    "event": {
      "cumulus_meta": {
        "cumulus_version": "18.2.0",
        "execution_name": "a9f02389-e45a-437d-b943-7d3a081304e3",
        "message_source": "sfn",
        "queueExecutionLimits": {
          "https://sqs.us-west-2.amazonaws.com/772632177894/cumulus-ali-backgroundJobQueue": 200.0,
          "https://sqs.us-west-2.amazonaws.com/772632177894/cumulus-ali-backgroundProcessing": 5.0
        },
        "state_machine": "arn:aws:states:us-west-2:772632177894:stateMachine:cumulus-ali-CNMExampleWorkflow",
        "system_bucket": "csda-cumulus-ali-internal-7894",
        "workflow_start_time": 1.730076769328E12,
        "queueUrl": "arn:aws:sqs:us-west-2:772632177894:cumulus-ali-startSF"
      },
      "meta": {
        "buckets": {
          "dashboard": {
            "name": "csda-cumulus-ali-dashboard-7894",
            "type": "dashboard"
          },
          "internal": {
            "name": "csda-cumulus-ali-internal-7894",
            "type": "internal"
          },
          "orca_default": {
            "name": "csda-cumulus-cba-uat-orca-archive",
            "type": "orca"
          },
          "orca_reports": {
            "name": "csda-cumulus-cba-uat-orca-reports",
            "type": "orca"
          },
          "private": {
            "name": "csda-cumulus-ali-private-7894",
            "type": "private"
          },
          "protected": {
            "name": "csda-cumulus-ali-protected-7894",
            "type": "protected"
          },
          "provider": {
            "name": "csda-cumulus-ali-provider-7894",
            "type": "provider"
          },
          "public": {
            "name": "csda-cumulus-ali-public-7894",
            "type": "public"
          }
        },
        "cmr": {
          "clientId": "csdap-cumulus-ali",
          "cmrEnvironment": "UAT",
          "cmrLimit": 100.0,
          "cmrPageSize": 50.0,
          "oauthProvider": "launchpad",
          "passwordSecretName": "",
          "provider": "CSDA",
          "username": ""
        },
        "collection": {
          "createdAt": 1.729634300936E12,
          "updatedAt": 1.72963777353E12,
          "name": "GE01_Pan_L1B",
          "version": "1",
          "url_path": "{cmrMetadata.CollectionReference.ShortName}___{cmrMetadata.CollectionReference.Version}/{dateFormat(cmrMetadata.TemporalExtent.SingleDateTime, YYYY)}/{dateFormat(cmrMetadata.TemporalExtent.SingleDateTime, DDDD)}/{cmrMetadata.GranuleUR}",
          "duplicateHandling": "skip",
          "granuleId": ".*",
          "granuleIdExtraction": "^(GE01_.+-P1BS-.+_P\\\\d{3}).+$",
          "files": [
            {
              "type": "browse",
              "regex": ".*-thumb[.]jpg$",
              "bucket": "public",
              "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001-thumb.jpg"
            },
            {
              "type": "data",
              "regex": ".*-BROWSE[.]jpg$",
              "bucket": "protected",
              "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001-BROWSE.jpg"
            },
            {
              "type": "metadata",
              "regex": ".*-cmr[.]json$",
              "bucket": "protected",
              "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001-cmr.json"
            },
            {
              "type": "data",
              "regex": ".*[.]tar$",
              "bucket": "protected",
              "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001.tar"
            },
            {
              "type": "data",
              "regex": ".*[.]ntf$",
              "bucket": "protected",
              "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001.ntf"
            },
            {
              "type": "data",
              "regex": ".*[.]xml$",
              "bucket": "protected",
              "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001.xml"
            }
          ],
          "sampleFileName": "GE01_20220412012140_105001002B1B4700_22APR12012140-P1BS-507956386020_01_P001-BROWSE.jpg",
          "ignoreFilesConfigForDiscovery": false,
          "meta": {
            "preferredQueueBatchSize": 1.0
          }
        },
        "distribution_endpoint": "https://ir1mh0lwu9.execute-api.us-west-2.amazonaws.com/dev/",
        "launchpad": {
          "api": "https://api.launchpad.nasa.gov/icam/api/sm/v1",
          "certificate": "launchpad.pfx",
          "passphraseSecretName": "cumulus-ali-message-template-launchpad-passphrase20241020004234462700000003"
        },
        "provider": {},
        "stack": "cumulus-ali",
        "template": "s3://csda-cumulus-ali-internal-7894/cumulus-ali/workflow_template.json",
        "workflow_name": "CNMExampleWorkflow",
        "workflow_tasks": {
          "0": {
            "name": "cumulus-ali-CNMToCMA",
            "version": "$LATEST",
            "arn": "arn:aws:lambda:us-west-2:772632177894:function:cumulus-ali-CNMToCMA"
          }
        },
        "triggerRule": "GE01_Pan_L1B___1_CNMTest",
        "cnmResponseMethod": "sns",
        "cnmResponseStream": "arn:aws:sns:us-west-2:772632177894:cumulus-ali-cnm-outbound",
        "snsSourceArn": "arn:aws:sns:us-west-2:772632177894:cumulus-ali-cnm-inbound",
        "granule": {
          "queryFields": {
            "cnm": {
              "version": "1.5.1",
              "provider": "maxar",
              "deliveryTime": "2024-10-12T16:50:23.458100",
              "collection": "GE01_Pan_L1B",
              "identifier": "c5c828ac328c97b5d3d1036d08898b30-12",
              "product": {
                "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001",
                "dataVersion": "1",
                "files": [
                  {
                    "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001-BROWSE.jpg",
                    "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001-BROWSE.jpg",
                    "size": 92168.0,
                    "checksum": "66891025235e76269668f6ea574192c5c03296621f03ccf2f976e2ea5ac8559f",
                    "checksumType": "SHA-256",
                    "type": "browse"
                  },
                  {
                    "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.tar",
                    "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.tar",
                    "size": 1648640.0,
                    "checksum": "99496734affcd2c51a37a6c3351ae5c9617d5cfc1bb362a06b0edbe5c38a94f8",
                    "checksumType": "SHA-256",
                    "type": "ancillary"
                  },
                  {
                    "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.ntf",
                    "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.ntf",
                    "size": 2.18914893E8,
                    "checksum": "9f5505a51fc0b54d50e26703fd1ebc561dfa6797c671244ecc0dcc46ebe3edab",
                    "checksumType": "SHA-256",
                    "type": "data"
                  },
                  {
                    "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.xml",
                    "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.xml",
                    "size": 1661126.0,
                    "checksum": "824483c06b651802150686ddf4ba96ca4ea8b305c2487ffe7d14137badd59bf5",
                    "checksumType": "SHA-256",
                    "type": "metadata"
                  }
                ]
              },
              "receivedTime": "2024-10-28T00:53:31.305Z"
            }
          }
        },
        "cnm": {
          "version": "1.5.1",
          "provider": "maxar",
          "deliveryTime": "2024-10-12T16:50:23.458100",
          "collection": "GE01_Pan_L1B",
          "identifier": "c5c828ac328c97b5d3d1036d08898b30-12",
          "product": {
            "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001",
            "dataVersion": "1",
            "files": [
              {
                "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001-BROWSE.jpg",
                "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001-BROWSE.jpg",
                "size": 92168.0,
                "checksum": "66891025235e76269668f6ea574192c5c03296621f03ccf2f976e2ea5ac8559f",
                "checksumType": "SHA-256",
                "type": "browse"
              },
              {
                "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.tar",
                "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.tar",
                "size": 1648640.0,
                "checksum": "99496734affcd2c51a37a6c3351ae5c9617d5cfc1bb362a06b0edbe5c38a94f8",
                "checksumType": "SHA-256",
                "type": "ancillary"
              },
              {
                "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.ntf",
                "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.ntf",
                "size": 2.18914893E8,
                "checksum": "9f5505a51fc0b54d50e26703fd1ebc561dfa6797c671244ecc0dcc46ebe3edab",
                "checksumType": "SHA-256",
                "type": "data"
              },
              {
                "uri": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.xml",
                "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.xml",
                "size": 1661126.0,
                "checksum": "824483c06b651802150686ddf4ba96ca4ea8b305c2487ffe7d14137badd59bf5",
                "checksumType": "SHA-256",
                "type": "metadata"
              }
            ]
          },
          "receivedTime": "2024-10-28T00:53:31.305Z"
        }
      },
      "payload": {
        "granules": [
          {
            "granuleId": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001",
            "version": "1",
            "dataType": "GE01_Pan_L1B",
            "files": [
              {
                "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001-BROWSE.jpg",
                "path": "css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01",
                "url_path": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001-BROWSE.jpg",
                "bucket": "csda-cumulus-ali-provider-7894",
                "source_bucket": "csda-cumulus-ali-provider-7894",
                "size": 92168.0,
                "checksumType": "SHA-256",
                "checksum": "66891025235e76269668f6ea574192c5c03296621f03ccf2f976e2ea5ac8559f",
                "type": "browse",
                "fileName": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001-BROWSE.jpg",
                "key": "css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001-BROWSE.jpg"
              },
              {
                "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.tar",
                "path": "css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01",
                "url_path": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.tar",
                "bucket": "csda-cumulus-ali-provider-7894",
                "source_bucket": "csda-cumulus-ali-provider-7894",
                "size": 1648640.0,
                "checksumType": "SHA-256",
                "checksum": "99496734affcd2c51a37a6c3351ae5c9617d5cfc1bb362a06b0edbe5c38a94f8",
                "type": "ancillary",
                "fileName": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.tar",
                "key": "css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.tar"
              },
              {
                "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.ntf",
                "path": "css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01",
                "url_path": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.ntf",
                "bucket": "csda-cumulus-ali-provider-7894",
                "source_bucket": "csda-cumulus-ali-provider-7894",
                "size": 2.18914893E8,
                "checksumType": "SHA-256",
                "checksum": "9f5505a51fc0b54d50e26703fd1ebc561dfa6797c671244ecc0dcc46ebe3edab",
                "type": "data",
                "fileName": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.ntf",
                "key": "css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.ntf"
              },
              {
                "name": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.xml",
                "path": "css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01",
                "url_path": "s3://csda-cumulus-ali-provider-7894/css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.xml",
                "bucket": "csda-cumulus-ali-provider-7894",
                "source_bucket": "csda-cumulus-ali-provider-7894",
                "size": 1661126.0,
                "checksumType": "SHA-256",
                "checksum": "824483c06b651802150686ddf4ba96ca4ea8b305c2487ffe7d14137badd59bf5",
                "type": "metadata",
                "fileName": "GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.xml",
                "key": "css/nga/GE01/1B/2015/100/GE01_10504100127F6300_M1BS_502547447070_01/GE01_20150410011735_10504100127F6300_15APR10011735-M1BS-502547447070_01_P001.xml"
              }
            ]
          }
        ]
      },
      "task_config": {
        "collection": "{$.meta.collection}",
        "cumulus_message": {
          "outputs": [
            {
              "source": "{$.cnm}",
              "destination": "{$.meta.granule.queryFields.cnm}"
            },
            {
              "source": "{$.cnm}",
              "destination": "{$.meta.cnm}"
            },
            {
              "source": "{$.output}",
              "destination": "{$.payload}"
            }
          ]
        }
      },
      "exception": "None"
    }
  }
}

  1. Invoke cumulus-ali-CNMToCMA either through the console or CLI
    => expect $.meta.cnm to capture original CNM
    => expect $.payload to reflect CNM content as granules

  2. Invoke cumulus-ali-CnmResponse either through the console or CLI
    => expect $.meta.cnmResponse to reflect original CNM
    => expect $.meta.cnmResponse to contain additional fields
    $.meta.cnmResponse.receivedTime
    $.meta.cnmResponse.response.status
    $.meta.cnmResponse.processCompleteTime

@aliziel aliziel marked this pull request as draft November 4, 2024 18:20
@aliziel
Copy link
Contributor Author

aliziel commented Nov 4, 2024

Kris and I decided it might be best to use this branch for the workflow + other infra changes as well so they can be reviewed all together — will reopen at that time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@aliziel