Automate Geoglam & NO2 dataset ingestion

[smohiudd]

Description

NO2 (#89) and Geoglam (#167, #173) datasets requires monthly ingestion as new assets are created. This is currently a manual process however should be automated. veda-data-airflow has a feature that allows scheduled ingestion by creating dataset specific DAGs. The file must still be transferred to the collection s3 bucket. A json file must be uploaded to the airflow event bucket. Here is an example json:

{
    "collection": "emit-ch4plume-v1",
    "bucket": "lp-prod-protected",
    "prefix": "EMITL2BCH4PLM.001/",
    "filename_regex": ".*.tif$",
    "schedule": "00 05 * * *",
    "assets": {
        "ch4-plume-emissions": {
            "title": "EMIT Methane Point Source Plume Complexes",
            "description": "Methane plume complexes from point source emitters.",
            "regex": ".*.tif$"
        }
    }
}

Acceptance Criteria

• transfer no2-monthly and no2-monthly-diff from s3://covid-eo-data bucket to s3://veda-data-store-staging and s3://veda-data-store using MWAA transfer dag
• scheduled ingestion (bi weekly) json files are created and uploaded for NO2 (no2-monthly, no2-monthly-diff) in mwaa event bucket for staging (UAH) and production (MCP)
• scheduled ingestion (bi weekly) json files are created and uploaded for Geoglam (geoglam) in mwaa event bucket for staging (UAH) and production (MCP)
• Validate that new ingestions are initiated in staging and production. Add new geoglam files from GEOGLAM August 2024 #167, GEOGLAM September & October 2024 #173 into veda-data-store-staging bucket.
• transfer, automated ingestion json configs are in veda-data

[slesaad]

Putting the discovery-items config within s3://<EVENT_BUCKET>/collections/ in the following format: https://github.com/US-GHG-Center/ghgc-data/blob/add/lpdaac-dataset-scheduled-config/ingestion-data/discovery-items/scheduled/emit-ch4plume-v1-items.json will trigger the discovery and subsequent ingestion of the collection items based on the schedule attribute

[smohiudd]

mcp-prod will need a new release of airflow to include automated ingestion

[smohiudd]

aws s3 ls s3://covid-eo-data/OMNO2d_HRMDifference/

aws s3 ls s3://covid-eo-data/OMNO2d_HRM/

[sandrahoang686]

Update: We have decided to run these weekly instead of bi-weekly

[anayeaye]

I added the scheduled collection configs from veda-data #177 to mcp-test and mcp-production