Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dev to main BLOCKED by issue #252 #245

Open
wants to merge 67 commits into
base: main
Choose a base branch
from
Open

dev to main BLOCKED by issue #252 #245

wants to merge 67 commits into from

Commits on Jul 19, 2024

  1. WIP: dynamic ingest mapping for continuous tracing

    @ividito
    ividito committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    c1d67da View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2024

  1. feat: use taskflow for concurrent task runs

    @ividito
    ividito committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    55744c6 View commit details
    Browse the repository at this point in the history

  2. clean up debug code

    @ividito
    ividito committed Jul 31, 2024
    Configuration menu
    Copy the full SHA
    ee98cf7 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. Bugfix vector ingest

    @ividito
    ividito committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    6e151a9 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. merge dev

    @smohiudd
    smohiudd committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    82c724f View commit details
    Browse the repository at this point in the history

  2. add generic vector pipeline

    @smohiudd
    smohiudd committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    54b885d View commit details
    Browse the repository at this point in the history

  3. remove process generic vector pipeline

    @smohiudd
    smohiudd committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    a7b51e1 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. Update dags/veda_data_pipeline/veda_generic_vector_pipeline.py

    @ividito
    ividito authored Aug 27, 2024
    Configuration menu
    Copy the full SHA
    318d39a View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Restructure dataset get_files output, add retries to some tasks to av… 

    …oid SIGKILL errors
    @ividito
    ividito committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    0ab867b View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Changes to make vector ingest work in shared VPC environment

    @ividito
    ividito committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    f1411fe View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. feat:port event-driven vector automation to terraform

    @ividito
    ividito committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    4db9b2c View commit details
    Browse the repository at this point in the history

  2. fix: feature flag for vector automation lambda

    @ividito
    ividito committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    ebb9993 View commit details
    Browse the repository at this point in the history

  3. fix: adjust log group count

    @ividito
    ividito committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    4eca40e View commit details
    Browse the repository at this point in the history

  4. fix tf vector automation vars

    @smohiudd
    smohiudd committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    0005892 View commit details
    Browse the repository at this point in the history

  5. update vars in event bridge tf

    @smohiudd
    smohiudd committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    38bff94 View commit details
    Browse the repository at this point in the history

  6. Merge pull request #239 from NASA-IMPACT/fix/vector-automation 

    Fix tf vector automation vars
    @smohiudd
    smohiudd authored Oct 18, 2024
    Configuration menu
    Copy the full SHA
    90aed8f View commit details
    Browse the repository at this point in the history

  7. add new line

    @smohiudd
    smohiudd committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    021f4ff View commit details
    Browse the repository at this point in the history

  8. Merge pull request #240 from NASA-IMPACT/fix/vector-automation 

    add new line
    @smohiudd
    smohiudd authored Oct 18, 2024
    Configuration menu
    Copy the full SHA
    8ec0cd4 View commit details
    Browse the repository at this point in the history

  9. fix tag filter

    @smohiudd
    smohiudd committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    abba102 View commit details
    Browse the repository at this point in the history

  10. Merge pull request #241 from NASA-IMPACT/fix/vector-automation 

    Fix tag filter
    @smohiudd
    smohiudd authored Oct 18, 2024
    Configuration menu
    Copy the full SHA
    006e989 View commit details
    Browse the repository at this point in the history

  11. update vector subnet ref

    @smohiudd
    smohiudd committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    f181ca7 View commit details
    Browse the repository at this point in the history

  12. Merge pull request #242 from NASA-IMPACT/fix/vector-automation 

    update vector subnet ref
    @smohiudd
    smohiudd authored Oct 18, 2024
    Configuration menu
    Copy the full SHA
    39401a0 View commit details
    Browse the repository at this point in the history

  13. fix vector subnet ids

    @smohiudd
    smohiudd committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    857e2fa View commit details
    Browse the repository at this point in the history

  14. Merge pull request #243 from NASA-IMPACT/fix/vector-automation 

    fix vector subnet ids
    @smohiudd
    smohiudd authored Oct 18, 2024
    Configuration menu
    Copy the full SHA
    9b43b89 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. conditional vector subnets

    @smohiudd
    smohiudd committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    e3870e7 View commit details
    Browse the repository at this point in the history

  2. feat: disable default api gateway endpoint for workflows api

    @botanical
    botanical committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    266be89 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #246 from NASA-IMPACT/fix/vector-ingest-subnets 

    Conditional vector subnets
    @smohiudd
    smohiudd authored Oct 21, 2024
    Configuration menu
    Copy the full SHA
    65f0a8c View commit details
    Browse the repository at this point in the history

  4. feat: add variable to configure disabling default apigw endpoint

    @botanical
    botanical committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    d4a57e3 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Merge pull request #197 from NASA-IMPACT/fix/asset-handling 

    Concurrency and taskflow refactors
    @smohiudd
    smohiudd authored Oct 22, 2024
    Configuration menu
    Copy the full SHA
    b9a9323 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Update infrastructure/terraform.tfvars.tmpl

    @botanical
    botanical authored Oct 23, 2024
    Configuration menu
    Copy the full SHA
    d77d946 View commit details
    Browse the repository at this point in the history

  2. Update infrastructure/terraform.tfvars.tmpl

    @botanical
    botanical authored Oct 23, 2024
    Configuration menu
    Copy the full SHA
    7af892a View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. Using pythonoperators instead of ECS operator

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    6d3e29a View commit details
    Browse the repository at this point in the history

  2. Merge pull request #247 from NASA-IMPACT/jt/issue-452-disable-default… 

    …-endpoint

feat: disable default api gateway endpoint for workflows api
    @botanical
    botanical authored Oct 24, 2024
    Configuration menu
    Copy the full SHA
    61a797d View commit details
    Browse the repository at this point in the history

  3. Switching to pythonOperator

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    686c4c8 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Merge pull request #249 from NASA-IMPACT/feature/use-pythonoperators 

    Feature/use pythonoperators
    @amarouane-ABDELHAK
    amarouane-ABDELHAK authored Oct 25, 2024
    Configuration menu
    Copy the full SHA
    c190564 View commit details
    Browse the repository at this point in the history

  2. test: temporarily use test buckets

    @anayeaye
    anayeaye committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    305dd43 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2024

  1. Add S3 event bridge

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    a619098 View commit details
    Browse the repository at this point in the history

  2. Add S3 event bridge

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    4c9753f View commit details
    Browse the repository at this point in the history

  3. Add S3 event bridge

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    59e1938 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2024

  1. Add env deployment example

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    685e7db View commit details
    Browse the repository at this point in the history

  2. Merge pull request #251 from NASA-IMPACT/feat/use-sm2a-in-eventbrige-… 

    …lambda

Feat/use sm2a in eventbrige lambda
    @amarouane-ABDELHAK
    amarouane-ABDELHAK authored Oct 30, 2024
    Configuration menu
    Copy the full SHA
    dd0ea9e View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. only execute deploy action on push to dev branch

    @anayeaye
    anayeaye committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    8a3c584 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #254 from NASA-IMPACT/ci/remove-staging-action 

    ci: remove automated deployment for staging environment
    @anayeaye
    anayeaye authored Nov 6, 2024
    Configuration menu
    Copy the full SHA
    4a277d3 View commit details
    Browse the repository at this point in the history

  3. ci: do not automatically deploy mwaa to any environment

    @anayeaye
    anayeaye committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    db73561 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2024

  1. Merge pull request #255 from NASA-IMPACT/ci/remove-all-mwaa-deploy-ac… 

    …tions

ci: do not automatically deploy mwaa to any environment
    @anayeaye
    anayeaye authored Nov 8, 2024
    Configuration menu
    Copy the full SHA
    c450be2 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. Adding condition on s3 event bridge

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    b258d41 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. Adding condition on s3 event bridge

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    eb5e32c View commit details
    Browse the repository at this point in the history

  2. Fix some missing variables

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    4909f43 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2024

  1. Remove .env file

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    87a08e2 View commit details
    Browse the repository at this point in the history

  2. Remove .env file

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    cf19bd3 View commit details
    Browse the repository at this point in the history

  3. Remove the airflow config to be created by CICD

    @amarouane-ABDELHAK
    amarouane-ABDELHAK committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    1335a92 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2024

  1. print all transfer exceptions

    @anayeaye
    anayeaye committed Nov 21, 2024
    Configuration menu
    Copy the full SHA
    dbcd631 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2024

  1. Merge pull request #257 from NASA-IMPACT/feat/use-sm2a-in-eventbrige-… 

    …lambda

Adding condition on s3 event bridge
    @amarouane-ABDELHAK
    amarouane-ABDELHAK authored Nov 22, 2024
    Configuration menu
    Copy the full SHA
    32b2acb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' into fix/reveal-transfer-exception

    @anayeaye
    anayeaye committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    7872364 View commit details
    Browse the repository at this point in the history

  3. print client err on failed transfer

    @anayeaye
    anayeaye committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    4374f34 View commit details
    Browse the repository at this point in the history

  4. raise transfer exception

    @anayeaye
    anayeaye committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    091bc80 View commit details
    Browse the repository at this point in the history

  5. update secret variable type to string and add assume role arns to air… 

    …flow_custom_variables"
    @anayeaye
    anayeaye committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    4f14c96 View commit details
    Browse the repository at this point in the history

  6. Merge pull request #260 from NASA-IMPACT/fix/reveal-transfer-exception 

    fix(transfer-dag): add needed assume role and stop discarding exceptions
    @anayeaye
    anayeaye authored Nov 22, 2024
    Configuration menu
    Copy the full SHA
    ef7be91 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2024

  1. fix(dockerfile) remove pypgstac install

    @anayeaye
    anayeaye committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    fe09938 View commit details
    Browse the repository at this point in the history

  2. fix(dag-requirements) remove pypgstac install

    @anayeaye
    anayeaye committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    56738e9 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #262 from NASA-IMPACT/fix/remove-pypgstac 

    fix: remove pypgstac install
    @anayeaye
    anayeaye authored Nov 25, 2024
    Configuration menu
    Copy the full SHA
    66193a8 View commit details
    Browse the repository at this point in the history

  4. let ingest api handle datetime validation and just pass through input… 

    …s in collection generation dag
    @anayeaye
    anayeaye committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    4fbd229 View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2024

  1. Merge pull request #263 from NASA-IMPACT/fix/collection-datetime-format 

    fix: remove python version dependent datetime formatter in collection generation
    @anayeaye
    anayeaye authored Nov 26, 2024
    Configuration menu
    Copy the full SHA
    d9799cc View commit details
    Browse the repository at this point in the history

  2. temp print statement

    @smohiudd
    smohiudd committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    926be29 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #264 from NASA-IMPACT/fix/collection-print 

    Collection generation print statement
    @smohiudd
    smohiudd authored Nov 26, 2024
    Configuration menu
    Copy the full SHA
    7e05d2f View commit details
    Browse the repository at this point in the history

  4. remove comma

    @smohiudd
    smohiudd committed Nov 26, 2024
    Configuration menu
    Copy the full SHA
    2974701 View commit details
    Browse the repository at this point in the history

  5. Merge pull request #265 from NASA-IMPACT/fix/collection-print 

    Remove comma in collection generation
    @smohiudd
    smohiudd authored Nov 26, 2024
    Configuration menu
    Copy the full SHA
    5d8757e View commit details
    Browse the repository at this point in the history