Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

upgrade to python 3.13, add mypy #2548

Merged
merged 11 commits into from
Jan 8, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/deploy-daac.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: 3.9
python-version: 3.13

- uses: ./.github/actions/deploy-hyp3
if: github.ref == matrix.deploy_ref
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-enterprise-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: 3.9
python-version: 3.13

- uses: ./.github/actions/deploy-hyp3
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-enterprise.yml
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: 3.9
python-version: 3.13

- uses: ./.github/actions/deploy-hyp3
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-multi-burst-sandbox.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: 3.9
python-version: 3.13

- uses: ./.github/actions/deploy-hyp3
with:
Expand Down
14 changes: 10 additions & 4 deletions .github/workflows/static-analysis.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
name: Static code analysis

permissions:
contents: read

on: push

jobs:
call-ruff-workflow:
# Docs: https://github.com/ASFHyP3/actions
uses: ASFHyP3/actions/.github/workflows/[email protected]

call-mypy-workflow:
uses: ASFHyP3/actions/.github/workflows/[email protected]

cfn-lint:
runs-on: ubuntu-latest
strategy:
Expand All @@ -17,7 +23,7 @@ jobs:
- uses: actions/[email protected]
- uses: actions/setup-python@v5
with:
python-version: 3.9
python-version: 3.13
- run: |
python -m pip install --upgrade pip
make install
Expand All @@ -31,7 +37,7 @@ jobs:
- uses: actions/[email protected]
- uses: actions/setup-python@v5
with:
python-version: 3.9
python-version: 3.13
- run: |
python -m pip install --upgrade pip
make install
Expand All @@ -48,7 +54,7 @@ jobs:
- run: gem install statelint
- uses: actions/setup-python@v5
with:
python-version: 3.9
python-version: 3.13
- run: |
python -m pip install --upgrade pip
make install
Expand All @@ -65,7 +71,7 @@ jobs:
- uses: snyk/actions/[email protected]
- uses: actions/setup-python@v5
with:
python-version: 3.9
python-version: 3.13
- run: |
python -m pip install --upgrade pip
make install
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:

- uses: actions/setup-python@v5
with:
python-version: 3.9
python-version: 3.13

- run: |
python -m pip install --upgrade pip
Expand Down
10 changes: 7 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,17 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [9.1.2]
## [9.2.0]

### Added
- OPERA_DISP_TMS job type is now available in EDC UAT deployment
- Add `mypy` to [`static-analysis`](.github/workflows/static-analysis.yml) workflow
- `OPERA_DISP_TMS` job type is now available in EDC UAT deployment

### Changed
- Upgrade to Python 3.13

### Removed
- hyp3-opera-disp-sandbox deployment
- Remove `hyp3-opera-disp-sandbox` deployment

## [9.1.1]

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ From the repository root,
make install
```

- Install Python dependencies for AWS Lambda functions (requires pip for python 3.9)
- Install Python dependencies for AWS Lambda functions (requires pip for python 3.13)
```sh
make build
```
Expand Down
2 changes: 1 addition & 1 deletion apps/api/api-cf.yml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ Resources:
Handler: hyp3_api.lambda_handler.handler
MemorySize: 3008
Role: !GetAtt LambdaRole.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 30
{% if security_environment == 'EDC' %}
VpcConfig:
Expand Down
3 changes: 1 addition & 2 deletions apps/api/src/hyp3_api/auth.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
import time
from os import environ
from typing import Optional

import jwt


def decode_token(token) -> Optional[dict]:
def decode_token(token) -> dict | None:
try:
return jwt.decode(token, environ['AUTH_PUBLIC_KEY'], algorithms=environ['AUTH_ALGORITHM'])
except (jwt.ExpiredSignatureError, jwt.DecodeError):
Expand Down
13 changes: 7 additions & 6 deletions apps/api/src/hyp3_api/routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from decimal import Decimal
from os import environ
from pathlib import Path
from typing import Any

import yaml
from flask import abort, g, jsonify, make_response, redirect, render_template, request
Expand Down Expand Up @@ -98,10 +99,10 @@ def default(self, o):


class CustomJSONProvider(JSONProvider):
def dumps(self, o):
return json.dumps(o, cls=CustomEncoder)
def dumps(self, obj: Any, **kwargs) -> str:
return json.dumps(obj, cls=CustomEncoder)

def loads(self, s):
def loads(self, s: str | bytes, **kwargs) -> Any:
return json.loads(s)


Expand All @@ -111,15 +112,15 @@ def __init__(self):

def __call__(self, errors):
response = super().__call__(errors)
error = response.json['errors'][0]
error = response.json['errors'][0] # type: ignore[index]
return handlers.problem_format(error['status'], error['title'])


app.json = CustomJSONProvider(app)

openapi = FlaskOpenAPIViewDecorator(
api_spec,
response_cls=None,
response_cls=None, # type: ignore[arg-type]
errors_handler_cls=ErrorHandler,
)

Expand All @@ -138,7 +139,7 @@ def jobs_post():
@app.route('/jobs', methods=['GET'])
@openapi
def jobs_get():
parameters = request.openapi.parameters.query
parameters = request.openapi.parameters.query # type: ignore[attr-defined]
start = parameters.get('start')
end = parameters.get('end')
return jsonify(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ Resources:
Handler: check_processing_time.lambda_handler
MemorySize: 128
Role: !GetAtt Role.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 30
{% if security_environment == 'EDC' %}
VpcConfig:
Expand Down
7 changes: 2 additions & 5 deletions apps/check-processing-time/src/check_processing_time.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,11 @@
from typing import Union


def get_time_from_result(result: Union[list, dict]) -> Union[list, float]:
def get_time_from_result(result: list | dict) -> list | float:
if isinstance(result, list):
return [get_time_from_result(item) for item in result]

return (result['StoppedAt'] - result['StartedAt']) / 1000


def lambda_handler(event, _) -> list[Union[list, float]]:
def lambda_handler(event, _) -> list | float:
processing_results = event['processing_results']
result_list = [processing_results[key] for key in sorted(processing_results.keys())]
return get_time_from_result(result_list)
2 changes: 1 addition & 1 deletion apps/disable-private-dns/disable-private-dns-cf.yml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Resources:
Handler: disable_private_dns.lambda_handler
MemorySize: 128
Role: !GetAtt Role.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 5
Environment:
Variables:
Expand Down
2 changes: 1 addition & 1 deletion apps/get-files/get-files-cf.yml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ Resources:
Handler: get_files.lambda_handler
MemorySize: 128
Role: !GetAtt Role.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 30
{% if security_environment == 'EDC' %}
VpcConfig:
Expand Down
3 changes: 1 addition & 2 deletions apps/get-files/src/get_files.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
from os import environ
from os.path import basename
from pathlib import Path
from typing import Union

import boto3

Expand Down Expand Up @@ -36,7 +35,7 @@ def get_object_file_type(bucket, key):
return None


def visible_product(product_path: Union[str, Path]) -> bool:
def visible_product(product_path: str | Path) -> bool:
return Path(product_path).suffix in ('.zip', '.nc', '.geojson')


Expand Down
2 changes: 1 addition & 1 deletion apps/handle-batch-event/handle-batch-event-cf.yml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ Resources:
Handler: handle_batch_event.lambda_handler
MemorySize: 128
Role: !GetAtt Role.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 30
{% if security_environment == 'EDC' %}
VpcConfig:
Expand Down
4 changes: 2 additions & 2 deletions apps/render_cf.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ def get_map_state(job_spec: dict, step: dict) -> dict:

def get_batch_submit_job_state(job_spec: dict, step: dict, filter_batch_params=False) -> dict:
if filter_batch_params:
batch_job_parameters = get_batch_job_parameters(job_spec, step)
batch_job_parameters: dict | str = get_batch_job_parameters(job_spec, step)
parameters_key = 'Parameters'
else:
batch_job_parameters = '$.batch_job_parameters'
Expand Down Expand Up @@ -120,7 +120,7 @@ def parse_map_statement(map_statement: str) -> tuple[str, str]:
return tokens[1], tokens[3]


def get_batch_job_parameters(job_spec: dict, step: dict, map_item: str = None) -> dict:
def get_batch_job_parameters(job_spec: dict, step: dict, map_item: str | None = None) -> dict:
job_params = {'bucket_prefix', *job_spec['parameters'].keys()}
step_params = get_batch_param_names_for_job_step(step)
batch_params = {}
Expand Down
4 changes: 2 additions & 2 deletions apps/scale-cluster/scale-cluster-cf.yml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ Resources:
Handler: scale_cluster.lambda_handler
MemorySize: 128
Role: !GetAtt Role.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 30
{% if security_environment == 'EDC' %}
VpcConfig:
Expand Down Expand Up @@ -161,7 +161,7 @@ Resources:
Handler: scale_cluster.lambda_handler
MemorySize: 128
Role: !GetAtt Role.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 30
{% if security_environment == 'EDC' %}
VpcConfig:
Expand Down
2 changes: 1 addition & 1 deletion apps/set-batch-overrides/set-batch-overrides-cf.yml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ Resources:
Handler: set_batch_overrides.lambda_handler
MemorySize: 128
Role: !GetAtt Role.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 30
{% if security_environment == 'EDC' %}
VpcConfig:
Expand Down
2 changes: 1 addition & 1 deletion apps/set-batch-overrides/src/set_batch_overrides.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
}


def get_container_overrides(memory: str, omp_num_threads: str = None) -> dict:
def get_container_overrides(memory: str, omp_num_threads: str | None = None) -> dict:
container_overrides = {
'ResourceRequirements': [
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Resources:
Handler: start_execution_manager.lambda_handler
MemorySize: 128
Role: !GetAtt Role.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 10
{% if security_environment == 'EDC' %}
VpcConfig:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Resources:
Handler: start_execution_worker.lambda_handler
MemorySize: 128
Role: !GetAtt Role.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 45
{% if security_environment == 'EDC' %}
VpcConfig:
Expand Down
2 changes: 1 addition & 1 deletion apps/update-db/update-db-cf.yml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Resources:
Handler: main.lambda_handler
MemorySize: 128
Role: !GetAtt Role.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 30
{% if security_environment == 'EDC' %}
VpcConfig:
Expand Down
3 changes: 1 addition & 2 deletions apps/upload-log/src/upload_log.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import json
from os import environ
from typing import Optional

import boto3
from botocore.config import Config
Expand All @@ -11,7 +10,7 @@
S3 = boto3.client('s3')


def get_log_stream(result: dict) -> Optional[str]:
def get_log_stream(result: dict) -> str | None:
if 'Error' in result:
result = json.loads(result['Cause'])
return result['Container'].get('LogStreamName')
Expand Down
2 changes: 1 addition & 1 deletion apps/upload-log/upload-log-cf.yml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ Resources:
Handler: upload_log.lambda_handler
MemorySize: 128
Role: !GetAtt Role.Arn
Runtime: python3.9
Runtime: python3.13
Timeout: 30
{% if security_environment == 'EDC' %}
VpcConfig:
Expand Down
2 changes: 1 addition & 1 deletion environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ channels:
- conda-forge
- nodefaults
dependencies:
- python=3.9
- python=3.13
- pip
- pip:
- -r requirements-all.txt
Loading
Loading