Increase in PutLogEvents for cloud watch due to batch job

[lacoak21]

I am trying to disable cloud watch logging because we had a multifold increase in Cloud Watch PutLogEvents. We didn't change anything for our cextflow pipelines and configuration so we are still unclear what happened. It was confirmed that the increase came from nextflow tower resources. In the meantime I would like to switch from forwarding logs to CloudWatch to
"logConfiguration": {
"logDriver": "json-file",
"options": {},
"secretOptions": []
},

It seems like there is no easy way to do this since next flow automatically creates the job definitions.

I could manually update the job definitions through the AWS-cli and then use process.container = 'job-definition://your-job-definition-name' for each one but we have many and that would be a tedious solution.

Do you have any suggestions?

[bentsherman]

Maybe the aws.batch.logsGroup config option?

https://nextflow.io/docs/latest/reference/config.html#aws

[mearatjames]

@bentsherman I'm having similar issue to @lacoak21 where I would like to disable the cloudwatch logs for the nextflow run using aws-batch to save on the aws cost.

I tried by removing the PutLogEvents permission from the Compute Environment instance role (based on what I found here https://nextflow.slack.com/archives/C02T97HAV5M/p1666983382733349?thread_ts=1664982143.297759&cid=C02T97HAV5M). However that resulted in the run failing because it cannot find the cloudwatch log for the jobs.

I tried looking into the aws.batch.logsGroup config option, but that seems to only allow to change the name of the cloudWatch logsGroup.

Is there a way that I can disable the cloudwatch logs for the nextflow run using aws-batch? Any suggestion is greatly appreciated.