Your request was too big error on stock RSV repo build via aws-batch

[corneliusroemer]

Current Behavior

When trying to run the nextstrain/rsv workflow using nextstrain build --aws-batch ... I get an error: botocore.exceptions.ClientError: An error occurred (MaxMessageLengthExceeded) when calling the PutObject operation: Your request was too big.

Expected behavior

It works, I don't get such an error, or at least cli can recover and advise me on how to workaround the issue.

How to reproduce

Steps to reproduce the current behavior:

1. gh repo clone nextstrain/rsv
2. Run:

nextstrain build \         
   --aws-batch \
   --detach \
   --no-download \
   --cpus 16 \
   --memory 64gib \
   --exec env \
   . \
     snakemake \                  
       --configfiles config/configfile.yaml  \                                 
       --printshellcmds

3. See error

Your environment: if running Nextstrain locally

• Operating system: macOS 13.3.1 on M1
• shell: zsh
• version: nextstrain.cli 6.2.1

Full logs

nextstrain build \         
     --aws-batch \
     --detach \
     --no-download \
     --cpus 16 \
     --memory 64gib \
     --exec env \
     . \
       snakemake \                  
         --configfiles config/configfile.yaml  \                                 
         --printshellcmds
 Nextstrain Run ID: b1df3dba-e7b9-4336-8eff-754cd2727e17
 Uploading /Users/corneliusromer/code/rsv to S3
 zipping: /Users/corneliusromer/code/rsv
 zipping: /Users/corneliusromer/code/rsv/bin
 zipping: /Users/corneliusromer/code/rsv/bin/notify-on-deploy
 zipping: /Users/corneliusromer/code/rsv/bin/set-branch-ingest-config
 zipping: /Users/corneliusromer/code/rsv/bin/notify-on-error
 zipping: /Users/corneliusromer/code/rsv/bin/write-envdir
 zipping: /Users/corneliusromer/code/rsv/bin/notify-on-start
 zipping: /Users/corneliusromer/code/rsv/bin/notify-on-success
 zipping: /Users/corneliusromer/code/rsv/config
 zipping: /Users/corneliusromer/code/rsv/config/clades_genome_a.tsv
 zipping: /Users/corneliusromer/code/rsv/config/clades_genome_b.tsv
 zipping: /Users/corneliusromer/code/rsv/config/color_orderings.tsv
 zipping: /Users/corneliusromer/code/rsv/config/clades_G_b.tsv
 zipping: /Users/corneliusromer/code/rsv/config/areference.gbk
 zipping: /Users/corneliusromer/code/rsv/config/configfile.yaml
 zipping: /Users/corneliusromer/code/rsv/config/clades_G_a.tsv
 zipping: /Users/corneliusromer/code/rsv/config/description.md
 zipping: /Users/corneliusromer/code/rsv/config/outliers.txt
 zipping: /Users/corneliusromer/code/rsv/config/breference.gbk
 zipping: /Users/corneliusromer/code/rsv/config/areference.fasta
 zipping: /Users/corneliusromer/code/rsv/config/color_schemes.tsv
 zipping: /Users/corneliusromer/code/rsv/config/nextstrain_automation.yaml
 zipping: /Users/corneliusromer/code/rsv/config/breference.fasta
 zipping: /Users/corneliusromer/code/rsv/config/auspice_config.json
 zipping: /Users/corneliusromer/code/rsv/ingest
 zipping: /Users/corneliusromer/code/rsv/ingest/bin
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/notify-on-job-start
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/transform-authors
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/transform-field-names
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/upload-to-s3
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/gene-coverage.py
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/transform-date-fields
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/sort.py
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/csv-to-ndjson.py
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/transform-genbank-location
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/ndjson-to-tsv-and-fasta
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/notify-slack
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/merge-user-metadata
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/notify-on-job-fail
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/notify-on-record-change
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/apply-geolocation-rules
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/join-metadata-and-clades.py
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/metadata_dedup.py
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/fasta-to-ndjson
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/sha256sum
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/s3-object-exists
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/sequencesandmetadata.py
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/transform-string-fields
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/cloudfront-invalidate
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/genbank-url
 zipping: /Users/corneliusromer/code/rsv/ingest/bin/transform-strain-names
 zipping: /Users/corneliusromer/code/rsv/ingest/config
 zipping: /Users/corneliusromer/code/rsv/ingest/config/b_1_reference.fasta
 zipping: /Users/corneliusromer/code/rsv/ingest/config/config.yaml
 zipping: /Users/corneliusromer/code/rsv/ingest/config/a_1_reference.fasta
 zipping: /Users/corneliusromer/code/rsv/ingest/config/a_3_reference.fasta
 zipping: /Users/corneliusromer/code/rsv/ingest/config/b_2_reference.fasta
 zipping: /Users/corneliusromer/code/rsv/ingest/config/b_3_reference.fasta
 zipping: /Users/corneliusromer/code/rsv/ingest/config/a_2_reference.fasta
 zipping: /Users/corneliusromer/code/rsv/ingest/config/optional.yaml
 zipping: /Users/corneliusromer/code/rsv/ingest/source-data
 zipping: /Users/corneliusromer/code/rsv/ingest/source-data/geolocation-rules.tsv
 zipping: /Users/corneliusromer/code/rsv/ingest/source-data/annotations.tsv
 zipping: /Users/corneliusromer/code/rsv/ingest/Snakefile
 zipping: /Users/corneliusromer/code/rsv/ingest/workflow
 zipping: /Users/corneliusromer/code/rsv/ingest/workflow/envs
 zipping: /Users/corneliusromer/code/rsv/ingest/workflow/envs/nextstrain.yaml
 zipping: /Users/corneliusromer/code/rsv/ingest/workflow/snakemake_rules
 zipping: /Users/corneliusromer/code/rsv/ingest/workflow/snakemake_rules/fetch_sequences.smk
 zipping: /Users/corneliusromer/code/rsv/ingest/workflow/snakemake_rules/upload.smk
 zipping: /Users/corneliusromer/code/rsv/ingest/workflow/snakemake_rules/sort.smk
 zipping: /Users/corneliusromer/code/rsv/ingest/workflow/snakemake_rules/transform.smk
 zipping: /Users/corneliusromer/code/rsv/README.md
 zipping: /Users/corneliusromer/code/rsv/env.d
 zipping: /Users/corneliusromer/code/rsv/Snakefile
 zipping: /Users/corneliusromer/code/rsv/logs
 zipping: /Users/corneliusromer/code/rsv/logs/traits_rsv_rsv.txt
 zipping: /Users/corneliusromer/code/rsv/workflow
 zipping: /Users/corneliusromer/code/rsv/workflow/envs
 zipping: /Users/corneliusromer/code/rsv/workflow/envs/nextstrain.yaml
 zipping: /Users/corneliusromer/code/rsv/workflow/snakemake_rules
 zipping: /Users/corneliusromer/code/rsv/workflow/snakemake_rules/glycosylation.smk
 zipping: /Users/corneliusromer/code/rsv/workflow/snakemake_rules/core.smk
 zipping: /Users/corneliusromer/code/rsv/workflow/snakemake_rules/clades.smk
 zipping: /Users/corneliusromer/code/rsv/workflow/snakemake_rules/export.smk
 zipping: /Users/corneliusromer/code/rsv/workflow/snakemake_rules/nextstrain_automation.smk
 zipping: /Users/corneliusromer/code/rsv/workflow/snakemake_rules/download.smk
 zipping: /Users/corneliusromer/code/rsv/scripts
 zipping: /Users/corneliusromer/code/rsv/scripts/clade_names.py
 zipping: /Users/corneliusromer/code/rsv/scripts/newreference.py
 zipping: /Users/corneliusromer/code/rsv/scripts/glycosylation.py
 zipping: /Users/corneliusromer/code/rsv/scripts/cut.py
 zipping: /Users/corneliusromer/code/rsv/scripts/align_for_tree.py
 zipping: /Users/corneliusromer/code/rsv/scripts/assign-colors.py
 zipping: /Users/corneliusromer/code/rsv/scripts/set_final_strain_name.py
 zipping: /Users/corneliusromer/code/rsv/scripts/wrangle_metadata.py
 zipping: /Users/corneliusromer/code/rsv/scripts/metadatadedup.py
 zipping: /Users/corneliusromer/code/rsv/.github
 zipping: /Users/corneliusromer/code/rsv/.github/workflows
 zipping: /Users/corneliusromer/code/rsv/.github/workflows/fetch-and-ingest.yaml
 zipping: /Users/corneliusromer/code/rsv/.github/workflows/rebuild.yaml
 /opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/botocore/utils.py:1720: FutureWarning: The S3RegionRedirector class has been deprecated for a new internal replacement. A future version of botocore may remove this class.
   warnings.warn(
 Traceback (most recent call last):
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/s3fs/core.py", line 112, in _error_wrapper
     return await func(*args, **kwargs)
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/aiobotocore/client.py", line 358, in _make_api_call
     raise error_class(parsed_response, operation_name)
 botocore.exceptions.ClientError: An error occurred (MaxMessageLengthExceeded) when calling the PutObject operation: Your request was too big.
 
 The above exception was the direct cause of the following exception:
 
 Traceback (most recent call last):
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/bin/nextstrain", line 8, in <module>
     sys.exit(main())
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/nextstrain/cli/__main__.py", line 19, in main
     return cli.run( argv[1:] )
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/nextstrain/cli/__init__.py", line 36, in run
     return opts.__command__.run(opts)
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/nextstrain/cli/command/build.py", line 195, in run
     return runner.run(opts, working_volume = opts.build, cpus = opts.cpus, memory = opts.memory)
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/nextstrain/cli/runner/__init__.py", line 232, in run
     return opts.__runner__.run(opts, argv, working_volume = working_volume, extra_env = extra_env, cpus = cpus, memory = memory)
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/nextstrain/cli/runner/aws_batch/__init__.py", line 129, in run
     remote_workdir = s3.upload_workdir(local_workdir, bucket, run_id)
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/nextstrain/cli/runner/aws_batch/s3.py", line 68, in upload_workdir
     with fsspec.open(object_url(remote_workdir), "wb", auto_mkdir = False) as remote_file:
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/fsspec/core.py", line 121, in __exit__
     self.close()
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/fsspec/core.py", line 141, in close
     f.close()
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/fsspec/spec.py", line 1789, in close
     self.flush(force=True)
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/fsspec/spec.py", line 1660, in flush
     if self._upload_chunk(final=force) is not False:
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/s3fs/core.py", line 2215, in _upload_chunk
     self.commit()
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/s3fs/core.py", line 2230, in commit
     write_result = self._call_s3(
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/s3fs/core.py", line 2082, in _call_s3
     return self.fs.call_s3(method, self.s3_additional_kwargs, *kwarglist, **kwargs)
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/fsspec/asyn.py", line 115, in wrapper
     return sync(self.loop, func, *args, **kwargs)
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/fsspec/asyn.py", line 100, in sync
     raise return_result
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/fsspec/asyn.py", line 55, in _runner
     result[0] = await coro
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/s3fs/core.py", line 347, in _call_s3
     return await _error_wrapper(
   File "/opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/lib/python3.10/site-packages/s3fs/core.py", line 139, in _error_wrapper
     raise err
 OSError: [Errno 40] Your request was too big.

[tsibley]

@corneliusroemer What's the output of nextstrain version --verbose?

[tsibley]

Hm. This is coming from inside the aiobotocore client used by fsspec/s3fs. My first guess (probably wrong) is that you have an incompatible combination of s3fs and aiobotocore in whatever environment nextstrain itself is running.

[corneliusroemer]

$ nextstrain version --verbose
nextstrain.cli 6.2.1

Python
  /opt/homebrew/Caskroom/miniforge/base/envs/nextstrain/bin/python3.10
  3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:31:57) [Clang 14.0.6 ]

Runners
  docker 
    nextstrain/base:build-20230411T103027Z (bb05df2db0ce, 2023-04-11 13:38:55 +0200 CEST)
    augur 21.1.0
    auspice v2.45.2
    fauna e3ed8e1
    sacra not present

  conda 
    nextstrain-base 20230407T195218Z (h0dc7051_1_locked, nextstrain)
    augur 21.1.0
    auspice 2.45.1

  singularity 
    docker://nextstrain/base (not present)

  ambient (default)
    augur 21.1.0
    auspice 2.45.1

  aws-batch 
    unknown

[tsibley]

I ask because I wonder if bioconda/bioconda-recipes#39711 is implicated. Though not certain if the expectation would be that you have a version of nextstrain-cli from Bioconda before or after that…

[tsibley]

Ah, the version is 6.2.1, which is after that Conda packaging change. Hmm.

[corneliusroemer]

This is my pip list:

Package                       Version
----------------------------- ---------
aiobotocore                   2.4.2
aioeasywebdav                 2.4.0
aiohttp                       3.8.4
aioitertools                  0.11.0
aiosignal                     1.3.1
amply                         0.1.5
appdirs                       1.4.4
appnope                       0.1.3
asttokens                     2.2.1
async-timeout                 4.0.2
attmap                        0.13.2
attrs                         22.2.0
autopep8                      2.0.2
backcall                      0.2.0
backports.functools-lru-cache 1.6.4
backrefs                      5.2
bcbio-gff                     0.6.9
bcrypt                        3.2.2
biopython                     1.80
black                         23.3.0
boto3                         1.26.111
botocore                      1.29.111
bracex                        2.2.1
brotlipy                      0.7.0
bx-python                     0.9.0
cachetools                    5.3.0
certifi                       2022.12.7
cffi                          1.15.1
charset-normalizer            2.1.1
click                         8.1.3
colorama                      0.4.6
comm                          0.1.3
commonmark                    0.9.1
ConfigArgParse                1.5.3
connection-pool               0.0.3
constellations                0.1.10
contourpy                     1.0.7
crc32c                        2.3.post0
cryptography                  39.0.0
cvxopt                        1.3.0
cycler                        0.11.0
dataclasses                   0.8
datrie                        0.8.2
debugpy                       1.6.7
decorator                     5.1.1
deepdiff                      6.3.0
defusedxml                    0.7.1
distlib                       0.3.6
docutils                      0.19
dpath                         2.1.5
dropbox                       11.36.0
entrypoints                   0.4
epiweeks                      2.1.4
exceptiongroup                1.1.1
executing                     1.2.0
fasteners                     0.17.3
fastjsonschema                2.16.3
filechunkio                   1.8
filelock                      3.11.0
fonttools                     4.39.3
frozenlist                    1.3.3
fsspec                        2023.4.0
ftputil                       5.0.4
future                        0.18.3
gitdb                         4.0.10
GitPython                     3.1.31
google-api-core               2.10.0
google-api-python-client      2.85.0
google-auth                   2.17.2
google-auth-httplib2          0.1.0
google-cloud-core             2.3.2
google-cloud-storage          2.8.0
google-crc32c                 1.1.2
google-resumable-media        2.4.1
googleapis-common-protos      1.57.0
grpcio                        1.46.3
httplib2                      0.22.0
humanfriendly                 10.0
idna                          3.4
importlib-metadata            6.3.0
importlib-resources           5.12.0
iniconfig                     2.0.0
ipdb                          0.13.13
ipykernel                     6.22.0
ipython                       8.7.0
isal                          1.1.0
isodate                       0.6.1
isort                         5.12.0
jedi                          0.18.2
Jinja2                        3.1.2
jmespath                      1.0.1
joblib                        1.2.0
jsonschema                    3.2.0
jupyter_client                8.1.0
jupyter_core                  5.3.0
kiwisolver                    1.4.4
libcst                        0.4.9
llist                         0.7.1
logmuse                       0.2.6
MarkupSafe                    2.1.2
matplotlib                    3.7.1
matplotlib-inline             0.1.6
memory-profiler               0.61.0
MonkeyType                    23.3.0
multidict                     6.0.4
munkres                       1.1.4
mypy                          1.2.0
mypy-extensions               1.0.0
natsort                       8.3.1
nbformat                      5.8.0
nest-asyncio                  1.5.6
networkx                      2.8.8
nextstrain-augur              21.1.0
nextstrain-cli                6.2.1
nodeenv                       1.7.0
numpy                         1.24.2
nwkfmt                        0.1.1
oauth2client                  4.1.3
ordered-set                   4.1.0
orjson                        3.8.10
packaging                     23.0
pandas                        1.5.3
pango-aliasor                 0.3.0
pango-designation             1.19
paramiko                      3.1.0
parso                         0.8.3
pathspec                      0.11.1
peppy                         0.35.5
pexpect                       4.8.0
phylo-treetime                0.9.4
pickleshare                   0.7.5
Pillow                        9.2.0
pip                           23.0.1
pipenv                        2023.3.20
plac                          1.3.5
platformdirs                  3.2.0
pluggy                        1.0.0
ply                           3.11
polars                        0.17.1
pooch                         1.7.0
prettytable                   3.7.0
prompt-toolkit                3.0.38
protobuf                      3.18.3
psutil                        5.9.4
ptpython                      3.0.20
ptyprocess                    0.7.0
PuLP                          2.7.0
pure-eval                     0.2.2
py                            1.11.0
pyasn1                        0.4.8
pyasn1-modules                0.2.7
pycodestyle                   2.10.0
pycparser                     2.21
pyfastx                       0.8.4
Pygments                      2.15.0
pygraphviz                    1.10
PyJWT                         2.6.0
pyllist                       0.3
PyNaCl                        1.5.0
pyOpenSSL                     23.1.1
pyparsing                     3.0.9
pyright                       1.1.302
pyrsistent                    0.19.3
pysam                         0.20.0
pysftp                        0.2.9
PySocks                       1.7.1
pytest                        7.3.0
python-dateutil               2.8.2
python-irodsclient            1.1.6
python-lzo                    1.14
pytz                          2023.3
pyu2f                         0.1.5
PyYAML                        6.0
pyzmq                         25.0.2
ratelimiter                   1.2.0
regex                         2023.3.23
requests                      2.28.2
reretry                       0.11.8
retry                         0.9.2
rich                          12.6.0
rsa                           4.9
s3fs                          2023.3.0
s3transfer                    0.6.0
scikit-learn                  1.2.2
scipy                         1.10.1
setuptools                    67.6.1
setuptools-scm                7.1.0
shellingham                   1.5.1
six                           1.16.0
slacker                       0.14.0
smart-open                    6.3.0
smmap                         3.0.5
snakefmt                      0.8.4
snakemake                     7.25.0
stack-data                    0.6.2
stone                         3.3.1
stopit                        1.1.2
tabulate                      0.9.0
threadpoolctl                 3.1.0
throttler                     1.2.1
toml                          0.10.2
tomli                         2.0.1
toposort                      1.10
tornado                       6.2
tqdm                          4.65.0
traitlets                     5.9.0
typer                         0.7.0
typing_extensions             4.5.0
typing-inspect                0.8.0
ubiquerg                      0.6.2
unicodedata2                  15.0.0
uritemplate                   4.1.1
urllib3                       1.26.15
veracitools                   0.1.3
virtualenv                    20.21.0
virtualenv-clone              0.5.4
wcmatch                       8.3
wcwidth                       0.2.6
wheel                         0.40.0
wrapt                         1.15.0
xopen                         1.7.0
xxhash                        0.0.0
yarl                          1.8.2
yte                           1.5.1
zipp                          3.15.0
zstandard                     0.19.0

Is it ok for boto3 and botocore to have different versions?

boto3                         1.26.111
botocore                      1.29.111

[tsibley]

This seems worth digging into, but I'd love if someone else in @nextstrain/core could pick it up.