GridToMeshESMFRegridder: TypeError: buffer is too small for requested array

[arjclark]

🐛 Bug Report

@stephenworsley - When working with large source data and trying to generate a regridder am hitting TypeError: buffer is too small for requested array

How To Reproduce

Steps to reproduce the behaviour:

1. Start a SPICE interactive session with 180GB memory (not actually necessary but gives sufficient overhead to prove this isn't a memory availability problem)
2. module load scitools
3. set: ulimit -u unlimited
4. start python
5. load in large source file (please contact separately for path)
6. load in a C48 target cube
7. generate regridder
8. wait for traceback to be generated

Expected behaviour

regridder to be generated

Environment

• SPICE interactive
• scitools/default-current
• esmf_regrid version: 0.5.0

Additional context

We need to be able to regrid this particular large source file to a range of UGrid resolutions (target C896 for now) in order to generate ancillary files for starting LFRic. An example script and source files can be provided on request but am not able to advertise on a public repo.

[stephenworsley]

As mentioned offline, this seems to be due to the way esmpy handles arrays and I'm not sure how possible it would be to solve this directly so I think this could be solved with a work around. If it is only the source data which is too large, it ought to be possible to create a series of regridders from slices of the large source which are each small enough to fit in memory. once regridding is done, the result can be concatenated back together. If this can be demonstrated as a viable solution, it should be possible to refine this process to be something that iris-esmf-regrid can handle internally.

[github-actions]

@SciTools-incubator/esmf-regrid-devs This issue is stale due to a lack of activity in the last 90 days. Remove stale label or comment, otherwise this issue will close automatically in 7 days time.

[arjclark]

Still relevant and proof of concept provided by @stephenworsley seems to be working in getting round this issue.

[github-actions]

@SciTools-incubator/esmf-regrid-devs This issue is stale due to a lack of activity in the last 180 days. Remove stale label or comment, otherwise this issue will close automatically in 14 days time.

[arjclark]

@stephenworsley - any chance this is going to get worked on any time soon?

[stephenworsley]

@arjclark We're hoping to work on this as part of a focus on performance as planned in the roadmap here #278. The next release (0.10) ought to finish the work on regridder unification so that this should be included in work on 0.11 which ought to be starting up around March. If things go well there should be a release by summer, though that depends on how successful attempts at performance improvements are so I can't guarantee when these improvements will be released

[github-actions]

@SciTools-incubator/esmf-regrid-devs This issue is stale due to a lack of activity in the last 180 days. Remove stale label or comment, otherwise this issue will close automatically in 14 days time.

[arjclark]

@stephenworsley - where are we at with this?

[stephenworsley]

This is lined up to be the next bit of big work we do on iris-esmf-regrid. The timeline on this depends on how big this turns out to be, but I'd hope to start looking into this next month.

[hdyson]

Just to raise that we've encountered this issue in the wild with an additional dataset beyond the one from the original comment here. I'm happy to share the path for the other data set offline too, if that would be helpful.

[stephenworsley]

I'd be interested to see if this is addressed by #396. Since ESMPy v8.1.0, there has been a "large_file" argument which looks like the kind of thing which could help here.