Memory usage optimization

[YooSunYoung]

The workflow crashes a kernel in a 16 GB VISA instance.

Number of Pixels: 4915200
(McStas file) Number of Rows: ?

16 GB of memory is expected to be enough.
We need to see why it crashes or see how we can optimize it.

[aaronfinke]

Please note the crash happens on a VISA instance at ILL, not ESS. However, the performance on the ESS VISA instance is very poor (might be an unrelated issue).

[SimonHeybrock]

Please add more details:

• Which workflow.
• Which file(s).
• In a Jupyter notebook? Is output caching disabled?

[aaronfinke]

https://confluence.ess.eu/display/~aaronfinke/ESSNMX+on+VISA+performance+comparison

[SimonHeybrock]

One thing that should in my opinion be avoided is the "merging of panels". It just adds overhead, and implies that just for the merge step we need a minimum 6x the size of a single panel in terms of memory. If all we want is write all panels to the same file (or even dataset), that can be achieved without merging as well.