Revised calcHTranspose operator

[johnmauff]

This PR represents a complete rewrite of the calcHTranspose operator. The previous implementation only stored the H matrix and accessed it using several indirect addressing arrays for the calcHTranspose operation. The previous storage format was non-standard and on large problems like the hurricane_4panel test case actually consumed more memory than just explicitly storing the H^t matrix. This PR changes the form of the calcHTranspose operator such that the H^t matrix is explicitly stored in a CSR format. This change enables both a reduction in memory usage for computationally large problems and execution time on the CPU. Due to reduction in memory usage, both the hurricane_4panel configuration can be run on 80GB H100 GPU's and the hurricane case can be run on a 40 GB A100 GPU without code changes. The version of the code has been tested on the following platforms and input configurations:

Derecho CPU

beltrami
supercell
hurricane
typhoonChanthu2020
hurricane_4panel

Derecho A100 GPU

beltrami
supercell
hurricane
typhoonChanthu2020

Casper H100 GPU

beltrami
supercell
hurricane
typhoonChanthu2020
hurricane_4panel

[sjsprecious]

Thanks @johnmauff for working on this. It is still unclear to me what you have done to reduce the memory usage but I have some clarification questions first.

[sjsprecious]

why shall we define *IH and *JH with different types?

[johnmauff]

Good question and this relates to your overall question as to why these changes save memory. The JH array can have values that range from (0,nstate-1) while IH can have values that range from (0,nnz-1). While nnz exceeds the threshold for 32-bit integers, nState does not. So we can save memory here by using 32-bit inttegers for the JH array which has a total number of elements of nnz. The previous implementation had multiple 64-bit integer index arrays that had nnz elements. I have added comments to the CostFunction3D.h to help clarify the differences in word size.

[sjsprecious]

Aha, I see. Thanks for your clarification and it now makes sense. As long as nState won't exceed the threshold for 32-bit integer in the future SAMURAI production test, this change seems safe to me.

[sjsprecious]

Thanks @johnmauff for addressing my comments. Here is the second round of my questions.

[sjsprecious]

Thanks @johnmauff for addressing all my comments.

I could confirm that I am able to run a 4-panel hurricane case on Casper's H100 GPU with your branch (it takes about 2 hours in total and runtime: Cost3D minimize: 5222).

One minor and optional suggestion: I prefer to merge my PR #66 first (still waiting for @mmbell 's review) so that we can merge it into your PR and test if it works as expected.

[johnmauff]

@sjsprecious Thanks for your thorough review. I am fine with waiting for PR #66 to be committed first.

[mmbell]

Thank you for making these changes and for the thorough discussion about the reasoning.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 32.62%. Comparing base (12e3fc1) to head (1367543).

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #68   +/-   ##
=======================================
  Coverage   32.62%   32.62%           
=======================================
  Files          51       51           
  Lines       16815    16815           
=======================================
  Hits         5486     5486           
  Misses      11329    11329           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.