[BUG] cuFile driver closing causes segfault upon program termination

[kingcrimsontianyu]

Describe the bug

cuDF accesses the cuFile API via the cufile_shim object that has a static storage duration, meaning its destructor is called after the main function returns. The cuFileDriverClose() internally calls CUDA API, resulting in UB (usually manifested as segfault), and therefore should not be called here. However, even in the absence of cuFileDriverClose(), cuFile will implicitly close the driver, during which process some CUDA calls are still made, likely causing segfault. The best way to clean up the resources needs to be revisited in the future.

For the time being, segfault cannot be completely avoided under the GDS I/O path, but at least cuFileDriverClose() should be removed from the destructor of cufile_shim.

Related issues from KvikIO:
rapidsai/kvikio#497

Steps/Code to reproduce bug
Run any program using GDS I/O.

Expected behavior
Free of segmentation fault.

Environment overview (please complete the following information)
N/A

Environment details
N/A

Additional context
N/A

[vyasr]

We've tackled similar issues in the past by leaking the resource and allow the OS to reclaim the objects when the process is terminated (see e.g. rapidsai/rmm#1375). That seems like the thing you're suggesting with removing the explicit close, but it doesn't resolve the other issue if cufile is also executing past main due to the static duration of the object created by cufile.

[kingcrimsontianyu]

I filed an internal bug, and the cuFile issue will be addressed in a future CUDA release 😀

[wence-]

@kingcrimsontianyu I suppose this means this needs to remain open until we are on the newer cuFile?

[kingcrimsontianyu]

@wence- Yes. I think so.

Summary of an update from cuFile team:

• For cuFile's implicit driver close that happens at program exit, cuFile still needs to clean up the CUDA resources, such as internal stream, and cannot let them leak, even though making CUDA API calls after main returns means UB.
• The cufile library does not have any means to know if the user has called cudaDeviceReset already (such as in our case where nvbench resets the device before main returns). They will reach out to the CUDA team to explore viable options. Ideally, during implicit driver close, cuFile should skip the cleanup step when it is aware that the user has reset the device.
• For now, the workaround is to explicitly call cuFileDriverClose prior to calling cudaDeviceReset (before main returns).

cc. @madsbk who has rolled out rapidsai/kvikio#514 for KvikIO using the same workaround.