Should a user defined VFD capability be introduced?

[jedwards4b]

Following up on this discussion, should we allow for
a user defined VFD?

[jedwards4b]

@DennisHeimbigner you suggested that I follow H5FDhttp.c as an example, but I think that that part is already in the
vfd-gds hdf5 plug-in that I am using. I only needed one additional HDF5 call at the application level:
H5Pset_driver_by_name(fapl_id, "gds", NULL) and a couple of cuda memory management calls. I can't call the H5Pset_driver_by_name directly from my application because the fapl_id needs to be the same as that used by netcdf. Unless you can suggest a way to do that?

[DennisHeimbigner]

Sorry I was not clear. You will have to modify and rebuild the netcdf-c library code.
My reference to H5FDhttp was as a pointer into the netcdf-c code in libhdf5 where
you will need to set the VFD.

[jedwards4b]

I've added the user defined VFD at the hdf5 level - the question is how to access it from the netcdf level or above?
Do I copy the nc4_open_file and nc4_create_file and rename them GDS_open_file and GDS_create_file? Or more generally VDF_open_file and VDF_create_file - I suppose that if I do that I can redefine the mode flags. I can use a dispatch table to call these. Is that what you have in mind?

[DennisHeimbigner]

Ok, these issues should be addressed in the discussion you created.

[gsjaardema]

[Moved this from #2637]
I am looking at using the HDF5 subfiling VFD with netCDF. I have a similar issue as the above in that there is not a good way to pass down into netCDF the option to pick a particular VFD and then also how to set the parameters on that VFD.

There is the void *parameters argument down at the NC_create and below level, but I don't see any existing netCDF create or open calls that would allow me to set values in parameters and then interpret those down at the appropriate level.

For my experimentation, I can just work with environment variables and hard-coding, but eventually it would be nice to have a general mechanism for accessing the various HDF5 VFD and VOL capabilities without writing new UDF* options for each one.

The .rc file mechanism mentioned by @DennisHeimbigner in #2637 might work, but that puts some responsibility on the user and there are issues with running multiple outputs each with a different option (e.g., I want a restart/checkpoint file using subfiling VFD and a results file not using subfiling all in the same execution).

Would it make sense to add another nc_create_??? function which took the void *parameters option and passed it down through the call stack to the bottom-level function(s) that actually deal with creating (and opening) the HDF5 file?

@edhartnett @WardF

[gsjaardema]

An example of usage of the subfiling VFD:

  /* Set up File Access Property List with MPI parameters for the Subfiling VFD to use */
  hid_t    fapl_id = H5Pcreate(H5P_FILE_ACCESS);
  MPI_Info info    = MPI_INFO_NULL;
  H5Pset_mpi_params(fapl_id, comm, info);

  /* Get a default Subfiling and IOC configuration */
  H5FD_subfiling_config_t subf_config;
  H5Pget_fapl_subfiling(fapl_id, &subf_config);

  H5FD_ioc_config_t ioc_config;
  H5Pget_fapl_ioc(fapl_id, &ioc_config);

  /* Set Subfiling configuration to use a 1MiB stripe size. */
  subf_config.shared_cfg.stripe_size = 1048576;

  /* Set IOC configuration to use 2 worker threads per IOC instead of the default setting. */
  ioc_config.thread_pool_size = 2;

  /* Set our new configuration on the IOC FAPL used for Subfiling */
  H5Pset_fapl_ioc(subf_config.ioc_fapl_id, &ioc_config);

  /* Finally, set our new Subfiling configuration on the original FAPL */
  H5Pset_fapl_subfiling(fapl_id, &subf_config);

  /* Create a new Subfiling-based HDF5 file */
  hid_t file_id = H5Fcreate("my_h5_file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl_id);

Would want to be able to specify that user wanted subfiling VFD, and the stripe size of 1MiByte and the thread_pool_size of 2 in addition to some other user-settable parameters [user may be the actual user, or the application calling the netCDF library]

Maybe it is best to have a new UDF* that can be used for all/most VFD and VOL options instead of trying to push everything down through a void *parameters option...

[DennisHeimbigner]

I am considering the following solution: comments welcome.

Allow nc_open/create to take path of the form file://<absolute|relative file path>#hdf5.vfd=XXXX
where XXXX is some encoding of the VFD to use: "subfile", for example.
This has the advantage that it allows passing arbitrary info to the HDF5 library.
One could also shorthand this by not requiring the leading file://.

[gsjaardema]

That would solve part of the problem. It leaves the issue of how to specify parameters to the VFD. For example, subfiling might need setting of stripe size, stripe count, thread count, ...

[DennisHeimbigner]

That not a big problem if we can encode those params as strings, so we might have:

#hdf5.vfd=subfile(param,param,...)

Or, more verbosely:

#hdf5.vfd.subfile.stripesize=50&hdf5.vfd.subfile.threadcount=4...

[gsjaardema]

Yes, that should work and would be relatively easy to at least prototype with the current library.

[edwardhartnett]

I don't like having parameters passed as part of the string. I think that's overly complex.

Adding a new nc_create/nc_open variant would be preferable, IMO.

[DennisHeimbigner]

The disadvantage of extending the API is that it could impact all sorts of existing client programs.
It basically means that existing programs could not access new, VFD enabled, file formats.

[WardF]

So the main concerns about this I have are (as usual, I should make a template) the following:

• What about netCDF-Java? With netCDF-Java underpinning netCDF files in TDS, we have to pay attention to the impact new functionality in the C library will have on the Java project. I'm not certain there will be much impact here, but it will need to be considered. @haileyajohnson?
• Technical debt. vs. demand: We're fairly resource constrained; this has an impact not just on the implementation, but on supporting it in the future. Our community support (you guys!) are fantastic, but I don't want to take your availability for granted. We need to ensure whatever we do is maintainable with the resources on hand.
• Allocation of resources: Currently, we are seeing a lot more questions/interest around our ncZarr (and associated S3 access) support than we are fielding questions around 'will netCDF support user-defined VFDs?'. We also have a long laundry list of projects and tasks and regular maintenance (revamping the cmake integration in C/Fortran/C++, getting libnetcdf-cxx4 back up to speed, bringing our CI system into alignment with more modern artifact-generation systems, etc). Not to suggest we are any more resource constrained than any other open-source public-service project out there, but it remains a consideration.

Dennis already mentioned the potential impact on other existing software, so I won't reiterate those here.

I agree that this could be useful, and am not trying to end the conversation, I just wanted to share my thoughts while they were fresh in my mind. I don't see any technical reason why we shouldn't permit user-defined VFD's, but since they're a relatively new feature in HDF5, there's no urgency to incorporate them into libnetcdf, particularly at the expense of other, higher priority features and bug fixes.

[gsjaardema]

I understand completely. I will probably continue testing with a locally-modified version of netCDF and that will hopefully help drive me toward a better thought out design that I can then present as a prototype. As you noted, the VFD and VOL are fairly new and the VOL at least is/was churning the HDF5 API.

If I find that the VFD/VOL can give significant improvements in speed and/or capability, then we can revisit at that point and see how to best provide the capability to a larger netCDF user audience.