transient vs committed types

[kmuehlbauer]

Updates:

• Wrapped the output blocks in collapsible markdown.
• added code comments

This is a more general question how to use transient types in connection with committed types especially when using user types (Compound, Enum, Vlen).

I've currently asked Unidata/netcdf4-python#1308 where I'm able to create a malformed netcdf file using netcdf4-python where a variable correctly written can't be read back properly.

Over at h5netcdf (h5netcdf/h5netcdf#228) I've taken a deeper look into that issue. IIUC netcdf-c creates a committed type (eg. for an enum). When creating a variable a copy of that committed enum type is written into the underlying hdf5 dataset as transient type [*].

I've created a workflow using h5netcdf/h5py to create a similar file with same committed types but with shared/linked types in the underlying dataset. That file can be read perfectly fine by netcdf-c, ncdump leads to same output. Using h5dump (and h5ls) reveal the difference in the two files.

Unfortunately I'm not that fluent in C so I hope you will excuse me providing just the python code and the two resulting files.

Questions:

1. What is the correct workflow of creating a committed type and correctly bind it to the dataset as well as to the _FillValue-attribute?
2. Is my assumption above [*] correct?
3. Is the provided file (test_h5.nc) up to netcdf standards, although it uses linked/shared committed type for dataset and _FillValue-attribute?

Please find below the python code to create the two files and the two resulting files (gzipped to be attachable). Thanks for any comments and suggestions.

import netCDF4
import numpy as np
import h5netcdf
import h5py

print(netCDF4.__version__)
print(netCDF4.getlibversion())
print(h5netcdf.__version__)
print(h5py.version.info)

def create_enum_file(tmp_local_netcdf, nc_module):
    # two enum dicts with just a minor difference
    enum_dict1 = dict(one=1, two=2, three=3, missing=255)
    enum_dict2 = dict(one=1, two=2, three=3, missing=254)
    
    with nc_module.Dataset(tmp_local_netcdf, "w") as ds:
        # create dimension to fit all enum member values
        ds.createDimension("enum_dim", 4)
        # create subgroup test1
        g1 = ds.createGroup("test1")
        # create two committed types
        # enum_type1 in root group named enum_t1 with enum_dict1 contents
        enum_type1 = ds.createEnumType(np.uint8, "enum_t1", enum_dict1)
        # enum_type2 in subgroup test1 named enum_t2 with enum_dict2 contents
        enum_type2 = g1.createEnumType(np.uint8, "enum_t2", enum_dict2)
    
        # create two variables in subgroup test1 corresponding to the above two enum types
        v = g1.createVariable(
            "enum_var1", enum_type1, ("enum_dim",), fill_value=enum_dict1["missing"]
        )
        v[:] = [1, 2, 255, 3]
        print(v[:])
        
        v = g1.createVariable(
            "enum_var2", enum_type2, ("enum_dim",), fill_value=enum_dict2["missing"]
        )
        v[:] = [1, 2, 254, 3]
        print(v[:])

# create with netCDF4
create_enum_file("test_nc.nc", netCDF4)
# create with h5netcdf (this branch https://github.com/kmuehlbauer/h5netcdf/tree/user-types)
create_enum_file("test_h5.nc", h5netcdf.legacyapi)

Output of the above:

  1.7.0
  4.9.2 of Dec 10 2023 17:23:27 $
  1.4.0.dev19+g2d56f9e
  Summary of the h5py configuration
  ---------------------------------
  
  h5py    3.10.0
  HDF5    1.14.3
  Python  3.12.1 | packaged by conda-forge | (main, Dec 23 2023, 08:03:24) [GCC 12.3.0]
  sys.platform    linux
  sys.maxsize     9223372036854775807
  numpy   1.26.3
  cython (built with) 0.29.36
  numpy (built against) 1.26.2
  HDF5 (built against) 1.14.3
  
  [1 2 -- 3]
  [1 2 -- 3]
  [  1   2 255   3]
  [  1   2 254   3]

$ ncdump test_nc.nc

netcdf test_nc {
types:
  ubyte enum enum_t1 {one = 1, two = 2, three = 3, missing = 255} ;
dimensions:
    enum_dim = 4 ;

group: test1 {
  types:
    ubyte enum enum_t2 {one = 1, two = 2, three = 3, missing = 254} ;
  variables:
    enum_t1 enum_var1(enum_dim) ;
	    enum_t1 enum_var1:_FillValue = missing ;
    enum_t2 enum_var2(enum_dim) ;
	    enum_t2 enum_var2:_FillValue = missing ;
  data:

   enum_var1 = one, two, _, three ;

   enum_var2 = one, two, _, three ;
  } // group test1
}

$ ncdump test_h5.nc

netcdf test_h5 {
types:
  ubyte enum enum_t1 {missing = 255, one = 1, three = 3, two = 2} ;
dimensions:
    enum_dim = 4 ;

group: test1 {
  types:
    ubyte enum enum_t2 {missing = 254, one = 1, three = 3, two = 2} ;
  variables:
    enum_t1 enum_var1(enum_dim) ;
	    enum_t1 enum_var1:_FillValue = missing ;
    enum_t2 enum_var2(enum_dim) ;
	    enum_t2 enum_var2:_FillValue = missing ;
  data:

   enum_var1 = one, two, _, three ;

   enum_var2 = one, two, _, three ;
  } // group test1
}

$ h5dump test_nc.nc

HDF5 "test_nc.nc" {
GROUP "/" {
   ATTRIBUTE "_NCProperties" {
      DATATYPE  H5T_STRING {
         STRSIZE 34;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "version=2,netcdf=4.9.2,hdf5=1.14.3"
      }
   }
   DATASET "enum_dim" {
      DATATYPE  H5T_IEEE_F32BE
      DATASPACE  SIMPLE { ( 4 ) / ( 4 ) }
      DATA {
      (0): 0, 0, 0, 0
      }
      ATTRIBUTE "CLASS" {
         DATATYPE  H5T_STRING {
            STRSIZE 16;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SCALAR
         DATA {
         (0): "DIMENSION_SCALE"
         }
      }
      ATTRIBUTE "NAME" {
         DATATYPE  H5T_STRING {
            STRSIZE 64;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SCALAR
         DATA {
         (0): "This is a netCDF dimension but not a netCDF variable.         4"
         }
      }
      ATTRIBUTE "REFERENCE_LIST" {
         DATATYPE  H5T_COMPOUND {
            H5T_REFERENCE { H5T_STD_REF_OBJECT } "dataset";
            H5T_STD_U32LE "dimension";
         }
         DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
         DATA {
         (0): {
               DATASET 93826121327728 "/test1/enum_var1",
               0
            },
         (1): {
               DATASET 93826121398336 "/test1/enum_var2",
               0
            }
         }
      }
      ATTRIBUTE "_Netcdf4Dimid" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SCALAR
         DATA {
         (0): 0
         }
      }
   }
   DATATYPE "enum_t1" H5T_ENUM {
      H5T_STD_U8LE;
      "one"              1;
      "two"              2;
      "three"            3;
      "missing"          255;
   };
   GROUP "test1" {
      DATATYPE "enum_t2" H5T_ENUM {
         H5T_STD_U8LE;
         "one"              1;
         "two"              2;
         "three"            3;
         "missing"          254;
      };
      DATASET "enum_var1" {
         DATATYPE  H5T_ENUM {
            H5T_STD_U8LE;
            "one"              1;
            "two"              2;
            "three"            3;
            "missing"          255;
         }
         DATASPACE  SIMPLE { ( 4 ) / ( 4 ) }
         DATA {
         (0): one, two, missing, three
         }
         ATTRIBUTE "DIMENSION_LIST" {
            DATATYPE  H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }
            DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
            DATA {
            (0): (DATASET 93826121393792 "/enum_dim")
            }
         }
         ATTRIBUTE "_FillValue" {
            DATATYPE  H5T_ENUM {
               H5T_STD_U8LE;
               "one"              1;
               "two"              2;
               "three"            3;
               "missing"          255;
            }
            DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
            DATA {
            (0): missing
            }
         }
         ATTRIBUTE "_Netcdf4Coordinates" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
            DATA {
            (0): 0
            }
         }
      }
      DATASET "enum_var2" {
         DATATYPE  H5T_ENUM {
            H5T_STD_U8LE;
            "one"              1;
            "two"              2;
            "three"            3;
            "missing"          254;
         }
         DATASPACE  SIMPLE { ( 4 ) / ( 4 ) }
         DATA {
         (0): one, two, missing, three
         }
         ATTRIBUTE "DIMENSION_LIST" {
            DATATYPE  H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }
            DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
            DATA {
            (0): (DATASET 93826121402320 "/enum_dim")
            }
         }
         ATTRIBUTE "_FillValue" {
            DATATYPE  H5T_ENUM {
               H5T_STD_U8LE;
               "one"              1;
               "two"              2;
               "three"            3;
               "missing"          254;
            }
            DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
            DATA {
            (0): missing
            }
         }
         ATTRIBUTE "_Netcdf4Coordinates" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
            DATA {
            (0): 0
            }
         }
      }
   }
}
}

$ h5dump test_h5.nc

HDF5 "test_h5.nc" {
GROUP "/" {
   ATTRIBUTE "_NCProperties" {
      DATATYPE  H5T_STRING {
         STRSIZE 63;
         STRPAD H5T_STR_NULLPAD;
         CSET H5T_CSET_ASCII;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SCALAR
      DATA {
      (0): "version=2,h5netcdf=1.4.0.dev19+g2d56f9e,hdf5=1.14.3,h5py=3.10.0"
      }
   }
   DATASET "enum_dim" {
      DATATYPE  H5T_IEEE_F32BE
      DATASPACE  SIMPLE { ( 4 ) / ( 4 ) }
      DATA {
      (0): 0, 0, 0, 0
      }
      ATTRIBUTE "CLASS" {
         DATATYPE  H5T_STRING {
            STRSIZE 16;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SCALAR
         DATA {
         (0): "DIMENSION_SCALE"
         }
      }
      ATTRIBUTE "NAME" {
         DATATYPE  H5T_STRING {
            STRSIZE 64;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_ASCII;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SCALAR
         DATA {
         (0): "This is a netCDF dimension but not a netCDF variable.         4"
         }
      }
      ATTRIBUTE "REFERENCE_LIST" {
         DATATYPE  H5T_COMPOUND {
            H5T_REFERENCE { H5T_STD_REF_OBJECT } "dataset";
            H5T_STD_U32LE "dimension";
         }
         DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
         DATA {
         (0): {
               DATASET 94231148364832 "/test1/enum_var1",
               0
            },
         (1): {
               DATASET 94231148433872 "/test1/enum_var2",
               0
            }
         }
      }
      ATTRIBUTE "_Netcdf4Dimid" {
         DATATYPE  H5T_STD_I32LE
         DATASPACE  SCALAR
         DATA {
         (0): 0
         }
      }
   }
   DATATYPE "enum_t1" H5T_ENUM {
      H5T_STD_U8LE;
      "missing"          255;
      "one"              1;
      "three"            3;
      "two"              2;
   };
   GROUP "test1" {
      DATATYPE "enum_t2" H5T_ENUM {
         H5T_STD_U8LE;
         "missing"          254;
         "one"              1;
         "three"            3;
         "two"              2;
      };
      DATASET "enum_var1" {
         DATATYPE  "/enum_t1"
         DATASPACE  SIMPLE { ( 4 ) / ( 4 ) }
         DATA {
         (0): one, two, missing, three
         }
         ATTRIBUTE "DIMENSION_LIST" {
            DATATYPE  H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }
            DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
            DATA {
            (0): (DATASET 94231148338032 "/enum_dim")
            }
         }
         ATTRIBUTE "_FillValue" {
            DATATYPE  "/enum_t1"
            DATASPACE  SCALAR
            DATA {
            (0): missing
            }
         }
         ATTRIBUTE "_Netcdf4Dimid" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SCALAR
            DATA {
            (0): 0
            }
         }
      }
      DATASET "enum_var2" {
         DATATYPE  "/test1/enum_t2"
         DATASPACE  SIMPLE { ( 4 ) / ( 4 ) }
         DATA {
         (0): one, two, missing, three
         }
         ATTRIBUTE "DIMENSION_LIST" {
            DATATYPE  H5T_VLEN { H5T_REFERENCE { H5T_STD_REF_OBJECT } }
            DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
            DATA {
            (0): (DATASET 94231148367504 "/enum_dim")
            }
         }
         ATTRIBUTE "_FillValue" {
            DATATYPE  "/test1/enum_t2"
            DATASPACE  SCALAR
            DATA {
            (0): missing
            }
         }
         ATTRIBUTE "_Netcdf4Dimid" {
            DATATYPE  H5T_STD_I32LE
            DATASPACE  SCALAR
            DATA {
            (0): 0
            }
         }
      }
   }
}
}

test_nc.nc.gz
test_h5.nc.gz

[kmuehlbauer]

Citing from NetCDF Users Guide v1.1, section NetCDF-4 User Defined Data Types:

Types may be defined in any group in the data file, but they are always available globally in the file.

Is this correct? It seems like it's not (see, Unidata/netcdf4-python#1308). Or is it the same as for dimensions Enhanced Data Model in NetCDF-4/HDF5 Files:

Dimensions are scoped such that they can be seen in all descendant groups. That is, dimensions can be shared between variables in different groups, if they are defined in a parent group.

[DennisHeimbigner]

Frankly, I cannot give a good answer to your issue. The use of duplicate transient types has along history. AFAIK, when netcdf started using HDF5, the early
versions of HDF5 there were only transient types available. At some point this all changed and netcdf has
a rather hack'ish use of HDF5 types internally.
Using committed types + type references would be desirable to match the semantics of netcdf more closely.
But backward compatibility must also be considered. There is a further complication of the Java implementation
of netcdf. So this may all take some time to figure out.

[kmuehlbauer]

Thanks Dennis, I've thought it would be a complex case.

It looks like the approach I'm currently following with file test_h5.nc is fully backwards compatible with at least netcdf4-python and netcdf-c (tested with ncdump). I'd very much appreciate if you could have a check on that file and share your thoughts.