CUSPARSE_VERSION used to select CUSPARSE_SPM*/M*_ALG_DEFAULT

[maartenarnst]

Kokkos kernels provides specialisations of spMV and spMM that call CUSPARSE functions. As part of the Cuda 11 release cycle, Nvidia changed the names of the enumerator values that set the default algorithm. Thus, we can find in several files in Kokkos kernels preprocessor conditionals that select between CUSPARSE_SPMV_ALG_DEFAULT and CUSPARSE_MV_ALG_DEFAULT for spMV and between CUSPARSE_SPMM_ALG_DEFAULT and CUSPARSE_MM_ALG_DEFAULT for spMM depending on the CUSPARSE_VERSION.

However, the CUSPARSE_VERSION used is not consistent across these files:

• In Kokkos-kernels/perf_test/sparse/KokkosSparse_spmv_struct_tuning.cpp, the conditional is #if CUSPARSE_VERSION >= 11201.
• In Kokkos-kernels/sparse/tpls/KokkosSparse_spmv_tpl_spec_decl.hpp, the conditional is #if CUSPARSE_VERSION >= 11301.
• In Kokkos-kernels/sparse/tpls/KokkosSparse_spmv_mv_tpl_spec_decl.hpp, the conditional is #if CUSPARSE_VERSION < 11010.

Moreover, the value may not always be well chosen. Indeed, as @etphipp pointed out in trilinos/Trilinos#12238, the CUSPARSE_VERSION does not necessarily match the cuda toolkit version, and Nvidia changed the enumerator values as part of the incremental subreleases in the Cuda 11 release cycle. In particular, in Cuda toolkit 11.1.1, the CUSPARSE_VERSION is 11301, and the names of the spMV default algorithm enumerator value is still CUSPARSE_MV_ALG_DEFAULT . Thus, choosing 11201 or 11010 as in the Kokkos kernels files mentioned above can be expected to lead to a compilation error in Cuda toolkit 11.1.1.

In trilinos/Trilinos#12238, @etphipp and I went through subsequent subreleases of Cuda 11 (see details in that PR). And we found that using a CUSPARSE_VERSION of 11401 would work. It's a subrelease in which both names are available and the older one is marked deprecated.

@etphipp is in favour of addressing this issue simultaneously in all concerned packages in Trilinos and setting the same threshold (Kokkos kernels, Stokhos, muelu). In this regard, I wanted to ask your suggestion how to proceed? Should I introduce a PR in Kokkos kernels with a change to a CUSPARSE_VERSION threshold of 11401? Thanks in advance!

[maartenarnst]

@lucbv @kliegeois @brian-kelley

[cwpearson]

Can't just grep the cuSPARSE version out of the CUDA 10 source code, so I don't know what it is.

CUDA Release	cuSPARSE Version	cusparseSpMVAlg_t	Citation
10.0	???	generic API not present	https://docs.nvidia.com/cuda/archive/10.0/cusparse/index.html
10.1	???	CUSPARSE_MV_ALG_DEFAULT	https://docs.nvidia.com/cuda/archive/10.1/cusparse/index.html
11.0.1	11.0.0	CUSPARSE_MV_ALG_DEFAULT	https://docs.nvidia.com/cuda/archive/11.0_GA/cusparse/index.html#cusparse-generic-function-spmv
11.1.1	11.3.0	...	https://docs.nvidia.com/cuda/archive/11.1.1/cusparse/index.html#cusparse-generic-function-spmv
11.2.0	11.3.1	...	https://docs.nvidia.com/cuda/archive/11.2.0/cusparse/index.html#cusparse-generic-function-spmv
11.2.1	11.4.0	CUSPARSE_MV_ALG_DEFAULT [D], CUSPARSE_SPMV_ALG_DEFAULT	https://docs.nvidia.com/cuda/archive/11.2.1/cusparse/index.html#cusparse-generic-function-spmv
11.2.2	11.4.1	...	https://docs.nvidia.com/cuda/archive/11.2.2/cusparse/index.html#cusparse-generic-function-spmv
11.8.0	11.7.5	...	https://docs.nvidia.com/cuda/archive/11.8.0/cusparse/index.html#cusparse-generic-function-spmv
12.0.0	12.0.0	Only lists CUSPARSE_SPMV_COO_ALG2 in table, but references others in text	https://docs.nvidia.com/cuda/archive/12.0.0/cusparse/index.html#cusparsespmv
12.0.1	12.0.1	CUSPARSE_SPMV_ALG_DEFAULT	https://docs.nvidia.com/cuda/archive/12.0.1/cusparse/index.html#cusparsespmv

[cwpearson]

Looks like the SpMV enums changed in cuSPARSE version 11.4.0 and became required in cuSPARSE version 12.0.0 (the abbreviated documentation is probably an error, Nvidia changed their documentation format in CUDA 12)

[cwpearson]

We need to do a similar thing to check the SpMM enums

[cwpearson]

CUDA Release	cuSPARSE Version	cusparseSpMMAlg_t	Citation
10.0	???	generic API not present	https://docs.nvidia.com/cuda/archive/10.0/cusparse/index.html
10.2	???	CUSPARSE_MM_ALG_DEFAULT	https://docs.nvidia.com/cuda/archive/10.2/cusparse/index.html#cusparse-generic-function-spmm
11.0.1	11.0.0	CUSPARSE_MM_ALG_DEFAULT [D] CUSPARSE_SPMM_ALG_DEFAULT	https://docs.nvidia.com/cuda/archive/11.0_GA/cusparse/index.html#cusparse-generic-function-spmm
11.8.0	11.7.5	...	https://docs.nvidia.com/cuda/archive/11.8.0/cusparse/index.html#cusparse-generic-function-spmm
12.0.0	12.0.0	Doesn't list a default in the table	https://docs.nvidia.com/cuda/archive/12.0.0/cusparse/index.html#cusparsespmm
12.0.1	12.0.1	CUSPARSE_SPMM_ALG_DEFAULT	https://docs.nvidia.com/cuda/archive/12.0.1/cusparse/index.html#cusparsespmm

[cwpearson]

The SpMM enums changed in CUDA 11.0.1 / cuSPARSE 11.0.0

[etphipp]

Has this been merged to Trilinos? I am still getting errors like

/ascldap/users/etphipp/project/Trilinos/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_tpl_spec_decl.hpp(98): error: identifier "CUSPARSE_SPMV_ALG_DEFAULT" is undefined

/ascldap/users/etphipp/project/Trilinos/Trilinos/packages/kokkos-kernels/sparse/tpls/KokkosSparse_spmv_tpl_spec_decl.hpp(112): error: identifier "CUSPARSE_SPMV_CSR_ALG2" is undefined

[brian-kelley]

@etphipp No, they are only in KokkosKernels develop at the moment.

@ndellingwood @cwpearson Is it too late for #2011 and #2012 to get cherry picked into release-candidate-4.2.00?

[lucbv]

Not yet but the release is underway

[brian-kelley]

@lucbv Carl's fixes for this aren't in the 4.2 candidate actually