ValueError: max() arg is an empty sequence

[argavalen]

Hello, i installed gisola using this flags because i'm using CUDA 12.2

FFLAGS := -O2 -mcmodel=medium -acc=gpu,multicore -gpu=all,nofma -Minfo=accel

but when i do Test Functionality benchmark, i encounter this error at inversion calculation

Accelerator Fatal Error: call to cuInit returned error 802: Other

 This is ISOLA

 Initializing array w
 Time required:    0.2446730

 Reading input file gr.hes

2024-05-07 04:00:57,158 INFO Function 'calculateInversions' executed 2211.19 seconds
2024-05-07 04:00:57,161 ERROR Fatal Error occurred
Traceback (most recent call last):
  File "/root/Gisola/Gisola-main/src/./gisola.py", line 235, in <module>
    isola.gatherResults()
  File "/root/Gisola/Gisola-main/src/config.py", line 181, in inner
    func(*args, **kwargs)
  File "/root/Gisola/Gisola-main/src/isola.py", line 464, in gatherResults
    config.bestinvdir=inversions[corr.index(max(corr))]
ValueError: max() arg is an empty sequence
2024-05-07 04:00:57,163 INFO Moving to next event, if any

is there any way to fix this?

[nikosT]

Hi @argavalen , are you using the exact version (NVIDIA HPC SDK v. 20.9) mentioned here? https://github.com/nikosT/Gisola/wiki/Installation#prerequisites

This is important because we have encountered a version (and probably there are more) that was not working properly such as v. 21.3.

Did you try with openMP (running on CPU)? Is it working?

According to this thread: https://forums.developer.nvidia.com/t/error-802-system-not-yet-initialized-cuda-11-3/234955 it seems that there is a setup issue. Which GPU card do you use?

[argavalen]

It run succesfully when I use the OpenMP model for benchmark testing. Previously, I used Nvidia HPC SDK 23.9, and now I've switched to 20.9, but I still encounter the same error message. Here's the message.

 This is ISOLA

 Initializing array w
 Time required:    0.2542979

 Reading input file gr.hes
call to cuInit returned error 802: Other


2024-05-07 14:22:23,963 INFO Function 'calculateInversions' executed 2227.75 seconds
2024-05-07 14:22:23,965 ERROR Fatal Error occurred
Traceback (most recent call last):
  File "/root/Gisola/Gisola-main/src/./gisola.py", line 235, in <module>
    isola.gatherResults()
  File "/root/Gisola/Gisola-main/src/config.py", line 181, in inner
    func(*args, **kwargs)
  File "/root/Gisola/Gisola-main/src/isola.py", line 464, in gatherResults
    config.bestinvdir=inversions[corr.index(max(corr))]
ValueError: max() arg is an empty sequence
2024-05-07 14:22:23,966 INFO Moving to next event, if any

For the GPU, I'm using Nvidia H100, and the fabric manager is already installed.

nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    NODE    NODE    NODE    SYS     0-63,128-191    0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    PIX     PHB     PHB     SYS     0-63,128-191    0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    NODE    NODE    SYS     

Here are the compiler or library versions.

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.171.04             Driver Version: 535.171.04   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA H100 80GB HBM3          Off | 00000000:04:00.0 Off |                    0 |
| N/A   35C    P0              75W / 700W |      0MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          Off | 00000000:23:00.0 Off |                    0 |
| N/A   32C    P0              73W / 700W |      0MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA H100 80GB HBM3          Off | 00000000:44:00.0 Off |                    0 |

 nvfortran --version

nvfortran 20.9-0 LLVM 64-bit target on x86-64 Linux -tp zen
NVIDIA Compilers and Tools
Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.

 nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

Is the issue possibly due to different CUDA versions?

[nikosT]

Hmm, it could be. As I can see here https://docs.nvidia.com/hpc-sdk/archive/20.9/hpc-sdk-release-notes/index.html up to CUDA 11.0 is needed. Could you please use this CUDA version? Also, do not forget to recompile the code.
Perhaps, prof. @ivenetis can provide us with more suggestions on that.

[ivenetis]

We had indeed issues with different versions and combinations of NVidia HPC SDK and CUDA. I would suggest you stay with the recommended versions.