Config to limit CUDA memory usage

[BennetLeff]

Howdy! I got everything building and running on my 3080ti (so there's a new card confirmed working).

The truck demo scene worked fine. Later I tried to import my own scene and torch reserved too much GPU memory. I don't have this problem with the same dataset in the python/original implementation. This project would be more accessible if some configs existed to control this! I don't have time at the moment to fix it but might soon :)

[MrNeRF]

Salute!

Thanks for testing and reporting. I've got a spare 3080 Ti on my desk but not enough space to fit it in my PC. Could you provide insights on how you managed with the original setup? Which cmdline arguments did you use? Any configuration tips are welcome. If you could help us out with this issue it would be also great.

Thanks in advance!

[BennetLeff]

Yes I just followed the README to setup however I did have to ensure version compatibility. I think I started with 11.8 and it's true that 12.2 is necessary. I did notice the CMake files list the cuda arch as 89 in some places though it's commented out. I only note that because the 3080ti arch is 86 if I recall correctly.

And I'll need to read through your code a bit more. I haven't used torch in C++, just in python so I'm not completely familiar with the API. As I'm reading I'll see if I can find places to reduce VRAM usage. Btw it seems like you're doing some profiling, could you list any tools you've found useful? Thanks!

[MrNeRF]

I think CUDA 12.2 is overspecified. I don't have another parallel CUDA installation, thats why its there. @ihorizons2022 suggest that 11.8 is enough. Libtorch version is also optimized for 11.8. So I will adjust this requirement.

Libtorch is not that different from pytorch although there are some differences and at times it is more complicated to use.

CUDA profiling works best with NVIDIA tools, i.e. nsight compute for low level kernel profiling and nsight systems for a high level overview of what is going on in the entire application

[ihorizons2022]

I have OOM error on my own dataset too.

1. CUDA 11.8 is enough, the project works perfectly after changing the code in CMAKE file
2. In my Tesla T4 16G, T&T dataset works, but any other complex scenes will OOM
3. I will post my error later

[oliver-batchelor]

I've a lot of big datasets which OOM easily depends on what the "percent_dense" and image resolution I'm using. One of the most straightforward means of using less memory is not storing all the images in CUDA memory and instead loading them in at each iteration, they can be loaded asynchronously at the same time as the image is rendered e.g.

image_gpu = image_cpu.to(device = 'cuda', non_blocking=True).to(dtype= torch.float32) / 255

Perhaps has some overhead in terms of processing speed but in my usage it's not much different, especially when the images are pinned beforehand.

[BennetLeff]

Sounds like a good idea to investigate, thank you. I'm also looking into why some of my bigger datasets aren't crashing in the python implementation but crashing here.

[MrNeRF]

I have created an issue: #21
Today I can implement the changes/solutions suggested here unless anybody else wants to grab them. All of it should be straight forward.
Can someone provide me a dataset that fails? This would make it easier to test. It might also fail for me?

[MrNeRF]

Compute capability and cuda dependencies are solved. However, this comes with a cmake upgrade to 3.24.
@BennetLeff You want to try yourself to solve the loading issue because you can also immediately verify if it works. This should be not that hard.

[MrNeRF]

@BennetLeff @oliver-batchelor @ihorizons2022
There is a new flag --empty-gpu-cache that comes with commit 0068d9e. Hopefully, this solves your memory problems in most cases. I would appreciate if you could report if this helps. However, this comes with a performance penalty.

[MrNeRF]

I think we have a solution if nobody complains. I will close this discussion.