Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

torch.cuda.OutOfMemoryError #91

Open
mneyanmels opened this issue Nov 2, 2024 · 7 comments
Open

torch.cuda.OutOfMemoryError #91

mneyanmels opened this issue Nov 2, 2024 · 7 comments

Comments

@mneyanmels
Copy link

mneyanmels commented Nov 2, 2024 •
edited
Loading

Anybody facing this error ?

image

nitFluxLoRATraining
'C:\Users\User\Desktop\Train\New folder' is not a directory

ComfyUI Error Report

Error Details

  • Node Type: FluxTrainLoop
  • Exception Type: torch.cuda.OutOfMemoryError
  • Exception Message: Allocation on device

Stack Trace
@mneyanmels mneyanmels changed the title ValueError- C:\Users\User\Desktop\Train\New folder' is not a directory torch.cuda.OutOfMemoryError Nov 3, 2024
@camoody1
Copy link

camoody1 commented Nov 4, 2024 •
edited
Loading

CPU: Intel 8700K
GPU: RTX 4070 Ti 12GB
RAM: 64GB
pytorch version: 2.5.0+cu124

Yes. I am also getting this error, now. It's been awhile since I trained a lora using this Kijai's trainer, but the last two loras I used it for ran perfectly well. Now, the workflow runs OOM before even one step of the first Flux Train Loop node can complete. This is my error:

# ComfyUI Error Report
## Error Details
- **Node Type:** FluxTrainLoop
- **Exception Type:** torch.OutOfMemoryError
- **Exception Message:** Allocation on device 
## Stack Trace

File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 323, in execute
output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 198, in get_output_data
return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 169, in _map_node_over_list
process_inputs(input_dict, i)

File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 158, in process_inputs
results.append(getattr(obj, func)(**inputs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\nodes.py", line 798, in train
steps_done = training_loop(
^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\train_network.py", line 1173, in training_loop
noise_pred, target, timesteps, huber_c, weighting = self.get_noise_pred_and_target(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\flux_train_network_comfy.py", line 427, in get_noise_pred_and_target
model_pred = unet(img=intermediate_img, txt=intermediate_txt, vec=vec, pe=pe, txt_attention_mask=t5_attn_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\accelerate\utils\operations.py", line 820, in forward
return model_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\accelerate\utils\operations.py", line 808, in call
return convert_to_fp32(self.model_forward(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\amp\autocast_mode.py", line 44, in decorate_autocast
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.py", line 1255, in forward
img = block(img, vec=vec, pe=pe, txt_attention_mask=txt_attention_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.py", line 835, in forward
return checkpoint(self._forward, x, vec, pe, txt_attention_mask, use_reentrant=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_compile.py", line 32, in inner
return disable_fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_dynamo\eval_frame.py", line 632, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\utils\checkpoint.py", line 496, in checkpoint
ret = function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.py", line 826, in _forward
attn = attention(q, k, v, pe=pe, attn_mask=attn_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-FluxTrainer\library\flux_models.py", line 445, in attention
x = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

## System Information
- **ComfyUI Version:** v0.2.6-9-g6c9dbde
- **Arguments:** ComfyUI\main.py --windows-standalone-build --fp8_e4m3fn-text-enc --normalvram --dont-upcast-attention --reserve-vram 1.25
- **OS:** nt
- **Python Version:** 3.11.9 (tags/v3.11.9:de54cf5, Apr  2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]
- **Embedded Python:** true
- **PyTorch Version:** 2.5.0+cu124
## Devices

- **Name:** cuda:0 NVIDIA GeForce RTX 4070 Ti : cudaMallocAsync
  - **Type:** cuda
  - **VRAM Total:** 12878086144
  - **VRAM Free:** 5931185884
  - **Torch VRAM Total:** 5737807872
  - **Torch VRAM Free:** 166115036

## Logs

2024-11-04 12:05:53,697 - root - INFO - Total VRAM 12282 MB, total RAM 65463 MB
2024-11-04 12:05:53,698 - root - INFO - pytorch version: 2.5.0+cu124
2024-11-04 12:05:53,699 - root - INFO - Set vram state to: NORMAL_VRAM
2024-11-04 12:05:53,699 - root - INFO - Device: cuda:0 NVIDIA GeForce RTX 4070 Ti : cudaMallocAsync
2024-11-04 12:05:56,162 - root - INFO - Using pytorch cross attention
2024-11-04 12:05:59,870 - root - INFO - [Prompt Server] web root: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\web
2024-11-04 12:05:59,873 - root - INFO - Adding extra search path checkpoints H:\Stable_Diffusion_Models\models/Checkpoints
2024-11-04 12:05:59,873 - root - INFO - Adding extra search path configs H:\Stable_Diffusion_Models\models/Checkpoints
2024-11-04 12:05:59,873 - root - INFO - Adding extra search path vae H:\Stable_Diffusion_Models\models/VAE
2024-11-04 12:05:59,873 - root - INFO - Adding extra search path loras H:\Stable_Diffusion_Models\models/Lora
2024-11-04 12:05:59,873 - root - INFO - Adding extra search path loras H:\Stable_Diffusion_Models\models/LyCORIS
2024-11-04 12:05:59,874 - root - INFO - Adding extra search path upscale_models H:\Stable_Diffusion_Models\models/upscale_models
2024-11-04 12:05:59,874 - root - INFO - Adding extra search path upscale_models H:\Stable_Diffusion_Models\models/BSRGAN
2024-11-04 12:05:59,874 - root - INFO - Adding extra search path upscale_models H:\Stable_Diffusion_Models\models/ESRGAN
2024-11-04 12:05:59,874 - root - INFO - Adding extra search path upscale_models H:\Stable_Diffusion_Models\models/GFPGAN
2024-11-04 12:05:59,874 - root - INFO - Adding extra search path upscale_models H:\Stable_Diffusion_Models\models/LDSR
2024-11-04 12:05:59,874 - root - INFO - Adding extra search path upscale_models H:\Stable_Diffusion_Models\models/RealESRGAN
2024-11-04 12:05:59,874 - root - INFO - Adding extra search path upscale_models H:\Stable_Diffusion_Models\models/SwinIR
2024-11-04 12:05:59,874 - root - INFO - Adding extra search path embeddings H:\Stable_Diffusion_Models\models/Embeddings
2024-11-04 12:05:59,874 - root - INFO - Adding extra search path hypernetworks H:\Stable_Diffusion_Models\models/hypernetworks
2024-11-04 12:05:59,874 - root - INFO - Adding extra search path controlnet H:\Stable_Diffusion_Models\models/ControlNet
2024-11-04 12:05:59,875 - root - INFO - Adding extra search path ipadapter H:\Stable_Diffusion_Models\models/IPAdapter
2024-11-04 12:05:59,875 - root - INFO - Adding extra search path clip_vision H:\Stable_Diffusion_Models\models/Clip_Vision
2024-11-04 12:05:59,875 - root - INFO - Adding extra search path clip H:\Stable_Diffusion_Models\models/Clip
2024-11-04 12:05:59,875 - root - INFO - Adding extra search path unet H:\Stable_Diffusion_Models\models/Unet
2024-11-04 12:05:59,875 - root - INFO - Adding extra search path svd H:\Stable_Diffusion_Models\models/Checkpoints/SVD
2024-11-04 12:05:59,875 - root - INFO - Adding extra search path deepbump H:\Stable_Diffusion_Models\models/Deepbump
2024-11-04 12:05:59,875 - root - INFO - Adding extra search path photomaker H:\Stable_Diffusion_Models\models/Photomaker
2024-11-04 12:05:59,875 - root - INFO - Adding extra search path inpaint H:\Stable_Diffusion_Models\models/Inpaint
2024-11-04 12:05:59,875 - root - INFO - Adding extra search path instantid H:\Stable_Diffusion_Models\models/InstantID
2024-11-04 12:05:59,876 - root - INFO - Adding extra search path layer_model H:\Stable_Diffusion_Models\models/layer_model
2024-11-04 12:05:59,876 - root - INFO - Adding extra search path stablesr H:\Stable_Diffusion_Models\models/StableSR
2024-11-04 12:05:59,876 - root - INFO - Adding extra search path image2text H:\Stable_Diffusion_Models\models/image2text
2024-11-04 12:05:59,876 - root - INFO - Adding extra search path llm_checkpoints H:\Stable_Diffusion_Models\models/LLM_checkpoints
2024-11-04 12:05:59,876 - root - INFO - Adding extra search path llavacheckpoints H:\Stable_Diffusion_Models\models/LLavacheckpoints
2024-11-04 12:05:59,876 - root - INFO - Adding extra search path ella H:\Stable_Diffusion_Models\models/Ella
2024-11-04 12:05:59,876 - root - INFO - Adding extra search path flux H:\Stable_Diffusion_Models\models/Flux
2024-11-04 12:06:05,087 - ComfyUI-CogVideoXWrapper.custom_cogvideox_transformer_3d - INFO - sageattn not found, using sdpa
2024-11-04 12:06:05,100 - ComfyUI-CogVideoXWrapper.cogvideox_fun.transformer_3d - INFO - sageattn not found, using sdpa
2024-11-04 12:06:05,103 - ComfyUI-CogVideoXWrapper.cogvideox_fun.fun_pab_transformer_3d - INFO - sageattn not found, using sdpa
2024-11-04 12:06:06,255 - dinov2 - WARNING - xFormers not available
2024-11-04 12:06:06,258 - dinov2 - WARNING - xFormers not available
2024-11-04 12:06:08,468 - root - INFO - Total VRAM 12282 MB, total RAM 65463 MB
2024-11-04 12:06:08,468 - root - INFO - pytorch version: 2.5.0+cu124
2024-11-04 12:06:08,470 - root - INFO - Set vram state to: NORMAL_VRAM
2024-11-04 12:06:08,470 - root - INFO - Device: cuda:0 NVIDIA GeForce RTX 4070 Ti : cudaMallocAsync
2024-11-04 12:06:09,498 - root - INFO - --------------
2024-11-04 12:06:09,498 - root - INFO - �[91m ### Mixlab Nodes: �[93mLoaded
2024-11-04 12:06:09,508 - root - INFO - ChatGPT.available True
2024-11-04 12:06:09,509 - root - INFO - edit_mask.available True
2024-11-04 12:06:09,667 - root - INFO - ClipInterrogator.available True
2024-11-04 12:06:10,076 - root - INFO - PromptGenerate.available True
2024-11-04 12:06:10,076 - root - INFO - ChinesePrompt.available True
2024-11-04 12:06:10,077 - root - INFO - RembgNode_.available True
2024-11-04 12:06:10,652 - root - INFO - TripoSR.available
2024-11-04 12:06:10,653 - root - INFO - MiniCPMNode.available
2024-11-04 12:06:10,709 - root - INFO - Scenedetect.available
2024-11-04 12:06:10,803 - root - INFO - FishSpeech.available
2024-11-04 12:06:10,831 - root - INFO - SenseVoice.available
2024-11-04 12:06:11,056 - root - INFO - Whisper.available False
2024-11-04 12:06:11,068 - root - INFO - FalVideo.available
2024-11-04 12:06:11,068 - root - INFO - �[93m -------------- �[0m
2024-11-04 12:06:15,369 - root - INFO -
Import times for custom nodes:
2024-11-04 12:06:15,369 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Photopea
2024-11-04 12:06:15,370 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\websocket_image_save.py
2024-11-04 12:06:15,370 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_UNet_bitsandbytes_NF4
2024-11-04 12:06:15,370 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Image-Selector
2024-11-04 12:06:15,370 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_bnb_nf4_fp4_Loaders
2024-11-04 12:06:15,370 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Miaoshouai-Tagger
2024-11-04 12:06:15,370 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-FBCNN
2024-11-04 12:06:15,370 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-mxToolkit
2024-11-04 12:06:15,370 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Noise
2024-11-04 12:06:15,371 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Inpaint-CropAndStitch
2024-11-04 12:06:15,371 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-sampler-lcm-alternative
2024-11-04 12:06:15,371 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Detail-Daemon
2024-11-04 12:06:15,371 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\Comfyui_TTP_Toolset
2024-11-04 12:06:15,371 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_segment_anything
2024-11-04 12:06:15,371 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\sd-dynamic-thresholding
2024-11-04 12:06:15,371 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_SUNoise
2024-11-04 12:06:15,372 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ASTERR
2024-11-04 12:06:15,372 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\FreeU_Advanced
2024-11-04 12:06:15,372 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\A8R8_ComfyUI_nodes
2024-11-04 12:06:15,372 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\efficiency-nodes-comfyui
2024-11-04 12:06:15,373 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\SD-Advanced-Noise
2024-11-04 12:06:15,373 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\cg-use-everywhere
2024-11-04 12:06:15,373 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\sigmas_tools_and_the_golden_scheduler
2024-11-04 12:06:15,373 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-previewlatent
2024-11-04 12:06:15,373 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus
2024-11-04 12:06:15,373 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_TiledKSampler
2024-11-04 12:06:15,373 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-enricos-nodes
2024-11-04 12:06:15,373 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-post-processing-nodes
2024-11-04 12:06:15,374 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-List-Utils
2024-11-04 12:06:15,374 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-eesahesNodes
2024-11-04 12:06:15,374 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\cg-image-picker
2024-11-04 12:06:15,374 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Image-Saver
2024-11-04 12:06:15,374 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-VideoHelperSuite
2024-11-04 12:06:15,374 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AutomaticCFG
2024-11-04 12:06:15,374 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WD14-Tagger
2024-11-04 12:06:15,374 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Stable-Video-Diffusion
2024-11-04 12:06:15,374 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-inpaint-nodes
2024-11-04 12:06:15,374 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux
2024-11-04 12:06:15,375 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_JPS-Nodes
2024-11-04 12:06:15,375 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\Comfyui-StableSR
2024-11-04 12:06:15,375 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\execution-inversion-demo-comfyui
2024-11-04 12:06:15,375 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Custom-Scripts
2024-11-04 12:06:15,375 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion
2024-11-04 12:06:15,375 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfy-plasma
2024-11-04 12:06:15,375 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\steerable-motion
2024-11-04 12:06:15,375 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Chibi-Nodes
2024-11-04 12:06:15,375 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-various
2024-11-04 12:06:15,376 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyMath
2024-11-04 12:06:15,376 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_essentials
2024-11-04 12:06:15,376 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack
2024-11-04 12:06:15,376 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_UltimateSDUpscale
2024-11-04 12:06:15,376 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfy-image-saver
2024-11-04 12:06:15,376 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Marigold
2024-11-04 12:06:15,376 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Frame-Interpolation
2024-11-04 12:06:15,376 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\Derfuu_ComfyUI_ModdedNodes
2024-11-04 12:06:15,376 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\x-flux-comfyui
2024-11-04 12:06:15,376 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-DepthAnythingV2
2024-11-04 12:06:15,377 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet
2024-11-04 12:06:15,377 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-GGUF
2024-11-04 12:06:15,377 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-prompt-reader-node
2024-11-04 12:06:15,377 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_ExtraModels
2024-11-04 12:06:15,377 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\rgthree-comfy
2024-11-04 12:06:15,377 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-SuperBeasts
2024-11-04 12:06:15,377 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Florence2
2024-11-04 12:06:15,377 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-PyramidFlowWrapper
2024-11-04 12:06:15,377 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_Comfyroll_CustomNodes
2024-11-04 12:06:15,377 - root - INFO - 0.0 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved
2024-11-04 12:06:15,377 - root - INFO - 0.1 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Inspire-Pack
2024-11-04 12:06:15,377 - root - INFO - 0.1 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-FluxTrainer
2024-11-04 12:06:15,378 - root - INFO - 0.1 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-segment-anything-2
2024-11-04 12:06:15,378 - root - INFO - 0.1 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfy_mtb
2024-11-04 12:06:15,378 - root - INFO - 0.1 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-KJNodes
2024-11-04 12:06:15,378 - root - INFO - 0.1 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-SAM2
2024-11-04 12:06:15,378 - root - INFO - 0.1 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_bitsandbytes_NF4
2024-11-04 12:06:15,378 - root - INFO - 0.2 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_MileHighStyler
2024-11-04 12:06:15,378 - root - INFO - 0.3 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-reactor-node
2024-11-04 12:06:15,378 - root - INFO - 0.3 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-SUPIR
2024-11-04 12:06:15,379 - root - INFO - 0.3 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_LayerStyle
2024-11-04 12:06:15,379 - root - INFO - 0.4 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_dagthomas
2024-11-04 12:06:15,379 - root - INFO - 0.4 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-CogVideoXWrapper
2024-11-04 12:06:15,379 - root - INFO - 0.5 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Easy-Use
2024-11-04 12:06:15,379 - root - INFO - 0.6 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Manager
2024-11-04 12:06:15,379 - root - INFO - 0.9 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AdvancedLivePortrait
2024-11-04 12:06:15,379 - root - INFO - 1.1 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Crystools
2024-11-04 12:06:15,379 - root - INFO - 1.2 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-art-venture
2024-11-04 12:06:15,379 - root - INFO - 1.3 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Allor
2024-11-04 12:06:15,379 - root - INFO - 1.5 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Inspyrenet-Rembg
2024-11-04 12:06:15,380 - root - INFO - 1.9 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-mixlab-nodes
2024-11-04 12:06:15,380 - root - INFO - 2.1 seconds: H:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\was-node-suite-comfyui
2024-11-04 12:06:15,380 - root - INFO -
2024-11-04 12:06:15,400 - root - INFO - Starting server

2024-11-04 12:06:15,400 - root - INFO - To see the GUI go to: http://127.0.0.1:8188
2024-11-04 13:04:15,111 - root - INFO - got prompt

@camoody1
Copy link

camoody1 commented Nov 4, 2024 •
edited
Loading

Update: I upgraded my pytorch version from 2.5.0+cu124 to 2.5.1+cu124. Now, my workflow no longer throws an IMMEDIATE out of memory error, but it just sits on the Flux Train Loop node for a minute or so before finally throwing the same OOM error. (On a second attempt, I never actually received an OOM error. I just sat on the node doing nothing.)

Is there any way to DOWNGRADE my version of pytorch back to 2.4.0? I tried following the steps on the pytorch.org website, but my pytorch version never actually changed. 🤷🏼

@camoody1
Copy link

camoody1 commented Nov 4, 2024

I was able to downgrade to pytorch version 2.4.1+cu124 and the workflow is now running as it used to.

Is there any way we can get this to run on the newer versions of pytorch? The newest ComfyUI installs come with 2.5.1 built it.

@kijai
Copy link
Owner

kijai commented Nov 5, 2024 •
edited
Loading

I was able to downgrade to pytorch version 2.4.1+cu124 and the workflow is now running as it used to.

Is there any way we can get this to run on the newer versions of pytorch? The newest ComfyUI installs come with 2.5.1 built it.

Seems to be some memory leak when using attn_mask, confirmed it doesn't work with 2.5.0, updating to torch nightly 2.6.0 (currently testing with torch-2.6.0.dev20241104) does fix it though.

Edit: downgraded to 2.5.1 now and it's also fine for me.

@camoody1
Copy link

camoody1 commented Nov 5, 2024

Edit: downgraded to 2.5.1 now and it's also fine for me.

This is interesting. I'm using 2.5.1 on a separate install and it's not working for me. Downgrading to 2.4.0 works fine, though.

@arcfun
Copy link

arcfun commented Dec 2, 2024

Why do I keep encountering the error "FluxTrainLoop Allocation on device" even after trying multiple versions, including 2.4, 2.5, and the 2.6 series?

@arcfun
Copy link

arcfun commented Dec 2, 2024

编辑：现在降级到 2.5.1，对我来说也很好。

这很有趣。我在单独安装中使用 2.5.1，但它无法正常工作。不过，降级到 2.4.0 可以正常工作。

Why do I keep encountering the error "FluxTrainLoop Allocation on device" even after trying multiple versions, including 2.4, 2.5, and the 2.6 series?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@arcfun @kijai @camoody1 @mneyanmels