[Bug]: API Call Uses More Memory than WebUI (Causes CUDA OOM)

[thechrismorin]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

Have you read FAQ on README?

• I have updated WebUI and this extension to the latest version

What happened?

When I send the below payload to WebUI via API, the app will crash with a Cuda out of memory error. I have a 4070, 12GB dedicated and 8GB shared. When I configure the web GUI with the same settings, it processes just fine. If I lower the video_length to something like 12 or 24 frames, i don't his the memory limit. But critically, the web GUI can seem to accomidate the higher frame lengths without issue.

Steps to reproduce the problem

API Request to Automatic1111:
payload = {
"alwayson_scripts": {
"AnimateDiff": {
"args": [
{
"batch_size" : 16,
"closed_loop" : "R-P",
"enable" : True,
"format" : ["PNG"],
"fps" : 12,
"interp" : "FILM",
"interp_x" : 10,
"model" : "animatediffMotion_v14.ckpt",
"overlap" : 1,
"stride" : 1,
"video_length" : 96,
}
]
}
},
"batch_size" : 1,
"cfg_scale" : 7,
"denoising_strength" : 0.7,
"height" : 540,
"negative_prompt" : negative_prompt,
"prompt" : prompt,
"sampler_name" : "DPM++ 2M Karras",
"seed" : -1,
"steps" : 20,
"width" : 960
}

What should have happened?

It should return a payload with 96 png frames included under "images". I can then convert this into a gif on my own.

Commit where the problem happens

webui: 1.8.0-RC
extension: Updated 2024-04-23

What browsers do you use to access the UI ?

No response

Command Line Arguments

API Request

Console logs

2024-05-05 12:21:02,254 - AnimateDiff - INFO - Hacking SD1.5 GroupNorm32 forward function.
2024-05-05 12:21:02,254 - AnimateDiff - INFO - Injecting motion module animatediffMotion_v14.ckpt into SD1.5 UNet input blocks.
2024-05-05 12:21:02,254 - AnimateDiff - INFO - Injecting motion module animatediffMotion_v14.ckpt into SD1.5 UNet output blocks.
2024-05-05 12:21:02,254 - AnimateDiff - INFO - Setting DDIM alpha.
2024-05-05 12:21:02,257 - AnimateDiff - INFO - Injection finished.
*** Error running before_process: C:\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "C:\stable-diffusion-webui\modules\scripts.py", line 776, in before_process
        script.before_process(p, *script_args)
      File "C:\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py", line 65, in before_process
        params.set_p(p)
      File "C:\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_ui.py", line 180, in set_p
        extract_frames_from_video(self)
      File "C:\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_utils.py", line 108, in extract_frames_from_video
        assert params.video_source, "You need to specify cond hint for ControlNet."
    AssertionError: You need to specify cond hint for ControlNet.

---
Warning: field infotext in API payload not found in <modules.processing.StableDiffusionProcessingTxt2Img object at 0x0000022B078AB820>.
  0%|                                                                                                                                                                                                                                | 0/20 [00:00<?, ?it/s]*** Error executing callback cfg_denoiser_callback for C:\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff.py
    Traceback (most recent call last):
      File "C:\stable-diffusion-webui\modules\script_callbacks.py", line 230, in cfg_denoiser_callback
        c.callback(params)
      File "C:\stable-diffusion-webui\extensions\sd-webui-animatediff\scripts\animatediff_infv2v.py", line 102, in animatediff_on_cfg_denoiser
        ad_params.text_cond = ad_params.prompt_scheduler.multi_cond(cfg_params.text_cond, prompt_closed_loop)
    AttributeError: 'NoneType' object has no attribute 'multi_cond'

---
  0%|                                                                                                                                                                                                                                | 0/20 [00:21<?, ?it/s]
*** API error: POST: http://192.168.134.138:7860/sdapi/v1/txt2img {'error': 'OutOfMemoryError', 'detail': '', 'body': '', 'errors': 'CUDA out of memory. Tried to allocate 3.68 GiB. GPU 0 has a total capacty of 11.99 GiB of which 0 bytes is free. Of the allocated memory 15.93 GiB is allocated by PyTorch, and 1.15 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF'}
    Traceback (most recent call last):
      File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio\streams\memory.py", line 98, in receive
        return self.receive_nowait()
      File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio\streams\memory.py", line 93, in receive_nowait
        raise WouldBlock
    anyio.WouldBlock

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 78, in call_next
        message = await recv_stream.receive()
      File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio\streams\memory.py", line 118, in receive
        raise EndOfStream
    anyio.EndOfStream

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\stable-diffusion-webui\modules\api\api.py", line 186, in exception_handling
        return await call_next(request)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 84, in call_next
        raise app_exc
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 70, in coro
        await self.app(scope, receive_or_disconnect, send_no_error)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 108, in __call__
        response = await self.dispatch_func(request, call_next)
      File "C:\stable-diffusion-webui\modules\api\api.py", line 150, in log_and_time
        res: Response = await call_next(req)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 84, in call_next
        raise app_exc
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 70, in coro
        await self.app(scope, receive_or_disconnect, send_no_error)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\cors.py", line 84, in __call__
        await self.app(scope, receive, send)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 24, in __call__
        await responder(scope, receive, send)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 44, in __call__
        await self.app(scope, receive, self.send_with_gzip)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
        raise exc
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
        await self.app(scope, receive, sender)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
        raise e
      File "C:\stable-diffusion-webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
        await self.app(scope, receive, send)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 718, in __call__
        await route.handle(scope, receive, send)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 276, in handle
        await self.app(scope, receive, send)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 66, in app
        response = await func(request)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\fastapi\routing.py", line 237, in app
        raw_response = await run_endpoint_function(
      File "C:\stable-diffusion-webui\venv\lib\site-packages\fastapi\routing.py", line 165, in run_endpoint_function
        return await run_in_threadpool(dependant.call, **values)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\starlette\concurrency.py", line 41, in run_in_threadpool
        return await anyio.to_thread.run_sync(func, *args)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
        return await get_asynclib().run_sync_in_worker_thread(
      File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
        return await future
      File "C:\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
        result = context.run(func, *args)
      File "C:\stable-diffusion-webui\modules\api\api.py", line 477, in text2imgapi
        processed = process_images(p)
      File "C:\stable-diffusion-webui\modules\processing.py", line 785, in process_images
        res = process_images_inner(p)
      File "C:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 48, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "C:\stable-diffusion-webui\modules\processing.py", line 921, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
      File "C:\stable-diffusion-webui\modules\processing.py", line 1257, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
      File "C:\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 234, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "C:\stable-diffusion-webui\modules\sd_samplers_common.py", line 261, in launch_sampling
        return func()
      File "C:\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 234, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\modules\sd_samplers_cfg_denoiser.py", line 237, in forward
        x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict(cond_in, image_cond_in))
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
        eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
      File "C:\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
        return self.inner_model.apply_model(*args, **kwargs)
      File "C:\stable-diffusion-webui\modules\sd_hijack_utils.py", line 18, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
      File "C:\stable-diffusion-webui\modules\sd_hijack_utils.py", line 32, in __call__
        return self.__orig_func(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
        x_recon = self.model(x_noisy, t, **cond)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
        out = self.diffusion_model(x, t, context=cc)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\modules\sd_unet.py", line 91, in UNetModel_forward
        return original_forward(self, x, timesteps, context, *args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 797, in forward
        h = module(h, emb, context)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward
        x = layer(x, context)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 334, in forward
        x = block(x, context=context[i])
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 269, in forward
        return checkpoint(self._forward, (x, context), self.parameters(), self.checkpoint)
      File "C:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 121, in checkpoint
        return CheckpointFunction.apply(func, len(inputs), *args)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\function.py", line 539, in apply
        return super().apply(*args, **kwargs)  # type: ignore[misc]
      File "C:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\util.py", line 136, in forward
        output_tensors = ctx.run_function(*ctx.input_tensors)
      File "C:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 274, in _forward
        x = self.ff(self.norm3(x)) + x
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 76, in forward
        return self.net(x)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\container.py", line 215, in forward
        input = module(input)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "C:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\stable-diffusion-webui\modules\sd_hijack_utils.py", line 18, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
      File "C:\stable-diffusion-webui\modules\sd_hijack_utils.py", line 32, in __call__
        return self.__orig_func(*args, **kwargs)
      File "C:\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 56, in forward
        return x * F.gelu(gate)
    torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.68 GiB. GPU 0 has a total capacty of 11.99 GiB of which 0 bytes is free. Of the allocated memory 15.93 GiB is allocated by PyTorch, and 1.15 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

---

Additional information

GUI Attempt (which works) console logs:
Startup time: 58.6s (prepare environment: 35.1s, import torch: 5.4s, import gradio: 1.0s, setup paths: 3.1s, initialize shared: 0.1s, other imports: 0.6s, load scripts: 3.4s, create ui: 4.7s, gradio launch: 4.3s, add APIs: 0.5s).
2024-05-05 12:04:39,681 - AnimateDiff - INFO - AnimateDiff process start.
2024-05-05 12:04:39,683 - AnimateDiff - INFO - Loading motion module animatediffMotion_v14.ckpt from C:\stable-diffusion-webui\extensions\sd-webui-animatediff\model\animatediffMotion_v14.ckpt
2024-05-05 12:04:40,824 - AnimateDiff - INFO - Guessed animatediffMotion_v14.ckpt architecture: MotionModuleType.AnimateDiffV1
2024-05-05 12:04:43,692 - AnimateDiff - INFO - Hacking SD1.5 GroupNorm32 forward function.
2024-05-05 12:04:43,692 - AnimateDiff - INFO - Injecting motion module animatediffMotion_v14.ckpt into SD1.5 UNet input blocks.
2024-05-05 12:04:43,693 - AnimateDiff - INFO - Injecting motion module animatediffMotion_v14.ckpt into SD1.5 UNet output blocks.
2024-05-05 12:04:43,694 - AnimateDiff - INFO - Setting DDIM alpha.
2024-05-05 12:04:43,725 - AnimateDiff - INFO - Injection finished.
2024-05-05 12:04:43,725 - AnimateDiff - INFO - AnimateDiff + ControlNet will generate 96 frames.
Warning: field infotext in API payload not found in <modules.processing.StableDiffusionProcessingTxt2Img object at 0x0000022A481B67D0>.
0%| | 0/20 [00:00<?, ?it/s]2024-05-05 12:04:43,840 - AnimateDiff - INFO - inner model forward hooked
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [06:23<00:00, 19.20s/it]
2024-05-05 12:11:40,264 - AnimateDiff - INFO - Restoring DDIM alpha.███████████████████| 20/20 [06:04<00:00, 19.18s/it]
2024-05-05 12:11:40,264 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet input blocks.
2024-05-05 12:11:40,266 - AnimateDiff - INFO - Removing motion module from SD1.5 UNet output blocks.
2024-05-05 12:11:40,266 - AnimateDiff - INFO - Restoring SD1.5 GroupNorm32 forward function.
2024-05-05 12:11:40,266 - AnimateDiff - INFO - Removal finished.
2024-05-05 12:11:40,290 - AnimateDiff - INFO - Saving output formats: PNG
2024-05-05 12:11:40,341 - AnimateDiff - ERROR - Deforum not found. Please install: https://github.com/deforum-art/deforum-for-automatic1111-webui.git
2024-05-05 12:11:56,526 - AnimateDiff - INFO - AnimateDiff process end.
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [06:52<00:00, 20.65s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [06:52<00:00, 19.18s/it]

[zappityzap]

I haven't been able to reproduce this. For me with those settings the API uses 9.6 GB and web UI is the same.

Could you try this again with latest versions of A1111 and AD? If you still see the same problem, it would be helpful to see a screenshot of the UI settings used.

Also, did you intend to enable FILM interpolation with Interp X in the JSON? That's going to return about 960 frames.