Full Docker and BYO-ComfyUI setup: Initial impressions (👍👍👍)

[unphased]

Took much less work than I expected to spin this thing up. My approach was

• For the past 6 months I have been using a1111 and comfyui in docker via https://github.com/AbdBarho/stable-diffusion-webui-docker
• I really appreciate that you are providing docker file and scripts for the workflow. I went ahead and set it up for ComfyUI API by URL backend to my already existing comfyui setup which is set up for loading roughly 4TB of model files.
• I am under the impression that given I am able to get past configuration issues, the use of a Comfy API by URL backend will provide me the flexibility of continuing to use those backends directly without disruption as well as leverage them for rendering via stableswarm.

Couple of notes on little issues I had to address, with this somewhat custom setup, on my way to generating the first image:

• First, API access fails because the docker launch has no network access. So even to access the local comfyui backend on localhost I had to add --network=host

• suggest addition of flags to docker run --rm and --name stableswarmui for consistent docker container naming

• I found that the path of the model specified in stableswarm must match up with the path seen by comfy. This alignment is surely taken care of for you with a ComfyUI Self-Starting backend setup. I just wanted to bring up that the log output of

   15:19:30.603 [Info] User local requested 1 image(s)...
   15:19:30.618 [Warning] [BackendHandler] No backends match the request! Cannot generate anything.
   15:19:30.619 [Error] [BackendHandler] Backend request #1 failed: System.InvalidOperationException: No backends match the settings of the request given!
  

  could be a lot better, aka in particular I would make an effort to parrot the error comfyui would emit, e.g. something like

   Prompt outputs failed validation
   CheckpointLoaderSimple: 
   - Value not in list: ckpt_name: 'from-incorrect-dir-for-models/Toon/alphaLyrae_avaloriaedFP16Pruned.safetensors' not in (list of length 614)
  

  (dear lord i have 614 models ouch)
  ... As I was able to get past this error by ensuring via bind mounts that the model paths seen by stableswarm match up perfectly with what comfyui sees.

I'm not going to go too far into questions since I have a lot more exploration to do ahead of me. But one of the things I hope to answer first is whether I can get up an effective multiGPU solution for the comfyui workflow experiments. I'm talking about like some really complex workflows. They are the ones that can benefit the most from a GPU swarm. Generating a batch of 16 or 25 images with ADv3 or SVD already takes about a minute each. Then add on upscaling and other touches to that.

I was reading this page and the last part about the "colors" is intruiging. Color is a built in comfy feature to visually style a node. This appears to be a silly/fun piggyback on the output node to communicate for swarm to control dispatch of workflows across GPUs, without requiring a functional extension to comfyui, which is clever. I'll need more time to play with it but I'll go ahead and request an overview of how this is supposed to work. According to this document, if I have 5 GPUs and define 5 separate workflow branches including ksampler, each leading to an output node with a different color, what this will mean is i could within the comfy workflow interface specify 5 specific custom sets of inputs for those ksamplers (and all the rest of the nodes really) and by launching once get 5 parallel generations across 5 GPUs (or, say 3 GPUs round-robining that job, i hope, if i only have 3). That would be FREAKING AWESOME. I'd like to know how that is implemented because from where I'm standing, you would need to walk through the workflow node graph and re-generate (with e.g. a DFS) a separate workflow to dispatch to each comfyui backend for actual execution. If this is true this should basically be a full realization of the holy grail multigpu implementation for ComfyUI. The only way to push this any further would be if you had certain workflow edges that you could declare as being possible to cut over to continue from a different backend, transmitting it over the network. That would be really cool but can be hard to deal with, since network performance can vary wildly

...If that's not how it works then I assume the way it works is just to have the full workflow launched on each backend. I mean, I'd be quite happy just getting that.

Thanks for everyones hard work on this software and SAI for embracing open source.

[unphased]

A request I forgot to put in the initial post.

For use cases like mine where I brought a very full featured comfyui into stableswarm, instead of forcing me to replicate the whole set of model loading paths, for when we send the comfy workflow into the Generate tab, maybe in that use case we can just ... trust the model paths that comfy gave? I noticed it just switches out the model for the first one in the list (stableswarm's known models list) so it leads to discrepancies.

[mcmonkey4eva]

suggest addition of flags to docker run --rm and --name stableswarmui for consistent docker container naming
--rm is a flag to autodelete short/temporary containers after they're done, which is probably not the intended use for most people running swarm - persistence makes more sense. (Though of course if you want it anyway it's easy to just toss it in)
re --name sure will add

First, API access fails because the docker launch has no network access. So even to access the local comfyui backend on localhost I had to add --network=host
That makes sense. The default behavior is to use comfy inside the container, so yeah doing it separately would require that.
I added a comment in the reference launch script about it for future users.

I found that the path of the model specified in stableswarm must match up with the path seen by comfy. This alignment is surely taken care of for you with a ComfyUI Self-Starting backend setup. I just wanted to bring up that the log output of
Correct. It's again recommended that you use comfy inside the swarm container with self-start.
If you want multiple distinct containers working together, the best strategy is to use swarm containers as the backend, and a swarm frontend, and connect them with "Swarm API Backend". This automatically handles everything as it should, and in the case of models will handle the misalignment by just marking inconsistent models in gray (still works just notifies that the frontend doesn't have the same models that the backend does, which usually indicates multiple backends and only some have certain models)
If you choose to use comfy independently under api-by-url I strongly recommend you add the swarm nodeset to it, that's the most important thing self-start adds. Some of these nodes are handy feature (eg variation seed support) while others are key QoL that make way less of a mess, eg SwarmSaveImageWS is used by default in swarm whenever available instead of comfy default SaveImage or PreviewImage, as the WS option sends directly over websocket to the user (or to swarm) and is guaranteed to work, whereas save/preview both save to data drive first and have some reliability issues along side causing drive clutter / redundant readwrites / etc. Also SwarmKSampler allows changing preview mode on the fly and has more advanced previewing (eg animated previews for SVD) and etc.

I just wanted to bring up that the log output of... could be a lot better
This is true. I try to be good on clear error messages, but I've been a bit lazy on messages that can only show to very advanced users on the assumption that they're just looking at raw debug logs all the time anyway. This hasn't ended up being how it works out so yeah I need to improve that.
EDIT: errors should be clearer now here

I'd like to know how that is implemented because from where I'm standing, you would need to walk through the workflow node graph and re-generate (with e.g. a DFS) a separate workflow to dispatch to each comfyui backend for actual execution.
Yes basically that. All nodes that lead to any output of a given color get stored in an independent workflow (and any nodes that didn't lead to it ignored) and that it sent to an independent comfy backend worker. Each color gets its own sub-workflow and generation. I'd recommend only using as many colors as you have GPUs but you can do more and just have the latter colors reuse the GPUs taken by the first colors.

If this is true this should basically be a full realization of the holy grail multigpu implementation for ComfyUI. I mean it's really nice but it's not quite the holy grail. The 'holy grail' would be all GPUs used all the time automatically splitting and recombining as-needed, and never bottlenecking, ...

The only way to push this any further would be if you had certain workflow edges that you could declare as being possible to cut over to continue from a different backend, transmitting it over the network. That would be really cool but can be hard to deal with, since network performance can vary wildly that would be extremely difficult to implement in practice as most data types between nodes are not network friendly (transferring an entire model over network is naturally worse than just loading local copy, latents would require some custom encoding, there's all sorts of custom and transient data types that just can't be networked at all). So unfortunately not a planned option.

...If that's not how it works then I assume the way it works is just to have the full workflow launched on each backend. I mean, I'd be quite happy just getting that. I mean technically if you enable multi-GPU and then don't use colors and then spam the queue button it will do that, but, uh, I wouldn't recommend it. The comfy graph is extremely confusing to look at when multiple backends are all generating the same nodes at same time. It kinda just aggressively flickers and then you can only see 1 of n outputs at the end.
If you want that you should probably just use the main swarm interface as it's always multi-backend-enabled but also has an images # to generate, has image history & recent batch view, etc.

can just ... trust the model paths that comfy gave? no
Swarm doesn't just have a list of model files, it reads a lot more data than that.

Like, obviously the name/icon/description - but also, that Type: Stable Diffusion XL 1.0-Base and all is key. Know what type of model is in use changes how swarm operates. SDv1 and SDXL need different resolutions, favor different conditioning values, etc. and thus get entirely different workflows generated.
(Using Swarm's generate tab will get you better results on average than inexperienced usage of the comfy workflow tab, as swarm generates with things like custom improved rescond values for XL, or better sigma values for SVD, or etc. You can of course always just "import from generate tab" on the comfy tab to see what swarm did and replicate it)
Also can't use swarm's metadata but comfy's paths as there's no reliable way to determine "oh, comfy's my/weird/path/to/sdxl.safetensors is the same as swarm's OfficialStableDiffusion/sdxl_base_1.0.safetensors"
Should really just maintain consistent folder paths.
If you want to cheat and force it for a custom workflow specifically, you can add a CheckpointLoader and title it SwarmUI: Model and then just ignore it and use a different loader (it will assign that loader to use swarm's model selector, and your actually-in-use loader will be treated as a general node and not overwrite it unless you tell it to)

[unphased]

Thank you so much. All of these nuances would have taken an insane amount of time to learn on my own. Really appreciate it. Will take me a bit to digest and i'll be back with any follow-ups especially once i test out running this thing with 3 GPUs.

I have one question from some quick troubleshooting so far. This extension (which provides an essential macOS UX improvement) is failing to load, only within the comfyui tab in stableswarm:

[unphased]

when multiple backends are all generating the same nodes at same time. It kinda just aggressively flickers and then you can only see 1 of n outputs at the end.

I would imagine Pythongosssss' custom nodes kit (with its image feed) has a chance of having that experience be somewhat streamlined. But I hear ya

[unphased]

The 'holy grail' would be all GPUs used all the time automatically splitting and recombining as-needed, and never bottlenecking, ...

If you don't mind could you elaborate on this a little bit. splitting at the workflow level? But most workflows won't have recombining if they split. and any split would imply to me transferring the data - at the point of split - across to more GPUs.

Assuming any amount of GPU related transfers there is a potential world of difference in connectivity between them, as two backends could be running on the same machine (best case scenario) or on two machines across the world over flaky WAN (worst case)

I have a near worst case test coming up as my workstation is across the world right now and i suspect throughput would be bad if completed image download bottlenecks further generation. i hope batching would address that.

[mcmonkey4eva]

re the extension failing to load: post an issue on that extension's github repo and link them to comfyanonymous/ComfyUI#867
That issue is full of links to other extensions that had absolute paths but fixed to use relative that can serve as good examples of how to fix that common issue.

re split+recombine 'holy grail' multi-gpu: An example of a workflow that would take advantage of this would be:

Say you have an 8xA100 server node. Load model (once, then use a good direct local multiGPU impl to push to all 8 GPUs all at once. This is possible in torch already it's just not clean to do in python due to threading limits), then generate 32 images (batch 4 per GPU) (do this in one node of ksampler on batch 32, autosplit to use all 8 gpus intelligently), then pass all images to an aesthetic predictor model, and take the best 8 images of the total set (this is the recombine, you put all 32 images in one set and sort them in a group), then have a second generation pass (refiner) upscale/enhance every image (resplitting to all 8), then recombine again at the endpoint to display in a saveimage node. You would implement as just a few nodes (loadcheckpoint/condition, just the 1 first ksampler, the 1 aesth score + select 8, 1 refiner ksampler, 1 save. All the splitting handles itself). Essentially: the graph processor is a single process with raw access to all GPUs and intelligently uses them as relevant without limits.

That's the dream. That's also not possible with current tech and would need major redesigns to even have a chance. And to either not use python, or wait until the planned future python multicore support update. Maybe "Comfy 2.0: The C# Remake" will have something like that.

(That also favors local GPUs. Wide networks would suffer bad from latency doing it that way... Swarm's sending one job out per node is about the best you can do with remote servers if you don't want latency hell)