[Feature request] Flux regional conditioning support

[tristan22mc]

The new controlnets for flux are not the best quality however I would love to be able to use the controlnets regionally so that we can apply the controlnet only to the subject we are trying to control for, as the current controlnet implementations crush the backgrounds quite badly. I used to do this with SDXL before we got good controlnets. It helps to let the model still generate the backgrounds freely without influence from the controlnet. Essentially Im asking if we can implement Flux with regional conditioning support

[mikebilly]

Yes, please implement it, I cannot use controlnet without attention mask

[Kosinkadink]

Since flux controlnet is supported via vanilla comfyui as of a day ago, I should be able to make it work with all the Advanced-ControlNet features relatively easy.

I haven't messed with Flux at all yet as I have a lot on my plate, but in the next few days I should have it ready. I'll update this issue when it's pushed out.

[tristan22mc]

Amazing! Best news today!

[mikebilly]

Is this implemented yet? @Kosinkadink

[Kosinkadink]

Not yet, been working on AnimateDiff-Evolved stuff this week. I should be able to get to it sometime this coming week.

[tristan22mc]

Just checking in! Waiting patiently hah

[Kosinkadink]

In develop branch, last night I pushed up support for most ACN features for flux (scheduling, soft scaled weights, etc).

Masking will still need work, as unlike SD, Flux's tensors while sampling don't have simple x,y dimensions - the dims get combined at some point, so I'd need to dig through the code to figure out exactly how it gets combined so I can do the same to my own mask tensor.

General cond masking that ComfyUI supports does work currently, but I will try to make the ACN attn masking work before I do any merging into main.

[tristan22mc]

Ahh super interesting. So flux is down sampling or combining spatial dimensions between layers? Jealous of your knowledge, been trying to gain a deeper understanding of how things work lately. Looking forward to seeing how you tackle that!

[tristan22mc]

By the way when you mentioned that Comfyui supports cond masking atm did you mean with controlnets too? Went and tried to test and couldnt get it working

[alex-mitov]

I've tried the develop branch and it works with the InstantX controlnet Union model, but only with the alpha version of that model. The final release produces poor quality results with the tile controlnet. Is this a problem with the controlnet or is it because Flux support is still in WIP stage? Thanks for the great work so far! :)

[mikebilly]

@Kosinkadink any updates?

[Kosinkadink]

@mikebilly
In develop branch, all ACN features besides masking work for the xlabs flux models and the instantx depth model. flux is very odd with how it deals with encoding width and height into images, thought I figured it out but the results are not as expected. I'll give it another go today, but either way I will probably merge things into main branch regardless of if I can get masking working prior to initial release.

@tristan22mc yes, cond masking works for controlnets too, assuming everything's plugged in correctly. the attention-level masking is not currently supported in dev branch for flux controlnets through the Apply Advanced ControlNet nodes. I also added an Apply Advanced ControlNet(1) node that only receives one cond to simplify connecting for certain workflows.

As for the flux union model, comfy added the initial support for it in ComfyUI, so I can piggy back off of that and make it possible to modify the type soon.

[Kosinkadink]

I pushed what I have so far into main branch, i'll keep the thread posted with progress on masking.

[tristan22mc]

Awesome thanks for the update! Also Im currently working with Xinsir to help him find compute for Flux controlnets. Wanted to see if you knew of any possible resources or people in your network that would be interested in getting involved in the project. Also happy to connect you guys if it would be of any value

[tristan22mc]

Any update on this feature or did this end up being too difficult?

[Suprhimp]

I hope this function works soon :)

[Kosinkadink]

I'll revisit this feature this week. In the rework-modelpatcher branch that I will be merging as soon as my ComfyUI hook PR gets merged around Thanksgiving, I've solved an issue with soft weights/custom weights not working as intended with flux, so I'll see if that in any way helps me resolve attention masking the controlnet output onto the conditioning.