Latest technology brings unprecedented levels of control to Stable Diffusion

ControlNet is revolutionary. With a recent paper submitted last week, the boundaries of AI image and video creation have been pushed even further: It’s now possible to make use of sketches, outlines, depth maps, or human poses to regulate diffusion models in ways in which haven’t been possible before. Here’s how that is changing the sport and bringing us closer to unlimited control of AI imagery and fully customized design:

The revolutionary thing about ControlNET is its solution to the issue of . Whereas previously there was simply no efficient strategy to tell an AI model which parts of an input image to maintain, ControlNet changes this by introducing a technique to enable Stable Diffusion models to make use of additional input conditions Reddit user IWearSkin with an apt summary:

ControlNet Examples

To display ControlNet’s capabilities a bunch of pre-trained models has been released that showcase control over image-to-image generation based on different conditions, e.g. edge detection, depth information evaluation, sketch processing, or human pose, etc.

For instance, ControlNet’s uses an edge detection algorithm to derive a Canny edge image from a given input image (“Default”), after which uses each for further diffusion-based image generation:

In the identical way, ControlNet’s showcases control over an input image via HED boundary detection:

And here’s ControlNet’s pose detection model:

ControlNet’s casually enhances sketch-based diffusion as well

ControlNET also works with the Stable Diffusion’s default masked diffusion. For instance, the Canny Edge model may be used to regulate image manipulation with manual editing:

And these presented within the original paper, which have already triggered the event of a recent generation of toolkits for creators (interestingly, ControlNet casually removed “strange hands” already).

As well as, with spatial consistency solved, recent advances in temporal consistency and AI cinema may be expected!