ControlNet

ControlNet provides structural guidance during image generation, helping maintain shapes, edges, depth, and poses from your input image. This is particularly useful for 360° content where you want to preserve specific structural elements while applying style changes.

Supported Pipelines

ControlNet is currently supported on:

Enabling ControlNet

1. Select a ControlNet-compatible pipeline (SD 1.5 or SD 2.1)
2. Navigate to the ControlNet tab in the configuration panel
3. Check Enable ControlNet
4. Select the ControlNet type for your use case
5. Adjust strength and timing settings

ControlNet Types

Canny Edge Detection

Best for: Preserving sharp edges and line art

Canny detection finds edges in your image and uses them to guide generation. Ideal for architectural scenes, product shots, and any content with defined edges.

Settings: - Low Threshold (0-255, default: 100): Lower values detect more subtle edges - High Threshold (0-255, default: 200): Higher values focus on stronger edges

Canny Presets:

Depth Map

Best for: Preserving spatial relationships and 3D structure

Uses depth information to guide generation. Objects maintain their relative positions and distances.

Settings: - Depth Boost (0.5-2.0, default: 1.5): Amplifies depth contrast for stronger depth cues

Tips: - Higher boost values (1.5-2.0) work better for scenes with subtle depth variation - Lower values (0.5-1.0) prevent over-emphasis of depth in already high-contrast scenes

OpenPose

Best for: Preserving human poses and body positions

Detects human skeletons and uses them to maintain poses during generation. Ideal for scenes with people.

Requirements: - Requires controlnet_aux package for pose detection - Works best with clearly visible human figures

Scribble/HED

Best for: Loose structural guidance from sketchy outlines

Uses soft edge detection (HED - Holistically-nested Edge Detection) to create scribble-like guidance. More forgiving than Canny for organic shapes.

Tips: - Good for natural scenes, landscapes - Allows more creative freedom than Canny

Normal Map (SD 1.5 only)

Best for: Surface detail and lighting direction

Uses normal map information to preserve surface orientations. Useful when you have normal maps from 3D renders.

Lineart (SD 1.5 only)

Best for: Clean line drawings and illustrations

Extracts clean lineart from images. Better than Canny for illustrated or cartoon-style content.

Softedge (SD 1.5 only)

Best for: Soft, organic edge guidance

Similar to scribble but with softer edges. Good for portraits and organic subjects.

Tile (SD 1.5 only)

Best for: Upscaling and detail enhancement

Specialized for tile-based processing. Helps maintain consistency across tile boundaries.

ControlNet Settings

Strength

Range: 0.0 - 2.0 (default: 0.85)

Controls how strongly ControlNet influences the generation:

Guidance Start

Range: 0.0 - 1.0 (default: 0.0)

When ControlNet starts influencing generation (as a fraction of total steps):

• 0.0: ControlNet active from the beginning
• 0.3: ControlNet kicks in after 30% of steps (allows more initial creativity)

Guidance End

Range: 0.0 - 1.0 (default: 1.0)

When ControlNet stops influencing generation:

• 1.0: ControlNet active until the end
• 0.7: ControlNet releases after 70% (allows final details to deviate)

Timing Strategies

Model Installation

ControlNet models are automatically downloaded when first used. The ControlNet tab shows installation status for each model type:

• Installed: Model ready to use
• Not Installed: Will download on first use (~1-2GB per model)

Models are stored in the controlnet-models/ directory within your models folder.

Model Sources

SD 2.1 ControlNets: - thibaud/controlnet-sd21-canny-diffusers - thibaud/controlnet-sd21-depth-diffusers - thibaud/controlnet-sd21-openpose-diffusers - thibaud/controlnet-sd21-scribble-diffusers

SD 1.5 ControlNets: - lllyasviel/control_v11p_sd15_canny - lllyasviel/control_v11f1p_sd15_depth - lllyasviel/control_v11p_sd15_openpose - lllyasviel/control_v11p_sd15_scribble - And more from the ControlNet v1.1 collection

Best Practices

For 360° Content

1. Use moderate strength (0.6-0.8) to maintain structure without visible artifacts at tile boundaries
2. Canny works best for architectural and urban environments
3. Depth works best for outdoor scenes with natural depth variation
4. Start with defaults and adjust based on results

Common Issues

Too rigid/artificial results: - Lower the strength (try 0.5-0.7) - Use early release timing (guidance_end: 0.7)

Structure not maintained: - Increase strength (try 0.9-1.2) - Lower Canny thresholds to capture more edges - Use full control timing

Visible seams at tile boundaries: - Reduce strength slightly - Ensure consistent ControlNet settings across all tiles

Example Configuration

{
  "controlnet": {
    "enabled": true,
    "type": "canny",
    "strength": 0.85,
    "guidance_start": 0.0,
    "guidance_end": 1.0,
    "preprocessor": {
      "canny": {
        "low_threshold": 100,
        "high_threshold": 200
      }
    }
  }
}

API Configuration

When using the API, ControlNet can be configured in two formats:

Nested format (preferred):

{
  "controlnet": {
    "enabled": true,
    "type": "canny",
    "strength": 0.85,
    "guidance_start": 0.0,
    "guidance_end": 1.0
  }
}

Flat format (legacy):

{
  "controlnet_enabled": true,
  "controlnet_type": "canny",
  "controlnet_strength": 0.85,
  "controlnet_guidance_start": 0.0,
  "controlnet_guidance_end": 1.0
}

See also: - Pipelines Guide - Supported pipelines and their features - Sequences Guide - Using ControlNet with image sequences - API Reference - Complete API documentation