Z-image Advanced Tutorial: Controlnet, inpainting, 4K+ upscaling & more

Zimage Advanced Techniques: A Detailed Summary

Key Concepts: Zimage, ControlNet, ComfyUI, Image Upscaling, Latent Space, VAE, SeedVR2, Diffusion Models, Text Encoders, Model Patches, Inpainting, Workflow Nodes.

I. Introduction & ControlNet Integration

The video details advanced functionalities of Zimage, a recently released open-source image generator, beyond basic text-to-image creation. A primary focus is utilizing ControlNet to exert greater control over image composition and character posing. The presenter emphasizes watching a prior installation tutorial before proceeding.

• ControlNet Workflow: A “Zimage turbo fun ControlNet workflow” is available (link in description) requiring a JSON file download (saved via “save link as”). This workflow is implemented within ComfyUI.
• ComfyUI Update: Before use, ComfyUI must be updated via the “Manager” section, followed by a restart.
• Required Models: Beyond the base Zimage installation, the following models are necessary:
  • Quen 3 Text Encoder: 7.8 GB, located in ComfyUI/models/text_encoders.
  • Zimage Turbo Model: 11.4 GB, located in ComfyUI/models/diffusion_models.
  • AE Safe Tensors: 327 MB, located in ComfyUI/models/VAE.
  • ControlNet Union Safe Tensors: 2.9 GB, located in ComfyUI/models/model_patches. This is the new model required for ControlNet functionality.
• Workflow Overview (Canny Edge Detection): The initial example demonstrates controlling image composition using a reference image and Canny edge detection. The workflow involves:
  • Image upload.
  • Optional scaling using a “scaler” node (resizing the longest edge to a specified value, maintaining aspect ratio).
  • Canny edge detection to create an edge map.
  • Combining the edge map with a text prompt (e.g., “a cozy bedroom at night with Christmas lights and decorations”).
  • Hidden Zimage workflow processing (model loading, sampling).
  • Influence control via a node adjusting the edge map’s impact (default 100%, example adjusted to 8).
  • Shift and K Sampler (settings detailed in the previous tutorial).
  • Image generation.
• Performance: On a GPU with 16GB VRAM, a 9-step generation using the full Zimage Turbo model took under 10 seconds.

II. Pose and Depth Control with ControlNet

The video expands on ControlNet, demonstrating pose estimation and depth map control.

• Pose Estimation: Replacing the Canny node with an “OpenPose” detector allows controlling character pose based on a reference image.
• Custom Nodes: Utilizing ControlNet detectors may require installing the “comfyui-controlnet-auxiliary” custom node (downloaded via ComfyUI’s Custom Nodes Manager).
• OpenPose Settings: The OpenPose node allows enabling detections for hands, fingers, body, and face.
• Prompt Example: Using a reference pose with the prompt “an evil sorceress in a black robe in a dark forest” generates an image maintaining the pose. Influence set to 7 (70% pose influence).
• Depth Map Control: Replacing the OpenPose node with a “Depth Anything v2” detector extracts depth information from the reference image. This requires automatic model download from Hugging Face. Prompt example: “a cosplayer girl at a convention”.
• ControlNet Detectors: The presenter highlights edge, pose, and depth as primary ControlNet detectors, with others available (line art, anime line art, scribble).

III. Inpainting with Zimage

While a dedicated “ZimageEdit” model is forthcoming, the current Zimage Turbo model can be used for basic inpainting.

• Workflow Setup:
  • Load an image to edit.
  • Utilize a “Load Image” node.
  • Encode the image into latent space using a “VAE Encode” node.
  • Use the “Mask Editor” (accessed via right-click on a node) to draw a mask over the area to be replaced. Brush size and hardness are adjustable.
  • Employ a “Set Latent Noise Mask” node to transfer the mask to the latent image.
  • Connect the latent image to the K Sampler.
• Prompting: Provide a text prompt describing the desired replacement (e.g., “sleeping cat” to replace a diary).
• Denoise Value: Adjust the “denoise” value to control the extent of change (1.0 = complete replacement, lower values retain more of the original image).
• Limitations: The presenter acknowledges this method is “janky” and less refined than dedicated inpainting tools like Nano Banana or the upcoming ZimageEdit.

IV. Image Upscaling to 4K and Beyond

Two methods for upscaling images are presented.

• Multipass Workflow (No Additional Upscaler):
  • Duplicate the standard Zimage workflow.
  • Connect the output image from the first pass to the input of a second Zimage workflow instance.
  • Use an “Upscale Latent by” node to increase the image resolution (e.g., 2x).
  • Adjust the “denoise” value to control the level of detail retention.
  • This method generates a higher-resolution image by iteratively refining the original.
• SeedVR2 Upscaler (Recommended):
  • Download the SeedVR2 upscaler (approximately 16 GB).
  • Utilize a dedicated workflow designed for SeedVR2.
  • This method provides significantly more detail and allows for 4K+ resolution.
  • The workflow automatically downloads a VAE model (478 MB).
• Comparison: The SeedVR2 upscaler demonstrably produces sharper, more detailed images than the multipass method, particularly when zooming in.

V. Conclusion & Resources

The presenter concludes by summarizing the advanced techniques demonstrated and encouraging viewers to experiment.

• HubSpot Sponsorship: A sponsored segment highlights HubSpot’s “AI Assistant Showdown” resource (link in description), comparing ChatGPT, Claude, and Gemini.
• Newsletter: The presenter promotes a free weekly AI newsletter (link in description).
• Community Support: Viewers are encouraged to share feedback and troubleshooting questions in the comments.

Notable Quote: "AI is the hottest topic right now. Everyone is talking about it and using it, but there are just so many different AI models out there, it can get overwhelming." - Presenter, emphasizing the need for resources like the video and HubSpot's guide.

This summary aims to provide a detailed and specific overview of the video's content, preserving the technical language and providing actionable insights for Zimage users.