Dream textures, stable diffusion wraps in babylon?

bigrig · December 23, 2022, 3:16am

Anyone understand how this works in blender?

How does he generate a texture to wrap the mesh exactly? Think we can do something similar in babylon?

carolhmj · December 23, 2022, 12:37pm

Hmmm I didn’t know this project I read the dream-textures/TEXTURE_PROJECTION.md at main · carson-katri/dream-textures (github.com) docs, and saw this video: (4) BSLIVE Blender Dream Texture Update - Depth To Image Projection - YouTube to see what’s doing, and it’s basically this:

1 - Generate the stable diffusion image from the prompt and extract depth information from the image (it mentions that there are a few depth models)
2 - Generate a UV projection of the selected geometry that matches the perspective of the generated image, based on that depth information. You can see the generated UVs here and how they try to match the perspective of the building:

That’s very much doable in Babylon, with a lot of ~mathz~ it’s an interesting idea @PirateJC might be interested as he was thinking about AI and Babylon these weeks

bigrig · December 23, 2022, 5:34pm

Hm, what i see in this video you posted is more how i’d expected it to work… like a random projection of a generated image on some geometry. But in these demos by the creator, somehow he’s getting Stable Diffusion to create an image that matches the geometry of the model. I’m assuming he’s using the model itself to prompt SD? Need to look into that code when get a moment.

But see awesome demos here and on his twitter feed:

jelster · December 23, 2022, 5:55pm

I thought this YT vid is decent explanation Dream Textures - New Blender A.I Tool For All! - YouTube

bigrig · December 23, 2022, 6:10pm

Ah, here’s the diffusers pipeline. Got to study this! Carson says:
It’s very similar to the inpainting model, except instead of concatenating the latents, mask, and masked image latents, you concatenate the latents and a depth map.
But still, i wonder why it works great in Carson’s demo, but doesn’t seem to be matching the geometry in other demos. I got to imagine it’s more than a depth map - b/c depth map wouldn’t be able to specify “Render this specific part as a Window and this other part as a Wall”, right?

github.com

carson-katri/dream-textures/blob/2f4e0d4d60b8c6881b8cd213a8cbbc7fd539bb67/generator_process/actions/depth_to_image.py#L52


      
          final_size = depth.shape[:2]
          
          
def prepare_depth(depth):
              if isinstance(depth, PIL.Image.Image):
                  depth = np.array(depth.convert("L"))
                  depth = depth.astype(np.float32) / 255.0
              depth = depth[None, None]
              depth = torch.from_numpy(depth)
              return depth
          
          
class GeneratorPipeline(diffusers.StableDiffusionInpaintPipeline):
              def prepare_depth_latents(
                  self, depth, batch_size, height, width, dtype, device, generator, do_classifier_free_guidance
              ):
                  # resize the mask to latents shape as we concatenate the mask to the latents
                  # we do that before converting to dtype to avoid breaking in case we're using cpu_offload
                  # and half precision
                  depth = torch.nn.functional.interpolate(
                      depth, size=(height // self.vae_scale_factor, width // self.vae_scale_factor)
                  )
                  depth = depth.to(device=device, dtype=dtype)

bigrig · December 26, 2022, 4:50pm

Ah some more explanation ‘It works by using the scene as an initial image, then projects generated images from several views back onto the object, assigning a separate material for each angle.’