How to draw (heat map) something on arbitrary meshes using dynamic texture on the exact point I clicked?

[babylonlearn]

Hey guys, I am having the issue where I cannot apply drawing to only the point I click on a dynamic texture for (any) mesh I imported.
– new user can only put two links? I need to write something in the comment –

problem here ------------
However, when I import another mesh: https://www.babylonjs-playground.com/#R5URDG#129, and click on the right side large white house, the heat circle appears not only on the place I clicked, but on other faces as well.

I am using dynamic textures, I think it is a normal behavior for the texture to apply to all faces, like this: https://playground.babylonjs.com/#5ZCGRM#3559. And I know there is a way to tackle specific meshes like cubes, cylinder, etc so that we can draw on exactly the face we want using faceUVs.

my question here ---------------
but generally, I wanna know if there is a way to draw something using dynamic textures for any random meshes exactly on the point I clicked.
I also wanna ask if there is a better way to draw heat map on the surface of a 3D mesh. The logic of drawing is in the javascript functions at the bottom of the code; I don’t think it has much to do with the problem I have, if any

[babylonlearn]

prologue ------------------
My code was inspired by a post here: Drawing on a Dynamic Texture over existing image .
I tried to simplify it a little bit to: https://www.babylonjs-playground.com/#T100FS#105 (here it works fine, that’s the effect I wanna have; you click on the plane and it draws heat map).

[sebavan]

Maybe you could look into decals for this with a dynamic texture: Decals | Babylon.js Documentation

[babylonlearn]

Hey @sebavan, I tried to add dynamic texture to the cat (the main mesh), but I am having the issue that I cannot draw exactly on the place I clicked.
see my code: https://playground.babylonjs.com/#1BAPRM#1098. I wanna simply draw a circle on the place I clicked.
the <pickInfo.getTextureCoordinates()> function (line 74/75) returns wrong coordinates.
===== note ======

1. <pickInfo.getTextureCoordinates().y> is always greater than 1. If I understand the concept correctly, it should be between [0, 1].
2. it might be more correct if we need to multiply <pickInfo.getTextureCoordinates().x> by <canvasSize.width> (and y coordinate), but this still cannot generate a good result.

There must be something wrong with my code (mapping between 2D canvas and 3D surface), do you have any ideas on how to fix this?

[mawa]

Do you want to draw on a 2D plane from a pick on the 3D mesh, is that it?
There’s been a recent post around this. May be that can help:

Decal on a Flat view

Hello everyone, I need to display a decal on a plane view of my model’s texture. As you can see in this playground: https://playground.babylonjs.com/#XWBES1#387 There are the model and the texture associated with the albedo above it on a plane. When I click on the model, a decal is added to it and what I need is to display the decal on my plane in the correct position based on where I clicked. I tried several things but none worked so far. Thanks for your help PS: @Deltakosh, It seems I ha…

[Evgeni_Popov]

It is the cat which doesn’t have proper texture coordinates. If you try with a sphere you will see it works once you fix these problems:

• centerX and centerY should be multiplied with the texture width/height when passed to context.arc
• you should use 1-centerY instead of centerY

Here’s a corrected PG:

Babylon.js Playground

[babylonlearn]

hey @Evgeni_Popov , thanks so much for your reply. I have more questions:

===== I haven’t read the fourth post yet (How to draw (heat map) something on arbitrary meshes using dynamic texture on the exact point I clicked? - #6 by mawa), maybe it contains answers to some of my questions below, but if you have time to answer them all, that would be really helpful ======

(1) why does the cat don’t have proper texture coordinates?
(2) how can I make the cat to have proper texture coordinates?
(3) does this mean that if a mesh doesn’t have proper texture coordinates, we cannot draw something on the surface of it on exactly where I clicked using dynamic texture?
(4) regarding my original post, I wanna know if there is a way to draw (something) only on the point I clicked on the surface of a random 3D mesh. (so decal doesn’t seem to be a good idea because I would like the texture for the main mesh (cat) to re-render every time I clicked. if I placed decal_1, is it easy to change decal_1 when I place decal_2? this is similar to heat map functionalities).
(5) my ultimate goal is to draw a heat map on a (random) 3D mesh. I probably don’t need to use dynamic texture to draw heat maps. Do you have any idea how to realize this?

[Evgeni_Popov]

@babylonlearn

(1) why does the cat don’t have proper texture coordinates?

The cat does have texture coordinates but not layed out for what you want to do to work. In your case, you need the uv partition to be somewhat “homogeneous”, meaning two points that are close on the 3D mesh should be close in the uv map, which is not always easy/possible to do.

@babylonlearn

(2) how can I make the cat to have proper texture coordinates?

You can try to use a DCC tool like Blender to modify the uv mapping. @PatrickRyan will know better than me about how to do that.

@babylonlearn

(3) does this mean that if a mesh doesn’t have proper texture coordinates, we cannot draw something on the surface of it on exactly where I clicked using dynamic texture?

Yes indeed. This method relies on mapping the dynamic texture over the mesh, so it relies on proper uv coordinates for the mesh.

@babylonlearn

(4) regarding my original post, I wanna know if there is a way to draw (something) only on the point I clicked on the surface of a random 3D mesh. (so decal doesn’t seem to be a good idea because I would like the texture for the main mesh (cat) to re-render every time I clicked. if I placed decal_1, is it easy to change decal_1 when I place decal_2? this is similar to heat map functionalities).

I’m not sure I understand the problem with decals… You can create any amount of decals and they are not related to each other: each decal is a separate mesh.

@babylonlearn

(5) my ultimate goal is to draw a heat map on a (random) 3D mesh. I probably don’t need to use dynamic texture to draw heat maps. Do you have any idea how to realize this?

So, decals are probably not a good idea as you want the texture to fully cover the mesh.

In the PG you did from Drawing on a Dynamic Texture over existing image it does work because the plane has proper uvs => I think it’s the right method, you simply need good uvs coordinates.

[PatrickRyan]

@babylonlearn, the main issue you are running into is that the asset you are using was designed to have different materials on the base of the house and the roof. This means that the UVs are laid out in a way that they expect two different textures. When you attach a single texture and a single material to both the base and the roof, your UV coordinates from different parts of each mesh will reference the same texture coordinates. This is why your heatmap color is showing up on several parts of the mesh.

If you open the glTF version of this asset and look at the UVs of the mesh, you can see that they are all overlapping.

The way to fix this issue is to move each UV island into a unique part of the UV coordinates. No faces from the model should overlap in UV space. This means you will likely have some wasted space in your texture based on how your UV islands fit together, but you need to give in on this tradeoff to make sure all of your UVs have unique texture coordinates.