How to convert a single mesh into an image

[nuguru]

I have a scene where I am importing a .glb file using ImportMesh. When user clicks on a mesh I am highlighting it with the ActionManager.OnPickTrigger event.

The task I am having trouble with is, I need an image or an image string that represents the highlighted mesh. I tried using ‘CreateScreenshotUsingRenderTargetAsync’ as follows

              const screenshotAsDataUrlPromised = new Promise((resolve: (value: string) => void) => {
              scene.onAfterRenderObservable.addOnce(async () => {
                const screenshot = await Tools.CreateScreenshotUsingRenderTargetAsync(
                  scene.getEngine(),
                  scene.activeCamera,
                  {
                    width: 200,
                    height: 150,
                  },
                  "image/png",
                  8,
                  false
                )
                resolve(screenshot)
              })
            })
            let imageString = await screenshotAsDataUrlPromised;

this gives me the screenshot of the entire canvas. Is there a way to only capture the mesh of interest? If not is there a way to crop the screenshot of the entire scene based on click point?

I appreciate any suggestions
Thanks

[sharp]

Hi

you can try to

• Hide the other meshes
• Take your screenshot
• Unhide the other meshes

[nuguru]

how do I hide/unhide meshes? Is there a visibility property for the meshes?

[sharp]

Mesh.isVisible

Scene.meshes

[nuguru]

Good suggestion. It works and after looking at the result I realized that its not quite what I am looking for. To elaborate
this is the scene

if I select left thigh

using your suggestion when I tried taking a snapshot of just the mesh that was selected this is what I get

It is the mesh that was selected but its not very intuitive. how can I get something like this

[nuguru]

Based on the click point, is there a way I can create a cropped image of set width and height from the scene ?

[bghgary]

If you create a second camera that focuses on the red section, you should be able to capture a screenshot using that camera.

[nuguru]

I am new to Babylonjs. Can I get an example of using multiple cameras? Thank you

[bghgary]

I don’t see an example that fits this exactly, but you basically need to create another camera using the normal way, as described in documentation, but don’t attach controls, and I think also add the camera to scene.activeCameras.

If you can create a playground with what you have so far, it will be easier to help.

[nuguru]

I have created a second camera
var targetCamera = new ArcRotateCamera(“TargetCamera”, alpha, beta, radius, target, scene);
using same alpha, beta and radius as the first camera. target for the first camera is set to new Vector3(0, 0, 0);
where as for the second camera I did
targetCamera.setTarget(new Vector3(clickPoint.pointerX, clickPoint.pointerY, 0));
is this the correct way of getting the second camera to focus on the red section?
after that I am saying
scene.activeCamera = targetCamera;
const screenshotAsDataUrlPromised = new Promise((resolve: (value: string) => void) => {
scene.onAfterRenderObservable.addOnce(async () => {
const screenshot = await Tools.CreateScreenshotUsingRenderTargetAsync(
scene.getEngine(),
scene.activeCamera,
{
width: 200,
height: 150,
},
“image/png”,
8,
false
)
resolve(screenshot)
})
})

I must be doing something wrong as I am not seeing the correct area of interest. I tried setting viewport for the second camera but that still didn’t do what I want.

I appreciate any guidance in this.
Thanks

[Takemura]

How do you get clickPoint? Do you use Action on pickable mesh, to get pickingInfo.pickedPoint? That would be a way or you can get pickedPoint like this:

var pickResult = scene.pick(scene.pointerX, scene.pointerY);

Then use pickResult.pickedPoint as target.

Or you might want to check other possibilities like pickedMesh.position as target:

PickingInfo | Babylon.js Documentation

[nuguru]

I didn’t use pickedPoint. I did this to get click point
SceneLoader.ImportMesh(“”, ${process.env.PUBLIC_URL}/models/, “MaleScene.glb”, scene, function (theMeshes) {
for (var i = 0, max = theMeshes.length; i < max; i++) {
theMeshes[i].actionManager!.registerAction(new ExecuteCodeAction(ActionManager.OnPickTrigger,
function (evt: any) {
clickPoint = evt;
}))
}}

[Takemura]

Does it work to set target like this?

var pickResult = scene.pick(evt.pointerX, evt.pointerY);
targetCamera.setTarget(pickResult.pickedPoint)

Your evt is an ActionEvent:

ActionEvent | Babylon.js Documentation

Here I made a PG, that camera looks at pickedPoint:

Babylon.js Playground

[nuguru]

I tried using pickResult as follows

const alpha = Math.PI / 1;
const beta = Math.PI / 3;
const radius = 2;
const target = new Vector3(0, 0, 0);

var targetCamera = new ArcRotateCamera(“TargetCamera”, alpha, beta, radius, target, scene);
var pickResult = scene.pick(clickPoint.pointerX, clickPoint.pointerY);
targetCamera.setTarget(pickResult.pickedPoint)
scene.activeCamera = targetCamera;
const screenshotAsDataUrlPromised = new Promise((resolve: (value: string) => void) => {
scene.onAfterRenderObservable.addOnce(async () => {
const screenshot = await Tools.CreateScreenshotUsingRenderTargetAsync(
scene.getEngine(),
scene.activeCamera,
{
width: 200,
height: 150,
},
“image/png”,
8,
false
)
resolve(screenshot)
})
})

click point was
clickpoint ActionEvent {source: Mesh, pointerX: 741.3333129882812, pointerY: 392.3333435058594, meshUnderPointer: Mesh, sourceEvent: PointerEvent, …}

pickedPoint was
Vector3 {_isDirty: true, _x: -0.005583208781954507, _y: 0.46242410555715685, _z: -0.10015882496113007}

and the snapshot captured was

which is not bad but it is still taking snapshot of the entire body. Is there a way I can get a zoomed in view only the surrounding regions of the selected mesh?

Thanks for all the input

[Takemura]

You can reduce arc camera radius:

targetCamera.radius = 0.4

[Blake]

Can also try using zoomOnMesh

Babylon.js Playground

[nuguru]

I tried zoomOnMesh as shown in the example but its not quite zooming properly. Here is a simple example that I modified to include zoom on scene pointer down. I am seeing a similar behavior in my project

Babylon.js Playground

[Blake]

Looks like the issue is that minZ is too high so your mesh is getting clipped. Here I used the same formula to set min and max z as CreateDefaultCamera() uses and it seems to be working well.

Babylon.js Playground

[nuguru]

Thanks for that example. For me it didn’t work right away so I tried playing with the minZ and maxZ values.
When I switched to use ‘scene.createDefaultCameraOrLight(true, true, true);’ instead of ArcRotateCamera the values you provided for me. So I am able to zoom on mouse down which helps me take closer snapshots of target meshes.

Few questions:

1. Is there a way to reset the zoom back to original?
2. Is there a way to rotate the imported mesh? This is how it now looks on load
   
   
   I need it facing forward

Thanks

[Takemura]

1.
If it is ArcRotateCamera, maybe saving camera.radius, -.alpha, -.beta and -.position as default and set default on reset would be a solution?

There also seems to be usable functions, like:

Camera | Babylon.js Documentation

Camera | Babylon.js Documentation

2.
If you dont want to use Quaternion for rotation, this is an example of how to rotate by 90 degress:

newMeshes[0].rotation.y += Math.PI/2;

Alternatively, you can rotate Quaternion like this.

newMeshes[0].rotate(BABYLON.Axis.Y, Math.PI/2, BABYLON.Space.WORLD);

If you have animations running on the mesh, then you will have to stop animations before, i.e.:

scene.stopAllAnimations()