Screen x,y coordinates to world coordinates

Questions
1

Hi,

I’m using tracking.js to get the faces coordinates from an image.

coords = {
  x : 38.12355469963794,
  y : 96.47419166998122,
  width : 106.33406420946123,
  height : 102.4978113412857
}

I’m an absolute 3D and Babylon beginner, so far I managed to project the image texture on a plane with the right aspect ratio. But when it comes to draw a box over a face with the given coords, I just can’t position get the right position.

Here is my attempt : https://www.babylonjs-playground.com/#0CSKL1 (the black box is suppose to be over the guy with the glasses)

Tried many configurations :

scene.createPickingRay(
   coords.x,
   coords.y,
   BABYLON.Matrix.Identity(),
   camera
).direction;

BABYLON.Vector3.Unproject(
  new BABYLON.Vector3(coords.x, coords.y, 0),
  engine.getRenderWidth(),
  engine.getRenderHeight(),
  BABYLON.Matrix.Identity(),
  scene.getViewMatrix(),
  scene.getProjectionMatrix()
)

faceBox.setPositionWithLocalVector(...)

faceBox.position = ...

Thanks.

2

Hello and welcome!

It looks like what you need is to transform the picture’s local coordinates to the world coordinates, like this:

https://www.babylonjs-playground.com/#0CSKL1#3

In the code on your playground, you create a picking ray, but you don’t actually use it to pick anything with! (scene.pickWithRay). (You can also pick with scene.pick or create a new BABYLON.Ray(...) and use pickWithRay).
A ray has an origin, a direction, and optionally a length. Those are used by BJS to cast the ray into the scene and determine what gets hit, so setting the position of the box to the direction of the ray doesn’t achieve much of anything… BUT if you get a distance from a picking function, you can do mesh.position = ray.origin.add(ray.direction.scale(distance))

3 Likes
4

Thanks a lot for the explanation and the clear code.

I guess, still have a few more weeks to master Babylon, but this is encouraging, so thanks again.

2 Likes