How do I determine intersection at a particular point in the view matrix

Hallo there, I have a requirement to track which mesh user is looking at and the create a heat map based on the eye gaze data. The script that is doing the eye tracking is asynchronously sending socket messages to a server that then tells me back about the (x, y) in the DOM that the user is looking at. At which point user might have moved the camera and casting a ray at the current view matrix will not tell exactly what user was looking at that produced the (x, y).

What I don’t yet understand is what exactly I need to save at the point of sending the eye web camera data to the server then use that later to determine the mesh that was looked at when server finally sends me back document (x, y) of where user is looking at.

Hi!

Hmmm, it feels to me that you’d have to somehow “intercept” the (x,y) script data before it is sent, and cast the ray at that exact moment to find out the mesh the user is looking at. And send the mesh information directly instead of the (x,y), because as you’ve said, the user might have changed the camera position in the meantime.

Thanks @carolhmj, but I don’t yet know the (x, y) at the point when the data is sent. When the data is sent it’s just things like head position etc then server sends me back the (x,y) after some black box calculations. Only at this point can I make the ray cast.

I see, hmm, then can you send the camera’s view matrix at the same time that the other data is sent to the server? Because if you have the camera’s view matrix at the same point in time than the other information, this should be enough to reconstruct it.

@carolhmj yeah I guess that was my exact question right there phrased even better, how do I use the camera view matrix and (x, y) numbers to determine intersection?

You can decompose the view matrix into a position, rotation using decomposeToTransformNode, then plug those values into a new camera and use that to calculate the ray intersection :slight_smile:

hmmm interesting, let me try that

@carolhmj I cannot figure out what exactly I am doing wrong but no matter which cordinates I pass into the function here, I always don’t get a hit so I am getting back a null.

castRayAndFindIntersection = (x, y, cameraViewMatrix) => {
    const scene = this.shelfRootStore.initStore._scene;
    const camera = scene.activeCamera.clone('dummy');
    const transformationNode = new BABYLON.TransformNode('dummy_node');
    cameraViewMatrix.decomposeToTransformNode(transformationNode);
    camera.position = transformationNode.position;
    camera.rotation = transformationNode.rotation;

    const ray = scene.createPickingRay(x, y, null, camera);

    const pickInfo = scene.pickWithRay(ray);

    if (pickInfo.hit) {
      const mesh = pickInfo.pickedMesh;
      const localIntersectionPoint = pickInfo.pickedPoint;

      const worldIntersectionPoint = BABYLON.Vector3.TransformCoordinates(
        localIntersectionPoint,
        mesh.getWorldMatrix()
      );

      transformationNode.dispose();
      camera.dispose();

      return {
        mesh,
        point: worldIntersectionPoint,
      };
    }

    transformationNode.dispose();
    camera.dispose();

    return null;
  }

Code looks alright, I wonder what’s on the view matrix? You can use the Ray Helper to show you where is the picking ray: Mesh Picking | Babylon.js Documentation, to check if it’s where you expect

The ray helper is definitely not at the right place on the screen, way away from the screen. I don’t know if this scene.useRightHandedSystem = true; has an effect on that. But when I use the activeCamera to create the ray, then the ray is correctly placed at the right position on the screen. I therefore know that the problem is not with the 2d coordinates but with the view matrix but I don’t know how to fix it.

It does change the view matrix computation, see: https://github.com/BabylonJS/Babylon.js/blob/4b508e3a5aabf2637158d67da5dba0295a014e65/packages/dev/core/src/Cameras/targetCamera.ts#L504

@carolhmj I see, but then again I am working with the same scene so this setting should apply for all cameras created within the same scene since I am see it’s getting the flag from the scene. Since the camera is a clone of the active camera, I expect that they share the same scene.

not sure to understand the issue anymore :frowning:

@DANIEL_S can you share a repro in the playground ?

@sebavan, unfortunately my configuration would take significant effort and time to replicate in a playground but thank God I managed to fix the issue by actually creating a new camera every for and using the transform node to set it’s position and rotation rather than clone the active camera. I have a feeling it had to do with the fact that cloning a camera only copies bits of it, or something but yeah fixed it that way. I am not happy with my current solution since I don’t understand it very well but I have been stuck on this for too long, so will maybe look for a better solution later. Works a bit like the code below but not exactly.

castRayAndFindIntersection = (x, y, cameraViewMatrix) => {
    const scene = this.shelfRootStore.initStore._scene;
   const transformationNode = new BABYLON.TransformNode('dummy_node');
    cameraViewMatrix.decomposeToTransformNode(transformationNode);
    const camera = new BABYLON.UniversalCamera(
      'UniversalCamera',
      transformationNode.position,
      _scene
    );

    camera.minZ = 0.1;
    
    const position = transformationNode.position;  
    camera.rotation = transformationNode.rotation;

    // I am doing the same thing for the scene camera when setting it up and think this was missing in the clone
    camera.setTarget(new BABYLON.Vector3(position.x, position.y, -0));

    const ray = scene.createPickingRay(x, y, null, camera);

    const pickInfo = scene.pickWithRay(ray);

    if (pickInfo.hit) {
      const mesh = pickInfo.pickedMesh;
      const localIntersectionPoint = pickInfo.pickedPoint;

      const worldIntersectionPoint = BABYLON.Vector3.TransformCoordinates(
        localIntersectionPoint,
        mesh.getWorldMatrix()
      );

      transformationNode.dispose();
      camera.dispose();

      return {
        mesh,
        point: worldIntersectionPoint,
      };
    }

    transformationNode.dispose();
    camera.dispose();

    return null;
  }