I’m trying to achieve picking using a scene that has clipping planes.
I need to access the actual 3D location of the clicked point visible in the viewport, the picking will not take clipping planes into account.

so clipping planes are a GPU only operation but I totally get your issue. This is not something we support at the moment. Would you be willing to contribute it ?

Well, I have been using babylonJS for a while now, but I never took the time to look into the source code. I really do need this feature though, so I’m going to try to find a way.
If it can benefit to other, that even better.

But your reply implies that picking is not done on GPU as well. I wasn’t expecting that.
I’m very familliar with OpenGL (I have my own engine but it’s not web compatible) and I implemented picking on the GPU. Is there a WebGL limitation that forces the use of CPU ray/triangle collision to implement picking ?

not sure there was any restriction, but just the way it has been done

If you can not do it and so as I consider it more as a bug I can volunteer @PirateJC to have a look into it as he is extremely keen on contributing code more and more

I tried to use the optional parameter of scene.pick
“trianglePredicate?: BABYLON.TrianglePickingPredicate”

But the documentation is not quite clear on what that does.
My idea was to reject triangles that is at least partiarly clipped. But I can get if to work.

I’m cloning Babylon git right now, i’ll try to figure out something.

About trianglePredicate you could then depending on the 3 vertices of each triangle force reject rays which would intersects by returning false inside of it like any triangles outside the clip planes would return false in your case and true for the others (except the edge case where vertices are half in half out where the intersection points should decide of the rejection)

Got it from reading the source code. The points and ray in the predicate are not provided in world space but in mesh space, while clipping planes are in world space.

But we have no idea which object is beeing tested, so it’s not possible in the current state to perform any matrix computation to have them in the same space.

OK, I found a “hack/solution”. Fell free to use it to find a more permanent solution in BabalonJS

I overrided Ray.TransformToRef to store the matrix of the ray space and it’s inverse (four last lines)

BABYLON.Ray.TransformToRef = (ray: BABYLON.DeepImmutable<BABYLON.Ray>, matrix:
BABYLON.DeepImmutable<BABYLON.Matrix>, result: BABYLON.Ray) => {
BABYLON.Vector3.TransformCoordinatesToRef(ray.origin, matrix, result.origin);
BABYLON.Vector3.TransformNormalToRef(ray.direction, matrix, result.direction);
result.length = ray.length;
var dir = result.direction;
var len = dir.length();
if (!(len === 0 || len === 1)) {
var num = 1.0 / len;
dir.x *= num;
dir.y *= num;
dir.z *= num;
result.length *= len;
}
if (!ray["matrix"]) { ray["matrix"] = BABYLON.Matrix.Identity(); ray["inv_matrix"] = BABYLON.Matrix.Identity(); }
result["matrix"] = ray["matrix"].multiply(matrix);
result["inv_matrix"] = BABYLON.Matrix.Identity();
result["matrix"].invertToRef(result["inv_matrix"]);
}

Then I used

let pickingRay = scene.createPickingRay( X, Y, null, camera);
let picking = scene.pickWithRay(pickingRay, null, false, (p0, p1, p2, ray) => {
let m: BABYLON.Matrix = ray["inv_matrix"];
if (m) {
let wp0 = BABYLON.Vector3.TransformCoordinates(p0, m);
let wp1 = BABYLON.Vector3.TransformCoordinates(p1, m);
let wp2 = BABYLON.Vector3.TransformCoordinates(p2, m);
let not_clipped = true;
if (scene.clipPlane) {
not_clipped = not_clipped
&& scene.clipPlane.signedDistanceTo(wp0) < 0
&& scene.clipPlane.signedDistanceTo(wp1) < 0
&& scene.clipPlane.signedDistanceTo(wp2) < 0;
}
return not_clipped;
}
else
return true;
});

Note : it does not work with a simple “scene.pick”, the TransformToRef is never called, I guess the ray managment is done diffently for some unknown reason.

Note 2 : I don’t manage “on the fence” triangles. I can only collide with triangles fully visible. That’s ok in my current application, but it’s a limitation.

As far as I’m concerned, it’s ok for now.
Maybe I’ll remove this “hack” if babylon fix this bug.

I found another approach which does not require to override any methods
I use the meshPredicate to store the current mesh matrix in a local variable, the use it in the trianglePredicate.
I think it’s a bit cleaner.

let currentMeshMatrix: BABYLON.Matrix = null;
let pickingRay = scene.createPickingRay(event.offsetX, event.offsetY, null, camera);
let picking = scene.pickWithRay(pickingRay,
// MESH PREDICATE
(mesh) => {
if (mesh.isPickable) {
currentMeshMatrix = mesh.computeWorldMatrix();
}
return mesh.isPickable;
},
false,
// TRIANGLE PREDICATE
(p0, p1, p2, ray) => {
let m: BABYLON.Matrix = currentMeshMatrix;
if (m) {
let wp0 = BABYLON.Vector3.TransformCoordinates(p0, m);
let wp1 = BABYLON.Vector3.TransformCoordinates(p1, m);
let wp2 = BABYLON.Vector3.TransformCoordinates(p2, m);
let not_clipped = true;
if (scene.clipPlane) {
not_clipped = not_clipped
&& scene.clipPlane.signedDistanceTo(wp0) < 0
&& scene.clipPlane.signedDistanceTo(wp1) < 0
&& scene.clipPlane.signedDistanceTo(wp2) < 0;
}
return not_clipped;
}
else
return p0.y < currentYClipping && p1.y < currentYClipping && p2.y < currentYClipping;
});