You have to use the indices of the mesh:
var indices = mesh.getIndices();
var positions = mesh.getVerticesData(BABYLON.VertexBuffer.PositionKind);
var triangles = [];
for (var i = 0; i < indices.length; i+=3) {
var index1 = indices[i];
var index2 = indices[i+1];
var index3 = indices[i+2];
var triangle = [
new BABYLON.Vector3(positions[index1*3],positions[index1*3+1],positions[index1*3+2]),
new BABYLON.Vector3(positions[index2*3],positions[index2*3+1],positions[index2*3+2]),
new BABYLON.Vector3(positions[index3*3],positions[index3*3+1],positions[index3*3+2])
]
triangles.push(triangle);
}
And if your mesh has in any way changed position away from 0,0,0 or rotated, scaled etc. you’ll have to bake the positions of the vertices as well:
mesh.bakeCurrentTransformIntoVertices();
Now for the rest, I would strongly suggest that you use a convex mesh vs. triangle mesh, as it yields both better performance and results(at least if you plan on using it for collision detection). I’m sure that whatever you want to achieve, there must be a better approach.
If you really want to do triangle vs. triangle intersection, you should use some kind of spatial partitioning as mentioned already. Then use the combined overlaps of the bounding boxes as an AABB to traverse the two bounding volumes, the resulting two lists of triangles should then be checked against each other using triangle-triangle intersection tests.
Here’s the BVH I use for static meshes: BVH · GitHub
If your models move around,a dynamic BVH with reinsertion or tree-rotation won’t really cut it, so you’ll have to recreate the BVH every time your models move/rotate. I have done this for simple models without big performance drops