Hi again, actually I could do with some help getting the 3d space to 2d coordinates, I haven’t been able to do a deep dive into the matrices involved.
As for my progress, I had another idea for this where I suspended a transparent plane in the scene and parented it to the camera so it was always static relative to the screen, and then used texture coordinates taken from rays that hit it to replace the pointers. I had a lot of help from this thread:
HOWEVER the next issue i faced was that i couldn’t get multiple points at the same time using this technique - it just duplicated the first coordinate. So still no pinch…
So that leaves another possible avenue to get screen coordinates, if theres a way to get multiple texture coordinates from different rays at the same time. Or maybe some sort of multi-mesh grid, though that seems a bit wasteful