How can we pick meshes or tap UI elements when using the virtual joystick? Its canvas covers the whole screen and consumes tap events. I tried resizing the joystick canvas, but that screws up its position calculations.
Cc @RaananW and I’ll also take a look at this when possible 
1 Like
Hi! Apologies for the delay. There is documentation showing how to do that already 
Virtual Joysticks | Babylon.js Documentation (babylonjs.com). From it:
“Note: This will create an overlay canvas on top of the canvas for the scene. This will disable interaction with the Babylon GUI elements and scene pointer events. To avoid this the overlay joystick canvas’s z index can be modified to toggle between scene interaction and joysticks input as seen in the playground above. Another option would be to create a custom joystick using the Babylon GUI as described below.”
Hi @carolhmj,
The demo on the docs page uses a HTML button… I’m using the Babylon GUI.
Maybe I can make a custom joystick from DOM elements, instead of the canvas-based one.
Then in that case the picking should work as expected. Can you share your implementation so we can see better what’s going on?
Sorry, I don’t mean picking with rays… I meant tapping on Babylon UI elements.
There’s a menu button in the corner of the screen, but it’s not clickable when the virtual joystick is shown. My solution was to change it to an HTML button, with a higher z-order. When the menu button is clicked, I hide the joystick canvas. It works, but it would be nice to resize the joystick canvas so it doesn’t have to take up the whole screen.
Oooh, I see what you mean now 
yeah, currently there is no way to resize the JoystickCanvas.
@alekop, we are open to PR if you want to add the support for this ?
Yeah, I should… I’ll put it on my board.
2 Likes
To follow up on an old thread…
What if many meshes in a scene are interactible, i.e.
mesh.actionManager = new ActionManager(scene);
mesh.actionManager.registerAction(new ExecuteCodeAction(
ActionManager.OnPickTrigger,
myFunctionOnPickTrigger
));
AND I would like to use the virtual joystick as a camera input (as one of several camera inputs, in fact, … but that is way off-topic so we downplay that here).
Explicit toggling between canvases that receive the events isn’t an option here because I would like “touching and dragging” to move the camera but “tapping” (without dragging) on thaty mesh call myFunctionOnPickTrigger. Nothing special, just the default behavior when standard mouse and touch camera inputs (and no virtual joysticks input) are used.
Has anyone ever tried (succeeded?) implementing such a thing?
“Forwarding” all or certain subset of events from the joysticks’ canvas to the scene canvas is one idea that comes to mind, but implementing that would’ve been very tricky, I imagine.
If anyone thought about this or tried implementing, please share any ideas and experience 
Thanks!
seva
Maybe your case would be better suited to a custom joystick? Something like Virtual Joysticks | Babylon.js Documentation
@carolhmj, I appreciate that you cared to respond.
To answer your question, no. The amount of work the so called “custom joystick” takes to implement is prohibitively high for for my case. Also, the “custom joysticks” example that you refer to helps convincing that it’s probably not a good idea: it crashes upon page load and doesn’t even start to work in Chrome (Linux desktop with and without a touchscreen and Android phone) and is unusable (although doesn’t crash outright) in Firefox desktop.
Adding @PolygonalSun to check the input issue
Which Android version and model are you using? I checked Chrome and FF on mine and it ran normally, even if the controls were on top of each other. But crashing is not good so I’d like to investigate that better if possible.
1 Like
I connected Chrome devtools to my phone’s browser just now in order to check for errors in the console, etc, opened the page ( https://playground.babylonjs.com/#C6V6UY#5 ) and the page rendered. Not that the joysticks are usable or anything, but I think it’s now working as intended.
On the desktop it also rendered completely and works as probably intended.
I do see “Failed to create WebGPU Context Provide” errors in the console in both Android and Linux desktop (and I remember seeing that when opening this same URL before on Linux desktop, too), but it functions now.
Yesterday and a few times I tried that before, same URL wasn’t rendering but I didn’t save the console output. If I see that again (though I am unlikely to try again), I will update this thread.
For reference, the phone I tried this on previously as well as today is a Google Fi’s Moto G Stylus 2021 (or 2020?..) model. Android 11 and up-to-date Google Chrome of the stable channel.
Thanks
+1
Any luck here?
Are you having issues with the Virtual Joystick?