Cardboard / Merge VR + Gaze Control + Babylon GUI

Some variants of the current google cardboard designs as well as the merge vr goggles use simple touch interactions (without any additional controller). How would it be possible to use the babylon gui with these kind of goggles, concering following issues:

  • gaze interaction: a press on a touch-button (seperate from the gaze target) needs to trigger a babylon gui element. pickWithRay is not an option, as this doesnt work with gui elements (as far is i know).

  • the non stereo buttons, needed for the touch interaction, probably prevent the use of the webvr helper (as the webvr button only renders the canvas element fullscreen and therefore “removes” the html buttons). Also touch interactions dont seem to work on some phones during webvr-mode.

The webvr helper probably holds the solution to this problem, as the gazecontrollers somehow simulates pointer events.

Also see:
Simple Scene which shows the setup: the blue buttons represent buttons, which could be used for google cardboard / merge vr, the fullscreen button calls a custom fullscreen function, which uses the document as fullscreen target (to keep the single rendered html buttons)
Scenario: the gaze target is over a babylon gui element -> the user presses one of the blue buttons ->
a gui event should be triggered (emulating a regular touch / click on the gui element)

Similar questions:

Adding @RaananW as I’m on my phone.
The gaze controller shoukd be easy to wire with babylon gui

Thanks for the fast reply, but like I mentioned in the initial post I probably can’t use the webvr helper (due to the needed html buttons and the missing of touch events in vr-mode) and therefore I don’t have access to the gaze controller.

I’m currently trying to to create my own emulated click event (fired by one of the buttons, and with the location in the middle of the half-canvas), but I’m not sure if that is even possible for the canvas object - no luck so far.

Any hint to a simpler solution would be great, at his is a somewhat complex hack for such a (seemingly) simple requirement.

Yeah sorry, what I meant was perhaps to make the gaze controller available outside of the webvr helper :slight_smile:

Anyway it is really easy to simulate click by calling scene.simulatePointerDown for instance

Thank you so much @Deltakosh, I think im close now with your help.
Could you take a look at this playground: Babylon.js Playground

It seems as if the gui events arent fired (neither for 2d nor 3d buttons).
Also the regular 2d gui hover / hittest area seems off in stereo view, or are only 3d gui elements working properly in this view mode?

Thanks again for your time.


I ran some tests and tried to extract a simple version of the VR gazer, and came up with this:

Notice that mouse events no longer works, just sensors. the buttons can be pressed using the f button, hover works, and all other goodies of the UI work as well.

This is not an optimal solution, and should be cleaned a bit, but this is a wonderful starting point!

Hop this helps :slight_smile:

1 Like

Wow, that looks really promising. I will look into your solution in detail,
Thanks a lot, this i huge help!