Setting setHardwareScalingLevel breaks GUI button on HDP / Retina displays

Hi Raanan,

Thanks for looking into the issue.
I checked the latest PG and i still have my issue. Button does not get any input.

This has not been deployed yet and will be in the next nightly in a couple hours.

What device are you testing with?

it is deploying at the moment.

I’m using a xiaomi mi 8.

The PR is about mawa’s issue i think ?

yep, unrelated to the base issue here.
I will have to debug on my android, will keep you posted :slight_smile:

1 Like

Hey @Tom_Krikorian , does this work:

Test UI Minsar | Babylon.js Playground (babylonjs-playground.com)

The UI works well in both mobile web and AR mode now, and stays the same size. It’s an issue with the render size in AR (that we cannot really control). I might add this code to the XR implementation so that it will happen automatically, but I still need to find the right place for it :slight_smile:

2 Likes

Yes it works !

Thank you :slight_smile:

1 Like

I talked a bit too soon.
The issue is not really fixed because now the UI looks blurry (https://www.babylonjs-playground.com/#J1KG86#48) since the Engine’s setHardwareScalingLevel(1 / window.devicePixelRatio) was removed. As soon as i put it back, everything breaks again regarding input (https://www.babylonjs-playground.com/#J1KG86#49)

1 Like

OMG! I’m also still stuck with (some other and new) mobile issues. Oh, how I just hate mobile. If you’d ask Apple, they’d say you can still make it an app. If you’d ask Apple EVERYTHING would need to go through (the amazing and unique!:wink: bloody appstore.

@RaananW is the wizard for this :slight_smile:

Is that better?

Is this the expected behavior? We’ll get there :slight_smile: Test UI Minsar | Babylon.js Playground (babylonjs-playground.com)

The UI doesn’t react to hardware scaling level, you need to implement it yourself. It seems now that that the UI stays the same size and keeps the hardware scaling level correct.

Sadly input on the button does not work again.
Looks like as soon as we set hardwarescalinglevel to something different to 1 it breaks.

In the first example i linked we tried to adapt the UI when we resize the window, it’s not enough ?
https://www.babylonjs-playground.com/#J1KG86#31

with this version we don’t have issue on Retina / HDP displays for the UI.

Feels like the infernal loop. I will continue to follow. I’m really very much interested in the outcome of this one. Just too bad I haven’t yet acquired enough XP to solve the bloody topic. If only I would become better at this, my life and work with mobile would become just so much easier :wink:

1 Like

There are a few issues with every example. They vary, but the general gist is:

  1. When entering and existing AR the UI changes its scale (unexpectedly) under certain conditions.
  2. When setting scale values to the GUI, picked point is not being calculated correctly.

and a side issue:

  • When setting hardware scaling rate, the UI doesn’t react to it, only the scene does. So the UI gets smaller and needs to be rescaled manually. This should be automated IMO.

About your example:

  1. Setting the scale values change the scale of the objects and sets the container as dirty (forcing it to rescale), but
  2. (as far as I see) doesn’t force an internal resize of the texture, that will be important if you want hi res ui.

I think I am also missing the use case - we are talking about UI that works both in AR and on mobile, and should work the same, right?
AR doesn’t respect the hardware scaling rate, it has its own scaling of the XR framebuffer. What I find odd is that when getting out of XR the scaling doesn’t work as expected, and I mean to check that very soon (sadly a bit swamped at the moment). This is solved with the setTimeout I introduced in one of the PRs (probably need to wait a single frame to run the rescale).

I will debug this further. The solution we are looking for here is - highest resolution UI for both modes - mobile and AR, that looks the same and works the same. Seems reasonable :slight_smile:

I don’t know about mawa but for me i need the UI & Input to work on mobile AR.

My use case is UI that works while in AR, i don’t really care about non AR mode tbh. We hide the UI when the AR mode is not enabled.

Should i try to rescale the UI without setting hardwarescalinglevel then ? I’m getting a bit lost on the best way to support Retina / HDP displays :sweat_smile:

Thanks again for taking so much time looking into this issue :pray:

Oh, if it is AR only, setting the hardware scaling rate should not influence your scene. AR takes the entire screen area (i believe it is the image’s resolution as well), so UI is automatically “smaller”. So you need to scale it up. So what you are looking for is to stay in this resolution, but make the elements bigger. My solution with rescaling the texture will not work for you (due to problematic quality), but rescaling all elements prevents the clicks from passing to the ui elements.

Ok! I now have a proper understanding :wink: checking…

1 Like

(Jumping in to say thank you @RaananW for helping us (and probably many others!) on this issue! :pray:)

1 Like

Wanted to share the progress -

The issue with scaling the container is that it is scaling the size of the container, making the elements render out of the screen. Setting the scale of the container’s children works much better -

Test UI Minsar | Babylon.js Playground (babylonjs-playground.com)

The button seems to slide out of the screen, so I still need to understand what happens there. But the right way to go at the current state would be to scale the direct children and not the container itself.

2 Likes