Real time reflection in web AR, how to create it enviroment reflection from camera feed apply to the obj in Web AR
This is related to a feature called light estimation which is in progress here : Light estimation by brianzinn · Pull Request #10461 · BabylonJS/Babylon.js · GitHub
1 Like
You can read more about it here:
lighting-estimation/lighting-estimation-explainer.md at main · immersive-web/lighting-estimation (github.com)
What has been implemented is just the part where the primary light direction/intensity. The spherical harmonics coefficients are available (on my phone I have 27 coefficients), but I believe are a higher order that what is currently supported by babylonjs. Also, the reflection cube texture is available, but they all need to be abstracted to babylon equivalents such as Vector3, but also it would be nice to pass in a light, for example, to the feature and have it cast shadows and act as the primary light direction/intensity. I have been too busy to take the PR to the next step and left it at that stage - would be great if you were able to move that along. Raanan was looking at making changes there as well, but the API as-is is not ready to publish and be public (not just because it exposes XR specific types)
Also, since you are asking specifically about real-time reflection – I did some testing on the Reflection Cube. It works a bit different than the primary light and spherical harmonics. Reflection Cube has a different eventing mechanism than you may expect. Once the reflection cube texture is provided by your device (maybe not on first frame!), it will not update every XR frame (every render). It was updating around every 8-9 frames on an iphone 11, if I recall correctly - so a few times/second at least. The XR pose does update per frame and needs to be taken into account. That is part of the spec and by design since updating textures every frame will have performance implications. You can see the additional listener on reflection cube texture updates in the PR. I think it would be great if the feature was able to manage those updates transparently for an environment reflection. I do not have an ETA on when that capability will be included in that new feature though.