Jumping between scenes while in WebXR

Hi guys, Proud to be a Babyloner :slight_smile:

I Have a question about navigating between scene while being in the webXR in VR experience.

To switch scenes, i would have to run the following (pseudo) code:

oldScene.xrHelper.baseExperience.exitXRAsync();
newScene.xrHelper.baseExperience.enterXRAsync("immersive-vr", "local-floor");

The problem is that the user experiencing a jump out of the XR session (as the code instructs) back to the browser and after a short while, entering to the XR back again.
I feel that this is a bad UX specially in a scenario when a lot of scene navigation is presented.

Any other way it can be done without the jumping out back to the browser and then jumping in to the webXR ?

thanks.

2 Likes

As I think, @JCPalmer should to know how to do this correct =)

1 Like

Also,
While rendering after the navigation a new scene with the same engine and the same canvas, i get a warning:

Can’t perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in the componentWillUnmount method.

Do I need to do additional operations before starting a new scene?

cc @RaananW our XR guru

1 Like

Oh, interesting scenario!

There are a few ways to achieve it cleanly. The one I would personally choose is to switch to a different link while using the sessiongranted event which is already integrated in the default xr experience. The idea is this - every scene has its own URL. Switching between those 2 URLs will move the session to the new URL (and with it to the new scene) and the user will have a seamless experience.

If you want to move from one scene to the other in a single experience (like you are trying to do here) you will need to block the rendering while the session starts again.

Another way (which I have never tested TBH) is to implement the XR pipeline yourself using the babylon classes and move those classes from one scene to the other. It won’t be trivial, but more than possible. The WebXRSessionManager is your main entry point - the scene defined there is passed to all other modules. If you dispose all other modules apart from the Session Manager you will still have a running session with a new scene.

3 Likes

@Dok11, thanks for the ping. I have been heads down recently changing over lower level systems / mesh classes getting ready for multi-scene within the same engine instance, same as this topic. While i am a week or more behind here, but think there are a couple of things that go along with this.


First, a reminder that someone might want to do this in Babylon Native (it could happen). It may be that the desired goal here, may end up being implemented in Native by force. That is not a problem, nor is it a problem if there will be some other difference in the mechanism. Knowing that there will be a difference, Now though, would be nice to know for code organization.


Visual & Audio / transitions are obviously going to make the change over more graceful. I’ll leave the visual part up to “you can do whatever you can do on both sides”, but the audio context is another matter. Clearly having an audio buffer started prior to the change over and part of an transition animation is highly desire-able.

I use WebAudioFonts for stuff, so I could just do this outside of engine’s knowledge. But Audio Context is at the Engine level, so there should also be an option for starting a BABYLON.Sound completely independent of a scene, right?

I also notice that there is an option to pass a Web Audio Context in the Scene constructor, which is not really documented. What it is for & should that be utilized for scene switching?

Well still not actively looking at this, but a little IDE code completion “searching” shows that WebXRSessionManager has an isNative member. Probably need to do a complete read of that class.

@RaananW, your option 3 of replacing the scene right underneath an existing session seems best by a wide margin:

  • Synchronous. The code snippet by @lior3790 is 2 async calls. For production worthy code, you cannot do that. They must be nested. Beyond that there are a bunch of other things which need switched out / handled like render loop starting / stopping & canvas to scene attaching / detaching. If you wished to run on either a device or desktop, you always need to do the non-XR stuff. Inline switching seems less of a kluge to integrate.

  • I cannot imagine any other way that depends on dropping an old session & creating a new one being faster than this. That lightens the lift required for any transition process.

I don’t mind looking at a “moveXRToScene” function to switch scenes. I can create an issue for that, but sadly my backlog is, as always, rather long :slight_smile:

1 Like

I’ll try doing it outside of the framework, when I actually get to that point, rather than a direct function in SessionManager. Subclassing is my cheap way of developing for the framework without actually doing it yet, but not going work here. Will just hack any privates or just not write in Typescript.

You can say I did that. I just stole yours via sub-classing, so it is going to look a little similar. 3 classes needed changes (WebXRDefaultExperience, WebXRExperienceHelper, & WebXRSessionManager).

Default Experience Sub

/**
 * WebXRDefaultExperience Notes:
 *     - Made a seperate Create Call, so if integrated nothing need change,
 *       unless code reduction for working in functions from here.
 *     - probably should add an engine disposer observer for here
 */
export class DefExperience extends BABYLON.WebXRDefaultExperience {
    public options : BABYLON.WebXRDefaultExperienceOptions;
    public persistent = false; // only really needed, if changes end up directly in framework

    /**
     * Creates the default xr experience for multi scene
     * @param engine
     * @param options options for basic configuration
     * @returns resulting WebXRDefaultExperience
     */
    public static CreatePersistentAsync(engine : BABYLON.Engine, options: BABYLON.WebXRDefaultExperienceOptions = {}) {
        const result = new DefExperience();
        result.options = options;
        result.persistent = true;

        const throwAwayScene = new BABYLON.Scene(engine);

        // Create base experience.  Must pass a valid scene, since it does not check
        return ExpHelper.CreateAsync(throwAwayScene)
            .then((xrHelper) => {
                (<ExpHelper> xrHelper).persistent = true;
                result.baseExperience = xrHelper;

                // Clears observer of the important scene disposers (baseExperience & session manager).
                // They are littered everywhere, but in those classes they are just going to be
                // disposed themselves before the scene is, if using moveXRToScene() correctly.
                throwAwayScene.onDisposeObservable.clear();
                throwAwayScene.dispose();
                return result;
            })
            .catch((error) => {
                BABYLON.Logger.Error("Error initializing XR");
                BABYLON.Logger.Error(error);
                return result;
            });
        }

    /**
     * broken out from CreateAsync(); could possibly reduce replication by calling
     * this from there too
     */
    private _addUI(scene: BABYLON.Scene) : void {
        // init the UI right after construction
        if (!this.options.disableDefaultUI) {
            const uiOptions: BABYLON.WebXREnterExitUIOptions = {
                renderTarget: this.renderTarget,
                ...(this.options.uiOptions || {}),
            };
            if (this.options.optionalFeatures) {
                if (typeof this.options.optionalFeatures === "boolean") {
                    uiOptions.optionalFeatures = ["hit-test", "anchors", "plane-detection", "hand-tracking"];
                } else {
                    uiOptions.optionalFeatures = this.options.optionalFeatures;
                }
            }
            this.enterExitUI = new BABYLON.WebXREnterExitUI(scene, uiOptions);
        }
    }

    /**
     * broken out from CreateAsync(); could possibly reduce replication by calling
     * this from there too
     */
    private async _initialize() : Promise<void> {
        if (this.options.ignoreNativeCameraTransformation) {
            this.baseExperience.camera.compensateOnFirstFrame = false;
        }

        // Add controller support
        this.input = new BABYLON.WebXRInput(this.baseExperience.sessionManager, this.baseExperience.camera, {
            controllerOptions: {
                renderingGroupId: this.options.renderingGroupId,
            },
            ...(this.options.inputOptions || {}),
        });

        if (!this.options.disablePointerSelection) {
            // Add default pointer selection
            const pointerSelectionOptions = {
                ...this.options.pointerSelectionOptions,
                xrInput: this.input,
                renderingGroupId: this.options.renderingGroupId,
            };

            this.pointerSelection = <BABYLON.WebXRControllerPointerSelection>(
                this.baseExperience.featuresManager.enableFeature(
                    BABYLON.WebXRControllerPointerSelection.Name,
                    this.options.useStablePlugins ? "stable" : "latest",
                    <BABYLON.IWebXRControllerPointerSelectionOptions>pointerSelectionOptions
                )
            );

            if (!this.options.disableTeleportation) {
                // Add default teleportation, including rotation
                this.teleportation = <BABYLON.WebXRMotionControllerTeleportation>this.baseExperience.featuresManager.enableFeature(
                    BABYLON.WebXRMotionControllerTeleportation.Name,
                    this.options.useStablePlugins ? "stable" : "latest",
                    <BABYLON.IWebXRTeleportationOptions>{
                        floorMeshes: this.options.floorMeshes,
                        xrInput: this.input,
                        renderingGroupId: this.options.renderingGroupId,
                        ...this.options.teleportationOptions,
                    }
                );
                this.teleportation.setSelectionFeature(this.pointerSelection);
            }
        }

        if (!this.options.disableNearInteraction) {
            // Add default pointer selection
            this.nearInteraction = <BABYLON.WebXRNearInteraction>this.baseExperience.featuresManager.enableFeature(
                BABYLON.WebXRNearInteraction.Name,
                this.options.useStablePlugins ? "stable" : "latest",
                <BABYLON.IWebXRNearInteractionOptions>{
                    xrInput: this.input,
                    farInteractionFeature: this.pointerSelection,
                    renderingGroupId: this.options.renderingGroupId,
                    useUtilityLayer: true,
                    enableNearInteractionOnAllControllers: true,
                    ...this.options.nearInteractionOptions,
                }
            );
        }

        // Create the WebXR output target
        this.renderTarget = this.baseExperience.sessionManager.getWebXRRenderTarget(this.options.outputCanvasOptions);

        if (!this.options.disableDefaultUI) {
            // Create ui for entering/exiting xr
            // changed a little here to contain promise chaining if persistent
            const promise = this.enterExitUI.setHelperAsync(this.baseExperience, this.renderTarget);
            if (this.persistent) await promise;
            else return promise;
        } else {
            return Promise.resolve();
        }
    }

    /**
     * @overide
     * not calling super as purpose is to remove nuking of baseExperience.
     * Why are not these members part of dispose:
     *  - pointerSelection
     *  - teleportation
     *  - nearInteraction
     */
    public dispose() {
        if (this.baseExperience && !this.persistent) {
            this.baseExperience.dispose();
        }
        if (this.input) {
            this.input.dispose();
        }
        if (this.enterExitUI) {
            this.enterExitUI.dispose();
        }
        if (this.renderTarget) {
            this.renderTarget.dispose();
        }
    }

    public moveXRToScene(nextScene : BABYLON.Scene, hookUp : (defExperience : DefExperience) => void) : void {
        // sanity check
        if (!this.persistent) throw 'DefaultExperience must be instanced with CreatePersistentAsync() to move XR';

        // call a persistence aware dispose
        this.dispose();

        // re-add the enter
        this._addUI(nextScene);
        this._initialize();

        // cascade down
        (<ExpHelper> this.baseExperience)._moveXRToScene(nextScene);
        (<XRSM> this.baseExperience.sessionManager)._moveXRToScene(nextScene, this, hookUp);
    }
}

Experience Helper Sub

//==========================================================================
/**
 * WebXRExperienceHelper Notes:
 * `  - Constructor is passed a scene, which adds a scene.onDisposeObservable.
 *      As this is only instanced once, disposer is premanently removed in
 *      DefExperience.CreatePersistentAsync()
 *
 *    - no dispose of feature manager; bug?
 *
 *    - The constructor & a # of privates need to be changed to protected to 
 *      avoid using an altered babylon.d.ts to transpile.
 */
export class ExpHelper extends BABYLON.WebXRExperienceHelper {
    public persistent = false; // only really needed, if changes end up directly in framework

     // super constructor's scene.onDisposeObservable() nuked in DefExperience
    constructor(scene: BABYLON.Scene) {
        super(scene);

        // replace some parts with sub classes
        this.sessionManager.dispose();
        this.sessionManager = new XRSM(scene);
    }

    /**
     * broken out from enterXRAsync(), could possibly reduce replication by calling
     * this from there too
     */
    private _adjustScene() {
        // Cache pre xr scene settings
        this._originalSceneAutoClear = this._scene.autoClear;
        this._nonVRCamera = this._scene.activeCamera;
        this._attachedToElement = !!this._nonVRCamera?.inputs?.attachedToElement;
        this._nonVRCamera?.detachControl();

        this._scene.activeCamera = this.camera;
        // do not compensate when AR session is used
        if (this.sessionManager.sessionMode !== "immersive-ar") {
            this._nonXRToXRCamera();
        } else {
            // Kept here, TODO - check if needed
            this._scene.autoClear = false;
            this.camera.compensateOnFirstFrame = false;
            // reset the camera's position to the origin
            this.camera.position.set(0, 0, 0);
            this.camera.rotationQuaternion.set(0, 0, 0, 1);
        }
    }

    /**
     * broken out from enterXRAsync(), could possibly reduce replication by calling
     * this from there too
     */
    public _onSessionEnded() : void {
        // when using the back button and not the exit button (default on mobile), the session is ending but the EXITING state was not set
        if (this.state !== BABYLON.WebXRState.EXITING_XR) {
            this._setState(BABYLON.WebXRState.EXITING_XR);
        }
        // Reset camera rigs output render target to ensure sessions render target is not drawn after it ends
        this.camera.rigCameras.forEach((c) => {
            c.outputRenderTarget = null;
        });

        // Restore scene settings
        this._scene.autoClear = this._originalSceneAutoClear;
        this._scene.activeCamera = this._nonVRCamera;
        if (this._attachedToElement && this._nonVRCamera) {
            this._nonVRCamera.attachControl(!!this._nonVRCamera.inputs.noPreventDefault);
        }
        if (this.sessionManager.sessionMode !== "immersive-ar" && this.camera.compensateOnFirstFrame) {
            if ((<any>this._nonVRCamera).setPosition) {
                (<any>this._nonVRCamera).setPosition(this.camera.position);
            } else {
                this._nonVRCamera!.position.copyFrom(this.camera.position);
            }
        }

        this._setState(BABYLON.WebXRState.NOT_IN_XR);
    }

    /**
     * Intended to be called by DefExperience.moveXRToScene()
     */
    public _moveXRToScene(nextScene : BABYLON.Scene) : void {
        this.dispose();

        // the 3 things assigned / instanced by the constructor
        this._scene = nextScene;
        this.camera = new BABYLON.WebXRCamera("webxr", this._scene, this.sessionManager);
        this.featuresManager = new BABYLON.WebXRFeaturesManager(this.sessionManager);

        this._adjustScene();

        // will possibly clear too much, but no choice
        nextScene.onDisposeObservable.clear();
    }

    /**
     * @overide
     * Not calling super as purpose is to stay inside an XR session.
     * Also, added disposal of featureManager.  Seems like a bug not to have.
     */
    public dispose() {
        if (!this.persistent) {
            this.exitXRAsync();
            this.sessionManager.dispose();
        }
        this.featuresManager.dispose(); // added from stock class
        this.camera.dispose();
        this.onStateChangedObservable.clear();
        this.onInitialXRPoseSetObservable.clear();
        this._spectatorCamera?.dispose();
        if (this._nonVRCamera) {
            this._scene.activeCamera = this._nonVRCamera;
        }
    }
}

Session Manager Sub

//==========================================================================
/**
 * WebXRSessionManager Notes:
 * `  - constructor is passed a scene, but all it wants is an Engine, except for a disposer.
 *      As this is only instanced once, disposer is premanently fixed in
 *      DefExperience.CreatePersistentAsync()
 *
 *   - _moveXRToScene() is passed a scene, which is not used; toss, maybe
 */
export class XRSM extends BABYLON.WebXRSessionManager {

    /**
     * Intended to be called by DefExperience.moveXRToScene()
     */
    public _moveXRToScene(nextScene : BABYLON.Scene, defExperience : DefExperience, hookUp : (defExperience : DefExperience) => void) : void {
        //very similar to dispose, but not calling it
        this.onXRFrameObservable.clear();
        this.onXRSessionEnded.clear();
        this.onXRReferenceSpaceChanged.clear();
        this.onXRSessionInit.clear();

        // all the old sessionInit observers are now gone; time to make some new ones
        hookUp(defExperience);

        // add back a cleared observer, if current session is ever exited
        const eHelper = <ExpHelper>defExperience.baseExperience;
        this.onXRSessionEnded.addOnce(eHelper._onSessionEnded);

        // now simulate that event for the new stuff
        this.onXRSessionInit.notifyObservers(this.session);
    }
}

I have transpiled successfully, & will start next week to try to get it to run.

4 Likes

Well, this has started to turn over in the last hour. The “framework” that is going to call this & manage each of the scenes is also being written at the same time, so things are really messy.

I ended up Forking the BJS repo & adding an “XR” branch, because it was easier to set up to test & probably needed to be done at some point anyway. No more throw away scene to fix things, going in thru the front door now.

I only have one scene right now. It gets setup for XR using moveXRToScene(), though. One test yet to do is to actually get a 2nd scene, and go to it using a button in the first scene.

There is one issue so far that I have found when leaving XR. The camera is still somehow using the rig cameras. Here is a UI Test (not being shown here) / AR simulation scene. Upon leaving you get 2 renders side by side (as seen from the remote debugger). Going to have to track through that to find what I missed.

You can repeatedly go in and out of XR, no problem. Not sure I have a question in there.



edit:
Ok, have posted my changes to GitHub

Everything works, except nulling of Engine in WebXRSessionManager.dispose(). Do not know why. It is commented out ans is the least of my worries.

I made a very small 2nd scene, and go to it off a click in the first scene. It seems pretty immediate, but I do get a message in console of

Note: The XRSession has completed multiple animation frames without drawing anything to the baseLayer's framebuffer, resulting in no visible output.

I am now in the process of getting commits in the repo which calls this stuff. Will also start to remove some of the scaffolding which was required.

When I run “xrHelper baseExperience. EnterXRAsync (” immersive vr -“, “local - floor”);” “Error: Failed to execute ‘requestSession’ on’ XRSystem’: The requested session requires user activation.” But I didn’t find a way to fix it…

cc @RaananW

the error states exactly what the issue is :slight_smile:

Entering XR requires user interaction. meaning - a click (for example).