VideoTexture not rendered to in Apple Vision Pro

Questions
1

I want to play a video inside of a WebXR experience. This is working great for Meta 2/3 devices, but I am having issues with Apple Vision Pro. I get audio, but no video.

If I disable webxr (comment out lines 88-92 in the playground sample), and just play it in the browser, it works correctly in Safari and Chrome.

For the playground, run the script, click on the sphere (this should count as my user interaction to allow for autoplay), enter VR mode. The difficult part of course is if you don’t have a AVP then you can’t actually see the issue occuring.

Side note: The jwt token for nanocosmos will only be good for about two weeks, then this playground will cease functioning altogether without a new token

2

What format is the video?

3

I believe it is being sent as variant of HLS, we are getting the video from nanocosmos and running it through their video player.

I did try just plugging in a mp4 video as a source, and that worked without issue.

4

Unless this was fixed, it won’t work on safari mobile/ vision OS - html - Hls video streaming on iOS/Safari - Stack Overflow

5

Hmm, shouldn’t the native HLS support of Safari running on Apple Vision Pro allow this to work?

The hls.js referenced in the stack overflow seems to suggest that - though I haven’t actually tried using hls.js and set the src to the underlying low-level nanocosmos URL (and skip using nanocosmos provided video player).

Another quick test on my end (not using the nanoplayer), but just the built in browsers video tag:
video.src = https://bintu-splay.nanocosmos.de/h5live/authhttp/playlist.m3u8?url=rtmp://localhost/splay&stream={streamId}&jwtoken=${nanoToken};

Works in safari without webxr, but again provides audio only with webxr on AVP. Is this an issue with WebGL.

Is the “won’t work on safari mobile” pretty much the end of the line here? Will other formats that work for live streaming work, mpeg-dash?

6

Sorry, i’m being a bit too harsh on apple because I love them so much.

Apple has limited support for MediaSource, which might be used by the player you are using, and this might be the reason it is not working using the nano player.

About webxr vs desktop, if you can reproduce this on the playground I will be happy to check. I don’t have a vision pro yet, but I do have the emulator and different webxr devices. If it works outside of webxr, it should work in webxr.

7

Must… resist… adding fuel to that fire :slight_smile:

My original post has a playground link… but the nano jwt token in it has expired early (have some theories on that that are completely irrelevant to the question at hand).

But here is an updated playground with a new jwt token

If you comment out lines 88-92 it will run in browser without webXR (and works in Safari). With those lines in it runs with webxr and fails on AVP (and presumably the emulator, though I might spend some time tomorrow setting up an AVP emulator and see if that gives any new insights).

8

Had a bit of a problem with the playground, so i updated the code a bit - Nanocosmos Video Test | Babylon.js Playground (babylonjs.com)

There is no need to recreate a webxr camera, or to dispose the old one - the experience helper does everything for you. With the playground i pasted above I can see the video after clicking the sphere inside webxr. I wonder if this works on the VisionPro

BTW, i put everything on 42 and won, let me know where i collect my winnings.

Oh, and if you ever feel like ranting about the wonderful fruity company, i’m the right address! :slight_smile:

9

Still broken on AVP.

I am going to try some other video formats/protocols to see if it is just HLS that is the problem child. I’ll post back here once I have had a chance to test those. (Also going to clean up my code to remove the redundant camera setup calls).

As for your winnings, payouts are being handled by the company that shall not be named. Best of luck on redeeming your prizes.

1 Like
10

Well I have a working version using wowza streaming instead of nanocosmos. Not sure what wowza is doing differently in their player than what nano is doing.

1 Like
11

Further update. After working with nanocosmos support some minor changes to how the nanoplayer gets called and setup has resolved the issue

Changes:
Commented out the code to attach the playerDiv to the html body
//document.body.appendChild(playerDiv);

Instead attach it to the BJS rendering canvas prior to calling the nano player setup

var canvas = scene.getEngine().getRenderingCanvas();
canvas.appendChild(playerDiv);

nanoPlayer = new window.NanoPlayer(‘playerDiv’);
nanoPlayer.setup(config).then(() => connectVideoStream(scene));

12

nice solution. thanks for sharing!