Sound created with a remote webrtc stream track does not seem to work

Bugs
22

I got this working.

First there has to be an element, muted, but playing. Then,

  voice = new BABYLON.Sound("voice", mediaStream, scene, null, { spatialSound: true, streaming: true });
  voice.attachToMesh(mesh);
  
  var ctx = voice._inputAudioNode.context;
  var gainNode = voice.getSoundGain();
  voice._streamingSource.connect(voice._soundPanner);
  voice._soundPanner.connect(gainNode);
  gainNode.connect(ctx.destination);

Go figure.

Quick look at Babylon.js/sound.ts at master · BabylonJS/Babylon.js · GitHub, it seems that GainNode is just not connected…

23

This is weird… Can you repro in the playground ? so I could fix it, thx.

24

I don’t think I can, I don’t even think it’s doable. I.e. in order to reproduce it I need a way to open some WebRTC stream from some (same?) server.
In my case, I’m running OpenVidu locally, like described here:

Is there an existing playground example playing WebRTC stream I could build upon?

25

Would smthg like this work ???

navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(handleSuccess);

Basically capturing from the mike

26

Well that’s the catch, it’s not WebRTC. Remote media has to conform to totally different set of rules. One would think that same set of rules apply for streaming remote mp3/4 and WebRTC, but that’s not the case.

And the fact that I got this working does not mean that I understand these rules. But I found out that nothing plays unless there’s a html element playing the stream. In Chrome and Edge that is, Mozilla just works.
And to make that html element play the stream in the first place, user has to explicitly click on something. I’ve seen hacks like scripts sending click events, didn’t try them myself yet.

I don’t think any of the above is related to a babylon bug, seems like google messed it up.
Best we can do at the moment is document a workaround.

Unrelated to WebRTC, I may have encountered another bug related to remote streams. I was streaming that mp3, it was stereo and spatial and everything, until I entered VR. In Edge, sound suddenly became non-spatial. That was months ago though. I assumed something’s wrong with camera tracking.
While working on this, I’ve encountered similar effect, before I connected GainNode to the destination. I.e. I had PannerNode connected to the destination, like it seems to me that Sound implementation does. But once I connected panner to gain and then gain to destination, everything was fine.
So in case there’s some sound issue like that, I might have stumbled on a fix.

27

Confirmed that is the case.
Here is what I ended up doing to make it work on chrome -

const peerAudio = document.createElement(“audio”);
peerAudio.srcObject = stream;
const remoteSound = new BABYLON.Sound(“audio”, peerAudio.srcObject, scene, null, { streaming: true,
autoplay: true,
loop: true
});

1 Like
28

Before I give it a try, please confirm - did you get spatial sound?

29

Yes, It is spatial sound.