Sound created with a remote webrtc stream track does not seem to work

Sound does not seem to work with a MediaStream object created with a remote track (coming from a SFU).

  • The same MediaStream object works fine with audio and video html elements.
  • Sound does work with a local media stream coming from userMedia.

Update: When autoPlay property of html audio element is set to false my ubuntu system makes a pop. I can hear the same pop when I create a Sound. Therefore, the problem might be that “autoplay” property in the Sound class has no effect somehow.

Sound options: {
autoplay: true,
streaming: true,
spatialSound: true,
}

The image below shows the local (first) and remote tracks (second)

Stream Object:

Sound object:

This is probably because now, browsers will not let you play the sound without a user interaction. Try to run your code behind a click event (on a button for instance)

Nope, that did not work. The system pops even when the stream is set to null. Just to mention again, html audio element works as expected.

I created a new project just to test all of this. It’s a “create-react-app” app which renders a canvas an an audio element.

Some code snippets:

Creating a stream:

  const stream = new MediaStream([track]);
  const audio = document.getElementById("audio") as HTMLAudioElement;
  audio.srcObject = stream;

  addSound(stream);

Creating a sound:

export function addSound(stream: MediaStream) {
  const sound = new Sound(`sound:${stream.id}`, stream, scene, () => {}, {
    autoplay: true,
    streaming: true,
    spatialSound: true
  });

  sound.setVolume(1);

  sounds.set(stream.id, sound);
}

Findings:

  • With audio.srcObject commented out there is no sound
  • With addSound commented out there is sound through the html audio element (autoPlay = true)
  • With nothing commented out, there is sound but it’s echoing because now the sound object actually works. sound.setVolume() also works and when the volume is set to 0 the echoing goes away. Also, this happens when autoPlay (audio element) is set to false - same echoing.

So, there is definitely something weird going on with the Sound class. It only works when an audio element is rendered and srcObject is set to the stream as well. The sound is echoing, feels like the stream is “cloned” multiple times.

Can you repro in the playground?

There is no way I can get a track created through webrtc in the playground. A local stream from user media works as expected, probably because its audio track is not produced by webrtc (not sure about this) . But then again, if there was an issue with that webrtc track how would html audio/video elements work just fine ?

I think the issue could be the integration of the webrtc track/stream with AudioContext/AudioNode interfaces.

I can create a github repo for this. If someone has a UNIX system they can test it.

By the way, the hack from https://github.com/BabylonJS/Babylon.js/pull/4531 works just fine as well.

Without the hack, there must be an audio element and srcObject set to the stream for the sound to work. If an audio element is present and autoPlay is set to false, everything seems to work as expected - no echo, sound stop/play/setVolume all work as expected.

https://stackoverflow.com/questions/41784137/webrtc-doesnt-work-with-audiocontext - i think this is the exact issue here.

The second implementation from https://stackoverflow.com/questions/41784137/webrtc-doesnt-work-with-audiocontext could be a solution. Since, the first implementation (similar to Sound) did not work for the author.

Solution:

var ctx = new AudioContext();
var audio = new Audio();
audio.srcObject = stream;
var gainNode = ctx.createGain();
gainNode.gain.value = .5;   
audio.onloadedmetadata = function() {
    var source = ctx.createMediaStreamSource(audio.srcObject);
    audio.play();
    audio.muted = true;
    source.connect(gainNode);
    gainNode.connect(ctx.destination);
}

Congrats! Do you think we should fix it at framework level?

I think so, it’s definitely important to properly support communication through webrtc.

Fancy doing a PR? (that would be cool ;))

Oh boy, I am just starting with babylon and I’m not familiar with these audio interfaces. So, it’s gonna take me a while to dig into all of this :slight_smile:

1 Like

No rush! It is always cool to contribute :wink:

Hey @Deltakosh,

Finally got some time to play around with this. A couple of issues, my ubuntu system fails to build babylon. Constant firefox crashes, multiple firefox windows created, timeouts, builds fail at different tests (some passed until the 126th and then failed). Weird random stuff. My little surface laptop completes the build though.

Is there anyway to include dependencies in the build, so somehow I could import and use them in index.js in ./localDev ? I need socket.io to communicate with a server to create a stream. I could use the dist directory, but it would be a chore to rebuild every time.

Just update the index.html to reference them in the localdev folder :wink: