Sound created with a remote webrtc stream track does not seem to work

renozzz · November 17, 2019, 10:26pm

Sound does not seem to work with a MediaStream object created with a remote track (coming from a SFU).

The same MediaStream object works fine with audio and video html elements.
Sound does work with a local media stream coming from userMedia.

Update: When autoPlay property of html audio element is set to false my ubuntu system makes a pop. I can hear the same pop when I create a Sound. Therefore, the problem might be that “autoplay” property in the Sound class has no effect somehow.

Sound options: {
autoplay: true,
streaming: true,
spatialSound: true,
}

The image below shows the local (first) and remote tracks (second)

Stream Object:

Sound object:

Deltakosh · November 18, 2019, 4:51pm

This is probably because now, browsers will not let you play the sound without a user interaction. Try to run your code behind a click event (on a button for instance)

renozzz · November 19, 2019, 10:42am

Nope, that did not work. The system pops even when the stream is set to null. Just to mention again, html audio element works as expected.

renozzz · November 19, 2019, 12:17pm

I created a new project just to test all of this. It’s a “create-react-app” app which renders a canvas an an audio element.

Some code snippets:

Creating a stream:
…

  const stream = new MediaStream([track]);
  const audio = document.getElementById("audio") as HTMLAudioElement;
  audio.srcObject = stream;

  addSound(stream);

Creating a sound:

export function addSound(stream: MediaStream) {
  const sound = new Sound(`sound:${stream.id}`, stream, scene, () => {}, {
    autoplay: true,
    streaming: true,
    spatialSound: true
  });

  sound.setVolume(1);

  sounds.set(stream.id, sound);
}

Findings:

With audio.srcObject commented out there is no sound
With addSound commented out there is sound through the html audio element (autoPlay = true)
With nothing commented out, there is sound but it’s echoing because now the sound object actually works. sound.setVolume() also works and when the volume is set to 0 the echoing goes away. Also, this happens when autoPlay (audio element) is set to false - same echoing.

So, there is definitely something weird going on with the Sound class. It only works when an audio element is rendered and srcObject is set to the stream as well. The sound is echoing, feels like the stream is “cloned” multiple times.

Deltakosh · November 19, 2019, 4:57pm

Can you repro in the playground?

renozzz · November 19, 2019, 10:39pm

There is no way I can get a track created through webrtc in the playground. A local stream from user media works as expected, probably because its audio track is not produced by webrtc (not sure about this) . But then again, if there was an issue with that webrtc track how would html audio/video elements work just fine ?

I think the issue could be the integration of the webrtc track/stream with AudioContext/AudioNode interfaces.

I can create a github repo for this. If someone has a UNIX system they can test it.

renozzz · November 19, 2019, 11:48pm

By the way, the hack from https://github.com/BabylonJS/Babylon.js/pull/4531 works just fine as well.

Without the hack, there must be an audio element and srcObject set to the stream for the sound to work. If an audio element is present and autoPlay is set to false, everything seems to work as expected - no echo, sound stop/play/setVolume all work as expected.

renozzz · November 20, 2019, 3:26am

https://stackoverflow.com/questions/41784137/webrtc-doesnt-work-with-audiocontext - i think this is the exact issue here.

The second implementation from https://stackoverflow.com/questions/41784137/webrtc-doesnt-work-with-audiocontext could be a solution. Since, the first implementation (similar to Sound) did not work for the author.

Solution:

var ctx = new AudioContext();
var audio = new Audio();
audio.srcObject = stream;
var gainNode = ctx.createGain();
gainNode.gain.value = .5;   
audio.onloadedmetadata = function() {
    var source = ctx.createMediaStreamSource(audio.srcObject);
    audio.play();
    audio.muted = true;
    source.connect(gainNode);
    gainNode.connect(ctx.destination);
}

Deltakosh · November 20, 2019, 2:53pm

Congrats! Do you think we should fix it at framework level?

renozzz · November 20, 2019, 3:03pm

I think so, it’s definitely important to properly support communication through webrtc.

Deltakosh · November 20, 2019, 3:04pm

Fancy doing a PR? (that would be cool ;))

renozzz · November 20, 2019, 3:08pm

Oh boy, I am just starting with babylon and I’m not familiar with these audio interfaces. So, it’s gonna take me a while to dig into all of this

Deltakosh · November 20, 2019, 7:18pm

No rush! It is always cool to contribute

renozzz · November 30, 2019, 7:54am

Hey @Deltakosh,

Finally got some time to play around with this. A couple of issues, my ubuntu system fails to build babylon. Constant firefox crashes, multiple firefox windows created, timeouts, builds fail at different tests (some passed until the 126th and then failed). Weird random stuff. My little surface laptop completes the build though.

Is there anyway to include dependencies in the build, so somehow I could import and use them in index.js in ./localDev ? I need socket.io to communicate with a server to create a stream. I could use the dist directory, but it would be a chore to rebuild every time.

Deltakosh · December 2, 2019, 3:48pm

Just update the index.html to reference them in the localdev folder

renozzz · December 29, 2019, 3:16am

Just an update. This issue is a known bug in Chrome, Firefox works perfectly.
In the last comment in https://bugs.chromium.org/p/chromium/issues/detail?id=121673#c96 they say that every Audio Node should be connected to AudioContext.destination. I’ve tried that locally but got no success as a bunch of other people. The only fix is to create an audio element in the constructor of Sound and mute it.

                switch (this._urlType) {
                    case "MediaStream":
                        this._streaming = true;
                        this._isReadyToPlay = true;

                        this._streamingSource = Engine.audioEngine.audioContext.createMediaStreamSource(urlOrArrayBuffer);
                        const chromeBugAudio = new Audio()
                        // @ts-ignore
                        chromeBugAudio.srcObject = urlOrArrayBuffer
                        chromeBugAudio.muted = true
                        ...

I am going to use the fix in my fork and see how it goes. Don’t know if you want to use it in the source code, but if you do let me know and I’ll create a PR.

Links:
https://bugs.chromium.org/p/chromium/issues/detail?id=121673#c96
https://bugs.chromium.org/p/chromium/issues/detail?id=687574
https://stackoverflow.com/questions/54514273/webrtc-via-web-audio-api-silent-on-google-chrome

bovesan · March 18, 2020, 2:57pm

What did you construct the audio object from in the end? audio? source.mediaStream?

AlexSeoven · May 8, 2020, 9:11am

Is there a PR for that issue?

Deltakosh · May 8, 2020, 2:13pm

Not yet but all contributions are welcome

aravindh_akshay · June 17, 2020, 11:00pm

Try adding this to chrome shortcut --disable-web-security --disable-gpu --user-data-dir=~/chromeTemp

or else try a plugin to disable cors

Topic		Replies	Views
Create a sound with a webrtc media stream Questions	1	1018	April 17, 2019
Browser audio cancellation doesn't work using babylon's sound component Questions	12	1337	November 14, 2022
Steaming Audio into BabylonJS Questions audio	2	625	May 26, 2023
setDirectionalCone from a WebRTC MediaStream doesn't update the soundPanner directionalCone Bugs	3	699	April 15, 2020
Sound - Multiple Sounds with same source Questions	14	442	January 19, 2023

Sound created with a remote webrtc stream track does not seem to work

Related topics