AudioEngineV2 upgrade questions

Questions
1

Hi,

I see that Sound.attachToMesh() is gone. I see there’s spatialPositon property in options passed to CreateSoundAsync, so I can set it to mesh.position.
Then, what happens when the mesh moves, e.g. animation is applied to the position?
And what happens if I set mesh position directly, like mesh.position = new Vector3(whatever)?

Furthermore, unlike Sound() constructor, CreateStreamingSoundAsync does not take MediaStream argument, only HTMLMediaElement.
That’s kinda blocker for WebRTC streaming: when new stream arrives, I don’t get event with HTMLMediaElement.
Is there a workaround?

Thanks!

2

You have the same thing with spatial.attach :

this.audioEngine = await BABYLON.CreateAudioEngineAsync({ listenerEnabled: true, disableDefaultUI: true});
this.audioEngine.listener.attach(this.scene.activeCamera, true);


const sound = await BABYLON.CreateSoundAsync(nameSound, pathSound, options);
sound.spatial.maxDistance = 10;
sound.spatial.position = mesh.position;
sound.spatial.rotation = mesh.rotation;
sound.spatial.attach(meshToAttach, true);
3

And what about my second question?

Looking at the source, I may have a RFE :slight_smile:

packages/dev/core/src/AudioV2/webAudio/webAudioStreamingSound.ts, _initFromMediaElement:

this._sourceNode = new MediaElementAudioSourceNode(this._sound._audioContext, { mediaElement: mediaElement });

I need something like this, but with AudioContext.createMediaStreamSource() instead.

Use case - new user joins the chatroom/virtual world. WebRTC client creates new MediaStream that needs to be attached to user’s avatar.
This would require some constructor changes, also setting up different event listeners etc, but overall, new method _initFromMediaStream is pretty much the same as this one.

4

Let me add @RaananW to the topic as he is pretty well versed in audio

5

You’re right that CreateStreamingSoundAsync only takes an HTMLMediaElement. For a MediaStream, use CreateSoundSourceAsync instead — it wraps any Web Audio AudioNode, so you feed it a MediaStreamAudioSourceNode:

const audioEngine = await BABYLON.CreateAudioEngineAsync();

// stream = your incoming WebRTC MediaStream
const node = new MediaStreamAudioSourceNode(audioEngine.audioContext, { mediaStream: stream });
const source = await BABYLON.CreateSoundSourceAsync("webrtc", node, { spatialEnabled: true });
source.spatial.attach(mesh); // spatialization should work here too

This is exactly what the built-in CreateMicrophoneSoundSourceAsync does internally — it just calls getUserMedia and hands the resulting MediaStreamAudioSourceNode to CreateSoundSourceAsync. When a new WebRTC stream arrives, build a fresh MediaStreamAudioSourceNode and create a new sound source for it. Disposing the source stops the underlying stream tracks for you.

6

Thanks, this kinda works, but not good enough to be useful.
First thing that I got was silence. Tinkering with sound options, silence was caused by spatialMaxDistance of 50. With default 100, I do have sound, but it’s not spatial. Wherever I go and turn, it sounds exactly the same.

BTW AudioContext is private, so audioEngine._audioContext

7

This could be an issue with spatial audio and WebRTC. Let me look into it. hard to debug, as I don’t have a webrtc audio stream. It does work with a microphone, so it might be a hardware limitation that we will not be able to fix.
I’ll put it on my todo list.

8

Yeah tell me about it :wink:
I do have publicly available webrtc streaming server you could use. But running the server is simple like
docker run -p 4443:4443 --rm -e FORCE_PLAIN_HTTP=false -e SERVER_SSL_ENABLED=true -e OPENVIDU_SECRET=YOUR_SECRET -e DOMAIN_OR_PUBLIC_IP=YOUR_IP openvidu/openvidu-dev:2.32.1
You’d need to include client lib though. I’m not sure if that can be set up for playground. But that’s also fairly straightforward: Hello World - OpenVidu Docs
No big deal anyway, old audio engine still works.

9

can’t promise to dive into it today, but I will.