Spatial Sound Sourced from A Video Texture File

Questions
1

I did not see an example of this, is there a way to redirect the audio output from a video texture to a spatial sound object instead of it just playing omnipresent?

I believe audio textures are handled at DOM natively in not in engine, so was thinking probably not, but was not 100% sure.

I know I could fake it with a second audio only file, but I rather I did not have to given the videos will play with audio at some point when full screened, all the pieces mesh+sound file for spatial sound are already present when a video texture is applied to a mesh so seems like a logical thing to do, and doing so seems like a recipe for unsynchronized video+audio.

2

This is a great question, let us summon the brilliant @sebavan and see what he has to say on the subject :slight_smile:

3

As you noted, the video texture is basically handing over everything to the dom sounds wise so if you find a way to link it to WebAudio, we could do it in Babylon too :slight_smile: Let me know your findings.

4 Likes
4

Hi @Calsa just checking in, were you able to make progress on this issue? Anything more we can help with? :smiley:

5

It’s been put on my todo list for this project, but realistically 1-3 months before I can get to it. I will share it back when done unless someone beats me to a solution.

1 Like
6

Here is intermediate example - https://playground.babylonjs.com/#CHQ4T#343
Audio source from video is connected to lowpass audio filter (check slider).
Now it is just a matter of some time to find the right syntax how to bind the audio to a mesh.

2 Likes
7

Oh then we should simply allow Sound to be created from mediaElement as well :slight_smile:

Let me check if that works.

8

Support will be in the next nightly

4 Likes
9

Hi @sebavan , curious about how I can just use the Sound element to control a Video’s sound. Right now, I’m passing in a video mediaElement into a VideoTexture, and the same mediaElement to the Sound class in hopes of controlling it fully from the Sound. I can’t avoid passing the mediaElement into the videoTexture, because that’s where the video is. But then when I set the mediaElement to unmuted, it will play from both the VideoTexture and the Sound element at once. Any ideas no how I could just control a video’s sound from an instance of the Sound class?

1 Like
10

Here are some hints from the commit mentioned - Nightly by sebavan · Pull Request #11601 · BabylonJS/Babylon.js · GitHub

1 Like
11

That shows me how to pass the video element as a source for the Sound class, but it’s not clear to me how to manipulate the sound without affecting the sound from the same media element being passed to the videotexture. @labris , thanks for the help !

1 Like
12

@sebavan Could you check this example with video as sound source - https://playground.babylonjs.com/#PCY1J#198 ?

image
image1103×677 10.9 KB

When you press Unmute button music starts to play, as it is seen in the console

image
image634×543 4.79 KB

The problem is that there is no sound :slight_smile:

13

The entire point was to have the mediaElement (video) as an input of the sound class to allow for instance spatial audio from a video stream:

It does not seem to play from both in the playground above ? at least not on Chromium based browsers

1 Like
14

Thanks @sebavan ! Will take this for a try today and report back.

2 Likes
15

So it is impossible to use .mp4 for the Sound?
(I believe it may be related to codecs)

16

not that I am aware of :frowning:

17

I will formulate my question in other way:
Is it possible to hear the sound from this example - https://playground.babylonjs.com/#PCY1J#198 ?

18

So with a simple audio element it is working : https://playground.babylonjs.com/#PCY1J#203

In theory it should then also work with the sound class… let me check

1 Like
19

I pushed a fix for it. should be in tomorrow s nightly

2 Likes
20

This works beautifully @sebavan . For anyone interested, I’ve tweaked Seb’s playground to set spatialAudio on the video’s audio and give a universal camera for an easy way to test that aspect working out: Sounds not ready to play | Babylon.js Playground (babylonjs.com)

3 Likes