Dears,
Quick question for you.
I have a scene where I connect an analyzer to the audio engine.
const analyser = new BABYLON.Analyser(scene)
BABYLON.Engine.audioEngine.connectToAnalyser(analyser)
Works fine for all of my sounds but obviously doesn’t connect to my audio in my new version displays using a videoTexture.
Any way to connect to the audio from the videoTexture (or to the audio context get from the stream somehow)?
Thanks in advance for your help and meanwhile, have a great sunday 
There are probably a few ways to do it depending on what you need. Here is a hacked together PG showing one approach:
(For this PG, I combined code from the BJS Analyser docs and the VideoTexture docs.)
This approach uses the captureStream of VideoElement. The key code:
let isSoundInitialized = false
demoVideoVidTex.onLoadObservable.add(() => {
demoVideoVidTex.video.addEventListener('playing', () => {
if (!isSoundInitialized) {
// Example setting up audio context directly, however
// this will bypass handling by Babylon's AudioEngine:
//const audioCtx = new AudioContext();
//const source = audioCtx.createMediaElementSource(demoVideoVidTex.video);
//source.connect(audioCtx.destination)
// Instead, use Babylon's Sound object, where the second
// parameter can also be a MediaStream
new BABYLON.Sound("music", demoVideoVidTex.video.captureStream(), scene, null, { loop: true, autoplay: true });
isSoundInitialized = true
}
})
})
Note that this approach probably does not work on Safari. (I haven’t tested it). Getting custom audio processing to work on Safari and iOS can be a complicated topic.
Thanks a lot for your reply.
This indeed looks like a method I imagined possible. I’m just new to the audio and video in BJS so I thought I would better ask here instead of making too many attempts myself, not really knowing what I’m doing 
I had a busy day today so I didn’t get the chance to try implement this in my scene.
I will check it out tomorrow morning and also check it with Safari. Will let you know how it goes (or ask for further help… hopefully not 
Meanwhile, thanks again and have a great day
Hi, Hope you are well.
Thanks again for this solution because it is (a solution 
) However, nope, it does not work with Safari (as expected 
) Though, I have another ‘LOL’ issue where you can may be enlight me.
Obviously, by doing this I have two audio tracks running simultaneously. The one from the video and the one from the new sound object. Fair enough. Where I get stuck is, if I mute the video I get no stream capture. If I setVolume of sound object to zero, I have nothing showing on the analyzer. If I attempt to set the volume of the audio to zero (even after creating the sound object)…well, since it’s a stream, obviously not a solution either Any ideas?
Edit: I was just thinking of another ‘very dirty hack’ Eventually I could set the volume of the sound object to something like 0.1 and multiply all values in the equalizer. But this thingy would start to become a bit messy. Not to mention that I will also need to handle the Safari client seperately 
Edit1: So I confirm my dirty hack works. I set the volume of the sound object to 0.01 and use a multiplier to compensate on the frequencies. Dirty but functional and not noticeable by the end-user. Still, the issue remains with Safari not recognizing this function/handling. And then, if I may, I was just wondering what would be these ‘other ways’ you mentioned in your reply? Because, I really can’t figure it myself (alas not in a ‘reasonably’ simple environment, without playing with buffers and workers)?
I’m gonna mark this as a solution, although it excludes Safari (and iOS, but my app is not for mobile anyway). I still don’t really like the need of handling the Safari client separetely. I’m also not a great fan of my dirty hack for rendering the track on the analyzer, despite that it does not impact the user experience.
May be @RaananW would kindly give me your thought about this and if you have any other idea that would improve this handling. I mean somehow we took the approach of generating a sound object from the stream and connect the analyzer to this. A more direct approach (although I at least sort of understand the complexity behind it)… but yes, would be better
Edit: Don’t know if this might help for understanding but … Here’s my quick and dirty PG simulation of dynamically changing the freq/norm on the equalizer.
On load, the rendering of the analyser is based on the reduced to 0.01 volume of the sound object generated from the video stream (while the audio on the video is set to 1).
After timeout, the new values are the standards I use in my scene for a ‘regular’ sound object with a volume of 1.
Note: forgot to mention in my post that I’m using the circular analyzer from the ribbon version of greasedLines. However doesn’t change anything to the topic… at least, I suppose so
Edit2: I know the values are not the maths ones. It’s a design tweak of mine. Because of the video rendering in the background, I’m lowering and softening the analyser displayed on top. Again, nothing related to the actual question/issue.
@mawa - I don’t completely understand everything you are trying to accomplish. Some thoughts though, simply fwiw:
Calling captureMediaStream should not start a separate audioTrack. I suspect one possibility for what was happening in your tests, is that a browser’s audioContext can easily get in to a strange state if the audioContext all resources be used in the audioContext are not cleaned up before restarting. Most BJS playground examples do not show how to clean up the audio context, and so it’s easy to get in to strange states if modifying code and clicking play over-and-over. i.e. Multiple audio streams playing simultaneously.
As you discovered, captureMediaStream is not available on Safari.
It depends on exactly what it is you are trying to do as to if you can use the existing BJSAudioEngine. I believe the BJS audio engine is provided as a convenience, primarily for users that want spatial sound. The implementation had to make some assumptions, and for normal audio (non-spatial), the audio handling is not complex. At some point, it might make sense to use your own audio engine instead. To better understand how the Babylon.js AudioEngine works, or to consider creating your own AudioEngine, I would suggest to start by looking at these resources:
A small note on your last PG. The ~4th parameter of Sound takes a readyToPlayCallback. You might consider using that instead of the arbitrary timeout. Here is your PG slightly modified to use that callback.
Sorry for the confusion. My english is not the best and my playground does not really show the entire use case. The time out was just to simulate the change on the analyzer frequencies when loading a video instead of a sound (from the BJS audio engine). So, no the media stream does not create a separate audio track on itself. I create it from your solution kindly provided.
I also do not have multiple audio or video streams. All works well on this side. It’s really just the Safari issue that causes a real trouble. For now, I’m detecting Safari and simply remove the analyzer when a video is played. I believe I will keep this at least for the time being. I know (but only little) of the web audio API. I don’t really want to invest time and efforts into a custom audio context setup at this moment (and with my current knowledge) since everything else is working fine. But thanks again for your detailed answer and for the solution. Meanwhile, have a great day