Analyze GetUserMedia() Audio without creating feedback loop

Questions
1

I am trying to add a visual Analyser for the audio input of getUserMedia(). The problem is that the only way I can see to do it creates an audio feedback loop which is super painful :frowning:

First I create an audio stream from getUserMedia() and use it to make a new BABYLON.Sound in my scene.

var stream = navigator.getUserMedia({ video: false, audio: true }, 
        function( localMediaStream ){
          sound = new BABYLON.Sound("MicAudio", localMediaStream,
                                    scene, null, { streaming: true, autoplay: true, loop:true, volume:1 });
        },
        function(){ console.log("failed mic for visualizer"); }
);

To analyze that stream of sound I do this:

myAnalyser = new BABYLON.Analyser(scene);
BABYLON.Engine.audioEngine.connectToAnalyser(myAnalyser);
myAnalyser.FFT_SIZE = 64;

The visual analysis is working (yay), but because I had to add the getUserMedia stream sound to the scene, it is playing back what the mic takes in, creating the horrible loop.

I was hoping that I could set the volume of the sound to zero so it’s not actually audible to the microphone, but then the Analyser doesn’t pick it up either. Is there a way for the Analyser to respond to a sound’s data without playing it audibly?

This will be part of a big performance on a stage with a large speaker system, so it will be HORRIBLE for the audience if the feedback loop goes wild :slight_smile:

THANK YOU SO MUCH

I threw together a Playground for this, but strangely it’s not loading anymore with a JSON parse error:
https://www.babylonjs-playground.com/#238E6Z#2

2

FIXED:

I gave up on a Babylon based solution, but having seen getUserMedia audio visualization demos before, I went searching and found some great tutorials. I learned the basics from these two tutorials and hacked away.

One surprising thing that caught me off guard was that the browser’s native function “analyserNode.getByteFrequencyData( fftArray )” didn’t return an array like Babylon’s similarly named function but instead modified the array passed in… (which seems like an uncommon pattern for JS). But after I realized what was going on, everything was smooth sailing.

I’m still doing my visual updates as part of the scene.registerBeforeRender(callback), but the audio analysis doesn’t feed through Babylon and because of that it doesn’t need to actually play audibly.

And subsequently the people in the audience of our performance won’t be ripping our their eardrums. Yay :slight_smile:

2 Likes
3

Hi @jgmakes… Thanks for this info! I went experimenting and reading yesterday morning a bit… didn’t find anything useful.

Can you show us what you did, or perhaps reproduce it in the above playground? Probably difficult. At line 64, where that playground does BABYLON.Engine.audioEngine.connectToAnalyser(analyzer);, you probably do something like lines 6-9 HERE.

Here’s src code for BABYLON.Engine.audioEngine.connectToAnalyser()

I see this._audioContext.destination within line 364… had to find out what it is: Web Audio API

Then I tried a little goofing around with a “hijack/overload” of BABYLON.Engine.audioEngine.connectToAnalyser… in lines 3-13 here:

Early play with destinations… not much success. I got analyser turned-off and music still playing, but we need the opposite of that. :slight_smile: Destinations are audioNodes, and they have certain requirements… not easily replaced with null. :slight_smile:

Fascinating! :slight_smile:

With that thought (discovery)… are you using the browser’s native function analyserNode.getByteFrequencyData( fftArray )… to step thru the song samples… 1/2 FFT at-a-time… via the scene render-loop? Are you having to advance the buffer-pointer, manually?

I’m noob to this, obviously. Essentially, you are “sampling” the microphone… continuously, in the render loop? The only way a song gets involved, is if someone plays one into the mic-input, right? The render-loop speed… controls how-often you “gulp” a half-FFT-sized “chunk” from the mic, yes?

I learned about the half-FFT… here…

[url] “The length of the array is half the value of analyzer.fftSize. So if analyzer.fftSize = 1024 analyser.getByteFrequencyData will return an array with 512 values. Also see What does the FFT data in the Web Audio API correspond to? - Stack Overflow for more information.”

It’s all very interesting! Thanks for sharing your experiments/discoveries!

1 Like