Sound element not playing after window blur on iPadOS

Bugs
1

This is quite specific, but basically the sound element won’t play a second time.

Playground

  • iPad Pro 12.9" 4th gen, iPadOS 17.5.1

Steps to replicate:

  • Tap on scene to play audio
  • Open a new tab with anything on it (audio should still be playing).
  • Go back into the playground and tap to pause audio.
  • Tap again to play audio (this should not work, contrary to repeating the steps without opening a new tab).

Devices with Android and iOS seem to work.

I tested on another iPad and the issue did not appear (unfortunately I don’t have access to more):

  • iPad 6th gen, iPadOS 16

I found that this issue only happens when the streaming option is turned on.
Playground

2

cc @docEdub

3

Thanks for reporting this. I can reproduce the issue on my iPhone with that playground, so I’m investigating now.

4

Also repros on desktop Safari.

5

I’m not finding a way to fix this, yet, and I’m not finding a workaround. Safari is broken in this case. It tells me everything is fine and the sound is playing when it’s clearly not playing. AFAICT there is no workaround other than just not setting streaming: true.

Even when a streaming sound does pause/resume correctly in Safari, I hear large audio glitches and buffering issues (the current stream buffer plays back twice in rapid succession when pause/resume is done), so it’s best not to use the streaming: true option anyway if you’re going to be pausing and resuming the sound.

1 Like
6

If Safari ever starts supporting HTMLMediaElement: captureStream(), we can use it as a workaround for issues like this. Until then I don’t think we can get the streaming: true option to be robust for pause/resume.

7

Thanks for the fast response, I won’t be setting streaming on then. Would I have any other options for audio optimization then?

8

So I’m also drawing an audio progress indicator and I had to use sound._htmlAudioElement to access the duration of the sound and mark it as @ts-expect-error. On the other hand If I don’t set streaming I can use sound.getAudioBuffer().

Playground

1 Like
9

How big are the audio files you’re working with? Is memory use what you want to optimize? Time to start? Both? There may not be much we can do now for those factors, but the new audio engine I’m making may be able to do something about it.

10

The issue is with the amount of sound files, so memory. Currently, I only create the sound element when necessary (when the mesh that allows to play it is visible), which helps for the scene loading. However, if the visibility of these elements is toggled fast this could cause problems. On the other hand, I could create them at start, but this becomes heavier the more files I have.

For reference, I have a scene with 75 audio files ranging from 200KB to 700KB.

11

I’m thinking now we may be able to recreate the Sound object each time. So pause would just store the Sound.currentTime and play would create a new Sound and start it at the previously stored currentTime. This doesn’t work right now because streaming sounds don’t respect the Sound.play offset argument, but we should be able to get that working fairly easily.

I’m going to give it a try to see if it works.

1 Like
12

PR 15526 makes streaming sounds work with the Sound.play offset parameter, so when you want to pause the sound you can store the sound’s currentTime and dispose it, recreate the sound, and then play it with an offset that matches the stored currentTime.

See https://playground.babylonjs.com/#RF3XZS for an example.

1 Like
13

Thanks, I reverted the streaming option to true. Tested on the device I specified on this issue’s description and it works.

1 Like