Syncing to MP3 in the browser
category: code [glöplog]
Hi....
As the title suggests.. I'm trying to sync visuals to an MP3 that's playing in the browser. I also want to be able to start/stop and seek around the track as it's playing without the sync drifting.
The naive approach of just getting currentTime on an Audio element isn't anywhere near good enough. I _think_ if I play from the start of the track then it's consistent and OK, but if I start seeking around then the timing jumps out.
I've messed around with AudioContext and createMediaElementSource but I'm not getting any better results.. which I expected. I suspect I might be able to get good results with this approach if I played around a bit more.. but I'm sorta worried that I might be straying into "it works on my machine" territory.
Can anyone point me in the right direction for how to do this properly?
Also.. if the answer doesn't anger the CORS gods then that's soooo much better. I'd like to be able to load the audio from a local file without having to turn off browser security. A plain audio element works fine, but as soon as I pipe that through createMediaElementSource then CORS steps in and silences the track for me :(
As the title suggests.. I'm trying to sync visuals to an MP3 that's playing in the browser. I also want to be able to start/stop and seek around the track as it's playing without the sync drifting.
The naive approach of just getting currentTime on an Audio element isn't anywhere near good enough. I _think_ if I play from the start of the track then it's consistent and OK, but if I start seeking around then the timing jumps out.
I've messed around with AudioContext and createMediaElementSource but I'm not getting any better results.. which I expected. I suspect I might be able to get good results with this approach if I played around a bit more.. but I'm sorta worried that I might be straying into "it works on my machine" territory.
Can anyone point me in the right direction for how to do this properly?
Also.. if the answer doesn't anger the CORS gods then that's soooo much better. I'd like to be able to load the audio from a local file without having to turn off browser security. A plain audio element works fine, but as soon as I pipe that through createMediaElementSource then CORS steps in and silences the track for me :(
Quote:
I'd like to be able to load the audio from a local file without having to turn off browser security
Use a file input and select it from disk (you can also drag and drop files onto it) you can then access the file and pass it to the respective APIs (either create a blob URL and pass it to an audio tag or use WebAudio API). I didn't really have any issues syncing using currentTime as far as I can remember, but it's been a while.
i use mousewheel to scroll back and forth on frames on my web based demos and never really noticed anything out of sync, do use audio context and do suffer from CORS issues but i got used to running a small http-server and also supplying link to live online version and a video capture to orgas. some parties will force you to make an electron build out of it (because pedantic orgas will be pedantic and refuse to run a damn http-server because omg think of the children reasons or whatever) which can be a headache if you didn't prepare your code properly but should also be doable.
anyways, i think mp3 isn't supported by all browser vendors properly, i use ogg/m4a. maybe the mp3 encoding itself is weirded out? tried re-encoding it to something with other settings (CBR instead of VBR or something) or just using ogg instead?
some reference code if you want to look: https://github.com/psenough/heart_of_molvania hope the debug timer var still works properly, didn't really check after blackpawn's final version, but it should.
anyways, i think mp3 isn't supported by all browser vendors properly, i use ogg/m4a. maybe the mp3 encoding itself is weirded out? tried re-encoding it to something with other settings (CBR instead of VBR or something) or just using ogg instead?
some reference code if you want to look: https://github.com/psenough/heart_of_molvania hope the debug timer var still works properly, didn't really check after blackpawn's final version, but it should.
one way around CORS is to have gfx/music assets in the html base64 encoded. no need for electron or external html-servers. our demos did just that.
as for sync we did it old school. knowing the BPM, having tick in the engine and a calculator handy
as for sync we did it old school. knowing the BPM, having tick in the engine and a calculator handy
Quote:
pedantic orgas will be pedantic and refuse to run a damn http-server because omg think of the children reasons or whatever
I know out of own experience that compo orgas can have a stressful job, because of tight deadlines which are a couple of hours before the compo and then people still coming up after the deadline with improved/fixed versions and last-minute submissions.
Therefore it might just be a time issue, because setting up a web server for that one demo only on top of all other things is just added hassle. Please cut orgas some slack there
and come to inercia demoparty!
Okay.. it seems that it was the vbr mp3 that was the problem. Both ogg and cbr mp3 work fine. Thanks ps :)
CORS isn't currently a problem - the whole demo runs by just opening index.html in the browser. I load the audio from index.html like so..
..and then just call play() on it from a user action and it works fine. I was just concerned because I didn't want to end up with a solution that didn't work without a server
CORS isn't currently a problem - the whole demo runs by just opening index.html in the browser. I load the audio from index.html like so..
Code:
<audio id="music"><source src="assets/zik.mp3" type="audio/mpeg"></audio>..and then just call play() on it from a user action and it works fine. I was just concerned because I didn't want to end up with a solution that didn't work without a server
Quote:
pedantic orgas will be pedantic and refuse to run a damn http-server
Why should they? I've yet to see a web-based demo that didn't run with "--allow-file-access-from-files".
Someone™️ should really make a tiny standalone exe that starts a local HTTP server on port 8000something, sandboxed to the directory it's launched from, with support for 206 Partial Content responses (which I mention specifically because last time I checked, it wasn't guaranteed that Random Tiny HTTP Servers did, and certain browsers relied on them for media playback), and then opens the default browser to that page.
That way, you could just drop it into the folder of any browser demo and have a nice easy icon to click on like any normal demo. Browser demo authors don't have to hack around restrictions with data: URLs or find that one setting in cables.gl to disable the "you're launching this from a local file like some kind of freaking dinosaur" message, and compo orgas don't have to set up shortcuts with --allow-file-access-from-files switches and can just write "browser demos will be launched from Browserlaunchomatic" in their rules.
I realise that PC compo orgas tend to be naturally resistant to any new unknown things in their workflow, but I really think that it will be outweighed by them not having to give more of a shit about browsers than absolutely necessary.
That way, you could just drop it into the folder of any browser demo and have a nice easy icon to click on like any normal demo. Browser demo authors don't have to hack around restrictions with data: URLs or find that one setting in cables.gl to disable the "you're launching this from a local file like some kind of freaking dinosaur" message, and compo orgas don't have to set up shortcuts with --allow-file-access-from-files switches and can just write "browser demos will be launched from Browserlaunchomatic" in their rules.
I realise that PC compo orgas tend to be naturally resistant to any new unknown things in their workflow, but I really think that it will be outweighed by them not having to give more of a shit about browsers than absolutely necessary.
EvilPaul: AudioBuffer and AudioBufferSourceNode work well with seeking. Of course they decode the whole file into RAM so you probably shouldn't try that with a 4 hours DJ set but for single tracks they're ok :)
or just "python -m http.server"
KeyJ: i've been at parties where orgas were refusing to run that because "it wouldn't be fair to others".
v3nom: I try to give orgas some slack, but web-based demos always got some ammount of shit from party orgas and it's sad that modern party rules still seem to largely enforce the "if you want your web-based demo to run on the big boys executable demos compo you need to jump through hoops and make it a click away for us to even consider playing it" mindset. while i was brought up under the "you have a demo for us? i'll do everything i possibly can to get it running on our compo" mindset. call it little party syndrome ;)
v3nom: I try to give orgas some slack, but web-based demos always got some ammount of shit from party orgas and it's sad that modern party rules still seem to largely enforce the "if you want your web-based demo to run on the big boys executable demos compo you need to jump through hoops and make it a click away for us to even consider playing it" mindset. while i was brought up under the "you have a demo for us? i'll do everything i possibly can to get it running on our compo" mindset. call it little party syndrome ;)
Quote:
or just "python -m http.server"
Apparently that one still doesn't support partial content / range requests (although to be fair I can't remember which browsers that actually mattered in - it might have only been Safari...), and from the compo organiser / casual watcher perspective it's just swapping "you need to fuck around with browser command line switches" for "you need to fuck around with installing Python" :-)
