GLSL Sandbox Audio
category: code [glöplog]
Had a crazy idea for Audio in the GLSL sandbox - wondering if it's possible/easy doable?
have a
uniform int uPacketCount;
the idea is to grab the first 1024 pixels from the frame buffer (or whatever audio packet size) and dump that out to the soundcard (prolly need a float framebuffer), using winapi waveout or openal or whatever. Once a packet has been grabbed, increase the PacketCount uniform every time a new audio packets worth has been grabbed.
Packet count can then be used in the shader to sync and could then write crazy 7 supersaw synths with moog ladder filters and lush reverbs e.t.c. easily in the frag shader code.
Easily doable or sunday morning booze talking madness?
Shabby
have a
uniform int uPacketCount;
the idea is to grab the first 1024 pixels from the frame buffer (or whatever audio packet size) and dump that out to the soundcard (prolly need a float framebuffer), using winapi waveout or openal or whatever. Once a packet has been grabbed, increase the PacketCount uniform every time a new audio packets worth has been grabbed.
Packet count can then be used in the shader to sync and could then write crazy 7 supersaw synths with moog ladder filters and lush reverbs e.t.c. easily in the frag shader code.
Easily doable or sunday morning booze talking madness?
Shabby
Something like this but with shaders?
Kind of - but using that would be overkill and counter creative.
as above dumping the first x pixels to the soundcard would keep all the code unified and in one place, and u'd have to do some serious creative coding to get decent toons out - which is half the fun :P. It would mean u could have micro demos - with totally awesome audio in the sandbox (or total shite depending on code skillz :) ). Could be fun!
as above dumping the first x pixels to the soundcard would keep all the code unified and in one place, and u'd have to do some serious creative coding to get decent toons out - which is half the fun :P. It would mean u could have micro demos - with totally awesome audio in the sandbox (or total shite depending on code skillz :) ). Could be fun!
Awesome - super band limited additive synthesis glitch cubes here we come :P
can't think of a way of doing FII's in shaders because of shaders stateless nature, so is there any way to do FIR's with resonance or even sneaky FII's?
can't think of a way of doing FII's in shaders because of shaders stateless nature, so is there any way to do FIR's with resonance or even sneaky FII's?
I got around to testing the Audio stuff in my own shader rig last night - it's totally awesome - so much power you can do some real awesome synths:
http://www.youtube.com/watch?v=JDl5J83cAPI
I noticed one thing tho - you really need tweak scrollbars for the complex stuff - i.e. cymbals - since it's so hit and miss.
http://www.youtube.com/watch?v=JDl5J83cAPI
I noticed one thing tho - you really need tweak scrollbars for the complex stuff - i.e. cymbals - since it's so hit and miss.
Wtf. Hat's off for crazyness! :D
Also that mrdoob page doesn't work for me on FF. What am I doing wrong?
Also that mrdoob page doesn't work for me on FF. What am I doing wrong?
Interesting stuff Shabby, nice work :)
Shabby: Wow! Do you see this becoming suitable for 1k/4k intros ? 1k is probably too tight.
Yes!
It's actually been years since I've heard about people starting to think about using GPUs in order to synthesize sounds, really nice work Shabby!
It's actually been years since I've heard about people starting to think about using GPUs in order to synthesize sounds, really nice work Shabby!
when i asked if this is possible some time ago all i got was along the lines of "lol, why would you wanna do sth like that?".
nice to see theres actually some substance behind the idea.
nice to see theres actually some substance behind the idea.
Hey thx for the comments doods - I had so much fun coding that - in fact too much spent hours "tweaking" the synth :P
@p01: 1k-4k demos - maybe, the audio setup code is a one liner using waveoutopen and preparing a load of buffers. The actual audio sync is a few lines - but still small.
Finding it totally awesome for audio experiments/r&d since you can recompile on the fly and you have massive processing power - it's amazing for DSP math voodoo fu!
@p01: 1k-4k demos - maybe, the audio setup code is a one liner using waveoutopen and preparing a load of buffers. The actual audio sync is a few lines - but still small.
Code:
if ((((int)(gTime*44100))/gAudioBufferSize)!=LastAudioBlock)
{
int RH=(gAudioBufferSize/Form1->ClientWidth)+1;
i16 temp[4096];
glReadPixels(0, 0, Form1->ClientWidth, RH, GL_RGBA, GL_UNSIGNED_BYTE, temp);
for (int a=0;a<gAudioBufferSize*2;a++)
gAudioBlocks[gCurrentBlock][a]=(int)temp[a]-32767;;
SendAudio();
}
Finding it totally awesome for audio experiments/r&d since you can recompile on the fly and you have massive processing power - it's amazing for DSP math voodoo fu!
I don't know if you'd find this example useful. It does run in both FF and Chrome though
https://github.com/greggman/html5bytebeat
There is code in there to make it so you can edit the GLSL shaders as well but it's commented out
https://github.com/greggman/html5bytebeat
There is code in there to make it so you can edit the GLSL shaders as well but it's commented out
oh my... I would have never though I would see gman here. ^^
I tried this audio-from-shader approach a while ago, and it turned out that:
- code cost is quite huge for 4k (song data texture + framebuffer grab); might be not that costly in OpenGL,
- no way to do any kind of IIR filter in shader,
+ enough power to go for additive synthesis (and using this, you can apply filtering directly from frequency response function)
- code cost is quite huge for 4k (song data texture + framebuffer grab); might be not that costly in OpenGL,
- no way to do any kind of IIR filter in shader,
+ enough power to go for additive synthesis (and using this, you can apply filtering directly from frequency response function)
Shabby: can you find my email on my homepage and mail me? i'm looking for sound for a javascript 4k to try and do something for payback demoparty (first days of february), if you have something i can use it would be cool to try and fit some visuals on it.
Did a bit more work on the audio - got delays and visualization working:
http://youtu.be/FeK338ojIq0
@KK
I agree at first the lack of IIR's is a real pain, but it actually clears up the audio quality because of phase issues with those filters (yeah I know u can linear phase a IIR - but it's not 100%) and I'm find phase distortion gives a much more stable pure sound. I'm also finding the code size for the whole thing tiny - less than 400 bytes at the moment - and that's not creatively compressed/obfuscated. I wouldn't use a texture for song data storage - much simpler to code with simple arrays and bit switches for turning parts off and on.
Another advantage is since the audio generation is 100% parametric you can do crazy fx (reverse play, wild psytrance pitch wobbles e.t.c).
I think the biggest disadvantage is glitching - any kind of cpu os slowdown can cause a audio glitch and make ur 4k masterpiece seem buggy and inconsistent.
@psenough - I couldn't find ur email so I post here, man I'd love to work on a 4k demo with you, but I'm really tied up with work and stuff at the moment.
http://youtu.be/FeK338ojIq0
@KK
I agree at first the lack of IIR's is a real pain, but it actually clears up the audio quality because of phase issues with those filters (yeah I know u can linear phase a IIR - but it's not 100%) and I'm find phase distortion gives a much more stable pure sound. I'm also finding the code size for the whole thing tiny - less than 400 bytes at the moment - and that's not creatively compressed/obfuscated. I wouldn't use a texture for song data storage - much simpler to code with simple arrays and bit switches for turning parts off and on.
Another advantage is since the audio generation is 100% parametric you can do crazy fx (reverse play, wild psytrance pitch wobbles e.t.c).
I think the biggest disadvantage is glitching - any kind of cpu os slowdown can cause a audio glitch and make ur 4k masterpiece seem buggy and inconsistent.
@psenough - I couldn't find ur email so I post here, man I'd love to work on a 4k demo with you, but I'm really tied up with work and stuff at the moment.
with the glitching, would it be possible to buffer a certain amount in main memory?
Totally - I actually send 2 blank audio packets at the start before I render anything - and that gives me 2 frame dropouts before I start to get glitching. My audio packet size is 2048 samples, so as long as the frame rate doesn't drop below 44100/2048= 21fps everything syncs perfectly. You can increase the initial blank packets sent to give a more safety space to stop glitches - but at the expense that you will be x packets time behind in the visual frame (which you can compensate for by fiddling with the uniform time).