Skip to content

jes5199/synthesizing-video-and-turning-it-into-music-bang

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Synthesizing Video & Turning It Into Music!

on the GPU! in the browser! using WebGL 3D! and the Web Audio API!

I originally gave this talk at ‼️con, which requires all talk titles to include at least one exclamation point.

Video of that talk is available: https://www.youtube.com/watch?v=nwsg-ZTRRoI
The audio gets wonky because their HDMI system used 48kHz audio sample rate rather than the 44.1kHz that my code expects, oops.

I reprised the talk at 🍩.js in Portland, and video is available here: https://www.youtube.com/watch?v=u0umvcH7kIc
In this version, the audio gets wonky because instance of Skype running on the PA system was secretly trying to automatically readjust the input levels while the hosts wrestled against it, in confusion.

Hopefully future talks will demonstrate even more varied and interesting ways of failing to record this music.

What do you call this thing

VIDEOSYNTH AURALIZER

can I see it working

blue skies and red trees

The piece of video/music that I try to show off that never seems to work quite right on stage is available here:

"blue skies and red trees"

I have no idea what browsers it will work in. I use Chrome. Feel free to view the source, but it's a bit of a mess and kind of overengineered- I pushed some of the video-to-audio conversion into GPU code, and added the oscilliscope and FFT visualizations, and I don't remember what else.

composition b

The initial inspiration for the videosynth auralizer came as "composition b", which I wrote in a single night's worth of insomnia, October 2016.

without incidental complexity

The first piece that made me think I was making something musically interesting was "without incidental complexity"

code from the talk

  1. A440 in Javascript
  2. A Major triad in Javascript
  3. A440 in WebGL Jump directly to: (javascript audiolizer function) (WebGL waveform)
  4. the ruby script that converts wav files to images
  5. auralizing images of my own voice

Sites I consulted when creating these demos, these are all really amazing resources:

other stuff

The algorithmic art I used in a slide is "chaos portal (red/blue)" on shadertoy: https://www.shadertoy.com/view/MtlyDj

A related but different piece titled "chaos pyramid" was on display during the intermission at !!con: https://www.shadertoy.com/view/ldscWN

The Patreon page that I haven't managed to actually use yet is at https://www.patreon.com/jes5199

I put some images I generated onto https://wolfe-interval.tumblr.com/

I ran a version of the auralizer on a family of aliasing patterns that I had been playing with (it's a long story.) and I found the noise surprisingly pleasing and so I posted it to soundcloud as if it was music: https://soundcloud.com/wolfe-interval/chromogeographic-oscillation

magic number

44,100 ÷ 1024 = 43.06640625 Hz

maybe someone else will stumble across this number in the process of doing something strange with audio, google it, and find this page

after all, that's how I found this one: https://www.adventurekid.se/akrt/glossary/one-cycle-waveforms/1024-at-44-1-khz/

About

Info from my talk "Synthesizing Video & Turning It Into Music!"

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published