Procedurally generating music, scoring generations based on an audio service (like SoundCloud) identifying it as an existing song/artist. The more famous the track/artist the better.
Machines identifying audio tend to:
- Reduce the audio features to their essence (facilitating fast lookup or accuracy on a sliding scale).
- Rely on computer vision techniques to create audio fingerprints.
- Account for differing audio quality and ambient background noise.
- Use a training set of sane music to teach the algorithm to recognise music.
Audiofinger print generated by Chromaprint:
We use these propeties to guide us in creating new music for machines that explores the smudged edges around machine listening. Highlighting how differently humans and machines identify music. And for fun.
To try and match our generative audio to songs we will use a number of music services and some open-source audio fingerprinting tools. Most commercial audio fingerprinting algorithms are secret and patented up to the eyeballs. So there is a lot of trial and errors. Some services used to evalute a song match:
- Soundcloud (copyright detection)
- Youtube (copyright detection)
- Shazam (audio lookup)
- Chromaprint (Open-source audio fingerpinter)
Music for Machines
All generated tracks have comments exactly when a song was detected.
Warning: The audio clips are designed for machines and not your weak human ears 
. Hence keep your volume low before listening
Audio generations by @Finger_smudger
Generation #1 – 1467636802259
Artists/Songs identified:
- Sophonic Media – Make It Shine
- Pachanga Boys – Time
- Johan Vilborg – Second Wind (Illuminor Remix)
- Oelki – Galileo
- Lipps, Inc – Funkytown
- Spaceanddisco – Nights
- Matt Fax – Voyage orignal mix Bullistik
- Katty Perry – Birthday (Cash Cash Remix)
Generation #1.1 – 1469483772764
Artists/Songs identified:
- George Michael – A Different Corner
- Dezt – Last Year
- Axiuw – Be Yourself (Original Mix)
- Duran Duran – Thank You
Generation #2.0 – 1470683054969
Artist/Songs identified:
- Dimension – Mangata
- Michael Jackson – You Are Not Alone (tempo 60 bpm / B major)
Generation #2.0 – 1470700413305
Artist/Songs identified:
- T-Pain Vs Chuckie Feat. Pitbull – Its Not You (Its Me)
- Pink Floyd – Cymbaline
Machines listening with ambient background noise
While I experimented with lots of different services the above examples were most successful when using Shazam for identification. This focuses on operating in noisy environments and identifying a sound as quickly as possible based only on partial song information. This tolerance makes it easy to get Shazam to mis-match audio to songs.
The other services also had a nasty habit of banning new accounts uploading what appeared to be copyrighted infringing content (who would have thought!). Which makes the whole mass procedural generation somewhat challenging.
Shazam has a desktop app which will run detection on audio for 8 hours continuously. So over that time we generate a large set of audio and pick the winners from each generation.
Overtone synths & score generation
Using Overtone and Clojure a single audio generation works by:
Dynamically generating Overtone synths using generative testing framework test.check. Using QuickCheck style generators is a cheap way of exploring a permutation space given grammar rules, like those of a synth definition in Overtone. Supports selection of:
- Audio wave (potentially many combined)
- Envelope type
- Effects (reverb/echo/pitchshift/delays)
- Filters (low pass/high pass)
The various properties of the audio units are selected randomly.
Dynamically generating a score varying:
- Clock tempo
- Note lengths
- Root note / scale
- Octaves
Dynamically generating synth control parameters:
- Distortion
- Changing audio wave (sin/saw/triangle/square/etc)
Running for 3 minutes with a random chance of mutation to score and parameters.
- Store state of a generation for future evolution. We store the state and mutations as edn: Example state for one generation
Each generation scored based on number of Shazam matches (scraped from Notification alerts on OS X).
Each generation scored by popularity of artists matched (manually
).
To avoid any background noise or messing with microphones we use SoundFlower with the following setup:
Overtone main audio ->
Soundflower input device
Soundflower output device -> Shazam Desktop.
Conclusion
There is a clear difference in the strength of accuracy when it comes to fingerprinting audio for copyright infringement. It’s noticeable that Soundcloud or YouTube are matching when processing the entire track (even though it will check for partial matches) while Shazam focuses on as small a segment as possible LIVE. Open-source alternatives (like Chromaprint) while useful, provided little help tricking the commercial services.
Coming back to Shazam, what actually made the tracks match remains somewhat of a mystery. If we look at one example “Michael Jackson - You Are Not Alone” our generative score was not even in the same scale or tempo! We can identify things that made it hard to match, for example adding drums patterns killed off all matches. More layers of audio, more permutations to explore.
One thing is clear, the way machines learn and the specialisation on a single application rules out a whole subset of sound that is unlikely to enter the realm of music. Hence for the creators of the algorithms, a mismatch of this type is of little relevance.
This lost ghost music is perhaps just for the machines.
Source code
Finger Smudge: https://github.com/josephwilk/finger-smudge. Core engine behind generating audio and container for all failed and future audio fingerprint smudge experiments:
Synthatron: https://github.com/josephwilk/synthatron. Overtone Synth generator using
test.check: