I wrote Breakage back in 2006 to explore using neural networks in drum machine programming. The last download is available at the end of this page. It’s unsupported and buggy, but some people have got it working. Have a look in the comments for help and ideas running it.
This article describes the thinking and challenges behind the making of Breakage. You might also enjoy my latest music and data science project, Akin To.
Neural networks and drum machines
Neural networks are a remarkably versatile machine learning technology which can find patterns in just about anything. In a nutshell, you train a neural net by showing it a set of input and output pairs, and it works out how the outputs relate to the inputs. Then you can show it a new input it’s not seen before and it will produce an output for it. There’s all sorts of complexities and caveats in practice, but the central idea is that you give it data to train on, it finds patterns in it, then it can create new outputs from fresh input by itself – it’s a supervised machine learning technology.
Around the time I learned about these things I was playing with Propellerhead Software’s ReBirth, a software music studio which simulates two 303s synthesisers, and the 808 and 909 drum machine. Each of these instruments has a row of sixteen buttons, one for each note in a pattern. All four instruments play in sync, and every control in ReBirth can be edited live. It looks good, sounds great, has a very intuitive UX and is a delight to perform with.
I got to wondering what would happen if ReBirth could learn how you used it, detecting patterns in your performances. If it could spot a correlation between how you used the instruments, treating them as inputs and outputs, then it could learn to drop the bass when you triggered a kick drum. I wondered how feasible this was, and what it would be like to perform with. Would it feel like having an extra pair of hands, or would the machine seem to have a mind of its own, familiar to yours but different in some way?
I also wanted to test out what I’d learned about neural networks in theory by building and using one in practice, so I went and made a drum machine with a neural network inside it. I called it Breakage.
Breakage is a four channel drum machine, usually set to play kick, snare, open and closed hi-hat samples on each channel. If you’ve not used software like this before, the grid of circles in the middle is called a step-sequencer. This is where you write drum patterns by turning hits on or off at each time step in each channel.
To train the neural network in Breakage, you’d write a collection of patterns, tell the net which ones to use as inputs and outputs, then set it to train. By default the kick and snare are inputs, open and closed hats are outputs. After training you can switch the net into performance mode. When performing, it will create new hi hat parts as you change the kick and snare hits.
Here’s an example. In this pattern, there are drum hits set in the kick, snare channels and closed hi-hat channels:
This pattern has an extra kick drum hit in the fourteenth step, and hits on every other step in the closed hat channel:
This one has another hit in the eighth kick step, and hits on every closed hat step:
The kick and snare channels are inputs for the neural net, and the closed hats are outputs. After training with these three patterns, it learns the correlations between them. If we change the kick hits in performance mode, it produces this hi-hat pattern:
This happens in real-time, so you can edit the inputs and get new output parts in a live performance.
The darker shades of red in the output hats are hits at lower volumes. When developing Breakage I learned that having the net just turn hits on or off gave a jumpy response. It was hard to predict which hits would appear or disappear when changing and input, and the transitions were too abrupt. I changed the net to give a continuous output, so each of the input hits can be set to a value between 0 (silent) and 255 (maximum volume), and the outputs also have this range. This lets you slowly increase the volume on a hit and have the outputs gradually change, which I found much more enjoyable to perform with.
Written around 2006, Breakage is built in Java with a Swing GUI. I used ChucK underneath as a sound library, a language for real-time music composition and performance. Java was never so great with sound. Or GUIs… but anyway, it worked, and I had some fun with it. It had its own website for a while with a forum and a space to share your patterns and nets, but I’ve taken the site down. It never really took off and was more of a personal learning exercise.
I learned a lot about neural networks and the subtleties of applying them to real world problems from writing Breakage, and a few things about software engineering. Looking back on it now, I have a few thoughts about Breakage and my original goals:
It’s really fun
This thing was great fun to play with. There’s a few people in the academic field of music informatics writing journals, running conferences and teaching courses about this kind of thing, but I’ve not seen much that’s easily accessible to casual musicians or being developed commercially. That’s a real loss, and I hope a “casual machine assisted composition and performance” market gets going one day.
The training target isn’t clear
Neural networks start off behaving randomly. With repeated exposure to training data they become more accurate. On each training cycle, you test how accurate the net is and stop training it when it reaches a cutoff point you’ve set. Too much training gives you a brittle network that can match the patters it’s seen perfectly but can’t generalise from them or respond well to new input. Too little training and you get an inaccurate net that gives near random results.
This challenge is known as overfitting and is well discussed in statistical modelling and machine learning. In a musical context however, there are unclear definitions of what correct and incorrect outputs are. Sometimes deliberately overfitting to create a rigid and predictable network is desirable, as it’s easier to perform with an instrument that responds predictably to your actions. Other times it’s preferable to leave it able to generalise, enjoying the moments where it does something unexpected and delightful, and tolerating the odd sounds that come with that.
If I returned to Breakage or a similar app, I’d consider creating several networks trained to different error levels and adding a control to switch between them during a performance. Beyond my work, I’d like to see some approaches to recording and evaluating the qualitative, aesthetic, and interactive experiential outcomes of machine learning algorithms.
UX and design are crucial
If you work on the consumer facing internet you probably know this already. I didn’t realise how important this was at the time I wrote Breakage, and when I showed it to musicians they struggled with the controls before they could understand and grapple with the ideas and the musical possibilities.
It takes a lot of work to develop a simple interface to a complicated instrument, especially a novel one where there are few well-known points of reference or easy metaphors. I got a lot of things clearer in my head by writing Breakage, solving some interesting technical and conceptual challenges, but if I were approaching this again I’d invest more work in the design and the UX. At the time I wanted to scratch a personal itch, but today I’d want it to be used by others.
Here’s the last version of Breakage I published, version 23 for Windows, Mac and Linux. I’d be interested to hear your thoughts if you try it out, and if you can get it working! I stopped developing it years ago and I’m not sure if ChucK or that old Java code will run on a modern OS.
As well as all of the points above, if I were developing it now I’d look to build it into existing music software. No step sequencer software at the time offered an API on the note data, perhaps that’s changed with Ableton’s new Max/MSP and Python interfaces.