Neural networks are a remarkably versatile machine learning technology which can find patterns in just about anything. In a nutshell, you train a neural net by showing it a set of input and output pairs, and it works out how the outputs relate to the inputs. Then you can show it a new input it’s not seen before and it will produce an output for it. There’s all sorts of complexities and caveats in practice, but the central idea is that you give it data to train on, it finds patterns in it, then it can create new outputs from fresh input by itself – it’s a supervised machine learning technology.

Around the time I learned about these things I was playing with Propellerhead Software’s ReBirth, a software music studio which simulates two 303s synthesisers, and the 808 and 909 drum machine. Each of these instruments has a row of sixteen buttons, one for each note in a pattern. All four instruments play in sync, and every control in ReBirth can be edited live. It looks good, sounds great, has a very intuitive UX and is a delight to perform with.

I got to wondering what would happen if ReBirth could learn how you used it, detecting patterns in your performances. If it could spot a correlation between how you used the instruments, treating them as inputs and outputs, then it could learn to drop the bass when you triggered a kick drum. I wondered how feasible this was, and what it would be like to perform with. Would it feel like having an extra pair of hands, or would the machine seem to have a mind of its own, familiar to yours but different in some way?

I also wanted to test out what I’d learned about neural networks in theory by building and using one in practice, so I went and made a drum machine with a neural network inside it. I called it Breakage.

Breakage

Breakage is a four channel drum machine, usually set to play kick, snare, open and closed hi-hat samples on each channel. If you’ve not used software like this before, the grid of circles in the middle is called a step-sequencer. This is where you write drum patterns by turning hits on or off at each time step in each channel.

To train the neural network in Breakage, you’d write a collection of patterns, tell the net which ones to use as inputs and outputs, then set it to train. By default the kick and snare are inputs, open and closed hats are outputs. After training you can switch the net into performance mode. When performing, it will create new hi hat parts as you change the kick and snare hits.

Here’s an example. In this pattern, there are drum hits set in the kick, snare channels and closed hi-hat channels:

This pattern has an extra kick drum hit in the fourteenth step, and hits on every other step in the closed hat channel:

This one has another hit in the eighth kick step, and hits on every closed hat step:

The kick and snare channels are inputs for the neural net, and the closed hats are outputs. After training with these three patterns, it learns the correlations between them. If we change the kick hits in performance mode, it produces this hi-hat pattern:

This happens in real-time, so you can edit the inputs and get new output parts in a live performance.

The darker shades of red in the output hats are hits at lower volumes. When developing Breakage I learned that having the net just turn hits on or off gave a jumpy response. It was hard to predict which hits would appear or disappear when changing and input, and the transitions were too abrupt. I changed the net to give a continuous output, so each of the input hits can be set to a value between 0 (silent) and 255 (maximum volume), and the outputs also have this range. This lets you slowly increase the volume on a hit and have the outputs gradually change, which I found much more enjoyable to perform with.

Written around 2006, Breakage is built in Java with a Swing GUI. I used ChucK underneath as a sound library, a language for real-time music composition and performance. Java was never so great with sound. Or GUIs… but anyway, it worked, and I had some fun with it. It had its own website for a while with a forum and a space to share your patterns and nets, but I’ve taken the site down. It never really took off and was more of a personal learning exercise.

I learned a lot about neural networks and the subtleties of applying them to real world problems from writing Breakage, and a few things about software engineering. Looking back on it now, I have a few thoughts about Breakage and my original goals:

It’s really fun

This thing was great fun to play with. There’s a few people in the academic field of music informatics writing journals, running conferences and teaching courses about this kind of thing, but I’ve not seen much that’s easily accessible to casual musicians or being developed commercially. That’s a real loss, and I hope a “casual machine assisted composition and performance” market gets going one day.

Ge Wang, creator of the ChucK language, is now running Smule. They make social music apps for the iPhone-like I Am T-Pain and Sonic Lighter, which are in the same kind of space.

The training target isn’t clear

Neural networks start off behaving randomly. With repeated exposure to training data they become more accurate. On each training cycle, you test how accurate the net is and stop training it when it reaches a cutoff point you’ve set. Too much training gives you a brittle network that can match the patters it’s seen perfectly but can’t generalise from them or respond well to new input. Too little training and you get an inaccurate net that gives near random results.

This challenge is known as overfitting and is well discussed in statistical modelling and machine learning. In a musical context however, there are unclear definitions of what correct and incorrect outputs are. Sometimes deliberately overfitting to create a rigid and predictable network is desirable, as it’s easier to perform with an instrument that responds predictably to your actions. Other times it’s preferable to leave it able to generalise, enjoying the moments where it does something unexpected and delightful, and tolerating the odd sounds that come with that.

If I returned to Breakage or a similar app, I’d consider creating several networks trained to different error levels and adding a control to switch between them during a performance. Beyond my work, I’d like to see some approaches to recording and evaluating the qualitative, aesthetic, and interactive experiential outcomes of machine learning algorithms.

UX and design are crucial

If you work on the consumer facing internet you probably know this already. I didn’t realise how important this was at the time I wrote Breakage, and when I showed it to musicians they struggled with the controls before they could understand and grapple with the ideas and the musical possibilities.

It takes a lot of work to develop a simple interface to a complicated instrument, especially a novel one where there are few well-known points of reference or easy metaphors. I got a lot of things clearer in my head by writing Breakage, solving some interesting technical and conceptual challenges, but if I were approaching this again I’d invest more work in the design and the UX. At the time I wanted to scratch a personal itch, but today I’d want it to be used by others.

Breakage today

Here’s the last version of Breakage I published, version 23 for Windows, Mac and Linux. I’d be interested to hear your thoughts if you try it out, and if you can get it working! I stopped developing it years ago and I’m not sure if ChucK or that old Java code will run on a modern OS.

As well as all of the points above, if I were developing it now I’d look to build it into existing music software. No step sequencer software at the time offered an API on the note data, perhaps that’s changed with Ableton’s new Max/MSP and Python interfaces.

Update Thanks to Owen Vallis for pointing out that “the chuck binary doesn’t work on OSX lion. Here is a link to a patched binary of Chuck for OSX 10.7”

Pusher is a great streaming data service, sparing you from writing up a slew of workarounds for the various degrees of WebSockets support in various browsers. As they put it: “We believe that your developer time is best spent making awesome features, not creating infrastructure.”

At NixonMcInnes, my colleague Edd Parris and Pusher’s Developer Evangelist Phil Leggetter put Twitter’s Bootstrap on the front end and made a server-side optimisation to push the total count each second instead of sending every Tweet. Edd’s written about it on the NM blog and we’ve hosted it with Heroku at bieber.nixonmcinnes.co.uk.

I’ve stripped it down to a much simpler example with one div, no css, a minimal data set and only the most crucial RGraph settings, put it all in one HTML file and made a jit-boilerplace project on GitHub for future reference. Perhaps you’ll find it useful too.

DOT

Here’s the DOT language for describing graphs:

It’s simple: nodes are text, edges are two dashes.

Rendering tools like Pixelglow’s Graphviz can generate images from this syntax, like so:

Graphviz handles the layout and the rendering of the image. DOT is so lightweight I often use it to sketch out systems and connections between things. Graphviz detects changes in files, so if you work with Graphviz and a text editor next to each other, the image will update automatically when you save.

jQuery Mobile

I learned about jQuery Mobile at Remy Sharp‘s mobile HTML5 workshops at the Update Conference in 2011. You can create mobile web apps from a single page of html using divs, data attributes and links to specify pages and links between them. Wikipedia have a great basic example of this, here’s the central idea:

Here’s how that condiments graph looks as a jQuery Mobile site, generated from the dot with dot2jqmobile.

The dot2jqmobile script

jQuery Mobile sites are essentially graphs, with the pages as nodes and the links being edges between them. Translating between the DOT and jQuery Mobile graph representations is the central idea in dot2jqmobile. It reads and parses a DOT file using pydot, creates a Page object for each node, adds links to the Page objects, then renders them all as jQuery Mobile ready HTML with a header and footer.

Exploring spaces in the first and third person

Sometimes I write directly in DOT to capture my thoughts about a system as I’m mentally traversing it. The simple syntax makes it easy to add connections as you realise them and record a stream of consciousness. My DOT writing process often involves me following a train of thought like this:

apples -- oranges -- bananas -- ...

Then following another:

ice_cream -- pudding -- pie -- flan -- ...

And revisiting nodes in the trains, making links between them:

bananas -- banana_pie -- banana_ice_cream -- ice_cream
ice_cream -- vanilla -- cinnamon -- apple -- apple_pie -- pie

Writing the connections straight into the text lets me explore the system from a first person point of view, wandering through the space and capturing spontaneous thoughts like “an orange is like an apple, and a banana is like an orange”. Unlike a diagram editor which could be used to the same end, or more structured writing like an essay or blog post, writing directly in DOT doesn’t require any thinking about the layout of things or connections between trails and entities.

On rendering, the layout algorithms and image rendering in Graphviz give me a third-person overview of my thinking. Looking at the graph I can see the bigger picture with observations like “this is how the fruits and deserts fit together as a whole” and notice trends at this level. In the example above, I might spot that there’s an unclear boundary between flavours and desert types – should there also be a vanilla ice cream to follow the banana ice cream, or should I not join flavours and deserts? Together, the language and the renderer give me the ability to switch between first flow and third person views of systems.

But neither make it easy for me to share routes through the graph with others, or see the routes they find when they are traversing through the same space. Graphviz isn’t quite polished or simple enough for most non-technical users to feel comfortable with it, not is the DOT format. I often export rendered graphs to images, print them out and walk through them with other people. It’s interesting to ask people to think out loud while they read and make sense of graphs.

With dot2jqmobile, I hope to make it easier to follow along side others as they navigate through spaces, and to capture some of the spontaneous thinking and observations that come out of traversing in a first-person point of view.