The System

Sonic Multiplicities (SM) is a real-time machine listening and response system for solo improvisers, built on SuperCollider. A performer plays an acoustic instrument; SM listens, analyzes, learns, and answers with sound of its own. It plays only alongside a live performer, in real time. Every sound it makes reacts to what the human just played, and every sound the human plays next is shaped by what SM answered.

The learning engines train on the project's own fifteen-year archive of SuperCollider performance history logs, accumulated since 2010. The corpus is the project's own past performances. Nothing is scraped from anyone else: what SM has learned, SM has played.

The full slide deck this page condenses is available as a PDF.

Datacoustics

Datacoustics: data (a representation of facts or ideas in a formalized manner, capable of being communicated or manipulated by some process) joined to acoustic (pertaining to the sense of hearing; an agent to assist with hearing). Sonic Multiplicities is an attempt at musical datacoustics.

The term exists because the established ones don't fit. Electroacoustic music describes composers using technology to manipulate the timbres of acoustic sounds. Computer music describes computing applied to composition, helping composers write, or letting machines compose alone. SM works differently. Every well-intended musical action carries a lot of data (bow noise, the onset of tone, a decay, a strong finish), and the system's job is to extract, reveal, and manipulate that data in real time, as a listening partner.

The Five Engines

SM is five interlocking engines. Each maps to a layer of human musicianship: gestural recognition, ear training, music theory, extended technique, and state management. For a human, those layers are ordered easiest to hardest. For a machine, they are ordered hardest to easiest. The inversion is the point. SM is only kinda smart, but it's a great listener.

1. Generative deep learning (gestures)

A non-realtime peephole convolutional LSTM, RNN-modeled, used to classify musical gestures. Its output is normalized into a probability distribution, and "intent" assignments are made by classifying those normalized outputs against the performance's SuperCollider history logs: when the predicted outcome matches what the logs show actually happened, the intent is associated with that output.

2. Live audio analysis (ear training)

A real-time OSC stream of analysis over one or two mono audio-rate buffers: peak, RMS, spectral content, pitch, MFCC, and k-means clustering. Outputs include 2-D k-means clusters, interval vectors, amplitude and spectral value arrays, and binary prediction trees, with conversions between pattern sequences, randomized/aleatoric coefficients, and unit-generator wildcard values.

3. Discriminatory deep learning (music theory)

A real-time MFC discrimination engine, labeled with self-defined "outcomes." It is a musical differentiator: it determines in what ways the current musical input is distinct from previous ones. The broader those deltas, the more restrictive the analysis becomes.

4. Audio extension (extended technique)

Audio unit generation, DSP node management, and auto-mixing across 20 output channels. Temporal modulation: granular sampling, rate shifting, single/multi-tap/complex delay lines, and filtered, equalized capture buffers with complex routing. Spatial modulation: mono, stereo, and quad upmixed to B-Format, with A-Format and B-Format conversion to AmbiX and FuMa ambisonics up to 3rd order. Dynamics: performance-wide limiting and an intelligent compander with lookahead prediction. Pitch and harmonics: independent manipulation of tones and their partials: chord manipulation, inversion, arpeggiation.

5. State management

The engine that brings the other four into cohesion: a two-way non-deterministic finite automaton whose tape is the performance itself. Unlike an ordinary state machine it can revisit already-processed material, and its transition table is not static. A move right is a transition to a new accepted musical concept; a move left is a return to a prior one; null remains on the current concept. Which analysis data streams are active at any moment is determined by which gates the automaton opens, and gate sensitivity is set by comparing the activity of the performance's first five minutes against the current musical gesture. Behind every musical event is a clock; ensembles are musicians whose clocks are synchronized. SM's state machine makes the performer's time a first-class citizen.

The Performance-Completion Loop

SM's output is how the system communicates ideas back to the performer, and that exchange between machine and musician is itself part of the performance. The deck calls this bi-reactionary 4-dimensional feedback. The same outputs also feed external contexts (video synthesis via FFmpeg and OpenCV), so the loop closes around the audience as well.

"Sonic Multiplicities feels like you're playing chamber music with yourself."
- Garrett Semmelink, violinist

"The software is not a musical partner - it's more like a ghost, a presence. You can choose to believe it is there, and that it is listening to you, or you can choose to ignore it and not believe."
- Tyler Dinner, guitarist

Questions