Joseph Wilk

Joseph Wilk

Things with code, creativity and computation.

Emacs as a Musical Instrument

A simple goal, to directly connect Emacs to running hardware and software synthesisers with realtime control. Emacs as a musical instrument.

I already use code to control synths (as Repl-Electric) but there is a level of indirection between the code and the effect on the music. You push keys on your computer keyboard and nothing happens. Only when you run the code does the music change. I wanted to add realtime control to my performances while still remaining in code and Emacs. Bringing the performance closer to musical instruments were instant feedback is a core part of the performance experience.

I could use external hardware, like a MidiFighterTwister (https://www.midifighter.com/#Twister) or TouchOSC (https://hexler.net/software/touchosc) but I’ve found the context switch of moving between coding to twiddling dials on hardware expensive. The code is my composition process, so it makes sense for the direct control to also be there.

Playing the Emacs

Sculpting sound live with Emacs.

Scratching Samples with Emacs

Doing crazy things with Emacs starts to open more doors in musical expression. Since we can control music hardware with Emacs, we can control the playhead position within a sample. Much like the needle of a record player.

Say we map the position of your cursor in the Emacs buffer to the position of the playback head. By moving around in your buffer you can scratch the sample.

Emacs sending Musical Controls

Sending messages to musical synths using midi and OSC.

Emacs Communicating with Open Sound Control

First we need a client within Emacs which will send OSC (https://en.wikipedia.org/wiki/Open_Sound_Control) using UDP packets. There is thankfully already an Emacs package for this: osc: https://delysid.org/emacs/osc.html

Creating a client and sending OSC messages:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

(require 'osc)
(defvar osc-client nil "Connection to send OSC From Emacs")

(defun osc-make-client (host port)
  (make-network-process
   :name "OSC client"
   :host host
   :service port
   :type 'datagram
   :family 'ipv4))

(defun osc-connect ()
  (if osc-client   (delete-process osc-client))
  (setq osc-client (osc-make-client "localhost" 4561)))

(osc-connect)

(osc-send-message osc-client ADDRESS *ARGUMENTS)

Emacs Communicating with Midi

A lot of musical hardware and software only supports midi. We are sending OSC 😕. Hence we need a quick way of converting our OSC message to a midi message. Why don’t you send midi from Emacs directly? It’s not a new thought (https://www.emacswiki.org/emacs/EmacsMidi), but I’ve seen no examples of getting it working. My answer here is path of least resistance, and I don’t feel like implementing the midi standard in Elisp.

Luis Lloret (https://github.com/llloret) has create the lovely C++ lib osmid (https://github.com/llloret/osmid) for converting between midi and OSC. And its FAST.

It compiles into two binary servers:

  • m2o – midi to OSC
  • o2m – OSC to midi

We launch the o2m server in a shell which will be listening on port 4561 and will forward on OSC messages as midi.

1

osmid/o2m -b -i 4561 -O 4562 -m 6

Midi has a fixed argument format while OSC is very flexible. Hence this a little jiggery pokery in get the source OSC message mapping to the midi arguments.

OSC message format:

1

(osc-send-message osc-client MIDI_HOST MIDI_CHANNEL MIDI_CONTROL_CODE MIDI_VALUE)

For MIDI_HOST I’m using the Inter-application communication (IAC) driver on Mac. This registers IAC Bus 1 as a midi device. I’m also sending a control change message, shaping the parameters rather than triggering notes. This could be any of the supported midi messages (note_on, note_off, pitch_bend, poly_pressure, etc).

Example of OSC to midi message:

1

(osc-send-message osc-client "/IAC Bus 1/control_change" 9 100 127)

The channel 9 and control code 100 seem magical. I’m routing those to synths via Ableton Live.

Any DAW would support this or you could send direct to a synth and not use IAC. Your MIDI_HOST would be the name your midi device registers as.

Emacs GUI for Musical Controls

We have the backend for sending live midi controls to hardware. Now we need some interface within Emacs to trigger it.

Modelling Encoders in Emacs

The design is based on the idea of dial encoders commonly found in music hardware.

Our dials in code are float numbers. To turn our dials we borrow the idea from the Chrome inspector of scrolling through possible numeric values with the arrow keys.

In our case every increment/decrement will send a live message to our synths.

We need to extract from the active Emacs buffer the thing we are trying to control. Finding the:

  • Synth (the musical instrument)
  • Parameter (ie. amp, pan, lfo, filter)
  • Value 0-127 (whatever value we want to set param)

Example changing the filter param:

1
2
3
4
5

zerocoast_cc lfo: 0.49, filter: <0.80>

;;; Synth => zerocoast
;;; Parameter => filter
;;; Value => 101.6 (127*0.80)

Here comes a lot of Elisp and some hairy regexes.

Emacs Code

Thingatpt (https://www.emacswiki.org/emacs/ThingAtPoint) which is part of the standard lib is extremely useful here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

(require 'thingatpt)

;;The patterns for matching the beginning and
;;end of something that looks like a float

(put 'float 'end-op       (lambda () (re-search-forward "[0-9-]*\.[0-9-]*" nil t)))
(put 'float 'beginning-op (lambda () (if (re-search-backward "[^0-9-\.]" nil t) (forward-char))))

(defun change-number-at-point (fn-float-op)
  "Check if the thing under the cursor looks like a
  float and if so change it with `fn-float-op`.
  `fn-float-op` is passed the name of the synth,
  parameter and float value."

  (let* ((bounds (bounds-of-thing-at-point 'float))
         (float-val (buffer-substring (car bounds) (cdr bounds)))
         (cursor-point (point)))
    (goto-char (car bounds))
    (re-search-backward "^\\([^\n]+\\): " (line-beginning-position) t)
    (let ((synth-str (match-string 1 nil)))
      (goto-char (car bounds))
      (delete-char (length float-val))
      (let* ((parts (split-string (string-trim synth-str) " "))
             (synth-and-param (if m
                                  (concat
                                    (replace-regexp-in-string
                                     "^#" "" (first parts))
                                     "/"
                                     (first (reverse parts)))
                                 nil)))
        (insert
         (format "%.2f"
                 (funcall fn-float-op
                   (string-to-number float-val)
                   synth-and-param)))))
    (goto-char cursor-point)))

(defun inc-float-at-point ()
  "Increase a float value and send OSC message."
  (interactive)
  (change-number-at-point
    (lambda (float-val synth-and-param)
      (let ((new-float (min 1.00 (+ float-val 0.01))))
        (route-osc-message synth-and-param new-float)
        new-float))))

(defun dec-float-at-point ()
  "Decrease a float value and send OSC message."
  (interactive)
  (change-number-at-point
    (lambda (float-val synth-and-param)
      (let ((new-float (max 0.00 (- float-val 0.01))))
        (route-osc-message synth-and-param new-float)
        new-float))))

We map our synth-and-param to the correct midi port and channel.

1
2
3
4
5
6
7
8

(defun route—osc-msg (synth-and-param float-val)
  (when synth-and-param
    (let ((midi-val (round (* 127.0 float-val)))
          (host "/IAC Bus 1/control_change"))
      (if (not osc-client) (osc-connect))
      (cond
       ((string-match synth-and-param "zerocoast/filter")
         (osc-send-message osc-client host 9 100 midi-val)))))))))

Finally the keybindings that trigger our instrument mode:

1
2

(global-set-key [(meta up)]    'inc-float-at-point)
(global-set-key [(meta down)]  'dec-float-at-point)

Encoder ASCII Art

We are almost done. For fun I’ve added a visually aid to the position of the float encoder. Midi messages are generally limited to 0-127 values, so if we map that to 0-100% we can create a visual representation of the current setting.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

(defun code->pots (beg end)
  "Add a ASCII encoder bar to anything that looks like a tweakable float"

  (interactive "r")
  (save-excursion
    (let ((inhibit-read-only t))
      (remove-text-properties beg end '(read-only t))
      (goto-char end)
      (setq i 0)
      (while (re-search-backward "_cc .+:\s*\\([0-9]*.[0-9]+\\)\s*\n" beg t)
        (setq i (+ 1 i))
        (let ((full     (round (* MAX-LENGTH (string-to-number (match-string 1))))))
          (let ((empty  (- MAX-LENGTH full)))
            (goto-char (match-beginning 0))
            (end-of-line)
            (let ((start-pnt (point))
                  (pad (if (>= empty 0)
                           (make-string empty ?╌)
                         "")))
              (insert (concat " #╟"
                              (make-string full ?▓)
                              "▒░"
                              pad
                              "╢"))
              (put-text-property start-pnt (point) 'read-only t)))))
      (align-regexp beg (+ (* i 106) end) "\\(\\s-*\\)#"))))

(defun pots-update (new-float old-float)
  "Update ASCII encoder bar for float being changed"

  (interactive)
  (if (not (= new-float old-float))
      (let ((old-full (round (* 100.0 old-float)))
            (full     (round (* 100.0 new-float))))
        (let ((empty  (- MAX-LENGTH full))
              (movement (- full old-full))
              (bmovement (- old-full full)))
          (save-excursion
            (when (re-search-forward "#╟" (line-end-position) t)
              (goto-char (match-end 0))
              (let ((inhibit-read-only t)
                    (start-pnt (point)))
                (if (> new-number old-float)
                    (progn ;;forwards
                      (forward-char old-full)
                      (insert (make-string movement ?▓))
                      (forward-char 2)
                      (delete-char movement))
                  (progn ;;backwards
                    (forward-char (- old-full bmovement))
                    (delete-char bmovement)
                    (forward-char 2)
                    (insert (make-string bmovement ?╌))))
                (put-text-property start-pnt (point) 'read-only t))))))))

The Ascii dials:

1
2
3

zerocoast_cc amp: 0.18 #╟▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒░╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╢

zerocoast_cc amp: 1.00 #╟▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒░╢

Binding into a Performance

All this work was used in my recent performance:

“You Fall into Your Screen”: https://vimeo.com/replelectric/you-fall-into-your-screen

I’m using Emacs to simultaneous control Synths and Unity3D in much the same way discussed in this post. It feels pretty amazing to augment the performance with this live control.

Comments