Parsing SysEx messages

Electra One supports MIDI communication in both directions. Obviously, you can use Electra to send MIDI messages to your instruments, but you can also use it to receive MIDI messages and update the values of your Controls accordingly.

You do not have to do much extra work for channel messages such as CC7, CC14, and NRPN. Electra will automatically update the value of Controls that are linked to a given MIDI parameter/controller. With SysEx, however, Electra must be provided information how to parse incoming SysEx data. This tutorial provides an introduction to parsing SysEx messages.

A simple example

The SysEx parsing instructions can be included in both Electra presets and instrument files. We encourage users to develop instrument files. Instrument files allow other people to easily integrate supported instruments in their own presets.

The following example is a simplified Instrument file that allows bi-directional synchronization of two parameters. We use one Voice parameter and one Performance parameter on purpose, to show that multiple SysEx messages can be used to synchronize Electra and the connected synthesizer.

{
  "id": "yamaha-tx7",
  "name": "Yamaha TX7",
  "manufacturer": "Yamaha",
  "manufacturerId": "yamaha",
  "categories": [
    {
      "id": "op1",
      "label": "Operator 1"
    }
  ],
  "overlays": [
    {
      "id": 1,
      "name": "Op Modes",
      "items": [
        {
          "value": 0,
          "label": "Fixed"
        },
        {
          "value": 1,
          "label": "Ratio"
        }
      ]
    }
  ],
  "parameters": [
    {
      "id": 1,
      "type": "fader",
      "name": "Rate 2",
      "min": 0,
      "max": 99,
      "categoryId": "op6",
      "msg": "SysEx",
      "data": ["43", "10", "00", "01",
        { "type": "value",
          "rules": [
            {
              "parameterNumber": 1,
              "byteBitPosition": 0,
              "bitWidth": 7
            }
          ]
        }
      ]
    },
    {
      "id": 161,
      "name": "Keyboard mode",
      "categoryId": "setup",
      "type": "list",
      "overlayId": 3,
      "msg": "SysEx",
      "data": ["43", "10", "04", "02",
        { "type": "value",
          "rules": [
            {
              "parameterNumber": 161,
              "byteBitPosition": 0,
              "bitWidth": 7
            }
          ]
        }
      ]
    }
  ],
  "patch": [
    {
      "request": ["43", "20", "00"],
      "responses": [
        {
          "header": ["43", "00", "00", "01", "1B"],
          "rules": [
            {
              "type": "sysex",
              "parameterNumber": 1,
              "parameterBitPosition": 0,
              "byte": 1,
              "byteBitPosition": 0,
              "bitWidth": 7
            }
          ]
        }
      ]
    },
    {
      "request": ["43", "20", "01"],
      "responses": [
        {
          "header": ["43", "00", "01", "00", "5E"],
          "rules": [
            {
              "type": "sysex",
              "parameterNumber": 161,
              "parameterBitPosition": 0,
              "byte": 2,
              "byteBitPosition": 0,
              "bitWidth": 7
            }
          ]
        }
      ]
    }
  ]
}

This is a complete Instrument file, the part that takes care of SysEx message processing is located in the patch element.

  "patch": [
    {
      "request": ["43", "20", "00"],
      "responses": [
        {
          "header": ["43", "00", "00", "01", "1B"],
          "rules": [
            {
              "type": "sysex",
              "parameterNumber": 1,
              "parameterBitPosition": 0,
              "byte": 1,
              "byteBitPosition": 0,
              "bitWidth": 7
            }
          ]
        }
      ]
    },
    {
      "request": ["43", "20", "01"],
      "responses": [
        {
          "header": ["43", "00", "01", "00", "5E"],
          "rules": [
            {
              "type": "sysex",
              "parameterNumber": 161,
              "parameterBitPosition": 0,
              "byte": 2,
              "byteBitPosition": 0,
              "bitWidth": 7
            }
          ]
        }
      ]
    }
  ]

The Patch element

The patch element may consist of a number of objects that describe request and possible responses.

A request is a sequence of bytes (expressed in hexadecimal notation) that is sent to your synthesizer when you want to read its settings. Requests are never sent automatically, not even when the preset is loaded. They are sent when you press the [PATCH REQUEST] button in the menu, run the requestPatch command from a hardware button or over MIDI, or call patch.requestAll(), patch.request() or device:requestPatches() from Lua.

The responses array then describes the format of expected SysEx messages that the instrument will send after receiving the request message. The response consists of a message header and the rules. The header is used to identify the incoming SysEx message and the rules provide instructions on how to parse the bytes of that SysEx message and how to assign parsed data to Electra's controls.

A response is not tied to its request. Any incoming SysEx message from an interface the device is wired to, whose leading bytes match a response header, is parsed, whether it was requested or not - a dump the instrument sends on its own is parsed too. The responses of a device are tried in order and the first one whose header matches is used. A checksum in the dump is not verified.

Before the rules of a matched response are applied, the values of all parameters the rules map to are reset to 0, and the extracted bits are then combined into them.

Let's look at an example...

When Yamaha TX7 receives SysEx message

F0h 43h 20h 00h F7h

It is supposed to respond with One Voice data dump SysEx message. It is a 163 bytes long SysEx message starting with a number of constant bytes followed by the sequence of bytes with voice parameter values. A fragment of such a response might look like this:

F0h 43h 00h 00h 01h 1Bh 01h 02h 03h .... F7h

These are the above messages translated to Electra's Instrument file:

  "patch": [
    {
      "request": ["43", "20", "00"],
      "responses": [
        {
          "header": ["43", "00", "00", "01", "1B"],
          "rules": [
            {

The parsing rules

The format of the parsing rules is essentially identical to the syntax that you use for the composition of SysEx templates, you might want to take a look at SysEx templates tutorial..

Each rule describes extraction of a byte or individual bits from given SysEx message byte and an application of this extracted value to Electra's internal parameter storage.

{
  "type": "sysex",
  "parameterNumber": 1,
  "parameterBitPosition": 0,
  "byte": 1,
  "byteBitPosition": 0,
  "bitWidth": 7
}

attribute	function
type	type of parameter, sysex when omitted
parameterNumber	id of parameter (id is accepted as an older name)
parameterBitPosition	bit position within the parameter
byte	position of the byte in the SysEx message, byte 0 is the first one after the header bytes
byteBitPosition	bit position within the SysEx byte
bitWidth	number of bits to be used

The best is to describe on real world examples:

get 7-bit value of SysEx byte 1 and assign it to parameter 1 of the SysEx type:

{
  "type": "sysex",
  "parameterNumber": 1,
  "parameterBitPosition": 0,
  "byte": 1,
  "byteBitPosition": 0,
  "bitWidth": 7
}

get value expressed with bits 2 and 3 in the SysEx byte 10 and assign it to parameter 5:

{
  "type": "sysex",
  "parameterNumber": 5,
  "parameterBitPosition": 0,
  "byte": 10,
  "byteBitPosition": 2,
  "bitWidth": 2
}

compose a 14-bit value of parameter 65 out of SysEx bytes 10 (MSB) and 11 (LSB). SysEx data bytes carry 7 bits, so the low 7 bits of the parameter, bits 0 .. 6, come from the LSB byte, and the MSB byte goes above them, at parameterBitPosition 7:

{
  "type": "sysex",
  "parameterNumber": 65,
  "parameterBitPosition": 7,
  "byte": 10,
  "byteBitPosition": 0,
  "bitWidth": 7
},
{
  "type": "sysex",
  "parameterNumber": 65,
  "parameterBitPosition": 0,
  "byte": 11,
  "byteBitPosition": 0,
  "bitWidth": 7
}

The bits of one rule must fit in a 7-bit byte (byteBitPosition + bitWidth at most 7) and in a 14-bit parameter (parameterBitPosition + bitWidth at most 14); the controller logs an error when the preset loads if they do not.

extract parameter 1 (bits 0 .. 2) and parameter 5 (bits 4 .. 6) out of SysEx byte 10:

{
  "type": "sysex",
  "parameterNumber": 1,
  "parameterBitPosition": 0,
  "byte": 10,
  "byteBitPosition": 0,
  "bitWidth": 3
},
{
  "type": "sysex",
  "parameterNumber": 5,
  "parameterBitPosition": 0,
  "byte": 10,
  "byteBitPosition": 4,
  "bitWidth": 3
}

Receiving individual parameter changes

A patch response parses a dump the instrument sends when it is asked. Many instruments also send a short SysEx message whenever a parameter changes on the instrument itself. Electra can match those against SysEx templates and update the controls, without a request.

Receiving is opt-in. A template receives when its direction is both or in:

"message": {
  "type": "sysex",
  "deviceId": 1,
  "parameterNumber": 1,
  "direction": "both",
  "data": ["43", "10", "00", "01", { "type": "value" }]
}

With both, the same template is sent when the control is turned and matched when the instrument sends it back. Here, F0 43 10 00 01 3F F7 arriving from the TX7 sets parameter 1 to 63.

When the instrument sends its changes in a different format from the one it accepts, describe the reply as a separate device message with direction set to in. It is never sent:

"devices": [
  {
    "id": 1,
    "name": "Yamaha TX7",
    "port": 1,
    "channel": 1,
    "messages": [
      {
        "id": 1,
        "direction": "in",
        "data": ["43", { "type": "any" }, "00", { "type": "parameter" }, { "type": "value" }]
      }
    ]
  }
]

One device message with a parameter placeholder serves every parameter: the placeholder says which parameter the message is for, and value carries the value.

Matching

• constant bytes have to be equal, and the message has to be exactly as long as the template (at most 64 bytes)
• { "type": "any" } matches any byte - a device ID, or a channel nibble
• a checksum element matches any byte; the checksum is not verified
• a function element matches any byte; a Lua function computes a byte on the way out and cannot be run backwards

Which parameter is updated

1. the parameter named in the value placeholder's rules, if it names one
2. otherwise the parameter carried by the parameter placeholder
3. otherwise the parameterNumber of the control that uses the template

A template that none of these settle is not matched, and the controller logs a warning when the preset loads. That is a device message shared by controls with different parameter numbers that has no parameter placeholder: a message matching it could be for any of them.

Performance

Electra matches an incoming message against all receiving templates at once, so the number of templates does not slow it down: with 300 receiving templates, a message takes about 15 microseconds to match. The matching structure is built when the preset loads.