LinnStrument : the ultimate open-source hacker instrument

LinnStrument:
the ultimate open-source hacker instrument
Geert Bevin

Who am I?
• Geert Bevin, Twitter @gbevin
• XRebel Product Manager at ZeroTurnaround
• Java Champion, Musician, Composer, Arranger, Producer, Singer,
Guitarist, Gamer, Kung-Fu
• Many open-source projects including Gentoo Linux, OpenLaszlo,
RIFE, Juce, …

LinnStrument firmware and tools

Revolutionary Music Performance
Controller with 3D Note Expression

Front-panel
Chassis + circuit boards
Translucent silicone sheet
Final prototype before production - actual units are slightly different

Metal chassis with wooden sides
Sensor board

LEDs board

Connection between circuit boards

Arduino Due’s
ARM chip Serial <-> USB MIDI Shield

What you need to know about
the Arduino Due

ARM Cortex-M3
32-bit 84MHz
CPU
SPI signals
shared by
LED/Sensor
Digital 33-37
Footpedals
DIN <-> USB
LED control
MIDI <-> Serial

Arduino Due and LinnStrument
• 32-bit core, 4 bytes wide operations within single CPU clock
• CPU Clock at 84Mhz
• 96 KBytes of SRAM
• 512 KBytes of Flash memory for code
• Digital I/O pins
• Serial Peripheral Interface (SPI) pins with Slave Select

Very simple Arduino program
// the setup function runs once when you press reset or power the board
void setup() {
// initialize digital pin 13 as an output.
pinMode(13, OUTPUT);
}
// the loop function runs over and over again forever
void loop() {
digitalWrite(13, HIGH); // turn the LED on (HIGH is the voltage level)
delay(1000); // wait for a second
digitalWrite(13, LOW); // turn the LED off by making the voltage LOW
delay(1000); // wait for a second
}

Arduino code
• language based on C/C++
• concise reference with all language structures, values and functions
• Arduino IDE to get started
• the ‘setup’ function runs once, when the board starts up
• the ‘loop’ function runs at 100% CPU, over and over again

Only one execution thread
• what happens in the ‘loop’ function is all that’s happening
• if something takes too long, something else isn’t happening
• guerrilla coding tactics: powerful, short strikes and get out of there
• design your algorithms to be sub-dividable
• avoid dynamic memory allocation, it’s very slow

I don’t like the Arduino IDE
• non-HDPI fonts
• extremely basic editor
• bad OS integration
• I use Sublime Text 3 with the Stino plugin

How the LinnStrument hardware
is accessed through software

Easy LED functions
• Change the color and brightness of a single LED
void setLed(byte col, // Column of LED to be changed
byte row, // Row of LED to be changed
byte color, // Color of LED to be changed
byte brightness) // Brightness of LED (0, 1, 2 or 3)
• Light up a single LED with the default color and brightness
void lightLed(byte col, // Column of LED to be changed
byte row ) // Row of LED to be changed
• Clear a single LED
void clearLed(byte col, // Column of LED to be changed
byte row ) // Row of LED to be changed

Details of LED control
• LED control is done through SPI using pin 10, with mode 0, running at 21MHz
SPI.begin(10);
SPI.setDataMode(10, SPI_MODE0);
SPI.setClockDivider(10, 4); // 84MHz divided by 4
• Digital pin 37 is an output and connected to the LED driver chips
• Write 32-bit data structure to SPI to control the LEDs, refreshed every 100μs
digitalWrite(37, HIGH); // enable the outputs of the LED driver chips
SPI.transfer(10, column, SPI_CONTINUE); // send column (left-shifted 2 bits + special bit 7)
SPI.transfer(10, blue, SPI_CONTINUE); // send blue byte
SPI.transfer(10, green, SPI_CONTINUE); // send green byte
SPI.transfer(10, red); // send red byte
digitalWrite(37, LOW); // disable the outputs of the LED driver chips

Easy sensor functions
• Send 16-bit word over SPI to touch sensor to set the analog switches
void selectSensorCell(byte col, // Column used by analog switches
byte row, // Row used
byte switch) // Switch to read X (0), Y (1) or Z (2)
• Read stable raw X value at the current col and row (returns 0-4095)
int readX()
• Read stable raw Y value at the current col and row (returns 0-127)
int readY()
• Read stable raw Z value at the current col and row (returns 0-127, 255 feather)
byte readZ()

Details of touch sensor control
• Touch sensor control is done through SPI using pin 4,
with mode 0, running at 21MHz
SPI.begin(4);
• Write 16-bit data to SPI to set analog switches (see ls_sensor.ino)
SPI.transfer(4, lsb, SPI_CONTINUE); // first byte of data structure
SPI.transfer(4, msb); // second byte of data structure

Read touch sensor data
• Touch sensor A/D input is using SPI through pin 52,
with mode 0, running at 21MHz
SPI.begin(52);
• Read sensor data
delayUsec(7); // wait for stable current after sensor
// control changes the analog switches
// delay different for each analog switch
byte msb = SPI.transfer(4, 0, SPI_CONTINUE); // first byte of sensor data
byte lsb = SPI.transfer(4, 0); // second byte of sensor data
int raw = (int(msb) << 8 | lsb) >> 2; // pack into int, shift from 16 to 14 bit

Reading the foot pedals
• Done in ls_switches.ino, modify this method to add custom behavior
void handleFootSwitchState(byte whichSwitch, boolean state)
• Digital pin 33 and 34, respectively for left and right foot switches, configured as pull-up
inputs (inverted inputs: high is off, low is on)
pinMode(33, INPUT_PULLUP);
pinMode(34, INPUT_PULLUP);
• Read the digital value of foot pedal states (typically every 20ms)
boolean leftPedalState = digitalRead(33);
boolean rightPedalState = digitalRead(34);

Details of MIDI / Serial - USB / DIN
• Setting digital switches changes the communication methods from the LinnStrument
to the outside world
• Digital pin 35 switches between Serial and MIDI
digitalWrite(35, HIGH); // high switches to Serial input/output
digitalWrite(35, LOW); // low switches to MIDI input/output
• Digital pin 36 switches between USB and DIN connectors
digitalWrite(36, HIGH); // high switches to USB input/output
digitalWrite(36, LOW); // low switches to DIN input/output

That’s all the hardware stuff!

Uncomment line in ls_debug.h
Recompile and upload

Enable Update OS mode
Which essentially turns on Serial instead of MIDI

Click on the Looking Glass icon
To enable Serial Monitor and reset LinnStrument

Column 17 selects the debug level
By default it’s -1, meaning that no debug info is printed

This is printed in the Serial Console
Along with the actual debug messages of corresponding levels

The relevant debug code
if (sensorCol == 17 && sensorRow < 4) {
debugLevel = sensorRow - 1;
DEBUGPRINT((-1,"debugLevel = "));
DEBUGPRINT((-1,debugLevel));
DEBUGPRINT((-1,"n"));
}
void handleNewTouch(byte z) {
DEBUGPRINT((1,"handleNewTouch"));
DEBUGPRINT((1," col="));DEBUGPRINT((1,(int)sensorCol));
DEBUGPRINT((1," row="));DEBUGPRINT((1,(int)sensorRow));
DEBUGPRINT((1," z="));DEBUGPRINT((1,(int)z));
DEBUGPRINT((1,"n"));
void handleTouchRelease() {
DEBUGPRINT((1,"handleTouchRelease"));
DEBUGPRINT((1," col="));DEBUGPRINT((1,(int)sensorCol));
DEBUGPRINT((1," row="));DEBUGPRINT((1,(int)sensorRow));
DEBUGPRINT((1,"n"));

Core Files
• linnstrument-firmware.ino: global data structures, setup and main loop
• ls_displayModes.ino: illuminate LEDs for the different display modes
• ls_handleTouches.ino: driven by main loop, handles touch tracking
• ls_rtos.ino: primitive scheduler calling continuous tasks during delay
• ls_settings.ino: switch behavior from UI, store and recall settings
• ls_touchInfo.ino: encapsulate sensor data into touched cells

Low-level Files
• ls_calibration.ino: calibration procedure and data conversion
• ls_leds.ino: low-level communication with the LEDs
• ls_midi.ino: MIDI input, NRPN control, clock, output and queue
• ls_sensor.ino: low-level touch sensing with bias and curve
• ls_test.ino: debug functions and low-level reports

Auxilliary Features Files
• ls_arpeggiator.ino: arpeggiator logic, tied to internal MIDI clock
• ls_faders.ino: MIDI CC faders touch handling and data sending
• ls_font.ino: tiny, small, and big font display, including scrolling
• ls_lowRow.ino: continuous cell evaluation for low-row features, driven by
the main touch tracking
• ls_noteTouchMapping.ino: track MIDI notes to touched cells, mainly used
by arpeggiator
• ls_switches.ino: handles control switches and foot pedals

Support Header Files
• ls_bytebuffer.h: circular byte buffer with independent push and pop
locations, used by ls_midi.ino output queue
• ls_channelbucket.h: hands out MIDI channels from a bucket of
allowed channel numbers.
• ls_debug.h: debug macros and defines
• ls_midi.h: standard MIDI status codes, used by ls_midi.ino

Global variables
• byte sensorCol: the column number of the current sensor cell
• byte sensorRow: the row number of the current sensor cell
• byte sensorSplit: the split of the current sensor cell (0: left, 1: right)
• DisplayMode displayMode: the active display mode
(see DisplayMode enum in linnstrument-firmware.ino for the values)

Global functions
• TouchInfo &cell(): the last touch data for the current cell
• TouchInfo &cell(byte col, byte row): the last touch data for a cell
• FocusCell &focus(byte split, byte channel): the current cell that has
the focus for a particular split and channel (FocusCell is a structure
that just contains col and row)

Some specific firmware
examples

Per-finger touch-tracking
• With a general purpose CPU, you’d model this as touch ‘sessions’
that are dynamically created and have a life of their own
• Too much memory churn for Arduino, too much book-keeping also
• Instead, have a touch state for each cell
• Transfer data between cells since we don’t support two fingers
touching the same cell

Inside ls_handleTouches.ino
void handleNewTouch(byte z) {
// ... snip ...
// check if the new touch could be an ongoing slide to the right
if (potentialSlideTransferCandidate(sensorCol-1)) {
// if the pressure gets higher than adjacent cell,
// the slide is transitioning over
if (isReadyForSlideTransfer(sensorCol-1)) {
transferFromSameRowCell(sensorCol-1);
handleXYZupdate(z);
}
// otherwise act as if this new touch never happened
else {
cellTouched(transferCell);
}
}
// similar for slide to the left

Check potential slide transfer
boolean potentialSlideTransferCandidate(int col) {
if (col < 1) return false;
if (sensorSplit != getSplitOf(col)) return false;
if (!isLowRow() && (!Split[sensorSplit].sendX ||
!isFocusedCell(col, sensorRow))) return false;
if (isLowRow() && !lowRowRequiresSlideTracking()) return false;
if (isStrummingSplit(sensorSplit)) return false;
// the sibling cell has an active touch
return cell(col, sensorRow).touched != untouchedCell &&
// either a release is pending to be performed, or
(cell(col, sensorRow).pendingReleaseCount ||
// both cells are touched simultaneously on the edges
abs(cell().calibratedX() - cell(col, sensorRow).calibratedX())
< TRANSFER_SLIDE_PROXIMITY);
}

Is the sibling cell ready for the transfer?
boolean isReadyForSlideTransfer(int col) {
// there's a pending release waiting
return cell(col, sensorRow).pendingReleaseCount ||
// the cell pressure is higher
cell().rawZ > cell(col, sensorRow).rawZ;
}

Perform the data transfer 1/3
void transferFromSameRowCell(byte col) {
cell().initialX = cell(col, sensorRow).initialX;
cell().initialReferenceX = cell(col, sensorRow).initialReferenceX;
cell().lastMovedX = cell(col, sensorRow).lastMovedX;
cell().fxdRateX = cell(col, sensorRow).fxdRateX;
cell().rateCountX = cell(col, sensorRow).rateCountX;
cell().initialY = cell(col, sensorRow).initialY;
cell().note = cell(col, sensorRow).note;
cell().channel = cell(col, sensorRow).channel;
cell().fxdPrevPressure = cell(col, sensorRow).fxdPrevPressure;
cell().fxdPrevTimbre = cell(col, sensorRow).fxdPrevTimbre;
cell().velocity = cell(col, sensorRow).velocity;
cell().vcount = cell(col, sensorRow).vcount;
noteTouchMapping[sensorSplit]
.changeCell(cell().note, cell().channel, sensorCol, sensorRow);

if (cell(col, sensorRow).touched != untouchedCell) {
cell(col, sensorRow).touched = transferCell;
}
cell(col, sensorRow).initialX = -1;
cell(col, sensorRow).initialReferenceX = 0;
cell(col, sensorRow).lastMovedX = 0;
cell(col, sensorRow).fxdRateX = 0;
cell(col, sensorRow).rateCountX = 0;
cell(col, sensorRow).initialY = -1;
cell(col, sensorRow).note = -1;
cell(col, sensorRow).channel = -1;
cell(col, sensorRow).fxdPrevPressure = 0;
cell(col, sensorRow).fxdPrevTimbre = 0;
cell(col, sensorRow).velocity = 0;
cell(col, sensorRow).pendingReleaseCount = 0;
// do not reset vcount!

// transfer the focus if this was the focused cell
byte channel = cell().channel;
if (channel != -1 && col == focus(sensorSplit, channel).col &&
sensorRow == focus(sensorSplit, channel).row) {
focus(sensorSplit, channel).col = sensorCol;
focus(sensorSplit, channel).row = sensorRow;
}
}

Finding the right MIDI channels
• Note-per-channel distributes up to 16 MIDI channels across notes
• Notes should reuse same channel as late as possible (release trails)
• Intuitively you’d scan all the active notes and determine which channel is
available for a new note, which is again too much overhead
• We use a bucket of available channels, channels bubble up or sink down
• A new note merely has to take the next channel from the top
• Fully encapsulated inside ls_channelbucket.h

Low-row functionalities
• Intuitively you’d detect a touch on low-row cells when it’s active
• Then evaluate state of every other cell and trigger behavior
• This is again too much overhead
• Instead keep track of low-row start/stop in a state machine
• Piggy-back when processing each cell in the main loop to evaluate
appropriate low-row behavior

Inside ls_handleTouches.ino
void handleXYZupdate(byte z) {
// ... snip ...
handleLowRowState(z);
if (isLowRow()) {
if (newVelocity) {
lowRowStart();
}
return;
}
// ... snip ...
}
void handleTouchRelease() {
// ... snip ...
if (isLowRow()) {
lowRowStop();
}
// ... snip ...
}

Inside ls_lowRow.ino 1/2
void lowRowStart() {
switch (Split[sensorSplit].lowRowMode) {
case lowRowStrum:
lowRowState[sensorCol] = pressed;
break;
// ... snip, different for each low-row mode
}
}
void lowRowStop() {
case lowRowStrum:
lowRowState[sensorCol] = inactive;
break;
}
}

Inside ls_lowRow.ino 2/2
void handleLowRowState(byte z) {
// this is a low-row cell
if (isLowRow()) {
// send out the continuous data for low-row cells
if (cell().velocity) {
}
}
// this is a non low-row cell
else {
case lowRowStrum:
// uses lowRowState to correlate with column
handleLowRowStrum();
break;
// ... snip, other cases
}
}
}

Sending MIDI bytes
• MIDI was causing the LEDs to flicker
• Too much time was spent at once (need more guerrilla!)
• Created a MIDI queue to continuously send byte-by-byte from our
RTOS
• Arduino UART classes still caused problems: synchronous wait for
readiness when sending

Patched UARTClass
--- UARTClass.cpp 2014-11-10 14:55:10.000000000 +0100
+++ UARTClass.cpp 2014-10-10 19:39:43.000000000 +0200
@@ -109,9 +109,15 @@
size_t UARTClass::write( const uint8_t uc_data )
{
+ return write(uc_data, true);
+}
+
+size_t UARTClass::write( const uint8_t uc_data, const bool wait )
+{
// Check if the transmitter is ready
- while ((_pUart->UART_SR & UART_SR_TXRDY) != UART_SR_TXRDY)
- ;
+ while ((_pUart->UART_SR & UART_SR_TXRDY) != UART_SR_TXRDY) {
+ if ( !wait ) return 0;
+ }
// Send character
_pUart->UART_THR = uc_data;

Queuing of messages
ByteBuffer<4096> midiOutQueue;
// called for each MIDI message that is sent
void queueMidiMessage(MIDIStatus type, byte param1, byte param2, byte channel) {
param1 &= 0x7F; param2 &= 0x7F;
midiOutQueue.push((byte)type | (channel & 0x0F));
midiOutQueue.push(param1);
if (type != MIDIProgramChange && type != MIDIChannelPressure) {
midiOutQueue.push(param2);
}
}
// continuously called by our RTOS
void handlePendingMidi() {
if (!midiOutQueue.empty()) {
if (Serial.write(midiOutQueue.peek(), false) > 0) { // patched UART method
midiOutQueue.pop();
}
}
}

More information at
http://www.rogerlinndesign.com
Questions?
@gbevin

LinnStrument : the ultimate open-source hacker instrument

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to LinnStrument : the ultimate open-source hacker instrument

Similar to LinnStrument : the ultimate open-source hacker instrument (20)

Recently uploaded

Recently uploaded (20)

LinnStrument : the ultimate open-source hacker instrument