BlindSight aims to explore synesthesia by associating certain body positions with visual hallucinations induced by photic stimulation. In other words, flashing lights at various frequencies seems to cause visual patterns to appear for the viewer. Simple wearable sensors adjust the frequency of the flashes and thus provide various patterns.

[download PDF]

Much of this project stems from two separate influences. First, William Gibson’s “Neuromancer” where he depicts the protagonist using electrodes/glasses to jack into cyberspace, which is depicted as a three-dimensional space largely visualized as colored geometric shapes. Secondly, my general jealousy of individuals whom describe themselves as synesthetes, those who experience certain senses crossing over, ie. seeing certain letters as colored or hearing certain visual motion.

An article in the journal Cortex (#44, May 2008,  Dominic H. ffytche), describes a study which employed flashing goggles to induce visual hallucinations for the purpose of analyzing brain activity with MRI. The study indicates that flashing frequencies between 5-30Hz at a certain intensity caused participants to describe perceiving hallucinations (Purkinje patterns) and produced a measurable change in the brain activity.

I’d like to use this visual stimulation to represent a small portion of our sense of proprioception. The ideal scenario would be to enable a user to generate an association between certain Purkinje patterns and the orientation of the body relative to itself. I’m working closely with Andrew Styer on the input devices and we’ve been discussing the best sensing for our end. Trying to visualize all of proprioception in this manner would be arrogant at best, so for this prototype we are looking to limit our focus to the upper body, possibly the arms alone.

Initial user testing of the glasses have provided strong reactions and most participants report perceiving patterns which change in response to frequency shift. Participants generally indicate that the experience is pleasant, however the prototype glasses may provide too much illumination intensity, overpowering the effect.

Next steps are to experiment with various sensors / inputs and to determine the levels of discernible patterns within the available frequency range. Another direction for exploration is the event as frequencies change. Several participants have expressed strong reactions when the patterns initially change. Perhaps alternating between frequencies could provide another dimension to the expressiveness of the glasses.

Corey Menscher and I worked together on ThinkBig for the Whole Body Interface exercise in Thinking Physically. Corey had the basic idea of making soft switches which would be foot activated. Initially he was going to make three which you’d have to skip between.

We had a discussion about other implementations of the idea, including a hopscotch-type arrangement but ended up riffing off the large keyboard depicted at FAO Schwartz in the movie Big. To this end we made 8 switches which would be mapped to notes on a musical scale. After more discussion we decided to allow the switches to be arbitrarily positioned, including on walls, and the actions would trigger sound samples rather than notes.

Building the soft switches was easy enough – two layers of conductive fabric with a layer of dense foam between them. We wanted to put them on flexible surfaces, but for the prototype affixed each to a small wooden block.

The class seemed to enjoy using them, especially when they were on the wall rather than the floor. The scale of the interface encouraged multiple simultaneous users to play, and the distance between each switch introduced a natural limit to how quickly you could move from one switch to the next. As with the Brauswitch the output was simple sound samples, but it would be interesting to experiment with other actions.

Video of the class trying it out below. Enjoy.

There is something really nice about the amplification of a small facial movement and the larger audio/visual response of the sketch. It’s also nice to interact in a handsfree way. Oh! Fun. Code after the video.

Arduino code:

/* brauswitch
* robert carlsen | robertcarlsen.net
* 2-2008
*
* the brauswitch is a headband mounted switch activated by raising the eyebrows.
* the prototype version is made from burlap with conductive fabric on opposite sides
* of a small gap in the headband just above the eyebrows. raising the eyebrows closes the gap
* which closes the switch. the position of the headband needs to be adjusted carefully for proper
* action - however once situated well the brauswitch works very consistently.
*
* this code sends a byte via serial when the switch is closed:
* 1, 2 or 3 for left, right and both switches respectively
*/

#define LEFT_BROW 8
#define RIGHT_BROW 9
#define LED 3

void setup() {
Serial.begin(9600);

pinMode(LEFT_BROW, INPUT);
pinMode(RIGHT_BROW, INPUT);
pinMode(LED, OUTPUT);

}

void loop(){
int var1 = digitalRead(LEFT_BROW);
int var2 = digitalRead(RIGHT_BROW);

byte msg = 0;

if(var1 == HIGH){
digitalWrite(LED, HIGH);
msg += 1;
delay(10);
digitalWrite(LED, LOW);
}

if(var2 == HIGH){
digitalWrite(LED, HIGH);
msg += 2;
delay(100);
digitalWrite(LED, LOW);
}

if(msg>0)
Serial.print(msg,BYTE);

delay(100);
}

Processing (java) code:

// Project:         brauswitch
// File:             Brauswitch.java
// Created by:         rcarlsen, Feb 21, 2009

// Imports
import processing.core.*;
import ddf.minim.*;
import processing.serial.*;

public class Brauswitch extends PApplet {
Serial myPort;

// holder for the incoming data
byte[] data = new byte[1];

// color array
int[] c = {0x33000000,0x33ff0000,0x3300ff00,0x330000ff};
int cIndex = 0;

int timer;
int timeout = 1000;

Minim minim;
AudioSnippet bothSound,leftSound,rightSound;

public void setup() {
size(500,300);
smooth();
background(0);
noStroke();

//println(Serial.list());
myPort = new Serial(this,Serial.list()[0],9600);

minim = new Minim(this);
bothSound = minim.loadSnippet("beat.wav");
leftSound = minim.loadSnippet("msgstart.wav");
rightSound = minim.loadSnippet("msgend.wav");
}

public void draw() {
// draw a partially transparent rect over the previous frame
fill(0x33000000);
rect(0,0,width,height);

// read the serial data is available
if(myPort.available()>0){
println(myPort.available() + " bytes available");

//only expecting one byte
data = myPort.readBytes();
myPort.clear();
}

// act on the read data. it will be 0,1,2,3
if(data[0]>0){
println("Data read: " + data[0]);
cIndex = data[0];
// clear the data
data[0] = 0;

switch(cIndex){
case 1:
if(!leftSound.isPlaying())
leftSound.loop(0);
break;
case 2:
if(!rightSound.isPlaying())
rightSound.loop(0);
break;
case 3:
if(!bothSound.isPlaying())
bothSound.loop(0);
break;
}

// keep the timer going
timer = millis();
}

// fade out if the brauswitch is open
if(millis() - timer > timeout){
cIndex = 0;
if(bothSound.isLooping())
bothSound.play(); //finish the sound and stop
if(leftSound.isPlaying())
leftSound.pause();
if(rightSound.isPlaying())
rightSound.pause();
timer = millis();
}

// draw the indicator ellipse
fill(c[cIndex]);
ellipseMode(CENTER);
ellipse(this.width/2, this.height/2,200,200);
}

public void stop()
{
// always close Minim audio classes
bothSound.close();
leftSound.close();
rightSound.close();
// always stop Minim before exiting
minim.stop();

super.stop();
}
}

some quick background. the touchscreen on the iPhone and iPod Touch (as well as the older click wheel iPods and trackpad on Macbook/Pro’s) use the capacitance of skin to track touches. gloves generally prevent these type of sensors from reading (except perhaps very thin gloves). taking off a glove to use the phone is frustrating, especially when trying to momentarily check something that would only take a few seconds (text messages, e-mails, etc).

there are gloves currently available which work in a variety of ways, but i already have gloves which i use – specifically glove liners that i wear under my flip-back mittens when riding a bike in the cold. (of course, i’m not advocating using the device while riding, but it’s nice to check an incoming message while waiting at a traffic light).

the solution i used was to sew two pads of conductive thread into the tips of the forefinger and thumb. they are simple, knit polypropylene gloves from performance bike but any material would likely work just the same. the thread goes through the cloth and touches the skin of the fingers inside the glove. anyway, they work as expected – super convenient.

The Brauswtich is a digital switch to be worn on the forehead, just above the eyebrows. Raising one or both eyebrows will close the switch. There is a small gap between the upper and lower portions of the headband. It is made with a heavy burlap; the upper portion is stiffer than the lower portion which generally moves less as the eyebrows are raised and enables the switching motion.

It was designed without a specific output in mind; the task was to work within the confines of the assigned body part – in this case the head and torso. In class we attached each of our switches to an Arduino +WaveSheild and used them to trigger sound effects.

When adjusted correctly, it takes a very subtle movement to activate the switch although larger movements will work as well.

Nothing groundbreaking here, but this was my first experience working with conductive fabric to build a wearable circuit. I may be hooked.

Corey wearing the Brauswitch.