In a nut shell:
* What: Mind controlled video installation
* Why: To Investigate human experience: symbiosis between the emotive nature of stimuli and technological quantification/adaptation, manipulating & creating new stimuli
* Where: Inside a 360 degree ‘Immersive cinema’, a concave screen also used as a planetarium. Plymouth, England.
* When: Project completed 19 June 2009
* How: Main Software = PureData + Biotrace and Python code. Technologies = Electroencephalography + other biometric ‘readers’ +Bluetooth + Networking. Hardware= Nexus device + electrodes + digital dome cinema + 2 PC’s + Photographic and video equipment + patience (limitless resource required)
Using EEG electrodes attached to one’s head the viewer can interact with an immersive video installation via brain waves and stress levels. Different video clips and audio clips play dependant on the immersants individual brain waves and stress levels.
What the Immersant is thinking or feeling is reflected in the video and sounds for other people who were inside the dome.
The Final version requires 3 instances of Pd run at the same time communicating via a local network. I have also use limiters on every single audio channel, along with delays; this helps save CPU usage and lets the project run well. One of the final (and most simple) things was reducing the pixel res by 50%… GEM seems a lot happier than when it had to cope with 1400 x 1050 video.
I’m using a a high and low pass filter, a decibel limiter, and an fx.limiter object…all of which will hopefully protect speakers from breaking. I can’t seem to erradicate the feedback loop that occurs from having the mic on (maybe there is no solution though).
Unfortunately I’m having to delete some parts of my patch, as it seems that some EEG frequencies are not a viable source for interaction (regrettably). I’m keeping parts that have an in-depth use of SMR and Theta frequencies, as there is more control over them (and comparatively free from erraticism).
I have also been working more on the visuals. There have been various problems with running multiple videos and audio, I am now using MPEG’s and may have a way to switch the right ones off and on depending on GSR/SMR and Theta rhythms.
The video that i have shot, hopefully will evoke calm or stressful feelings. I have tried to go for a film noir (or more accurately neo-noir aesthetic), as it would seem that the dome would lend it self well to it. There should be less light-scattering and better contrast, and this look seems to have been ignored so far in dome content.
Maybe this is possible, if so using the same tactics within the immersive vision theatre could be key to providing an increased sense of immersion.
Today i have been looking more at PD (pure data) and especially Open Sound Control (osc), allowing me to send a data stream (bio-sensory data) from one pc via a python script to another. The other pc in turn picks up the signal with PD and OSC. There is a successful (yet seemingly temperamental) connection, providing everything is set up right.
My next plan of action was to get the data stream controlling some visual element; GEM is the visual element that is built into PD and having not really managed to get the stream to control much other than the pure data terminal window it would seem i have an awful lot to learn, and not a huge amount of time to learn it.
I started experimenting with GEM, this patch below i built using DjCypod’s tutorial on video mixing:
It plays two separate movie files, one is a Tomb Raider Underworld advertisement, one is me on a bmx jumping into lake Taupo, NZ. The central slider fades between either one. Certain more usefull aplications of this patch could include fading between live video and pre-recorded video, as a way to blend the past and the present, or the real and hyper-real etc…
Below is a screen shot of the GEM window in action:
As part of the research for my dissertation, I went along to see ‘The Mind Cupola’ an “Affective Environment” it is an installation that has built in heat sensor equipment. It also has face recognition software, to allow the viewer to “passively” interact with a visualization in front of them in order to achieve a state of ‘equilibrium’. Sound is generated by the heat from the person, and fans are built into the machine to heat or cool the person to help them achieve a neural state.
This is an example of an immersive environment different to the dome, and using different modalities than i will attempt to use.
Equipment running, signals being gathered… Seems to make sense… now for the (first) hard part, putting it into PD/Processing.
I downloaded ‘Pure Data’ this evening, which is an open source application that is similar to max MSP… I have done one video tutorial. For a start it sounds atrocious, as expected to be honest… but it barley does what its supposed to according to the tutorial either… So headache inducing all round! Oh dear, still a long way to go!
I want to investigate developments in neuroscience technology and technologies within immersive environments. I want to eventually create a user experience in Plymouth’s Immersive Vision Theatre where by the user will be able to change the visualisations on the screen with nothing but his or her own reaction. Is it possible for technology to read peoples minds? Can ones mental state affect the physical realm? Is that persons mental state conscious will or subconscious reaction?
I have been reading a book called “A Brief Tour Of Human Consciousness” by Vilayanur S. Ramachandran which deals with why the brain acts a certain way and he argues that:
“your concious life in short, is nothing but a post-hoc rationalization of things you really do for other reasons” ( Ramachandran, 2004)
I would like to investigate the subconscious mind, and how this relates to our conscious decisions and reactions. I wish to develop my project within a phycological and perhaps sociological context, including the different variables that may lead to changing the visualisations up on screen, such as a crowd in the dome at one time, or a single user un-influenced by anyone around them. I am also interested in philosophical concepts, such as free will, qualia and metaphysics. Using this space as one form of interactive architecture I am interested in the user unique user experience…how certain elements may lead to a different visualisation and then will the altered visualisation produce further brain activity and reactions that will change the look once again.
The Immersive media and neuroscience are moving forward very rapidly, with a lot of research being conducted and a lot of developments are being made.
I want to use biofeedback (such as EEG or galvanic skin responses) as an input device to change the visualisation by reading the reaction of the user. The device will be dependant on the research from my dissertation.
Possible software I may need to use may include are processing, vvvv, arduino or quarts composer a to create this real time user experience…
