Tag Archives: MIT

Echoes from the Future of Opera: A Mashup of Music and Technoloy at the MIT Media Lab

On Saturday, my classmate Emily Elert and I visited the Opera of the Future research group at the MIT Media Lab. Graduate student Elena (Elly) Jessop was kind enough to spend the day with us and demonstrate some really neat fusions of music and technology.

Now, I can’t talk in detail about absolutely everything we saw, because some of it is not yet ready for the public’s eye. To respect the Media Lab and the researchers who work there, I will only discuss what can already be found online in some form or another. I’ll be linking out a lot in case you’re interested in reading more.

The Opera of the Future group, headed by innovative composer Tod Machover, “explores concepts and techniques to help advance the future of musical composition, performance, learning, and expression.” Their work includes Hyperinstruments, traditional musical instruments enhanced by technology to actively interact with performers: as the musicians play, sensors and computers allow the instruments to interpret the nuances of live performance and accordingly change the sounds they produce, creating a new spectrum of sonic power, range and finesse. Hyperscore is a computer program that translates complex musical concepts into intuitive and colorful visual representations, making relatively sophisticated music composition a possibility for those with no formal training.

Tod Machover playing a hypercello (Credit: Tod Machover, MIT)

A large ongoing project is Death and the Powers, a new opera that makes unprecedented use of technology to modify live musical performance.  The creative team includes Machover (composer), former US Poet Laureate Robert Pinsky (librettist), Artistic Director of the American Repertory Theater Diane Paulus (director), and Hollywood’s Alex McDowell (production designer), who has worked on such visually stunning films as Fight Club, Minority Report and Watchmen.

Proposed Set for Death and the Powers

Proposed Set for Death and the Powers (Credit: Opera of the Future)

The opera – currently scheduled to premiere in Monaco in September, 2010 – centers on the rich and powerful businessman Simon Powers, who dies in the second act. Powers, however, has uploaded his consciousness onto a massive computer called The System, so his essence persists throughout the performance. The stage, which largely represents the Powers household, comes alive as Simon Powers learns to manipulate the external world in his new computerized existence. Power’s daughter Miranda, his wife Evvy, and his research assistant Nicholas struggle to cope with his posthumous influence and understand the choices he made.

To achieve this transcendent effect, researchers at the Opera of the Future Group – in particular Peter Torpey – have developed what they call Disembodied Performance, a collection of technological innovations that allow an absent performer to remain actively involved in a live performance. Although Powers won’t be directly visible, his voice will be heard, his emotions will be felt, his reactions will be known. All kinds of sensors on the singer and actor playing the role of Simon Powers will wirelessly transmit data to a central computer, where the data will be used to modify the operatic performance in real time. Large bookcases onstage will present the audience with waves of color to reflect Powers’s mood, bristling in anger when a character walks by that Powers doesn’t like, for example. Ambisonic sound – which reverberates in any and every direction – will wrap the audience in song, placing them at the very center of Powers’s voice. Many-legged furniture – inspired by the beautiful kinetic sculptures of Theo Jansen – will scurry across the stage.

Elly Jessop has been working on a remarkable device for the opera called the Vocal Augmentation and Manipulation Prosthesis (VAMP): a glove that gives its wearer the power to modify their singing voice in real time using a gesture-based vocabulary.


Vocal Augmentation and Manipulation Prosthesis (Credit: Elly Jessop)

Currently, VAMP is a long fabric glove outfitted with pressure sensors and accelerometers that wirelessly transmit data to a computer, which interprets the gestures and – working with a microphone and speakers – changes the singer’s voice. If you want to preserve a note you’re singing for example, you use a plucking gesture to pull the note from your mouth, which signals the computer to continue playing that note as you sing something else. This allows for vocal layering and textured singing. Bending of the elbow controls volume and shaking your hand induces vibrato. You can watch Elly’s fantastic demo on her web site.

VAMP is specifically designed for Nicholas, Simon Power’s research assistant, who – according to the script – has a prosthetic arm than endows him with special abilities. Since their work is for an opera, Elly and her colleagues decided that the arm should give Nicholas special vocal abilities: thus, VAMP. Eventually the glove will look far more like a robotic arm than it does now.

Other technological innovations designed for Death and the Powers include a group of advanced animatronic-like robots, ranging in height from four to seven feet, with giant triangular lights for heads. These robots, which will be partly pupeteered, can sense and avoid one another, perform semi-autonomous movement, and may eventually have their own voices. Within the opera’s narrative framework, they function as a kind of Greek chorus, perpetually retelling the story of Simon Powers in a choreographed pageantry. The also serve as sculptural set pieces and, since they’re so bright, as stage lights.

Stage Robot

Stage Robot for Death and The Powers (Credit: Opera of the Future)

Another example of music and technology working in harmony is a giant stringed instrument called The Chandelier, which can be stimulated by both a human player and by electromagnets, allowing for a much greater range of notes than a standard string instrument. The Chandelier provides the opportunity for a powerful duet between Powers and his wife, after his physical death – another form of disembodied communication.

The Chandelier

The Chandelier (Credit: Opera of the Future)

After the 2010 premiere in Monaco, a US and World tour is planned. I for one can’t wait to see how it turns out.


Illuminating the Secrets of Mind Control in Light of the Coming Singularity

This past weekend, the 92nd Street Y hosted the 2009 Singularity Summit. Maybe you’ve already heard of Ray Kurzweil and his widely cited book The Singularity is Near, but here’s the gist just in case: Kurzweil and many others claim the exponentially increasing rate of technological development is evidence for a shift in the near future —a shift called the Singularity—during which artificial intelligence will surpass human intelligence and machines will supersede humans as the dominant sentient forces on the planet. If this sounds like something out of The Matrix and more than a little kooky—don’t worry, you’re not alone. But the Singularity Summit wasn’t just an excuse for enthusiastic futurists and computer science geeks to stand up on a soapbox and spout speculation—the Summit welcomed a diverse range of scientists whose presentations described some really fascinating current research. One that caught my attention in particular was Ed Boyden of MIT.

Boyden’s speech focused on synthetic neurobiology—a field in which researchers create technology that interacts directly with the brain—and a remarkable technique for specifically manipulating individual neurons, a technique Boyden helped pioneer. Many organisms—such as jellyfish, algae and bacteria—produce light-activated protein pumps. Using harmless viruses as vehicles to transport DNA coding for these proteins from one organism to another, researchers can make neurons in the animals they study light-sensitive as well. What’s more, by using different kinds of protein pumps—one which excites neurons in response to blue light and one which inhibits them in response to yellow light—they can precisely determine whether the neurons fire or not. Eager for an unprecedented level of control over different parts of the brain and nervous system, researchers have readily adopted the technique, successfully applying it to a range of animal models, from zebra fish to primates.

So what does any of this have to do with the Singularity? Well, during his presentation at the Summit, Boyden described attempts to make fiber optic implants for the human brain, implants that could directly stimulate or inhibit neurons with light. Let’s think about this: brain implants that precisely determine whether our neurons are firing or not? Sure, there’s great therapeutic potential here—especially for diseases that involve abnormal firing patterns, like epilepsy and Parkinson’s—but there’s also something a bit alarming. The technology Boyden described is similar to deep brain stimulation (DBS)—in which an implanted brain pacemaker regulates specific areas of neurons—but there is a crucial difference: present day DBS uses electrical stimulation, which is not nearly as precise as light stimulation. The more sophisticated the technology with which we study the brain becomes, the more we learn about the brain’s function and the better we become at treating psychological disorders; on the other hand, one can’t help but imagine how precise control of individual neurons could turn into the kind of mind control science fiction has long warned us against. Scarily, fiber optics have already been used to stimulate the brain in mice, as this little fellow demonstrates: when the light goes on, he involuntarily runs in circles:

If Kurzweil is right—and computers will soon be smarter than us—I’m sure they’ll take full advantage of any mind-controlling technology at their disposal. Who knows: maybe the Singularity already happened and we’re nothing more than a bunch of brains in vats, lapping up the rays of light that power our dream reality—all for the amusement of our supercomputer overlords.