Tag Archives: computers

Echoes from the Future of Opera: A Mashup of Music and Technoloy at the MIT Media Lab

On Saturday, my classmate Emily Elert and I visited the Opera of the Future research group at the MIT Media Lab. Graduate student Elena (Elly) Jessop was kind enough to spend the day with us and demonstrate some really neat fusions of music and technology.

Now, I can’t talk in detail about absolutely everything we saw, because some of it is not yet ready for the public’s eye. To respect the Media Lab and the researchers who work there, I will only discuss what can already be found online in some form or another. I’ll be linking out a lot in case you’re interested in reading more.

The Opera of the Future group, headed by innovative composer Tod Machover, “explores concepts and techniques to help advance the future of musical composition, performance, learning, and expression.” Their work includes Hyperinstruments, traditional musical instruments enhanced by technology to actively interact with performers: as the musicians play, sensors and computers allow the instruments to interpret the nuances of live performance and accordingly change the sounds they produce, creating a new spectrum of sonic power, range and finesse. Hyperscore is a computer program that translates complex musical concepts into intuitive and colorful visual representations, making relatively sophisticated music composition a possibility for those with no formal training.

Tod Machover playing a hypercello (Credit: Tod Machover, MIT)

A large ongoing project is Death and the Powers, a new opera that makes unprecedented use of technology to modify live musical performance.  The creative team includes Machover (composer), former US Poet Laureate Robert Pinsky (librettist), Artistic Director of the American Repertory Theater Diane Paulus (director), and Hollywood’s Alex McDowell (production designer), who has worked on such visually stunning films as Fight Club, Minority Report and Watchmen.

Proposed Set for Death and the Powers

Proposed Set for Death and the Powers (Credit: Opera of the Future)

The opera – currently scheduled to premiere in Monaco in September, 2010 – centers on the rich and powerful businessman Simon Powers, who dies in the second act. Powers, however, has uploaded his consciousness onto a massive computer called The System, so his essence persists throughout the performance. The stage, which largely represents the Powers household, comes alive as Simon Powers learns to manipulate the external world in his new computerized existence. Power’s daughter Miranda, his wife Evvy, and his research assistant Nicholas struggle to cope with his posthumous influence and understand the choices he made.

To achieve this transcendent effect, researchers at the Opera of the Future Group – in particular Peter Torpey – have developed what they call Disembodied Performance, a collection of technological innovations that allow an absent performer to remain actively involved in a live performance. Although Powers won’t be directly visible, his voice will be heard, his emotions will be felt, his reactions will be known. All kinds of sensors on the singer and actor playing the role of Simon Powers will wirelessly transmit data to a central computer, where the data will be used to modify the operatic performance in real time. Large bookcases onstage will present the audience with waves of color to reflect Powers’s mood, bristling in anger when a character walks by that Powers doesn’t like, for example. Ambisonic sound – which reverberates in any and every direction – will wrap the audience in song, placing them at the very center of Powers’s voice. Many-legged furniture – inspired by the beautiful kinetic sculptures of Theo Jansen – will scurry across the stage.

Elly Jessop has been working on a remarkable device for the opera called the Vocal Augmentation and Manipulation Prosthesis (VAMP): a glove that gives its wearer the power to modify their singing voice in real time using a gesture-based vocabulary.


Vocal Augmentation and Manipulation Prosthesis (Credit: Elly Jessop)

Currently, VAMP is a long fabric glove outfitted with pressure sensors and accelerometers that wirelessly transmit data to a computer, which interprets the gestures and – working with a microphone and speakers – changes the singer’s voice. If you want to preserve a note you’re singing for example, you use a plucking gesture to pull the note from your mouth, which signals the computer to continue playing that note as you sing something else. This allows for vocal layering and textured singing. Bending of the elbow controls volume and shaking your hand induces vibrato. You can watch Elly’s fantastic demo on her web site.

VAMP is specifically designed for Nicholas, Simon Power’s research assistant, who – according to the script – has a prosthetic arm than endows him with special abilities. Since their work is for an opera, Elly and her colleagues decided that the arm should give Nicholas special vocal abilities: thus, VAMP. Eventually the glove will look far more like a robotic arm than it does now.

Other technological innovations designed for Death and the Powers include a group of advanced animatronic-like robots, ranging in height from four to seven feet, with giant triangular lights for heads. These robots, which will be partly pupeteered, can sense and avoid one another, perform semi-autonomous movement, and may eventually have their own voices. Within the opera’s narrative framework, they function as a kind of Greek chorus, perpetually retelling the story of Simon Powers in a choreographed pageantry. The also serve as sculptural set pieces and, since they’re so bright, as stage lights.

Stage Robot

Stage Robot for Death and The Powers (Credit: Opera of the Future)

Another example of music and technology working in harmony is a giant stringed instrument called The Chandelier, which can be stimulated by both a human player and by electromagnets, allowing for a much greater range of notes than a standard string instrument. The Chandelier provides the opportunity for a powerful duet between Powers and his wife, after his physical death – another form of disembodied communication.

The Chandelier

The Chandelier (Credit: Opera of the Future)

After the 2010 premiere in Monaco, a US and World tour is planned. I for one can’t wait to see how it turns out.


Insect Cyborg Sentinels: DARPA’s Dream for Remote Control Bugs Generates Buzz

Since before the dawn of time, man has relied on his animal brethren. But technology threatens to make animals obsolete. The car replaced the horse-drawn carriage. E-mail murdered millions of carrier pigeons. And before we had JetBlue and Virgin Atlantic, we relied primarily on penguins.

Today, science is bringing animals and technology back into harmony. Yesterday, at 5:23 PM, super secret government agents working in a subterranean laboratory six miles below Area 51, breathed life into an army of tiny six-legged Frankensteins. 70% living thing, 30% machine, 100% badass, the Pentagon’s army of insect cyborgs will soon be secretly spying through a window near you.

Hahaha, very funny—now let’s get serious. From this point forward, I am leaving all fictional embellishments and exaggerations behind and only reporting the truth of the matter. And, believe me, the truth is plenty bizarre on its own.

The United States Department of Defense has a pretty famous agency called the Defense Advanced Research Projects Agency (DARPA) whose primary goal is to create new technology for the government and military. As you might imagine, this agency funds some pretty interesting research at various universities and institutions across the country.

One of DARPA’s programs is called—brace yourself—Hybrid Insect Microelectromechanical Systems (HI-MEMS). So what on earth does that mean? As the HI-MEMS program says on its web site, their goal is to develop “tightly coupled machine-insect interfaces by placing micro-mechanical systems inside the insects during the early stages of metamorphosis.”

In other words, DARPA is trying to fuse living insects with tiny electronic implants. Why?

“The goal of the MEMS, inside the insects, will be to control the locomotion by obtaining motion trajectories either from GPS coordinates, or using RF, optical, ultrasonic signals based remote control. The control of locomotion will be investigated using several approaches. These include direct electrical muscle excitation, electrical stimulation of neurons, projection of ultrasonic pulses simulating bats, projection of pheromones, electromechanical stimulation of insect sensory cells, and presentation of optical cues with micro-optical visual presentation. The intimate control of insects with embedded microsystems will enable insect cyborgs, which could carry one or more sensors, such as a microphone or a gas sensor, to relay back information gathered from the target destination.”

Let’s parse this.

Step 1: combine living insect with tiny electronic chip.

Step 2: Figure out how to make that chip control the insect’s movement.

Step 3: Use the remote-controlled insect cyborg as a means of military surveillance.

Well that’s certainly intriguing, but just how feasible are these goals? It turns out researchers have made some surprising advances.

At Cornell University’s Laboratory for Intelligent Machine Systems, among other labs, electrical engineers have not only been able to control the wing movements of hawk moths using direct electrical stimulation, they have created devices that harvest the energy produced by the moth’s own vibrations during flight. If these devices could be improved to power the kind of very tiny GPS sensors, cameras and microphones that would be useful for military surveillance, the moths could be sent out as self-sufficient spies, easily accessing areas humans never could. What’s more, other Cornell researchers are developing a means of temporarily paralyzing the moths at a routine pit stop. Let’s say you needed the moth to stay still while you collect some data: you could give it a quick injection of spider venom—just enough to paralyze it—and then another higher dose to excite it and send it on its way again.

Adult female Hawk Moth (Manduca sexta)

Here’s a video of the hawk moth and energy harvesting device by Tim Reissman of Cornell:

And a video by New Scientist:

Researchers at the University of California, Berkeley, have just published a remarkable study describing how they remotely controlled free flying beetles. After implanting tiny electronic chips into the brains and muscles of adult Green June beetles and Giant Flower beetles, the researchers were able to initiate and stop flight, as well as make the beetles change direction and altitude. Interestingly, no one knows the specific neural pathways involved in the beetles’ flight. The implants stimulate relatively large areas of the brain, yet somehow enable a relatively sophisticated level of control.

The moths and beetles with which researchers have achieved the greatest success so far are pretty giant, as far as insects go. The Giant Flower beetle, for example, can grow as large as a human palm. So they’re not exactly the most inconspicuous surveillance units.


Giant Flower Beetles (Credit: pet_insects, photobucket)

Here’s a video of the beetles in controlled flight at Berkeley:

What’s perhaps more promising is the kind of information scientists are learning from this fantastical synthesis of insect and machine. First, researchers are getting access to data about living things they have never had before. Tiny chips that can measure the internal changes in insects as they go about their normal mating and feeding behaviors in the wild would prove invaluable for biologists. And the need to build machine parts small enough for insects to carry is pushing researchers to build smaller, lighter and more efficient electronics—with applications far beyond the insect world and DARPA’s dream of a buzzing surveillance squad.