MIT Students Turn Internet Into a Sixth Human Sense

Students at the MIT Media Lab have developed a wearable computing system that turns any surface into an interactive display screen. The wearer can summon virtual gadgets and internet data at will, then dispel them like smoke when they're done.

Pattie Maes of the lab's Fluid Interfaces group said the research is aimed at creating a new digital "sixth sense" for humans.

In the tactile world, we use our five senses to take in information about our environment and respond to it, Maes explained. But a lot of the information that helps us understand and respond to the world doesn't come from these senses. Instead, it comes from computers and the internet. Maes' goal is to harness computers to feed us information in an organic fashion, like our existing senses.

The prototype was built from an ordinary webcam and a battery-powered 3M projector, with an attached mirror -- all connected to an internet-enabled mobile phone. The setup, which costs less than $350, allows the user to project information from the phone onto any surface -- walls, the body of another person or even your hand.

Maes showed a video of her student Pranav Mistry who she describes as the brains behind the project. Mistry wore the device on a lanyard around his neck, and colored Magic Marker caps on four fingers (red, blue, green and yellow) helped the camera distinguish the four fingers and recognize his hand gestures with software that Mistry created.

The gestures can be as simple as using his fingers and thumbs to create a picture frame that tells the camera to snap a photo, which is saved to his mobile phone. When he gets back to an office, he projects the images onto a wall and begins to size them.



When he encounters someone at a party, the system projects a cloud of words on the person's body to provide more information about him -- his blog URL, the name of his company, his likes and interests. "This is a more controversial [feature]," Maes said over the audience's laughter.

In another frame, Mistry picks up a boarding pass while he's sitting in a car. He projects the current status of his flight and gate number he's retrieved from the flight-status page of the airline onto the card.

"If you need to know what time it is, it's as simple as drawing a watch on your arm," Maes said, while Mistry used his right finger to draw a circle on his left wrist. The face of a watch popped up on his hand, which the audience liked.

When Mistry folds his hands in "namaste" fashion, the system opens a menu to allow him to choose an application. If he wants to read e-mail on his phone, he draws an @ symbol in the air with his finger. He can project a phone pad onto his palm and dial a number without removing the phone from his pocket. As he reads the newspaper on the subway he can project a video onto the page that provides more information about the topic he's reading.

Maes and Mistry told Wired they've been working on the project for four months, day and night, and have filed a patent for it.

Maes' MIT group, which includes seven graduate students, were thinking about how a person could be more integrated into the world around them and access information without having to do something like take out a phone. They initially produced a wristband that would read an RFID tag to know, for example, which book a user is holding in a store.



They also had a ring that used infrared to communicate by beacon to supermarket smart shelves to give you information about products. As you grab a package of macaroni, the ring would glow red or green to tell you if the product was organic or free of peanut traces -- whatever criteria you program into the system.

"We wanted to make information more useful to people in real time with minimal effort in a way that doesn't require any behavior changes," Maes said. "The wristband was getting close, but you still had to take out your cell phone to look at the information."

That's when they struck on the idea of accessing information from the internet and projecting it. So someone wearing the wristband could pick up a paperback in the bookstore and immediately call up reviews about the book, projecting them onto a surface in the store or doing a keyword search through the book by accessing digitized pages on Amazon or Google books.

They started with a larger projector that was mounted on a helmet. But that proved cumbersome if someone was projecting data onto a wall then turned to speak to friend -- the data would project on the friend's face. Last month, they switched to a smaller projector and created the pendant prototype to be worn around the neck.

The TED demo was the first time they've shown it in public, though they're far from making a commercial product or forming a company around their invention. "But we're really excited about the potential," Maes said.

They learned recently that cellphone makers soon plan to release cellphones with projectors integrated in them, which will simplify their system even more.