FROM THE ARCHIVES: reality bites

Brand new posts are coming down the pike, but over the weekend I was on a panel at the State of the Arts Symposium in downtown LA, part of Vision Lab, held at Los Angeles Center Studios. I wasn't quite sure why I was on a panel on immersive media, art and entertainment, except that the Science & Entertainment Exchange recently hosted an event on the topic of 3D technology and augmented learning. My co-panelists were an impressive bunch, being heavily involved in 3D film, virtual worlds like Second Life, virtual gaming, immersive media, and "augmented reality." Yes, augmented reality is on the verge of commercialization, thanks to companies like Ogmento, who are looking into applying the technology to gaming. We're a ways off from achieving true augmented reality, but the panel reminded me of this 2008 blog post I wrote on mixed reality states — the goal of future gamers, if Ogmento is any indication. Enjoy!

We're in the Big Easy, baby! And we're just a wee bit frazzled because there's simply too much to do and see, both in and around Nawlins, and at the APS March Meeting. They were handing out Mardi Gras beads in the press room (minus the usual requirement of flashing one's chest), where conversations covered everything from Eliot Spitzer's disgrace and the future of science publishing, to which one of the myriad competing parallel sessions one should attend. Should one go
hear about block copolymer thin films or pyrochlores (rare earth metals)? Graphene transport or silicon photonics? Locomotion in complex fluids, or biological networks?

Or should one simply blow off an
afternoon and take in the local sites and sounds? It's tough to resist the temptation in an iconic city like New Orleans — one of the few towns (along with Montreal) where folks can pronounce my last name. Mostly, I have been working, although I did stop off at the legendary Cafe Du Monde for chicory-flavored cafe au lait and fresh beignets, and last night I indulged in a scrumptious dinner at Commander's Palace (in the Garden District) with Physics of NASCAR author Diandra Leslie-Pelecky — gumbo to die for, and we both heartily recommend the bread pudding souffle for dessert. Alas, my shellfish allergy is severely limiting my dining options around these parts, where crawfish, crab and other crustaceans appear in some form in virtually every dish. But we can deal. At least I'm not allergic to beignets, which would be tragic indeed.

Perhaps in some future virtual world, I will be able to savor all the forbidden shellfish-based delicacies with no ill effects, and it will be almost like the real thing, because, according to University of Illinois physicist Alfred Hubler, I will inhabit a "mixed reality" state where there is no clear boundary between the real system and the virtual system: "The line blurs between what's real and what isn't."  Really, he said that. In a charming Teutonic accent, no less. It reminded me of a scene in The Matrix, where that despicable turncoat Cypher is dining with the Agents and observes that even though he knows the meal isn't technically "real," it feels and tastes just like it's real, and if it's good enough to fool the human brain, it's good enough for him after all those years struggling aboard the rebel ship Nebuchadnezzar. "Ignorance is bliss!"

Personally, I prefer the real world, with all its imperfections, but it's tough to deny the appeal of virtual worlds like Second Life. Not only can you have cafe au lait and beignets in a virtual New Orleans, but your avatar can literally make like Neo and fly there, with no need for cramped, overcrowded airplanes and the indignities routinely inflicted at airport security checkpoints by the TSA, just because they can. Oh yes — we can all be The One, and wreak appropriate revenge on that over-zealous TSA employee in the Philly airport who confiscated the Spousal Unit's contact lens solution and my favorite lip gloss. Not that I'm bitter….

*ahem* Anyway, Hubler's just completed an experiment (the link is to his blog — how cool is Hubler?) that he believes supports the existence of mixed reality states, using a real system — in this case, a standard mechanical pendulum — coupled with a virtual system (a virtual pendulum) that was programmed to follow the well-known equations of motion. He and his  colleagues sent data about the real pendulum to the virtual one, while sending information about the virtual pendulum to a motor that influenced the motion of the real pendulum. They found that when the two pendulums were of different lengths, they remained in a "dual reality state" in which their motion was uncorrelated, and thus not synchronized.

That in itself is not especially enlightening. But then they discovered that when the pendulum lengths were similar, they reached a critical transition point and became correlated, or, in Hubler's words, "They suddenly noticed each other, synchronized their motions, and danced together indefinitely." It's a lot like a typical phase transition, in fact: the critical temperature/pressure point wherein matter moves from one state (gas) to another (liquid). In this case, the phase transition occurs when the boundary between reality and virtual reality disappears.

This is the "mixed reality" state, where a real pendulum and a virtual pendulum move together as one. The trick is real-time feedback. Scientists have coupled mechanical pendulums with springs to create correlated motion, but without the staggering computational speed now achievable, coupling pendulums with a virtual system simply hadn't been possible. Per Hubler: "Computers are now fast enough that we can detect the position of the real pendulum, compute the dynamics of the virtual pendulum, and compute appropriate feedback to the real pendulum, all in real time." [That's the quote from the press release, but he said almost exactly the same thing in yesterday morning's press conference.]

As flight simulations, immersive VR, and online virtual games and worlds become increasingly accurate in their depictions of the real world, Hubler believes such "mixed reality" states via such "bidirectional coupling" will become more common. They could be very useful. Lasers, for example, are useful because of the synchronization of molecular motion. Laser light is "coherent," and thus all the molecules "help each other." Hubler thinks his lab-induced mixed reality states could be used to better understand real complex systems with a large number of parameters, by coupling a real system to a virtual one until their constant interactions result in a mixed reality state — for instance, modeling neurons by coupling a real neuron with a virtual one.

Of course, Hubler admits that while there are benefits to be realized from such an effect, it could present problems. As an example, he pointed to the infamous wobbling of London's celebrated Millennium Bridge when it opened to much fanfare in June 2000. Thousands of pedestrians started walking across the bridge, and at first, nothing happened. Then, the bridge began to sway slightly until, quite suddenly, the wobble intensified to such an extent that people started walking in near-perfect unison without meaning to: left, right, left, right — an enforced Sherman's March or something. The people began exhibiting the same coherent behavior as the molecules in laser light when the number and density of pedestrians on the bridge reached a critical threshold, resulting in the sudden transition to synchronized motion. This, in turn, exacerbated the bridge's wobble even more. The city was forced to shut down the $32 million bridge immediately (although it reopened in 2002 after being outfitted with strategic dampers, at an added cost of $8.9 million).

Similarly, while the human brain can be tricked, in The Matrix, into thinking one is consuming actual haute cuisine in a virtual environment because that environment so closely depicts "reality," there is also a downside. Remember what Morpheus told Neo after he failed to make that first virtual jump in the simulator, and, finding he was bleeding from the impact with the virtual pavement, said, "I thought it wasn't reality." Saith Morpheus: "The mind makes it real." You die in the Matrix, you die in the real world, because "The body cannot live without the mind."

Sure, it's just a sci-fi movie, but Hubler mentioned a series of recent experiments in what he unfortunately termed "out-of-body" experiences. He didn't mean anything mystical by it. He was just describing the effects when study participants were outfitted with 3D goggles that allowed them to see themselves from behind, via real-time video feedback. Then someone poked the participant from behind with a stick, making sure the "virtual" version of said participant was also poked. Eventually, they were able to only poke the virtual participant, yet the "meat world" version would see the action and immediately react as if he/she had actually been poked. The line between the real and virtual worlds had blurred to the point that study participants inhabited a "mixed reality" state, according to Hubler.

To me, this says that future generations of Second Life and other online games could become very exciting indeed, and almost indistinguishable from "reality." We're not there yet, if anyone finds this worrying. (As someone noted during the press conference, imagine if economic transactions in Second Life became so strongly coupled to, say, the real New York Stock Exchange that it caused a major market meltdown and folks didn't just lose their virtual shirts, but were bankrupted in the Real World as well.) Instantaneous interaction is a critical requirement and while we can manage this in the lab with real and virtual pendulums, expanding that to an entire virtual world will require even faster computers, as well as far better probes and actuators and other supporting device technologies. We've got a decade or so to prepare before The Matrix becomes "real" – or whatever "real" is going to mean by then.