You may doubt that you’ll ever fall for the illusion of virtual reality, but your brain is already working against you.
Kimberly Voll has a Ph.D. in computer science, specializing in artificial intelligence, and an honors degree in cognitive science. She is also one of the developers of the puzzle game Fantastic Contraption, built in virtual reality for the HTC Vive. Voll knows how your brain works, and at the 2016 Game Developers Conference she explained how it’s specifically affected by VR.
“We have historically, particularly in games, really tried to bring the player into that experience,” Voll said. “We have spent time taking flat screens and trying to pull people into those screens. But with virtual reality we throw all of that out, because in many respects we are actually, literally putting the person in the game, or in the experience.”
The key to achieving this goal is to understand the tools and senses that the human brain uses to figure out what is real, in the real world, and then give those tools and senses the same data, but in the virtual world.
The human brain works to generate a consistent, sensible worldview, using a variety of tools. The brain is also gullible and easy to fool. Have you ever seen an optical illusion where two objects are actually the same size, but one looks larger than the other? Your brain falls for it every time. Or you could check out an Ames room, another optical illusion, that depends on the shape of a room, and controlling the way you view it.
The way a VR developer fools your brain into thinking that a virtual space is real is by knowing what tools your brain uses to construct reality, and then giving your brain the same information, but presented in virtual reality. VR developers in a simulation give your brain all the building blocks it needs to say “This is real.”
Your brain reads visual cues in the environment to decide what’s real. If you are sitting in a chair reading this article on a computer, or riding the train and reading this article on a smartphone, your brain knows where you are because it knows what a chair and a computer, or a train and a smartphone look like.
In Job Simulator, for instance, when you see a computer on a desk in front of you, along with a photocopier and a coffee maker and a telephone and a power strip, your brain sees a whole bunch of things it recognizes and says “This is an office,” so you feel like you’re standing in an office.
The vestibular system in our ears tells our brain about our position in 3D space by helping us achieve balance. Nausea in VR is such a horrible thing because if someone in VR feels nauseous it means they feel like they’ve lost their balance, even though in the real world they might be sitting on a chair, or a couch, or standing up perfectly straight. The brain can’t reconcile what it feels in the real world, versus what it sees in virtual reality.
Our brains know where and how we can extend ourselves into space, like where your hands are when you reach for an object. VR games like Job Simulator and I Expect You To Die give you virtual hands to manipulate objects, and as long as the orientation of your hand in the simulation matches the orientation of your hand in the real world, it feels like you’re actually reaching into the digital space.
Whenever VR taps into a trick like an illusion of extension, the developer also has to make sure that the trick doesn’t turn against them. VR developers are already learning that once you give the player the opportunity to pick up and manipulate objects, for example, they’re going to want to pick up everything they can see that their brain thinks they should be able to pick up.
Putting doorknobs on doors in VR might not be a good idea, because players will want to reach out and turn the knob. If they can, and the door opens, the sense of reality is maintained. If not, the illusion might be broken.
It’s not possible for VR developers to give your brain the exact same information it gets in the real world, however, and that’s when VR developers have to employ different tricks, like manipulating the player’s attention.
The HTC Vive game Arizona Sunshine begins at a barbecue in the desert. You see a hamburger on a grill. If you’ve had any previous experience with virtual reality, you may try to reach out and flip the burger. When you can’t, you’ll feel frustrated because the simulation is not matching up to your brain’s expectations as to what is real. Burgers on grills are supposed to be flipped, right?
Then the zombies show up, and Arizona Sunshine turns into a shooting gallery. Now your brain couldn’t care less about whether or not you can flip that burger, because you’re too busy shooting the undead. Now what matters is making sure your brain thinks you are actually holding guns and shooting them.
VR developers are getting good enough at fooling your brain in virtual reality that they also have to start worrying about your safety. “We can lead people to do things without intending to, that they shouldn’t be doing,” Voll said.
What if you try to lean against a wall in virtual reality that actually isn’t there? Or accidentally punch a wall in the real world, because the borders of the VR simulation aren’t clearly marked?
Because your brain can be so thoroughly fooled into thinking the virtual is the real, players may need to be warned about the content that awaits them in the simulation. Not warning people that a VR game is scary and then giving them a jump scare can make people really upset.
Voll also made clear that scientists and researchers really have no idea whether and how long-term exposure to VR changes the brain. There are decades’ worth of research into how motion simulators affect people, but that research may not apply in this case.
“What we’re able to do today with the technology that has just emerged in the last couple days is so much more robust, and so much more powerful that so many of the theories are proving to not hold water,” Dr. Voll said. “They’re just not relevant when we’re able to do things at this level of quality. So I think we really need to look at it. We need to think long and hard about what long-term exposure looks like.”
Illustration by Jason Reed