Grand illusions

YOUR correspondent stands, in a pleasingly impossible way, in orbit. The Earth is spread out beneath. A turn of the head reveals the blackness of deep space behind and above. In front is a table full of toys and brightly coloured building blocks, all of which are resolutely refusing to float away—for, despite his being in orbit, gravity’s pull does not seem to have vanished. A step towards the table brings that piece of furniture closer. A disembodied head appears, and pair of hands offer a toy ray-gun. “Go on, shoot me with it,” says the head, encouragingly. Squeezing the trigger produces a flash of light, and the head is suddenly a fraction of its former size, speaking in a comic Mickey-Mouse voice (despite the lack of air in low-Earth orbit) as the planet rotates majestically below.

It is, of course, an illusion, generated by a virtual-reality (VR) company called Oculus. The non-virtual reality is a journalist wearing a goofy-looking headset and clutching a pair of controllers in a black, soundproofed room at a video-gaming trade fair in Germany. But from the inside, it is strikingly convincing. The virtual world surrounds the user. A turn of the head shifts the view exactly as it should. Move the controllers and, in the simulation, a pair of virtual arms and hands moves with them. The disembodied head belongs to an Oculus employee in another room, who is sharing the same computer-generated environment. The blocks on the table obey the laws of physics, and can be stacked up and knocked down just like their real-world counterparts. The effect, in the words of one VR enthusiast, is “like sticking your head into a wormhole that leads to some entirely different place”.

Matrix algebra

The idea of virtual reality—of building a convincing computer-generated world to replace the boring old real one—has fuelled science fiction’s novels and movies since the 1950s. In the 1990s, as computers became commonplace, several big firms tried to build headsets as a first attempt to realise the idea. They failed. The feeble computers of the time could not produce a convincing experience. Users suffered from nausea and headaches, and the kit was expensive and bulky. Although VR found applications in a few bits of engineering and science, the consumer version was little more than a passing fad in the world’s video-game arcades. But now a string of companies are betting that information technology, both hardware and software, has advanced enough to have another go. They are convinced that their new, improved virtual reality will shake up everything from video-gaming to social media, and from films to education.

Oculus, based in Menlo Park, California, is the emblem of this VR revival—partly because it was the first to demonstrate a plausible headset, partly because of its fairy-tale rise to prominence. As a teenager the firm’s now-22-year-old founder, Palmer Luckey, used to collect old VR headsets and tinker with them in his parents’ garage. Frustrated by their limitations, he hacked together a headset of his own and in 2012 turned to Kickstarter, a crowdfunding website, hoping to raise $250,000. The idea was to distribute the headsets to other members of a small online community of VR-loving hackers.

One of these turned out to be John Carmack, a legendary video-game and graphics programmer, who made some modifications to one of Mr Luckey’s headsets and demonstrated it, in all its taped-together glory, at a gaming conference in 2012. Partly thanks to Mr Carmack’s evangelism (he is now Oculus’s chief technology officer), Mr Luckey’s Kickstarter project ended up raising $2.4m, and he dropped out of university to pursue the idea full-time. In 2014 his work attracted the interest of Mark Zuckerberg, the founder of Facebook, which bought Oculus for $2 billion.

Oculus plans to launch its “Rift” headset early next year. But it is not the only firm with such ambitions. Sony’s offering, called “Morpheus”, will go on sale at around the same time. “Vive”, a joint product of Valve, a big American computer-game firm, and HTC, a Taiwanese smartphone-maker, is planned to appear later this year. Other, lesser-known companies are working on similar products. Meanwhile, Google and Samsung are dipping their toes in the water, making cheap kits that let people turn their smartphones into bare-bones VR headsets.

Each firm hopes that its headset will become next year’s big consumer product for geeky early-adopters. So far none has announced prices, but a few hundred dollars for a full rig seems a good bet. That is well within the means of many people of the sort likely to be attracted to VR, and has led some to suggest it will be the Next Big Thing in consumer electronics. Digi-Capital, a consultancy in San Francisco, reckons the market for virtual reality could be worth $30 billion a year by 2020—if, of course, people actually want to buy it.

The reason that VR failed in the 1990s, it is widely believed, was that computers back then could not create graphics good enough to persuade users they were in a different world. Brendan Iribe, Oculus’s chief executive, disagrees. He reckons high-quality graphics are not the most important piece of the puzzle. “You have to remember,” he says, “that VR is, essentially, a hack on the human sensory system.” In the 1990s that hack was clumsy and inelegant. Nowadays it is much slicker. Three things, according to Mr Iribe, have made this possible. Better graphics, a big consequence of the extra computational power available these days, is certainly one of them. But better screens and improvements in the sensors needed to keep track of what a user is doing are more important.

If at first you don’t succeed...

Start with the screens. The headsets now in development have two tiny, high-resolution liquid-crystal displays (LCDs), one for each eye. A computer creates the scene to be displayed, and each screen shows part of it to the eye it is in front of. This is an old trick, called stereoscopy, which takes advantage of the fact that human brains create a perception of depth by noting differences between the images received by the left and right eyes. Displaying appropriately different images to each eye fools the brain into thinking it is looking at a fully three-dimensional world.

Making this illusion comfortable, though, is harder. It is true that people will happily tolerate poor-quality, badly animated images on television screens. (Standard TV is low resolution, using as few as 300,000 pixels per image, and is displayed at no more than 30 frames per second.) But a TV screen is merely part of a much wider environment. Things get more difficult when the picture on a screen fills a viewer’s entire visual field.

A low frame-rate, meaning things move choppily rather than smoothly, is one cause of “VR sickness”—a motion-sickness-like affliction that can make a user lose his lunch. And, because the screens on a VR headset are mounted so close to its wearer’s eyes, low resolutions leave the individual pixels visible, breaking the illusion that what is being seen is real.

To combat these effects, the headsets from Oculus, Sony and Valve will all show between 2m and 2.6m pixels per image, half for each eye, and those images will be updated between 90 and 120 times a second. Even this, though, is not enough to banish VR sickness entirely. In a standard LCD, each frame of a moving picture remains on screen until it is time to display the next one. It is then replaced as instantly as the technology will permit. For reasons not yet properly understood, but probably something to do with the speed with which the brain is processing the images in question, this contributes to the feeling of nausea. VR engineers have learned that inserting short-lived black frames, lasting about 2 milliseconds, between each frame of the picture, can help. Such blank frames are too short to be perceived consciously, but they make motion appear smoother, which helps to calm stomachs.

Satisfying all the requirements of effective VR is hard. One thing that has allowed the new generation of headsets to be created is the development of organic-light-emitting-diode screens. These have high resolution, can update themselves rapidly, can be made small and light enough for use in headsets and are cheap enough to be part of a consumer product. Another requirement for VR, however, is that the computer running the show must be aware of the position of a user’s head, so that it knows which part of the scene to display on the screens. VR headsets therefore employ a mixture of cameras and the sorts of miniaturised gyroscopes and accelerometers found in smartphones, to keep track of what that user is doing.

These sensors must report back to the computer hundreds of times a second, and the image must be updated to reflect the new information as rapidly as possible. Even a tiny amount of delay is enough to have users reaching for the barf bags. Such quick-reacting sensors were not readily available even a few years ago. Indeed, pioneers of VR such as Mr Carmack had to ask specifically that they be made. It is this tracking technology that has allowed firms to develop controllers which create for the user a pair of virtual hands that can be moved around almost as naturally as the real things. Tracking technology can even follow the user’s body, meaning that as he walks around in the real world, he also seems to walk around in the virtual one.

Ready player one

Many people, after having had VR explained to them but before trying it themselves, assume it will be a bit like a gigantic television set. But the illusion is more convincing than that. A well-built VR program creates a sense of presence—of actually being inside an alternative reality—that, though far from perfect, is much better than any TV can manage. And to do that, a VR headset must obscure a user’s view of the real world, which makes it more akin to a blindfold than a TV.

Nor do tricks from the world of TV and its cousin, video gaming, necessarily work in VR. Shaking the image on a screen to suggest an impact, for instance, is a common technique in video games. Experience has shown that a player does not need to feel himself shaken in order to believe an impact has happened. In the immersive environment of VR, however, the conflict between seeing an impact’s effects and not feeling them can provoke nausea.

Because VR is so new, no one is quite sure, in the matter of suspended disbelief, what does and does not work. Video-game developers, though, often have a better idea than most. And Oculus, Valve and Sony are all aiming their headsets at gamers, at least at first. Gamers tend to be open-minded and technologically astute, and many of the people making the headsets are gamers themselves.

Patrick O’Luanaigh, for example, is the boss of nDreams, a British games studio that is developing a VR adventure game called “The Assembly”. In the course of the game’s development, he and his team have learned a lot about what does and does not work in a virtual world. “The Assembly” begins with the player restrained on a trolley, so that all he can move is his head. “The idea is to ease players gently into the illusion,” says Mr O’Luanaigh. Cut-scenes, in which the camera cuts to a new angle or a different scene, are a staple of video-game storytelling. But they are a no-no in VR. “You generally don’t want to take camera control away from the player,” notes Jackie Tetley, a senior designer at nDreams, “because if you do, you’ve effectively sent their head zooming around the room.”

One striking effect of VR, says Mr O’Luanaigh, is that it boosts the emotional intensity of whatever a user is experiencing. Partly, that is because of the experience’s all-enveloping nature. But it may also be because audiences have not had time to become jaded and genre-savvy. Analogies with the early days of cinema abound. One much-discussed example is the (possibly apocryphal) story of a film called “Train Pulling into a Station”, made in 1895, which is said to have induced naive viewers to scramble out of their seats in fear of an impending collision.

SCE London Studio, a British games developer that has been experimenting with Sony’s VR headset, is playing on this heightened sensitivity in its products. One is a sedate deep-sea dive that culminates in a shark attack. Another is a game that opens with the user tied to a chair in an anonymous London warehouse, about to be “interrogated” with the help of a blowtorch. Even hardened gamers, according to Dave Ranyard, the studio’s director, report feeling more than a little nervous as the virtual torturer looms over them.

That the new generation of VR’s first software products will mostly be games, then, seems in little doubt. But the industry’s boosters point out that it could have plenty of other uses as well. One is film. All of the proposed headsets will come with cinema apps that put the user inside a virtual picture palace with an ordinary flat screen. But immersive films that place the viewer at the centre of the action, and which are made with special panoramic cameras, are possible too. One, called “Clouds over Sidra”, which chronicles life inside a refugee camp in Jordan, has already proved a hit online.

Pornography, an industry which has often been at the cutting edge of technology, is also pondering the possibilities of VR. Its practitioners, such as BaDoinkVR, are already making VR films intended for use with the cheap headsets that transform smartphones into low-spec VR machines. And, for those in need of other, less vigorous, forms of relaxation, VR may provide an alternative to noise-cancelling headphones when it comes to the matter of shutting out the outside world. nDreams, for example, makes a program called “Perfect Beach”, in which users can lounge in the sun on virtual versions of real tropical beaches. Glance to a radio on your left, and you can listen to music. Look at a book on a table to your right, and the program will open up e-books for you to read.

For those who prefer their pleasure shared, rather than solitary, VR might also improve social media, using the ability to create shared tele-reality of the sort described at the beginning of this article. Indeed, this is thought to be one reason why Facebook bought Oculus. And there is also talk of creating, once the industry is properly established, shared tele-real applications in education, news broadcasting and collaborative working. Architects, for instance, might use VR to plonk a planned new building into a realistic simulation of its surroundings. Museums might let visitors wander around a virtual version of an object that, in the real world, must be kept safe behind glass.

Curb your enthusiasm

That, at least, is what the boosters promise. Not everyone is quite so enthusiastic. The current generation of headsets, though impressive, is not perfect. Neither is the software. VR sickness is rarer than it was, but some programs still cause it. And taking full advantage of the technology may be tricky. Although modern VR systems can track their users as they walk around, few people will have enough space at home to create a personal version of the holodeck from “Star Trek”.

There are other, more fundamental criticisms, too. The effort of putting on the headset and adjusting it makes using one much more of a commitment than simply glancing at a screen. And the very immersiveness of the illusion makes VR feel less sociable than using a traditional flat screen with friends, where viewers can chat about what they are watching. (Though sometimes, that may be an advantage. For instance, long-distance passengers in cramped economy-class cabins might quite like the idea of blotting out their dreary reality in favour of an entertaining computer-generated one.)

Similarly, there is no getting around the fact that watching VR users turn their heads to stare at things that are not there, while pawing the air with the hand-held controllers, looks odd. None of the proposed headsets is what you would call stylish. Even the most ardent technophiles may be wary of kit that makes them look silly. And one lesson of consumer electronics is that just because something is possible—nifty, even—does not mean it will come to pass. Attempts to revive 3D movies, for example, have been less transformative than the initial hype suggested.

Still, there is an infectious sense of experimentation and ferment around the technology. Hardware-makers are already researching improvements, such as tracking the pupils of a user’s eyes (which would let him glance around without moving his head) and haptic feedback (which employs special gloves to provide the sensation of touch, alongside sound and vision). And even if full virtual reality proves too overwhelming either to produce or to consume, many of the lessons learned will transfer to the field’s cousin, augmented reality, in which computer-generated images are overlaid on the real world rather than replacing it completely.

Moreover, it is hard to overstate the improvement in the hardware, which, unlike the abortive attempts of the 1990s, does more or less exactly what it promises. Virtual reality is still far from the verisimilitude imagined by novelists and screen writers. Like the smartphone, which went through many failed iterations before Apple hit on a winning formula with the iPhone, it is hard to tell which (if any) of the forthcoming headsets will take off. But it is also hard to believe that nothing of interest will be done with it. This time, for virtual reality, actual reality beckons.