The promise of augmented reality

SCIENCE fiction both predicts the future and influences the scientists and technologists who work to bring that future about. Mobile phones, to take a famous example, are essentially real-life versions of the hand-held communicators wielded by Captain Kirk and his crewmates in the original series of “Star Trek”. The clamshell models of the mid-2000s even take design cues directly from those fictional devices.

If companies ranging from giants like Microsoft and Google to newcomers like Magic Leap and Meta have their way, the next thing to leap from fiction to fact will be augmented reality (AR). AR is a sci-fi staple, from Arnold Schwarzenegger’s heads-up display in the “Terminator” films to the holographic computer screens that Tom Cruise slings around as a futuristic policeman in “Minority Report”.

AR is a close cousin to virtual reality (VR). There is, though, a crucial difference between them: the near-opposite meanings they ascribe to the term “reality”. VR aims to drop users into a convincing, but artificial, world. AR, by contrast, supplements the real world by laying useful or entertaining computer-generated data over it. Such an overlay might be a map annotated with directions, or a reminder about a meeting, or even a virtual alien with a ray gun, ripe for blasting. Despite the hype and prominence given recently to VR, people tend to spend more time in real realities than computer-generated ones. AR thus has techies licking their lips in anticipation of a giant new market. Digi-Capital, a firm of merger and acquisitions advisors in California, reckons that of the $108 billion a year which it predicts will be spent by 2021 on VR and AR combined, AR will take three-quarters.

Improving on the world

Like many science-fictional technologies, AR is in fact already here—just unevenly distributed. An early version was the heads-up displays that began to be fitted to jet fighters in the 1950s. These projected information such as compass headings, altitude and banking angles onto the cockpit canopy. Such displays occasionally turn up in cars, too. But only now, as computers have shrunk enough and become sufficiently powerful, has it become possible to give people a similar sort of experience as they go about their daily lives.

Last year, for instance, the world was briefly entranced by an AR smartphone game called Pokémon Go. Players had to wander the world collecting virtual monsters that were, thanks to their phones’ cameras, drawn over a phone’s-eye view of a building’s lobby or a stand of trees. Apps such as Snapchat, which features image filters that permit users to take pictures of themselves and others wearing computer-generated rabbit ears or elaborate virtual make-up, are another example.

There are less frivolous uses, too. Google’s Translate app employs computer vision, automatic translation and a smartphone’s camera to show an image of the world that has text, such as items on menus and street signs, interpreted into any of several dozen languages.

Apps like Snapchat and Translate rely on machine-vision algorithms to work their magic. Snapchat is designed to detect faces. This works well enough, but means that the bunny ears can be applied only to heads. Translate, similarly, looks for text in the world upon which to work its magic. But smartphone-makers have bigger plans.

At the end of last year Google and Lenovo, a Chinese hardware manufacturer, unveiled the Phab 2 Pro, the first phone to implement a piece of Google technology called Tango. The idea is that, by giving the phone an extra set of sensors, it can detect the shape of the world around it. Using information from infra-red detectors, a wide-angle lens and a “time-of-flight” camera (which measures how long pulses of light take to reflect off the phone’s surroundings) Tango is able to build up a three-dimensional image of those surroundings. Armed with all this, a Tango-enabled phone can model a house, an office or any other space, and then use that model as a canvas upon which to draw things.

To give an idea of what is possible, Google has written apps that would be impossible on Tango-less phones. “Measure”, for instance, overlays a virtual tape measure on the phone’s screen. Point it at a door, and it will tell you how wide and high that portal is. Point it at a bed, and you get the bed’s dimensions—letting you work out whether it will fit through the door. Another Tango app is the oddly spelled “Woorld”, which lets users fill their living rooms with virtual flowers, houses and rocket ships, all of which will interact appropriately with the scenery. Place the rocket behind a television, for instance, and the set will block your view of it.

Through a pair of glasses, virtually

The effect Tango gives is impressive, but the technology is still in its early stages. Building 3D models of the world is computationally demanding, and quickly drains even the Phab 2 Pro’s beefy battery. The models themselves quickly use up the phone’s data-storage capacity. And the touchscreen of a phone is a clumsy way of communicating with the software. Some enthusiasts of augmented reality therefore think that the technology will not take off properly until smartphones can be abandoned in favour of smart spectacles that can superimpose images on whatever their wearers happen to be looking at.

Such glasses do exist. So far, though, they have made a bigger impact on the workplace than in the home. Companies such as Ubimax, in Germany, or Vuzix, in New York, make AR spectacles that include cameras and sensors, and which use a projector mounted on the frame to place what looks like a small, two-dimensional screen into one corner of the wearer’s vision.

Used in warehouses, for instance, that screen—in combination with technology which tracks workers and parcels—can give an employee instructions on where to go, the fastest route to get there and what to pick up when he arrives, all the while leaving both of his hands free to move boxes around. Ubimax reckons that could bring a 25% improvement in efficiency. At a conference in London in October, Boeing, a big American aeroplane-maker, described how it was using AR glasses to give workers in its factories step-by-step instructions on how to assemble components, as well as to check that the job had been done properly. The result, said Paul Davies of Boeing’s research division, is faster work with fewer mistakes.

The one serious attempt to offer individual consumers such technology did not, though, go well. Like Vuzix’s and Ubimax’s products, Google’s “Glass”, released in 2013, was a pair of spectacles with a small projector mounted on one arm. The idea was, in effect, to create a wearable smartphone that would let its user make calls, read e-mails, see maps and use the Glass’s built-in GPS to navigate, all the while leaving his hands free for other tasks.

The problem was not with the users. Google’s “Glass Explorers”—those willing to pay $1,500 for early access to the hardware—seemed happy enough. But, often, those they interacted with were not. Glass Explorers quickly attracted the nickname “Glassholes” from those annoyed by their proclivity to glance at e-mails in the middle of a conversation, or worried that the device let wearers record everything going on around them. (Some restaurants banned Glass users on privacy grounds.) Google stopped making Glass early in 2015, although it is working on a new version aimed at businesses instead of individuals.

Other firms have more limited ambitions, but may do better for that. RideOn, for instance, is an Israeli outfit founded by three engineers with experience in designing heads-up displays for aircraft. It will soon start selling augmented-reality ski goggles. The idea is to turn skiing into a video game, by showing users routes, letting them time runs, compete with their friends, shoot footage and the like.

Some companies are building much more capable displays. Instead of 2D images, they propose to create augmented reality in three dimensions. In March 2016 Microsoft began making early versions of a headset called the HoloLens available to software developers around the world. Unlike the AR glasses produced by Vuzix and Ubimax, or Google’s Glass, the HoloLens can draw 3D images that appear to exist in the real world. Users can walk around a virtual motorbike, for instance, to inspect it from behind, or place virtual ornaments on real tables or shelves.

It is, in other words, like a Tango-enabled smartphone—only much more capable. The device’s cameras, derived from the Kinect (an accessory originally developed for Microsoft’s Xbox 360 games console), scan the world around it. Those cameras generate such a flood of information that Microsoft has had to design a special chip to process all the incoming data. Armed with that understanding, and with the ability to track the position of its user’s head, the machine can tailor its graphics accordingly: making a virtual motorbike appear to be standing on a real floor, for instance. The same cameras let the wearer interact with the machine via voice commands, by making gestures in mid-air, or by tracking precisely where he is looking.

Unlike VR headsets, which must be connected to either a PC or a smartphone to work, the HoloLens is a self-contained computer that needs no accessories. Users view the world through a pair of thick, transparent lenses. A pair of projectors feed light into the top of these lenses. Three optical waveguides (one each for red, green and blue light—the primary colours from which others can be created) funnel that light down the lenses before bending it through 90° and into the user’s eyes.

By overlaying its images onto the real world, the HoloLens headset turns reality into a computer monitor. A window containing a Skype call can be placed onto an office wall, disappearing when the user looks away and returning when he looks back at it. A computerised calendar can be placed on the desk (or the ceiling, if you prefer). All this information can be seen without having to cut yourself off completely from the outside world, as a VR headset would require.

Some of the first demonstrations of the HoloLens involved games. In one, users blasted aliens that took cover behind their living-room sofas. In a second, they played with blocks from Minecraft, a sort of virtual Lego, on their living-room tables. More recent apps have focused on business and training. One such, developed in collaboration with Case Western Reserve University, in Cleveland, projects a human body into the room to help with the teaching of anatomy. A wave of the hands can add muscles to the skeleton, or bring the heart out of the chest to examine it more closely.

Augmenting the enterprise

The HoloLens can be used collaboratively, as well. Another demo has someone being instructed how to repair a light-switch by someone else, who is employing videoconferencing software in another room to do so. The guide can see what the HoloLens user sees, and can draw on top of his field of view—putting circles around objects of interest or highlighting the correct tool in a box. ThyssenKrupp, a German engineering firm, is experimenting with giving the devices to its lift repairmen. Should anyone encounter a particularly difficult job, he can call head office for specialist advice. Users can also connect to each other and see the same augmented reality (in true science-fiction style, other users appear as golden, androgynous, vaguely Art Deco-looking figures).

Aecom, an international firm of architects and engineers, is already using the HoloLens to help design buildings. Modern building projects can be very complicated, says John Endicott, one of Aecom’s executive directors—to the point where even experienced designers have trouble keeping everything in their heads.

In 2016 the firm designed buildings around the Serpentine art gallery, in London. Mr Endicott observes that, “the roofs of these things had very complex geometry. We simply couldn’t check it on a 2D screen, but the HoloLens let us all review it together.” Trimble, an American engineering firm, helped Aecom develop the system. “We’re also finding it has applications in everything from mining to agriculture to facilities management,” says Aviad Almagor, the director of Trimble’s “mixed reality” programme. “You can do things like track assets [such as miners, lorries or equipment] as they move round a 3D model of a mine, in real time.”

The HoloLens is far from perfect, however. The AR magic happens in only a small slice of a user’s view (some have likened it to looking in on the computer-generated world through a letterbox). Though the headset is light (weighing around 600g) and comfortable, it is bulky and not exactly fashionable. And using the gesture-tracking system to interact with the illusions the headset generates can feel clunky and awkward. It is not yet on general sale, but when it is (Microsoft has given no firm date) its price tag—also unknown, though the versions sold to software developers go for at least $3,000—is likely to make it a business-only proposition.

Microsoft is not the only firm working on advanced AR headsets. One rival is Meta, in San Mateo, California. Compared with Microsoft this firm is a tiddler, having raised only $73m in funding so far. But its engineers promise a much wider field of view than the HoloLens’s. Microsoft’s product can track a few hand gestures. Meta’s is designed to keep a constant eye on exactly what a user’s hands are up to, letting him “handle” virtual objects simply by picking them up and rotating them.

Another potential rival, Osterhout Design Group, in San Francisco, which makes AR glasses for industrial and medical companies, has announced two products aimed at individuals. Though less technically capable than the HoloLens, both are sleeker than their rival. Microsoft’s best-known competitor in this area, though, is Magic Leap, a firm founded in Florida in 2010, which has attracted $1.4 billion in investment from companies such as Google and Ali Baba, China’s biggest online retailer, as well as plenty of attention for its snazzy promotional videos. It has kept its technological cards close to its chest—to the point where some sceptics think that its technology has been oversold. But the demos it has released show images much clearer and crisper than those Microsoft can manage with the HoloLens.

Curb your enthusiasm

For all the hype, AR is still at an early stage, especially as a consumer technology. Forecasts of markets worth squillions by the end of the decade should be taken with a good deal of salt, especially since virtual reality, AR’s close and even-more-hyped cousin, has so far proved a bit of a damp squib. No VR headset-maker has yet released official sales figures, but the numbers that have trickled out look modest.

In October 2016 Cher Wang, chairwoman of HTC, a Taiwanese consumer-electronics company, told 87870 News, a Chinese website, that her firm had sold 140,000 of its Vive headsets since their launch the previous April. (By way of comparison, Apple sells more than 870,000 iPhones a day.) In November SuperData, a market-research firm in New York, described VR as “the biggest loser” in the American shopping season around Thanksgiving, and cut its sales forecasts for Sony’s PlayStation VR headset in 2016 from 2.6m to 750,000. Even among keen techies, enthusiasm for VR seems limited. A survey by Steam, an online shop that dominates the market for PC gaming, found that just 0.38% of its customers owned a VR headset in December, a number unchanged from the previous month.

If AR is not to go the same way, it will have to be made easier to use. That probably means consumer versions will be adapted for peoples’ phones. As Tim Merel, Digi-Capital’s boss, points out, phones are a known quantity that people are comfortable with. They have become, for many, their default computing device. Their existing app stores offer developers an easy way to sell software, and their business model—in which the cost of the hardware is often subsidised by network operators, who recoup this investment with fees and rental charges as they go along—could help draw some of the financial sting of the initial outlay a customer must make. On the other hand, a phone’s screen is small and fiddly, and holding it up every time you want to use an AR app could become tedious.

Headsets such as the HoloLens offer a way around this problem. Those currently in development will cost thousands of dollars and look more than a little silly. For now, that will limit their uptake to companies, which can afford the hardware and are less worried about the aesthetics. But the hope is that the mix of sensors and computing power needed to run AR can be shrunk to the point where, as Mark Zuckerberg, Facebook’s boss, put it at a show for developers last April: “we’re going to have what look like normal-looking glasses that can do both virtual and augmented reality.” Others want to go further still. Samsung and Apple, for instance, are exploring the idea of AR-enabled contact lenses.

For now, such devices remain far away. Those in the computing industry like to talk of an “iPhone moment”, when a well-crafted product launches, almost single-handedly, a new phase of the computing revolution. But such moments are the culmination of years of research into, and development of, many different technologies. The iPhone was not the first smartphone. No self-respecting salaryman of the mid-2000s was without a BlackBerry, and the basic idea can trace its ancestry back at least as far as the hand-held personal digital assistants of the 1990s. None of the present approaches to AR seems likely to change the world as the iPhone did. But those behind them hope that, one day, a combination of them will.

Correction: An earlier version of this story referred to Digi-Capital as a firm of analysts, rather than as the mergers and acquisitions advisors they actually are.