In praise of celestial mechanics

THE launch on March 1st of the Falcon 9 rocket, carrying a Dragon cargo capsule (pictured) to the International Space Station, went perfectly. But once the capsule had reached orbit, a problem emerged: only one of its four sets of thrusters was working. The spacecraft was drifting, unable to stabilise itself, let alone dock with the space station. Its tumbling motion meant it could not turn its solar panels towards the sun and made communication difficult. Engineers at SpaceX, the private firm that has a $1.6 billion contract with NASA, America’s space agency, to fly at least 12 cargo missions to the station, had to find a way to rescue the craft in the few hours before its battery ran out.

Assuming that the problem was caused by blocked valves, the SpaceX team wrote software telling the Dragon to increase and then suddenly decrease the pressure upstream of the valves, hoping that this “pressure hammering” would unblock things. The code was uploaded to the spinning Dragon using powerful radio dishes provided by the US Air Force. Once activated, the software forced the blocked valves open one by one, and within five hours control of the spacecraft had been restored. At a press briefing, Elon Musk, SpaceX’s founder, called it “the equivalent of the Heimlich manoeuvre”.

The problem, it transpired, had been caused by a tiny change made by a supplier to the design of the valves. Once rescued, the Dragon completed the rest of its mission smoothly. This was the latest example in a decades-long tradition of long-distance fixes, in which mission controllers restore stricken spacecraft to health, or extend their working lives, using a combination of technical skill, improvisation, good fortune—and the old standby of simply switching things off and on again and hoping for the best.

An early instance of such space-hacking involved Mariner 10, an accident-prone NASA probe that performed fly-bys of Venus and Mercury in 1973. It suffered from short-circuits, faulty radio and power systems, an unreliable tape-recorder, the failure of its camera heaters and, most worryingly, the loss of much of the gas that supplied its stabilising thrusters. This last problem was cleverly overcome by repositioning the craft’s solar panels and main antenna so that they acted as solar sails, stabilising the probe via the gentle pressure of sunlight.

In the 1990s Galileo, a NASA probe sent to Jupiter, was another beneficiary of the long-distance fixers’ art. Its main antenna, folded up like a giant umbrella, failed to open properly after launch. The probe was repeatedly turned towards and away from the sun, in the hope that thermal expansion and contraction would free the antenna, but to no avail. Next, the antenna-deployment motors were switched on and off thousands of times, but the antenna remained jammed. Further efforts involved switching on the motors while spinning Galileo on its axis, and also when it reached the point of its maximum acceleration around Jupiter. Eventually the team gave up on the main antenna and remotely reprogrammed the craft to use its smaller secondary dish, which transmitted data at one-hundredth of the speed of the main one. But by using compression software and getting Galileo to record data on its tape-recorder for later transmission, it was ultimately possible to achieve almost all of the probe’s objectives.

Even more elaborate measures were called for in 1998 after a gyroscope malfunction on the Solar and Heliospheric Observatory (SOHO) probe, a joint project between NASA and the European Space Agency (ESA). SOHO was designed to monitor solar activity, but lost its lock on the sun, went into an emergency mode and began spinning uncontrollably, losing power as it did so.

The probe was slowly spinning, with sunlight falling on its solar panels once a minute, causing its computer to boot up for 25 seconds each time. SOHO’s controllers sent control sequences to it thousands of times a day, at various frequencies, for six weeks, ordering the probe to switch on its transmitter so that they could tell that commands were getting through. Eventually it did so, and the controllers told it to charge its batteries so it could stay on for more than a few seconds at a time.

A solar probe in deep-freeze

Once the batteries had been charged, it was possible to download telemetry from SOHO. This revealed that the propulsion systems used to stabilise the craft were in shadow and had frozen solid. Thawing SOHO’s 200kg of hydrazine fuel involved a repeating cycle of turning on its heaters and then recharging its batteries, gradually raising the temperature of the tank each time. Eventually a command sequence was sent which enabled SOHO to stabilise itself using its warmed-up thrusters.

The probe was then reprogrammed to operate without two of its three gyroscopes, and its scientific instruments were gradually brought back online. Having survived this three-month crisis, SOHO has continued to operate ever since, far exceeding its design life of two years. “It’s engineering, really hard engineering,” said Bernhard Fleck of the ESA, the head of the SOHO science team, after the probe’s recovery. “You have to try the impossible.”

The recovery of SOHO relied in part on the ESA’s experience with Olympus, an spectacularly unlucky experimental communications satellite launched in 1989. It malfunctioned in 1991, went into a spin and was lost for a year before contact was re-established in 1992. Its propulsion systems then had to be thawed out before it could be stabilised. Olympus was damaged during the Perseid meteor shower in 1993 and had to be taken out of service. But one of its engineers went on to work on SOHO, which proved useful when it, too, had to be rescued from the deep-freeze.

Similarly, NASA had applied lessons from its Pioneer probes to the Voyager probes during the 1970s. Launched in 1977, Voyager 1 and Voyager 2 have been in continuous operation longer than any other space probes. Edward Stone, the project scientist and head of the Voyager programme since its inception at NASA’s Jet Propulsion Laboratory (JPL), says the two spacecraft were radiation-hardened after the Pioneer 10 probe, which flew past Jupiter in 1973, revealed that the intensity of radiation from the planet was higher than expected. The result was a nine-month redesign effort to replace parts of the Voyager design that were “radiation soft”. This made the Voyager probes more reliable than the Pioneer craft, and may have contributed to their unusual longevity: they have now been delivering data for more than 35 years, and continue to break new scientific ground.

The original plan was for the two Voyager probes to fly past Jupiter and Saturn, returning both photographs and information from an array of instruments. Launching two nearly identical probes just days apart made it more likely at least one would succeed; happily, both did. Each one has three separate computers, each of which has its own backup. The tiny on-board memory in each system necessitated sending complete changes of program code from Earth to get the probes to perform different tasks: this was done 18 times during the Jupiter fly-by alone. Such reprogramming also made it easier to cope with unexpected problems and meant the probes could be taught new tricks.

Both Voyager probes delivered images and scientific readings as planned from Jupiter and Saturn, sending back data at what were, at the time, high speeds: 115 kilobits per second (kbps) from Jupiter and 45kbps from Saturn. But signal strength (and hence data rate) drops off as the probes head out into space, following an inverse-square law: double the distance between sender and receiver and you get only one-quarter of the signal strength.

So when Voyager 2’s mission was extended to include fly-bys of Uranus and Neptune, mission controllers had to find a way to cope with the much lower data rates achievable at such great distances: about 9kbps at Uranus and 3kbps at Neptune. This would have made downloading images taken by Voyager 2’s camera, each of which is 640 kilobytes (around five megabits) in size, slow and painful. But JPL and NASA had a few tricks up their sleeves. On Earth, the radio dishes of the Deep Space Network, which are used to communicate with the Voyager probes, were expanded from 64 to 70 metres in size, and their focusing was improved. A new technique was also developed to combine multiple dishes of varying sizes, further increasing sensitivity. This boosted the data rate threefold for the Uranus fly-by and, with an even larger array of dishes involved, sevenfold for Neptune.

Voyager 2 also had two subtle bits of futureproofing waiting to be awoken. First, the backup flight computer could be activated simultaneously with the main unit, and run a separate program. A simple compression algorithm was designed for and uploaded to this secondary system, which reduced images to about 40% of their initial size. Second, a bit of prototype hardware, called a Reed-Solomon encoder, was installed but not activated at launch. This efficiently adds error-correction data to a transmitted stream, to allow the recovery of garbled information.

Teaching an old Voyager new tricks

Both Voyager probes were launched with a more primitive error-correction system in place that added one bit of correction for each bit transmitted. The new encoder reduced that to one bit for every five. However, Dr Stone stays, effective decoders that could strip the error-correction bits and restore the original data did not exist on Earth at the time of launch. By the time Voyager 2 reached Uranus, they did. All these tricks meant that it could send back far more images.

Redundancy, resiliency and adaptability, plus human ingenuity, keep distant hardware going.

A further hack was necessary to help Voyager 2 point its camera. As it completed its pass of Saturn, the gears that controlled the azimuth of its camera platform got stuck. The JPL controllers coped with this by reprogramming Voyager 2 to rotate by 90° and use its elevation motor to pan the camera instead. They also reprogrammed the probe’s attitude-control system with a motion-compensation algorithm to slew the craft as it passed Uranus and its moons, enabling it to produce much crisper images.

Unblocking valves, using solar panels as sails and thawing propulsion systems are all useful tricks that have saved free-flying probes. The various rovers that NASA has sent to the surface of Mars operate in a very different environment, and they too have been kept going using some clever tinkering. In 2004 two identical rovers, Spirit and Opportunity, landed on opposite sides of the red planet. Their mission was intended to last 90 days, but this was soon extended to eight months, and in the event the solar-powered, six-wheeled rovers continued to operate for several years, covering long distances on the Martian surface, examining minerals and rock formations and looking for evidence of water in Mars’s distant past.

Driving backwards on Mars

In 2006 the gearbox for Spirit’s right front wheel experienced a current spike and stopped working. Its controllers kept it going by driving it backwards instead. (It eventually became mired in a region of flour-like Martian soil in 2009, and performed stationary science until it stopped working in 2010.) When Opportunity began to experience a similar problem in its left front wheel in 2008, its controllers gently reversed it out of the crater it was exploring. As with Spirit, running the wheel backwards seemed to loosen it up and may have spread lubrication around. In 2009 a second current spike led to an important decision: Opportunity would also drive backwards from then on, rather than risk the loss of a wheel.

The rover is nearly as nimble going backwards as it was going forwards—but driving backwards it could not initially cover as much distance each day. When moving forwards, Opportunity sends images of the ground in front of it, allowing controllers to devise a route for it. The rover follows that route and then switches to driving autonomously until it receives further instructions. When driving backwards, however, an antenna blocks part of the view of the camera that is used to spot obstacles. This limited Opportunity’s ability to drive backwards autonomously.

But Scott Maxwell, the rover’s driver, devised a clever workaround. He realised that if Opportunity rotated itself 17.5° clockwise to the drive direction and its camera mast then turned anticlockwise by the same amount, its rear view was unimpeded. This dramatically increased the distance Opportunity could cover autonomously, despite having to repeat the manoeuvre every 1.2 metres. There have been no problems since, despite the awkwardness of the rover looking over its shoulder for obstacles.

Jon Callas, the mission leader, says Opportunity has continued to perform well, despite a few ailments. Because of “arthritis” in its robot arm, it now drives with the arm extended, to avoid the risk that it will become stuck in its retracted position. And a ribbon cable (a set of wires arrayed side by side like a ribbon) began to fray on one side. After testing duplicate gear on Earth, Dr Callas says the team reprogrammed the rover’s computer to use different measurements of signals from the ribbon cable to avoid problems. Opportunity has now travelled more than 35km and continues to perform useful science, nine years after landing.

In August 2012 Opportunity was joined on Mars by a much larger rover, Curiosity. Its mission leader, Mike Watkins, says its design was influenced by the experience of operating other rovers on the red planet. The goal, he says, was to avoid “learned problems from previous missions”, a form of advance troubleshooting. Curiosity has a notional mission length of 700 days. But as with Spirit and Opportunity, the work required to ensure that a rover has a 99% chance of working for its full mission length means it is very likely to work for much longer, says Dr Watkins.

Curiosity is the size of a small car and has greater articulation in its wheels, which should make it better at handling obstacles. According to Mr Maxwell, who shifted to driving Curiosity last year, it could drive over a coffee table without even noticing, and has “more eyes than a potato”, with 17 cameras in all. (He has since left NASA to work at Google.) The falling cost of computer power also means that Curiosity has more powerful on-board computers, capable of storing multiple versions of the control software, and switching between them if necessary. This provides extra flexibility when testing new ideas and fixing problems, says Dr Watkins. Curiosity went into an emergency “safe mode” for two days in March, seemingly because of a problem with its primary computer, which may have been caused by cosmic rays. But it continues to operate using its secondary computer.

Indeed, redundancy, resiliency, adaptability and programmability, along with human ingenuity, seem to be the keys to keeping distant hardware going, years or even decades longer than planned. In May the planet-hunting Kepler probe suffered the loss of a second of its four reaction wheels, three of which are needed to keep it precisely pointed at a particular region of the sky. Kepler, which has found 132 planets around other stars and another 2,740 planetary candidates, completed its primary mission in October 2012 and was granted a four-year extension. As The Economist went to press, its prospects were uncertain, and NASA engineers were devising plans to try to get the failed reaction wheels working again. As decades of long-distance repair work have demonstrated, if anyone can fix it, they can.