Kepler Loses Another Reaction Wheel.

It was Deja Vu all over again yesterday, as the Kepler team at NASA announced the probable failure of the spacecraft's second reaction wheel. Kepler is a NASA satellite searching for exoplanets (planets outside our solar system) by finding stars with periodic variation in their brightness, which would indicate planets passing in front of them. Kepler needs three working reaction wheels in order to maintain the necessary level of pointing precision for science operations. It lost reaction wheel #2 back in 2012, so it's been operating without a spare ever since. I first wrote about Kepler in January 2013, when Reaction Wheel #4 (the second reaction wheel to exhibit problems) was showing signs of degradation and poor performance. After a ten day rest period, which operators hoped would redistribute lubricant in the reaction wheel and allow it to spin more easily, the Kepler team returned the spacecraft to full operations with the hopes that the wheel could hang in there. Turns out, it couldn't.

The location of Kepler's reaction wheels.
Source: NASA

On Wednesday NASA announced that the wheel seemed to have failed again. During a semi-weekly contact with Kepler, operators found the spacecraft unexpectedly in safe mode, which is what the spacecraft does when it detects something is wrong. In this mode it ceases science operations and faces the sun to gather as much power as possible, allowing the communicating line with Earth to drop in and out as it spins around the Sun. When operators commanded the reaction wheels to stop rotating the vehicle in order to maintain the communication link, reaction wheel 4 stopped spinning, but maintained full torque, meaning that it was trying as hard as it could to spin up but was not moving at all. This behavior likely indicates a structural failure of the wheel bearing. In other words, bad news bears.

NASA says the Kepler mission was originally only 3.5 years, so technically Kepler completed all of its mission goals. However, it is standard for missions like this to low-ball the life expectancy so that they can tell congress it will cost less, and claim 100 percent mission success even if something goes wrong later in the life of the spacecraft. In reality, missions are often extended for 2-5 times the original mission life. When Kepler reached its original 3.5 year mission goal in 2012, funding for the satellite was extended for six more years. There had been more noise than anticipated in Kepler's data, due to worse than expected signal and more noise than anticipated in the spacecraft, so observations were taking longer than originally expected. It was estimated that Kepler would take until 2015 to fulfill all of it's mission goals. So, if Kepler is headed for hospice, it will have reached it's originally planned mission life, but not entirely fulfilled it's mission goals.

But all is not lost! Kepler operators have not given up on attempts to resuscitate the wheel. They plan to try jump starting it by sending start and stop commands in rapid succession, and perhaps spinning it in the opposite direction. If all attempts fail, they may investigate pursuing other science goals using a scanning mode, where 3-axis control is not necessary.

Even if rejuvenation in some form is not possible, the mission is far from a bust. Kepler has made significant contributions to science and the search for planets. It has identified 2740 candidate stars, and led scientists to estimate that there are at least 17 billion Earth-sized exoplanets in the Milky Way Galaxy alone. There is also a huge trove of data from Kepler that has not been analyzed yet, and will provide fodder for research and discovery for years to come. There may very well be data revealing an Earth-sized star in the habitable zone of a far-away planet, just sitting on a server somewhere waiting to be extracted. And Kepler is just the beginning of that search - NASA is already planning the next generation of exoplanet-finding telescopes, starting with TESS, which would launch in 2017.

Risky Rescue of an Air Force Satellite

Wired had a great article recently about the risky rescue of a malfunctioning Air Force communications satellite. The series of anomalies that occurred are a great example of some of the things that satellite operations teams should be prepared to confront, and made for a very exciting 2 years for the AEHF-1 operations team! First, a bit of background on the satellite, from Air-Force Magazine:

"The AEHF program, one of the largest space programs of the decade, is designed to augment and eventually replace the legacy Milstar satellite communications network. Lockheed Martin is the prime contractor, Northrop Grumman built the payload, and everything is run by Space and Missile Systems Center at Los Angeles AFB, Calif. The constellation of four cross-linked AEHF satellites is expected to provide a communications capacity exceeding that of Milstar by a factor of 10."

The 7 ton, $2 billion satellite, AEHF-1, was launched aboard an Atlas V rocket back in August 2010, and at first it seemed to be functioning perfectly. It was placed into an elliptical orbit about 220 km above the Earth's surface, and the first step in the mission was to fire the satellite's hydrazine engine to booste it into a circular, geosynchronous orbit where it would become part of the Air Force's comms satellite constellation. 

AEHF-1 in the clean room.
Credit: Lockheed Martin

The first sign that something was not right came when the operators first tried to ignite the engine to boost the satellite into geo. Nothing happened. They tried again; the engine did not fire. (In my opinion, firing the engine again was a huge mistake on the part of the operators. As Albert Einstein said, "Insanity is doing the same thing over and over again and expecting different results". In operations, repeating something that didn't work the first time, without first figuring out why it did not work, can mean the end of the mission.) At this point David Madden, the head of the comms satellites at the Space and Missile Systems Center at Los Angeles Air Force Base, stepped in and rounded up a group of engineers who analyzed the telemetry and detrmined that the anomaly was likely caused by a piece of fabric left in the fuel line during the manufacturing process. They also realized that repeated attempts to fire the engine would flood the fuel line and cause the satellite to explode. (Needless to say, they quickly abandoned the "try, try again" tactic.)

With the oxidizer tanks sealed off, rendering the main engine useless, AEHF-1 was stranded in an ineffective and slowly decaying orbit, losing 3 miles of altitude each day.  Compounding the problem was the fact that at its altitude AEHF-1 was sharing an orbit with huge amounts of space junk. Operators were having to fire the small thrusters  in small maneuvers to avoid debris, wasting valuable fuel. 



The operations team was barricaded in a conference room for a week - pizzas were literally slipped to them under the door - and they emerged with a plan to salvage the mission: They would fire the satellite's small thrusters, hydrazine-fueled reaction engine assemblies (REAs) and tiny xenon-fueled Hall Current Thrusters (HCTs). These were originally intended for small orbit adjustments and momentum dumps, but would now be used in a series of more than 450 maneuvers over 14 months to get the satellite into GEO. This would require painstaking planning and precision.

The boosting maneuvers commenced with phase 1 of the recovery procedure: a big orbit-boosting burn to get the satellite out of the danger zone. Very quickly, the second anomaly popped up: The small thrusters required burns that were several hours long in order to be effective, and during these burns the satellite was left with the same side facing the sun, causing components on the sun-facing side to overheat. Overheating can at best diminish mechanism performance, and at worst quickly destroy crucial satellite components. The operations team quickly devised a set of maneuvers to flip the satellite periodically. These maneuvers were inserted during the boosting maneuvers so that no part of the satellite was facing the sun for too long. Flipping the spacecraft, while continuing to keep the thrusters pointed in the right direction and the spacecraft on-target, required an entirely new operations strategy.

Once the schedule of mission-saving maneuvers was in place, operators soon realized that the fuel supply on board was not sufficient to complete the rescue. Every ounce of fuel means another ounce of mass to launch into space (and find space to store onboard the satellite), so the fuel supply is budgeted very carefully based on the minimum mission requirements. Even if AEHF-1 had enough fuel to get to geo, it would still need propellant for station-keeping and momentum management maneuvers once in place, and thus each firing of the minor thrusters meant a shortened mission life. To solve this problem, the software engineers buckled down and re-wrote a chunk of the flight software. The new software allowed operators to position the satellite using the reaction wheels rather than the thrusters, saving fuel. It also optimized fuel consumption in the REA thrusters. The new software was uplinked and implemented successfully.


Engineers calculated the amount of fuel necessary to execute the mission once the satellite was in GEO, and allotted the rest of the fuel to the REA thrusters for the duration of phase 2. When this fuel ran out, the REA thrusters could no longer be used for the rescue effort. At this point AEHF-1  was in the Van Allen radiation belt, a high radiation zone where energetic charged particles are held in place by the Earth’s magnetic field. These particles can damage a satellite's electrical system very quickly, especially the solar panels. This presented a catch-22 for the ops team, because escaping the Van Allen Belt without the REA thrusters would require using the HCT thrusters, which run on electric current. Generating electric current, of course, would require deploying the solar arrays and exposing them to the dangerous radiation of the Van Allen Belt. After much deliberation, the team came up with a strategy to deploy the arrays and fire the thrusters quickly to escape the danger zone with minimal damage.


Once out of the Van Allen Belt, Stage 3 of the rescue continued, with the ops team relying solely on the HCTs to perform the necessary boosting maneuvers. HCTs are generally used for station-keeping maneuvers, and they use electricity and Xenon fuel to emit short, relatively weak puffs of power. However, they can burn for thousands of hours. They had never been used for extended amounts of time in zero G. 

Source: Air-Force Magazine

The HCT motors were optimized to fire at the apogee (the point in the orbit farthest from the Earth) of the AEHF satellite’s orbit, in order to increase the orbit's perigee (the point in the orbit closest to the Earth). (Think of it like a lever - the farther out you push, the more thrust you get).  From late October 2010 to June 2011, the HCTs burned for 10 to 12 hours per day. This required frequent analysis and tuning. Madden described the tedious process: "They’re like a finicky old car, one that you’ve got to constantly adjust to get it to optimize. There’s no instruction manual for how to do that. It’s basically an art."


Once the satellite reached its target altitude of 17,000 miles, the orbit had to be circularized at the correct inclination, which required additional burns to increase the perigree to 22,000 miles, decrease the apogee by 10,000 miles, and drive the inclination down closer to the equator to allow the satellite to see more of the Earth. The ops team timed the burns to take advantage of the beneficial effects of Earth’s gravitational pull, thereby conserving valuable fuel.


Finally, after almost two years, the satellite reached it's target location, and the payload was successfully deployed. Madden says that it still has enough remaining fuel to continue operating for it's full planned lifetime of 14 years. (Of course, this is likely due to the habit of aerospace companies to grossly and purposefully under-predict lifetimes to ensure mission success; with a full tank of gas I'm sure the team was hoping the satellite would last much, much longer.) Nevertheless, it was a remarkable and complex recovery effort. I am sure it was a giant relief to finally deploy the payload, which had been stored inside the satellite in order to fit in the Atlas V launch cone, and see that it worked. Next up: the team will launch and deploy AEHF-2. Best of luck to them, although I bet the operations team is pretty sick of the entire AEHF family at this point.


Summary of Anomalies:

Anomaly 1: Clogged fuel line
Solution: Use REA thrusters to boost orbit


Anomaly 2: Potential Collisions with Space Debris
Solution: Small maneuvers with REA thrusters, and rapid exit from LEO


Anomaly 3: Overheating during orbit-boosting maneuvers
Solution: Periodic spacecraft flips during maneuvers

Anomaly 4: Fuel Shortage

Solution: Re-write/load of the flight software to optimize fuel consumption for new operations plan; Strategic scheduling of burns to take advantage of orbital and gravitational affects.


Anomaly 5: High radiation from Van Allen Belt
Solution: Rapid Solar Array deployment and use of Hall Current Thrusters to quickly boost altitude.

SpaceX Launches Dragon Capsule

After a two day delay due to an anomaly in one of the rocket engines, SpaceX's falcon 9 rocket blasted off early this morning, launching a Dragon capsule filled with cargo for the ISS.

There was plenty of news coverage of the event, but there were three little tidbits that I especially enjoyed.

1. The description of the humorous juxtaposition of the young, casual, SpaceX mission controllers with the suited NASA old guard in the control room: "Many of the SpaceX controllers wore untucked T-shirts and jeans or even shorts, a stark contrast to NASA’s old suit-and-tie shuttle team."



View from Spacex's Dragon spacecraft looking outward
at one of two solar array panels in the process of deploying.
Source: SpaceX

2. The picture that @SpaceX tweeted of the Dragon capsule solar arrays after they deployed. News reports say the cheering in the control room after the successful deployment of the solar arrays was almost louder than the cheering following the successful launch. In many ways, deploying the solar arrays is the most stressful, high stakes activity of commissioning. It takes less force, and different mechanics, to deploy arrays in a zero G environment than it does on Earth. In fact, many spacecraft (like the James Webb Telescope) are never able to fully test solar array deployment on Earth; gravity makes the solar arrays too heavy to operate with the motors that will be used in space. Operators must trust in the capability of the mechanical engineers and software simulations without ever testing the full deployment procedure. I have a friend who was working on a satellite at Space Systems Loral when one of the solar arrays failed to deploy during commissioning. In a last ditch effort to save the spacecraft they fired a spacecraft thruster at the array to try to push it out, and ended up losing whatever capability they had from the partially deployed array by blowing a hole through it. The satellite is now a very expensive piece of space junk, floating around in GEO with 100 million dollars worth of expensive, cutting edge technology and equipment, and no power. 
3. The smaller, public interest news articles about Celestis, the for-profit company that sent the ashes of 300 people to orbit aboard the Falcon 9. The "special payload" included the remains of James Doohan (Scotty, from Star Trek) and Gordon Cooper (one of the famous Mercury 7 astronauts). The ashes were in the second stage of the Falcon 9 rocket, which was jettisoned about 10 minutes after launch and will remain in orbit for about a year before its orbit decays and it burns up in Earth's atmosphere. This was actually the backup flight for Celestis - the first round of ashes failed to make it to orbit on the botched 2008 SpaceX launch that crashed into the Pacific Ocean.

Congrats to SpaceX! I am excited to follow the mission over the next two weeks as the Dragon performs several maneuvers and is eventually grabbed by ISS astronauts using the station's robotic arm. The SpaceX Ops team has a lot on their plate - It must be an incredibly exciting experience, but I don't envy the pressure on them or their likely sleep schedule for the next two weeks!