Kepler Loses Another Reaction Wheel.

It was Deja Vu all over again yesterday, as the Kepler team at NASA announced the probable failure of the spacecraft's second reaction wheel. Kepler is a NASA satellite searching for exoplanets (planets outside our solar system) by finding stars with periodic variation in their brightness, which would indicate planets passing in front of them. Kepler needs three working reaction wheels in order to maintain the necessary level of pointing precision for science operations. It lost reaction wheel #2 back in 2012, so it's been operating without a spare ever since. I first wrote about Kepler in January 2013, when Reaction Wheel #4 (the second reaction wheel to exhibit problems) was showing signs of degradation and poor performance. After a ten day rest period, which operators hoped would redistribute lubricant in the reaction wheel and allow it to spin more easily, the Kepler team returned the spacecraft to full operations with the hopes that the wheel could hang in there. Turns out, it couldn't.

The location of Kepler's reaction wheels.
Source: NASA

On Wednesday NASA announced that the wheel seemed to have failed again. During a semi-weekly contact with Kepler, operators found the spacecraft unexpectedly in safe mode, which is what the spacecraft does when it detects something is wrong. In this mode it ceases science operations and faces the sun to gather as much power as possible, allowing the communicating line with Earth to drop in and out as it spins around the Sun. When operators commanded the reaction wheels to stop rotating the vehicle in order to maintain the communication link, reaction wheel 4 stopped spinning, but maintained full torque, meaning that it was trying as hard as it could to spin up but was not moving at all. This behavior likely indicates a structural failure of the wheel bearing. In other words, bad news bears.

NASA says the Kepler mission was originally only 3.5 years, so technically Kepler completed all of its mission goals. However, it is standard for missions like this to low-ball the life expectancy so that they can tell congress it will cost less, and claim 100 percent mission success even if something goes wrong later in the life of the spacecraft. In reality, missions are often extended for 2-5 times the original mission life. When Kepler reached its original 3.5 year mission goal in 2012, funding for the satellite was extended for six more years. There had been more noise than anticipated in Kepler's data, due to worse than expected signal and more noise than anticipated in the spacecraft, so observations were taking longer than originally expected. It was estimated that Kepler would take until 2015 to fulfill all of it's mission goals. So, if Kepler is headed for hospice, it will have reached it's originally planned mission life, but not entirely fulfilled it's mission goals.

But all is not lost! Kepler operators have not given up on attempts to resuscitate the wheel. They plan to try jump starting it by sending start and stop commands in rapid succession, and perhaps spinning it in the opposite direction. If all attempts fail, they may investigate pursuing other science goals using a scanning mode, where 3-axis control is not necessary.

Even if rejuvenation in some form is not possible, the mission is far from a bust. Kepler has made significant contributions to science and the search for planets. It has identified 2740 candidate stars, and led scientists to estimate that there are at least 17 billion Earth-sized exoplanets in the Milky Way Galaxy alone. There is also a huge trove of data from Kepler that has not been analyzed yet, and will provide fodder for research and discovery for years to come. There may very well be data revealing an Earth-sized star in the habitable zone of a far-away planet, just sitting on a server somewhere waiting to be extracted. And Kepler is just the beginning of that search - NASA is already planning the next generation of exoplanet-finding telescopes, starting with TESS, which would launch in 2017.

Kepler Reacts to a Sticky Wheel

In a Kepler Mission Manager Update, NASA announced today that the Kepler mission is temporarily ceasing science operations due to increased friction in reaction wheel #4. Kepler is a NASA satellite that has been searching for Earth-like planets around nearby starts since it launched in 2009. Reaction wheels are used to maneuver satellites and other spacecraft requiring fine, continuous pointing control. A motor spins up the reaction wheel in one direction, which causes the spacecraft to spin in the opposite direction. Spacecraft almost always have 4 reaction wheels: one for each axis of rotation, and one spare. If two or more reaction wheels fail, the spacecraft loses it's ability to attain and hold commanded positions. For a telescope like Kepler, where steady pointing is critical to collecting high quality images of faint stars, this is a mission ending-anomaly.

The type of reaction wheel used on Kepler
Source: Ball Aerospace

This isn't the first road bump for Kepler. The spacecraft experienced problems with reaction wheel #2 back in July 2012, and took steps to mitigate future risk at that time. Among other things, they decided to increase operational temperatures  (likely to keep the lubricant in the wheels warmer);  increase spin-rates (because the faster wheels spin, the less likely they are to get stuck, and the more the lubricant gets distributed nicely), and implement bi-directional rotation (to even out any use-related wear and tear).

Reaction wheel side view

The Kepler operations team is dealing with this latest threat to the mission by putting Kepler in a ten day "rest period" to give the wheel's lubricant time to become more evenly distributed. During this time the wheels will not be used, and a modified safe mode attitude (where the solar arrays are pointed at the sun to keep the spacecraft power-positive) will be maintained using thrusters.

Reaction wheel problems seem to be prevalent lately, especially in high profile missions. In August, Operators for the Dawn Mission also experienced complications or failures with two of its reaction wheels. Dawn had already lost a reaction wheel in June 2010 when it lost a second wheel in August. It is now using an inventive combination of the two remaining reaction wheels and its hydrazine thrusters to point the spacecraft (this solution is not sufficient for the strict pointing requirements of Kepler).

It is interesting to note that the operations teams identified the anomaly differently in each case. Kepler operators noticed that the amount of torque needed to change the spin rate of its reaction wheel was higher than normal during a semiweekly contact with the spacecraft, indicating increased friction in the wheel. Dawn operators became aware of the problem when the spacecraft software detected the problem and initiated an automatic shut down of the wheel.

Proba-1 is Back in Business.

ESA's Proba-1 microsat was built and launched by the ESA as a proof-of-concept for semi-autonomous Earth observation - operators can upload coordinates and the satellite will autonomously image that location. The satellite ended up turning into a heavily-used science tool, but it almost shut down in May due to star tracker degradation. 

Proba-1 captured this image of London's Olympic Park
 neighborhood.   Credit: ESA 

Many small satellites use star trackers to orient themselves. The star trackers point behind the satellite (away from the Sun and Earth) and take pictures of star fields, which they can identify and use to calculate the satellite's orientation.  The CCD camera's attached to Proba-1's star trackers had sustained severe radiation (they were 5 years older than the planned lifetime for Proba-1), and the radiation had created permanent damaged or "hot" pixels which  show up in images as white spots. The star trackers were mistaking hot pixels for stars, and sending incorrect coordinates to the main satellite computer, causing it to error out or mispoint. 

Proba-1 was saved by a software patch written specifically for the satellite, which - once loaded - allowed the star trackers to distinguish between hot pixels and real stars. The star trackers are apparently working "as good as new" now, even with degraded CCDs. The proof is in the pudding - at the right is a farewell shot of Olympic Park that the satellite shot last week.

Intelsat IS-19 Deploys Damaged Solar Array

Intelsat's IS-19 communication satellite, which will provide telecommunications services and broadband coverage in the Pacific Ocean region, finally deployed its south solar array yesterday, after keeping it stowed for 13 days. The solar array sustained severe damage during launch, and was stuck in the stowed position for several days. Once deployed, operators estimated that the damage reduced the array's power-collecting capacity by half. The spacecraft is still operational, but will only be able to reach 75% power capacity.

IntelSat-19.
Credit: Space Systems/Loral

IS-19 was built by Space Systems/Loral, in Palo Alto, CA, and launched by the Swiss company Sea Launch. Both sides continue to deflect blame for the damage, but Sea Launch did report an "unexplained pressure event" 72 seconds into the launch, which mirrors a similar event detected in the Sea Launch rocket in 2004, when launching another Space Systems/Loral satellite (Telstar 14/Estrela do Sul 1) whose solar array never deployed.

Sea Launch has released launch telemetry indicating that "all systems performed nominally throughout the launch profile...Boeing engineers did notice an unexpected, isolated event around 72 seconds after launch" . They cheekily point the finger by adding "while it is premature to speculate on its origin…it bears a striking resemblance to a prior Space Systems/Loral mission". 

It is worth noting that Space Systems/Loral has a small history of challenged solar arrays. The company also built Telstar 14R (Estrela so Sul 2), which experienced a problem deploying the solar arrays after launch as well, but was launched on a Russian Proton rocket.

The Sea Launch platform - a converted
oil rig. Credit: Sea Launch

Sea Launch is an interesting company - it launches its Zenith 3SL rockets from a converted oil rig which is towed from Long Beach, California to the middle of the Pacific Ocean right on the equator, an optimal location for launching satellites to geosynchronous transfer orbits. The company was formed in 1999 as a joint venture between companies from Norway, Russia, the Ukraine, and the United States, and is managed by Boeing. It recently emerged from bankruptcy in 2010 and has had two successful launches in as many years.

Risky Rescue of an Air Force Satellite

Wired had a great article recently about the risky rescue of a malfunctioning Air Force communications satellite. The series of anomalies that occurred are a great example of some of the things that satellite operations teams should be prepared to confront, and made for a very exciting 2 years for the AEHF-1 operations team! First, a bit of background on the satellite, from Air-Force Magazine:

"The AEHF program, one of the largest space programs of the decade, is designed to augment and eventually replace the legacy Milstar satellite communications network. Lockheed Martin is the prime contractor, Northrop Grumman built the payload, and everything is run by Space and Missile Systems Center at Los Angeles AFB, Calif. The constellation of four cross-linked AEHF satellites is expected to provide a communications capacity exceeding that of Milstar by a factor of 10."

The 7 ton, $2 billion satellite, AEHF-1, was launched aboard an Atlas V rocket back in August 2010, and at first it seemed to be functioning perfectly. It was placed into an elliptical orbit about 220 km above the Earth's surface, and the first step in the mission was to fire the satellite's hydrazine engine to booste it into a circular, geosynchronous orbit where it would become part of the Air Force's comms satellite constellation. 

AEHF-1 in the clean room.
Credit: Lockheed Martin

The first sign that something was not right came when the operators first tried to ignite the engine to boost the satellite into geo. Nothing happened. They tried again; the engine did not fire. (In my opinion, firing the engine again was a huge mistake on the part of the operators. As Albert Einstein said, "Insanity is doing the same thing over and over again and expecting different results". In operations, repeating something that didn't work the first time, without first figuring out why it did not work, can mean the end of the mission.) At this point David Madden, the head of the comms satellites at the Space and Missile Systems Center at Los Angeles Air Force Base, stepped in and rounded up a group of engineers who analyzed the telemetry and detrmined that the anomaly was likely caused by a piece of fabric left in the fuel line during the manufacturing process. They also realized that repeated attempts to fire the engine would flood the fuel line and cause the satellite to explode. (Needless to say, they quickly abandoned the "try, try again" tactic.)

With the oxidizer tanks sealed off, rendering the main engine useless, AEHF-1 was stranded in an ineffective and slowly decaying orbit, losing 3 miles of altitude each day.  Compounding the problem was the fact that at its altitude AEHF-1 was sharing an orbit with huge amounts of space junk. Operators were having to fire the small thrusters  in small maneuvers to avoid debris, wasting valuable fuel. 



The operations team was barricaded in a conference room for a week - pizzas were literally slipped to them under the door - and they emerged with a plan to salvage the mission: They would fire the satellite's small thrusters, hydrazine-fueled reaction engine assemblies (REAs) and tiny xenon-fueled Hall Current Thrusters (HCTs). These were originally intended for small orbit adjustments and momentum dumps, but would now be used in a series of more than 450 maneuvers over 14 months to get the satellite into GEO. This would require painstaking planning and precision.

The boosting maneuvers commenced with phase 1 of the recovery procedure: a big orbit-boosting burn to get the satellite out of the danger zone. Very quickly, the second anomaly popped up: The small thrusters required burns that were several hours long in order to be effective, and during these burns the satellite was left with the same side facing the sun, causing components on the sun-facing side to overheat. Overheating can at best diminish mechanism performance, and at worst quickly destroy crucial satellite components. The operations team quickly devised a set of maneuvers to flip the satellite periodically. These maneuvers were inserted during the boosting maneuvers so that no part of the satellite was facing the sun for too long. Flipping the spacecraft, while continuing to keep the thrusters pointed in the right direction and the spacecraft on-target, required an entirely new operations strategy.

Once the schedule of mission-saving maneuvers was in place, operators soon realized that the fuel supply on board was not sufficient to complete the rescue. Every ounce of fuel means another ounce of mass to launch into space (and find space to store onboard the satellite), so the fuel supply is budgeted very carefully based on the minimum mission requirements. Even if AEHF-1 had enough fuel to get to geo, it would still need propellant for station-keeping and momentum management maneuvers once in place, and thus each firing of the minor thrusters meant a shortened mission life. To solve this problem, the software engineers buckled down and re-wrote a chunk of the flight software. The new software allowed operators to position the satellite using the reaction wheels rather than the thrusters, saving fuel. It also optimized fuel consumption in the REA thrusters. The new software was uplinked and implemented successfully.


Engineers calculated the amount of fuel necessary to execute the mission once the satellite was in GEO, and allotted the rest of the fuel to the REA thrusters for the duration of phase 2. When this fuel ran out, the REA thrusters could no longer be used for the rescue effort. At this point AEHF-1  was in the Van Allen radiation belt, a high radiation zone where energetic charged particles are held in place by the Earth’s magnetic field. These particles can damage a satellite's electrical system very quickly, especially the solar panels. This presented a catch-22 for the ops team, because escaping the Van Allen Belt without the REA thrusters would require using the HCT thrusters, which run on electric current. Generating electric current, of course, would require deploying the solar arrays and exposing them to the dangerous radiation of the Van Allen Belt. After much deliberation, the team came up with a strategy to deploy the arrays and fire the thrusters quickly to escape the danger zone with minimal damage.


Once out of the Van Allen Belt, Stage 3 of the rescue continued, with the ops team relying solely on the HCTs to perform the necessary boosting maneuvers. HCTs are generally used for station-keeping maneuvers, and they use electricity and Xenon fuel to emit short, relatively weak puffs of power. However, they can burn for thousands of hours. They had never been used for extended amounts of time in zero G. 

Source: Air-Force Magazine

The HCT motors were optimized to fire at the apogee (the point in the orbit farthest from the Earth) of the AEHF satellite’s orbit, in order to increase the orbit's perigee (the point in the orbit closest to the Earth). (Think of it like a lever - the farther out you push, the more thrust you get).  From late October 2010 to June 2011, the HCTs burned for 10 to 12 hours per day. This required frequent analysis and tuning. Madden described the tedious process: "They’re like a finicky old car, one that you’ve got to constantly adjust to get it to optimize. There’s no instruction manual for how to do that. It’s basically an art."


Once the satellite reached its target altitude of 17,000 miles, the orbit had to be circularized at the correct inclination, which required additional burns to increase the perigree to 22,000 miles, decrease the apogee by 10,000 miles, and drive the inclination down closer to the equator to allow the satellite to see more of the Earth. The ops team timed the burns to take advantage of the beneficial effects of Earth’s gravitational pull, thereby conserving valuable fuel.


Finally, after almost two years, the satellite reached it's target location, and the payload was successfully deployed. Madden says that it still has enough remaining fuel to continue operating for it's full planned lifetime of 14 years. (Of course, this is likely due to the habit of aerospace companies to grossly and purposefully under-predict lifetimes to ensure mission success; with a full tank of gas I'm sure the team was hoping the satellite would last much, much longer.) Nevertheless, it was a remarkable and complex recovery effort. I am sure it was a giant relief to finally deploy the payload, which had been stored inside the satellite in order to fit in the Atlas V launch cone, and see that it worked. Next up: the team will launch and deploy AEHF-2. Best of luck to them, although I bet the operations team is pretty sick of the entire AEHF family at this point.


Summary of Anomalies:

Anomaly 1: Clogged fuel line
Solution: Use REA thrusters to boost orbit


Anomaly 2: Potential Collisions with Space Debris
Solution: Small maneuvers with REA thrusters, and rapid exit from LEO


Anomaly 3: Overheating during orbit-boosting maneuvers
Solution: Periodic spacecraft flips during maneuvers

Anomaly 4: Fuel Shortage

Solution: Re-write/load of the flight software to optimize fuel consumption for new operations plan; Strategic scheduling of burns to take advantage of orbital and gravitational affects.


Anomaly 5: High radiation from Van Allen Belt
Solution: Rapid Solar Array deployment and use of Hall Current Thrusters to quickly boost altitude.