Boeing changes testing methods
Announcement after probe found multiple problems with Starliner
Boeing plans to change how it does testing for its Starliner astronaut capsule following an investigation that found the company could have caught the issues it faced during a test flight of the spacecraft in December — if it had done more robust testing.
The news comes after an Orlando Sentinel report that members of NASA’s safety advisory panel were “surprised” to learn Boeing had not performed a full, end-to-end test of the entire mission integrated with United Launch Alliance’s Atlas V rocket prior to its troubled flight in December. During the mission, the company experienced numerous problems that, in one instance, could have caused two spacecraft to collide with each other.
In a press conference Friday, Boeing’s vice president and pro
gram manager for the Commercial Crew Program — a joint venture with NASA to return astronauts to space from American soil — at first said the Sentinel mischaracterized Boeing’s testing processes, before going on to confirm the Sentinel’s reporting.
Boeing did do extensive testing integrated with the rocket, John Mulholland said, but it didn’t run it through the entire mission — the concern NASA’s safety panel raised.
Instead, it tested the mission in “chunks,” a relatively standard procedure in aerospace testing.
Still, had Boeing run the software through the entire process, a potentially 25-hour-plus ordeal, it would have caught the problems that later cropped up in flight.
On Dec. 20, the capsule, called Starliner, failed to reach the correct orbit because it was running on a timer that was 11 hours ahead. The problem happened because Starliner incorrectly pulled the time from the Atlas V rocket. But during verification, Boeing had chosen to stop testing the two systems together from the point they separate in launch, instead of running a fully integrated test from start to finish.
Had the full test happened in the lab, Mulholland said the issue would have manifested itself and Boeing could have corrected it before the mission.
“From a hindsight standpoint, I think it’s very easy to see what we should have done because we uncovered an error,” Mulholland said. “This is a tough business. It’s a game of inches. And so you had a highly talented, very dedicated team that made that error. And going forward, we just need to make sure we have the discipline that won’t occur again.”
In the future, Boeing said it will still do extensive testing in chunks, but also add a full end-to-end test with the spacecraft and rocket responding to each other to ensure similar unexpected problems don’t occur.
Boeing will also make changes to what it uses to test its software.
The December mission also uncovered another coding error that could have potentially led the spacecraft to collide with its service module, which separates from Starliner before it returns to Earth, while in space.
Boeing used emulators — computers that simulate how other pieces of software works — to test certain components. One of those emulators had faulty code, blocking Boeing from catching the potential collision issue. The actual hardware Boeing should have tested was being used on a different test, so the company opted to use the emulator instead when it ran a test of that part of the mission.
Luckily, the company caught the problem midflight and was able to quickly correct the issue before Starliner returned to Earth on Dec. 22. In fact, much of the mission went well, with certain components even performing better than anticipated.
But the problems have led Boeing to implement new requirements about what exact hardware needs to be used to test certain parts of the mission, Mulholland said. More focus will also put into checking the validity of its emulators.
The software problems have drawn questions about the parallel with the software issues experienced by Boeing’s 737 Max airplanes, which have led to the deaths of 346 people. Costcutting has been associated with the problems plaguing Boeing’s commercial aviation division.
Mulholland denied that the same could be happening with Starliner.
“Cost has never been in any way a key factor in any of our decisions on how we test and verify our systems,” Mulholland said. “… It was not a matter at all of the team consciously short cutting or not doing what they believe was appropriate.”
Boeing has not yet said whether it will repeat a test of its astronaut capsule. The company’s next launch was supposed to be with astronauts onboard, but the errors in the December flight have put that timeline in question. NASA is expected to release the full results of an independent investigation into Boeing’s software issues after instructing the company to re-verify 1 million lines of code.
Under the Commercial Crew Program, Boeing got $4.2 billion to develop Starliner, while the program’s other contractor, SpaceX, got $2.6 billion for its astronaut capsule called Crew Dragon. SpaceX performed a successful test flight of Crew Dragon without astronauts aboard in March 2019 and is expected to launch with humans later this year — marking the first time since 2011 that American astronauts have launched from the U.S.