by Matt Rosenberg June 10th, 2012
A U.S. Senate’s Armed Services Committee report sharply critiques Boeing Company for failing to notify the U.S Navy for 17 months after it issued its own internal “suspect discrepancy report” that a key component in an ice detection system to be used on the first P8-A Poseidon “submarine killer” plane test model likely contained “reworked parts” that were “unacceptable for use.” The Navy has committed to take delivery of 117 P8-As from Boeing, with 13 slated for the contract’s first phase. The enhanced 737 is manufactured at Boeing’s Renton, Wash. plant. As the Puget Sound Business Journal reported, the first operational model was delivered in March of this year, after six test models that won’t see battle action. The committee report noted there were suspected false parts originally included on the five other P8-A test models as well.
It was the first P8-A flight test model’s troubled history that raised the hackles of the Senate committee, which stressed in the report that suspected counterfeit parts are a scourge on the U.S. defense supply chain. Investigators found about 1,800 such cases in 2009 and 2010, and only 271 reported to a related and recommended government-industry data exchange program. The total number of suspect parts exceeded one million. “Unvetted independent distributors” primarily in China are the main source of the suspect parts, and the defense industry “routinely” fails to report their use to the government, according to the report.
Three subcontractors and a shady Hong Kong supplier
The supply chain for the questionable part on the P8-A flight test model involved three subcontractors plus a shady Hong Kong supplier, and false assurances of close inspection. There was a “significant reliability risk” involving increased likelihood of malfunction of the ice detection system in the P8-A test jet that Boeing had delivered to the Navy, and likewise for eight commercial 737s delivered to foreign airlines with the same problematic part, the report said. The committee concluded that by using so-called “reconditioned” or “remanufactured” supplies without permission from the Navy, Boeing violated federal acquisition regulation 52.211-5 and the Navy’s Aerospace Standard 9100.
Boeing technician finds problem in November 2009
In November 2009 problems surfaced with a component called a field programmable gate array (FGPA) when a Boeing technician “heard something rattling around inside the [ice detection] module,” the Senate committee report said. It turned out to be an FGPA that had “fallen out of the socket.” The FGPAs were to have come from an original equipment manufacturer named Xilinx even though none had been made since 2003. Through a four-step supply chain, Boeing believed it was in possession of the real thing. Until one of its subcontractors looked more closely.
Lead subcontractor’s inspections show “a large number of anomalies”
Further visual inspection of 249 Xilinx FGPAs by the lead Boeing subcontractor on the ice detection unit, BAE, found in December 2009 there were “a large number of anomalies” including “different ceramic body sizes and …metal tabs,” plus different lengths and signs the parts had been “sanded down and resurfaced.” There were chips and dents and “FGPAs from four different date codes were represented to be from the same lot.” BAE notified Boeing in January 2010 the FGPAs were “unacceptable for use” and identified the nine ice detection modules it had delivered to Boeing so far which contained them.
OEM cites “significant reliability risk”
The original equipment manufacturer Xilinx, later told the Senate committee that based on that description “we would consider the devices to be of dubious origin,” quite possibly “reclaimed and potentially exposed to excessive heat in order to dismount them from a circuit board” and thus posing “a significant reliability risk owing to the potential exposure to excessive solder heat and electro-static discharge damage.”
The abused parts had come to BAE from a California firm named Tandex which falsely claimed to have closely inspected them. Tandex gots the parts from a Florida distribution firm named Abacus Technologies which in turn had purchased them from the Japanese affiliate of Hong Kong based A Access Electronics.
February 2010 Boeing internal report calls parts “unacceptable for use”
A Boeing internal report issued in February 2010 noted that the ice detection system being built by BAE had FPGAs that appeared not to be original equipment from the manufacturer, but rather to be “reworked parts and unacceptable for use.”
“‘We did not alert the customers’”
According to the Senate committee’s investigative report, BAE continued to dog Boeing about notifying the commercial 737 clients and the Navy about the planes delivered with ice detection units containing the sub-par components, but made little headway. Boeing service engineer Robert Kertesz emailed Boeing procurement agent Theotonius Rozario in late July 2011, “components may have a somewhat lower reliability, (but) the engineering consensus is that the units can remain on the airplane and be repaired on an attrition basis…We did not alert the customers.” Rozario disagreed, replying Boeing should notify the Navy because of the importance of P8-A project and because “the U.S. Senate Armed Services Committee is inquiring.” Another Boeing engineer named Jeffrey Look emailed his agreement with Rozario. Finally in mid-August Boeing notified the Navy and a few weeks later, the commercial clients.
The message to the P8-A’s program office was marked “Priority: Critical” and said, “It is suspected that the [ice detection] module may be a reworked part that should not have been put on the airplane originally and should be replaced immediately.”
Problems with another P8-A component
Another type of suspected counterfeit Xilinx FGPA came to Boeing for the P8-A from Honeywell via two sub-contractors, for use in the plane’s distance measuring equipment which shows en-route angled-range distance to the ground. Over 3,000 of 5,500 received showed signs of having been incorrectly formed. Some also showed signs of remarking, and delamination that Honeywell said “can lead to increased failure rates over time.” FGPAs from these lots went into the distance measuring equipment of five more P8-As, also test models. Boeing’s P8-A program manager Charles Dabundo said he only found out from Armed Services Committee staff in November 2011, a year after Honeywell notified Boeing of the problem. Other Boeing officials explained no notification was given because the planes hadn’t been officially delivered to the government, only to Patuxent River Naval Air Station for use by test pilots.
Amendments approved to the 2012 National Defense Authorization Act incude a provision that contractors pay for replacing identified suspect parts, not the military.