Appendix O Analysis of Solenoid 103. Appendix O Analysis of Solenoid 103

Transcription

1 Appendix O Analysis of Solenoid 103

2 The Deepwater Horizon blowout preventer (BOP) stack was supplied with a Cameron Mark II MUX control system. The system had two subsea MUX pods (called the blue and yellow pods) to control the BOP functions. Each pod had one subsea electronic module (SEM) with two redundant electronic control systems (SEM A and SEM B). To control hydraulic valves and regulators, the SEMs were connected by individual cables to 87 dual-coil solenoids in each pod. Each solenoid had two coils for redundancy, one was controlled and powered by SEM A, and the other was controlled and powered by SEM B. In normal subsea operation of the pods, both SEM A and SEM B applied power to one of the solenoid coils in the dual solenoid when the function is activated. These solenoids controlled 1/4-in. hydraulic pilot valves that (when activated) controlled hydraulic supply valves and regulators in each pod to operate the BOP functions. Yellow pod solenoid 103 (when activated) supplied hydraulic pilot pressure to the high pressure (HP) shear circuit control valve mounted on the lower BOP stack. The HP shear circuit caused the blind shear rams to close, the ST Locks to lock, and inner failsafe valves to close. Solenoid 103 was powered and activated by the surface BOP control system or by the automatic mode function (AMF) system mounted in each SEM. 1 In normal system operation, when the AMF system was placed into the arm mode from a surface control panel, all four AMF processor boards received the signal to arm and stayed in the armed mode until given a disarm signal. Yellow Pod Solenoid 103 February 2010 In the course of routine maintenance to the pods during the Deepwater Horizon move to Macondo, several solenoids were replaced on the yellow pod. At that time, solenoid 103 was replaced with a rebuilt spare and function tested prior to lowering the BOP stack to the wellhead. No problems were noted with the yellow pod solenoid 103 while the BOP stack was deployed on the Macondo well. Testing on the Q4000 May 5, 2010 Fifteen days after the incident, as part of the response to secure the well, the yellow pod was lifted off the Deepwater Horizon BOP stack and pulled to surface by an ROV intervention vessel. The pod was transferred immediately to the Q4000 to prepare the pod to operate the BOP functions. A The yellow pod AMF battery voltages were checked by a Cameron service technician and found to be at acceptable levels 8.85 volts (V) for both 9V SEM batteries and 26V for the 27V pod battery. 2 May 6, 2010 During the first function test of the AMF system on board the Q4000, yellow pod solenoid 103 was tested on the surface by Cameron following Cameron s AMF test procedure and using an electromagnetic pin technique to determine whether the coil(s) were activated. With one AMF armed and activated per test, there was no indication of solenoid 103 activating. 3, B A test was then performed using the Cameron portable electronic test unit (PETU) activating one SEM per test; again, there was no indication of solenoid 103 activating. The team sent offshore to prepare the yellow pod to again operate the Deepwater Horizon BOP had very limited access to the pod, as it was considered evidence. No access was allowed into the subsea electronic module, and no checking or manipulation was allowed of the components or electrical connections. A B The Q4000 is a dynamically positioned offshore intervention vessel. Cameron AMF system consists of one AMF electronic processor board and 9V battery pack for SEM A and SEM B (two per pod) and one 27V battery pack shared by both SEM A and SEM B AMF processor boards (one per pod).

3 May 8, 2010 With no further testing performed and in preparation to re-run the yellow pod, solenoid 103 was removed from the pod and replaced with a spare solenoid. At that time it was noted that the plug end connection of solenoid 103 was different than the plug end on the replacement spare that was to be installed. 4 Solenoid 103 was then locked in an evidence box and taken into custody by the U.S. Coast Guard and MMS. The replacement solenoid 103 was successfully function tested by activation of the PETU. 5 May 12, 2010 The yellow pod AMF system was tested and functioned as expected following the Cameron AMF test procedure. 6 May 19, 2010 The yellow pod was lowered to the Deepwater Horizon BOP stack and latched in place. 7 The yellow pod was used to operate functions remotely on the Deepwater Horizon BOP stack from the Q4000 for 114 days (May 19 Sept. 10, 2010) when the BOP stack was loaded on a barge to be sent ashore. Testing at NASA s Michoud Assembly Facility Feb. 25, 2011 Nearly eight months after removal from the yellow pod on board the Q4000, solenoid 103 was bench tested by DNV at NASA s Michoud Assembly Facility (Michoud) in New Orleans using Cameron s test criteria. Solenoid 103 passed all of the tests performed. 8 March 2, 2011 Solenoid 103 was re-installed in the yellow pod by the DNV at Michoud in preparation to test the AMF system in the blue and yellow pods. 9 March 3, 2011 Solenoid 103 was tested using the Cameron PETU. When functioned on SEM A only or SEM B only, there was no indication of solenoid 103 fully activating. 10 During the activations of the solenoid, no electrical faults were found with the PETU. 11 When solenoid 103 was activated with SEM A or SEM B, the meter on the PETU for monitoring electrical current was indicating current flow, as expected during normal operation. 12 Solenoid 103 was again removed from the pod and replaced with the same spare that was installed May 8, 2010, on the Q4000. The AMF system in the yellow pod was tested with the spare solenoid and functioned as expected. Solenoid 103 was then re-installed on the yellow pod. The AMF system in the pod was tested with SEM A and SEM B active (normal operation). Solenoid 103 functioned (at 43 seconds) after a 22-second delay from the expected activation time. 13 The AMF system in the yellow pod was tested two additional times with SEM A and SEM B active. Solenoid 103 functioned (at 21 seconds) as expected at the correct activation time. 14 Note: Each time the AMF was activated in the yellow and blue pods by DNV at Michoud, both the SEM A and SEM B AMF systems were armed and functioned in tandem.

4 March 4, 2011 The DNV and the investigation Technical Working Group decided that additional tests were required to reconfirm the results. The group decided both solenoid coils in solenoid 103 should be activated from the Cameron PETU to simulate the normal control system operation when the BOP is subsea. The No. 1 PETU that was connected to the yellow pod was configured to operate only SEM A or SEM B. The No. 2 PETU that was connected to the blue pod was configured to operate both SEM A and SEM B at the same time. The No. 2 PETU was connected to the yellow pod and power supplied to SEM A and SEM B. Hydraulic pilot pressure of 3,000 psi was supplied to the solenoids. Solenoid 103 was activated three times by PETU No. 2 with both SEM A and SEM B, then with only SEM A or SEM B, and functioned as expected each time. 15 The No. 1 PETU was connected to the yellow pod and power supplied to SEM A. Hydraulic pilot pressure of 3,000 psi was supplied to the solenoids. Solenoid 103 was activated six times by PETU No. 1 with only SEM A or SEM B. Solenoid 103 functioned as expected on the first activation of SEM A, but did not function again on SEM B or SEM A. 16 The No. 2 PETU was connected to the yellow pod and power supplied to both SEM A and SEM B. The plug end connection of solenoid 103 was pulled from the SEM PIE receptacle. Solenoid 103 was activated on both SEM A and SEM B. Voltage readings were taken at the PIE receptacle for SEM A and SEM B and were in the correct range per Cameron s specification. 17 The No. 1 PETU was connected to the yellow pod and power supplied to both SEM A and SEM B. Solenoid 103 was activated on both SEM A and SEM B. Voltage readings were taken at the PIE receptacle for SEM A and SEM B and were in the correct range per Cameron s specification. 18 The No. 2 PETU was connected to the yellow pod and power supplied to both SEM A and SEM B. The plug end connection of solenoid 103 was plugged in to the SEM PIE receptacle. 3,000 psi pilot pressure was supplied to the solenoids. Solenoid 103 was activated three times by PETU No. 2 with only SEM A or SEM B, then with both SEM A and SEM B, and functioned as expected each time. 19 Solenoid E-cable and Plug Questions were raised by a Cameron service technician s comments written in the Cameron service report on the Q4000 about the plug end of the solenoid cable. 20 These comments caused DNV at Michoud to investigate. The plug on solenoid 103 was not the same as the spare replacement solenoid installed on the yellow pod. Photos and measurements were taken to compare the two versions. 21 The Transocean investigation team further researched the solenoid cable plug issue by contacting the manufacturer of the cable and plug system directly. The cable and plug on solenoid 103 were the later revision for improved performance that was released by the manufacturer in All solenoid cable and plug assemblies purchased from 2009 forward would be this revision, whether supplied directly to Transocean by the manufacturer, or from the manufacturer through Cameron. 22

5 Additional Inspection of Solenoid 103 After review of the test results of solenoid 103 from Michoud and history from the Q4000, it appeared there could be a mechanical issue requiring more electrical current to pull in (actuate) the solenoid armature that operates the hydraulic pilot valve. Requests were submitted to DNV by the BOP investigation Technical Working Group to perform additional testing and to disassemble the solenoid for inspection. May 2011 DNV and NASA engineers performed additional inspections and tests of Solenoid 103. Solenoid 103 was bench tested by NASA engineers. When power was applied to coil A or B, the solenoid functioned correctly. When power was applied to both coil A and B, solenoid 103 did not function. Solenoid 103 was installed in the Yellow Pod and successfully functioned with the PETU three times. Solenoid 103 was removed from the Yellow Pod and the wiring was inspected. It was found that one of the coils in the solenoid was wired in reverse polarity to the other coil.

6 1. See Appendix N. 2. Daily Report Sheet, Subsea Pod Intervention, May 5, 2010, CAM_CIV_ Daily Report Sheet, Subsea Pod Intervention, May 5, 2010, CAM_CIV_ Daily Report Sheet, Subsea Pod Intervention, May 5, 2010, CAM_CIV_ Ibid. 6. Ibid. 7. Ibid. 8. DNV, Forensic Examination of Deepwater Horizon Blowout Preventer, March 20, Ibid. 10. Ibid. 11. Ibid. 12. Ibid. 13. DNV, Forensic Examination of Deepwater Horizon Blowout Preventer, March 20, Ibid. 15. DNV, Forensic Examination of Deepwater Horizon Blowout Preventer, March 20, Ibid. 17. Ibid. 18. Ibid Ibid. 20. Daily Report Sheet, Subsea Pod Intervention, May 5, 2010, CAM_CIV_ DNV, Forensic Examination of Deepwater Horizon Blowout Preventer, March 20, Ibid.