Case Study of FMEA and FTA toward Effective AADL Error Modeling for Infusion Pump Systems (Progress Report) BaekGyu Kim

Size: px

Start display at page:

Download "Case Study of FMEA and FTA toward Effective AADL Error Modeling for Infusion Pump Systems (Progress Report) BaekGyu Kim"

Lionel Howard
5 years ago
Views:

1 Case Study of FMEA and FTA toward Effective AADL Error Modeling for Infusion Pump Systems (Progress Report) BaekGyu Kim

2 Architecture Analysis and Design Language The AADL is designed for the specification, analysis, and automated integration of real-time performance-critical distributed computer systems. Supporting analysis Architectural analysis Schedulability analysis Security analysis And more analysis by applying AADL language extensions Error Model annex Behavior annex 2

3 Example of AADL Model Example: Infusion Pump Controller 3

4 Overview of AADL Error Model Annex The Error Model Annex is a mean to express causal chains of system failures. [Fault] -> [Error] -> [Failure] Possible dependability analysis Fault-Tree Analysis (FTA) Failure modes and effects analysis (FMEA) 4

5 Example: Keypad Component Error Model Param_Repair: dh KB_Detect: 1-ph Incorrect Parameter Detected ErrorFree KB_: lambda Incorrect Parameter KB_Nondetect: ph Incorrect Parameter Non Detected KB out:1

6 Example: Keypad Component Error Model Textual format model Keypad_ErrorModel_v3 features ErrorFree: initial state; IncorrectParameter_Nondetected: state; (states) KB_: event {Occurrence => poisson lambda}; (events) KB out: out propagation {Occurrence => fixed 1}; end KeyPad_ErrorModel_v3 model implementation Keypad_ErrorModel_v3.impl transitions ErrorFree-[KB_]->IncorrectParameter; IncorrectParameter_Nondetected-[out KB out]->incorrectparameter_nondetected; (transitions) end Keypad_ErrorModel_v3.impl

7 The Issues AADL Error Model provides syntax/semantics to represent the []-[]-[failure] causal chains Lack of guideline in mapping the AADL model to Fault Trees ErrorFree Param_Repair: dh Incorrect KB_: lambda Parameter Incorrect Parameter Detected KB_Detect: 1-ph KB_Nondetect: ph Incorrect Parameter Non Detected KB out:1 AADL Error Model Fault Tree Analysis 7

8 The Objective The infusion pump case study using FMEA and FTA. 1. Battery 2. Keypad 3. Pump motor 4. Drop sensor 5. Buzzer To answer the questions: 1. What information should be described in creating AADL model toward dependability analysis using FTA? 2. How such information should be organized in the AADL Error Model toward dependability analysis using FTA? 8

9 The Approach 1. Study reported accidents related to the use of infusion pumps (e.g., from FDA s recall cases) Identify the related components. 2. Create FMEA What can go wrong in each component? 3. Create FTA What can go wrong to patients? 4. Suggest the effective way of creating AADL Error Model for infusion pump systems. 9

10 Failure Mode and Effects Analysis (FMEA) The purpose of the FMEA An inductive failure analysis of failure modes within a system for classification by the severity and likelihood of the failures. FMEA is typically used as a bottom-up tool 10

11 Example of RPN rating - FMEA applied to the Use of Infusion Pumps, IEEE EMBS,

12 Fault Tree Analysis The tree analysis is a popular technique for hazard analysis. A top-down hierarchical decomposition approach. To identify which combinations of hardware, software, human, or other s could cause safety-threatening hazards. 12

13 Preview of the Conclusion Over/under infusion Sustain > 10min Undetected Incorrect flow Incorrect flow rate Alarm Fail *Keypad failure Incorrect Infusion Parameter *Pump Motor failure Incorrect Motor-rate Inappropriate Alarm Sound *Buzzer failure Drop-Detection Failure *Drop Sensor failure Key-bounce Key-stuck Parameter conversion Wrong parameter by users Motor-Stall Unstable voltage Miscalculation of pump stroke Unstable Wiring Signal Generation Noise place Noise Interrupt-Processing Droplets stuck Misaligned light 13

14 Case Study: Batteries The use of batteries in infusion pump systems Provide mobility of patients during the infusion treatments. Provide reliable operations in case of unexpected power shortage. Examples of reported battery failures * over-heating of the battery and leads to premature battery failure. A patient returns from ambulating and forgets to plug in the infusion pump. The battery is not replaced during the recommended end of life routine maintenance. The infusion pump alarms with a low battery message, but the speaker volume is set too low, and the alarm goes unnoticed. *Example of Reported Infusion Pump Problems, FDA, 14

Recall Case 07/19/2005 Electronic problems caused the pump to shut down unexpectedly. 09/19/2005 The infusion pump battery could suffer damage, rendering it unable to operate on battery power.

15 Recall Case 07/19/2005 Electronic problems caused the pump to shut down unexpectedly. 09/19/2005 The infusion pump battery could suffer damage, rendering it unable to operate on battery power. 12/13/2005 Battery indicators could overstate the battery level; sensors could misinterpret tension in the tubing as air in the line. 07/20/2007 Upgraded replacements for the Baxter triple channel infusion pumps contain software defects which could cause the therapy to be interrupted. 01/23/2009 and battery usage failures could result in interruption of therapy, damaged battery messages, smoke and fire hazards, and serious injury and/or death. 03/12/2009 FDA issues class 1 recall on Baxter infusion pumps. 15

16 Low Battery Alarm Low Battery Alarm Battery Low Alarm will be triggered with audible alarm and visual flash light. KDS EZFlow 2040 Precise Infusion Pump 16

17 Low Battery Alarm Low Battery (Instruction in the User Guide) After receiving this alarm, DO NOT go to sleep without replacing the battery Paradigm Infusion Pump Model MMT

18 How Low Battery Status can be detected? Voltage Reference Circuit Power supply voltage regulators Analog-to-Digital converters Digital-to-Analog converters 18

19 Battery-Related Hazard - from A Hazard Analysis for GIIP Battery related issues are one of the source of hazardous situations in Insulin pumps. ID Cause Hazardous Situation Battery depletes without the user s awareness Underdose Battery depletes rapidly, giving the user insufficient time to respond 19 Underdose Voltage level of the battery is too low Overdose Underdose Voltage level of the battery varies greatly Overdose Underdose Battery life is unpredictable Overdose Underdose Battery is inadvertently disconnected from the pump. Underdose Battery impedance or contact impedance becomes too high Overdose Underdose * A Hazard Analysis for a Generic Insulin Infusion Pump, Yi Zhang et al., Journal of Diabetes Science and Technology, 2010

20 Case Study: FMEA of the Battery Component *Potential causes of the failure mode - perspective (HW) - perspective (SW) - perspective (ENV) Component Potential failure mode Potential failure effects S E V Potential causes O C C Current process controls D E T R P N Battery Battery indicators overstate the battery level. The pump may stop without user's awareness, which may lead to under infusion. 10 (HW) The voltage circuit is not able to detect the correct voltage level. 10 (SW) The software miscalcuate the voltage level based on the sensor reading. 5 None 10 3 None

21 Construction of FT at the Component Level -Battery component Overstated Battery-Level *Battery failure Voltage-circuit Miscalculated Battery-Level 21

22 Case Study: Key Bounce Defect Key bounce occurs when a number pressed on the pump registers twice although the operator only pressed the key once. Key bounce 4.8 ml/hr 44.8 ml/hr 22

Context: Key Bounce Defect The device manufacturer was issued warning letters by FDA in August 1998 and October 1999 Outlining the violations and was given opportunities to correct the

23 Context: Key Bounce Defect The device manufacturer was issued warning letters by FDA in August 1998 and October 1999 Outlining the violations and was given opportunities to correct the violations, but failed to take appropriate actions. The seized devices valued at more than an estimated 1.8 million dollars. 23

24 Analyzing the cascade effect of Keypad Defect Keypad defect may generate key-bounce. Key-bounce the infusion pump interprets a single keystroke as multiple keystrokes. Key-bounce may lead to the incorrect infusion parameter. E.g.) the user programs an infusion rate of 10 ml/hour, but the device registers an infusion rate of 100 ml/hour. Incorrect infusion parameter may lead to the system failure. Deliver a larger amount of drugs than expected. The system failure may lead to over-infusion hazard.

25 Case Study: FMEA of the Keypad Component Component Potential failure mode Potential failure effects S E V Potential causes O C C Current process controls D E T R P N Keypad Incorrect infusion parameter is entered. If it is not detected before pressing confirmation button, it may lead to over/under infusion depending on the entered parameters. 10 (HW) Key bounce 7 Caregiver's attention to the display (HW) Key stuck 4 Caregiver's attention to the display (SW) The entered infusion parameters do not match with the actual parameters used in software routines to calculate infusion rate/duration. 2 None (ENV)User enters wrong infusion parameters by mistake Education for caregivers 4 200

26 Construction of FT at the Component Level -Keypad component Incorrect Infusion Parameter *Keypad failure Key-bounce Key-stuck Parameter conversion Wrong parameter by users 26

27 Context: Pump-Motor Stall Motor stall is a critical failure causing a top-priority alarm. Important Considerations for Infusion Pump and Portable Medical Designs Cause Gear shaft wear. 27

28 Context: Pump-Motor Stall Reason for recall The pumps can stall at a higher rate due to gear shaft wear. If a pump motor stalls, drug delivery will stop suddenly and without warning. This stoppage will result in loss of therapy, return of the patient s symptoms, and/or symptoms of drug under-infusion or withdrawal. 28

29 Analyzing the cascade effect of Pump-Motor Defect Nondurable gear shaft may generate the motor stall. This is a mechanical happening due to the worn gear-shaft. The motor stall may lead to the incorrect flow rate. Typically flow rate becomes stopped or irregular. The incorrect flow rate may lead to the system failure. Drugs are not delivered at the specified rate. Lead to Under-infusion hazard.

30 Case Study: FMEA of the Pump Motor Component Component Potential failure mode Potential failure effects S E V Potential causes O C C Current process controls D E T R P N Pump Motor Pump motor is running at Incorrect motorrate. If the incorrect rate is sustained for a long time, it leads to under/over infusion 10 (HW)Motor stall 2 None (HW)Unstable supply voltage 5 None (SW) the device driver miscalculates the number/rate of pump stroke(rotation) given the infusion parameter. 3 None

31 Construction of FT at the Component Level -Pump motor component Incorrect Motor-rate *Pump Motor failure Motor-Stall Unstable voltage Miscalculation of pump stroke 31

32 Drop Sensor (Flow Rate Sensor) Purpose Detection of the drop of fluid that can be used to know the flow rate. Infrared LED is typically used. *Figure from Design and Implementation of controlled drug infusion system, R C Gupta et al. Journal of Scientific and Industrial Research 32

33 Drop Sensor(KDS EZFlow 2040 Model) 33

34 Drop Sensor Failure Cause of the failure 1. Tiny droplets get stuck to the walls of the drip chamber due to condensation in the chamber. These tiny droplets deflect the light rays from the source IR-LED away from the detector. 2. The light does not fall on the detector and it misses to register the drop count. Consequences 1. False negative case: missing drop count If the actual flow rate is higher than the normal flow rate, this could lead to over-infusion. 2. False positive case: wrongly sensing drop count If the actual flow rate is lower than the normal flow rate, this could lead to under-infusion. *Information from Design and Implementation of controlled drug infusion system, R C Gupta et al. Journal of Scientific and Industrial Research 34

35 Example: FMEA of the Drop Sensor Component Component Potential failure mode Potential failure effects S E V Potential causes O C C Current process controls D E T R P N Drop Sensor Drop Sensor fails to detect drops. In case of false negative, it leads to over-infusion. In case of false positive, it leads to under-infusion. 10 (HW) the sensor produces excessive noises. 3 None (SW) the device driver is not able to process the dropinterrupt. 5 None (ENV) tiny droplets get stuck to the wall of the drip chamber due to condensation in the chamber. 7 None (ENV) The light does not fall on the detector. 6 None

36 Construction of FT at the Component Level -Drop Sensor component Drop-Detection Failure *Drop Sensor failure Noise Interrupt- Processing Droplets stuck Misaligned light 36

37 Buzzer Buzzer is a critical hardware component to make audible alarms. 37

Recall Case: Plum A+ Infusion Pumps (Hospira) The reason of the recall Hospira has received customer reports of the Plum A+ infusers in which the audible

a delay or interruption of therapy which may result in serious injury and/or death Recall was issued on 07/03/2011 Root Cause (identified by Hospira)

38 Recall Case: Plum A+ Infusion Pumps (Hospira) The reason of the recall Hospira has received customer reports of the Plum A+ infusers in which the audible alarm has failed. Consequences of the failure the user may not be aware of the alarming conditions such as air-in-line or occlusion. a delay or interruption of therapy which may result in serious injury and/or death Recall was issued on 07/03/2011 Root Cause (identified by Hospira) failure of the piezoelectric assembly ("buzzer") due to improper mounting of components on the board assembly, poor solder application and breakage of internal wiring connections. 38

39 Example: FMEA of the Buzzer Component Component Buzzer Potential failure mode Buzzer fails to make appropriate alarm sound. Potential failure effects If alarming conditions do not trigger the buzzer sound, it may lead to over/under infusion. S E Potential causes V 10 (HW) the wire is not properly connected to PWM channel of the MCU. O C C Current process controls D E T 3 None 6 R P N (SW) the software fails to generate signals to buzzer in case an alarm condition is detected. 4 None (ENV) the pump is located in a place whose noise level is above the preset buzzer sound level. 4 None

40 Construction of FT at the Component Level -Buzzer component Inappropriate Alarm Sound *Buzzer failure Unstable Wiring Signal Generation Noise place 40

41 Extending the Component-Level Fault Tree Over/under infusion *Hazard level (What can go wrong to the patients?) What information should be filled? How such information should be organized in the tree? Keypad component Pump-motor component Buzzer component Drop sensor component 41

42 Idea: FTA Hierarchy for Infusion Pumps Separate the level of failure modes Infusion Pumps Physical Dynamics Patients C1 C2 C3 C4 C5 <System Components> Category 1) The mission is to provide infusion operation Category 2) The mission is to detect failure modes of (operation) components 42

43 The Idea: FTA Hierarchy for Infusion Pumps Level 1: Failure mode in patients E.g.) Over/under infusion Level 2: Failure mode in physical dynamics E.g.) Incorrect flow rate Level 3: Failure mode in operation/detection mechanism Failure in operations (e.g., incorrect motor rate) Failure in detections (e.g., alarm failure) Level 4: Failure mode in components E.g.) Incorrect infusion parameter is entered (Keypad component) E.g.) Drop is not detected (Drop sensor component) 43

44 Example of the Fault Tree Over/under infusion Undetected Incorrect flow Sustain > 10min Incorrect motor rate Alarm Fail Keypad fail Pump-Motor fail Buzzer Fail Drop-Sensor Fail 44

45 Example of the Fault Tree 1 st level: Patient Failure Mode Over/under infusion 2 nd level: Physical Dynamics Failure Mode Undetected Incorrect flow Sustain > 10min 3rd level: Operation/Detection Failure Mode Incorrect motor rate Alarm Fail 4th level: Component Failure Mode Keypad fail Pump-Motor fail Buzzer Fail Drop-Sensor Fail 45

46 Example of the Fault Tree Over/under infusion Sustain > 10min Undetected Incorrect flow Incorrect flow rate Alarm Fail *Keypad failure Incorrect Infusion Parameter *Pump Motor failure Incorrect Motor-rate Inappropriate Alarm Sound *Buzzer failure Drop-Detection Failure *Drop Sensor failure Key-bounce Key-stuck Parameter conversion Wrong parameter by users Motor-Stall Unstable voltage Miscalculation of pump stroke Unstable Wiring Signal Generation Noise place Noise Interrupt-Processing Droplets stuck Misaligned light 46

47 Example of the Fault Tree 1 st level: Patient Failure Mode Over/under infusion Sustain > 10min Undetected Incorrect flow Incorrect flow rate Alarm Fail *Keypad failure Incorrect Infusion Parameter *Pump Motor failure Incorrect Motor-rate Inappropriate Alarm Sound *Buzzer failure Drop-Detection Failure *Drop Sensor failure Key-bounce Key-stuck Parameter conversion Wrong parameter by users Motor-Stall Unstable voltage Miscalculation of pump stroke Unstable Wiring Signal Generation Noise place Noise Interrupt-Processing Droplets stuck Misaligned light 47

48 Example of the Fault Tree 2 nd level: Physical Dynamics Failure Mode Over/under infusion Sustain > 10min Undetected Incorrect flow Incorrect flow rate Alarm Fail *Keypad failure Incorrect Infusion Parameter *Pump Motor failure Incorrect Motor-rate Inappropriate Alarm Sound *Buzzer failure Drop-Detection Failure *Drop Sensor failure Key-bounce Key-stuck Parameter conversion Wrong parameter by users Motor-Stall Unstable voltage Miscalculation of pump stroke Unstable Wiring Signal Generation Noise place Noise Interrupt-Processing Droplets stuck Misaligned light 48

49 Example of the Fault Tree 3rd level: Operation/Detection Failure Mode Over/under infusion Sustain > 10min Undetected Incorrect flow Incorrect flow rate Alarm Fail *Keypad failure Incorrect Infusion Parameter *Pump Motor failure Incorrect Motor-rate Inappropriate Alarm Sound *Buzzer failure Drop-Detection Failure *Drop Sensor failure Key-bounce Key-stuck Parameter conversion Wrong parameter by users Motor-Stall Unstable voltage Miscalculation of pump stroke Unstable Wiring Signal Generation Noise place Noise Interrupt-Processing Droplets stuck Misaligned light 49

50 Example of the Fault Tree 4 th level: Component Failure Mode Over/under infusion Sustain > 10min Undetected Incorrect flow Incorrect flow rate Alarm Fail *Keypad failure Incorrect Infusion Parameter *Pump Motor failure Incorrect Motor-rate Inappropriate Alarm Sound *Buzzer failure Drop-Detection Failure *Drop Sensor failure Key-bounce Key-stuck Parameter conversion Wrong parameter by users Motor-Stall Unstable voltage Miscalculation of pump stroke Unstable Wiring Signal Generation Noise place Noise Interrupt-Processing Droplets stuck Misaligned light 50

51 A Proposed Strategy in Creating AADL Error Model 1. Create component model based on FMEA analysis Categorize each AADL component into two groups depending on its mission 1. Infusion operation 2. Detection of failure mode 3. Compose component models into higher level of model (through propagation semantics) 1. Error model for operation 2. Error model for detection 51

52 Summary and Next Work Case study of causal-effect analysis using FMEA and FTA. Separation of failure mode level in order to construct the tree. Next work Construct the AADL model according to the proposed modeling strategy. 52

53 53

54 Definitions - David Kalinsky, Architecture of safety-critical systems, EETimes, 2005 Fault defects or situations that can lead to failures. E.g.) a frozen memory bit, an uninitialized variable in software, or a cosmic ray ionizing its way through our embedded system. A may (or may not) lead to an. Error a manifestation of a as an unexpected behavior within our system. E.g.) an incorrect result of a calculation or a mistaken value of a state variable. Errors may (or may not) lead to failure. Failure A situation in which a system (or part of a system) is not performing its intended function. Hazard The potential to threaten injury or loss of life.

55 Definitions Fault--failure cascade can lead to life-threatening hazards. Component-Level Fault Error Failure Sub-System Level Fault Error Failure System-Wide Fault Error Failure Hazard Accident *Cited from David Kalinsky, Architecture of safety-critical systems, EETimes, 2005

Compatibility of STPA with GM System Safety Engineering Process. Padma Sundaram Dave Hartfelder

Compatibility of STPA with GM System Safety Engineering Process Padma Sundaram Dave Hartfelder Table of Contents Introduction GM System Safety Engineering Process Overview Experience with STPA Evaluation