PCPC QUALITY METRICS BENCHMARK SURVEY YEAR SEVEN 1
7 th Year Quality Metric Survey Business Success Through Continuous Quality Improvement 2
Background of PCPC Quality Metrics Survey Based on the Malcolm Baldridge Model for Excellence How does your performance compare to your overall industry performance? Includes Metrics used by member companies to manage product quality and corporate performance Metrics: FDA survey focuses on drug products PCPC survey focuses on cosmetics and drugs 3
Baldrige Excellence Framework A systems approach to improving your organization Baldrige Performance Excellence Program 2016 4
The Baldrige Approach in 3 Questions Is your organization doing as well as it could? How do you know? What and how should your organization improve or change? 5
A systems perspective Core values and concepts Criteria for Performance Excellence Scoring system 6
A Systems Perspective 7
Category Point Values 1 Leadership 120 2 Strategy 85 3 Customers 85 4 Measurement, Analysis, and Knowledge Management 90 5 Workforce 85 6 Operations 85 7 Results 450 (45%) Total 1,000 8
Evaluating Results (45% of Scoring Criteria) Levels: What is your current performance? Trends: Are the results improving, staying the same, or getting worse? Comparisons: How does your performance compare with others in your industry? Integration: Are you tracking important results? Are you using the results? 9
For more information Baldrige Excellence Framework booklets and free content Self-assessment tools Organizational assessments Training, conferences, executive education Award recipient profiles Case studies Connections to the Baldrige community Baldrige Performance Excellence Program www.nist.gov/baldrige www.nist.gov/baldrige 301-975-2036 baldrige@nist.gov 10
Survey Data Analysis While the final Metric report for 2016 will include detailed analysis of each of survey question, a top line historical summary is available now. Where appropriate, averages are shown and improvement or declining trends are highlighted. Best in Class is defined as the best 20% of the responding companies. 11
20 Metrics have been surveyed Company Identification Business Model Company Size Business Type Drug Registration # Sites Registered Quality Metrics % QA Budget PPM FG Defects Complaints Per Million % 1 st Pass Batch Approval % 1 st FG Approval % FG Micro Rejects % QA FTE / Mfg. FTE % Batch Record Accuracy 12
20 Metrics have been surveyed Regulatory Data PPM Annual Adverse Events PPM Annual Serious Adverse Events # FDA Inspections # 483 Observations OTC Metrics (New in 2016 Aware of OTC Qmetrics program Readiness for OYC Qmetrics Program Willingness to Benchmark OTC Qmetrics Program 13
Company Identification 14
#1 What is the size of your company? Size is defined by gross sales for the latest reporting period. Calculate as annual $ amount. 2016 Number of Companies Annual Sales (n=30) 14 12 12 12 10 8 6 4 2 0 4 2 0 <1 M 1 M - 10 M 10 M - 100 M 100 M - 1 B > 1 Billion 15
#2 Please identify whether your data are based on global or North American markets. 2016 9 21 Global North America 16
#3 Which of the following descriptions best fits your primary business? 2016 Primary Business Supplier of Packaging Components 0 Supplier of Raw Materials 1 Contract Manufacturer 5 Manufacturer of Finished Goods for Retail/Online Distribution 24 0 5 10 15 20 25 30 17
#4 Does your company have any sites that are registered with FDA as drug establishments? 2016 Companies Registered as Drug Establishment 13% Yes 87% No 18
# Companies #5 How many sites do you have that are registered with the FDA (drug and/or cosmetic)? # Sites Registered with FDA (2016) (n=30) Average = 4.5 Sites/Company 8 7 7 7 6 5 4 3 4 3 3 2 1 0 1 1 1 1 1 1 0 0 0 0 1 2 3 4 5 6 7 8 9 10 11 ~ 15 16 ~ 20 >20 # Sites Registered 19
Quality Metrics 20
Percent # Companies #6 What is your company's Total Quality Assurance Budget/Company Gross Sales? Total Quality Assurance Budget refers to total quality assurance/control spending and is defined as a % of Gross Revenue. Calculate as: $ Annual QA budget/$ Gross Annual Sales expressed as a % 7 6 5 4 3 2 1 0 2016 Percent QA Budget / Gross $Sales - All Responses Average = 1.3 % 0.1 0.2 0.3 0.3 0.4 0.5 0.6 0.6 0.8 0.8 1.0 1.0 1.1 1.3 1.3 1.4 1.5 1.5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 2.1 3.2 4.0 6.6 n = 23 8.0 7.0 6.0 5.0 4.0 3.0 2.0 1.0 0.0 History Percent QA Budget / Gross $ Sales 7.0 6.6 5.0 3.7 4.0 3.0 3.2 1.5 0.8 1.0 1.0 1.3 1.3 0.9 0.2 0.3 0.1 0.0 0.2 0.0 0.1 2010 2011 2012 2013 2014 2015 2016 Average Min Max 21
#7 What is your company's annual parts per million (ppm) Defective Finished Goods (FG)? A finished good unit is defined as the final packaged product in the shipping case. A defective FG unit has one or more defects. Calculate as: (# Defective FG units/# FG units produced) X 1,000,000 20,000 2016 Finished Goods PPM Defects By Company (n=18) 19,500 P P M 15,000 10,000 Industry Average 4,865 PPM 7,300 9,700 11,340 Lower is better 5,000 0 10 165 5,528 5,000 5,000 4,600 3,500 3,725 1,056 1,500 1,800 2,546 2,635 2,658 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Best In Class 22
#8 What is your company's annual quality complaints per million units sold? Quality complaints are defined as product and/or package quality complaints Calculate as: (Total # of product and package complaints / total units sold) X 1,000,000 2016 Complaints Per Million Units Sold (n=22) C O M P L A I N T S P E R M I L L I O N 1300 1200 1100 1000 900 800 700 600 500 Lower is Better Industry Avg 151 1164 400 300 200 100 0 279 296 214 141 162 85 86 100 104 106 108 115 117 45 52 60 61 1 1 1 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Best In Class 1-45 # Companies 23
#9 What is your company's annual % 1st pass batches? A batch is defined as the final in-process product ready for filling. A reject is defined as a batch that does not meet specification and is reworked or scrapped. Calculate as: (Total batches manufactured total batches rejected) X 100/Total batches manufactured 100 2016 % First Pass Batch Approval (n=22) Best In Class >99.2% 99 98 97 96 95 Higher is 94 93 92 91 Industry Avg 98.2 % 90 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
% 1st Pass FG #10 What is your company's annual % 1st pass finished goods? 1st pass finished goods is defined as the % finished goods that meet specification without any rework. Rework is defined as any additional labor expended to bring off-quality goods back into specification. Reject is defined as any product that does not meet specification and is scrapped. Calculate as: [Total units produced (total units reworked + total units rejected)] X 100 / Total Units produced 100.0 2016 % 1st Pass Finished Goods (n=22) Best In Class >99.6 % 99.0 98.0 Industry Avg 99.0% 97.0 96.0 95.0 Higher is Better 94.0 93.0 92.0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 25
% Micro Rejects #11 What is your company's annual % Finished Goods (FG) microbiological rejects? FG microbiological reject units are defined as the number of FG units that are out of microbiological specification. Calculate as: (# FG micro reject units / # FG micro units produced) X 100 0.30 2016 % Finished Goods Micro Rejects (n=23) 0.25 0.20 Lower is Better 0.15 0.10 Industry Avg.03% 0.05 0.00 0 0 0 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 BIC = 0% 26
% QA FTE/MFG FTE #12 What is your company's annual % for QA and QC Full Time Equivalent per Total Manufacturing Full Time Equivalent? QA and QC Full Time Equivalent or FTE is defined as total personnel engaged in QC and QA activities vs. true product or process development work. Manufacturing (Mfg) FTE is defined as total personnel engaged in manufacturing and direct support of manufacturing (not development or distribution.) Calculate as QA FTE/Total Mfg FTE x 100 26 24 22 20 18 16 14 12 10 8 6 4 2 0 2016 % QA FTE/Manufacturing FTE (n=23) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 # Companies Industry Avg 11% 27
QA FTE History 7 Year History % QA FTE/Manufacturing FTE 60.0 50.0 40.0 30.0 20.0 10.0 9.5 8.6 8.7 12.1 12.3 12.8 11.0 0.0 2010 2011 2012 2013 2014 2015 2016 Average: Min: Max: 28
#13 What is your company s annual accuracy level (right the first time) for batch records? Batch record is defined as the sum of all records/documentation used to produce a batch per your company s SOPs. Calculate as: (Total Number of Batch Records Meeting QA Requirements / Total Number of Batch Records Reviewed) X 100. Report calculation as numerical value, e.g., 85, 92.3, etc. Do not include symbols in answer box, (e.g., <, >, %, etc.) 100 90 80 70 60 50 40 30 20 10 2016 Annual % Batch Record Accuracy (n=17) Average = 78.9% 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Annual % Batch Record Accuracy 29
Regulatory Data 30
#14 For personal care products, what is your company s annual Adverse Event (AE) rate per million units sold? An Adverse Event is any undesirable health related experience associated with the use of a product. The event usually occurs within the normal and expected use of the product. Personal care products include makeup, hair care, skin care, fragrance, and related OTC products. Calculate a (Total AE / Total Products Sold) X 1,000,000. Report calculation as numerical value, e.g. 0.05. Do not include units or symbols in answer box (e.g., <, >, ppm.) 2016 Annual Adverse Events per Million Units Sold (n=13) 120 107 100 80 Average = 18.97 ppm BIC = >3 ppm 60 57 40 28 27 20 0 8.9 6.67 5.26 3 2 1.41 0.32 0.01 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Annual Adverse Events per Million Sold 31
#15 For personal care products, what is your company s annual Serious Adverse Event (SAE) rate per million units sold? Personal care products include makeup, hair care, skin care, fragrance, and related OTC products. SAE is defined in accordance with FDA s definition for drugs in 21 CFR 314.80(a) and is any adverse experience that: Results in any of the following outcomes: death, a life-threatening experience, inpatient hospitalization or prolongation of existing hospitalization, a persistent or significant disability or incapacity, or a congenital anomaly/birth defect; Requires, based on reasonable medical judgment, a medical or surgical intervention to prevent an outcome described above. Calculate as: (Total SAE / Total Products Sold) X 1,000,000 Report calculation as numerical value, e.g., 0. Do not include units or symbols in answer box (e.g., <, >, ppm.) 3.5 2016 Serious Adverse Events per Million Units Sold (n=12) 3 2.5 2 Average = 0.4 ppm 1.5 1 0.5 0 0 0 0 1 2 3 4 5 6 7 8 9 10 11 12 Serous Adverse Events per Million Units Sold 32
#16 How many FDA inspections (domestic and international) did you receive during this 12-month reporting period? 7 2016 # Annual FDA Inspections (n=23) (8/23 Had no Visits) 6 5 4 Average = 2 3 2 1 0 0 0 0 0 0 0 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 33
#17 How many 483 observations did you receive from these inspections? 2016 # 483's Issued (n=22)(10 companies had no 483 observations) 25 23 20 Average = 4 16 15 10 10 9 8 7 5 0 5 3 2 2 2 1 0 0 0 0 0 0 0 0 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 34
OTC Qmetrics 35
#18 Are you aware of FDA s quality metrics program which will create a reporting obligation for quality data related to drug products (including OTCs) produced in or for the US market? 2016 Awareness of FDA's Qmetrics program (n=25) Yes 40% No 60% 36
#19 On a scale of 1-5, what is your company s level of readiness to report quality metrics data to FDA? (1=just getting started; 5=very prepared) 18 16 17 2016 Level of Readiness (n=24)(average = 1.5) 14 12 10 8 6 4 2 0 3 1 0 0 1 2 3 4 5 37
#20 Would your company be interested in participating in an OTC industry quality metrics initiative to facilitate benchmarking via parallel analytics on the quality metrics data reported to FDA? 2016 Level of Interest Participating in OTC Qmetrics Benchmarking (n=25) Yes 56% 44% No 38
Summary 39
S U M M A R Y Company Identification Responses Company Size Business Type Registered as Drug Manufacturer # Sites Registered North American 2010 2011 2012 2013 2014 2015 2016 10 9 12 14 17 13 9 Global 11 8 16 27 39 11 21 Total 21 17 28 41 56 24 30 >$1B 10 10 12 18 24 13 12 $100M - $1B 6 3 6 16 21 8 12 $10M - $100M 4 3 8 5 9 2 4 $1M - $10M 1 0 2 1 2 0 0 <$1M 0 1 0 1 0 1 2 Finished Goods 43 20 29 Supplier 13 4 1 Yes 22/92% 26/87% No 2/8% 4/13% # 82 135 40
Quality Metrics S U M M A R Y % QA Budget / Revenue PPM Finished Goods Defects Complaints per Million Units Sold % 1 st Pass Batch Approval % 1 st Pass Finished Goods % Finished Goods Micro Rejects % QA FTE / Mfg. FTE 2010 2011 2012 2013 2014 2015 2016 Average 1.5% 0.8% 1.0% 1.0% 1.0% 0.9% 1.3% Average Average Average Average Average 3,759 ppm 2,563 ppm 13,358 ppm 24,581 ppm 22,555 ppm 11,743 ppm BIC 20 50 BIC 0-1 BIC 10-1,214 BIC 0 1,057 BIC 217-3,200 4,865 ppm BIC 25-452 BIC 10-1,500 156 cpm 140 cpm 264 cpm 165 cpm 166 cpm 189 cpm 151 cpm BIC.3 12 BIC 4-20 BIC 0.2-95 BIC 0-51 BIC.05-50 BIC 1-50 BIC 1-45 96.80% 98.40% 96.00% 98.40% 98.30% 98.50% 98.20% BIC 99.2 99.8 BIC 99.9-100 BIC 99.4-99.9 BIC 99.5-100 BIC 99.5-99.8 BIC 99.5-99.8 BIC >99.2% 98.4% 99.0% 98.8% 98.0% 98.3% 98.50% 99.00% BIC 99.4 99.9 BIC 99.7-99.9 BIC 99.5-99.8 BIC 99.6-100 BIC 99.7-100 BIC 99.3-99.5 BIC >99.6% 0.06% 0.18% 0.17% 0.06% 0.04% 0.08% 0.03% BIC 0 BIC 0 BIC 0 BIC 0 BIC 0 BIC 0 BIC 0 Average 9.50% 8.60% 8.50% 12.10% 12.30% 12.80% 11.00% % Batch Record Accuracy Average 79% 74% 79.00% BIC >90% BIC >90% BIC >90% 41
S U M M A R Y Regulatory Data Annual Advese Events (ppm) Serious Adverse Events (ppm) # FDA Inspections # 483 Observations Average 2010 2011 2012 2013 2014 2015 2016 11.95 ppm 13.65 ppm 18.97 ppm Average 0.005 ppm 0.010 ppm 0.4 ppm Average 0.76 1 2.00 Average 1.25 2 4 42
S U M M A R Y OTC Qmetrics Aware of Qmetrics Program Readiness for Qmetric Program (1-5) Willing to Participate in OTC Industry Metric Initiative 2010 2011 2012 2013 2014 2015 2016 Yes 60% 1= Just Started 5= Very Prepared Yes 44% 1.5 43
Reminder -- The complete 2016 annual metrics report will shortly be available on the PCPC website: http://www.personalcarecouncil.org/ QUESTIONS? Thank You!! 44