ISTA Referee Test Programme

Transcription

1 ISTA Referee Test Programme ISTA Referee Committee Doug Ashton Ottawa Laboratory (Carling) - Seeds Canadian Food Inspection Agency ISTA Statistics Seminar, Corvallis, Oregon, June 25,

2 Topics!Part I - Overview of ISTA Referee "Purpose "Design and preparation "Evaluation of results "Consequences "Benefits Oregon, June 25,

3 Topics!Part II - Statistics "Homogeneity of samples "Data handling (outliers) "Tattersfield Method (Purity and Germ) "Other Seed Determination Data "Laboratory rating Oregon, June 25,

4 Definitions! Referee "For this programme, Referee test = Proficiency test! Purity "2500 seed subsample separated into 3 components: Pure Seed; Other Seeds; Inert Matter "Report percent by weight! Other Seed Determination (OSD) "Impurities retrieved from the purity subsample "Report numbers found! Germination "400 seedlings separated into 3 components: Normal seedlings; abnormal seedlings; dead seeds "Report percent by number Oregon, June 25,

5 Part I Overview of ISTA Referee Programme ISTA Statistics Seminar, Corvallis, Oregon, June 25,

6 Purpose Confidence in results issued by accredited laboratories.

7 Why?!Proficiency testing "Method for labs to demonstrate ability to report accurate results "An essential element of laboratory accreditation "Credibility of ISTA Certificate!Continuous improvement Oregon, June 25,

8 Considerations!Must be fair!must be meaningful!use typical seed lots (in commerce)!quality control of samples essential!use consensus results!transparent evaluation of results!useful feedback to participants Oregon, June 25,

9 Design (purity, germ, OSD)!3 Rounds per year "Each Round has 3 samples (same species)!six crop groups "grasses, cereals, small legumes, pulses, other field crops, vegetables "Representative species for each group e.g. Wheat = cereals!add impurities to each sample "2 to 4 seeds of 3 or 4 species Oregon, June 25,

10 Participants!Mandatory for ISTA Accredited Labs "80 accredited member labs "Depends on scope of accreditation Crop groups (grasses, legumes, etc.) Methods (Purity, Germ, Other Seeds, Moisture, Seed Health, Vigour, etc.)!voluntary for non-accredited "73 non-accredited member labs Oregon, June 25,

11 Sample preparation Test Leader Brassica 1 Brassica 2 Brassica 3 2 Silene dichotoma 3 Vicia cracca 2 Argemone sp. 4 Galium aparine

12 Participating Laboratory - Purity and other seeds Brassica 1 No. 42 Brassica 2 No. 42 Brassica 3 No. 42

13 Participating Laboratory - Germination Brassica 1 No. 42 Brassica 2 No. 42 Brassica 3 No. 42

14 Results from approx. 120 labs Secretariat Data Entry Preliminary Report to Participating Labs Data to Test Leader Final Report to Participating Labs

15 Analysis of results!purity and germ "Statistical analysis using Z-scores "Mean of all participants used "Scores added across three samples!other seed determination "Percentage of retrieved and correctly identified seeds!review of reporting procedures and use of ISTA Rules Oregon, June 25,

16 Output - Purity and Germ Oregon, June 25,

17 Rating System!For each round, lab rated: "A, B, C, BMP (Below Minimum Performance) "General summary report for A and B labs "Detailed report for C and BMP labs!cumulative six-round rating: "A, B, C, BMP "Warning letter to C labs "Suspension of BMP labs Oregon, June 25,

18 Challenges!Global membership in ISTA "Difficult to select globally representative crop and impurity species "Starting with different levels of lab competencies!partial accreditation "One or two crop groups only "Delays accumulation of data for rating Oregon, June 25,

19 Benefits!Continuous improvement!independent evaluation of lab competency!credibility of system Oregon, June 25,

20 Part II Statistical Considerations ISTA Statistics Seminar, Corvallis, Oregon, June 25,

21 Homogeneity of sub-samples!to eliminate doubt!10 randomly selected samples!test for purity, germination, other seeds!use Miles tolerances to detect heterogeneity Oregon, June 25,

22 Homogeneity testing results - Germination Lot 1 Subsample Normal Sdlgs Lot 1 Mean 93.8 Tolerance 6.45 Range between tests 6.00 Lot 1 Homogeneity OK S = Fσ F = 5.16 (1% prob. 10 tests) σ = ( pq) n S = 5.16 ( p 0.5)*((100 p) + 0.5) 400

23 Data handling - Outliers!Tattersfield Method "SST, 1979, 7(2): !Median of all results (m)!variance of results above median (s 2 )!Outliers - Results outside m ± ts Oregon, June 25,

24 Example: Chickpea Germ!16 Labs!Sand vs Soil "Sand data used for example Oregon, June 25,

25 Chickpea Sand

26 Chickpea Soil

27 Chickpea germination - Sand Lot 1 Lot 2 Lot 3 Lab Normal Lab Normal Lab Normal Mean

28 Chickpea germination - Identification of outliers Lot 1 Lot 2 Lot 3 Lab Normal Lab Normal Lab Normal Median Mean Count Median (m) Variance above m St. Dev. (s) Degrees Freedom Student's t Upper limit (m+ts) Lower limit (m-ts)

29 Final Mean and S.D.!Arithmetic mean of non-outlier results!standard deviation around new mean Oregon, June 25,

30 Chickpea germination - Calculation of final mean and standard deviation Lot 1 Lot 2 Lot 3 Lab Normal Lab Normal Lab Normal New Mean (X) Count New Variance New St. Dev. (s) Degrees Freedom Student's t Upper limit (X+ts) Lower limit (X-ts)

31 Normal Scores (Z-scores)!Uses consensus mean and st. deviation!allows combining results from different samples of the same species!deviation of individual result from mean, divided by standard deviation "Outliers excluded from mean and s.d. "Z-scores also calculated for outliers Oregon, June 25,

32 Normal Scores (Z-scores) Z -score = (participant result) - (mean of acceptable results) (standard deviation of acceptable results) Z score = x X (x X ) n 1 2 Z score = = 1.6 Oregon, June 25,

33 Chickpea germination - Calculation of Z-scores Lot 1 Lot 2 Lot 3 Lab Normal Z Lab Normal Z Lab Normal Z Mean St. Dev

34 Z-scores - addition!measure of lab performance with a species and method!measured against peers!not influenced by sample quality Oregon, June 25,

35 Chickpea germination - Sum of Z-scores Lot 1 Lot 2 Lot 3 Lab Normal Z Normal Z Normal Z Sum Z Mean St. Dev

36 Lab Performance - Within Round!Still under discussion!critical value of sum of Z-score absolute values: 5.3!Include count of out of tolerance results Sum Abs Z Pure seed: 3.9 Other seed: 6.6 Inert: 0.7 Oregon, June 25,

37 Critical value: 5.3 (from Tattersfield, 1979)

38 Chickpea germination - Lab Rating using sum of Z scores - Under discussion Lot 1 Lot 2 Lot 3 Lab Normal Z Normal Z Normal Z Sum Z Absolute Sum Z Lab Rating A A A A A A A A A A B BMP BMP BMP BMP BMP Mean St. Dev

39 Another method?!magruder Check Sample Program "Feed and Fertilizer proficiency testing!uses Z-scores to calculate bias, precision, accuracy!lab rating based on accuracy score Oregon, June 25,

40 High Bias + High Precision = Low Accuracy

41 Low Bias + Low Precision = Low Accuracy

42 Low Bias + High Precision = High Accuracy

43 Bias!Average Z-score for lab!indicator of systematic error!significance of values not determined!0 = No bias!>1.5 = Lab should review procedures Bias = B = n Z Oregon, June 25,

44 Precision!Measure of consistency!low value = consistent results!high value = variable results Precision = ( Z n B) 2 Oregon, June 25,

45 Accuracy!Combination of bias and precision "Low = lab consistently near overall mean "High = lab has bias and/or is inconsistent >1.5 cause for concern >2.0 lab may have serious problem Accuracy Z = n 2 = ( bias) 2 + ( precision) 2 Oregon, June 25,

46 Laboratory Rating (Round) Magruder Check Sample Program Accuracy Rating A Acceptable B Critical and over C Unacceptable Oregon, June 25,

47 Chickpea germination - Lab Rating using Magruder method Lot 1 Lot 2 Lot 3 Lab Normal Z Normal Z Normal Z Sum Z Absolute Sum Z Bias Precision Accuracy Rating A A A A A A A A A A A BMP BMP BMP BMP BMP Mean St. Dev

48 Soybean germination - 62 labs Comparison of rating methods: Sum of absolute Z scores vs. Magruder Accuracy Sum Abs Z-scores Accuracy More More

49 Rating Other Seeds Retrieval!Based on retrieval rate: "90% - 100% retrieval "85% - 89% retrieval "80% - 84% retrieval " 0% - 79% retrieval A B C BMP Oregon, June 25,

50 Other Seeds Retrieval (Round) No. Added No. Retrieved % Retrieved Sample 1 Silene dichotoma 2 2 Vicia cracca 3 3 Argemone sp. 2 1 Galium aparine 4 0 Sample 2 Species Species Species Species Sample 3 Species Species Species %