REVIEW OF RDE EVALUATION METHODS RDE-LDV meeting 19 July 2017 Ligterink, N.E. (Norbert), Bernd Rietberg, Willem Hekman, Pim van Mensch, Veerle Heijne, Rob Cuelenaere, Sam van Goethem
REVIEW OF THE EVALUATION METHODS SUMMARY OF MAIN ISSUES Large variation in emissions, seemingly unrelated to test conditions and driving dynamics probably, effects of payload, cold start, ambient temperature, etc. not reflected in the methods or boundary conditions. Many invalid tests, from combinations of different boundary conditions also invalidity from contradictory, or complementary, conditions sensitive for details of the input, e.g., WLTP values Limited number of multiple tests with the same vehicle variations over multiple axes at the same time with one-test-per-vehicle separate analyses on multiple tests with same vehicle (19 vehicles) Limited data on PN, focus on NOx only AECC (including PHEV) and Vienna data: difficult to draw conclusions on PN specifics.
STRUCTURE OF THE PRESENTATION Update on received data General set-up and processing Elements of RDE to check: evaluation methods and boundary conditions altitude, vapos and RPA Invalid tests: severity and complementarity of boundary conditions Comparisons of evaluations Comparisons of multiple tests with the same vehicle PHEV NOx/CO2 versus NOx/CO2*ICE Robustness: sensitivity study by varying input parameters Systematic effects: room for misuse Bounds and outliers Effect of the use of default input variables Conclusions
ORIGINAL LIST OF METHODS Annex IIIa (trip composit ion) Appendix 5 (or 6) Appendix 7a (trip dynamics) +7b Remarks distinct values in the results Option 1 X Boundaries without trip dynamics + no evaluation REFERENCE method =raw emissions (mg/km) Option 2 X X Full boundaries + no evaluation method REFERENCE =REFERENCE Option NL X X Raw emissions/co2 NOx/CO2 Baseline EMROAD X X 5 X RDE 3 Baseline (EMROAD) MAW Baseline CLEAR X X 6 X RDE 3 Baseline (CLEAR) PB EMROAD modified X X 5 modified In-between (T&E) x Mod 5 boundaries? EMROAD + Circular calculation LOOPED x Validity by Appen.5 + new boundaries for % as below, but Raw emissions (i.e. no calculation) REFERENCE ACEA new proposal x Mod 5 boundaries x Validity by Appen.5 + new boundaries for % New calculation method as NOx/CO2 except for PHEV vehicle
5 METHODS AND ONE ADDITIONAL BCS 4 VALUES + COLOURS REFERENCE (RAW + trip + dynamic BCS) with and without EMROAD BCS MAW (EMROAD) LOOPED MAW (windows going round to the beginning) PB (CLEAR) NOx/CO2 NOx/CO2 * ICE (ACEA) with and without EMROAD BCS (MODIFIED to be checked later)
UPDATE ON DATA SINCE 31/5 From many different parties: ACEA, AECC, DfT, EMPA, FIA, ICCT, JRC, TüV- Hessen, TüV-Nord, UTAC, TU-Vienna: late delivery 28/6 of AECC PHEV data (presented in RDE-LDV meetings) +/- 350 data files, 252 RDE-like trips (> 4000 secs, with all velocities) SUMMARY OF THE RESULTS max difference with RAW (completely valid) min difference with RAW (completely valid) total trips, all relevant boundary conditions from 252 method Option 1: only trip Composition 0% 0% 217 REFERENCE (RAW) with trip dynamics 0% 0% 168 RAW with MAW boundary conditions 0% 0% 81 MAW (EMROAD) 46.9% -51.2% 81 LOOPED MAW (windows round to the start) 46.9% -51.2% 59 PB (CLEAR) 24.9% -34.4% 34 NOx/CO2 58.3% -48.5% 168 NOx/CO2 * ICE with MAW boundary (ACEA) 48.5% -62.6% 81
PETROL --OR-- DIESEL Not all parties provided basic vehicle information. of 252 tests it is deduced that 77 are with petrol vehicles, 95 with diesel vehicles, and the remainder 80 tests with unknown vehicle type. The distinction between petrol and diesel vehicles should made. Additional useful information would be: Payload Cold start Euro class
IMPLEMENTATION & PRACTICAL ISSUES SIGNALS AND LEGAL TEXT Method implementation based on the comitology text: http://ec.europa.eu/transparency/regcomitology/index.cfm?do=se arch.documentdetail&dos_id=13782&ds_id=48924&version=4 Varying meaning of urban in the different methods, generally based on 0-60 km/h instantaneous velocity (also at rural roads and motorway). Several unclear points in the implementation. Eventually, correctness of the methods checked against EMROAD and CLEAR software, for a few tests. Velocity signal appears not always accurate: both noisy and (too) smooth Euro-5 vehicles excluded, for the vehicles it was known. Difference between legal text and CLEAR software on maximal data in first power bin (PB1: 60% (legal text: used) vs. 65%)
LEGAL TEXT A NUMBER OF ISSUES Our (deviating?) interpretation of the text: For PB: Up to class 6... including class 6 for more than 5 counts is a severe bound on the test currently used in TNO implementation For MAW: v < 1 km/h excluded, or skipped, or v set to zero only... not skipped (seems not consistent with JRC EMROAD software, but differences are very limited) textual and minor issues: A5.6.1 k = u, r,m k should be on a new line, K 22 should read k 22 A5.7.1 M CO should read M CO2 A5.6.3 f u, f r f m cumbersome, and difficult to read. A5.5.3 if tol1 can be increased to 50%,, 1/(tol1-tol2) will lead to singularities...
WLTP INPUT DATA RELEVANCE, AVAILABILITY AND USE Not all vehicles have all the input data needed to perform MAW and PB evaluation: for PB more often necessary input was absent than for MAW. In the case of limited input data, the proposal from JRC was adopted: 1. use input data when provided (72% for MAW, 50% for PB) 2. if certain input data is not available interpolated values on the basis of WLTP CO2 or test mass (most other cases) 3. If no data is available, the average values are used Effect of generic input is checked for vehicles for which all input data is provided WLTP data is specific and not in the full range of RDE: in RDE test mass can be higher than WLTP test mass high. Cold start part of WLTP, but not necessarily so in RDE and (likely) different (e.g ambient temperature).
DEFAULT WLTP INPUT VALUES PROVIDED BY JRC BASED ON CO2MPAS DB CO2MPAS DB vehicle_mass[kg] 1760.490873 WLTC_low[g/km] 201.4496572 WLTC_high[g/km] 139.5868138 CLEAR default values as provided, in the absence of input data. WLTC_extra_high[g/km] 167.4759841 Mref 1872.602653 f_0 164.1789662 f_1 0.622031363 f_2 0.037966265 Fits if Mref (WLTP [g/km] CO2) is provided: WLTC low = 1.3684 * (Mref*2/23.266) - 16.598 WLTC high = 0.8694 * (Mref*2/23.266) - 0.3216 WLTC xhigh = 1.0656 * (Mref*2/23.266) - 4.862
BOUNDARY CONDITIONS (BCS) MANY FLAVOURS Three main type of boundary conditions: trip composition and velocity length data provided with too short trips (not considered) velocity velocities over 170 km/h observed stopping 6%-30% lead to a number of invalid trips limited stops altitude, altitude gain, and altitude difference (Appendix 7b) raw signal poor, map data missing. trip dynamics (Appendix 7a) RPA seldom a problem v*apos some exclusions evaluation method CO2 band in MAW complementary to trip dynamics bin fractions in PB PB6 (total)/pb5 (urban) leads to many invalids
V*A POS [95%]: ALL DATA GREEN = TRIP VALID, RED = INVALID
RPA: ALL DATA GREEN = TRIP VALID, RED = INVALID
COMPLEMENTARITY OF BCS MAW + DYNAMICS + COMPOSITION = 56 EMROAD primary tolerance 1 #trips 25% 206 Valid (25 30]% 8 Invalid (25% + 38 3 invalids from other BCS in 25-30% range (u/r/m check) remainder after all BCS are applied Venn diagram without order: all BCS checked independently
COMPLEMENTARITY OF BCS PB + DYNAMICS + COMPOSITION = 34
SEVERITY OF POWER BINNING BCS MOST INVALIDS ON URBAN
TAKE AWAY: BOUNDARY CONDITIONS MOST EXCLUSIONS FROM MAW AND PB BOUNDARY CONDITIONS. DROPPING MORE THAN HALF OF THE TESTS AFTER TRIP COMPOSITION AND TRIP DYNAMICS ALREADY REMOVED A THIRD OF THE TESTS. GENERAL TEST VALIDITY IS AN ISSUE FOR MAW AND PB: MANY TESTS EXCLUDED
THE PURPOSE OF THE EVALUATION METHOD IS TO REDUCE THE VARIATION OF THE RESULTS (FOR THE SAME DETERMINISTIC VEHICLE) FOR THE VARIATION IN THE TEST EXECUTION. THE VARIATION IN THE TEST (REFLECTED IN THE EVALUATION METHODS) COMPRISES MAINLY OF VELOCITY AND CO2-EMISSION RELATED ASPECTS. EFFECT OF PAYLOAD AND ROAD SLOPE IS PARTLY REFLECTED IN THE EVALUATION, I.E., VIA CO2. THE EFFECT OF COLD START AND AMBIENT CONDITIONS ARE NOT REFLECTED.
only trip composition OPTION 1: (NO CHANGE IN THE RESULTS) INVALID ON TRIP COMPOSITION ALONE number of valid tests varies trip composition and driving dynamics
COMPARISON OF METHODS: NOX/CO2 ALL TRIPS VERSUS VALID TRIPS no additional conditions
COMPARISON OF METHODS: MAW
COMPARISON OF METHODS: PB
COMPARISON OF METHODS: ACEA including MAW boundary condition minor differences in data points with NOx/CO2 (only for PHEV data, see later)
TAKE AWAY: EVALUATION METHODS REMAIN WITHIN A BAND AROUND THE REFERENCE RESULT. FOR VALID TESTS LIMITED SYSTEMATIC EFFECTS BASED ON A LIMITED NUMBER OF VALID TESTS FOR MAW AND PB.
EVALUATION METHODS MEANT TO REDUCE THE VARIATION IN TEST RESULTS IN A SIMILAR MANNER, I.E., WITH VARIATION IN TEST EXECUTION. HENCE, IT IS EXPECTED THAT THE EFFECTS OF THE DIFFERENT METHODS ACT IN THE SAME DIRECTION.
COMPARISON TWO METHODS MAW VERSUS PB
COMPARISON TWO METHODS MAW VERSUS NOX/CO2
COMPARISON TWO METHODS PB VERSUS ACEA (NOX/CO2 * ICE)
COMPARISON TWO METHODS MAW VERSUS ACEA (NOX/CO2 * ICE) same MAW boundary condition
COMPARISON TWO METHODS MAW VERSUS MAW LOOPED looped: to better account for cold start, effect mainly in reduced test validity
COMPARISON TWO METHODS NOX/CO2 VERSUS NOX/CO2*ICE (PHEV)
COMPARISON TWO URBAN METHODS PN/CO2 VERSUS PN/CO2*ICE (PHEV)
TAKE AWAY: LIMITED CONSISTENCY IN THE EFFECTS OF THE DIFFERENT EVALUATION METHODS RESULTS DO NOT CHANGE IN THE SAME DIRECTION FOR MAW AND PB (NOX/CO2)*ICE (ACEA) REDUCES THE HIGHEST NOX AND PN EMISSIONS ASSOCIATED WITH COLD START WITH RESPECT TO NOX/CO2 METHOD
SYSTEMATIC EFFECTS: DIFFERENT RDE TRIPS WITH THE SAME VEHICLE IF EVALUATION METHODS REDUCE VARIATION IN RESULTS PER VEHICLE, THE EFFECT WILL BE BEST VISIBLE FOR DIFFERENT TESTS WITH THE SAME VEHICLE. ANALYSES OF 18 VEHICLES WITH MULTIPLE RDE-VALID TESTS (3 OR MORE PER VEHICLE)
VEHICLES WITH MULTIPLE TESTS CHANGE IN VARIANCE FOR PB s y expectation for a effective evaluation method: s x > s y lines and colours to indicate fit for each individual vehicle s x 10% reduction If PB BCS are applied only 4 vehicles remain. The variance also increases.
VEHICLES WITH MULTIPLE TESTS CHANGE IN VARIANCE FOR MAW 20% increase, large spread including invalid MAW
VEHICLES WITH MULTIPLE TESTS CHANGE IN VARIANCE FOR NOX/CO2 no change
VEHICLES WITH MULTIPLE TESTS CHANGE IN VARIANCE FOR ACEA no change excluding invalid (difference with NOx/CO2)
VEHICLES WITH MULTIPLE TESTS MAW URBAN ZOOM-IN ON 0-200 MG/KM 20% increase, large spread
VEHICLES WITH MULTIPLE TESTS PB URBAN ZOOM-IN AT 0-200 MG/KM 10% increase
VEHICLES WITH MULTIPLE TESTS NOX/CO2 URBAN ZOOM-IN 0-200 MG/KM 20% reduction if MAW boundary (ACEA) is not used: too few vehicles remaining
VEHICLES WITH MULTIPLE TESTS ACEA URBAN ZOOM-IN 0-200 MG/KM
TAKE AWAY: LIMITED EFFECTIVENESS OF REDUCING THE VARIATION IN DIFFERENT TEST RESULTS FOR THE SAME VEHICLE MAW INCREASES VARIATION BY 20%, URBAN AND TOTAL PB REDUCES VARIATION BY 10% ON TOTAL TRIP AND INCREASED URBAN BY 10% NOX/CO2 AND NOX/CO2*ICE (ACEA) TOTAL TRIPS REMAINS CONSTANT, NOX/CO2 AND NOX/CO2*ICE (ACEA) URBAN TRIPS YIELDS A 20%URBAN REDUCTION
varied emission evaluation result ROBUSTNESS OR- LIMITED SENSITIVITY AGAINST VARIATION OF THE INPUT AND PARAMETERS Small variation (~10%) of the input (e.g. WLTP CO 2 value or test mass) or the parameters (e.g. velocity bins) should not affect the outcome significantly. This should hold for both the emission evaluation result and the boundary conditions. always invalid always valid x-valid, y-invalid x-invalid, y-valid expected bandwidth on the basis of the input variation standard emission evaluation result
VARYING THE INPUT BY 10% AFFECTS THE RESULT BY MORE THAN 10%
CHANGING THE WLTP INPUT 10% CHANGES SOME VALID/INVALID
PB INPUT VERSUS DEFAULT INPUT SHOWS A LARGE EFFECT ON RESULTS AND VALIDITY worst case, based on the default values (not scaled)
TAKE AWAY: STRONG SENSITIVITY FOR INPUT DATA NOT ALWAYS A CHANGE IN RESULTS, BUT THE CHANGE IN TEST VALIDITIES IS GENERALLY LARGE. PB RESULTS MOST SENSITIVE FOR VARIATION IN INPUT DATA
PRELIMINARY GENERAL CONCLUSIONS No clear variation in test executions (i.e. driving behaviour and trip composition) with the same vehicles (except for a few vehicles) affecting the evaluation. The variation in the results not significantly reduced by the evaluation methods. However, evaluation methods bring limited benefit over raw results to justify the risks of deviating results, which include: high invalid rates, which vary strongly with input values possibility of large (outlier) corrections from evaluation, both up and down Some systematic effects per vehicle observed, prone to exploitation The observed variation in test results are treated differently by the different evaluation methods.
PRELIMINARY CONCLUSIONS ON METHODS Validity of tests: many invalid tests, also on unexpected conditions, like stop fraction invalid inconsistent between methods (e.g. valid for MAW, invalid for PB) method-invalid complementary to trip-invalid (dynamic, composition) MAW BCS as additional condition on trip BCS: add to the bounds on the trip, but for the PHEV it leads to a high number of invalid trips Possibly, it may lead to problems to achieve a valid test for high payload or cold start variation NOx/CO2 * ICE (ACEA) reduces variation on PHEV data, reducing the highest NOx/CO2 values down (i.e. cold starts with low ICE ratios) Variation remains large in all methods, reduction of variation limited to 20% maximal.
THANK YOU FOR YOUR ATTENTION Take a look: TIME.TNO.NL