IMPROVING MOTOR CARRIER SAFETY MEASUREMENT

Transcription

1 IMPROVING MOTOR CARRIER SAFETY MEASUREMENT Panel on the Review of the Compliance, Safety, and Accountability (CSA) Program of the Federal Motor Carrier Safety Administration Initial Comments by Steve Bryan, President & Founder of Vigillo, A SambaSafety Company June 28, 2017 Yesterday, June 27, 2017, 18 months after the FAST Act removed CSA Scores from public view, the National Academies of Science, Engineering and Medicine (NAS) published their preliminary findings outlining their recommendations to reform the Compliance, Safety and Accountability program we all know and love as CSA. Their 132 page report starts with a nod to the FMCSA for the work they have done on CSA and for the funding for this study (amount undisclosed). NAS starts by thanking Joseph DeLorenzo, director, FMCSA Office of Enforcement and Compliance, for his presentations, which were not just useful, but enormously useful, way to go Joe! That was followed by acknowledgements to the other twenty-five presenters the first of which was yours truly (it was alphabetical). We are all thanked but our contributions are not characterized on a usefulness scale. Item Response Theory (IRT) applied to the relative informational value of each of us would have been nice, but hey, more on that later. NAS categorizes their report into the following four sections: 1. Assessment of the Current Safety Measurement System 2. A More Natural Statistical Model 3. Data Improvements 4. Transparency, Reproducibility, and Public Disclosure of Safety Rankings In one paragraph, I ll summarize. My take on this entire 132 page report is stated as follows.

2 Good job FMCSA on what you have done with CSA so far, we (NAS) give you a solid 5 out of 10 for your efforts so far. But there needs to be a sound statistical model applied to replace the ad hoc nature of how CSA is formulated today. More, and more reliable data needs to be gathered, and the data utilized needs to be more transparent and software developed to make such data more useful by all stakeholders. Only then should CSA Scores be released to the public. My take on the Report So I m going to very briefly summarize my thoughts on each of the 4 categories of the NAS Report. I have distilled 132 pages to 5 ½. There are some specific topics down in the trees such as how to blend relative and absolute scoring, I ll leave those for a follow up, I ve had 24 hours with the report and so has the Vigillo team, I wanted to get out some thoughts at a high level, so here we go 1. ASSESSMENT OF THE CURRENT SAFETY MEASUREMENT SYSTEM NAS Recommendation: SMS is structured in a reasonable way, and its method of identifying motor carriers for alert status is defendable. However, much of what is now done is ad hoc and based on subject-matter expertise that has not been sufficiently empirically validated. This argues for FMCSA adopting a more statistically principled approach that can include the expert opinion that is implicit in SMS in a natural way. My Commentary: NAS states that Roadside Inspection data (MCMIS) cannot be utilized to build a predictive model that identifies the crash risk of individual motor carriers. Identifying the crash risk of individual carriers is a specific requirement of the FAST Act. NAS has, in its first conclusion supported the structure of SMS but called out the lack of sound statistical underpinnings. The industry has always supported the concept of CSA but identified early on that FMCSA s ad hoc approach to the methodology, severity weights and BASICs largely lacked any statistical support. My Summary: I agree that the CSA concept is sound. I also agree that there has never been an identifiable statistical model that FMCSA has ever relied upon. NAS gives more credit than I would to the subject matter expertise that originally architected CSA. Something like CSA makes sense, it needs science behind it. 2. A MORE NATURAL STATISTICAL MODEL NAS Recommendation: FMCSA should develop the suggested IRT model over the next 2 years. If it is then demonstrated to perform well in identifying motor carriers for alerts, FMCSA should use it to replace SMS in a manner akin to the way SMS replaced SafeStat. My Commentary: NAS proposes an IRT Model. So what is an IRT model? Item Response Theory has been largely used for years in transportation, aviation, health care, and education and it can be explained easily how it might be applied to CSA. Below the surface, IRT lies in the domain of PhD statisticians and is monstrously complex. But the concept is easily explained. Think of a test you d take in school. It has ten questions. Sally takes the test and gets 2 incorrect. Sally gets an 80%. Steve takes the test and gets 2 incorrect, a different 2. Steve gets an 80%. Little

3 can be learned about Steve vs Sally because we got the same score but got different questions (items) wrong. What is hidden and what IRT reveals is that Sally s questions were far more difficult than Steve s. Only by applying sound statistics to the items, the individual questions, can we begin to get insight into the differences between Sally and Steve. IRT does this analysis There is a very good analogy to CSA. Carrier A has a BASIC % of 80% in the Maintenance BASIC and Carrier B has an 80% in the same BASIC. They are equivalent, right? Maybe not. Carrier A operates a proportionately high number of miles in Texas and receives many violations because of the Texas focus on Maintenance. Carrier B does not operate in Texas, they just have really crappy trucks. IRT will help explain this, and can begin to explain and balance some of the defects in CSA like disparate State Enforcement. NAS also believes that IRT can assist in the areas of: account for the probability of being selected for inspection provide a basis with which to evaluate how data insufficiency could impact safety ratings provide a basis to more rigorously and empirically evaluate the utility of individual violations allow severity weights to change over time (e.g., as violations become more or less prevalent); determine empirically whether severity weights should be different for trucks versus passenger carriers; enable adjustment for factors that may be outside a carrier s direct control accommodate new violations over time My Summary: IRT can improve CSA, even with data that is not as rich as we d like it to be, and even in a complex environment like trucking, with 50 enforcement regimes, millions of trucks, differing cargo types (tasks) and a huge variety of operating environments, weather, mountains and congested cities. Warning: IRT, while well understood by statisticians and widely used for years and in many different applications, is far more complex than the current CSA methodology. Safety professionals in trucking are going to have to learn to explain to drivers, customers, bosses and shareholders how all of this works. Let s look at the existing methodology for Vehicle Maintenance vs the IRT Modeling Strategy. Existing Vehicle Maintenance Methodology: Proposed IRT Model (excerpt from NAS Report)

4 The Science of the IRT model is far superior to the ad hoc pseudo-science of the current methodology. Great statistically. Now its Monday morning, your phone just rang, a shipper wants to know why you have an Alert in Vehicle Maintenance, what do you say? 3. Improvement of MCMIS Data NAS Recommendation: FMCSA should continue to collaborate with states and other agencies to improve the quality of MCMIS data in support of SMS. Two specific data elements require immediate attention: carrier exposure and crash data. The current exposure data are missing with high frequency, and data that are collected are likely of unsatisfactory quality. Further, to improve the exposure data collected involves not only collecting higher-quality VMT data, but also collecting this information by state and by month. This will enable SMS to (partially) accommodate existing heterogeneity in the environments where carriers travel. Crash data are also missing too often. Also, there is information available from police reports currently not represented on MCMIS that could be helpful in understanding the contributing factors in a crash. Such information could help to validate the assumptions linking violations to crash frequency. To address these issues, FMCSA should support the states in collecting more complete crash data, and in universal adoption of the Model Minimum Uniform Crash Criteria, as well as developing and supplying the code needed to automatically extract the data needed for the MCMIS crash file. My Commentary: NAS breaks down the goal of improving data into quality of data that is currently collected, and quantity of new data that might be collected.

5 Quality of existing data focuses on VMT (miles) and APU (truck count). These are both reported today on the MCS-150, but as NAS correctly points out the impact of flawed or out-of-date VMT and APU data on SMS percentile ranks may not be fully appreciated by the carriers. Better reporting of these two data elements would improve the data quality of CSA and make an IRT model more trustworthy. When we move to Quantity, the NAS suggestion that more data be provided, while a worthy goal statistically, becomes problematic (huge understatement) in practice. NAS suggests that VMT by State, cargo type, driver pay and pay model and driver retention rates all be reported to FMCSA. There is also a recognition of the data that could be available as a result of the ELD mandate, and NAS suggests that that data could be reported to FMCSA. I m trying to think of the right word here My Summary: Aint not never gonna happen. Where I drive, what cargo I carry, what and how I pay my drivers and how successfully I retain them, combined with HOS data that would show location, speed, direction, time/date, perhaps shipper detail, is never, on this planet, in this universe, going to be provided by motor carriers to the FMCSA. This is the soul of a very competitive industry and this data is solid gold and will never be made available to FMCSA. The competitive nature, combined with years of distrust of the FMCSA makes this, in my opinion, the weakest component of this entire 132 page report. If this data is to be aggregated, FMCSA is not the one to do it, they have no authority and their reputation for aggressive interventions based on flawed data precludes them as a trusted candidate. Sorry Joe, you ll never see this data. 4. TRANSPARENCY, REPRODUCIBILITY, AND PUBLIC DISCLOSURE OF SAFETY RANKINGS NAS Recommendation: FMCSA should structure a user-friendly version of the MCMIS data file used as input to SMS without any personally identifiable information to facilitate its use by external parties, such as researchers, and by carriers. In addition, FMCSA should make userfriendly computer code used to compute SMS elements available to individuals in accordance with reproducibility and transparency guidelines. Steve Commentary: MCMIS data is a mess, its always been a mess. Vigillo spends 20-30% of our time cleaning the data before we even begin to analyze it. NAS has seen this, and that combined with the far more complex IRT model, will require a new focus on MCMIS. Better and more consistent collection from the states, a much better understanding of some of the flaws in how its reported and now combined with the need to give non-phd s the ability to run the analysis, is absolutely critical if CSA is to move forward. My Summary: I hope the FMCSA accepts the recommendations of NAS and seriously begins to work on the next phase of CSA, or whatever we call it. FMCSA must strike a more collaborative relationship with the industry, and demonstrate absolute transparency in how they are collecting, analyzing, and taking action on this new data model. NAS suggests that outside experts be hired,

6 perhaps from academia, maybe there are some of us in the industry that know a little about this too. It s a moment in time where perhaps, a less combative, more inclusive partnership can be developed and we can re-boot from the very controversial launch of CSA in I m hopeful, I m appreciative of the work NAS has done, now we need to see what FMCSA s response is in 120 days oops 119. Then we can discuss when scores are public again. Steve s.bryan@vigillo.com