January 12, Dear Mr. Myers and Mr. Cohen:

Transcription

1 January 12, 2017 Kurt J. Myers, Deputy Secretary of Transportation Roger J. Cohen, Policy Director Pennsylvania Department of Transportation Keystone Building 400 North St. Harrisburg, PA Dear Mr. Myers and Mr. Cohen: The Pennsylvania Department of Transportation (PennDOT) has requested comments on the Pennsylvania Autonomous Vehicle Testing Policy. The Insurance Institute for Highway Safety (IIHS) is encouraged by the potential for driving automation systems to help reduce, if not someday eliminate, crash injuries and deaths. Our research has already documented the benefits of some crash avoidance technologies that underpin driving automation. However, it also has shown that not all such technologies are preventing crashes. It will be important to continually monitor the safety and performance of driving automation systems in the real world to guarantee that it delivers on its promise. IIHS encourages PennDOT to require testing companies to provide information about every crash and disengagement of an automated driving system that occurs during testing on public roads in Pennsylvania. We also would like to offer some suggestions for improving the quality of the information the department plans on collecting to better support evaluations of safety and performance. The importance of collecting consistent and reliable information about driving automation systems cannot be stressed enough. We already have observed how imprecise requirements can result in incomplete and inconsistent information about the number and nature of safetyrelated disengagements and total use of driving automation systems among companies testing these systems on public roads in California. In our comment to the California Department of Motor Vehicles (attached), we recommended that California s draft regulations for the deployment of autonomous vehicles be amended to provide specific details and a structured method for companies to report information about crashes and safety-related disengagements. The Pennsylvania Autonomous Vehicle Testing Policy also should include detailed guidance about the information companies are required to provide in order to ensure consistent, complete, and reliable information. It is crucial that this information be as precise and accurate as possible because researchers are using such data to draw insights into how driving automation systems perform in the real world (e.g., Dixit, Chand, & Nair, 2016). Several areas of the policy could be strengthened to require companies to report more information that would be useful for evaluating the safety and performance of driving automation systems. Testing companies are required to report crashes involving injury or death or damage sufficient to render the vehicle inoperable. Form AA-600 Commonwealth of Pennsylvania Driver s Accident Report will collect precise detail regarding the circumstances surrounding

2 Mr. Myers and Mr. Cohen January 12, 2017 Page 2 crashes involving vehicles with driving automation systems and conventional vehicles. This information will be valuable for understanding the differences between crashes involving highly automated vehicles and conventional vehicles. We encourage PennDOT to require companies to complete this form for every crash involving a vehicle with driving automation when operating in manual mode or automated mode, regardless of severity. We reviewed crashes reported by companies testing driving automation systems in California, and most of the crashes were not severe enough to be reported to police. For example, Google reported 10 crashes to the California DMV during , but we determined that only three were likely severe enough to be reported to police. None of the 10 crashes reported by Google during this period were singlevehicle crashes, and in most of the crashes, Google s test vehicle was rear-ended by another vehicle. This information suggests that vehicles with automated driving systems will continue to be involved in crashes, but the crashes will be different in nature from those involving conventional vehicles. It would have taken much longer to arrive at these insights if Google only reported crashes severe enough to report to police. This is particularly important in Pennsylvania, given the Commonwealth s stringent requirements for considering a crash to be reportable; we expect that even fewer crashes involving a vehicle with driving automation will be considered police-reportable under Pennsylvania s standards than we observed in California. We suggest that PennDOT require testing companies to report the number of times that automated driving systems are disengaged, whether the disengagement is initiated automatically or by a human operator. While disengagements of automated driving systems are not as safety-relevant as crashes, they can provide key information concerning the performance and limitations of the technology. Critically, information about disengagements, both routine (e.g., approaching operational boundary; weather testing) and safety-related (e.g., technology failure), will inform PennDOT and the public about how best to evaluate and test the technology and gauge its performance in the real world. It also would be valuable to know, generally, the operational design domain for the driving automation on each test vehicle represented in every semi-annual report. Some of the Autonomous Vehicle Testing Policy Task Force members were concerned that providing detailed information about a test vehicle s operational design domain would disclose proprietary information. We agree and think that a general description of the automated driving system s operational design domain would suffice. This description could include the roadway type(s), speed range, weather conditions, and other characteristics of the locations and conditions under which the technology operates. According to Policy Topic #5 Section I Part B, PennDOT will collect data that testing companies voluntarily disclose to the National Highway Traffic Safety Administration (NHTSA). Companies testing automated driving systems are asked to voluntarily submit a copy of the Safety Assessment Letter to NHTSA, so we expect PennDOT also will collect this information. We identified five areas of the Federal Automated Vehicles Policy that could be strengthened in our comment to NHTSA (attached), some of which may be relevant to PennDOT s Policy. Specifically, we felt that companies should be given more guidance about the contents of the Safety Assessment Letter to improve the quality of the information that is collected; additional information such as the vehicle identification number of vehicles equipped with automated

3 Mr. Myers and Mr. Cohen January 12, 2017 Page 3 driving systems that are sold to consumers should be collected to facilitate the evaluation of the technology; and companies deploying Level 2 driving automation systems also should be required to submit a Safety Assessment Letter because the distinction between Level 2 and Level 3 systems may not be apparent from a user or regulator s point of view and could be used as a means for avoiding regulatory requirements. Finally, we encourage PennDOT to make the information it collects from testing companies available to the public. Crash reports and semi-annual reports should be made available as soon as practical. Making these data publicly available makes the evaluation process more transparent. Access to precise, consistent, and reliable information about the safety and performance of automated driving systems that are operating on public roads is crucial for PennDOT and research organizations interested in evaluating the technology. The extent that PennDOT can ensure that information about crashes, disengagements, and other safety and performance measures are consistently and precisely reported by companies testing and deploying these technologies will further enhance its value. Sincerely, David Kidd, Ph.D. Senior Research Scientist Reference Dixit, V.V.; Chand, S.; and Nair, D.J Autonomous vehicles: disengagements, accidents, and reaction times. PLOS ONE, 11(12): e DOI: /journal.pone Attachments Insurance Institute for Highway Safety Comment to the California Department of Motor Vehicles concerning proposed regulations for the deployment of autonomous vehicles on public roads for general public use. Arlington, VA. Insurance Institute for Highway Safety Comment to the National Highway Traffic Safety Administration s request for comment on Federal Automated Vehicles Policy; Docket No. NHTSA Arlington, VA.

4 April 7, 2016 Brian G. Soublet Chief Counsel California Department of Motor Vehicles st Ave Sacramento, CA Dear Mr. Soublet: The California Department of Motor Vehicles (DMV) has requested comments on its draft regulations for the deployment of autonomous vehicles on public roads for general public use. The draft regulations detail the steps for obtaining a permit to deploy autonomous vehicles on public roads and specify requirements for monthly reporting of crashes during autonomous driving, safety-related disengagements of the autonomous vehicle technology, total use of the autonomous vehicle technology, and cyber security. The monthly reporting requirements provide an opportunity to collect the data needed to evaluate the safety and performance of autonomous vehicle technology in the real world. The Insurance Institute for Highway Safety (IIHS) would like to offer some suggestions for improving the monthly reporting of these safety and performance data. Crash reporting requirements IIHS understands that the California DMV intends to continue the same crash reporting practice that is currently required for companies with permission to test autonomous technology on public roads. These companies are required to report any crashes involving an autonomous vehicle operating in autonomous mode. Information about these incidents is reported using form OL 316 titled Report of a Traffic Accident Involving an Autonomous Vehicle. An advantage of collecting this information using a structured form is that it reduces the variability in reporting and increases the quality and uniformity of the data. Comparing the crash experience of autonomous vehicles with that of conventional vehicles is an important research question, and police-reported crash information will be a valuable data source for answering it. It is important to make sure the information collected from crash reports involving a vehicle in autonomous mode through form OL 316 and crash reports for incidents involving conventional vehicles through form SR-1 titled Traffic Accident Report are consistent. For the most part, forms OL 316 and SR-1 collect similar information in the same way, but IIHS has noted a handful of data fields present in form SR-1 that are missing from form OL 316. The information missing from form OL 316 is listed below. We encourage the California DMV to add these data fields to make form OL 316 consistent with form SR-1. Section 2 of OL 316 does not specifically require the Hour for the time of the crash. Sections 2, 3, and 4 of OL 316 do not include a section to indicate if the damages to the autonomous vehicle, other party, or property damage exceeded $750. Section 3 of OL 316 does not include an option to indicate if the other party s vehicle was parked. Section 3 of OL 316 does not include a section to indicate if the other party was driving for an employer. Section 4 of OL 316 does not include an option to indicate if a pedestrian was the other party that was injured or killed in the crash.

5 Mr. Brian Soublet April 7, 2016 Page 2 Safety-related disengagements The draft regulations also require autonomous vehicle companies to report safety-related disengagements of the autonomous technology where the operator manually takes over control from a vehicle operating in autonomous mode, the autonomous vehicle hands over control to the driver unexpectedly due to an external condition, or transfers control to the operator due to an internal failure or fault with the autonomous vehicle technology. The information that companies are required to report about each disengagement is not described in the draft deployment regulations document D.1, but IIHS anticipates that the information would be consistent with the current requirement for companies testing autonomous vehicles on public roads specified in Title 13 Article B.3.A. That is, the reports would include: Total number of disengagements of the autonomous technology and the circumstances or testing conditions at the time of the disengagements. Location (i.e., highway, rural road, street, parking facility). Description of the facts causing the disengagements including weather conditions, road surface conditions, construction, emergencies, accidents or collisions, and whether the disengagement was the result of a planned test of the autonomous technology. Total number of miles each vehicle has traveled in autonomous mode on public roads. Period of time elapsed from when the autonomous vehicle test driver was alerted of the technology failure and the driver assumed manual control of the vehicle. Every company with approval to test autonomous vehicles on California public roads is required to submit an annual disengagement report summarizing the information listed above. IIHS reviewed the annual disengagement reports submitted by Volkswagen, Mercedes-Benz, Google, Delphi, Tesla, Bosch, and Nissan for the period ending November 30, Based on our review, there is ample reason to believe that the information provided in these reports will be incomplete, inconsistent, and insufficient for drawing inferences about the real-world safety and performance of the technology. We noted three major issues with the way disengagements were documented. First, the required information listed above was not included in every report. Second, the type of information and detail varied across companies, making it difficult to combine and summarize in a meaningful way. Third, it was often difficult to determine which disengagements were safety-relevant and which were routine. IIHS urges the California DMV to precisely state the information that must be reported about every disengagement of autonomous vehicle technology, both routine and safety related, in the requirements for deploying autonomous vehicles for general public use. We encourage the California DMV to develop a structured form with predefined answer choices that companies must use when reporting this information. Some data fields to consider include the party that initiated the disengagement (e.g., automation, operator), the safety relevance of the disengagement (e.g., safety related, routine), and the suspected reason for it (e.g., end of autonomous operating range, hardware failure). Additional data fields could be consistent with those in forms OL 316 and SR-1 and information in California s Statewide Integrated Traffic Records Systems. For example, it would be useful to know the date, time, location type (e.g., highway, intersection), direction of travel (e.g., north, south), weather (e.g., clear, cloud, raining), road condition (e.g., no unusual condition, flooded), and lighting (e.g., daylight, dusk-dawn) when the disengagement occurred. Circumstantial information about the principal other road user involved (e.g., pedestrian, other motor vehicle), if present, the principal other road user s action preceding the disengagement (e.g., proceeding straight, making right turn, changing lanes), the autonomous vehicle s action preceding the disengagement, and the type of incident that was preempted

6 November 22, 2016 The Honorable Mark R. Rosekind Administrator National Highway Traffic Safety Administration 1200 New Jersey Avenue, SE Washington, DC Request for Comment on Federal Automated Vehicles Policy; Docket No. NHTSA Dear Administrator Rosekind: The National Highway Traffic Safety Administration (NHTSA) has requested comments on the Federal Automated Vehicles Policy, which provides guidance for the testing and deployment of automated driving systems on public roads. The Insurance Institute for Highway Safety (IIHS) congratulates NHTSA on developing a broad set of guidelines for evaluating, monitoring, and regulating the deployment of driving automation technology. IIHS is pleased the guidance attempts to balance public safety without hindering technology development, and we support the agency s efforts to dramatically reduce and eliminate crashes using driving automation technology. IIHS welcomes this opportunity to comment publicly on the Safety Assessment Letter, information gathering, and data-sharing practices described in the guidance. Our comments focus on five areas of the policy we believe can be strengthened: 1. NHTSA should give more guidance about the contents of the Safety Assessment Letter that the agency is requesting from developers of driving automation technology; 2. Vehicle performance guidance should be explicitly applied to Level 2 systems; 3. Guidance concerning the operational design domain should recommend that driving automation systems self-enforce their use within the intended operational design domain rather than relying on users to do so; 4. NHTSA should collect additional information to facilitate the evaluation of the safety and performance of driving automation systems by the agency and the public; and 5. NTHSA should encourage developers to address possible misuse and errors primarily through intuitive design. Additional guidance about the contents of the Safety Assessment Letter is needed The Safety Assessment Letter could be a tool to help NHTSA and the public assess the extent to which developers of driving automation systems have addressed safety. However, the policy only asks developers to attest to whether the system covered by the letter adheres to the guidance in the 15 areas outlined in the policy, along with a summary of how compliance was proven. IIHS is concerned that without further guidance, the information collected by the Safety Assessment Letter will be incomplete and insufficient for NHTSA and the public to evaluate the veracity of these claims. The policy emphasizes the conciseness of the Safety Assessment Letter, and IIHS is concerned this brevity may come at the expense of the quality of the information supplied in response to the letter. We strongly urge NHTSA to identify a set of data elements for each guidance area that is crucial for evaluating whether the system s developer has adequately adhered to the guidance outlined in the policy.

7 Mark R. Rosekind November 22, 2016 Page 2 For instance, the agency could specify that the summary showing adherence to the guidance about Data Recording and Sharing includes a list of data elements collected by the developer, under what conditions these data are collected, and what provisions have been made to share these data with NHTSA and the public. Likewise, the agency could specify that summaries showing adherence with guidance addressing the Human Machine Interface include a description of how the system communicates to users its operating status, operating mode, availability, and takeover requests. As expressed in our prior comment to the agency on this topic, the importance of collecting data from developers in a structured manner cannot be overstated. Making these data publicly available is equally important and makes the evaluation process more transparent. More detailed specifications about the contents of the Safety Assessment Letter will help NHTSA and the public better assess the legitimacy of claims made by driving automation system developers about complying with the guidance. Developers deploying Level 2 driving automation systems should submit a Safety Assessment Letter Currently, NHTSA is only requesting Safety Assessment Letters from developers deploying SAE Levels 3-5 automated driving systems. IIHS strongly urges NHTSA to request the same information from developers deploying Level 2 driving automation systems. Our concern is that the distinction between Level 2 and Level 3 systems may not be apparent from a user s point of view. Already, YouTube is populated by numerous videos exhibiting the misuse of Level 2 automation, and NHTSA along with the National Transportation Safety Board are investigating the first fatality in the United States, possibly involving overreliance on such a system. Without collecting Safety Assessment Letters for Level 2 systems, NHTSA and the public will not be able to evaluate whether developers are properly classifying the systems they put on U.S. roads. IIHS believes Level 2 systems should be equipped with safeguards that keep the human driver fully engaged in the driving task to adequately convey to users the difference between them and Level 3 systems. A Level 2 system that does not enforce human driver engagement may be perceived by some users as more capable than it is. Many current driver assistance systems that provide sustained lateral control alert the driver if the driver does not provide sufficient steering input, but steering input alone may be insufficient to reliably determine driver engagement. Lack of steering input could arise on a straight section of road or because the driver has removed their hands from the steering wheel. In the latter case, it may take several seconds for the driver to reengage in active control of the vehicle. Level 2 systems should constrain use to the intended operational design domain The published guidance recommends that developers ensure safe operation of their systems within designated design domains and that the conditions defining the domain are clearly communicated to users through the Human Machine Interface, but does not recommend the system self-enforce its use within the domain. Some current Level 2 systems provide sustained lateral and longitudinal control when the vehicle reaches a minimum speed and lane markings are detected. These systems seem to be intended for use on limited-access high-speed roadways, but the minimum operating conditions can be met in other circumstances where the technology may be unreliable or use would be dangerous, such as in inclement weather or on arterial roadways with signalized intersections. Existing on-board information such as location information from GPS, clock time, and windshield wiper status can be used to constrain the availability and use of Level 2 systems to roadways and conditions that meet the intended operational design domain. Levels 3 and 4 are similarly defined by their limited operational domains and should likewise include some means to prevent their activation outside the limits defining their design domain.

8 Mark R. Rosekind November 22, 2016 Page 3 Vehicle feature information must be shared to assess the real-world effectiveness of driving automation technology It seems likely that driving automation systems initially will be offered as optional features by automakers. This situation presents an opportunity to evaluate their effectiveness at improving the safety on our roads if the equipped vehicles can be distinguished from their counterparts without driving automation systems. IIHS and its affiliate, the Highway Loss Data Institute, have successfully evaluated the effects that various crash avoidance technologies have on crash involvement using insurance claims and police reports in combination with information from cooperating automakers linking specific vehicle identification numbers (VIN) with the presence of optional features. These systems are the foundation of driving automation systems, and our research shows that some systems provide a measurable safety benefit, some do not, and none are eliminating all of the crashes they potentially could address. It will likewise be important to evaluate the real effects of driving automation systems as they enter the marketplace. Access to precise information about the vehicles equipped with driving automation systems will help NHTSA, IIHS, and other research organizations evaluate the technologies and push developers to improve them. To this end, we strongly encourage NHTSA to collect and share build sheets with particular VINs for all vehicles equipped with Levels 1-5 driving automation systems and the driver assistance and crash avoidance technologies they build on, whether or not these vehicles are involved in crashes. Consumer education and training alone is not sufficient for ensuring consumers can use driving automation technology safely and as intended IIHS surveys of vehicle owners, observational studies, and internal evaluations of driver assistance and crash avoidance technology all have indicated ways that drivers can be confused or surprised by the way the technology behaves. We feel that consumer education and training alone will not be sufficient for ameliorating confusion and preventing mistakes when consumers use driving automation technology. In addition to describing a plan for educating and training consumers, IIHS urges NHTSA to request that developers provide evidence that the system is designed intuitively such that an uninformed consumer can use the technology safely and successfully as intended. One component of this will be demonstrating that the technology is resilient to human error either by preventing or reducing the opportunity for drivers to make a mistake and having successful error recovery strategies in place to mitigate the costs of errors that occur. Some sources of error may be easy to address and only require improving the design of system controls and inputs. For instance, the comments below reflect just some of the confusion and challenges IIHS employees encountered when using adaptive cruise control in three different vehicles over multiple days of normal driving: The lever was too close to the turn signal/[lane Keeping Assistance System] lever. I mistakenly hit the wrong levers repeatedly during the week. Not always clearly on, difficult finding and operating [the] control stick. the small buttons on the steering wheel were sometimes hard to locate quickly. Poorly designed system controls may not compromise safety, but other design issues may. For example, some IIHS employees stated the following when asked about lane keeping assistance technology: I didn t like that when I intentionally [gave] a tractor trailer extra room because it was close to the line, [that] the system would try to get me to move back toward the truck. The system often got confused at places where lanes quickly drop or are added, such as toll lanes and some highway exits.

9 Mark R. Rosekind November 22, 2016 Page 4 Each driver in the IIHS program received step-by-step training on how to use each vehicle s adaptive cruise control system and other driver assistance and crash avoidance technologies in stationary and onroad demonstrations. This training likely exceeded the amount that most consumers receive when purchasing vehicles with these features. Nevertheless, the subset of comments above illustrates that training was not entirely successful and cannot replace or replicate continual learning from extended use of a vehicle technology. It is critically important to ensure that developers of driving automation technology implement strategies to encourage appropriate use, prevent errors, and prevent misuse that extend beyond consumer education and training. Summary In conclusion, IIHS is pleased that NHTSA is providing guidance to developers and policymakers concerning the safe deployment of driving automation systems. Although the potential safety benefits of the technology are tremendous, we must proceed with care to balance technology development with public safety. The Safety Assessment Letter is a reasonable first step for collecting the information needed to evaluate the safety and performance of driving automation technology, but, as proposed, it will provide insufficient detail to achieve its stated objective. The guidance is directed at Highly Automated Vehicles, but there is much that can be learned from lower levels of driving automation technology present in current vehicles. Collecting information from developers deploying Level 2 systems will help to sharpen the tools and processes used to collect information to monitor, evaluate, and improve the technology. Similarly, collecting and sharing VIN information for vehicles equipped with driving automation systems and the underpinning technologies will support safety and performance assessment. Sincerely, David G. Kidd, Ph.D. Senior Research Scientist References Cicchino, J.B Effectiveness of forward collision warning and autonomous emergency braking systems in reducing front-to-rear crash rates. Arlington, VA: Insurance Institute for Highway Safety.