Incident response standards, data sharing, and not blaming humans unfairly for the failures of machines

More than a week after an Uber vehicle driving in autonomous mode killed a pedestrian in Tempe, Arizona — the first pedestrian death by a self-driving car — we still don’t know what exactly went wrong. Video of the crash shows that the pedestrian, Elaine Herzberg, walked in front of a moving vehicle. But the vehicle didn’t appear to react, and there are many unanswered questions as to why it did not. Did the car’s Velodyne Light Detection and Ranging (LIDAR) or other sensors get enough signal to detect her? Did Uber’s decision to scale down to a single LIDAR sensor from the seven LIDAR sensors on earlier vehicle models, which created more LIDAR blindspots, play a role? Where the vehicle’s LIDAR sensors disabled? Did the fact that she was a pedestrian walking a bicycle confuse any of the car’s vision systems? Did the vehicle in fact slow down?

Regardless of the details, the most important question we should all be asking is: What can Uber and its competitors do to learn collectively from this incident and (hopefully) avoid similar incidents in the future? 

One thing all self-driving car companies could and should do is develop incident-response protocols, and those protocols should include sharing data about collisions and other safety incidents. That data needs to be shared between autonomous car makers, government regulators, academic research labs, and ideally the public,^[1] so they can analyze what went wrong, learn from each other’s mistakes, and all get safer faster. This seems fairly obvious, but self-driving car companies are racing to develop the first fully autonomous, “Level 5” vehicle. Acting in isolation, they have few if any incentives to share data. But if sharing is the rule, their vehicles will be collectively safer, and the public will be much better off. 

While autonomous vehicles are hailed for their promise of reducing vehicle fatalities, the Uber accident has raised questions about whether and when autonomous vehicles will really be safer than human drivers. If accidents continue at this initial rate, some of the early self-driving car fleets might be much more dangerous than regular vehicles. That isn’t a reason to stop. We are very early in the technology’s development; early airplanes were disastrously dangerous, and dramatic safety gains have continued to the present day. But it is a reason to ask, how can we ensure that safety improvements happen as fast as possible?

This is especially true given that, unlike the pilots who died flying early airplanes, pedestrians injured or killed by autonomous vehicles are not the ones who decided to get into them and that it was worth the risk. It’s the companies who are deciding what risk it will impose on the rest of us. We have the right to understand that risk, what companies are doing to mitigate it, and whether they’ve put us at any unnecessary risk, which it appears Uber may have done here. We also have a right to demand that they take reasonable steps to help make the technology safer for everyone, such as sharing incident sensor data.

After last week’s incident, Uber immediately halted testing of autonomous vehicles in cities across North America and has reached an undisclosed settlement with the victim’s family. It is also currently cooperating with Tempe officials, the National Highway Traffic Safety Administration (NHTSA), and the National Transportation Safety Board (NTSB) on their investigations into the incident.

Regulators have, up until now, largely adopted a light-touch approach to regulating autonomous cars. Arizona, for example, has virtually no rules dictating where and when testing can occur, and imposes no reporting or disclosure requirements, even about crashes, though following the accident it has banned Uber from testing self-driving cars in the state. California has granted 50 manufacturers permits to test autonomous cars within the state, so long as there is a safety driver behind the wheel; next month, manufacturers will be able to apply to test and deploy cars without a safety driver. NHTSA, which we criticized last year for trying to push through an ill-thought-out proposal to force connected cars to talk to each other,^[2] prefers “voluntary guidance” over mandatory standards for autonomous driving systems. Waymo, Uber, and other self-driving car companies, just days before the recent accident, urged Congress to pass legislation that would facilitate the deployment of self-driving cars throughout the United States.

“Whenever you release a new technology there’s a whole bunch of unanticipated situations,” Arun Sundararajan, a professor at New York University’s business school, told Bloomberg. “Despite the fact that humans are also prone to error, we have as a society many decades of understanding of those errors." When it comes to machines and algorithms, many people expect them to always be right. But they won’t always be right — especially as new technologies are being developed. And because of this misperception, how companies respond when things do go wrong is going to play an increasingly important role in the development the autonomous and intelligent systems they are trying to build.

Rare events will always have the potential to cause unexpected behavior in robotic systems. Sharing data about them when they happen is necessary for making autonomous vehicles safer.

The autonomous car industry has not always done a great job with this. Tesla, for instance, responded to two incidents involving vehicles traveling in “autopilot” mode in January by simply reiterating their policy — the driver is supposed to remain fully attentive and keep their their hands on the wheel at all times — rather than by trying to address the underlying consumer confusion generated by the technology’s misleading name. And after the company’s first autopilot death in June 2016, it “repeatedly went out of its way to shift blame for the accident” in its 537-word statement, even while acknowledging that the car’s sensors had failed to distinguish between a large white truck and the bright sky in the background. It also referred to the driver Joshua Brown’s death as a “statistical inevitability” on its blog. One crisis management consultant has called Tesla’s response a “perfect case study in the wrong way to handle this sort of crisis.”

Even after last week’s tragic Uber accident, the instinct of many (though not Uber’s, as far as we know) was to blame the humans. Many initial reports assumed that the pedestrian jumped off the median in front a car, a theory which the incident video disproved. Later, questions were raised over whether the safety driver was paying adequate attention. We are somewhat concerned by that reaction. Decades of research show that humans are notoriously bad at doing exactly what the safety drivers are supposed to be doing: paying constant attention when they are not actively engaged in the activity. We aren’t even all that good at paying complete attention while we are actively driving. We must avoid relying on humans as liability sponges, or “moral crumple zones” that “bear the brunt of the moral and legal penalties when the overall system fails.”

Instead of pointing fingers, we need to focus on making the technology safer, and quickly. And the very first step in doing so is to ensure that when a terrible accident like this occurs, the company involved in the accident shares all of the underlying sensor data with other autonomous car makers so that no autonomous vehicle has to repeat the same mistake.

***

[1] The exact scope of data that should be shared, and who it should be shared with, involves some privacy tradeoffs. At minimum, companies should share the sensor data immediately preceding accidents or circumstances that could contribute to accidents (such as when a human safety driver needs to take control, or when a computer vision system fails to detect an obstacle that was found by LIDAR). It could also potentially include computer vision architectures and neural network models, as well as sensor data. Even when vehicles have different types of sensors, there will often be opportunities for cross-training or cross-testing.

When data is hard to sufficiently anonymize, this may require extra protections, such as contractual restrictions against de-anonymizing humans present in the data. If there were reliable ways to anonymize large amounts of vehicle sensor data, it could be desirable to share all of the data from the self-driving vehicle fleets, to enable its inclusion in training datasets, but we are not presently optimistic that such anonymization methods are available.

[2] The agency thankfully backed away from its plan, but out of concern over placing too much of a burden on automanufactuers rather than security or privacy.