Waymo pitted its autonomous vehicles against a virtual superhuman driver to see which was safer

Waymo’s latest effort to prove that autonomous vehicles are safer than silly, accident-prone humans involves creating a virtual representation of a hyper-attentive driver and then pitting this fake person against its own AVs in a series of simulated tests to see which is better at crash avoidance.

(Not to spoil it or anything, but the Waymo vehicle did better.)

Improved safety has been one of the main predictions of the autonomous vehicle (AV) industry. With millions of people dying in auto crashes globally every year, AV operators are increasingly leaning on this safety case to spur regulators to pass legislation allowing more fully autonomous vehicles on the road. But while the argument seems convincing on the surface — AVs don’t get drunk or distracted like humans, nor do they speed or break the law — there is scant data that proves that fully automated vehicles are safer than human drivers.

To provide more statistical support for its argument, Waymo produced two new scientific papers comparing autonomous vehicle performance to human driving. The first analyzes and models response times when a crash is imminent, while the other “presents a novel methodology to evaluate how well autonomous driving systems avoid crashes.”

The novel methodology involves creating a model of “the response time and evasive action of a human driver that is non-impaired, with eyes always on the conflict (NIEON).” In other words, unlike a normal human driver who experiences fatigue and distraction, this virtual superhuman driver is always attentive, never gets tired or distracted, and is always ready to react. Waymo is calling the type of analysis the first of its kind.

Waymo then simulated a number of different imminent-crash scenarios to compare the NIEON driver to its own autonomous vehicles. Predictably, the superhuman driver was very good at preventing collisions, avoiding 62.5 percent of the simulated crashes and reducing the risk of serious injury in 84 percent of the situations.

But despite that admirable performance, the Waymo vehicle did better, avoiding 75 percent of collisions and reducing serious injury risk by a whopping 93 percent.

“We consistently outperform this high bar of human performance,” Trent Victor, Waymo’s director of safety research, told The Verge.

Predicting how human drivers respond when a crash is about to happen has been a challenge for generations of traffic safety researchers. Response time is usually measured in controlled experiments, where subjects are instructed to respond to stimuli such as a sound or a brake light. 

The key question in response time studies is when to “start the clock”: from which point do you calculate the reaction time? This becomes even more crucial in a naturalistic setting where pedestrians and cyclists do not always behave in a predictable way. Many traditional methods for determining reaction time do not account for a sense of urgency. They tend to overestimate drivers’ reaction time by assuming they respond slower in fast-moving situations or faster in slow-moving situations.

But Waymo approached the problem differently. In its study, it decided to start the clock at the moment when the driver is surprised or when their expectations are broken.

“In urgent events, drivers react as fast as possible,” Victor said. “And they do that because they’re very surprised.”

Using data from more naturalistic driving studies, Waymo said it created an internal benchmark for collision avoidance that exceeds the typical driver. This gave it a new way to evaluate the performance of its automated driving system, which it calls the Waymo Driver.

Previously, the company had sought to measure the safety of its AVs by simulating dozens of real-world fatal crashes that took place in Arizona over nearly a decade. The Google spinoff discovered that replacing either vehicle in a two-car collision with its robot-guided vehicles would nearly eliminate all deaths.

But Waymo decided to take things a step further. Since it didn’t know how many of those real-world crashes involved drunk, impaired, or overly fatigued driving, the company decided to create a model of a superhuman driver who never got tired or drunk and was always attentive with their eyes on the road. And using its new benchmark in response time measurement, Waymo re-simulated the crashes in its previous study to see how this driver would respond.

The superhuman driver performed really well, but the Waymo Driver did better — sometimes by completely avoiding collisions altogether by just being better than even the best human driver.

There’s no standard approach for evaluating AV safety. A recent study by RAND concluded that, in the absence of a framework, customers are most likely to trust the government — even though US regulators appear content to let the private sector dictate what’s safe. In this vacuum, Waymo hopes that by publicizing this data, policymakers, researchers, and even other companies may begin to take on the task of developing a universal framework.

Victor said both studies have been peer-reviewed and will be submitted to a journal for publication. “We’re following the way the scientific process works,” he said. “So then it’s up to others to comment and do research based off of that.”