Each year, California’s Department of Motor Vehicles (DMV) requires the companies testing self-driving vehicles in the state of California to write a Disengagement Report. The report provides insight into the number of miles each company has driven autonomously on California roads and the total number of disengagements enacted in a year. Disengagements occur when test vehicles switch from autonomous to manual mode either automatically when the system experiences a failure or manually by a vehicle operator to ensure safe operation.
This data has historically been viewed as a “report card” for self-driving companies. But as we have repeatedly said, using these statistics as a measuring stick is misguided.
The self-driving teams I’ve led over the past 17 years have championed various attitudes toward on-road testing.
While huge numbers of on-road miles may initially seem impressive, experience has taught me which approaches merely look like progress and which can actually move the needle toward real progress.
Do more on-road miles mean that vehicles are performing harder maneuvers or navigating more complex environments? Are miles and “disengagement rates” on an empty highway even comparable to those in dense urban areas? The answer to both of these questions is, unequivocally: No.
At Aurora, our ultimate goal is for the Aurora Driver to safely move people and goods without the need of a human driver. We’re incredibly pleased with the rapid and meaningful advancements we made toward that goal in 2019, which are enabled due to investments in our Virtual Testing Suite. Thanks to extensive virtual testing, we’re able to make more progress more quickly today than we did a year ago, despite driving fewer miles on California roads.
Why? As we’ve said before, a single virtual mile can be orders of magnitude more useful than one collected on the open road.
Here are the highlights from our 2019 California Disengagement Report.
- Our Approach: Instead of prioritizing the collection of on-road miles in 2019, our Virtual Testing Suite has grown to over 735,000 tests per day, an increase of over 100 times from 2018. We find that on-road autonomous testing is a slow, expensive and inefficient approach to development as compared to more sophisticated, virtual techniques. We leverage on-road miles for targeted data collection in increasingly complex environments. And, in a typical mile of driving in 2019, our vehicles encountered approximately three times more vehicles and nearly ten times more pedestrians than they did in 2018.
Since Day One, we’ve invested heavily in our Virtual Testing Suite to make progress. While we optimize the data we collect from our on-road miles and disengagements, Virtual Testing has enabled us to make rapid progress at scale.
- Our Progress: Thanks to our Virtual Testing Suite, we saw the Aurora Driver perform dramatically more complex maneuvers, like unprotected left-hand turns, in more dense urban areas, like downtown Palo Alto. We also integrated the Aurora Driver with three new vehicle platforms, performed two major hardware upgrades that significantly increased both computational power and sensor capability, and made countless improvements to infrastructure across the company.
- Miles and Disengagements: We drove 39,729 miles on California public roads, 26,300 miles of which were manually driven. These numbers show that prioritizing virtual testing enables faster development than on-road driving. Our rate of disengagements per 1,000 autonomous miles is similar this year (10.6) to what it was in last year’s report (11.5). This is by design and is consistent with the philosophy we outlined in our 2018 report.
Overall, it’s been a great year for the development of the Aurora Driver. For the first time, we were able to demonstrate our initial thesis for development: an expansive suite of carefully crafted virtual tools allows quicker development than on-road driving. Our engineers make more progress more quickly today than they did a year ago, despite driving fewer miles on California roads. While we recognize that on-road testing miles are an essential tool for the ultimate safety and deployment of self-driving vehicles, we are excited about the virtual testing infrastructure we’ve created, the driving capabilities we’ve built, and the trajectory we’re on.
