The NHTSA’s first ADAS crash report and how it looks for Tesla

Better than most reporting suggests, though many questions remain unanswered

Updated 6/16 with additional notes based on the raw data NHTSA released.

Today the NHTSA released a preliminary report on accidents involving Advanced Driver Assistance Systems in the United States. This report reveals data provided to NHTSA by auto manufacturers in response to last year’s General Order mandating that they begin reporting data about accidents involving such “ADAS” systems.

The scope of this data is limited to SAE “Level 2” automation, which is defined as any partially automated driving mode where the system controls both the vehicle speed and steering, but where a human driver is always ultimately responsible for the vehicle’s behavior and may have to take over or override the system at any time.

SAE provides this helpful summary graphic to describe the various levels of automation they have defined:

Note that this report excludes Level 1 automation, which includes more common Adaptive Cruise Control and Lane Keep Assist features when used independently. It also does not cover any ADAS-powered safety features such as Forward Collision Warning, Automatic Emergency Braking, Lane Departure Warning, etc.

What NHTSA says about the data

An NHTSA spokesperson warned that the data released today should not be used to draw comparisons between ADAS systems nor to make assessments of their safety. Of course, this hasn’t stopped many sensationalist news outlets and Twitter personalities from doing exactly that, often completely misconstruing what the data does tell us.

NHTSA Administrator Steven Cliff said “I would advise caution before attempting to draw conclusions based only on the data that we’re releasing. In fact, the data alone may raise more questions than they answer.”

In the report, NHTSA highlights some specific issues and limitations with the data, including:

“Access to Crash Data May Affect Crash Reporting”
The report explains that crash data recording and telemetry capabilities vary widely among car manufacturers. Tesla, for example, has over-the-air telemetry from every vehicle they sell. They use this to report to NHTSA every single accident where their Autopilot system was activated within the 30 seconds leading up to an accident.

For the most part, NHTSA says, other manufacturers only know about these accidents if they are contacted by the vehicle owner, or if they are able to discover it through analysis of police reports and insurance claims. Even when a vehicle is involved in a crash which results in a police report and/or insurance claim, NHTSA warns that the manufacturer may never be made aware of the accidents, and when they do - there can be a substantial time delay.

In fact, NHTSA provides some raw data about the sources used. Of the 258 accidents with reports from telematics data, 251 were from Tesla. Telematics data was included in reports on three accidents from Subaru, two from GM, one from Honda, and one from Lucid.

By far the largest source relied upon by other manufacturers were customer complaints and insurance claims, suggesting they are likely woefully undercounted.

“Incident Report Data May Be Incomplete or Unverified”
The report warns that manufacturers were required to report incidents even if they could not verify the accuracy of the report, and regardless of whether they agree with the information in the report. They call out that a manufacturer like Tesla, with access to real-time vehicle telemetry, is required under the order to report all incidents where an airbag was deployed. However, they may not immediately (or ever) be aware of all circumstances related to the crash (surface conditions, whether an injury occurred, etc).

“The same crash may have multiple reports”
NHTSA warns that their methodology requires manufacturers to submit multiple reports for a single crash. In addition, multiple entities may be required to report incidents, and may report the same one twice. They say, “Consequently, the overall number of reports submitted does not equate to the total number of incidents and is not a meaningful safety metric”.

“Summary Incident Report Data Are Not Normalized”
Finally, NHTSA calls out that the data reported here is not normalized across manufacturers, and that data required to contextualize these figures is limited. They say: “For example, a reporting entity could report an absolute number of crashes that is higher than another reporting entity but operate a higher number of vehicles for many more miles.”

What are the numbers they released?

As of May 15, 2022, 12 companies have reported a total of 392 incident reports for vehicles involving the use of Level 2 ADAS functionality. 367 of these reported crashes occurred between July 2021 and May 15, 2022. The other 25 either occurred before that or were lacking a valid incident date.

The numbers by reporting entity (usually the auto manufacturer) are shown in this chart:

An initial read of this data may make it appear problematic or concerning for Tesla. However, keep in mind all of the limitations described above. For one thing, NHTSA says that in total, only 258 crashes were reported via telematics, and 251 of those were from Tesla. Tesla was essentially the only one to report data this way, and thus it is likely to be much more complete and timely.

Further, recall that this data is not normalized across usage. Unfortunately, the report does not include data about the number of vehicles equipped with these systems, nor the number of miles traveled with these systems engaged. These would be the bare minimum needed to start making any sort of comparisons.

Most ADAS crashes are not severe

It’s worth noting that NHTSA provides some (limited) data about the severity of the reported crashes.

The large number of “unknown” values is unfortunate, but for those where this information was provided, approximately 11% resulted in severe injuries or fatalities.

What can we conclude from this data?

Unfortunately, not much. However, despite the limitations that NHTSA called out, we do have some data points we can look at here to contextualize these numbers ourselves.

Some manufacturers have reported data about usage of their ADAS systems. We don’t have perfect data for the exact locations (US) and time frames that NHTSA looked at, but we can make some estimates based on the data that is available.

For example, Ford announced in May that owners have used their Blue Cruise ADAS system for approximately 4.5 million miles since it was introduced last year. Unfortunately, it is unclear if this encompasses all driving which would count as ADAS operation for the purposes of this report, but it’s a starting point. If it is accurate, this figure would put Ford at a ballpark of 900,000 miles of ADAS driving between accident reports. However, this is likely undercounting the denominator (then again, the numerator is also likely undercounted for the reasons described above).

Tesla, however, announced in late November 2018 that their customers had already driven over 1 billion miles using their ADAS system worldwide - with the vast majority being in the US. Further, they stated that this accounted for 10% of all miles driven by Tesla owners. They then announced in April of 2020 that the number had reached over 3 billion (an increase of 2 billion in 15 months). Again, that was a worldwide number, but the vast majority of which we know was in the US. We can surmise that this rate has only increased as the number of Tesla vehicles on US roads has more than doubled since that 3 billion number was reported.

If we take a ballpark, conservative estimate that 1 billion miles were driven on Autopilot in the US during the 10.5 month period that NHTSA looked at, then this would come out to a rate of 3.66 million miles between accidents. This is of course a very rough estimate, but it comes out reasonably close to Tesla’s own Vehicle Safety Report number, which for Q4 2021 was 4.31 million miles between accidents.

Note that Tesla’s number includes accidents where Autopilot was in use at the time of the crash or within the 5 seconds prior. The NHTSA reports (which do not assign blame to the system) include any accidents where the ADAS was enabled within the 30 seconds prior to the accident. The NHTSA number likely includes a somewhat larger number of cases where ADAS usage was not relevant to the accident.

It is very possible, perhaps even likely, that more than one billion miles were driven on AP in the US during this timeframe - meaning the 3.66 million number I estimated above is too low. While it’s possible that the one billion mile estimate is too high, I very strongly doubt it. If anyone at Tesla reads this - please consider publishing the actual number!

These numbers appear to be on par with or better than what is seen for non-ADAS miles in similar circumstances. Certainly, with available data, we cannot conclude that this number is significantly different from what we should expect were Autopilot not available. Ideally, we will get more data in the future that can enable a more meaningful assessment, as well as comparisons across systems.

On that point, I would like to address the tendency of some to compare Tesla’s numbers to the NHTSA general number of 479,000 miles per accident. For several reasons I described in detail last year, that is not a particularly valuable comparison. Simply controlling for vehicle age brings that number up to roughly 1 million miles per accident, and if you focus on similar premium vehicles and demographics of buyers, that number goes up further - I estimate to roughly 2 million. If you then control for road type where Autopilot and similar ADAS systems are almost exclusively used (limited access highways), this goes even higher - quite likely in the 3-4 million miles-per-accident ballpark.

If I were to follow the example of many outlets covering Tesla, I could say that this suggests that drivers using Ford’s ADAS are 4-5 times more likely to be involved in accidents than drivers using Tesla’s. However, given all the caveats noted above, this would not be a fair comparison given the lack of quality, normalized data available at this time. If Ford or NHTSA could release data about the mileage that Ford’s ADAS systems saw during this specific timeframe, that could help us come up with a more comparable estimate.

Ideally, we (or NHTSA) would be able to produce a table like this (with firmer figures, and all manufacturers represented):

Unfortunately, Tesla is likely the only manufacturer in a position to even have this kind of usage data for their fleet - for the same reason they’re pretty much the only ones providing crash data via telematics. A better comparison might be viable between specific models, though, such as between the Tesla Model Y and the Ford Mach-E (which has a cellular modem and OTA update capabilities - so presumably has telematics as well, or could be updated to support them).

Key takeaways

So what did we learn? Unfortunately, not very much. The data seems to roughly corroborate Tesla’s reported numbers in terms of accident rates on Autopilot, and suggests there is no obvious correlation between Autopilot use and accidents. If there is a positive or negative safety impact, it is far too small for us to assess with the limited data available today. I would love for manufacturers, including Tesla, to release more data and better data. Perhaps the NHTSA will be able to use the reports they’re now receiving to produce greater insights in the future.