Security Metrics and Tesla’s Safety Statistics

by C. Warren Axelrod on April 23, 2019

I have long railed
against the inadequacy of popular easy-to-record security metrics. They usually
lack critical information about the nature and severity of vulnerabilities and
are therefore misleading in providing support for decision-making. I addressed
this point in my article “Accounting for Value and Uncertainty in Security
Metrics,” in the ISACA IS Control Journal (November 2008).

Tesla has just published its fourth quarter 2019
safety report , available at https://www.tesla.com/VehicleSafetyReport
and showed that the number of crashes per mile driven is significantly lower
than for other vehicles for Tesla cars without Autopilot engaged and lower
still when Autopilot is engaged.

Further, Tesla’s Model 3 was tested by the NHTSA and
found to have the lowest probability of injury of any vehicle ever tested. https://www.tesla.com/blog/model-3-lowest-probability-injury-any-vehicle-ever-tested-nhtsa

These are clearly great achievements by any standard
and Tesla should be praised for them. However, there are some critical aspects
omitted from these reports, recognizing that the additional information may be difficult
and/or expensive to come by.

First is the relative severity of accidents that occur
with regular everyday cars and Teslas, with Autopilot on or off. One might
assume that if Teslas have fewer crashes and the probability of injury is
lower, then the conditional probability of injury given a crash has taken place
is less. Obvious? Well, no! If the accidents with Teslas were to be much more
severe than average accidents and if injuries are greater, despite their lower
probability, then we have an entirely different picture. Of course, the press
broadcasts Tesla accidents that result in fatalities with inappropriate fanfare,
but that does not imply that there are more of them.

Sponsorships Available

The statistics need to be normalized for other
differences too. The average Tesla is really new relative to the average age of
all vehicles. Does this affect accidents? Probably. To what extent are failing
components responsible for crashes? We should know this.

The average Tesla may not be driven under poor driving
conditions. Does Autopilot work in heavy snow, for example? If not, you are
missing road conditions in which many crashes occur.

Teslas have limited range. Does the length of journey
have anything to do with rate of accidents? It is arguable that when drivers
exceed a certain number of hours driving, they are more prone to accidents.

Do Teslas cause accidents that are not recorded or
assigned to Tesla? I recall reading a report of an autonomous vehicle forcing
another vehicle off the road. Do we know how common such situations might be? I
don’t think so.

None of this is an attempt to diminish Teslas laudable
achievements in safety and accident reduction. It’s just that the published
statistics are not the whole story.

The same is true of security metrics. We often lack
severity and context n these statistics. We also don’t know the indirect
effects. If we are able to deter or avoid a cyberattack, then the hackers
likely move on to more vulnerable targets. That’s good for the first
organization, but not for the second. Are we interested in the impact of
cyberattacks across the whole of cyberspace? Or are we just interested in our
own wellbeing?

Yes,
superficial statistics are easy together. But are they truly representative of
the real world? Probably not. Whether that is Tesla’s world or that of
cyberspace. Yes, we draw comfort from supportive statistics, but the frequency
with which such statistics are challenged or debunked (especially in the
medical field) should give us pause. Let’s petition for more useful statistics
and metrics. If we don’t, we will continue to make decisions on statistics that
are not particularly meaningful. I have been convinced that Teslas are safer
than other vehicles on the road, but I cannot be certain, and I don’t know if
there are other aspects that reduce the value of the statistics that are
published. There is a lot of hype about autonomous and electric vehicles, but
there are other things to consider that might detract from their long-term
value. Talking about autonomous cars, you should read Todd Litman’s report, Autonomous
Vehicle Implementation Predictions –Implications for Transport Planning,
Victoria Transport Policy Institute, March 18, 2019. It is available at https://www.vtpi.org/avip.pdf Litman presents some very good ideas about
some of the costs and disadvantages of driverless road vehicles that are
omitted elsewhere. We need to consider similar factors for cybersecurity.