Much has been said in the security world about the recent release of data on vulnerabilities discovered in 2015. Due to the way this data has been presented, many news outlets have been reporting that Mac OS X was the “most vulnerable” OS in 2015. But was it really?
It turns out, there are some issues with the way the data is presented and the conclusions that are being drawn from that presentation. The table shows Mac OS X at the top, with 384 vulnerabilities, and iOS in a close second with 374. Well below that are different versions of Windows, with the worst being Windows Server 2012, in 10th place with 155 vulnerabilities, a mere 40 percent of the vulnerabilities of OS X.
Therein lies the first issue with this data, however. All versions of OS X are clumped together in one group, while different versions of Windows are broken down and shown separately. This means that comparing the numbers in that table does not amount to comparing apples to apples, so to speak.
If we look into the data a little more closely, it is possible to group vulnerabilities by vendor. Doing that, we see that Apple had a total of 654 vulnerabilities in 2015, while Microsoft had 571. Throwing Adobe into the mix, it achieves third place at 460 vulnerabilities.
This is a slightly more useful number, as it includes all versions of each company’s systems as well as their web browsers (Safari and Internet Explorer). However, this includes the vulnerabilities for iOS, while it’s unclear to me whether any Windows mobile systems are included in the Windows vulnerability counts.
Still, even those numbers are not particularly meaningful as-is. Comparing them is like comparing the number of hits made by two different baseball teams in a season without considering how many of those hits were foul balls, grounders, home runs, etc. One team may have a higher number of hits than another, but if they hit a lot more foul balls, while the team with fewer hits had a higher proportion of home runs and RBIs (runs batted in), the higher number of hits is revealed as a misleading statistic.
Similarly, vulnerabilities come in all kinds. There are relatively minor vulnerabilities that can’t really do much harm, and then there are the ones that allow a remote attacker to fully compromise your machine. Fortunately, each vulnerability in the data set is given a severity rating between 0.0 and 10.0. With some filtering, it’s possible to see all the vulnerabilities for each platform that were very severe (higher than 9).
Doing so turns up some interesting results. Apple, it turns out, had 91 vulnerabilities of this severity in 2015, a mere 14 percent of their total vulnerabilities. Microsoft, however, had 332 very severe vulnerabilities, at 58 percent of their total. Adobe leads both with a whopping 389 very severe vulnerabilities, almost 85 percent of their total.
This tells us more clearly about the severity of the vulnerabilities in the data, which is a more important metric than just how many total vulnerabilities there are.
Now, don’t get me wrong… I’m not trying to say that Apple’s systems are the most secure systems on the planet. Although the number of highly dangerous vulnerabilities is interesting, there’s still a lot left out.
For example, how many of the vulnerabilities were actually exploited by malware in the wild? There’s no way to know, barring another source of information or a lot of research. It doesn’t matter that Apple’s counts of dangerous vulnerabilities are lower if they got exploited more frequently than the competition.
There’s also no information about existing vulnerabilities. As an example, there are no new vulnerabilities listed for Windows XP. However, there are existing, unfixed vulnerabilities in Windows XP, and many people still use that, despite the fact that Microsoft has discontinued support for it. These won’t show up, but it would still be fair to count them if we’re trying to put some kind of number on how vulnerable Windows users as a whole are.
The same is probably true of Mac OS X 10.6 (aka Snow Leopard), which is no longer supported by Apple but has known vulnerabilities. That should also count against Apple, and although I don’t know for sure (due to the way the Mac OS X data was categorized), I’m guessing there are probably no Snow Leopard vulnerabilities in the 2015 data.
Plus, there could very well be minor vulnerabilities in the systems that were listed that were found in 2014 but haven’t been fixed yet. Such things do happen, and that would be particularly interesting to examine, since it would give insight into the delay between discovery of a vulnerability and patching it for each vendor.
So, bottom line, I’d advise you to ignore any click-bait headlines about how Mac OS X is the “most vulnerable system” based on this data. As I’ve shown, if you look at the data in a different way, you could come to a very different – but still probably inaccurate – conclusion. Ultimately, based solely on vulnerability counts, there’s no valid way to say whether Apple or Microsoft win the “most vulnerable” award, and any attempt to do so is extremely misleading.
Comments
Post a Comment