Methodology, Sources and Assumptions for Monthly Vulnerability Scorecards

Tue, 2007-02-27 20:20

This page outlines the methodology, sources and assumptions made for the Monthly Vulnerability Scorecards and also defines which components were excluded from analysis in the case of Linux distributions in order to give them the benefit of the doubt for their modularity and make the stats more comparable.

I did the vulnerability compilation and analysis myself, but the sources for the vulnerabilities are vendor security advisories on the vendor web sites:

For severity information, I used the US Department of Homeland Security sponsored National Vulnerability Database (NVD, http://nvd.nist.gov) as a source for independent severity ratings that were defined across all of the products.

The vulnerabilities included in the analysis only include those vulnerabilities for which the vendor has confirmed applicability, typically via a security advisory or patch notice. The analysis here does not include publicly disclosed vulnerabilities during the period that have not yet been fixed by the vendor. I think this information (publicly disclosed, but unpatched) is also useful for decision makers, but is complicated enough to deserve a separate analysis in order to be accurate. Rest assured that all products had publicly disclosed issues during the periods studied that had not yet be addressed.

For each of Server and Workstation, I built two charts for each of two time periods:

Previus Three Months. These values will represent totals for a recent time period that will smooth comparisons between vendors where a vendor might have had one bad month. Additionally, it accommodates potential vendor policies that for releasing fixes on schedule longer than month (e.g. the Oracle quarterly schedule).
Year to Date. These values will represent totals for the calendar year to date starting from January through the current month.

For Workstation products, the scorecard charts will include these products:

Windows Vista. Stats for Windows Vista will include vulnerabilities that affect any product shipping as part of Windows Vista. It includes Internet Explorer and Windows Defender, for example.
Windows XP SP2. Stats for Windows XP SP2 will include vulnerabilities that affect any product shipping as part of Windows XP SP2. It includes Internet Explorer, but would not include Windows Defender, for example, as that component is optional and did not ship with Windows XP.
Red Hat Enterprise Linux 4 Workstation (rhel4ws). Stats for rhel4ws will include the default installed components, excluding thunderbird, text-internet, graphics(which is the gimp stuff), and office (which is OpenOffice) installation groups. These exclusions are to remove applications packages for which Windows XP does not have comparable packages. For more detail on this decision process, read Red Hat and Windows - Defining an Apples-to-Apples Workstation Build.
[UPDATE 03/29/2007] Ubuntu 6.60 LTS. Stats for Ubuntu will include the default client installation software (not the server CD), excluding packages that do not have equivalents on Windows, such as bittorrent, evolution, gimp, openoffice and thunderbird.
[UPDATE 03/29/2007] Novell SLED10. Stats for SLED 10 will include the default set of packages, excluding packages that do not have equivalents on Windows, such as gimp, ImageMagick, mono and openoffice.
Mac OS X. Stats for Mac OS X will include vulnerabilities that affect shipping components of Mac OS X. This includes Quicktime, for example, in a similar way that Windows Media Player (WMP) would be counted against Windows XP

For the Server products, the scorecard charts will include these products:

Stats for Windows Server will include vulnerabilities that affect any product shipping as part of Windows Server 2003, including Internet Explorer, for example.
Red Hat Enterprise Linux 4 AS (rhel4as). Stats for rhel4ws will include only the minimum required installation group packages, plus the package groups necessary to build basic server configurations: file server, print server, network server, and basic web server. X-Graphics, Gnome, Firefox, OpenOffice, Sound-n-Video and other optional packages are explicitly excluded from analysis in order to grant Red Hat the benefit of the doubt for its modularity. MySQL is also excluded since Windows Server does not ship with SQL Server included.
Sun Solaris 10. Stats for Solaris 10 include vulnerabilities that affect Solaris 10 as confirmed by a Sun Security Advisory.

The monthly scorecards are limited in scope by intent to give a snapshot view of recent (3 month) and year-to-date information. Deeper analysis for different and longer time periods can give further insight into OS vulnerability comparisons.