Introduction

Ask the victims of Code Red, SQL Slammer, or Nimda how effective their security measures were when their networks were compromised and you’ll instantly understand how vulnerable the computing infrastructure is, and how easily its vulnerabilities can be exploited. The growth in security vulnerabilities and supplier-issued patches is now flooding the enterprise with information that IT managers don’t dare ignore, forcing a significant diversion of resources in most enterprises.

According to IDC, 90 percent of businesses and government agencies suffered hacker attacks within the past year. Managed security services provider RipTech Inc. recently published the Internet Security Threat Report Volume 2 which noted that for the first 6 months in 2002, attack activity was up to a projected growth rate of 64%. The threat of a severe attack is very real to all levels and sizes of business and the recovery cost of such an attack is typically 50% more expensive than prevention costs.

Estimates of actual dollar costs for the disruption caused by Code Red, for example, vary, but are generally thought to be in excess of $2.6 billion ("Code Red's Costs and Hunt for Creator Mount," by Anne Saita, Information Security 2001). Approximately 359,000 computers in enterprises around the world were brought to their knees within just 14 hours of the worm’s release onto the Internet (The Spread of Code Red Worm: An analysis. By David Moore and Colleen Shannon, Cooperative Association for Internet Data Analysis, 2001). On January 25, 2003, the SQL Slammer worm caused between $950 million and $1.2 billion in lost productivity according to London-based Mi2g. During its rampage, 13,000 automated teller machines belonging to Bank of America were disabled, 5 of the 13 root servers of the Internet’s domain name service shut down, Internet availability around the world was reduced by 25 percent and South Korea was completely cut off due to equipment failures and excessive traffic loads ("SQL Slammer slows the Internet," by Iain Thomson, vnunet.com, 2003).

Despite a suspect track record on applying patches, enterprises worldwide already spend in excess of $2 billion annually to investigate, prioritize, test and deploy patches; according to a report by The Aberdeen Group and these costs will escalate with no end in sight as enterprises continue to move critical business assets into a connected environment. As systems and software get more complex, the number of security vulnerabilities and the corresponding volume of patches will increase exponentially, further taxing already overburdened IT staffs, draining budgets and diverting resources from core business purposes.

About 90 percent of today’s security breaches are preventable, according to analysts at Gartner, Inc., but known vulnerabilities continue to be successfully exploited because organizations fail to apply available patches or inadvertently misconfigure software. Microsoft released the patch that could have prevented the SQL Slammer disaster six months before that worm was unleashed, and had notified end users of its availability, yet many organizations did not heed the warning for one reason or another.

As high-visibility network security attacks like Code Red and the SQL Slammer worm have shown, traditional security technologies are limited in their capability to combat the effects of new and evolving attacks. Organizations require robust application security methods that prevent security attacks from compromising their network systems and critical applications.

To that end, this whitepaper will address how Primary Response, the application security solution from Sana Security, can help organizations prevent security breaches and improve the efficiency of the vulnerability patching process to reduce the costs associated with patching as well as the costs of failing to patch.

Preventing Exploits is the Real Objective

Enterprises continue to increase the number of critical application servers, the speed at which new services are deployed, and the rate at which servers and software are changed. Balancing risk, functionality, efficiency and budget can seem like an insurmountable task. A proactive approach to preventing security breaches is, at least in theory, simple – block all attacks before they cause harm. Internet-based attacks can be blocked by ensuring that every server and network element is impervious to attack. However, keeping those systems perfectly safe over time is virtually impossible; technologies change, software gets updated, administrators make mistakes, and hackers continue to develop and propagate new attacks at an ever increasing rate.

Recognizing this, some organizations take significant steps to employ compensating controls in addressing a potentially increased threat level associated with the publication of a new vulnerability. In certain cases, they may choose to restrict access to particular applications or even take them off-line entirely, depending on the level of risk and the value of information assets at risk, adopting a stronger security posture for the duration of the threat assessment. A critical requirement for information security systems is the ability to provide a straightforward mechanism for increasing the security footing in a manner that allows business systems to continue to operate normally.

The Gartner Group reported in 2002 that 75 percent of successful exploits targeted vulnerabilities at the application level. These are exploits that have successfully run the gauntlet established by firewalls, network intrusion detection systems and other devices focused on monitoring network traffic. Externally-oriented applications such as web, DNS and e-mail servers present a significant area of concern for enterprises attempting to reduce their security exposure, and the issue is at least as significant for databases and other off-the-shelf and custom applications.

Based on Sana Profile Technology, and developed as a result of extensive research into how the human immune system correctly identifies and repels invaders, Primary Response is deployed on servers and monitors and protects applications at the operating system level. Primary Response automatically builds a profile of a protected application’s normal application behavior based on the code paths of the running program, and then continually monitors those code paths for deviations from the norm. Because vulnerability exploits are not part of normal application behavior, Primary Response automatically detects and prevents these exploits as soon as they take applications down unexpected code paths, blocking bad behavior while otherwise leaving the application running and intact. Primary Response also provides organizations with a compensating control capability by being able to specify with fine granularity which application activities are blocked when an exploit is detected. The ability to configure a range of blocking responses on an application specific basis enables administrators to take steps to eliminate risk even in advance of the availability of a vendor-provided patch.

Gartner asserts that 90 percent of cyber attacks through 2005 will involve known vulnerabilities for which a patch or solution already exists. During that time, Gartner predicts, 20 percent of enterprises will suffer a serious Internet security incident and, as noted above, the clean-up costs will exceed the prevention costs by 50 percent. According to CERT, more than 73,000 security incidents occurred in the first three quarters of 2002. That figure represents a nearly 40 percent increase from about 52,000 incidents in 2001.

Even the most effective IT department can be challenged when faced with the limited window in which to assess their exposure to a new exploit, validate a patch provided by a software vendor and implement that patch. Recently the MS Blaster worm caused significant operational downtime and expense for many companies, despite the fact that a patch was made available at the same time the vulnerability was publicly identified. Blaster was effective because it came on the scene less than thirty days after the patch was made available and many companies could not react that rapidly, especially for an issue affecting thousands of systems. By preventing the exploits associated with unpatched application software, Primary Response offers IT departments the ability to take a rational, measured approach to vulnerability management, both eliminating risk and saving time and money.

The Patch Proliferation Headache

Security administrators don't have it easy. Between performing system updates, backing up servers, monitoring intrusion detection systems, and completing other tasks, they frequently have to operate in interrupt mode to apply software patches. In an economy that is forcing the IT organization to do more with less, attempting to use the current staff to manually address the patching issue does not scale. Patching presents security administrators, as well as business managers with a number of other issues. Many patches require systems to be rebooted and managers may be reluctant to make their key systems unavailable during the business day. In the extreme case, patches to fix one security issue may introduce another, more serious (and undocumented) vulnerability.

However, one of the greatest challenges that enterprises face in managing security vulnerabilities today is coping with the growing volume of patches, the accelerating rate at which they are released and the general unpredictability of their occurrence. The reasons for this are many: Software is getting more complex, which increases the likelihood of errors. Windows 2000, for example, has 40 million to 60 million lines of code. Software companies are under pressure to get products out the door more quickly, often at the expense of proper but time-consuming testing, resulting in buggy or vulnerable software and patches that often need additional repair after the customer installs them. In fact, empirical data suggests nearly 18 percent of patches were either revised or pulled altogether, and nearly 15 percent of the revised patches were revised again ("Timing the Application Security Patches for Optimal Uptime," WireX Communications, Inc. and Zero Knowledge Systems, Inc.). Software vendors seeking to limit their exposure to liability may compensate by issuing more patches. The uncertainty and disruptive effects of this situation make it impossible for IT departments to efficiently allocate resources.

So, should you patch now and risk a costly system failure caused by defective software, or patch later and risk a costly and damaging exploit of an exposed vulnerability? Should vulnerabilities be tracked and analyzed every day? Should resources be taken offline to deploy a patch every time there is any doubt about the extent of a vulnerability? Should testing be shortchanged in the name of expediency? Are there ways to adjust the organization’s security posture when the threat level changes? These are some of the questions every IT security professional struggles with.

Ironically, says an Aberdeen analyst, "The more time you spend on patch management, the less you focus on security." Caution is warranted. Because patching with this frequency is a relatively new phenomenon, the practice has raised as many security issues as it has solved. Software vendors are under tremendous pressure to generate fixes in the limited window between third-party discovery and public disclosure of a new vulnerability, and quick fixes are the bane of responsible information security practices.

Primary Response lays a foundation to effectively transform this exercise from a disorganized, reactive and costly fire drill into a planned and budgeted vulnerability management process that benefits from economies of scale. Because Primary Response protects application vulnerabilities against exploits, patching can be safely deferred and managed to make more efficient use of IT resources. Instead of constantly searching for patches and analyzing them, this task can be performed once a week or once a month. Instead of assigning personnel to patch and reboot each server several times a month, patching can be consolidated, validated by security personnel and deployed by the operations group on a monthly or quarterly basis, or as roll-ups and service packs are released. This process significantly reduces downtime and lowers overhead while maintaining an effective level of security.

The Patching Money Pit

Ultimately the risk associated with a compromise of information systems and the burden imposed by the requirement to patch vulnerabilities must stand up to a reasonable cost-benefit analysis that relates to a business bottom line. Organizations depend on information technology resources and require them to be trustworthy and reliable. The operational cost of a day's downtime can be calculated, but what if the information with which others entrust your organization is compromised publicly? A breach of corporate security and the resulting loss of credibility (with customers, partners, and governments) can put the very core of a business at risk.

Some enterprises, in an effort to avoid the stigma of becoming the next vulnerability poster child, in addition to incurring the costs associated with a successful attack, have reacted with an extreme sense of urgency. These enterprises invest considerable resources to track, analyze, test and deploy patches on a daily basis. One Sana Security customer, a global financial services organization with a particularly high number of servers, reports spending $1.5 million each time Microsoft issues a security patch, and maintains a full-time staff of 21 people in the U.S. alone dedicated to security patching. Unfortunately, time pressures can cause some patches to be deployed before they are thoroughly tested, increasing the risk of a costly system failure or other disruption.

The cumulative cost for deploying patches can be prohibitive. But, as is well known, deployment is only part of the story. The Aberdeen Group estimates that more than half the $2 billion annual total cost of patching is spent on researching, evaluating, and testing patches, and there are additional costs associated with downtime in terms of business continuity, productivity, and goodwill.

For other enterprises, the cost of patching is simply too great, both in terms of real dollars spent and in the ripple effect it has on other aspects of the business. Recent research ("Remote Net Probe Reveals Sloppy Software Upkeep," New Scientist, 20 November 2002) indicates that 7 out of 10 systems administrators did not fix a major software bug when it was first announced, and 3 out of 10 still failed to apply the patch even after the vulnerability had been exploited. Some enterprises don’t take the risks seriously until it affects them, and view patching as money spent to prevent something that might only potentially occur.

In fact, the cost associated with downtime is a major issue in vulnerability management. If a server must be rebooted every time a patch is installed, it becomes impossible to get beyond “two nines” availability. If it were possible to defer all the individual patches and instead apply them all at once, in batches, roll-ups or service packs, that figure could be improved to between four and five nines uptime, or the difference between days of downtime per year and about an hour of downtime per year.

The Cost of Downtime

In the 2001 Cost of Downtime Survey put forth by Contingency Planning Research (CPR), forty-six percent of participating companies said that each hour of downtime would cost their companies up to $50,000; 28 percent said each hour would cost between $51,000 and $250,000, 18 percent said each hour would cost between $251,000 and $1 million and 8 percent said it would cost their companies more than $1 million per hour (2001 Cost of Downtime Survey Results, 2001). In additional CPR research, also reported by InternetWeek (InternetWeek, April 2000), the following data was gathered on the cost of one hour of downtime in the following industry sectors:

Primary Response and the Bottom Line

To illustrate how effectively Primary Response can reduce vulnerability management costs, consider the example of a two-tiered architecture using a Windows 2000 server with IIS 5.0 on the front end and another Windows 2000 server running SQL Server 2000 on the back end. If we assume it takes 10 minutes to reboot a server after patching, and if each of these vulnerabilities were addressed individually, that would mean a downtime of approximately 15 hours annually for the server running IIS 5.0 and around another 13 hours of downtime for the server running SQL Server 2000, for a total of close to 28 hours of downtime per year. And, these downtimes often cannot be scheduled for the most convenient (i.e., least costly) times. Assuming a conservative average downtime cost of $25,000 per hour,, that’s a total downtime cost of $700,000 per year and an uptime availability of just over two nines.

Now consider the scenario in which Primary Response is protecting the applications, allowing patching to be accomplished on a scheduled basis as part of the deployment of two service packs per year, instead of ad hoc patching. This reduces the downtimes to 0.33 hour per year for the IIS server and 0.67 hour per year for SQL Server 2000, or a total downtime of just 1 hour per year at a cost of $25,000. That’s a savings of $675,000 per year and an uptime availability of almost five nines. Additional savings may be realized because it is now possible to schedule downtimes during periods of least usage.

Table 1: Illustrates downtime hours and annual costs assuming 10 minutes per patch (Numbers have been rounded for illustration purposes)

As noted above, this approach reduces the likelihood of system failures and their associated costs by affording IT departments the opportunity to adequately evaluate and test patches before deploying them. Primary Response can offer similar benefits with automated patching systems. Although more efficient than manual patching, a server still must be rebooted each time a patch is applied. Setting up each patching job and testing it can take anywhere from ten minutes to an hour, and so by consolidating patches in a roll-up, additional savings can be realized here, too.

Conclusion

Implementing sound vulnerability management practices costs time and money. And, organizations will always need to weigh these costs against the potential consequences of intrusions. Companies lost millions last year to attacks that exploited known vulnerabilities. Code Red and Nimda cost companies worldwide an estimated $2 billion in damaged computing resources and downtime, according to Computer Economics. More than 600,000 servers were infected by Code Red, although the vulnerabilities it exploited had been published and a patch was available six months before the worm was released. The patch for Nimda was available up to a year before it made its debut, but the worm still infected 160,000 hosts at its peak. While actual numbers for the damage associated with SQL Slammer and MS Blaster may not be know for some time, it is clear that the impact is real and significant. Effective vulnerability management could have saved much of the cost of these and other worms and electronic exploits, in addition to saving much of the cost associated with the unsuccessful attempts to patch these vulnerabilities.

Primary Response mitigates the risks created by application vulnerabilities and reduces the costs associated with “reactive” security patching by transforming time and resource-consuming patching activities into a proactive vulnerability management procedure with greater predictability, fewer disruptions, less downtime, and more efficient use of IT resources. Primary Response achieves this by improving the security posture of application and information assets, and facilitating a scheduled process of updates wherein the current uncertain state of information system patching gives way to predictability and planning, thereby reducing costs. By protecting applications from exploits of both known and unknown vulnerabilities before, during, and after the update process, Primary Response enables organizations to significantly reduce their vulnerability exposure, and, as a matter of course, offers administrators the freedom to patch on their own terms. Significant cost savings can be realized through greater uptime and operational efficiencies when compared with the cost of the ad hoc methods routinely employed today, making Primary Response a critical asset in a proactive information security arsenal.

Sana Security
2121 S. El Camino Real, Suite 700
San Mateo , CA 94403
Tel: 650.292.7100
Fax: 650.292.7110

www.sanasecurity.com

About Sana Security

Sana Security creates award-winning, autonomous intrusion prevention software that is aware of environment change, adaptive to new threats and active in preventing attacks before they do harm across mission-critical computer systems. Sana Security's Primary Response® and Attack Shield brand products improve security and business continuity for large enterprises and government organizations on servers, personal computers, and industrial systems, while driving the emerging standards for simple, secure computing in an Internet-connected world. Sana Security is headquartered in San Mateo, Calif. ( Silicon Valley), with offices in global business and technology centers and can be reached at www.sanasecurity.com or 866-900-SANA.