| Written by Beth Breier and Dennis Sutton,
|
 
Introduction
On February 14, 2005, while most folks were celebrating Valentines Day and most businesses were operating as usual, the City of Tallahassee was beginning to experience network problems that would bring the City to its knees. At first, the City’s computer experts thought that network equipment was failing, but soon the realization hit that the City’s network was infected by a computer virus. When the virus was detected, auditors from the Office of the City Auditor got involved by attending key meetings and observing virus eradication activities. Then, after the network was restored, audit staff interviewed key departmental staff, reviewed related virus and security documentation, examined payroll information, and calculated estimated lost productivity. Lastly, the auditors produced an Inquiry Report for the city commission and executive management that answered the following five questions:
1. How was the City infected with the virus?
2. What were the impacts on the City (i.e., financial, customer service, data integrity)?
3. Was the infection preventable?
4. How was the virus detected, identified, and eradicated?
5. What issues were identified and what lessons were learned from this experience?
In this article, the authors will share the pain we experienced in eradicating the virus and implementing new virus prevention techniques, describe the lessons learned by the City, and describe the role of the Office of the City Auditor.
Background
The City has developed a computer network to support business operations conducted by approximately 3,000 full-time employees that consists of approximately 2,400 computers, 56 servers, related network infrastructure equipment (i.e., switches, routers, and hubs), and numerous mobile computing devices (i.e., laptops, personal digital assistants). The network is maintained by Distributed Network Services, a section within the City’s Information Systems Services (ISS) Department.
Computer viruses, in general terms, are computer programs that take advantage of weaknesses in computer software in order to perform malicious activities on computers. These viruses can spread to other computers either by human actions (e.g., opening an e-mail with the virus or opening a computer document with the virus), or without human assistance (e.g., the computer coding within the virus directs the virus to replicate and spread automatically).
The City was infected by “Agobot,” which is a type of virus that can replicate without human assistance. This type of virus is referred to as a “worm”. Worms can infect entire networks very quickly because they are self-replicating and self-propagating. They are particularly troubling because they “flood or clog” the network with so much computer activity that the network slows down or stops entirely.
Answers to the Five Questions Asked by Audit
1. How was the City infected with the virus?
Neither ISS nor the Office of the City Auditor was able to determine how or when the City was first infected with the virus. ISS management determined that the cost involved to identify the specific cause would far exceed any benefits that may be derived from obtaining that information.
What we do know is that the City was first infected with the “Win32.Agobot” virus, which is one of hundreds of variants of the “Agobot” virus strain. Internet virus sites noted that this particular virus source code has been widely distributed to various hacker groups, with each making minor changes (hence the hundreds of variants). However, over time the core functionality of the Agobot virus has remained consistent.
The Agobot strain that infected the City copied itself to the City’s network Microsoft Windows? operating system directory and added commands to run the virus program when the users started their computers. Agobot was able to: (1) scan the network looking for other computers to infect; (2) initiate a connection to specific Internet Relay Chat (IRC) websites; and (3) disable the anti-virus software on the computers. It is possible that once connected to that IRC, a hacker could connect to the infected computer and perform multiple functions, including:
• Find and change administrative passwords;
• Launch “denial of service” attacks;
• Retrieve detailed system information;
• Download and execute files from Internet sites (including other viruses);
• Start and stop processes (including security-related processes);
• Execute local files;
• Access or manipulate data; and
• Modify host files to disable or redirect antivirus software scanning and prevent updating.
Because the anti-virus software had been disabled, some computers were found during the cleaning process to be infected with as many as 200 different viruses. Agobot also opened the door for numerous other viruses and other malicious programs (i.e., spy-ware, pop-ups, etc.) to be installed without notice.
2. What were the impacts on the City (i.e., financial, customer service, data integrity)?
Just like other companies, City business processes have been developed such that departments rely heavily on the City’s network to conduct their day-to-day operations. Without these resources, most departments were unable to conduct business in their normal manner.
City operations were impacted in three main ways: a) the direct cost of the ISS staff work time (including overtime) to eradicate the virus; b) lost productivity due to the inaccessibility of computer network resources; and c) diminished customer service because of that inaccessibility.
3. Was the infection preventable?
Yes, the spread of the “Agobot” virus across the network appears to have been preventable if the operating system software updates had been installed timely.
ISS faces many challenges trying to keep the City’s computers and network operating systems current. Challenges include the complexity of the network, the potential impact of updates to critical applications, and the size and geographical dispersion of the computer network. These challenges are discussed briefly below.
Complexity of the City’s network - The City’s computer network is a complex assembly of many different pieces of hardware and software. Parts of the network that appear to be the same may be very different. For example, all computers in the City use a Microsoft Windows operating system. However, depending on when the computer was purchased it could be running any one of four different versions (Windows 2000, Windows NT, Windows 98, or Windows 95). Each version has different security weaknesses and must have different procedures performed to maintain and protect them. Microsoft, Inc., provides separate software updates to fix weaknesses for each operating system version as they are identified. For example, an update for Windows NT cannot be used to fix a Windows 2000 vulnerability.
Potential impact of updates to critical applications - The updates that are provided for each of the operating systems sometimes create conflicts with other applications that a City department may depend on for business operations. For example, an update to a Windows NT operating system may cause a conflict that will prohibit the City’s PeopleSoft Human Resources Management System (HRMS) software application from functioning properly. If the PeopleSoft HRMS application is not working properly, then department timekeepers would not be able to enter payroll information.
To keep a “bad” update from disrupting business operations, ISS management decided no updates would be installed without first testing how that update affected key City applications. At the time, that decision appeared to be prudent as an untested update could have caused disruptions to those key applications. Looking back, it proved to be a very costly decision reminding us of the old television commercial encouraging drivers to change their oil filters to prevent their engines from needing replacement. The commercial showed a mechanic tearing apart an engine saying, “you can pay me now or you can pay me later.”
Size and location of the City’s computer network - As noted earlier, there are approximately 2,400 personal computers (PCs) in the City’s network that are spread among numerous locations where City business is conducted. Those locations include, for example, the airport, police department, fire department stations, electric and water plants, various warehouses, and City Hall. There are 14 employees maintaining those 2,400 PCs; a 1 to 170 ratio. ISS management indicated that this section struggles to maintain that large number of computers.
4. How was the virus detected, identified, and eradicated?
On Monday, February 14, 2005, ISS noticed a performance problem throughout the City’s network. The City’s main application systems and network users experienced intermittent problems. Some users were affected more severely than others.
Early Tuesday morning, February 15, after the hardware initially believed to be the problem was replaced, ISS noted the network performance was getting worse. ISS network staff ran additional diagnostics and determined that the problems were due to two types of network activity: (1) excessive Internet requests (called “broadcasts”) to a limited number of suspicious websites; and (2) individual computers scanning the City network for other vulnerabilities that could be exploited. As a result, ISS declared the City network the victim of a virus infection. Network services were not stopped.
For the remaining Tuesday and all day Wednesday, ISS worked with vendors to develop a solution. ISS initially attempted to keep the network functioning properly. However, there were still periods of time when the network was unavailable. Consequently, the decision was made to completely shut down the network (but this did not occur until 6:00 p.m. on Thursday) and begin the process of eradicating the virus.
On Thursday, the cleaning process began. It took much longer than anticipated, anywhere from 2-6 hours on each machine. At the conclusion of the first two shifts, it was obvious that there were simply too many computers and not enough people to clean all the computers by Monday morning.
On Friday, ISS management began working with a vendor to find an alternative automated solution while staff and volunteers continued the manual process.
At 4:00 p.m., Saturday, with only 25% of the City’s computers cleaned and updated, a determination was made that an automated solution was feasible. ISS management shifted all resources to implement an automated solution that would clean the virus, update the operating system patches and virus signatures, and install an additional software utility to monitor network traffic which would automatically disconnect PCs exhibiting abnormal behavior from the network. With this utility monitoring network traffic, ISS felt confident that the network could be reactivated without allowing the virus to propagate. When the network was reactivated with the utility monitoring the network traffic, approximately 80 computers were identified across the City that were exhibiting at least one of the above mentioned abnormal behaviors. These computers were automatically disconnected from the network. With the network reactivated and the virus unable to spread, ISS was able to automate the manual process that was previously being utilized to clean and update the computers.
On Monday morning, February 21, the City’s network was available and major business processes had been reestablished. The automated process to clean and update the PCs was completed by Tuesday afternoon.
5. What issues were identified and what lessons were learned?
After the virus incident, ISS management developed a list of areas where improvements were needed, including:
• Increasing the number of ISS staff with the needed skills (due to budgetary constraints, this will most likely be done through cross-training);
• Improving communication to keep City departments informed of network status in ways other than email;
• Segmenting the network so that virus infections are isolated, thereby minimizing the spread of the virus;
• Updating operating system software in a timely manner;
• Automating virus scanning and eliminating the reliance on users to manually scan electronic media; and
• Increasing the availability of computer security training for users and ISS staff.
Additionally, during our discussions with representatives from City departments, we noted the need for more business continuity planning, i.e., plans for alternative means of conducting business without computers.
Summary
Unfortunately, computer virus infections are now a way of life. Your organization will most likely be infected at some time. What will make the difference is how you limit the infection from spreading and what alternative processes are put in place to continue business operations without the use of computers. Our City had a painful experience when Agobot infected our network, and we can only hope that we have learned enough to implement some measures to mitigate the risks associated with future virus infections. We hope your organization already has such measures in place!
Beth Breier, CPA, CISA, Audit Manager, and Dennis Sutton, CPA, CIA, work in the City of Tallahassee Office of the City Auditor. The full report, “Inquiry into the February 2005 Network Computer Virus,” can be found on their website at: http://www.ci.tallahassee.fl.us/citytlh/auditing/index.html (choose Audit Reports, Reports Issued in FY 2005, then report title).
|
Quarterly Articles